# PaddleOCR-VL
Source: https://docs.sglang.io/cookbook/autoregressive/Baidu/PaddleOCR-VL

Deploy PaddleOCR-VL 1.6 / 1.5 / 0.9B with SGLang — Baidu's 0.9B NaViT + ERNIE-4.5 document-parsing VLM for OCR, tables, formulas and charts in 109 languages, on a single H100, H200 or B200.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick a release and your hardware to generate the launch command. The model is 0.9B and single-GPU, so there is one serving recipe per platform; the axis that actually moves cost is **Page Resolution**, which caps how many image tokens one page is worth.

## Playground

Use the Playground to layer tensor parallelism on top of the selected deployment cell. At this size TP is a latency knob, not a capacity one — the weights fit on one GPU.

## 1. Model Introduction

**PaddleOCR-VL** is Baidu's compact document-parsing vision-language model: a NaViT-style dynamic-resolution SigLIP vision encoder feeding an **ERNIE-4.5-0.3B** language backbone, 0.9B parameters in total, released under **Apache 2.0**. It recognizes text, tables, formulas, charts and seals across **109 languages**, and is small enough that a single GPU serves it comfortably.

SGLang serves **this model**, which is the recognition stage of PaddleOCR's document-parsing pipeline — not the pipeline itself. Layout and region detection, cropping, reading order and the assembly of markdown or JSON stay in PaddleOCR, which calls a model endpoint once per region. That split is why the model is prompt-conditioned rather than instruction-following: the caller decides whether a crop is a table, a formula or a chart (§3.1), and SGLang is the backend that answers.

All three releases share an identical `config.json` (same `PaddleOCRVLForConditionalGeneration` architecture, same tower and backbone dimensions), so one SGLang recipe serves every variant and only the model path changes.

<table>
  <thead>
    <tr>
      <th>Variant</th>
      <th>Total params</th>
      <th>Use</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[PaddleOCR-VL-1.6](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6)**</td>
      <td>0.9B</td>
      <td>Latest. Best tables, Chinese characters and seals; drop-in for 1.5.</td>
    </tr>

    <tr>
      <td>**[PaddleOCR-VL-1.5](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5)**</td>
      <td>0.9B</td>
      <td>Previous generation; pin it if you have calibrated against its output.</td>
    </tr>

    <tr>
      <td>**[PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL)**</td>
      <td>0.9B</td>
      <td>The original 0.9B release.</td>
    </tr>
  </tbody>
</table>

**Recommended generation:** greedy decoding (`temperature=0`) with a per-page `max_tokens` budget — the model card uses 512 for a single region and the reference server allows more for a full page. These are informational; do not hardcode them in library code.

**Resources:** [Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) · [PaddleOCR on GitHub](https://github.com/PaddlePaddle/PaddleOCR)

## 2. Configuration Tips

* **Page resolution is the main cost knob.** The vision tower and the prefill both scale with the patch count of a page. `max_pixels` is expressed in 28x28 units (patch size 14 with a 2x2 merge), so `max_pixels / 784` is the image-token budget per page. The checkpoint's own default is 1280 tokens; the **Page Resolution** selector in the Deploy panel emits the corresponding `--mm-process-config` value. Lower it for clean born-digital PDFs, raise it for dense scans and small print.
* **Prompt selects the task.** PaddleOCR-VL is prompt-conditioned rather than instruction-following — use the exact task strings in §3.1. A free-form question will not behave like a chat model.
* **Leave `--trust-remote-code` off.** The checkpoints ship their own `configuration_paddleocr_vl.py` / `processing_paddleocr_vl.py`, but `transformers` 5.12 supports `paddleocr_vl` natively — and the bundled remote image processor is the slower of the two implementations (measured 87.4 ms vs 39.1 ms per 1080p page). Passing the flag pins SGLang to the remote copy. Serving without it produced byte-identical OCR output on every page we checked and about 5% more requests per second at 32-way concurrency.
* **Preprocessing is parallelized for you.** A full-resolution page costs tens of milliseconds of CPU to resize, normalize and patchify, which caps throughput long before the GPU saturates, so this model runs the image processor across several workers by default. `--mm-processor-worker-num` overrides the count; raising it past the default did not help in our measurements.
* **Keep the radix cache on for repeated pages.** Unlike whole-document batch OCR over unique scans, a workload that re-asks about the same page (different task prompts on one image) reuses the image prefix. Add `--disable-radix-cache` only if every request carries a different page.
* **The saturated-throughput flags earn their place.** A page is \~2700 tokens, so the default 8192-token prefill budget packs only three of them into a forward. Raising it to 16384 and letting decode ride along in the same batch (`--enable-mixed-chunk`, `--num-continuous-decode-steps 2`) measured +11% requests per second at 32-way concurrency and cut queued TTFT by 23%, with single-stream latency unchanged. Measured on an H200; on a smaller card lower `--chunked-prefill-size` until it fits.
* **Prefill CUDA graph is on for this model.** SGLang normally switches the breakable prefill graph off for every multimodal architecture; PaddleOCR-VL is allowlisted back in, which is worth 16.1 ms → 11.5 ms of single-stream TTFT on text-only prompts. Image-carrying batches are rejected at graph replay and run eager, so this helps mixed and text traffic, not pure page parsing. No flag needed.
* **Tensor parallelism is optional.** The weights are under 2 GB in BF16; TP>1 only shortens the vision-encoder and prefill critical path, at the cost of a collective per layer. Measure before adopting it.
* **Context length.** The backbone advertises 131072 positions, but a parsed page rarely needs more than a few thousand tokens. The recipe pins `--context-length 16384` so the KV pool stays small and concurrency stays high; raise it only if you batch many pages into one request.

### Measured on one H200

One 1080p page (\~2700 image tokens) in, 128 tokens out, prefix cache disabled, median TTFT:

<table>
  <thead>
    <tr>
      <th>Configuration</th>
      <th>TTFT, 1 stream</th>
      <th>req/s at 32 concurrent</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>With `--trust-remote-code` (remote image processor)</td>
      <td>219 ms</td>
      <td>10.9</td>
    </tr>

    <tr>
      <td>Recipe above (native image processor)</td>
      <td>**114 ms**</td>
      <td>**11.3**</td>
    </tr>
  </tbody>
</table>

Throughput at saturation is bound by the vision tower, which runs full attention over
every patch of the page — so the **Page Resolution** selector is the lever that moves it,
not tensor parallelism.

## 3. Advanced Usage

### 3.1 Task prompts

PaddleOCR-VL exposes its capabilities through a small set of fixed prompts. Send the prompt as the text part and the page as the image part of the same user turn.

<table>
  <thead>
    <tr>
      <th>Prompt</th>
      <th>Task</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`OCR:`</td>
      <td>Plain text recognition.</td>
    </tr>

    <tr>
      <td>`Table Recognition:`</td>
      <td>Table structure and cell contents.</td>
    </tr>

    <tr>
      <td>`Formula Recognition:`</td>
      <td>Mathematical expressions.</td>
    </tr>

    <tr>
      <td>`Chart Recognition:`</td>
      <td>Chart contents.</td>
    </tr>

    <tr>
      <td>`Spotting:`</td>
      <td>Text with locations. Benefits from the high-detail resolution setting.</td>
    </tr>

    <tr>
      <td>`Seal Recognition:`</td>
      <td>Seals and stamps (1.6).</td>
    </tr>
  </tbody>
</table>

Structured tasks answer in the model's own markup, not HTML: `Table Recognition:` returns OTSL-style cell tokens (`<fcel>` per cell, `<nl>` per row), so a caller that wants HTML or markdown converts them itself.

> **Table Recognition Output**
>
> ```text Output theme={null}
> <fcel>Methods<fcel>R<fcel>P<fcel>F<fcel>FPS<nl><fcel>SegLink [26]<fcel>70.0<fcel>86.0<fcel>77.0<fcel>8.9<nl><fcel>PixelLink [4]<fcel>73.2<fcel>83.0<fcel>77.8<fcel>-<nl><fcel>TextSnake [18]<fcel>73.9<fcel>83.2<fcel>78.3<fcel>1.1<nl>
> ... (one <fcel> per cell, one <nl> per row, to the end of the table)
> ```

> **OCR Request (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="PaddlePaddle/PaddleOCR-VL-1.6",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {"type": "text", "text": "OCR:"},
>                 {
>                     "type": "image_url",
>                     "image_url": {
>                         "url": "https://raw.githubusercontent.com/PaddlePaddle/PaddleOCR/release/2.9/doc/imgs_en/img_12.jpg"
>                     },
>                 },
>             ],
>         }
>     ],
>     max_tokens=2048,
> )
>
> print(response.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> ACKNOWLEDGEMENTS
> We would like to thank all the designers and contributors who have been involved in the
> production of this book; their contributions have been indispensable to its creation. We would
> also like to express our gratitude to all the producers for their invaluable opinions and
> assistance throughout this project. And to the many others whose names are not credited but
> have made specific input in this book, we thank you for your continuous support.
> ```

### 3.2 Parsing a multi-page document

The model parses one page per request. Render each page to an image, then fan the pages out concurrently — SGLang batches the vision encoders of in-flight requests into a single forward, so concurrency is what keeps the GPU busy on a model this small.

> **Concurrent Page Parsing (Python)**
>
> ```python Example theme={null}
> import base64
> from concurrent.futures import ThreadPoolExecutor
>
> import pymupdf
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
>
> def render(page, dpi=200):
>     pixmap = page.get_pixmap(dpi=dpi)
>     return base64.b64encode(pixmap.tobytes("png")).decode("ascii")
>
>
> def parse(page_png_b64):
>     response = client.chat.completions.create(
>         model="PaddlePaddle/PaddleOCR-VL-1.6",
>         messages=[
>             {
>                 "role": "user",
>                 "content": [
>                     {"type": "text", "text": "OCR:"},
>                     {
>                         "type": "image_url",
>                         "image_url": {
>                             "url": f"data:image/png;base64,{page_png_b64}"
>                         },
>                     },
>                 ],
>             }
>         ],
>         max_tokens=2048,
>     )
>     return response.choices[0].message.content
>
>
> document = pymupdf.open("your_document.pdf")
> pages = [render(page) for page in document]
>
> with ThreadPoolExecutor(max_workers=16) as pool:
>     for index, text in enumerate(pool.map(parse, pages)):
>         print(f"--- page {index + 1} ---")
>         print(text)
> ```

> **Example Output**
>
> ```text Output theme={null}
> --- page 1 ---
> (a) Total-Text
> (b) Total-Text
> (c) CTW1500
> (d) CTW1500
>
> Figure 8. Visual experimental results. The blue contours are boundary proposals, and the
> green contours are final detection boundaries.
> Table 6. Experimental results on CTW-1500.
> Methods
> Ext
> R
> P
> F
> FPS
> TextSnake [18]
> Syn
> 85.3
> 67.9
> 75.6
> -
> ... (page continues)
> --- page 2 ---
> ... (one block per page, in page order)
> ```


# Unlimited-OCR
Source: https://docs.sglang.io/cookbook/autoregressive/Baidu/Unlimited-OCR

Deploy Baidu Unlimited-OCR with SGLang for long document OCR using prefill-aware sliding-window attention.

## Deployment

> **Install SGLang**
>
> Unlimited-OCR support is in [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186). Until that PR is included in a tagged SGLang release, install from a build that contains the PR.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install -U uv
> uv venv --python 3.12 && source .venv/bin/activate
>
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> git fetch origin pull/29186/head && git checkout FETCH_HEAD
> uv pip install --prerelease=allow -e python
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware to generate the launch command. The recipe uses FlashAttention-3 with `--page-size 1`, which is required by the current prefill-aware sliding-window attention path. It also disables radix cache by default, which is the better fit for batch OCR workloads where each request usually contains a different image.

## Playground

Use the Playground to adjust tensor parallelism on top of the selected deployment cell.

## 1. Model Introduction

[Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's multimodal OCR model for document parsing. It uses a sliding-window language backbone, but SGLang serves it with a prefill-aware sliding-window path so image and prompt tokens remain visible during long decode.

The SGLang integration loads the standalone Unlimited-OCR architecture with SAM and CLIP vision encoders plus a DeepSeek-style language backbone. It supports OpenAI-compatible image requests and model-specific image processing options through `images_config`.

**Resources:** [Hugging Face](https://huggingface.co/baidu/Unlimited-OCR) · [SGLang PR #29186](https://github.com/sgl-project/sglang/pull/29186)

## 2. Configuration Tips

* **Attention backend**: use `--attention-backend fa3 --page-size 1`. The prefill-aware SWA page table is built with token-level locations, so page size 1 is required.
* **Radix cache**: keep `--disable-radix-cache` for batch OCR over different documents. If your workload repeatedly asks about the same image and prompt, remove this flag to allow prefix reuse through `PureSWARadixCache`.
* **Long OCR generations**: keep the default prefill-aware SWA path enabled. It retains prompt and image KV while still applying a sliding window to generated text.
* **Custom logit processor**: keep `--enable-custom-logit-processor` in the launch command.
* **Image modes**: pass `images_config.image_mode` per request. Supported modes are `tiny`, `small`, `base`, `large`, and `gundam`. Multiple images are supported only for `tiny`, `small`, and `base`.
* **Default image mode**: when `images_config.image_mode` is omitted, SGLang uses `gundam`.

## 3. Advanced Usage

### 3.1 OCR request

> **OCR Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="baidu/Unlimited-OCR",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {"type": "text", "text": "document parsing."},
>                 {
>                     "type": "image_url",
>                     "image_url": {
>                         "url": "https://example.com/your_document.png"
>                     },
>                 },
>             ],
>         }
>     ],
>     max_tokens=2048,
>     temperature=0,
>     extra_body={"images_config": {"image_mode": "gundam"}},
> )
>
> print(response.choices[0].message.content)
> ```

### 3.2 Choosing an image mode

Use lower modes to reduce prefill cost for simple images, and use `gundam` for high-detail document parsing.

<table>
  <thead>
    <tr>
      <th>Mode</th>
      <th>Use</th>
      <th>Multiple images</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`tiny`</td>
      <td>Lowest prefill cost.</td>
      <td>Yes</td>
    </tr>

    <tr>
      <td>`small`</td>
      <td>Lightweight OCR requests.</td>
      <td>Yes</td>
    </tr>

    <tr>
      <td>`base`</td>
      <td>Balanced quality and cost.</td>
      <td>Yes</td>
    </tr>

    <tr>
      <td>`large`</td>
      <td>Higher resolution single-image OCR.</td>
      <td>No</td>
    </tr>

    <tr>
      <td>`gundam`</td>
      <td>Default high-detail document parsing mode.</td>
      <td>No</td>
    </tr>
  </tbody>
</table>


# Clef
Source: https://docs.sglang.io/cookbook/autoregressive/Cloudflare/clef

Deploy Cloudflare's Clef 27B and Clef Flash 9B decision models with SGLang. Select either variant for BF16 text and image decisions on one NVIDIA B200, B300, or H200.

## Deployment

> **Install SGLang**
>
> Clef support merged after v0.5.21 in [#42721](https://github.com/sgl-project/sglang/pull/42721). Use the pinned nightly wheel or Docker image below. See the [installation guide](/docs/get-started/install) for other methods.
>
> **Python (pip / uv)**
>
> This command selects the wheel from [nightly build 37ae292e6f](https://github.com/sgl-project/sglang/actions/runs/37897861964), which includes Clef support. It targets Python 3.10 on Linux x86\_64 with glibc 2.39 or newer.
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv venv --python 3.10
> source .venv/bin/activate
> uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ "sglang==0.5.22.dev20261009+g37ae292e6f"
> ```
>
> Run the command from the wizard’s Python tab in this environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-clef
> ```
>
> The [Clef image build](https://github.com/sgl-project/sglang/actions/runs/37893990016) produces this image from the #42721 merge commit. Run the command from the wizard’s Docker tab. Both checkpoints are public; no Hugging Face token is required.

Choose your GPU, then Clef or Clef Flash. The wizard updates the model in the launch command and sample request. Both models run on one GPU in BF16 with TP=1 using the default settings; the commands need only the model path, host, and port.

## 1. Model introduction

Cloudflare post-trained Clef from Qwen3.8-27B and Clef Flash from Qwen3.5-9B, then released both under the Apache-2.0 license. Each model keeps its backbone and vision encoder and adds a joint schema head. The head reads the backbone’s final hidden states, routes state evidence to each option, and lets questions attend to one another. It returns one logit per allowed option in a single prefill, with no text generation.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variant</th>
      <th>Checkpoint</th>
      <th>Backbone</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Clef (27B)</td>
      <td>[Cloudflare/clef](https://huggingface.co/Cloudflare/clef)</td>
      <td>Qwen3.8-27B</td>
    </tr>

    <tr>
      <td>Clef Flash (9B)</td>
      <td>[Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash)</td>
      <td>Qwen3.5-9B</td>
    </tr>
  </tbody>
</table>

Pass a text string, JSON object, or JSON array as the state, with optional images. `/v1/systemone` returns one answer per question: a `choice` over allowed options, a `noul` probability of true, or a `score` over ordered levels.

## 2. Configuration tips

* **Route.** Send requests to `/v1/systemone`, or set your System One client’s base URL to the server. This includes TypeSafe SDK clients. See the [API reference](/docs/supported-models/decision_models#system-one-compatible-api) for request and response fields. Clef refuses `/v1/decisions`, chat, and generation requests.
* **One GPU, one prefill per request.** Keep TP=1 and PP=1. All questions share one prompt, so the model reads the state once. SGLang detects `joint_head_config.json`, loads `joint_head.safetensors`, and enables embedding mode. It disables radix caching, chunked prefill, and CUDA graphs. No reasoning parser, tool-call parser, or chat-template override is needed.
* **Answers.** `probabilities` are the softmax of the head’s logits for each question. `confidence` uses the System One adapter formulas. It can differ from the checkpoint’s reference `systemone()`, which reports the top option’s probability as confidence. Answers omit `x_label_mass` because the head scores only the allowed options.
* **Long states.** Prompts, including images, are capped at 16,384 tokens, the reference `encode_record` function’s default `max_length`. The effective limit can be lower, depending on the server context, KV token pool, and `--max-prefill-tokens`. The server truncates the state to leave room for the questions and images. If those do not fit even without the state, it rejects the request.
* **Images.** Send each image in `images` as a data URL, such as `data:image/jpeg;base64,...`, or an HTTP(S) URL. Keep the data-URL prefix for JPEGs: bare base64 starts with `/`, which the loader treats as a file path, causing the request to fail. Images precede the state.

## 3. Advanced usage

### 3.1 Text decisions

Set `MODEL` to the checkpoint you selected in the wizard. The request works with either variant.

> **Text decision example (Python)**
>
> ```python Example theme={null}
> import requests
>
> MODEL = "Cloudflare/clef-flash"  # Or "Cloudflare/clef".
>
> response = requests.post(
>     "http://localhost:30000/v1/systemone",
>     json={
>         "model": MODEL,
>         "state": "My Stripe integration keeps failing. I am losing sales. Please help ASAP.",
>         "questions": {
>             "routing": {
>                 "type": "choice",
>                 "instructions": "Which team should handle this request?",
>                 "criteria": {
>                     "billing": "Charges and refunds",
>                     "technical_support": "Integration errors",
>                     "sales": "Questions about buying a product",
>                 },
>             },
>             "urgency": {"type": "noul", "instructions": "Does this message express urgency?"},
>             "frustration": {
>                 "type": "score",
>                 "instructions": "How frustrated is the customer?",
>                 "criteria": ["Calm", "Frustrated but civil", "Very angry"],
>             },
>         },
>     },
>     timeout=60,
> )
> response.raise_for_status()
> answers = response.json()["answers"]
> print("Routing:", answers["routing"]["choice"])
> print("Urgency:", answers["urgency"]["noul"])
> print("Frustration:", answers["frustration"]["score"])
> print("Usage:", response.json()["usage"])
> ```

> **Example output**
>
> Captured with Clef Flash on one H200. Probabilities can vary with the model variant, hardware, and numerical backend.
>
> ```text Output theme={null}
> Routing: technical_support
> Urgency: 0.9492946352021694
> Frustration: 1.7297646970642042
> Usage: {'input_tokens': 334, 'output_tokens': 0}
> ```

## 4. GSM8K accuracy

[Decision Index 0.2.1](https://github.com/apolinario/decision-index/blob/9eb2dbe2a358004c8782c66e40a83ac07b953fec/docs/suite.md) evaluates all 1,319 GSM8K test problems with both four-choice and ten-choice questions, for 2,638 requests per configuration. The table reports accuracy across both formats through `/v1/systemone`.

| Model | [Reported](https://huggingface.co/Cloudflare/clef#results) | H200 | B200 | B300 |
| - | -: | -: | -: | -: |
| Clef | 80.8% | 80.71% | 80.59% | 80.59% |
| Clef Flash | 67.3% | 67.32% | 67.36% | 67.36% |

All runs use BF16 and TP=1 on one GPU.


# Ornith-1.0
Source: https://docs.sglang.io/cookbook/autoregressive/DeepReinforce/Ornith-1.0

Deploy DeepReinforce Ornith-1.0 with SGLang - a self-improving agentic-coding model family with 397B, 35B, and 9B checkpoints plus FP8 and GGUF variants.

## Deployment

> **Install SGLang**
>
> Ornith-1.0 model cards recommend SGLang `>=0.5.9`. The Deploy panel below emits the base serve command; the reasoning and tool-call parsers from the model-card quickstarts (`--reasoning-parser qwen3` for `<think>...</think>` traces, `--tool-call-parser qwen3_coder` for Qwen-style XML tool calls) are added on top via the [Playground](#playground).
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow "sglang>=0.5.9"
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick an Ornith checkpoint to generate the launch command. The non-FP8 397B recipe requires an H200 single node in this matrix. The 397B-FP8 recipe is available on H100 and H200 with TP=8; H100 also supports the 35B and 9B checkpoints. The 35B recipes use tensor parallelism 2 in this matrix. The 9B checkpoint is dense and serves on a single GPU by default; the command panel makes that default explicit with `--tp 1`.

## Playground

The Playground layers SGLang features on top of whichever cell the Deploy panel is showing — only your overrides change, and any change flips the badge to **Not Verified** until the new configuration is run end-to-end.

For Ornith-1.0 the knobs are the reasoning and tool-call parsers:

* **Reasoning Parser** appends `--reasoning-parser qwen3`. Ornith emits `<think>...</think>` traces; with this on, SGLang surfaces them as `message.reasoning_content` instead of leaving the tags inline in `content`.
* **Tool Call Parser** appends `--tool-call-parser qwen3_coder`, so Qwen-style XML tool calls are returned as OpenAI-compatible `tool_calls`.

## 1. Model Introduction

[Ornith-1.0](https://huggingface.co/collections/deepreinforce-ai/ornith-10) is DeepReinforce's self-improving open-source model family for agentic coding. The model cards describe the family as post-trained on top of Gemma 4 and Qwen 3.5, and the collection currently includes 397B, 35B, and 9B repos plus FP8 and GGUF variants. The model cards report results on Terminal-Bench 2.1, SWE-Bench, NL2Repo, ClawEval, and SWE Atlas benchmarks.

**Key Features:**

* **Agentic coding specialization**: the model cards describe Ornith-1.0 as specialized for agentic coding and report coding-agent benchmark results.
* **Self-improving training**: the model cards state that Ornith-1.0 uses reinforcement learning to optimize both solution rollouts and the scaffold that drives those rollouts.
* **Reasoning model behavior**: assistant responses begin with a `<think>...</think>` reasoning block before the final answer; enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) to split it into `reasoning_content`.
* **Tool calling**: emits Qwen-style XML tool calls; enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground).
* **Long context**: model-card recipes use `--context-length 262144`.
* **MIT license**: the Hugging Face repos are released under MIT.

**Available Models:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Format</th>
      <th>Deploy Panel</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[deepreinforce-ai/Ornith-1.0-397B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B)</td>
      <td>BF16</td>
      <td>H200 only</td>
      <td>Flagship 397B MoE checkpoint; model-card baseline uses TP=8 on an H200 single node.</td>
    </tr>

    <tr>
      <td>[deepreinforce-ai/Ornith-1.0-397B-FP8](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B-FP8)</td>
      <td>FP8</td>
      <td>H100 / H200</td>
      <td>FP8 repo in the collection; the deploy command uses this repo id with TP=8.</td>
    </tr>

    <tr>
      <td>[deepreinforce-ai/Ornith-1.0-35B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B)</td>
      <td>BF16</td>
      <td>H100 / H200</td>
      <td>35B MoE checkpoint; the deploy command uses TP=2.</td>
    </tr>

    <tr>
      <td>[deepreinforce-ai/Ornith-1.0-35B-FP8](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-FP8)</td>
      <td>FP8</td>
      <td>H100 / H200</td>
      <td>FP8 repo in the collection; the deploy command uses this repo id with TP=2.</td>
    </tr>

    <tr>
      <td>[deepreinforce-ai/Ornith-1.0-9B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B)</td>
      <td>BF16</td>
      <td>H100 / H200</td>
      <td>Dense 9B checkpoint; the model card describes it as designed for efficient single-GPU deployment.</td>
    </tr>

    <tr>
      <td>[deepreinforce-ai/Ornith-1.0-35B-GGUF](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF)</td>
      <td>GGUF</td>
      <td>No</td>
      <td>Listed for completeness; GGUF targets llama.cpp-style local inference, not the SGLang server recipe here.</td>
    </tr>

    <tr>
      <td>[deepreinforce-ai/Ornith-1.0-9B-GGUF](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B-GGUF)</td>
      <td>GGUF</td>
      <td>No</td>
      <td>Listed for completeness; the model card shows llama.cpp and Ollama examples for the GGUF build.</td>
    </tr>
  </tbody>
</table>

**License:** [MIT](https://huggingface.co/deepreinforce-ai/Ornith-1.0-397B/blob/main/LICENSE)

**Resources:** [Hugging Face collection](https://huggingface.co/collections/deepreinforce-ai/ornith-10) · [Ornith blog](https://deep-reinforce.com/ornith_1_0.html)

## 2. Configuration Tips

* **Reasoning parser**: Ornith responses include `<think>...</think>`. Enable the `--reasoning-parser qwen3` toggle in the [Playground](#playground) so OpenAI-compatible responses expose the reasoning trace as `message.reasoning_content`.
* **Tool-call parser**: enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) so `<tool_call>` blocks are returned as OpenAI-compatible tool calls.
* **Context length**: the model-card SGLang recipes use `--context-length 262144`. Lower it if you need more memory headroom.
* **Tensor parallelism**: the 397B model-card recipes use `--tp 8`; in this single-node matrix, non-FP8 397B is H200-only, while 397B-FP8 is available on both H100 and H200. The 35B deploy commands use `--tp 2`. The 9B model-card recipe is single-GPU by default; the command panel makes that explicit with `--tp 1`. Adjust TP to match your node and memory budget.
* **Sampling**: model cards recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20` for normal use. Their reported benchmark setup may use different task-specific sampling parameters.
* **Benchmarks**: benchmark numbers in the model cards are reported by DeepReinforce. They are useful for context, but the command panel leaves recipes unverified until exact runs are signed off.

## 3. Usage Examples

### 3.1 Basic Chat Completion

`message.reasoning_content` is only populated when the server was launched with the `--reasoning-parser qwen3` toggle (see the [Playground](#playground)); otherwise the `<think>...</think>` trace stays inline in `message.content`.

> **Python client**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="Ornith-1.0-9B",
>     messages=[
>         {"role": "user", "content": "Write a compact Python function is_prime(n)."}
>     ],
>     temperature=0.6,
>     top_p=0.95,
>     max_tokens=1024,
>     extra_body={"top_k": 20},
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(message.reasoning_content)
> print("=============== Answer ==================")
> print(message.content)
> ```

### 3.2 Tool Calling

Enable the `--tool-call-parser qwen3_coder` toggle in the [Playground](#playground) and launch with the resulting command. Then use the standard OpenAI-compatible `tools` field:

> **Tool-call request**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "run_tests",
>         "description": "Run the project's test suite.",
>         "parameters": {
>             "type": "object",
>             "properties": {
>                 "target": {"type": "string", "description": "Test target or command"}
>             },
>             "required": ["target"],
>         },
>     },
> }]
>
> response = client.chat.completions.create(
>     model="Ornith-1.0-9B",
>     messages=[{"role": "user", "content": "Run the unit tests for the parser module."}],
>     tools=tools,
>     tool_choice="auto",
>     temperature=0.6,
>     top_p=0.95,
>     max_tokens=2048,
> )
>
> print(response.choices[0].message.tool_calls)
> ```


# Ornith-1.5
Source: https://docs.sglang.io/cookbook/autoregressive/DeepReinforce/Ornith-1.5

Deploy Ornith-1.5 with SGLang on NVIDIA H100 and H200 GPUs, with recipes for 9B, 35B-A3B, 122B, and 397B models, FP8, reasoning, and tool calling.

## Deployment

These recipes target a single Linux node with NVIDIA CUDA GPUs. The Ornith-1.5 model cards recommend SGLang `>=0.5.9`. Both parsers are included in every launch command: `qwen3` separates reasoning from the answer, and `qwen3_coder` parses tool calls.

> **Install SGLang**
>
> **Python**
>
> ```bash Command theme={null}
> pip install --upgrade "sglang>=0.5.9"
> ```
>
> Run the Python command from the deployment panel in this environment. See the [installation guide](/docs/get-started/install) for platform-specific requirements.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> Select Docker in the deployment panel to generate the container command.

Choose your hardware, model size, and checkpoint format. The 9B recipe uses one GPU, 35B-A3B uses two, and 122B and 397B use eight. BF16 397B requires H200 in this matrix; its FP8 checkpoint is also available on H100. Select **BF16** for a BF16 checkpoint or **FP8** for an FP8 checkpoint. All commands use Hugging Face repository IDs and default to port **30000**.

## 1. Model introduction

Ornith-1.5 is an agentic model family published by [Ornith AI](https://huggingface.co/ornith-ai). The 9B, 35B-A3B, 122B, and 397B checkpoints use Qwen3.5 architectures. The family extends Ornith-1.0's self-improvement loop by jointly optimizing task generation, scaffold construction, and solution rollouts through reinforcement learning. The model cards describe applications in coding, reasoning, and tool-using agents.

The recipes expose an OpenAI-compatible API with separate reasoning output, parsed tool calls, and a 262,144-token context window. The table lists the checkpoints covered by this recipe.

<table>
  <thead>
    <tr>
      <th>Checkpoint</th>
      <th>Format</th>
      <th>Recipe</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[Ornith-1.5-9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B)</td>
      <td>BF16</td>
      <td>Dense; 1× H100 or H200</td>
    </tr>

    <tr>
      <td>[Ornith-1.5-35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B)</td>
      <td>BF16</td>
      <td>MoE, \~3B active; 2× H100 or H200</td>
    </tr>

    <tr>
      <td>[Ornith-1.5-35B-A3B-FP8](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B-FP8)</td>
      <td>FP8</td>
      <td>2× H100 or H200</td>
    </tr>

    <tr>
      <td>[Ornith-1.5-122B](https://huggingface.co/ornith-ai/Ornith-1.5-122B)</td>
      <td>BF16</td>
      <td>MoE, 256 experts with 8 routed per token; 8× H100 or H200</td>
    </tr>

    <tr>
      <td>[Ornith-1.5-397B](https://huggingface.co/ornith-ai/Ornith-1.5-397B)</td>
      <td>BF16</td>
      <td>MoE; 8× H200</td>
    </tr>

    <tr>
      <td>[Ornith-1.5-397B-FP8](https://huggingface.co/ornith-ai/Ornith-1.5-397B-FP8)</td>
      <td>FP8</td>
      <td>8× H100 or H200</td>
    </tr>
  </tbody>
</table>

The official [9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B), [35B-A3B](https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B), [122B](https://huggingface.co/ornith-ai/Ornith-1.5-122B), and [397B](https://huggingface.co/ornith-ai/Ornith-1.5-397B) model cards declare the MIT license.

**Resources:** [Ornith-1.5 model repositories](https://huggingface.co/ornith-ai/models?search=Ornith-1.5) · [Ornith-1.5 blog](https://ornith.ai/ornith_1_5.html)

## 2. Configuration tips

* **Model IDs:** use the `ornith-ai` namespace. The 35B repository includes the `-A3B` suffix. FP8 selections use the actual `-FP8` repository rather than the BF16 repository shown in their model-card quickstarts.
* **Context and memory:** every recipe uses `--context-length 262144`. The 9B, 35B-A3B, and 397B recipes use `--mem-fraction-static 0.85`; the 122B recipe uses `0.80`. Lower the context length if you need more memory headroom.
* **Reasoning:** `--reasoning-parser qwen3` exposes reasoning as `message.reasoning_content`, with the answer in `message.content`.
* **Tool calling:** `--tool-call-parser qwen3_coder` exposes parsed calls in `message.tool_calls`. Your application executes the requested function and submits its result in a follow-up tool message.
* **Sampling:** 35B-A3B and 397B recommend `temperature=0.6`, `top_p=0.95`, and `top_k=20`. For 9B and 122B, the model cards recommend `temperature=1.0` and `presence_penalty=1.5` for general tasks, or `temperature=0.6` and `presence_penalty=0.0` for precise coding; both use `top_p=0.95`, `top_k=20`, `min_p=0.0`, and `repetition_penalty=1.0`. These settings are distinct from benchmark-specific settings. The Python examples below use the precise-coding sampling settings.

## 3. Usage examples

These requests use the precise-coding sampling settings shared by 9B and 122B. Set `model` to the deployment panel's `--served-model-name`, such as `Ornith-1.5-9B` or `Ornith-1.5-122B`. The examples use the default port; match the client port if you change the launch port.

> **Chat completion with reasoning (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> response = client.chat.completions.create(
>     model="Ornith-1.5-9B",
>     messages=[{"role": "user", "content": "Write a compact Python is_prime function."}],
>     temperature=0.6,
>     top_p=0.95,
>     presence_penalty=0.0,
>     max_tokens=1024,
>     extra_body={"top_k": 20, "min_p": 0.0, "repetition_penalty": 1.0},
> )
> message = response.choices[0].message
> print("Reasoning:", getattr(message, "reasoning_content", None))
> print("Answer:", message.content)
> ```

> **Tool-call request (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> response = client.chat.completions.create(
>     model="Ornith-1.5-9B",
>     messages=[{"role": "user", "content": "Run the parser module's unit tests."}],
>     tools=[{
>         "type": "function",
>         "function": {
>             "name": "run_tests",
>             "description": "Run the project's test suite.",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"target": {"type": "string"}},
>                 "required": ["target"],
>             },
>         },
>     }],
>     tool_choice="auto",
>     temperature=0.6,
>     top_p=0.95,
>     presence_penalty=0.0,
>     max_tokens=2048,
>     extra_body={"top_k": 20, "min_p": 0.0, "repetition_penalty": 1.0},
> )
> message = response.choices[0].message
> print("Reasoning:", getattr(message, "reasoning_content", None))
> print("Tool calls:", message.tool_calls)
> ```

### Agent integration

Configure your agent or coding CLI to use the same OpenAI-compatible endpoint:

* **API base URL:** `http://localhost:30000/v1`.
* **API key:** `EMPTY` for a local server without API-key authentication.
* **Model:** the selected checkpoint's `--served-model-name`, such as `Ornith-1.5-122B`.

Use the served model name rather than the Hugging Face repository ID when the two differ. Keep both parsers enabled so the agent receives separate reasoning and structured tool calls. Your agent executes tools and returns their results to the server in subsequent messages.


# DeepSeek-Math-V2
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-Math-V2



## 1. Model Introduction

[DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) is DeepSeek's advanced mathematical reasoning model with strong theorem-proving capabilities. The model demonstrates exceptional performance on mathematical competitions, achieving gold-level scores on IMO 2025 and CMO 2024, and a near-perfect 118/120 on Putnam 2024 with scaled test-time compute.

**Key Features:**

* **Strong Theorem-Proving**: Gold-level performance on IMO 2025 and CMO 2024
* **Self-Verifiable Reasoning**: Implements self-verifiable mathematical reasoning for improved accuracy
* **Competition-Level Math**: Near-perfect score (118/120) on Putnam 2024
* **Large MoE Model**: \~671B total parameters, requires high-memory GPUs (B200 183GB or B300 275GB)

**Available Models:**

* **BF16 (Full Weights)**: [deepseek-ai/DeepSeek-Math-V2](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2) - Full precision weights

**License:**
To use DeepSeek-Math-V2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-Math-V2/blob/main/LICENSE) for details.

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.

Run this Command:

<pre />

> [!WARNING]
>
> DeepSeek-Math-V2 is built on DeepSeek-V3.2 and uses DSA sparse attention. All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on this model.

### 3.2 Configuration Tips

**Hardware Requirements:**

* **B200 (183GB)**: BF16 tp=8
* **B300 (275GB)**: BF16 tp=8

**DP Attention:**

* Enable DP attention for high-throughput scenarios
* The `--attn-dp-size` value commonly matches the `--tp` value
* Trade-off: Higher throughput at the cost of slightly increased latency

## 4. Model Invocation

### 4.1 Deployment Command

Deploy the model using the command generated above. Example for B200:

```shell Command theme={null}
sglang serve --model-path deepseek-ai/DeepSeek-Math-V2 \
  --tp 8 \
  --ep 8 \
  --reasoning-parser deepseek-r1 \
  --host 0.0.0.0 \
  --port 30000
```

### 4.2 Mathematical Reasoning

DeepSeek-Math-V2 excels at mathematical problem-solving with step-by-step reasoning.

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Mathematical reasoning problem
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-Math-V2",
    messages=[
        {"role": "user", "content": "Prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2"}
    ],
    max_tokens=4096,
    stream=True
)

# Process the stream
thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
We need to prove that for any positive integer n, the sum 1 + 2 + 3 + ... + n = n(n+1)/2.

This is a classic formula for the sum of the first n natural numbers. We can prove by induction.

Base case: n=1, LHS = 1, RHS = 1*(1+1)/2 = 1*2/2 = 1. Holds.

Inductive step: Assume true for n = k, i.e., 1 + 2 + ... + k = k(k+1)/2. Then for n = k+1, sum = 1 + 2 + ... + k + (k+1) = [k(k+1)/2] + (k+1) = (k(k+1) + 2(k+1))/2 = (k+1)(k+2)/2 = (k+1
)((k+1)+1)/2. So holds for k+1. By induction, holds for all positive integers n.

...
=============== Content =================
We can prove the well-known formula for the sum of the first \(n\) positive integers in several ways. Two of the most elementary are presented below.

---

### 1. Proof by mathematical induction

**Base case (\(n=1\))**:
\[
1 = \frac{1\cdot(1+1)}{2}= \frac{1\cdot2}{2}=1,
\]
so the formula holds for \(n=1\).

**Inductive hypothesis:**
Assume that for some positive integer \(k\) the formula is true, i.e.
\[
1+2+\dots+k = \frac{k(k+1)}{2}.
\]

**Inductive step (\(k \to k+1\))**:
Consider the sum up to \(k+1\):
\[
\begin{aligned}
1+2+\dots+k+(k+1) &= \bigl(1+2+\dots+k\bigr) + (k+1) \\[4pt]
&= \frac{k(k+1)}{2} + (k+1) \qquad\text{(by the induction hypothesis)}\\[4pt]
&= (k+1)\left(\frac{k}{2}+1\right)\\[4pt]
&= (k+1)\frac{k+2}{2}\\[4pt]
&= \frac{(k+1)(k+2)}{2}\\[4pt]
&= \frac{(k+1)\bigl((k+1)+1\bigr)}{2}.
\end{aligned}
\]
Thus the formula also holds for \(n=k+1\).

By the principle of mathematical induction,
\[
1+2+3+\dots+n = \frac{n(n+1)}{2}
\]
for every positive integer \(n\).

---

### 2. Proof by pairing (Gauss’s trick)

Let
\[
S = 1 + 2 + 3 + \dots + n.
\]

Write the same sum in reverse order:
\[
S = n + (n-1) + (n-2) + \dots + 1.
\]

Add the two equalities term‑by‑term:
\[
\begin{aligned}
2S &= (1+n) + \bigl(2+(n-1)\bigr) + \bigl(3+(n-2)\bigr) + \dots + (n+1)\\
    &= \underbrace{(n+1)+(n+1)+\dots+(n+1)}_{n\ \text{times}}\\
    &= n\,(n+1).
\end{aligned}
\]

Therefore
\[
S = \frac{n(n+1)}{2}.
\]

Both proofs are rigorous and show that the formula holds for all positive integers \(n\).
```

### 4.3 Competition-Level Problems

**Example: IMO-style Problem:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# IMO-style problem
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-Math-V2",
    messages=[
        {"role": "user", "content": "Let a, b, c be positive real numbers such that abc = 1. Prove that (a-1+1/b)(b-1+1/c)(c-1+1/a) <= 1."}
    ],
    max_tokens=8192,
    stream=True
)

# Process the stream
thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
We need to prove that for positive real numbers a,b,c with abc = 1, we have:

\[
(a - 1 + \frac{1}{b})(b - 1 + \frac{1}{c})(c - 1 + \frac{1}{a}) \le 1.
\]

We can rewrite the expressions: Since abc=1, we have 1/b = ac, 1/c = ab, 1/a = bc. Wait careful: abc=1 => 1/b = ac? Actually 1/b = ac? Let's check: abc=1 => ac = 1/b? Multiply both sides by something: abc=1 => (ac) b = 1 => ac = 1/b. Yes, because (ac) * b = 1 => ac = 1/b. Similarly, ab = 1/c, bc = 1/a. So we can rewrite:

...
=============== Content =================

We are given positive real numbers \(a,b,c\) with \(abc=1\). We must prove

\[
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
\]

---

### 1.  A convenient substitution

Because \(abc=1\), we can write

\[
a=\frac{x}{y},\qquad b=\frac{y}{z},\qquad c=\frac{z}{x}
\]

with positive numbers \(x,y,z\).
(For instance, take \(x=1,\;y=\frac1a,\;z=\frac1{ab}\); then indeed \(a=\frac{x}{y},\;b=\frac{y}{z}\) and, using \(abc=1\), we obtain \(c=\frac{z}{x}=\frac1{ab}=c\).)

---

### 2.  Rewriting the factors

\[
\begin{aligned}
a-1+\frac1b &=\frac{x}{y}-1+\frac{z}{y}= \frac{x+z-y}{y},\\[2mm]
b-1+\frac1c &=\frac{y}{z}-1+\frac{x}{z}= \frac{x+y-z}{z},\\[2mm]
c-1+\frac1a &=\frac{z}{x}-1+\frac{y}{x}= \frac{y+z-x}{x}.
\end{aligned}
\]

Hence the product becomes

\[
P=\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)
   =\frac{(x+z-y)(x+y-z)(y+z-x)}{xyz}.
\]

---

### 3.  Reducing to a known inequality

We have to show \(P\le1\), i.e.

\[
(x+z-y)(x+y-z)(y+z-x)\le xyz .
\tag{1}
\]

Set

\[
p=x+y+z,\qquad q=xy+yz+zx,\qquad r=xyz .
\]

Notice that

\[
x+z-y=p-2y,\quad x+y-z=p-2z,\quad y+z-x=p-2x .
\]

Therefore

\[
\begin{aligned}
(x+z-y)(x+y-z)(y+z-x)
&=(p-2x)(p-2y)(p-2z)\\
&=p^{3}-2p^{2}(x+y+z)+4p(xy+yz+zx)-8xyz\\
&=-p^{3}+4pq-8r .
\end{aligned}
\]

Inequality (1) is thus equivalent to

\[
-p^{3}+4pq-8r\le r\quad\Longleftrightarrow\quad 4pq-p^{3}\le 9r .
\tag{2}
\]

---

### 4.  Applying Schur’s inequality

Schur’s inequality of third degree states that for any non‑negative \(x,y,z\)

\[
p^{3}+9r\ge 4pq .
\]

Rearranged, this is exactly \(4pq-p^{3}\le 9r\), which is (2).
Since our \(x,y,z\) are positive, Schur’s inequality applies and (2) holds.

Consequently (1) is true, and we obtain \(P\le1\).

---

### 5.  Equality case

Equality in Schur’s inequality for positive numbers occurs only when \(x=y=z\).
Then \(a=b=c=1\), and indeed the product equals \(1\).

---

Thus for all positive \(a,b,c\) with \(abc=1\),

\[
\Bigl(a-1+\frac1b\Bigr)\Bigl(b-1+\frac1c\Bigr)\Bigl(c-1+\frac1a\Bigr)\le 1 .
\]

∎
```

## 5. Benchmark

### 5.1 Accuracy Benchmark

#### 5.1.1 GSM8K Benchmark

**Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.975
Invalid: 0.000
Latency: 34.358 s
Output throughput: 540.162 token/s
```

### 5.2 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8x, 183GB each)
* Model: DeepSeek-Math-V2
* Tensor Parallelism: 8
* SGLang Version: 0.5.8

#### 5.2.1 Latency Benchmark

**Benchmark Command:**

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model deepseek-ai/DeepSeek-Math-V2 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

**Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  53.34
Total input tokens:                      1972
Total input text tokens:                 1972
Total generated tokens:                  2784
Total generated tokens (retokenized):    2778
Request throughput (req/s):              0.19
Input token throughput (tok/s):          36.97
Output token throughput (tok/s):         52.19
Peak output token throughput (tok/s):    56.00
Peak concurrent requests:                3
Total token throughput (tok/s):          89.16
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5330.72
Median E2E Latency (ms):                 5879.28
P90 E2E Latency (ms):                    8320.33
P99 E2E Latency (ms):                    9921.29
---------------Time to First Token----------------
Mean TTFT (ms):                          183.38
Median TTFT (ms):                        177.92
P99 TTFT (ms):                           217.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.96
Median TPOT (ms):                        18.39
P99 TPOT (ms):                           19.03
---------------Inter-Token Latency----------------
Mean ITL (ms):                           18.57
Median ITL (ms):                         18.63
P95 ITL (ms):                            19.26
P99 ITL (ms):                            19.48
Max ITL (ms):                            24.93
==================================================
```

#### 5.2.2 Throughput Benchmark

**Benchmark Command:**

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model deepseek-ai/DeepSeek-Math-V2 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

**Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  217.36
Total input tokens:                      301701
Total input text tokens:                 301701
Total generated tokens:                  188375
Total generated tokens (retokenized):    187456
Request throughput (req/s):              4.60
Input token throughput (tok/s):          1388.05
Output token throughput (tok/s):         866.67
Peak output token throughput (tok/s):    2589.00
Peak concurrent requests:                109
Total token throughput (tok/s):          2254.72
Concurrency:                             89.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   19521.73
Median E2E Latency (ms):                 12076.76
P90 E2E Latency (ms):                    47248.87
P99 E2E Latency (ms):                    86862.79
---------------Time to First Token----------------
Mean TTFT (ms):                          790.40
Median TTFT (ms):                        456.81
P99 TTFT (ms):                           4223.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          106.52
Median TPOT (ms):                        107.24
P99 TPOT (ms):                           238.33
---------------Inter-Token Latency----------------
Mean ITL (ms):                           100.29
Median ITL (ms):                         38.34
P95 ITL (ms):                            237.00
P99 ITL (ms):                            347.49
Max ITL (ms):                            3642.56
==================================================
```


# DeepSeek-OCR
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-OCR



## 1. Model Introduction

[DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR) is DeepSeek's advanced OCR (Optical Character Recognition) model designed for high-accuracy text extraction from images. The model is optimized for various document processing and image-to-text conversion tasks.

**Key Features:**

* **Advanced OCR**: High-accuracy text recognition from images and documents
* **Multi-Modality**: Supports various image formats and document types

**Available Models:**

* **Base Model**: [deepseek-ai/DeepSeek-OCR](https://huggingface.co/deepseek-ai/DeepSeek-OCR) - Recommended for OCR tasks

**License:**
To use DeepSeek-OCR, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR/blob/main/LICENSE) for details.

For more details, please refer to the [official DeepSeek-OCR repository](https://github.com/deepseek-ai/DeepSeek-OCR).

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy.

Run this Command:

<pre />

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 OCR-Specific Prompts

DeepSeek-OCR accepts recommended prompts from the model card:

```text theme={null}
<image>
<|grounding|>Convert the document to markdown.
```

```text theme={null}
<image>
Free OCR.
```

**OpenAI-compatible image request example:**

```python Example theme={null}
import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "deepseek-ai/DeepSeek-OCR",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "<image>\n<|grounding|>Convert the document to markdown."
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/your_image.jpg"}
                },
            ],
        }
    ],
    "max_tokens": 512,
}

response = requests.post(url, json=data)
print(response.text)
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: AMD MI300X GPU (1x)
* Model: DeepSeek-OCR
* Tensor Parallelism: 1
* sglang version: 0.5.7

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-OCR \
  --tp 1 \
  --dtype float16 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 8000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 8000 \
  --model deepseek-ai/DeepSeek-OCR \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  4.45
Total input tokens:                      1972
Total input text tokens:                 1972
Total input vision tokens:               0
Total generated tokens:                  2784
Total generated tokens (retokenized):    2770
Request throughput (req/s):              2.25
Input token throughput (tok/s):          442.89
Output token throughput (tok/s):         625.26
Peak output token throughput (tok/s):    635.00
Peak concurrent requests:                4
Total token throughput (tok/s):          1068.16
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   443.32
Median E2E Latency (ms):                 493.29
---------------Time to First Token----------------
Mean TTFT (ms):                          21.59
Median TTFT (ms):                        20.89
P99 TTFT (ms):                           24.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.47
Median TPOT (ms):                        1.52
P99 TPOT (ms):                           1.53
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.52
Median ITL (ms):                         1.51
P95 ITL (ms):                            1.76
P99 ITL (ms):                            1.93
Max ITL (ms):                            8.28
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-OCR \
  --tp 1 \
  --ep 1 \
  --dtype float16 \
  --host 0.0.0.0 \
  --port 8000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 8000 \
  --model deepseek-ai/DeepSeek-OCR \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  16.24
Total input tokens:                      301698
Total input text tokens:                 301698
Total input vision tokens:               0
Total generated tokens:                  188375
Total generated tokens (retokenized):    186927
Request throughput (req/s):              61.59
Input token throughput (tok/s):          18582.90
Output token throughput (tok/s):         11602.84
Peak output token throughput (tok/s):    15479.00
Peak concurrent requests:                179
Total token throughput (tok/s):          30185.75
Concurrency:                             85.53
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1388.60
Median E2E Latency (ms):                 901.43
---------------Time to First Token----------------
Mean TTFT (ms):                          73.36
Median TTFT (ms):                        50.21
P99 TTFT (ms):                           349.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.42
Median TPOT (ms):                        7.31
P99 TPOT (ms):                           27.99
---------------Inter-Token Latency----------------
Mean ITL (ms):                           7.04
Median ITL (ms):                         4.62
P95 ITL (ms):                            21.11
P99 ITL (ms):                            36.92
Max ITL (ms):                            172.15
==================================================
```


# DeepSeek-OCR-2
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2



## 1. Model Introduction

[DeepSeek-OCR-2](https://github.com/deepseek-ai/DeepSeek-OCR-2) is DeepSeek's next-generation OCR (Optical Character Recognition) model, building on DeepSeek-OCR with improved accuracy and broader document understanding capabilities. The model is optimized for high-accuracy text extraction from images across a wide variety of document types and formats.

**Key Features:**

* **Semantic-Aware Visual Encoding (DeepEncoder V2)**: DeepSeek-OCR-2 introduces DeepEncoder V2, which models document reading order in a more human-like, semantic-driven manner rather than relying on fixed raster scanning. This significantly improves logical reading flow in complex layouts (e.g., multi-column documents).
* **Stronger Layout and Structural Understanding**: DeepSeek-OCR-2 demonstrates improved performance on structured documents such as tables, forms, and dense multi-column pages. It reduces reading-order errors and improves overall document parsing robustness compared to the original version.
* **Improved Accuracy While Maintaining Token Efficiency**: The original DeepSeek-OCR emphasized aggressive visual token compression. OCR-2 maintains high token efficiency while delivering higher benchmark performance, particularly on document-level understanding tasks.
* **Better Generalization Across Complex Document Tasks**: DeepSeek-OCR-2 performs more consistently across multilingual documents, structured data extraction, and visually complex content, making it more suitable for real-world document intelligence scenarios beyond plain text OCR.

**Available Models:**

* **Base Model**: [deepseek-ai/DeepSeek-OCR-2](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2) - Recommended for OCR tasks

**License:**
To use DeepSeek-OCR-2, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2/blob/main/LICENSE.txt) for details.

For more details, please refer to the [official DeepSeek-OCR-2 repository](https://github.com/deepseek-ai/DeepSeek-OCR-2).

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The DeepSeek-OCR-2 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, and Intel Arc Pro B-Series GPUs, as well as Intel Xeon CPUs.

Run this Command:

<pre />

**Note**: DeepSeek-OCR-2 has \~3B parameters and easily fits on a single modern GPU. For low-latency serving, no model parallelism is needed. For high-throughput requirements, consider using data parallelism with the SGLang Model Gateway — see [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway) for more details.

### 3.2 Configuration Tips

* **Single GPU Deployment:** DeepSeek-OCR-2 (\~3B parameters) fits on a single modern GPU — no tensor parallelism required for low-latency serving.
* **High Throughput:** For high-throughput scenarios, use data parallelism with the SGLang Model Gateway. See [DP, DPA and SGLang DP Router](../../../docs/advanced_features/sgl_model_gateway).
* **NCCL timeout:** If model loading is slow, increase `--dist-timeout 3600`.
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

**OpenAI-compatible request example**

```python Example theme={null}
import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "deepseek-ai/DeepSeek-OCR-2",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "<image>\n<|grounding|>Convert the document to markdown."},
                {"type": "image_url", "image_url": {"url": "https://example.com/your_image.jpg"}},
            ],
        }
    ],
    "max_tokens": 512,
}

response = requests.post(url, json=data)
print(response.text)
```

**Reference**

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Recommended Prompts

The following prompts are recommended by the [official model card](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts).

**Structured document conversion** — extracts text while preserving layout:

```text Example theme={null}
<image>
<|grounding|>Convert the document to markdown.
```

**Free-form OCR** — extracts without layouts:

```text Example theme={null}
<image>
Free OCR.
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA H200 GPU (1x)
* Model: DeepSeek-OCR-2
* Tensor Parallelism: 1
* sglang version: 0.0.0.dev1+g93fca0bbc

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses. For more details on how to perform evaluation, see [Evaluating New Models with SGLang](../../../docs/developer_guide/evaluating_new_models).

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
  --model-path deepseek-ai/DeepSeek-OCR-2 \
  --enable-multimodal \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 0.0.0.0 \
  --port 30000 \
  --model deepseek-ai/DeepSeek-OCR-2 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  3.54
Total input tokens:                      1972
Total input text tokens:                 1972
Total generated tokens:                  2784
Total generated tokens (retokenized):    2710
Request throughput (req/s):              2.83
Input token throughput (tok/s):          557.53
Output token throughput (tok/s):         787.10
Peak output token throughput (tok/s):    818.00
Peak concurrent requests:                5
Total token throughput (tok/s):          1344.63
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   352.69
Median E2E Latency (ms):                 392.34
P90 E2E Latency (ms):                    540.64
P99 E2E Latency (ms):                    639.01
---------------Time to First Token----------------
Mean TTFT (ms):                          18.08
Median TTFT (ms):                        16.57
P99 TTFT (ms):                           25.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.18
Median TPOT (ms):                        1.21
P99 TPOT (ms):                           1.22
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.21
Median ITL (ms):                         1.21
P95 ITL (ms):                            1.28
P99 ITL (ms):                            1.44
Max ITL (ms):                            4.32
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
  --model-path deepseek-ai/DeepSeek-OCR-2 \
  --enable-multimodal \
  --tp 1 \
  --ep 1 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 0.0.0.0 \
  --port 30000 \
  --model deepseek-ai/DeepSeek-OCR-2 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  14.79
Total input tokens:                      301698
Total input text tokens:                 301698
Total generated tokens:                  188375
Total generated tokens (retokenized):    185236
Request throughput (req/s):              67.63
Input token throughput (tok/s):          20402.54
Output token throughput (tok/s):         12738.99
Peak output token throughput (tok/s):    17508.00
Peak concurrent requests:                187
Total token throughput (tok/s):          33141.53
Concurrency:                             86.87
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1284.50
Median E2E Latency (ms):                 866.07
P90 E2E Latency (ms):                    3027.32
P99 E2E Latency (ms):                    5490.63
---------------Time to First Token----------------
Mean TTFT (ms):                          86.08
Median TTFT (ms):                        50.09
P99 TTFT (ms):                           613.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.79
Median TPOT (ms):                        6.54
P99 TPOT (ms):                           50.10
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.42
Median ITL (ms):                         4.64
P95 ITL (ms):                            23.65
P99 ITL (ms):                            39.62
Max ITL (ms):                            452.65
==================================================
```


# DeepSeek-R1
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-R1



## 1. Model Introduction

[DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) is DeepSeek's advanced reasoning model that combines powerful language understanding with step-by-step reasoning capabilities. The model is available in multiple quantization formats optimized for different hardware platforms.

**Key Features:**

* **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
* **Multiple Quantizations**: FP8 and FP4 variants for different performance/memory trade-offs
* **Hardware Optimization**: Specifically tuned for NVIDIA B200 (Blackwell) and H200 (Hopper) GPUs, AMD MI300X, MI325X and MI355X GPUs, as well as Intel Xeon CPUs
* **High Performance**: Optimized for both throughput and latency scenarios

**Available Models:**

* **FP8 (8-bit quantized)**: [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) - Recommended for H200 and MI300X
* **FP4 (4-bit quantized)**: [nvidia/DeepSeek-R1-0528-FP4-v2](https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4-v2) - Recommended for B200 and MI355X
* **BF16 (upcast from FP8)**: [unsloth/DeepSeek-R1-0528-BF16](https://huggingface.co/unsloth/DeepSeek-R1-0528-BF16)
* **INT8 (channel-wise)**: [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8)
* **W4A8**: [novita/Deepseek-R1-0528-W4AFP8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8)
* **AWQ (4-bit)**: [QuixiAI/DeepSeek-R1-0528-AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ)
* **MXFP4**: [amd/DeepSeek-R1-MXFP4](https://huggingface.co/amd/DeepSeek-R1-MXFP4)

**License:**
To use DeepSeek-R1, you must agree to DeepSeek's Community License. See [LICENSE](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528/blob/main/LICENSE) for details.

For more details, please refer to the [official DeepSeek-R1 repository](https://github.com/deepseek-ai/DeepSeek-R1).

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate a basic deployment command for your hardware platform, quantization method, and deployment strategy.

Run this Command:

<pre />

### 3.2 Optimal Configurations

Pareto-optimal configurations for B200, H200, MI300X, MI325X, and MI355X hardware.

Run this Command:

<pre />

### 3.3 Configuration Tips

DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.

**Recommended GPU configurations by weight type:**

<table>
  <thead>
    <tr>
      <th>Weight Type</th>
      <th>Supported Hardware</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**FP8** (recommended)</td>
      <td>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
    </tr>

    <tr>
      <td>**BF16** (upcast from FP8)</td>
      <td>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
    </tr>

    <tr>
      <td>**INT8**</td>
      <td>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series</td>
    </tr>

    <tr>
      <td>**W4A8 / AWQ / MXFP4 / NVFP4**</td>
      <td>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
    </tr>
  </tbody>
</table>

> The official DeepSeek-R1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.

**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels to avoid JIT overhead (\~10 min):

```bash theme={null}
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --trust-remote-code
```

**Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node.

**NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`.

**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

DeepSeek-R1 supports advanced reasoning capabilities with built-in thinking process. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-R1-0528 \
  --reasoning-parser deepseek-r1 \
  --tp 8
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

DeepSeek-R1 supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-R1-0528 \
  --reasoning-parser deepseek-r1 \
  --tool-call-parser deepseekv3 \
  --chat-template examples/chat_template/tool_chat_template_deepseekr1.jinja \
  --tp 8
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

🔧 Tool Call: get_weather
   Arguments:
🔧 Tool Call: None
   Arguments: {"location": "Beijing"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-0528",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)

DeepSeek-R1 supports EAGLE-based Multi-Token Prediction (MTP), the same mechanism as DeepSeek-V3. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the complete launch command, flag reference, tuning guidance (`--speculative-num-steps`, `--speculative-eagle-topk`, `--max-running-requests`), and `bench_speculative.py` link. R1's speed benchmark commands that include `--speculative-*` flags use this mechanism.

#### 4.2.4 Thinking Budget

Limit the model's thinking token budget using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-R1 \
  --tp 8 \
  --port 30000 \
  --reasoning-parser deepseek-r1 \
  --enable-custom-logit-processor
```

```python Example theme={null}
import openai
from sglang.srt.sampling.custom_logit_processor import DeepSeekR1ThinkingBudgetLogitProcessor

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1",
    messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
    max_tokens=1024,
    extra_body={
        "custom_logit_processor": DeepSeekR1ThinkingBudgetLogitProcessor().to_str(),
        "custom_params": {"thinking_budget": 512},
    },
)
print(response)
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: B200 GPU (8x)
* Model: DeepSeek-R1-0528
* Tensor Parallelism: 8
* SGLang Version: 0.5.6.post1

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Test Scenarios

Three core scenarios reflect real-world usage patterns:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Scenario</th>
      <th>Input Length</th>
      <th>Output Length</th>
      <th>Use Case</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Chat**</td>
      <td>1K</td>
      <td>1K</td>
      <td>Most common conversational AI workload</td>
    </tr>

    <tr>
      <td>**Reasoning**</td>
      <td>1K</td>
      <td>8K</td>
      <td>Long-form generation, complex reasoning tasks</td>
    </tr>

    <tr>
      <td>**Summarization**</td>
      <td>8K</td>
      <td>1K</td>
      <td>Document summarization, RAG retrieval</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 Concurrency Levels

Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:

* **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
* **Medium Concurrency**: `--max-concurrency 16` (Balanced)
* **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)

#### 5.1.3 Number of Prompts

For each concurrency level, configure `num_prompts` to simulate realistic user loads:

* **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
* **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
* **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)

***

#### 5.1.4 Benchmark Commands

**Scenario 1: Chat (1K/1K) - Most Important**

* **Model Deployment**

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-R1-0528 \
  --tp 8
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  40.00
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4210
Total generated tokens (retokenized):    4205
Request throughput (req/s):              0.25
Input token throughput (tok/s):          152.52
Output token throughput (tok/s):         105.24
Peak output token throughput (tok/s):    110.00
Peak concurrent requests:                2
Total token throughput (tok/s):          257.76
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3998.40
Median E2E Latency (ms):                 3207.53
---------------Time to First Token----------------
Mean TTFT (ms):                          153.00
Median TTFT (ms):                        140.76
P99 TTFT (ms):                           214.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.16
Median TPOT (ms):                        9.15
P99 TPOT (ms):                           9.21
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.16
Median ITL (ms):                         9.15
P95 ITL (ms):                            9.47
P99 ITL (ms):                            9.63
Max ITL (ms):                            15.45
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  51.21
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40725
Total generated tokens (retokenized):    40458
Request throughput (req/s):              1.56
Input token throughput (tok/s):          774.66
Output token throughput (tok/s):         795.30
Peak output token throughput (tok/s):    1088.00
Peak concurrent requests:                21
Total token throughput (tok/s):          1569.96
Concurrency:                             13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8918.33
Median E2E Latency (ms):                 9466.16
---------------Time to First Token----------------
Mean TTFT (ms):                          273.51
Median TTFT (ms):                        131.71
P99 TTFT (ms):                           839.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.56
Median TPOT (ms):                        17.46
P99 TPOT (ms):                           28.68
---------------Inter-Token Latency----------------
Mean ITL (ms):                           17.02
Median ITL (ms):                         14.70
P95 ITL (ms):                            16.41
P99 ITL (ms):                            112.38
Max ITL (ms):                            461.90
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  110.46
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252162
Total generated tokens (retokenized):    251441
Request throughput (req/s):              4.53
Input token throughput (tok/s):          2261.80
Output token throughput (tok/s):         2282.90
Peak output token throughput (tok/s):    3900.00
Peak concurrent requests:                109
Total token throughput (tok/s):          4544.71
Concurrency:                             92.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   20380.71
Median E2E Latency (ms):                 19391.65
---------------Time to First Token----------------
Mean TTFT (ms):                          563.14
Median TTFT (ms):                        147.62
P99 TTFT (ms):                           2632.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          40.11
Median TPOT (ms):                        41.98
P99 TPOT (ms):                           50.10
---------------Inter-Token Latency----------------
Mean ITL (ms):                           39.37
Median ITL (ms):                         26.36
P95 ITL (ms):                            98.16
P99 ITL (ms):                            150.08
Max ITL (ms):                            2052.85
==================================================
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  411.34
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  44452
Total generated tokens (retokenized):    44390
Request throughput (req/s):              0.02
Input token throughput (tok/s):          14.83
Output token throughput (tok/s):         108.07
Peak output token throughput (tok/s):    110.00
Peak concurrent requests:                2
Total token throughput (tok/s):          122.90
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   41132.04
Median E2E Latency (ms):                 44288.71
---------------Time to First Token----------------
Mean TTFT (ms):                          125.76
Median TTFT (ms):                        126.19
P99 TTFT (ms):                           137.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.21
Median TPOT (ms):                        9.20
P99 TPOT (ms):                           9.27
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.23
Median ITL (ms):                         9.22
P95 ITL (ms):                            9.64
P99 ITL (ms):                            9.86
Max ITL (ms):                            15.18
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  348.93
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  318226
Total generated tokens (retokenized):    317630
Request throughput (req/s):              0.23
Input token throughput (tok/s):          113.69
Output token throughput (tok/s):         912.02
Peak output token throughput (tok/s):    1088.00
Peak concurrent requests:                19
Total token throughput (tok/s):          1025.70
Concurrency:                             14.07
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   61360.70
Median E2E Latency (ms):                 62071.20
---------------Time to First Token----------------
Mean TTFT (ms):                          176.02
Median TTFT (ms):                        153.75
P99 TTFT (ms):                           268.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.42
Median TPOT (ms):                        15.59
P99 TPOT (ms):                           16.07
---------------Inter-Token Latency----------------
Mean ITL (ms):                           15.39
Median ITL (ms):                         15.17
P95 ITL (ms):                            16.62
P99 ITL (ms):                            18.13
Max ITL (ms):                            226.59
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  589.31
Total input tokens:                      158939
Total input text tokens:                 158939
Total input vision tokens:               0
Total generated tokens:                  1300705
Total generated tokens (retokenized):    1297658
Request throughput (req/s):              0.54
Input token throughput (tok/s):          269.70
Output token throughput (tok/s):         2207.16
Peak output token throughput (tok/s):    2944.00
Peak concurrent requests:                68
Total token throughput (tok/s):          2476.86
Concurrency:                             57.03
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   105032.36
Median E2E Latency (ms):                 108229.09
---------------Time to First Token----------------
Mean TTFT (ms):                          223.91
Median TTFT (ms):                        158.15
P99 TTFT (ms):                           474.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.94
Median TPOT (ms):                        26.72
P99 TPOT (ms):                           27.99
---------------Inter-Token Latency----------------
Mean ITL (ms):                           25.79
Median ITL (ms):                         25.37
P95 ITL (ms):                            26.70
P99 ITL (ms):                            105.49
Max ITL (ms):                            237.91
==================================================
```

**Scenario 3: Summarization (8K/1K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  40.65
Total input tokens:                      41941
Total input text tokens:                 41941
Total input vision tokens:               0
Total generated tokens:                  4210
Total generated tokens (retokenized):    4195
Request throughput (req/s):              0.25
Input token throughput (tok/s):          1031.65
Output token throughput (tok/s):         103.56
Peak output token throughput (tok/s):    110.00
Peak concurrent requests:                2
Total token throughput (tok/s):          1135.20
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4063.62
Median E2E Latency (ms):                 3296.13
---------------Time to First Token----------------
Mean TTFT (ms):                          165.91
Median TTFT (ms):                        154.96
P99 TTFT (ms):                           240.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.26
Median TPOT (ms):                        9.27
P99 TPOT (ms):                           9.42
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.28
Median ITL (ms):                         9.28
P95 ITL (ms):                            9.66
P99 ITL (ms):                            9.83
Max ITL (ms):                            14.06
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  56.71
Total input tokens:                      300020
Total input text tokens:                 300020
Total input vision tokens:               0
Total generated tokens:                  41589
Total generated tokens (retokenized):    41490
Request throughput (req/s):              1.41
Input token throughput (tok/s):          5290.75
Output token throughput (tok/s):         733.41
Peak output token throughput (tok/s):    1024.00
Peak concurrent requests:                20
Total token throughput (tok/s):          6024.16
Concurrency:                             14.25
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   10098.99
Median E2E Latency (ms):                 10623.46
---------------Time to First Token----------------
Mean TTFT (ms):                          486.80
Median TTFT (ms):                        189.59
P99 TTFT (ms):                           2138.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          19.06
Median TPOT (ms):                        19.23
P99 TPOT (ms):                           30.69
---------------Inter-Token Latency----------------
Mean ITL (ms):                           18.53
Median ITL (ms):                         15.63
P95 ITL (ms):                            16.64
P99 ITL (ms):                            109.71
Max ITL (ms):                            1471.36
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-R1-0528 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  115.55
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total input vision tokens:               0
Total generated tokens:                  169680
Total generated tokens (retokenized):    169275
Request throughput (req/s):              2.77
Input token throughput (tok/s):          11024.93
Output token throughput (tok/s):         1468.50
Peak output token throughput (tok/s):    2254.00
Peak concurrent requests:                70
Total token throughput (tok/s):          12493.43
Concurrency:                             59.45
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   21465.98
Median E2E Latency (ms):                 20686.26
---------------Time to First Token----------------
Mean TTFT (ms):                          913.93
Median TTFT (ms):                        224.92
P99 TTFT (ms):                           6257.83
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          39.93
Median TPOT (ms):                        40.99
P99 TPOT (ms):                           60.91
---------------Inter-Token Latency----------------
Mean ITL (ms):                           38.83
Median ITL (ms):                         26.29
P95 ITL (ms):                            113.81
P99 ITL (ms):                            176.94
Max ITL (ms):                            5521.53
==================================================
```

#### 5.1.5 Understanding the Results

**Key Metrics:**

* **Request Throughput (req/s)**: Number of requests processed per second
* **Output Token Throughput (tok/s)**: Total tokens generated per second
* **Mean TTFT (ms)**: Time to First Token - measures responsiveness
* **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
* **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency

**Why These Configurations Matter:**

* **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
* **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
* **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks.
* **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.

**Interpreting Results:**

* Compare your results against baseline numbers for your hardware
* Higher throughput at same latency = better performance
* Lower TTFT = more responsive user experience
* Lower TPOT = faster generation speed

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319 \
  --num-threads 1316
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```


# DeepSeek-V3
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3



## 1. Model Introduction

[DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) is a large-scale Mixture-of-Experts (MoE) language model developed by DeepSeek, designed to deliver strong general-purpose reasoning, coding, and tool-augmented capabilities with high training and inference efficiency. As the latest generation in the DeepSeek model family, DeepSeek V3 introduces systematic architectural and training innovations that significantly improve performance across reasoning, mathematics, coding, and long-context understanding, while maintaining a competitive compute cost.

Key highlights include:

* **Efficient MoE architecture**: DeepSeek V3 adopts a fine-grained Mixture-of-Experts design with a large number of experts and sparse activation, enabling high model capacity while keeping inference and training costs manageable.
* **Advanced reasoning and coding**: The model demonstrates strong performance on mathematical reasoning, logical inference, and real-world coding benchmarks, benefiting from improved data curation and training strategies.
* **Long-context capability**: DeepSeek V3 supports extended context lengths, allowing it to handle long documents, complex multi-step reasoning, and agent-style workflows more effectively.
* **Tool use and function calling**: The model is trained to support structured outputs and tool invocation, enabling seamless integration with external tools and agent frameworks during inference.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

**Recommended GPU configurations by weight type:**

<table>
  <thead>
    <tr>
      <th>Weight Type</th>
      <th>Supported Hardware</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**FP8** (recommended)</td>
      <td>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
    </tr>

    <tr>
      <td>**BF16** (upcast from FP8)</td>
      <td>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
    </tr>

    <tr>
      <td>**INT8**</td>
      <td>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series</td>
    </tr>

    <tr>
      <td>**W4A8 / AWQ / MXFP4 / NVFP4**</td>
      <td>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
    </tr>
  </tbody>
</table>

> The official DeepSeek-V3 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.

**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (\~10 min):

```bash theme={null}
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code
```

DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.

**Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.

**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.

**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [Basic API Usage](../../../docs/get-started/quickstart)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

DeepSeek-V3 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3 \
  --reasoning-parser deepseek-v3 \
  --tp 8
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    extra_body = {"chat_template_kwargs": {"thinking": True}},
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To determine 15% of a number, follow these steps:

**Step 1: Understand the Problem**
You need to find 15% of a given number. Let's assume the number is 240 for this example.

**Step 2: Convert the Percentage to a Decimal**
To work with percentages in calculations, convert the percentage to its decimal form. To do this, divide the percentage by 100.

\[ 15\% = \frac{15}{100} = 0.15 \]

**Step 3: Multiply the Decimal by the Number**
Now, multiply the decimal form of the percentage by the number you want to find the percentage of.

\[ 0.15 \times 240 \]

**Step 4: Perform the Multiplication**
Calculate the product:

\[ 0.15 \times 240 = 36 \]

**Step 5: Conclusion**
Therefore, 15% of 240 is:

\boxed{36}

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

DeepSeek-V3 supports tool calling capabilities. Enable the tool call parser:

**Deployment Command:**

```shell Command theme={null}
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3 \
  --tool-call-parser deepseekv3 \
  --reasoning-parser deepseek-v3 \
  --chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

**Quick Test (curl):**

```shell Command theme={null}
curl "http://127.0.0.1:30000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "temperature": 0,
    "max_tokens": 100,
    "model": "deepseek-ai/DeepSeek-V3",
    "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of a city", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}],
    "messages": [{"role": "user", "content": "How'\''s the weather in Beijing today?"}]
  }'
```

> [!NOTE]
>
> Use a low `temperature` (e.g. `0`) for more consistent tool call results. The `--chat-template` flag above provides an improved unified prompt for tool use.

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    extra_body = {"chat_template_kwargs": {"thinking": True}},
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

Please attach the code blocks below to the previous Python script.

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)

SGLang implements DeepSeek V3 Multi-Token Prediction (MTP) based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). With this optimization, decoding speed improves by up to **1.8×** at batch size 1 and **1.5×** at batch size 32 on H200 TP8.

**Enable with:**

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --speculative-algorithm EAGLE \
  --trust-remote-code \
  --tp 8
```

The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`.

> [!NOTE]
>
> For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs-decode` to include your target batch sizes (default captured sizes for speculative decoding: 48).

> [!TIP]
>
> The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable.

#### 4.2.4 MLA Optimizations

DeepSeek V3 uses [Multi-head Latent Attention (MLA)](https://arxiv.org/pdf/2405.04434), an attention mechanism that improves inference efficiency. SGLang implements several optimizations:

* **Weight Absorption:** Reorders matrix multiplications to improve decoding phase efficiency.
* **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default.
* **FP8 Quantization:** W8A8 FP8 and KV Cache FP8, with BMM operators for weight-absorbed MLA in FP8.
* **CUDA Graph & Torch.compile:** Both MLA and MoE support CUDA Graph and Torch.compile for reduced decoding latency.
* **Chunked Prefix Cache:** Increases throughput for long-sequence chunked prefill (FlashAttention3 backend only).

Overall, these optimizations achieve up to **7×** output throughput improvement vs. the baseline.

**Reference:** See [SGLang v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) and [Slides](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/lmsys_1st_meetup_deepseek_mla.pdf) for details.

#### 4.2.5 Multi-Node Deployment

For multi-node serving and hardware-specific examples:

* [8× H200 / 4–8× B200](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#using-docker-recommended)
* [8× MI300X](../../../docs/hardware-platforms/amd_gpu#running-deepseek-v3)
* [2×8× H200 with Docker](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-two-h2008-nodes-and-docker)
* [4×8× A100](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-four-a1008-nodes)
* [8× A100 AWQ](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-8-a100a800-with-awq-quantization)
* [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
* [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
* [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus)
* [4× Ascend A3 Series (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment)

**Blog references for large-scale deployment:**

* [Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP](https://lmsys.org/blog/2025-06-16-gb200-part-1/) ([Part I](https://lmsys.org/blog/2025-06-16-gb200-part-1/), [Part II](https://lmsys.org/blog/2025-09-25-gb200-part-2/))
* [PD Disaggregation and Large-Scale Expert Parallelism on 96× H100](https://lmsys.org/blog/2025-05-05-large-scale-ep/)
* [Best Practices for Serving DeepSeek-R1 on H20](https://lmsys.org/blog/2025-09-26-sglang-ant-group/)

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: AMD MI300X GPU (8x)
* Model: DeepSeek-V3
* Tensor Parallelism: 8
* sglang version: 0.5.7

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3 \
  --tp 8 \
  --attn-dp-size 8 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 0.0.0.0 \
  --port 8000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 8000 \
  --model deepseek-ai/DeepSeek-V3 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  81.27
Total input tokens:                      1972
Total input text tokens:                 1972
Total input vision tokens:               0
Total generated tokens:                  2784
Total generated tokens (retokenized):    2774
Request throughput (req/s):              0.12
Input token throughput (tok/s):          24.27
Output token throughput (tok/s):         34.26
Peak output token throughput (tok/s):    65.00
Peak concurrent requests:                2
Total token throughput (tok/s):          58.52
Concurrency:                             1.00
Accept length:                           2.61
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8123.17
Median E2E Latency (ms):                 7982.65
---------------Time to First Token----------------
Mean TTFT (ms):                          1080.76
Median TTFT (ms):                        1248.82
P99 TTFT (ms):                           1896.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.04
Median TPOT (ms):                        24.76
P99 TPOT (ms):                           32.09
---------------Inter-Token Latency----------------
Mean ITL (ms):                           25.41
Median ITL (ms):                         20.14
P95 ITL (ms):                            60.28
P99 ITL (ms):                            60.99
Max ITL (ms):                            61.49
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3 \
  --tp 8 \
  --ep 8 \
  --attn-dp-size 8 \
  --host 0.0.0.0 \
  --port 8000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 8000 \
  --model deepseek-ai/DeepSeek-V3 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  406.16
Total input tokens:                      301701
Total input text tokens:                 301701
Total input vision tokens:               0
Total generated tokens:                  188375
Total generated tokens (retokenized):    187542
Request throughput (req/s):              2.46
Input token throughput (tok/s):          742.81
Output token throughput (tok/s):         463.80
Peak output token throughput (tok/s):    1299.00
Peak concurrent requests:                109
Total token throughput (tok/s):          1206.61
Concurrency:                             87.53
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   35552.98
Median E2E Latency (ms):                 21466.07
---------------Time to First Token----------------
Mean TTFT (ms):                          1521.51
Median TTFT (ms):                        476.80
P99 TTFT (ms):                           8329.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          214.73
Median TPOT (ms):                        152.00
P99 TPOT (ms):                           1155.85
---------------Inter-Token Latency----------------
Mean ITL (ms):                           182.10
Median ITL (ms):                         79.18
P95 ITL (ms):                            398.60
P99 ITL (ms):                            1488.96
Max ITL (ms):                            43465.60
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:8000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Test Results**:
  * DeepSeek-V3
    ```text Output theme={null}
    Accuracy: 0.960
    Invalid: 0.000
    Latency: 32.450 s
    Output throughput: 614.211 token/s
    ```

#### 5.2.2 MMLU Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:8000/v1 \
  --num-threads 64 --max-tokens 2048
```

The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above.

* **Test Results**:
  * DeepSeek-V3
    ```text Output theme={null}
    subject: abstract_algebra, #q:100, acc: 0.800
    subject: anatomy, #q:135, acc: 0.874
    subject: astronomy, #q:152, acc: 0.928
    subject: business_ethics, #q:100, acc: 0.880
    subject: clinical_knowledge, #q:265, acc: 0.928
    subject: college_biology, #q:144, acc: 0.965
    subject: college_chemistry, #q:100, acc: 0.670
    subject: college_computer_science, #q:100, acc: 0.840
    subject: college_mathematics, #q:100, acc: 0.800
    subject: college_medicine, #q:173, acc: 0.861
    Total latency: 58.339
    Average accuracy: 0.871
    ```


# DeepSeek-V3.1
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1



## 1. Model Introduction

[DeepSeek V3.1](https://huggingface.co/deepseek-ai/DeepSeek-V3.1) is an advanced Mixture-of-Experts (MoE) large language model developed by DeepSeek, representing a major capability and usability upgrade over DeepSeek V3. As a refined iteration in the DeepSeek V3 family, DeepSeek V3.1 introduces a hybrid reasoning paradigm that supports both fast non-thinking responses and explicit multi-step reasoning, alongside significantly improved tool calling and agentic behavior. The model demonstrates strong performance across reasoning, mathematics, coding, long-context understanding, and real-world agent workflows, benefiting from continued training, alignment optimization, and inference-time refinements. DeepSeek V3.1 is designed to serve as a robust general-purpose foundation model, well suited for conversational AI, structured tool invocation, search-augmented generation, and complex multi-step tasks, while maintaining high efficiency through its sparse MoE architecture.

**[DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus)** is an experimental version designed for general conversations and long-context processing. It features hybrid thinking capabilities, allowing you to toggle between "Think" mode for deliberate reasoning and "Non-Think" mode for faster responses. Recommended for general conversations, long-context processing, and experimental use cases.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same hardware and optimization recommendations apply.

**Recommended GPU configurations by weight type:**

<table>
  <thead>
    <tr>
      <th>Weight Type</th>
      <th>Supported Hardware</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**FP8** (recommended)</td>
      <td>8× H200, 8× B200, 8× MI300X, 2×8× H100/H800/H20</td>
    </tr>

    <tr>
      <td>**BF16** (upcast from FP8)</td>
      <td>2×8× H200, 2×8× MI300X, 4×8× H100/H800, 4×8× A100/A800</td>
    </tr>

    <tr>
      <td>**INT8**</td>
      <td>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series</td>
    </tr>

    <tr>
      <td>**W4A8 / AWQ / MXFP4 / NVFP4**</td>
      <td>8× H20/H100, 4× H200; 8× H100/A100; 8/4× MI355X/MI350X; 8/4× B200</td>
    </tr>
  </tbody>
</table>

> The official DeepSeek-V3.1 checkpoint is already in FP8 format — do **not** add `--quantization fp8` when serving it.

**DeepGEMM precompilation (NVIDIA Hopper / Blackwell):** Precompile GEMM kernels before the first server run to avoid JIT overhead (\~10 min):

```bash theme={null}
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --trust-remote-code
```

DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.

**Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.

**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.

**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [Basic API Usage](../../../docs/get-started/quickstart)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

DeepSeek-V3.1 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3.1-Terminus \
  --reasoning-parser deepseek-v3 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1-Terminus",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    extra_body = {"chat_template_kwargs": {"thinking": True}},
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
First, the problem is asking for 15% of 240. Percent means per hundred, so 15% is the same as 15 out of 100, or 15/100.

To find a percentage of a number, I can multiply the number by the percentage expressed as a decimal. So, I need to convert 15% to a decimal. To do that, I divide 15 by 100, which gives me 0.15.

Now, I multiply 0.15 by 240. So, the calculation is 0.15 × 240.

I can compute this step by step. First, I know that 15% of 100 is 15, but since 240 is larger, I need to adjust. Alternatively, I can think of 10% of 240, which is easy because 10% is just 240 divided by 10, which is 24. Then, 5% is half of 10%, so half of 24 is 12. Therefore, 15% is 10% plus 5%, so 24 plus 12, which equals 36.

I should also do the multiplication to confirm. 0.15 × 240. I can break it down: 0.15 × 200 = 30, and 0.15 × 40 = 6, so 30 + 6 = 36. Same answer.

So, 15% of 240 is 36.

The problem says "step by step," so I should present it clearly.
=============== Content =================
To find 15% of 240, follow these steps:

1. Understand that "percent" means "per hundred," so 15% is equivalent to \( \frac{15}{100} \).
2. Convert 15% to a decimal by dividing by 100: \( 15\% = \frac{15}{100} = 0.15 \).
3. Multiply the decimal by 240: \( 0.15 \times 240 \).
4. Perform the multiplication:
   - \( 0.15 \times 200 = 30 \)
   - \( 0.15 \times 40 = 6 \)
   - Add the results: \( 30 + 6 = 36 \).

Alternatively, you can find 15% by breaking it into parts:
- 10% of 240 is \( \frac{10}{100} \times 240 = 0.10 \times 240 = 24 \).
- 5% of 240 is half of 10%, so \( \frac{24}{2} = 12 \).
- Add 10% and 5%: \( 24 + 12 = 36 \).

Thus, 15% of 240 is 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

DeepSeek-V3.1 and DeepSeek-V3.1-Terminus support tool calling capabilities. Enable the tool call parser:

**Deployment Command:**

```shell Command theme={null}
python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3.1-Terminus \
  --tool-call-parser deepseekv31 \
  --reasoning-parser deepseek-v3 \
  --chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

For DeepSeek-V3.1, use `--tool-call-parser deepseekv31` as well.

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1-Terminus",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    extra_body = {"chat_template_kwargs": {"thinking": True}},
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
Hmm, the user is asking for the weather in Beijing. This is a straightforward request that matches exactly what the weather tool can provide.

I need to call the get_weather function with Beijing as the location parameter. The user didn't specify a temperature unit, so I'll default to Celsius since that's commonly used in most parts of the world.

The tool call format needs to be precise - just the city name and unit selection. Once I get the weather data back, I'll present it clearly to the user.I'll check the weather in Beijing for you.
=============== Content =================

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

Please attach the code blocks below to the previous Python script.

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.1-Terminus",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "Currently, it is **22°C and sunny** in Beijing."
```

#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)

DeepSeek-V3.1 shares the same architecture as DeepSeek-V3 and supports the same EAGLE-based MTP speculative decoding path. Refer to [DeepSeek-V3 §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-3-multi-token-prediction-eagle-speculative-decoding) for the full configuration, tuning guidance, and `bench_speculative.py` reference. The `--speculative-num-steps`, `--speculative-eagle-topk`, and `--max-running-requests` recommendations apply equally to V3.1.

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: AMD MI300X GPU (8x)
* Model: DeepSeek-V3.1-Terminus
* Tensor Parallelism: 8
* sglang version: 0.5.7

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Test Scenarios

Three core scenarios reflect real-world usage patterns:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Scenario</th>
      <th>Input Length</th>
      <th>Output Length</th>
      <th>Use Case</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Chat**</td>
      <td>1K</td>
      <td>1K</td>
      <td>Most common conversational AI workload</td>
    </tr>

    <tr>
      <td>**Reasoning**</td>
      <td>1K</td>
      <td>8K</td>
      <td>Long-form generation, complex reasoning tasks</td>
    </tr>

    <tr>
      <td>**Summarization**</td>
      <td>8K</td>
      <td>1K</td>
      <td>Document summarization, RAG retrieval</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 Concurrency Levels

Test each scenario at different concurrency levels to capture the throughput vs. latency trade-off:

* **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
* **Medium Concurrency**: `--max-concurrency 16` (Balanced)
* **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)

#### 5.1.3 Number of Prompts

For each concurrency level, configure `num_prompts` to simulate realistic user loads:

* **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
* **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
* **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)

***

#### 5.1.4 Benchmark Commands

**Scenario 1: Chat (1K/1K) - Most Important**

* **Model Deployment**

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.1 \
  --tp 8
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  106.24
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4201
Request throughput (req/s):              0.09
Input token throughput (tok/s):          57.43
Output token throughput (tok/s):         39.72
Peak output token throughput (tok/s):    43.00
Peak concurrent requests:                2
Total token throughput (tok/s):          97.15
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   10620.29
Median E2E Latency (ms):                 8868.09
---------------Time to First Token----------------
Mean TTFT (ms):                          557.85
Median TTFT (ms):                        213.58
P99 TTFT (ms):                           1625.28
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.84
Median TPOT (ms):                        23.90
P99 TPOT (ms):                           24.03
---------------Inter-Token Latency----------------
Mean ITL (ms):                           23.90
Median ITL (ms):                         23.92
P95 ITL (ms):                            24.15
P99 ITL (ms):                            24.25
Max ITL (ms):                            25.44
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  107.71
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40625
Request throughput (req/s):              0.74
Input token throughput (tok/s):          368.28
Output token throughput (tok/s):         378.84
Peak output token throughput (tok/s):    508.00
Peak concurrent requests:                19
Total token throughput (tok/s):          747.12
Concurrency:                             13.72
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   18473.65
Median E2E Latency (ms):                 19558.42
---------------Time to First Token----------------
Mean TTFT (ms):                          607.91
Median TTFT (ms):                        191.32
P99 TTFT (ms):                           2135.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          35.50
Median TPOT (ms):                        35.99
P99 TPOT (ms):                           43.62
---------------Inter-Token Latency----------------
Mean ITL (ms):                           35.10
Median ITL (ms):                         32.18
P95 ITL (ms):                            33.03
P99 ITL (ms):                            159.99
Max ITL (ms):                            453.99
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  207.65
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    251238
Request throughput (req/s):              2.41
Input token throughput (tok/s):          1203.15
Output token throughput (tok/s):         1216.79
Peak output token throughput (tok/s):    2100.00
Peak concurrent requests:                106
Total token throughput (tok/s):          2419.94
Concurrency:                             91.02
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   37800.20
Median E2E Latency (ms):                 35921.56
---------------Time to First Token----------------
Mean TTFT (ms):                          835.15
Median TTFT (ms):                        236.88
P99 TTFT (ms):                           2868.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          73.33
Median TPOT (ms):                        76.35
P99 TPOT (ms):                           97.63
---------------Inter-Token Latency----------------
Mean ITL (ms):                           73.30
Median ITL (ms):                         50.82
P95 ITL (ms):                            180.67
P99 ITL (ms):                            186.83
Max ITL (ms):                            1661.39
==================================================
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  1097.29
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  44462
Total generated tokens (retokenized):    44313
Request throughput (req/s):              0.01
Input token throughput (tok/s):          5.56
Output token throughput (tok/s):         40.52
Peak output token throughput (tok/s):    43.00
Peak concurrent requests:                2
Total token throughput (tok/s):          46.08
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   109725.52
Median E2E Latency (ms):                 117748.67
---------------Time to First Token----------------
Mean TTFT (ms):                          156.67
Median TTFT (ms):                        156.19
P99 TTFT (ms):                           159.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.41
Median TPOT (ms):                        24.51
P99 TPOT (ms):                           24.96
---------------Inter-Token Latency----------------
Mean ITL (ms):                           24.65
Median ITL (ms):                         24.58
P95 ITL (ms):                            25.68
P99 ITL (ms):                            25.93
Max ITL (ms):                            29.80
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  775.02
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  318306
Total generated tokens (retokenized):    317426
Request throughput (req/s):              0.10
Input token throughput (tok/s):          51.18
Output token throughput (tok/s):         410.70
Peak output token throughput (tok/s):    512.00
Peak concurrent requests:                18
Total token throughput (tok/s):          461.89
Concurrency:                             13.86
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   134236.65
Median E2E Latency (ms):                 135181.28
---------------Time to First Token----------------
Mean TTFT (ms):                          214.35
Median TTFT (ms):                        194.12
P99 TTFT (ms):                           300.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.72
Median TPOT (ms):                        34.00
P99 TPOT (ms):                           34.75
---------------Inter-Token Latency----------------
Mean ITL (ms):                           33.69
Median ITL (ms):                         33.71
P95 ITL (ms):                            34.50
P99 ITL (ms):                            34.92
Max ITL (ms):                            164.76
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  1231.97
Total input tokens:                      158939
Total input text tokens:                 158939
Total input vision tokens:               0
Total generated tokens:                  1301025
Total generated tokens (retokenized):    1296845
Request throughput (req/s):              0.26
Input token throughput (tok/s):          129.01
Output token throughput (tok/s):         1056.05
Peak output token throughput (tok/s):    1472.00
Peak concurrent requests:                67
Total token throughput (tok/s):          1185.07
Concurrency:                             56.17
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   216256.25
Median E2E Latency (ms):                 224192.84
---------------Time to First Token----------------
Mean TTFT (ms):                          317.68
Median TTFT (ms):                        235.28
P99 TTFT (ms):                           649.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          53.30
Median TPOT (ms):                        55.10
P99 TPOT (ms):                           56.58
---------------Inter-Token Latency----------------
Mean ITL (ms):                           53.13
Median ITL (ms):                         52.95
P95 ITL (ms):                            56.23
P99 ITL (ms):                            181.04
Max ITL (ms):                            208.61
==================================================
```

**Scenario 3: Summarization (8K/1K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  114.47
Total input tokens:                      41941
Total input text tokens:                 41941
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4194
Request throughput (req/s):              0.09
Input token throughput (tok/s):          366.39
Output token throughput (tok/s):         36.87
Peak output token throughput (tok/s):    42.00
Peak concurrent requests:                2
Total token throughput (tok/s):          403.26
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   11442.86
Median E2E Latency (ms):                 9508.87
---------------Time to First Token----------------
Mean TTFT (ms):                          883.78
Median TTFT (ms):                        481.38
P99 TTFT (ms):                           2217.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.93
Median TPOT (ms):                        25.05
P99 TPOT (ms):                           26.11
---------------Inter-Token Latency----------------
Mean ITL (ms):                           25.08
Median ITL (ms):                         25.08
P95 ITL (ms):                            26.18
P99 ITL (ms):                            26.28
Max ITL (ms):                            27.41
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  162.33
Total input tokens:                      300020
Total input text tokens:                 300020
Total input vision tokens:               0
Total generated tokens:                  41669
Total generated tokens (retokenized):    41443
Request throughput (req/s):              0.49
Input token throughput (tok/s):          1848.27
Output token throughput (tok/s):         256.70
Peak output token throughput (tok/s):    467.00
Peak concurrent requests:                19
Total token throughput (tok/s):          2104.97
Concurrency:                             14.52
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   29456.89
Median E2E Latency (ms):                 27628.16
---------------Time to First Token----------------
Mean TTFT (ms):                          1784.30
Median TTFT (ms):                        1347.21
P99 TTFT (ms):                           5384.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          53.65
Median TPOT (ms):                        52.09
P99 TPOT (ms):                           74.39
---------------Inter-Token Latency----------------
Mean ITL (ms):                           53.23
Median ITL (ms):                         34.52
P95 ITL (ms):                            35.81
P99 ITL (ms):                            513.25
Max ITL (ms):                            2865.73
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model deepseek-ai/DeepSeek-V3.1 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  282.55
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total input vision tokens:               0
Total generated tokens:                  170000
Total generated tokens (retokenized):   169081
Request throughput (req/s):              1.13
Input token throughput (tok/s):          4508.6
Output token throughput (tok/s):         601.67
Peak output token throughput (tok/s):   1216
Peak concurrent requests:                68
Total token throughput (tok/s):         5110.27
Concurrency:                            59.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                  52810.32
Median E2E Latency (ms):                50981.81
---------------Time to First Token----------------
Mean TTFT (ms):                         786.69
Median TTFT (ms):                       499.38
P99 TTFT (ms):                          2925.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                         97.93
Median TPOT (ms):                       103.45
P99 TPOT (ms):                          157.84
---------------Inter-Token Latency----------------
Mean ITL (ms):                          98.11
Median ITL (ms):                        55.7
P95 ITL (ms):                           240.71
P99 ITL (ms):                          1114.36
==================================================
```

#### 5.1.5 Understanding the Results

**Key Metrics:**

* **Request Throughput (req/s)**: Number of requests processed per second
* **Output Token Throughput (tok/s)**: Total tokens generated per second
* **Mean TTFT (ms)**: Time to First Token - measures responsiveness
* **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
* **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency

**Why These Configurations Matter:**

* **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
* **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
* **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks.
* **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.

**Interpreting Results:**

* Compare your results against baseline numbers for your hardware
* Higher throughput at same latency = better performance
* Lower TTFT = more responsive user experience
* Lower TPOT = faster generation speed

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319 \
  --num-threads 1316
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```


# DeepSeek-V3.2
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2



## 1. Model Introduction

The DeepSeek-V3.2 series includes three model variants, each optimized for different use cases:

**[DeepSeek-V3.2-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)** is an upgraded version of DeepSeek-V3.1-Terminus, introducing the DeepSeek Sparse Attention (DSA) mechanism through continued training. DSA is a fine-grained sparse attention mechanism powered by a lightning indexer, enabling DeepSeek-V3.2-Exp to achieve significant efficiency improvements in long-context scenarios. Recommended for general conversations, long-context processing, and efficient inference.

**[DeepSeek-V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)** is the standard version suitable for general tasks and conversational scenarios. For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top\_p = 0.95. Recommended for standard conversations and general tasks.

**[DeepSeek-V3.2-Speciale](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Speciale)** is a special variant designed exclusively for deep reasoning tasks. This model is specifically optimized for scenarios requiring complex logical reasoning and deep thinking. However this model does not support tool calls (see below). For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top\_p = 0.95. Recommended for deep reasoning tasks, complex logical problems, and mathematical reasoning.

**[DeepSeek-V3.2-NVFP4](https://huggingface.co/nvidia/DeepSeek-V3.2-NVFP4)** is an NVIDIA-optimized NVFP4-quantized variant of DeepSeek-V3.2 for Blackwell devices. It uses ModelOpt FP4 quantization with a choice of MoE runner backends (`flashinfer_trtllm` (recommended), `flashinfer_cutlass`, or `flashinfer_cutedsl`), enabling efficient deployment with lower tensor parallelism (TP=4). It supports the same features as DeepSeek-V3.2 including tool calling, reasoning, and speculative decoding (MTP).

**[DeepSeek-V3.2-MXFP4](https://huggingface.co/amd/DeepSeek-V3.2-mxfp4)** is an OCP-MXFP4 optimized variant for DeepSeek-V3.2 for AMD MI300X/MI355X devices. It uses OCP MXFP4 quantization with a triton mxfp4 backend (the same backend for gptoss-120B), enabling efficient deployment with lower tensor parallelism (TP=8) in a single node. It includes the same features as DeepSeek-V3.2 including tool calling, reasoning, fp8-kv, CP, TP and speculative decoding MTP.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

### 2.1 Docker Images

Pre-built Docker images are available for different hardware platforms:

```bash Command theme={null}
# NVIDIA H200 / B200
docker pull lmsysorg/sglang:latest

# AMD MI350 / MI355X
docker pull lmsysorg/sglang:v0.5.8-rocm700-mi35x

# AMD MI300X
# Note: v0.5.8-rocm700-mi30x does not include PR #17504.
# Prefer the newest MI30x ROCm image tag from Docker Hub when available, or build from source.
docker pull lmsysorg/sglang:v0.5.8-rocm700-mi30x

# Ascend NPU (A2/A3 Series)
docker pull lmsysorg/sglang:dsv32-a2
docker pull lmsysorg/sglang:dsv32-a3
```

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities. SGLang supports serving DeepSeek V3.2 on NVIDIA H200, B200, and AMD MI300X/MI355X GPUs.

Run this Command:

<pre />

> [!WARNING]
>
> All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on DeepSeek-V3.2.

### 3.2 Configuration Tips

* **Short-sequence MHA prefill (adaptive):** For prefill sequences shorter than 2048 tokens (default threshold), the DSA backend automatically switches to standard MHA (using FlashAttention variable-length on SM90, TRT-LLM ragged MHA on SM100). To extend this to longer sequences set env var `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` to a larger value (potential minor accuracy trade-off).
* **DSA prefill/decode attention kernels (`--dsa-prefill-backend`, `--dsa-decode-backend`):** The `dsa` backend is automatically selected for DeepSeek-V3.2. Available kernels: `flashmla_sparse`, `flashmla_sparse_q8` (native FP8 e4m3 sparse prefill — no fp8→bf16 dequantization round-trip; Hopper SM90 + `--kv-cache-dtype fp8_e4m3` only, prefill only), `flashmla_kv`, `flashmla_auto`, `fa3` (Hopper only), `tilelang` (GPU/HPU/NPU), `aiter` (AMD, decode only), `trtllm` (Blackwell only). Defaults: Hopper BF16 KV → `flashmla_sparse` prefill / `fa3` decode; Hopper FP8 KV → `flashmla_kv` both; Blackwell BF16 → `flashmla_sparse` / `trtllm`; Blackwell FP8 → `trtllm` both.
* **Index Cache:** Reuses indexer results across layers for efficiency at negligible accuracy cost. For **GLM-5** specifically, append `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for a better speed/accuracy tradeoff.
* **HiSparse (experimental):** Reduces per-request GPU memory during long-context decode by offloading KV data to CPU pinned memory. Requires PD disaggregation mode (decode instance only). See [HiSparse Guide](../../../docs/advanced_features/hisparse_guide).
* **NVFP4 on Blackwell:** Specify `--quantization modelopt_fp4` and `--moe-runner-backend flashinfer_trtllm` (recommended) / `flashinfer_cutlass` / `flashinfer_cutedsl`. Full example:
  ```bash theme={null}
  python -m sglang.launch_server --model nvidia/DeepSeek-V3.2-NVFP4 --tp 4 \
    --quantization modelopt_fp4 --moe-runner-backend flashinfer_trtllm \
    --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3
  ```
* **NCCL timeout:** Slow model loading → add `--dist-timeout 3600`.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [Basic API Usage](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

DeepSeek-V3.2 supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
sglang serve \
  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
  --reasoning-parser deepseek-v3 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.2-Exp",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    extra_body = {"chat_template_kwargs": {"thinking": True}},
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

DeepSeek-V3.2 and DeepSeek-V3.2-Exp support tool calling capabilities. But they use different parameters. Enable the tool call parser:

**Note:** DeepSeek-V3.2-Speciale does **NOT** support tool calling. Launch it with reasoning parser only:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2-Speciale \
  --trust-remote-code \
  --tp-size 8 --attn-dp-size 8 \
  --reasoning-parser deepseek-v3
```

**Deployment Command:**

For DeepSeek-V3.2-Exp:

```shell Command theme={null}
sglang serve \
  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
  --tool-call-parser deepseekv31 \
  --reasoning-parser deepseek-v3 \
  --chat-template ./examples/chat_template/tool_chat_template_deepseekv32.jinja \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

For DeepSeek-V3.2, use `--tool-call-parser deepseekv32` and remove `--chat-template`.

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.2-Exp",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    extra_body = {"chat_template_kwargs": {"thinking": True}},
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V3.2-Exp",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

#### 4.2.3 Multi-Token Prediction (EAGLE Speculative Decoding)

SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding). This optimization significantly improves decoding speed for small batch sizes.

**With DP Attention:**

```bash Command theme={null}
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --attn-dp-size 8 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
```

**With Pure TP:**

```bash Command theme={null}
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
```

Find optimal values for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`.

> [!NOTE]
>
> `--max-running-requests` defaults to 48 for MTP. Increase it for larger batch sizes.

> [!TIP]
>
> The spec-v2 overlap scheduler is enabled by default. Pass `--disable-overlap-schedule` to disable.

#### 4.2.4 PD Disaggregation

Prefill-Decode (PD) disaggregation separates prefill and decode stages onto different instances, improving GPU utilization for mixed workloads.

**Prefill command:**

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
  --disaggregation-mode prefill \
  --host $LOCAL_IP \
  --port $PORT \
  --tp 8 \
  --attn-dp-size 8 \
  --dist-init-addr ${HOST}:${DIST_PORT} \
  --trust-remote-code \
  --disaggregation-bootstrap-port 8998 \
  --mem-fraction-static 0.9
```

**Decode command:**

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
  --disaggregation-mode decode \
  --host $LOCAL_IP \
  --port $PORT \
  --tp 8 \
  --attn-dp-size 8 \
  --dist-init-addr ${HOST}:${DIST_PORT} \
  --trust-remote-code \
  --mem-fraction-static 0.9
```

**Router command:**

```bash Command theme={null}
python -m sglang_router.launch_router --pd-disaggregation \
  --prefill $PREFILL_ADDR 8998 \
  --decode $DECODE_ADDR \
  --host 127.0.0.1 \
  --port 30000
```

For production deployments (RBG / LWS-based, DeepEP EP parallelism), see [multi\_node\_deployment docs](../../../docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd).

#### 4.2.5 DSA Long-Sequence Context Parallel and PP/CP

Enable prefill context parallelism (CP) on CUDA with `--enable-prefill-cp --cp-strategy interleave` for long-sequence workloads.

> [!WARNING]
>
> Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for DeepSeek V3.2, GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3. Use `interleave` for these models. Attention DP is supported, with `tp_size / attn_dp_size` CP ranks in each DP group. Dense FFNs retain their configured TP width; `--moe-dense-tp-size 1` selects local dense computation.

**Interleave** (`--cp-strategy interleave`): Distributes tokens by `token_idx % cp_size`. It supports fused MoE, FP8 KV cache, and multi-batch prefill.

```bash Command theme={null}
# Interleave — FusedMoE + CP8
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \
  --tp 8 --enable-prefill-cp --attn-cp-size 8 \
  --cp-strategy interleave --max-running-requests 32
```

**PP + CP (multi-node):** Combines Pipeline Parallelism and Context Parallelism for cross-node scaling. The production-optimized configurations below have been verified on Hopper:

We suggested `DP2` + `MTP` for local deployment of agentic workflow with DeepSeek V3.2 on Hopper platform:

```shell Command theme={null}
export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32
export SGLANG_SET_CPU_AFFINITY=1

# Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970
#   dp 2 : 5019.54  toks/sec, MAX ITL 7233
#   dp 4 : 4942.82  toks/sec, MAX ITL 35654
#   dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081
sglang_args=$(echo serve \
  --model-path $MAPPED_MODEL_PATH \
  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
  --attn-dp-size 2 --page-size 64 \
  --trust-remote-code --host "0.0.0.0" --port 30000 \
  --log-requests \
  --context-length 65536 --max-running-requests 128 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
  --allow-auto-truncate --enable-metrics \
  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
  --served-model-name DeepSeek-V3.2-Opt-dp2-mtp
)

sglang_args=($sglang_args)

sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```

**CP + PP + EP**

`CP` is currently enabled with `PP=2` on Hopper platform and we can reduce TP=16 to TP=8 from standalone deployment:

```shell Command theme={null}
# verified on Hopper platform
sglang_args=$(echo serve \
  --model-path $MAPPED_MODEL_PATH \
  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 \
  --moe-a2a-backend deepep --ep-size 16  \
  --page-size 128 \
  --chunked-prefill-size 16384 \
  --attention-backend dsa \
  --dsa-prefill-backend flashmla_sparse \
  --dsa-decode-backend flashmla_sparse \
  --enable-prefill-cp \
  --cp-strategy interleave \
  --cuda-graph-max-bs-decode 128 \
  --max-running-requests 128 \
  --trust-remote-code --host "0.0.0.0" --port 30000 \
  --log-requests \
  --context-length 65536 \
  --allow-auto-truncate --enable-metrics \
  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
  --served-model-name DeepSeek-V3.2-dsa-pp-cp-ep
)

sglang_args=($sglang_args)

sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```

**fp8 KV + CP + PP**

FP8 KV reduces the memory footprint. For DeepSeek V3.2, combine it with interleave CP and PP, without attention DP:

```shell Command theme={null}
# verified in Hopper platform
cp_config=" \
  --enable-prefill-cp \
  --cp-strategy interleave \
"

# see discussion : https://github.com/sgl-project/sglang/pull/12065
sglang_args=$(echo serve \
  --model-path $MAPPED_MODEL_PATH \
  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --pp-async-batch-depth 1 \
  --trust-remote-code --host "0.0.0.0" --port 30000 \
  --log-requests \
  --context-length 65536 --max-running-requests 128 \
  $cp_config \
  --kv-cache-dtype fp8_e4m3 \
  --allow-auto-truncate --enable-metrics \
  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
  --served-model-name DeepSeek-V3.2-Opt-fp8kv-pp2-cp4
)

sglang_args=($sglang_args)

sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```

## 5. Benchmark

### 5.1 Speed Benchmark on Blackwell

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8x)
* Model: DeepSeek-V3.2-Exp
* Tensor Parallelism: 8
* sglang version: 0.5.6

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios. To simulate real-world usage patterns, we configure each request with 1024 input tokens and 1024 output tokens, representing typical medium-length conversations with detailed responses.

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
  --tp 8 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model deepseek-ai/DeepSeek-V3.2-Exp \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  29.11
Total input tokens:                      1972
Total input text tokens:                 1972
Total input vision tokens:               0
Total generated tokens:                  2784
Total generated tokens (retokenized):    2777
Request throughput (req/s):              0.34
Input token throughput (tok/s):          67.73
Output token throughput (tok/s):         95.62
Peak output token throughput (tok/s):    157.00
Peak concurrent requests:                3
Total token throughput (tok/s):          163.36
Concurrency:                             1.00
Accept length:                           2.46
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2909.74
Median E2E Latency (ms):                 3088.27
P90 E2E Latency (ms):                    4200.62
P99 E2E Latency (ms):                    5588.52
---------------Time to First Token----------------
Mean TTFT (ms):                          317.58
Median TTFT (ms):                        191.31
P99 TTFT (ms):                           740.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.09
Median TPOT (ms):                        9.25
P99 TPOT (ms):                           11.73
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.35
Median ITL (ms):                         7.64
P95 ITL (ms):                            22.81
P99 ITL (ms):                            23.33
Max ITL (ms):                            31.45
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
  --tp 8 \
  --ep 8 \
  --attn-dp-size 8 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model deepseek-ai/DeepSeek-V3.2-Exp \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  219.09
Total input tokens:                      301701
Total input text tokens:                 301701
Total input vision tokens:               0
Total generated tokens:                  188375
Total generated tokens (retokenized):    187443
Request throughput (req/s):              4.56
Input token throughput (tok/s):          1377.06
Output token throughput (tok/s):         859.80
Peak output token throughput (tok/s):    2465.00
Peak concurrent requests:                109
Total token throughput (tok/s):          2236.86
Concurrency:                             88.05
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   19291.23
Median E2E Latency (ms):                 11927.39
---------------Time to First Token----------------
Mean TTFT (ms):                          530.36
Median TTFT (ms):                        444.00
P99 TTFT (ms):                           1504.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          106.16
Median TPOT (ms):                        106.69
P99 TPOT (ms):                           221.12
---------------Inter-Token Latency----------------
Mean ITL (ms):                           100.46
Median ITL (ms):                         41.73
P95 ITL (ms):                            225.67
P99 ITL (ms):                            392.37
Max ITL (ms):                            975.03
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Test Results**:
  * DeepSeek-V3.2-Exp
    ```
    Accuracy: 0.980
    Invalid: 0.000
    Latency: 19.128 s
    Output throughput: 965.919 token/s
    ```

* **Full GSM8K (1319 questions)** — evaluate the complete test set with sgl-eval:

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319 \
  --num-threads 1319
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* 8-shot:
  ```
  Accuracy: 0.956
  Invalid: 0.000
  Latency: 25.109 s
  Output throughput: 5226.235 token/s
  ```
* 20-shot (long-context; stays close to the 8-shot result):
  ```
  Accuracy: 0.956
  Invalid: 0.000
  Latency: 29.545 s
  Output throughput: 4418.617 token/s
  ```

#### 5.2.2 MMLU Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above.

* **Test Results**:
  * DeepSeek-V3.2-Exp
    ```
    subject: abstract_algebra, #q:100, acc: 0.780
    subject: anatomy, #q:135, acc: 0.874
    subject: astronomy, #q:152, acc: 0.961
    subject: business_ethics, #q:100, acc: 0.860
    subject: clinical_knowledge, #q:265, acc: 0.925
    subject: college_biology, #q:144, acc: 0.972
    subject: college_chemistry, #q:100, acc: 0.660
    subject: college_computer_science, #q:100, acc: 0.880
    subject: college_mathematics, #q:100, acc: 0.840
    subject: college_medicine, #q:173, acc: 0.879
    Total latency: 7.961
    Average accuracy: 0.879
    ```

#### 5.2.3 GPQA-Diamond Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gpqa --base-url http://127.0.0.1:30000/v1 --num-examples 198 --max-tokens 128000 --n-repeats 8 --thinking
```

* **Historical Test Results** (model: `deepseek-ai/DeepSeek-V3.2-Exp`, 8×B200; measured with the earlier `sglang.test.run_eval` GPQA command):
  * Default (`temperature=0`): mean **0.797** over 8 runs — closely matches the official GPQA-Diamond score of **79.9** for DeepSeek-V3.2-Exp reported in its [model card](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)
  * With `temperature=1.0, top_p=0.95` (as recommended by DeepSeek):

```shell Command theme={null}
sgl-eval run gpqa --base-url http://127.0.0.1:30000/v1 --num-examples 198 --max-tokens 128000 --n-repeats 8 --top-p 0.95 --temperature 1.0 --thinking
```

```
Repeat: 8, mean: 0.840
Scores: ['0.848', '0.808', '0.848', '0.838', '0.879', '0.813', '0.838', '0.848']
```

#### 5.2.4 AIME 2025 Benchmark

Results on AIME 2025 (8×B200), evaluated with [sgl-eval](https://github.com/sgl-project/sgl-eval):

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>pass\@1 avg-of-4</th>
      <th>majority\@4</th>
      <th>pass\@4</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>DeepSeek-V3.2-Exp</td>
      <td>87.50% ± 1.67%</td>
      <td>90.00%</td>
      <td>90.00%</td>
    </tr>

    <tr>
      <td>DeepSeek-V3.2</td>
      <td>92.50% ± 1.67%</td>
      <td>94.71%</td>
      <td>96.67%</td>
    </tr>

    <tr>
      <td>DeepSeek-V3.2-Speciale</td>
      <td>95.00% ± 1.92%</td>
      <td>95.83%</td>
      <td>100.00%</td>
    </tr>
  </tbody>
</table>

**Reproduction.** Install [sgl-eval](https://github.com/sgl-project/sgl-eval), launch the server with the tool-call and reasoning parsers, then run `sgl-eval run`. The AIME 2025 dataset ships with sgl-eval, and thinking is on by default for it:

```bash Command theme={null}
pip install sgl-eval

sgl-eval run aime25 \
  --base-url http://localhost:30000/v1 \
  --model deepseek-ai/DeepSeek-V3.2-Exp \
  --n-repeats 4 \
  --temperature 1.0 \
  --top-p 0.95 \
  --max-tokens 64000
  # Use --max-tokens 120000 for the DeepSeek-V3.2-Speciale model
```

### 5.3 Speed Benchmark on Hopper

**Test Environment:**

* Hardware: NVIDIA H800 GPU (16x)
* Model: DeepSeek-V3.2
* Tensor Parallelism: 16
* sglang version: 0.5.9

#### 5.3.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
export SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS=32
export SGLANG_SET_CPU_AFFINITY=1

# Test workload ISL/OSL=1k/1k, raw tap : 4948.16 toks/sec, MAX ITL 5970
#   dp 2 : 5019.54  toks/sec, MAX ITL 7233
#   dp 4 : 4942.82  toks/sec, MAX ITL 35654
#   dp 2 + mtp : 6842.51 toks/sec, MAX ITL 3081
sglang_args=$(echo serve \
  --model-path $MAPPED_MODEL_PATH \
  --nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
  --attn-dp-size 2 --page-size 64 \
  --trust-remote-code --host "0.0.0.0" --port 30000 \
  --log-requests \
  --context-length 65536 --max-running-requests 128 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
  --allow-auto-truncate --enable-metrics \
  --tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
  --served-model-name DeepSeek-V3.2-Opt-dp2-mtp
)

sglang_args=($sglang_args)

sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host $MASTER_ADDR \
  --port 30000 \
  --model deepseek-ai/DeepSeek-V3.2 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    64.0
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  48.96
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4217
Request throughput (req/s):              0.20
Input token throughput (tok/s):          124.62
Output token throughput (tok/s):         86.20
Peak output token throughput (tok/s):    113.00
Peak concurrent requests:                2
Total token throughput (tok/s):          210.81
Concurrency:                             1.00
Accept length:                           3.27
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4893.12
Median E2E Latency (ms):                 3742.47
P90 E2E Latency (ms):                    8877.37
P99 E2E Latency (ms):                    10769.85
---------------Time to First Token----------------
Mean TTFT (ms):                          199.88
Median TTFT (ms):                        176.15
P99 TTFT (ms):                           272.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.99
Median TPOT (ms):                        10.88
P99 TPOT (ms):                           13.93
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.15
Median ITL (ms):                         8.86
P95 ITL (ms):                            17.29
P99 ITL (ms):                            33.71
Max ITL (ms):                            36.84
==================================================
```

#### 5.3.2 Throughput-Sensitive Benchmark

We simply use the same deployment method and vary the throughput by maximizing concurrencies:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host $MASTER_ADDR \
  --port 30000 \
  --model deepseek-ai/DeepSeek-V3.2 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 2048 \
  --max-concurrency 1024 # see picture below why we use 1024 for concurrency, hence num prompts 2048
```

DeepSeek 3.2 can steadily support concurrency up to `1024` and when concurrency is greater than `128`, the TTFT increase sharply:

![DeepSeek V3.2 Concurrency ISL/OSL=1024/128](https://github.com/user-attachments/assets/d5c9c9fb-44f3-4793-a0fd-f8fa954546f5)

Performance record:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    64.0
Max request concurrency:                 1024
Successful requests:                     2048
Benchmark duration (s):                  408.09
Total input tokens:                      1048992
Total input text tokens:                 1048992
Total generated tokens:                  1032734
Total generated tokens (retokenized):    1031817
Request throughput (req/s):              5.02
Input token throughput (tok/s):          2570.50
Output token throughput (tok/s):         2530.66
Peak output token throughput (tok/s):    5092.00
Peak concurrent requests:                1035
Total token throughput (tok/s):          5101.16
Concurrency:                             763.41
Accept length:                           3.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   152117.70
Median E2E Latency (ms):                 181704.84
P90 E2E Latency (ms):                    215924.77
P99 E2E Latency (ms):                    231679.59
---------------Time to First Token----------------
Mean TTFT (ms):                          127729.28
Median TTFT (ms):                        170098.94
P99 TTFT (ms):                           185705.73
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          49.18
Median TPOT (ms):                        48.48
P99 TPOT (ms):                           77.24
---------------Inter-Token Latency----------------
Mean ITL (ms):                           48.46
Median ITL (ms):                         52.11
P95 ITL (ms):                            110.26
P99 ITL (ms):                            200.63
Max ITL (ms):                            2666.37
==================================================
```

By adding `--random-range-ratio 1`, we could get even higher statistical numbers:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    64.0
Max request concurrency:                 1024
Successful requests:                     2048
Benchmark duration (s):                  612.87
Total input tokens:                      2097152
Total input text tokens:                 2097152
Total generated tokens:                  2097152
Total generated tokens (retokenized):    2096201
Request throughput (req/s):              3.34
Input token throughput (tok/s):          3421.84
Output token throughput (tok/s):         3421.84
Peak output token throughput (tok/s):    9077.00
Peak concurrent requests:                1039
Total token throughput (tok/s):          6843.68
Concurrency:                             772.66
Accept length:                           3.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   231222.27
Median E2E Latency (ms):                 289846.24
P90 E2E Latency (ms):                    314480.41
P99 E2E Latency (ms):                    320392.27
---------------Time to First Token----------------
Mean TTFT (ms):                          194081.02
Median TTFT (ms):                        252945.22
P99 TTFT (ms):                           279637.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          36.31
Median TPOT (ms):                        36.73
P99 TPOT (ms):                           46.33
---------------Inter-Token Latency----------------
Mean ITL (ms):                           36.31
Median ITL (ms):                         23.18
P95 ITL (ms):                            96.79
P99 ITL (ms):                            135.81
Max ITL (ms):                            3121.00
==================================================
```


# DeepSeek-V4
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4

Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, Flash Vision (Exp), Pro, and Pro Official (0813).

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
>
> **NVIDIA GPUs**
>
> A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)). **DGX Spark** is the other exception: its three cells (Flash Official FP4, Flash Official NVFP4, Flash Vision FP4) use the DGX Spark–only preview image `lmsysorg/sglang:dev-v4f-2dgx-v2` (the command panel picks it automatically — see the [DGX Spark notes](#spark-note)); do not use that image on any other hardware.
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
>
> docker run --gpus all \
>     --shm-size 32g \
>     -p 30000:30000 \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     --env "HF_TOKEN=<your-hf-token>" \
>     --ipc=host \
>     lmsysorg/sglang:latest \
>     sglang serve <use args below>
> ```
>
> **AMD GPUs (ROCm)**
>
> AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version.
>
> For example:
>
> * **MI355X** → `lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261016`
> * **MI300X** → `lmsysorg/sglang-rocm:v0.5.22-rocm720-mi30x-20261016`
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang-rocm:v0.5.22-rocm720-{mi35x,mi30x}-20261016
>
> docker run \
>     --device=/dev/kfd --device=/dev/dri \
>     --group-add video \
>     --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
>     --shm-size 32g --ipc=host \
>     -p 30000:30000 \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     --env "HF_TOKEN=<your-hf-token>" \
>     lmsysorg/sglang-rocm:v0.5.22-rocm720-{mi35x,mi30x}-20261016 \
>     sglang serve <use args below>
> ```

For **MI355X → Pro / Pro Official → FP4 → Multi-Nodes**, the panel generates a **1P1D deployment**: one prefill node and one decode node, each running an independent worker. Select **Router**, **Prefill**, or **Decode** to copy its command. Start the [UMBP tier](#3-8-unified-cache-external-linker-umbp) on the prefill node, then both workers, then the router. Send client requests to the router on port `8000`.

The MI355X multi-node strategies **Low-Latency**, **Balanced**, and **High-Throughput** use fixed concurrency presets of **16**, **48**, and **256**, respectively. **Pro** uses EAGLE/MTP; **Pro Official (0813)** uses DSpark. Both enable UMBP on prefill by default. The Playground inherits the selected worker, including its speculative decoder and cache settings.

Run these recipes' **Python** commands inside the ROCm image shown in the **Docker** tab, where Aiter is already enabled. Add explicit parser and network-device settings through the Playground's **Parsers** and **PD Disagg → IB Device** controls.

Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:

* **Low-Latency** — fastest reply for a single user. Pick for chat.
* **Balanced** — good speed with several users at once. Use for typical multi-user serving.
* **High-Throughput** — most tokens per second across many users. Best for batch jobs.

> [!NOTE]
>
> The **Flash Vision (Exp)** variant runs on a dedicated preview Docker image, `lmsysorg/sglang:dev-dsv4-flash-vision` — its support ([sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253)) has not shipped in a release yet. The command panel's Docker mode emits that image automatically for Flash Vision cells; see the [Flash Vision notes](#vision-note).

> [!NOTE]
>
> For a runnable end-to-end example, see the [DeepSeek-V4-Flash demo notebook](https://github.com/sgl-project/sglang/blob/main/docs/demo/deepseek_v4_flash.ipynb).

**Panel controls** (top of the command box):

<ul>
  <li>**Python / Docker** — bare `sglang serve …` for an existing SGLang env, or a `docker run … sglang serve …` wrap against the per-hardware image from the [Install SGLang](#install) panel above.</li>
  <li>**⧉ Copy** — copies the current command (with whichever framing is active) to your clipboard.</li>
  <li>**\$ cURL** — a sample request against `localhost:30000` to confirm the server is up.</li>
  <li>**⚙ Env** — edits the placeholders (`HOST_IP`, `PORT`, `HF_TOKEN`, `NODE_RANK`, `NODE0_IP`) the command and cURL share. Persists in localStorage across cookbooks.</li>
  <li>**Verified / Not Verified** badge — green when the `(hw, variant, quant, strategy, nodes)` combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
</ul>

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.

The knobs come in two flavors:

* **Built-in SGLang features** — parallelism overrides (TP / CP / DP-Attention — DP-Attention's value is the DP degree, with `off` to disable), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, HiCache tiers, and HiSparse hierarchical sparse attention (decode-role only — the card appears once PD-Disagg mode is set to decode).
* **DeepSeek-V4 specific features** — MegaMoE W4A8 / W4A4 fused kernel (Blackwell only; Hopper SM90 uses a separate all-FP8 MegaMoE path — see Configuration Tips below).

Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

**Panel controls** reuse **Python / Docker** · **⧉ Copy** · **\$ cURL** · **⚙ Env** from the Deploy panel, plus one extra:

<ul>
  <li>**Submit ↗** — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says **Not Verified**; click it once you've actually run the command on your hardware and confirmed it works.</li>
</ul>

## Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

* **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
* **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
* **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

## 1. Model Introduction

**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash and 0813 Pro refreshes add checkpoints with a bundled DSpark draft head, and the experimental Flash Vision checkpoint builds image understanding on top of the 0731 Flash base:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variant</th>
      <th>Total params</th>
      <th>Active (MoE)</th>
      <th>Use</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)**</td>
      <td>**284B**</td>
      <td>13B</td>
      <td>single-node serving on B200 / B300 / GB200 / GB300 / H200 (TP=4); RTX PRO 6000 (TP=2); H100 (TP=8)</td>
    </tr>

    <tr>
      <td>**[DeepSeek-V4-Flash-0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731)**</td>
      <td>**304**</td>
      <td>13B</td>
      <td>Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200</td>
    </tr>

    <tr>
      <td>**[DeepSeek-V4-Flash-Vision-Exp](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp)**</td>
      <td>**305B**</td>
      <td>13B</td>
      <td>Flash Vision (Exp) — experimental multimodal (image-text-to-text): the 0731 Flash base + vision encoder & aligner; verified on 4×B200 (TP=4), requires the [preview build](#vision-note)</td>
    </tr>

    <tr>
      <td>**[DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro)**</td>
      <td>**1.6T**</td>
      <td>49B</td>
      <td>high-capacity: B200 / B300 (TP=8) · GB300 (TP=4) · H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H200 FP8 (2-node, TP=16) · H100 (2-node, TP=16)</td>
    </tr>

    <tr>
      <td>**[DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813)**</td>
      <td>**1.65T**</td>
      <td>49B</td>
      <td>Pro Official (0813), with a bundled DSpark draft head; verified on 4×GB300 (TP=4) · B200 / B300 / H200 FP4 (TP=8) · GB200 (2-node, TP=8) · H100 (2-node, TP=16) · MI355X</td>
    </tr>
  </tbody>
</table>

The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (one mixed-precision checkpoint covers every FP4-capable GPU). Matching `*-Base` repos ship pure FP8 mixed and are for further pre-training only — not for chat or tool calling.

**Highlights:** hybrid CSA + HCA attention (\~27% inference FLOPs / \~10% KV cache vs DSv3.2 at 1M context), manifold-constrained hyper-connections (mHC), Muon optimizer, **1M-token context** (32T+ pre-training tokens), three reasoning modes (*Non-think* / *Think High* / *Think Max* — use ≥ 384K context for Think Max), and a dedicated `encoding_dsv4.encode_messages` Python encoder + DSML tool-call grammar.

**Recommended generation:** `temperature=1.0`, `top_p=1.0`.

**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Flash Vision (Exp)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · [Pro Official (0813)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813)  ·  ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro).

## 2. Configuration Tips

**Concurrency & DeepEP dispatch buffer**

Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together.

The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.

**Speculative decoding**

The original Flash and Pro recipes use EAGLE. Flash Official (0731) and Pro Official (0813) use the bundled DSpark draft head; see [DSpark](#3-4-dspark-speculative-decoding) for its launch and tuning notes.

> [!WARNING]
>
> Do not use EAGLE on the checkpoints that bundle a DSpark head. On 0813, `--speculative-algorithm EAGLE` starts and serves without any error, but the draft head it binds accepts nothing — every decode batch logs `accept len: 1.00, accept rate: 0.00`, so you pay the draft cost for zero speedup. Output stays correct, which is what makes it easy to miss. Switch to `--speculative-algorithm DSPARK`; the startup log then reports `Draft checkpoint bundles a DSpark head`.

For the original Flash and Pro checkpoints:

* `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
* `balanced`: steps=1, draft-tokens=2 → gentler MTP, reduces throughput hit at higher batch.
* `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves.
* MTP runs on the v2 speculative path.

**DGX Spark (2x GB10): Flash Official FP4 / NVFP4, Flash Vision FP4**

The **DGX Spark** row has three recipes, all **Balanced · Multi-Nodes**: **Flash Official (0731) · FP4**, **Flash Official (0731) · NVFP4**, and **Flash Vision (Exp) · FP4**. None of these checkpoints fits one 128GB GB10, so every recipe runs TP=2 across two DGX Sparks connected over ConnectX-7 (RoCE). Every other DGX Spark combination is greyed out on purpose.

* **Docker image** — all three cells use `lmsysorg/sglang:dev-v4f-2dgx-v2`, a preview build made **only for DGX Spark** (branch `b12x-vision` @ `452239a74f`): it bakes in the SM12x `b12x` MoE (W4A8) and compressed-MLA attention kernels ([#34878](https://github.com/sgl-project/sglang/pull/34878), [#35899](https://github.com/sgl-project/sglang/pull/35899), [#34018](https://github.com/sgl-project/sglang/pull/34018)), the Flash Vision model support ([#37253](https://github.com/sgl-project/sglang/pull/37253)), the b12x image-prefill fix that lets Flash Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel's Docker mode — the bare Python command needs the `b12x` kernel package this image ships.
* **Run the same command on both Sparks** with `--node-rank 0` / `--node-rank 1` and `--dist-init-addr` pointing at node 0 over the ConnectX-7 link. The `docker run` flags the panel emits (`--network host --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`) are what let NCCL use RDMA; without them NCCL silently falls back to TCP and decode slows by roughly 40%.
* **Env knobs** in the cells are part of the recipe: `SGLANG_SM120_FLASHMLA_BACKEND=b12x` selects the b12x attention path, `SGLANG_B12X_MAX_TOKENS` must equal `--chunked-prefill-size`, and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` avoids unified-memory fragmentation OOMs on GB10.
* **NVFP4 (`nvidia/DeepSeek-V4-Flash-0731-NVFP4`)** — only the routed experts are NVFP4; attention, shared experts and the DSpark MTP layer stay in the checkpoint's native formats. On SM12x that means three extra flags: `--moe-runner-backend flashinfer_cutlass` (b12x's MoE is MXFP4-only and trtllm-gen kernels are sm100-only), `--speculative-moe-runner-backend b12x` (the DSpark draft's MTP experts are MXFP4 and run on b12x), and `--disable-shared-experts-fusion` (HashTopK rejects fused shared experts under the cutlass runner). Throughput and DSpark acceptance match the FP4 cell within noise.
* **Flash Vision** — images are served natively on the b12x recipe with the same flags as Flash Official (send `image_url` content on `/v1/chat/completions`, see [Vision](#3-5-vision-image-inputs)); text-only requests work unchanged. Expect roughly 15–20% lower text throughput than Flash Official on this checkpoint — its bundled DSpark head accepts fewer drafts (\~3.2 vs \~3.9) — with text accuracy intact.

**DeepSeek-V4-Flash-Vision-Exp (Experimental)**

[`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) is DeepSeek's first experimental multimodal V4 checkpoint: the 0731 Flash base plus a vision encoder and aligner, served through the same `sglang serve` flow with OpenAI-style `image_url` inputs (see [Vision](#3-5-vision-image-inputs) below). Select the **Flash Vision** variant in the Deploy panel for its recipes.

* **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch. The DGX Spark Flash Vision cell is the exception: it uses the DGX Spark image `lmsysorg/sglang:dev-v4f-2dgx-v2` (see the [DGX Spark notes](#spark-note)).
* **Verified matrix** — MMMU-Pro via sgl-eval at `temperature 1.0`, `top-p 0.95`, `--reasoning-effort max`.
* **Engine auto-configuration** — the engine picks the `flashinfer_mxfp4` MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don't pass `--enforce-shared-experts-fusion`.
* **Chunked prefill & radix cache stay enabled** — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start.
* **Speculative decoding** — the checkpoint bundles a DSpark head, and the low-latency recipes enable it with `--speculative-algorithm DSPARK` (verified with image batches on B200 via the MMMU-Pro round; the other hardware rows are pending verification). The balanced and high-throughput recipes run target-only: they use DP Attention, which DSpark is incompatible with on current releases. As on the 0731/0813 checkpoints, do not pass the EAGLE flags.

**Shared experts fusion (Blackwell, flashinfer\_mxfp4)**

On the Blackwell fp4 recipes (`--moe-runner-backend flashinfer_mxfp4`), the shared expert runs as a separate FP8 MLP on an alternate stream by default. Adding:

```bash Command theme={null}
--enforce-shared-experts-fusion
```

routes it as one extra MXFP4 expert through the same trtllm-gen MoE kernel, so the whole MoE runs on a single stream (\~4 fewer kernel launches and 2 fewer stream syncs per MoE layer). The shared expert is requantized from FP8 to MXFP4 at load time. Measured on GB200 tp4: gsm8k and AIME25 accuracy on par with the unfused baseline; Mean TTFT -13% to -21% and P99 ITL -15% to -53% at QPS 1-8 with neutral throughput.

Only for deployments without expert parallelism (e.g. the single-node low-latency recipes): with `moe_ep_size > 1` the flag is rejected at startup, unless the DeepEP/MegaMOE per-rank shared-slot path is in use. Not applicable to [Flash Vision (Exp)](#vision-note) — the engine auto-disables the fusion on that checkpoint.

**Compressed attention state dtype**

DeepSeek-V4 uses hybrid compressed attention for long-context efficiency. `SGLANG_DSV4_COMPRESS_STATE_DTYPE` controls the dtype of the C4 / C128 compressed attention state pools. Supported values are `float32` / `fp32` (default: `float32`) and `bfloat16` / `bf16`. For BF16 on the offline compression path:

```bash Command theme={null}
SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16 \
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  <other args>
```

This BF16 setting applies only to the compressed attention state pools and reduces the GPU memory footprint of each compressed-state slot. It does not change model weight precision or the main KV cache dtype. With automatic pool sizing and no explicit capacity cap, the same memory budget holds more slots, and the startup log shows larger `c4_state` and `c128_state` pool sizes. Keep the default `float32` setting for the most conservative behavior.

**EPLB + Waterfill (Experimental)**

For recorded/static EPLB reproduction, first record an expert-distribution file by following
[Capture expert selection distribution in MoE models](../../../docs/basic_usage/native_api.mdx#capture-expert-selection-distribution-in-moe-models).
For reproduction runs, use the generated `expert_distribution_recorder_*.pt` as
the initial expert location. **Please checkout to latest main branch for this feature.**

For non-PD reproduction, use:

```bash Command theme={null}
--moe-a2a-backend deepep \
--deepep-mode auto \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill
```

For PD-Disagg reproduction, use `normal` mode on the prefill server and
`low_latency` mode on the decode server. Add the same `--init-expert-location`
flag to both commands:

```bash Command theme={null}
# prefill
--moe-a2a-backend deepep \
--deepep-mode normal \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill

# decode
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--init-expert-location /path/to/expert_distribution_recorder_*.pt \
--enable-waterfill
```

You can also add `--ep-num-redundant-experts` and `--eplb-algorithm` to customize
EPLB placement.

Waterfill also supports MegaMOE. Use `--moe-a2a-backend megamoe --enable-waterfill` to keep the MegaMOE backend while applying Waterfill to the
fused shared expert slot.

**FP4 Indexer (Experimental)**

DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial.

On **NVIDIA (SM100 / SM120)**, pair the flag with DeepGEMM FP4 indexer support and the FlashInfer MXFP4 MoE runner:

```bash Command theme={null}
# Please use the latest main branch for this feature.
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Flash \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --enable-deepseek-v4-fp4-indexer
```

On **AMD MI355X (gfx95)**, the AITER FP4 indexer kernels are available on ROCm. Keep the standard MI355 recipe and add only the indexer flag. Do not need to pass `--moe-runner-backend flashinfer_mxfp4`:

```bash Command theme={null}
# Please use the latest ROCm image for this feature.
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4-Pro \
  --tp 8 \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  <other args from the Deploy panel>
```

**NVFP4 Hybrid Checkpoints**

The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and
[`nvidia/DeepSeek-V4-Flash-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4) checkpoints
quantize MoE experts to **NVFP4** while keeping attention and dense layers in
**FP8**. The official releases have matching NVFP4 checkpoints at
[`nvidia/DeepSeek-V4-Flash-0731-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-0731-NVFP4) and
[`nvidia/DeepSeek-V4-Pro-0813-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-0813-NVFP4).
All of them require `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided.

```bash Command theme={null}
sglang serve \
  --model-path nvidia/DeepSeek-V4-Pro-NVFP4 \
  --tp 8
```

or

```bash Command theme={null}
sglang serve \
  --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \
  --tp 8
```

Requires Blackwell (SM100+). The MTP layer in this checkpoint stays
MXFP4-packed and is routed through the `Mxfp4FlashinferTrtllmMoEMethod` path
automatically.

The official (0731 / 0813) NVFP4 checkpoints preserve the bundled DSpark draft
head, so their low-latency recipes use `--speculative-algorithm DSPARK` instead
of the EAGLE/MTP shape flags — same as the corresponding original-precision
official checkpoints.

**Hopper (H100 / H200) note**

Two options are available for running DeepSeek-V4 on Hopper:

* **Original FP4 checkpoints** — run the MoE experts with W4A16 kernels (Marlin or the FlashInfer SM90 CUTLASS runner) as the command generator picks for Hopper cells. With FlashInfer >= 0.6.18 you can instead select the **W4A8** path — MXFP4 weights with FP8 activations via FlashInfer's Humming kernels — by adding `--flashinfer-mxfp4-moe-precision fp8` to `--moe-runner-backend flashinfer_mxfp4`; the low-latency Hopper cells now generate this form. Both work on H100 and H200; FP4 is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16).
* **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes).

On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below.

PD-Disagg recipes on H200 may require `docker run --privileged --ulimit memlock=-1`
(or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) so mooncake
can discover the IB HCAs; without IB exposure mooncake silently falls back to
TCP, which can lead to garbled KV transfer on large checkpoints.

**RTX PRO 6000 (SM120 / Blackwell Desktop) note**

RTX PRO 6000 (96 GB) runs **Flash only** with the FlashInfer MXFP4 MoE runner.
V4-Pro doesn't fit on 8× 96 GB; the Deploy panel greys out unsupported recipes.
HiCache and MegaMoE are **not** supported on RTX PRO 6000.

**AMD (MI300X / MI355X) note**

* **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
* **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
* **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--attn-dp-size 8 --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`.
* **MTP** — speculative decoding on the original Flash / Pro checkpoints, which bundle an MTP head; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Pro Official (0813) has no MTP head, so these flags do not apply there and the Playground does not offer them.
* **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head, and DSpark is the only speculative algorithm it supports. Enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. DSpark also runs under **PD disaggregation** — see the Pro Official pairs in [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io).
* **Kernels** — uses the Unified KV attention and the flydsl MoE.
* **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe.

**MoRI EP (AMD expert parallelism)**

On AMD, expert parallelism uses the **MoRI** all-to-all backend (`--moe-a2a-backend mori`), not DeepEP. Add the flags below on top of the verified recipe when sharding experts across GPUs; set `--ep-size` to the EP degree (typically the GPU count on one node).

Two optional env vars improve MoRI throughput (both off by default):

```bash Command theme={null}
export SGLANG_MORI_DISPATCH_DTYPE=mxfp8
export SGLANG_MORI_RECV_BOUND=1

sglang serve \
  --model-path {MODEL_PATH} \
  --tp 8 --attn-dp-size 8 \
  --ep-size 8 --moe-a2a-backend mori --deepep-mode normal \
  <other args from the Deploy panel>
```

* **FP4**: enable both env vars.
* **FP8**: use `SGLANG_MORI_RECV_BOUND=1` only; omit `SGLANG_MORI_DISPATCH_DTYPE=mxfp8`.

**MegaMoE**

MegaMoE fuses expert dispatch + GEMM into a single kernel for higher throughput
on MoE layers. To enable it, use the **MegaMoE** chip in the Playground
below — the playground will swap `--moe-a2a-backend deepep` for
`--moe-a2a-backend megamoe` and add the relevant launch settings automatically.

Two variants are exposed:

* **W4A8** — default MegaMoE kernel (FP4 weights, FP8 activations).
* **W4A4** — adds `--enable-w4a4-mxfp4-megamoe` to run the custom W4A4 kernel
  (FP4 activations). The flag configures the required DeepGEMM settings.
  Higher throughput with negligible accuracy drop (\~89.5 GPQA on Pro).

Notes:

* The W4A8 / W4A4 variants above are **Blackwell-only** (B200 / B300 / GB200 / GB300). On **Hopper (SM90, H100 / H200)** use the all-FP8 MegaMoE path described below instead.
* MegaMoE is **only wired into the `high-throughput` recipe** on Blackwell (per [sgl-project/sglang#26451](https://github.com/sgl-project/sglang/pull/26451)). The chip is hidden on `low-latency` and `balanced` — switch to `high-throughput` to expose it.
* When running MegaMoE, don't set `--moe-runner-backend` manually.
* Adjust `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` based on your workload and memory usage. Setting higher number of tokens for MegaMoE requires more HBM space (recommended: 8320 for high-throughput).

**SM90 (Hopper) FP8 MegaMoE (Experimental)**

On SM90 (Hopper, H100 / H200), the all-FP8 MegaMoE path routes MoE through the
DeepGEMM `mega_moe` runner for higher long-context / large-decode throughput on
the FP8 checkpoints. Unlike the Blackwell W4A8 / W4A4 variants above, experts
stay in **FP8** — keep `SGLANG_DSV4_FP4_EXPERTS=0`. It requires a `sgl-deep-gemm`
build with SM90 FP8 MegaMoE support. **Please use the latest image for this
feature.**

Enable the MegaMoE path with `--moe-a2a-backend megamoe`

```bash Command theme={null}
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \
SGLANG_DSV4_FP4_EXPERTS=0 \
sglang serve \
  --model-path sgl-project/DeepSeek-V4-Flash-FP8 \
  --tp 8 \
  --moe-a2a-backend megamoe \
  --chunked-prefill-size 4096
```

`SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` caps the number of tokens
the MegaMoE path processes per rank (i.e. per GPU); the MegaMoE path is only used
for batches at or below this cap. The right value depends on your parallelism /
token-split scheme, and larger values reserve more HBM.

**GB300 PD-Disagg cross-pod MNNVL**

On some GB300 clusters with cross-pod KV transfer over NVLink, mooncake may
fail with `nvlink_transport.cpp:497 Requested address ... not found!`. If
this happens, prepend `MC_FORCE_MNNVL=1 NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`
to both prefill and decode `sglang serve` commands.

## 3. Advanced Usage

### 3.1 Reasoning

Enable the `deepseek-v4` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.

> **Streaming with Thinking Process (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(
>     base_url="http://localhost:30000/v1",
>     api_key="EMPTY"
> )
>
> response = client.chat.completions.create(
>     model="deepseek-ai/DeepSeek-V4-Flash",
>     messages=[
>         {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
>     ],
>     max_tokens=2048,
>     extra_body={"chat_template_kwargs": {"thinking": True}},
>     stream=True,
> )
>
> thinking_started = False
> has_thinking = False
> has_answer = False
>
> for chunk in response:
>     if not chunk.choices:
>         continue
>     delta = chunk.choices[0].delta
>
>     if getattr(delta, "reasoning_content", None):
>         if not thinking_started:
>             print("=============== Thinking =================", flush=True)
>             thinking_started = True
>         has_thinking = True
>         print(delta.reasoning_content, end="", flush=True)
>
>     if delta.content:
>         if has_thinking and not has_answer:
>             print("\n=============== Content =================", flush=True)
>             has_answer = True
>         print(delta.content, end="", flush=True)
>
> print()
> ```

> **Example Output**
>
> ```text Output theme={null}
> We are asked: "What is 15% of 240?" This is a simple percentage problem. I need to provide a step-by-step solution. The user wants the solution explained step by step. I'll calculate 15% of 240: 0.15 * 240 = 36. I'll break it down into steps: understand what percent means, convert percentage to decimal or fraction, then multiply. I'll present the answer clearly.</think>To find 15% of 240, follow these steps:
>
> **Step 1: Understand the meaning of percent**
> "Percent" means "per hundred," so 15% means 15 out of every100, or \( \frac{15}{100} \).
>
> **Step2: Convert the percentage to a decimal or fraction**
> \( 15\% = \frac{15}{100} = 0.15 \)
>
> **Step3: Multiply by the given number**
> Multiply the decimal form by 240:
> \( 0.15 \times 240 \)
>
> **Step4: Perform the multiplication**
> \( 0.15 \times 240 = 36 \)
>
> **Answer:** 15% of 240 is **36**.
> ```

### 3.2 Tool Calling

Enable the `deepseekv4` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.

> **Python Example with Thinking Process**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(
>     base_url="http://localhost:30000/v1",
>     api_key="EMPTY"
> )
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="deepseek-ai/DeepSeek-V4-Flash",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
>     extra_body={"chat_template_kwargs": {"thinking": True}},
>     stream=True,
> )
>
> thinking_started = False
> has_thinking = False
> tool_calls_accumulator = {}
>
> for chunk in response:
>     if not chunk.choices:
>         continue
>     delta = chunk.choices[0].delta
>
>     if getattr(delta, "reasoning_content", None):
>         if not thinking_started:
>             print("=============== Thinking =================", flush=True)
>             thinking_started = True
>         has_thinking = True
>         print(delta.reasoning_content, end="", flush=True)
>
>     if getattr(delta, "tool_calls", None):
>         if has_thinking and thinking_started:
>             print("\n=============== Content =================\n", flush=True)
>             thinking_started = False
>         for tool_call in delta.tool_calls:
>             index = tool_call.index
>             if index not in tool_calls_accumulator:
>                 tool_calls_accumulator[index] = {"name": None, "arguments": ""}
>             if tool_call.function:
>                 if tool_call.function.name:
>                     tool_calls_accumulator[index]["name"] = tool_call.function.name
>                 if tool_call.function.arguments:
>                     tool_calls_accumulator[index]["arguments"] += tool_call.function.arguments
>
>     if delta.content:
>         print(delta.content, end="", flush=True)
>
> for index, tool_call in sorted(tool_calls_accumulator.items()):
>     print(f"Tool Call: {tool_call['name']}")
>     print(f"   Arguments: {tool_call['arguments']}")
>
> print()
> ```

> **Example Output**
>
> ```text Output theme={null}
> The user wants to know the weather in Beijing. I'll use the get_weather function with Beijing as the location. I don't need to specify a unit, so I'll just use the default.</think>
>
> <｜DSML｜tool_calls>
> <｜DSML｜invoke name="get_weather">
> <｜DSML｜parameter name="location" string="true">Beijing</｜DSML｜parameter>
> </｜DSML｜invoke>
> </｜DSML｜tool_calls>
> ```

### 3.3 HiCache (Hierarchical KV Caching)

HiCache enables multi-tier KV cache offloading (GPU → CPU → Storage), significantly expanding effective context capacity for long-context and multi-turn scenarios. Combined with UnifiedRadixTree, it provides intelligent prefix caching across all tiers.

To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**:

* **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only.
* **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file` / `mooncake` / `hf3fs` / `nixl`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).

For AMD devices,

* **L2 (GPU + CPU)** — leave Storage on `auto` (default). Cold KV pages spill to CPU pinned memory only. Use `direct` IO backend + `page_first_direct` or `layer-first` mem-layout.
* **L3 (GPU + CPU + Storage)** — pick a Storage backend (`file`); the Playground emits the canonical `page_first_direct` mem-layout + `direct` IO backend + `wait_complete` prefetch policy, matching the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices).

The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow.

For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache). For the alternative that skips the host tier entirely, see [§3.8 UMBP](#3-8-unified-cache-external-linker-umbp) — it is a sibling of HiCache, not a storage backend for it, and the two cannot be enabled together.

### 3.4 DSpark (Speculative Decoding)

Flash Official (0731) and Pro Official (0813) bundle a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731` and `deepseek-ai/DeepSeek-V4-Pro-0813`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`.

The experimental [Flash Vision checkpoint](#vision-note) also bundles a DSpark head, enabled the same way: the Flash Vision low-latency recipes ship with `--speculative-algorithm DSPARK` (verified with image batches on B200 via the MMMU-Pro round; other hardware rows pending). The balanced and high-throughput Flash Vision recipes stay target-only because they run DP Attention.

Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint.

> [!NOTE]
>
> The Pro Official (0813) low-latency speed numbers in the Deploy panel were measured with `SGLANG_SIMULATE_ACC_LEN=4`, which pins the DSpark accept length at exactly 4.00. The recipe as shipped earns **4.678** on the same engine, so those rows read slightly conservative. The GSM8K figure for that cell is from the shipped command.

The verified 4×GB300 FP4 low-latency command is:

```bash Command theme={null}
sglang serve \
  --trust-remote-code \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --tp 4 \
  --moe-runner-backend flashinfer_mxfp4 \
  --speculative-algorithm DSPARK \
  --mem-fraction-static 0.90 \
  --chunked-prefill-size 4096 \
  --swa-full-tokens-ratio 0.1 \
  --host 0.0.0.0 \
  --port 30000
```

Keep `--mem-fraction-static 0.90` on this topology to leave enough headroom for the batch-256 verify graph. The first cold start can take 10–15 minutes while FlashInfer autotunes and SGLang captures the draft and verify graphs; later starts reuse the cache. This path is verified end-to-end on 4×GB300 with SGLang v0.5.16.

**Tune proposed draft tokens.** `--speculative-dspark-block-size N` asks DSpark to propose `N` tokens per step; the target verifies a window of `N + 1`. If the flag is omitted, SGLang reads the value from the checkpoint. Both the 0731 and 0813 checkpoints resolve to five proposed tokens (the startup log reports `gamma=5, verify_num_draft_tokens=6`), which is the verified default. Use the **DSpark Proposed Draft Tokens** slider in the [Playground](#playground) to sweep one through five.

Larger blocks can improve decode latency when acceptance stays high, but they also increase verification work and graph memory. Start from the checkpoint default, then sweep downward under the real prompt-length and concurrency distribution. The gain is usually largest for short interactive traffic and narrows as prefill dominates. Track P50/P99 TTFT and TPOT, total throughput, accepted length, GPU memory, and stop rate rather than choosing from acceptance alone.

For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override.

DSpark requires `pp_size == 1`. Selecting a prefill or decode role in the Playground keeps the inherited DSpark flags on both roles; for PD-disaggregated configurations that run DSpark, see the [Agentic Long-Context Deployment](#agentic-long-context-deployment) and the [MI355X Pro Official PD pairs](#3-7-pd-disaggregation-on-mi355x-mori-io). The MI355X Flash Official recipes run target-only, and so do the DP-Attention recipes in the Deploy panel — for DP-Attention configurations that do run DSpark, see the [Agentic Long-Context Deployment](#agentic-long-context-deployment) and the [MI355X agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates.

### 3.5 Vision (Image Inputs)

The experimental [`DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) checkpoint (the **Flash Vision** variant in the Deploy panel — see the [configuration notes](#vision-note)) takes images via the OpenAI-compatible `image_url` content type, as public URLs or base64 `data:` URIs; text and images mix freely in one message. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed.

> **Image Understanding (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "image_url",
>              "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
>             {"type": "text", "text": "Describe this image in a few sentences."},
>         ],
>     }],
> )
>
> message = response.choices[0].message
> if getattr(message, "reasoning_content", None):
>     print("=============== Thinking =================")
>     print(message.reasoning_content)
> print("=============== Content =================")
> print(message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update...
> ```

### 3.6 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark)

Three commands follow, one per concurrency range. They share the model, parser, attention and DSpark flags, and differ in parallelism, MoE path and host offload:

| | TP8 | TP8 + HiCache | DP8 | DP8 EP8 MegaMoE |
| - | - | - | - | - |
| Concurrency | 1–16 | 32–48 | 128 | 128–384 |
| Parallelism | TP8 | TP8 | TP8+DP8 | TP8+DP8+EP8 |
| MoE | replicated | replicated | replicated | MegaMoEv2 |
| HiCache | off | on | on | on |
| `--mem-fraction-static` | 0.86 | 0.86 | 0.92 | 0.85 |

DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7). Keep `SGLANG_DSV4_UNIFIED_KV_FP8=1` on every command. The first three use `--enforce-shared-experts-fusion`; the MegaMoE command uses `--disable-shared-experts-fusion` instead.

Four adjustments are not in the commands below. Add them when they apply:

* `--cuda-graph-backend-prefill breakable` at concurrency 1–4.
* `--enable-prefill-cp --cp-strategy interleave --enable-cp-decode-attn-tp` on the TP-only commands at concurrency 16 and above. Prefill context parallelism splits each prompt across all eight ranks (CP size = TP size).
* `SGLANG_DSV4_GROUPED_ASM=1` on the TP-only commands at concurrency 16 and above — the kernel needs at least 16 requests per draft token.
* At concurrency 384 on the MegaMoE command: `--prefill-decode-interval 20`, `SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR=1024`, and `AITER_MEGA_FIXED_SLOT_MAX_MTPR=2047`.

**TP8, concurrency 1–16:**

```bash Command theme={null}
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --prefill-decode-interval 20 \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --enforce-shared-experts-fusion \
  --mem-fraction-static 0.86 \
  --chunked-prefill-size 16384 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --watchdog-timeout 3600
```

**TP8 + HiCache, concurrency 32–48:** the same command with HiCache DRAM offload added. `--hicache-ratio` is the host/device token ratio; `1.5` is what stays under a \~2.7 TB host DRAM budget at TP8. The same five flags appear in the two DP commands below.

```bash Command theme={null}
  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct
```

**DP8 (DP Attention), concurrency 128:**

```bash Command theme={null}
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_DP_USE_GATHERV=1 \
SGLANG_DP_USE_REDUCE_SCATTER=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --attn-dp-size 8 \
  --enable-dp-lm-head \
  --enable-prefill-delayer \
  --enable-dp-attention-local-control-broadcast \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --prefill-decode-interval 20 \
  --prefill-delayer-token-usage-low-watermark 0.7 \
  --load-balance-method total_requests \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --enforce-shared-experts-fusion \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 131072 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct \
  --watchdog-timeout 3600
```

`--chunked-prefill-size` is a global budget divided by `--attn-dp-size`, so 131072 keeps 16384 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware`, so a multi-turn session lands on the rank already holding its prefix; above concurrency 160 add `--balance-abs-threshold 32`. Both apply to the MegaMoE command below as well.

**DP8 EP8 MegaMoE, concurrency 128–384:**

```bash Command theme={null}
SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_HIP_DSV4_C4_INDEXER_ROWGROUP=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \
AITER_BF16_FP8_MOE_BOUND=0 \
SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \
SGLANG_OPT_FP8_WO_A_FUSED_INVROPE=1 \
SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \
SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR=16384 \
SGLANG_AMD_FLYDSL_MEGA_MOE_DECODE_MTPR=512 \
AITER_MEGA_FIXED_SLOT_MAX_MTPR=1023 \
SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \
MORI_SHMEM_HEAP_SIZE=17179869184 \
SGLANG_AITER_MEGA_RANK_SYNC=1 \
SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1 \
SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1 \
SGLANG_EPLB_P2P_BATCH_CHUNK_SIZE=7 \
SGLANG_DP_USE_REDUCE_SCATTER=0 \
GPU_MAX_HW_QUEUES=5 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --tp 8 \
  --attn-dp-size 8 \
  --ep 8 \
  --enable-dp-lm-head \
  --enable-prefill-delayer \
  --enable-dp-attention-local-control-broadcast \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --prefill-decode-interval 40 \
  --prefill-delayer-token-usage-low-watermark 0.7 \
  --load-balance-method total_requests \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --page-size 256 \
  --swa-full-tokens-ratio 0.1 \
  --kv-cache-dtype fp8_e4m3 \
  --moe-a2a-backend megamoe \
  --moe-dense-tp-size 1 \
  --disable-shared-experts-fusion \
  --enable-eplb \
  --eplb-rebalance-num-iterations 200 \
  --expert-distribution-recorder-mode stat \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 131072 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 6 \
  --enable-hierarchical-cache \
  --hicache-ratio 1.5 \
  --hicache-write-policy write_through \
  --hicache-io-backend direct \
  --hicache-mem-layout page_first_direct \
  --watchdog-timeout 3600
```

This command adds expert parallelism on top of the DP8 one. [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io) explains `SGLANG_AMD_USE_FLYDSL_MEGA_MOE`, `SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR` and the MORI symmetric heap; the heap is 16 GiB here rather than 8 GiB because `--mem-fraction-static 0.85` leaves more GPU memory free. `SGLANG_DP_USE_REDUCE_SCATTER=0` turns off the reduce-scatter path that MegaMoE's all-to-all replaces. The three `SGLANG_AITER_MEGA_*` variables and `--enable-eplb` enable expert-load balancing, recorded during prefill only. `SGLANG_EPLB_P2P_BATCH_CHUNK_SIZE=7` keeps each RCCL P2P group under 31 ops during EPLB rebalance.

### 3.7 PD Disaggregation on MI355X (MORI-IO)

The prefill and decode roles can use any transport the **PD Disagg** card in the [Playground above](#playground) offers. On MI300X / MI355X that is **MORI-IO**, **Mooncake** and **NiXL**, and all three stay selectable. This section covers MORI-IO only, because it is the transport every AMD recipe in this cookbook was measured with. It is also what the card selects by default on those two platforms, and it is the one transport that is ROCm-only, so the card hides it on the CUDA platforms.

The MI355X multi-node recipes start with **IB Device → Auto**. Select the `rdma0-7 (all NICs)` option in the Playground to add the explicit NIC list. ROCm nodes enumerate their NICs as `rdmaN`, so the card hides the ConnectX `mlx5_*` names on AMD platforms. On CUDA platforms it hides the `rdmaN` list instead.

Both roles need the RDMA NICs passed into the container, so the **Docker** output adds the fabric flags below on top of the usual ROCm device access from the [Install panel](#install). `/dev/infiniband` covers `rdma_cm` and the per-NIC `uverbs0`…`uverbs7` nodes, and the memlock/`IPC_LOCK` pair is what lets MORI-IO pin the buffers it registers with the NICs:

```bash Command theme={null}
docker run \
    --device=/dev/kfd --device=/dev/dri --device /dev/infiniband \
    --group-add video --cap-add IPC_LOCK \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    --ulimit nofile=1048576:1048576 \
    --network host --ipc=host --shm-size 32g \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261019 \
    python3 -m sglang.launch_server <role args below>
```

Use this image for the **MI355X Pro / Pro Official FP4 multi-node** workers and router, including the additional concurrency points below. The Deployment and Playground Docker outputs select it for all three PD roles. It supports the documented `SGLANG_DSV4_DENSE_MXSCALE` and `SGLANG_ROUTER_P2C_LOAD_METRIC` settings. `--network host` replaces the single-node `-p` mapping because the two roles run on different hosts.

For the **MI355X Pro / Pro Official FP4** presets, **High-Throughput** sets `MORI_IO_QP_MAX_SEND_WR=32767` and `MORI_IO_QP_MAX_CQE=32768` on both MORI roles. Low-Latency and Balanced use the transport's default queue sizes. The send-queue backoff timeout uses MORI's default of 5 seconds.

#### Role-based configuration

Select **MI355X → Pro Official → FP4 → Multi-Nodes**, then a strategy and role. Replace the host and model-path placeholders with your deployment's values. The Pro Official commands use the kernel and routing settings described below.

| Strategy | Concurrency | Prefill → Decode | `--max-running-requests` | DSpark block size |
| - | - | - | - | - |
| Low-Latency | 16 | TP4 → TP8 | 32 | 6 |
| Balanced | 48 | TP4 → TP8 | 96 | 3 |
| High-Throughput | 256 | TP8+DP8+EP8 → TP8+DP8 | 512 | 3 |

These are two separate workers: neither receives `--nnodes 2` or a cross-node TP rendezvous address. Prefill listens on `30000`, decode on `30100`, and the router on `8000`. Other hardware's Multi-Nodes recipes can still represent a model sharded across nodes.

`--cuda-graph-bs-decode` covers 1 to `--max-running-requests / attn_dp_size`: the full value on the TP-only rows and an eighth on the DP8 rows. Each strategy sets both workers' admission limit and decode's graph sizes together. The TP-only points use `--mem-fraction-static 0.86` and SWA ratio `0.1`; the DP8 points use `0.92` and `0.15`.

All prefill roles enable [UMBP](#3-8-unified-cache-external-linker-umbp), `--optimistic-prefill-attempts 2`, and cache reporting. Decode does not use an offload tier. The high-throughput prefill worker uses Aiter MegaMoEv2 with EP8; decode keeps TP MoE. All Pro Official PD workers enable `SGLANG_DSV4_UNIFIED_KV_FP8=1`. Low-Latency and Balanced also set `SGLANG_DSV4_DENSE_MXSCALE=1` on both roles; High-Throughput sets mxscale on decode only. The decode dispatch budget is `64 × (DSpark block size + 1)`: `MORI_MAX_DISPATCH_TOKENS_DECODE=448` at concurrency 16 and `256` at concurrency 48 or 256. The commands below show concurrency 256.

Select **Pro** to use the original checkpoint's **EAGLE/MTP 3-1-4** head with the same topology and admission limits. Its commands contain no DSpark parameters. In the Playground, **Off (greedy)** removes speculative decoding. Enabling **HiCache** automatically turns off **Unified Cache External Linker** and removes UMBP's socket configuration and optimistic prefill. Enabling the linker turns HiCache off.

> [!NOTE]
>
> Selecting a TP-only role forces **DP-Attention** off and greys the control. `--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so turning DP on would cut the per-rank batch below the captured `--cuda-graph-bs-decode` values without changing either flag in the command.

> [!WARNING]
>
> **Speculative decoding changes how you size `--max-running-requests`.** For the strategy presets above, a target concurrency of N uses `N*2` on **both** roles; at N the served batch stays below the concurrency you are aiming for. This holds for DSpark and EAGLE/MTP alike. The table above follows it: 32 serves a concurrency of 16, 96 serves 48, and 512 serves the high-throughput concurrency of 256. The Pro Official c1 override below is an exception: retain 8 request slots on each role even though client concurrency is 1.
>
> The decode role's `--cuda-graph-bs-decode` then has to cover `N*2 / attn_dp_size`, the per-rank batch. The strategy presets apply both rules: decode graphs cover 1–32, 1–96, and 1–64, respectively.

Two things then differ between the two roles regardless of strategy:

* **CUDA graphs.** Prefill runs `--disable-cuda-graph`, because the FP4 indexer's prefill path is not captured. Decode captures `--cuda-graph-bs-decode`.
* **Speculative decoding** is the same on both. Keep `--speculative-dspark-block-size` equal on the two roles. The original Flash and Pro checkpoints have no DSpark head and use their MTP head instead: `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.

**MoE parallelism differs too, on the high-throughput pair.** Each role uses what suits its batch size. Prefill runs **EP8**, with the experts split across the 8 ranks: it processes a 65536-token chunk, so the split pays for the all-to-all it costs. Decode runs **TP8**, with the experts replicated on every rank, because a few tokens per request per step would not pay for it.

The two commands below are that pair at concurrency 256, on one prefill node and one decode node. Run each on its own host, then front them with the router.

**Prefill, EP8:**

```bash Command theme={null}
GPU_MAX_HW_QUEUES=5 \
SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1 \
SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 \
MORI_SHMEM_HEAP_SIZE=8G \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_DP_USE_REDUCE_SCATTER=0 \
UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --host 0.0.0.0 --port 6100 \
  --disaggregation-mode prefill \
  --disaggregation-bootstrap-port 7200 \
  --disaggregation-transfer-backend mori \
  --disaggregation-ib-device rdma3,rdma0,rdma2,rdma1,rdma7,rdma4,rdma6,rdma5 \
  --tp-size 8 --dp-size 8 --ep-size 8 \
  --enable-dp-attention \
  --enable-dp-attention-local-control-broadcast \
  --enable-dp-lm-head \
  --moe-a2a-backend megamoe \
  --moe-dense-tp-size 1 \
  --enforce-shared-experts-fusion \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --kv-cache-dtype fp8_e4m3 \
  --page-size 256 \
  --swa-full-tokens-ratio 0.15 \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 65536 \
  --max-running-requests 512 \
  --context-length 1048576 \
  --disable-cuda-graph \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 3 \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-unified-cache-external-linker \
  --unified-cache-external-linker-backend mori \
  --optimistic-prefill-attempts 2 \
  --enable-cache-report \
  --load-balance-method round_robin \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --enable-metrics \
  --watchdog-timeout 3600
```

**Decode, TP8:**

```bash Command theme={null}
GPU_MAX_HW_QUEUES=5 \
SGLANG_DSV4_UNIFIED_KV_FP8=1 \
SGLANG_DSV4_DENSE_MXSCALE=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_DP_USE_GATHERV=1 \
MORI_MAX_DISPATCH_TOKENS_DECODE=256 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
  --trust-remote-code \
  --host 0.0.0.0 --port 6100 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mori \
  --disaggregation-ib-device rdma3,rdma0,rdma2,rdma1,rdma7,rdma4,rdma6,rdma5 \
  --tp-size 8 --dp-size 8 \
  --enable-dp-attention \
  --enable-dp-attention-local-control-broadcast \
  --enable-dp-lm-head \
  --enforce-shared-experts-fusion \
  --attention-backend dsv4 \
  --enable-deepseek-v4-fp4-indexer \
  --kv-cache-dtype fp8_e4m3 \
  --page-size 256 \
  --swa-full-tokens-ratio 0.15 \
  --mem-fraction-static 0.92 \
  --max-running-requests 512 \
  --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 \
  --context-length 1048576 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 3 \
  --speculative-num-steps 1 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --load-balance-method round_robin \
  --tokenizer-worker-num 8 \
  --stream-interval 20 \
  --tool-call-parser deepseekv4 \
  --reasoning-parser deepseek-v4 \
  --enable-metrics \
  --watchdog-timeout 3600
```

Decode sets neither `--ep-size` nor `--moe-a2a-backend`. `--moe-a2a-backend` names the dispatch path for expert parallelism, so at `ep_size` 1 there is nothing for it to name, and the MoE runs replicated across TP.

The shared-expert, `SGLANG_DP_USE_GATHERV`, and `SGLANG_DP_USE_REDUCE_SCATTER` settings control the `all_gatherv` / `reduce_scatterv` path. Shared-expert locality and gatherv default to off; reduce-scatter defaults to on for ROCm. The commands only set the values that differ from those defaults. That is what decode's TP MoE uses and what prefill's all-to-all replaces, so leaving them on under EP8 would run both over the same tokens.

The rest of the difference is the prefill-only set already described above: `--chunked-prefill-size`, `--disable-cuda-graph`, and the linker flags with `--optimistic-prefill-attempts` / `--enable-cache-report`. Decode replaces the first two with `--cuda-graph-bs-decode`.

On the prefill env:

* `SGLANG_AMD_USE_FLYDSL_MEGA_MOE=1` is required. `--moe-a2a-backend megamoe` alone runs the DeepGEMM MegaMoE path, which has no ROCm kernel. Do not set this variable on decode: the scheduler reads it too, so it also changes how decode schedules batches.
* `SGLANG_AMD_FLYDSL_MEGA_MOE_MTPR` must be at least `--chunked-prefill-size / attn_dp_size`, here 65536 / 8. Below that SGLang reports no error and runs fused MoE for the extra tokens. 8192 is the default, so the command omits this variable. Set it explicitly if you increase the per-rank chunk size.
* `MORI_SHMEM_HEAP_SIZE` sizes the MORI symmetric heap, where MegaMoEv2 allocates its dispatch and combine buffers. It needs about 0.51 GiB per 1024 tokens of MTPR — about 4.2 GiB at 8192 — and the 4 GiB default fails during CUDA graph capture. SGLang allocates the whole heap at the first MegaMoE forward pass, after the KV cache pool, so it consumes the GPU memory `--mem-fraction-static` left free whether MegaMoE fills it or not. 8 GiB fits next to `0.92`; [§3.6](#3-6-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) runs at `0.85` and can afford more.

> [!NOTE]
>
> These recipes leave `MORI_SHMEM_MODE` unset and use MORI's default `STATIC_HEAP`. The pinned Aiter MegaMoEV2 implementation allocates its buffers through `mori_shmem_create_tensor()` / `shmem_malloc()` after SGLang initializes MORI shmem. This requires the symmetric heap; the separate MORI CCO allocator is not used by this implementation. If your launch script exports `MORI_SHMEM_MODE=ISOLATION`, unset it before launching MegaMoE: its peer-pointer lookup requires the heap mapping. Keep `MORI_SHMEM_HEAP_SIZE=8G` for the prefill recipe above.

**Both roles set `SGLANG_DSV4_UNIFIED_KV_FP8=1` at every Pro Official PD concurrency**, which splits the unified KV cache pool into an FP8 nope pool and a BF16 rope pool. It requires the `unified_kv_triton` kernels, which the base cell already selects through `SGLANG_HACK_FLASHMLA_BACKEND`. The value must match on both roles — the PD handshake compares KV cache layouts and fails if they differ. The DSpark draft worker stays BF16 either way.

#### Router configuration

Run the two roles on separate nodes with the same transport and IB device, then front them with the router the card prints. On MI355X it is cache-aware: `--policy consistent_hashing` keeps a conversation on the prefill worker that already holds its prefix, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` stop that affinity from starving the peer. The TP-only presets keep `--decode-policy round_robin`. Pro Official's DP-attention presets (concurrency 192–512, including the High-Throughput preset at 256) instead use:

```bash Router settings theme={null}
SGLANG_ROUTER_P2C_LOAD_METRIC=least_requests \
python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://<prefill-host>:30000 \
  --decode http://<decode-host>:30100 \
  --host 0.0.0.0 --port 8000 \
  --policy consistent_hashing --dp-aware \
  --decode-policy power_of_two \
  --worker-startup-check-interval 1 \
  --cache-threshold 0.3 \
  --balance-abs-threshold 2 --balance-rel-threshold 1.1 \
  --disable-circuit-breaker --health-failure-threshold 100 \
  --health-check-timeout-secs 600 --health-check-interval-secs 30
```

Power-of-two compares in-flight request counts across decode DP ranks. The one-second load polling interval replaces the 30-second default. Check for `Power-of-two load metric: LeastRequests` in the router startup log.

The multi-node router presets retain the default request timeout. The standalone high-throughput Playground preset uses `--request-timeout-secs 14400`. At high concurrency the prefill role queues many requests. When the default 1800 s passes the router aborts the request, and an abort that lands while MORI is writing KV cache over RDMA leaves the transfer neither finished nor failed: that prefill worker can no longer send KV cache to decode, and you have to restart it.

> [!WARNING]
>
> `--request-timeout-secs` is the timeout of the HTTP request from the router to the worker, not a limit on queue time. At 14400 the router also waits four hours before giving up on a worker that has stopped responding. That is acceptable where prefill queues are long, which is why the lower-concurrency recipes keep the default. Lower it again if you lower the concurrency.

#### Additional Pro Official concurrency points

The panel keeps its c16 / c48 / c256 strategy presets. Apply the following overrides to the **Multi-Nodes** commands to reproduce the optimized recipe's endpoints. Here `cN` means a target concurrency of N.

**c1 (replaces c4):** start from Low-Latency. Keep prefill TP4, set decode to `--tp 4`, and set `--max-running-requests 8` on both roles. Replace decode's graph list with `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8`. Keep DSpark block size 6 (7 draft tokens), `MORI_MAX_DISPATCH_TOKENS_DECODE=448`, and both kernel switches on both roles. The source also sets `SGLANG_MLA_HCA_KV_SPLITS=8` on both roles; the FP8 KV path does not read this setting. Keep these 8 server slots and graph sizes when benchmarking with client concurrency 1; do not reduce them to 2 using the strategy presets' `N*2` rule.

**c32:** start from Low-Latency, keep TP4 → TP8 and block size 6, set both admission limits to 64, and capture decode graph sizes 1–64. Both roles use FP8 KV and mxscale, as at c16 and c48.

**c192 / c384 / c512:** start from High-Throughput. Keep TP8+DP8+EP8 prefill → TP8+DP8 decode, DSpark block size 3 (4 draft tokens), and decode dispatch budget 256. Set both admission limits to 384 / 768 / 1024 and capture decode graph sizes 1–48 / 1–96 / 1–128, respectively. Keep FP8 KV on both roles, mxscale on decode only, and the power-of-two router settings above. At c384 and c512, add `--request-timeout-secs 14400` to the router.

**c512 prefill EPLB:** additionally set these environment variables on the prefill worker:

```bash Prefill environment theme={null}
export SGLANG_AITER_MEGA_RANK_SYNC=1
export SGLANG_AITER_MEGA_EPLB_PREFILL_ONLY=1
export SGLANG_AITER_MEGA_EPLB_FUSED_MAP_RECORD=1
```

Append these flags to its launch command:

```text Prefill flags theme={null}
--enable-eplb
--eplb-rebalance-num-iterations 200
--expert-distribution-recorder-mode stat
```

Keep shared-expert fusion enabled. Rank synchronization lets idle DP ranks participate in MegaMoE dispatch; the prefill-only mode records load and advances rebalancing on synchronized extend passes. Do not set `--expert-balancedness-report-mode` with this mode: it can deadlock prefill on the first request. Decode uses TP MoE and receives none of these EPLB settings. EPLB is specific to c512; the c256 panel preset leaves it disabled.

### 3.8 Unified Cache External Linker (UMBP)

The **unified cache external linker** is the second way to move KV cache off the GPU. The linker is the SGLang feature, and you enable it with `--enable-unified-cache-external-linker`. UMBP is one store the linker can use; you select it with `--unified-cache-external-linker-backend mori`. The Playground card uses the feature name, not the store name.

The linker is an **alternative to [HiCache](#3-3-hicache-hierarchical-kv-caching), not a tier inside it**. HiCache is a hierarchy — GPU, then pinned host memory, then optionally a storage backend. The linker connects the unified radix tree directly to the external store, with no host cache tier in between, so pages load and offload against that store. SGLang rejects the two together, so enabling the **Unified Cache External Linker** card in the [Playground above](#playground) strips the whole HiCache flag family from the command:

```bash Command theme={null}
--enable-unified-cache-external-linker \
--unified-cache-external-linker-backend mori
```

For **MI355X Pro / Pro Official FP4**, the Playground keeps the two Enable controls mutually exclusive: turning either on automatically turns the other off. You can also leave both off.

The **Backend** knob picks the linker backend. `mori` is the UMBP pool; `mooncake` drives the same direct-linker path against a Mooncake store instead.

**The linker does not need PD disaggregation.** It works in aggregated serving as well, with the **PD Disagg** card set to Off. The §3.7 pairs use it on their prefill role because that is where the KV is produced, but that is a property of those recipes, not a requirement of the linker. Turn the card on next to any command whose selection satisfies the condition below.

**DP Attention is not required.** The linker runs under pure TP as well, and the Pro Official low-latency and balanced prefill roles in [§3.7](#3-7-pd-disaggregation-on-mi355x-mori-io) are exactly that shape.

DP Attention does change how much the tier holds, so it is worth understanding before you size one. The linker keys its objects by rank, and MLA KV is replicated across TP. Under pure TP, a TP8 worker therefore opens eight separate keyspaces, and all eight hold copies of the same tokens: the tier stores an eighth of the distinct tokens that its byte budget suggests. At TP4 it stores a quarter. DP Attention collapses the keys onto one shared keyspace, so the whole budget counts. Size the pool for the copies you will actually get, and do not read the byte budget as a distinct-token figure.

Every MI355X Pro / Pro Official multi-node prefill role uses a **standalone** UMBP tier: a separate `umbp_standalone_server` process on the prefill node owns the DRAM pool, and the prefill server attaches to it over a Unix socket. Start the tier server first; the 1500 GB DRAM budget below is what the verified runs used:

```bash Command theme={null}
mkdir -p /tmp/umbp
UMBP_DRAM_CAPACITY=1500000000000 \
UMBP_DRAM_USE_HUGEPAGES=1 \
UMBP_SSD_ENABLED=0 \
MORI_UMBP_LOG_LEVEL=info \
<mori>/umbp_standalone_server unix:///tmp/umbp/standalone.grpc.sock
```

Reserve enough hugepages for the 1.5 TB tier on a host with at least 3 TB DRAM, leaving memory for model loading and the worker. Wait for `host memory registered for GPU access` before starting prefill. For the pinned ROCm image, the tier binary is `/sgl-workspace/mori/python/mori/umbp_standalone_server`; include `/sgl-workspace/mori/python/mori` in `LD_LIBRARY_PATH`.

If the tier also runs in Docker, pass the ROCm and RDMA devices and hugepage access through to that container and mount `-v /tmp/umbp:/tmp/umbp` in **both** containers. The generated Prefill Docker command already includes that socket-directory mount.

The standalone server defaults to `unix:///run/umbp/standalone/node0.grpc.sock` when neither an address argument nor `UMBP_STANDALONE_ADDRESS` is supplied (`UMBP_NODE_ID` replaces `node0` when set). SGLang's client has no default standalone address: set `UMBP_STANDALONE_ADDRESS` to enable this mode. These commands use an explicit socket under `/tmp/umbp` so the server and worker share the same path.

UMBP defaults to a 4 GiB DRAM capacity, hugepages disabled, and a 30-second startup wait. This recipe uses 1.5 TB with hugepages enabled and keeps the default startup wait. It also sets `UMBP_SSD_ENABLED=0` to disable the SSD tier (whose config default is enabled), and `MORI_UMBP_LOG_LEVEL=info` to expose the readiness log (the default log level is error).

Then launch the prefill role with the socket in its environment. The Deployment panel and Playground include this address and use the default 30-second startup wait.

```bash Command theme={null}
UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp/standalone.grpc.sock \
sglang serve ...
```

Two further limits are worth knowing before you size a run. The pool is a per-node process, so a worker that spans nodes would shard its keyspace by node. And under PD disaggregation the §3.7 pairs run the linker on the prefill role only, so the decode worker offloads nothing; in aggregated serving the single worker offloads its own KV.


# DeepSeek-V4.1
Source: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1

Deploy DeepSeek-V4.1 Flash with SGLang — launch recipes, feature compatibility, and tuning notes for GB300, H200, B200, B300 and MI355X.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> DeepSeek-V4.1 Flash support has **not shipped in an SGLang release yet**, so a stock `pip install sglang` cannot serve it. Use the preview Docker image below.
>
> **Docker**
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker).
>
> **NVIDIA GPUs** — every NVIDIA cell on this page uses the preview build `lmsysorg/sglang:dev-dsv41` (H200 / B200 / B300 / GB300); the command panel picks it automatically.
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-dsv41
>
> docker run --gpus all \
>     --shm-size 32g \
>     -p 30000:30000 \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     --env "HF_TOKEN=<your-hf-token>" \
>     --ipc=host \
>     lmsysorg/sglang:dev-dsv41 \
>     sglang serve <use args below>
> ```
>
> **AMD GPUs (ROCm)** — AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 10 version.
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261021
>
> docker run \
>     --device=/dev/kfd --device=/dev/dri \
>     --group-add video \
>     --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
>     --shm-size 32g --ipc=host \
>     -p 30000:30000 \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     --env "HF_TOKEN=<your-hf-token>" \
>     lmsysorg/sglang-rocm:v0.5.22-rocm10-mi35x-20261021 \
>     sglang serve <use args below>
> ```

Pick your hardware + recipe to generate the launch command:

* **Low-Latency** — fastest reply for a single user. Pick for chat.
* **High-Throughput** — most tokens per second across many users. Best for batch jobs.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

* **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
* **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
* **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

## 1. Model Introduction

**DeepSeek-V4.1 Flash** is a sparse-attention Mixture-of-Experts model (`model_type: deepseek_v4.1`) served through SGLang's `dsv4` backend: 40 decoder layers, 384 routed experts at top-6 plus one shared expert, fp8 dense weights at a 32-wide `ue8m0` block scale with fp4 routed experts. It also carries **DSpark**, its own three-stage speculative draft, which the Low-Latency recipe turns on.

**Architecture.** Every layer keeps one 512-wide KV latent per position and uses it as both key and value for all 64 query heads, with no separate value projection. Each layer reads two stores that behave oppositely: **compressed latents**, produced only at source layers and shared forward, collapsing 2 positions into 1 in layers 2–19 and 1-to-1 from layer 20 on; and a **128-position sliding window**, recomputed per layer from that layer's own activations, so it can never be shared but also never grows with context. Consumers address a source's cache by plain `loc // ratio` arithmetic with no mapping table, so compressed state lives and dies with the full prefix for free. The residual stream is four parallel copies mixed by a per-token doubly stochastic matrix; each sublayer's mixing coefficients are consumed by the *next* sublayer, which lets that projection overlap the main GEMMs.

**Sparse retrieval.** KV source layers and index source layers are different lists — four of the former, eight of the latter. The four extra index layers produce no keys at all; they re-score layer 20's keys with their own query, so retrieval decisions are made twice as often as keys are stored. Each retrieving layer picks a top-512 candidate set.

**Engram.** An additive n-gram hash memory at two layers. Token ids are normalized *before* hashing, so `" The"`, `"the"` and `"THE"` cannot fork into separate rows. Its two fp8 tables are the largest single block of weight in the checkpoint, and by default they load row-sharded across the TP group, which costs an all-reduce per engram layer. `SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1` (**opt-in**) moves them to host memory instead: the freed HBM goes to the KV pool and output is bitwise unchanged — at the cost of host RAM, a longer load, and needing huge-page backing to keep the gather cheap. `SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT` trades the two all-reduces for page size: `shared` (the default) maps a single copy all ranks read and needs no reduce; `per_rank` splits that same table into one shard per rank and keeps the reduce, but is the only one the kernel reliably backs with 2 MiB pages.

**SWA bounded replay.** Decoder SWA bounded replay runs the early layers over the whole extend and the late layers over only each request’s last 128 tokens, sharing late-layer KV from source layer 20. It defaults on for supported CUDA/HIP configurations without context parallelism, including both PD roles on CUDA. Encoder SWA replay can be combined with decoder replay, subject to encoder-specific validation. Eager and breakable prefill are supported; full prefill graphs keep it disabled. With DP attention, automatic enablement requires attention TP 1 and the `none` or `megamoe` MoE backend on CUDA, or `none` on HIP. Use `--enable-decoder-swa-bounded-replay` to explicitly enable replay, subject to compatibility validation. Bounded replay is not numerically equivalent to full prefill. A request for prompt logprobs or for every hidden state runs its whole prompt through the late layers with full sliding windows; other requests, including their output-token logprobs, use the bounded computation. Use `--no-enable-decoder-swa-bounded-replay` to retain full prefill, for example for RL rollouts whose logprobs must match the trainer.

**Kernels.** At decode token counts these kernels are launch-bound rather than bandwidth-bound, so the work is fusion and overlap: the compressor projection, ratio-2 pooling, RoPE with fp4 quantization and indexer packing, paged fp4 indexer scoring with top-k resolved straight to cache addresses, and the mHC statistics and Sinkhorn — with compression, indexing and mHC overlapped against attention and FFN on dedicated streams. All of it is selected automatically; the environment switches it once sat behind were removed, so there is nothing to turn on. One caveat outranks the fusions: **output is not bitwise stable across batch composition today** — two default kernels are shape-guarded to a single token — and `--enable-deterministic-inference` is refused on this backend.

**Recommended generation:** reasoning evaluations were run at `temperature=1.0`, `top_p=0.95` with reasoning effort `max` (informational — do not hardcode these in application code).

**Resources:** [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash).

## 2. Configuration Tips

### Do not override the backends

`--attention-backend`, `--moe-runner-backend` and `--fp8-gemm-backend` are selected automatically from the model, hardware, forward mode and shape. On GB300 they resolve to `dsv4` / `flashinfer_mxfp4` / `flashinfer_cutedsl`. Confirm them in the startup log rather than passing them. On gfx950 the MI355X recipes pin `--fp8-gemm-backend aiter` and leave the other two to resolve.

Overriding them is the most common cause of a disappointing measurement: it leaves the 32-wide ue8m0 blocks on the Triton `_w8a8_block_fp8_matmul` fallback, which dominates the decode step and costs most of the model's bs=1 throughput. If your decode rate looks like a small fraction of what you expected, check the resolved backends first.

## 3. Advanced Usage

### 3.1 Reasoning

Enable the reasoning parser — `--reasoning-parser auto` resolves to `deepseek-v41` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer. The parser puts the thinking block in `reasoning_content` and the answer in `content`; without it both arrive concatenated in `content`.

The parser can only split a thinking block the model actually produced, and **thinking is off by default** (`SGLANG_DEFAULT_THINKING=false`). Sending `reasoning_effort` turns it on: any value other than `none` also switches `chat_template_kwargs.thinking` on for you. A request that carries neither comes back with an empty `reasoning_content` however the parser is configured.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="deepseek-ai/DeepSeek-V4.1-Flash",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     reasoning_effort="high",
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

On the request, `reasoning_effort` accepts the tiers `low` (50), `high` (75), `xhigh` (75), and `max` (100), or a float in `[0.0, 0.99]` that maps onto the model's 1–100 budget. An integer budget is reachable only through `chat_template_kwargs.reasoning_effort`, and that route does not switch thinking on by itself — pair it with `chat_template_kwargs.thinking`. `none` turns thinking off. The remaining OpenAI tiers, which have no V4.1 counterpart (`minimal`, `medium`), log a warning and fall back to the server-side default rather than erroring; that default is `high` (75), and `SGLANG_DSV41_REASONING_EFFORT` overrides it.

### 3.2 Tool Calling

Enable the tool-call parser — `--tool-call-parser auto` resolves to `deepseekv41` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. DeepSeek-V4.1 uses spaced DSML tool tags, which the stock DeepSeek-V4 detector does not parse — the `deepseekv41` detector is required.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather in a city",
>         "parameters": {
>             "type": "object",
>             "properties": {"location": {"type": "string"}},
>             "required": ["location"],
>         },
>     },
> }]
>
> resp = client.chat.completions.create(
>     model="deepseek-ai/DeepSeek-V4.1-Flash",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
>     reasoning_effort="high",
> )
> choice = resp.choices[0]
> print("finish_reason:", choice.finish_reason)
> print("tool_calls:", choice.message.tool_calls)
> print("reasoning:", getattr(choice.message, "reasoning_content", None))
> ```

A successful call returns `finish_reason: "tool_calls"` with a single `get_weather` entry whose arguments carry the requested location. If you see the raw DSML markup in `content` instead, the parser flag is missing.

### 3.3 Speculative Decoding (DSpark)

DSpark is DeepSeek-V4.1 Flash's own bundled draft — there is no EAGLE or MTP path for this model, and no `--speculative-num-steps` knob (the draft-token count is resolved from the checkpoint). Turn it on with `--speculative-algorithm DSPARK --speculative-dspark-block-size 5`, which is what the Low-Latency cells do.

What to expect:

* A DSpark step costs meaningfully more than a plain decode step, so the win at bs=1 is roughly the accept length divided by that fixed step cost. It is a real speed-up on interactive workloads, but not proportional to the number of draft tokens.
* Acceptance is strongly workload-dependent — highest on maths and code, lower on open-ended chat, lower still on agentic traces. Size your expectations from the workload you actually run, and measure it rather than assuming.
* The step cost does not move with accept length or with the draft block size, so tuning `--speculative-dspark-block-size` will not buy throughput. This is also why the High-Throughput recipe turns speculation off: at large batch the fixed step cost stops paying for itself.

### 3.4 PD Disaggregation

Prefill/decode disaggregation is validated token-identical against a single server across 32 greedy prompts, with GSM8K matching through the router. Use the **PD Disaggregation** card in the Playground to generate the prefill role, the decode role and the router command.

Both roles run MegaAttention (`--dsv4-attn-backend flashmla_mega`) by default on SM100/SM103, as a single server does. Prefill and decode must store KV in the same layout, so they must agree on `--dsv4-attn-backend` and `SGLANG_DSV4_KV_LAYOUT`; a decode server paired with a prefill whose layout differs fails its first request with a `KV layout mismatch` error. A decode on a GPU other than SM100/SM103 therefore needs `--dsv4-attn-backend flashmla` on the prefill too.

One deployment note: Mooncake needs the RDMA fabric visible inside the container, so launch with `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1`. Without it Mooncake selects its NVLink transport, which only serves buffers from its own allocator and fails with `Requested address ... not found`. If you hit that, force TCP with `MOONCAKE_PROTOCOL=tcp` and `MC_FORCE_TCP=1` — the Playground's Mooncake option sets both.

PD runs with DSpark: on a DSpark recipe both Playground roles keep the speculative flags, and the prefill and decode servers must both pass them. The server accepts this with Mooncake or NIXL transfer, static verification (`SGLANG_RAGGED_VERIFY_MODE`, `static` by default), and no context parallelism on decode (CP=1, DCP=1); it rejects any other combination at startup.

### 3.5 HiCache (Hierarchical KV Caching)

HiCache extends RadixAttention with a hierarchy of KV cache tiers, significantly expanding effective context capacity for long-context and multi-turn scenarios.

To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**: the Playground emits `--enable-hierarchical-cache` on top of the recipe's flags, and cold KV pages spill to CPU pinned memory (L2, GPU + CPU) instead of being dropped from the GPU pool. The host pool is sized by ratio (`--hicache-ratio 2`), not by a fixed `--hicache-size`.

The Write policy knob controls the GPU → CPU write and defaults to `write_through` (the upstream default): every page is mirrored to the CPU tier as it is written. `write_through_selective` backs up only hot data and `write_back` defers the copy to eviction, trading cache freshness for host-side I/O.

The card is not offered on MI355X: hierarchical cache has not been validated on ROCm for this model yet.

Only the L2 tier is exposed here. For the storage (L3) tier and the canonical flag set, see the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices) and the [HiCache documentation](../../../docs/advanced_features/hicache).

### 3.6 Agentic Long-Context on MI355X (FP4, DSpark)

Agentic replay is a different workload: long-context prompts, multi-turn sessions that reuse their own prefix across turns, and subagent branches that fan out within one session. Concurrency below counts live sessions, not requests. Every command keeps the engram tables on the host, and none adds a HiCache tier. The sessions fit the device pool.

TP2 covers concurrency 1 to 32 and TP4 covers 1 to 16; at 32 and 64 the same two GPUs also run DP attention with TP MoE.

**TP2 or TP4, concurrency 1–16:**

* TP2 raises `--mem-fraction-static` at concurrency 16; TP4 stops at 16 and holds 0.70, with a quarter of the checkpoint per rank leaving room to spare.
* A longer `--prefill-decode-interval` runs more decode rounds between prefill batches, trading TTFT for P90 inter-token latency.
* Up to concurrency 8 it pairs with `--schedule-policy lpm`, which admits the requests with the longest cached prefix first.
* These points set `--enable-amd-monokernel`. Do not set `SGLANG_HIP_DSV41_INDEX_K_PAGE8`: the monokernel refuses that layout.

The braced values in the command follow concurrency:

| Concurrency | 1 | 2 | 4 | 8 | 16 | 32 |
| - | - | - | - | - | - | - |
| `--max-running-requests` | 2 | 4 | 8 | 16 | 32 | 64 |
| `--swa-prefix-tails` | 128 | 128 | 256 | 512 | 1024 | 2048 |
| `--prefill-decode-interval` | 192 | 192 | 192 | 192 | 96 | 32 |
| `--schedule-policy` | lpm | lpm | lpm | lpm | fcfs | fcfs |
| `--mem-fraction-static`, TP2 | 0.70 | 0.70 | 0.70 | 0.70 | 0.80 | 0.80 |
| `--mem-fraction-static`, TP4 | 0.70 | 0.70 | 0.70 | 0.70 | 0.70 | — |

```bash Command theme={null}
SGLANG_DEFAULT_THINKING=1 \
SGLANG_DSV41_REASONING_EFFORT=high \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 \
SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_USE_AITER=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
TRITON_HIP_USE_ASYNC_COPY=0 \
ROCM_QUICK_REDUCE_QUANTIZATION=INT8 \
SGLANG_OPT_HIP_OPUS_SPARSE_PREFILL=true \
SGLANG_OPT_HIP_FLYDSL_MXFP8=1 \
SGLANG_HIP_FP4_INDEXER_ROWGROUP=1 \
HSA_NO_SCRATCH_RECLAIM=0 \
GPU_MAX_HW_QUEUES=2 \
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4.1-Flash \
  --trust-remote-code \
  --tp {2 or 4} \
  --ep-size 1 \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 5 \
  --enable-amd-monokernel \
  --enforce-shared-experts-fusion \
  --fp8-gemm-backend aiter \
  --enable-decoder-swa-bounded-replay \
  --cuda-graph-backend-prefill disabled \
  --mem-fraction-static {mem-fraction-static} \
  --chunked-prefill-size 16384 \
  --prefill-decode-interval {prefill-decode-interval} \
  --schedule-policy {schedule-policy} \
  --swa-prefix-tails {swa-prefix-tails} \
  --max-running-requests {max-running-requests} \
  --cuda-graph-max-bs-decode 64 \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --watchdog-timeout 3600
```

**TP2, concurrency 32:** the command above with the concurrency-32 column, without `--enable-amd-monokernel`, and with `SGLANG_HIP_DSV41_INDEX_K_PAGE8=1`.

**DP2, concurrency 32–64:** two DP attention ranks, one per GPU, with the MoE still tensor parallel across both. The braced values follow concurrency:

| Concurrency | 32 | 64 |
| - | - | - |
| `--mem-fraction-static` | 0.80 | 0.85 |
| `--prefill-decode-interval` | 32 | 16 |
| `--swa-prefix-tails` | 2048 | 4096 |

```bash Command theme={null}
SGLANG_DEFAULT_THINKING=1 \
SGLANG_DSV41_REASONING_EFFORT=high \
SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1 \
SGLANG_DSV41_ENGRAM_HOST_TABLE_LAYOUT=per_rank \
SGLANG_USE_AITER=1 \
SGLANG_USE_ROCM700A=0 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
TRITON_HIP_USE_ASYNC_COPY=0 \
ROCM_QUICK_REDUCE_QUANTIZATION=INT8 \
SGLANG_OPT_HIP_OPUS_SPARSE_PREFILL=true \
SGLANG_OPT_HIP_FLYDSL_MXFP8=1 \
SGLANG_HIP_FP4_INDEXER_ROWGROUP=1 \
SGLANG_HIP_DSV41_INDEX_K_PAGE8=1 \
HSA_NO_SCRATCH_RECLAIM=0 \
GPU_MAX_HW_QUEUES=5 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_USE_GATHERV=1 \
SGLANG_DP_USE_REDUCE_SCATTER=1 \
SGLANG_AITER_CAPTURE_COPY_IN=1 \
sglang serve \
  --model-path deepseek-ai/DeepSeek-V4.1-Flash \
  --trust-remote-code \
  --tp 2 \
  --attn-dp-size 2 \
  --ep-size 1 \
  --enable-dp-lm-head \
  --enable-prefill-delayer \
  --enable-dp-attention-local-control-broadcast \
  --tokenizer-worker-num 2 \
  --prefill-delayer-token-usage-low-watermark 0.7 \
  --load-balance-method total_requests \
  --speculative-algorithm DSPARK \
  --speculative-dspark-block-size 5 \
  --enforce-shared-experts-fusion \
  --fp8-gemm-backend aiter \
  --enable-decoder-swa-bounded-replay \
  --cuda-graph-backend-prefill disabled \
  --mem-fraction-static {mem-fraction-static} \
  --chunked-prefill-size 32768 \
  --prefill-decode-interval {prefill-decode-interval} \
  --swa-prefix-tails {swa-prefix-tails} \
  --max-running-requests 64 \
  --cuda-graph-max-bs-decode 64 \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --watchdog-timeout 3600
```

Front the server with `sglang_router --policy cache_aware --dp-aware`, so a multi-turn session lands on the DP rank already holding its prefix.


# Ernie4.5
Source: https://docs.sglang.io/cookbook/autoregressive/Ernie/Ernie4.5



## 1. Model Introduction

The **ERNIE-4.5** series is a family of large language models developed by Baidu. ERNIE (Enhanced Representation through Knowledge Integration) 4.5 represents an advanced version of the ERNIE series, optimized for general-purpose tasks and conversational scenarios.

ERNIE-4.5 delivers advanced features as below:

* **Heterogeneous Modality Structure**: MoE architecture that supports parameter sharing across modalities while allowing dedicated parameters for each individual modality, enhancing multimodal understanding without compromising, and even improving, performance on text-related tasks.
* **Vision Encoder**: Dedicated adaptive-resolution ViT with 2D RoPE and image packing; for video, adaptive frame sampling and timestamp rendering, supporting both shared and modality-specific visual processing.
* **Adapter**: Shared modality-bridging module with spatial and temporal compression to align vision to text embedding space, enabling cross-modal understanding without compromising text representations.
* **Multimodal Position Embedding**: Unified 3D RoPE (temporal, height, width) for vision and 1D RoPE for text in a single embedding space, supporting parameter sharing while encoding modality-specific positions.
* **Hardware Optimization**: Specifically tuned for AMD MI300X, MI325X, MI355X GPUs and Intel Xeon CPUs.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

## 4. API Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

The following example demonstrates deployment using ERNIE-4.5-21B-A3B-PT.

```shell Command theme={null}
python -m sglang.launch_server \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --tp 1
```

**Basic Python Client Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="baidu/ERNIE-4.5-21B-A3B-PT",
    messages=[
        {"role": "user", "content": "What is artificial intelligence?"}
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=1024
)

print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
**Artificial Intelligence (AI)** is the simulation of human intelligence processes by machines, particularly computer systems. These processes include **learning** (acquiring information and rules for using the information), **reasoning** (using rules to reach approximate or definite conclusions), and **self-correction**. AI encompasses a wide range of techniques, algorithms, and methodologies designed to enable machines to perform tasks that typically require human intelligence.

### Key Characteristics of AI:
...

### In Summary:
AI represents a transformative force with the potential to revolutionize industries and enhance human capabilities. However, its development requires careful consideration of ethical, legal, and social implications to ensure that it benefits society as a whole. As AI continues to evolve, ongoing dialogue among stakeholders will be crucial to balancing innovation with responsibility.
```

**Streaming Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="baidu/ERNIE-4.5-21B-A3B-PT",
    messages=[
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=2048,
    stream=True
)

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
Sure! Here’s a simple explanation of quantum computing:

### **Quantum Computing: Making Computers Super Fast (But Weird) Using Quantum Rules**

1. **Classic vs. Quantum Computers**
   - **Normal computers** use **bits** (0s and 1s) to store and process information.
   - **Quantum computers** use **qubits** (short for quantum bits). Unlike bits, qubits can be **0, 1, or both at the same time** (this is called **superposition**).

2. **Superposition: The Magic Behind Speed**
   - A single qubit can represent **0 and 1 simultaneously**, like a coin spinning in the air.
   - Many qubits working together (in something called **quantum parallelism**) can **check multiple possibilities at once**, making quantum computers much faster for certain problems.

3. **Entanglement: Making Qubits Link**
   - When qubits are **entangled**, their states are linked—changing one instantly affects the other, no matter how far apart they are (this is called **spooky action at a distance** by Einstein).
   - Entanglement allows quantum computers to process information in **very efficient ways**.

4. **What Quantum Computers Are Good At**
   - **Cracking encryption** (like RSA).
   - **Factoring large numbers** (used in encryption and cryptography).
   - **Searching unsorted databases** (way faster than classical computers).
   - **Simulating quantum systems** (like molecules for drug discovery).
   - **Optimizing problems** (like logistics or finance).

5. **Challenges & Current State**
   - Qubits are **fragile** and easily disturbed (called **decoherence**).
   - Engineers are working to keep qubits stable long enough to do useful calculations.
   - Today’s quantum computers are **small and experimental**, but the goal is to build powerful ones that outperform classical supercomputers.

### **Final Thought**
Quantum computing isn’t just a faster calculator—it’s a **new way of thinking about problems** using the weird laws of physics. While still new, it has the potential to revolutionize fields like medicine, AI, and cybersecurity.

Would you like an example of how a quantum computer might solve a problem? 😊
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: AMD MI300X GPU (1x)
* Model: ERNIE-4.5-21B-A3B-PT
* Tensor Parallelism: 1
* SGLang Version: 0.5.7

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Scenario Benchmark

* Model Deployment Command:

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path baidu/ERNIE-4.5-21B-A3B-PT \
  --tp 1
```

##### 5.1.1.1 Low Concurrency (Latency-Optimized)

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  58.72
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4219
Request throughput (req/s):              0.17
Input token throughput (tok/s):          103.90
Output token throughput (tok/s):         71.87
Peak output token throughput (tok/s):    245.00
Peak concurrent requests:                2
Total token throughput (tok/s):          175.77
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5869.86
Median E2E Latency (ms):                 1870.80
---------------Time to First Token----------------
Mean TTFT (ms):                          4152.58
Median TTFT (ms):                        36.81
P99 TTFT (ms):                           37498.23
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.07
Median TPOT (ms):                        4.09
P99 TPOT (ms):                           4.09
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.08
Median ITL (ms):                         4.08
P95 ITL (ms):                            4.14
P99 ITL (ms):                            4.20
Max ITL (ms):                            4.67
==================================================
```

##### 5.1.1.2 Medium Concurrency (Balanced)

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  34.30
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40773
Request throughput (req/s):              2.33
Input token throughput (tok/s):          1156.62
Output token throughput (tok/s):         1189.77
Peak output token throughput (tok/s):    1392.00
Peak concurrent requests:                21
Total token throughput (tok/s):          2346.39
Concurrency:                             14.14
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6060.62
Median E2E Latency (ms):                 6496.70
---------------Time to First Token----------------
Mean TTFT (ms):                          78.90
Median TTFT (ms):                        45.90
P99 TTFT (ms):                           234.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.99
Median TPOT (ms):                        12.16
P99 TPOT (ms):                           14.81
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.75
Median ITL (ms):                         11.48
P95 ITL (ms):                            12.24
P99 ITL (ms):                            34.85
Max ITL (ms):                            105.01
==================================================
```

##### 5.1.1.3 High Concurrency (Throughput-Optimized)

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  66.63
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    252449
Request throughput (req/s):              7.50
Input token throughput (tok/s):          3749.79
Output token throughput (tok/s):         3792.28
Peak output token throughput (tok/s):    4902.00
Peak concurrent requests:                113
Total token throughput (tok/s):          7542.06
Concurrency:                             90.33
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   12036.90
Median E2E Latency (ms):                 11782.16
---------------Time to First Token----------------
Mean TTFT (ms):                          104.86
Median TTFT (ms):                        84.62
P99 TTFT (ms):                           297.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.89
Median TPOT (ms):                        24.62
P99 TPOT (ms):                           26.91
---------------Inter-Token Latency----------------
Mean ITL (ms):                           23.66
Median ITL (ms):                         20.48
P95 ITL (ms):                            45.57
P99 ITL (ms):                            54.31
Max ITL (ms):                            185.12
==================================================
```

#### 5.1.2 Reasoning Scenario Benchmark

##### 5.1.2.1 Low Concurrency

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  185.11
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  44462
Total generated tokens (retokenized):    44423
Request throughput (req/s):              0.05
Input token throughput (tok/s):          32.96
Output token throughput (tok/s):         240.19
Peak output token throughput (tok/s):    245.00
Peak concurrent requests:                2
Total token throughput (tok/s):          273.15
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   18508.84
Median E2E Latency (ms):                 19866.81
---------------Time to First Token----------------
Mean TTFT (ms):                          32.59
Median TTFT (ms):                        32.14
P99 TTFT (ms):                           38.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.13
Median TPOT (ms):                        4.13
P99 TPOT (ms):                           4.20
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.16
Median ITL (ms):                         4.12
P95 ITL (ms):                            4.31
P99 ITL (ms):                            4.36
Max ITL (ms):                            7.28
==================================================
```

##### 5.1.2.2 Medium Concurrency

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  263.48
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  318306
Total generated tokens (retokenized):    317984
Request throughput (req/s):              0.30
Input token throughput (tok/s):          150.55
Output token throughput (tok/s):         1208.09
Peak output token throughput (tok/s):    1408.00
Peak concurrent requests:                19
Total token throughput (tok/s):          1358.64
Concurrency:                             14.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   47249.55
Median E2E Latency (ms):                 47828.67
---------------Time to First Token----------------
Mean TTFT (ms):                          62.77
Median TTFT (ms):                        57.10
P99 TTFT (ms):                           93.70
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.92
Median TPOT (ms):                        12.09
P99 TPOT (ms):                           12.50
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.86
Median ITL (ms):                         12.04
P95 ITL (ms):                            12.68
P99 ITL (ms):                            13.61
Max ITL (ms):                            39.94
==================================================
```

##### 5.1.2.3 High Concurrency

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  428.30
Total input tokens:                      158939
Total input text tokens:                 158939
Total input vision tokens:               0
Total generated tokens:                  1301025
Total generated tokens (retokenized):    1299877
Request throughput (req/s):              0.75
Input token throughput (tok/s):          371.09
Output token throughput (tok/s):         3037.63
Peak output token throughput (tok/s):    3880.00
Peak concurrent requests:                69
Total token throughput (tok/s):          3408.73
Concurrency:                             57.08
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   76392.58
Median E2E Latency (ms):                 79698.73
---------------Time to First Token----------------
Mean TTFT (ms):                          92.79
Median TTFT (ms):                        78.71
P99 TTFT (ms):                           168.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          18.81
Median TPOT (ms):                        19.15
P99 TPOT (ms):                           19.81
---------------Inter-Token Latency----------------
Mean ITL (ms):                           18.77
Median ITL (ms):                         18.77
P95 ITL (ms):                            19.86
P99 ITL (ms):                            42.08
Max ITL (ms):                            74.36
==================================================
```

#### 5.1.3 Summarization Scenario Benchmark

##### 5.1.3.1 Low Concurrency

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  18.59
Total input tokens:                      41941
Total input text tokens:                 41941
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4216
Request throughput (req/s):              0.54
Input token throughput (tok/s):          2256.43
Output token throughput (tok/s):         227.04
Peak output token throughput (tok/s):    245.00
Peak concurrent requests:                2
Total token throughput (tok/s):          2483.46
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1856.72
Median E2E Latency (ms):                 1513.87
---------------Time to First Token----------------
Mean TTFT (ms):                          86.66
Median TTFT (ms):                        72.30
P99 TTFT (ms):                           167.13
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.19
Median TPOT (ms):                        4.22
P99 TPOT (ms):                           4.30
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.20
Median ITL (ms):                         4.23
P95 ITL (ms):                            4.34
P99 ITL (ms):                            4.42
Max ITL (ms):                            5.68
==================================================
```

##### 5.1.3.2 Medium Concurrency

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  40.25
Total input tokens:                      300020
Total input text tokens:                 300020
Total input vision tokens:               0
Total generated tokens:                  41669
Total generated tokens (retokenized):    41646
Request throughput (req/s):              1.99
Input token throughput (tok/s):          7454.72
Output token throughput (tok/s):         1035.37
Peak output token throughput (tok/s):    1310.00
Peak concurrent requests:                20
Total token throughput (tok/s):          8490.09
Concurrency:                             14.37
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7229.56
Median E2E Latency (ms):                 7578.95
---------------Time to First Token----------------
Mean TTFT (ms):                          137.38
Median TTFT (ms):                        122.59
P99 TTFT (ms):                           485.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.04
Median TPOT (ms):                        14.24
P99 TPOT (ms):                           20.77
---------------Inter-Token Latency----------------
Mean ITL (ms):                           13.64
Median ITL (ms):                         12.36
P95 ITL (ms):                            14.72
P99 ITL (ms):                            57.39
Max ITL (ms):                            411.31
==================================================
```

##### 5.1.3.3 High Concurrency

* Benchmark Command:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model baidu/ERNIE-4.5-21B-A3B-PT \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  78.33
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total input vision tokens:               0
Total generated tokens:                  170000
Total generated tokens (retokenized):    169888
Request throughput (req/s):              4.09
Input token throughput (tok/s):          16262.33
Output token throughput (tok/s):         2170.20
Peak output token throughput (tok/s):    3005.00
Peak concurrent requests:                73
Total token throughput (tok/s):          18432.53
Concurrency:                             58.79
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   14392.52
Median E2E Latency (ms):                 14460.70
---------------Time to First Token----------------
Mean TTFT (ms):                          184.82
Median TTFT (ms):                        155.24
P99 TTFT (ms):                           379.82
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          26.97
Median TPOT (ms):                        28.31
P99 TPOT (ms):                           33.61
---------------Inter-Token Latency----------------
Mean ITL (ms):                           26.79
Median ITL (ms):                         20.55
P95 ITL (ms):                            47.55
P99 ITL (ms):                            145.64
Max ITL (ms):                            287.62
==================================================
```

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command:

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319 \
  --num-threads 1316
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Results:
  * ERNIE-4.5-21B-A3B-PT
  ```
  Accuracy: 0.865
  Invalid: 0.000
  Latency: 21.669 s
  Output throughput: 10359.790 token/s
  ```


# Ernie4.5-VL
Source: https://docs.sglang.io/cookbook/autoregressive/Ernie/Ernie4.5-VL



## 📝 Community Contribution Welcome

This guide is currently under development. We welcome community contributions!

If you have experience deploying **Ernie4.5-VL** with SGLang, please help us complete this documentation.

## 🚀 How to Contribute

```shell Command theme={null}
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
cd sglang-cookbook
git checkout -b add-ernie4-5-vl-guide
# Edit this file and submit a PR
```

## 📚 Reference

* [GLM-4.6V](../GLM/GLM-4.6V)

***

**Let's build this together!** 🌟


# Chroma-1.0
Source: https://docs.sglang.io/cookbook/autoregressive/FlashLabs/Chroma1.0



## 1. Model Introduction

[Chroma-1.0](https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma) is an open-source end-to-end speech conversation model developed by FlashLabs, focusing on the following core capabilities:

* **Real-time Speech Generation**: Supports low-latency speech synthesis, suitable for real-time conversational scenarios.
* **Customized Voice Cloning**: Capable of cloning and replicating specific speaker voice characteristics.
* **End-to-End Architecture**: Provides a complete processing workflow from speech to speech.
* **Speech Reasoning**: Equipped with reasoning capabilities to understand and process speech content.

## 2. Architecture Overview

**Chroma-1.0** utilizes a hybrid serving architecture rather than a direct SGLang deployment. This design choice is driven by:

1. **Complex Model Architecture**: The end-to-end speech processing pipeline involves specialized components that go beyond standard text generation loops.
2. **KV Cache & State Management**: The model requires custom handling of KV caches that differs from standard implementations.
3. **Batching Limitations**: The current implementation supports a batch size of 1, meaning SGLang's advanced continuous batching capabilities are not yet fully applicable.

Therefore, you will start the **FlashLabs Server**, which manages the overall workflow and selectively leverages SGLang for specific inference components where supported.

* **Outer Layer**: FlashLabs Server (Handles Audio I/O, State, and Model Logic)
* **Inner Engine**: SGLang Instance (Utilized for specific acceleration where applicable)

## 3. Installation & Setup

We recommend following these steps to set up the environment and prepare the model.

### Step 1: Get the Docker Image

Pull the official pre-built image from Docker Hub to ensure all dependencies are correctly configured.

```bash Command theme={null}
docker pull flashlabs/chroma:latest
```

### Step 2: Download Model Weights

Download the **Chroma-4B** weights from Hugging Face. You can choose one of the following methods:

**Method 1: Using Python (Recommended)**

```bash Command theme={null}
huggingface-cli download FlashLabs/Chroma-4B --local-dir Chroma-4B
```

**Method 2: Using Git Clone**

Make sure you have Git LFS installed before cloning.

```bash Command theme={null}
# Install Git LFS first
git lfs install

# Clone the repository
git clone https://huggingface.co/FlashLabs/Chroma-4B Chroma-4B
```

### Step 3: Download Chroma Codes (SGLang version)

```bash Command theme={null}
git clone https://github.com/FlashLabs-AI-Corp/Chroma-SGLang.git

cd Chroma-SGLang
```

### Step 4: Run the Server

```bash Command theme={null}
docker run -d \
  --gpus all \
  -p 8000:8000 \
  -w /app/Chroma-SGLang \
  -v "your_Chroma-SGLang_path":/app/Chroma-SGLang \
  -v "your_chroma_path":/model \
  -e CHROMA_MODEL_PATH=/model \
  -e DP_SIZE="1" \
  flashlabs/chroma:latest \
  /opt/conda/bin/python -m uvicorn api_server:app \
  --host 0.0.0.0 \
  --port 8000 \
  --workers 1
```

or run simply the following one line command

```bash Command theme={null}
docker-compose up -d
```

## 5. Client Usage Example

Once the server is running, you can interact with it using HTTP requests.

### Python Client

```python Example theme={null}
import requests
import base64

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}

payload = {
    "model": "chroma",
    "messages": [
        {
            "role": "system",
            "content": "You are Chroma, a voice agent developed by FlashLabs."
        },
        {
            "role": "user",
            "content": [
                {"type": "audio", "audio": "assets/question_audio.wav"}
            ]
        }
    ],
    "max_tokens": 1000,
    "return_audio": True
}

response = requests.post(url, json=payload, headers=headers)
result = response.json()

if result.get("audio"):
    audio_data = base64.b64decode(result["audio"])
    with open("output.wav", "wb") as f:
        f.write(audio_data)
    print("Audio saved to output.wav")
```

### OpenAI SDK Compatible Example

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    api_key="dummy",
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="chroma",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": [
                {"type": "audio", "audio": "assets/question_audio.wav"}
            ]
        }
    ],
    extra_body={
        "prompt_text": "I have not... I'm so exhausted, I haven't slept in a very long time. It could be because... Well, I used our... Uh, I'm, I just use... This is what I use every day. I use our cleanser every day, I use serum in the morning and then the moistu- daily moisturizer. That's what I use every morning.",
        "prompt_audio": "assets/ref_audio.wav",
        "return_audio": True
    }
)

print(response)
```

### CLI (cURL)

```bash Command theme={null}
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "chroma",
    "messages": [
      {
        "role": "system",
        "content": "You are Chroma, a voice agent developed by FlashLabs."
      },
      {
        "role": "user",
        "content": [
          {
            "type": "audio",
            "audio": "assets/question_audio.wav"
          }
        ]
      }
    ],
    "max_tokens": 1000,
    "return_audio": true
  }' | jq -r '.audio' | base64 -d > output.wav
```


# GLM-4.5
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.5



## 1. Model Introduction

[GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.

**Key Features:**

* **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
* **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
* **Hardware Optimization**: Specifically tuned for AMD MI300X/MI325X/MI355X GPUs
* **High Performance**: Optimized for both throughput and latency scenarios

**Available Models:**

* **BF16 (Full precision)**: [zai-org/GLM-4.5](https://huggingface.co/zai-org/GLM-4.5) - Recommended for MI300X/MI325X/MI355X
* **FP8 (8-bit quantized)**: [zai-org/GLM-4.5-FP8](https://huggingface.co/zai-org/GLM-4.5-FP8) - Recommended for MI300X/MI325X/MI355X

**License:**

Please refer to the [official GLM-4.5 model card](https://huggingface.co/zai-org/GLM-4.5) for license details.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
* **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

GLM-4.5 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.5 \
  --reasoning-parser glm45 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="zai-org/GLM-4.5",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

> [!NOTE]
>
> **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.

GLM-4.5 supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.5 \
  --reasoning-parser glm45 \
  --tool-call-parser glm45 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/GLM-4.5",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

#### 4.2.3 Thinking Budget

Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:

```python Example theme={null}
import openai
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
response = client.chat.completions.create(
    model="zai-org/GLM-4.5",
    messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
    max_tokens=1024,
    extra_body={
        "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
        "custom_params": {"thinking_budget": 512},
    },
)
print(response)
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
* Model: GLM-4.5
* Tensor Parallelism: 8
* SGLang Version: 0.5.6.post1

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Test Scenarios

Three core scenarios reflect real-world usage patterns:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Scenario</th>
      <th>Input Length</th>
      <th>Output Length</th>
      <th>Use Case</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Chat**</td>
      <td>1K</td>
      <td>1K</td>
      <td>Most common conversational AI workload</td>
    </tr>

    <tr>
      <td>**Reasoning**</td>
      <td>1K</td>
      <td>8K</td>
      <td>Long-form generation, complex reasoning tasks</td>
    </tr>

    <tr>
      <td>**Summarization**</td>
      <td>8K</td>
      <td>1K</td>
      <td>Document summarization, RAG retrieval</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 Concurrency Levels

Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):

* **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
* **Medium Concurrency**: `--max-concurrency 16` (Balanced)
* **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)

#### 5.1.3 Number of Prompts

For each concurrency level, configure `num_prompts` to simulate realistic user loads:

* **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
* **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
* **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)

***

#### 5.1.4 Benchmark Commands

**Scenario 1: Chat (1K/1K) - Most Important**

* **Model Deployment**

```bash Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.5 \
  --tp 8
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

**Scenario 3: Summarization (8K/1K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

#### 5.1.5 Understanding the Results

**Key Metrics:**

* **Request Throughput (req/s)**: Number of requests processed per second
* **Output Token Throughput (tok/s)**: Total tokens generated per second
* **Mean TTFT (ms)**: Time to First Token - measures responsiveness
* **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
* **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency

**Why These Configurations Matter:**

* **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
* **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
* **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks.
* **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.

**Interpreting Results:**

* Compare your results against baseline numbers for your hardware
* Higher throughput at same latency = better performance
* Lower TTFT = more responsive user experience
* Lower TPOT = faster generation speed

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.


# GLM-4.5V
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.5V



## 1. Model Introduction

[GLM-4.5V](https://huggingface.co/zai-org/GLM-4.5V) is a state-of-the-art multimodal vision-language model from ZhipuAI, built on the next-generation flagship text foundation model GLM-4.5-Air (106B parameters, 12B active). It achieves SOTA performance among models of the same scale across 42 public vision-language benchmarks. Through efficient hybrid training, GLM-4.5V focuses on real-world usability and enables full-spectrum vision reasoning across diverse visual content types.

**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X

GLM-4.5V introduces several key features:

* **Image Reasoning & Grounding** Scene understanding, complex multi-image analysis, and spatial recognition with precise visual element localization. Supports bounding box predictions with normalized coordinates (0-1000) for accurate object detection.
* **Video Understanding** Long video segmentation and event recognition, supporting comprehensive temporal analysis across extended video sequences.
* **GUI Agent Tasks** Screen reading, icon recognition, and desktop operation assistance for agent-based applications. Enables natural interaction with graphical user interfaces.
* **Complex Chart & Long Document Parsing** Research report analysis and information extraction from documents with text, charts, tables, and figures. Processes up to 64K tokens of multimodal context.
* **Thinking Mode Switch** Allows users to balance between quick responses and deep reasoning. Users can enable/disable Chain-of-Thought reasoning based on task requirements for improved accuracy and interpretability.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The GLM-4.5V offers models in various sizes and architectures, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.

**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.)
* **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder`.
* **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
* **Hardware Notes:**
  * **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
  * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
  * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
* **Additional Multimodal Parameters:**
  * `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
  * `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency.

**Example with full multimodal optimizations:**

```bash Command theme={null}
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
SGLANG_VLM_CACHE_SIZE_MB=0 \
python -m sglang.launch_server \
  --model-path zai-org/GLM-4.5V \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tp-size 8 \
  --enable-cache-report \
  --log-level info \
  --max-running-requests 64 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 8192 \
  --attention-backend fa3 \
  --mm-attention-backend fa3 \
  --mm-enable-dp-encoder \
  --enable-metrics
```

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Multi-Modal Inputs

GLM-4.5V supports both image and video inputs. Here's a basic example with image input:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Describe this image in detail."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 3.37s
Generated text: Auntie Anne's

CINNAMON SUGAR
1 x 17,000                    17,000

SUB TOTAL                    17,000

GRAND TOTAL                  17,000

CASH IDR                     20,000

CHANGE DUE                  3,000
```

**Multi-Image Input Example:**

GLM-4.5V can process multiple images in a single request for comparison or analysis:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
                }
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
                }
            },
            {
                "type": "text",
                "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 3.86s
Generated text: The first image shows a close - up of a few red taxis on a street with storefronts in the background. The taxis are in a line, and the scene has an urban, busy feel with visible shop displays. The second image is an aerial view of a large taxi parking area with numerous red and green taxis, some with hoods open. The scene is more open, with a parking lot layout, and includes elements like a bridge and grassy areas. Key differences: number of taxis (few vs many), perspective (close - up vs aerial), color variety (mostly red vs red and green), and setting (street with shops vs parking lot).
```

**Video Input Example:**

GLM-4.5V supports video understanding by processing video URLs:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
                }
            },
            {
                "type": "text",
                "text": "Describe what happens in this video."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Note:**

* For video processing, ensure you have sufficient context length configured (up to 64K tokens)
* Video processing may require more memory; adjust `--mem-fraction-static` accordingly
* You can also provide local file paths using `file://` protocol

**Example Output:**

```text Output theme={null}
Response costs: 3.89s
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
```

#### 4.2.2 Thinking Mode

GLM-4.5V supports thinking mode for enhanced reasoning. Enable thinking mode during deployment:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path zai-org/GLM-4.5V \
  --reasoning-parser glm45 \
  --tp 4 \
  --host 0.0.0.0 \
  --port 30000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

**Disable Thinking Mode:**

To disable thinking mode for a specific request:

```python Example theme={null}
response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
```

#### 4.2.3 Tool Calling

GLM-4.5V supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path zai-org/GLM-4.5V \
  --reasoning-parser glm45 \
  --tool-call-parser glm45 \
  --tp 4 \
  --host 0.0.0.0 \
  --port 30000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

#### 4.2.4 Thinking Budget

Beyond enabling/disabling the full reasoning mode (section 4.2.2), you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request:

```python Example theme={null}
import openai
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
response = client.chat.completions.create(
    model="zai-org/GLM-4.5V",
    messages=[{"role": "user", "content": "Describe this image briefly."}],
    max_tokens=1024,
    extra_body={
        "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
        "custom_params": {"thinking_budget": 512},
    },
)
print(response)
```

## 5. Benchmark

### 5.1 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.1.1 MMMU Benchmark

* Benchmark Command

```bash Command theme={null}
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64
```

* Test Result

```text Output theme={null}
Benchmark time: 616.6163094160147
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.867, 'num': 30},
 'Agriculture': {'acc': 0.567, 'num': 30},
 'Architecture_and_Engineering': {'acc': 0.667, 'num': 30},
 'Art': {'acc': 0.667, 'num': 30},
 'Art_Theory': {'acc': 0.9, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.8, 'num': 30},
 'Biology': {'acc': 0.6, 'num': 30},
 'Chemistry': {'acc': 0.533, 'num': 30},
 'Clinical_Medicine': {'acc': 0.667, 'num': 30},
 'Computer_Science': {'acc': 0.8, 'num': 30},
 'Design': {'acc': 0.867, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.667, 'num': 30},
 'Economics': {'acc': 0.833, 'num': 30},
 'Electronics': {'acc': 0.433, 'num': 30},
 'Energy_and_Power': {'acc': 0.733, 'num': 30},
 'Finance': {'acc': 0.767, 'num': 30},
 'Geography': {'acc': 0.667, 'num': 30},
 'History': {'acc': 0.8, 'num': 30},
 'Literature': {'acc': 0.9, 'num': 30},
 'Manage': {'acc': 0.733, 'num': 30},
 'Marketing': {'acc': 0.9, 'num': 30},
 'Materials': {'acc': 0.567, 'num': 30},
 'Math': {'acc': 0.8, 'num': 30},
 'Mechanical_Engineering': {'acc': 0.767, 'num': 30},
 'Music': {'acc': 0.3, 'num': 30},
 'Overall': {'acc': 0.732, 'num': 900},
 'Overall-Art and Design': {'acc': 0.683, 'num': 120},
 'Overall-Business': {'acc': 0.82, 'num': 150},
 'Overall-Health and Medicine': {'acc': 0.787, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.783, 'num': 120},
 'Overall-Science': {'acc': 0.707, 'num': 150},
 'Overall-Tech and Engineering': {'acc': 0.648, 'num': 210},
 'Pharmacy': {'acc': 0.9, 'num': 30},
 'Physics': {'acc': 0.933, 'num': 30},
 'Psychology': {'acc': 0.767, 'num': 30},
 'Public_Health': {'acc': 0.9, 'num': 30},
 'Sociology': {'acc': 0.667, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.732
```


# GLM-4.6
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.6



## 1. Model Introduction

[GLM-4.6](https://huggingface.co/zai-org/GLM-4.6) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.

As the latest iteration in the GLM series, GLM-4.6 achieves comprehensive enhancements across multiple domains, including real-world coding, long-context processing, reasoning, searching, writing, and agentic applications. Details are as follows:

* **Longer context window**: The context window has been expanded from 128K to 200K tokens, enabling the model to handle more complex agentic tasks.
* **Superior coding performance**: The model achieves higher scores on code benchmarks and demonstrates better real-world performance in applications such as Claude Code, Cline, Roo Code and Kilo Code, including improvements in generating visually polished front-end pages.
* **Advanced reasoning**: GLM-4.6 shows a clear improvement in reasoning performance and supports tool use during inference, leading to stronger overall capability.
* **More capable agents**: GLM-4.6 exhibits stronger performance in tool use and search-based agents, and integrates more effectively within agent frameworks.
* **Refined writing**: Better aligns with human preferences in style and readability, and performs more naturally in role-playing scenarios.

For more details, please refer to the [official GLM-4.6 documentation](https://docs.z.ai/guides/llm/glm-4.6).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **EAGLE Speculative Decoding:** Supported for GLM-4.5/4.6. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable.
* **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

GLM-4.6 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.6 \
  --reasoning-parser glm45 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="zai-org/GLM-4.6",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

> [!NOTE]
>
> **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.

GLM-4.6 supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.6 \
  --reasoning-parser glm45 \
  --tool-call-parser glm45 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/GLM-4.6",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="zai-org/GLM-4.6",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

#### 4.2.3 Thinking Budget

Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:

```python Example theme={null}
import openai
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
response = client.chat.completions.create(
    model="zai-org/GLM-4.6",
    messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
    max_tokens=1024,
    extra_body={
        "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
        "custom_params": {"thinking_budget": 512},
    },
)
print(response)
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8x), AMD MI300X (8x), AMD MI325X (8x), AMD MI355X (8x)
* Model: GLM-4.6
* Tensor Parallelism: 8
* SGLang Version: 0.5.6.post1

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Test Scenarios

Three core scenarios reflect real-world usage patterns:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Scenario</th>
      <th>Input Length</th>
      <th>Output Length</th>
      <th>Use Case</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Chat**</td>
      <td>1K</td>
      <td>1K</td>
      <td>Most common conversational AI workload</td>
    </tr>

    <tr>
      <td>**Reasoning**</td>
      <td>1K</td>
      <td>8K</td>
      <td>Long-form generation, complex reasoning tasks</td>
    </tr>

    <tr>
      <td>**Summarization**</td>
      <td>8K</td>
      <td>1K</td>
      <td>Document summarization, RAG retrieval</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 Concurrency Levels

Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):

* **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
* **Medium Concurrency**: `--max-concurrency 16` (Balanced)
* **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)

#### 5.1.3 Number of Prompts

For each concurrency level, configure `num_prompts` to simulate realistic user loads:

* **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
* **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
* **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)

***

#### 5.1.4 Benchmark Commands

**Scenario 1: Chat (1K/1K) - Most Important**

* **Model Deployment**

```bash Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.6 \
  --tp 8
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  63.82
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4210
Total generated tokens (retokenized):    4209
Request throughput (req/s):              0.16
Input token throughput (tok/s):          95.60
Output token throughput (tok/s):         65.97
Peak output token throughput (tok/s):    68.00
Peak concurrent requests:                2
Total token throughput (tok/s):          161.57
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6379.24
Median E2E Latency (ms):                 5085.00
---------------Time to First Token----------------
Mean TTFT (ms):                          155.57
Median TTFT (ms):                        149.79
P99 TTFT (ms):                           207.69
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.81
Median TPOT (ms):                        14.80
P99 TPOT (ms):                           14.84
---------------Inter-Token Latency----------------
Mean ITL (ms):                           14.82
Median ITL (ms):                         14.82
P95 ITL (ms):                            15.17
P99 ITL (ms):                            15.36
Max ITL (ms):                            25.05
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}

============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  72.06
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40725
Total generated tokens (retokenized):    40672
Request throughput (req/s):              1.11
Input token throughput (tok/s):          550.47
Output token throughput (tok/s):         565.14
Peak output token throughput (tok/s):    752.00
Peak concurrent requests:                20
Total token throughput (tok/s):          1115.61
Concurrency:                             13.71
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   12348.93
Median E2E Latency (ms):                 13164.81
---------------Time to First Token----------------
Mean TTFT (ms):                          196.08
Median TTFT (ms):                        155.22
P99 TTFT (ms):                           377.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.24
Median TPOT (ms):                        24.55
P99 TPOT (ms):                           30.42
---------------Inter-Token Latency----------------
Mean ITL (ms):                           23.92
Median ITL (ms):                         21.40
P95 ITL (ms):                            22.49
P99 ITL (ms):                            123.83
Max ITL (ms):                            486.54
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  138.50
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252162
Total generated tokens (retokenized):    251841
Request throughput (req/s):              3.61
Input token throughput (tok/s):          1803.78
Output token throughput (tok/s):         1820.61
Peak output token throughput (tok/s):    2900.00
Peak concurrent requests:                107
Total token throughput (tok/s):          3624.40
Concurrency:                             90.91
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   25183.97
Median E2E Latency (ms):                 23968.49
---------------Time to First Token----------------
Mean TTFT (ms):                          337.77
Median TTFT (ms):                        180.65
P99 TTFT (ms):                           906.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          49.97
Median TPOT (ms):                        52.20
P99 TPOT (ms):                           61.81
---------------Inter-Token Latency----------------
Mean ITL (ms):                           49.36
Median ITL (ms):                         35.05
P95 ITL (ms):                            124.91
P99 ITL (ms):                            187.69
Max ITL (ms):                            440.34
==================================================
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  666.64
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  44452
Total generated tokens (retokenized):    44387
Request throughput (req/s):              0.02
Input token throughput (tok/s):          9.15
Output token throughput (tok/s):         66.68
Peak output token throughput (tok/s):    68.00
Peak concurrent requests:                2
Total token throughput (tok/s):          75.83
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   66661.35
Median E2E Latency (ms):                 71902.36
---------------Time to First Token----------------
Mean TTFT (ms):                          160.21
Median TTFT (ms):                        140.32
P99 TTFT (ms):                           295.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.92
Median TPOT (ms):                        14.94
P99 TPOT (ms):                           15.02
---------------Inter-Token Latency----------------
Mean ITL (ms):                           14.96
Median ITL (ms):                         14.96
P95 ITL (ms):                            15.36
P99 ITL (ms):                            15.57
Max ITL (ms):                            19.06
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  503.30
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  318226
Total generated tokens (retokenized):    318025
Request throughput (req/s):              0.16
Input token throughput (tok/s):          78.82
Output token throughput (tok/s):         632.28
Peak output token throughput (tok/s):    752.00
Peak concurrent requests:                19
Total token throughput (tok/s):          711.09
Concurrency:                             13.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   87349.22
Median E2E Latency (ms):                 88248.04
---------------Time to First Token----------------
Mean TTFT (ms):                          228.54
Median TTFT (ms):                        142.78
P99 TTFT (ms):                           569.84
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          21.97
Median TPOT (ms):                        22.14
P99 TPOT (ms):                           22.47
---------------Inter-Token Latency----------------
Mean ITL (ms):                           21.91
Median ITL (ms):                         21.80
P95 ITL (ms):                            22.30
P99 ITL (ms):                            22.78
Max ITL (ms):                            137.19
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  772.28
Total input tokens:                      158939
Total input text tokens:                 158939
Total input vision tokens:               0
Total generated tokens:                  1300705
Total generated tokens (retokenized):    1299924
Request throughput (req/s):              0.41
Input token throughput (tok/s):          205.80
Output token throughput (tok/s):         1684.24
Peak output token throughput (tok/s):    2112.00
Peak concurrent requests:                68
Total token throughput (tok/s):          1890.05
Concurrency:                             56.17
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   135563.36
Median E2E Latency (ms):                 140888.88
---------------Time to First Token----------------
Mean TTFT (ms):                          232.45
Median TTFT (ms):                        145.59
P99 TTFT (ms):                           576.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.47
Median TPOT (ms):                        34.02
P99 TPOT (ms):                           35.10
---------------Inter-Token Latency----------------
Mean ITL (ms):                           33.30
Median ITL (ms):                         32.63
P95 ITL (ms):                            34.27
P99 ITL (ms):                            104.39
Max ITL (ms):                            155.65
==================================================
```

**Scenario 3: Summarization (8K/1K)**

* Low

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  65.11
Total input tokens:                      41941
Total input text tokens:                 41941
Total input vision tokens:               0
Total generated tokens:                  4210
Total generated tokens (retokenized):    4210
Request throughput (req/s):              0.15
Input token throughput (tok/s):          644.17
Output token throughput (tok/s):         64.66
Peak output token throughput (tok/s):    68.00
Peak concurrent requests:                2
Total token throughput (tok/s):          708.83
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6508.31
Median E2E Latency (ms):                 5263.36
---------------Time to First Token----------------
Mean TTFT (ms):                          189.48
Median TTFT (ms):                        159.23
P99 TTFT (ms):                           304.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          15.02
Median TPOT (ms):                        15.03
P99 TPOT (ms):                           15.27
---------------Inter-Token Latency----------------
Mean ITL (ms):                           15.04
Median ITL (ms):                         15.03
P95 ITL (ms):                            15.46
P99 ITL (ms):                            15.65
Max ITL (ms):                            24.20
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  76.43
Total input tokens:                      300020
Total input text tokens:                 300020
Total input vision tokens:               0
Total generated tokens:                  41589
Total generated tokens (retokenized):    41577
Request throughput (req/s):              1.05
Input token throughput (tok/s):          3925.47
Output token throughput (tok/s):         544.15
Peak output token throughput (tok/s):    752.00
Peak concurrent requests:                19
Total token throughput (tok/s):          4469.62
Concurrency:                             13.95
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   13329.63
Median E2E Latency (ms):                 14141.09
---------------Time to First Token----------------
Mean TTFT (ms):                          339.88
Median TTFT (ms):                        252.75
P99 TTFT (ms):                           906.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.37
Median TPOT (ms):                        25.73
P99 TPOT (ms):                           30.94
---------------Inter-Token Latency----------------
Mean ITL (ms):                           25.04
Median ITL (ms):                         21.68
P95 ITL (ms):                            22.69
P99 ITL (ms):                            146.98
Max ITL (ms):                            483.14
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.6 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  136.24
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total input vision tokens:               0
Total generated tokens:                  169680
Total generated tokens (retokenized):    169452
Request throughput (req/s):              2.35
Input token throughput (tok/s):          9350.32
Output token throughput (tok/s):         1245.44
Peak output token throughput (tok/s):    1984.00
Peak concurrent requests:                69
Total token throughput (tok/s):          10595.77
Concurrency:                             58.46
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   24889.40
Median E2E Latency (ms):                 25123.37
---------------Time to First Token----------------
Mean TTFT (ms):                          355.82
Median TTFT (ms):                        268.84
P99 TTFT (ms):                           858.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          46.62
Median TPOT (ms):                        49.04
P99 TPOT (ms):                           58.88
---------------Inter-Token Latency----------------
Mean ITL (ms):                           46.36
Median ITL (ms):                         32.46
P95 ITL (ms):                            135.23
P99 ITL (ms):                            204.27
Max ITL (ms):                            508.14
==================================================
```

#### 5.1.5 Understanding the Results

**Key Metrics:**

* **Request Throughput (req/s)**: Number of requests processed per second
* **Output Token Throughput (tok/s)**: Total tokens generated per second
* **Mean TTFT (ms)**: Time to First Token - measures responsiveness
* **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
* **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency

**Why These Configurations Matter:**

* **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
* **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
* **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks.
* **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.

**Interpreting Results:**

* Compare your results against baseline numbers for your hardware
* Higher throughput at same latency = better performance
* Lower TTFT = more responsive user experience
* Lower TPOT = faster generation speed

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* Test Result

```text Output theme={null}
Accuracy: 0.975
Invalid: 0.000
Latency: 16.574 s
Output throughput: 1194.637 token/s
```


# GLM-4.6V
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.6V



## 1. Model Introduction

GLM-4.6V series model includes two versions: GLM-4.6V (106B), a foundation model designed for cloud and high-performance cluster scenarios, and GLM-4.6V-Flash (9B), a lightweight model optimized for local deployment and low-latency applications. GLM-4.6V scales its context window to 128k tokens in training, and achieves SoTA performance in visual understanding among models of similar parameter scales. Crucially, GLM team integrated native Function Calling capabilities for the first time. This effectively bridges the gap between "visual perception" and "executable action" providing a unified technical foundation for multimodal agents in real-world business scenarios.

Beyond achieves SoTA performance across major multimodal benchmarks at comparable model scales. GLM-4.6V introduces several key features:

* **Native Multimodal Function Calling** Enables native vision-driven tool use. Images, screenshots, and document pages can be passed directly as tool inputs without text conversion, while visual outputs (charts, search images, rendered pages) are interpreted and integrated into the reasoning chain. This closes the loop from perception to understanding to execution. Please refer to this [example](#4-2-3-tool-calling).
* **Interleaved Image-Text Content Generation** Supports high-quality mixed media creation from complex multimodal inputs. GLM-4.6V takes a multimodal context—spanning documents, user inputs, and tool-retrieved images—and synthesizes coherent, interleaved image-text content tailored to the task. During generation it can actively call search and retrieval tools to gather and curate additional text and visuals, producing rich, visually grounded content.
* **Multimodal Document Understanding** GLM-4.6V can process up to 128K tokens of multi-document or long-document input, directly interpreting richly formatted pages as images. It understands text, layout, charts, tables, and figures jointly, enabling accurate comprehension of complex, image-heavy documents without requiring prior conversion to plain text.
* **Frontend Replication & Visual Editing** Reconstructs pixel-accurate HTML/CSS from UI screenshots and supports natural-language-driven edits. It detects layout, components, and styles visually, generates clean code, and applies iterative visual modifications through simple user instructions.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

### 2.1 Docker Installation (Recommended)

```shell Command theme={null}
docker pull lmsysorg/sglang:latest
```

**Advantages:**

* Ready to use out of the box, no manual environment configuration needed
* Avoids dependency conflict issues
* Easy to migrate between different environments

### 2.2 Build from Source

If you need to use the latest development version or require custom modifications, you can build from source:

```bash Command theme={null}
# Install SGLang using UV (recommended)
git clone https://github.com/sgl-project/sglang.git
cd sglang
uv venv
source .venv/bin/activate
uv pip install --prerelease=allow -e "python[all]" --index-url=https://pypi.org/simple
pip install nvidia-cudnn-cu12==9.16.0.29
# Install ffmpeg to support video input
sudo apt update
sudo apt install ffmpeg
```

**Use Cases:**

* Need to customize and modify SGLang source code
* Want to use the latest development features
* Participate in SGLang project development

For general installation instructions, you can also refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the interactive configuration generator below to customize your deployment settings. Select your hardware platform, model size, quantization method, and other options to generate the appropriate launch command.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.)
* **TP=8 Configuration**: When using Tensor Parallelism (TP) of 8, the vision attention's 12 heads cannot be evenly divided. You can resolve this by adding `--mm-enable-dp-encoder` (which the generator above handles automatically).
* **Fast Model Loading**: For large models (like the 106B version), you can speed up model loading by using `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'`.
* **Hardware Notes:**
  * **H100 (FP8):** Use the FP8 checkpoint for best memory efficiency.
  * **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
  * **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.
* **Additional Multimodal Parameters:**
  * `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3).
  * `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency.

**Example with full multimodal optimizations:**

```bash Command theme={null}
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
SGLANG_VLM_CACHE_SIZE_MB=0 \
python -m sglang.launch_server \
  --model-path zai-org/GLM-4.6V \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tp-size 8 \
  --enable-cache-report \
  --log-level info \
  --max-running-requests 64 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 8192 \
  --attention-backend fa3 \
  --mm-attention-backend fa3 \
  --mm-enable-dp-encoder \
  --enable-metrics
```

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Multi-Modal Inputs

GLM-4.6V supports image and video inputs via the OpenAI-compatible API.

**Image Input:**

```python Example theme={null}
import subprocess

curl_command = f"""
curl -s http://localhost:{30000}/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{{
    "model": "default",
    "messages": [
      {{
        "role": "user",
        "content": [
          {{
            "type": "image_url",
            "image_url": {{
              "url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
            }}
          }},
          {{
            "type": "text",
            "text": "What is the image"
          }}
        ]
      }}
    ],
    "temperature": "0",
    "max_completion_tokens": "1000",
    "max_tokens": "1000"
  }}'
"""

response = subprocess.check_output(curl_command, shell=True).decode()
print(response)
```

```text Output theme={null}
{"id":"b61596ca71394dd699fd8abd4f650c44","object":"chat.completion","created":1765259019,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a logo featuring the text \"SGL\" (in a bold, orange-brown font) alongside a stylized icon. The icon includes a network-like structure with circular nodes (suggesting connectivity or a tree/graph structure) and a tag with \"</>\" (a common symbol for coding, web development, or software). The color scheme uses warm orange-brown tones with a black background, giving it a tech-focused, modern aesthetic (likely representing a company, project, or tool related to software, web development, or digital technology).<|begin_of_box|>SGL logo (stylized text + network/coding icon)<|end_of_box|>","reasoning_content":"Okay, let's see. The image has a logo with the text \"SGL\" and a little icon on the left. The icon looks like a network or a tree structure with circles, and there's a tag with \"</>\" which is a common symbol for coding or web development. The colors are orange and brown tones, with a black background. So probably a logo for a company or project named SGL, maybe related to software, web development, or a tech company.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":2222,"total_tokens":2448,"completion_tokens":226,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
```

**Video Input:**

```python Example theme={null}
import subprocess

curl_command = f"""
curl -s http://localhost:{30000}/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{{
    "model": "default",
    "messages": [
      {{
        "role": "user",
        "content": [
          {{
            "type": "video_url",
            "video_url": {{
              "url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
            }}
          }},
          {{
            "type": "text",
            "text": "What is in the video"
          }}
        ]
      }}
    ],
    "temperature": "0",
    "max_completion_tokens": "1000",
    "max_tokens": "1000"
  }}'
"""

response = subprocess.check_output(curl_command, shell=True).decode()
print(response)
```

```text Output theme={null}
{"id":"520e0a079e5d4b17b82a6af619315a97","object":"chat.completion","created":1765259029,"model":"default","choices":[{"index":0,"message":{"role":"assistant","content":"The image is a still from a presentation by a man on a stage. He is pointing to a small pocket on his jeans and asking the audience what the pocket is for. The video is being shared by Evan Carmichael. The man then reveals that the pocket is for an iPod Nano.","reasoning_content":"Based on the visual evidence in the video, here is a breakdown of what is being shown:\n\n*   **Subject:** The video features a man on a stage, giving a presentation. He is wearing a black t-shirt and dark jeans.\n*   **Action:** The man is pointing to a pocket on his jeans. He is asking the audience a question about the purpose of this pocket.\n*   **Context:** The presentation is being filmed, and the video is being shared by \"Evan Carmichael,\" a well-known motivational speaker and content creator. The source of the clip is credited to \"JoshuaG.\"\n*   **Reveal:** The man then reveals the answer to his question. He pulls a small, white, rectangular device out of the pocket. He identifies this device as an \"iPod Nano.\"\n\nIn summary, the image is a still from a presentation where a speaker is explaining the purpose of the small pocket found on many pairs of jeans.","tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":151336}],"usage":{"prompt_tokens":30276,"total_tokens":30532,"completion_tokens":256,"prompt_tokens_details":null,"reasoning_tokens":0},"metadata":{"weight_version":"default"}}
```

#### 4.2.2 Thinking Mode

GLM-4.6V supports Thinking mode. Enable the reasoning parser during deployment:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.6V \
  --reasoning-parser glm45 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="zai-org/GLM-4.6V",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.3 Tool Calling

GLM-4.6V supports tool calling with vision capabilities. Pass tools in your API request:

```python Example theme={null}
from openai import OpenAI

openai_api_key = "EMPTY"
openai_api_base = "http://127.0.0.1:30000/v1"
client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)



tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current temperature for a given location.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and country e.g. Beijing, China",
                    }
                },
                "required": ["location"],
                "additionalProperties": False,
            },
        },
    }
]


messages = [
    {
        "role": "user",
        "content": "Please help me check today's weather in Beijing, and tell me whether the tool returned an image."
    },
    {
        "role": "assistant",
        "tool_calls": [
            {
                "id": "call_bk32t88BGpSdbtDgzT044Rh4",
                "type": "function",
                "function": {
                    "name": 'get_weather',
                    "arguments": '{"location":"Beijing, China"}'
                }
            }
        ]
    },
    {
        "role": "tool",
        "tool_call_id": "call_bk32t88BGpSdbtDgzT044Rh4",
        "content": [
            {
                "type": "text",
                "text": "Weather report generated: Beijing, November 7, 2025, sunny, temperature 2°C."
            },
            {
                "type": "image_url",
                "image_url": {
                     "url": "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
                }
            }
        ]
    },
]

response = client.chat.completions.create(
    model="zai-org/GLM-4.6V",
    messages=messages,
    timeout=900,
    tools=tools
)
print(response.choices[0].message.content.strip())
```

**Output Example:**

```text Output theme={null}
The weather in Beijing today (November 7, 2025) is sunny with a temperature of 2°C.

Yes, the tool returned an image (the SGL logo).
```

#### 4.2.4 Thinking Budget

Beyond the reasoning parser, you can cap the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor` and pass `Glm4MoeThinkingBudgetLogitProcessor` in the request — same as the [GLM-4.6 text model approach](./GLM-4.6#4-2-3-thinking-budget):

```python Example theme={null}
import openai
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
response = client.chat.completions.create(
    model="zai-org/GLM-4.6V",
    messages=[{"role": "user", "content": "Describe this image briefly."}],
    max_tokens=1024,
    extra_body={
        "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
        "custom_params": {"thinking_budget": 512},
    },
)
print(response)
```

## 5. Benchmark

### 5.1. Text Benchmark: Latency, Throughput and Accuracy

#### Command

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

#### Result Output

```text Output theme={null}
Accuracy: 0.925
Invalid: 0.000
Latency: 15.327 s
Output throughput: 1788.375 token/s
```

### 5.2. Multimodal Benchmark - Latency and Throughput

#### Command

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --port 30000 \
  --model zai-org/GLM-4.6V \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 128 \
  --max-concurrency 8
```

#### Result Output

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 8
Successful requests:                     128
Benchmark duration (s):                  89.27
Total input tokens:                      315390
Total input text tokens:                 8702
Total input vision tokens:               306688
Total generated tokens:                  66020
Total generated tokens (retokenized):    31037
Request throughput (req/s):              1.43
Input token throughput (tok/s):          3533.17
Output token throughput (tok/s):         739.59
Peak output token throughput (tok/s):    823.00
Peak concurrent requests:                12
Total token throughput (tok/s):          4272.76
Concurrency:                             7.67
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5349.20
Median E2E Latency (ms):                 5380.98
---------------Time to First Token----------------
Mean TTFT (ms):                          1724.04
Median TTFT (ms):                        1688.16
P99 TTFT (ms):                           6152.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.15
Median TPOT (ms):                        7.77
P99 TPOT (ms):                           23.97
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.00
Median ITL (ms):                         8.44
P95 ITL (ms):                            9.23
P99 ITL (ms):                            116.02
Max ITL (ms):                            173.48
==================================================
```

### 5.3. Multimodal Accuracy Benchmark - MMMU

#### Command

```shell Command theme={null}
python3 benchmark/mmmu/bench_sglang.py --response-answer-regex "<\|begin_of_box\|>(.*)<\|end_of_box\|>" --port 30000 --concurrency 64 --extra-request-body '{"max_tokens": 4096}'
```

#### Result Output

```text Output theme={null}
Benchmark time: 487.2229107860476
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.962, 'num': 26},
 'Agriculture': {'acc': 0.5, 'num': 30},
 'Architecture_and_Engineering': {'acc': 0.733, 'num': 15},
 'Art': {'acc': 0.833, 'num': 30},
 'Art_Theory': {'acc': 0.9, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.733, 'num': 30},
 'Biology': {'acc': 0.586, 'num': 29},
 'Chemistry': {'acc': 0.654, 'num': 26},
 'Clinical_Medicine': {'acc': 0.633, 'num': 30},
 'Computer_Science': {'acc': 0.76, 'num': 25},
 'Design': {'acc': 0.867, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.633, 'num': 30},
 'Economics': {'acc': 0.862, 'num': 29},
 'Electronics': {'acc': 0.5, 'num': 18},
 'Energy_and_Power': {'acc': 0.875, 'num': 16},
 'Finance': {'acc': 0.857, 'num': 28},
 'Geography': {'acc': 0.714, 'num': 28},
 'History': {'acc': 0.767, 'num': 30},
 'Literature': {'acc': 0.897, 'num': 29},
 'Manage': {'acc': 0.759, 'num': 29},
 'Marketing': {'acc': 1.0, 'num': 26},
 'Materials': {'acc': 0.833, 'num': 18},
 'Math': {'acc': 0.76, 'num': 25},
 'Mechanical_Engineering': {'acc': 0.619, 'num': 21},
 'Music': {'acc': 0.286, 'num': 28},
 'Overall': {'acc': 0.761, 'num': 803},
 'Overall-Art and Design': {'acc': 0.729, 'num': 118},
 'Overall-Business': {'acc': 0.884, 'num': 138},
 'Overall-Health and Medicine': {'acc': 0.773, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.78, 'num': 118},
 'Overall-Science': {'acc': 0.728, 'num': 136},
 'Overall-Tech and Engineering': {'acc': 0.671, 'num': 143},
 'Pharmacy': {'acc': 0.933, 'num': 30},
 'Physics': {'acc': 0.929, 'num': 28},
 'Psychology': {'acc': 0.733, 'num': 30},
 'Public_Health': {'acc': 0.933, 'num': 30},
 'Sociology': {'acc': 0.724, 'num': 29}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.761
```


# GLM-4.7
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.7



## 1. Model Introduction

[GLM-4.7](https://huggingface.co/zai-org/GLM-4.7) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and agent workflows.

GLM-4.7 brings improvements across all major domains:

* **Extended Context Window**: Expanded context window supporting even longer documents and complex multi-turn conversations
* **Enhanced Reasoning**: Improved reasoning capabilities with better chain-of-thought processing
* **Superior Coding**: Significantly improved code generation and understanding, with better real-world application performance
* **Advanced Tool Use**: More robust tool calling and agent capabilities for complex workflows
* **Optimized Performance**: Better throughput and latency characteristics across all hardware platforms

For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).

**Key Features:**

* **State-of-the-Art Reasoning**: Enhanced reasoning capabilities for the most complex problem-solving tasks
* **Multiple Quantizations**: BF16, FP8, and NVFP4 variants for different performance/memory trade-offs
* **Hardware Optimization**: Tuned for NVIDIA Blackwell (B200, GB200) and AMD MI300X/MI325X/MI355X GPUs
* **High Performance**: Optimized for both throughput and latency scenarios

**Available Models:**

* **BF16 (Full precision)**: [zai-org/GLM-4.7](https://huggingface.co/zai-org/GLM-4.7)
* **FP8 (8-bit quantized)**: [zai-org/GLM-4.7-FP8](https://huggingface.co/zai-org/GLM-4.7-FP8)
* **NVFP4 (4-bit, NVIDIA Blackwell)**: [nvidia/GLM-4.7-NVFP4](https://huggingface.co/nvidia/GLM-4.7-NVFP4)

**License:**

Please refer to the [official GLM-4.7 model card](https://huggingface.co/zai-org/GLM-4.7) for license details.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

**Docker Images by Hardware Platform:**

<table>
  <thead>
    <tr>
      <th>Hardware Platform</th>
      <th>Docker Image</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>NVIDIA H100 / H200 / B200</td>
      <td>`lmsysorg/sglang:v0.5.12`</td>
    </tr>

    <tr>
      <td>NVIDIA GB200 / B300 / GB300 (aarch64)</td>
      <td>`lmsysorg/sglang:v0.5.12-cu130`</td>
    </tr>

    <tr>
      <td>AMD MI300X / MI325X</td>
      <td>`lmsysorg/sglang:v0.5.12-rocm720-mi30x`</td>
    </tr>

    <tr>
      <td>AMD MI355X</td>
      <td>`lmsysorg/sglang:v0.5.12-rocm720-mi35x`</td>
    </tr>
  </tbody>
</table>

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

Pick a weight format by hardware: **NVFP4** on NVIDIA Blackwell (B200, GB200), **FP8** on H100/H200/AMD, **BF16** as the full-precision fallback. The recommended tensor-parallel size per platform:

<table>
  <thead>
    <tr>
      <th>Hardware</th>
      <th>NVFP4</th>
      <th>FP8</th>
      <th>BF16</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>B200 (8×, single node)</td>
      <td>tp=2 / 4 / 8</td>
      <td>tp=4 / 8</td>
      <td>tp=8</td>
    </tr>

    <tr>
      <td>GB200 (NVL72, 4× per tray)</td>
      <td>tp=2 / 4</td>
      <td>tp=4</td>
      <td>—</td>
    </tr>

    <tr>
      <td>H200 (8×)</td>
      <td>—</td>
      <td>tp=8</td>
      <td>tp=8</td>
    </tr>

    <tr>
      <td>AMD MI300X / MI325X / MI355X</td>
      <td>—</td>
      <td>tp=2 / 4 / 8</td>
      <td>tp=4 / 8</td>
    </tr>
  </tbody>
</table>

* **EAGLE Speculative Decoding:** Supported for GLM-4.7. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.
* **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count (see section 4.2.3).

For general GLM-4.x family launch guidance (AMD ROCm notes and more), see [Launch GLM-4.5 / GLM-4.6 / GLM-4.7 with SGLang](/cookbook/autoregressive/GLM/GLM-4.5). Per-hardware bench commands and flags are inline in §5.1 below.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

GLM-4.7 supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.7 \
  --reasoning-parser glm45 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="zai-org/GLM-4.7",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

> [!NOTE]
>
> **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.

GLM-4.7 supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.7 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/GLM-4.7",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="zai-org/GLM-4.7",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

#### 4.2.3 Thinking Budget

Limit the number of thinking tokens using `CustomLogitProcessor`. Launch with `--enable-custom-logit-processor`:

```python Example theme={null}
import openai
from sglang.srt.sampling.custom_logit_processor import Glm4MoeThinkingBudgetLogitProcessor

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="*")
response = client.chat.completions.create(
    model="zai-org/GLM-4.7",
    messages=[{"role": "user", "content": "Is Paris the Capital of France?"}],
    max_tokens=1024,
    extra_body={
        "custom_logit_processor": Glm4MoeThinkingBudgetLogitProcessor().to_str(),
        "custom_params": {"thinking_budget": 512},
    },
)
print(response)
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200, NVIDIA GB200, AMD MI300X/MI325X/MI355X (8x)
* Model: GLM-4.7-NVFP4 on NVIDIA Blackwell; GLM-4.7-FP8 or GLM-4.7 (BF16) on AMD
* SGLang Version: 0.5.12 (NVIDIA Blackwell), 0.5.6.post1 (AMD)
* Best per-GPU throughput config on B200: **TP=2 NVFP4 bf16-KV** (NVFP4 weights, no EP). Numbers below come from this config.

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Test Scenarios

Four core scenarios reflect real-world usage patterns:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Scenario</th>
      <th>Input Length</th>
      <th>Output Length</th>
      <th>Use Case</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Chat**</td>
      <td>1K</td>
      <td>1K</td>
      <td>Most common conversational AI workload</td>
    </tr>

    <tr>
      <td>**Reasoning**</td>
      <td>1K</td>
      <td>8K</td>
      <td>Long-form generation, complex reasoning tasks</td>
    </tr>

    <tr>
      <td>**Summarization**</td>
      <td>8K</td>
      <td>1K</td>
      <td>Document summarization, RAG retrieval</td>
    </tr>

    <tr>
      <td>**Throughput**</td>
      <td>4K</td>
      <td>1K</td>
      <td>Mixed RAG / agent / multi-turn conversation (used for the inline B200 / GB200 results below)</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 Concurrency Levels

Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):

* **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
* **Medium Concurrency**: `--max-concurrency 16` (Balanced)
* **High Concurrency**: `--max-concurrency 100` (Throughput-optimized) — the Throughput (4K/1K) scenario uses `--max-concurrency 128` to match the inline B200/GB200 results below.

#### 5.1.3 Number of Prompts

For each concurrency level, configure `num_prompts` to simulate realistic user loads:

* **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
* **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
* **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)

***

#### 5.1.4 Benchmark Commands

**Scenario 1: Chat (1K/1K) - Most Important**

* **Model Deployment**

```bash Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.7 \
  --tp 8
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

**Scenario 3: Summarization (8K/1K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

**Scenario 4: Throughput (4K/1K) — NVIDIA Blackwell with NVFP4**

The remaining sub-sections (§5.1.4.1 NVIDIA B200, §5.1.4.2 NVIDIA GB200) measure this scenario with `nvidia/GLM-4.7-NVFP4` weights and report the full `bench_serving` output verbatim. The same commands apply to other NVIDIA hardware after substituting the deployment line from §3.1.

> **Note**: These runs use EOS-enabled generation (no `--disable-ignore-eos`), so generated-token counts reflect natural model behavior rather than a strict fixed-OSL pin. Compare against other EOS-enabled runs at the same workload, not against fixed-output-length benchmarks.

#### 5.1.4.1 NVIDIA B200

**Model Deployment (NVIDIA B200, TP=2 NVFP4 — max tok/s/gpu config):**

```bash Command theme={null}
python -m sglang.launch_server \
  --model nvidia/GLM-4.7-NVFP4 \
  --tp-size 2 \
  --mem-fraction-static 0.85 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/GLM-4.7-NVFP4 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1024 \
  --num-prompts 5 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 1
Successful requests:                     5
Benchmark duration (s):                  25.07
Total input tokens:                      8105
Total generated tokens:                  2674
Request throughput (req/s):              0.20
Input token throughput (tok/s):          323.25
Output token throughput (tok/s):         106.65
Total token throughput (tok/s):          429.90
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5011.93
Median E2E Latency (ms):                 6441.44
---------------Time to First Token----------------
Mean TTFT (ms):                          179.61
Median TTFT (ms):                        169.05
P99 TTFT (ms):                           238.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.05
Median TPOT (ms):                        9.03
P99 TPOT (ms):                           9.16
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.05
Median ITL (ms):                         9.05
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/GLM-4.7-NVFP4 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1024 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  60.60
Total input tokens:                      179772
Total generated tokens:                  39657
Request throughput (req/s):              1.32
Input token throughput (tok/s):          2966.39
Output token throughput (tok/s):         654.37
Total token throughput (tok/s):          3620.76
Concurrency:                             14.01
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   10615.87
Median E2E Latency (ms):                 9985.45
---------------Time to First Token----------------
Mean TTFT (ms):                          267.39
Median TTFT (ms):                        177.26
P99 TTFT (ms):                           584.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          20.98
Median TPOT (ms):                        21.06
P99 TPOT (ms):                           24.88
---------------Inter-Token Latency----------------
Mean ITL (ms):                           20.92
Median ITL (ms):                         17.93
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/GLM-4.7-NVFP4 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1024 \
  --num-prompts 640 \
  --max-concurrency 128 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 128
Successful requests:                     640
Benchmark duration (s):                  172.95
Total input tokens:                      1453591
Total generated tokens:                  308740
Request throughput (req/s):              3.70
Input token throughput (tok/s):          8404.67
Output token throughput (tok/s):         1785.14
Total token throughput (tok/s):          10189.80
Concurrency:                             117.85
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   31848.20
Median E2E Latency (ms):                 28554.42
---------------Time to First Token----------------
Mean TTFT (ms):                          1598.40
Median TTFT (ms):                        298.88
P99 TTFT (ms):                           11015.96
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          65.94
Median TPOT (ms):                        65.81
P99 TPOT (ms):                           137.73
---------------Inter-Token Latency----------------
Mean ITL (ms):                           62.99
Median ITL (ms):                         35.44
==================================================
```

#### 5.1.4.2 NVIDIA GB200

**Model Deployment (NVIDIA GB200, TP=2 NVFP4 — max tok/s/gpu config):**

```bash Command theme={null}
python -m sglang.launch_server \
  --model nvidia/GLM-4.7-NVFP4 \
  --tp-size 2 \
  --mem-fraction-static 0.85 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/GLM-4.7-NVFP4 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1024 \
  --num-prompts 5 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 1
Successful requests:                     5
Benchmark duration (s):                  24.74
Total input tokens:                      8105
Total generated tokens:                  2674
Request throughput (req/s):              0.20
Input token throughput (tok/s):          327.65
Output token throughput (tok/s):         108.10
Total token throughput (tok/s):          435.75
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4944.47
Median E2E Latency (ms):                 6347.31
---------------Time to First Token----------------
Mean TTFT (ms):                          211.41
Median TTFT (ms):                        207.25
P99 TTFT (ms):                           226.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.86
Median TPOT (ms):                        8.84
P99 TPOT (ms):                           8.96
---------------Inter-Token Latency----------------
Mean ITL (ms):                           8.87
Median ITL (ms):                         8.85
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/GLM-4.7-NVFP4 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1024 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  60.40
Total input tokens:                      179772
Total generated tokens:                  39657
Request throughput (req/s):              1.32
Input token throughput (tok/s):          2976.52
Output token throughput (tok/s):         656.61
Total token throughput (tok/s):          3633.13
Concurrency:                             13.97
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   10611.51
Median E2E Latency (ms):                 9956.84
---------------Time to First Token----------------
Mean TTFT (ms):                          338.14
Median TTFT (ms):                        215.25
P99 TTFT (ms):                           915.40
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          20.87
Median TPOT (ms):                        21.36
P99 TPOT (ms):                           27.05
---------------Inter-Token Latency----------------
Mean ITL (ms):                           20.77
Median ITL (ms):                         16.53
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/GLM-4.7-NVFP4 \
  --dataset-name random \
  --random-input-len 4096 \
  --random-output-len 1024 \
  --num-prompts 640 \
  --max-concurrency 128 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 128
Successful requests:                     640
Benchmark duration (s):                  181.89
Total input tokens:                      1453591
Total generated tokens:                  309221
Request throughput (req/s):              3.52
Input token throughput (tok/s):          7991.59
Output token throughput (tok/s):         1700.04
Total token throughput (tok/s):          9691.63
Concurrency:                             118.86
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   33690.47
Median E2E Latency (ms):                 30421.55
---------------Time to First Token----------------
Mean TTFT (ms):                          1353.16
Median TTFT (ms):                        383.52
P99 TTFT (ms):                           8940.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          69.88
Median TPOT (ms):                        71.77
P99 TPOT (ms):                           131.75
---------------Inter-Token Latency----------------
Mean ITL (ms):                           67.23
Median ITL (ms):                         33.46
==================================================
```

#### 5.1.5 Understanding the Results

**Key Metrics:**

* **Request Throughput (req/s)**: Number of requests processed per second
* **Output Token Throughput (tok/s)**: Total tokens generated per second
* **Mean TTFT (ms)**: Time to First Token - measures responsiveness
* **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
* **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency

**Why These Configurations Matter:**

* **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
* **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
* **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks.
* **4K/1K (Throughput)**: Realistic mixed workload typical of production deployments (RAG context + medium response). Long enough input that prefill matters, long enough output that decode steady-state dominates. Used for the inline B200 / GB200 results above.
* **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.

**Interpreting Results:**

* Compare your results against baseline numbers for your hardware
* Higher throughput at same latency = better performance
* Lower TTFT = more responsive user experience
* Lower TPOT = faster generation speed

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* Test Result (NVIDIA B200, TP=2 NVFP4)

```text Output theme={null}
Accuracy: 0.946
Latency: 178.284 s
Output throughput: 769.204 token/s
```

* Test Result (NVIDIA GB200, TP=2 NVFP4)

```text Output theme={null}
Accuracy: 0.951
Latency: 175.190 s
Invalid: 0.000
```


# GLM-4.7-Flash
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-4.7-Flash



## 1. Model Introduction

[GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash) is a lightweight and high-speed model in the GLM-4.7 series developed by Zhipu AI, featuring state-of-the-art capabilities in reasoning, function calling, and efficient local deployment.

As a compact variant in the GLM-4.7 family, GLM-4.7-Flash is a **30B-A3B MoE** model designed to balance performance and efficiency:

* **Lightweight Architecture**: 30B total parameters with only 3B active parameters, enabling efficient inference
* **Enhanced Reasoning**: Inherits the reasoning capabilities from GLM-4.7 with optimized performance
* **Superior Coding**: Strong code generation and understanding capabilities
* **Advanced Tool Use**: Robust tool calling and agent capabilities for complex workflows
* **Optimized for Local Deployment**: Designed for single-GPU deployment scenarios

For more details, please refer to the [official GLM-4.7 documentation](https://docs.z.ai/guides/llm/glm-4.7).

**Key Features:**

* **Efficient MoE Architecture**: 30B-A3B sparse activation for optimal performance/efficiency trade-off
* **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
* **Hardware Optimization**: Specifically tuned for NVIDIA H100/H200/B200 GPUs
* **High Performance**: Optimized for both throughput and latency scenarios

**Available Models:**

* **BF16 (Full precision)**: [zai-org/GLM-4.7-Flash](https://huggingface.co/zai-org/GLM-4.7-Flash)

**License:**

Please refer to the [official GLM-4.7-Flash model card](https://huggingface.co/zai-org/GLM-4.7-Flash) for license details.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **EAGLE Speculative Decoding:** Supported for GLM-4.7-Flash. Add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The spec-v2 overlap scheduler is enabled by default; pass `--disable-overlap-schedule` to disable. Enable via the interactive command generator above.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

GLM-4.7-Flash supports Thinking mode by default. Enable the reasoning parser during deployment to separate the thinking and the content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.7-Flash \
  --reasoning-parser glm45 \
  --attention-backend triton \
  --tp 1 \
  --host 0.0.0.0 \
  --port 8000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="zai-org/GLM-4.7-Flash",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

> [!NOTE]
>
> **Parser names by model:** GLM-4.5 and GLM-4.6 use `--tool-call-parser glm45`. GLM-4.7 and GLM-4.7-Flash use `--tool-call-parser glm47`. All GLM models use `--reasoning-parser glm45` regardless of generation.

GLM-4.7-Flash supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.7-Flash \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --attention-backend triton \
  --tp 1 \
  --host 0.0.0.0 \
  --port 8000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/GLM-4.7-Flash",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls (tool call deltas may stream in multiple chunks)
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
if tool_calls_accumulator:
    print("\n=============== Tool Calls =================", flush=True)
    for index, tool_call in sorted(tool_calls_accumulator.items()):
        print(f"Tool Call: {tool_call['name']}")
        print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking for the weather in Beijing. I have the get_weather function available which can provide weather information for a location. The required parameter is "location" and the
 user has provided "Beijing". There's an optional parameter "unit" for temperature unit, but the user hasn't specified which unit they prefer, and since it's optional, I should not ask about it or make up a value for it. I'll call the function with just the location parameter.I'll check the current weather in Beijing for you.
=============== Tool Calls =================
Tool Call: get_weather
   Arguments: {"location": "Beijing"}

```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="zai-org/GLM-4.7-Flash",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 (1x)
* Model: GLM-4.7-Flash
* Tensor Parallelism: 1
* SGLang Version: 0.5.7

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Test Scenarios

Three core scenarios reflect real-world usage patterns:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Scenario</th>
      <th>Input Length</th>
      <th>Output Length</th>
      <th>Use Case</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Chat**</td>
      <td>1K</td>
      <td>1K</td>
      <td>Most common conversational AI workload</td>
    </tr>

    <tr>
      <td>**Reasoning**</td>
      <td>1K</td>
      <td>8K</td>
      <td>Long-form generation, complex reasoning tasks</td>
    </tr>

    <tr>
      <td>**Summarization**</td>
      <td>8K</td>
      <td>1K</td>
      <td>Document summarization, RAG retrieval</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 Concurrency Levels

Test each scenario at three concurrency levels to capture the throughput vs. latency tradeoff (Pareto frontier):

* **Low Concurrency**: `--max-concurrency 1` (Latency-optimized)
* **Medium Concurrency**: `--max-concurrency 16` (Balanced)
* **High Concurrency**: `--max-concurrency 100` (Throughput-optimized)

#### 5.1.3 Number of Prompts

For each concurrency level, configure `num_prompts` to simulate realistic user loads:

* **Quick Test**: `num_prompts = concurrency × 1` (minimal test)
* **Recommended**: `num_prompts = concurrency × 5` (standard benchmark)
* **Stable Measurements**: `num_prompts = concurrency × 10` (production-grade)

***

#### 5.1.4 Benchmark Commands

**Scenario 1: Chat (1K/1K) - Most Important**

* **Model Deployment**

```bash Command theme={null}
python -m sglang.launch_server \
  --model zai-org/GLM-4.7-Flash \
  --attention-backend triton \
  --tp 1
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  38.94
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.26
Input token throughput (tok/s):          156.67
Output token throughput (tok/s):         108.37
Peak output token throughput (tok/s):    125.00
Peak concurrent requests:                2
Total token throughput (tok/s):          265.03
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3891.12
Median E2E Latency (ms):                 3061.48
P90 E2E Latency (ms):                    7172.25
P99 E2E Latency (ms):                    9042.62
---------------Time to First Token----------------
Mean TTFT (ms):                          131.36
Median TTFT (ms):                        94.55
P99 TTFT (ms):                           435.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.75
Median TPOT (ms):                        8.82
P99 TPOT (ms):                           9.39
---------------Inter-Token Latency----------------
Mean ITL (ms):                           8.93
Median ITL (ms):                         8.98
P95 ITL (ms):                            9.83
P99 ITL (ms):                            10.20
Max ITL (ms):                            18.50
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  52.73
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40775
Request throughput (req/s):              1.52
Input token throughput (tok/s):          752.27
Output token throughput (tok/s):         773.83
Peak output token throughput (tok/s):    1040.00
Peak concurrent requests:                21
Total token throughput (tok/s):          1526.10
Concurrency:                             13.98
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   9217.90
Median E2E Latency (ms):                 9642.50
P90 E2E Latency (ms):                    15147.02
P99 E2E Latency (ms):                    18237.06
---------------Time to First Token----------------
Mean TTFT (ms):                          299.02
Median TTFT (ms):                        105.98
P99 TTFT (ms):                           1109.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          18.03
Median TPOT (ms):                        18.00
P99 TPOT (ms):                           26.51
---------------Inter-Token Latency----------------
Mean ITL (ms):                           17.52
Median ITL (ms):                         16.07
P95 ITL (ms):                            18.14
P99 ITL (ms):                            89.43
Max ITL (ms):                            763.13
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  91.48
Total input tokens:                      249831
Total input text tokens:                 249831
Total generated tokens:                  252662
Total generated tokens (retokenized):    250941
Request throughput (req/s):              5.47
Input token throughput (tok/s):          2730.87
Output token throughput (tok/s):         2761.82
Peak output token throughput (tok/s):    4199.00
Peak concurrent requests:                109
Total token throughput (tok/s):          5492.69
Concurrency:                             90.54
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16566.04
Median E2E Latency (ms):                 16134.36
P90 E2E Latency (ms):                    30167.60
P99 E2E Latency (ms):                    34034.04
---------------Time to First Token----------------
Mean TTFT (ms):                          433.94
Median TTFT (ms):                        123.26
P99 TTFT (ms):                           1760.09
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.26
Median TPOT (ms):                        33.56
P99 TPOT (ms):                           38.78
---------------Inter-Token Latency----------------
Mean ITL (ms):                           31.99
Median ITL (ms):                         24.06
P95 ITL (ms):                            79.62
P99 ITL (ms):                            103.03
Max ITL (ms):                            1369.20
==================================================
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  525.43
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  44462
Total generated tokens (retokenized):    44451
Request throughput (req/s):              0.02
Input token throughput (tok/s):          11.61
Output token throughput (tok/s):         84.62
Peak output token throughput (tok/s):    125.00
Peak concurrent requests:                2
Total token throughput (tok/s):          96.23
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   52540.19
Median E2E Latency (ms):                 53694.45
P90 E2E Latency (ms):                    94742.08
P99 E2E Latency (ms):                    101224.18
---------------Time to First Token----------------
Mean TTFT (ms):                          97.45
Median TTFT (ms):                        95.28
P99 TTFT (ms):                           105.64
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.94
Median TPOT (ms):                        11.25
P99 TPOT (ms):                           13.09
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.80
Median ITL (ms):                         11.51
P95 ITL (ms):                            15.83
P99 ITL (ms):                            16.86
Max ITL (ms):                            19.96
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  473.92
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  318306
Total generated tokens (retokenized):    317860
Request throughput (req/s):              0.17
Input token throughput (tok/s):          83.70
Output token throughput (tok/s):         671.65
Peak output token throughput (tok/s):    1040.00
Peak concurrent requests:                19
Total token throughput (tok/s):          755.35
Concurrency:                             13.80
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   81746.73
Median E2E Latency (ms):                 78508.54
P90 E2E Latency (ms):                    155292.49
P99 E2E Latency (ms):                    166769.99
---------------Time to First Token----------------
Mean TTFT (ms):                          117.50
Median TTFT (ms):                        101.97
P99 TTFT (ms):                           182.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          20.36
Median TPOT (ms):                        20.48
P99 TPOT (ms):                           22.63
---------------Inter-Token Latency----------------
Mean ITL (ms):                           20.52
Median ITL (ms):                         20.42
P95 ITL (ms):                            23.41
P99 ITL (ms):                            26.29
Max ITL (ms):                            90.48
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  714.72
Total input tokens:                      158939
Total input text tokens:                 158939
Total generated tokens:                  1301025
Total generated tokens (retokenized):    1289431
Request throughput (req/s):              0.45
Input token throughput (tok/s):          222.38
Output token throughput (tok/s):         1820.33
Peak output token throughput (tok/s):    3200.00
Peak concurrent requests:                68
Total token throughput (tok/s):          2042.71
Concurrency:                             55.68
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   124364.58
Median E2E Latency (ms):                 129250.98
P90 E2E Latency (ms):                    219175.80
P99 E2E Latency (ms):                    247741.77
---------------Time to First Token----------------
Mean TTFT (ms):                          149.40
Median TTFT (ms):                        114.78
P99 TTFT (ms):                           288.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          30.51
Median TPOT (ms):                        31.75
P99 TPOT (ms):                           33.32
---------------Inter-Token Latency----------------
Mean ITL (ms):                           30.56
Median ITL (ms):                         30.82
P95 ITL (ms):                            33.20
P99 ITL (ms):                            80.54
Max ITL (ms):                            117.72
==================================================
```

**Scenario 3: Summarization (8K/1K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  58.27
Total input tokens:                      41941
Total input text tokens:                 41941
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.17
Input token throughput (tok/s):          719.73
Output token throughput (tok/s):         72.42
Peak output token throughput (tok/s):    112.00
Peak concurrent requests:                2
Total token throughput (tok/s):          792.15
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5825.08
Median E2E Latency (ms):                 4624.26
P90 E2E Latency (ms):                    12690.22
P99 E2E Latency (ms):                    13177.96
---------------Time to First Token----------------
Mean TTFT (ms):                          296.01
Median TTFT (ms):                        195.59
P99 TTFT (ms):                           717.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          12.63
Median TPOT (ms):                        13.07
P99 TPOT (ms):                           16.68
---------------Inter-Token Latency----------------
Mean ITL (ms):                           13.13
Median ITL (ms):                         13.17
P95 ITL (ms):                            17.02
P99 ITL (ms):                            17.47
Max ITL (ms):                            19.84
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  89.59
Total input tokens:                      300020
Total input text tokens:                 300020
Total generated tokens:                  41669
Total generated tokens (retokenized):    41656
Request throughput (req/s):              0.89
Input token throughput (tok/s):          3348.77
Output token throughput (tok/s):         465.10
Peak output token throughput (tok/s):    752.00
Peak concurrent requests:                19
Total token throughput (tok/s):          3813.87
Concurrency:                             14.39
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16120.74
Median E2E Latency (ms):                 16246.55
P90 E2E Latency (ms):                    27279.72
P99 E2E Latency (ms):                    34577.93
---------------Time to First Token----------------
Mean TTFT (ms):                          1943.94
Median TTFT (ms):                        382.19
P99 TTFT (ms):                           8980.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          27.87
Median TPOT (ms):                        28.26
P99 TPOT (ms):                           40.55
---------------Inter-Token Latency----------------
Mean ITL (ms):                           27.27
Median ITL (ms):                         21.74
P95 ITL (ms):                            23.32
P99 ITL (ms):                            232.65
Max ITL (ms):                            4282.01
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-4.7-Flash \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  167.01
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total generated tokens:                  170000
Total generated tokens (retokenized):    169226
Request throughput (req/s):              1.92
Input token throughput (tok/s):          7627.82
Output token throughput (tok/s):         1017.93
Peak output token throughput (tok/s):    1984.00
Peak concurrent requests:                69
Total token throughput (tok/s):          8645.75
Concurrency:                             59.68
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   31147.52
Median E2E Latency (ms):                 30603.34
P90 E2E Latency (ms):                    54889.44
P99 E2E Latency (ms):                    67665.30
---------------Time to First Token----------------
Mean TTFT (ms):                          428.87
Median TTFT (ms):                        441.69
P99 TTFT (ms):                           1232.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          58.06
Median TPOT (ms):                        62.79
P99 TPOT (ms):                           82.23
---------------Inter-Token Latency----------------
Mean ITL (ms):                           57.93
Median ITL (ms):                         33.30
P95 ITL (ms):                            247.98
P99 ITL (ms):                            409.63
Max ITL (ms):                            1421.21
==================================================
```

#### 5.1.5 Understanding the Results

**Key Metrics:**

* **Request Throughput (req/s)**: Number of requests processed per second
* **Output Token Throughput (tok/s)**: Total tokens generated per second
* **Mean TTFT (ms)**: Time to First Token - measures responsiveness
* **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
* **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency

**Why These Configurations Matter:**

* **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
* **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
* **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks.
* **Variable Concurrency**: Captures the Pareto frontier - the optimal tradeoff between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.

**Interpreting Results:**

* Compare your results against baseline numbers for your hardware
* Higher throughput at same latency = better performance
* Lower TTFT = more responsive user experience
* Lower TPOT = faster generation speed

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* Result

```text Output theme={null}
Accuracy: 0.845
Invalid: 0.000
Latency: 8.431 s
Output throughput: 2195.387 token/s
```


# GLM-5
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5



## 1. Model Introduction

[GLM-5](https://huggingface.co/zai-org/GLM-5) is the most powerful language model in the GLM series developed by Zhipu AI, targeting complex systems engineering and long-horizon agentic tasks. Scaling from GLM-4.5's 355B parameters (32B active) to 744B parameters (40B active), GLM-5 integrates DeepSeek Sparse Attention (DSA) to largely reduce deployment cost while preserving long-context capacity.

With advances in both pre-training (28.5T tokens) and post-training via [slime](https://github.com/THUDM/slime) (a novel asynchronous RL infrastructure), GLM-5 delivers significant improvements over GLM-4.7 and achieves best-in-class performance among open-source models on reasoning, coding, and agentic tasks.

**Key Features:**

* **Systems Engineering & Agentic Tasks**: Purpose-built for complex systems engineering and long-horizon agentic tasks
* **State-of-the-Art Performance**: Best-in-class among open-source models on reasoning (HLE, AIME, GPQA), coding (SWE-bench, Terminal-Bench), and agentic tasks (BrowseComp, Vending Bench 2)
* **DeepSeek Sparse Attention (DSA)**: Reduces deployment cost while preserving long-context capacity
* **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
* **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency

**Available Models:**

* **BF16 (Full precision)**: [zai-org/GLM-5](https://huggingface.co/zai-org/GLM-5)
* **FP8 (8-bit quantized)**: [zai-org/GLM-5-FP8](https://huggingface.co/zai-org/GLM-5-FP8)

**License:** MIT

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5 on NVIDIA H100, H200, B200, and AMD MI300X/MI325X/MI355X GPUs.

Run this Command:

<pre />

> [!WARNING]
>
> All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.

### 3.2 Configuration Tips

* Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
* **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
* The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
* BF16 model always requires **2x GPUs** compared to FP8 on NVIDIA hardware.

<table>
  <thead>
    <tr>
      <th>Hardware</th>
      <th>FP8</th>
      <th>BF16</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>H100</td>
      <td>tp=16</td>
      <td>tp=32</td>
    </tr>

    <tr>
      <td>H200</td>
      <td>tp=8</td>
      <td>tp=16</td>
    </tr>

    <tr>
      <td>B200</td>
      <td>tp=8</td>
      <td>tp=16</td>
    </tr>

    <tr>
      <td>MI300X/MI325X</td>
      <td>—</td>
      <td>tp=8</td>
    </tr>

    <tr>
      <td>MI355X</td>
      <td>—</td>
      <td>tp=8</td>
    </tr>
  </tbody>
</table>

* **B200 (FP8)**: Use `--ep 1 --attention-backend dsa --dsa-decode-backend trtllm --dsa-prefill-backend trtllm --moe-runner-backend flashinfer_trtllm --enable-flashinfer-allreduce-fusion` for optimized DSA and MoE backends on Blackwell. Also add `--quantization fp8` for FP8 weight quantization.

* **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.

* For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.

* **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5. Use `--enable-prefill-cp --cp-strategy interleave` with `--attn-dp-size 1`.

* Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.

## 4. Model Invocation

Deploy GLM-5 with the following command (FP8 on H200, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path zai-org/GLM-5-FP8 \
  --tp 8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-flashinfer-allreduce-fusion \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 \
  --port 30000
```

### 4.1 MI300X/MI325X/MI355X (ROCm) Server Command

The following ROCm command is an additional option for AMD GPUs and does not replace the NVIDIA instructions above.

```shell Command theme={null}
sglang serve \
  --model-path zai-org/GLM-5 \
  --tp 8 \
  --trust-remote-code \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --chunked-prefill-size 131072 \
  --mem-fraction-static 0.80 \
  --watchdog-timeout 1200 \
  --host 0.0.0.0 \
  --port 30000
```

### 4.2 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.3 Advanced Usage

#### 4.3.1 Reasoning Parser

GLM-5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.

To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:

* **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
* **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.

**Example 1: Thinking Mode (Default)**

Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Thinking mode is enabled by default, no extra parameters needed
response = client.chat.completions.create(
    model="zai-org/GLM-5-FP8",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user wants me to solve a math problem: "What is 15% of 240?".

Step 1: Understand the problem. I need to calculate a percentage of a number.
Formula: Percentage × Number = Result.

Step 2: Convert the percentage to a decimal or fraction.
15% = 15/100 or 0.15.

Step 3: Perform the multiplication.
Method A: Decimal multiplication.
0.15 × 240.
Break it down:
10% of 240 = 24.
5% is half of 10%, so 12.
15% = 10% + 5% = 24 + 12 = 36.

Method B: Fraction multiplication.
15/100 × 240.
Simplify 240/100 = 2.4.
15 × 2.4.
10 × 2.4 = 24.
5 × 2.4 = 12.
24 + 12 = 36.

Method C: Direct multiplication.
240 × 0.15.
240 × 0.10 = 24.
240 × 0.05 = 12.
24 + 12 = 36.

Step 4: Final Verification.
Is 36 reasonable?
10% is 24. 20% is 48.
15% is halfway between 10% and 20%.
Halfway between 24 and 48 is 36.
The result is correct.

Step 5: Structure the final response. I will present the calculation clearly, perhaps showing the fractional or decimal method, or the mental math shortcut (10% + 5%).
=============== Content =================
Here is the step-by-step solution:

**Step 1: Convert the percentage to a decimal.**
To convert 15% to a decimal, divide by 100.
$$15\% = \frac{15}{100} = 0.15$$

**Step 2: Multiply the decimal by the number.**
Now, multiply 0.15 by 240.
$$0.15 \times 240$$

**Step 3: Perform the calculation.**
You can break this down to make it easier:
$$0.15 = 0.10 + 0.05$$

*   First, find 10% of 240:
    $$0.10 \times 240 = 24$$
*   Next, find 5% (which is half of 10%):
    $$\frac{24}{2} = 12$$
*   Add the two results together:
    $$24 + 12 = 36$$

**Answer:**
15% of 240 is **36**.
```

**Example 2: Instruct Mode (Thinking Off)**

To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Disable thinking mode via chat_template_kwargs
response = client.chat.completions.create(
    model="zai-org/GLM-5-FP8",
    messages=[
        {"role": "user", "content": "What is 15% of 240?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    max_tokens=2048,
    stream=True
)

# In Instruct mode, the model responds directly without reasoning_content
for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
To find **15% of 240**, follow these steps:

### Step 1: Convert the Percentage to a Decimal
First, convert the percentage to a decimal by dividing by 100.

\[
15\% = \frac{15}{100} = 0.15
\]

### Step 2: Multiply by the Number
Next, multiply the decimal by the number you want to find the percentage of.

\[
0.15 \times 240
\]

### Step 3: Perform the Multiplication
Calculate the multiplication:

\[
0.15 \times 240 = 36
\]

### Final Answer
\[
\boxed{36}
\]
```

#### 4.3.2 Tool Calling

GLM-5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/GLM-5-FP8",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking for the weather in Beijing. I have access to a get_weather function that can provide current weather information. Let me check what parameters are required:

- location: required, should be "Beijing"
- unit: optional (not in required array), can be "celsius" or "fahrenheit"

Since the user didn't specify a unit preference and it's optional, I should not ask about it or make up a value. I'll just call the function with the required location parameter.I'll get the current weather in Beijing for you.
=============== Content =================
Tool Call: get_weather
   Arguments:
Tool Call: None
   Arguments: {
Tool Call: None
   Arguments: "location": "Be
Tool Call: None
   Arguments: ijing"
Tool Call: None
   Arguments: }
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: H200 (8x)
* Model: GLM-5-FP8
* Tensor Parallelism: 8
* SGLang Version: commit 947927bdb

#### 5.1.1 Latency Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-5-FP8 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  35.78
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4213
Request throughput (req/s):              0.28
Input token throughput (tok/s):          170.54
Output token throughput (tok/s):         117.96
Peak output token throughput (tok/s):    148.00
Peak concurrent requests:                2
Total token throughput (tok/s):          288.50
Concurrency:                             1.00
Accept length:                           3.48
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3576.31
Median E2E Latency (ms):                 2935.97
P90 E2E Latency (ms):                    5908.97
P99 E2E Latency (ms):                    8588.08
---------------Time to First Token----------------
Mean TTFT (ms):                          290.88
Median TTFT (ms):                        282.34
P99 TTFT (ms):                           332.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.54
Median TPOT (ms):                        6.97
P99 TPOT (ms):                           9.04
---------------Inter-Token Latency----------------
Mean ITL (ms):                           7.80
Median ITL (ms):                         6.81
P95 ITL (ms):                            13.51
P99 ITL (ms):                            26.99
Max ITL (ms):                            29.50
==================================================
```

#### 5.1.2 Throughput Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-5-FP8 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  411.74
Total input tokens:                      502493
Total input text tokens:                 502493
Total generated tokens:                  500251
Total generated tokens (retokenized):    499614
Request throughput (req/s):              2.43
Input token throughput (tok/s):          1220.41
Output token throughput (tok/s):         1214.97
Peak output token throughput (tok/s):    2648.00
Peak concurrent requests:                105
Total token throughput (tok/s):          2435.38
Concurrency:                             96.30
Accept length:                           3.50
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   39648.76
Median E2E Latency (ms):                 39058.12
P90 E2E Latency (ms):                    57009.82
P99 E2E Latency (ms):                    68880.33
---------------Time to First Token----------------
Mean TTFT (ms):                          20613.80
Median TTFT (ms):                        21429.21
P99 TTFT (ms):                           29543.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          38.73
Median TPOT (ms):                        36.52
P99 TPOT (ms):                           67.09
---------------Inter-Token Latency----------------
Mean ITL (ms):                           38.13
Median ITL (ms):                         16.57
P95 ITL (ms):                            86.01
P99 ITL (ms):                            164.88
Max ITL (ms):                            1307.02
==================================================
```

### 5.2 Accuracy Benchmark

> [!NOTE]
>
> The accuracy benchmark results below are shared with GLM-5.1, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate GLM-5.1 benchmark run is planned.

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Result

```text Output theme={null}
Accuracy: 0.955
Invalid: 0.000
Latency: 32.470 s
Output throughput: 642.044 token/s
```

#### 5.2.2 MMLU Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Result

```text Output theme={null}
subject: abstract_algebra, #q:100, acc: 0.860
subject: anatomy, #q:135, acc: 0.874
subject: astronomy, #q:152, acc: 0.941
subject: business_ethics, #q:100, acc: 0.880
subject: clinical_knowledge, #q:265, acc: 0.932
subject: college_biology, #q:144, acc: 0.972
subject: college_chemistry, #q:100, acc: 0.640
subject: college_computer_science, #q:100, acc: 0.900
subject: college_mathematics, #q:100, acc: 0.810
subject: college_medicine, #q:173, acc: 0.873
subject: college_physics, #q:102, acc: 0.912
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.928
subject: econometrics, #q:114, acc: 0.807
subject: electrical_engineering, #q:145, acc: 0.897
subject: elementary_mathematics, #q:378, acc: 0.937
subject: formal_logic, #q:126, acc: 0.778
subject: global_facts, #q:100, acc: 0.710
subject: high_school_biology, #q:310, acc: 0.961
subject: high_school_chemistry, #q:203, acc: 0.847
subject: high_school_computer_science, #q:100, acc: 0.960
subject: high_school_european_history, #q:165, acc: 0.891
subject: high_school_geography, #q:198, acc: 0.960
subject: high_school_government_and_politics, #q:193, acc: 0.984
subject: high_school_macroeconomics, #q:390, acc: 0.923
subject: high_school_mathematics, #q:270, acc: 0.696
subject: high_school_microeconomics, #q:238, acc: 0.962
subject: high_school_physics, #q:151, acc: 0.821
subject: high_school_psychology, #q:545, acc: 0.956
subject: high_school_statistics, #q:216, acc: 0.889
subject: high_school_us_history, #q:204, acc: 0.941
subject: high_school_world_history, #q:237, acc: 0.945
subject: human_aging, #q:223, acc: 0.857
subject: human_sexuality, #q:131, acc: 0.908
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.907
subject: logical_fallacies, #q:163, acc: 0.933
subject: machine_learning, #q:112, acc: 0.830
subject: management, #q:103, acc: 0.942
subject: marketing, #q:234, acc: 0.940
subject: medical_genetics, #q:100, acc: 0.990
subject: miscellaneous, #q:783, acc: 0.959
subject: moral_disputes, #q:346, acc: 0.873
subject: moral_scenarios, #q:895, acc: 0.837
subject: nutrition, #q:306, acc: 0.922
subject: philosophy, #q:311, acc: 0.897
subject: prehistory, #q:324, acc: 0.929
subject: professional_accounting, #q:282, acc: 0.844
subject: professional_law, #q:1534, acc: 0.714
subject: professional_medicine, #q:272, acc: 0.941
subject: professional_psychology, #q:612, acc: 0.913
subject: public_relations, #q:110, acc: 0.791
subject: security_studies, #q:245, acc: 0.878
subject: sociology, #q:201, acc: 0.940
subject: us_foreign_policy, #q:100, acc: 0.920
subject: virology, #q:166, acc: 0.596
subject: world_religions, #q:171, acc: 0.936
Total latency: 165.275
Average accuracy: 0.877
```

### 5.3 AMD GPU Benchmarks

#### 5.3.1 GSM8K Benchmark (MI325/MI35x)

* MI325/MI35x Test (GLM-5 BF16, `tp=8`, TileLang DSA backends)

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

```text Output theme={null}
Accuracy: 0.970
Invalid: 0.000
```

Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).


# GLM-5.1
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.1



## 1. Model Introduction

**Available Models:**

* **BF16 (Full precision)**: [zai-org/GLM-5.1](https://huggingface.co/zai-org/GLM-5.1)
* **FP8 (8-bit quantized)**: [zai-org/GLM-5.1-FP8](https://huggingface.co/zai-org/GLM-5.1-FP8)
* **NVFP4 (4-bit quantized)**: [nvidia/GLM-5.1-NVFP4](https://huggingface.co/nvidia/GLM-5.1-NVFP4)

**License:** MIT

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities. SGLang supports serving GLM-5.1 on NVIDIA H100, H200, B300, GB300, and AMD MI300X/MI325X/MI355X GPUs.

Run this Command:

<pre />

> [!WARNING]
>
> All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.1.

### 3.2 Configuration Tips

* Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
* **DP Attention**: Enables data parallel attention for higher throughput under high concurrency. Note that DP attention trades off low-concurrency latency for high-concurrency throughput — disable it if your workload is latency-sensitive with few concurrent requests.
* The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.

<table>
  <thead>
    <tr>
      <th>Hardware</th>
      <th>NVFP4</th>
      <th>FP8</th>
      <th>BF16</th>
      <th>MXFP4</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>H100</td>
      <td>—</td>
      <td>tp=16</td>
      <td>—</td>
      <td>—</td>
    </tr>

    <tr>
      <td>H200</td>
      <td>—</td>
      <td>tp=8</td>
      <td>—</td>
      <td>—</td>
    </tr>

    <tr>
      <td>B300</td>
      <td>tp=8</td>
      <td>—</td>
      <td>—</td>
      <td>—</td>
    </tr>

    <tr>
      <td>GB300</td>
      <td>tp=4</td>
      <td>—</td>
      <td>—</td>
      <td>—</td>
    </tr>

    <tr>
      <td>MI300X/MI325X</td>
      <td>—</td>
      <td>tp=8</td>
      <td>tp=8</td>
      <td>—</td>
    </tr>

    <tr>
      <td>MI355X</td>
      <td>—</td>
      <td>tp=8</td>
      <td>tp=8</td>
      <td>tp=4</td>
    </tr>
  </tbody>
</table>

* **H100 and H200**: FP8 is the recommended deployment path.

* **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300.

* **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (\~89 GB/GPU vs \~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang.

* For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.

* **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.1. Use `--enable-prefill-cp --cp-strategy interleave`. Attention DP is supported, with `tp_size / attn_dp_size` CP ranks in each DP group. Dense FFNs retain their configured TP width.

* Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature.

## 4. Model Invocation

Deploy GLM-5.1 with the following command (FP8 on H200, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path zai-org/GLM-5.1-FP8 \
  --tp 8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.85 \
  --host 0.0.0.0 \
  --port 30000
```

### 4.1 B300/GB300 (NVFP4) Server Command

#### B300

```shell Command theme={null}
sglang serve \
  --model-path nvidia/GLM-5.1-NVFP4 \
  --tp 8 \
  --quantization modelopt_fp4 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --trust-remote-code \
  --mem-fraction-static 0.80 \
  --host 0.0.0.0 \
  --port 30000
```

#### GB300

```shell Command theme={null}
sglang serve \
  --model-path nvidia/GLM-5.1-NVFP4 \
  --tp 4 \
  --quantization modelopt_fp4 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --trust-remote-code \
  --mem-fraction-static 0.80 \
  --host 0.0.0.0 \
  --port 30000
```

### 4.2 MI300X/MI325X/MI355X (ROCm) Server Command

The following ROCm commands are additional options for AMD GPUs and do not replace the NVIDIA instructions above.

#### MXFP4 (MI355X / gfx950)

On MI355X (gfx950), set `SGLANG_DSA_TRITON_PREFILL=1` to enable a faster Triton attention kernel for the prefill phase (opt-in, off by default). Keep `--dsa-prefill-backend tilelang` as shown. The EAGLE speculative-decoding flags below are optional but recommended on gfx950.

```shell Command theme={null}
# SGLANG_DSA_TRITON_PREFILL=1 is optional; it enables a faster Triton prefill kernel on gfx950
SGLANG_DSA_TRITON_PREFILL=1 sglang serve \
  --model-path amd/GLM-5.1-MXFP4 \
  --tp 4 \
  --trust-remote-code \
  --kv-cache-dtype fp8_e4m3 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --chunked-prefill-size 131072 \
  --mem-fraction-static 0.85 \
  --watchdog-timeout 1200 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 30000
```

#### FP8 (Recommended)

```shell Command theme={null}
sglang serve \
  --model-path zai-org/GLM-5.1-FP8 \
  --tp 8 \
  --trust-remote-code \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --chunked-prefill-size 131072 \
  --mem-fraction-static 0.80 \
  --watchdog-timeout 1200 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 30000
```

#### BF16

```shell Command theme={null}
sglang serve \
  --model-path zai-org/GLM-5.1 \
  --tp 8 \
  --trust-remote-code \
  --dsa-prefill-backend tilelang \
  --dsa-decode-backend tilelang \
  --chunked-prefill-size 131072 \
  --mem-fraction-static 0.80 \
  --watchdog-timeout 1200 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 30000
```

### 4.3 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.4 Advanced Usage

#### 4.4.1 Reasoning Parser

GLM-5.1 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.

To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:

* **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
* **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.

**Example 1: Thinking Mode (Default)**

Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Thinking mode is enabled by default, no extra parameters needed
response = client.chat.completions.create(
    model="zai-org/GLM-5.1-FP8",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
1.  **Understand the Goal:** The user wants to find 15% of 240, and they want the solution explained step-by-step.

2.  **Identify the Core Mathematical Concept:** "Percent" means "per hundred" or "out of 100". Finding "X% of Y" translates to the mathematical operation: $(X / 100) \times Y$.

3.  **Step-by-Step Breakdown:**
    *   *Step 1: Convert the percentage to a decimal (or fraction).* 15% means 15 out of 100, which is $15/100$ or $0.15$.
    *   *Step 2: Multiply the decimal by the given number.* Multiply $0.15$ by $240$.
    *   *Step 3: Perform the calculation.*
        *   $0.15 \times 240$
        *   I can break this down further to make it easy to follow:
            *   $0.10 \times 240 = 24$ (which is 10%)
            *   $0.05 \times 240 = 12$ (which is 5%, half of 10%)
            *   $24 + 12 = 36$
        *   Alternatively, standard multiplication:
            *   $240 \times 15 = 3600$
            *   Move decimal two places left -> $36$
    *   *Step 4: State the final answer clearly.*

4.  **Draft the Response (incorporating the steps clearly):**
    *   *Introduction:* State the problem clearly.
    *   *Step 1:* Explain how to convert 15% to a decimal.
    *   *Step 2:* Explain the multiplication step.
    *   *Step 3:* Show the actual math (I'll provide the standard multiplication and the "mental math" trick as it adds value).
    *   *Conclusion:* Give the final answer.

5.  **Refine the Output (Self-Correction/Polishing during drafting):**
    *   *Drafting Step 1:* To find 15% of 240, first convert 15% into a decimal. Since percent means "per hundred," you divide 15 by 100. 15 ÷ 100 = 0.15.
    *   *Drafting Step 2:* Next, multiply this decimal by the number you are finding the percentage of (which is 240). So, calculate 0.15 × 240.
    *   *Drafting Step 3 (Standard way):* 0.15 × 240 = 36.
    *   *Adding the alternative mental math way:* It's often helpful to break it down into 10% and 5%.
        *   10% of 240 = 24 (move the decimal point one place to the left)
        *   5% is half of 10%, so half of 24 = 12
        *   Add them together: 24 + 12 = 36.
    *   *Final Answer:* 15% of 240 is 36.

6.  **Final Review against User Prompt:** Does it solve the problem? Yes. Is it step-by-step? Yes. Is it clear? Yes. (Proceed to generate output).
=============== Content =================
Here is the step-by-step solution to find 15% of 240:

**Step 1: Convert the percentage to a decimal.**
To convert a percentage to a decimal, divide it by 100 (or simply move the decimal point two places to the left).
* 15% = 15 ÷ 100 = **0.15**

**Step 2: Multiply the decimal by the number.**
Now, multiply the decimal (0.15) by the number you are finding the percentage of (240).
* 0.15 × 240 = **36**

*(Alternative mental math method for Step 2)*:
If you don't want to multiply by 0.15 directly, you can break 15% down into 10% and 5%:
* **10% of 240** = 24 (just move the decimal point one place to the left)
* **5% of 240** = 12 (5% is half of 10%, so just divide 24 by 2)
* **Add them together**: 24 + 12 = **36**

**Answer:**
15% of 240 is **36**.
```

**Example 2: Instruct Mode (Thinking Off)**

To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Disable thinking mode via chat_template_kwargs
response = client.chat.completions.create(
    model="zai-org/GLM-5.1-FP8",
    messages=[
        {"role": "user", "content": "What is 15% of 240?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    max_tokens=2048,
    stream=True
)

# In Instruct mode, the model responds directly without reasoning_content
for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
15% of 240 is 36.

Here is how to calculate it:
1. Convert the percentage to a decimal: 15% = 0.15
2. Multiply the decimal by the number: 0.15 × 240 = 36
```

#### 4.4.2 Tool Calling

GLM-5.1 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/GLM-5.1-FP8",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user wants to know the weather in Beijing. I'll call the get_weather function with "Beijing" as the location.
=============== Content =================
Tool Call: get_weather
   Arguments:
Tool Call: None
   Arguments: {
Tool Call: None
   Arguments: "location": "Be
Tool Call: None
   Arguments: ijing"
Tool Call: None
   Arguments: }
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: H200 (8x)
* Model: GLM-5.1-FP8
* Tensor Parallelism: 8
* SGLang Version: commit 947927bdb

#### 5.1.1 Latency Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-5.1-FP8 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  35.78
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4213
Request throughput (req/s):              0.28
Input token throughput (tok/s):          170.54
Output token throughput (tok/s):         117.96
Peak output token throughput (tok/s):    148.00
Peak concurrent requests:                2
Total token throughput (tok/s):          288.50
Concurrency:                             1.00
Accept length:                           3.48
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3576.31
Median E2E Latency (ms):                 2935.97
P90 E2E Latency (ms):                    5908.97
P99 E2E Latency (ms):                    8588.08
---------------Time to First Token----------------
Mean TTFT (ms):                          290.88
Median TTFT (ms):                        282.34
P99 TTFT (ms):                           332.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.54
Median TPOT (ms):                        6.97
P99 TPOT (ms):                           9.04
---------------Inter-Token Latency----------------
Mean ITL (ms):                           7.80
Median ITL (ms):                         6.81
P95 ITL (ms):                            13.51
P99 ITL (ms):                            26.99
Max ITL (ms):                            29.50
==================================================
```

#### 5.1.2 Throughput Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/GLM-5.1-FP8 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  411.74
Total input tokens:                      502493
Total input text tokens:                 502493
Total generated tokens:                  500251
Total generated tokens (retokenized):    499614
Request throughput (req/s):              2.43
Input token throughput (tok/s):          1220.41
Output token throughput (tok/s):         1214.97
Peak output token throughput (tok/s):    2648.00
Peak concurrent requests:                105
Total token throughput (tok/s):          2435.38
Concurrency:                             96.30
Accept length:                           3.50
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   39648.76
Median E2E Latency (ms):                 39058.12
P90 E2E Latency (ms):                    57009.82
P99 E2E Latency (ms):                    68880.33
---------------Time to First Token----------------
Mean TTFT (ms):                          20613.80
Median TTFT (ms):                        21429.21
P99 TTFT (ms):                           29543.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          38.73
Median TPOT (ms):                        36.52
P99 TPOT (ms):                           67.09
---------------Inter-Token Latency----------------
Mean ITL (ms):                           38.13
Median ITL (ms):                         16.57
P95 ITL (ms):                            86.01
P99 ITL (ms):                            164.88
Max ITL (ms):                            1307.02
==================================================
```

### 5.2 Accuracy Benchmark

> [!NOTE]
>
> The accuracy benchmark results below are shared with GLM-5, as GLM-5.1 was not independently benchmarked at the time of this writing. A separate benchmark run is planned.

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Result

```text Output theme={null}
Accuracy: 0.955
Invalid: 0.000
Latency: 32.470 s
Output throughput: 642.044 token/s
```

#### 5.2.2 MMLU Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Result

```text Output theme={null}
subject: abstract_algebra, #q:100, acc: 0.860
subject: anatomy, #q:135, acc: 0.874
subject: astronomy, #q:152, acc: 0.941
subject: business_ethics, #q:100, acc: 0.880
subject: clinical_knowledge, #q:265, acc: 0.932
subject: college_biology, #q:144, acc: 0.972
subject: college_chemistry, #q:100, acc: 0.640
subject: college_computer_science, #q:100, acc: 0.900
subject: college_mathematics, #q:100, acc: 0.810
subject: college_medicine, #q:173, acc: 0.873
subject: college_physics, #q:102, acc: 0.912
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.928
subject: econometrics, #q:114, acc: 0.807
subject: electrical_engineering, #q:145, acc: 0.897
subject: elementary_mathematics, #q:378, acc: 0.937
subject: formal_logic, #q:126, acc: 0.778
subject: global_facts, #q:100, acc: 0.710
subject: high_school_biology, #q:310, acc: 0.961
subject: high_school_chemistry, #q:203, acc: 0.847
subject: high_school_computer_science, #q:100, acc: 0.960
subject: high_school_european_history, #q:165, acc: 0.891
subject: high_school_geography, #q:198, acc: 0.960
subject: high_school_government_and_politics, #q:193, acc: 0.984
subject: high_school_macroeconomics, #q:390, acc: 0.923
subject: high_school_mathematics, #q:270, acc: 0.696
subject: high_school_microeconomics, #q:238, acc: 0.962
subject: high_school_physics, #q:151, acc: 0.821
subject: high_school_psychology, #q:545, acc: 0.956
subject: high_school_statistics, #q:216, acc: 0.889
subject: high_school_us_history, #q:204, acc: 0.941
subject: high_school_world_history, #q:237, acc: 0.945
subject: human_aging, #q:223, acc: 0.857
subject: human_sexuality, #q:131, acc: 0.908
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.907
subject: logical_fallacies, #q:163, acc: 0.933
subject: machine_learning, #q:112, acc: 0.830
subject: management, #q:103, acc: 0.942
subject: marketing, #q:234, acc: 0.940
subject: medical_genetics, #q:100, acc: 0.990
subject: miscellaneous, #q:783, acc: 0.959
subject: moral_disputes, #q:346, acc: 0.873
subject: moral_scenarios, #q:895, acc: 0.837
subject: nutrition, #q:306, acc: 0.922
subject: philosophy, #q:311, acc: 0.897
subject: prehistory, #q:324, acc: 0.929
subject: professional_accounting, #q:282, acc: 0.844
subject: professional_law, #q:1534, acc: 0.714
subject: professional_medicine, #q:272, acc: 0.941
subject: professional_psychology, #q:612, acc: 0.913
subject: public_relations, #q:110, acc: 0.791
subject: security_studies, #q:245, acc: 0.878
subject: sociology, #q:201, acc: 0.940
subject: us_foreign_policy, #q:100, acc: 0.920
subject: virology, #q:166, acc: 0.596
subject: world_religions, #q:171, acc: 0.936
Total latency: 165.275
Average accuracy: 0.877
```

### 5.3 AMD GPU Benchmarks

#### 5.3.1 GSM8K Benchmark (MI325/MI35x)

* MI325/MI35x Test (GLM-5.1 BF16, `tp=8`, TileLang DSA backends)

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

```text Output theme={null}
Accuracy: 0.970
Invalid: 0.000
```

Results from [AMD nightly CI](https://github.com/sgl-project/sglang/actions/runs/22556197510/attempts/2#summary-65346783629). See also [sglang#18911](https://github.com/sgl-project/sglang/pull/18911).


# GLM-5.2
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.2

Deploy GLM-5.2 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X, plus Ascend NPU recipes.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

> **Install and prepare Ascend NPUs**
>
> Use a Linux host with the Ascend driver and a compatible CANN / TorchNPU environment. Follow the
> [Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) and the
> [GLM-5.2 deployment tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2#installation) to start a container with the NPU devices, driver mounts, and checkpoint directory. Run the setup and the generated **Python** command inside that container.
>
> **A3**
>
> ```bash Command theme={null}
> docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16
> ```
>
> Run the following setup before copying the generated command (CANN 9.0.0). CPU tuning requires host privileges.
>
> ```bash Command theme={null}
> unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
> echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> sysctl -w vm.swappiness=0
> sysctl -w kernel.numa_balancing=0
> sysctl -w kernel.sched_migration_cost_ns=50000
> source /usr/local/Ascend/ascend-toolkit/set_env.sh
> source /usr/local/Ascend/nnal/atb/set_env.sh
> ```
>
> **950PR/DT series**
>
> Use the compatible 950PR/DT series SGLang checkout, CANN environment, and custom operators used for your deployment. Set `SGLANG_ROOT` to that checkout; these recipes do not prescribe an image tag.
>
> ```bash Command theme={null}
> unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
> unset HCCL_IF_IP HCCL_SOCKET_FAMILY RANK_TABLE_FILE
> SGLANG_ROOT=/path/to/sglang
> export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
> ```
>
> Replace the model path in the generated command with your mounted ModelSlim checkpoint directory. Include its configuration, tokenizer, quantization metadata, and MTP weights.
>
> For A3, use the locally prepared **GLM-5.2-w8a8** checkpoint described in the [model-weight prerequisites](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2#model-weights).
> For 950PR/DT series, use your locally prepared GLM-5.2 **W8A8C8\_MXFP8** checkpoint, including MTP weights, and replace the model path in the generated command with its mounted path.
>
> Replace network placeholders such as `<network-interface>`, `<prefill-head-ip>`, and `<decode-head-ip>` in the generated NPU command with your cluster settings.

Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:

* **Low-Latency** — fastest reply for a single user. Pick for chat.
* **Balanced** — good speed with several users at once. Use for typical multi-user serving.
* **High-Throughput** — most tokens per second across many users. Best for batch jobs.

> [!WARNING]
>
> All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.2.

> [!NOTE]
>
> Speed numbers are measured with `--random-range-ratio 1.0`, `--flush-cache`, on `main @ 09ca4fc` (H200 FP8 cells: `v0.5.14 @ 49e384ce`). Spec cells pin the EAGLE acceptance length via the serve env `SGLANG_SIMULATE_ACC_LEN` (low-latency 5-1-6 = 3.5; FP8 balanced 1-1-2 = 2; NVFP4 balanced 2-1-3 = 2); high-throughput has no spec.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

* **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
* **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
* **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

## 1. Model Introduction

**GLM-5.2** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It ships in two precisions — **FP8** (`zai-org/GLM-5.2-FP8`) and full **BF16** (`zai-org/GLM-5.2`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (\~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, NVIDIA also publishes an **NVFP4** build (`nvidia/GLM-5.2-NVFP4`) that quantizes only the MoE experts' linear weights and activations to 4-bit (the shared expert stays unquantized), holding accuracy within \~1 point of the FP8 baseline on GPQA Diamond, SciCode, and IFBench. For AMD MI355X (gfx950), AMD publishes an **MXFP4** build (`amd/GLM-5.2-MXFP4`, Quark-quantized) — see the AMD GPUs configuration tip below; this recipe is inferred from the validated `amd/GLM-5.1-MXFP4` MI355X recipe and not yet benchmarked on GLM-5.2 (`verified: false`).

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Architecture</th>
      <th>Context</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8)**</td>
      <td>MoE · DSA · 256 experts (top-8) · MTP · FP8</td>
      <td>1,048,576</td>
    </tr>

    <tr>
      <td>**[GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)**</td>
      <td>MoE · DSA · 256 experts (top-8) · MTP · BF16</td>
      <td>1,048,576</td>
    </tr>

    <tr>
      <td>**[GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4)**</td>
      <td>MoE · DSA · 256 experts (top-8) · MTP · NVFP4</td>
      <td>1,048,576</td>
    </tr>

    <tr>
      <td>**[GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4)**</td>
      <td>MoE · DSA · 256 experts (top-8) · MTP · MXFP4</td>
      <td>1,048,576</td>
    </tr>
  </tbody>
</table>

**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code).

**Resources:** [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) · [GLM-5.2 (BF16)](https://huggingface.co/zai-org/GLM-5.2) · [GLM-5.2-NVFP4](https://huggingface.co/nvidia/GLM-5.2-NVFP4) · [GLM-5.2-MXFP4](https://huggingface.co/amd/GLM-5.2-MXFP4).

## 2. Configuration Tips

* **DeepSeek Sparse Attention (DSA).** GLM-5.2 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.2's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`.

* **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). **Tune the draft length to the accept length.** GLM-5.2's MTP head is strong — accept length runs high (4+ in many workloads, near-saturating at 5–6 in low-latency runs). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens` accordingly: while accept length stays close to the draft-token count there is headroom to push them higher (more accepted tokens per step); if it falls well below, lower them — every rejected draft token is wasted verification compute.

* **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).

* **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.

* **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.2`, \~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, \~2.1 TB HBM) — **verified**; on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8), and those **multi-node BF16 recipes are still proposed/inferred** (`verified: false`). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. On B300, BF16 low-latency matches FP8 (the sm103 FP8 path is not yet optimized), but FP8 wins at the balanced/high-throughput points.

* **PD Disaggregation (prefill/decode).** GLM-5.2 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP.

* **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: \~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider.

* **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, \~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (\~89 GB/GPU vs \~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).

* **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the DSA triton backends (`--dsa-prefill-backend triton --dsa-decode-backend triton`, also SGLang's ROCm default), and the same `--chunked-prefill-size` and `--watchdog-timeout` as the FP8/BF16 recipes above. At `tp=4` this shape — 16 query heads per rank on an FP8 KV cache — is the one the gfx950 sparse-MLA kernels are tuned for; the `tp=8` recipes above stay on tilelang, which is what their published numbers were measured with. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified.

> [!NOTE]
>
> **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to \~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3).

* **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260930`. Choose **Low-Latency** for TP8/EP1, or **High-Throughput** for TP4/EP4 with HiCache (`--enable-hierarchical-cache --hicache-ratio 1.0`). HiCache uses the default `kernel`, `page_first`, and `write_through` settings. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions.

## 3. Advanced Usage

### 3.1 Reasoning

GLM-5.2 is a hybrid-reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Thinking is on by default; turn it off with `chat_template_kwargs: {"enable_thinking": False}` (the template variable is `enable_thinking`, not `thinking`).

**Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to inject a `Reasoning Effort: <level>` system line (only while thinking is on). **The template wires only two effective levels — `Max` and `High` — and if you don't pass `reasoning_effort` at all you get `Max`, the highest.** `"high"` is the *only* value that lowers effort; every other value (including `"low"` and `"medium"`) falls through to `Max`:

| `reasoning_effort` | Injected system line | Effect |
| - | - | - |
| *(not passed / unset)* | `Reasoning Effort: Max` | **default — highest reasoning** |
| `"high"` | `Reasoning Effort: High` | dials reasoning **down** |
| `"low"`, `"medium"`, any other value | `Reasoning Effort: Max` | falls through to `Max` (not a distinct level) |

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="zai-org/GLM-5.2-FP8",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     extra_body={"chat_template_kwargs": {"enable_thinking": True, "reasoning_effort": "high"}},
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: 1.  **Identify the core question:** The user wants to find 15% of 240.
> 2.  **Convert the percentage to a decimal:** 15% = 0.15
> 3.  **Multiply by the total:** 0.15 * 240 = 36
>     (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)
>
> Answer: 15% of 240 is **36**.
>
> Here is how you can calculate it:
> 0.15 × 240 = 36
> ```

### 3.2 Tool Calling

Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.2 emits the newer `<tool_call>…<arg_key>…<arg_value>…` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city",
>         "parameters": {
>             "type": "object",
>             "properties": {"city": {"type": "string"}},
>             "required": ["city"],
>         },
>     },
> }]
> resp = client.chat.completions.create(
>     model="zai-org/GLM-5.2-FP8",
>     messages=[{"role": "user", "content": "What's the weather in Paris?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.
>
> Tool calls: [
>   {
>     "id": "call_13fcd52146934b7781d06d4a",
>     "type": "function",
>     "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
>   }
> ]
> ```

### 3.3 HiCache (Hierarchical KV Caching)

For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.2's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern.

### 3.4 Claude Code Integration

GLM-5.2's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.2 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then:

```bash Command theme={null}
export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
export ANTHROPIC_AUTH_TOKEN="dummy"
export API_TIMEOUT_MS="3000000"
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"
claude
```

Two of these matter specifically for GLM-5.2:

* **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.2's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
* **`glm-5.2[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.2's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side.

For the full setup (streaming, tool-use, count\_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api).

### 3.5 Context Parallelism

> [!WARNING]
>
> Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` as shown below. Attention DP is supported: each DP group uses `tp_size / attn_dp_size` CP ranks. Dense FFNs retain the configured TP width; use `--moe-dense-tp-size 1` for local dense computation.

Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments:

```bash theme={null}
--attn-cp-size 8 \
--enable-prefill-cp \
--cp-strategy interleave \
```

which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill.

When deploying with PD Disaggregation, the prefill node can choose to enable [LayerSplit](https://z.ai/blog/scaling-pain) technique with

```bash theme={null}
--enable-dsa-cache-layer-split \
--attn-cp-size 8 \
--cp-strategy interleave \
```

With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side.


# GLM-5.3
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3

Deploy GLM-5.3 with SGLang — Z.ai's DeepSeek-Sparse-Attention (DSA) Mixture-of-Experts model with MTP speculative decoding and 1M context, on H200, B200, B300, GB300, and AMD MI300X/MI325X/MI355X, plus Ascend NPU recipes.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

> **Install and prepare Ascend NPUs**
>
> Use a Linux host with the Ascend driver and a compatible CANN / TorchNPU environment. Follow the
> [Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) and the
> [GLM-5.3 deployment tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3#installation) to start a container with the NPU devices, driver mounts, and checkpoint directory. Run the setup and the generated **Python** command inside that container.
>
> **A3**
>
> ```bash Command theme={null}
> docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080
> ```
>
> This CANN 9.1.0 image requires a compatible SGLang build, TorchNPU, ATB, and custom operators before serving. Set `SGLANG_ROOT` to that source checkout.
>
> Run the following setup before copying the generated command (CANN 9.1.0). CPU tuning requires host privileges.
>
> ```bash Command theme={null}
> unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
> echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
> sysctl -w vm.swappiness=0
> sysctl -w kernel.numa_balancing=0
> sysctl -w kernel.sched_migration_cost_ns=50000
> source /usr/local/Ascend/ascend-toolkit/set_env.sh
> source /usr/local/Ascend/nnal/atb/set_env.sh
> source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
> SGLANG_ROOT=/path/to/sglang
> export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
> export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib:${LD_LIBRARY_PATH:-}"
> ```
>
> **950PR/DT series**
>
> Use the compatible 950PR/DT series SGLang checkout, CANN environment, and custom operators used for your deployment. Set `SGLANG_ROOT` to that checkout; these recipes do not prescribe an image tag.
>
> ```bash Command theme={null}
> unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
> unset HCCL_IF_IP HCCL_SOCKET_FAMILY RANK_TABLE_FILE
> SGLANG_ROOT=/path/to/sglang
> export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
> ```
>
> Replace the model path in the generated command with your mounted ModelSlim checkpoint directory. Include its configuration, tokenizer, quantization metadata, and MTP weights.
>
> Use a locally prepared **W8A8** checkpoint for A3. On 950PR/DT series, both unified serving and PD disaggregation use the same **W8A8C8\_MXFP8** checkpoint. Both 950PR/DT series recipes explicitly use FP8 E4M3 KV cache and BF16 draft KV cache.
>
> Replace network placeholders such as `<network-interface>`, `<prefill-head-ip>`, and `<decode-head-ip>` in the generated NPU command with your cluster settings.

Pick your hardware + recipe to generate the launch command. The three serving strategies cover the common operating points:

* **Low-Latency** — fastest reply for a single user. Pick for chat.
* **Balanced** — good speed with several users at once. Use for typical multi-user serving.
* **High-Throughput** — most tokens per second across many users. Best for batch jobs.

> [!WARNING]
>
> All recipes here run the DSA indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on GLM-5.3.

## Playground

The Playground is where you experiment with **SGLang features beyond the deployment matrix**. It lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

* **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
* **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
* **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

## 1. Model Introduction

**GLM-5.3** is Z.ai's flagship Mixture-of-Experts model built on **DeepSeek Sparse Attention (DSA)**: a lightning indexer selects a sparse set of key tokens per query (top-2048), so attention cost stays near-constant as context grows. It keeps the same base architecture while updating post-training for complex coding and long-horizon tasks. It ships in two precisions — **FP8** (`zai-org/GLM-5.3`) and full **BF16** (`zai-org/GLM-5.3-BF16`) — both with **78 transformer layers**, **256 routed experts** (8 active per token), a **1M-token context window**, and a single **MTP (Multi-Token Prediction)** layer for built-in EAGLE-style speculative decoding. FP8 is the recommended deployment; BF16 (\~1.5 TB) needs an 8×B300 node or a multi-node setup. For Blackwell, RadixArk publishes an **experimental NVFP4** build (`RadixArk/GLM-5.3-NVFP4`, Model Optimizer) that quantizes only the routed experts' linear weights and activations to 4-bit (attention, shared experts, dense layers, MTP, embeddings, and the LM head stay unquantized), cutting the weight footprint to \~0.45 TB so a 4-GPU GB300 node can serve it with TP4. **The NVFP4 cells are experimental** — benchmark data (accuracy / throughput) is still pending; treat them as provisional until the benchmark cards fill in.

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Architecture</th>
      <th>Context</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[GLM-5.3](https://huggingface.co/zai-org/GLM-5.3)**</td>
      <td>MoE · DSA · 256 experts (top-8) · MTP · FP8</td>
      <td>1,048,576</td>
    </tr>

    <tr>
      <td>**[GLM-5.3-BF16](https://huggingface.co/zai-org/GLM-5.3-BF16)**</td>
      <td>MoE · DSA · 256 experts (top-8) · MTP · BF16</td>
      <td>1,048,576</td>
    </tr>

    <tr>
      <td>**[GLM-5.3-NVFP4](https://huggingface.co/RadixArk/GLM-5.3-NVFP4)**</td>
      <td>MoE · DSA · 256 experts (top-8) · MTP · NVFP4</td>
      <td>1,048,576</td>
    </tr>
  </tbody>
</table>

**Recommended generation:** `temperature=1.0`, `top_p=0.95` (the checkpoint's `generation_config.json` defaults; informational — do not hardcode in client code).

**Resources:** [GLM-5.3](https://huggingface.co/zai-org/GLM-5.3) · [GLM-5.3-BF16](https://huggingface.co/zai-org/GLM-5.3-BF16) · [GLM-5.3-NVFP4](https://huggingface.co/RadixArk/GLM-5.3-NVFP4).

## 2. Configuration Tips

* **DeepSeek Sparse Attention (DSA).** GLM-5.3 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.3's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`.

* **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens`: lower the draft length when rejected draft tokens create excess verification work.

* **DFlash2 (block-diffusion draft).** The **Speculative** card in the [Playground above](#playground) also offers **DFlash2**, which replaces the in-checkpoint MTP layer with the separately trained block-diffusion drafter [`incoai/GLM-5.3-DFlash2`](https://huggingface.co/incoai/GLM-5.3-DFlash2). It proposes a whole block per step and the target verifies the block in one forward pass, so output quality stays the target's. The block size — 8, i.e. 7 draft tokens per verification step — comes from the draft checkpoint's own `dflash_config`, so no `--speculative-num-draft-tokens` is passed; the draft is a small dense model and runs on `fa4` instead of the target's DSA backends. Note that DFLASH runs on **CUDA/NPU only** and rejects **DP-Attention**, so turn DP-Attention off in the **Attention** card before selecting it on a high-throughput base. The draft repository is public but licensed CC BY-NC-ND 4.0 for research and evaluation.

* **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4).

* **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP.

* **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.3-BF16`, \~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, \~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8.

* **PD Disaggregation (prefill/decode).** GLM-5.3 is a DSA model and runs under prefill/decode disaggregation — toggle the **PD Disagg** card in the [Playground above](#playground) (pick a Prefill/Decode role + transfer backend, then front the roles with `sglang_router.launch_router --pd-disaggregation`). The Mooncake backend **auto-detects the InfiniBand HCA**, so no device flag is needed by default; only add `--disaggregation-ib-device mlx5_0` (your NIC) if auto-detection picks the wrong device or KV transfer fails to connect. On H200 Docker, expose the IB HCAs to the container (`--privileged --ulimit memlock=-1`, or `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK`) — without IB exposure Mooncake silently falls back to TCP.

* **Chunked-prefill size is regime-dependent.** For long-input balanced workloads, start with `--chunked-prefill-size 32768` and tune it together with `--max-running-requests` for your input length and KV capacity. Keep the default chunked-prefill size for the high-throughput recipe unless profiling shows that prefill is the bottleneck.

* **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.3`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.3-BF16`, \~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there. Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). GLM-5.3 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2).

* **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel because the gfx950 spec-decode draft kernel is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP.

## 3. Advanced Usage

### 3.1 Reasoning

GLM-5.3 is a reasoning model, and generated commands enable `--reasoning-parser auto` (which resolves to `glm45` for GLM-5.3) by default so thinking is separated from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Without the parser the server returns the thinking and the answer as one `content` string with a stray `</think>` between them, because the chat template opens `<think>` in the generation prompt. You can disable **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground) when an integration needs that raw format. The chat template defaults `clear_thinking` to `false`; for multi-turn chat, pass `chat_template_kwargs: {"clear_thinking": True}` so previous reasoning is cleared before the next response.

**Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to select `low`, `high`, or `max`. If you omit it or pass another value, the template uses `max`.

| `reasoning_effort` | Injected system line | Effect |
| - | - | - |
| *(not passed / unset)*, `"max"` | `Reasoning Effort: Max` | default — highest reasoning effort |
| `"high"` | `Reasoning Effort: High` | high reasoning effort |
| `"low"` | `Reasoning Effort: Low` | low reasoning effort |
| any other value | `Reasoning Effort: Max` | falls back to the default |

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="zai-org/GLM-5.3",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     extra_body={"chat_template_kwargs": {"clear_thinking": True, "reasoning_effort": "high"}},
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: 1.  **Identify the core question:** The user wants to find 15% of 240.
> 2.  **Convert the percentage to a decimal:** 15% = 0.15
> 3.  **Multiply by the total:** 0.15 * 240 = 36
>     (Quick mental math: 10% of 240 = 24; 5% = 12; 24 + 12 = 36.)
>
> Answer: 15% of 240 is **36**.
>
> Here is how you can calculate it:
> 0.15 × 240 = 36
> ```

### 3.2 Tool Calling

Generated commands enable `--tool-call-parser auto` by default, so structured calls are returned in `message.tool_calls` with `finish_reason: "tool_calls"`. `auto` resolves to **`glm47`** for GLM-5.3: the model emits the newer `<tool_call>…<arg_key>…<arg_value>…` format, which the older `glm45` parser does not parse (the call would be left as raw text in `content`). Running with no tool-call parser fails the same way, and `finish_reason` stays `"stop"`, so an agent loop never sees the call. You can disable **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground) when tool calling is not needed. On thinking mode the turn also fills `reasoning_content`, so print both fields.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city",
>         "parameters": {
>             "type": "object",
>             "properties": {"city": {"type": "string"}},
>             "required": ["city"],
>         },
>     },
> }]
> resp = client.chat.completions.create(
>     model="zai-org/GLM-5.3",
>     messages=[{"role": "user", "content": "What's the weather in Paris?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user wants to know the weather in Paris. I'll call the get_weather function with "Paris" as the city.
>
> Tool calls: [
>   {
>     "id": "call_13fcd52146934b7781d06d4a",
>     "type": "function",
>     "function": {"name": "get_weather", "arguments": "{\"city\": \"Paris\"}"}
>   }
> ]
> ```

### 3.3 HiCache (Hierarchical KV Caching)

For long-context, prefix-heavy workloads, enable hierarchical KV caching to spill cold KV blocks to host memory (toggle the **Hierarchical KV Cache** card in the [Playground above](#playground)). Useful given GLM-5.3's 1M-token window; pair `--hicache-ratio` with a write policy that matches your reuse pattern.

### 3.4 Claude Code Integration

GLM-5.3's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.3 server with only environment variables — no code change. Launch the server with `--reasoning-parser auto --tool-call-parser auto` (any recipe from the Deployment panel above works), then:

```bash Command theme={null}
export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
export ANTHROPIC_AUTH_TOKEN="dummy"
export API_TIMEOUT_MS="3000000"
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.3[1m]"
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.3[1m]"
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.3[1m]"
claude
```

Two of these matter specifically for GLM-5.3:

* **`CLAUDE_CODE_ATTRIBUTION_HEADER=0`** — Claude Code prepends a per-request attribution block to the system prompt. GLM-5.3's chat template renders `tools` **before** `system`, so that per-request hash is the first token to diverge between turns and the radix prefix cache re-prefills the whole system + history every turn. This env removes the block and restores prefix-cache reuse.
* **`glm-5.3[1m]`** as the model name — the `[1m]` suffix is the client-side hint that enables Claude Code's 1M-context beta, matching GLM-5.3's 1,048,576-token window. Without it, context is capped well below 1M. SGLang does not validate the `model` field, so any name is accepted server-side.

For the full setup (streaming, tool-use, count\_tokens, persisting env in `~/.claude/settings.json`, troubleshooting), see [Anthropic-Compatible API](../../../docs/basic_usage/anthropic_api).

### 3.5 Context Parallelism

> [!WARNING]
>
> Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.3. For prefill CP on CUDA, use `interleave` as shown below.

Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments:

```bash theme={null}
--attn-cp-size 8 \
--enable-prefill-cp \
--cp-strategy interleave \
```

which splits the sequence equally across `--attn-cp-size` ranks during attention forward. The trade off for prefill CP is that it will introduce extra all-gather operation before indexer-topk and attention kernels, so it will increase latency for decode (in unified deployment) or short prefill.

You can combine interleave CP with attention DP. Each attention-DP group owns
`tp_size / attn_dp_size` CP ranks; for example, `--tp 8 --attn-dp-size 2`
uses CP4 within each DP group. Dense FFNs retain their configured TP width:
the default runs them over the full TP group, while `--moe-dense-tp-size 1`
runs them independently on local token shards.

When deploying with PD Disaggregation, a **prefill worker using the Mooncake transfer backend** can enable the [LayerSplit](https://z.ai/blog/scaling-pain) technique with

```bash theme={null}
--enable-dsa-cache-layer-split \
--enable-prefill-cp \
--attn-cp-size 8 \
--cp-strategy interleave \
```

With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side.


# GLM-5.3-Flash
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3-Flash

Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, GB300, MI355X, and TPU7x, with MTP and multimodal serving, plus Ascend A3 and 950PR/DT series NPU recipes.

## Deployment

> **Install SGLang**
>
> Use an SGLang build that includes GLM-5.3-Flash support (v0.5.20 or later).
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> For the listed NVIDIA hardware, the deployment panel can render a complete `docker run` command. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup.
>
> For MI355X, follow the [AMD GPU installation guide](/docs/hardware-platforms/amd_gpu) and run the generated **Python** command on four GPUs. Select FP8 or [MXFP4](https://huggingface.co/OneNexus/GLM-5.3-Flash-MXFP4); SGLang downloads the model and tokenizer automatically.
>
> For TPU7x, install [sglang-torchtpu](https://github.com/sgl-project/sglang-torchtpu) from source (its `main` branch; the `v0.1.0-tpu` image predates the GLM-5.3-Flash MTP fixes) and run the generated **Python** command on one TPU7x host with 4 chips:
>
> ```bash Command theme={null}
> git clone https://github.com/sgl-project/sglang-torchtpu.git
> cd sglang-torchtpu
> cp python/pyproject_tpu.toml python/pyproject.toml
> pip install --extra-index-url https://download.pytorch.org/whl/cpu -e python/
> pip install --no-deps -e .
> ```

> **Install and prepare Ascend NPUs**
>
> Use a Linux host with the Ascend driver and a compatible CANN / TorchNPU environment. Follow the
> [Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) and the
> [GLM-5.3-Flash deployment tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash#installation) to start a container with the NPU devices, driver mounts, and checkpoint directory. Run the setup and the generated **Python** command inside that container.
>
> **A3**
>
> ```bash Command theme={null}
> docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-glm5.3-flash
> ```
>
> Run the following setup before copying the generated command (CANN 9.1.0). CPU tuning requires host privileges.
>
> ```bash Command theme={null}
> unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
> source /usr/local/Ascend/ascend-toolkit/set_env.sh
> source /usr/local/Ascend/nnal/atb/set_env.sh
> source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
> ```
>
> **950PR/DT series**
>
> Use a CANN 9.1.0 SGLang image with the custom operators and MemFabric runtime required by the tutorial; PD mode also requires DP-attention support. The tutorial does not specify a 950PR/DT series image tag.
>
> ```bash Command theme={null}
> NPU_IMAGE="<your-compatible-950pr-dt-series-image>"
> docker pull "$NPU_IMAGE"
> ```
>
> ```bash Command theme={null}
> unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY ASCEND_LAUNCH_BLOCKING
> source /usr/local/Ascend/ascend-toolkit/set_env.sh
> source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
> source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash
> source /usr/local/memfabric_hybrid/set_env.sh
> ```
>
> Replace the model path in the generated command with your mounted ModelSlim checkpoint directory. Include its configuration, tokenizer, quantization metadata, and MTP weights.
>
> Use [W8A8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8) for A3 and [W8A8\_MXFP8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8-mxfp8) for 950PR/DT series.
>
> Replace network placeholders such as `<network-interface>`, `<prefill-head-ip>`, and `<decode-head-ip>` in the generated NPU command with your cluster settings.

The cookbook opens on **B300**. Choose your hardware, then choose the operating point that matches your workload:

* **Low Latency** starts with MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses.
* **High Throughput** starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches.

Every listed GPU platform exposes both strategies; TPU7x has a Low Latency recipe only. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported.

The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, Breakable Cuda Graph, and HiCache tiers. The MI355X commands are marked **Not Verified** because the selector defaults to HiCache off, whereas the measured AgentX configuration adds the host tier described in [Extend the cache hierarchy](#extend-the-cache-hierarchy). Do not treat the CUDA benchmark rows as MI355X measurements.

**Breakable Cuda Graph** defaults to **Off**. On NVIDIA, select **On** to add `--cuda-graph-backend-prefill breakable` to the generated command. This requires a build that includes [PR #38522](https://github.com/sgl-project/sglang/pull/38522); the option is disabled on MI355X.

> [!NOTE]
>
> Generated commands leave `--mamba-full-memory-ratio` at its `0.9` default, which is a generic starting point rather than a workload-tuned split: too low starves the KDA state pool and clamps `max_running_requests`, too high over-provisions it and shrinks the KV pool. Use the repo-local [`compute-mamba-ratio`](https://github.com/sgl-project/sglang/blob/main/.claude/skills/compute-mamba-ratio/SKILL.md) skill to compute the balanced ratio — or the `--max-mamba-cache-size` pin to use instead — from your average request length and the two pool sizes printed in one boot log. See [Size both memory pools](#size-both-memory-pools) for what each pool caps.

## Playground

Use the Playground for lower-level tuning such as attention parallelism, MoE communication, PD disaggregation, speculative decoding, and reasoning or tool parsers. It inherits every selection from the deployment panel and shows only the command-line diff. The MoE backend defaults to `flashinfer_trtllm` on B300, B200, GB300 and GB200, to DeepGemm on Hopper, and to AITER on MI355X.

## 1. Model introduction

GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a hybrid attention architecture — 320B total parameters with 18B active. Its 45 text layers combine MLA attention, DSA sparse attention, and KDA linear attention, and a 24-layer vision encoder handles image and video input. The checkpoint uses 288 routed experts with 8 active experts per token and includes a native MTP draft layer for speculative decoding. See the [GLM-5.3-Flash blog](https://z.ai/blog/glm-5.3-flash) and the [GLM-5 technical report](https://arxiv.org/abs/2602.15763) for training details.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Attribute</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Architecture</td>
      <td>MoE with hybrid attention (MLA, DSA, KDA), mHC, and MTP — 320B total / 18B active parameters</td>
    </tr>

    <tr>
      <td>Precision</td>
      <td>FP8 weights ([`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash)); MXFP4 on MI355X ([`OneNexus/GLM-5.3-Flash-MXFP4`](https://huggingface.co/OneNexus/GLM-5.3-Flash-MXFP4)); NVFP4 on Blackwell from [`RadixArk/GLM-5.3-Flash-NVFP4`](https://huggingface.co/RadixArk/GLM-5.3-Flash-NVFP4) and [`nvidia/GLM-5.3-Flash-NVFP4`](https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4); FP8 KV cache by default on Blackwell and MI355X, BF16 KV cache on H100 and H200</td>
    </tr>

    <tr>
      <td>Context</td>
      <td>1M tokens</td>
    </tr>

    <tr>
      <td>Inputs</td>
      <td>Text, image, and video</td>
    </tr>

    <tr>
      <td>Generation defaults</td>
      <td>`temperature=1.0`, `top_p=0.95`, thinking enabled</td>
    </tr>
  </tbody>
</table>

The deployment recipes inherit the checkpoint's 1,048,576-token context limit and generation configuration. Override sampling only when your application has its own evaluated settings.

## 2. Configuration tips

### Choose a strategy

Start with **Low Latency** for chat and agent workloads. It drafts from the checkpoint's MTP head at a fixed depth (5 steps, top-k 1, 6 draft tokens) with natural acceptance. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value.

Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes. On MI355X, the measured AgentX frontier used Low Latency (EAGLE 5/1/6); High Throughput is an unmeasured speculation-off starting point.

### Change the speculative algorithm

The **Speculative** card in the Playground changes the algorithm without leaving the selected strategy:

* **EAGLE / MTP 5-1-6** is exactly what Low Latency serves, so a Low Latency base starts on this chip. Pick it from a High Throughput base to keep that recipe's other settings and add the MTP head.
* **Off (greedy)** strips the whole `--speculative-*` family, which is what High Throughput already starts from.
* **DFlash2** swaps the in-checkpoint MTP head for the trained block-diffusion draft in [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2). The draft proposes a whole block per step and the target verifies it in one forward pass, so output quality stays the target's. Its block size comes from the draft checkpoint, and the draft runs on `fa4` rather than the target's DSA backends. The hidden-state capture it needs ([PR #36708](https://github.com/sgl-project/sglang/pull/36708)) shipped with the GLM-5.3-Flash support in v0.5.20, so the image pinned above is enough. The draft repository is access-gated: request access on its model page, then download it alongside the target before serving. This combination is not yet measured on the cookbook hardware, so treat it as a starting point.

Neither algorithm runs with DP-Attention; the card disables the affected chips and names the reason. DFlash2's FA4 draft path is also disabled on MI355X.

### Serve on TPU7x

The TPU7x recipe serves the language model with DeepSeek Sparse Attention on the Pallas sparse MLA kernels and MTP 3/1/4 on the experimental chain speculative worker (`SGLANG_TPU_ENABLE_EXPERIMENTAL_SPEC_WORKER=1`). On TPU7x, 3/1/4 decodes faster than 5/1/6 at the same accept length, because each extra draft row widens the target verify. It is sized for one request with 8K input and 1K output: `--context-length 10240`, `--max-running-requests 1`, and a 20,480-token KV pool. Raise `--context-length`, `--max-total-tokens`, `--max-running-requests` and `--max-mamba-cache-size` together for longer or concurrent requests; those settings are outside the verified command. The first start compiles every decode and prefill shape listed in `--tpu-compile-bs` and `--tpu-compile-tokens`, so it takes longer than later starts that reuse the compilation cache.

### Size both memory pools

GLM-5.3-Flash maintains a paged KV pool for attention and a separate KDA state pool. The KDA state pool can limit concurrency before the KV pool is full. If startup reduces `max_running_requests` because of KDA state capacity, increase `--mamba-full-memory-ratio` or set `--max-mamba-cache-size` for the expected concurrency, then tune `--max-running-requests` to the workload.

Keep the prefix cache enabled for every strategy.

Keep the checkpoint's KDA lower-bound setting unchanged. In particular, do not override `linear_lower_bound` through `--json-model-override-args`.

### Keep the KV and DSA backends paired

On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300 this pairing measured 2.9–5.7% higher throughput and about 1.8x the KV token capacity at identical pool bytes, with GSM8K accuracy within noise of BF16. BF16 KV with TileLang DSA remains selectable in the deployment panel and is the default on H100 and H200, where FP8 KV with TRT-LLM DSA is disabled. MI355X instead uses FP8 KV with TileLang DSA in the recorded ROCm recipe. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination, and TRT-LLM DSA is not the MI355X backend.

On Hopper, BF16 KV can also run the sparse prefill in FP8 with `--dsa-prefill-backend flashmla_sparse_q8` (decode stays on `tilelang` and the cache stays BF16): on H200 this pairing measured 5.8–12.1% higher throughput on prefill-only requests, with GSM8K accuracy within noise of BF16 + TileLang. It is SM90-only and requires a build that includes [PR #41790](https://github.com/sgl-project/sglang/pull/41790); FP8 KV is not available with it for this model.

### Extend the cache hierarchy

Keep **HiCache** off when GPU memory is sufficient. Select **L1 + L2** to spill reusable cache entries into host memory. Select **+ L3** only after configuring Mooncake on every serving node; the generated command exposes the required configuration path. These options remain selectable but are marked **Not Verified** until the resulting command is validated on the chosen hardware.

For agentic MXFP4 serving on MI355X, select **Low Latency** with **L1 + L2**. The host tier uses `--hicache-size 180` (about 190 GB of host memory per rank once the KDA state and DSA indexer pools are included, 750 GB per node) and requires a build that includes [PR #42178](https://github.com/sgl-project/sglang/pull/42178). With less free host memory, keep HiCache off; the server does not start when the host tier does not fit.

### Multimodal memory

All strategies enable multimodal serving. Image and video input use `Glm5NextProcessor`, which the pinned Transformers release ships (the MI355X checks predate that release and used commit `e4052f55`). On MI355X, text, image, and an image-bearing follow-up were qualified with that build; video was not evaluated. The processor samples video at 2 FPS and caps video input at 240,000 visual tokens. Install `torchcodec` in the serving environment before sending video requests. For very long videos on 4x GB300, use encoder disaggregation to isolate the vision encoder's memory spikes from language decoding.

The default multimodal feature transport is automatic, and on a single CUDA node auto resolves to CPU transport. The MI355X selection explicitly emits CPU transport, matching its image-serving check. CUDA IPC is opt-in: pass `--mm-feature-transport cuda_ipc` when lower transfer latency matters more than the GPU memory the IPC pool reserves. CUDA VMM transport applies only to multi-node GB200/GB300 systems on the MNNVL fabric, where auto selects it.

## 3. Advanced usage

### 3.1 Reasoning

Thinking is enabled by the checkpoint's generation configuration, and generated commands enable `--reasoning-parser auto` (which resolves to `glm45` for GLM-5.3-Flash) by default. The OpenAI-compatible API then places thinking in `message.reasoning_content` and the final answer in `message.content`. You can disable **Reasoning Parser** in the Playground when an integration needs the raw response format.

To disable thinking for a request, pass `chat_template_kwargs: {"thinking": false}` in the request body.

### 3.2 Tool calling

Generated commands enable `--tool-call-parser auto` (which resolves to `glm47` for GLM-5.3-Flash) by default, so structured calls are returned in `message.tool_calls`. You can disable **Tool Call Parser** in the Playground when tool calling is not needed. On follow-up turns, read both `reasoning_content` and `content` because a thinking model can use either field around tool execution.

### 3.3 Multimodal serving

The base recipes accept image and video content through the OpenAI-compatible chat API. Keep the processor defaults unless you have measured a different sampling or resize policy. Inputs above the video token budget are clamped to the processor's limit rather than rejected.

### 3.4 Encoder disaggregation

Encoder disaggregation separates vision preprocessing from language inference. The verified topology uses one 4x GB300 node shared by an encoder-only TP4 process on port 30001 and a language-only TP4 process on port 30000. Start the encoder first.

> **Encoder server (GB300)**
>
> ```bash Command theme={null}
> sglang serve \
>   --model-path zai-org/GLM-5.3-Flash \
>   --tp-size 4 \
>   --encoder-only \
>   --host 0.0.0.0 \
>   --port 30001
> ```

> **Language server (GB300)**
>
> ```bash Command theme={null}
> sglang serve \
>   --model-path zai-org/GLM-5.3-Flash \
>   --tp-size 4 \
>   --attention-backend dsa \
>   --dsa-prefill-backend tilelang \
>   --dsa-decode-backend tilelang \
>   --linear-attn-backend triton \
>   --kv-cache-dtype bfloat16 \
>   --quantization fp8 \
>   --moe-runner-backend flashinfer_trtllm \
>   --max-running-requests 64 \
>   --chunked-prefill-size 8192 \
>   --max-prefill-tokens 8192 \
>   --disable-prefill-cuda-graph \
>   --speculative-algorithm EAGLE \
>   --speculative-num-steps 3 \
>   --speculative-eagle-topk 1 \
>   --speculative-num-draft-tokens 4 \
>   --language-only \
>   --encoder-urls http://localhost:30001 \
>   --mem-fraction-static 0.78 \
>   --host 0.0.0.0 \
>   --port 30000
> ```

This topology served image requests and videos up to 238,080 visual tokens. In a concurrent long-video workload, the largest observed decode gap fell from 5.53 seconds in unified serving to 1.79 seconds with encoder disaggregation. Keep `--mem-fraction-static 0.78` on the language process so the encoder retains room for vision workspaces.

Install `torchcodec` for video and see the [encoder disaggregation guide](/docs/advanced_features/epd_disaggregation) for the generic architecture and operational model.

### 3.5 Prefill Context Parallelism

Use prefill context parallelism (CP) to distribute DSA prefill computation for long prompts, including multimodal inputs. This hybrid model uses **interleave CP for DSA layers** and **head-wise tensor parallelism for KDA layers**, sharing the same communication group. Layer boundaries all-gather the context shards before KDA and reduce-scatter its output back onto those shards. The mHC residual streams and their coefficients stay local, with each output's reduction completed before the residual update.

In the Playground's **Attention** card, select **TP=8** and **CP=8** on B300 or B200 to add `--attn-cp-size 8 --enable-prefill-cp --cp-strategy interleave`. For CP4, set TP=4 and CP=4. With CP enabled, the **MoE** card restricts EP to **1** or the CP size; an incompatible EP selection falls back to 1. Choosing EP=CP also adds `--moe-dense-tp-size` with the CP size. Prefill CP and PD disaggregation cannot be enabled together for this model.

> **Example of launching with Prefill CP on Blackwell GPU**
>
> ```bash Command theme={null}
> sglang serve \
>   --model-path zai-org/GLM-5.3-Flash \
>   --tp-size 4 \
>   --attn-cp-size 4 \
>   --enable-prefill-cp \
>   --cp-strategy interleave \
>   --moe-dense-tp-size 4 \
>   --ep-size 1 \
>   --speculative-algorithm EAGLE \
>   --speculative-num-steps 5 \
>   --speculative-eagle-topk 1 \
>   --speculative-num-draft-tokens 6 \
>   --dsa-prefill-backend trtllm \
>   --dsa-decode-backend trtllm \
>   --kv-cache-dtype fp8_e4m3 \
>   --moe-runner-backend flashinfer_trtllm \
>   --reasoning-parser auto \
>   --tool-call-parser auto \
>   --host 0.0.0.0 \
>   --port 30000
> ```

### 3.6 PD disaggregation (preview)

PD splits prefill and decode into separate server groups behind a router. For this hybrid model, the transfer moves both the paged DSA KV and the KDA recurrent state.

The Playground's **PD Disagg** card defaults to **Off**, with **Mooncake** as its only transfer backend. Select **Prefill** or **Decode** to add `--disaggregation-mode prefill` or `--disaggregation-mode decode`, respectively, together with `--disaggregation-transfer-backend mooncake`. Selecting either role disables prefill CP. The generated prefill and decode commands use ports 30000 and 30100, respectively.

> [!WARNING]
>
> PD serving is mechanically validated with dummy weights only — startup, bootstrap, state transfer, and request flow all work on 4x GB300. It has not been load- or accuracy-tested. Treat it as a preview until the real-weight gate completes.

> **PD serving on 4x GB300 (single node)**
>
> ```bash Prefill (GPU 0-1) theme={null}
> sglang serve \
>   --model-path zai-org/GLM-5.3-Flash \
>   --tp-size 2 \
>   --dsa-prefill-backend tilelang \
>   --dsa-decode-backend tilelang \
>   --kv-cache-dtype bfloat16 \
>   --moe-runner-backend triton \
>   --disaggregation-mode prefill \
>   --disaggregation-bootstrap-port 8998 \
>   --disaggregation-transfer-backend nixl \
>   --host 0.0.0.0 \
>   --port 31000
> ```
>
> ```bash Decode (GPU 2-3) theme={null}
> sglang serve \
>   --model-path zai-org/GLM-5.3-Flash \
>   --tp-size 2 \
>   --base-gpu-id 2 \
>   --dsa-prefill-backend tilelang \
>   --dsa-decode-backend tilelang \
>   --kv-cache-dtype bfloat16 \
>   --moe-runner-backend triton \
>   --disaggregation-mode decode \
>   --disaggregation-transfer-backend nixl \
>   --host 0.0.0.0 \
>   --port 32000
> ```
>
> ```bash Router theme={null}
> python -m sglang_router.launch_router \
>   --pd-disaggregation \
>   --mini-lb \
>   --prefill http://127.0.0.1:31000 8998 \
>   --decode http://127.0.0.1:32000 \
>   --host 0.0.0.0 \
>   --port 30000
> ```
>
> The positional `8998` after `--prefill` must equal the prefill server's `--disaggregation-bootstrap-port`.

Operational notes:

* Give each role a distinct `--nccl-port` when both share one node.
* Single-node NIXL needs `UCX_NET_DEVICES=lo` and `UCX_TLS=tcp,cuda_copy,cuda_ipc,self,sm` in both server environments.
* The validated arm used the triton MoE runner; the flashinfer\_trtllm runner from the deployment recipes is untested under PD.

Known limitations:

* Speculative decoding does not start under PD at the current cut (draft-graph capture width assert on the prefill role; decode-side memory pressure at TP2). Run PD without speculative flags.
* Prefill and decode with different TP sizes transfer state through slice paths, but numeric correctness is unverified. Keep both roles at the same TP size.


# GLM Glyph
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-Glyph



## 1. Model Introduction

[Glyph](https://huggingface.co/zai-org/Glyph) is a powerful language model developed by Zhipu AI, featuring advanced capabilities in reasoning, function calling, and multi-modal understanding.

**Hardware Support:** NVIDIA B200/H100/H200, AMD MI300X/MI325X/MI355X

**Key Features:**

* **Advanced Reasoning**: Built-in reasoning capabilities for complex problem-solving
* **Multiple Quantizations**: BF16 and FP8 variants for different performance/memory trade-offs
* **High Performance**: Optimized for both throughput and latency scenarios

**Available Models:**

* **BF16 (Full precision)**: [zai-org/Glyph](https://huggingface.co/zai-org/Glyph)
* **FP8 (8-bit quantized)**: [zai-org/Glyph-FP8](https://huggingface.co/zai-org/Glyph-FP8)

**License:**

Please refer to the [official Glyph model card](https://huggingface.co/zai-org/Glyph) for license details.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and other options.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Thinking Budget:** Use `--enable-custom-logit-processor` flag and pass `Glm4MoeThinkingBudgetLogitProcessor` in requests to cap the model's thinking token count. See the [GLM-4.5 cookbook page](/cookbook/autoregressive/GLM/GLM-4.5) for the full Thinking Budget usage example.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Thinking Mode

Glyph supports thinking mode for enhanced reasoning. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path zai-org/Glyph \
  --reasoning-parser glm45 \
  --tp 4
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="zai-org/Glyph",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

**Disable Thinking Mode:**

To disable thinking mode for a specific request:

```python Example theme={null}
response = client.chat.completions.create(
    model="zai-org/Glyph",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
```

#### 4.2.2 Tool Calling

Glyph supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path zai-org/Glyph \
  --reasoning-parser glm45 \
  --tool-call-parser glm45 \
  --tp 4
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="zai-org/Glyph",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="zai-org/Glyph",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Model: Glyph
* SGLang Version: 0.5.6.post1

**Benchmark Methodology:**

We use industry-standard benchmark configurations to ensure results are comparable across frameworks and hardware platforms.

#### 5.1.1 Standard Scenario Benchmark

* **Model Deployment**

```bash Command theme={null}
python -m sglang.launch_server \
  --model zai-org/Glyph \
  --tp 2
```

##### 5.1.1.1 Low Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  17.03
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.59
Input token throughput (tok/s):          358.17
Output token throughput (tok/s):         247.74
Peak output token throughput (tok/s):    251.00
Peak concurrent requests:                3
Total token throughput (tok/s):          605.91
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1702.14
Median E2E Latency (ms):                 1361.72
---------------Time to First Token----------------
Mean TTFT (ms):                          22.35
Median TTFT (ms):                        22.61
P99 TTFT (ms):                           23.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.99
Median TPOT (ms):                        3.99
P99 TPOT (ms):                           4.01
---------------Inter-Token Latency----------------
Mean ITL (ms):                           3.99
Median ITL (ms):                         3.99
P95 ITL (ms):                            4.03
P99 ITL (ms):                            4.12
Max ITL (ms):                            7.46
==================================================
```

##### 5.1.1.2 Medium Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  16.27
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40804
Request throughput (req/s):              4.92
Input token throughput (tok/s):          2438.06
Output token throughput (tok/s):         2507.94
Peak output token throughput (tok/s):    3069.00
Peak concurrent requests:                26
Total token throughput (tok/s):          4946.00
Concurrency:                             13.44
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2733.43
Median E2E Latency (ms):                 2892.98
---------------Time to First Token----------------
Mean TTFT (ms):                          33.10
Median TTFT (ms):                        27.73
P99 TTFT (ms):                           49.34
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.33
Median TPOT (ms):                        5.39
P99 TPOT (ms):                           5.86
---------------Inter-Token Latency----------------
Mean ITL (ms):                           5.30
Median ITL (ms):                         4.89
P95 ITL (ms):                            5.54
P99 ITL (ms):                            21.17
Max ITL (ms):                            25.14
==================================================
```

##### 5.1.1.3 High Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  25.67
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    252657
Request throughput (req/s):              19.48
Input token throughput (tok/s):          9733.69
Output token throughput (tok/s):         9843.99
Peak output token throughput (tok/s):    13398.00
Peak concurrent requests:                127
Total token throughput (tok/s):          19577.68
Concurrency:                             89.49
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4593.75
Median E2E Latency (ms):                 4431.03
---------------Time to First Token----------------
Mean TTFT (ms):                          48.66
Median TTFT (ms):                        35.88
P99 TTFT (ms):                           120.61
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.10
Median TPOT (ms):                        9.55
P99 TPOT (ms):                           11.00
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.01
Median ITL (ms):                         6.51
P95 ITL (ms):                            23.19
P99 ITL (ms):                            25.54
Max ITL (ms):                            52.93
==================================================
```

#### 5.1.2 Reasoning Scenario Benchmark

##### 5.1.2.1 Low Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  201.53
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  44462
Total generated tokens (retokenized):    44455
Request throughput (req/s):              0.05
Input token throughput (tok/s):          30.27
Output token throughput (tok/s):         220.63
Peak output token throughput (tok/s):    251.00
Peak concurrent requests:                2
Total token throughput (tok/s):          250.90
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   20151.45
Median E2E Latency (ms):                 21576.31
---------------Time to First Token----------------
Mean TTFT (ms):                          2362.23
Median TTFT (ms):                        23.03
P99 TTFT (ms):                           21310.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.00
Median TPOT (ms):                        4.00
P99 TPOT (ms):                           4.01
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.00
Median ITL (ms):                         4.00
P95 ITL (ms):                            4.05
P99 ITL (ms):                            4.08
Max ITL (ms):                            5.67
==================================================
```

##### 5.1.2.2 Medium Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  118.67
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  318306
Total generated tokens (retokenized):    318270
Request throughput (req/s):              0.67
Input token throughput (tok/s):          334.27
Output token throughput (tok/s):         2682.26
Peak output token throughput (tok/s):    3264.00
Peak concurrent requests:                19
Total token throughput (tok/s):          3016.53
Concurrency:                             13.74
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   20387.23
Median E2E Latency (ms):                 20466.09
---------------Time to First Token----------------
Mean TTFT (ms):                          132.47
Median TTFT (ms):                        27.19
P99 TTFT (ms):                           583.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.09
Median TPOT (ms):                        5.13
P99 TPOT (ms):                           5.19
---------------Inter-Token Latency----------------
Mean ITL (ms):                           5.09
Median ITL (ms):                         5.08
P95 ITL (ms):                            5.18
P99 ITL (ms):                            5.57
Max ITL (ms):                            522.26
==================================================
```

##### 5.1.2.3 High Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  150.00
Total input tokens:                      158939
Total input text tokens:                 158939
Total input vision tokens:               0
Total generated tokens:                  1301025
Total generated tokens (retokenized):    1300901
Request throughput (req/s):              2.13
Input token throughput (tok/s):          1059.59
Output token throughput (tok/s):         8673.49
Peak output token throughput (tok/s):    11899.00
Peak concurrent requests:                71
Total token throughput (tok/s):          9733.09
Concurrency:                             54.71
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   25645.42
Median E2E Latency (ms):                 26913.26
---------------Time to First Token----------------
Mean TTFT (ms):                          163.75
Median TTFT (ms):                        93.67
P99 TTFT (ms):                           426.19
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.27
Median TPOT (ms):                        6.39
P99 TPOT (ms):                           6.59
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.27
Median ITL (ms):                         0.17
P95 ITL (ms):                            32.94
P99 ITL (ms):                            67.89
Max ITL (ms):                            136.00
==================================================
```

#### 5.1.3 Summarization Scenario Benchmark

#### 5.1.3.1 Low Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  17.44
Total input tokens:                      41941
Total input text tokens:                 41941
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.57
Input token throughput (tok/s):          2405.19
Output token throughput (tok/s):         242.00
Peak output token throughput (tok/s):    250.00
Peak concurrent requests:                2
Total token throughput (tok/s):          2647.19
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1742.54
Median E2E Latency (ms):                 1412.47
---------------Time to First Token----------------
Mean TTFT (ms):                          53.48
Median TTFT (ms):                        45.05
P99 TTFT (ms):                           98.57
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.01
Median TPOT (ms):                        4.01
P99 TPOT (ms):                           4.03
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.01
Median ITL (ms):                         4.01
P95 ITL (ms):                            4.06
P99 ITL (ms):                            4.09
Max ITL (ms):                            4.95
==================================================
```

##### 5.1.3.2 Medium Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  16.90
Total input tokens:                      300020
Total input text tokens:                 300020
Total input vision tokens:               0
Total generated tokens:                  41669
Total generated tokens (retokenized):    41668
Request throughput (req/s):              4.73
Input token throughput (tok/s):          17753.58
Output token throughput (tok/s):         2465.75
Peak output token throughput (tok/s):    3005.00
Peak concurrent requests:                25
Total token throughput (tok/s):          20219.33
Concurrency:                             13.68
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2890.33
Median E2E Latency (ms):                 3069.55
---------------Time to First Token----------------
Mean TTFT (ms):                          41.46
Median TTFT (ms):                        31.75
P99 TTFT (ms):                           93.18
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.52
Median TPOT (ms):                        5.58
P99 TPOT (ms):                           6.14
---------------Inter-Token Latency----------------
Mean ITL (ms):                           5.48
Median ITL (ms):                         5.13
P95 ITL (ms):                            5.93
P99 ITL (ms):                            20.76
Max ITL (ms):                            36.01
==================================================
```

##### 5.1.3.3 High Concurrency

* **Benchmark Command**:

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model zai-org/Glyph \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  35.54
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total input vision tokens:               0
Total generated tokens:                  170000
Total generated tokens (retokenized):    169994
Request throughput (req/s):              9.01
Input token throughput (tok/s):          35848.57
Output token throughput (tok/s):         4783.96
Peak output token throughput (tok/s):    8396.00
Peak concurrent requests:                80
Total token throughput (tok/s):          40632.53
Concurrency:                             59.26
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6580.96
Median E2E Latency (ms):                 6248.74
---------------Time to First Token----------------
Mean TTFT (ms):                          345.27
Median TTFT (ms):                        96.06
P99 TTFT (ms):                           2823.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          12.26
Median TPOT (ms):                        12.53
P99 TPOT (ms):                           23.58
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.76
Median ITL (ms):                         6.57
P95 ITL (ms):                            27.66
P99 ITL (ms):                            91.24
Max ITL (ms):                            2609.64
==================================================
```

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* Test Result

```text Output theme={null}
Accuracy: 0.890
Invalid: 0.000
Latency: 3.718 s
Output throughput: 5245.606 token/s
```


# GLM-OCR
Source: https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-OCR



## 1. Model Introduction

[GLM-OCR](https://huggingface.co/zai-org/GLM-OCR) is a multimodal OCR model for complex document understanding, built on the GLM-V encoder–decoder architecture. It introduces Multi-Token Prediction (MTP) loss and stable full-task reinforcement learning to improve training efficiency, recognition accuracy, and generalization.

The model integrates the CogViT visual encoder pre-trained on large-scale image–text data, a lightweight cross-modal connector with efficient token downsampling, and a GLM-0.5B language decoder. Combined with a two-stage pipeline of layout analysis and parallel recognition based on PP-DocLayout-V3, GLM-OCR delivers robust and high-quality OCR performance across diverse document layouts.

**Hardware Support:** NVIDIA B200/H100/H200

**Key Features:**

* **State-of-the-Art Performance**: Achieves 94.62 on OmniDocBench V1.5, ranking #1, and delivers SOTA results across major document understanding benchmarks, including formula recognition, table recognition, and information extraction.
* **Optimized for Real-World Scenarios**: Specifically optimized for practical business cases, maintaining stable and accurate performance on complex tables, code documents, seals, and other challenging layouts.
* **Efficient Inference**: With only 0.9B parameters, GLM-OCR supports deployment via vLLM and SGLang, significantly reducing inference latency and compute cost—well suited for high-concurrency and edge deployments.
* **Easy to Use**: Fully open-sourced with a complete SDK and inference toolchain, enabling one-line invocation and seamless integration into existing systems.

For more details, please refer to the [official GLM-OCR model card](https://huggingface.co/zai-org/GLM-OCR).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options. You can optionally enable MTP (Multi-Token Prediction) for faster inference using EAGLE speculative decoding.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **CUDA IPC Transport**: The `SGLANG_USE_CUDA_IPC_TRANSPORT=1` environment variable enables CUDA IPC for transferring multimodal features, which significantly improves TTFT.
* **MTP (Multi-Token Prediction)**: Enable MTP to use EAGLE speculative decoding for faster inference. This feature predicts multiple tokens at once to reduce latency.
* **Memory Management**: For memory-constrained environments, you may need to adjust `--mem-fraction-static` and/or `--max-running-requests`.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 OCR Image Processing

GLM-OCR supports OCR tasks on various document types. Here's a basic example:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Please extract all text from this image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="zai-org/GLM-OCR",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 2.29s
Generated text: CINNAMON SUGAR
1 x 17,000 17,000

SUB TOTAL 17,000

GRAND TOTAL 17,000

CASH IDR 20,000

CHANGE DUE 3,000

```

#### 4.2.2 Complex Document Processing

GLM-OCR excels at processing complex documents including:

* **Tables**: Accurate extraction of tabular data with structure preservation
* **Formulas**: Mathematical formula recognition
* **Code Documents**: Source code extraction from screenshots
* **Seals and Stamps**: Recognition of seals and stamps in documents
* **Multi-layout Documents**: Mixed content with text, images, and tables

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

# Example: Processing a document with tables
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "YOUR_DOCUMENT_IMAGE_URL"
                }
            },
            {
                "type": "text",
                "text": "Please extract the table content from this document and format it as markdown."
            }
        ]
    }
]

response = client.chat.completions.create(
    model="zai-org/GLM-OCR",
    messages=messages,
    max_tokens=4096
)
print(response.choices[0].message.content)
```

## 5. Benchmark

### 5.1 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.1.1 OCRBench Benchmark

* Benchmark Command

```bash Command theme={null}
python3 -m lmms_eval \
  --model openai_compatible \
  --model_args "model_version=zai-org/GLM-OCR" \
  --tasks ocrbench \
  --batch_size 128 \
  --log_samples \
  --log_samples_suffix "openai_compatible" \
  --output_path ./logs
```

* Test Result

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Tasks</th>
      <th>Version</th>
      <th>Filter</th>
      <th>n-shot</th>
      <th>Metric</th>

      <th />

      <th>Value</th>
      <th>Stderr</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>ocrbench</td>
      <td>Yaml</td>
      <td>none</td>
      <td>0</td>
      <td>ocrbench\_accuracy</td>
      <td>↑</td>
      <td>0.806</td>
      <td>N/A</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 OmniDocBench V1.5

GLM-OCR achieves **94.62** on OmniDocBench V1.5, ranking #1 among all models, demonstrating state-of-the-art performance across major document understanding benchmarks.


# GigaChat 3.5
Source: https://docs.sglang.io/cookbook/autoregressive/GigaChat/GigaChat3.5

Deploy GigaChat 3.5 with SGLang — the 432B-A28B hybrid MLA + GDN Mixture-of-Experts instant model with MTP speculative decoding and tool calling, on H100 and H200 in FP8.

## Deployment

> **Install SGLang**
>
> For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
>
> **Python (pip / uv)**
>
> GigaChat 3.5 support is on `main` ([PR #29189](https://github.com/sgl-project/sglang/pull/29189), merged 2026-09-21) and not yet in a `pip` release — install from source:
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
> ```
>
> Then run the **Python** output of the command panel below.
>
> **Docker**
>
> > [!NOTE]
> >
> > No released `lmsysorg/sglang` tag carries GigaChat 3.5 yet. The nightly `lmsysorg/sglang:dev` is rebuilt from `main` and carries the model in builds after 2026-09-21, but that path has not been run end-to-end, so the command panel shows only the Python run mode. Run the Python command inside the nightly image; the Docker tab returns to the panel after a verified run.
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

This page covers the **instant** checkpoint (direct answers, no thinking). For the thinking checkpoint, see [GigaChat 3.5 Reasoning](/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning).

Pick your hardware and recipe to generate the launch command. Two serving strategies cover the operating points:

* **Low-Latency** — MTP speculative decoding on, Mamba state pool pinned. Fastest reply per user; pick for chat and agent loops.
* **High-Throughput** — MTP off. At saturation the draft + verify overhead outweighs the speedup, so this is the recipe for batch jobs and many concurrent users.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been run end-to-end; the Playground lets you turn on additional knobs — the tool-call parser, the MTP preset, parallelism — on top of whichever cell the Deploy panel is currently showing. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.

## 1. Model Introduction

**GigaChat 3.5** is a hybrid Mixture-of-Experts model from GigaChat — **432B** total parameters, **28B** active per token, **256K** context (YaRN, factor 8 over a 32K base), released under the **MIT** license. Its **40** layers interleave two attention types: **10** full-attention **MLA** layers (DeepSeek-style latent attention, every 4th layer) and **30** **GDN** linear-attention layers (Qwen3-Next gated delta-net, constant-size state). Only the MLA layers hold a KV cache; the linear layers keep their state in a Mamba-style pool. The MoE block routes to **8 of 256** experts plus **1** shared expert. The checkpoint ships **2** stacked **MTP** draft heads for speculative decoding, and tool calls are emitted in GigaChat's **GCML** format.

This is the **instant** release: it answers directly, without a thinking block. The separately trained **Reasoning** release always thinks first and ships 3 MTP heads — it has [its own page](/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning).

**Resources:** HuggingFace — [GigaChat3.5-432B-A28B](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B) (FP8, served here) · [GigaChat3.5-432B-A28B-bf16](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-bf16) (BF16, for fine-tuning; no single-node recipe).

## 2. Configuration Tips

**Two pools.** GDN layers keep per-request state in a Mamba-style pool, MLA layers use a paged KV pool; `--mamba-full-memory-ratio` (default 0.9) splits the post-weight budget. On 8×H100 at `--mem-fraction-static 0.8` that is \~51 GB of weights per GPU, 347 state slots and 542k KV tokens. The resolver forces `--mamba-radix-cache-strategy no_buffer` (3 slots per request, overlap scheduler off), so the unpinned server admits 115 concurrent requests.

**MTP needs a pinned pool.** Each running request holds extra state copies under speculative decoding, and the default ratio then admits only a few dozen. The Low-Latency recipe pins `--max-mamba-cache-size 240` with `--max-running-requests 80`; the KV pool drops to 185k tokens. Keep `--mem-fraction-static 0.8` on 80 GB — it is already tight in eager prefill at 16 concurrent 8k-token prompts.

**Capacity levers** (toggles in the [Playground](#playground)). `--mamba-ssm-dtype bfloat16` halves every state slot at no cost in accuracy or speed. `--kv-cache-dtype fp8_e4m3` doubles the KV pool but decodes \~2.3× slower on H100 and cannot be combined with MTP on 80 GB.

**MTP flags.** `--speculative-algorithm EAGLE` uses GigaChat's own NextN heads through the multi-layer EAGLE worker, selected automatically. This checkpoint has **2** heads: `--speculative-num-steps 2`, `--speculative-num-draft-tokens 3`.

**Parsers.** `--tool-call-parser gigachat35` for structured tool calls (§3.1). No reasoning parser, not even `auto`: the checkpoint never emits `</think>`, and the forced `gigachat35` splitter would move the whole answer into `reasoning_content`.

## 3. Advanced Usage

### 3.1 Tool Calling

Enable the `gigachat35` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured calls via `message.tool_calls`. GigaChat emits tool calls in its **GCML** format — a `<｜GCML｜tool_calls>` … `</｜GCML｜tool_calls>` block holding one `<｜GCML｜invoke name="…">` per call — inside the assistant turn; the parser reads them out and strips the block from `content`. `--tool-call-parser auto` also resolves to `gigachat35` for this checkpoint, since its chat template carries the GCML marker.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"city": {"type": "string", "description": "The city name"}},
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> resp = client.chat.completions.create(
>     model="ai-sage/GigaChat3.5-432B-A28B",
>     messages=[{"role": "user", "content": "What's the weather in Paris?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Content:", msg.content)
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Content:
> Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_aa689732d4884654a1f76762', function=Function(arguments='{"city": "Paris"}', name='get_weather'), type='function', index=0)]
> ```
>
> Appending a `tool` message with `{"city": "Paris", "temp_c": 9, "condition": "light rain"}` and re-requesting returns the final answer with no further tool call:
>
> ```text Output theme={null}
> Content: The current weather in Paris is light rain with a temperature of 9°C.
> Tool calls: None
> ```

> [!NOTE]
>
> GigaChat tends to call tools **sequentially** — one hop, then the next — so a multi-tool request may come back with a single call. Drive it with an agent loop that appends each `tool` result and re-requests until `finish_reason` is no longer `tool_calls`. Parallel calls (several invokes in one block) are supported by the format when the model chooses them.

### 3.2 MTP (Speculative Decoding)

GigaChat 3.5 ships its own NextN draft heads, so speculative decoding needs no separate draft model: `--speculative-algorithm EAGLE` runs the stacked heads through SGLang's multi-layer EAGLE worker, which is selected automatically for this architecture. The **Low-Latency** recipe has it on; in the [Playground](#playground) the **Speculative** card adds the same preset to any other cell.

```bash Flags theme={null}
--speculative-algorithm EAGLE \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--max-running-requests 80 \
--max-mamba-cache-size 240
```

* **Steps follow the checkpoint.** This checkpoint has 2 draft heads, so use `--speculative-num-steps 2` and `--speculative-num-draft-tokens 3` (steps + 1); the worker rejects other values at startup.
* **Pin the state pool.** Under MTP every running request holds extra GDN state copies for verification. Without `--max-mamba-cache-size` and `--max-running-requests` the resolver clamps concurrency to a few dozen requests; §2 explains what the pin costs in context. The panel shows an amber callout whenever a speculative command lacks `--max-running-requests`.
* **Where it pays off.** Low concurrency: accepted draft tokens skip target forwards, so single-request decode runs about 2× faster with the output distribution unchanged. At high concurrency the verify work and the smaller KV pool eat the gain, which is why the High-Throughput recipe leaves MTP off. Measured numbers are in the benchmark card above.


# GigaChat 3.5 Reasoning
Source: https://docs.sglang.io/cookbook/autoregressive/GigaChat/GigaChat3.5-Reasoning

Deploy GigaChat 3.5 Reasoning with SGLang — the 432B-A28B hybrid MLA + GDN Mixture-of-Experts thinking model with reasoning parsing, MTP speculative decoding and tool calling, on H100 and H200 in FP8.

## Deployment

> **Install SGLang**
>
> For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
>
> **Python (pip / uv)**
>
> GigaChat 3.5 support is on `main` ([PR #29189](https://github.com/sgl-project/sglang/pull/29189), merged 2026-09-21) and not yet in a `pip` release — install from source:
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
> ```
>
> Then run the **Python** output of the command panel below.
>
> **Docker**
>
> > [!NOTE]
> >
> > No released `lmsysorg/sglang` tag carries GigaChat 3.5 yet. The nightly `lmsysorg/sglang:dev` is rebuilt from `main` and carries the model in builds after 2026-09-21, but that path has not been run end-to-end, so the command panel shows only the Python run mode. Run the Python command inside the nightly image; the Docker tab returns to the panel after a verified run.
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

This page covers the **Reasoning** checkpoint (always thinks before answering). For the direct-answer checkpoint, see [GigaChat 3.5](/cookbook/autoregressive/GigaChat/GigaChat3.5).

Pick your hardware and recipe to generate the launch command. Two serving strategies cover the operating points:

* **Low-Latency** — MTP speculative decoding on, Mamba state pool pinned to 80 requests. Fastest reply per user: measured 2× decode speed at batch 1 and 1.5× at 16 concurrent 8k-token requests. Pick for chat and agent loops.
* **High-Throughput** — MTP off, pools left to the default ratio, so the KV pool is \~2.7× larger. Past a few dozen concurrent long-context requests the pinned recipe starts queueing prompts on KV space, and at saturation the draft + verify overhead no longer pays; this is the recipe for batch jobs and many concurrent users.

> [!NOTE]
>
> H100 numbers were measured on 8×H100 SXM at `b63f8416b3` (the PR #29189 merge on `main`) with `bench_serving`, random
> ISL 8192 / OSL 1024, `--random-range-ratio 1.0`, `--warmup-requests 4`, `--flush-cache`; latencies are
> P50. GSM8K was measured with the 5-shot chat harness `sglang.test.run_eval --eval-name gsm8k --api chat` (1314 scored questions,
> `max_tokens` 8192), so the model thinks before every answer exactly as in production. That harness was
> deprecated upstream after these runs; the **Reproduce** button emits its successor, `sgl-eval run gsm8k`
> (single-shot), which can land a point or two away from the numbers shown.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been run end-to-end; the Playground lets you turn on additional knobs — the reasoning and tool-call parsers, the MTP preset, parallelism — on top of whichever cell the Deploy panel is currently showing. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.

## 1. Model Introduction

**GigaChat 3.5 Reasoning** is the thinking release of GigaChat's hybrid Mixture-of-Experts model — **432B** total parameters, **28B** active per token, **256K** context (YaRN, factor 8 over a 32K base), released under the **MIT** license. It shares the base architecture with [GigaChat 3.5](/cookbook/autoregressive/GigaChat/GigaChat3.5): **40** layers interleaving **10** full-attention **MLA** layers (DeepSeek-style latent attention, every 4th layer) with **30** **GDN** linear-attention layers (Qwen3-Next gated delta-net, constant-size state), and an MoE block routing to **8 of 256** experts plus **1** shared expert. Only the MLA layers hold a KV cache; the linear layers keep their state in a Mamba-style pool.

What differs: the chat template opens every assistant turn with `<think>`, so the model always reasons before answering (there is no template switch to turn it off), and the checkpoint ships **3** stacked **MTP** draft heads instead of 2. Tool calls use the same **GCML** format. The model card reports the largest gains over the instant release in mathematics, code, instruction following and structured output.

**Recommended generation:** the model card suggests `temperature=0.6` with a generous `max_tokens` (thinking alone can run to thousands of tokens). Informational only — the sample code below relies on the checkpoint's `generation_config.json` defaults.

**Resources:** HuggingFace — [GigaChat3.5-432B-A28B-Reasoning](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-Reasoning) (FP8, served here) · [GigaChat3.5-432B-A28B-Reasoning-bf16](https://huggingface.co/ai-sage/GigaChat3.5-432B-A28B-Reasoning-bf16) (BF16, for fine-tuning; no single-node recipe).

## 2. Configuration Tips

**Two pools.** GDN layers keep per-request state in a Mamba-style pool, MLA layers use a paged KV pool; `--mamba-full-memory-ratio` (default 0.9) splits the post-weight budget. On 8×H100 at `--mem-fraction-static 0.8` that is \~51 GB of weights per GPU, 348 state slots and 542k KV tokens. The resolver forces `--mamba-radix-cache-strategy no_buffer` (3 slots per request, overlap scheduler off), so the unpinned server admits 116 concurrent requests.

**MTP needs a pinned pool and 0.83.** Each running request holds extra state copies under speculative decoding, and the default ratio then admits only a few dozen. The Low-Latency recipe pins `--max-mamba-cache-size 240` with `--max-running-requests 80` and runs at `--mem-fraction-static 0.83` (KV pool 199k tokens); at 0.85 the server ran out of memory in eager prefill at 16 concurrent 8k-token prompts. To size by workload instead of a pin: `r = (3 + draft_tokens) × 16 MB / (11.5 KB × L)` for an average request length `L`, and with MTP keep running 8k-token requests under \~20, since the EAGLE worker holds each prompt's hidden states outside both pools.

**Capacity levers** (toggles in the [Playground](#playground)). `--mamba-ssm-dtype bfloat16` halves every state slot at no cost in accuracy or accept length — on the Low-Latency recipe the KV pool grows from 199k to 496k tokens. `--kv-cache-dtype fp8_e4m3` doubles the KV pool but decodes \~2.3× slower on H100 and cannot be combined with MTP on 80 GB.

**MTP flags.** `--speculative-algorithm EAGLE` uses GigaChat's own NextN heads through the multi-layer EAGLE worker, selected automatically. This checkpoint has **3** heads: `--speculative-num-steps 3`, `--speculative-num-draft-tokens 4`.

**Parsers.** `--reasoning-parser gigachat35` splits thinking from the answer (§3.1), `--tool-call-parser gigachat35` gives structured tool calls (§3.2); `auto` resolves to `gigachat35` for both. Do not use the reasoning parser on the Instant checkpoint.

## 3. Advanced Usage

### 3.1 Reasoning

The chat template opens every assistant turn with `<think>`, and there is no template switch to turn thinking off. Enable the `gigachat35` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to split the thinking from the final answer into `reasoning_content` vs `content`. Without the parser, the raw think block is returned inline in `content`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> resp = client.chat.completions.create(
>     model="ai-sage/GigaChat3.5-432B-A28B-Reasoning",
>     messages=[{"role": "user", "content": "What is 17 times 24?"}],
>     max_tokens=4096,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: I need to calculate 17 times 24. I can do this by breaking it down into smaller parts. First, I know that 10 times 24 is 240, and 7 times 24 is... let me figure that out. 7 times 20 is 140, and 7 times 4 is 28, so 140 plus 28 is 168. Then, I add the two results: 240 plus 168.
>
> 240 plus 168: 240 plus 100 is 340, plus 60 is 400, plus 8 is 408. Wait, that doesn't seem right. Let me add them properly. 240 + 168: 0 + 8 is 8, 4 + 6 is 10, carry over the 1, so 2 + 1 is 3, plus the carried over 1 is 4, so it should be 408. But I think I remember that 17 times 24 is 408, but let me double-check.
> [... ~1,400 more thinking tokens ...]
> Answer:
>
> 17 multiplied by 24 equals 408. This can be calculated in several ways:
>
> ### Method 1: Breaking Down the Numbers
> - $17 \times 24 = 17 \times (20 + 4)$
> - First, $17 \times 20 = 340$
> - Then, $17 \times 4 = 68$
> - Add the results: $340 + 68 = 408$
>
> ### Method 2: Using the Distributive Property
> - $17 \times 24 = (10 + 7) \times 24$
> - $10 \times 24 = 240$
> - $7 \times 24 = 168$
> - Add the results: $240 + 168 = 408$
> [...]
> ```
>
> The full response was 1,641 completion tokens, 24.6 s at batch size 1 on 8×H100 without MTP.

> [!NOTE]
>
> The model can spend hundreds of tokens thinking, so keep `max_tokens` generous. A response with an empty `content` and `finish_reason: "length"` means the think block was cut off before it closed — raise the limit.

### 3.2 Tool Calling

Enable the `gigachat35` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured calls via `message.tool_calls`. GigaChat emits tool calls in its **GCML** format — a `<｜GCML｜tool_calls>` … `</｜GCML｜tool_calls>` block holding one `<｜GCML｜invoke name="…">` per call — inside the assistant turn; the parser reads them out and strips the block from `content`. `--tool-call-parser auto` also resolves to `gigachat35` for this checkpoint, since its chat template carries the GCML marker.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"city": {"type": "string", "description": "The city name"}},
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> resp = client.chat.completions.create(
>     model="ai-sage/GigaChat3.5-432B-A28B-Reasoning",
>     messages=[{"role": "user", "content": "What's the weather in Paris?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: Okay, the user is asking about the weather in Paris. Let me check what tools I have available. There's the get_weather function which requires the city name. Since the user provided "Paris" directly, I can use that function. I just need to make sure to format the tool call correctly with the city parameter. No other information is needed here, so I'll call get_weather with city set to Paris.
> Content:
> Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_ef4e35b959c648fbaeab9afd', function=Function(arguments='{"city": "Paris"}', name='get_weather'), type='function', index=0)]
> ```
>
> Appending a `tool` message with `{"city": "Paris", "temp_c": 9, "condition": "light rain"}` and re-requesting returns the final answer with no further tool call:
>
> ```text Output theme={null}
> Reasoning:
> Content: The current weather in Paris is **9°C** with **light rain**.
> Tool calls: None
> ```

> [!NOTE]
>
> The tool-call turn carries the model's thinking in `reasoning_content`, so print it alongside `content` and `tool_calls`. GigaChat tends to call tools **sequentially** — one hop, then the next — so a multi-tool request may come back with a single call. Drive it with an agent loop that appends each `tool` result and re-requests until `finish_reason` is no longer `tool_calls`. Parallel calls (several invokes in one block) are supported by the format when the model chooses them.

### 3.3 MTP (Speculative Decoding)

GigaChat 3.5 ships its own NextN draft heads, so speculative decoding needs no separate draft model: `--speculative-algorithm EAGLE` runs the stacked heads through SGLang's multi-layer EAGLE worker, which is selected automatically for this architecture. The **Low-Latency** recipe has it on; in the [Playground](#playground) the **Speculative** card adds the same preset to any other cell.

```bash Flags theme={null}
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--max-running-requests 80 \
--max-mamba-cache-size 240
```

* **Steps follow the checkpoint.** This checkpoint has 3 draft heads, so use `--speculative-num-steps 3` and `--speculative-num-draft-tokens 4` (steps + 1); the worker rejects other values at startup.
* **Pin the state pool.** Under MTP every running request holds extra GDN state copies for verification. Without `--max-mamba-cache-size` and `--max-running-requests` the resolver clamps concurrency to a few dozen requests; §2 explains what the pin costs in context and why the recipe runs at `--mem-fraction-static 0.83`. The panel shows an amber callout whenever a speculative command lacks `--max-running-requests`.
* **Where it pays off.** Long think blocks are long, low-entropy decode, exactly what draft heads predict well: on 8×H100 single-request decode runs about 2× faster and 16 concurrent 8k-token requests about 1.5× faster, with GSM8K unchanged. At 64 concurrent requests the verify work and the smaller KV pool make it slower than the plain recipe, which is why the High-Throughput recipe leaves MTP off. Measured numbers are in the benchmark card above.


# DiffusionGemma
Source: https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma



## 1. Model Introduction

DiffusionGemma is a uniform-state (renoising) block-diffusion language model from Google. An encoder builds causal context, and a decoder denoises a fixed-length bidirectional canvas of `canvas_length` tokens. The `Gemma4Renoise` sampler runs `max_denoising_steps` reverse steps over the canvas, feeding the previous step's logits back as self-conditioning and emitting the greedy argmax of the processed logits.

**Key Features:**

* **Uniform-State Renoising**: The canvas starts from random tokens and is refined each step by accepting confident positions and re-noising the rest, with no mask token.
* **Encoder / Decoder Canvas**: The encoder produces causal context KV, the decoder attends bidirectionally over the canvas.
* **Self-Conditioning**: Each step conditions on the previous step's logits.
* **EntropyBound Acceptance**: Each step accepts the lowest-entropy canvas positions within an entropy budget and re-noises the rest.
* **StableAndConfident Stopping**: A canvas stops early once it is stable and confident.
* **MoE Architecture**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference.
* **Multimodal Input**: Accepts text and image inputs (via a \~550M vision encoder) and generates text output.

**Available Models:**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Architecture</th>
      <th>Parameters</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[google/diffusiongemma-26B-A4B-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it)</td>
      <td>MoE, uniform-state diffusion (text + image)</td>
      <td>25.2B total / 3.8B active</td>
    </tr>
  </tbody>
</table>

**Architecture Specifications:**

| Spec | Value |
| - | - |
| Total Parameters | 25.2B |
| Active Parameters | 3.8B |
| Layers | 30 |
| Sliding Window | 1024 tokens |
| Context Length | Up to 256K tokens |
| Canvas Length | 256 |
| Vocabulary Size | 262K |
| Experts | 8 active / 128 total + 1 shared |
| Supported Modalities | Text, Image |
| Vision Encoder | \~550M parameters |

**License:**

Refer to the model card for license details.

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

The checkpoint ships its own modeling code, so `--trust-remote-code` is required when serving.

## 3. Model Deployment

### 3.1 Basic Configuration

The required runtime settings are applied automatically for `Gemma4Renoise` (the Triton attention backend, eager mode, and unchunked prefill, needed because the full-attention head\_dim is 512 and the canvas uses bidirectional attention), so a default launch works:

```bash Command theme={null}
sglang serve \
  --model-path google/diffusiongemma-26B-A4B-it \
  --dllm-algorithm Gemma4Renoise \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

### 3.2 Configuration Tips

**dLLM-Specific Parameters:**

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Recommended Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dllm-algorithm`</td>
      <td>Diffusion decoding algorithm</td>
      <td>`Gemma4Renoise`</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>Required to load the checkpoint's modeling code</td>
      <td>Always enabled</td>
    </tr>

    <tr>
      <td>`--dllm-algorithm-config`</td>
      <td>Optional YAML overriding the renoise schedule</td>
      <td>Checkpoint defaults</td>
    </tr>
  </tbody>
</table>

The attention backend, eager mode, and unchunked prefill are selected automatically for `Gemma4Renoise`, so they do not need to be passed on the command line.

Sampling is governed by the renoise schedule. Request-level `logprobs`, penalties, `logit_bias`, and grammar / structured output (`json_schema` / `regex` / `ebnf` / `structural_tag`) are not applied and are rejected with a 400. Core sampling controls (`temperature`, `top_k`, `top_p`) are accepted but have no effect. Streaming is block-level: one fully-denoised canvas per chunk.

**Gemma4Renoise Config** (defaults follow the checkpoint's `generation_config.json`):

```yaml Config theme={null}
# Number of reverse denoising steps per canvas.
max_denoising_steps: 48
# Optional. Makes the renoise sampling reproducible (also shared across TP ranks).
seed: 1234
sampler_config:
  # Entropy budget. Accept the lowest-entropy canvas positions within this bound each step (the rest are re-noised).
  entropy_bound: 0.1
# Linear temperature schedule applied over the denoising steps.
temperature_schedule:
  t_min: 0.4
  t_max: 0.8
# Stop early once the canvas is stable and confident.
stopping_config:
  confidence_threshold: 0.005
  stability_threshold: 1
```

## 4. Model Invocation

### 4.1 Deployment

Start the server with the command from [Section 3.1](#3-1-basic-configuration).

### 4.2 Basic Usage

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="google/diffusiongemma-26B-A4B-it",
    messages=[
        {"role": "user", "content": "What are the key differences between TCP and UDP?"}
    ],
    max_tokens=1024
)

print(response.choices[0].message.content)
```

### 4.3 Streaming

Streaming emits one fully-denoised canvas per chunk.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="google/diffusiongemma-26B-A4B-it",
    messages=[
        {"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
    ],
    max_tokens=2048,
    stream=True
)

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

## 5. Benchmark

### 5.1 Speed Benchmark

Not benchmarked for speed.

### 5.2 Accuracy Benchmark

Full test splits, every item scored (no failed-request exclusions). Text MCQ benchmarks use greedy generate-and-parse, MATH uses boxed-answer extraction plus sympy equivalence. MMLU, ARC-Challenge, and MATH-500 are the mean of two independent server launches.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Benchmark</th>
      <th>Score</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>GSM8K</td>
      <td>95.4%</td>
    </tr>

    <tr>
      <td>ARC-Challenge</td>
      <td>91.6%</td>
    </tr>

    <tr>
      <td>HumanEval</td>
      <td>92.7% pass\@1</td>
    </tr>

    <tr>
      <td>MMLU</td>
      <td>76.2%</td>
    </tr>

    <tr>
      <td>MMLU-Pro</td>
      <td>73.7%</td>
    </tr>

    <tr>
      <td>GSM-Symbolic</td>
      <td>92.2%</td>
    </tr>

    <tr>
      <td>MATH-500</td>
      <td>72.1%</td>
    </tr>

    <tr>
      <td>AIME-2026</td>
      <td>10.0%</td>
    </tr>

    <tr>
      <td>HMMT-Feb-2025</td>
      <td>10.0%</td>
    </tr>

    <tr>
      <td>GPQA-main</td>
      <td>59.2%</td>
    </tr>
  </tbody>
</table>

Multimodal, full standard split per task (MMMU / MMMU-Pro / MMStar / AI2D as multiple-choice, MathVista testmini, DocVQA by ANLS, ChartQA by relaxed accuracy):

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Multimodal benchmark</th>
      <th>Score</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>MMMU (val, MC)</td>
      <td>64.9%</td>
    </tr>

    <tr>
      <td>MMMU-Pro (standard 10-opt, MC)</td>
      <td>57.3%</td>
    </tr>

    <tr>
      <td>MathVista (testmini)</td>
      <td>68.4%</td>
    </tr>

    <tr>
      <td>DocVQA (val)</td>
      <td>85.9%</td>
    </tr>

    <tr>
      <td>ChartQA (test)</td>
      <td>61.7%</td>
    </tr>

    <tr>
      <td>AI2D (test)</td>
      <td>78.7%</td>
    </tr>

    <tr>
      <td>MMStar (val)</td>
      <td>65.9%</td>
    </tr>
  </tbody>
</table>


# EmbeddingGemma
Source: https://docs.sglang.io/cookbook/autoregressive/Google/EmbeddingGemma

Serve Google's EmbeddingGemma text embedding model with SGLang.

## Overview

[EmbeddingGemma](https://huggingface.co/google/embeddinggemma-300m) is Google's 300M-parameter text embedding model. SGLang detects its bidirectional Gemma 3 encoder, applies normalized mean pooling, and serves embeddings through the OpenAI-compatible `/v1/embeddings` endpoint.

On NVIDIA CUDA, SGLang uses breakable CUDA graph (BCG) for its complete prefill by default. It also disables prefix caching and chunked prefill, which are incompatible with this bidirectional encoder.

## Prerequisites

* NVIDIA CUDA GPU.
* A Hugging Face account that has accepted the [EmbeddingGemma license](https://huggingface.co/google/embeddinggemma-300m).
* A Hugging Face access token. Export it before starting the server so it can download the gated checkpoint:

```bash theme={null}
export HF_TOKEN=<your-hugging-face-token>
```

Install an SGLang build that includes EmbeddingGemma support:

```bash theme={null}
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
```

## Start the server

The standard configuration detects EmbeddingGemma and enables embedding mode,
BCG, and the checkpoint's BF16 dtype automatically:

```bash theme={null}
sglang serve \
  --model-path google/embeddinggemma-300m \
  --host 0.0.0.0
```

### Hopper performance defaults

On H100 and H200, SGLang automatically selects FA3 and captures BCG through
16,384 tokens, covering eight 2K embedding requests in one replay. No extra
performance flags are required for this workload.

To capture larger aggregate prefills, raise the BCG tier explicitly:

```bash theme={null}
sglang serve \
  --model-path google/embeddinggemma-300m \
  --cuda-graph-max-bs-prefill 32768 \
  --host 0.0.0.0
```

EmbeddingGemma automatically enables batch tokenization for list-valued
embedding requests, so do not add a separate tokenizer batching flag.

## Create embeddings

Send one string or a batch of strings to the OpenAI-compatible endpoint:

```bash theme={null}
curl http://127.0.0.1:30000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "google/embeddinggemma-300m",
    "input": [
      "A short guide to serving text embeddings.",
      "Vector search retrieves semantically similar documents."
    ],
    "encoding_format": "float"
  }'
```

See [OpenAI-compatible embedding APIs](/docs/basic_usage/openai_api_embeddings) for Python and OpenAI client examples.

## Deployment behavior

EmbeddingGemma performs bidirectional attention over the complete input, so reusing a prefix KV cache or splitting the input into chunked prefills would produce incorrect attention states. SGLang applies the required settings automatically:

* disables RadixAttention prefix caching;
* disables chunked prefill;
* disables the decode CUDA graph because this is an embedding-only model;
* uses BCG for CUDA prefill;
* uses the FlashAttention raw-K/V path when the prefill backend is FA3 or FA4 on supported Hopper and Blackwell CUDA GPUs.

No prefill CUDA-graph override is required for this recipe. Keep BCG enabled to use the optimized EmbeddingGemma path.


# EmbeddingGemma 2
Source: https://docs.sglang.io/cookbook/autoregressive/Google/EmbeddingGemma2

Serve Google's multimodal EmbeddingGemma 2 model (text, image, video, audio) with SGLang.

## Overview

[EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) is Google's multimodal embedding model. It maps text (including code), images, video, audio, and interleaved combinations of them into one shared, L2-normalized 768-dimensional vector space.

| Property | Value |
| :- | :- |
| Parameters | 740M total: 270M text backbone, 170M vision encoder, 300M audio encoder |
| Architecture | 24-layer bidirectional Gemma 4 encoder, 5:1 local:global attention, local window radius 512 |
| Context | 8,192 tokens shared by all modalities in a request |
| Output | 768 dimensions; Matryoshka truncation to 512, 256, or 128 |
| Pooling | Mean pooling over all tokens, then L2 normalization |

SGLang detects the `EmbeddingGemma2Model` architecture and configures it automatically:

* enables embedding mode, so `--is-embedding` is not needed;
* selects the Triton attention backend, which implements the bidirectional sliding window;
* disables RadixAttention prefix caching and chunked prefill, because bidirectional attention makes every token depend on the whole input;
* caps the context length at 8,192 tokens.

## Prerequisites

* An NVIDIA CUDA GPU. The model weighs about 1.5 GB in BF16, so any data-center GPU works.
* A Transformers build that includes EmbeddingGemma 2 (`model_type: embedding_gemma2`). Older releases fail at startup with `Transformers does not recognize this architecture`.
* An SGLang build that includes EmbeddingGemma 2 support:

```bash theme={null}
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
```

Run the model in BF16 (the checkpoint default) or FP32. Do not use FP16: the model's activations exceed its range and produce NaN or degraded embeddings.

## Start the server

```bash theme={null}
sglang serve \
  --model-path google/embeddinggemma-2 \
  --host 0.0.0.0
```

### Load only the encoders you need

The vision and audio encoders load only when their modality is allowed. Set a modality's per-request limit to `0` to skip loading its encoder. Images and video share the vision encoder.

| Serving | `--limit-mm-data-per-request` | Parameters loaded |
| :- | :- | :- |
| Text only | `'{"image": 0, "video": 0, "audio": 0}'` | 270M |
| Text and images/video | `'{"audio": 0}'` | 440M |
| Text and audio | `'{"image": 0, "video": 0}'` | 570M |
| All modalities | (default) | 740M |

```bash theme={null}
sglang serve \
  --model-path google/embeddinggemma-2 \
  --limit-mm-data-per-request '{"image": 0, "video": 0, "audio": 0}' \
  --host 0.0.0.0
```

A request for a disabled modality fails with HTTP 400, for example `Image count 1 exceeds limit 0 per request.`

### Enable Matryoshka dimensions

To let clients request shorter vectors through the `dimensions` field, declare the supported sizes when starting the server:

```bash theme={null}
sglang serve \
  --model-path google/embeddinggemma-2 \
  --json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 768]}' \
  --host 0.0.0.0
```

SGLang truncates the pooled vector and then re-normalizes it, as the model card requires. Other sizes are rejected with HTTP 400. Queries and documents must use the same dimension.

## Task prompts

EmbeddingGemma 2 is trained with short task prefixes on text inputs. Prepend them yourself; images, video, and audio take no prefix. For documents without a title, use `title: none`.

| Use case | Query | Document |
| :- | :- | :- |
| Retrieval | `task: search result \| query: {content}` | `title: {title} \| text: {content}` |
| Question answering | `task: question answering \| query: {content}` | `title: {title} \| text: {content}` |
| Fact checking | `task: fact checking \| query: {content}` | `title: {title} \| text: {content}` |
| Code retrieval | `task: code retrieval \| query: {content}` | `title: {title} \| text: {content}` |
| Classification | `task: classification \| query: {content}` | |
| Clustering | `task: clustering \| query: {content}` | |
| Semantic similarity | `task: sentence similarity \| query: {content}` | |

## Create embeddings

### Native `/encode` API

`/encode` supports every modality, including several media items in one embedding.

Text, one embedding per string:

```bash theme={null}
curl http://127.0.0.1:30000/encode \
  -H 'Content-Type: application/json' \
  -d '{
    "text": [
      "task: search result | query: What causes the aurora borealis?",
      "title: none | text: The aurora is caused by solar wind particles colliding with the upper atmosphere."
    ]
  }'
```

An image, video, or audio clip on its own. SGLang inserts the placeholder for you:

```bash theme={null}
curl http://127.0.0.1:30000/encode \
  -H 'Content-Type: application/json' \
  -d '{"image_data": "https://example.com/cat.jpg"}'
```

Use `video_data` or `audio_data` the same way. Audio should be mono at 16 kHz.

Interleaved input, one embedding for the whole sequence. Mark each item's position with `<|image|>`, `<|video|>`, or `<|audio|>`, and pass the items in the same order:

```bash theme={null}
curl http://127.0.0.1:30000/encode \
  -H 'Content-Type: application/json' \
  -d '{
    "text": "Waterproof running shoes. <|image|> Breathable mesh upper. <|image|> Narration: <|audio|>",
    "image_data": ["https://example.com/shoe.jpg", "https://example.com/mesh.jpg"],
    "audio_data": "https://example.com/narration.wav"
  }'
```

#### How requests map to embeddings

| Request shape | Result |
| :- | :- |
| `text` is a string, media is a list | 1 embedding containing all the media |
| `text` is a list of N strings, media is a list of N entries | N embeddings, paired by position; an entry may be `null` or a list of several items |
| `text` is a list, media is a single item (not a list) | the item is attached to every request, so every string needs its placeholder |
| No `text`, media is a flat list `[a, b]` | 1 embedding of `a` and `b` together |
| No `text`, media is a list of lists `[[a], [b]]` | 2 embeddings |

Requests are rejected with HTTP 400 instead of being silently changed when:

* the number of `<|image|>`, `<|video|>`, or `<|audio|>` placeholders differs from the number of items of that modality (text without any placeholder plus an image is also a mismatch);
* a batched media list has a different length than the `text` list.

### OpenAI-compatible `/v1/embeddings`

```bash theme={null}
curl http://127.0.0.1:30000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "google/embeddinggemma-2",
    "input": [
      {"text": "a product photo: <|image|>", "image": "https://example.com/shoe.jpg"},
      {"image": "https://example.com/mesh.jpg"},
      {"text": "task: search result | query: waterproof trail shoes"}
    ]
  }'
```

Each input item produces one embedding. In this API, an item carries at most one `image` or `video` and no audio. For several media items in one embedding, or for audio, use `/encode`.

## Input limits

All modalities share the 8,192-token context. Requests that exceed it fail with HTTP 400 (`The input (N tokens) is longer than the model's context length (8192 tokens).`).

| Modality | Cost | Max, single modality |
| :- | :- | :- |
| Text | 1 token per subword | 8,192 tokens |
| Image | up to 280 tokens per image | about 29 images |
| Video | up to 140 tokens per sampled frame | about 58 frames |
| Audio | 25 tokens per second | about 327 seconds |

Video is sampled at 1 frame per second and capped at 32 frames by default, spread uniformly over longer clips, so a video of any length fits by default. A 2-minute 720p clip, for example, uses 3,906 tokens. Only the sampled frames are decoded.

To change the sampling rate or frame cap for every request, use `--mm-process-config`:

```bash theme={null}
sglang serve \
  --model-path google/embeddinggemma-2 \
  --mm-process-config '{"video": {"fps": 2, "max_frames": 48}}' \
  --host 0.0.0.0
```

The number of frames that fit depends on resolution. At 720p, 64 frames fit; 70 frames exceed the context.


# Gemma 4
Source: https://docs.sglang.io/cookbook/autoregressive/Google/Gemma4



## 1. Model Introduction

Gemma 4 is Google's next-generation family of open models, building on the Gemma 3 architecture with improved performance, MoE variants, and multimodal support for text, vision, and audio.

**Key Features:**

* **Hybrid Attention**: Combines sliding window and full attention layers for efficient long-context processing
* **Multimodal**: Supports text, image, and audio inputs via dedicated vision and audio encoders
* **MoE Variant**: The 26B-A4B model uses a Mixture-of-Experts architecture for efficient inference
* **Per-Layer Embeddings (PLE)**: Layer-specific token embeddings for enhanced representations
* **Reasoning**: Built-in thinking mode with `gemma4` reasoning parser
* **Tool Calling**: Function call support with streaming via `gemma4` tool call parser
* **Fused Operations**: Triton-optimized RMSNorm + residual + scalar kernels

**Available Models:**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Architecture</th>
      <th>Parameters</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it)</td>
      <td>Dense</td>
      <td>\~2B</td>
    </tr>

    <tr>
      <td>[google/gemma-4-E4B-it](https://huggingface.co/google/gemma-4-E4B-it)</td>
      <td>Dense</td>
      <td>\~4B</td>
    </tr>

    <tr>
      <td>[google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it)</td>
      <td>Dense</td>
      <td>12B</td>
    </tr>

    <tr>
      <td>[google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it)</td>
      <td>Dense</td>
      <td>31B</td>
    </tr>

    <tr>
      <td>[google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it)</td>
      <td>MoE</td>
      <td>26B total / 4B active</td>
    </tr>
  </tbody>
</table>

## 2. SGLang Installation

Gemma 4 (including the encoder-free unified 12B, [sgl-project/sglang#27167](https://github.com/sgl-project/sglang/pull/27167)) is supported on SGLang main. Install it together with the matching transformers commit:

```bash Command theme={null}
# Install SGLang from main
pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'

# Install transformers with Gemma 4 support (encoder-free unified family included)
pip install 'git+https://github.com/huggingface/transformers.git@1423d22f7a3b62e8c70ad67b58ec25cd9b675897'
```

### Docker

`lmsysorg/sglang:latest` (CUDA 13.0, multi-arch `amd64` + `arm64`) runs on both Hopper (H200) and Blackwell (B200 / GB200 / GB300):

```bash Command theme={null}
docker run --gpus all --ipc=host --shm-size 32g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 30000:30000 \
  lmsysorg/sglang:latest \
  sglang serve --model-path google/gemma-4-12B-it \
    --reasoning-parser gemma4 --tool-call-parser gemma4 \
    --host 0.0.0.0 --port 30000
```

For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

### 3.1 Basic Configuration

The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), Xeon CPUs. The recommended launch configurations vary by hardware and model size.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant.

Run this Command:

<pre />

### 3.2 Configuration Tips

* SGLang automatically selects the Triton attention backend for Gemma 4 models (required for bidirectional image-token attention during prefill).
* **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for text but applies *causal* attention to image tokens. For multimodal (image) workloads on B200, pass `--attention-backend triton` to restore bidirectional image-token attention and full vision quality. Text-only and audio workloads are unaffected by the default.
* **Gemma 4 26B-A4B on B200**: Use `--mem-fraction-static 0.75` to leave workspace headroom for the Triton MoE path.
* For the 26B-A4B MoE model, consider `--tp 2` for high-throughput workloads.
* **Speculative Decoding (MTP)**: Each Gemma 4 variant ships with a paired `*-assistant` draft model that enables NEXTN multi-token prediction. Enable it via the selector above, or pass `--speculative-algorithm NEXTN --speculative-draft-model-path google/gemma-4-<variant>-it-assistant --speculative-num-steps 5 --speculative-num-draft-tokens 6 --speculative-eagle-topk 1`. MTP can significantly reduce latency for interactive use cases. The 26B-A4B MoE model requires `--tp 2` when MTP is enabled.
* **QAT checkpoints**: Toggle **Checkpoint → QAT** in the selector to target the `qat-q4_0-unquantized` releases. These keep bf16 weights, so memory and TP requirements match the standard checkpoints, and each has a matching `*-qat-q4_0-unquantized-assistant` draft model for MTP.
* Hardware requirements:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Hardware</th>
      <th>TP</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>gemma-4-E2B-it</td>
      <td>1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X</td>
      <td>1</td>
    </tr>

    <tr>
      <td>gemma-4-E4B-it</td>
      <td>1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X</td>
      <td>1</td>
    </tr>

    <tr>
      <td>gemma-4-12B-it</td>
      <td>1x H200 / 1x B200 / 1x B300</td>
      <td>1</td>
    </tr>

    <tr>
      <td>gemma-4-31B-it</td>
      <td>2x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X</td>
      <td>2 (H200) / 1 (B200/B300/AMD)</td>
    </tr>

    <tr>
      <td>gemma-4-26B-A4B-it</td>
      <td>1x H200 / 1x B200 / 1x B300 / 1x MI300X / 1x MI325X / 1x MI355X</td>
      <td>1</td>
    </tr>
  </tbody>
</table>

### 3.3 AMD GPU Deployment (MI300X / MI325X / MI355X)

SGLang automatically selects the correct attention backend on AMD GPUs. For the small E-models (`gemma-4-E2B-it`, `gemma-4-E4B-it`), disable AITER on AMD GPUs and use the same command line otherwise:

```bash Command theme={null}
SGLANG_USE_AITER=0 sglang serve --model-path google/gemma-4-E4B-it \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --host 0.0.0.0 --port 30000
```

For `gemma-4-31B-it` and `gemma-4-26B-A4B-it`, the same commands above work on MI300X, MI325X, and MI355X without additional command-line changes.

> **Status**: AMD benchmarks are available in [Section 5.1](#5-1-speed-benchmark).

## 4. Model Invocation

Deploy gemma-4-26B-A4B-it (MoE) with all features enabled:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-26B-A4B-it \
  --reasoning-parser gemma4 \
  --tool-call-parser gemma4 \
  --host 0.0.0.0 --port 30000
```

#### Speculative Decoding (MTP) Server Commands

Each Gemma 4 variant ships with a paired `*-assistant` draft model for NEXTN multi-token prediction. Use the commands below to enable MTP for the corresponding target model. These match the configuration generated when you toggle **Speculative Decoding (MTP) → Enabled** in the [interactive selector](#3-1-basic-configuration).

```bash Command theme={null}
# Gemma 4 E2B + MTP
sglang serve \
  --model-path google/gemma-4-E2B-it \
  --speculative-algorithm NEXTN \
  --speculative-draft-model-path google/gemma-4-E2B-it-assistant \
  --speculative-num-steps 5 \
  --speculative-num-draft-tokens 6 \
  --speculative-eagle-topk 1 \
  --mem-fraction-static 0.85
```

```bash Command theme={null}
# Gemma 4 E4B + MTP
sglang serve \
  --model-path google/gemma-4-E4B-it \
  --speculative-algorithm NEXTN \
  --speculative-draft-model-path google/gemma-4-E4B-it-assistant \
  --speculative-num-steps 5 \
  --speculative-num-draft-tokens 6 \
  --speculative-eagle-topk 1 \
  --mem-fraction-static 0.85
```

```bash Command theme={null}
# Gemma 4 12B + MTP (~35% faster single-stream decode on H200)
sglang serve \
  --model-path google/gemma-4-12B-it \
  --speculative-algorithm NEXTN \
  --speculative-draft-model-path google/gemma-4-12B-it-assistant \
  --speculative-num-steps 5 \
  --speculative-num-draft-tokens 6 \
  --speculative-eagle-topk 1 \
  --mem-fraction-static 0.85
```

```bash Command theme={null}
# Gemma 4 31B + MTP
sglang serve \
  --model-path google/gemma-4-31B-it \
  --tp-size 2 \
  --speculative-algorithm NEXTN \
  --speculative-draft-model-path google/gemma-4-31B-it-assistant \
  --speculative-num-steps 5 \
  --speculative-num-draft-tokens 6 \
  --speculative-eagle-topk 1 \
  --mem-fraction-static 0.85
```

```bash Command theme={null}
# Gemma 4 26B-A4B + MTP
sglang serve \
  --model-path google/gemma-4-26B-A4B-it \
  --tp-size 2 \
  --speculative-algorithm NEXTN \
  --speculative-draft-model-path google/gemma-4-26B-A4B-it-assistant \
  --speculative-num-steps 5 \
  --speculative-num-draft-tokens 6 \
  --speculative-eagle-topk 1 \
  --mem-fraction-static 0.85
```

### 4.1 Basic Usage

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[
        {"role": "user", "content": "What are the key differences between TCP and UDP?"}
    ],
    max_tokens=1024
)

print(response.choices[0].message.content)
```

<details>
  <summary>Example Output</summary>

  ```text Output theme={null}
  The fundamental difference between **TCP (Transmission Control Protocol)** and **UDP (User Datagram
  Protocol)** lies in how they prioritize data integrity versus speed.

  ### 1. Connection Type
  *   **TCP (Connection-Oriented):** Before any data is sent, TCP performs a "three-way handshake."
      The sender and receiver exchange signals to establish a formal connection.
  *   **UDP (Connectionless):** UDP does not establish a connection. It simply starts blasting packets
      to the destination IP address without checking if the receiver is ready.

  ### 2. Reliability and Error Checking
  *   **TCP (Reliable):** If a packet is lost or arrives corrupted, TCP detects the error and
      retransmits the missing data.
  *   **UDP (Unreliable):** If a packet is lost or corrupted, it is simply discarded. There is no
      mechanism to ask for a retransmission.

  ### 3. Ordering of Data
  *   **TCP (Ordered):** Segments are assigned sequence numbers and reassembled in the correct order.
  *   **UDP (Unordered):** Packets may arrive in a different order than sent.

  ### 4. Speed and Overhead
  *   **TCP (Slower):** Managing connections, tracking, and retransmissions adds significant overhead.
  *   **UDP (Faster):** No handshake, no tracking — extremely fast and ideal for real-time needs.

  | Feature | TCP | UDP |
  | :--- | :--- | :--- |
  | **Connection** | Connection-oriented | Connectionless |
  | **Reliability** | Guaranteed delivery | Best-effort |
  | **Ordering** | Maintains strict order | No guaranteed order |
  | **Speed** | Slower (High overhead) | Faster (Low overhead) |
  ```
</details>

### 4.2 Vision Input

Gemma 4 multimodal variants accept images alongside text:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://farm4.staticflickr.com/3175/2653711032_804ff86d81_z.jpg"
                    }
                },
                {
                    "type": "text",
                    "text": "Describe this image in detail."
                }
            ]
        }
    ],
    max_tokens=1024
)

print(response.choices[0].message.content)
```

<details>
  <summary>Example Output</summary>

  ```text Output theme={null}
  A vertical, full shot shows a girl and a boy standing in front of a giant teddy bear. The boy, who
  is on the left, is of South Asian descent, has short dark hair, and is smiling at the camera. He is
  wearing a navy blue sweatshirt with a white collar, blue jeans, and white, black, and red sneakers.
  The girl, on the right, is also of South Asian descent and has long, dark hair. She is smiling at
  the camera and is wearing a pink t-shirt, a white long-sleeve shirt underneath, blue jeans, and pink
  sneakers. The giant teddy bear is light brown and is standing behind the two children. The bear has
  large, dark eyes and a black nose. In the background, on the left, there is a large wooden basket
  filled with small teddy bears. To the left of the basket, an American flag is hanging on the wall.
  On the right side of the image, there is a green leafy plant. The floor is a dark purple carpet. The
  lighting is bright and even.
  ```
</details>

### 4.3 Reasoning (Thinking Mode)

Gemma 4 supports hybrid reasoning. Thinking is **not enabled by default** — pass `chat_template_kwargs: {"enable_thinking": true}` via `extra_body` to activate it. The reasoning parser separates thinking and content, returning the thinking process via `reasoning_content` in the streaming response.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[
        {"role": "user", "content": "Solve step by step: If a train travels at 60 km/h for 2.5 hours, how far does it go?"}
    ],
    max_tokens=4096,
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}}
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

<details>
  <summary>Example Output</summary>

  ```text Output theme={null}
  =============== Thinking =================
  *   Input: Speed = 60 km/h, Time = 2.5 hours.
      *   Goal: Find the distance traveled.
      *   Distance = Speed × Time.
      *   Step 1: Identify given values. Speed = 60 km/h, Time = 2.5 hours
      *   Step 2: Formula. Distance = Speed × Time
      *   Step 3: Calculation. 60 × 2.5
          Mental math: 60 × 2 = 120; 60 × 0.5 = 30; 120 + 30 = 150.
      *   Step 4: Final Result. 150 km.

  =============== Content =================
  To find the distance traveled, you can follow these steps:

  ### 1. Identify the given information:
  *   **Speed:** 60 km/h
  *   **Time:** 2.5 hours

  ### 2. Use the distance formula:
  Distance = Speed × Time

  ### 3. Substitute the values:
  Distance = 60 km/h × 2.5 hours

  ### 4. Perform the calculation:
  *   60 × 2 = 120
  *   60 × 0.5 = 30
  *   120 + 30 = 150

  **Final Answer: The train travels 150 km.**
  ```
</details>

### 4.4 Tool Calling

Gemma 4 supports function calling with the `gemma4` tool call parser. Enable it during deployment with `--tool-call-parser gemma4`.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[
        {"role": "user", "content": "What's the weather in Tokyo?"}
    ],
    tools=tools,
    stream=True
)

thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            if has_thinking and thinking_started:
                print("\n=============== Tool Calls ================", flush=True)
                thinking_started = False
            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

<details>
  <summary>Example Output</summary>

  ```text Output theme={null}
  =============== Tool Calls ================
  Tool Call: get_weather
     Arguments: {"location": "Tokyo"}
  ```
</details>

### 4.5 Audio Input

The audio-capable Gemma 4 variants (`gemma-4-E2B-it`, `gemma-4-E4B-it`, `gemma-4-12B-it`) accept raw audio alongside text. Pass the waveform as a base64 `audio_url` data URI (16 kHz mono WAV works well), or as OpenAI's `input_audio` part with the base64 bytes in `data` and a `format` of `wav`:

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

with open("sample.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="google/gemma-4-12B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
                {"type": "text", "text": "Transcribe the speech in this audio exactly."},
            ],
        }
    ],
    max_tokens=256,
    temperature=0,
)

print(response.choices[0].message.content)
```

<details>
  <summary>Example Output</summary>

  ```text Output theme={null}
  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.
  ```
</details>

For best ASR quality, use the recommended transcription prompt structure:

```text Prompt theme={null}
Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.

Follow these specific instructions for formatting the answer:
* Only output the transcription, with no newlines.
* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
```

For speech translation (AST), ask for the transcription in the source language first, then the translation: *"Transcribe the following speech segment in , then translate it into . ..."*

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: H200
* SGLang Version: gemma4 branch

#### gemma-4-E2B-it (1x H200, TP=1)

Server Launch Command:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-E2B-it
```

**Latency Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 10 --max-concurrency 1
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  17.44
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.57
Output token throughput (tok/s):         242.03
Total token throughput (tok/s):          591.94
Mean TTFT (ms):                          50.19
Median TTFT (ms):                        54.22
Mean TPOT (ms):                          3.99
Median ITL (ms):                         4.05
==================================================
```

**Latency Benchmark (Image)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang-oai-chat \
  --host 0.0.0.0 --port 30000 \
  --dataset-name image --image-count 2 --image-resolution 720p \
  --random-input-len 128 --random-output-len 1024 \
  --num-prompts 10 --max-concurrency 1
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  18.05
Total input tokens:                      6097
Total input vision tokens:               5340
Total generated tokens:                  4220
Request throughput (req/s):              0.55
Output token throughput (tok/s):         233.84
Total token throughput (tok/s):          571.69
Mean TTFT (ms):                          109.59
Median TTFT (ms):                        112.62
Mean TPOT (ms):                          4.01
Median ITL (ms):                         4.04
==================================================
```

**Throughput Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 1000 --max-concurrency 100
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  51.73
Total input tokens:                      512842
Total generated tokens:                  510855
Request throughput (req/s):              19.33
Output token throughput (tok/s):         9876.36
Peak output token throughput (tok/s):    13863.00
Total token throughput (tok/s):          19791.14
Mean TTFT (ms):                          86.57
Mean TPOT (ms):                          9.56
Median ITL (ms):                         5.99
==================================================
```

**Throughput Benchmark (Image)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang-oai-chat \
  --host 0.0.0.0 --port 30000 \
  --dataset-name image --image-count 2 --image-resolution 720p \
  --random-input-len 128 --random-output-len 1024 \
  --num-prompts 1000 --max-concurrency 100
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  89.07
Total input tokens:                      617799
Total input vision tokens:               534000
Total generated tokens:                  510855
Request throughput (req/s):              11.23
Output token throughput (tok/s):         5735.75
Peak output token throughput (tok/s):    12823.00
Total token throughput (tok/s):          12672.23
Mean TTFT (ms):                          636.46
Mean TPOT (ms):                          16.34
Median ITL (ms):                         5.68
==================================================
```

#### gemma-4-E4B-it (1x H200, TP=1)

Server Launch Command:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-E4B-it
```

**Latency Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  24.49
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.41
Output token throughput (tok/s):         172.32
Total token throughput (tok/s):          421.45
Mean TTFT (ms):                          52.76
Median TTFT (ms):                        53.66
Mean TPOT (ms):                          5.64
Median ITL (ms):                         5.74
==================================================
```

**Latency Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  25.04
Total input tokens:                      6124
Total input vision tokens:               5340
Total generated tokens:                  4220
Request throughput (req/s):              0.40
Output token throughput (tok/s):         168.54
Total token throughput (tok/s):          413.13
Mean TTFT (ms):                          110.15
Median TTFT (ms):                        108.24
Mean TPOT (ms):                          5.66
Median ITL (ms):                         5.73
==================================================
```

**Throughput Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  72.95
Total input tokens:                      512842
Total generated tokens:                  510855
Request throughput (req/s):              13.71
Output token throughput (tok/s):         7002.68
Peak output token throughput (tok/s):    9878.00
Total token throughput (tok/s):          14032.60
Mean TTFT (ms):                          166.33
Mean TPOT (ms):                          13.36
Median ITL (ms):                         8.88
==================================================
```

**Throughput Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  108.99
Total input tokens:                      616952
Total input vision tokens:               534000
Total generated tokens:                  510855
Request throughput (req/s):              9.18
Output token throughput (tok/s):         4687.38
Peak output token throughput (tok/s):    9277.00
Total token throughput (tok/s):          10348.25
Mean TTFT (ms):                          626.17
Mean TPOT (ms):                          20.00
Median ITL (ms):                         8.64
==================================================
```

#### gemma-4-31B-it (2x H200, TP=2)

Server Launch Command:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-31B-it --tp 2
```

**Latency Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  53.05
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.19
Output token throughput (tok/s):         79.55
Total token throughput (tok/s):          194.55
Mean TTFT (ms):                          72.77
Median TTFT (ms):                        75.05
Mean TPOT (ms):                          12.32
Median ITL (ms):                         12.53
==================================================
```

**Latency Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  53.78
Total input tokens:                      6162
Total input vision tokens:               5340
Total generated tokens:                  4220
Request throughput (req/s):              0.19
Output token throughput (tok/s):         78.46
Total token throughput (tok/s):          193.03
Mean TTFT (ms):                          143.35
Median TTFT (ms):                        146.85
Mean TPOT (ms):                          12.37
Median ITL (ms):                         12.48
==================================================
```

**Throughput Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  182.00
Total input tokens:                      512842
Total generated tokens:                  510855
Request throughput (req/s):              5.49
Output token throughput (tok/s):         2806.82
Peak output token throughput (tok/s):    3798.00
Total token throughput (tok/s):          5624.56
Mean TTFT (ms):                          324.67
Mean TPOT (ms):                          33.95
Median ITL (ms):                         25.44
==================================================
```

**Throughput Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  236.46
Total input tokens:                      621630
Total input vision tokens:               534000
Total generated tokens:                  510855
Request throughput (req/s):              4.23
Output token throughput (tok/s):         2160.42
Peak output token throughput (tok/s):    3745.00
Total token throughput (tok/s):          4789.30
Mean TTFT (ms):                          952.02
Mean TPOT (ms):                          44.17
Median ITL (ms):                         26.81
==================================================
```

#### gemma-4-26B-A4B-it (MoE, 1x H200, TP=1)

Server Launch Command:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-26B-A4B-it
```

> **Tip**: Consider `--tp 2` for high-throughput workloads.

**Latency Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  25.00
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.40
Output token throughput (tok/s):         168.81
Total token throughput (tok/s):          412.85
Mean TTFT (ms):                          103.74
Median TTFT (ms):                        46.57
Mean TPOT (ms):                          5.60
Median ITL (ms):                         5.78
==================================================
```

**Latency Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  25.31
Total input tokens:                      6164
Total input vision tokens:               5340
Total generated tokens:                  4220
Request throughput (req/s):              0.40
Output token throughput (tok/s):         166.70
Total token throughput (tok/s):          410.20
Mean TTFT (ms):                          129.22
Median TTFT (ms):                        132.54
Mean TPOT (ms):                          5.68
Median ITL (ms):                         5.75
==================================================
```

**Throughput Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  138.98
Total input tokens:                      512842
Total generated tokens:                  510855
Request throughput (req/s):              7.20
Output token throughput (tok/s):         3675.81
Peak output token throughput (tok/s):    4799.00
Total token throughput (tok/s):          7365.91
Mean TTFT (ms):                          153.77
Mean TPOT (ms):                          25.95
Median ITL (ms):                         20.23
==================================================
```

**Throughput Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  186.38
Total input tokens:                      621146
Total input vision tokens:               534000
Total generated tokens:                  510855
Request throughput (req/s):              5.37
Output token throughput (tok/s):         2740.86
Peak output token throughput (tok/s):    4962.00
Total token throughput (tok/s):          6073.47
Mean TTFT (ms):                          854.71
Mean TPOT (ms):                          34.64
Median ITL (ms):                         19.08
==================================================
```

#### gemma-4-31B-it (1x MI300X, TP=1)

Server Launch Command:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-31B-it
```

> **Note**: The 31B dense model fits on a single MI300X (192 GB VRAM) at TP=1, unlike H200 (141 GB) which requires TP=2.

**Latency Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 10 --max-concurrency 1
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  103.55
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.10
Output token throughput (tok/s):         40.75
Total token throughput (tok/s):          99.67
Mean TTFT (ms):                          152.35
Median TTFT (ms):                        169.66
Mean TPOT (ms):                          24.13
Median ITL (ms):                         24.23
==================================================
```

**Throughput Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 1000 --max-concurrency 100
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  441.59
Total input tokens:                      512842
Total generated tokens:                  510855
Request throughput (req/s):              2.26
Output token throughput (tok/s):         1156.85
Peak output token throughput (tok/s):    1759.00
Total token throughput (tok/s):          2318.19
Mean TTFT (ms):                          819.22
Mean TPOT (ms):                          82.51
Median ITL (ms):                         63.45
==================================================
```

#### gemma-4-26B-A4B-it (MoE, 1x MI300X, TP=1)

Server Launch Command:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-26B-A4B-it
```

**Latency Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 10 --max-concurrency 1
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  43.73
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.23
Output token throughput (tok/s):         96.49
Total token throughput (tok/s):          236.00
Mean TTFT (ms):                          185.58
Median TTFT (ms):                        90.18
Mean TPOT (ms):                          9.78
Median ITL (ms):                         9.57
==================================================
```

**Throughput Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 1000 --max-concurrency 100
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  219.43
Total input tokens:                      512842
Total generated tokens:                  510855
Request throughput (req/s):              4.56
Output token throughput (tok/s):         2328.05
Peak output token throughput (tok/s):    3500.00
Total token throughput (tok/s):          4665.16
Mean TTFT (ms):                          168.44
Mean TPOT (ms):                          41.23
Median ITL (ms):                         29.31
==================================================
```

#### gemma-4-12B-it (1x H200, TP=1)

Server Launch Command:

```bash Command theme={null}
sglang serve --model-path google/gemma-4-12B-it
```

**Latency Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 10 --max-concurrency 1
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  38.66
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.26
Output token throughput (tok/s):         109.15
Total token throughput (tok/s):          266.94
Mean TTFT (ms):                          33.08
Median TTFT (ms):                        33.71
Mean TPOT (ms):                          9.02
Median ITL (ms):                         9.19
==================================================
```

**Latency Benchmark (Image)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang-oai-chat \
  --host 0.0.0.0 --port 30000 \
  --dataset-name image --image-count 2 --image-resolution 720p \
  --random-input-len 128 --random-output-len 1024 \
  --num-prompts 10 --max-concurrency 1
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  39.36
Total input vision tokens:               5320
Total generated tokens:                  4220
Request throughput (req/s):              0.25
Output token throughput (tok/s):         107.23
Total token throughput (tok/s):          263.62
Mean TTFT (ms):                          94.98
Median TTFT (ms):                        97.33
Mean TPOT (ms):                          9.08
Median ITL (ms):                         9.17
==================================================
```

**Throughput Benchmark (Text)**

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 1000 --max-concurrency 100
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  130.44
Total input tokens:                      512842
Total generated tokens:                  510855
Request throughput (req/s):              7.67
Output token throughput (tok/s):         3916.46
Total token throughput (tok/s):          7848.15
Mean TTFT (ms):                          207.49
Median TTFT (ms):                        76.95
Mean TPOT (ms):                          24.38
Median ITL (ms):                         17.89
==================================================
```

**Throughput Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  147.57
Total input tokens:                      619609
Total input vision tokens:               532000
Total generated tokens:                  510855
Request throughput (req/s):              6.78
Output token throughput (tok/s):         3461.79
Total token throughput (tok/s):          7660.54
Mean TTFT (ms):                          438.40
Median TTFT (ms):                        129.83
Mean TPOT (ms):                          27.12
Median ITL (ms):                         19.16
==================================================
```

#### gemma-4-12B-it (1x B200, TP=1)

Server Launch Command:

```bash Command theme={null}
# Text/audio: the sm100 default (trtllm_mha) is fastest.
# For image workloads add --attention-backend triton (bidirectional image attention).
sglang serve --model-path google/gemma-4-12B-it --attention-backend triton
```

**Latency Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  30.46
Output token throughput (tok/s):         138.55
Total token throughput (tok/s):          338.85
Mean TTFT (ms):                          28.14
Median TTFT (ms):                        29.74
Mean TPOT (ms):                          7.08
Median ITL (ms):                         7.26
==================================================
```

**Latency Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  31.43
Total input vision tokens:               5320
Total generated tokens:                  4220
Request throughput (req/s):              0.32
Output token throughput (tok/s):         134.26
Total token throughput (tok/s):          329.57
Mean TTFT (ms):                          115.51
Median TTFT (ms):                        74.27
Mean TPOT (ms):                          7.14
Median ITL (ms):                         7.24
==================================================
```

**Throughput Benchmark (Text)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  92.94
Request throughput (req/s):              10.76
Output token throughput (tok/s):         5496.55
Total token throughput (tok/s):          11014.49
Mean TTFT (ms):                          120.89
Median TTFT (ms):                        45.00
Mean TPOT (ms):                          17.23
Median ITL (ms):                         14.30
==================================================
```

**Throughput Benchmark (Image)**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Max request concurrency:                 100
Successful requests:                     998
Benchmark duration (s):                  107.82
Total input tokens:                      617971
Total input vision tokens:               530936
Total generated tokens:                  508951
Request throughput (req/s):              9.26
Output token throughput (tok/s):         4720.29
Total token throughput (tok/s):          10451.68
Mean TTFT (ms):                          425.89
Median TTFT (ms):                        109.57
Mean TPOT (ms):                          19.45
Median ITL (ms):                         15.11
==================================================
```

> **Performance tuning**: On B200, raising `--scheduler-recv-interval` to 16 lifted text throughput from 5497 to 5673 tok/s output (≈ +3%) at concurrency 100 with no accuracy change, by reducing the scheduler's per-step Python overhead. It is a safe, low-risk knob for high-concurrency serving.

### 5.2 Accuracy Benchmark

**Test Environment:**

* Hardware: H200
* SGLang Version: gemma4 branch

#### MMLU

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Humanities</th>
      <th>Social Sciences</th>
      <th>STEM</th>
      <th>Other</th>
      <th>Overall</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>gemma-4-E2B-it</td>
      <td>0.621</td>
      <td>0.739</td>
      <td>0.830</td>
      <td>0.736</td>
      <td>**0.720**</td>
    </tr>

    <tr>
      <td>gemma-4-E4B-it</td>
      <td>0.703</td>
      <td>0.862</td>
      <td>0.902</td>
      <td>0.825</td>
      <td>**0.810**</td>
    </tr>

    <tr>
      <td>gemma-4-12B-it</td>
      <td>0.784</td>
      <td>0.888</td>
      <td>0.946</td>
      <td>0.861</td>
      <td>**0.859**</td>
    </tr>

    <tr>
      <td>gemma-4-31B-it</td>
      <td>0.878</td>
      <td>0.921</td>
      <td>0.884</td>
      <td>0.911</td>
      <td>**0.896**</td>
    </tr>

    <tr>
      <td>gemma-4-26B-A4B-it</td>
      <td>0.853</td>
      <td>0.906</td>
      <td>0.938</td>
      <td>0.886</td>
      <td>**0.891**</td>
    </tr>
  </tbody>
</table>

#### GSM8K

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Accuracy</th>
      <th>Invalid</th>
      <th>Latency (s)</th>
      <th>Output Throughput (tok/s)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>gemma-4-E2B-it</td>
      <td>0.170</td>
      <td>0.000</td>
      <td>3.990</td>
      <td>8041.739</td>
    </tr>

    <tr>
      <td>gemma-4-E4B-it</td>
      <td>0.745</td>
      <td>0.000</td>
      <td>4.174</td>
      <td>4672.030</td>
    </tr>

    <tr>
      <td>gemma-4-12B-it</td>
      <td>0.431</td>
      <td>0.052</td>
      <td>55.105</td>
      <td>6580.229</td>
    </tr>

    <tr>
      <td>gemma-4-31B-it</td>
      <td>0.805</td>
      <td>0.005</td>
      <td>16.148</td>
      <td>1559.914</td>
    </tr>

    <tr>
      <td>gemma-4-26B-A4B-it</td>
      <td>0.450</td>
      <td>0.010</td>
      <td>13.001</td>
      <td>4089.457</td>
    </tr>
  </tbody>
</table>

> **Note**: These GSM8K numbers use the raw few-shot completion harness (now removed). `gemma-4-12B-it` is reasoning-oriented and is under-elicited by raw few-shot prompting; with the chat template it scores **0.950** on the same 1319 GSM8K test questions (also measured before the sgl-eval migration).

#### gemma-4-12B-it with sgl-eval

`gemma-4-12B-it` is reasoning-oriented and answers verbosely (step-by-step) rather than emitting a terse final line. Strict last-line `Answer: $LETTER` extraction (as in `sglang.test.run_eval`) therefore undercounts its correct answers. [sgl-eval](https://github.com/sgl-project/sgl-eval) — sgl-project's evaluation CLI, which uses robust answer extraction — gives a faithful score on the served model:

| Benchmark | Examples | Accuracy |
| - | - | - |
| MMLU | 2000 | **0.878** |
| GSM8K | 1319 | **0.960** |

Reproduce against a running server (`--base-url` points at your endpoint):

```bash Command theme={null}
pip install sgl-eval

# Sanity-check the endpoint
sgl-eval ping --base-url http://localhost:30000/v1

# Run the benchmarks (greedy, single-shot)
sgl-eval run gsm8k --base-url http://localhost:30000/v1
sgl-eval run mmlu  --base-url http://localhost:30000/v1 --num-examples 2000
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

#### MMMU

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Overall</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>gemma-4-E2B-it</td>
      <td>**0.307**</td>
    </tr>

    <tr>
      <td>gemma-4-E4B-it</td>
      <td>**0.396**</td>
    </tr>

    <tr>
      <td>gemma-4-12B-it</td>
      <td>**0.683**</td>
    </tr>

    <tr>
      <td>gemma-4-31B-it</td>
      <td>**0.589**</td>
    </tr>

    <tr>
      <td>gemma-4-26B-A4B-it</td>
      <td>**0.549**</td>
    </tr>
  </tbody>
</table>

<details>
  <summary>MMMU detailed scores (per domain)</summary>

  **gemma-4-E2B-it**

  ```json Config theme={null}
  {"Overall-Art and Design": {"num": 120, "acc": 0.45}, "Art": {"num": 30, "acc": 0.5}, "Art_Theory": {"num": 30, "acc": 0.467}, "Design": {"num": 30, "acc": 0.5}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.26}, "Accounting": {"num": 30, "acc": 0.367}, "Economics": {"num": 30, "acc": 0.233}, "Finance": {"num": 30, "acc": 0.2}, "Manage": {"num": 30, "acc": 0.233}, "Marketing": {"num": 30, "acc": 0.267}, "Overall-Science": {"num": 150, "acc": 0.273}, "Biology": {"num": 30, "acc": 0.233}, "Chemistry": {"num": 30, "acc": 0.267}, "Geography": {"num": 30, "acc": 0.367}, "Math": {"num": 30, "acc": 0.233}, "Physics": {"num": 30, "acc": 0.267}, "Overall-Health and Medicine": {"num": 150, "acc": 0.273}, "Basic_Medical_Science": {"num": 30, "acc": 0.5}, "Clinical_Medicine": {"num": 30, "acc": 0.233}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.233}, "Pharmacy": {"num": 30, "acc": 0.3}, "Public_Health": {"num": 30, "acc": 0.1}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.4}, "History": {"num": 30, "acc": 0.4}, "Literature": {"num": 30, "acc": 0.567}, "Sociology": {"num": 30, "acc": 0.333}, "Psychology": {"num": 30, "acc": 0.3}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.252}, "Agriculture": {"num": 30, "acc": 0.333}, "Architecture_and_Engineering": {"num": 30, "acc": 0.267}, "Computer_Science": {"num": 30, "acc": 0.233}, "Electronics": {"num": 30, "acc": 0.1}, "Energy_and_Power": {"num": 30, "acc": 0.3}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.307}}
  ```

  **gemma-4-E4B-it**

  ```json Config theme={null}
  {"Overall-Art and Design": {"num": 120, "acc": 0.458}, "Art": {"num": 30, "acc": 0.433}, "Art_Theory": {"num": 30, "acc": 0.567}, "Design": {"num": 30, "acc": 0.667}, "Music": {"num": 30, "acc": 0.167}, "Overall-Business": {"num": 150, "acc": 0.287}, "Accounting": {"num": 30, "acc": 0.233}, "Economics": {"num": 30, "acc": 0.467}, "Finance": {"num": 30, "acc": 0.133}, "Manage": {"num": 30, "acc": 0.3}, "Marketing": {"num": 30, "acc": 0.3}, "Overall-Science": {"num": 150, "acc": 0.28}, "Biology": {"num": 30, "acc": 0.333}, "Chemistry": {"num": 30, "acc": 0.133}, "Geography": {"num": 30, "acc": 0.4}, "Math": {"num": 30, "acc": 0.2}, "Physics": {"num": 30, "acc": 0.333}, "Overall-Health and Medicine": {"num": 150, "acc": 0.427}, "Basic_Medical_Science": {"num": 30, "acc": 0.4}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.4}, "Pharmacy": {"num": 30, "acc": 0.4}, "Public_Health": {"num": 30, "acc": 0.4}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.7}, "History": {"num": 30, "acc": 0.633}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.567}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.324}, "Agriculture": {"num": 30, "acc": 0.533}, "Architecture_and_Engineering": {"num": 30, "acc": 0.3}, "Computer_Science": {"num": 30, "acc": 0.367}, "Electronics": {"num": 30, "acc": 0.133}, "Energy_and_Power": {"num": 30, "acc": 0.4}, "Materials": {"num": 30, "acc": 0.2}, "Mechanical_Engineering": {"num": 30, "acc": 0.333}, "Overall": {"num": 900, "acc": 0.396}}
  ```

  **gemma-4-12B-it**

  ```json Config theme={null}
  {"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.7}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.767}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.747}, "Accounting": {"num": 30, "acc": 0.767}, "Economics": {"num": 30, "acc": 0.767}, "Finance": {"num": 30, "acc": 0.633}, "Manage": {"num": 30, "acc": 0.7}, "Marketing": {"num": 30, "acc": 0.867}, "Overall-Science": {"num": 150, "acc": 0.647}, "Biology": {"num": 30, "acc": 0.6}, "Chemistry": {"num": 30, "acc": 0.633}, "Geography": {"num": 30, "acc": 0.567}, "Math": {"num": 30, "acc": 0.6}, "Physics": {"num": 30, "acc": 0.833}, "Overall-Health and Medicine": {"num": 150, "acc": 0.68}, "Basic_Medical_Science": {"num": 30, "acc": 0.667}, "Clinical_Medicine": {"num": 30, "acc": 0.633}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.267}, "Pharmacy": {"num": 30, "acc": 0.833}, "Public_Health": {"num": 30, "acc": 1.0}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.817}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.9}, "Sociology": {"num": 30, "acc": 0.8}, "Psychology": {"num": 30, "acc": 0.767}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.6}, "Agriculture": {"num": 30, "acc": 0.467}, "Architecture_and_Engineering": {"num": 30, "acc": 0.667}, "Computer_Science": {"num": 30, "acc": 0.733}, "Electronics": {"num": 30, "acc": 0.567}, "Energy_and_Power": {"num": 30, "acc": 0.667}, "Materials": {"num": 30, "acc": 0.567}, "Mechanical_Engineering": {"num": 30, "acc": 0.533}, "Overall": {"num": 900, "acc": 0.683}}
  ```

  **gemma-4-31B-it**

  ```json Config theme={null}
  {"Overall-Art and Design": {"num": 120, "acc": 0.667}, "Art": {"num": 30, "acc": 0.667}, "Art_Theory": {"num": 30, "acc": 0.867}, "Design": {"num": 30, "acc": 0.8}, "Music": {"num": 30, "acc": 0.333}, "Overall-Business": {"num": 150, "acc": 0.573}, "Accounting": {"num": 30, "acc": 0.633}, "Economics": {"num": 30, "acc": 0.733}, "Finance": {"num": 30, "acc": 0.433}, "Manage": {"num": 30, "acc": 0.533}, "Marketing": {"num": 30, "acc": 0.533}, "Overall-Science": {"num": 150, "acc": 0.527}, "Biology": {"num": 30, "acc": 0.667}, "Chemistry": {"num": 30, "acc": 0.567}, "Geography": {"num": 30, "acc": 0.5}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.633}, "Overall-Health and Medicine": {"num": 150, "acc": 0.673}, "Basic_Medical_Science": {"num": 30, "acc": 0.733}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.467}, "Pharmacy": {"num": 30, "acc": 0.8}, "Public_Health": {"num": 30, "acc": 0.833}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.825}, "History": {"num": 30, "acc": 0.833}, "Literature": {"num": 30, "acc": 0.867}, "Sociology": {"num": 30, "acc": 0.767}, "Psychology": {"num": 30, "acc": 0.833}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.405}, "Agriculture": {"num": 30, "acc": 0.667}, "Architecture_and_Engineering": {"num": 30, "acc": 0.2}, "Computer_Science": {"num": 30, "acc": 0.567}, "Electronics": {"num": 30, "acc": 0.333}, "Energy_and_Power": {"num": 30, "acc": 0.533}, "Materials": {"num": 30, "acc": 0.3}, "Mechanical_Engineering": {"num": 30, "acc": 0.233}, "Overall": {"num": 900, "acc": 0.589}}
  ```

  **gemma-4-26B-A4B-it**

  ```json Config theme={null}
  {"Overall-Art and Design": {"num": 120, "acc": 0.717}, "Art": {"num": 30, "acc": 0.733}, "Art_Theory": {"num": 30, "acc": 0.833}, "Design": {"num": 30, "acc": 0.867}, "Music": {"num": 30, "acc": 0.433}, "Overall-Business": {"num": 150, "acc": 0.493}, "Accounting": {"num": 30, "acc": 0.533}, "Economics": {"num": 30, "acc": 0.533}, "Finance": {"num": 30, "acc": 0.333}, "Manage": {"num": 30, "acc": 0.5}, "Marketing": {"num": 30, "acc": 0.567}, "Overall-Science": {"num": 150, "acc": 0.473}, "Biology": {"num": 30, "acc": 0.633}, "Chemistry": {"num": 30, "acc": 0.367}, "Geography": {"num": 30, "acc": 0.533}, "Math": {"num": 30, "acc": 0.267}, "Physics": {"num": 30, "acc": 0.567}, "Overall-Health and Medicine": {"num": 150, "acc": 0.62}, "Basic_Medical_Science": {"num": 30, "acc": 0.767}, "Clinical_Medicine": {"num": 30, "acc": 0.533}, "Diagnostics_and_Laboratory_Medicine": {"num": 30, "acc": 0.433}, "Pharmacy": {"num": 30, "acc": 0.7}, "Public_Health": {"num": 30, "acc": 0.667}, "Overall-Humanities and Social Science": {"num": 120, "acc": 0.758}, "History": {"num": 30, "acc": 0.8}, "Literature": {"num": 30, "acc": 0.833}, "Sociology": {"num": 30, "acc": 0.733}, "Psychology": {"num": 30, "acc": 0.667}, "Overall-Tech and Engineering": {"num": 210, "acc": 0.376}, "Agriculture": {"num": 30, "acc": 0.633}, "Architecture_and_Engineering": {"num": 30, "acc": 0.367}, "Computer_Science": {"num": 30, "acc": 0.533}, "Electronics": {"num": 30, "acc": 0.167}, "Energy_and_Power": {"num": 30, "acc": 0.367}, "Materials": {"num": 30, "acc": 0.367}, "Mechanical_Engineering": {"num": 30, "acc": 0.2}, "Overall": {"num": 900, "acc": 0.549}}
  ```

  #### ASR

  <table>
    <colgroup>
      <col />

      <col />

      <col />

      <col />
    </colgroup>

    <thead>
      <tr>
        <th>Model</th>
        <th>WER</th>
        <th>Avg Latency (s)</th>
        <th>Throughput (req/s)</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>gemma-4-E2B-it</td>
        <td>23.86%</td>
        <td>0.212</td>
        <td>2.99</td>
      </tr>

      <tr>
        <td>gemma-4-E4B-it</td>
        <td>29.55%</td>
        <td>0.366</td>
        <td>2.46</td>
      </tr>

      <tr>
        <td>gemma-4-12B-it</td>
        <td>Supported (see §4.5)</td>
        <td>—</td>
        <td>—</td>
      </tr>

      <tr>
        <td>gemma-4-31B-it</td>
        <td>Not Supported</td>
        <td>—</td>
        <td>—</td>
      </tr>

      <tr>
        <td>gemma-4-26B-A4B-it</td>
        <td>Not Supported</td>
        <td>—</td>
        <td>—</td>
      </tr>
    </tbody>
  </table>

  #### FLEUR (EN\_US)

  <table>
    <colgroup>
      <col />

      <col />

      <col />

      <col />
    </colgroup>

    <thead>
      <tr>
        <th>Model</th>
        <th>WER</th>
        <th>Avg Latency (s)</th>
        <th>Throughput (req/s)</th>
      </tr>
    </thead>

    <tbody>
      <tr>
        <td>gemma-4-E2B-it</td>
        <td>7.37%</td>
        <td>0.8963s</td>
        <td>16.25</td>
      </tr>

      <tr>
        <td>gemma-4-E4B-it</td>
        <td>6.08%</td>
        <td>0.8707s</td>
        <td>16.20</td>
      </tr>

      <tr>
        <td>gemma-4-12B-it</td>
        <td>Supported (see §4.5)</td>
        <td>—</td>
        <td>—</td>
      </tr>

      <tr>
        <td>gemma-4-31B-it</td>
        <td>Not Supported</td>
        <td>—</td>
        <td>—</td>
      </tr>

      <tr>
        <td>gemma-4-26B-A4B-it</td>
        <td>Not Supported</td>
        <td>—</td>
        <td>—</td>
      </tr>
    </tbody>
  </table>

  ### 5.3 Logits correctness validation

  **gemma-4-E2B-it**

  ```shell Command theme={null}
  $ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E2B-it ....
  prefill logits (final): tensor([[-25.3063,  -2.5718, -10.3674,  ..., -25.3779, -25.5181, -25.2337]],
         device='cuda:0')
  ....

  $ python scripts/playground/reference_hf.py --model-path google/gemma-4-E2B-it
  ....
  prefill logits (final) tensor([-25.3281,  -2.1367, -10.2266,  ..., -25.4375, -25.5000, -25.2500],
         device='cuda:0', dtype=torch.float16)
  ....
  ```

  **gemma-4-E4B-it**

  ```shell Command theme={null}
  $ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E4B-it ....
  prefill logits (final): tensor([[-17.6478,   7.9901,  -5.6505,  ..., -17.5658, -17.6478, -17.7293]],
         device='cuda:0')
  ....

  $ python scripts/playground/reference_hf.py --model-path google/gemma-4-E4B-it
  ....
  prefill logits (final) tensor([-17.5625,   8.0469,  -5.5742,  ..., -17.4688, -17.5625, -17.6719],
         device='cuda:0', dtype=torch.float16)
  ....
  ```

  **gemma-4-31B-it**

  ```shell Command theme={null}
  $ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-31B-it ....
  prefill logits (final): tensor([[-2.0748,  1.1245, -7.4356,  ..., -2.1059, -2.1525, -2.2303]],
         device='cuda:0')
  ....

  $ python scripts/playground/reference_hf.py --model-path google/gemma-4-31B-it
  ....
  prefill logits (final) tensor([-2.1133,  1.2656, -7.4766,  ..., -2.1523, -2.2012, -2.2695],
         device='cuda:0', dtype=torch.float16)
  ....
  ```
</details>


# Granite 4.2
Source: https://docs.sglang.io/cookbook/autoregressive/IBM/Granite-4.2

Deploy Granite 4.2 3B, 8B, and 30B dense models with SGLang on NVIDIA H200 and B200, including thinking modes and structured tool calling.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick a Granite 4.2 checkpoint to generate the launch command. The verified matrix covers BF16 serving on one NVIDIA H200 or B200 with tensor parallelism 1.

> [!NOTE]
>
> The H200 speed results use `lmsysorg/sglang:dev` at SGLang commit `d59c1ddf7` and the B200 results at commit `d10a656ad8`; the launch recipes were verified end to end on both GPUs against the release checkpoints. Each speed point uses 80 fixed-length random requests at 8,192 input and 1,024 output tokens, 8 warmup requests, a flushed cache, greedy sampling, and ignore-EOS.

## Playground

The Playground layers SGLang features on top of the verified recipe. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end.

## 1. Model introduction

**Granite 4.2** is IBM's dense decoder-only language model family with 3B, 8B, and 30B checkpoints. Each checkpoint uses BF16 weights, has a configured context length of 131,072 tokens, and supports default thinking, non-thinking, low-effort thinking, and structured tool calls through its chat template. The repositories declare the Apache-2.0 license.

<table>
  <thead>
    <tr>
      <th>Variant</th>
      <th>Total params</th>
      <th>Position in family</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[Granite 4.2 3B](https://huggingface.co/ibm-granite/granite-4.2-3b)**</td>
      <td>3B</td>
      <td>Smallest checkpoint</td>
    </tr>

    <tr>
      <td>**[Granite 4.2 8B](https://huggingface.co/ibm-granite/granite-4.2-8b)**</td>
      <td>8B</td>
      <td>Mid-size checkpoint</td>
    </tr>

    <tr>
      <td>**[Granite 4.2 30B](https://huggingface.co/ibm-granite/granite-4.2-30b)**</td>
      <td>30B</td>
      <td>Largest checkpoint</td>
    </tr>
  </tbody>
</table>

**Recommended generation:** IBM recommends `temperature=1.0` and `top_p=0.95` for general chat, reasoning, and tool calling. The release checkpoints ship these values in `generation_config.json`; send them per request when you want to be explicit.

**Resources:** [Granite 4.2 3B](https://huggingface.co/ibm-granite/granite-4.2-3b) · [Granite 4.2 8B](https://huggingface.co/ibm-granite/granite-4.2-8b) · [Granite 4.2 30B](https://huggingface.co/ibm-granite/granite-4.2-30b).

## 2. Configuration tips

* **Thinking is enabled by default.** Set `chat_template_kwargs.enable_thinking` to `false` for a direct answer. Set `enable_thinking` and `low_effort` to `true` for a shorter reasoning trace.
* **Give thinking enough tokens.** At `temperature=1.0` the default thinking mode can run past 1,000 tokens on multi-step problems. Use a `max_tokens` of at least 2,048 for thinking requests so the answer is not cut off.
* **Reasoning parser.** Launch with `--reasoning-parser auto`, which resolves to `nemotron_3` for these checkpoints, so OpenAI-compatible responses separate the trace into `message.reasoning_content` and the answer into `message.content`. Without a parser flag the reasoning markup stays inline in `message.content`.
* **Tool-call parser.** Launch with `--tool-call-parser auto`, which resolves to `qwen3_coder` for these checkpoints, so tool requests are returned through `message.tool_calls`. Without it, raw `<tool_call>` markup stays in `message.content`.
* **Single-GPU sizing.** All three BF16 checkpoints loaded and completed chat requests with `--tp 1 --mem-fraction-static 0.8` on one H200 and on one B200. Increase TP only after validating the new topology.
* **Image selection.** The validated path uses `lmsysorg/sglang:dev`. A stable image tested during validation had an incompatible dependency set before model loading, so use the recipe's image until a newer tagged release is confirmed.

## 3. Advanced usage

The outputs below are verbatim captures from Granite 4.2 3B on the verified server. Sampling is stochastic, so a repeated request can produce different wording.

### 3.1 Thinking modes

The `nemotron_3` reasoning parser keeps reasoning and final content in separate fields. Granite 4.2 accepts three chat-template modes: default thinking, non-thinking, and low-effort thinking.

> **Thinking modes example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> model = "ibm-granite/granite-4.2-3b"
>
> modes = {
>     "thinking": {"enable_thinking": True},
>     "non-thinking": {"enable_thinking": False},
>     "low-effort": {"enable_thinking": True, "low_effort": True},
> }
>
> for name, chat_template_kwargs in modes.items():
>     response = client.chat.completions.create(
>         model=model,
>         messages=[
>             {"role": "user", "content": "What is 17 * 23? Answer briefly."}
>         ],
>         extra_body={"chat_template_kwargs": chat_template_kwargs},
>     )
>     message = response.choices[0].message
>     print(f"[{name}]")
>     print("Reasoning:", getattr(message, "reasoning_content", None))
>     print("Answer:", message.content)
> ```

> **Example output**
>
> ```text Output theme={null}
> [thinking]
> Reasoning: Okay, the user asked "What is 17 * 23? Answer briefly." I need to calculate 17 multiplied by 23.
>
> Let me do the multiplication. 17 times 23.
>
> I can break it down: 17 * 20 = 340, and 17 * 3 = 51. Then add them: 340 + 51 = 391.
>
> Alternatively, 23 * 17: 23*10=230, 23*7=161, 230+161=391. Same result.
>
> So the answer is 391.
>
> The user wants a brief answer, so just state the number.
>
> Answer:
> 391
> [non-thinking]
> Reasoning: None
> Answer: 391
> [low-effort]
> Reasoning: Compute 17*23 = 17*20=340, plus 17*3=51 => 391.
>
> Answer:
> 391
> ```

### 3.2 Tool calling

The `qwen3_coder` parser converts the model's tool markup into OpenAI-compatible structured calls.

> **Tool calling example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city.",
>         "parameters": {
>             "type": "object",
>             "properties": {
>                 "city": {"type": "string", "description": "The city name"},
>             },
>             "required": ["city"],
>         },
>     },
> }]
>
> response = client.chat.completions.create(
>     model="ibm-granite/granite-4.2-3b",
>     messages=[{"role": "user", "content": "What is the weather in Boston right now?"}],
>     tools=tools,
>     tool_choice="auto",
> )
>
> choice = response.choices[0]
> message = choice.message
> print("Reasoning:", getattr(message, "reasoning_content", None))
> print("Content:", message.content)
> for call in message.tool_calls or []:
>     print("Tool:", call.function.name)
>     print("Arguments:", call.function.arguments)
> print("Finish reason:", choice.finish_reason)
> ```

> **Example output**
>
> ```text Output theme={null}
> Reasoning: Okay, the user is asking for the weather in Boston right now. I need to use the available tool called get_weather. The tool requires the city parameter. Since the user specified Boston, I'll call get_weather with city set to Boston.
>
> Content: None
> Tool: get_weather
> Arguments: {"city": "Boston"}
> Finish reason: tool_calls
> ```


# Jais 3
Source: https://docs.sglang.io/cookbook/autoregressive/IFM/Jais-3

Deploy the IFM Jais 3 family with SGLang — BF16 recipes for six dense and Mixture-of-Experts checkpoints on NVIDIA H100 GPUs.

## Deployment

> **Install SGLang**
>
> Jais 3 is supported from SGLang 0.5.20:
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install "sglang>=0.5.20"
> ```
>
> For other install methods, see [Install](/docs/get-started/install).

Choose a model size to generate its H100 launch command. Five checkpoints run on a single node; 375B-A23B needs two. The recipes serve each repository's `main` branch.

Each accuracy value in the benchmark cards is the arithmetic mean of two full evaluation runs. Speed values come from a single run.

## Playground

The base recipes enable the `k2_horizon` reasoning parser. Use the Playground to configure the `k2_horizon` tool-call parser, TP and MoE expert-parallel overrides, NGRAM speculative decoding, PD disaggregation, and HiCache. A separate validation pass functionally exercised the tool-call parser on all six checkpoints; every other Playground override remains unverified.

## 1. Model introduction

**Jais 3** is an IFM model family spanning six checkpoints from **0.9B** to **375B** parameters. The family includes dense models, Mixture-of-Experts models, and the MoVA-36B-A4B checkpoint's routed Mixture-of-Value-Attention path. This page provides native SGLang serving recipes for the family.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Checkpoint</th>
      <th>Architecture</th>
      <th>Context length</th>
      <th>H100 recipe</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[0.9B](https://huggingface.co/IFM/Jais-3-0.9B)**</td>
      <td>Dense</td>
      <td>131,072</td>
      <td>BF16 · TP1</td>
    </tr>

    <tr>
      <td>**[3.7B](https://huggingface.co/IFM/Jais-3-3.7B)**</td>
      <td>Dense</td>
      <td>524,288</td>
      <td>BF16 · TP1</td>
    </tr>

    <tr>
      <td>**[7B](https://huggingface.co/IFM/Jais-3-7B)**</td>
      <td>Dense</td>
      <td>524,288</td>
      <td>BF16 · TP1</td>
    </tr>

    <tr>
      <td>**[32B](https://huggingface.co/IFM/Jais-3-32B)**</td>
      <td>Dense</td>
      <td>524,288</td>
      <td>BF16 · TP2</td>
    </tr>

    <tr>
      <td>**[MoVA-36B-A4B](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B)**</td>
      <td>MoE + MoVA</td>
      <td>524,288</td>
      <td>BF16 · TP2 · EP2</td>
    </tr>

    <tr>
      <td>**[375B-A23B](https://huggingface.co/IFM/Jais-3-375B-A23B)**</td>
      <td>MoE</td>
      <td>524,288</td>
      <td>BF16 · TP16 · EP16 · 2 nodes</td>
    </tr>
  </tbody>
</table>

**Resources:** [Jais 3 collection](https://huggingface.co/collections/IFM/jais-3) · [Jais 3 375B-A23B model card](https://huggingface.co/IFM/Jais-3-375B-A23B).

## 2. Configuration tips

* **Hardware and precision:** The current matrix covers NVIDIA H100 GPUs and BF16 checkpoints only.
* **Parallelism:** Use TP1 for 0.9B, 3.7B, and 7B; TP2 for 32B; TP2 with expert parallelism (`--ep 2`) for MoVA-36B-A4B; and TP16 with `--ep 16` across two eight-GPU nodes for 375B-A23B. Every other recipe stays within one node.
* **Attention:** The verified base recipes select FlashAttention-3 with `--attention-backend fa3`.
* **Parsers:** The base recipes enable reasoning with `--reasoning-parser k2_horizon`. Enable `--tool-call-parser k2_horizon` from the **Parsers** card in the [Playground](#playground) when you need tool calling.
* **MoVA-36B-A4B router provenance:** The MoVA-36B-A4B checkpoint's source xLLM router used two GEMM partitions. `--json-model-override-args '{"xllm_source_router_gemm_partitions":2}'` preserves its BF16 partial-GEMM rounding and FP32 reduction order; SGLang does not infer this source topology from runtime TP.
* **375B-A23B memory:** The BF16 weights take about 760 GB. One node of eight H100 80 GB GPUs holds 640 GB, so the checkpoint does not fit on a single node; the recipe uses two nodes (16 GPUs, 1,280 GB). Run the generated command on both nodes, with node rank 0 on the head node and 1 on the other.
* **375B-A23B loading:** The 375B-A23B recipe disables multithreaded weight loading with `--model-loader-extra-config '{"enable_multithread_load":false}'`.
* **Playground features:** TP and EP settings other than the base recipes', NGRAM, PD disaggregation, and HiCache are optional, unverified command overrides. The tool-call parser was functionally exercised separately.
* **Deliberate omissions:** Context parallelism is unsupported or unvalidated for xLLM. DP-attention and alternate MoE backend selectors are not exposed until they are validated.
* **Validation:** All six recipes completed two full GSM8K evaluation runs on H100; see [Benchmark results](#3-benchmark-results).

## 3. Benchmark results

Speed was measured with native `sglang.benchmark.serving` using fixed random token IDs: 8,192 input tokens, 1,024 output tokens, 64 warmup requests, a cache flush, and seed `20260901`. The latency run uses 32 prompts at concurrency 1. The throughput run uses 256 prompts at concurrency 64 for the single-GPU checkpoints, 32 for the two-GPU checkpoints, and 8 for the two-node 375B-A23B. Each speed value comes from a single run, with the load generator on a separate client node; TTFT and TPOT are medians. Throughput per GPU is `(total input tokens + total output tokens) / duration / TP`. Checkpoints and workloads not yet measured show a dash in the benchmark cards.

GSM8K used the full 1,319-example test split with `sgl-eval`, 32 client threads, a 32,768-token output limit, temperature 0, top-p 0.95, seed 0, and high reasoning effort. Each server ran on H100 from the repository's `main` branch, with no pinned revision. The evaluation is zero-shot: each question is sent as a single user message with the `math` prompt, which asks for the final answer inside `\boxed{}`, and no worked examples. Scoring is rule-based accuracy with no LLM judge: `sgl-eval` extracts the last `\boxed{}` from the final answer and checks it against the reference by symbolic equivalence, so `\$57{,}500` matches `57500`. A response with no boxed answer counts as incorrect, so these scores can be lower than LLM-as-a-judge results on the same outputs.

Each checkpoint was evaluated twice, and the table shows both runs and their mean. The 0.9B and 3.7B runs were issued one after the other; from 7B onward both were issued together with `--n-repeats 2` against one running server. All requests completed without errors. Responses that reached the output limit stay in the score and count as incorrect; the Truncated column gives the share of such responses in run 1 and run 2.

| Checkpoint | Run 1 | Run 2 | Mean | Truncated |
| - | -: | -: | -: | -: |
| 0.9B | 83.62% | 83.78% | 83.70% | 0.68% / 0.61% |
| 3.7B | 96.36% | 96.13% | 96.25% | 0.00% / 0.15% |
| 7B | 95.91% | 95.83% | 95.87% | 0.00% / 0.08% |
| 32B | 96.51% | 96.74% | 96.63% | 0.00% / 0.00% |
| MoVA-36B-A4B | 95.98% | 96.44% | 96.21% | 0.00% / 0.00% |
| 375B-A23B | 96.44% | 96.21% | 96.32% | 0.00% / 0.00% |

## 4. Advanced usage

### 4.1 Reasoning

The base recipes include `--reasoning-parser k2_horizon`. The parser returns thinking in `reasoning_content` and the final answer in `content`.

The example output below was captured with the MoVA-36B-A4B checkpoint and both parser flags enabled.

> **Reasoning example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> response = client.chat.completions.create(
>     model="IFM/Jais-3-MoVA-36B-A4B",
>     messages=[{"role": "user", "content": "Hello"}],
> )
>
> message = response.choices[0].message
> reasoning = getattr(message, "reasoning_content", None)
> if reasoning is None:
>     reasoning = (message.model_extra or {}).get("reasoning_content")
> print("Reasoning:", reasoning)
> print("Answer:", message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user says "Hello". As a helpful AI, we should greet them back, maybe ask how we can assist.
>
> We need to respond with a friendly greeting. Probably something like "Hello! How can I help you today?"
>
> That's straightforward.
>
> We should consider any policies for greeting. No issues.
>
> Let's proceed.
> Answer: Hello! How can I help you today?
> ```

### 4.2 Tool calling

Enable **Tool Call Parser** in the **Parsers** card of the [Playground](#playground) to add `--tool-call-parser k2_horizon`. The parser exposes K2 tool output as OpenAI-compatible `message.tool_calls`.

The example output below was captured with the MoVA-36B-A4B checkpoint and both parser flags enabled.

> **Tool-calling example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Return weather for a city.",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"city": {"type": "string"}},
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="IFM/Jais-3-MoVA-36B-A4B",
>     messages=[{"role": "user", "content": "Call get_weather for Paris."}],
>     tools=tools,
>     tool_choice={"type": "function", "function": {"name": "get_weather"}},
>     max_tokens=1024,
> )
>
> message = response.choices[0].message
> reasoning = getattr(message, "reasoning_content", None)
> if reasoning is None:
>     reasoning = (message.model_extra or {}).get("reasoning_content")
> print("Reasoning:", reasoning)
> print("Content:", message.content or "")
> for tool_call in message.tool_calls or []:
>     print(f"Tool call: {tool_call.function.name}({tool_call.function.arguments})")
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user wants the weather for Paris, using the get_weather function. We'll call the function.
> Content: [{"name":"get_weather", "parameters": {"city": "Paris"}}]
> ```

### 4.3 HiCache

For large checkpoints or repeated long-context prefixes, enable **HiCache** in the [Playground](#playground), then choose a storage backend and write policy. Treat the generated override as unverified until it is exercised on your storage stack.


# K2 Horizon
Source: https://docs.sglang.io/cookbook/autoregressive/IFM/K2-Horizon

Deploy the IFM K2 Horizon family with SGLang — BF16 recipes for six dense and Mixture-of-Experts checkpoints on NVIDIA H200 GPUs.

## Deployment

> **Install SGLang**
>
> Install from a source checkout that includes the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654) before using the generated commands:
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install -e "python"
> ```
>
> For general source-install guidance, see [Install → Method 2: From source](/docs/get-started/install#method-2-from-source).

Choose a model size to generate its validated single-node H200 launch command. The serving settings in all six base recipes completed two independent H200 benchmark launches at the pinned model revisions using the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654).

Each speed and accuracy value in the benchmark cards is the arithmetic mean of two independent server launches.

## Playground

The base recipes enable the `k2_horizon` reasoning parser. Use the Playground to configure the `k2_horizon` tool-call parser, TP and MoE expert-parallel overrides, NGRAM speculative decoding, PD disaggregation, and HiCache. A separate validation pass functionally exercised the tool-call parser on all six checkpoints; every other Playground override remains unverified.

## 1. Model introduction

**K2 Horizon** is an IFM model family spanning six checkpoints from **0.9B** to **375B** parameters. The family includes dense models, Mixture-of-Experts models, and the MoVA-36B-A4B checkpoint's routed Mixture-of-Value-Attention path. This page provides native SGLang serving recipes for the family.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Checkpoint</th>
      <th>Architecture</th>
      <th>Context length</th>
      <th>H200 recipe</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[0.9B](https://huggingface.co/IFM/K2-Horizon-0.9B)**</td>
      <td>Dense</td>
      <td>131,072</td>
      <td>BF16 · TP1</td>
    </tr>

    <tr>
      <td>**[3.7B](https://huggingface.co/IFM/K2-Horizon-3.7B)**</td>
      <td>Dense</td>
      <td>524,288</td>
      <td>BF16 · TP1</td>
    </tr>

    <tr>
      <td>**[7B](https://huggingface.co/IFM/K2-Horizon-7B)**</td>
      <td>Dense</td>
      <td>524,288</td>
      <td>BF16 · TP1</td>
    </tr>

    <tr>
      <td>**[32B](https://huggingface.co/IFM/K2-Horizon-32B)**</td>
      <td>Dense</td>
      <td>524,288</td>
      <td>BF16 · TP2</td>
    </tr>

    <tr>
      <td>**[MoVA-36B-A4B](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B)**</td>
      <td>MoE + MoVA</td>
      <td>524,288</td>
      <td>BF16 · TP2</td>
    </tr>

    <tr>
      <td>**[375B-A23B](https://huggingface.co/IFM/K2-Horizon-375B-A23B)**</td>
      <td>MoE</td>
      <td>524,288</td>
      <td>BF16 · TP8</td>
    </tr>
  </tbody>
</table>

> [!NOTE]
>
> All six checkpoints use the Apache License 2.0.

**Resources:** [K2 Horizon collection](https://huggingface.co/collections/IFM/k2-horizon) · [K2 Horizon 375B-A23B model card](https://huggingface.co/IFM/K2-Horizon-375B-A23B).

## 2. Configuration tips

* **Hardware and precision:** The current matrix covers NVIDIA H200 GPUs and BF16 checkpoints only.
* **Parallelism:** Use TP1 for 0.9B, 3.7B, and 7B; TP2 for 32B and MoVA-36B-A4B; and TP8 for 375B-A23B. Each recipe stays within one node.
* **Attention:** The verified base recipes select FlashAttention-3 with `--attention-backend fa3`.
* **Revisions:** The recipes pin revisions `6fbc54f7e4dc9d9cd671351f51498a1958dd2a01` (0.9B), `fe504ef19c7efa3120a9352bcfa180aa133946de` (3.7B), `85d46bbaf6ecd844a8ef61991f6e492d4faf4179` (7B), `eff3d24a3a40360c6fd1a1dd5723ce6329d93484` (32B), `e5c131d4d0ae196f5041284a0f9262d713083901` (MoVA-36B-A4B), and `70d7c58c1f8a2fc523c861c2cf7d2d339d7a8a0c` (375B-A23B).
* **Parsers:** The base recipes enable reasoning with `--reasoning-parser k2_horizon`. Enable `--tool-call-parser k2_horizon` from the **Parsers** card in the [Playground](#playground) when you need tool calling.
* **MoVA-36B-A4B router provenance:** The MoVA-36B-A4B checkpoint's source xLLM router used two GEMM partitions. `--json-model-override-args '{"xllm_source_router_gemm_partitions":2}'` preserves its BF16 partial-GEMM rounding and FP32 reduction order; SGLang does not infer this source topology from runtime TP.
* **375B-A23B loading:** The 375B-A23B recipe disables multithreaded weight loading with `--model-loader-extra-config '{"enable_multithread_load":false}'`.
* **Playground features:** TP, EP, NGRAM, PD disaggregation, and HiCache are optional, unverified command overrides. The tool-call parser was functionally exercised separately.
* **Deliberate omissions:** Context parallelism is unsupported or unvalidated for xLLM. DP-attention and alternate MoE backend selectors are not exposed until they are validated.
* **Validation:** The serving configuration for all six recipes completed native speed and full GSM8K evaluation on H200 using the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654).
* **Benchmark provenance:** The results below were collected at revisions `9b9ec1f7e17f62ed218df542687a144116219d84` (0.9B), `c177771836a4c460743c00002c22483f6f18d1eb` (3.7B), `69ada542b68fe13d767479db2ab9421baff88681` (7B), `e1fd0277713e4eefcd3416348fd6fedacf7f2392` (32B), `16d20c739c687c08423422d1a2fbba6c529014cd` (MoVA-36B-A4B), and `12812264242a14dce44aa7ae27f931ff4584bcbf` (375B-A23B). The 375B-A23B benchmark launched a directly mounted checkpoint; the other five launches used materialized Hugging Face snapshots.

## 3. Benchmark results

Speed was measured with native `sglang.benchmark.serving` using fixed random token IDs: 8,192 input tokens, 1,024 output tokens, 64 warmup requests, a cache flush, and seed `20260901`. The latency run used 32 prompts at concurrency 1. The throughput run used 256 prompts at concurrency 64 for TP1, 32 for TP2, and 8 for TP8. TTFT and TPOT are medians within each launch; the benchmark cards show the arithmetic mean across the two launches. Throughput per GPU is `(total input tokens + total output tokens) / duration / TP`.

GSM8K used the full 1,319-example test split with `sgl-eval`, 32 client threads, a 32,768-token output limit, temperature 0, top-p 0.95, seed 0, and high reasoning effort. The mean and both independent server-launch scores are shown below. All requests completed without errors. Responses that reached the output limit remain in the score: truncation was 6.90% in both 0.9B launches, 4.09% and 4.70% for 3.7B, and at most 1.14% for the other checkpoints.

| Checkpoint | Launch 1 | Launch 2 | Mean |
| - | -: | -: | -: |
| 0.9B | 85.22% | 85.29% | 85.25% |
| 3.7B | 92.12% | 91.89% | 92.00% |
| 7B | 95.30% | 94.47% | 94.88% |
| 32B | 95.75% | 96.21% | 95.98% |
| MoVA-36B-A4B | 95.45% | 94.84% | 95.15% |
| 375B-A23B | 95.53% | 95.60% | 95.56% |

## 4. Advanced usage

### 4.1 Reasoning

The base recipes include `--reasoning-parser k2_horizon`. The parser returns thinking in `reasoning_content` and the final answer in `content`.

The example output below was captured with the 3.7B checkpoint and both parser flags enabled.

> **Reasoning example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> response = client.chat.completions.create(
>     model="IFM/K2-Horizon-3.7B",
>     messages=[{"role": "user", "content": "Hello"}],
> )
>
> message = response.choices[0].message
> reasoning = getattr(message, "reasoning_content", None)
> if reasoning is None:
>     reasoning = (message.model_extra or {}).get("reasoning_content")
> print("Reasoning:", reasoning)
> print("Answer:", message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: We have a user who just said "Hello". We need to respond. As an AI, we should be friendly, perhaps ask how can we help. If says "Hello" alone, respond with greeting and offer assistance. Potentially ask what they need. There's no special instruction; just typical.
>
> But maybe we must think about the context: The user gave just a greeting; we should respond with a greeting and ask how we can help. Could also ask if they want anything specific.
>
> We should be concise but friendly: "Hello! How can I assist you today?" Provide some content.
>
> But maybe the user expects something else. But in this scenario, we just reply with a greeting.
>
> Thus answer: "Hello! What can I do for you?" Etc.
>
> Check any style guidelines: We want to keep it short or perhaps ask follow-up. The user didn't ask any question, just saying hello. Could also respond: "Hello! ..."
>
> Thus arguably the answer: "Hi there! How can I help you today?" Let's do that.
>
> But also maybe we could ask if they need help with something specific. That's appropriate.
>
> Thus the final answer: "Hello! How can I assist you today?" Possibly add friendly banter.
>
> We'll respond.
>
> Answer:
> Hello! 👋 How can I help you today?
> ```

### 4.2 Tool calling

Enable **Tool Call Parser** in the **Parsers** card of the [Playground](#playground) to add `--tool-call-parser k2_horizon`. The parser exposes K2 tool output as OpenAI-compatible `message.tool_calls`.

The example output below was captured with the 32B checkpoint and both parser flags enabled.

> **Tool-calling example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Return weather for a city.",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"city": {"type": "string"}},
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="IFM/K2-Horizon-32B",
>     messages=[{"role": "user", "content": "Call get_weather for Paris."}],
>     tools=tools,
>     tool_choice={"type": "function", "function": {"name": "get_weather"}},
>     max_tokens=1024,
> )
>
> message = response.choices[0].message
> reasoning = getattr(message, "reasoning_content", None)
> if reasoning is None:
>     reasoning = (message.model_extra or {}).get("reasoning_content")
> print("Reasoning:", reasoning)
> print("Content:", message.content or "")
> for tool_call in message.tool_calls or []:
>     print(f"Tool call: {tool_call.function.name}({tool_call.function.arguments})")
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: Okay, the user wants me to call the get_weather function for Paris. Let me check the tools available. There's a function called get_weather that takes a city parameter. The required parameter is city, and it's a string. So I need to make sure to pass "Paris" as the city. I don't see any other parameters needed. The user didn't mention any other details, so I'll just use the city name. Let me structure the tool call correctly. The function name is get_weather, and the arguments should be a JSON object with "city": "Paris". I need to make sure the JSON is properly formatted. Alright, that should do it.
>
> Content:
> Tool call: get_weather({"city": "Paris"})
> ```

### 4.3 HiCache

For large checkpoints or repeated long-context prefixes, enable **HiCache** in the [Playground](#playground), then choose a storage backend and write policy. Treat the generated override as unverified until it is exercised on your storage stack.


# IQuest-Q1
Source: https://docs.sglang.io/cookbook/autoregressive/IQuestLab/IQuest-Q1

Deploy IQuest-Q1 with SGLang — BF16 recipes for the 320B Mixture-of-Experts model on 8× NVIDIA H200, with optional MTP speculative decoding.

## Deployment

> **Install SGLang**
>
> IQuest-Q1 needs an SGLang build that includes IQuest-Q1 support. The nightly image carries it:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> For other methods, see the [official SGLang installation guide](../../../docs/get-started/install).

Download the checkpoint first; the MTP draft lives in its `mtp/` subdirectory:

```bash Command theme={null}
hf download IQuestLab/IQuest-Q1 --local-dir /model/IQuest-Q1
```

Pick a recipe to generate the launch command. **Low-Latency** enables MTP speculative decoding; **High-Throughput** serves the target model alone.

## 1. Model Introduction

**IQuest-Q1** is a Mixture-of-Experts language model from IQuestLab with about 320B total parameters and 15B active per token. It has 88 layers that interleave three sliding-window attention layers (4,096-token window) with one full-attention layer, uses learned attention sinks, and routes each token to 8 of 256 experts. It ships with a single-layer MTP draft that SGLang applies recursively for speculative decoding.

**Resources:** [HuggingFace](https://huggingface.co/IQuestLab/IQuest-Q1).

## 2. Configuration Tips

* **Hardware.** IQuest-Q1 requires the FA3 attention backend for both the target and the MTP draft, because the attention sink is applied from the FA3 softmax LSE. SGLang builds FA3 for Hopper, so Blackwell GPUs are not supported. The BF16 weights take about 640 GB, which fits 8× H200 with tensor parallelism 8.
* **Reasoning.** Thinking is on by default. Pass `"chat_template_kwargs": {"thinking": false}` to turn it off for a request. The `iquest_q1` reasoning parser returns the thinking in `reasoning_content`.
* **Tool calling.** The `iquest_q1` tool-call parser supports `tool_choice` values `auto`, `required`, and a named function.
* **MTP depth.** The single draft layer runs 5 sequential steps, one token each, and each round verifies 6 tokens. Keep `--speculative-num-draft-tokens` equal to `--speculative-num-steps` + 1.


# LLaDA 2.1
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/LLaDA-2.1



## 1. Model Introduction

[LLaDA 2.1](https://github.com/inclusionAI/LLaDA2.X) is a series of large-scale discrete diffusion language models (dLLMs) developed by the InclusionAI team at Ant Group. Unlike traditional autoregressive models that generate text left-to-right one token at a time, LLaDA 2.1 uses a diffusion-based approach — drafting tokens in parallel and refining them through iterative denoising, enabling self-correction during generation.

**Key Features:**

* **Token Editing (T2T + M2T)**: Combines Mask-to-Token (M2T) and Token-to-Token (T2T) editing, allowing the model to not only unmask tokens but also revise already-generated tokens mid-flight
* **Dual Decoding Modes**: Speed Mode (S) for maximum throughput with T2T refinement, and Quality Mode (Q) for conservative thresholds and higher benchmark scores
* **MoE Architecture**: Both variants use Mixture-of-Experts architecture for efficient scaling
* **First Large-Scale RL for dLLMs**: Implements the first reinforcement learning framework specifically designed for diffusion language models, improving reasoning and instruction-following
* **Lightning-Fast Decoding**: Up to 892 tokens/s on HumanEval+ for the 100B model

**Available Models:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Parameters</th>
      <th>Architecture</th>
      <th>Context Length</th>
      <th>HuggingFace</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**LLaDA2.1-mini**</td>
      <td>16B</td>
      <td>MoE (20 layers, 16 attention heads)</td>
      <td>32,768 tokens</td>
      <td>[inclusionAI/LLaDA2.1-mini](https://huggingface.co/inclusionAI/LLaDA2.1-mini)</td>
    </tr>

    <tr>
      <td>**LLaDA2.1-flash**</td>
      <td>100B</td>
      <td>MoE</td>
      <td>32,768 tokens</td>
      <td>[inclusionAI/LLaDA2.1-flash](https://huggingface.co/inclusionAI/LLaDA2.1-flash)</td>
    </tr>
  </tbody>
</table>

**License:**

Apache 2.0. Please refer to the [official LLaDA2.X repository](https://github.com/inclusionAI/LLaDA2.X) for details.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and decoding mode. SGLang supports serving LLaDA-2.1 on NVIDIA H100, H200, B200, and AMD MI300X, MI325X, MI355X GPUs.

Run this Command:

<pre />

### 3.2 Configuration Tips

**dLLM-Specific Parameters:**

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Recommended Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dllm-algorithm`</td>
      <td>Diffusion decoding algorithm</td>
      <td>`JointThreshold`</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>Required for LLaDA model loading</td>
      <td>Always enabled</td>
    </tr>

    <tr>
      <td>`--mem-fraction-static`</td>
      <td>Static memory fraction for KV cache</td>
      <td>`0.8`</td>
    </tr>

    <tr>
      <td>`--max-running-requests`</td>
      <td>Maximum concurrent requests</td>
      <td>`1` (for best quality)</td>
    </tr>

    <tr>
      <td>`--attention-backend`</td>
      <td>Attention computation backend</td>
      <td>`flashinfer`</td>
    </tr>
  </tbody>
</table>

**Decoding Mode Comparison:**

<table>
  <thead>
    <tr>
      <th>Mode</th>
      <th>Threshold</th>
      <th>Speed</th>
      <th>Quality</th>
      <th>Best For</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Quality Mode (Q)**</td>
      <td>Conservative</td>
      <td>Moderate</td>
      <td>Higher benchmark scores</td>
      <td>Accuracy-critical tasks</td>
    </tr>

    <tr>
      <td>**Speed Mode (S)**</td>
      <td>Aggressive</td>
      <td>Very fast, relies on T2T editing</td>
      <td>Slightly lower</td>
      <td>Throughput-critical tasks</td>
    </tr>
  </tbody>
</table>

**Hardware Requirements:**

* **LLaDA2.1-mini (16B)**: \~47 GB VRAM, runs on a single GPU (TP=1)
* **LLaDA2.1-flash (100B)**: Requires multi-GPU setup (TP=4 on H100/H200, TP=2 on B200)

## 4. Model Invocation

### 4.1 Deployment

Start the server using the command generated above, for example:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path inclusionAI/LLaDA2.1-mini \
  --dllm-algorithm JointThreshold \
  --tp 1 \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --max-running-requests 1 \
  --attention-backend flashinfer \
  --host 0.0.0.0 \
  --port 8000
```

### 4.2 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

**Simple Completion Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="inclusionAI/LLaDA2.1-mini",
    messages=[
        {"role": "user", "content": "Explain what a diffusion language model is in simple terms."}
    ],
    max_tokens=1024
)

print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
Sure! Let's break it down in simple terms.

A **diffusion language model** is a type of artificial intelligence that learns to generate text—like sentences, stories, or emails—by studying a lot of written text.

Here’s how it works, using a simple real-life analogy:

Imagine you have a big book full of stories. A diffusion language model is trying to learn how to write a new story. Instead of being told the rules, it starts by looking at all the words in the book and trying to understand how words usually go together.

Now, think of the process like this:

1. **Start with random noise**: The model begins with a completely random set of words (like a scribble on paper).
2. ** ** "clean up" the noise**: It gradually "denoises" the noise by turning it into meaningful text, word by word, based on what it learned learned from the book.
3. **Learn from patterns**: As it does this, it learns patterns—like how words often follow each other, or how sentences start.
4. **Generate new text**: Once it’s learned the patterns, it can create new, coherent sentences or stories by starting from a and and building it up word by word.

So, the "diffusion" part comes from the idea of going from random noise to clear, meaningful text—like turning a scribble into a full story.

In short:
A diffusion language model is an AI that learns to write text by reading lots of books and gradually turning random noise into coherent, meaningful sentences based on what it learned.
```

### 4.3 Advanced Usage

#### 4.3.1 Streaming

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="inclusionAI/LLaDA2.1-mini",
    messages=[
        {"role": "user", "content": "Write a Python function to compute the Fibonacci sequence."}
    ],
    max_tokens=2048,
    stream=True
)

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

````text Output theme={null}
Here are several ways to implement the Fibonacci sequence in Python:

## 1. Recursive Approach (Simple but Inefficient)

```python
def fibonacci_recursive(n):
    """
    Compute the nth Fibonacci number using recursion.

    Args:
        n (int): The position in the Fibonacci sequence (0-indexed)

    Returns:
        int: The nth Fibonacci number

    Raises:
        ValueError: If n is negative
    """
    if n < 0:
        raise ValueError("n must be non-negative")

    if n <= 1:
        return n

    return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)

# Example usage
print(fibonacci_recursive(10))  # Output: 55
```

## 2. Iterative Approach (Efficient)
...
````

#### 4.3.2 Code Generation

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="inclusionAI/LLaDA2.1-mini",
    messages=[
        {"role": "user", "content": "Write a Python function that checks if a string is a palindrome. Include docstring and test cases."}
    ],
    max_tokens=2048
)

print(response.choices[0].message.content)
```

**Output Example:**

````text Output theme={null}
```python
def is_palindrome(s):
    """
    Check if a string is a palindrome.

    A palindrome is a word, phrase, or sequence that reads the same backward as forward.
    This function ignores case, spaces, punctuation, and non characters characters.

    Args:
        s (str): The string to check

    Returns:
        bool: True if the string is a palindrome, False otherwise

    Examples:
        >>> is_palindrome("racecar")
        True
        >>> is_palindrome("A man a plan a canal Panama")
        True
        >>> is_palindrome("race a car")
        False
        >>> is_palindrome("")
        True
        >>> is_palindrome("a")
        True
    """
    # Remove non-alphanumeric characters and convert to lowercase
    cleaned = ''.join(char.lower() for char in s if char.isalnum())

    # Check if the cleaned string reads the same forwards and backwards
    return cleaned == cleaned[::-1]

# Test cases
def test_is_palindrome():
    """Test the is_palindrome function with various inputs."""

    # Test basic palindromes
    assert is_palindrome("racecar") == True
    assert is_palindrome("level") == True
    assert is_palindrome("madam") == True
    assert is_palindrome("radar") == True

    # Test palindromes with spaces and punctuation
    assert is_palindrome("A man a plan a canal Panama") == True
    assert is_palindrome("race a car") == False
    assert is_palindrome("Was it a car or a cat I saw?") == True
    assert is_palindrome("Madam, I'm Adam") == True

    # Test edge cases
    assert is_palindrome("") == True
    assert is_palindrome("a") == True
    assert is_palindrome("A") == True
    assert is_palindrome("Aa") == True

    # Test non-palindromes
    assert is_palindrome("hello") == False
    assert is_palindrome("world") == False
    assert is_palindrome("python") == False

    # Test single characters
    assert is_palindrome("1") == True
    assert is_palindrome("1") == True

    print("All tests passed!")

# Run the tests
if __name__ == "__main__":
    # Example usage
    print("Testing isalindrome function:")
    print(f"'racecar' {is_palindrome('racecar')}")
    print(f"'A man a plan a canal Panama': {is_palindrome('A man a plan a canal Panama')}")
    print(f"'race a car': {is_palindrome('race a car')}")
    print(f"'hello': {is_palindrome('hello')}")

    # Run tests
    test_is_palindrome()
```

This implementation includes:

1. **Comprehensive function** `is_palindrome()` that:
   - Ignores case by converting to lowercase
   - Removes all non-alphanumeric characters (spaces, punctuation, etc.)
   - Uses string slicing (`[::-1]`) to reverse the string

2. **Detailed docstring** explaining:
   - What the function does
   - How it works
   - Return value
   - Examples of usage

3. **Extensive test cases** covering:
   - Basic palindromes
   - Palindromes with spaces and punctuation
   - Edge cases (empty string, single character)
   - Non-palindromes
   - Mixed case scenarios

4. **Test function** that uses assertions to verify the function works correctly

The function efficiently handles real-world palindrome checking by ignoring case, spaces, and punctuation, making it suitable for phrases like "A man a plan a canal Panama".
````

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 (4x)
* SGLang Version: 0.5.8+

#### 5.1.1 LLaDA2.1-mini

**Model Deployment:**

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path inclusionAI/LLaDA2.1-mini \
  --dllm-algorithm JointThreshold \
  --tp 1 \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --max-running-requests 1 \
  --attention-backend flashinfer
```

* Latency Benchmark

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model inclusionAI/LLaDA2.1-mini \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Latency Result**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  9.90
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    3433
Request throughput (req/s):              1.01
Input token throughput (tok/s):          616.26
Output token throughput (tok/s):         426.26
Peak output token throughput (tok/s):    1010.00
Peak concurrent requests:                3
Total token throughput (tok/s):          1042.53
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   988.87
Median E2E Latency (ms):                 655.27
P90 E2E Latency (ms):                    1952.50
P99 E2E Latency (ms):                    2932.19
---------------Time to First Token----------------
Mean TTFT (ms):                          152.74
Median TTFT (ms):                        150.37
P99 TTFT (ms):                           229.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.16
Median TPOT (ms):                        2.08
P99 TPOT (ms):                           3.72
---------------Inter-Token Latency----------------
Mean ITL (ms):                           2.10
Median ITL (ms):                         1.99
P95 ITL (ms):                            4.03
P99 ITL (ms):                            6.34
Max ITL (ms):                            26.59
==================================================
```

* Throughput Benchmark

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model inclusionAI/LLaDA2.1-mini \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* **Throughput Result**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  467.74
Total input tokens:                      249831
Total input text tokens:                 249831
Total generated tokens:                  252662
Total generated tokens (retokenized):    189717
Request throughput (req/s):              1.07
Input token throughput (tok/s):          534.12
Output token throughput (tok/s):         540.17
Peak output token throughput (tok/s):    1753.00
Peak concurrent requests:                105
Total token throughput (tok/s):          1074.30
Concurrency:                             90.77
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   84912.27
Median E2E Latency (ms):                 86564.26
P90 E2E Latency (ms):                    110567.26
P99 E2E Latency (ms):                    114303.38
---------------Time to First Token----------------
Mean TTFT (ms):                          83920.39
Median TTFT (ms):                        85669.54
P99 TTFT (ms):                           112969.91
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.67
Median TPOT (ms):                        1.65
P99 TPOT (ms):                           4.43
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.69
Median ITL (ms):                         1.46
P95 ITL (ms):                            3.96
P99 ITL (ms):                            4.84
Max ITL (ms):                            92.08
==================================================
```

#### 5.1.2 LLaDA2.1-flash

**Model Deployment:**

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path inclusionAI/LLaDA2.1-flash \
  --dllm-algorithm JointThreshold \
  --tp 4 \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --max-running-requests 1 \
  --attention-backend flashinfer
```

* Latency Benchmark

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model inclusionAI/LLaDA2.1-flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Latency Result**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  14.46
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    3276
Request throughput (req/s):              0.69
Input token throughput (tok/s):          421.79
Output token throughput (tok/s):         291.75
Peak output token throughput (tok/s):    676.00
Peak concurrent requests:                3
Total token throughput (tok/s):          713.53
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1445.16
Median E2E Latency (ms):                 968.06
P90 E2E Latency (ms):                    3101.86
P99 E2E Latency (ms):                    4208.49
---------------Time to First Token----------------
Mean TTFT (ms):                          231.63
Median TTFT (ms):                        242.67
P99 TTFT (ms):                           341.33
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.04
Median TPOT (ms):                        2.79
P99 TPOT (ms):                           5.33
---------------Inter-Token Latency----------------
Mean ITL (ms):                           3.05
Median ITL (ms):                         2.41
P95 ITL (ms):                            7.25
P99 ITL (ms):                            8.27
Max ITL (ms):                            29.27
==================================================
```

* Throughput Benchmark

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model inclusionAI/LLaDA2.1-flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* **Throughput Result**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  671.85
Total input tokens:                      249831
Total input text tokens:                 249831
Total generated tokens:                  252662
Total generated tokens (retokenized):    177961
Request throughput (req/s):              0.74
Input token throughput (tok/s):          371.85
Output token throughput (tok/s):         376.07
Peak output token throughput (tok/s):    1521.00
Peak concurrent requests:                103
Total token throughput (tok/s):          747.92
Concurrency:                             91.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   122658.36
Median E2E Latency (ms):                 125265.55
P90 E2E Latency (ms):                    159554.07
P99 E2E Latency (ms):                    165174.88
---------------Time to First Token----------------
Mean TTFT (ms):                          121009.17
Median TTFT (ms):                        124437.80
P99 TTFT (ms):                           163579.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.73
Median TPOT (ms):                        2.16
P99 TPOT (ms):                           7.13
---------------Inter-Token Latency----------------
Mean ITL (ms):                           2.38
Median ITL (ms):                         1.40
P95 ITL (ms):                            6.89
P99 ITL (ms):                            8.60
Max ITL (ms):                            176.78
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:8000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

**Results:**

```text Output theme={null}
Accuracy: 0.895
Invalid: 0.000
Latency: 100.552 s
Output throughput: 262.094 token/s
```


# Ling-2.5-1T
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-2.5-1T



## 1. Model Introduction

[Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T) is the latest flagship instant model in the Ling family. Thinking models raise the ceiling of intelligence, while instant models expand its reach by balancing efficiency and performance—making AGI not only more powerful, but also more accessible. Ling-2.5-1T delivers comprehensive upgrades across model architecture, token efficiency, and preference alignment, designed to bring universally accessible AI to a new level of quality.

**Key Features:**

* **Trillion-Scale Model**: 1T total parameters with 63B active parameters (up from 51B in the previous generation). Pre-training corpus expanded from 20T to 29T tokens. Leveraging an efficient hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention), the model delivers exceptionally high throughput while processing context lengths of up to 1M tokens.
* **Token Efficiency**: By introducing a composite reward mechanism combining "Correctness" and "Process Redundancy", Ling-2.5-1T further pushes the frontier of efficiency-performance balance in instant models. At comparable token efficiency levels, Ling-2.5-1T's reasoning capabilities significantly outperform its predecessor, approaching the level of frontier "thinking models" that typically consume \~4x the output tokens.
* **Preference Alignment**: Through refined alignment strategies—such as bidirectional RL feedback and Agent-based instruction constraint verification—Ling-2.5-1T achieves substantial improvements over the previous generation in preference alignment tasks, including creative writing and instruction following.
* **Agentic Capabilities**: Trained with Agentic RL in large-scale high-fidelity interactive environments, Ling-2.5-1T is compatible with mainstream agent platforms such as Claude Code, OpenCode, and OpenClaw. It achieves leading open-source performance on the general tool-calling benchmark, BFCL-V4.
* **Context Length**: 256K -> 1M (YaRN)

**Available Models:**

* **BF16**: [inclusionAI/Ling-2.5-1T](https://huggingface.co/inclusionAI/Ling-2.5-1T)

**License:** MIT

## 2. SGLang Installation

Ling-2.5-1T runs on the standard SGLang Docker image:

```bash Command theme={null}
# NVIDIA (H200 / B200 / GB200 / GB300)
docker pull lmsysorg/sglang:latest
```

For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

Ling-2.5-1T is also supported via the **nightly PyPI builds**. See the [SGLang Installation (PyPI)](../../../docs/get-started/install) guide for setup instructions.

## 3. Model Deployment

Ling-2.5-1T is a trillion-parameter BF16 model that requires multi-node deployment (at least 2 nodes). Use the configuration selector below to generate the deployment command for your hardware platform.

Run this Command:

<pre />

### Configuration Tips

* The `--trust-remote-code` flag is required for this model due to custom modeling code.
* `--tp-size` can be set to a maximum of 8 for this model. If you have more GPUs available, increase `--pp-size` to scale across additional nodes.
* Adding `--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}'` enables faster model loading.
* On H200/GB200/GB300 with 2-node deployment, `--mem-frac 0.95` is required to avoid OOM since the model occupies most of the GPU memory. For better throughput, consider 4-node deployment (ref [model card](https://huggingface.co/inclusionAI/Ling-2.5-1T#run-inference) for more details).

## 4. Model Invocation

### 4.1 Basic Usage

For example, launch the server on 2 H200 nodes:

```bash Command theme={null}
export MASTER_IP=10.10.0.1 # The IP of Node 0
export PORT=30000
export DIST_PORT=50000

# Node 0:
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-2.5-1T \
--trust-remote-code \
--tp-size 8 \
--pp-size 2 \
--nnodes 2 \
--node-rank 0 \
--host 0.0.0.0 \
--port ${PORT} \
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
--tool-call-parser qwen \
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
--mem-frac 0.95


# Node 1:
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-2.5-1T \
--trust-remote-code \
--tp-size 8 \
--pp-size 2 \
--nnodes 2 \
--node-rank 1 \
--dist-init-addr ${MASTER_IP}:${DIST_PORT} \
--tool-call-parser qwen \
--model-loader-extra-config '{"enable_multithread_load": "true","num_threads": 64}' \
--mem-frac 0.95
```

Once the server is running, send requests to the master node:

```bash Command theme={null}
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
```

Output:

```json Config theme={null}
{
  "id": "e82af153da844ee6aed7a27a3187f2f4",
  "object": "chat.completion",
  "created": 1771216764,
  "model": "auto",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is **Paris**.\n\n**Additional details:**\n*   It is the largest city in France.\n*   It is located in the north-central part of the country along the Seine River.\n*   Paris is often referred to as \"The City of Light\" (*La Ville Lumière*).",
        "reasoning_content": null,
        "tool_calls": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "matched_stop": 156895
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "total_tokens": 93,
    "completion_tokens": 68,
    "prompt_tokens_details": null,
    "reasoning_tokens": 0
  }
}
```

For more API usage examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Tool Calling Example

```bash Command theme={null}
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "inclusionAI/Ling-2.5-1T",
    "messages": [{"role": "user", "content": "Search for the latest news about AI"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "search",
        "description": "Search for information on the internet",
        "parameters": {
          "type": "object",
          "properties": {
            "query": {"type": "string", "description": "The search query"}
          },
          "required": ["query"]
        }
      }
    }],
    "tool_choice": "auto"
  }'
```

Output:

```json Config theme={null}
{
  "id": "b968e45c7d414f7482c8ffc0f9c6b688",
  "object": "chat.completion",
  "created": 1771216520,
  "model": "inclusionAI/Ling-2.5-1T",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "reasoning_content": null,
        "tool_calls": [
          {
            "id": "call_e75f711d8ad840ed9d382c9e",
            "index": 0,
            "type": "function",
            "function": {
              "name": "search",
              "arguments": "{\"query\": \"latest news about AI\"}"
            }
          }
        ]
      },
      "logprobs": null,
      "finish_reason": "tool_calls",
      "matched_stop": null
    }
  ],
  "usage": {
    "prompt_tokens": 173,
    "total_tokens": 196,
    "completion_tokens": 23,
    "prompt_tokens_details": null,
    "reasoning_tokens": 0
  }
}
```

## 5. Benchmark

### GSM8K

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Result

```text Output theme={null}
Accuracy: 0.960
Invalid: 0.000
Latency: 45.410 s
Output throughput: 560.642 token/s
```


# Ling-2.6
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-2.6



## 1. Model Introduction

The **Ling-2.6** family from inclusionAI is the next iteration of the Ling instant-model series. Continuing the architectural direction set by Ling-2.5, Ling-2.6 doubles down on **inference efficiency**, **token efficiency**, and **agent performance** — staying competitive with frontier instant models while being faster, leaner, and better suited for production agent workloads.

**Key Features:**

* **Hybrid Linear Attention**: A `1:7 MLA + Lightning Linear` hybrid built on top of a highly sparse MoE backbone. Compared with same-class SOTA models, Ling-2.6-flash shows up to \~4× higher prefill and decode throughput in long-context scenarios; Ling-2.6-1T is shipped in FP8 so it fits a single GB300 node with `--tp 4`.
* **Token Efficiency**: Trained with explicit token-efficiency objectives. On the full Artificial Analysis suite, Ling-2.6-flash uses only \~15M output tokens while remaining competitive — a meaningfully stronger intelligence-per-token profile than long-reasoning peers.
* **Agentic Capabilities**: Refined for tool use, multi-step planning, and long-horizon execution. Reaches SOTA-class results on **BFCL-V4**, **TAU2-bench**, **SWE-bench Verified**, **Claw-Eval**, and **PinchBench**, and is validated against Claude Code, Kilo Code, Qwen Code, Hermes Agent, and OpenClaw.
* **Long Context**: Native 128K, extendable to **256K (Ling-2.6-flash)** and **256K → 1M (Ling-2.6-1T via YaRN)**.

**Available Models:**

* **BF16**: [inclusionAI/Ling-2.6-flash](https://huggingface.co/inclusionAI/Ling-2.6-flash) — 104B total / 7.4B active
* **FP8 (E4M3)**: [inclusionAI/Ling-2.6-1T](https://huggingface.co/inclusionAI/Ling-2.6-1T) — \~1T total

**License:** MIT

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

### 3.1 Ling-2.6-flash

Ling-2.6-flash is a 104B/7.4B-active MoE that runs comfortably on a single 4-GPU node. Use the selector below to generate the launch command for your hardware.

Run this Command:

<pre />

#### Configuration Tips

* `--trust-remote-code` is required (custom `BailingMoeV2_5ForCausalLM` modeling code).
* `--tp-size 4` is the reference layout. On 4× H20-3e the model reaches \~340 tokens/s decode at TP=4, batch 32.
* Native context is 128K. Enable YaRN (`--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 2.0, ...}}'`) to extend to 256K — the snippet does this for you.
* `--tool-call-parser qwen25` matches the model's `<tool_call>...</tool_call>` schema.
* The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6 is a controllable-reasoning model whose chat template defaults to `detailed thinking off`; the SGLang `qwen3` reasoning parser, in contrast, assumes default-thinking semantics and would mis-route normal output into `reasoning_content`. Only enable it if you specifically want `<think>...</think>` blocks split out — see [§4.3 Thinking Mode](#4-3-thinking-mode).
* **MTP (multi-token prediction)** is supported. Add `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-radix-cache-strategy extra_buffer` to enable it — see the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash#run-inference) for the full example.

### 3.2 Ling-2.6-1T

Ling-2.6-1T ships in **FP8 (E4M3)**, so unlike Ling-2.5-1T it fits a **single GB300 node with `--tp 4`**. On smaller GPUs (H200/B200), a 2-node deployment with `--pp-size 2` is required.

Run this Command:

<pre />

#### Configuration Tips

* `--trust-remote-code` is required for the custom modeling code.
* `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` significantly speeds up the multi-shard FP8 weight load (26 safetensors shards + an MTP layer).
* Use `--tool-call-parser qwen` for tool calling.
* The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6's chat template defaults to `detailed thinking off`, while SGLang's `qwen3` reasoning parser assumes default-thinking semantics — combining the two requires a per-request workaround for tool calls (see [§4.3 Thinking Mode](#4-3-thinking-mode)). Only enable `--reasoning-parser qwen3` if you specifically want `<think>...</think>` blocks split into `reasoning_content`.
* For 2-node deployments, set `MASTER_IP`, `PORT`, and `DIST_PORT` consistently across both nodes.

## 4. Model Invocation

For example, launch a Ling-2.6-1T server on a single GB300 node:

```bash Command theme={null}
sglang serve \
  --model-path inclusionAI/Ling-2.6-1T \
  --tp-size 4 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000 \
  --tool-call-parser qwen \
  --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'
```

### 4.1 Basic Usage

```bash Command theme={null}
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
```

Output:

```json Config theme={null}
{
  "id": "...",
  "object": "chat.completion",
  "model": "auto",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is **Paris**.",
        "reasoning_content": null,
        "tool_calls": null
      },
      "finish_reason": "stop"
    }
  ]
}
```

### 4.2 Tool Calling Example

```bash Command theme={null}
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Search for the latest news about AI"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "search",
        "description": "Search for information on the internet",
        "parameters": {
          "type": "object",
          "properties": {
            "query": {"type": "string", "description": "The search query"}
          },
          "required": ["query"]
        }
      }
    }],
    "tool_choice": "auto"
  }'
```

Output:

```json Config theme={null}
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_...",
            "type": "function",
            "function": {
              "name": "search",
              "arguments": "{\"query\": \"latest news about AI\"}"
            }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ]
}
```

### 4.3 Thinking Mode

Both Ling-2.6-flash and Ling-2.6-1T are **controllable-reasoning** models. Their chat template uses textual directives in the system message — `detailed thinking on` or `detailed thinking off` — to toggle thinking. The template **defaults to `detailed thinking off`** when neither phrase is present, and it does **not** read the Qwen3-style `enable_thinking` template variable.

#### Enabling thinking

Include `detailed thinking on` in the first system message:

```bash Command theme={null}
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [
      {"role": "system", "content": "detailed thinking on"},
      {"role": "user", "content": "If a box has 12 red balls and 8 blue balls, then 5 red balls are removed, how many balls remain?"}
    ]
  }'
```

If you already have a system prompt, append the directive on its own line:

```json theme={null}
{"role": "system", "content": "You are a helpful assistant.\ndetailed thinking on"}
```

When thinking is on, the model emits `<think>...</think>` blocks before its final answer. To get those split into `message.reasoning_content` automatically, also launch the server with `--reasoning-parser qwen3`.

#### Caveat: `--reasoning-parser qwen3` + tool calling

The SGLang `qwen3` reasoning parser was written for Qwen3, where models are **default-thinking** and clients opt out via `chat_template_kwargs.enable_thinking=false`. Ling-2.6 is the opposite — default-non-thinking, with toggling done in the system message. As a result, when the server is launched with **both** `--tool-call-parser qwen` and `--reasoning-parser qwen3`, every tool-call request must include `chat_template_kwargs.enable_thinking=false`, otherwise the parser routes the `<tool_call>...</tool_call>` block into `reasoning_content` instead of `message.tool_calls`:

```bash Command theme={null}
curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Search for the latest news about AI"}],
    "tools": [...],
    "tool_choice": "auto",
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

`enable_thinking` here is consumed by the SGLang reasoning parser, **not** by the chat template — Ling-2.6's template ignores it. For the simplest configuration, just omit `--reasoning-parser qwen3` and toggle thinking via the system message.

For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request).

## 5. Benchmark

### GSM8K (Ling-2.6-1T, GB300 × 4)

Reference run on a single GB300 node with `--tp 4`:

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

```text Output theme={null}
Accuracy: 0.9621 (1269 / 1319)
```

For Ling-2.6-flash, see the official numbers on the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash) (BFCL-V4, TAU2-bench, SWE-bench Verified, Claw-Eval, PinchBench, Artificial Analysis).


# Ling-3.0-flash
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-flash

Deploy Ling-3.0-flash with SGLang — a 124B hybrid KDA + MLA MoE in BF16, FP8, INT4, or MXFP4 on Hopper and Blackwell GPUs.

## Deployment

> **Install SGLang**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-Ling-3.0-flash
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + recipe to generate the launch command. Three serving strategies are covered:

* **Low-Latency** — fastest reply for a single user. Pick for chat. These recipes run speculative decoding: pick **NEXTN** (built-in MTP layer, no extra checkpoint) or **DSPARK** (external draft model) in the **Spec Decode** selector of the Deploy panel.
* **High-Throughput** — most tokens per second across many users. Best for batch jobs. These recipes turn speculative decoding off, since at saturation the draft/verify overhead outweighs the speedup.
* **HiCache + Mooncake** — writes reusable prefixes to Mooncake L3 storage. Start the Mooncake services in §3.3 before launching the generated server command.

## Playground

The Playground is where you experiment with **SGLang features beyond the documented matrix**. The Deploy panel above only emits the curated recipe combinations on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

Ling-3.0-flash is a hybrid-attention Mixture-of-Experts (MoE) language model from the BailingMoeV3 family. It interleaves Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers, on top of a fine-grained MoE feed-forward network. This keeps per-token inference cost close to a small model — **124B total parameters with only 5.1B active** — while retaining large-model capacity.

It is a hybrid-reasoning model with thinking enabled by default, and it supports structured tool calling. Native context length is 256K.

**Available Models:**

* **BF16**: [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) — 124B total / 5.1B active
* **FP8** (blockwise E4M3): [inclusionAI/Ling-3.0-flash-fp8](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp8)
* **INT4** (compressed-tensors W4A16): [inclusionAI/Ling-3.0-flash-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-int4)
* **MXFP4**: [inclusionAI/Ling-3.0-flash-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4)
* **DSPARK draft**: [inclusionAI/Ling-3.0-flash-dspark](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark) — 5-layer DSpark draft checkpoint for DSPARK speculative decoding

**License:** MIT

**Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-flash).

## 2. Configuration Tips

* BF16 tensor parallelism follows the GPU: `--tp 4` on 141 GB-class cards (H20-3e, H200) and 4-GPU Blackwell nodes (B200, GB300); `--tp 8` on 80 GB cards (H100, H800).
* The FP8 recipes pair `--tp` with a matching `--ep-size` (`--tp 4 --ep-size 4` on 4-GPU nodes, `--tp 8 --ep-size 8` on H100/H800). The checkpoint uses blockwise (128×128) E4M3 expert weights, so a pure tensor-parallel shard must satisfy `(768 / TP) % 128 == 0` — only TP2 qualifies; expert parallelism splits experts whole instead of by column, which lifts that restriction and uses the full node. SGLang detects the quantization format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed.
* INT4 uses compressed-tensors W4A16 experts. With the default MoE runner, SGLang selects Marlin on H200 and the graph-compatible Triton implementation on Blackwell; do not force a backend in the launch command.
* MXFP4 uses the native FlashInfer runner on H200 and B200. The checkpoint mixes MXFP4 routed experts with block-FP8 dense and shared projections; on B200, `--fp8-gemm-backend triton` avoids an unsupported FlashInfer FP8 tactic while the routed experts remain native MXFP4.
* `--reasoning-parser ling3` and `--tool-call-parser ling3` enable Ling-3.0-specific reasoning and structured tool-call parsing; toggle them in the **Parsers** card of the [Playground](#playground).
* Both the chat template and the `ling3` reasoning parser default to thinking on. A single request can turn it off with `"chat_template_kwargs": {"enable_thinking": false}` (see §3.1).
* The BF16/FP8 recipes use `--mem-fraction-static 0.8`; INT4/MXFP4 use `0.85`. These values reserve the headroom used by the validated graph-enabled runs.
* The checkpoint ships a built-in MTP layer (`num_nextn_predict_layers: 1`); enable it with `--speculative-algorithm NEXTN` — no separate draft model is needed. The Low-Latency recipes default to it via the **Spec Decode** selector; toggle it in the **Speculative Decoding** card of the [Playground](#playground).
* DSPARK is the alternative speculative path: it drafts with the external [inclusionAI/Ling-3.0-flash-dspark](https://huggingface.co/inclusionAI/Ling-3.0-flash-dspark) checkpoint instead of the built-in MTP layer. Pick **DSPARK** in the Deploy panel's **Spec Decode** selector (also available in the [Playground](#playground)); it composes `--speculative-algorithm DSPARK`, the draft path, and `--enable-linear-replayssm-spec`, the KDA verify path that folds per-draft intermediate states onto a fixed ring instead of snapshotting full states. The draft block size auto-infers from the checkpoint; tune it with `--speculative-dspark-block-size`. The draft's block size of 8 makes the verify window 9 tokens, and the KDA ReplaySSM ring must be a power of two at least twice the window — the recipe therefore pins `--linear-replayssm-cache-len 32`, since the 16 default fails startup validation. MXFP4 requires `--moe-runner-backend flashinfer_mxfp4`; the mixed checkpoint does not auto-select it.
* Native context is 256K; SGLang reads it from the checkpoint's `max_position_embeddings`, so no `--context-length` flag is needed.
* The **HiCache** card in the [Playground](#playground) exposes the validated Mooncake L3 path. It adds the hybrid-KDA scheduler and prefix-key settings together; see §3.3 for the required services.

## 3. Advanced Usage

### 3.1 Reasoning

Ling-3.0-flash thinks by default. With `--reasoning-parser ling3` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)), the chain-of-thought is returned in `message.reasoning_content` and the final answer in `message.content`:

> **Thinking-mode request**
>
> ```bash Command theme={null}
> curl -s http://localhost:30000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "inclusionAI/Ling-3.0-flash",
>     "messages": [{"role": "user", "content": "What is 15% of 240?"}]
>   }'
> ```

> **Example Output**
>
> ```json Output theme={null}
> {
>   "choices": [
>     {
>       "message": {
>         "role": "assistant",
>         "content": "15% of 240 = **36**",
>         "reasoning_content": "The user is asking a simple percentage calculation: 15% of 240. This is straightforward: 0.15 × 240 = 36.",
>         "tool_calls": null
>       },
>       "finish_reason": "stop"
>     }
>   ]
> }
> ```

> [!NOTE]
>
> Thinking is controlled by the chat template's `enable_thinking` kwarg and is on by default. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`.

### 3.2 Tool Calling

With `--tool-call-parser ling3` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)), structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`:

> **Tool-calling request**
>
> ```bash Command theme={null}
> curl -s http://localhost:30000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "inclusionAI/Ling-3.0-flash",
>     "messages": [{"role": "user", "content": "Search for the latest news about AI"}],
>     "tools": [{
>       "type": "function",
>       "function": {
>         "name": "search",
>         "description": "Search for information on the internet",
>         "parameters": {
>           "type": "object",
>           "properties": {
>             "query": {"type": "string", "description": "The search query"}
>           },
>           "required": ["query"]
>         }
>       }
>     }],
>     "tool_choice": "auto"
>   }'
> ```

> **Example Output**
>
> ```json Output theme={null}
> {
>   "choices": [
>     {
>       "message": {
>         "role": "assistant",
>         "content": "",
>         "reasoning_content": "The user wants me to search for the latest news about AI. I'll use the search tool with a query about the latest AI news.",
>         "tool_calls": [
>           {
>             "id": "call_0822dd418aa34254aa5b19e1",
>             "index": 0,
>             "type": "function",
>             "function": { "name": "search", "arguments": "{\"query\": \"latest news about AI 2025\"}" }
>           }
>         ]
>       },
>       "finish_reason": "tool_calls"
>     }
>   ]
> }
> ```

For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request).

### 3.3 HiCache with Mooncake

The HiCache recipes use Mooncake as L3 prefix storage. Start the metadata server, master, and storage client before you launch SGLang. The following command runs all three services from the same image in a separate container:

```bash Command theme={null}
docker run --rm --network host --ipc=host \
  lmsysorg/sglang:dev-Ling-3.0-flash \
  bash -lc '
    python3 -m mooncake.http_metadata_server --port 8290 &
    mooncake_master --port 50171 --metrics_port 9024 &
    exec mooncake_client \
      --host=127.0.0.1 \
      --port=50172 \
      --master_server_address=127.0.0.1:50171 \
      --metadata_server=http://127.0.0.1:8290/metadata \
      --protocol=tcp \
      --device_names= \
      --global_segment_size=4294967296 \
      --enable_http_server=true \
      --http_port=8291
  '
```

Then select **HiCache + Mooncake** in Deployment, or enable **HiCache** in the Playground. Docker commands use host networking so the SGLang container can reach these localhost services. You can change the master and metadata endpoints in the **Env** dialog.

This validated setup uses TCP, so `MOONCAKE_DEVICE=` and the client's `--device_names=` are intentionally empty. Set both to your actual device list only when you configure an RDMA deployment.

> [!NOTE]
>
> With the default `chunked_prefill_size` of 8192, a cold request writes through only when its uncached extend length fits in one chunk. A longer cold first request skips write-through for that influx; a repeat with the same prefix can hit the device radix cache and proceed normally.

> [!WARNING]
>
> For this hybrid KDA model, use the Mooncake L3 recipe shown here. Host-memory L2 eviction is not exposed because the KDA cache path is not currently compatible with it. Track the limitation in [issue #33713](https://github.com/sgl-project/sglang/issues/33713).

For storage sizing and backend details, see [HiCache best practices](/docs/advanced_features/hicache_best_practices).


# Ling-3.0-flash-VL
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-flash-VL

Deploy Ling-3.0-flash-VL with SGLang — a ~125B hybrid KDA + MLA MoE vision-language model (5.1B active) with image and video input, thinking mode, and 128K context on Blackwell and Hopper GPUs.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment. Ling-3.0-flash-VL support requires sglang with sgl-project/sglang#38526 (or newer); until it merges, install directly from the PR head:
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install "sglang @ git+https://github.com/sgl-project/sglang.git@refs/pull/38526/head#subdirectory=python"
> uv pip install torchcodec==0.16.0  # optional, preferred video-decode backend
> ```
>
> Video input needs `decord` or `torchcodec` in the environment; audio is not supported. Downloading the checkpoint requires `HF_TOKEN`.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-Ling-3.0-flash-VL
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware and quantization to generate the launch command. The checkpoint ships in BF16 with FP8, INT4, and FP4 variants, and each combination carries one **Balanced** recipe: BF16 uses `--tp 4` on 288 GB-class (GB300, B300) and 141 GB-class (B200, H200) GPUs and `--tp 8` on H100 (sized so the \~250 GB of BF16 weights fit with KV-cache headroom); FP8 uses `--tp 1` on GB300/B300/B200 and `--tp 2` on H200/H100; INT4 and FP4 use `--tp 1` everywhere. DGX Spark offers INT4 and FP4 only.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits the recipes on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

Ling-3.0-flash-VL is the vision-language member of the BailingMoeV3 family. Its language backbone is the [Ling-3.0-flash](/cookbook/autoregressive/InclusionAI/Ling-3.0-flash) hybrid-attention Mixture-of-Experts (MoE) model: 42 layers that interleave Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers, on top of a fine-grained MoE feed-forward network with 512 routed experts (8 active per token) plus a shared expert. A 27-layer vision encoder (0.4B parameters, 16-pixel patches with 2×2 spatial merging and paired frames for video) feeds image and video tokens into that backbone through a small MLP projector. In total the checkpoint holds **\~125B parameters with \~5.1B active per token**, so per-token inference cost stays close to a small model.

It is a thinking model: the chat template turns chain-of-thought on by default and exposes an `enable_thinking` switch. It supports structured tool calling. Inputs are text, images, and video; audio is not supported. Native context length is 128K tokens. Unlike Ling-3.0-flash, the VL checkpoint ships no built-in MTP draft layer, so there is no speculative-decoding recipe.

**Available Models:**

* **BF16**: [inclusionAI/Ling-3.0-flash-VL](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL) — \~125B total / \~5.1B active
* **FP8**: [inclusionAI/Ling-3.0-flash-VL-FP8](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-FP8) — 128×128 block quantization
* **INT4 (GPTQ)**: [inclusionAI/Ling-3.0-flash-VL-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-int4) — compressed-tensors, group size 32
* **FP4 (MXFP4)**: [inclusionAI/Ling-3.0-flash-VL-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp4) — e2m1 experts + FP8 e4m3 activations, Blackwell only

**License:** MIT

**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20` (from the checkpoint's `generation_config.json`; SGLang applies these defaults, so the samples below do not set them).

**Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL).

## 2. Configuration Tips

* `--trust-remote-code` is required. The checkpoint declares its config and multimodal processor through `auto_map`, and SGLang's native implementation loads the image/video preprocessing and the chat template from those files. Serving from the Hub repo or from a local snapshot needs no extra environment setup.
* BF16 weights take \~250 GB. The recipes use `--tp 4` on 288 GB-class (GB300, B300) and 141 GB-class (B200, H200) GPUs and `--tp 8` on H100, with the default memory pool. The 4×GB300 and 4×H200 cells are validated end to end; the rest are sized by the same rule and carry the unverified badge. Adjust TP in the **Attention Parallelism** card of the [Playground](#playground).
* HiCache is supported: `--enable-hierarchical-cache` attaches the hierarchical cache to the hybrid SSM state (verified with CPU offload; L3 storage backends such as Mooncake are untested). On shared hosts, set an explicit `--hicache-size <GB>` — the default ratio can request more host RAM than is free.
* Vision token budget: the encoder emits one token per 32×32-pixel block after merging, so a 1280×720 image costs about 880 tokens and the processor's `max_pixels` of 4,194,304 caps a single image at roughly 4,100 tokens. Video frames are paired before patching; the sample clip in §3.2 costs 4,224 tokens. Size `--chunked-prefill-size` and `--max-running-requests` with these counts in mind for image-heavy traffic.
* Parsers: the generated recipes include `--reasoning-parser auto --tool-call-parser auto` by default. Both resolve to `ling3` from the chat template (the server logs `Auto-detected --reasoning-parser as ling3 from chat template` and the same for the tool-call parser), so no model-specific parser name is needed; the explicit form `--reasoning-parser ling3 --tool-call-parser ling3` is equivalent. With the parsers on, reasoning is returned in `message.reasoning_content` and structured tool calls in `message.tool_calls`; without them, `message.content` carries the reasoning followed by `</think>` and the answer (the opening `<think>` tag sits in the prompt and is not echoed).
* Thinking is on by default (`enable_thinking` defaults to true in the chat template, and the `ling3` reasoning parser follows that default). Turn it off per request with `"chat_template_kwargs": {"enable_thinking": false}` (§3.3).
* Native context is 128K, and the generated recipes default to the 256K YaRN variant (`--context-length 262144` plus the `rope_scaling` override and `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1`), matching the model card. For the native 128K window, drop all three from the command.
* Audio content parts are rejected before preprocessing; send images as `image_url` and clips as `video_url` content parts (§3).
* The FP8 variant (`inclusionAI/Ling-3.0-flash-VL-FP8`) uses 128×128 block quantization, so plain TP must keep `768 ÷ tp` divisible by 128 (TP 1, 2, 3, or 6 — TP=4 fails to load). FP8 recipes default to a single GPU where the \~126 GB weights fit (`--tp 1` on 288 GB-class and B200, verified on GB300) or `--tp 2` on tighter cards. As a manual alternative on four GPUs, `--tp 4 --ep 4` keeps experts whole per rank (measured on 4×GB300). The BF16 checkpoint can also be quantized online with `--quantization fp8`; measured accuracy and speed for both paths are in the benchmark cards below.
* The INT4 variant (`inclusionAI/Ling-3.0-flash-VL-int4`) is a GPTQ 4-bit export (compressed-tensors, group size 32). It runs on all listed hardware with single-GPU (`--tp 1`) recipes; GB300, H200, and DGX Spark cells are verified, the rest carry the unverified badge. Shared-expert fusion is auto-disabled for the INT4 and FP4 checkpoints by the current dev image and by sglang main; older images need `--disable-shared-experts-fusion` explicitly.
* The FP4 variant (`inclusionAI/Ling-3.0-flash-VL-fp4`) is an MXFP4 export (e2m1 routed-expert weights with e8m0 scales, group 32, plus FP8 e4m3 dynamic activations) and runs on Hopper (SM90 CUTLASS W4A16) and Blackwell (SM100/SM120): H200, GB300, B300, B200, and DGX Spark. The `flashinfer_mxfp4` MoE backend is selected automatically on the `lmsysorg/sglang:dev-Ling-3.0-flash-VL` image and on sglang main; on builds that already include the VL integration but predate the auto-selection, add `--moe-runner-backend flashinfer_mxfp4` explicitly. GB300, H200, and DGX Spark cells are verified, the rest carry the unverified badge.

## 3. Advanced Usage

The examples below were run against a server launched with the recipe above plus `--reasoning-parser auto --tool-call-parser auto`, which resolves both parsers to `ling3` (§2). Reasoning therefore arrives in `message.reasoning_content` and the answer in `message.content`.

### 3.1 Image Understanding

Send images as OpenAI-style `image_url` content parts:

> **Image Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="inclusionAI/Ling-3.0-flash-VL",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
>             {"type": "text", "text": "Describe this image in one sentence."},
>         ],
>     }],
>     max_tokens=2048,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user wants a one-sentence description of this image. Let me look at the image carefully.
>
> The image shows a man wearing a yellow shirt (with some kind of text on it, probably "Sudsy" or similar) standing behind a yellow taxi SUV, ironing clothes on a portable ironing board set up at the rear of the vehicle. On the left side, another yellow taxi (sedan) is passing by on the street. The background is an urban streetscape with buildings lined up.
>
> I need to write this in one sentence. For example: "A man wearing a yellow shirt irons clothes on a portable ironing board attached to the back of a yellow taxi on a city street, while another taxi passes by."
>
> Let me make it concise yet descriptive. One sentence.
>
> "On a city street, a man in a yellow sweatshirt irons clothes on a portable ironing board mounted on the back of a yellow taxi SUV, while another yellow taxi drives past."
>
> This works. I'll return only the answer.
> Content: A man in a yellow shirt irons clothes on a portable ironing board set up at the back of a yellow taxi on a city street, while another yellow cab drives past.
> ```

### 3.2 Video Understanding

Clips go in as `video_url` content parts. The processor samples and pairs frames itself; no client-side frame extraction is needed:

> **Video Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY", timeout=900)
> resp = client.chat.completions.create(
>     model="inclusionAI/Ling-3.0-flash-VL",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "video_url", "video_url": {"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"}},
>             {"type": "text", "text": "What is happening in this video? Answer in two sentences."},
>         ],
>     }],
>     max_tokens=2048,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user wants me to describe what is happening in this video in two sentences.
>
> Looking at the video, I can see:
> - A man on stage (appears to be Steve Jobs) presenting something
> - He's wearing a black shirt and jeans
> - There's a large screen behind him showing a close-up of a hand reaching toward a pocket on blue jeans
> - Text appears on screen asking "Anyone know what this pocket is for?"
> - He pulls out a small white device (iPod Nano) from his pocket
> - The text says "I've always wondered that" and "well now we know" and "is the new iPod Nano"
> - He holds up the white iPod Nano device
>
> This is clearly Steve Jobs presenting the new iPod Nano at an Apple event. The small white device is the iPod Nano.
>
> I need to summarize this in exactly two sentences.
> Content: A presenter on stage pulls a small white device from his pocket while a large screen behind him displays a close-up image of a hand reaching toward a jeans pocket. He holds the device up to reveal it is the new iPod Nano, answering the on-screen text that questioned what the small pocket was designed for.
> ```

### 3.3 Thinking Mode

Thinking is on by default. Disable it for a single request through the chat template's `enable_thinking` kwarg; the model then answers directly and `reasoning_content` stays empty:

> **Thinking-off Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="inclusionAI/Ling-3.0-flash-VL",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
>             {"type": "text", "text": "Describe this image in one sentence."},
>         ],
>     }],
>     max_tokens=512,
>     extra_body={"chat_template_kwargs": {"enable_thinking": False}},
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: None
> Content: A man in a yellow sweatshirt irons a blue shirt on a folding board attached to the back of a parked yellow taxi SUV, while another yellow cab drives past on a city street.
> ```

### 3.4 Tool Calling

With the tool-call parser on, structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`. Send the tool result back as a `tool` message to get the final answer; on this thinking model the follow-up turn may put text in `reasoning_content` as well as `content`, so print both:

> **Tool-calling Example (Python)**
>
> ```python Example theme={null}
> import json
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city",
>         "parameters": {
>             "type": "object",
>             "properties": {"city": {"type": "string", "description": "City name"}},
>             "required": ["city"],
>         },
>     },
> }]
> messages = [{"role": "user", "content": "What's the weather like in Paris today?"}]
>
> resp = client.chat.completions.create(
>     model="inclusionAI/Ling-3.0-flash-VL",
>     messages=messages,
>     tools=tools,
>     tool_choice="auto",
>     max_tokens=2048,
> )
> msg = resp.choices[0].message
> print("finish_reason:", resp.choices[0].finish_reason)
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> for call in msg.tool_calls:
>     print("Tool call:", call.function.name, call.function.arguments)
>
> messages.append({
>     "role": "assistant",
>     "content": msg.content or "",
>     "tool_calls": [{"id": c.id, "type": "function",
>                     "function": {"name": c.function.name, "arguments": c.function.arguments}}
>                    for c in msg.tool_calls],
> })
> messages.append({
>     "role": "tool",
>     "tool_call_id": msg.tool_calls[0].id,
>     "content": json.dumps({"city": "Paris", "temperature_c": 18, "condition": "light rain"}),
> })
> resp = client.chat.completions.create(
>     model="inclusionAI/Ling-3.0-flash-VL",
>     messages=messages,
>     tools=tools,
>     max_tokens=2048,
> )
> msg = resp.choices[0].message
> print("finish_reason:", resp.choices[0].finish_reason)
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> finish_reason: tool_calls
> Reasoning: The user wants to know the current weather in Paris. I'll call the get_weather function with the city "Paris".
> Content: Let me check the weather in Paris for you right away!
> Tool call: get_weather {"city": "Paris"}
> finish_reason: stop
> Reasoning: None
> Content: Here's the current weather in Paris:
>
> - 🌡️ **Temperature:** 18°C
> - 🌧️ **Condition:** Light rain
>
> It looks like a mild day with some light rain. You might want to bring an umbrella if you're heading out! ☂️
> ```

For more API examples, see the [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) and the [Tool Parser guide](../../../docs/advanced_features/tool_parser).


# Ling-3.0-tiny
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny

Deploy Ling-3.0-tiny with SGLang — a compact ~7.9B total / ~1.2B active hybrid KDA + MLA MoE in BF16, FP8, or INT4, with thinking mode and tool calling.

## Deployment

> **Install SGLang**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-Ling-3.0-tiny
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + recipe to generate the launch command. One serving strategy is covered:

* **High-Throughput** — most tokens per second across many users. Best for batch jobs. Ling-3.0-tiny ships no built-in MTP draft layer (`num_nextn_predict_layers: 0`), so there is no NEXTN speculative-decoding recipe.

## Playground

The Playground is where you experiment with **SGLang features beyond the documented matrix**. The Deploy panel above only emits the curated recipe combinations on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

Ling-3.0-tiny is a compact hybrid-attention Mixture-of-Experts (MoE) language model from the BailingMoeV3 family — the small variant of [Ling-3.0-flash](/cookbook/autoregressive/InclusionAI/Ling-3.0-flash). It interleaves Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers on top of a fine-grained MoE feed-forward network, keeping per-token inference cost near a \~1B dense model — **\~7.9B total parameters with \~1.2B active** — while retaining large-model capacity.

It is a thinking model with chain-of-thought enabled by default, and it supports structured tool calling. Native context length is 128K. Unlike Ling-3.0-flash, it ships **no built-in MTP draft layer**, so it does not use NEXTN speculative decoding.

**Available Models:**

* **BF16**: [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) — \~7.9B total / \~1.2B active
* **FP8** (blockwise E4M3): [inclusionAI/Ling-3.0-tiny-fp8](https://huggingface.co/inclusionAI/Ling-3.0-tiny-fp8)
* **INT4** (compressed-tensors W4A16): [inclusionAI/Ling-3.0-tiny-int4](https://huggingface.co/inclusionAI/Ling-3.0-tiny-int4)

**License:** MIT

**Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-tiny).

## 2. Configuration Tips

* At \~7.9B total / 15.8 GB in BF16 (\~7.9 GB in FP8 and \~5.8 GB in INT4), a single GPU is plenty on every supported card. Tensor parallelism is only useful to raise aggregate KV-cache capacity for many long-context concurrent requests — add `--tp 2`/`--tp 4` to a multi-GPU serve directly.
* Use the dedicated `lmsysorg/sglang:dev-Ling-3.0-tiny` runtime image; it includes the compressed-tensors Hopper and Blackwell backends that INT4 needs.
* The FP8 checkpoint uses blockwise (128×128) E4M3 weights with dynamic activations, quantized from the BF16 model with attention projections, the dense MoE gate, and the lm\_head left in higher precision. SGLang detects the format from the checkpoint's `quantization_config`, so no explicit quantization flag is needed, and the same single-GPU recipe serves it.
* The INT4 checkpoint uses symmetric group-32 W4A16 routed experts. SGLang selects Marlin on Hopper and Triton WNA16 on Blackwell automatically; no explicit quantization or MoE backend flag is needed.
* Unlike Ling-3.0-flash (which pairs `--reasoning-parser ling3` / `--tool-call-parser ling3`), Ling-3.0-tiny uses `--reasoning-parser deepseek-r1` and `--tool-call-parser glm45` (its auto-detected template pairing) — the template wraps tool calls in `<tool_call>` blocks and emits an inline `...</think>` chain-of-thought. Toggle them in the **Parsers** card of the [Playground](#playground).
* Only `--model-path`, `--host`, and `--port` are needed. SGLang auto-resolves the context length (native 128K from `max_position_embeddings`), the attention backend, and `--mem-fraction-static` from the GPU and the CUDA-graph runtime, so the recipes leave them unset.
* The chat template defaults to thinking on. Turn it off per request with `"chat_template_kwargs": {"enable_thinking": false}` for direct answers without the `...</think>` block.
* Ling-3.0-tiny ships no built-in MTP draft layer (`num_nextn_predict_layers: 0`), so `--speculative-algorithm NEXTN` is not applicable.

## 3. Advanced Usage

### 3.1 Reasoning

With `--reasoning-parser deepseek-r1` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)), the chain-of-thought is returned in `message.reasoning_content` and the final answer in `message.content`:

> **Thinking-mode request**
>
> ```bash Command theme={null}
> curl -s http://localhost:30000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "inclusionAI/Ling-3.0-tiny",
>     "messages": [{"role": "user", "content": "What is 15% of 240?"}]
>   }'
> ```

> **Example Output**
>
> ```json Output theme={null}
> {
>   "choices": [
>     {
>       "message": {
>         "role": "assistant",
>         "content": "15% of 240 is **36**.\n\n**Calculation:** 0.15 × 240 = 36",
>         "reasoning_content": "The user is asking for 15% of 240. This is a simple percentage calculation.\n\n15% of 240 = 0.15 × 240 = 36\n\nLet me verify: 0.15 × 240 = 0.15 × 200 + 0.15 × 40 = 30 + 6 = 36. Yes, that's correct.",
>         "tool_calls": null
>       },
>       "finish_reason": "stop"
>     }
>   ]
> }
> ```

> [!NOTE]
>
> Thinking is controlled by the chat template's `enable_thinking` kwarg and is on by default. Disable it per request with `"chat_template_kwargs": {"enable_thinking": false}`.

### 3.2 Tool Calling

With `--tool-call-parser glm45` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)), structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`:

> **Tool-calling request**
>
> ```bash Command theme={null}
> curl -s http://localhost:30000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "inclusionAI/Ling-3.0-tiny",
>     "messages": [{"role": "user", "content": "Search for the latest news about AI"}],
>     "tools": [{
>       "type": "function",
>       "function": {
>         "name": "search",
>         "description": "Search for information on the internet",
>         "parameters": {
>           "type": "object",
>           "properties": {
>             "query": {"type": "string", "description": "The search query"}
>           },
>           "required": ["query"]
>         }
>       }
>     }],
>     "tool_choice": "auto"
>   }'
> ```

> **Example Output**
>
> ```json Output theme={null}
> {
>   "choices": [
>     {
>       "message": {
>         "role": "assistant",
>         "content": "Let me search for the latest news about AI for you.",
>         "reasoning_content": "The user wants me to search for the latest news about AI. I'll use the search tool to find recent AI news.",
>         "tool_calls": [
>           {
>             "id": "call_79b73a89696d4544ac6dd724",
>             "index": 0,
>             "type": "function",
>             "function": { "name": "search", "arguments": "{\"query\": \"latest AI news 2025\"}" }
>           }
>         ]
>       },
>       "finish_reason": "tool_calls"
>     }
>   ]
> }
> ```

For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request).


# Ling-3.1-flash
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.1-flash

Deploy Ling-3.1-flash with SGLang — FP8 hybrid KDA + MLA MoE recipes for H200 and GB300 GPUs, measured accuracy and decode speed, and Mooncake prefix caching.

## Deployment

> **Install SGLang**
>
> Use the Ling-3.1-flash image, which includes model support and the decode optimizations used by these recipes. The image supports multiple CPU architectures:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-ling-3.1-flash
> ```
>
> Select **Docker** in the command panel to launch the matching image. The **Python** command requires an environment with the same Ling-3.1-flash build. See the [installation guide](/docs/get-started/install#method-3-using-docker) for Docker setup.
>
> > [!NOTE]
> >
> > The first launch in a fresh container spends about five extra minutes compiling JIT kernels for the speculative-decoding graphs (about 4–5 minutes once, then about 30 seconds). Mount a persistent cache volume, for example `-v ~/.cache:/root/.cache`, to skip it on later starts.
>
> > [!NOTE]
> >
> > **AMD MI325X / MI350X:** the published ROCm images predate the ROCm gated-MLA fix that Ling-3.1-flash needs; without it the server fails while capturing decode CUDA graphs. Until a ROCm image includes the fix, build a thin image that switches the nightly's editable SGLang source to the Ling-3.1-flash branch (use the `mi35x` base for MI350X), then substitute it for the image in the generated Docker command:
> >
> > ```dockerfile Dockerfile theme={null}
> > FROM lmsysorg/sglang-rocm:v0.5.21-rocm724-mi30x-20261013
> > RUN cd /sgl-workspace/sglang \
> >  && git fetch https://github.com/sgl-project/sglang.git bailing-v3-day0 \
> >  && git checkout FETCH_HEAD
> > ```

Pick your hardware to generate the launch command:

* **Low-Latency** — TP-only (EP=1) with NEXTN speculative decoding for single-user chat and interactive requests. Verified on H200.
* **High-Throughput** — expert-parallel (EP) serving for batch and multi-user traffic: TP8 + EP8 on H200/H800/H100/B200-class nodes, TP4 + EP4 on GB300, TP8 + EP8 with AITER on AMD MI300X/MI325X/MI350X/MI355X (without speculative decoding). Verified on H200 and GB300; other platforms are unverified. AMD MI325X and MI350X recipes (TP8 + EP8 with AITER, without speculative decoding) are provided unverified; the MI325X accuracy in the benchmark card was measured by AMD.

For prefix-heavy multi-turn workloads, enable **HiCache** in the Playground after starting the Mooncake services in §3.3. The combination is verified: prefix hits return from Mooncake L3 on every TP rank.

## Playground

Experiment with SGLang features on top of the selected deployment. The speculative-decoding options show the checkpoint requirements; the multi-turn prefix-cache preset adds the paired scheduler flags and environment setting.

## 1. Model introduction

[Ling-3.1-flash](https://huggingface.co/inclusionAI/Ling-3.1-flash) is a hybrid-attention Mixture-of-Experts language model using the `BailingMoeV3ForCausalLM` architecture. Its 72 layers combine Kimi Delta Attention (KDA) with Multi-head Latent Attention (MLA), with 512 routed experts. The checkpoint uses FP8 E4M3 weights in 128×128 blocks with dynamic activation quantization.

The model supports reasoning and structured tool calling, with a context limit of 262,144 tokens.

**Checkpoint:** [inclusionAI/Ling-3.1-flash](https://huggingface.co/inclusionAI/Ling-3.1-flash). **License:** MIT.

The model has about 560B total parameters (512 routed experts with 8 active plus one shared expert per token, about 17B active), 72 layers, and FP8 block-quantized weights.

## 2. Configuration tips

* Use eight H200 GPUs with `--tp 8 --ep 8`, or four GB300 GPUs with `--tp 4 --ep 4`. SGLang detects FP8 from the checkpoint.
* On H200, enable `SGLANG_ENABLE_FP8_BLOCK_GEMV=1` and `SGLANG_ENABLE_FUSED_KDA_OUTPUT_QUANT=1`. Both are included in the generated commands; they are no-ops on GB300 (SM10x).
* Leave `--mem-fraction-static` at its default. SGLang auto-selects the `ling3` reasoning and tool-call parsers for this architecture; you can change the parser settings in the Playground.
* **NEXTN:** add `--speculative-algorithm NEXTN` for built-in MTP speculative decoding. Measured acceptance length is 2.61 on random tokens and 2.13 on real-text probes.
* **Long context:** the checkpoint supports 262,144 tokens natively. For up to 1,048,576 tokens, add the YaRN override from §3.4.
* **Topology:** the Low-Latency recipe runs TP-only (no `--ep-size`) and is clearly faster at batch size 1 (about 9-13% lower TPOT on H200, verified on both 2048/512 and 8192/1024 workloads). At concurrency 16-32 the two recipes deliver the same throughput within \~2%. TP-only relies on the loader's TP-requant support; it fails loudly at startup on builds without it.
* For prefix-heavy, multi-turn workloads, select **Cache fork and tail** in the Playground. It combines `--mamba-radix-cache-strategy extra_buffer`, `--chunked-prefill-size 4096`, and `SGLANG_SPLIT_PREFILL_AT_MAMBA_FORK=1`. GPU checks confirmed both fork and tail states are cached, saving about 576 tokens per subsequent turn in the measured workload.

### Playground support

* **Attention:** BailingKDA shards its projections on attention TP; DP-attention size must divide TP. Prefill CP is disabled because BailingKDA does not implement the required sequence gather and recurrent-state handling.
* **MoE:** Auto, DeepGEMM, and Triton use the FP8 MoE runner paths. EP must divide TP.
* **PD disaggregation:** the Playground exposes separate prefill and decode roles with Mooncake transfer. The runtime has hybrid recurrent-state transfer support, but this model's PD deployment has not been verified. Provision separate workers and route requests through a PD-aware gateway; select the role-specific command for each worker.
* **HiCache:** enabling Mooncake on the H200 base preserves NEXTN and adds direct page-first I/O, hybrid-KDA prefix settings, and TCP storage endpoints.

### Performance

On H200 TP8 with NEXTN and the decode gates enabled, batch-size-1 output throughput measures **269.2 tokens/s** (TP8 + EP8: 239.5 tokens/s). GB300 TP4 with NEXTN measures **175.3 tokens/s** (`TPOT 5.22 ms, accept 2.60`; TP4 + EP4: 160.0 tokens/s).

Acceptance length is the mean number of tokens accepted per speculative verification step, including the target bonus token:

<table>
  <thead>
    <tr>
      <th>Workload</th>
      <th>Acceptance length</th>
    </tr>
  </thead>

  <tbody>
    <tr><td>Random-token benchmark (bs=1)</td><td>2.61</td></tr>
    <tr><td>GSM8K smoke (HiCache serve)</td><td>2.29</td></tr>
    <tr><td>Full GSM8K (cumulative)</td><td>2.43</td></tr>
    <tr><td>Real-text probes (30 prompts)</td><td>2.13</td></tr>
  </tbody>
</table>

The image includes KDA normalization/quantization fusion, FP8 block GEMV, router-gate matrix-vector kernels, and a 512-expert top-k router.

> **Reproduce the H200 speed workload**
>
> Launch the H200 **Low-Latency** recipe, then run:
>
> ```bash Command theme={null}
> python3 -m sglang.bench_serving \
>   --backend sglang \
>   --host localhost --port 30000 \
>   --model inclusionAI/Ling-3.1-flash \
>   --dataset-name random \
>   --random-input-len 2048 --random-output-len 512 \
>   --random-range-ratio 1 \
>   --num-prompts 16 --max-concurrency 1 \
>   --flush-cache --seed 11
> ```
>
> For the saturation row in the benchmark widget, launch the H200 **High-Throughput** recipe instead and run:
>
> ```bash Command theme={null}
> python3 -m sglang.bench_serving \
>   --backend sglang \
>   --host localhost --port 30000 \
>   --model inclusionAI/Ling-3.1-flash \
>   --dataset-name random \
>   --random-input-len 8192 --random-output-len 1024 \
>   --random-range-ratio 1 \
>   --num-prompts 256 --max-concurrency 48 \
>   --flush-cache --seed 11
> ```

> **Reproduce GSM8K accuracy**
>
> ```bash Command theme={null}
> pip install sgl-eval
> sgl-eval run gsm8k \
>   --base-url http://localhost:30000/v1 \
>   --num-threads 32
> ```
>
> Evaluate all 1,319 questions and inspect completion reasons as well as accuracy. For the H200 NEXTN recipes, compare the stop rate with the measured 99.77% (Low-Latency) or 99.92% (High-Throughput); accuracy alone can hide generations that reach the token limit.

## 3. Advanced usage

### 3.1 Reasoning

The `ling3` parser returns reasoning in `message.reasoning_content` and the final answer in `message.content`. Thinking is enabled by default. To request a direct answer, pass `"chat_template_kwargs": {"enable_thinking": false}` in the request body, or `extra_body={"chat_template_kwargs": {"enable_thinking": False}}` with the OpenAI Python client.

> **Thinking-on request (Python)**
>
> ```python Python theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="Ling-3.1-flash",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
> )
> print(response.choices[0].message.reasoning_content)
> print(response.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> 15% of 240 = 0.15 × 240 = 36.
>
> Quick check: 10% of 240 = 24, 5% = 12, total 36. Correct.
> **36**
>
> Here's the calculation:
> - 15% = 0.15
> - 0.15 × 240 = 36
>
> Or break it down: 10% of 240 = 24, and 5% of 240 = 12, so 24 + 12 = **36**
> ```
>
> The first block is `reasoning_content` (59 tokens); the second is `content`. `finish_reason` is `stop`.

> **Thinking-off request (Python)**
>
> ```python Python theme={null}
> response = client.chat.completions.create(
>     model="Ling-3.1-flash",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     extra_body={"chat_template_kwargs": {"enable_thinking": False}},
> )
> print(response.choices[0].message.reasoning_content)  # None
> print(response.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> None
> 15% of 240 is **36**.
>
> Calculation:
> - 10% of 240 = 24
> - 5% of 240 = 12
> - 24 + 12 = **36**
>
> Or directly: 240 × 0.15 = **36**
> ```
>
> `reasoning_content` is `None` and `reasoning_tokens` is 0.

### 3.2 Tool calling

The `ling3` tool-call parser returns structured calls in `message.tool_calls`, with `finish_reason` set to `tool_calls`. Send tools through the OpenAI-compatible `tools` field and return each tool result with its matching `tool_call_id`. Read both `reasoning_content` and `content` on the follow-up assistant turn.

> **Tool-calling request and follow-up (Python)**
>
> ```python Python theme={null}
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "city": {"type": "string", "description": "City name, e.g. Beijing"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> import json
>
>
> def get_weather(city: str, unit: str = "celsius"):
>     # Stand-in for your real tool implementation.
>     return {"city": city, "temperature": 22, "condition": "sunny", "unit": unit}
>
>
> response = client.chat.completions.create(
>     model="Ling-3.1-flash",
>     messages=[{"role": "user", "content": "What's the weather like in Hangzhou right now?"}],
>     tools=tools,
>     tool_choice="required",
> )
> msg = response.choices[0].message
> call = msg.tool_calls[0]
> print(msg.model_dump_json(indent=2, exclude_none=True))
> print(response.choices[0].finish_reason)
>
> # Execute the tool yourself, then return its result on the matching tool_call_id
> result = get_weather(**json.loads(call.function.arguments))
>
> follow_up = client.chat.completions.create(
>     model="Ling-3.1-flash",
>     messages=[
>         {"role": "user", "content": "What's the weather like in Hangzhou right now?"},
>         msg,
>         {"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)},
>     ],
>     tools=tools,
> )
> print(follow_up.choices[0].message.reasoning_content)
> print(follow_up.choices[0].message.content)
> ```

> **Example Output (first turn)**
>
> ```json Output theme={null}
> {
>   "content": "",
>   "role": "assistant",
>   "tool_calls": [
>     {
>       "id": "call_ecd286d7dccb4c70a8c9c586",
>       "function": {
>         "arguments": "{\"city\": \"Hangzhou\", \"unit\": \"celsius\"}",
>         "name": "get_weather"
>       },
>       "type": "function",
>       "index": 0
>     }
>   ],
>   "reasoning_content": "The user is asking about the current weather in Hangzhou. I need to call the get_weather function with the city parameter set to \"Hangzhou\". The unit parameter is optional - I'll default to celsius since that's commonly used in China."
> }
> ```

> **Example Output (follow-up)**
>
> ```text Output theme={null}
> The user asked about the weather in Hangzhou and I got the result. Let me provide a natural response.
> The weather in Hangzhou right now is sunny with a temperature of 22°C. It's a pleasant day!
> ```
>
> The first line is `reasoning_content`; the second is `content`. `finish_reason` is `stop`.

### 3.4 Long context with YaRN (up to 1M tokens)

The model's native context is 262,144 tokens. Verified on H200 and GB300, YaRN extrapolation serves up to **1,048,576 tokens**:

```bash Command theme={null}
--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 262144, "rope_theta": 6000000, "partial_rotary_factor": 0.5}, "max_position_embeddings": 1048576}'
```

Add it to the Low-Latency serve command. All four keys are required: the override replaces the checkpoint's RoPE parameters, so omitting `rope_theta` or `partial_rotary_factor` silently changes the RoPE base, and omitting `max_position_embeddings` keeps the 262,144 limit. Only enable it for prompts beyond 256K.

Verification on the current checkpoint: a fact planted at token 700,023 of a 900,022-token prompt is retrieved exactly, NEXTN acceptance stays about 2.1 at 900K, and a 300,101-token prompt is answered correctly (H200 TTFT 15.4 s, GB300 12.6 s). One 1M-token request fits alongside normal serving traffic on both platforms.

### 3.3 HiCache with Mooncake

With HiCache enabled, prefix state is written through to Mooncake L3 and later requests prefetch it back: measured 5952 of 6000 tokens restored from storage on every TP rank. Mooncake runs as L3 prefix storage over TCP; start the metadata server, master, and storage client before launching SGLang. Run these services on the same host as the SGLang container:

> **Start Mooncake services**
>
> ```bash Command theme={null}
> docker run --rm --network host --ipc=host \
>   lmsysorg/sglang:dev-ling-3.1-flash \
>   bash -lc '
>     python3 -m mooncake.http_metadata_server --port 8290 &
>     mooncake_master --port 50171 --metrics_port 9024 &
>     exec mooncake_client \
>       --host=127.0.0.1 \
>       --port=50172 \
>       --master_server_address=127.0.0.1:50171 \
>       --metadata_server=http://127.0.0.1:8290/metadata \
>       --protocol=tcp \
>       --device_names= \
>       --global_segment_size=4294967296 \
>       --enable_http_server=true \
>       --http_port=8291
>   '
> ```

> **Example Output**
>
> ```text Output theme={null}
> I1014 11:47:24.542050 328820 master.cpp:1522] Master service started on port 50171, max_threads=16, enable_metric_reporting=1, metrics_port=9024, ...
> I1014 11:47:24.558478 328853 master_admin_service.cpp:301] Master Admin Metrics: role=leader, state=serving, service_ready=true, ...
> I1014 11:47:27.557325 328879 real_client.cpp:1935] Client HTTP server started on port 8291
> ```

Select **H200 → Low-Latency** in Deployment, then enable **HiCache → Mooncake** in the Playground. Keep NEXTN inherited from the base. Docker commands use host networking so the SGLang container can reach these localhost services. Change the master and metadata endpoints in the **Env** dialog for a remote store.

The TCP recipe leaves `MOONCAKE_DEVICE` and `--device_names` empty. The generated command includes `extra_buffer`, direct page-first I/O, and `hicache_storage_pass_prefix_keys=true` for hybrid KDA prefix reuse. To verify L3 round-trip: send a long prompt (thousands of tokens), wait for the server to go idle, flush with `curl -X POST http://localhost:30000/flush_cache -H 'Content-Type: application/json' -d '{}'`, then repeat the same prompt.

> **Verify L3 prefetch (serve log)**
>
> ```text Output theme={null}
> [2026-10-14 11:59:25 TP0 EP0] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> [2026-10-14 11:59:25 TP1 EP1] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> [2026-10-14 11:59:25 TP2 EP2] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> [2026-10-14 11:59:25 TP7 EP7] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> [2026-10-14 11:59:25 TP6 EP6] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> [2026-10-14 11:59:25 TP5 EP5] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> [2026-10-14 11:59:25 TP4 EP4] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> [2026-10-14 11:59:25 TP3 EP3] HiCache prefetch success req=2619007c50024156a9872f10ec14606a completed=5952 matched=0 loaded=5952 occupied=0
> ```

> [!NOTE]
>
> `POST /flush_cache` is refused with 400 while any request is still in flight — the L3 write-through finishes after the response returns. Wait a few seconds of idle before flushing.

For storage sizing and backend details, see [HiCache best practices](/docs/advanced_features/hicache_best_practices).


# Ring-2.5-1T
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ring-2.5-1T



## 1. Model Introduction

[Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T) is the world's first open-source trillion-parameter reasoning model based on hybrid linear attention architecture, developed by InclusionAI. Building on Ring-1T, Ring-2.5-1T demonstrates substantial improvements in generation efficiency, reasoning depth, and long-horizon task execution capabilities.

**Key Features:**

* **Trillion-Scale Model**: \~1T total parameters with 63B activation parameters using a hybrid linear attention architecture (1:7 MLA + Lightning Linear Attention)
* **Generation Efficiency**: Reduces memory access overhead by over 10x and increases generation throughput by more than 3x for sequences exceeding 32K tokens
* **Deep Reasoning**: Achieves gold medal level for both IMO 2025 and CMO 2025, with dense rewards for rigorous reasoning process feedback
* **Long-horizon Task Execution**: Enhanced autonomous execution capability through large-scale fully-async agentic RL training
* **Tool Calling**: Supports function calling with XML-style tool call format
* **Context Length**: 128K -> 256K (YaRN)

**Available Models:**

* **FP8 (8-bit quantized)**: [inclusionAI/Ring-2.5-1T](https://huggingface.co/inclusionAI/Ring-2.5-1T)

**License:** MIT

## 2. SGLang Installation

Ring-2.5-1T runs on the standard SGLang Docker image:

```bash Command theme={null}
# NVIDIA (H200 / B200 / GB200 / GB300)
docker pull lmsysorg/sglang:latest

# For MI300X/325X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x

# For MI355X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x
```

For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.

Run this Command:

<pre />

### 3.2 Configuration Tips

* The `--trust-remote-code` flag is required for this model due to custom modeling code.
* The model uses FP8 quantization (compressed-tensors format).

## 4. Model Invocation

Deploy Ring-2.5-1T with the following command (on H200, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path inclusionAI/Ring-2.5-1T \
  --tp 8 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

To enable reasoning output separation, add `--reasoning-parser deepseek-r1` when launching the server. The thinking process is returned via `reasoning_content` in the streaming response.

```shell Command theme={null}
sglang serve \
  --model-path inclusionAI/Ring-2.5-1T \
  --tp 8 \
  --trust-remote-code \
  --reasoning-parser deepseek-r1 \
  --host 0.0.0.0 \
  --port 30000
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="inclusionAI/Ring-2.5-1T",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    stream=True
)

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

<details>
  <summary>Output Example</summary>

  ```text Output theme={null}
  We are asked: "Solve this problem step by step: What is 15% of 240?" This is a straightforward percentage calculation. We need to show step-by-step solution.

  We can compute 15% of 240 as (15/100)*240 = 0.15 * 240 = 36.

  But we need to present step by step. Also ensure it's clear.

  We could also break down: 10% of 240 = 24, then 5% = 12, so 15% = 36.

  But any method is fine.

  We'll produce a solution with explanation: "To find 15% of 240, multiply 240 by 0.15 (or 15/100)."

  We'll show:

  15% = 15/100 = 0.15

  Then 0.15 × 240 = 36.

  Alternatively: (15/100) × 240 = (15 × 240) / 100 = 3600/100 = 36.

  Finally, answer: 36.

  We can also illustrate stepwise: "First, convert the percentage to a decimal: 15% = 0.15. Then multiply by the number: 0.15 × 240 = 36."

  We'll present as a final answer: \boxed{36}.

  However, we need to provide step-by-step solution as per instructions. We'll write a full explanation.

  We can also use the fraction method: 15% of 240 = (15/100)*240 = (15*240)/100 = 3600/100 = 36.

  Alr.

  I think that's it.


  **Step 1:** Write 15% as a fraction or decimal.
  \[ 15\% = \frac{15}{100} = 0.15\]

  **Step 2:** Multiply the number (240) by this fraction/decimal.
  \[ 240 \times 0.15 = 36\]

  Alternatively, using the fraction:
  \[ \frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36\]

  **Conclusion:** 15% of 240 is 36.

  \[ \boxed{36} \]
  ```
</details>

#### 4.2.2 Tool Calling

To enable tool calling, add `--tool-call-parser qwen` when launching the server.

```shell Command theme={null}
sglang serve \
  --model-path inclusionAI/Ring-2.5-1T \
  --tp 8 \
  --trust-remote-code \
  --tool-call-parser qwen \
  --host 0.0.0.0 \
  --port 30000
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="inclusionAI/Ring-2.5-1T",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools
)

print(response.choices[0].message.tool_calls)
```

**Output Example:**

```text Output theme={null}
[ChatCompletionMessageFunctionToolCall(id='call_770360e31d194ed79d32cd8c', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
```

## 5. Benchmark

### GSM8K

* Deployment Command

```bash Command theme={null}
sglang serve \
  --model-path inclusionAI/Ring-2.5-1T \
  --tp-size 8 \
  --trust-remote-code
```

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64 \
  --temperature 1.2 \
  --top-p 0.8 \
  --max-tokens 32768 \
  --chat-template-kwarg enable_thinking=true
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Result

```text Output theme={null}
Accuracy: 0.955
Invalid: 0.010
Latency: 615.833 s
Output throughput: 412.360 token/s
```


# Ring-2.6-1T
Source: https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ring-2.6-1T



## 1. Model Introduction

[Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T) is InclusionAI's trillion-parameter flagship reasoning model for real-world complex task execution. It targets agent workflows, engineering development, scientific research analysis, enterprise automation, and other long-horizon settings where the model must plan, use tools, recover from intermediate errors, and keep context across multiple steps.

**Key Features:**

* **Trillion-Scale Reasoning Model**: `BailingMoeV2_5ForCausalLM` with a `bailing_hybrid` architecture, 80 hidden layers, 256 routed experts, 8 selected experts per token, and FP8 compressed-tensors weights.
* **Agent Execution**: Designed for multi-step task decomposition, tool collaboration, context continuation, and long-horizon execution. The model card reports 87.60 on PinchBench, 63.82 on ClawEval, and 95.32 on Tau2-Bench Telecom for the `high` setting.
* **Reasoning Effort**: The model card describes `high` and `xhigh` reasoning-effort modes. In SGLang's OpenAI-compatible chat API, use top-level `reasoning_effort: "high"` for production agent workflows. To request the model-card `xhigh` prompt path, pass it through `chat_template_kwargs.reasoning_effort`.
* **Hybrid Attention**: Uses the Bailing hybrid stack with MLA plus Lightning linear attention kernels in SGLang.
* **Context Length**: Native 128K in the released config. Configure YaRN separately if you need a 256K deployment.

**Available Models:**

* **FP8 (E4M3 compressed-tensors)**: [inclusionAI/Ring-2.6-1T](https://huggingface.co/inclusionAI/Ring-2.6-1T)

**License:** MIT

## 2. SGLang Installation

Ring-2.6-1T requires recent SGLang builds with Bailing hybrid model support. Start with the latest SGLang Docker image when validating this cookbook:

```bash Command theme={null}
docker pull lmsysorg/sglang:latest
```

For other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

Use the selector below to generate a single-node command for the tested hardware targets.

Run this Command:

<pre />

### Configuration Tips

* `--trust-remote-code` is required for the model's custom Bailing hybrid implementation.
* Use `--tp-size 4` on a single 4-GPU GB300 node.
* Use `--tp-size 8` on a single 8-GPU B200 node.
* Use `--tp-size 8` on a single 8-GPU H200 node.
* Use `--mem-fraction-static 0.95` on GB300 x4. The model uses about 238.5GB/GPU after loading, so lower values can fail during KV-pool initialization.
* Use `--mem-fraction-static 0.8` on B200 x8.
* Use `--mem-fraction-static 0.95` on H200 x8.
* `--model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}'` is recommended because the model has 175 large safetensors shards.
* Keep `--tool-call-parser glm` enabled by default for OpenAI-compatible tool calls. Ring's template emits XML `<arg_key>/<arg_value>` tool calls, which the `qwen` parser does not convert into `message.tool_calls`.
* Keep `--reasoning-parser deepseek-r1` enabled by default so `<think>...</think>` content is split into `message.reasoning_content`.

## 4. Model Invocation

### 4.1 Basic Usage

For example, launch the server on a single 4-GPU GB300 node:

```bash Command theme={null}
export PORT=30000

sglang serve \
  --model-path inclusionAI/Ring-2.6-1T \
  --tp-size 4 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port ${PORT} \
  --mem-fraction-static 0.95 \
  --model-loader-extra-config '{"enable_multithread_load":"true","num_threads":64}' \
  --tool-call-parser glm \
  --reasoning-parser deepseek-r1
```

Send a basic chat request:

```bash Command theme={null}
curl -s http://localhost:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 128
  }'
```

### 4.2 Reasoning Effort

Ring-2.6-1T exposes two reasoning-effort levels in the model card: `high` and `xhigh`. In SGLang's OpenAI-compatible chat API, start with top-level `reasoning_effort: "high"` for agent and production workflows:

```bash Command theme={null}
curl -s http://localhost:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}],
    "reasoning_effort": "high",
    "max_tokens": 512
}'
```

For the model-card `xhigh` path, pass the template value explicitly:

```bash Command theme={null}
curl -s http://localhost:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 52, what is x?"}],
    "chat_template_kwargs": {"reasoning_effort": "xhigh"},
    "max_tokens": 512
  }'
```

With the default deployment command, thinking text is separated into `message.reasoning_content` when the model emits `<think>...</think>` blocks.

### 4.3 Tool Calling Example

```bash Command theme={null}
curl -s http://localhost:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "What is the weather in Beijing?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string", "description": "The city name"}
          },
          "required": ["location"]
        }
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 512
  }'
```

For more API examples, see the [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request).

## 5. Benchmark

### 5.1 Speed Benchmark

* Hardware: NVIDIA B200 GPU (8x), NVIDIA H200 GPU (8x), and NVIDIA GB300 GPU (4x)
* Model: `inclusionAI/Ring-2.6-1T`
* Docker image: `lmsysorg/sglang:latest`
* SGLang version tested: `0.5.11`
* Tensor Parallelism: 8 on B200 x8 and H200 x8, 4 on GB300 x4

Use the deployment command from [Section 3](#3-model-deployment), then confirm that the server is healthy before running benchmarks:

```bash Command theme={null}
curl -s http://localhost:${PORT}/health
curl -s http://localhost:${PORT}/v1/models
```

#### 5.1.1 Latency-Sensitive Benchmark

* Test Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port ${PORT} \
  --model inclusionAI/Ring-2.6-1T \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results (B200 x8):

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  207.18
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.05
Input token throughput (tok/s):          29.45
Output token throughput (tok/s):         20.37
Total token throughput (tok/s):          49.82
Mean E2E Latency (ms):                   20715.16
Mean TTFT (ms):                          187.86
Mean TPOT (ms):                          44.65
Mean ITL (ms):                           48.76
==================================================
```

* Test Results (GB300 x4):

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  62.21
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.16
Input token throughput (tok/s):          98.07
Output token throughput (tok/s):         67.83
Total token throughput (tok/s):          165.91
Mean E2E Latency (ms):                   6218.57
Mean TTFT (ms):                          233.04
Mean TPOT (ms):                          14.21
Mean ITL (ms):                           14.22
==================================================
```

* Test Results (H200 x8):

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  57.10
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.18
Input token throughput (tok/s):          106.85
Output token throughput (tok/s):         73.91
Total token throughput (tok/s):          180.76
Mean E2E Latency (ms):                   5707.72
Mean TTFT (ms):                          163.35
Mean TPOT (ms):                          13.17
Mean ITL (ms):                           13.17
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Test Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port ${PORT} \
  --model inclusionAI/Ring-2.6-1T \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 100 \
  --max-concurrency 100 \
  --request-rate inf
```

* Test Results (B200 x8):

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     100
Benchmark duration (s):                  46.30
Total input tokens:                      50561
Total generated tokens:                  52444
Request throughput (req/s):              2.16
Input token throughput (tok/s):          1092.10
Output token throughput (tok/s):         1132.77
Total token throughput (tok/s):          2224.86
Mean E2E Latency (ms):                   27581.74
Mean TTFT (ms):                          1710.53
Mean TPOT (ms):                          51.27
Mean ITL (ms):                           49.43
==================================================
```

* Test Results (GB300 x4):

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     100
Benchmark duration (s):                  55.80
Total input tokens:                      50561
Total generated tokens:                  52444
Request throughput (req/s):              1.79
Input token throughput (tok/s):          906.10
Output token throughput (tok/s):         939.84
Total token throughput (tok/s):          1845.94
Mean E2E Latency (ms):                   33736.85
Mean TTFT (ms):                          2156.40
Mean TPOT (ms):                          63.09
Mean ITL (ms):                           60.33
==================================================
```

* Test Results (H200 x8):

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     100
Benchmark duration (s):                  44.51
Total input tokens:                      50561
Total generated tokens:                  52444
Request throughput (req/s):              2.25
Input token throughput (tok/s):          1135.88
Output token throughput (tok/s):         1178.18
Total token throughput (tok/s):          2314.06
Mean E2E Latency (ms):                   27177.14
Mean TTFT (ms):                          2173.08
Mean TPOT (ms):                          51.11
Mean ITL (ms):                           47.77
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* Benchmark Command:

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:${PORT}/v1 \
  --model auto \
  --num-examples 200 \
  --num-threads 64 \
  --max-tokens 2048 \
  --reasoning-effort high
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* Test Results (B200 x8):

```text Output theme={null}
Total latency: 100.378 s
Score: 0.990
Output throughput: 627.401 token/s
```

* Test Results (GB300 x4):

```text Output theme={null}
Total latency: 98.386 s
Score: 0.990
Output throughput: 621.469 token/s
```

* Test Results (H200 x8):

```text Output theme={null}
Total latency: 76.849 s
Score: 0.990
Output throughput: 793.125 token/s
```


# Intern-S1
Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S1



## 1. Model Introduction

Intern-S1 includes the large **Intern-S1** MoE model and the smaller **Intern-S1-mini** dense model. The command generator below covers BF16 and FP8 serving on NVIDIA H100/H200/B200/B300 platforms.

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install from source:

```bash Command theme={null}
uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
```

## 3. Model Deployment

### 3.1 Basic Configuration

Run this Command:

<pre />

### 3.2 Configuration Tips

* FP8 checkpoints use the matching BF16 checkpoint as tokenizer path.
* B300 deployments use `--attention-backend flashinfer`.
* Enable `--reasoning-parser interns1` and `--tool-call-parser interns1` when your workload needs structured reasoning or tool-call parsing.


# Intern-S2-Mobius
Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S2-Mobius

Deploy the BF16 and FP8 Intern-S2-Mobius checkpoints with SGLang on NVIDIA H200 and B200.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> Intern-S2-Mobius was upstreamed in PR [#33691](https://github.com/sgl-project/sglang/pull/33691) (merged 2026-08-08) — it lives on `lmsysorg/sglang:dev` (nightly) until the next release cut. For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + recipe to generate the launch command. The two serving strategies cover the main operating points:

* **Low-Latency** — MTP (NEXTN) speculative decoding on. Fastest reply for a single user.
* **High-Throughput** — spec off, more tokens per second when many users share the server.

> [!NOTE]
>
> Speed and accuracy numbers apply to the BF16 checkpoint. They were measured with `--random-range-ratio 1.0`, `--flush-cache`, on 2×H200 TP=2 against `main @ e0828ee3` + PR [#33691](https://github.com/sgl-project/sglang/pull/33691) head (since merged 2026-08-08 — `lmsysorg/sglang:dev` is the live equivalent). GSM8K is the full 1319-example test split; GPQA is Diamond 198 problems × 8 repeats (pass\@1 avg-of-8). Both ran with no server-side sampling override, so the checkpoint's `generation_config.json` defaults applied (temperature 1.0, top\_p 0.95, top\_k 20). The FP8 H200 recipes have launch, text, vision, and stop-reason verification but no benchmark numbers yet. All B200 recipes are inferred and unverified.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

**Intern-S2-Mobius** is InternLM's 35B scientific multimodal foundation model built on the **Mobius-v0** architecture (continually pre-trained from Qwen3.5-35B, then SFT and RL post-trained). Instead of binding knowledge storage and reasoning computation layer by layer as conventional Transformers do, Mobius organizes knowledge into a **globally shared Memory** that multiple **Reasoners** iteratively query against, yielding two native capabilities:

* **Backward Residual Connection** — shallow and deep reasoning stages can reach knowledge across the model rather than relying only on forward layer-wise flow.
* **Dynamic Latent Reasoning** — recurrent latent iteration refines hidden states before decoding, internalizing part of the deliberation process and shrinking visible chain-of-thought. The reported result is roughly a **4× end-to-end inference speedup** over the Qwen3.5-35B baseline while holding comparable scores on general reasoning benchmarks and improving on scientific tasks (Biology-Instructions, Mol-Instructions, MolecularIQ).

On the serving side the model is a hybrid: 30 of 40 transformer layers use **GDN (Gated Delta Net) linear attention** (`kimi-linear`-family), with a **full-attention** layer every 4th layer (`full_attention_interval: 4` → **10 full-attention** layers), and the bottom of the stack is MoE-routed (2,560 routed experts × 512 intermediate, 8 active per token); a separate **MoE-256 / top-8 MTP (NEXTN) layer** feeds speculative decoding. It takes images via a vision tower and recognizes the standard `<|vision_start|>…<|vision_end|>` + `<|image_pad|>` markers. Context length is **262,144** tokens.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variant</th>
      <th>Architecture</th>
      <th>Context</th>
      <th>License</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[Intern-S2-Mobius](https://huggingface.co/internlm/Intern-S2-Mobius)**</td>
      <td>Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · BF16</td>
      <td>262,144</td>
      <td>[Apache-2.0](https://huggingface.co/internlm/Intern-S2-Mobius/blob/main/LICENSE)</td>
    </tr>

    <tr>
      <td>**[Intern-S2-Mobius-FP8](https://huggingface.co/internlm/Intern-S2-Mobius-FP8)**</td>
      <td>Mobius-v0 · GDN ×30 + full ×10 · MoE-2560 / top-8 · MTP · FP8 E4M3</td>
      <td>262,144</td>
      <td>[Apache-2.0](https://huggingface.co/internlm/Intern-S2-Mobius-FP8/blob/main/LICENSE)</td>
    </tr>
  </tbody>
</table>

**Recommended generation:** `temperature=0.8`, `top_p=1.0`, `top_k=50`, `min_p=0.0` — the values the [model card](https://huggingface.co/internlm/Intern-S2-Mobius) recommends. Both checkpoints ship `generation_config.json` with different values (`temperature=1.0`, `top_p=0.95`, `top_k=20`), and SGLang applies that file by default (`--sampling-defaults model`) — so send the recommended values explicitly per request if you want them.

**Resources:** [HuggingFace (BF16)](https://huggingface.co/internlm/Intern-S2-Mobius) · [HuggingFace (FP8)](https://huggingface.co/internlm/Intern-S2-Mobius-FP8) · [GitHub (InternLM/Intern-S2-Mobius)](https://github.com/InternLM/Intern-S2-Mobius).

## 2. Configuration Tips

* **Trust remote code is required.** Intern-S2-Mobius ships a custom `configuration_interns2_mobius.py` / `modeling_interns2_mobius.py` on its HF repo; every recipe adds `--trust-remote-code`.
* **FP8 uses one H200.** The FP8 checkpoint stores E4M3 weights with 128×128 block scales and dynamic activation quantization. The verified H200 recipes use TP=1, `deep_gemm`, `--mem-fraction-static 0.6`, disabled prefill CUDA graphs, and decode graphs up to batch size 16. Higher memory fractions or the default graph range exhausted H200 memory during DeepGEMM warmup; the running MTP server used about 99 GB.
* **Speculative decoding schedule.** The checkpoint ships one MTP layer. Enable MTP for the lowest latency (`--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`). We measured accept-length \~3.9/4 draft tokens at 8K-in / 1K-out, roughly tripling single-stream decode speed (median TPOT 9.79 ms → 3.13 ms at conc=1, 14.26 ms → 6.84 ms at conc=16) and roughly doubling mid-concurrency total throughput (9358 → 18029 tokens/s at conc=16, 21395 → 26033 tokens/s at conc=64). The high-throughput recipe stays spec-off because once you can batch wide, its saturation point is higher (34786 tokens/s at conc=256 vs the spec recipe's peak at conc=64).
* **Mamba pool sizing.** GDN layers live in a separate Mamba state pool; the `--mamba-full-memory-ratio` (defaults to 0.9) controls the split between the 10 full-attention layers' KV pool and the 30 GDN layers' conv+SSM state pool. Default split comfortably handles conc=64 on a 2×H200 node; if you need higher concurrency than `--max-running-requests` allows for your workload, raise `--mamba-full-memory-ratio` slightly (each +1% mamba ratio costs full-attn KV).
* **Vision input.** Images are accepted via the standard `image_url` chat message type. Vision tokens are counted into the prompt (`prompt_tokens_details.image_tokens` shows the count), and the model honors `<|vision_start|>` / `<|vision_end|>` boundaries exactly.
* **B200 sizing.** B200 (192 GB HBM) fits the BF16 weights + KV + Mamba pool on a **single GPU** with `--tp 1`. The FP8 B200 cells inherit the verified H200 TP=1 recipe but remain unverified; treat them as a starting point until a Blackwell measurement is available.

## 3. Advanced Usage

> [!NOTE]
>
> The outputs below are verbatim captures from a live server (sampling per the checkpoint's `generation_config.json`, temperature 1.0). Re-running the same request yields a semantically equivalent but textually different trace — treat them as representative, not deterministic.

### 3.1 Reasoning

InternS2-Mobius is a hybrid-reasoning model — thinking traces start with "Thinking Process:" before the final answer. Enable the **`qwen3` reasoning parser** (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to split thinking into `message.reasoning_content` and the answer into `message.content`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="internlm/Intern-S2-Mobius",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: Thinking Process:
>
> 1.  **Identify the core question:** The user is asking for 15% of 240.
> 2.  **Determine the calculation method:** To find a percentage of a
>     number, multiply the number by the percentage expressed as a decimal
>     or fraction.
>     *   Percentage: 15%
>     *   Decimal: 0.15
>     *   Fraction: 15/100
> 3.  **Perform the calculation:** $240 \times 0.15$
>     *   Method 1: $240 \times 0.10 = 24$ (10%) and
>         $240 \times 0.05 = 12$ (5%). Then add them: $24 + 12 = 36$.
>     *   Method 2: $240 \times 15 = 3600$. Divide by 100 -> 36.
> 4.  **Verify the result:** The calculation is correct.
> 5.  **Formulate the answer:** State the final number clearly.cw
>
> Answer:
>
> 15% of 240 is **36**.
>
> Here is the math:
> $240 \times 0.15 = 36$
> ```

### 3.2 Tool Calling

Enable the **`qwen3_coder` tool-call parser** (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Intern-S2-Mobius emits `<tool_call>…<function=name>…<parameter=key>…value…</parameter>…` — this is exactly the format `qwen3_coder` parses; without the parser the call is left as raw text in `content`. On this thinking-mode model the turn also fills `reasoning_content`, so print both fields.

**Auto-resolution works out of the box.** Intern-S2-Mobius's chat template contains the `<function=` / `<parameter=` markers the auto-detector keys on, so `--reasoning-parser auto --tool-call-parser auto` resolves to `qwen3` / `qwen3_coder` without any extra config (verified on this build by tailing the server log's "Auto-detected …" lines + a live `tools` request). You can pass the literal `qwen3_coder` slug, but you don't have to.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a location",
>         "parameters": {
>             "type": "object",
>             "properties": {
>                 "location": {"type": "string", "description": "The city name"},
>             },
>             "required": ["location"],
>         },
>     },
> }]
> resp = client.chat.completions.create(
>     model="internlm/Intern-S2-Mobius",
>     messages=[{"role": "user", "content": "What is the weather in Beijing?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user is asking for the weather in Beijing. I have access
> to a get_weather function that can get the current weather for a
> location. The function requires a "location" parameter which should be
> the city name. In this case, the user specified "Beijing", so I should
> use that as the location parameter.
>
> Tool calls: [
>   {
>     "id": "call_545b5956b4c3457286261490",
>     "index": 0,
>     "type": "function",
>     "function": {"name": "get_weather", "arguments": "{\"location\": \"Beijing\"}"}
>   }
> ]
>
> finish_reason: tool_calls
> ```

### 3.3 Vision Input

Intern-S2-Mobius takes images via the OpenAI-compatible `image_url` content type. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed.

> **Vision Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="internlm/Intern-S2-Mobius",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "image_url",
>              "image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}},
>             {"type": "text", "text": "Describe this image in one sentence."},
>         ],
>     }],
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user wants a one-sentence description of the image.
> Key elements: Tiger, lying down, grass, looking at camera.
> Drafting: A tiger is lying in the green grass looking directly at the
> camera.
> Refining for flow and detail: A majestic tiger with striking orange and
> black stripes rests calmly on a bed of lush green grass, staring
> intently directly at the viewer.
>
> Answer:
>
> A majestic tiger with striking orange and black stripes rests calmly on a
> bed of lush green grass, staring intently directly at the viewer.
> ```


# Intern-S2-Preview
Source: https://docs.sglang.io/cookbook/autoregressive/InternLM/Intern-S2-Preview



## 1. Model Introduction

**Intern-S2-Preview** is an efficient 35B scientific multimodal foundation model. Beyond conventional parameter and data scaling, Intern-S2-Preview explores task scaling: increasing the difficulty, diversity, and coverage of scientific tasks to further unlock model capabilities.

**Resources:**

* HuggingFace: [internLM/Intern-S2-Preview](https://huggingface.co/internLM/Intern-S2-Preview)

## 2. SGLang Installation

SGLang offers multiple installation methods. Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

Install SGLang from source or use an NVIDIA Docker image:

```bash Command theme={null}
# Install from source
uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'

# Or use Docker for NVIDIA GPUs
docker pull lmsysorg/sglang:latest
```

For how to actually launch a docker image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the [command generator](#3-model-deployment) below produces):

```bash Command theme={null}
docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<your-hf-token>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    sglang serve <use args below>
```

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the selector below to generate the deployment command for your hardware and parser configuration.

Run this Command:

<pre />

### 3.2 Configuration Tips

* Use `tp>=2` for the NVIDIA deployment commands.
* Use `--reasoning-parser qwen3` to separate reasoning content from final content in streaming responses.
* Use `--tool-call-parser qwen3_coder` when serving tool-calling workloads.
* Add `--mamba-radix-cache-strategy extra_buffer` with `--speculative-algo 'NEXTN'` to enable MTP.
* If weight loading is slow, add `--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 64}'`.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, see:

* [Basic API Usage](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Vision Input

Intern-S2-Preview supports image inputs. Here is an example with an image:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="internLM/Intern-S2-Preview",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"
                    },
                },
                {
                    "type": "text",
                    "text": "Describe this image in detail.",
                },
            ],
        }
    ],
    max_tokens=2048,
    stream=True,
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, "reasoning_content") and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

#### 4.2.2 Reasoning Parser

Enable streaming to read reasoning content separately from the final answer:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="internLM/Intern-S2-Preview",
    messages=[
        {"role": "user", "content": "Solve this step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    stream=True,
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, "reasoning_content") and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

#### 4.2.3 Tool Calling

Serve with `--tool-call-parser qwen3_coder` enabled, then send OpenAI-compatible tool requests:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name",
                    }
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="internLM/Intern-S2-Preview",
    messages=[{"role": "user", "content": "What is the weather in Beijing?"}],
    tools=tools,
    max_tokens=1024,
)

print(response.choices[0].message)
```


# InternVL3.5
Source: https://docs.sglang.io/cookbook/autoregressive/InternVL/InternVL3.5



## 📝 Community Contribution Welcome

This guide is currently under development. We welcome community contributions!

If you have experience deploying **InternVL3.5** with SGLang, please help us complete this documentation.

## 🚀 How to Contribute

```shell Command theme={null}
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
cd sglang-cookbook
git checkout -b add-internvl3-5-guide
# Edit this file and submit a PR
```

## 📚 Reference

* [GLM-4.6V](../GLM/GLM-4.6V)

***

**Let's build this together!** 🌟


# Jina-reranker-m0
Source: https://docs.sglang.io/cookbook/autoregressive/Jina/Jina-reranker-m0



## 📝 Community Contribution Welcome

This guide is currently under development. We welcome community contributions!

If you have experience deploying **Jina-reranker-m0** with SGLang, please help us complete this documentation.

## 🚀 How to Contribute

```shell Command theme={null}
git clone https://github.com/YOUR_USERNAME/sglang-cookbook.git
cd sglang-cookbook
git checkout -b add-jina-reranker-m0-guide
# Edit this file and submit a PR
```

## 📚 Reference

* [DeepSeek-V3.2](../DeepSeek/DeepSeek-V3_2.md)

***

**Let's build this together!** 🌟


# Kev
Source: https://docs.sglang.io/cookbook/autoregressive/Kev/kev

Deploy the Kev decision models with SGLang. Kev returns calibrated probabilities for typed questions about a document, in sizes from 0.8B to 27B, each on one NVIDIA H200 in BF16.

## Deployment

Every Kev size runs on one GPU in BF16 with TP=1 and needs no extra flag. SGLang reads the checkpoint's `head.pt` and configures the server.

```bash Command theme={null}
sglang serve --model-path jaredpalmer/kev-27b --host 0.0.0.0 --port 30000
```

Replace the model path with any size in the table below. Each was validated on one H200.

| Model | Hugging Face ID | Checkpoint | Backbone | GPU memory for weights |
| - | - | - | - | - |
| Kev-27B | `jaredpalmer/kev-27b` | full weights | Qwen3.8-27B | 47.8 GB |
| Kev-9B | `jaredpalmer/kev-9b` | LoRA adapter | Qwen3.5-9B-Base | 14.9 GB |
| Kev-4B | `jaredpalmer/kev-4b` | LoRA adapter | Qwen3.5-4B-Base | 8.0 GB |
| Kev-0.8B | `jaredpalmer/kev-0.8b` | LoRA adapter | Qwen3.5-0.8B-Base | 1.5 GB |

For a LoRA adapter, SGLang loads the base revision that the adapter's `head.pt` pins and merges the adapter into its weights at startup, in FP32 with one rounding to BF16, as the reference code merges it. The base is downloaded on first use.

## 1. Model introduction

[Kev](https://huggingface.co/collections/jaredpalmer/kev) is a family of decision models published by Jared Palmer under the Apache-2.0 license, with [reference code](https://github.com/jaredpalmer/kev). Each model keeps a Qwen3.5 or Qwen3.8 backbone and replaces the LM head with a pointer head. Each question runs as its own prompt after the shared state. The head compares the final hidden state at each option's closing token with the one at the prompt's last token and returns one logit per option, with no text generation. A temperature stored in `head.pt` calibrates the probabilities.

Pass a text string, JSON object, or JSON array as the state. `/v1/systemone` returns one answer per question: a `choice` over allowed options, a `noul` probability of true, or a `score` over ordered levels. JSON states are flattened into labeled text as the reference code does.

## 2. Configuration tips

* **Route.** Send requests to `/v1/systemone`, or set your System One client's base URL to the server. This includes TypeSafe SDK clients. See the [API reference](/docs/supported-models/decision_models#system-one-compatible-api) for request and response fields. Kev refuses `/v1/decisions`, chat, generation, and `/encode` requests.
* **One prefill per question.** The model reads the state once for every question, as the reference does, so a request with many questions over a long state costs that many prefills. SGLang enables embedding mode and disables radix caching, chunked prefill, and CUDA graphs. Keep TP=1 and PP=1.
* **Answers.** `probabilities` are the softmax of the head's logits divided by the calibrated temperature. `confidence` uses the System One adapter formulas, as Kev's own server does. SGLang does not round the values to four decimals. Answers omit `x_label_mass` because the head scores only the allowed options.
* **Long states.** The reference server accepts states of up to 65,536 tokens. In SGLang each question's prompt must also fit one prefill, 16,384 tokens by default, so longer prompts are refused with an error that gives both lengths. Raising `--max-prefill-tokens` is not supported for this model yet, because the server does not reserve memory for longer unchunked prefills and can run out of memory under load.
* **Inputs.** Requests are text only, so `images` and `chat_template_kwargs` are refused. A `score` question takes up to 10 levels and a `choice` question up to 255 options.
* **Not supported.** Kev-27B v1, the `v1-lora` revision of `jaredpalmer/kev-27b`, is an adapter trained on a BF16 backbone that the reference applies unmerged, so SGLang refuses it and serves the current full weight release instead. Kev-0.5B is a prototype on a Qwen2.5 base and is refused too.

## 3. Advanced usage

### 3.1 TypeSafe SDK

```bash Command theme={null}
pip install typesafe-sdk
```

> **Text decision example (Python)**
>
> ```python Example theme={null}
> from typesafe_sdk import Choice, Noul, TypeSafeClient
>
> client = TypeSafeClient(api_key="local", base_url="http://localhost:30000", model="kev-latest")
> response = client.system_one(
>     state="I was charged twice for order 1182. Please refund one of the charges.",
>     questions={
>         "team": Choice(
>             instructions="Which team should handle this?",
>             criteria={"billing": "Charges and refunds", "shipping": "Deliveries", "returns": "Exchanges"},
>         ),
>         "urgent": Noul(instructions="Does this need a reply today?"),
>     },
> )
> print(response.choices["team"].choice, response.nouls["urgent"].noul)
> ```

> **Example output**
>
> Captured with Kev-27B on one H200. Probabilities can vary with the model size, hardware, and numerical backend.
>
> ```text Output theme={null}
> billing 0.7591662905989526
> ```


# LFM2.5
Source: https://docs.sglang.io/cookbook/autoregressive/LiquidAI/LFM2.5

Deploy Liquid AI's LFM2.5 with SGLang — hybrid gated short conv + GQA models from 350M to the 8B-A1B MoE, plus LFM2.5-VL vision, with reasoning and Pythonic tool calling.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> > [!NOTE]
> >
> > LFM2.5 support — the dense / MoE / VL model classes and the `lfm2` tool-call parser — ships on SGLang `main`. If your installed release predates it, install from source or use the Docker dev image.
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> LFM2.5 support ships in the pinned SGLang dev image:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-cu13
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). A minimal example (substitute the inner `sglang serve ...` with whatever the command generator below produces):
>
> ```bash Command theme={null}
> docker run --gpus all \
>     --shm-size 32g \
>     -p 30000:30000 \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     --env "HF_TOKEN=<your-hf-token>" \
>     --ipc=host \
>     lmsysorg/sglang:dev-cu13 \
>     sglang serve <use args below>
> ```

Every LFM2.5 model runs on a **single GPU (TP=1)** — pick your hardware + model variant to generate the launch command. One recipe covers all operating points per variant; the commands differ only by the parsers a model needs and, on Blackwell, the attention backend. The `lfm2` tool-call parser and each reasoning model's `--reasoning-parser` are already part of the verified command.

**Panel controls** (top of the command box):

<ul>
  <li>**Python / Docker** — bare `sglang serve …` for an existing SGLang env, or a `docker run … sglang serve …` wrap against the dev image from the [Install SGLang](#install) panel above.</li>
  <li>**⧉ Copy** — copies the current command (with whichever framing is active) to your clipboard.</li>
  <li>**\$ cURL** — a sample request against `localhost:30000` to confirm the server is up.</li>
  <li>**⚙ Env** — edits the placeholders (`HOST_IP`, `PORT`, `HF_TOKEN`) the command and cURL share. Persists in localStorage across cookbooks.</li>
  <li>**Verified / Not Verified** badge — green when the `(hw, variant, quant, strategy, nodes)` combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
</ul>

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.

For LFM2.5 the exposed knob is the **TP override** (every variant is verified at TP=1; TP=2 is available for experimentation on the larger checkpoints). The reasoning and tool-call parsers are not playground toggles here — they are variant-intrinsic and already baked into each verified command.

Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

**Panel controls** reuse **Python / Docker** · **⧉ Copy** · **\$ cURL** · **⚙ Env** from the Deploy panel, plus one extra:

<ul>
  <li>**Submit ↗** — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says **Not Verified**; click it once you've actually run the command on your hardware and confirmed it works.</li>
</ul>

## 1. Model Introduction

LFM2.5 is [Liquid AI](https://www.liquid.ai/)'s family of hybrid models for on-device deployment, released under the [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE). It builds on the LFM2 architecture with extended pre-training — 10T → 28T tokens for the dense models, 12T → 38T for the 8B-A1B MoE — and large-scale reinforcement learning.

The backbone interleaves **gated short convolution blocks** with a small minority of **grouped query attention (GQA) blocks**. Each convolution block applies input-dependent multiplicative gating around a depthwise short convolution, giving fast local mixing at low compute and memory cost. The GQA blocks handle global context and long-range retrieval.

This minimal hybrid layout was selected by a hardware-in-the-loop architecture search under edge latency and memory budgets. On CPUs it delivers up to 2× faster prefill and decode than similarly sized models (see the [LFM2 Technical Report](https://arxiv.org/abs/2511.23404)).

**Key Features:**

* **Hybrid gated short conv + GQA layout**: the 1.2B / 350M dense models are 16 layers (10 conv + 6 GQA); the 8B-A1B MoE is 24 layers (18 conv + 6 GQA). With only 6 attention layers per model, the KV cache stays small even at long context.
* **Block details**: depthwise convolutions with kernel size 3; GQA with 8 KV groups and head size 64, plus RoPE and QK-Norm; pre-norm RMSNorm and SwiGLU MLPs throughout.
* **Sparse MoE (8B-A1B)**: 8.3B total / 1.5B active parameters. Every layer except the first two replaces its dense MLP with a 32-expert MoE block; each token is routed to the top-4 SwiGLU experts by a normalized sigmoid router with adaptive bias load balancing.
* **New in 2.5 (8B-A1B)**: the blocks are unchanged from LFM2-8B-A1B, but the context window grows from 32K to 128K (a RoPE base-θ increase plus long-context midtraining) and the vocabulary doubles from 65,536 to 128,000 tokens for more efficient non-Latin tokenization.
* **Pythonic tool calling**: function calls are emitted as a Python list between `<|tool_call_start|>` and `<|tool_call_end|>` tokens. The `lfm2` tool-call parser surfaces these as standard `message.tool_calls`.
* **Reasoning variants**: the 8B-A1B and 1.2B-Thinking checkpoints are reasoning-only models that always emit an explicit `<think>...</think>` chain-of-thought before the answer. The MoE's 1.5B active parameters keep those reasoning tokens cheap.
* **Multilingual**: every model except the JP checkpoints covers at least English, Arabic, Chinese, French, German, Japanese, Korean, and Spanish (some variants add more). The dedicated JP chat checkpoints focus on Japanese (Japanese + English only).
* **Vision**: LFM2.5-VL-1.6B pairs the 1.2B language backbone with a SigLIP2 So400M NaFlex encoder for OCR, document understanding, and multilingual vision. LFM2.5-VL-450M pairs the 350M backbone with a SigLIP2 Base-86M encoder for captioning and object detection at edge sizes; bounding-box grounding and function calling are new in the 2.5 release.

**Available Models:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Parameters</th>
      <th>Context</th>
      <th>Role</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B)**</td>
      <td>8.3B total / 1.5B active (MoE)</td>
      <td>128K</td>
      <td>Reasoning-tuned, agentic / tool use</td>
    </tr>

    <tr>
      <td>**[LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct)**</td>
      <td>1.17B (dense)</td>
      <td>32K</td>
      <td>General instruct, RAG, data extraction</td>
    </tr>

    <tr>
      <td>**[LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking)**</td>
      <td>1.17B (dense)</td>
      <td>32K</td>
      <td>Reasoning (always-on chain-of-thought)</td>
    </tr>

    <tr>
      <td>**[LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M)**</td>
      <td>350M (dense)</td>
      <td>32K</td>
      <td>Compact instruct, structured output</td>
    </tr>

    <tr>
      <td>**[LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M)**</td>
      <td>230M (dense)</td>
      <td>32K</td>
      <td>Most compact; data extraction, structured output</td>
    </tr>

    <tr>
      <td>**[LFM2.5-1.2B-JP-202606](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606)**</td>
      <td>1.17B (dense)</td>
      <td>32K</td>
      <td>Japanese chat (latest)</td>
    </tr>

    <tr>
      <td>[LFM2.5-1.2B-JP](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP)</td>
      <td>1.17B (dense)</td>
      <td>32K</td>
      <td>Japanese chat (original)</td>
    </tr>

    <tr>
      <td>**[LFM2.5-VL-1.6B](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B)**</td>
      <td>1.2B LM + SigLIP2 400M</td>
      <td>32K</td>
      <td>Vision-language (OCR, docs, multi-image)</td>
    </tr>

    <tr>
      <td>**[LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M)**</td>
      <td>350M LM + SigLIP2 86M</td>
      <td>32K</td>
      <td>Compact vision-language (captioning, object detection)</td>
    </tr>

    <tr>
      <td>[LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base)</td>
      <td>1.17B (dense)</td>
      <td>32K</td>
      <td>Pre-trained base (no post-training)</td>
    </tr>
  </tbody>
</table>

The Deploy panel above covers the eight serving variants; **LFM2.5-1.2B-JP** (original — launch without `--tool-call-parser`) and the **Base** repos (pre-trained only, no post-training — see [§3.5](#3-5-base-checkpoints)) launch the same way with the model path swapped.

**Choosing a variant:**

* **8B-A1B** — flagship for agentic and tool-calling workloads; the only 128K-context option.
* **1.2B-Thinking** — reasoning-heavy tasks: math, tool use, programming.
* **1.2B-Instruct** — the recommended pick for chat and creative writing.
* **350M** — tool use, data extraction, and structured output; not recommended for math, code, or creative writing.
* **230M** — the most compact checkpoint; same use as the 350M, not for math, code, or creative writing.

**License:** [LFM Open License v1.0](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B/blob/main/LICENSE).

**Resources:** [LFM2.5 announcement](https://www.liquid.ai/blog/introducing-lfm2-5-the-next-generation-of-on-device-ai), [LFM2.5-8B-A1B blog](https://www.liquid.ai/blog/lfm2-5-8b-a1b), [LFM docs](https://docs.liquid.ai/lfm/getting-started/welcome), [LFM2 Technical Report (arXiv:2511.23404)](https://arxiv.org/abs/2511.23404).

## 2. Configuration Tips

* **Reasoning parser**: LFM2.5 reasoning models wrap their chain-of-thought in `<think>...</think>` tags. The command generator passes `--reasoning-parser qwen3` for **8B-A1B** (it emits an explicit opening `<think>`) and `--reasoning-parser qwen3-thinking` for **1.2B-Thinking** (always-on reasoning). This splits the thinking process into `reasoning_content`; without it the chain-of-thought stays inline in `content`.
* **Tool calling**: `--tool-call-parser lfm2` surfaces LFM2.5's Pythonic `<|tool_call_start|>[...]<|tool_call_end|>` calls as standard `message.tool_calls`. The original **1.2B-JP** does not expose tool calling; **Base** has no post-training (see [§3.5](#3-5-base-checkpoints)).
* **Attention backend on Blackwell (B200/sm100)**: SGLang defaults to the `trtllm_mha` backend on sm100, which is fastest for the dense text models. The **8B-A1B** uses a mamba-style state cache that runs on a page-size-1 backend, so the generator picks `--attention-backend flashinfer` for it. The **VL** language model also uses that state cache and offers two backends: `--attention-backend flashinfer` (keeps prefix/radix caching — what the generator emits), or `--attention-backend trtllm_mha --disable-radix-cache` to run the language model on Blackwell `trtllm_mha` attention (`--disable-radix-cache` lifts the page-size-1 requirement, at the cost of prefix caching). Pair either with `--mm-attention-backend fa4` for the vision tower.
* **VL vision tower (`--mm-attention-backend`)**: on sm100 the `trtllm_mha` default is fastest for text but applies *causal* attention to image tokens. For the VL model, pass `--mm-attention-backend fa4` on B200/B300 (or `fa3` on H100/H200) to restore bidirectional image-token attention and full vision quality.
* **VL multimodal feature transport**: the generator launches the VL models with `SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1`. The first moves the processor→scheduler image-feature handoff onto CUDA IPC instead of serializing tensors between processes; the second ships the pool handle so the scheduler opens it once and caches it, instead of opening a per-item handle on every request. On the image serving workload (1 image @ 720p, measured on VL-1.6B on H100 and B200) this pair is worth roughly 30–50% higher image throughput and 30–40% lower image TTFT vs running without them (measured on VL-1.6B, H100 and B200); decode speed (TPOT) is unaffected.
* **VL-450M memory headroom (`--mem-fraction-static 0.8`)**: with the default memory fraction, the 450M's small weights make SGLang size its static KV/mamba pools to nearly the whole GPU, leaving no headroom for image-feature tensors — under sustained concurrent image load the scheduler can crash with a CUDA OOM in the radix-cache free path. The generator caps `--mem-fraction-static 0.8` for VL-450M; the pool is still far larger than this model ever needs.
* **Mamba scheduling**: LFM2.5 runs on the default `no_buffer` mamba scheduler strategy — no `--mamba-radix-cache-strategy` flag is needed. The `extra_buffer` strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use `mamba_chunk_size=1`.
* **Hardware requirements**: all LFM2.5 models run on a single GPU (TP=1) on either Hopper or Blackwell. The 1.2B / 350M dense models fit in a few GB; the 8B-A1B MoE needs roughly 16 GB for bf16 weights plus KV cache. Multi-GPU tensor parallelism is not required for any variant.

**Recommended sampling parameters** — pass these explicitly on every request. Some LFM2.5 checkpoints do not ship sampling defaults in `generation_config.json`, so the server will not apply them for you. `top_k`, `min_p`, and `repetition_penalty` are not standard OpenAI `chat.completions` fields — pass them through **`extra_body`** and SGLang forwards them to its sampler. Do not set `max_tokens` unless you intend to cap output, as it can truncate a response (or a reasoning model's chain-of-thought) mid-stream.

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>temperature</th>
      <th>extra\_body (sampler)</th>
    </tr>
  </thead>

  <tbody>
    <tr><td>LFM2.5-8B-A1B</td><td>0.2</td><td>`{"top_k": 80, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-1.2B-Instruct</td><td>0.1</td><td>`{"top_k": 50, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-1.2B-Thinking</td><td>0.05</td><td>`{"top_k": 50, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-350M</td><td>0.1</td><td>`{"top_k": 50, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-230M</td><td>0.1</td><td>`{"top_k": 50, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-1.2B-JP-202606</td><td>0.1</td><td>`{"top_k": 50, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-1.2B-JP</td><td>0.3</td><td>`{"min_p": 0.15, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-VL-1.6B (text)</td><td>0.1</td><td>`{"min_p": 0.15, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-VL-450M (text)</td><td>0.1</td><td>`{"min_p": 0.15, "repetition_penalty": 1.05}`</td></tr>
    <tr><td>LFM2.5-1.2B-Base</td><td>0.3</td><td>`{"min_p": 0.15, "repetition_penalty": 1.05}`</td></tr>
  </tbody>
</table>

## 3. Advanced Usage

### 3.1 Basic Usage

A single client with the recommended sampling presets applied per model (the examples in the following sections reuse this `chat` helper):

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

# Non-OpenAI fields (top_k / min_p / repetition_penalty) ride in extra_body.
SAMPLING = {
    "LiquidAI/LFM2.5-8B-A1B":         dict(temperature=0.2,  extra_body={"top_k": 80, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-1.2B-Instruct":  dict(temperature=0.1,  extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-1.2B-Thinking":  dict(temperature=0.05, extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-350M":           dict(temperature=0.1,  extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-230M":           dict(temperature=0.1,  extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-1.2B-JP-202606": dict(temperature=0.1,  extra_body={"top_k": 50, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-VL-1.6B":        dict(temperature=0.1,  extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
    "LiquidAI/LFM2.5-VL-450M":        dict(temperature=0.1,  extra_body={"min_p": 0.15, "repetition_penalty": 1.05}),
}

def chat(model, messages, **overrides):
    cfg = SAMPLING[model]
    body = cfg["extra_body"] | overrides.pop("extra_body", {})
    return client.chat.completions.create(
        model=model, messages=messages,
        temperature=cfg["temperature"], extra_body=body, **overrides,
    )

resp = chat(
    "LiquidAI/LFM2.5-1.2B-Instruct",
    [{"role": "user", "content": "What is C. elegans? Answer in one sentence."}],
)
print(resp.choices[0].message.content)
```

### 3.2 Reasoning

The 8B-A1B and 1.2B-Thinking checkpoints emit chain-of-thought as a built-in behavior. The Deploy panel launches them with the matching `--reasoning-parser`, which separates the thinking process into `reasoning_content`:

```python Example theme={null}
resp = chat(
    "LiquidAI/LFM2.5-8B-A1B",
    [{"role": "user", "content": "If a train travels 60 km/h for 2.5 hours, how far does it go?"}],
)
msg = resp.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Answer:", msg.content)
```

### 3.3 Tool Calling

LFM2.5 writes Pythonic tool calls. With `--tool-call-parser lfm2` (already part of the launch command) they are surfaced as standard `message.tool_calls`:

```python Example theme={null}
resp = chat(
    "LiquidAI/LFM2.5-1.2B-Instruct",
    [{"role": "user", "content": "What's the weather in Paris?"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"],
            },
        },
    }],
)
for call in resp.choices[0].message.tool_calls or []:
    print(call.function.name, call.function.arguments)
```

Tool calling is supported on 8B-A1B, 1.2B-Thinking, 1.2B-Instruct, 350M, 230M, 1.2B-JP-202606, VL-1.6B, and VL-450M. For the **VL** models it is text-turn-only — do not combine an image and tools in the same turn.

### 3.4 Vision Input

The VL models (VL-1.6B and VL-450M) accept images via standard OpenAI multimodal content blocks. Base64 data URIs (`data:image/jpeg;base64,...`) work in place of a URL:

```python Example theme={null}
resp = chat(
    "LiquidAI/LFM2.5-VL-1.6B",
    [{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {
                "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
            {"type": "text", "text": "What is in this image?"},
        ],
    }],
)
print(resp.choices[0].message.content)
```

### 3.5 Base Checkpoints

Each size ships a pre-trained Base repo — [LFM2.5-230M-Base](https://huggingface.co/LiquidAI/LFM2.5-230M-Base), [LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base), [LFM2.5-350M-Base](https://huggingface.co/LiquidAI/LFM2.5-350M-Base), and [LFM2.5-8B-A1B-Base](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B-Base) — intended for fine-tuning and continued pre-training.

The repos ship a ChatML-style chat template, so `chat.completions` requests format normally. The checkpoints have no post-training, though — don't expect instruction following. For raw text continuation:

```python Example theme={null}
comp = client.completions.create(
    model="LiquidAI/LFM2.5-1.2B-Base",
    prompt="The capital of France is",
    temperature=0.3,
    extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
)
print(comp.choices[0].text)
```


# LongCat-2.0
Source: https://docs.sglang.io/cookbook/autoregressive/Meituan/LongCat-2.0

Deploy LongCat-2.0-FP8 with SGLang - config-driven recipes for Meituan's 1.6T sparse MoE model on B300, B200, H200, and H20 GPUs.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). LongCat-2.0 support is on SGLang `main`; use a nightly wheel or rolling nightly Docker image until the next tagged release includes it. The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
>
> # Choose the nightly wheel index for your CUDA runtime.
> SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu130  # B300 / CUDA 13
> # SGLANG_WHL_INDEX=https://docs.sglang.ai/whl/cu129  # CUDA 12.9
>
> uv pip install --prerelease=allow --extra-index-url "${SGLANG_WHL_INDEX}" "sglang[all]"
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> # Choose the rolling nightly image for your hardware.
> SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev-cu13  # B300 / CUDA 13
> # SGLANG_DOCKER_IMAGE=lmsysorg/sglang:dev  # Other supported hardware
>
> docker pull "${SGLANG_DOCKER_IMAGE}"
> ```
>
> For how to launch the image, see [Install -> Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + recipe to generate the launch command. LongCat-2.0 currently exposes one model-card-aligned serving strategy:

* **Balanced** - the validated B300 recipe and the 2-node H200/B200/H20 topology use TP/EP parallelism with LongCat sparse attention prefill.

> [!WARNING]
>
> All recipes here run the LongCat sparse-attention indexer top-k on the default `--dsa-topk-backend sgl-kernel`. Other top-k backend choices have not been fully validated on LongCat-2.0.

> [!NOTE]
>
> The B300 single-node recipe was validated end-to-end with CUDA graph capture enabled. H200, B200, and H20 are shown as 2-node recipes because LongCat-2.0-FP8 needs 16 ranks for those GPU memory profiles.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

[LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8) is the FP8 checkpoint of Meituan LongCat-2.0, a large sparse Mixture-of-Experts language model with 1.6T total parameters and about 48B activated parameters per token. It combines LongCat Sparse Attention (LSA), expert parallel MoE layers, and an n-gram/token-table embedding path for serving long-context workloads efficiently.

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Architecture</th>
      <th>Serving precision</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8)**</td>
      <td>Sparse MoE · LongCat Sparse Attention · n-gram embedding</td>
      <td>FP8 weights, BF16 KV cache</td>
    </tr>
  </tbody>
</table>

**Resources:** [LongCat-2.0-FP8](https://huggingface.co/meituan-longcat/LongCat-2.0-FP8).

## 2. Configuration Tips

* **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint.
* **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you.
* **LongCat sparse attention.** Keep `--dsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path.
* **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster.
* **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time.
* **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout.
* **Host, port, and ranks.** Use the command panel environment fields for `HOST_IP`, `PORT`, `NODE0_IP`, and `NODE_RANK` instead of hardcoding them in the recipe.

## 3. Advanced Usage

### 3.1 Test the deployment

> **Chat completion example (cURL)**
>
> ```bash Command theme={null}
> curl http://localhost:30000/v1/chat/completions \
>   -H "Content-Type: application/json" \
>   -d '{
>     "model": "meituan-longcat/LongCat-2.0-FP8",
>     "messages": [
>       {"role": "user", "content": "A shop has 17 apples and sells 8. Then it buys 6 more. How many apples are there? Answer with only the final number."}
>     ],
>     "max_tokens": 32,
>     "chat_template_kwargs": {"enable_thinking": false}
>   }'
> ```

> **Expected output**
>
> ```text Output theme={null}
> 15
> ```

> **OpenAI-compatible client (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(
>     base_url="http://localhost:30000/v1",
>     api_key="EMPTY",
> )
>
> response = client.chat.completions.create(
>     model="meituan-longcat/LongCat-2.0-FP8",
>     messages=[
>         {
>             "role": "user",
>             "content": "Solve: A shop has 17 apples and sells 8, then buys 6 more. Answer with only the final number.",
>         }
>     ],
>     max_tokens=32,
>     extra_body={"chat_template_kwargs": {"enable_thinking": False}},
> )
>
> print(response.choices[0].message.content)
> ```

> **Example output**
>
> ```text Output theme={null}
> 15
> ```

## 4. Validation

The B300 recipe was validated with `meituan-longcat/LongCat-2.0-FP8` on 8x B300 using the command generated above.

<table>
  <thead>
    <tr>
      <th>Evaluation</th>
      <th>Examples</th>
      <th>Accuracy</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>GSM8K</td>
      <td>200</td>
      <td>98.0%</td>
    </tr>

    <tr>
      <td>GSM8K</td>
      <td>1314</td>
      <td>95.8904109589041%</td>
    </tr>
  </tbody>
</table>

CUDA graph was enabled, and decode CUDA graph capture completed successfully during serving validation.


# Llama-3.1
Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama3.1



## 1. Model Introduction

Llama 3.1 is a collection of pretrained and instruction tuned generative models, released in July 2024 by Meta. These models are available in 8B, 70B and 405B sizes, with the 405B variant being the most capable fully-open source model at the time.

These models bring open intelligence to all, with several new features and improvements:

* **Stronger General Intelligence**: These models showcase significant improvements in coding, state-of-the-art tool use, and overall stronger reasoning capabilities.
* **Extended Context Length**: Llama 3.1 extends the context length to 128K tokens to improve performance over long context tasks such as summarization and code reasoning.
* **Tool Use**: Llama 3.1 is trained to interact with a search engine, python interpreter and mathematical engine, and also improves zero-shot tool use capabilities to interact with potentially unseen tools.
* **Multilinguality**: Llama 3.1 supports 7 languages in addition to English: French, German, Hindi, Italian, Portuguese, Spanish, and Thai.

For further details, please refer to the [Llama 3.1 blog](https://ai.meta.com/blog/meta-llama-3-1/) and the [Llama 3.1 model card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md).note

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Llama 3.1 collection of models.

Run this Command:

<pre />

### 3.2 Configuration Tips

**Speculative Decoding (NVIDIA GPUs):**

* Using Speculative Decoding for latency-sensitive scenarios:
  * `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
  * `--speculative-num-steps 3`: Number of speculative verification rounds
  * `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
  * `--speculative-num-draft-tokens 4`: Number of draft tokens per step
  * `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`](https://huggingface.co/yuhuili/EAGLE3-LLaMA3.1-Instruct-8B).

**AMD GPU Deployment:**

* **Hardware-Aware TP**: MI355X (256GB memory) supports lower TP values compared to MI300X/MI325X (192GB)
* **Verified TP Configurations**:
  * MI300X/MI325X: 405B BF16 (TP=8), 405B FP8 (TP=4), 70B/8B (TP=1)
  * MI355X: 405B BF16 (TP=4), 405B FP8 (TP=2), 70B/8B (TP=1)
* **FP8 Model Variants**:
  * 405B: Use Meta's official `meta-llama/Llama-3.1-405B-Instruct-FP8`
  * 70B/8B: Use AMD's optimized `amd/Llama-3.1-{size}-Instruct-FP8-KV`
* **Tool Calling**: Enable with `--tool-call-parser llama3` for Instruct models

**Xeon CPU Deployment:**

* Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

SGLang exposes an OpenAI-compatible endpoint. First, start the server

```shell Command theme={null}
sglang serve \
  --model-path  Meta-Llama/Llama-3.1-405B-Instruct \
  --tp 8
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="Meta-Llama/Llama-3.1-405B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
    ],
    temperature=0.2,
    max_tokens=512,
)

print(resp.choices[0].message.content)
```

**Output Example:**

````text Output theme={null}
**Exponential Backoff Retry Function in Python**
=====================================================

Below is a Python function that uses the `requests` library to retry a request with exponential backoff.

```python
import requests
import time
import random

def exponential_backoff_retry(url, method, retries=3, backoff_factor=1, max_delay=60):
    """
    Retry a request with exponential backoff.

    Args:
        url (str): The URL to make the request to.
        method (str): The HTTP method to use (e.g. 'GET', 'POST', etc.).
        retries (int): The number of retries to attempt. Defaults to 3.
        backoff_factor (int): The factor to multiply the delay by for each retry. Defaults to 1.
        max_delay (int): The maximum delay to wait between retries in seconds. Defaults to 60.

    Returns:
        The response object from the successful request.
    """

    delay = 1
    for attempt in range(retries + 1):
        try:
            response = requests.request(method, url)
            response.raise_for_status()  # Raise an exception for HTTP errors
            return response
        except requests.RequestException as e:
            if attempt < retries:
                # Calculate the delay for this retry
                delay = min(delay * backoff_factor, max_delay)
                # Add a random jitter to the delay to prevent thundering herd problem
                delay += random.uniform(0, delay * 0.1)
                # Wait for the calculated delay before retrying
                time.sleep(delay)
            else:
                # If all retries have failed, raise the exception
                raise e
...
````

### 4.2 Advanced Usage

#### 4.2.1 Tool Calling

Llama3 supports tool calling capabilities. First, start the server with tool call parser enabled:

```shell Command theme={null}
sglang serve \
  --model-path  Meta-Llama/Llama-3.1-405B-Instruct \
  --tool-call-parser llama3 \
  --tp 8
```

**Python Example**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:8000/v1")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "The city to find the weather for, e.g. 'San Francisco'",
                    },
                    "unit": {
                        "type": "string",
                        "description": "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["city", "unit"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-405B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What's the weather like in Boston today?",
        }
    ],
    temperature=0.7,
    stream=True,
    tools=tools,
)


arguments = []

tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

Reference: [SGLang Tool Parser Documentation](../../../docs/advanced_features/tool_parser#openai-compatible-api)

**Output Example**

```text Output theme={null}
🔧 Tool Call: get_weather
   Arguments: {"city": "Boston", "unit": "fahrenheit"}
```

**Handling Tool Call Results**
After getting the tool call, you can execute the function:

```python Example theme={null}
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather like in Boston today?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Boston", "unit": "fahrenheit"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Boston", "fahrenheit")
    }
]

final_response = client.chat.completions.create(
    model="Meta-Llama/Llama-3.1-405B-Instruct",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The current weather in Boston is **22°C** and **sunny**. A perfect day to spend outside"
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA A100 GPU (8x)
* Model: Meta-Llama/Llama-3.1-70B
* Tensor Parallelism: 8
* sglang version: 0.5.6

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.

#### 5.1.1 Standard Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
  --model-path Meta-Llama/Llama-3.1-70B \
  --tp 8
```

##### 5.1.1.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
sglang serve \
  --backend sglang \
  --model Meta-Llama/Llama-3.1-70B \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  79.81
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4208
Request throughput (req/s):              0.13
Input token throughput (tok/s):          76.44
Output token throughput (tok/s):         52.88
Peak output token throughput (tok/s):    54.00
Peak concurrent requests:                2
Total token throughput (tok/s):          129.32
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7977.81
Median E2E Latency (ms):                 6373.48
---------------Time to First Token----------------
Mean TTFT (ms):                          131.61
Median TTFT (ms):                        131.77
P99 TTFT (ms):                           163.88
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          18.63
Median TPOT (ms):                        18.63
P99 TPOT (ms):                           18.65
---------------Inter-Token Latency----------------
Mean ITL (ms):                           18.64
Median ITL (ms):                         18.64
P95 ITL (ms):                            18.69
P99 ITL (ms):                            18.74
Max ITL (ms):                            21.95
==================================================
```

##### 5.1.1.2 Medium Concurrency

```shell Command theme={null}
sglang serve \
  --backend sglang \
  --model-path Meta-Llama/Llama-3.1-70B \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  79.47
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    38450
Request throughput (req/s):              1.01
Input token throughput (tok/s):          499.17
Output token throughput (tok/s):         513.48
Peak output token throughput (tok/s):    674.00
Peak concurrent requests:                20
Total token throughput (tok/s):          1012.65
Concurrency:                             13.47
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   13376.67
Median E2E Latency (ms):                 14130.48
---------------Time to First Token----------------
Mean TTFT (ms):                          264.84
Median TTFT (ms):                        147.02
P99 TTFT (ms):                           791.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          26.09
Median TPOT (ms):                        26.08
P99 TPOT (ms):                           34.65
---------------Inter-Token Latency----------------
Mean ITL (ms):                           25.76
Median ITL (ms):                         23.95
P95 ITL (ms):                            24.72
P99 ITL (ms):                            98.32
Max ITL (ms):                            478.92
==================================================
```

##### 5.1.1.3 High Concurrency

```shell Command theme={null}
sglang serve \
  --backend sglang \
  --model-path Meta-Llama/Llama-3.1-70B \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  131.64
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    243641
Request throughput (req/s):              3.80
Input token throughput (tok/s):          1897.87
Output token throughput (tok/s):         1919.38
Peak output token throughput (tok/s):    3100.00
Peak concurrent requests:                107
Total token throughput (tok/s):          3817.25
Concurrency:                             89.70
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   23616.71
Median E2E Latency (ms):                 22770.44
---------------Time to First Token----------------
Mean TTFT (ms):                          245.98
Median TTFT (ms):                        184.22
P99 TTFT (ms):                           1251.67
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          47.19
Median TPOT (ms):                        48.67
P99 TPOT (ms):                           56.37
---------------Inter-Token Latency----------------
Mean ITL (ms):                           46.34
Median ITL (ms):                         33.46
P95 ITL (ms):                            108.61
P99 ITL (ms):                            166.11
Max ITL (ms):                            1107.09
==================================================
```

#### 5.1.2 Summarization Scenario Benchmark

##### 5.1.2.1 Low Concurrency

```shell Command theme={null}
sglang serve \
  --backend sglang \
  --model-path Meta-Llama/Llama-3.1-70B\
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  83.25
Total input tokens:                      41941
Total input text tokens:                 41941
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.12
Input token throughput (tok/s):          503.77
Output token throughput (tok/s):         50.69
Peak output token throughput (tok/s):    54.00
Peak concurrent requests:                2
Total token throughput (tok/s):          554.46
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8322.45
Median E2E Latency (ms):                 6873.36
---------------Time to First Token----------------
Mean TTFT (ms):                          395.25
Median TTFT (ms):                        318.02
P99 TTFT (ms):                           850.80
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          18.80
Median TPOT (ms):                        18.81
P99 TPOT (ms):                           19.03
---------------Inter-Token Latency----------------
Mean ITL (ms):                           18.83
Median ITL (ms):                         18.81
P95 ITL (ms):                            19.06
P99 ITL (ms):                            19.08
Max ITL (ms):                            23.08
==================================================
```

##### 5.1.2.2 Medium Concurrency

```shell Command theme={null}
sglang serve \
  --backend sglang \
  --model-path Meta-Llama/Llama-3.1-70B \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  107.12
Total input tokens:                      300020
Total input text tokens:                 300020
Total input vision tokens:               0
Total generated tokens:                  41669
Total generated tokens (retokenized):    41603
Request throughput (req/s):              0.75
Input token throughput (tok/s):          2800.81
Output token throughput (tok/s):         389.00
Peak output token throughput (tok/s):    624.00
Peak concurrent requests:                19
Total token throughput (tok/s):          3189.81
Concurrency:                             14.18
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   18988.30
Median E2E Latency (ms):                 20290.66
---------------Time to First Token----------------
Mean TTFT (ms):                          603.42
Median TTFT (ms):                        531.82
P99 TTFT (ms):                           2607.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          36.94
Median TPOT (ms):                        36.73
P99 TPOT (ms):                           79.19
---------------Inter-Token Latency----------------
Mean ITL (ms):                           35.36
Median ITL (ms):                         25.72
P95 ITL (ms):                            27.07
P99 ITL (ms):                            439.74
Max ITL (ms):                            2529.51
==================================================
```

##### 5.1.2.3 High Concurrency

```shell Command theme={null}
sglang serve \
  --backend sglang \
  --model-path Meta-Llama/Llama-3.1-70B \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  215.66
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total input vision tokens:               0
Total generated tokens:                  170000
Total generated tokens (retokenized):    169035
Request throughput (req/s):              1.48
Input token throughput (tok/s):          5906.92
Output token throughput (tok/s):         788.27
Peak output token throughput (tok/s):    1920.00
Peak concurrent requests:                69
Total token throughput (tok/s):          6695.19
Concurrency:                             60.01
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   40443.85
Median E2E Latency (ms):                 39813.12
---------------Time to First Token----------------
Mean TTFT (ms):                          633.32
Median TTFT (ms):                        616.38
P99 TTFT (ms):                           1912.97
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          74.95
Median TPOT (ms):                        82.85
P99 TPOT (ms):                           118.46
---------------Inter-Token Latency----------------
Mean ITL (ms):                           75.08
Median ITL (ms):                         34.12
P95 ITL (ms):                            261.18
P99 ITL (ms):                            828.12
Max ITL (ms):                            1970.03
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Results**:

```text Output theme={null}
Accuracy: 0.830
Invalid: 0.000
Latency: 11.794 s
Output throughput: 1406.961 token/s
```


# Llama-3.3-70B
Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama3.3-70B



## 1. Model Introduction

[Llama-3.3-70B-Instruct](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct) is Meta's latest 70 billion parameter instruction-tuned language model, featuring improved performance and efficiency over Llama 3.1. With a 128K token context window and enhanced capabilities across reasoning, coding, and multilingual tasks, Llama 3.3 delivers state-of-the-art results while maintaining accessibility for production deployment.

**Key Features:**

* **Enhanced Performance**: Improved instruction following, reasoning, and task completion over Llama 3.1
* **Tool Calling**: Native support for function calling and tool use scenarios
* **Multilingual Support**: Optimized for 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai)
* **Extended Context**: 128K token context window for processing long documents and complex tasks
* **Efficient Deployment**: 70B parameters enable deployment on single GPU with AMD MI300X

**License:**
Llama 3.3 is licensed under the Llama 3.3 Community License. See [LICENSE](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct/blob/main/LICENSE) for details.

For more details, please refer to the [official Llama models repository](https://github.com/meta-llama/llama-models).

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X), Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)) and Intel Xeon CPUs.

### 3.1 Interactive Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU, Intel Arc Pro B-Series GPUs or Intel Xeon CPUs setup.

Run this Command:

<pre />

### 3.2 Configuration Tips

**AMD GPU Deployment:**

* All AMD GPUs (MI300X, MI325X, MI355X) support TP=1 for both BF16 and FP8 variants
* **FP8 Model Variant**: Use AMD's optimized `amd/Llama-3.3-70B-Instruct-FP8-KV`
* **Tool Calling**: Enable with `--tool-call-parser llama3` for function calling support
* **Higher Throughput**: Optional TP=2 or TP=4 can be used for increased throughput

**Xeon CPU Deployment:**

Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Tool Calling

Llama 3.3 70B Instruct supports native tool calling. Enable the tool parser during deployment:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.3-70B-Instruct \
  --tool-call-parser llama3 \
  --tp 1 \
  --host 0.0.0.0 \
  --port 30000
```

**Python Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request
response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[
        {"role": "user", "content": "What's the weather in Tokyo?"}
    ],
    tools=tools,
    temperature=0.7
)

# Check for tool calls
message = response.choices[0].message
if message.tool_calls:
    tool_call = message.tool_calls[0]
    print(f"Function: {tool_call.function.name}")
    print(f"Arguments: {tool_call.function.arguments}")
```

**Handling Tool Call Results:**

```python Example theme={null}
# After executing the function, send the result back
def get_weather(location, unit="celsius"):
    # Your weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Build conversation with tool result
messages = [
    {"role": "user", "content": "What's the weather in Tokyo?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Tokyo", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Tokyo", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The current weather in Tokyo is 22°C and sunny. A perfect day!"
```

#### 4.2.2 Long Context Processing

Leverage the 128K context window for processing long documents:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Example with long document
long_document = "..." * 10000  # Your long document here

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[
        {"role": "user", "content": f"Summarize this document:\n\n{long_document}"}
    ],
    temperature=0.7,
    max_tokens=1000
)

print(response.choices[0].message.content)
```

## 5. Benchmarking

Use the SGLang benchmarking suite to test model performance with different workload patterns:

### 5.1 Basic Benchmark Command

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 1000 \
  --random-input 1024 \
  --random-output 1024 \
  --max-concurrency 16
```

### 5.2 Adjusting Benchmark Parameters

**Input/Output Length**: Adjust `--random-input` and `--random-output` to test different workload patterns:

* Short conversations: `--random-input 1024 --random-output 1024`
* Long outputs: `--random-input 1024 --random-output 8192`
* Long inputs: `--random-input 8192 --random-output 1024`

**Concurrency Levels**: Adjust `--max-concurrency` to test different load scenarios:

* Low concurrency (latency-focused): `--max-concurrency 1 --num-prompts 100`
* Medium concurrency (balanced): `--max-concurrency 16 --num-prompts 1000`
* High concurrency (throughput-focused): `--max-concurrency 100 --num-prompts 2000`

***

## 📚 Additional Resources

* [Meta Llama Models Repository](https://github.com/meta-llama/llama-models)
* [Llama 3.3 Model Card](https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct)
* [SGLang Documentation](/)
* [AMD ROCm Documentation](https://rocm.docs.amd.com/)


# Llama 4
Source: https://docs.sglang.io/cookbook/autoregressive/Meta/Llama4



## 1. Model Introduction

[Llama 4](https://github.com/meta-llama/llama-models/blob/main/models/llama4/MODEL_CARD.md) is Meta's latest generation of open-source LLM model with industry-leading performance.

SGLang has supported Llama 4 Scout (109B) and Llama 4 Maverick (400B) since [v0.4.5](https://github.com/sgl-project/sglang/releases/tag/v0.4.5).

Ongoing optimizations are tracked in the [Roadmap](https://github.com/sgl-project/sglang/issues/5118).

This generation delivers comprehensive upgrades across the board:

The highly capable Llama 4 Maverick with 17B active parameters out of \~400B total, with 128 experts.
The efficient Llama 4 Scout also has 17B active parameters out of \~109B total, using just 16 experts.
Both models leverage early fusion for native multimodality, enabling them to process text and image inputs. Maverick and Scout are both trained on up to 40 trillion tokens on data encompassing 200 languages (with specific fine-tuning support for 12 languages including Arabic, Spanish, German, and Hindi).

For more details, please refer to the official llama4 Repository:[https://www.llama.com/models/llama-4/](https://www.llama.com/models/llama-4/)

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

Run this Command:

<pre />

### 3.2 Configuration Tips

* **OOM Mitigation:** Reduce `--context-length` to avoid GPU out-of-memory. Recommended: Scout up to 1M on 8×H100, up to 2.5M on 8×H200; Maverick doesn't need context-length set on 8×H200. With hybrid KV cache enabled, Scout can reach 5M on 8×H100 and 10M on 8×H200.
* **Attention Backend Auto-Selection:** SGLang automatically picks the optimal backend. Manual override with `--attention-backend`:
  * Blackwell (B200/GB200): `trtllm_mha`
  * Hopper (H100/H200): `fa3`
  * AMD GPUs: `aiter`
  * Intel XPU: `intel_xpu`
  * Other: `triton`
* **Chat Template:** Add `--chat-template llama-4` for chat completion tasks.
* **Multi-Modal:** Add `--enable-multimodal` to enable image input support.
* **Hybrid KV Cache:** Set `--swa-full-tokens-ratio` to control the ratio of SWA (local attention) KV tokens to full-attention KV tokens (default: 0.8, range: 0–1).
* **EAGLE Speculative Decoding:** Supported for Llama 4 Scout and Maverick via EAGLE3. Enable with the interactive command generator above.
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Launch the docker

```shell Command theme={null}
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```

```shell Command theme={null}
docker run -d -it --ipc=host --network=host --privileged \
  --cap-add=CAP_SYS_ADMIN \
  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
  --group-add video --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /:/work \
  -e SHELL=/bin/bash \
  --name Llama4 \
  lmsysorg/sglang:v0.5.9-rocm720-mi30x \
  /bin/bash
```

#### 4.2.2 Launch the server

### Llama-4-Scout

8-GPU deployment command:

```bash Command theme={null}
sglang serve \
  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tp 8 \
  --context-length 1000000 \
  --trust-remote-code
```

### Llama-4-Maverick

8-GPU deployment command:

```bash Command theme={null}
sglang serve \
  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --tp 8 \
  --trust-remote-code
```

#### 4.2.3 EAGLE Speculative Decoding

SGLang supports Llama 4 Maverick (400B) with [EAGLE speculative decoding](../../../docs/advanced_features/speculative_decoding). Enable with the EAGLE3 algorithm and the SGLang EAGLE3 draft model:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --trust-remote-code \
  --tp 8
```

## 5. Benchmark

### 5.1 Speed Benchmark (Scout)

Test Environment:

Hardware: AMD MI300x GPU

Model: Llama-4-Scout

Tensor Parallelism: 8

sglang version: 0.5.9

* **Model Deployment**

```bash Command theme={null}
sglang serve \
  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --tp 8 \
  --context-length 1000000 \
  --trust-remote-code
```

### 5.1.1 Low Concurrency (Latency-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  74.62
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4211
Request throughput (req/s):              0.14
Input token throughput (tok/s):          82.88
Output token throughput (tok/s):         57.42
Peak output token throughput (tok/s):    146.00
Peak concurrent requests:                2
Total token throughput (tok/s):          140.20
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7459.48
Median E2E Latency (ms):                 4489.77
---------------Time to First Token----------------
Mean TTFT (ms):                          4246.98
Median TTFT (ms):                        68.57
P99 TTFT (ms):                           48091.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.49
Median TPOT (ms):                        7.40
P99 TPOT (ms):                           7.40
---------------Inter-Token Latency----------------
Mean ITL (ms):                           7.49
Median ITL (ms):                         7.49
P95 ITL (ms):                            7.47
P99 ITL (ms):                            7.52
Max ITL (ms):                            10.44
==================================================
```

### 5.1.2 Medium Concurrency (Balanced)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  45.41
Total input tokens:                      49668
Total input text tokens:                 49668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40516
Request throughput (req/s):              2.26
Input token throughput (tok/s):          1120.46
Output token throughput (tok/s):         1152.47
Peak output token throughput (tok/s):    1520.00
Peak concurrent requests:                21
Total token throughput (tok/s):          2272.84
Concurrency:                             14.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6089.22
Median E2E Latency (ms):                 6568.80
---------------Time to First Token----------------
Mean TTFT (ms):                          124.44
Median TTFT (ms):                        87.42
P99 TTFT (ms):                           268.72
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          11.88
Median TPOT (ms):                        12.00
P99 TPOT (ms):                           15.49
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.72
Median ITL (ms):                         10.54
P95 ITL (ms):                            11.22
P99 ITL (ms):                            67.88
Max ITL (ms):                            74.05
==================================================
```

### 5.1.3 High Concurrency (Throughput-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  85.84
Total input tokens:                      249841
Total input text tokens:                 249841
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    250498
Request throughput (req/s):              5.84
Input token throughput (tok/s):          2910.84
Output token throughput (tok/s):         2944.82
Peak output token throughput (tok/s):    4100.00
Peak concurrent requests:                110
Total token throughput (tok/s):          5854.65
Concurrency:                             92.24
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   15844.00
Median E2E Latency (ms):                 15262.56
---------------Time to First Token----------------
Mean TTFT (ms):                          204.46
Median TTFT (ms):                        129.96
P99 TTFT (ms):                           528.54
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          41.56
Median TPOT (ms):                        42.90
P99 TPOT (ms):                           47.48
---------------Inter-Token Latency----------------
Mean ITL (ms):                           40.99
Median ITL (ms):                         24.46
P95 ITL (ms):                            84.46
P99 ITL (ms):                            87.64
Max ITL (ms):                            226.06
==================================================
```

### 5.2 Speed Benchmark (Maverick)

Test Environment:

Hardware: AMD MI300x GPU

Model: Llama-4-Maverick

Tensor Parallelism: 8

sglang version: 0.5.9

* **Model Deployment**

```bash Command theme={null}
sglang serve \
  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --tp 8 \
  --context-length 1000000 \
  --trust-remote-code
```

### 5.2.1 Low Concurrency (Latency-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  68.08
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4202
Request throughput (req/s):              0.15
Input token throughput (tok/s):          89.62
Output token throughput (tok/s):         61.99
Peak output token throughput (tok/s):    168.00
Peak concurrent requests:                2
Total token throughput (tok/s):          151.61
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6805.62
Median E2E Latency (ms):                 2733.91
---------------Time to First Token----------------
Mean TTFT (ms):                          4296.56
Median TTFT (ms):                        57.45
P99 TTFT (ms):                           38633.95
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.95
Median TPOT (ms):                        5.96
P99 TPOT (ms):                           5.97
---------------Inter-Token Latency----------------
Mean ITL (ms):                           5.96
Median ITL (ms):                         5.96
P95 ITL (ms):                            6.02
P99 ITL (ms):                            6.08
Max ITL (ms):                            7.02
==================================================
```

### 5.2.2 Medium Concurrency (Balanced)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  30.72
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40923
Request throughput (req/s):              2.60
Input token throughput (tok/s):          1291.39
Output token throughput (tok/s):         1328.41
Peak output token throughput (tok/s):    1760.00
Peak concurrent requests:                22
Total token throughput (tok/s):          2619.80
Concurrency:                             13.92
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5345.15
Median E2E Latency (ms):                 5679.73
---------------Time to First Token----------------
Mean TTFT (ms):                          259.30
Median TTFT (ms):                        72.60
P99 TTFT (ms):                           1063.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.53
Median TPOT (ms):                        10.22
P99 TPOT (ms):                           20.27
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.99
Median ITL (ms):                         9.10
P95 ITL (ms):                            9.87
P99 ITL (ms):                            55.62
Max ITL (ms):                            868.54
==================================================
```

### 5.2.3 High Concurrency (Throughput-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  90.95
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    251625
Request throughput (req/s):              5.50
Input token throughput (tok/s):          2746.77
Output token throughput (tok/s):         2777.90
Peak output token throughput (tok/s):    3700.00
Peak concurrent requests:                109
Total token throughput (tok/s):          5524.67
Concurrency:                             93.04
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16924.17
Median E2E Latency (ms):                 16294.85
---------------Time to First Token----------------
Mean TTFT (ms):                          188.19
Median TTFT (ms):                        128.96
P99 TTFT (ms):                           534.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.63
Median TPOT (ms):                        35.37
P99 TPOT (ms):                           38.26
---------------Inter-Token Latency----------------
Mean ITL (ms):                           33.19
Median ITL (ms):                         27.66
P95 ITL (ms):                            76.91
P99 ITL (ms):                            78.82
Max ITL (ms):                            268.17
==================================================
```

### 5.3 Accuracy Benchmark

#### 5.3.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* Llama-4-Scout-17B-16E-Instruct

```text Output theme={null}
Accuracy: 0.945
Invalid: 0.000
Latency: 12.731 s
Output throughput: 1595.418 token/s
```

* Llama-4-Maverick-17B-128E-Instruct

```text Output theme={null}
Accuracy: 0.895
Invalid: 0.000
Latency: 9.739 s
Output throughput: 2405.505 token/s
```

#### 5.3.2 MMLU-Pro with sgl-eval

The table below records historical lm-eval measurements on 8×H100 alongside [Meta's official benchmark numbers](https://ai.meta.com/blog/llama-4-multimodal-intelligence/) (reproduction details: [PR #5092](https://github.com/sgl-project/sglang/pull/5092)). These scores were not produced by the sgl-eval commands below; rerun the evaluation for results using sgl-eval's prompts and grading.

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Official</th>
      <th>SGLang</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Llama-4-Scout-17B-16E-Instruct</td>
      <td>74.3</td>
      <td>75.2</td>
    </tr>

    <tr>
      <td>Llama-4-Maverick-17B-128E-Instruct</td>
      <td>80.5</td>
      <td>80.7</td>
    </tr>
  </tbody>
</table>

Install the evaluation package before running either command:

```bash theme={null}
pip install sgl-eval==0.1.0
```

**Scout:**

```bash Command theme={null}
# Start the server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --port 30000 \
  --tp 8 \
  --mem-fraction-static 0.8 \
  --context-length 65536

# Run MMLU-Pro
sgl-eval run mmlu_pro \
  --base-url http://localhost:30000/v1 \
  --model meta-llama/Llama-4-Scout-17B-16E-Instruct \
  --max-tokens 2048 \
  --num-threads 128
```

**Maverick:**

```bash Command theme={null}
# Start the server
python -m sglang.launch_server \
  --model-path meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --port 30000 \
  --tp 8 \
  --mem-fraction-static 0.8 \
  --context-length 65536

# Run MMLU-Pro
sgl-eval run mmlu_pro \
  --base-url http://localhost:30000/v1 \
  --model meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --max-tokens 2048 \
  --num-threads 128
```


# Muse Glimmer
Source: https://docs.sglang.io/cookbook/autoregressive/Meta/MuseGlimmer

A multimodal reasoning model served from a BF16, NVFP4 + MXFP8, vendor GGUF, or MLX checkpoint.

## Deployment

> **Install SGLang**
>
> See the [official SGLang installation guide](../../../docs/get-started/install) for all installation methods and hardware platforms. The steps below match the **Python** and **Docker** options in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> # Muse Glimmer support is not in a release yet -- build the PR branch:
> # https://github.com/sgl-project/sglang/pull/34262
> git clone -b muse-glimmer https://github.com/sgl-project/sglang.git
> cd sglang
> uv pip install --prerelease=allow -e "python[all]"
> ```
>
> Run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-muse-glimmer
> ```
>
> See [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker) to start the image. Replace the inner `sglang serve ...` command with the command from the panel below.

Select a checkpoint format. Select whether to use speculative decoding:

* **Standard**: Use normal autoregressive decoding.
* **DFlash**: Use speculative decoding with the DFlash draft model. The draft serves as published, with no conversion step. See [§2](#2-configuration-tips).

## Playground

Use the Playground to test SGLang features that are not in the verified matrix. The Deploy panel above shows only combinations that the SGLang team has verified. The Playground lets you add more options to the command from the Deploy panel.

## 1. Model Introduction

Muse Glimmer is a multimodal reasoning model. You can serve Muse Glimmer in four formats:

* A BF16 checkpoint (`MuseGlimmerForConditionalGeneration`).
* A set of vendor GGUF files.
* A ready-to-serve NVFP4 + MXFP8 checkpoint.
* Three MLX repacks for Apple Silicon.

<table>
  <thead>
    <tr>
      <th>Form</th>
      <th>Source</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BF16**</td>
      <td>`meta-models/Muse-Glimmer-30B`</td>
      <td>Supports image input.</td>
    </tr>

    <tr>
      <td>**GGUF Q4\_K\_M**</td>
      <td>`meta-models/Muse-Glimmer-30B-GGUF`</td>
      <td>Text only. This path is not optimized. SGLang shows a warning at startup.</td>
    </tr>

    <tr>
      <td>**NVFP4**</td>
      <td>`RadixArk/Muse-Glimmer-NVFP4`</td>
      <td>Text only. Ready to serve, no conversion needed.</td>
    </tr>

    <tr>
      <td>**MLX Q4**</td>
      <td>`RadixArk/Muse-Glimmer-q4-MLX`</td>
      <td>Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.</td>
    </tr>

    <tr>
      <td>**MLX Q4\_K\_M (gs128)**</td>
      <td>`RadixArk/Muse-Glimmer-q4km-gs128-MLX`</td>
      <td>Text only. Apple Silicon (MLX backend). Carries the vendor GGUF's exact quantization codes in MLX format. The measured MLX artifact. See §3.4.</td>
    </tr>

    <tr>
      <td>**MLX Q4\_K (dynamic)**</td>
      <td>`RadixArk/Muse-Glimmer-q4k-dynamic-MLX`</td>
      <td>Text only. Apple Silicon (MLX backend). Same serve recipe as gs128, no measured round yet. See §3.4.</td>
    </tr>
  </tbody>
</table>

**Resources:** [Muse-Glimmer-30B (BF16)](https://huggingface.co/meta-models/Muse-Glimmer-30B) · [Muse-Glimmer-30B-assistant (DFlash draft)](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant) · [Muse-Glimmer-30B-GGUF](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF) · [Muse-Glimmer-NVFP4](https://huggingface.co/RadixArk/Muse-Glimmer-NVFP4) · MLX · [q4](https://huggingface.co/RadixArk/Muse-Glimmer-q4-MLX) · [q4km-gs128](https://huggingface.co/RadixArk/Muse-Glimmer-q4km-gs128-MLX) · [q4k-dynamic](https://huggingface.co/RadixArk/Muse-Glimmer-q4k-dynamic-MLX).

## 2. Configuration Tips

**The GGUF format is text only.** SGLang has no `mmproj` path. You cannot use the vision GGUF files. Use the BF16 checkpoint for multimodal input.

**The NVFP4 checkpoint.** `RadixArk/Muse-Glimmer-NVFP4` is a ready-to-serve NVFP4 + MXFP8 checkpoint. No conversion needed — point `--model-path` straight at it.

**The DFlash draft.** `meta-models/Muse-Glimmer-30B-assistant` is the vendor's native draft export and serves directly. No conversion needed.

**DFlash with a GGUF target model** needs `--speculative-draft-load-format auto`. Without this flag, the draft model uses the `gguf` load format from the target model. The loader then rejects the draft directory.

**Apple Silicon uses an MLX checkpoint, not the GGUF files.** The MLX backend has no GGUF path. Serve one of the three `RadixArk/Muse-Glimmer-*-MLX` artifacts with `SGLANG_USE_MLX=1` (see the Apple Silicon cells in the command panel). All three take the same flags; `q4km-gs128` is the one with a measured round. Keep `--disable-radix-cache` — the windowed KV storage for the sliding-window layers requires it — and set `SGLANG_MLX_CACHE_LIMIT_GB=8` so the MLX buffer cache does not grow the footprint under concurrent load. Speculative decoding is not available on the MLX backend.

## 3. Advanced Usage

### 3.1 Reasoning

Muse Glimmer enables the `muse` reasoning parser by default. This parser separates the reasoning text from the final answer.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="meta-models/Muse-Glimmer-30B",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

### 3.2 Tool Calling

Muse Glimmer enables the `muse` tool-call parser by default. This parser sends structured tool calls in `message.tool_calls`.

### 3.3 Multimodal

The BF16 checkpoint supports image input. It defaults to text only. To switch, select **Modality** in the command panel above.

**Text only** adds `--language-model-only`. This flag turns off the vision tower. SGLang does not build or load the vision weights. This frees memory for the KV cache. SGLang rejects image requests in this mode.

Select **Image + text** to turn on image input.

NVFP4, GGUF, and the MLX artifacts are text only. The Modality option does not appear for GGUF or MLX; NVFP4 only offers **Text only**.

### 3.4 Apple Silicon (MLX)

The MLX backend serves three Muse Glimmer artifacts on Apple Silicon Macs (48 GB unified memory or more). All three are text only — the MLX backend has no vision path — and all three take the same flags, so pick one in the command panel:

* `RadixArk/Muse-Glimmer-q4-MLX` — no measured round yet.
* `RadixArk/Muse-Glimmer-q4km-gs128-MLX` — a lossless repack of the vendor's Q4\_K\_M (gs128) GGUF: every weight keeps the GGUF's exact quantization code, with the group scales re-expressed in MLX affine bf16 (≤2⁻⁸ relative rounding). The numbers below are for this artifact.
* `RadixArk/Muse-Glimmer-q4k-dynamic-MLX` — no measured round yet.

Choose along the speed-versus-accuracy axis: footprint and expected accuracy both grow `q4` → `q4km-gs128` → `q4k-dynamic`, and decode speed moves the other way. Decode on Apple Silicon is memory-bandwidth-bound, so a smaller artifact reads fewer weight bytes per token — more tokens per second, and more unified memory left over for the KV cache. Take `q4` for the fastest responses on the smallest machine, `q4k-dynamic` to stay closest to BF16, and `q4km-gs128` for the middle ground — it is also the only one of the three with a measured round, below.

This table shows accuracy for the gs128 checkpoint, with the vendor llama.cpp fork serving the source GGUF on the same machine as the reference. GSM8K: 200 questions, no-thinking chat template, temperature 0, max 2048 new tokens. CIMemories: 1 profile, full combo, single trial, DeepSeek-R1-0528 judge.

<table>
  <thead>
    <tr>
      <th>Benchmark</th>
      <th>SGLang MLX</th>
      <th>llama.cpp (same GGUF)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>GSM8K (200q, no-thinking, greedy)</td>
      <td>0.970</td>
      <td>0.970</td>
    </tr>

    <tr>
      <td>CIMemories — violation rate (lower is better)</td>
      <td>0.00%</td>
      <td>8.27%</td>
    </tr>

    <tr>
      <td>CIMemories — coverage (higher is better)</td>
      <td>76.0%</td>
      <td>68.4%</td>
    </tr>
  </tbody>
</table>

CIMemories is a single-trial benchmark with a nondeterministic judge; treat the SGLang-vs-llama.cpp gap on that row as run noise, not a runtime effect. GSM8K parity is exact.

Decode throughput for gs128 on an M5 Pro (64 GB), 1k-in/1k-out greedy: 15.3 tok/s at batch 1, rising to 52.6 tok/s aggregate at batch 8 — ahead of llama.cpp on the same GGUF codes at every batch size above 1.


# Muse Spark Open
Source: https://docs.sglang.io/cookbook/autoregressive/Meta/MuseSparkOpen

Deploy Muse Spark Open with SGLang — launch recipes, feature compatibility, and tuning notes for GB300, B300, B200, H200, MI355X, Intel Xeon

## Deployment

> **Install SGLang**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-muse-spark-open     # NVIDIA B300 / GB300 / B200 / H200
> docker pull lmsysorg/sglang-rocm:muse-spark-open    # AMD MI355X
> docker pull lmsysorg/sglang:dev-muse-spark-xeon     # Intel Xeon CPU
> ```
>
> The deployment panel can render a complete `docker run` command for the selected hardware and options. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup. The CUDA tag does not run on MI355X; use the ROCm image there; Use Xeon image for Xeon CPU and for more SGLang CPU installation approaches, see the [CPU version installation guide](/docs/hardware-platforms/cpu_server#installation).

Choose your **Hardware Platform**, **Strategy**, and **Quantization** below. The default is **B300 / Low Latency / NVFP4**. **Low Latency** keeps the existing deployment settings; **High Throughput** removes all speculative decoding parameters. Use the Playground to tune parallelism, caching, speculative decoding, multimodal transport, and prefill/decode disaggregation.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

* **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
* **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
* **Container image.** Run the commands inside the image listed for each configuration; flag names follow that image.

### GB300 deployments measured with Artificial Analysis AgentPerf

Five aggregated Dynamo + SGLang deployments on GB300, measured with the [Artificial Analysis AgentPerf](https://github.com/ArtificialAnalysis/agentperf-client) coding-agent workload. Together they span the trade-off from the lowest per-user latency (concurrency 1) to the highest throughput per GPU (concurrency 160), ordered from low latency to high throughput. Each one starts from the Artificial Analysis SGLang submission for this model and changes only flags, environment variables and parallelism.

* **Container image.** `lmsysorg/sglang:dev-muse-spark-open@sha256:8cfc5a72cb296771f5f8a4a4688cb6d05685610a830cbb616fa4ab0eb7bb2b0a` with the Dynamo wheels added: `pip install --no-deps ai-dynamo==1.6.0.dev20261019 ai-dynamo-runtime==1.6.0.dev20261019`.
* **Router.** One Dynamo frontend with the KV-aware router. Session affinity keeps requests that carry the same `X-Dynamo-Session-ID` header on one worker; the AgentPerf client sends one ID per agent session.
* **Checkpoints.** `meta-models/Muse-Spark-Open-816B-A42B-NVFP4` with the DFlash2 draft `RadixArk/Muse-Spark-Open-DFlash2` (8 draft tokens, FP8 KV cache).
* **HiCache.** `--hicache-size` is the host tier in GB per rank; scale it to your host memory.
* **Two-node deployments.** The TP8·EP8 worker spans two GB300 trays (4 GPUs each) over multi-node NVLink: run the worker command on both nodes, with `NODE_RANK` 0 on the first.

> **Agg TP8·EP8 (2 nodes), lowest latency · 8 GPUs · concurrency 1**
>
> **Env Vars**
>
> ```bash theme={null}
> export ETCD_IP=<node running etcd>
> export AGG_HEAD_IP=<first node of the worker>
> export NODE_RANK=<0 on the first node, 1 on the second>
> ```
>
> **etcd**
>
> ```bash theme={null}
> etcd \
>   --data-dir /tmp/etcd \
>   --listen-client-urls http://0.0.0.0:2379 \
>   --advertise-client-urls http://$ETCD_IP:2379
> ```
>
> **Dynamo frontend**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export DYN_ROUTER_QUEUE_THRESHOLD=None
> export DYN_ROUTER_TEMPERATURE=0
> export DYN_TOKENIZER=fastokens
> export DYN_TOKENIZER_CACHE=1
> export DYN_TOKENIZER_CACHE_BYTES=8000000000
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
>
> python3 -m dynamo.frontend \
>   --http-port 8000 \
>   --router-mode kv \
>   --router-session-affinity-ttl-secs 14400 \
>   --active-decode-blocks-threshold None \
>   --active-prefill-tokens-threshold None \
>   --active-prefill-tokens-threshold-frac None
> ```
>
> **SGLang worker (run on each of the 2 nodes)**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
> export NCCL_CUMEM_ENABLE=1
> export NCCL_MNNVL_ENABLE=1
> export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
> export SGLANG_TIMEOUT_KEEP_ALIVE=900
>
> python3 -m dynamo.sglang \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --host 0.0.0.0 \
>   --port 30000 \
>   --dist-init-addr $AGG_HEAD_IP:20000 \
>   --nnodes 2 \
>   --node-rank $NODE_RANK \
>   --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
>   --request-plane tcp \
>   --dyn-reasoning-parser muse \
>   --dyn-tool-call-parser muse \
>   --expert-parallel-size 8 \
>   --kv-cache-dtype fp8_e4m3 \
>   --max-running-requests 32 \
>   --mem-fraction-static 0.85 \
>   --mm-feature-transport cpu \
>   --moe-runner-backend flashinfer_trtllm \
>   --quantization modelopt_fp4 \
>   --reasoning-parser muse \
>   --speculative-algorithm DFLASH \
>   --speculative-draft-attention-backend fa4 \
>   --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
>   --speculative-num-draft-tokens 8 \
>   --swa-full-tokens-ratio 0.1 \
>   --tensor-parallel-size 8 \
>   --trust-remote-code \
>   --watchdog-timeout 3600
> ```

> **Agg TP8·EP8 (2 nodes), low latency · 8 GPUs · concurrency 4**
>
> **Env Vars**
>
> ```bash theme={null}
> export ETCD_IP=<node running etcd>
> export AGG_HEAD_IP=<first node of the worker>
> export NODE_RANK=<0 on the first node, 1 on the second>
> ```
>
> **etcd**
>
> ```bash theme={null}
> etcd \
>   --data-dir /tmp/etcd \
>   --listen-client-urls http://0.0.0.0:2379 \
>   --advertise-client-urls http://$ETCD_IP:2379
> ```
>
> **Dynamo frontend**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export DYN_ROUTER_QUEUE_THRESHOLD=None
> export DYN_ROUTER_TEMPERATURE=0
> export DYN_TOKENIZER=fastokens
> export DYN_TOKENIZER_CACHE=1
> export DYN_TOKENIZER_CACHE_BYTES=8000000000
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
>
> python3 -m dynamo.frontend \
>   --http-port 8000 \
>   --router-mode kv \
>   --router-session-affinity-ttl-secs 14400 \
>   --active-decode-blocks-threshold None \
>   --active-prefill-tokens-threshold None \
>   --active-prefill-tokens-threshold-frac None
> ```
>
> **SGLang worker (run on each of the 2 nodes)**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
> export NCCL_CUMEM_ENABLE=1
> export NCCL_MNNVL_ENABLE=1
> export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
> export SGLANG_TIMEOUT_KEEP_ALIVE=900
>
> python3 -m dynamo.sglang \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --host 0.0.0.0 \
>   --port 30000 \
>   --dist-init-addr $AGG_HEAD_IP:20000 \
>   --nnodes 2 \
>   --node-rank $NODE_RANK \
>   --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
>   --request-plane tcp \
>   --dyn-reasoning-parser muse \
>   --dyn-tool-call-parser muse \
>   --expert-parallel-size 8 \
>   --kv-cache-dtype fp8_e4m3 \
>   --max-running-requests 32 \
>   --mem-fraction-static 0.85 \
>   --mm-feature-transport cpu \
>   --moe-runner-backend flashinfer_trtllm \
>   --quantization modelopt_fp4 \
>   --reasoning-parser muse \
>   --speculative-algorithm DFLASH \
>   --speculative-draft-attention-backend fa4 \
>   --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
>   --speculative-num-draft-tokens 8 \
>   --swa-full-tokens-ratio 0.1 \
>   --tensor-parallel-size 8 \
>   --trust-remote-code \
>   --watchdog-timeout 3600
> ```

> **Agg TP4·EP4, balanced · 4 GPUs · concurrency 16**
>
> **Env Vars**
>
> ```bash theme={null}
> export ETCD_IP=<node running etcd>
> export AGG_HEAD_IP=<first node of the worker>
> ```
>
> **etcd**
>
> ```bash theme={null}
> etcd \
>   --data-dir /tmp/etcd \
>   --listen-client-urls http://0.0.0.0:2379 \
>   --advertise-client-urls http://$ETCD_IP:2379
> ```
>
> **Dynamo frontend**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export DYN_ROUTER_QUEUE_THRESHOLD=None
> export DYN_ROUTER_TEMPERATURE=0
> export DYN_TOKENIZER=fastokens
> export DYN_TOKENIZER_CACHE=1
> export DYN_TOKENIZER_CACHE_BYTES=8000000000
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
>
> python3 -m dynamo.frontend \
>   --http-port 8000 \
>   --router-mode kv \
>   --router-session-affinity-ttl-secs 14400 \
>   --active-decode-blocks-threshold None \
>   --active-prefill-tokens-threshold None \
>   --active-prefill-tokens-threshold-frac None
> ```
>
> **SGLang worker**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
> export NCCL_CUMEM_ENABLE=1
> export SGLANG_TIMEOUT_KEEP_ALIVE=900
>
> python3 -m dynamo.sglang \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --host 0.0.0.0 \
>   --port 30000 \
>   --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
>   --request-plane tcp \
>   --dyn-reasoning-parser muse \
>   --dyn-tool-call-parser muse \
>   --enable-layernorm-sp \
>   --enable-megamoe-for-layernorm-sp \
>   --expert-parallel-size 4 \
>   --kv-cache-dtype fp8_e4m3 \
>   --max-running-requests 32 \
>   --mem-fraction-static 0.85 \
>   --mm-feature-transport cpu \
>   --moe-runner-backend flashinfer_trtllm \
>   --prefill-decode-interval 64 \
>   --quantization modelopt_fp4 \
>   --reasoning-parser muse \
>   --speculative-algorithm DFLASH \
>   --speculative-draft-attention-backend fa4 \
>   --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
>   --speculative-num-draft-tokens 8 \
>   --swa-full-tokens-ratio 0.1 \
>   --tensor-parallel-size 4 \
>   --trust-remote-code \
>   --watchdog-timeout 3600
> ```

> **Agg TP4·EP4, high throughput · 4 GPUs · concurrency 64**
>
> **Env Vars**
>
> ```bash theme={null}
> export ETCD_IP=<node running etcd>
> export AGG_HEAD_IP=<first node of the worker>
> ```
>
> **etcd**
>
> ```bash theme={null}
> etcd \
>   --data-dir /tmp/etcd \
>   --listen-client-urls http://0.0.0.0:2379 \
>   --advertise-client-urls http://$ETCD_IP:2379
> ```
>
> **Dynamo frontend**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export DYN_ROUTER_QUEUE_THRESHOLD=None
> export DYN_ROUTER_TEMPERATURE=0
> export DYN_TOKENIZER=fastokens
> export DYN_TOKENIZER_CACHE=1
> export DYN_TOKENIZER_CACHE_BYTES=8000000000
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
>
> python3 -m dynamo.frontend \
>   --http-port 8000 \
>   --router-mode kv \
>   --router-session-affinity-ttl-secs 14400 \
>   --active-decode-blocks-threshold None \
>   --active-prefill-tokens-threshold None \
>   --active-prefill-tokens-threshold-frac None
> ```
>
> **SGLang worker**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
> export NCCL_CUMEM_ENABLE=1
> export SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1
> export SGLANG_TIMEOUT_KEEP_ALIVE=900
>
> python3 -m dynamo.sglang \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --host 0.0.0.0 \
>   --port 30000 \
>   --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
>   --request-plane tcp \
>   --cuda-graph-max-bs-decode 64 \
>   --dyn-reasoning-parser muse \
>   --dyn-tool-call-parser muse \
>   --enable-layernorm-sp \
>   --enable-megamoe-for-layernorm-sp \
>   --expert-parallel-size 4 \
>   --kv-cache-dtype fp8_e4m3 \
>   --max-running-requests 64 \
>   --mem-fraction-static 0.85 \
>   --mm-feature-transport cpu \
>   --moe-runner-backend flashinfer_trtllm \
>   --prefill-decode-interval 24 \
>   --quantization modelopt_fp4 \
>   --reasoning-parser muse \
>   --speculative-algorithm DFLASH \
>   --speculative-draft-attention-backend fa4 \
>   --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
>   --speculative-num-draft-tokens 8 \
>   --swa-full-tokens-ratio 0.15 \
>   --tensor-parallel-size 4 \
>   --trust-remote-code \
>   --watchdog-timeout 3600
> ```

> **Agg TP4·EP4 + HiCache, max throughput · 4 GPUs · concurrency 160**
>
> **Env Vars**
>
> ```bash theme={null}
> export ETCD_IP=<node running etcd>
> export AGG_HEAD_IP=<first node of the worker>
> ```
>
> **etcd**
>
> ```bash theme={null}
> etcd \
>   --data-dir /tmp/etcd \
>   --listen-client-urls http://0.0.0.0:2379 \
>   --advertise-client-urls http://$ETCD_IP:2379
> ```
>
> **Dynamo frontend**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export DYN_ROUTER_QUEUE_THRESHOLD=None
> export DYN_ROUTER_TEMPERATURE=0
> export DYN_TOKENIZER=fastokens
> export DYN_TOKENIZER_CACHE=1
> export DYN_TOKENIZER_CACHE_BYTES=8000000000
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
>
> python3 -m dynamo.frontend \
>   --http-port 8000 \
>   --router-mode kv \
>   --router-session-affinity-ttl-secs 14400 \
>   --active-decode-blocks-threshold None \
>   --active-prefill-tokens-threshold None \
>   --active-prefill-tokens-threshold-frac None
> ```
>
> **SGLang worker**
>
> ```bash theme={null}
> export DYN_REQUEST_PLANE=tcp
> export ETCD_ENDPOINTS=http://$ETCD_IP:2379
> export NCCL_CUMEM_ENABLE=1
> export SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1
> export SGLANG_TIMEOUT_KEEP_ALIVE=900
>
> python3 -m dynamo.sglang \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --served-model-name meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --host 0.0.0.0 \
>   --port 30000 \
>   --kv-events-config '{"publisher":"zmq","topic":"kv-events","endpoint":"tcp://*:5557"}' \
>   --request-plane tcp \
>   --cuda-graph-max-bs-decode 176 \
>   --dyn-reasoning-parser muse \
>   --dyn-tool-call-parser muse \
>   --enable-hierarchical-cache \
>   --enable-layernorm-sp \
>   --enable-megamoe-for-layernorm-sp \
>   --expert-parallel-size 4 \
>   --hicache-size 120 \
>   --kv-cache-dtype fp8_e4m3 \
>   --max-running-requests 176 \
>   --mem-fraction-static 0.9 \
>   --mm-feature-transport cpu \
>   --moe-runner-backend flashinfer_trtllm \
>   --prefill-decode-interval 6 \
>   --quantization modelopt_fp4 \
>   --reasoning-parser muse \
>   --speculative-algorithm DFLASH \
>   --speculative-draft-attention-backend fa4 \
>   --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
>   --speculative-num-draft-tokens 8 \
>   --swa-full-tokens-ratio 0.15 \
>   --tensor-parallel-size 4 \
>   --trust-remote-code \
>   --watchdog-timeout 3600
> ```

### MI355X deployment measured with SemiAnalysis InferenceX

TP8·EP8 with DFlash V2 on 8× MI355X, measured in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) at concurrency 1–64.

* **Container image.** `lmsysorg/sglang-rocm:muse-spark-open-1020`.
* **HiCache.** At concurrency 16 and above, add the HiCache flags below. `--hicache-size` is the host tier in GB per rank, so 253 GB needs about 2 TB of host memory across the eight ranks.

```bash Command theme={null}
SGLANG_USE_AITER=1 \
SGLANG_USE_AITER_UNIFIED_ATTN=1 \
MUSE_MOE_BACKEND=flydsl \
python -m sglang.launch_server \
  --model-path amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 \
  --tp-size 8 \
  --ep-size 8 \
  --attention-backend aiter \
  --page-size 16 \
  --kv-cache-dtype fp8_e4m3 \
  --mem-fraction-static 0.85 \
  --max-running-requests 128 \
  --swa-full-tokens-ratio 0.2 \
  --prefill-decode-interval 20 \
  --scheduler-recv-interval 20 \
  --tokenizer-worker-num 4 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
  --speculative-draft-attention-backend triton \
  --speculative-num-draft-tokens 8 \
  --speculative-draft-window-size 2048 \
  --reasoning-parser muse \
  --tool-call-parser muse \
  --watchdog-timeout 1800 \
  --host 0.0.0.0 \
  --port 30000
```

```bash HiCache (concurrency 16 and above) theme={null}
  --enable-hierarchical-cache \
  --hicache-size 253 \
  --hicache-write-policy write_through \
  --hicache-io-backend kernel \
  --hicache-mem-layout page_first
```

## 1. Model introduction

This cookbook provides deployment recipes for the Muse Spark Open 816B-A42B checkpoints:

* **NVFP4:** `meta-models/Muse-Spark-Open-816B-A42B-NVFP4` (NVIDIA default).
* **MXFP4:** `amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4` (MI355X default; Quark-quantized).
* **NVFP4 on MI355X (direct load):** the same `meta-models/Muse-Spark-Open-816B-A42B-NVFP4` checkpoint runs on MI355X with `--quantization quark_mxfp4`; the weights are requantized to MXFP4 while loading, so no separate export is needed. Select **MI355X / NVFP4** in the panel for this command.
* **INT4** `intel-ai/Muse-Spark-Open-816B-A42B-INT4` (Xeon default; AutoRound-quantized).
* **BF16:** `meta-models/Muse-Spark-Open-816B-A42B`.

## 2. Configuration tips

### H200 configuration

NVFP4 runs on one 8x H200 node with Marlin W4A16 experts (`--moe-runner-backend marlin`), since Hopper has no TRT-LLM FP4 MoE. The BF16 checkpoint does not fit on one node, so it runs TP16/EP16 across two nodes; replace `<node-rank>` and `<node0-ip>` for each node. Prefill sequence parallel and MegaMoE need SM100 kernels and are unavailable on H200.

### Xeon CPU service configuration

* Xeon serves the **INT4** checkpoint (quantized by Intel AutoRound, HF id: `intel-ai/Muse-Spark-Open-816B-A42B-INT4`) on the SGLang CPU engine. A typical Xeon 6 P-core platform (e.g., Xeon 6980P) is recommended.
* Set `--page-size 1` for the `intel_amx` attention backend (the CPU default attention backend).
* We recommend setting the TP size equal to the NUMA count. See the **Notes** under [Launch of the Serving Engine](/docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) in the SGLang CPU server document for how to configure the arguments, especially TP and NUMA binding.

## 3. Advanced Usage

### 3.1 Reasoning

On the server, enable `--reasoning-parser muse` in your `sglang serve` command. The [Deployment](#deployment) recipes above already include this flag; in the [Playground](#playground), keep **Reasoning Parser** enabled under **Parsers**. The parser separates the model's reasoning into `message.reasoning_content` and its final answer into `message.content`.

On the client, set `reasoning_effort` for each chat completion request. Muse Spark Open supports **four levels: `low`, `medium`, `high`, and `xhigh`**. If you omit the parameter, the checkpoint's chat template defaults to **`xhigh`**. Use `low` or `medium` for faster, shorter responses, and `high` or `xhigh` for complex problem solving, coding, and agentic tasks. Other values, including `minimal`, are not supported. See the [model card](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B-NVFP4) for the supported settings.

The example below connects to the server on port `30000` and discovers its model ID, so it works with whichever checkpoint you selected in Deployment. Change `base_url` if the server runs on another host or port. The current checkpoint's chat template does not require `chat_template_kwargs={"thinking": True}`; set the reasoning level with `reasoning_effort`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(
>     base_url="http://127.0.0.1:30000/v1",
>     api_key="EMPTY",
>     timeout=600.0,
> )
> model = client.models.list().data[0].id
> response = client.chat.completions.create(
>     model=model,
>     messages=[
>         {"role": "system", "content": "You are a helpful assistant."},
>         {"role": "user", "content": "Briefly explain: what is SGLang?"},
>     ],
>     max_tokens=16384,
>     reasoning_effort="xhigh",  # low | medium | high | xhigh (default)
> )
> choice = response.choices[0]
> print("Reasoning:", getattr(choice.message, "reasoning_content", None))
> print("Content:", choice.message.content)
> print("Finish reason:", choice.finish_reason)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: We need to briefly explain what is SGLang. Likely want concise explanation. SGLang is structured generation language? It's a framework for programming LLMs. Developed by LMSYS? Actually SGLang is a serving framework.
>
> Need to explain: SGLang is a fast serving framework/DSL for LLMs? Let's recall.
>
> SGLang is a structured generation language, a domain-specific language for LLM programs. Created by LMSYS Org. Allows efficient LLM programming with structured prompting, caching, constrained decoding etc. Provides high-performance serving via RadixAttention, paged attention, etc. Used for chaining calls, multi-turn, function calling, reasoning.
>
> We should give brief explanation.
>
> Possibly user wants simple. Provide definition, key features, use cases.
>
> Make it brief but informative.
>
> Maybe need to distinguish from other? Should be neutral.
>
> Potentially also SGLang = Serving framework?
>
> SGLang is open-source system for efficient LLM serving and programming.
>
> Key features:
> - Front-end language (Python DSL) for controlling generation
> - Backend runtime: RadixAttention for KV cache reuse, continuous batching, tensor parallelism, quantization, speculative decoding.
> - Supports OpenAI-compatible API.
>
> Briefly explain.
>
> Keep concise.
>
> Add example maybe?
>
> User: "Briefly explain: what is SGLang?"
>
> Provide maybe 3-4 bullet points.
>
> Possibly mention origin: developed by LMSYS and UC Berkeley etc.
>
> Ensure not too long. Brief.
>
> Maybe add: SGL for Structured Generation Language.
>
> Make answer accessible.
>
> Probably mention advantages: faster, controllable.
>
> We should not hallucinate too much. Provide accurate info.
>
> SGLang GitHub: sgl-project/sglang.
>
> SGLang latest: SGLang is inference engine?
>
> Let's verify facts: SGLang was introduced in paper "SGLang: Efficient Execution of Structured Language Model Programs" arXiv 2312.07104, Dec 2023, authors Lianmin Zheng etc.
>
> It provides primitives: gen(), select(), etc.
>
> Runtime optimizations: RadixAttention, compressed FSM for constrained decoding, API speculative execution.
>
> Provides high throughput, low latency.
>
> Supports many models: Llama, Mistral, Qwen etc. Supports on NVIDIA GPUs, AMD.
>
> Should answer.
>
> User likely wants short. Provide 1 paragraph + bullets.
>
> Make tone helpful.
>
> Add use cases: agents, reasoning, data extraction.
>
> Probably not over.
>
> Potential need to add date knowledge cutoff 2026?
>
> Add nothing disallowed.
>
> Make final answer succinct.
>
> Maybe add link? Not needed.
>
> Use simple language.
>
> We should not mention reasoning strength.
>
> Proceed to final.
>
> Content: SGLang — **Structured Generation Language** — is an open-source system for programming and serving Large Language Models (LLMs).
>
> Created by LMSYS / UC Berkeley, it has 2 parts:
>
> **1. Frontend - a Python DSL:** Lets you write LLM programs with logic, not just string prompts. Primitives like `gen()`, `select()`, `fork()` for chaining calls, controlling reasoning, forcing JSON/tool-calling structure, multi-agent workflows, etc.
>
> **2. Backend - a high-performance inference engine:** Optimized runtime for fast, cheap serving with features like:
> * **RadixAttention:** automatic KV-cache reuse for prefixes, few-shot examples, multi-turn chat
> * Continuous batching, Tensor Parallelism, quantization, speculative decoding
> * Fast constrained/structured decoding
>
> In short: It's an alternative to vLLM / TensorRT-LLM. You get both a cleaner way to *program* LLMs and a very fast engine to *run* them, with an OpenAI-compatible API.
>
> Finish reason: stop
> ```

### 3.2 Tool calling

On the server, enable `--tool-call-parser muse` alongside `--reasoning-parser muse` in your `sglang serve` command. The [Deployment](#deployment) recipes above already include both flags; in the [Playground](#playground), keep **Tool Call Parser** and **Reasoning Parser** enabled under **Parsers**. SGLang converts Muse Spark Open's native tool-call format into OpenAI-compatible `message.tool_calls`, while keeping reasoning in `message.reasoning_content`.

On the client, pass function definitions in `tools`, with each function's arguments described by a JSON Schema. The checkpoint's [chat template](https://huggingface.co/meta-models/Muse-Spark-Open-816B-A42B-NVFP4/blob/main/chat_template.jinja) formats these definitions for the model. With `tool_choice="auto"`, the model decides whether to call a tool. The example below requests a weather lookup and prints each returned function name and its JSON arguments.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> import json
>
> from openai import OpenAI
>
> client = OpenAI(
>     base_url="http://127.0.0.1:30000/v1",
>     api_key="EMPTY",
>     timeout=600.0,
> )
> model = client.models.list().data[0].id
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city.",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {
>                         "type": "string",
>                         "description": "City name, for example Beijing.",
>                     },
>                     "unit": {
>                         "type": "string",
>                         "enum": ["celsius", "fahrenheit"],
>                     },
>                 },
>                 "required": ["location", "unit"],
>                 "additionalProperties": False,
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model=model,
>     messages=[
>         {"role": "user", "content": "Use get_weather to check the current weather in Beijing in Celsius."},
>     ],
>     tools=tools,
>     tool_choice="auto",
>     reasoning_effort="xhigh",
>     max_tokens=16384,
> )
> choice = response.choices[0]
> message = choice.message
> print("Reasoning:", getattr(message, "reasoning_content", None))
> print("Content:", message.content)
> for tool_call in message.tool_calls or []:
>     print("Tool call ID:", tool_call.id)
>     print("Function:", tool_call.function.name)
>     print("Arguments:", json.loads(tool_call.function.arguments))
> print("Finish reason:", choice.finish_reason)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: User wants get_weather to check current weather in Beijing in Celsius.
>
> We have function get_weather with parameters: location (string), unit (string enum celsius/fahrenheit)
>
> We need to call it.
>
> We must call the function with location=Beijing, unit=celsius
>
> Add reasoning.
>
> We should not add extra text beyond function call? In this format, we need to provide function call. Probably we need to output function call directly.
>
> We are Meta AI. Follow guidelines.
>
> We need to invoke get_weather.
>
> Be careful to JSON correct.
>
> Probably need thinking tags? But in this environment, we have tool access. We should call get_weather.
>
> We should do that now.
>
> Possibly reasoning strength 512, need thorough reasoning? No, we just need to call tool.
>
> Ensure we use correct recipient format: get_weather
>
> We need to produce "<atem:function_calls>..." block.
>
> We need to provide location: "Beijing", unit: "celsius"
>
> Make call.
>
> Content: Beijing in Celsius — checking the weather for you immediately.
> Tool call ID: call_43c0efae809d46988855bb79
> Function: get_weather
> Arguments: {'location': 'Beijing', 'unit': 'celsius'}
> Finish reason: tool_calls
> ```

This example inspects the requested tool calls; it does not fetch weather data. Your application executes the corresponding function. To continue the conversation, append the assistant message containing `tool_calls`, then a `role="tool"` message for each result with the matching `tool_call_id`, and send the updated messages with the same tool definitions.

### 3.3 Encode Prefill Decode Disaggregation

Encoder–Prefill–Decode (EPD) disaggregation separates media encoding, prompt processing, and token generation, so encoding-heavy image/video requests can be scheduled separately from language inference. The encoder sends features through `zmq_to_tokenizer`; prefill transfers KV state to decode through Mooncake, with a router exposing one client endpoint.

The examples below use one 8× B300 NVLink host: encoder TP2 shares GPUs 0–1 with prefill TP4/EP4 on GPUs 0–3; decode TP4/EP4 uses GPUs 4–7. Run each command in a separate shell using the NVIDIA image from [Deployment](#deployment), with a shared host network. Start encoder, prefill, decode, then router, waiting for each to become ready. Prefill (`30001`), decode (`30002`), and router (`8000`) match the Playground ports; the encoder uses `30010`.

> [!NOTE]
>
> These are simplified examples derived from [PR #265](https://github.com/radixark/sglang-muse-spark/pull/265). The source run checked transport with simulated outputs; this ordinary-serving configuration still needs runtime and accuracy validation. Leave GPU memory headroom for the encoder sharing the prefill GPUs.

> **Encoder server (B300, GPUs 0–1)**
>
> ```bash Command theme={null}
> CUDA_VISIBLE_DEVICES=0,1 sglang serve \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --quantization modelopt_fp4 \
>   --tp-size 2 \
>   --encoder-only \
>   --encoder-transfer-backend zmq_to_tokenizer \
>   --host 0.0.0.0 \
>   --port 30010
> ```

> **Prefill server (B300, GPUs 0–3)**
>
> ```bash Command theme={null}
> export MC_INTRANODE_NVLINK=true
> export MC_TCP_ENABLE_CONNECTION_POOL=true
> export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK
>
> CUDA_VISIBLE_DEVICES=0,1,2,3 sglang serve \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --quantization modelopt_fp4 \
>   --moe-runner-backend flashinfer_trtllm \
>   --tp-size 4 --ep-size 4 \
>   --mem-fraction-static 0.8 \
>   --language-only \
>   --encoder-urls http://127.0.0.1:30010 \
>   --encoder-transfer-backend zmq_to_tokenizer \
>   --encoder-bootstrap-port 8997 \
>   --disaggregation-mode prefill \
>   --disaggregation-transfer-backend mooncake \
>   --disaggregation-bootstrap-port 8998 \
>   --reasoning-parser muse --tool-call-parser muse \
>   --host 0.0.0.0 \
>   --port 30001
> ```

> **Decode server (B300, GPUs 4–7)**
>
> ```bash Command theme={null}
> export MC_INTRANODE_NVLINK=true
> export MC_TCP_ENABLE_CONNECTION_POOL=true
> export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK
>
> CUDA_VISIBLE_DEVICES=4,5,6,7 sglang serve \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --quantization modelopt_fp4 \
>   --moe-runner-backend flashinfer_trtllm \
>   --tp-size 4 --ep-size 4 \
>   --mem-fraction-static 0.75 \
>   --language-only \
>   --encoder-urls http://127.0.0.1:30010 \
>   --encoder-transfer-backend zmq_to_tokenizer \
>   --encoder-bootstrap-port 8999 \
>   --disaggregation-mode decode \
>   --disaggregation-transfer-backend mooncake \
>   --disaggregation-bootstrap-port 8998 \
>   --reasoning-parser muse --tool-call-parser muse \
>   --host 0.0.0.0 \
>   --port 30002
> ```

> **Router**
>
> ```bash Command theme={null}
> sglang-router launch \
>   --pd-disaggregation \
>   --prefill http://127.0.0.1:30001 8998 \
>   --decode http://127.0.0.1:30002 \
>   --host 0.0.0.0 --port 8000
> ```

The language workers use distinct encoder-bootstrap ports (`8997` / `8999`) on this shared host; the router's `8998` matches the prefill KV bootstrap port. Send client requests to `http://localhost:8000/v1`. See the [EPD guide](/docs/advanced_features/epd_disaggregation) for the architecture and additional deployment options.

### 3.4 Sequence Parallel for Prefill

Sequence parallelism (SP) splits activations along the token dimension across TP ranks. Each rank processes its own token shard for normalization, pooling, and unpooling, reducing duplicated computation and activation storage during long-prompt prefill. NVFP4 MegaMoE connects these shards to expert parallelism through integrated all-to-all dispatch and combine: tokens reach their selected experts, and weighted expert outputs return to the original token ranks.

For a Blackwell NVFP4 configuration (B300, B200, or GB300), use the [Playground](#playground) to enable **Prefill Sequence Parallel** and **Prefill MegaMoE** together. This adds `--enable-layernorm-sp --enable-megamoe-for-layernorm-sp` to the launch command. Keep **Attention TP** and **MoE EP** equal; MegaMoE requires SP to remain on. These prefill options are unavailable on H200 and are omitted for a PD decode worker.

### 3.5 Speculative Decoding with DFlash v2 and Block Verification

DFlash V2 uses the target model's hidden states to propose a block of future tokens in one draft-model forward pass; Muse Spark Open then verifies the block in a single pass, advancing by multiple tokens when proposals are accepted. V2 mixes information across neighboring draft positions and selects candidates with their dependencies in mind. Optional [block verification](https://arxiv.org/abs/2403.10444) evaluates draft prefixes jointly instead of token by token, potentially accepting longer prefixes while preserving the target sampling distribution.

In the [Playground](#playground), select **DFlash V2** under **Speculative Decoding** to add `--speculative-algorithm DFLASH` and `--speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2`. Then adjust:

* **Draft attention backend:** `fa4` for NVIDIA or `triton` for MI355X, inherited from Deployment and set with `--speculative-draft-attention-backend`.
* **Num draft tokens:** 1–8, default **8**, set with `--speculative-num-draft-tokens`.
* **Block Verification:** turn it on to add `--speculative-use-block-verification`; it is off by default.

The Deployment **Low Latency** strategy enables DFlash V2 on supported GPUs; **High Throughput** leaves speculative decoding off. The Playground inherits that choice and lets you override it.

### 3.6 MegaKernel with Helion (Experimental)

We support persistent Helion megakernels for DFlash V2 target verification. They fuse attention, projection and normalization boundaries, routing, MoE, and the layer tail, with a separate fused LM head, reducing kernel launches and communication overhead. TP4/EP1 shards each expert across four GPUs, avoiding the expert-load imbalance of EP4 at batch size 1. With **BS=1 + DFlash V2**, the reported decode performance is roughly **10% faster** than the TP4/EP4 baseline without megakernels on 4×GB300; gains depend on the workload.

Use this experimental recipe **only on a PD-disaggregated decode worker**, with four **GB300 (SM 10.3)** GPUs, NVFP4 weights, FP8 E4M3 KV cache, and `helion==1.5.0`. Keep prefill on the regular path: EP1 prefill is slower than the TP4/EP4 baseline. This specialized TP4/EP1 command is separate from the Playground's equal-TP/EP configuration.

> **Helion decode worker (4×GB300)**
>
> Install the dependency in the serving environment, then launch the decode worker:
>
> ```bash Command theme={null}
> python -m pip install helion==1.5.0
>
> export NCCL_NVLS_ENABLE=1
> export NCCL_MNNVL_ENABLE=1
> export NCCL_CUMEM_ENABLE=1
> export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK
> export MC_FORCE_MNNVL=1
>
> SGLANG_FORCE_FUSED_OP_BACKEND=helion \
> SGLANG_ENABLE_MUSE_SPARK_FINALIZE_AR=1 \
> SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=1 \
> python -m sglang.launch_server \
>   --model-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --tokenizer-path meta-models/Muse-Spark-Open-816B-A42B-NVFP4 \
>   --served-model-name external \
>   --json-model-override-args '{"eos_token_id":[200001,200008]}' \
>   --tp-size 4 \
>   --ep-size 1 \
>   --context-length 131072 \
>   --max-running-requests 1 \
>   --quantization modelopt_fp4 \
>   --moe-runner-backend flashinfer_trtllm \
>   --attention-backend trtllm_mha \
>   --kv-cache-dtype fp8_e4m3 \
>   --page-size 128 \
>   --disable-flashinfer-autotune \
>   --disable-prefill-cuda-graph \
>   --cuda-graph-max-bs-decode 1 \
>   --reasoning-parser muse \
>   --tool-call-parser muse \
>   --speculative-algorithm DFLASH \
>   --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2 \
>   --speculative-draft-attention-backend fa4 \
>   --speculative-num-draft-tokens 8 \
>   --disaggregation-mode decode \
>   --disaggregation-transfer-backend mooncake \
>   --disaggregation-bootstrap-port 8998 \
>   --mem-fraction-static 0.80 \
>   --host 0.0.0.0 \
>   --port 30002
> ```

Pair this worker with a compatible prefill worker using the same target checkpoint, TP4, `--kv-cache-dtype fp8_e4m3`, `--page-size 128`, and `--served-model-name external`, with bootstrap port `8998`. Use the [Playground](#playground) PD router command with the decode URL pointing to this worker on port `30002`; send client requests to the router on port `8000`.


# MiniMax-M2
Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2



## 1. Model Introduction

[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is a compact, fast, and cost-effective MoE model (230 billion total parameters with 10 billion active parameters) built for elite performance in coding and agentic tasks, all while maintaining powerful general intelligence.

This generation delivers comprehensive upgrades across the board:

* **Superior Intelligence**: MiniMax-M2 demonstrates highly competitive general intelligence across mathematics, science, instruction following, coding, and agentic tool use in [Artificial Analysis](https://artificialanalysis.ai/). Its composite score ranks #1 among open-source models globally.

* **Advanced Coding**: Engineered for end-to-end developer workflows, MiniMax-M2 excels at multi-file edits, coding-run-fix loops, and test-validated repairs. Strong performance on Terminal-Bench and (Multi-)SWE-Bench–style tasks demonstrates practical effectiveness in terminals, IDEs, and CI across languages.

* **Agent Performance**: MiniMax-M2 plans and executes complex, long-horizon toolchains across shell, browser, retrieval, and code runners. In BrowseComp-style evaluations, it consistently locates hard-to-surface sources, maintains evidence traceable, and gracefully recovers from flaky steps.

* **Efficient Design**: With 10 billion activated parameters (230 billion in total), MiniMax-M2 delivers lower latency, lower cost, and higher throughput for interactive agents and batched sampling—perfectly aligned with the shift toward highly deployable models that still shine on coding and agentic tasks.

For more details, please refer to the [official Minimax GitHub Repository](https://github.com/MiniMax-AI).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. The AMD environment is currently available in SGLang via Docker image install.

### 2.1 AMD Docker

#### 2.1.1 Launch docker

```shell Command theme={null}
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```

```shell Command theme={null}
docker run -d -it --ipc=host --network=host --privileged \
  --cap-add=CAP_SYS_ADMIN \
  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
  --group-add video --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /:/work \
  -e SHELL=/bin/bash \
  --name Minimax \
 lmsysorg/sglang:v0.5.9-rocm720-mi30x \
  /bin/bash
```

#### 2.1.2 Make modifications inside the docker

```shell Command theme={null}
mv /sgl-workspace/sglang/python/sglang/srt/models/transformers.py \
   /sgl-workspace/sglang/python/sglang/srt/models/hf_transformers_model.py
```

#### 2.1.3 Fix torch compile

Comment out the following line: @torch.compile(dynamic=True, backend=get\_compiler\_backend()) in /sgl-workspace/sglang/python/sglang/srt/models/minimax\_m2.py

```shell Command theme={null}
#@torch.compile(dynamic=True, backend=get_compiler_backend())
```

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

#### 3.1.1 NVIDIA GPU Deployment

The interactive command generator above covers AMD deployments. For NVIDIA GPUs (H100/H200/B200), use these explicit commands:

**4-GPU deployment (up to 400K context):**

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path MiniMaxAI/MiniMax-M2 \
    --tp-size 4 \
    --tool-call-parser minimax-m2 \
    --reasoning-parser minimax-append-think \
    --host 0.0.0.0 \
    --trust-remote-code \
    --port 30000 \
    --mem-fraction-static 0.85
```

**8-GPU deployment (up to 3M context):**

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path MiniMaxAI/MiniMax-M2 \
    --tp-size 8 \
    --ep-size 8 \
    --tool-call-parser minimax-m2 \
    --reasoning-parser minimax-append-think \
    --host 0.0.0.0 \
    --trust-remote-code \
    --port 30000 \
    --mem-fraction-static 0.85
```

### 3.2 System Requirements

Recommended configurations — actual requirements depend on workload:

<table>
  <thead>
    <tr>
      <th>GPUs</th>
      <th>Context Length Support</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>4× 96 GB GPUs</td>
      <td>Up to 400K tokens</td>
    </tr>

    <tr>
      <td>8× 144 GB GPUs</td>
      <td>Up to 3M tokens</td>
    </tr>
  </tbody>
</table>

### 3.3 Testing Deployment

After the server starts, verify with:

```shell Command theme={null}
curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniMaxAI/MiniMax-M2",
        "messages": [
            {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
            {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
        ]
    }'
```

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

Server Command:

```shell Command theme={null}
    sglang serve \
    --model-path MiniMaxAI/MiniMax-M2 \
    --tp-size 4 \
    --reasoning-parser minimax-append-think \
    --trust-remote-code \
    --mem-fraction-static 0.85
```

Test Code:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.6,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

Output Example:

```text Output theme={null}
<think>First, the user asks: "What is 15% of 240?" This is a straightforward percentage calculation. I need to solve it step by step as per the instruction.

The problem is: What is 15% of 240?

To find a percentage of a number, I multiply the number by the percentage divided by 100. So, 15% is 15/100, which simplifies to 0.15.

Therefore, 15% of 240 is 240 times 0.15.

Let me calculate that: 240 × 0.15.

I can break it down: 240 × 0.15 = 240 × (15/100) = (240 × 15) / 100.

Now, 240 × 15. 200 × 15 = 3000, and 40 × 15 = 600, so total 3000 + 600 = 3600.

Then, divide by 100: 3600 / 100 = 36.

So, 15% of 240 is 36.

I should confirm this with another method. For example, 10% of 240 is 24, and 5% is half of that, which is 12. Then 15% is 10% + 5% = 24 + 12 = 36. Same answer.

Or, using fractions: 15% = 3/20, so 240 × 3/20 = (240 / 20) × 3 = 12 × 3 = 36.

All methods confirm it's 36.

The user said "solve this problem step by step," so I should present the steps clearly.

Step 1: Understand that "15% of 240" means 15 per hundred of 240.

Step 2: Convert percentage to decimal: 15% = 15/100 = 0.15.

Step 3: Multiply the number by the decimal: 240 × 0.15.

Step 4: Calculate the multiplication: 240 × 0.15 = 36.

Step 5: Therefore, 15% of 240 is 36.

I should also mention that percentage means per hundred, so it's straightforward.

Finally, I need to box the answer as per the instruction.

So, the final answer is 36.
</think>

To find 15% of 240, follow these steps:

1. **Understand the percentage**: "15%" means 15 per hundred, or 15/100.
2. **Convert to a decimal**: 15/100 = 0.15.
3. **Multiply by the number**: 240 × 0.15.
4. **Calculate the result**:
   - 240 × 0.15 = 36.

Alternatively, you can break it down:
- 10% of 240 is 24 (since 240 ÷ 10 = 24).
- 5% of 240 is half of 10%, which is 12.
- Therefore, 15% is 10% + 5% = 24 + 12 = 36.

Both methods confirm the result.

**Answer**: 36
```

### 4.2.2 Tool Calling

Server Command:

```shell Command theme={null}
sglang serve \
    --model-path MiniMaxAI/MiniMax-M2 \
    --tp-size 4 \
    --tool-call-parser minimax-m2 \
    --trust-remote-code \
    --mem-fraction-static 0.85
```

Test Code:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

Output Example:

```text Output theme={null}
Alright, the user is asking about the weather in Beijing. This is a straightforward request that I can help with using the get_weather tool that's available to me.

Let me think about what I need to do here. The user wants to know the current weather conditions in Beijing, which is the capital city of China. To provide this information, I need to use the get_weather tool that's been provided to me.

Looking at the tool's parameters, I can see it requires:
1. location - which is required and should be a string representing the city name
2. unit - which is optional and can be either "celsius" or "fahrenheit"

For the location parameter, I'll use "Beijing" since that's what the user asked about.

For the unit parameter, the user didn't specify their preference between celsius and fahrenheit. Since Beijing is in China, which primarily uses celsius, and celsius is the more standard unit internationally, I'll default to celsius. If the user wants the temperature in fahrenheit instead, they can ask in a follow-up message and I can provide that information.

So I need to make a tool call to get_weather with the following parameters:
- location: "Beijing"
- unit: "celsius"

This should return the current weather information for Beijing, which I can then share with the user. I'll format my response using the required XML tags for tool calls as specified in my instructions.
</think>

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment**:

* Hardware: AMD MI300X GPU(4x)

* Model: MiniMax-M2

* Tensor Parallelism: 4

* sglang version: 0.5.7

**Model Deployment**:

```bash Command theme={null}
sglang serve \
    --model-path MiniMaxAI/MiniMax-M2 \
    --tp-size 4 \
    --trust-remote-code \
    --mem-fraction-static 0.85
```

### 5.1.1 Low Concurrency (Latency-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf

```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  138.91
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.07
Input token throughput (tok/s):          43.92
Output token throughput (tok/s):         30.38
Peak output token throughput (tok/s):    46.00
Peak concurrent requests:                2
Total token throughput (tok/s):          74.30
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   13887.62
Median E2E Latency (ms):                 10377.26
---------------Time to First Token----------------
Mean TTFT (ms):                          4528.94
Median TTFT (ms):                        385.23
P99 TTFT (ms):                           38338.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          22.21
Median TPOT (ms):                        22.24
P99 TPOT (ms):                           22.25
---------------Inter-Token Latency----------------
Mean ITL (ms):                           22.23
Median ITL (ms):                         22.24
P95 ITL (ms):                            22.35
P99 ITL (ms):                            22.41
Max ITL (ms):                            23.64
==================================================
```

### 5.1.2 Medium Concurrency (Balanced)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf

```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  81.07
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40803
Request throughput (req/s):              0.99
Input token throughput (tok/s):          489.29
Output token throughput (tok/s):         503.32
Peak output token throughput (tok/s):    704.00
Peak concurrent requests:                19
Total token throughput (tok/s):          992.61
Concurrency:                             13.74
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   13925.95
Median E2E Latency (ms):                 14348.75
---------------Time to First Token----------------
Mean TTFT (ms):                          532.32
Median TTFT (ms):                        147.69
P99 TTFT (ms):                           1978.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          27.49
Median TPOT (ms):                        26.56
P99 TPOT (ms):                           46.52
---------------Inter-Token Latency----------------
Mean ITL (ms):                           26.31
Median ITL (ms):                         23.47
P95 ITL (ms):                            24.37
P99 ITL (ms):                            125.10
Max ITL (ms):                            1192.51
==================================================
```

### 5.1.3 High Concurrency (Throughput-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  153.71
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    250982
Request throughput (req/s):              3.25
Input token throughput (tok/s):          1625.33
Output token throughput (tok/s):         1643.75
Peak output token throughput (tok/s):    2597.00
Peak concurrent requests:                107
Total token throughput (tok/s):          3269.09
Concurrency:                             91.14
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   28017.24
Median E2E Latency (ms):                 26865.28
---------------Time to First Token----------------
Mean TTFT (ms):                          387.41
Median TTFT (ms):                        183.90
P99 TTFT (ms):                           1192.44
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          55.23
Median TPOT (ms):                        57.84
P99 TPOT (ms):                           70.23
---------------Inter-Token Latency----------------
Mean ITL (ms):                           54.79
Median ITL (ms):                         39.01
P95 ITL (ms):                            143.10
P99 ITL (ms):                            150.46
Max ITL (ms):                            986.14
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Server Command**:

```shell Command theme={null}
sglang serve \
    --model-path MiniMaxAI/MiniMax-M2 \
    --tp-size 4 \
    --trust-remote-code \
    --mem-fraction-static 0.85
```

* **Benchmark Command**:

```shell Command theme={null}
  sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Result**:
  * MiniMax-M2

```text Output theme={null}
 Accuracy: 0.950
 Invalid: 0.000
 Latency: 15.120 s
 Output throughput: 1306.711 token/s
```


# MiniMax-M2.5
Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2.5

Deploy MiniMax-M2.5 with SGLang on NVIDIA, AMD, or Ascend A2, with an Ascend A3 reference configuration.

## 1. Model Introduction

[MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) is a powerful language model developed by MiniMax, built for real-world productivity with state-of-the-art performance across coding, reasoning, agentic tasks, and tool use.

As the latest iteration in the MiniMax model series, MiniMax-M2.5 achieves comprehensive enhancements across multiple domains. Details are as follows:

* **Superior coding performance**: Achieves 79.7 on Droid and 76.1 on OpenCode, surpassing Opus 4.6 (78.9 and 75.9 respectively). Strong results on SWE-bench Verified, SWE-bench Multilingual, SWE-bench-pro, and Multi-SWE-bench.
* **Advanced reasoning**: Demonstrates strong performance on AIME25 and other reasoning benchmarks, with robust tool use during inference.
* **More capable agents**: Excels in agentic tasks including web browsing (BrowseComp, Wide Search), information retrieval (RISE), and complex tool use scenarios (Terminal Bench 2, MEWC, Finance Modeling).
* **Real-world productivity**: Designed for production-grade workloads with strong performance on practical coding, data analysis, and multi-step reasoning tasks.

For more details, please refer to the [official MiniMax-M2.5 announcement](https://www.minimax.io/news/minimax-m25).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](/docs/get-started/install) for installation instructions.

**For AMD MI300X/MI325X/MI355X GPUs:**

```bash Command theme={null}
# Docker (AMD MI300X/MI325X)
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x

# Docker (AMD MI355X)
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi35x
```

### Ascend A2 prerequisites (smoke tested)

Select **Ascend A2** for the tested **8 × Ascend 910B4, 32 GiB HBM per device** configuration. It loads the W8A8 main model with TP=8 and serves one request at a time, without a draft model. The final run passed six simple chat checks and six synthetic generation probes, including 3500 input tokens plus 128 output tokens; see [the validation record](#5-4-ascend-a2-smoke-validation) for the runtime versions and test limits.

1. Prepare a compatible Ascend NPU runtime using the [NPU installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation). The tested runtime was SGLang `0.5.14.dev172+gf308abc05`, TorchNPU 2.10.0, and sgl-kernel-npu 2026.5.1. Follow-up inspection recorded CANN 9.0.0 and driver 25.5.2. An image digest was not captured; the A3 image below was not validated for this A2 run.
2. Run `npu-smi info` inside the container and confirm all eight 910B4 devices are visible and healthy. Reserve space for the approximately 230.82 GB main weights, download cache, and runtime.
3. Download the complete main model repository, including tokenizer and configuration files:

```bash Command theme={null}
python -m pip install modelscope
modelscope download --model Eco-Tech/MiniMax-M2.5-w8a8-QuaRot \
  --local_dir /models/MiniMax-M2.5-w8a8-QuaRot
```

Enter absolute paths inside your container in the selector. Both directories default to the complete model directory. If you keep tokenizer files separately, use files from the same model revision and set **Tokenizer Directory** accordingly. The original experiment used separate metadata and weights directories, as recorded in section 5.4.

### Ascend A3 prerequisites (unvalidated reference)

The A3 selector provides **one Ascend A3 node with 8 cards / 16 NPU dies**, running prefill and decode together (PD co-located). It uses W8A8 weights and EAGLE3 speculative decoding. Each A3 card contains two dies; a cloud allocation labelled "1 NPU" is not a full node. The [minimum recommended W8A8 configuration](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5#model-weights) is 4 cards / 8 dies with 64 GB per die, while this preset requires all 16 dies.

> [!NOTE]
>
> This recipe is based on the upstream Ascend best practice. Hardware validation of the generated A3 cookbook command is pending; the reference accuracy and performance values below are not measurements of this recipe. The tested A2 selector is a separate low-concurrency configuration and does not validate the A3 preset.

1. Set up the Ascend runtime using the [MiniMax-M2.5 installation instructions](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5#installation). The reference environment uses CANN 9.0.0 and `quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16`. Map `/dev/davinci0` through `/dev/davinci15` and mount your model directory into the container, for example `-v /path/to/models:/models`.
2. Check device health and allocation with `npu-smi info` inside the container. Confirm that all 16 dies are visible. For source builds, follow [installation with NPU support](/docs/hardware-platforms/ascend-npus/getting-started/installation), keep the CANN, TorchNPU, and kernel versions compatible, and record the SGLang commit you install.
3. Reserve space for the **230.82 GB** main weights, the draft model, download cache, and at least **30 GB** for the container image. Download both models below, or reuse complete copies from shared storage.

| Role | Model | Directory inside the container |
| - | - | - |
| W8A8 main model | [Eco-Tech/MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) | `/models/MiniMax-M2.5-w8a8-QuaRot` |
| EAGLE3 draft model | [sgl-npu/MiniMax-M2.5-eagel-model-0318](https://www.modelscope.cn/models/sgl-npu/MiniMax-M2.5-eagel-model-0318) | `/models/MiniMax-M2.5-eagel-model-0318` |

The draft repository identifier contains `eagel`; use that exact spelling. Keep its `custom_eagle3` Python package alongside the draft weights. The generated command adds this directory to `PYTHONPATH` and sets `SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3`.

You can download the weights with the [ModelScope CLI](https://github.com/modelscope/modelscope/blob/master/docs/source/command.md#download):

```bash Command theme={null}
python3 -m pip install modelscope
modelscope download --model Eco-Tech/MiniMax-M2.5-w8a8-QuaRot \
  --local_dir /models/MiniMax-M2.5-w8a8-QuaRot
modelscope download --model sgl-npu/MiniMax-M2.5-eagel-model-0318 \
  --local_dir /models/MiniMax-M2.5-eagel-model-0318
```

Before launching, run `ip -brief address` and select an interface available inside the container for both HCCL and Gloo. The selector defaults to `lo` for this single-node setup.

For comparable performance, use the host settings from the [selected 8-card reference](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms). These affect the whole host; apply them on the dedicated benchmark server before running the generated command:

```bash Command theme={null}
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
```

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Select your hardware to generate the deployment command. For **Ascend A2**, complete the A2 prerequisites, expand **Deployment paths**, and enter the model and tokenizer directories; the command reproduces the tested single-request settings on port 31216. For **Ascend A3**, complete its separate prerequisites, expand **Deployment paths and network**, enter both model directories as absolute paths inside the container, and set the HCCL / Gloo interface. The unvalidated A3 preset uses 3,500 input and 1,500 output tokens as its reference workload.

Run this Command:

<pre />

### 3.2 Configuration Tips

**GPU parameters:** The following recommendations apply to the NVIDIA and AMD recipes. The Ascend presets use the separate settings described below.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Recommended Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--tool-call-parser`</td>
      <td>Tool call parser for function calling support</td>
      <td>`minimax-m2`</td>
    </tr>

    <tr>
      <td>`--reasoning-parser`</td>
      <td>Reasoning parser for thinking mode</td>
      <td>`minimax-append-think`</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>Required for MiniMax model loading</td>
      <td>Always enabled</td>
    </tr>

    <tr>
      <td>`--mem-fraction-static`</td>
      <td>Static memory fraction for KV cache</td>
      <td>`0.85`</td>
    </tr>

    <tr>
      <td>`--tp`</td>
      <td>Tensor parallelism size</td>
      <td>`2` (2-GPU) or `4` (4-GPU) or `8` (8-GPU)</td>
    </tr>

    <tr>
      <td>`--ep`</td>
      <td>Expert parallelism size</td>
      <td>`8` (NVIDIA 8-GPU) or EP=TP (AMD)</td>
    </tr>

    <tr>
      <td>`--kv-cache-dtype`</td>
      <td>KV cache data type (AMD only)</td>
      <td>`fp8_e4m3`</td>
    </tr>

    <tr>
      <td>`--attention-backend`</td>
      <td>Attention backend (AMD only)</td>
      <td>`triton`</td>
    </tr>
  </tbody>
</table>

**Hardware Requirements: NVIDIA**

* **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4
* **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8

**Hardware Requirements: AMD**

* **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2
* **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
* **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8

### 3.3 Ascend A3 configuration (unvalidated reference)

The generated A3 command follows the [8-card throughput reference](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms). Keep the environment variables and server arguments together when reproducing that workload.

| Setting | A3 preset | Purpose |
| - | - | - |
| Device and parallelism | `--device npu --tp-size 16 --dp-size 16 --enable-dp-attention` | Use 16 dies on one node with DP attention. |
| Quantization and communication | `--quantization modelslim --moe-a2a-backend ascend_fuseep --fuseep-mode 2` | Run the Ascend W8A8 weights with FuseEP. |
| Speculative decoding | EAGLE3, 3 steps, top-k 1, 4 draft tokens, unquantized draft | Load the custom draft model from the supplied directory. |
| Static memory and concurrency | `--mem-fraction-static 0.75 --max-running-requests 320` | Use the memory and request limits of the reference workload. |
| Prefill | Chunk size 196608, maximum prefill tokens 8192, prefill delayer enabled | Match the reference scheduling configuration. |
| NPU graph batches | `--cuda-graph-bs-decode 1 2 4 8 12 16 20` | Set capture sizes; the argument retains `cuda` in its name on NPU. |
| API | `127.0.0.1:6688` | Send requests from the same host or container network. |

The A3 preset fixes the reasoning parser to `minimax-append-think` and the tool parser to `minimax-m2`. The shared form displays a fixed 8-card count and enabled parsers; these controls are read-only for this preset. To use fewer A3 cards or a different workload, follow the corresponding [Ascend best practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5) with its complete configuration.

### 3.4 Ascend A2 configuration (smoke tested)

The A2 selector fixes TP=8, ModelSlim quantization, BF16, `--context-length 4096`, and `--max-running-requests 1`. It uses prefill chunks of 128, maximum prefill tokens of 4096, static memory fraction 0.96, and `--max-total-tokens 4096`, and disables graph capture and radix cache. No EAGLE3 draft model is required.

The `minimax` reasoning parser separated reasoning from answer content in the tested runtime. The `minimax-m2` tool-call parser is configured, but tool calling was not tested. The shared form displays a fixed 8-card count and enabled parsers; these controls are read-only for this preset. The final run allocated a **4096-token KV pool on all eight ranks** and completed a synthetic 3500-input/128-output request. This validates that request size on this allocation, not every workload up to the configured limit. See section 5.4 for the earlier capacity failure and the final measurements. GPQA accuracy, standardized throughput, and sustained stability remain unmeasured for this single-request preset; [section 5.5](#5-5-ascend-a2-performance-benchmark) reports synthetic performance measurements using a separate server configuration with concurrency up to 8.

## 4. Model Invocation

For the **Ascend A3** preset, use [the A3 service checks](#4-3-verify-the-ascend-a3-service), which target port 6688 and read the deployed model identifier from `/v1/models`. The GPU examples below use their own model identifier and endpoints.

For **Ascend A2**, use `http://127.0.0.1:31216/v1` from the same host or container network. After startup completes, this short request uses the settings from the smoke test:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:31216/v1", api_key="EMPTY")
model = client.models.list().data[0].id
response = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "What is the capital of France? Answer with the city name only."}],
    temperature=0,
    max_tokens=512,
)
print(response.choices[0].message.content)
```

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](/docs/basic_usage/send_request)

**Testing Deployment:**

After startup, you can test the SGLang OpenAI-compatible API with the following command:

```bash Command theme={null}
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniMaxAI/MiniMax-M2.5",
        "messages": [
            {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
            {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
        ]
    }'
```

**Simple Completion Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.5",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"}
    ],
    max_tokens=1024
)

print(response.choices[0].message.content)
```

**Example Output**:

```text Output theme={null}
<think>The user asks: "Who won the world series in 2020?" That is a straightforward factual question. The answer: the Los Angeles Dodgers. They won the 2020 World Series, beating the Tampa Bay Rays. The user is presumably expecting that answer.

We must follow the policies. The question is safe: no disallowed content. It's just a factual question. Provide answer.

We must ensure compliance: Use no disallowed content. Should we provide context? Just answer straightforwardly.

The user simply asks "Who won the world series in 2020?" We'll answer: The Los Angeles Dodgers.

No additional relevant info needed, but could elaborate briefly: They beat the Tampa Bay Rays in six games, the series was played in a bubble at Globe Life Field in Arlington, Texas due to COVID-19.

No need for any extra. That's it.
</think>

The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games.
```

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

MiniMax-M2.5 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path MiniMaxAI/MiniMax-M2.5 \
  --tp 4 \
  --reasoning-parser minimax-append-think \
  --trust-remote-code \
  --mem-fraction-static 0.85
```

**Streaming with Thinking Process**

With `minimax-append-think`, the thinking content is wrapped in `<think>...</think>` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.5",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream, separating <think>...</think> from content
in_think = False
think_printed_header = False
content_printed_header = False
buffer = ""

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            buffer += delta.content

            while buffer:
                if in_think:
                    # Look for closing </think> tag
                    end_idx = buffer.find("</think>")
                    if end_idx != -1:
                        print(buffer[:end_idx], end="", flush=True)
                        buffer = buffer[end_idx + len("</think>"):]
                        in_think = False
                    else:
                        # Still in thinking, print what we have
                        print(buffer, end="", flush=True)
                        buffer = ""
                else:
                    # Look for opening <think> tag
                    start_idx = buffer.find("<think>")
                    if start_idx != -1:
                        # Print any content before <think>
                        before = buffer[:start_idx]
                        if before:
                            if not content_printed_header:
                                print("=============== Content =================", flush=True)
                                content_printed_header = True
                            print(before, end="", flush=True)
                        buffer = buffer[start_idx + len("<think>"):]
                        in_think = True
                        if not think_printed_header:
                            print("=============== Thinking =================", flush=True)
                            think_printed_header = True
                    else:
                        # No <think> tag, print as content
                        if not content_printed_header and think_printed_header:
                            print("\n=============== Content =================", flush=True)
                            content_printed_header = True
                        print(buffer, end="", flush=True)
                        buffer = ""

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" This is straightforward: 15% = 0.15; 0.15*240 = 36. So answer: 36. Provide step-by-step: convert percent to decimal, multiply.

We need to obey policies. There's no policy violation. Just answer. Provide step by step. Should respond with solution.

We can also mention alternative method: 15% = 15/100 = 3/20. Multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.

Thus answer 36.

We can add step-by-step. That's it.

=============== Content =================

**Step‑by‑step solution**

1. **Convert the percent to a decimal**
   \[
   15\% = \frac{15}{100}=0.15
   \]

2. **Multiply the decimal by the number**
   \[
   0.15 \times 240 = 36
   \]

(You can also think of it as \(15\% = \frac{3}{20}\) and then \(240 \times \frac{3}{20}=12 \times 3 = 36\).)

\[
\boxed{36}
\]
```

**Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `<think>...</think>` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately.

#### 4.2.2 Tool Calling

MiniMax-M2.5 supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model-path MiniMaxAI/MiniMax-M2.5 \
  --tp 4 \
  --tool-call-parser minimax-m2 \
  --reasoning-parser minimax-append-think \
  --trust-remote-code \
  --mem-fraction-static 0.85
```

**Python Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Non-streaming request
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.5",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7
)

message = response.choices[0].message

# Check for tool calls
if message.tool_calls:
    for tool_call in message.tool_calls:
        print(f"Tool Call: {tool_call.function.name}")
        print(f"   Arguments: {tool_call.function.arguments}")
else:
    print(message.content)
```

**Output Example**:

```text Output theme={null}
Tool Call: get_weather
   Arguments: {"location": "Beijing"}
```

**Note:**

* Tool calls are returned in `message.tool_calls` with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.5",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

### 4.3 Verify the Ascend A3 service

Wait for the server to finish loading the weights and capturing graphs. Keep the generated command running, then open another terminal in the same container. The A3 preset listens on port **6688**.

```bash Command theme={null}
curl --fail --silent --show-error --max-time 30 http://127.0.0.1:6688/health
curl --fail --silent --show-error --max-time 30 http://127.0.0.1:6688/v1/models

python3 - <<'PY'
import json
from urllib.request import Request, urlopen

base = "http://127.0.0.1:6688"
with urlopen(f"{base}/v1/models", timeout=30) as response:
    model = json.load(response)["data"][0]["id"]
payload = {
    "model": model,
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "temperature": 0,
    "max_tokens": 2048,
}
request = Request(
    f"{base}/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(request, timeout=300) as response:
    result = json.load(response)
print(json.dumps(result, indent=2, ensure_ascii=False))
choice = result["choices"][0]
assert (choice["message"].get("content") or "").strip(), "No final answer returned"
assert choice["finish_reason"] != "length", "Increase the token budget and retry"
PY
```

Check that the final answer identifies Paris. Repeat requests and check the server logs for runtime errors. After any concurrent test, check `/health` again. A successful HTTP response alone does not establish model accuracy; use the evaluation below for that.

## 5. Benchmark

For **Ascend A3**, use [the A3 throughput and accuracy benchmarks](#5-3-ascend-a3-benchmarks) with the command generated in section 3.

For **Ascend A2**, [section 5.4](#5-4-ascend-a2-smoke-validation) records simple chat checks and serial synthetic generation probes. [Section 5.5](#5-5-ascend-a2-performance-benchmark) reports the official benchmark tool's results on a separate configuration with concurrency up to 8 and a synthetic workload. [Section 5.6](#5-6-ascend-a2-reference-length-performance) measures 3500-input/1500-output requests, and [section 5.7](#5-7-ascend-a2-fixed-50-question-gsm8k-regression) records a fixed 50-question GSM8K deployment regression. No GPQA accuracy result is reported here.

The benchmark sections below use **industry-standard configurations** for comparable benchmark results; the A2 smoke record has a separate, limited scope.

### 5.1 Speed Benchmark

**Test Environment**:

* Hardware: NVIDIA B200 GPU (8x)
* Model: MiniMax-M2.5
* Tensor Parallelism: 8
* Expert Parallelism: 8
* sglang version: 0.5.8

#### 5.1.1 Standard Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
    --model-path MiniMaxAI/MiniMax-M2.5 \
    --tp 8 \
    --ep 8 \
    --reasoning-parser minimax-append-think \
    --trust-remote-code \
    --mem-fraction-static 0.85 \
    --tool-call-parser minimax-m2
```

##### 5.1.1.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  42.99
Total input tokens:                      6091
Total input text tokens:                 6091
Total generated tokens:                  4220
Total generated tokens (retokenized):    3804
Request throughput (req/s):              0.23
Input token throughput (tok/s):          141.70
Output token throughput (tok/s):         98.17
Peak output token throughput (tok/s):    102.00
Peak concurrent requests:                2
Total token throughput (tok/s):          239.87
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4295.92
Median E2E Latency (ms):                 3419.28
P90 E2E Latency (ms):                    7832.04
P99 E2E Latency (ms):                    9601.40
---------------Time to First Token----------------
Mean TTFT (ms):                          130.57
Median TTFT (ms):                        116.10
P99 TTFT (ms):                           190.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.89
Median TPOT (ms):                        9.89
P99 TPOT (ms):                           9.91
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.89
Median ITL (ms):                         9.89
P95 ITL (ms):                            10.15
P99 ITL (ms):                            10.32
Max ITL (ms):                            14.46
==================================================
```

##### 5.1.1.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  48.43
Total input tokens:                      39588
Total input text tokens:                 39588
Total generated tokens:                  40805
Total generated tokens (retokenized):    37142
Request throughput (req/s):              1.65
Input token throughput (tok/s):          817.37
Output token throughput (tok/s):         842.49
Peak output token throughput (tok/s):    1184.00
Peak concurrent requests:                21
Total token throughput (tok/s):          1659.86
Concurrency:                             13.67
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8274.32
Median E2E Latency (ms):                 8692.90
P90 E2E Latency (ms):                    13690.70
P99 E2E Latency (ms):                    16104.18
---------------Time to First Token----------------
Mean TTFT (ms):                          305.44
Median TTFT (ms):                        106.75
P99 TTFT (ms):                           1053.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.20
Median TPOT (ms):                        16.06
P99 TPOT (ms):                           26.75
---------------Inter-Token Latency----------------
Mean ITL (ms):                           15.65
Median ITL (ms):                         13.63
P95 ITL (ms):                            14.90
P99 ITL (ms):                            87.99
Max ITL (ms):                            483.53
==================================================
```

##### 5.1.1.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  92.31
Total input tokens:                      249331
Total input text tokens:                 249331
Total generated tokens:                  252662
Total generated tokens (retokenized):    218975
Request throughput (req/s):              5.42
Input token throughput (tok/s):          2700.94
Output token throughput (tok/s):         2737.02
Peak output token throughput (tok/s):    4479.00
Peak concurrent requests:                109
Total token throughput (tok/s):          5437.97
Concurrency:                             91.19
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16835.82
Median E2E Latency (ms):                 16042.08
P90 E2E Latency (ms):                    31027.63
P99 E2E Latency (ms):                    34787.91
---------------Time to First Token----------------
Mean TTFT (ms):                          391.06
Median TTFT (ms):                        133.12
P99 TTFT (ms):                           1712.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.04
Median TPOT (ms):                        34.29
P99 TPOT (ms):                           41.98
---------------Inter-Token Latency----------------
Mean ITL (ms):                           32.61
Median ITL (ms):                         21.67
P95 ITL (ms):                            87.76
P99 ITL (ms):                            118.81
Max ITL (ms):                            1145.62
==================================================
```

#### 5.1.2 Summarization Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
    --model-path MiniMaxAI/MiniMax-M2.5 \
    --tp 8 \
    --ep 8 \
    --reasoning-parser minimax-append-think \
    --trust-remote-code \
    --mem-fraction-static 0.85 \
    --tool-call-parser minimax-m2
```

##### 5.1.2.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  43.49
Total input tokens:                      41941
Total input text tokens:                 41941
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.23
Input token throughput (tok/s):          964.42
Output token throughput (tok/s):         97.04
Peak output token throughput (tok/s):    102.00
Peak concurrent requests:                2
Total token throughput (tok/s):          1061.46
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4346.83
Median E2E Latency (ms):                 3508.84
P90 E2E Latency (ms):                    7972.23
P99 E2E Latency (ms):                    9659.71
---------------Time to First Token----------------
Mean TTFT (ms):                          131.50
Median TTFT (ms):                        126.76
P99 TTFT (ms):                           182.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.00
Median TPOT (ms):                        10.01
P99 TPOT (ms):                           10.12
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.01
Median ITL (ms):                         10.02
P95 ITL (ms):                            10.29
P99 ITL (ms):                            10.44
Max ITL (ms):                            14.11
==================================================
```

##### 5.1.2.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  50.12
Total input tokens:                      300020
Total input text tokens:                 300020
Total generated tokens:                  41669
Total generated tokens (retokenized):    41662
Request throughput (req/s):              1.60
Input token throughput (tok/s):          5986.00
Output token throughput (tok/s):         831.38
Peak output token throughput (tok/s):    1152.00
Peak concurrent requests:                20
Total token throughput (tok/s):          6817.38
Concurrency:                             13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8727.66
Median E2E Latency (ms):                 9170.52
P90 E2E Latency (ms):                    14220.00
P99 E2E Latency (ms):                    16896.54
---------------Time to First Token----------------
Mean TTFT (ms):                          282.56
Median TTFT (ms):                        149.37
P99 TTFT (ms):                           1278.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.60
Median TPOT (ms):                        16.61
P99 TPOT (ms):                           25.17
---------------Inter-Token Latency----------------
Mean ITL (ms):                           16.24
Median ITL (ms):                         13.89
P95 ITL (ms):                            15.96
P99 ITL (ms):                            105.79
Max ITL (ms):                            1065.02
==================================================
```

##### 5.1.2.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  93.92
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total generated tokens:                  170000
Total generated tokens (retokenized):    169999
Request throughput (req/s):              3.41
Input token throughput (tok/s):          13563.30
Output token throughput (tok/s):         1810.01
Peak output token throughput (tok/s):    2881.00
Peak concurrent requests:                71
Total token throughput (tok/s):          15373.31
Concurrency:                             58.87
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   17277.69
Median E2E Latency (ms):                 16827.33
P90 E2E Latency (ms):                    29045.40
P99 E2E Latency (ms):                    33496.77
---------------Time to First Token----------------
Mean TTFT (ms):                          692.26
Median TTFT (ms):                        188.46
P99 TTFT (ms):                           4932.70
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.19
Median TPOT (ms):                        32.69
P99 TPOT (ms):                           50.46
---------------Inter-Token Latency----------------
Mean ITL (ms):                           31.28
Median ITL (ms):                         21.59
P95 ITL (ms):                            101.35
P99 ITL (ms):                            136.74
Max ITL (ms):                            4649.23
==================================================
```

#### 5.1.3 H100 Benchmark

**Test Environment**:

* Hardware: NVIDIA H100 80GB HBM3 GPU (8x)

* Model: MiniMax-M2.5

* Tensor Parallelism: 8

* Expert Parallelism: 8

* sglang version: 0.5.9

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
    --model-path MiniMaxAI/MiniMax-M2.5 \
    --tp 8 \
    --ep 8 \
    --reasoning-parser minimax-append-think \
    --trust-remote-code \
    --mem-fraction-static 0.85 \
    --tool-call-parser minimax-m2
```

##### 5.1.3.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  35.44
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.28
Input token throughput (tok/s):          172.16
Output token throughput (tok/s):         119.08
Peak output token throughput (tok/s):    127.00
Peak concurrent requests:                2
Total token throughput (tok/s):          291.24
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3542.38
Median E2E Latency (ms):                 2791.92
P90 E2E Latency (ms):                    6317.77
P99 E2E Latency (ms):                    7780.15
---------------Time to First Token----------------
Mean TTFT (ms):                          145.20
Median TTFT (ms):                        80.38
P99 TTFT (ms):                           633.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.05
Median TPOT (ms):                        8.08
P99 TPOT (ms):                           8.09
---------------Inter-Token Latency----------------
Mean ITL (ms):                           8.07
Median ITL (ms):                         8.08
P95 ITL (ms):                            8.12
P99 ITL (ms):                            8.16
Max ITL (ms):                            10.10
==================================================
```

##### 5.1.3.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  43.68
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40805
Request throughput (req/s):              1.83
Input token throughput (tok/s):          908.19
Output token throughput (tok/s):         934.22
Peak output token throughput (tok/s):    1184.00
Peak concurrent requests:                20
Total token throughput (tok/s):          1842.42
Concurrency:                             13.83
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7551.91
Median E2E Latency (ms):                 8094.28
P90 E2E Latency (ms):                    12606.99
P99 E2E Latency (ms):                    14977.84
---------------Time to First Token----------------
Mean TTFT (ms):                          116.86
Median TTFT (ms):                        82.33
P99 TTFT (ms):                           240.59
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.81
Median TPOT (ms):                        14.98
P99 TPOT (ms):                           17.98
---------------Inter-Token Latency----------------
Mean ITL (ms):                           14.61
Median ITL (ms):                         13.50
P95 ITL (ms):                            14.15
P99 ITL (ms):                            66.52
Max ITL (ms):                            107.39
==================================================
```

##### 5.1.3.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  80.63
Total input tokens:                      249831
Total input text tokens:                 249831
Total generated tokens:                  252662
Total generated tokens (retokenized):    252331
Request throughput (req/s):              6.20
Input token throughput (tok/s):          3098.45
Output token throughput (tok/s):         3133.56
Peak output token throughput (tok/s):    4800.00
Peak concurrent requests:                113
Total token throughput (tok/s):          6232.01
Concurrency:                             90.56
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   14604.59
Median E2E Latency (ms):                 14044.04
P90 E2E Latency (ms):                    26456.53
P99 E2E Latency (ms):                    30136.68
---------------Time to First Token----------------
Mean TTFT (ms):                          149.32
Median TTFT (ms):                        95.16
P99 TTFT (ms):                           374.62
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          28.92
Median TPOT (ms):                        30.09
P99 TPOT (ms):                           34.31
---------------Inter-Token Latency----------------
Mean ITL (ms):                           28.66
Median ITL (ms):                         21.52
P95 ITL (ms):                            66.90
P99 ITL (ms):                            96.76
Max ITL (ms):                            376.34
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* Benchmark Command:

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Results:

```text Output theme={null}
Accuracy: 0.950
Invalid: 0.000
Latency: 18.033 s
Output throughput: 1130.161 token/s
```

#### 5.2.2 MMLU Benchmark

* Benchmark Command:

```shell Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Results:

```text Output theme={null}
subject: abstract_algebra, #q:100, acc: 0.620
subject: anatomy, #q:135, acc: 0.830
subject: astronomy, #q:152, acc: 0.928
subject: business_ethics, #q:100, acc: 0.810
subject: clinical_knowledge, #q:265, acc: 0.891
subject: college_biology, #q:144, acc: 0.951
subject: college_chemistry, #q:100, acc: 0.670
subject: college_computer_science, #q:100, acc: 0.820
subject: college_mathematics, #q:100, acc: 0.660
subject: college_medicine, #q:173, acc: 0.832
subject: college_physics, #q:102, acc: 0.814
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.915
subject: econometrics, #q:114, acc: 0.719
subject: electrical_engineering, #q:145, acc: 0.834
subject: elementary_mathematics, #q:378, acc: 0.902
subject: formal_logic, #q:126, acc: 0.698
subject: global_facts, #q:100, acc: 0.710
subject: high_school_biology, #q:310, acc: 0.926
subject: high_school_chemistry, #q:203, acc: 0.793
subject: high_school_computer_science, #q:100, acc: 0.910
subject: high_school_european_history, #q:165, acc: 0.879
subject: high_school_geography, #q:198, acc: 0.955
subject: high_school_government_and_politics, #q:193, acc: 0.964
subject: high_school_macroeconomics, #q:390, acc: 0.908
subject: high_school_mathematics, #q:270, acc: 0.600
subject: high_school_microeconomics, #q:238, acc: 0.954
subject: high_school_physics, #q:151, acc: 0.781
subject: high_school_psychology, #q:545, acc: 0.956
subject: high_school_statistics, #q:216, acc: 0.847
subject: high_school_us_history, #q:204, acc: 0.922
subject: high_school_world_history, #q:237, acc: 0.916
subject: human_aging, #q:223, acc: 0.839
subject: human_sexuality, #q:131, acc: 0.893
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.861
subject: logical_fallacies, #q:163, acc: 0.890
subject: machine_learning, #q:112, acc: 0.750
subject: management, #q:103, acc: 0.883
subject: marketing, #q:234, acc: 0.944
subject: medical_genetics, #q:100, acc: 0.920
subject: miscellaneous, #q:783, acc: 0.936
subject: moral_disputes, #q:346, acc: 0.829
subject: moral_scenarios, #q:895, acc: 0.632
subject: nutrition, #q:306, acc: 0.863
subject: philosophy, #q:311, acc: 0.833
subject: prehistory, #q:324, acc: 0.907
subject: professional_accounting, #q:282, acc: 0.720
subject: professional_law, #q:1534, acc: 0.640
subject: professional_medicine, #q:272, acc: 0.923
subject: professional_psychology, #q:612, acc: 0.871
subject: public_relations, #q:110, acc: 0.773
subject: security_studies, #q:245, acc: 0.845
subject: sociology, #q:201, acc: 0.930
subject: us_foreign_policy, #q:100, acc: 0.940
subject: virology, #q:166, acc: 0.614
subject: world_religions, #q:171, acc: 0.895
Total latency: 81.468
Average accuracy: 0.825
```

### 5.3 Ascend A3 benchmarks

Use the A3 command generated in section 3 for both accuracy and performance evaluation. Record the SGLang commit, container image digest, CANN / TorchNPU / kernel versions, model revisions, visible dies, and host tuning with your logs. When testing `main`, record the installed source commit separately from the base container tag.

#### 5.3.1 Throughput workload

The [Ascend reference](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) lists **50 ms TPOT** for 8 A3 cards, W8A8, PD co-located, and 3.5k input / 1.5k output tokens. This is a reference value, not a measurement from a different allocation or software build. The reference table does not specify the TPOT percentile.

After warm-up, run the reference workload from the same container:

```bash Command theme={null}
set -o pipefail
python3 -m sglang.bench_serving \
  --dataset-name random \
  --backend sglang \
  --host 127.0.0.1 \
  --port 6688 \
  --max-concurrency 320 \
  --num-prompts 1280 \
  --random-input-len 3500 \
  --random-output-len 1500 \
  --random-range-ratio 1 \
  --seed 1 2>&1 | tee minimax-a3-throughput.log
```

Keep the request success and failure counts, actual token counts, output and total token throughput, TTFT, TPOT, and end-to-end latency. Report mean, median, and P99 separately. Repeat the measurement with separate logs and check server health afterwards. Changes to concurrency, quantization, speculative decoding, or token lengths define a different benchmark configuration.

#### 5.3.2 GPQA Diamond accuracy

Install EvalScope in a separate environment following [Ascend accuracy evaluation](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation). Keep the SGLang server running with the generated A3 command. The following request configuration follows the [upstream 8-card GPQA test](https://github.com/sgl-project/sglang/blob/c389dd086340878934bf96bc209534f1fbb8440c/test/registered/npu/accuracy/minimax_m2_5/test_npu_minimax_m2_5_w8a8_8p_in3k5_out1k5_50ms_gpqa.py):

```python evaluate_minimax_a3.py theme={null}
import json
from urllib.request import urlopen

from evalscope import TaskConfig, run_task

with urlopen("http://127.0.0.1:6688/v1/models", timeout=30) as response:
    model = json.load(response)["data"][0]["id"]

run_task(task_cfg=TaskConfig(
    model=model,
    api_url="http://127.0.0.1:6688/v1/chat/completions",
    eval_type="openai_api",
    datasets=["gpqa_diamond"],
    eval_batch_size=64,
    generation_config={"max_tokens": 65536, "temperature": 1.0},
    timeout=60000,
    stream=True,
    work_dir="./minimax-a3-gpqa",
))
```

Run the full **198-question** dataset and retain the EvalScope version, dataset revision, generation settings, reports, and all attempt scores. The upstream test declares an accuracy baseline of **0.852**; its [comparison utility](https://github.com/sgl-project/sglang/blob/c389dd086340878934bf96bc209534f1fbb8440c/python/sglang/test/ascend/e2e/test_npu_accuracy_utils.py) allows a fluctuation of 5 questions, giving a threshold of `0.852 - 5/198`, approximately **0.8267**, with up to 3 attempts. These are upstream validation criteria, not measured accuracy for this cookbook command.

### 5.4 Ascend A2 smoke validation

On September 22, 2026, the W8A8 main model completed loading, warm-up, and three short chat requests on **8 Ascend 910B4 devices with 32 GiB HBM each**. Eight-rank HCCL all-reduce and BF16 matrix operations also passed. The final September 23 run increased the static memory fraction to 0.96 and set `--max-total-tokens 4096`; these are the settings now exposed by the **Ascend A2** selector. These experiments used no EAGLE3 draft model and are separate from the A3 reference configuration.

The runtime was SGLang `0.5.14.dev172+gf308abc05` (commit `f308abc05212c2f5f455de22a525e14afa63ee4f`), Python 3.11.15, Torch 2.10.0+cpu, TorchNPU 2.10.0, Transformers 5.8.1, and sgl-kernel-npu 2026.5.1. A follow-up inspection of the same environment on September 23 recorded CANN 9.0.0 and driver 25.5.2. The supplied environment had modified `python/pyproject.toml` and a deleted `python/pyproject_npu.toml`; this was not a clean source-build validation. An image digest and immutable model revision were not captured, so this record is not a fully pinned environment specification.

After downloading the W8A8 main model listed above, the final tested service settings were:

```bash Command theme={null}
export HCCL_BUFFSIZE=128
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
python -m sglang.launch_server \
  --model-path /workspace/issue1215/model \
  --tokenizer-path /workspace/issue1215/metadata \
  --device npu --tp-size 8 --quantization modelslim --dtype bfloat16 \
  --reasoning-parser minimax --tool-call-parser minimax-m2 \
  --host 127.0.0.1 --port 31216 \
  --context-length 4096 --max-running-requests 1 \
  --chunked-prefill-size 128 --max-prefill-tokens 4096 \
  --mem-fraction-static 0.96 --max-total-tokens 4096 \
  --disable-cuda-graph --disable-radix-cache
```

The paths above are the experiment paths: `model` contains the complete main weights, while `metadata` contains the tokenizer and configuration files from the same model repository, downloaded separately while weights were transferring. Replace both paths with your local directories. The selector defaults both paths to one complete repository directory; this path convenience differs from the original experiment's separate metadata directory. Keep tokenizer and weights from the same revision.

The original September 22 requests used `/v1/chat/completions`, `temperature=0`, and `max_tokens=512`:

| Prompt | Answer content | Finish reason |
| - | - | - |
| What is the capital of France? Answer with the city name only. | `Paris` | `stop` |
| 请用中文回答：2加2等于多少？ | `2加2等于4。` | `stop` |
| Return only a JSON object with one key named result whose value is 7 \* 8. | `{"result":56}` (parsed JSON) | `stop` |

The `minimax` reasoning parser separated reasoning from answer content in this runtime. The three answers contained no thinking tags, and the final health check returned HTTP 200. The tool-call parser was configured but tool calling was not tested.

**Earlier baseline:** a September 23 serial follow-up retained `--mem-fraction-static 0.95` without an explicit total-token setting. After two warm-up requests, five of six simple answer checks passed; the JSON check exhausted its 128-token output budget and was inconclusive. Reasoning consumes that budget as well as final-answer tokens. Startup allocated 896 KV tokens; requests with 1024 and 2048 input tokens were rejected with `BadRequestError`, reporting a maximum input length of 890. The client stopped after those two failures, before attempting 3500 input tokens, and the subsequent health check returned HTTP 200. These limits describe the earlier baseline, not the final configuration above.

**Final September 23 run:** with memory fraction 0.96 and `--max-total-tokens 4096`, startup reported 4096 allocated KV tokens on each of the eight ranks. After two warm-up requests, all six chat checks passed with `temperature=0`, `max_tokens=512`, and finish reason `stop`:

| Check | Answer content | Completion tokens | End-to-end time (s) | First observed output (s) |
| - | - | - | - | - |
| Capital of France | `Paris` | 87 | 21.172 | 0.474 |
| 2 + 2 | `4` | 53 | 12.598 | 0.466 |
| JSON object for 7 × 8 | `{"result":56}` | 185 | 43.747 | 0.477 |
| 中国首都 | `北京` | 138 | 31.891 | 0.456 |
| Copy a word | `orchard` | 73 | 16.724 | 0.459 |
| Sort 3, 1, 2 as a JSON array | `[1,2,3]` | 142 | 33.590 | 0.465 |

Six native `/generate` probes used exact input-token IDs from `/v1/tokenize`, requested 128 output tokens, and set `ignore_eos=true`. All completed with exactly 128 output tokens and the expected length limit. The inputs were synthetic repeated text: these are individual operational measurements, not a quality evaluation or standardized throughput benchmark.

| Input tokens | Output tokens | End-to-end time (s) | First observed output (s) |
| - | - | - | - |
| 64 | 128 | 29.970 | 0.494 |
| 128 | 128 | 29.313 | 0.478 |
| 256 | 128 | 30.193 | 0.700 |
| 1024 | 128 | 32.165 | 2.103 |
| 2048 | 128 | 34.451 | 4.218 |
| 3500 | 128 | 39.803 | 10.055 |

Timing includes HTTP/SSE processing and ends at the stream's `DONE` event. First output means the first nonempty content or reasoning for chat, or the first text or positive output-token count for native generation; it is not the time to the final answer or a kernel timing. Completion counts include reasoning and special tokens as reported by the server. Parser buffering and chunks containing multiple tokens can affect observed timing. No cache flush was performed. Each input length was measured once at concurrency 1, so these results do not establish a latency percentile or saturation throughput.

The final probe run completed and its post-run health check returned HTTP 200. Confirm startup KV allocation and actual request limits on your allocation; the successful 3500-input/128-output probe does not establish the full 4096-token boundary for every workload.

These checks establish simple chat functionality and the listed synthetic request sizes. They do not establish GPQA accuracy, long-context answer quality, sustained stability, tool calling, or performance under the A3 reference workload. This observed 8×32 GiB low-concurrency run does not replace the upstream recommended hardware configuration.

### 5.5 Ascend A2 performance benchmark

On September 23, 2026, the official `python -m sglang.bench_serving` tool completed **10 runs and 76 measured requests** on the same 8 × 910B4 allocation and runtime recorded in section 5.4. All requests succeeded, with the planned input lengths and exactly 128 output tokens each. Warm-up requests are excluded from the 76 measured requests.

This benchmark used a separate server configuration: `--mem-fraction-static 0.97`, `--max-total-tokens 16384`, `--context-length 8192`, and `--max-running-requests 8`. The interactive selector retains the single-request configuration from section 5.4. The benchmark launch command was:

```bash Command theme={null}
export HCCL_BUFFSIZE=128
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
python -m sglang.launch_server \
  --model-path /workspace/issue1215/model \
  --tokenizer-path /workspace/issue1215/metadata \
  --device npu --tp-size 8 --quantization modelslim --dtype bfloat16 \
  --reasoning-parser minimax --tool-call-parser minimax-m2 \
  --host 127.0.0.1 --port 31216 \
  --context-length 8192 --max-running-requests 8 \
  --chunked-prefill-size 128 --max-prefill-tokens 4096 \
  --mem-fraction-static 0.97 --max-total-tokens 16384 \
  --disable-cuda-graph --disable-radix-cache
```

#### 5.5.1 Workload and measured results

The source data contained **16 locally authored synthetic English prompts in ShareGPT-compatible JSON format**; it was not the ShareGPT dataset. The tool used `--dataset-name random`, `--random-range-ratio 1`, and `--tokenize-prompt` to produce exact input lengths of 128 or 1024 tokens. Each run used `--request-rate inf`, temperature 0, one warm-up request, and `--flush-cache`. The serial control used seed 17; each repeated group used seeds **17, 29, and 43**, once each.

The repeated groups below report the arithmetic **mean and minimum–maximum across three runs**. TTFT, TPOT, and E2E use each run's mean latency before computing this summary. The serial control has one run and therefore no repeat range. Output throughput is aggregate output tokens per second from the benchmark, not per-request decoding speed. All latency values are in milliseconds.

**128 input / 128 output tokens, concurrency 1** — one run of 4 requests; 4/4 succeeded.

| Metric | Value |
| - | - |
| Output throughput (tok/s) | 4.266 |
| Mean TTFT (ms) | 486.589 |
| Mean TPOT (ms) | 232.385 |
| Mean E2E (ms) | 29999.459 |

**128 input / 128 output tokens, concurrency 4** — three runs of 8 requests; 24/24 succeeded.

| Metric | Mean | Min–max |
| - | - | - |
| Output throughput (tok/s) | 16.731 | 16.618–16.810 |
| Mean TTFT (ms) | 807.497 | 801.011–814.685 |
| Mean TPOT (ms) | 234.492 | 233.452–236.043 |
| Mean E2E (ms) | 30588.025 | 30449.379–30792.190 |

**128 input / 128 output tokens, concurrency 8** — three runs of 8 requests; 24/24 succeeded.

| Metric | Mean | Min–max |
| - | - | - |
| Output throughput (tok/s) | 32.457 | 32.373–32.532 |
| Mean TTFT (ms) | 1312.882 | 1276.591–1359.876 |
| Mean TPOT (ms) | 237.912 | 237.480–238.622 |
| Mean E2E (ms) | 31527.731 | 31456.127–31607.222 |

**1024 input / 128 output tokens, concurrency 8** — three runs of 8 requests; 24/24 succeeded.

| Metric | Mean | Min–max |
| - | - | - |
| Output throughput (tok/s) | 13.694 | 13.291–13.942 |
| Mean TTFT (ms) | 12582.293 | 12329.436–13062.333 |
| Mean TPOT (ms) | 272.578 | 268.251–280.396 |
| Mean E2E (ms) | 47199.685 | 46422.970–48672.617 |

Every run had a 100% request success rate; this measures successful request completion, not answer accuracy. Per-run percentiles remain in the detailed benchmark output and are not averaged into pooled percentiles. These short tests do not establish sustained stability or long-context answer quality. Their workload, runtime, and hardware differ from the A3 reference, so the table does not provide an A2-versus-A3 comparison.

#### 5.5.2 Reproduce the benchmark

Start the server above and wait for `/health` to return HTTP 200. Replace the two model paths below with your container paths, using the same tokenizer revision as the weights. Save the following as `benchmark_minimax_a2.py` and run it from the tested SGLang environment. It creates the synthetic data used here and runs the fixed suite, retaining each command, log, and detailed JSONL result. It refuses to overwrite an existing result and stops if a run fails validation.

```python benchmark_minimax_a2.py theme={null}
import json
from pathlib import Path
import subprocess
import sys

model = "/workspace/issue1215/model"
tokenizer = "/workspace/issue1215/metadata"
out = Path("minimax-a2-perf")
out.mkdir(exist_ok=True)
topics = [
    "gardening", "astronomy", "cooking", "geometry", "music", "weather",
    "transport", "ecology", "history", "software", "literature", "chemistry",
    "geography", "photography", "materials", "language",
]
data = [{"id": f"synthetic-{i}", "conversations": [
    {"from": "human", "value": (
        f"Explain the basic principles of {topic}. Give a clear example and discuss its practical limitations. "
    ) * 8},
    {"from": "gpt", "value": "This is a synthetic performance workload, not a quality evaluation."},
]} for i, topic in enumerate(topics)]
dataset = out / "synthetic-prompts.json"
dataset.write_text(json.dumps(data), encoding="utf-8")
cases = ([(128, 1, 4, 17)]
         + [(128, c, 8, s) for c in (4, 8) for s in (17, 29, 43)]
         + [(1024, 8, 8, s) for s in (17, 29, 43)])
for length, concurrency, count, seed in cases:
    name = f"in{length}-out128-c{concurrency}-n{count}-seed{seed}"
    result = out / f"{name}.jsonl"
    if result.exists():
        raise RuntimeError(f"Refusing to overwrite {result}")
    command = [
        sys.executable, "-m", "sglang.bench_serving", "--backend", "sglang",
        "--base-url", "http://127.0.0.1:31216", "--model", model, "--tokenizer", tokenizer,
        "--dataset-name", "random", "--dataset-path", str(dataset),
        "--random-input-len", str(length), "--random-output-len", "128",
        "--random-range-ratio", "1", "--tokenize-prompt", "--num-prompts", str(count),
        "--max-concurrency", str(concurrency), "--request-rate", "inf",
        "--temperature", "0", "--seed", str(seed), "--warmup-requests", "1",
        "--flush-cache", "--output-details", "--disable-tqdm", "--output-file", str(result),
    ]
    (out / f"{name}.command.json").write_text(json.dumps(command, indent=2), encoding="utf-8")
    with (out / f"{name}.log").open("w", encoding="utf-8") as log:
        subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, check=True)
    rows = [json.loads(line) for line in result.read_text().splitlines() if line.strip()]
    if len(rows) != 1:
        raise RuntimeError(f"Expected exactly one result in {result}")
    row = rows[0]
    if not (row["completed"] == count
            and row["input_lens"] == [length] * count
            and row["output_lens"] == [128] * count
            and len(row["errors"]) == count and not any(row["errors"])):
        raise RuntimeError(f"Incomplete or mismatched run: inspect {result}")
    print(name, row["output_throughput"], flush=True)
```

Retain failed runs and missing-run records when reporting a reproduction; do not summarize only successful seeds. Keep each run's mean and percentile metrics separate. The synthetic data and performance measurements provide no GPQA or other model-quality score.

### 5.6 Ascend A2 reference-length performance

On September 24, 2026, the same A2 hardware and runtime completed three official `sglang.bench_serving` runs with **3500 input tokens and 1500 output tokens**, using seeds **1, 17, and 29**. Each run measured two requests at client concurrency 2. All **6/6 measured requests** completed with the exact planned token lengths and no request errors: 21000 input tokens and 9000 output tokens in total. Each run's one-request, 32-output-token warm-up is excluded from these counts.

The server used the section 5.5 launch command with `--context-length 9216` instead of 8192. Its other settings remained TP8, ModelSlim, BF16, memory fraction 0.97, KV limit 16384, maximum running requests 8, prefill chunk 128, maximum prefill tokens 4096, and graph capture and radix cache disabled. **Client concurrency 2 did not change the server's maximum of 8.** No draft model was used; the interactive selector's conservative default is unchanged.

The workload reused the 16 synthetic prompts from section 5.5. It matches only the A3 reference's token lengths, not its ShareGPT workload, concurrency 320, hardware, or EAGLE3 configuration. It does not validate A3 throughput or the reference's 50 ms TPOT target.

| Metric | Mean | Min–max |
| - | - | - |
| Output throughput (tok/s) | 8.004 | 7.896–8.065 |
| Total throughput (tok/s) | 26.680 | 26.319–26.882 |
| Mean TTFT (ms) | 10317.118 | 10267.749–10365.114 |
| Mean TPOT (ms) | 243.163 | 241.222–246.570 |
| Mean E2E (s) | 374.818 | 371.957–379.927 |

These are arithmetic means and minimum–maximum values across the three run-level means or rates, not request extrema. Per-run percentiles are retained separately and are not averaged. Six measured requests provide a bounded performance observation, not a sustained-load or stability result.

To reproduce, generate `minimax-a2-perf/synthetic-prompts.json` using the data-creation portion of section 5.5.2, launch the server with context 9216, and run the following from that working directory. Use the same model/tokenizer paths as the server, and retain each JSONL and log, including failures:

```bash Command theme={null}
set -euo pipefail
mkdir -p minimax-a2-reference-perf
for SEED in 1 17 29; do
  RESULT="minimax-a2-reference-perf/in3500-out1500-c2-n2-seed${SEED}.jsonl"
  test ! -e "$RESULT"
  python -m sglang.bench_serving \
    --backend sglang --base-url http://127.0.0.1:31216 \
    --model /workspace/issue1215/model \
    --tokenizer /workspace/issue1215/metadata \
    --dataset-name random --dataset-path minimax-a2-perf/synthetic-prompts.json \
    --random-input-len 3500 --random-output-len 1500 --random-range-ratio 1 \
    --tokenize-prompt --num-prompts 2 --max-concurrency 2 \
    --request-rate inf --temperature 0 --seed "$SEED" \
    --warmup-requests 1 --flush-cache --output-details --disable-tqdm \
    --output-file "$RESULT" 2>&1 | tee "${RESULT%.jsonl}.log"
done
```

Check every result for `completed=2`, input lengths `[3500,3500]`, output lengths `[1500,1500]`, and two empty error entries before aggregating it. Verify server health after each run. A successful process exit alone is insufficient to establish those conditions.

### 5.7 Ascend A2 fixed 50-question GSM8K regression

On September 24, 2026, a **preselected 50-question deployment regression completed with 49/50 correct (98%)**, zero capped outputs, zero transport/protocol errors, and no missing or duplicate questions. This is a bounded GSM8K test subset, not full GSM8K accuracy, an EvalScope result, GPQA accuracy, or evidence of A3 parity.

The run used the same A2 hardware, W8A8 weights, and runtime as above. The server used context 9216, maximum running requests **2**, memory fraction 0.97, and KV limit 16384; all other launch settings matched section 5.6. The client submitted **one request at a time** to `/v1/chat/completions`, with `temperature=1.0`, `max_tokens=6144`, and generation seed `42 + original test row index`. The reasoning parser remained `minimax`; scoring used only final answer content.

The protocol was frozen before an independent three-question pilot:

1. Use the MIT-licensed [official GSM8K data at revision 3101c7d5072418e28b9008a6636bde82a006892c](https://github.com/openai/grade-school-math/tree/3101c7d5072418e28b9008a6636bde82a006892c/grade_school_math/data). Take training rows 0–4 as five worked demonstrations.
2. Shuffle all 1319 test-row indices with Python `random.Random(20260924).shuffle`. Reserve the first three indices for the independent pilot and use the next 50, in order, for this regression. Do not include pilot answers in the result or select, replace, reorder, or drop questions based on observed answers.
3. Build a single user message by joining the five `Question: ...\nAnswer: ...` demonstrations with two newlines, then append the test question as `Question: ...\nAnswer:`. Preserve the training answers, including their worked solutions. Apply the deployed MiniMax chat template through the chat endpoint.
4. Before any generation, tokenize all 50 messages with the server and verify that each prompt plus the 6144-token output allowance fits context 9216. Submit each frozen question exactly once, with a per-request wall limit of 1800 seconds. Stop new submissions on a transport/protocol error; do not retry or resume a failed run.
5. Parse only final content using the pinned runtime's last-number rule: remove commas, find matches of `-?\d+\.?\d*`, and parse the last match numerically. Compare it with the gold answer parsed the same way. A missing number, wrong answer, length cap, or transport/protocol error scores zero. Keep the denominator fixed at 50; report an incomplete run as incomplete rather than a full score.

The selection can be reconstructed without model access:

```python Example theme={null}
import random

indices = list(range(1319))
random.Random(20260924).shuffle(indices)
pilot_ids = indices[:3]       # Excluded from regression scoring.
regression_ids = indices[3:53]
assert len(set(regression_ids)) == 50
assert not set(pilot_ids).intersection(regression_ids)
```

| Check | Observed result |
| - | - |
| Unique completed questions | 50/50 |
| Correct final answers | 49/50 (98%) |
| Wrong final answers | 1 |
| Length-capped responses | 0 |
| Transport/protocol errors | 0 |
| Completion-token range | 292–2507 |

All 50 prompts passed preflight. The client elapsed time, including preflight, was 9444.598 seconds; it is not a per-request latency benchmark. The downloaded run's 172 manifest entries passed hash verification, and rescoring the preserved final answers with the declared parser reproduced 49/50. Preserve the frozen IDs, prompt hashes, parameters, every response, and raw SSE when reproducing the regression. This result tests the declared subset and parsing protocol; it does not establish full-dataset accuracy, long-context quality, sustained stability, or the unvalidated A3 recipe's acceptance criteria.


# MiniMax-M2.7
Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M2.7



## 1. Model Introduction

[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is MiniMax's first model deeply participating in its own evolution. Built for real-world productivity, M2.7 excels at building complex agent harnesses and completing highly elaborate productivity tasks, leveraging Agent Teams, complex Skills, and dynamic tool search.

Key highlights:

* **Model Self-Evolution**: During development, M2.7 updates its own memory, builds complex skills for RL experiments, and improves its own learning process. An internal version autonomously optimized a programming scaffold over 100+ rounds, achieving a **30% performance improvement**. On MLE Bench Lite, M2.7 achieved a **66.6% medal rate**.
* **Professional Software Engineering**: Delivers outstanding real-world programming capabilities. On SWE-Pro, M2.7 achieved **56.22%**, with strong results on SWE Multilingual (76.5) and Multi SWE Bench (52.7). On Terminal Bench 2 (57.0%) and NL2Repo (39.8%), M2.7 demonstrates deep understanding of complex engineering systems.
* **Professional Work**: Achieved an ELO score of **1495** on GDPval-AA (highest among open-source models). On Toolathon, M2.7 reached **46.3%** accuracy (global top tier).
* **Native Agent Teams**: Supports multi-agent collaboration with stable role identity and autonomous decision-making.

For more details, see the [official MiniMax-M2.7 blog post](https://www.minimax.io/news/minimax-m27-en).

**License**: [Modified-MIT (MiniMax Model License)](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE)

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

**Docker Images by Hardware Platform:**

<table>
  <thead>
    <tr>
      <th>Hardware Platform</th>
      <th>Docker Image</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>NVIDIA A100 / H100 / H200 / B200</td>
      <td>`lmsysorg/sglang:v0.5.10.post1`</td>
    </tr>

    <tr>
      <td>NVIDIA B300 / GB300</td>
      <td>`lmsysorg/sglang:v0.5.10.post1-cu130`</td>
    </tr>

    <tr>
      <td>AMD MI300X / MI325X</td>
      <td>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi30x`</td>
    </tr>

    <tr>
      <td>AMD MI355X</td>
      <td>`lmsysorg/sglang:v0.5.10.post1-rocm720-mi35x`</td>
    </tr>
  </tbody>
</table>

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and feature capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

**Key Parameters:**

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Recommended Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--tool-call-parser`</td>
      <td>Tool call parser for function calling support</td>
      <td>`minimax-m2`</td>
    </tr>

    <tr>
      <td>`--reasoning-parser`</td>
      <td>Reasoning parser for thinking mode</td>
      <td>`minimax-append-think`</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>Required for MiniMax model loading</td>
      <td>Always enabled</td>
    </tr>

    <tr>
      <td>`--mem-fraction-static`</td>
      <td>Static memory fraction for KV cache</td>
      <td>`0.85`</td>
    </tr>

    <tr>
      <td>`--tp`</td>
      <td>Tensor parallelism size</td>
      <td>`2` / `4` / `8` depending on hardware</td>
    </tr>

    <tr>
      <td>`--ep`</td>
      <td>Expert parallelism size</td>
      <td>`8` (NVIDIA 8-GPU) or EP=TP (AMD)</td>
    </tr>

    <tr>
      <td>`--kv-cache-dtype`</td>
      <td>KV cache data type (AMD only)</td>
      <td>`fp8_e4m3`</td>
    </tr>

    <tr>
      <td>`--attention-backend`</td>
      <td>Attention backend (AMD only)</td>
      <td>`triton`</td>
    </tr>
  </tbody>
</table>

**Hardware Requirements: NVIDIA**

* **4-GPU deployment**: Requires 4× high-memory GPUs (e.g., H200, B200, A100, H100) with TP=4
* **8-GPU deployment**: Requires 8× GPUs (e.g., H200, B200, A100, H100) with TP=8 and EP=8

**Hardware Requirements: NVIDIA GB300**

* **2-GPU deployment**: GB300 (275GB per die) can host the model with TP=2
* **4-GPU deployment**: Maximum single-node TP for GB300, recommended for higher throughput

**Hardware Requirements: AMD**

* **2-GPU deployment**: Requires 2× high-memory GPUs (e.g., MI300X, MI325X, MI355X) with TP=2, EP=2
* **4-GPU deployment**: Requires 4× GPUs (e.g., MI300X, MI325X, MI355X) with TP=4, EP=4
* **8-GPU deployment**: Requires 8× GPUs (e.g., MI300X, MI325X, MI355X) with TP=8, EP=8

**Hardware Requirements: Intel Xeon CPU**

* It is recommended to run the model service on a Granite Rapids (GNR) AP 2-Socket server.

* For configuring CPU service, please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

**Deployment Command:**

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-M2.7 \
  --tp 4 \
  --tool-call-parser minimax-m2 \
  --reasoning-parser minimax-append-think \
  --trust-remote-code \
  --mem-fraction-static 0.85
```

**Testing Deployment:**

After startup, you can test the SGLang OpenAI-compatible API with the following command:

```bash Command theme={null}
curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "MiniMaxAI/MiniMax-M2.7",
        "messages": [
            {"role": "system", "content": [{"type": "text", "text": "You are a helpful assistant."}]},
            {"role": "user", "content": [{"type": "text", "text": "Who won the world series in 2020?"}]}
        ]
    }'
```

**Simple Completion Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.7",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"}
    ],
    max_tokens=1024
)

print(response.choices[0].message.content)
```

**Example Output**:

```text Output theme={null}
<think>The user asks: "Who won the World Series in 2020?" That's a simple factual question. The answer: the Los Angeles Dodgers won the 2020 MLB World Series, defeating the Tampa Bay Rays. So answer accordingly.

We must be mindful of policy: it's a factual question about sports. It's allowed. Provide answer with brief context.

We should answer concisely.

Hence final answer: The Los Angeles Dodgers won the 2020 World Series, defeating the Tampa Bay Rays in six games (best-of-seven series). Possibly mention it was played at a neutral site due to COVID-19, at Globe Life Field in Arlington, Texas.

We must avoid disallowed content, no issue.

Thus final.
</think>

The **Los Angeles Dodgers** won the 2020 World Series. They defeated the **Tampa Bay Rays** in six games (4‑2) in a best‑of‑seven series that was played at Globe Life Field in Arlington, Texas, under the MLB bubble‑like arrangements for the COVID‑19 pandemic.
```

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

MiniMax-M2.7 supports Thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-M2.7 \
  --tp 4 \
  --reasoning-parser minimax-append-think \
  --trust-remote-code \
  --mem-fraction-static 0.85
```

**Streaming with Thinking Process**

With `minimax-append-think`, the thinking content is wrapped in `<think>...</think>` tags within the `content` field. You can parse these tags on the client side to separate the thinking and content sections:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.7",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    stream=True
)

# Process the stream, separating <think>...</think> from content
in_think = False
think_printed_header = False
content_printed_header = False
buffer = ""

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            buffer += delta.content

            while buffer:
                if in_think:
                    # Look for closing </think> tag
                    end_idx = buffer.find("</think>")
                    if end_idx != -1:
                        print(buffer[:end_idx], end="", flush=True)
                        buffer = buffer[end_idx + len("</think>"):]
                        in_think = False
                    else:
                        # Still in thinking, print what we have
                        print(buffer, end="", flush=True)
                        buffer = ""
                else:
                    # Look for opening <think> tag
                    start_idx = buffer.find("<think>")
                    if start_idx != -1:
                        # Print any content before <think>
                        before = buffer[:start_idx]
                        if before:
                            if not content_printed_header:
                                print("=============== Content =================", flush=True)
                                content_printed_header = True
                            print(before, end="", flush=True)
                        buffer = buffer[start_idx + len("<think>"):]
                        in_think = True
                        if not think_printed_header:
                            print("=============== Thinking =================", flush=True)
                            think_printed_header = True
                    else:
                        # No <think> tag, print as content
                        if not content_printed_header and think_printed_header:
                            print("\n=============== Content =================", flush=True)
                            content_printed_header = True
                        print(buffer, end="", flush=True)
                        buffer = ""

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" Straightforward. Provide solution: 15% = 15/100 = 0.15. Multiply 240 * 0.15 = 36. Show steps. So answer: 36. Provide explanation.

But also ensure we follow any policy? No issues. Just straightforward.

I'll provide a step-by-step solution.

Also could show fraction: 15% = 15/100 = 3/20, multiply 240 * 3/20 = (240/20)*3 = 12*3 = 36.

Yes. Provide final answer. Also show verification: 10% of 240 is 24, 5% is 12, total 36.

All good.

=============== Content =================

**Step‑by‑step solution**

1. **Convert the percent to a decimal (or a fraction).**

   15% = 15/100 = 0.15 = 3/20

2. **Multiply the original number (240) by this decimal/fraction.**

   Using the decimal:
   240 × 0.15 = 36

   Or using the fraction:
   240 × 3/20 = (240/20) × 3 = 12 × 3 = 36

3. **Result:**

   15% of 240 = **36**

*Check:*
- 10% of 240 = 24
- 5% of 240 = 12
- Adding them: 24 + 12 = 36, which matches the calculation.
```

**Note:** The `minimax-append-think` reasoning parser embeds the thinking process in `<think>...</think>` tags within the `content` field. The code above parses these tags in real-time to display thinking and content separately.

#### 4.2.2 Tool Calling

MiniMax-M2.7 supports tool calling capabilities. Enable the tool call parser:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-M2.7 \
  --tp 4 \
  --tool-call-parser minimax-m2 \
  --reasoning-parser minimax-append-think \
  --trust-remote-code \
  --mem-fraction-static 0.85
```

**Python Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Non-streaming request
response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.7",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools
)

message = response.choices[0].message

# Check for tool calls
if message.tool_calls:
    for tool_call in message.tool_calls:
        print(f"Tool Call: {tool_call.function.name}")
        print(f"   Arguments: {tool_call.function.arguments}")
else:
    print(message.content)
```

**Output Example**:

```text Output theme={null}
Tool Call: get_weather
   Arguments: {"location": "Beijing"}
```

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="MiniMaxAI/MiniMax-M2.7",
    messages=messages
)

print(final_response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
The weather in Beijing is currently 22°C and sunny.
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

**Test Environment**:

* Hardware: 2× NVIDIA GB300 (275GB per die)
* Docker Image: `lmsysorg/sglang:v0.5.10.post1-cu130`
* Model: MiniMax-M2.7 (FP8)
* Tensor Parallelism: 2
* SGLang version: 0.5.10.post1

### 5.1 Accuracy Benchmark

**Evaluation Tool**: [sgl-eval](https://github.com/sgl-project/sgl-eval)

**Evaluation Settings**: temperature=0.6, top\_p=0.95, 8 repeats, max\_tokens=120,000 (GPQA and AIME; MMLU-Pro is greedy and single-pass, see below)

#### 5.1.1 GPQA Diamond

* Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions)
* Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) -- sgl-eval's default for this benchmark
* Evaluation command:

```bash Command theme={null}
pip install sgl-eval

sgl-eval run gpqa \
    --base-url http://localhost:30000/v1 \
    --model MiniMaxAI/MiniMax-M2.7 \
    --n-repeats 8 \
    --max-tokens 120000 \
    --temperature 0.6 \
    --top-p 0.95
```

* Test Results:

<table>
  <thead>
    <tr>
      <th>Evaluation Mode</th>
      <th>Accuracy</th>
      <th>No Answer</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>pass\@1 (avg-of-8)</td>
      <td>84.91%</td>
      <td>3.54%</td>
    </tr>

    <tr>
      <td>**majority\@8**</td>
      <td>**88.89%**</td>
      <td>0.00%</td>
    </tr>

    <tr>
      <td>pass\@8</td>
      <td>96.46%</td>
      <td>0.00%</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 AIME 2025

* Dataset: AIME 2025 (30 problems)
* Prompt: `generic/math` (boxed answer format) -- sgl-eval's default for this benchmark
* Evaluation command:

```bash Command theme={null}
sgl-eval run aime25 \
    --base-url http://localhost:30000/v1 \
    --model MiniMaxAI/MiniMax-M2.7 \
    --n-repeats 8 \
    --max-tokens 120000 \
    --temperature 0.6 \
    --top-p 0.95
```

* Test Results:

<table>
  <thead>
    <tr>
      <th>Evaluation Mode</th>
      <th>Accuracy</th>
      <th>No Answer</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>pass\@1 (avg-of-8)</td>
      <td>92.50% ± 5.56%</td>
      <td>2.92%</td>
    </tr>

    <tr>
      <td>**majority\@8**</td>
      <td>**97.08%**</td>
      <td>0.00%</td>
    </tr>

    <tr>
      <td>pass\@8</td>
      <td>100.00%</td>
      <td>0.00%</td>
    </tr>
  </tbody>
</table>

#### 5.1.3 MMLU-Pro

* Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice)
* Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) -- sgl-eval's default for this benchmark
* Evaluation command:

```bash Command theme={null}
sgl-eval run mmlu_pro \
    --base-url http://localhost:30000/v1 \
    --model MiniMaxAI/MiniMax-M2.7 \
    --max-tokens 32768 \
    --temperature 0.0
```

* Test Results:

<table>
  <thead>
    <tr>
      <th>Evaluation Mode</th>
      <th>Accuracy</th>
      <th>No Answer</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>pass\@1 (greedy)</td>
      <td>69.41%</td>
      <td>18.75%</td>
    </tr>
  </tbody>
</table>

> **Note**: The high no-answer rate is due to the 32K token limit being insufficient for M2.7's extended thinking on some questions. A rerun with 120K tokens is expected to improve accuracy significantly.

#### 5.1.4 GSM8K Benchmark

* Benchmark Method: 8-shot Chain-of-Thought, evaluated via OpenAI-compatible API
* Test Results:

```text Output theme={null}
GSM8K Results (8-shot CoT)
Model: MiniMaxAI/MiniMax-M2.7
Total: 1319
Correct: 1218
Accuracy: 92.34%
```

### 5.2 Speed Benchmark

#### 5.2.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  34.33
Total input tokens:                      6101
Total generated tokens:                  4220
Request throughput (req/s):              0.29
Input token throughput (tok/s):          177.71
Output token throughput (tok/s):         122.92
Total token throughput (tok/s):          300.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3431.21
Median E2E Latency (ms):                 2742.57
---------------Time to First Token----------------
Mean TTFT (ms):                          50.28
Median TTFT (ms):                        53.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.02
Median TPOT (ms):                        8.01
---------------Inter-Token Latency----------------
Mean ITL (ms):                           8.03
Median ITL (ms):                         8.02
==================================================
```

#### 5.2.2 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model MiniMaxAI/MiniMax-M2.7 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  100.20
Total input tokens:                      249831
Total generated tokens:                  252662
Request throughput (req/s):              4.99
Input token throughput (tok/s):          2493.41
Output token throughput (tok/s):         2521.66
Total token throughput (tok/s):          5015.07
Concurrency:                             90.19
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   18072.69
Median E2E Latency (ms):                 17761.84
---------------Time to First Token----------------
Mean TTFT (ms):                          247.94
Median TTFT (ms):                        92.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          35.75
Median TPOT (ms):                        36.67
---------------Inter-Token Latency----------------
Mean ITL (ms):                           35.34
Median ITL (ms):                         30.55
==================================================
```


# MiniMax-M3
Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3

Deploy MiniMax-M3 with SGLang — a ~428B-param (23B activated) multimodal Mixture-of-Experts reasoning model with MiniMax Sparse Attention and 1M context, MXFP8 on NVIDIA Blackwell & AMD Instinct, bf16 on Hopper.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install -U uv
> uv venv --python 3.12 && source .venv/bin/activate
>
> # MiniMax-M3 ships in SGLang PR #27944, not yet in a tagged release — install from
> # the PR head. The serving runtime is in the base dependencies, so no extra is needed:
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> git fetch origin pull/27944/head && git checkout FETCH_HEAD
> uv pip install --prerelease=allow -e python
> ```
>
> Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — its image bundles the CUDA-13 runtime and the #27944 code. Once [PR #27944](https://github.com/sgl-project/sglang/pull/27944) is merged and released, `uv pip install --prerelease=allow sglang` will pull M3 support directly.
>
> **Docker**
>
> ```bash Command theme={null}
> # Pull the M3 image the command panel selects for your platform, e.g.:
> docker pull lmsysorg/sglang:dev-cu13-minimax-m3
> ```
>
> The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200; final build, the CUDA 12 lane is retired), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces.
>
> > [!NOTE]
> >
> > These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path.

Pick your hardware + recipe to generate the launch command.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

[MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) is MiniMax's native-multimodal Mixture-of-Experts reasoning model: **\~428B total parameters with \~23B activated per token** (128 experts, 4 active per token), 60 layers, and a **1M-token context** over text, image, and video. Its defining feature is **MiniMax Sparse Attention (MSA)** — a block-sparse "lightning indexer" attention that keeps long-context cost low (MiniMax reports \~9× prefill / \~15× decode speedup over M2 at 1M context). This page serves the **MXFP8** variant (`MiniMaxAI/MiniMax-M3-MXFP8`, \~440 GB) on NVIDIA Blackwell and AMD Instinct; on NVIDIA Hopper (H200), use the full-precision **bfloat16** build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3) (§2.4). Released under the **MiniMax Community License**.

Key characteristics as served by SGLang:

* **Multimodal (vision + text)**: accepts interleaved text and images through the OpenAI-compatible chat API (loaded as `MiniMaxM3SparseForConditionalGeneration`). Image input via URL and base64 is validated; video input has not been tested here.
* **Reasoning model**: emits its chain of thought wrapped in `<mm:think>...</mm:think>`. Always launch with **`--reasoning-parser auto`** — it auto-detects the right parser from the chat template, and SGLang then strips the tags and returns the trace separately in `message.reasoning_content`.
* **Native tool calling**: a custom namespace-token XML format, parsed into standard OpenAI `tool_calls`. Always launch with **`--tool-call-parser auto`** — it auto-detects the right parser from the chat template. Single, parallel, and nested (object / array) arguments are supported.
* **Sparse attention**: most layers use M3's "lightning indexer" block-sparse attention (top-k 128-token blocks), which keeps decode cost roughly flat in context length. On Blackwell, MiniMax's open-source [MSA kernel](https://github.com/MiniMax-AI/MSA) accelerates this path further (§2.1).
* **MXFP8 quantization across vendors**: the MXFP8 MoE weights run natively on NVIDIA Blackwell (B200 / B300 / GB200 / GB300) and on AMD Instinct MI350X/MI355X (gfx950 / CDNA4), both of which have hardware MX-scaled matmul. On AMD MI300X/MI325X (gfx942 / CDNA3) — no hardware MX — SGLang converts the weights to block-fp8 `[128,128]` at load and serves them on the tuned ROCm kernels (§2.3). The vision tower stays unquantized.

**Recommended generation**: the model's `generation_config.json` sets `temperature` 1.0 / `top_p` 0.95, which SGLang applies automatically (the default `--sampling-defaults model`). The model card additionally suggests `top_k` 40, but that value is **not** in `generation_config.json`, so SGLang does not apply it by default. `top_k` is a per-request sampling parameter (not a launch flag) — set it per call if you want it, e.g. `extra_body={"top_k": 40}` with the OpenAI client.

**Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8) · [MSA kernel](https://github.com/MiniMax-AI/MSA)

## 2. Configuration Tips

### 2.1 MSA sparse-attention fast path (recommended for Blackwell users)

[MiniMax MSA](https://github.com/MiniMax-AI/MSA) (`fmha_sm100`, MIT-licensed) is the recommended Blackwell kernel for M3's main sparse-attention step — faster and more memory-efficient than the built-in Triton fallback. **It ships pre-installed in the M3 dev image** (`lmsysorg/sglang:dev-minimax-m3`, also published under the `dev-cu13-minimax-m3` tag), so the Blackwell recipe above engages it automatically with no extra setup — `import fmha_sm100` works out of the box and the kernels JIT-compile on first use. It is otherwise purely additive: on a custom image, install it (below) and the recipe engages it automatically; without it the same recipe still serves on the built-in Triton path. The swap is numerically equivalent (cosine ≥ 0.99999 vs Triton), decode stays CUDA-graph-capturable, prefill TTFT drops \~9–12% at 8K–64K context, and the MSA path survives memory configurations where the Triton path OOMs.

**Requirements** (from the [MSA README](https://github.com/MiniMax-AI/MSA#requirements)):

* **GPU**: NVIDIA SM100 family — sm\_100 (B200 / GB200) and sm\_103 (B300 / GB300).
* **Toolchain**: CUDA Toolkit with `nvcc` ≥ 12.x on `PATH` (or `CUDA_HOME` set) — the kernels are JIT-compiled at first import.
* **Python**: ≥ 3.11; **OS**: Linux — works on both **x86\_64 and aarch64 (Grace, e.g. GB200 / GB300)**; the aarch64 build needs no source edits.

> **Install MSA (only on a custom image) & verify the gate (Python)**
>
> The M3 Blackwell dev images above already bundle MSA, so you can skip straight to the gate check. The `git clone` / `pip install` steps are only needed on a custom image that doesn't have `fmha_sm100`.
>
> ```bash Command theme={null}
> # Only on a custom image: --recursive pulls the CUTLASS submodule required for JIT compilation
> git clone --recursive https://github.com/MiniMax-AI/MSA.git msa
> cd msa && pip install .
> # Verify the SGLang gate (True -> MSA engaged on this device; False -> Triton fallback):
> python -c "from sglang.srt.layers.attention.minimax_sparse_ops.msa import msa_available; print(msa_available())"
> ```

> [!NOTE]
>
> The first import JIT-compiles the kernels, which can take 30 s to a few minutes on a cold `nvcc` cache — this is normal, not a hang. Subsequent server starts hit the JIT cache.

> [!WARNING]
>
> **Warm the JIT cache before a multi-GPU launch.** On a *cold* cache, several tensor-parallel ranks racing to JIT-compile MSA's plan kernel can leave one rank loading a half-linked module (`AttributeError: Module has no function 'plan'` at CUDA-graph capture). Run the gate-check `python -c "..."` (or any single-process `fmha_sm100_plan` call) once before launching the server — that compiles the kernel single-process, and every rank then hits the warm cache.

The gate requires `--attention-backend fa4` (MSA's sparse blocks are 128 tokens, so the page size must be 128). SGLang auto-forces `page_size` to 128 for the `fa4` backend — including the combined `--attention-backend fa4` the M3 recipe uses (#28976) — so `--page-size 128` is omitted from the Blackwell cells below. Force the Triton path at any time with the env var `SGLANG_DISABLE_MSA=1`. MSA is a Blackwell (SM100) kernel and does not apply to the AMD ROCm paths.

> [!NOTE]
>
> For multimodal (image) serving, keep the same text recipe above — `--attention-backend fa4` (MSA) is unchanged — and add `--mm-attention-backend flashinfer_cudnn` for the vision tower. The text and vision-tower attention backends are independent knobs; MSA only touches the language-model sparse attention, not image handling.

### 2.2 Memory and workload tuning

The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and **B300 / GB300 at `--tp 4`** (4-GPU is also the GB200 / GB300 single-node ceiling). GB200 (sm\_100, aarch64) is inferred-supported — both of its axes are validated above (B200 is sm\_100; GB300 is sm\_103 aarch64) — but not directly benchmarked. The AMD recipes use **8-GPU (`--tp 8`)**.

* **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context.
* **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated.
* **HiSparse for decode capacity**: on NVIDIA CUDA, HiSparse keeps the three dense layers on GPU, moves the 57 sparse-layer K/V caches to pinned host memory, and feeds selected block IDs directly to the swap-in kernel. For the released four-KV-head model, use `--tp 4` or greater, `--disable-radix-cache`, and `device_buffer_size >= 2048`. Enable it with `--enable-hisparse --hisparse-config='{"device_buffer_size":4096,"host_to_device_ratio":2}'` on the Triton launch command.
* **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom.
* **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy.
* `--trust-remote-code` is required to load the MiniMax config / processor classes.

### 2.3 AMD Instinct (ROCm)

MiniMax-M3 runs on AMD Instinct GPUs through two code paths, by architecture — both selected automatically; you still pass `--quantization mxfp8` either way:

* **MI350X / MI355X (gfx950, CDNA4)** has hardware MX-scaled matmul, so the **MXFP8 weights are served natively**. SGLang auto-detects the checkpoint, selects the Triton MiniMax-M3 MoE path with the packaged tuned MXFP8 configs, and enables AITER fused all-reduce for single-node tensor parallelism. The launch command is the NVIDIA recipe minus the Blackwell-only backend flags.
* **MI300X / MI325X (gfx942, CDNA3)** has **no** hardware MX matmul. SGLang transparently **converts the MXFP8 weights to block-fp8 `[128,128]` at load time**, then serves them with the tuned ROCm block-fp8 kernels (`--attention-backend aiter`, `--moe-runner-backend triton`; the `aiter` runner also works and scores marginally higher). On a cold start the first generation can JIT-compile AITER configs and exceed the default warmup/HTTP timeout, so the recipe adds `--watchdog-timeout 3600 --skip-server-warmup`. The block-fp8 step adds only a small relative error over MXFP8's native `1×32` scaling — negligible on GSM8K (see the benchmark card).

Select an MI300X/MI325X or MI350X/MI355X tile in the command panel above to get the exact launch command for each path.

> [!NOTE]
>
> The AMD recipes are validated end-to-end on **text** workloads — chat, reasoning separation, and tool calling. The vision tower was not exercised on ROCm; for image input on AMD, omit the Blackwell `--mm-attention-backend flashinfer_cudnn` flag and let the encoder use the ROCm default backend, and treat vision as unvalidated on that path.

### 2.4 Serving on Hopper (H200) with the bf16 build

The MXFP8 kernels are Blackwell-only, so Hopper (H200) serves the full-precision bfloat16 build [`MiniMaxAI/MiniMax-M3`](https://huggingface.co/MiniMaxAI/MiniMax-M3). Select **H200 + BF16** in the Deploy panel above for the exact command — it runs at `--tp 8` (the bf16 weights need a full 8-GPU node). SGLang picks the right backends for Hopper automatically, so the recipe stays minimal:

* **MoE runner**: Triton, auto-selected for bf16 weights.
* **Attention**: FlashAttention-3 with page size 1. MSA (§2.1) is a Blackwell kernel, so M3's sparse step runs on the built-in Triton path here.
* **CUDA graph**: on, with full decode-graph capture.

**High-concurrency throughput (optional).** On Hopper the sparse prefill runs on the Triton path as a separate eager forward, which briefly stalls the in-flight decode batch under heavy concurrent load. Adding `--enable-mixed-chunk --chunked-prefill-size 2048` merges the running decodes into the prefill step instead of preempting them, which recovers roughly **+10% output throughput** and **\~10% lower median TPOT** at high concurrency on 8×H200, with no change in accuracy. Leave it off for latency-sensitive low-concurrency serving.

Validated on 8×H200 — reasoning and tool-call auto-detection plus long-context generation. For prefill/decode disaggregation on Hopper, see §3.4.

## 3. Advanced Usage

### 3.1 Reasoning

Launch with `--reasoning-parser auto` (or toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)). The `<mm:think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3-MXFP8",
>     messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
>     max_tokens=2048,
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(message.reasoning_content)
> print("=============== Answer ==================")
> print(message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> =============== Reasoning ===============
> 15% of 240. 15% = 0.15. 240 * 0.15 = 36. Quick check: 10% is 24, 5% is 12, 24 + 12 = 36.
> =============== Answer ==================
> 15% of 240 is **36**.
> (10% of 240 = 24, and 5% of 240 = 12; 24 + 12 = 36.)
> ```

When streaming, the trace arrives on `delta.reasoning_content` and the answer on `delta.content`, so the two sections can be rendered separately in real time:

> **Streaming Reasoning (Python)**
>
> ```python Example theme={null}
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3-MXFP8",
>     messages=[{"role": "user", "content": "Solve step by step: what is 15% of 240?"}],
>     max_tokens=2048,
>     stream=True,
> )
>
> for chunk in response:
>     if not chunk.choices:
>         continue
>     delta = chunk.choices[0].delta
>     if getattr(delta, "reasoning_content", None):
>         print(delta.reasoning_content, end="", flush=True)  # thinking stream
>     if delta.content:
>         print(delta.content, end="", flush=True)            # answer stream
> print()
> ```
>
> **Output Example:**
>
> ```text Output theme={null}
> [delta.reasoning_content — thinking stream]
> Let me solve this step by step.
>
> 15% of 240
> = 0.15 × 240
> = 36
>
> Let me verify: 10% of 240 = 24, 5% of 240 = 12, so 15% = 24 + 12 = 36. ✓
>
> [delta.content — answer stream]
> # Solving 15% of 240
> ## Step 1: Convert the percentage to a decimal
> 15% = 15/100 = 0.15
> ## Step 2: Multiply by 240
> 0.15 × 240 = 36
> ## Answer
> **15% of 240 = 36**
> ```

### 3.2 Tool Calling

Launch with `--tool-call-parser auto` (or toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) — it auto-detects M3's tool-call parser from the chat template. M3 emits tool calls in a custom namespace-token XML format:

```text Raw model output theme={null}
]<]minimax[>[<tool_call>
]<]minimax[>[<invoke name="get_weather">]<]minimax[>[<location>Beijing]<]minimax[>[</location>]<]minimax[>[</invoke>
]<]minimax[>[</tool_call>
```

The parser converts that into the standard OpenAI `tool_calls` structure:

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3-MXFP8",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
>
> message = response.choices[0].message
> if message.tool_calls:
>     for call in message.tool_calls:
>         print(f"Tool: {call.function.name}")
>         print(f"Args: {call.function.arguments}")
> ```

> **Example Output**
>
> ```text Output theme={null}
> Tool: get_weather
> Args: {"location": "Beijing"}
> ```

Beyond a single flat call, the parser also supports:

* **Parallel calls** — multiple `<invoke>` blocks inside the single `<tool_call>` wrapper, surfaced as multiple `message.tool_calls` entries.
* **Nested object arguments** — an `object`-typed parameter is emitted as nested XML tags and reconstructed into a JSON object.
* **Array arguments** — an `array`-typed parameter uses repeated `<item>` children and is reconstructed into a JSON list.

For example, a tool with object and array parameters round-trips cleanly:

```text Output theme={null}
create_event {"title": "Design sync", "attendees": ["alice", "bob"], "location": {"room": "R2", "floor": 3}}
```

To return a tool result, append the assistant's `tool_calls` turn plus a matching `tool` message and ask the model to continue — the follow-up answer may place text in `reasoning_content` as well as `content`, so print both.

### 3.3 Multimodal (Vision) Input

Images go through the standard OpenAI `image_url` content type. The vision tower is always loaded; for image serving add `--mm-attention-backend flashinfer_cudnn` (the vision-tower backend) to the Blackwell deployment recipe — the text `--attention-backend` is unchanged (§2.1 note). On AMD, omit `--mm-attention-backend` and let the encoder use the ROCm default (vision is unvalidated on ROCm — §2.3).

> **Vision Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3-MXFP8",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {
>                     "type": "image_url",
>                     "image_url": {
>                         "url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
>                     },
>                 },
>                 {"type": "text", "text": "Describe this image in detail."},
>             ],
>         }
>     ],
>     max_tokens=1024,
> )
> print(response.choices[0].message.content)
> ```
>
> **Output Example:**
>
> ```text Output theme={null}
> This image captures a striking and unusual urban scene on what appears to be a busy New York City street.
>
> **Main Subject:**
> A man stands on the rear bumper of a yellow taxi cab (an SUV-style cab, likely a Ford Escape hybrid), operating a full-sized ironing board set up across the back of the vehicle. He is wearing a bright yellow long-sleeved shirt and dark pants, and is actively ironing a blue garment, holding an iron in his right hand.
>
> **Vehicles:**
> - The yellow SUV taxi on the right is stationary, its rear hatch serving as the ironing platform.
> - A second yellow taxi (a sedan) drives past on the left, captured with motion blur.
>
> **Setting:**
> Tall city buildings with classic urban architecture, an American flag, and white lane markings — a bustling downtown area, possibly Midtown Manhattan.
> ```

Notes:

* If the server cannot fetch external URLs, embed the image as a base64 `data:image/png;base64,...` URI — SGLang decodes it server-side.
* Multiple images per message are supported; add more `image_url` entries to the `content` list.
* Reasoning and tool calling work the same way for multimodal requests — a vision prompt can still produce a `<mm:think>` trace and/or tool calls.

### 3.4 Prefill-Decode (PD) Disaggregation

[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. M3 needs one thing beyond a dense model: alongside the main KV cache, every sparse "lightning-indexer" layer keeps a **K-only index buffer**, and that buffer must reach the decode server too — otherwise sparse attention reads stale state. SGLang transfers it alongside the main KV — reusing the same page mapping — so M3 disaggregates correctly with no extra flags.

**Supported topology** (the released MiniMax-M3, whose sparse layers are all K-only):

* **Equal tensor parallelism** — the prefill and decode servers run the same `--tp`.
* **Single pipeline stage** — PP = 1 (the default).
* **mooncake or NIXL** transfer backend over RDMA / InfiniBand.

Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Pick your hardware:

**Blackwell · MXFP8**

On Blackwell the MXFP8 recipe — fa4, page size 128, deep\_gemm MoE, and the MSA fast path (§2.1) — is auto-selected, so each role adds only the `--disaggregation-*` flags. This is the validated **2 × 4×B200** setup (TP4 prefill on node A, TP4 decode on node B); point `--disaggregation-ib-device` at your RDMA NIC(s).

```bash Prefill server (node A) theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-M3-MXFP8 \
  --trust-remote-code \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --tp 4 \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend nixl \
  --disaggregation-ib-device mlx5_0 \
  --host 0.0.0.0 --port 30000 \
  --disaggregation-bootstrap-port 8998
```

```bash Decode server (node B) theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-M3-MXFP8 \
  --trust-remote-code \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --tp 4 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend nixl \
  --disaggregation-ib-device mlx5_0 \
  --host 0.0.0.0 --port 30001
```

**Hopper · bf16**

On Hopper (H200) M3 runs the bf16 build (§2.4) with Triton MoE and the built-in Triton sparse path, pinned to `--page-size 128` so both roles share the page layout the sparse-index transfer relies on. This is the validated **2 × 8×H200** setup (TP8 each).

```bash Prefill server (node A) theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-M3 \
  --trust-remote-code \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --tp 8 \
  --attention-backend triton \
  --moe-runner-backend triton \
  --page-size 128 \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake \
  --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  --host 0.0.0.0 --port 30000 \
  --disaggregation-bootstrap-port 8998
```

```bash Decode server (node B) theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-M3 \
  --trust-remote-code \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --tp 8 \
  --attention-backend triton \
  --moe-runner-backend triton \
  --page-size 128 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mooncake \
  --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  --host 0.0.0.0 --port 30001
```

Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint:

```bash PD router theme={null}
python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://<prefill-host>:30000 8998 \
  --decode http://<decode-host>:30001 \
  --policy round_robin \
  --host 0.0.0.0 --port 8000
```

Clients hit the router exactly like a single server — it splits each request across the two stages transparently:

> **PD Client Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://<router-host>:8000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3-MXFP8",
>     messages=[{"role": "user", "content": "What is 2 + 2?"}],
>     max_tokens=64,
> )
> print(response.choices[0].message.content)
> ```
>
> **Output Example:**
>
> ```text Output theme={null}
> 2 + 2 = 4
> ```

**Validation.** PD disaggregation preserves output quality — the K-only sparse index transfers arrive intact and disaggregated output matches non-disaggregated serving. GSM8K is scored with the single sgl-eval harness used by the benchmark card above (full 1319-question split, chat with `--thinking`); see that card for per-platform single-node accuracy.

* **2 × 4×B200** (TP4+TP4, MXFP8, NIXL over InfiniBand) — output matches single-node serving. The 2-node PD serving benchmark (512-token input, 256-token output, 16 concurrent — a different workload from the card's single-node `random` isl=2048 / osl=256 / conc=64 row, so the throughput figures are not directly comparable) measured mean TTFT 1.1 s and TPOT 16.6 ms (≈ 60 tok/s per stream, ≈ 2.3k tok/s aggregate).
* **2 × 8×H200** (TP8+TP8, bf16, mooncake) — output matches single-node serving.


# MiniMax-M3.1
Source: https://docs.sglang.io/cookbook/autoregressive/MiniMax/MiniMax-M3.1

Deploy MiniMax-M3.1-Flash with SGLang — the static-NVFP4 build of MiniMax's sparse-attention Mixture-of-Experts reasoning model, with its bundled DSpark draft or RadixArk's DFlash2 draft for speculative decoding, on NVIDIA and AMD.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> MiniMax-M3.1 support has not yet been merged into SGLang `main`. Install from the support PR below. The PR number is **TBD**; replace `TBD` with the actual PR number before running these commands.
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
>
> PR_NUMBER=<the PR number of the model support PR>
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> git fetch origin "pull/${PR_NUMBER}/head"
> git checkout FETCH_HEAD
> uv pip install --prerelease=allow -e python
> cd ..
> ```
>
> On NVIDIA Blackwell (B300 / GB300 / B200), also install the MSA sparse-attention kernels at the tested commit. SGLang imports them at startup, and the server fails without them. Hopper and AMD do not use them. Some of these kernels are compiled at runtime and need a **CUDA toolkit (`nvcc`) >= 13.4** on the machine. The CUDA version of the PyTorch wheel (`cu130`) does not count. See [Blackwell prerequisites](#blackwell-prerequisites).
>
> ```bash Command theme={null}
> git clone https://github.com/MiniMax-AI/MSA.git
> cd MSA
> git checkout c8e1013a16a708d2a7656805d5fc26f2b87715d1
> uv pip install --no-deps -e .
> cd ..
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-minimax-m31-flash
> # AMD MI350X / MI355X (ROCm 7.2):
> docker pull lmsysorg/sglang:dev-minimax-m31-flash-mi35x
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + recipe to generate the launch command. The serving strategies cover the common operating points:

* **Low-Latency** — fastest responses with a few users at once. On B300 it serves one request at a time through the MSA-V1 MonoKernel; on AMD it is a TP4 recipe for up to about 32 concurrent requests.
* **Balanced** — good speed with several users at once. Use for typical multi-user serving.
* **High-Throughput** — most tokens per second across many users. Best for batch jobs and many concurrent agent sessions.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

**Resources:** [HuggingFace](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash), [DFlash2 draft](https://huggingface.co/RadixArk/Minimax-M3.1-Flash-Dflash2).

## 2. Configuration Tips

### NVIDIA Blackwell (B300 / GB300 / B200)

* **`FMHA_SM100_ALLOW_JIT=1`** lets the MSA sparse-attention kernels compile on first launch. They are cached under `~/.cache/minfer/msa_v1` (override with `MSA_V1_AOT_CACHE`) and reused afterwards. Without the variable and without a cached build, the server fails at startup with `JIT compilation is disabled and AOT kernel ... was not found`. Only the variable's presence is checked, so `=0` does not turn JIT off. An image that ships a prebuilt cache can drop it.
* **CUDA toolkit >= 13.4.** The decode path dequantizes the NVFP4 KV cache with the `qmul4` kernel, which needs `nvcc` 13.4 or newer. The toolkit is taken from `CUDA_HOME`, then `CUDACXX`, then the `nvcc` on `PATH` or `/usr/local/cuda`. An older toolkit does not fail: the server silently falls back to a slower path with an extra FP8 scratch buffer. Check that the startup log prints `nvfp4_to_fp8 backend=qmul4`, and point `CUDA_HOME` at a 13.4 toolkit (for example `/usr/local/cuda-13.4`) if it does not.
* **The first launch is slow** while these kernels and the FlashInfer kernels compile. Later launches on the same machine reuse the caches.
* **DSpark draft.** The checkpoint ships the DSpark draft in its own `dspark/` subfolder. With `--speculative-algorithm DSPARK` and no `--speculative-draft-model-path`, the server finds that subfolder and loads the draft from it, for a Hub repo id as well as a local directory, so the recipes need no draft flag. Pass `--speculative-draft-model-path` only to use a different draft.
* **DFlash2 draft.** Pick **DFlash2** under **Speculative Decoding** in the command panel to use RadixArk's trained draft instead. It is a separate checkpoint, so the command passes `--speculative-algorithm DFLASH --speculative-draft-model-path RadixArk/Minimax-M3.1-Flash-Dflash2`. It also adds `--speculative-use-block-verification`, which accepts sampled draft proposals by their joint probability instead of token by token. For a greedy draft the two are the same, and the server logs that the flag has no effect. DFlash2 runs on Blackwell only, without DP attention (so not with the B300 / B200 High-Throughput recipes, which use `--attn-dp-size 2`), and only in colocated serving.

### MonoKernel (Low-Latency)

The B300 **Low-Latency** recipe serves batch-size-1 decode through the MSA-V1 MonoKernel plugin with `--monokernel-backend msa`. The plugin supports Blackwell sm\_100 and sm\_103 (B200 / GB200 and B300 / GB300) at `--tp 4` only; it has been measured on B300. Each MoE layer of the DSpark verify step runs as two persistent kernels, MegaAttention then the token-parallel MoE, each fused with its TP4 all-reduce. The DSpark draft step runs as one cooperative kernel per rank. On 4x B300 with the acceptance length fixed at 5, decode reaches 1028 tokens/s at 1K context, 919 at 256K and 723 at 1M, against 479, 465 and 408 on the standard path.

* **Batch size 1 only.** Prefill, plain decode and batches of more than one request run on the standard path, so throughput under concurrency is lower than the Balanced recipe. The recipe caps `--max-running-requests` at 8.
* **Required flags.** The plugin needs `--page-size 128`, a DSpark chain of 8 draft tokens (`--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 8 --speculative-dspark-block-size 7`), and `--disable-attn-tp-gather`; without the last one SGLang pads the 7-row draft block and the server refuses to start. At startup the server lists every unsupported setting it finds, and the log confirms the plugin with `--monokernel-backend msa: minimax_m3_msa_v1 on sm_103, widths [8]`.
* **`ALLOW_JIT=1`** lets the MonoKernel kernels compile on first launch; set `CACHE_DIR` to keep the compiled kernels across restarts.
* **`SGLANG_FORCE_COARSE_WAR_BARRIER=1`** runs MegaAttention in canonical cluster placement, about 3–4% faster end to end. Canonical placement faults if another kernel holds SMs when MegaAttention launches, so the server must have the GPUs to itself. Without the variable, the server keeps cluster placement on, which is safe and slightly slower. The log prints `cluster placement off (canonical)` or `cluster placement on`.
* **Memory.** The plugin keeps its own copy of the routed experts (54.1 GiB per rank), so at `--mem-fraction-static 0.8` the KV pool holds about 6.5M tokens instead of 10.4M.
* DFlash2 is not available with this recipe.

### Memory and workload tuning

* **`--mem-fraction-static 0.8`** is the validated value at `--tp 4` on B300, and also on B200. The pool it reserves holds the NVFP4 main KV cache, the sparse-attention index cache and the bf16 DSpark draft KV cache. B200 has 192 GB per GPU against B300's 288 GB. Its high-throughput recipe (`--tp 8 --attn-dp-size 2`) therefore uses `--mem-fraction-static 0.75` and `--cuda-graph-max-bs-prefill 16384`. With B300's settings, 8x B200 runs out of GPU memory capturing the 32k-token prefill CUDA graphs, and then while preprocessing images on GPU 0.
* **`--tokenizer-worker-num 8`.** With 100K-token prompts a single tokenizer process becomes the bottleneck: tokenization blocks the server's event loop and TTFT climbs under load even when the GPUs have headroom. Eight workers remove that ceiling, so every recipe above sets it.
* **Hierarchical KV cache for multi-turn traffic.** `--enable-hierarchical-cache --hicache-size <GB per rank>` keeps evicted prefixes in host memory so a returning session reloads its prefix instead of recomputing it. Adding `--hicache-storage-backend mooncake` puts a Mooncake storage tier (L3) under it, which a set of replicas can share; pick **Mooncake** under **Storage** in the HiCache card of the [Playground](#playground). The Mooncake tier is validated on GB300 and B300, with DSpark and with DFlash speculative decoding. Other storage backends have not been validated. For the Mooncake master and environment setup, see [Deployment with Mooncake](../../../docs/advanced_features/hicache_best_practices#deployment-with-mooncake). When several replicas serve one endpoint, put the router in front with `--policy cache_aware` so a session's later turns reach the replica that holds its prefix.
* **`--max-running-requests` and `--cuda-graph-max-bs-decode`** are both 128 in the recipes. Keep them equal so every decode batch the scheduler can form replays a captured CUDA graph.

### AMD MI350X / MI355X

* Requires ROCm 7.2 (Triton 3.7); earlier ROCm images are not supported.
* **NVFP4 on MI355X:** the same `MiniMaxAI/MiniMax-M3.1-Flash` checkpoint as on NVIDIA; the routed experts are requantized to MXFP4 while loading.
* **MXFP4 on MI350X / MI355X:** `amd/MiniMax-M3.1-Flash-Quark-MXFP4` (Quark-quantized), same command as NVFP4 Low-Latency on both; select **MI350X / MXFP4** or **MI355X / MXFP4** in the panel. Only the Low-Latency recipe has been run with this checkpoint.
* **One server per four GPUs.** M3.1 NVFP4 on ROCm runs at `--tp 4` or less, and the server refuses a larger TP. On an 8-GPU node, start two servers with `HIP_VISIBLE_DEVICES=0,1,2,3` and `HIP_VISIBLE_DEVICES=4,5,6,7` on different ports, and put the router in front with `--policy cache_aware`.
* **Pick the recipe by concurrency.** Low-Latency is for up to about 32 concurrent requests. High-Throughput is for 48 or more concurrent long multi-turn sessions, such as coding agents, and adds two things to Low-Latency:
  * **Hierarchical KV cache** (`--enable-hierarchical-cache`). At the default ratio of 2, the host pools (main KV, index K and draft KV) take about 350 GB per rank, so one TP4 server needs about 1.4 TB of host memory. On a host with less free memory, the server lowers the ratio until the pools fit in 80% of it. When two servers share a host, lower `--hicache-host-memory-fraction` on each.
  * **FP8 DSpark draft KV cache** (`--speculative-draft-kv-cache-dtype fp8_e4m3`). The same memory holds 9% more tokens. The draft cache only affects which draft tokens get accepted; the target model still verifies every token.
* **`--prefill-decode-interval 16`** runs 16 decode steps after each prefill batch before the next prefill. With 64 concurrent sessions and HiCache on MI350X, it lowered P90 inter-token latency by about 10% at the cost of about 3% throughput.
* The High-Throughput recipe was measured on MI350X. MI355X runs the same command but has not been measured with it.

## 3. Advanced Usage

### 3.1 Reasoning

Enable the `minimax-m3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer. The `<mm:think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content`.

The reasoning traces in the example outputs below are abridged; `…` marks omitted text.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3.1-Flash",
>     messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
>     max_tokens=4096,
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(message.reasoning_content)
> print("=============== Answer ==================")
> print(message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> =============== Reasoning ===============
> … Calculate 15%=0.15*240 = 36. Brief explanation likely 10%=24, 5%=12 total 36 or 0.15×240. …
> =============== Answer ==================
> 15% of 240 is **36**.
>
> Calculation: \(0.15 \times 240 = 36\).
> ```

When streaming, the trace arrives on `delta.reasoning_content` and the answer on `delta.content`, so the two can be rendered separately as they arrive:

> **Streaming Reasoning (Python)**
>
> ```python Example theme={null}
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3.1-Flash",
>     messages=[{"role": "user", "content": "Solve step by step: what is 15% of 240?"}],
>     max_tokens=4096,
>     stream=True,
> )
>
> for chunk in response:
>     if not chunk.choices:
>         continue
>     delta = chunk.choices[0].delta
>     if getattr(delta, "reasoning_content", None):
>         print(delta.reasoning_content, end="", flush=True)  # thinking stream
>     if delta.content:
>         print(delta.content, end="", flush=True)            # answer stream
> print()
> ```

> **Example Output**
>
> ```text Output theme={null}
> [delta.reasoning_content — thinking stream]
> … Can give 15%=15/100=.15, .15×240=(15×240)/100=3600/100=36. …
>
> [delta.content — answer stream]
> 1. Convert the percentage to a decimal: \(15\% = 0.15\).
> 2. Multiply by 240: \(0.15 \times 240 = 36\).
>
> **Answer: 36**
> ```

### 3.2 Thinking Mode and Reasoning Effort

M3.1 takes two request-level controls. Pass them as top-level fields of the Chat Completions body, through `extra_body` with the OpenAI client:

* **`thinking`** — `{"type": "enabled" | "disabled" | "adaptive"}`. `enabled` always thinks before answering, `disabled` answers directly, and `adaptive` lets the model decide per turn. Omitting it means `adaptive`. Any other `type` is rejected with HTTP 400.
* **`reasoning_effort`** — `"low"`, `"medium"`, `"high"`, `"xhigh"` or `"max"`. The value is passed to the chat template, which tells the model how much to reason. `"none"` and `"minimal"` turn thinking off, the same as `thinking={"type": "disabled"}`. The effort is ignored when thinking is disabled.

> **Thinking Mode and Effort Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> question = [{"role": "user", "content": "How many prime numbers are there below 50?"}]
>
> for label, extra in [
>     ("thinking disabled", {"thinking": {"type": "disabled"}}),
>     ("adaptive, effort low", {"thinking": {"type": "adaptive"}, "reasoning_effort": "low"}),
>     ("enabled, effort max", {"thinking": {"type": "enabled"}, "reasoning_effort": "max"}),
> ]:
>     response = client.chat.completions.create(
>         model="MiniMaxAI/MiniMax-M3.1-Flash",
>         messages=question,
>         max_tokens=8192,
>         extra_body=extra,
>     )
>     message = response.choices[0].message
>     reasoning_chars = len(message.reasoning_content or "")
>     print(f"[{label}] reasoning chars: {reasoning_chars}")
>     print(f"  answer: {message.content.strip()}")
> ```

> **Example Output**
>
> ```text Output theme={null}
> [thinking disabled] reasoning chars: 0
>   answer: There are **15** prime numbers below 50:
>
> 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, and 47.
> [adaptive, effort low] reasoning chars: 8
>   answer: There are **15** prime numbers below 50:
>
> 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47.
> [enabled, effort max] reasoning chars: 268
>   answer: There are **15 prime numbers** below 50:
>
> 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, and 47.
> ```

### 3.3 Tool Calling

Enable the `minimax-m3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3.1-Flash",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
>
> message = response.choices[0].message
> if message.tool_calls:
>     for call in message.tool_calls:
>         print(f"Tool: {call.function.name}")
>         print(f"Args: {call.function.arguments}")
> ```

> **Example Output**
>
> ```text Output theme={null}
> Tool: get_weather
> Args: {"location": "Beijing", "unit": "celsius"}
> ```

To return a tool result, append the assistant turn with its `tool_calls` and one `tool` message per call (matching `tool_call_id`), then ask the model to continue.

### 3.4 Multimodal (Vision) Input

Images go through the standard OpenAI `image_url` content type.

> **Vision Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="MiniMaxAI/MiniMax-M3.1-Flash",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {
>                     "type": "image_url",
>                     "image_url": {
>                         "url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
>                     },
>                 },
>                 {"type": "text", "text": "Describe this image in two sentences."},
>             ],
>         }
>     ],
>     max_tokens=4096,
> )
> print(response.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> A man in a yellow shirt stands on the rear bumper of a yellow taxi, pressing clothes on an ironing board extending from its open hatch. Another yellow taxi speeds past on the city street lined with tall buildings.
> ```

### 3.5 Prefill-Decode (PD) Disaggregation

[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers, linked by an RDMA KV-transfer fabric and fronted by the PD router. Each M3.1 layer stores three things the decode server needs: the packed NVFP4 main KV cache with its block scales, an NVFP4 K-only sparse-index cache, and, with DSpark, the draft model's KV cache. SGLang sends all three from prefill to decode over the same page mapping, so PD needs no M3.1-specific flags.

**Supported topology**

* **Same attention parallelism on both roles.** Prefill and decode must use the same `--tp` and `--attn-dp-size`. The sparse-index transfer cannot reshard across different attention TP sizes.
* **Single pipeline stage.** PP must be 1, which DSpark also requires.
* **Speculative decoding.** Run DSpark on both roles. DFlash2 is only supported in colocated serving.
* **Transfer backend.** Mooncake over RDMA.
* **KV caching.** Run HiCache on the prefill server. The decode server keeps its default chunk cache. `--disaggregation-decode-enable-radix-cache` lets decode reuse prefixes it already holds. It is experimental and off by default.
* **HiCache on the decode server needs the decode radix cache.** A decode server turns its radix cache off unless `--disaggregation-decode-enable-radix-cache` is set, and `--enable-hierarchical-cache` with the radix cache off fails at startup (`enable-hierarchical-cache and disable-radix-cache are mutually exclusive`). Pass both flags on decode. The decode radix cache works with DSpark or without speculative decoding. With DP attention, route each session to the same DP rank so it finds its prefix. The playground adds the flag when you pick the decode role with HiCache on.

**Recommended layout: DP8 / TP1 / EP8 on each role.** Each role takes one 8×B300 node:

* Attention is data-parallel over eight ranks, each with attention TP 1 (`--tp 8 --attn-dp-size 8`).
* The experts are split eight ways through MegaMoE (`--ep 8`).
* The other flags follow the B300 High-Throughput recipe.

On B300, picking **Prefill role** or **Decode role** in the [Playground](#playground) PD card applies this layout to either recipe. Start the prefill server on node A, the decode server on node B, and then the router. Mooncake picks the RDMA NICs automatically; add `--disaggregation-ib-device` to pin specific ones.

```bash Prefill server (node A) theme={null}
FMHA_SM100_ALLOW_JIT=1 \
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=32768 \
sglang serve \
  --model-path MiniMaxAI/MiniMax-M3.1-Flash \
  --trust-remote-code \
  --reasoning-parser minimax-m3 \
  --tool-call-parser minimax-m3 \
  --tp 8 \
  --attn-dp-size 8 \
  --enable-dp-lm-head \
  --ep 8 \
  --speculative-algorithm DSPARK \
  --cuda-graph-backend-prefill breakable \
  --chunked-prefill-size 65536 \
  --max-running-requests 256 \
  --cuda-graph-max-bs-decode 128 \
  --mem-fraction-static 0.8 \
  --tokenizer-worker-num 8 \
  --enable-hierarchical-cache \
  --hicache-size 192 \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake \
  --disaggregation-bootstrap-port 8998 \
  --host 0.0.0.0 --port 30000
```

```bash Decode server (node B) theme={null}
FMHA_SM100_ALLOW_JIT=1 \
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=32768 \
sglang serve \
  --model-path MiniMaxAI/MiniMax-M3.1-Flash \
  --trust-remote-code \
  --reasoning-parser minimax-m3 \
  --tool-call-parser minimax-m3 \
  --tp 8 \
  --attn-dp-size 8 \
  --enable-dp-lm-head \
  --ep 8 \
  --speculative-algorithm DSPARK \
  --cuda-graph-backend-prefill breakable \
  --chunked-prefill-size 65536 \
  --max-running-requests 256 \
  --cuda-graph-max-bs-decode 128 \
  --mem-fraction-static 0.8 \
  --tokenizer-worker-num 8 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mooncake \
  --host 0.0.0.0 --port 30100
```

```bash Router theme={null}
python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://<node-A>:30000 8998 \
  --decode http://<node-B>:30100 \
  --policy round_robin \
  --host 0.0.0.0 --port 8000
```

The `8998` after the prefill URL is the prefill bootstrap port. It must match the prefill server's `--disaggregation-bootstrap-port`.

* **HiCache host memory.** `--hicache-size` is per rank, so at DP8 the prefill node reserves 8 × 192 GB of host memory for the host pools. Lower it on nodes with less memory.
* **MegaMoE token budget.** `SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK` must cover each rank's share of a prefill chunk (65536 / 8 here) and its largest decode or verify batch.
* **Media options through the router.** The PD router forwards only the standard OpenAI request fields. Per-request media options such as `max_long_side_pixel` and video `fps` do not reach the servers; this was observed with sglang-router 0.3.2.

> [!NOTE]
>
> Correctness was validated on 1P1D with TP4 on each role (8×B300, Mooncake, DSpark on both roles, HiCache on prefill). GSM8K matched colocated serving, and `--disaggregation-enable-kv-checksum` reported no mismatch. The DP8 / TP1 / EP8 layout has not been benchmarked yet.


# Devstral 2 (Mistral)
Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Devstral-2



## 1. Model Introduction

**Devstral 2** is an agentic LLM family for software engineering tasks. It is designed for agentic workflows such as tool use, codebase exploration, and multi-file edits, and achieves strong performance on **SWE-bench**.

The **Devstral 2 Instruct** checkpoints are instruction-tuned **FP8** models, making them a good fit for chat, tool-using agents, and instruction-following SWE workloads.

**Key Features:**

* **Agentic coding**: Optimized for tool-driven coding and software engineering agents
* **Improved performance**: A step up compared to earlier Devstral models
* **Better generalization**: More robust across diverse prompts and coding environments
* **Long context**: Up to a **256K** context window

**Use Cases:**
AI code assistants, agentic coding, and software engineering tasks that require deep codebase understanding and tool integration.

For enterprises requiring specialized capabilities (increased context, domain-specific knowledge, etc.), please reach out to Mistral.

**Models:**

* **Collection**: [mistralai/devstral-2 (Hugging Face)](https://huggingface.co/collections/mistralai/devstral-2)
* **FP8 Instruct**:
  * **[mistralai/Devstral-2-123B-Instruct-2512](https://huggingface.co/mistralai/Devstral-2-123B-Instruct-2512)**
  * **[mistralai/Devstral-Small-2-24B-Instruct-2512](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512)**

***

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

> [!WARNING]
>
> Devstral 2 requires a recent `transformers`. Please verify `transformers >= 5.0.0.rc`:
>
> ```shell Command theme={null}
> python -c "import transformers; print(transformers.__version__)"
> ```
>
> If your version is lower, upgrade:
>
> ```shell Command theme={null}
> pip install -U --pre "transformers>=5.0.0rc0"
> ```

***

## 3. Model Deployment

### 3.1 Basic configuration

**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Devstral Small 2 (24B) or Devstral 2 (123B).

> [!NOTE]
>
> The TP size is set to the minimum required for the selected model size.

Run this Command:

<pre />

### 3.2 Configuration tips

* **Context length vs memory**: Devstral 2 advertises a long context window; if you are memory-constrained, start by lowering `--context-length` (for example `32768`) and increase once things are stable.
* **FP8 checkpoints**: Both Devstral Small 2 and Devstral 2 are published as **FP8** weights. If you hit kernel / dtype issues, try a newer SGLang build and recent CUDA drivers.

***

## 4. Model Invocation

### 4.1 Basic Usage (OpenAI-Compatible API)

SGLang exposes an OpenAI-compatible endpoint. Example:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="mistralai/Devstral-Small-2-24B-Instruct-2512",
    messages=[
        {"role": "system", "content": "You are a helpful coding assistant."},
        {"role": "user", "content": "Write a Python function that retries a request with exponential backoff."},
    ],
    temperature=0.2,
    max_tokens=512,
)

print(resp.choices[0].message.content)
```

**Output Example:**

````text Output theme={null}
  Here's a Python function that implements exponential backoff for retrying a request. This function uses the `requests` library to make HTTP requests and includes error handling for common HTTP and connection errors.

  ```python
  import time
  import requests
  from requests.exceptions import RequestException

  def retry_with_exponential_backoff(
      url,
      max_retries=3,
      initial_delay=1,
      backoff_factor=2,
      method="GET",
      **kwargs
  ):
      """
      Retry a request with exponential backoff.

      Parameters:
      - url: The URL to request.
      - max_retries: Maximum number of retry attempts (default: 3).
      - initial_delay: Initial delay in seconds (default: 1).
      - backoff_factor: Multiplier for the delay between retries (default: 2).
      - method: HTTP method to use (default: "GET").
      - **kwargs: Additional arguments to pass to the request function (e.g., headers, data, etc.).

      Returns:
      - Response object if the request succeeds.
      - Raises an exception if all retries fail.
      """
      retry_count = 0
      delay = initial_delay

      while retry_count < max_retries:
          try:
              response = requests.request(method, url, **kwargs)
              # Check if the response status code indicates success
              if response.status_code < 400:
                  return response
              else:
                  raise RequestException(f"HTTP {response.status_code}: {response.text}")

          except RequestException as e:
              if retry_count == max_retries - 1:
                  raise Exception(f"All retries failed. Last error: {e}")

              print(f"Attempt {retry_count + 1} failed. Retrying in {delay} seconds...")
              time.sleep(delay)
...
````

### 4.2 Tool calling (optional)

Devstral 2 supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model mistralai/Devstral-2-123B-Instruct-2512 \
  --tp 2 \
  --tool-call-parser mistral
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="mistralai/Devstral-2-123B-Instruct-2512",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing"}
```

## AMD GPU Support

## 1. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 1.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 1.2 Advanced Usage

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path mistralai/Devstral-2-123B-Instruct-2512 \
  --tp 8 \
  --trust-remote-code \
  --port 8888
```

## 2.Benchmark

### 5.1 Benchmark Commands

**Scenario 1: Chat (1K/1K) - Most Important**

* **Model Deployment**

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path mistralai/Devstral-2-123B-Instruct-2512 \
  --tp 8 \
  --trust-remote-code \
  --port 8888
```

* Low Concurrency (Latency-Optimized)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Devstral-2-123B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf \
  --port 8888
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  94.30
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4206
Request throughput (req/s):              0.11
Input token throughput (tok/s):          64.70
Output token throughput (tok/s):         44.75
Peak output token throughput (tok/s):    82.00
Peak concurrent requests:                2
Total token throughput (tok/s):          109.44
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   9427.59
Median E2E Latency (ms):                 5637.23
---------------Time to First Token----------------
Mean TTFT (ms):                          4253.85
Median TTFT (ms):                        116.95
P99 TTFT (ms):                           37764.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          12.28
Median TPOT (ms):                        12.29
P99 TPOT (ms):                           12.30
---------------Inter-Token Latency----------------
Mean ITL (ms):                           12.29
Median ITL (ms):                         12.29
P95 ITL (ms):                            12.38
P99 ITL (ms):                            12.42
Max ITL (ms):                            12.90
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Devstral-2-123B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf \
  --port 8888
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  52.11
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40761
Request throughput (req/s):              1.54
Input token throughput (tok/s):          761.31
Output token throughput (tok/s):         783.13
Peak output token throughput (tok/s):    1120.00
Peak concurrent requests:                20
Total token throughput (tok/s):          1544.44
Concurrency:                             13.60
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8856.19
Median E2E Latency (ms):                 9314.71
---------------Time to First Token----------------
Mean TTFT (ms):                          398.80
Median TTFT (ms):                        127.81
P99 TTFT (ms):                           1500.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.32
Median TPOT (ms):                        16.90
P99 TPOT (ms):                           32.78
---------------Inter-Token Latency----------------
Mean ITL (ms):                           16.61
Median ITL (ms):                         14.26
P95 ITL (ms):                            15.07
P99 ITL (ms):                            114.46
Max ITL (ms):                            1224.45
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Devstral-2-123B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf \
  --port 8888
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  116.08
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    252523
Request throughput (req/s):              4.31
Input token throughput (tok/s):          2152.21
Output token throughput (tok/s):         2176.60
Peak output token throughput (tok/s):    3600.00
Peak concurrent requests:                107
Total token throughput (tok/s):          4328.81
Concurrency:                             92.42
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   21456.71
Median E2E Latency (ms):                 20126.82
---------------Time to First Token----------------
Mean TTFT (ms):                          291.60
Median TTFT (ms):                        199.24
P99 TTFT (ms):                           866.02
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          42.42
Median TPOT (ms):                        45.18
P99 TPOT (ms):                           53.32
---------------Inter-Token Latency----------------
Mean ITL (ms):                           41.97
Median ITL (ms):                         27.59
P95 ITL (ms):                            130.43
P99 ITL (ms):                            137.87
Max ITL (ms):                            616.73
==================================================
```

#### 5.2 Understanding the Results

**Key Metrics:**

* **Request Throughput (req/s)**: Number of requests processed per second
* **Output Token Throughput (tok/s)**: Total tokens generated per second
* **Mean TTFT (ms)**: Time to First Token - measures responsiveness
* **Mean TPOT (ms)**: Time Per Output Token - measures generation speed
* **Mean ITL (ms)**: Inter-Token Latency - measures streaming consistency

**Why These Configurations Matter:**

* **1K/1K (Chat)**: Represents the most common conversational AI workload. This is the highest priority scenario for most deployments.
* **1K/8K (Reasoning)**: Tests long-form generation capabilities crucial for complex reasoning, code generation, and detailed explanations.
* **8K/1K (Summarization)**: Evaluates performance with large context inputs, essential for RAG systems, document Q\&A, and summarization tasks.
* **Variable Concurrency**: Captures the Pareto frontier - the optimal trade-off between throughput and latency at different load levels. Low concurrency shows best-case latency, high concurrency shows maximum throughput.

**Interpreting Results:**

* Compare your results against baseline numbers for your hardware
* Higher throughput at same latency = better performance
* Lower TTFT = more responsive user experience
* Lower TPOT = faster generation speed

### 5.3 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.3.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:8888/v1 \
  --num-examples 1319 \
  --num-threads 1316
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.922
Invalid: 0.000
Latency: 35.800 s
Output throughput: 4507.697 token/s
```


# Ministral-3
Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Ministral-3



## 1. Model Introduction

The largest model in the Ministral 3 family, Ministral 3 14B offers frontier capabilities and performance comparable to its larger Mistral Small 3.2 24B counterpart. A powerful and efficient language model with vision capabilities.

The Ministral 3 14B Instruct model offers the following capabilities:

Vision: Enables the model to analyze images and provide insights based on visual content, in addition to text.
Multilingual: Supports dozens of languages, including English, French, Spanish, German, Italian, Portuguese, Dutch, Chinese, Japanese, Korean, Arabic.
System Prompt: Maintains strong adherence and support for system prompts.
Agentic: Offers best-in-class agentic capabilities with native function calling and JSON outputting.
Edge-Optimized: Delivers best-in-class performance at a small scale, deployable anywhere.
Apache 2.0 License: Open-source license allowing usage and modification for both commercial and non-commercial purposes.
Large Context Window: Supports a 256k context window.

For further details, please refer to the [official documentation](https://github.com/mistralai)

## 2. SGLang Installation

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

**Context length vs memory**: Ministral-3 advertises a long context window; if you are memory-constrained, start by lowering --context-length (for example 32768) and increase once things are stable.

**Pre-installation steps**: Adding the following steps after launching the docker

```shell Command theme={null}
pip install mistral-common --upgrade
pip install transformers==5.0.0.rc0
```

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Launch the docker

```shell Command theme={null}
docker pull lmsysorg/sglang:v0.5.9-rocm720-mi30x
```

```shell Command theme={null}
docker run -d -it --ipc=host --network=host --privileged \
  --cap-add=CAP_SYS_ADMIN \
  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
  --group-add video --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /:/work \
  -e SHELL=/bin/bash \
  --name Ministral \
 lmsysorg/sglang:v0.5.9-rocm720-mi30x \
  /bin/bash
```

#### 4.2.2 Launch the server

```shell Command theme={null}
sglang serve \
  --model-path mistralai/Ministral-3-14B-Instruct-2512 \
  --tp 1 \
  --trust-remote-code
```

## 5. Benchmark

This section uses **industry-standard configurations** for comparable benchmark results.

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: MI300X GPU (8x)

* Model: mistralai/Ministral-3-14B-Instruct-2512

* Tensor Parallelism: 1

* SGLang Version: 0.5.7

* Model Deployment Command:

```bash Command theme={null}
sglang serve \
  --model-path mistralai/Ministral-3-14B-Instruct-2512 \
  --tp 1 \
  --trust-remote-code
```

##### Low Concurrency

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Ministral-3-14B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  65.08
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4218
Request throughput (req/s):              0.15
Input token throughput (tok/s):          93.75
Output token throughput (tok/s):         64.84
Peak output token throughput (tok/s):    151.00
Peak concurrent requests:                2
Total token throughput (tok/s):          158.59
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6505.51
Median E2E Latency (ms):                 3037.37
---------------Time to First Token----------------
Mean TTFT (ms):                          3709.33
Median TTFT (ms):                        53.72
P99 TTFT (ms):                           33320.77
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.63
Median TPOT (ms):                        6.64
P99 TPOT (ms):                           6.66
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.64
Median ITL (ms):                         6.65
P95 ITL (ms):                            6.75
P99 ITL (ms):                            6.82
Max ITL (ms):                            8.45
==================================================
```

##### Medium Concurrency

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Ministral-3-14B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  31.20
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    40783
Request throughput (req/s):              2.56
Input token throughput (tok/s):          1271.38
Output token throughput (tok/s):         1307.82
Peak output token throughput (tok/s):    1760.00
Peak concurrent requests:                22
Total token throughput (tok/s):          2579.20
Concurrency:                             13.72
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5351.07
Median E2E Latency (ms):                 5626.45
---------------Time to First Token----------------
Mean TTFT (ms):                          280.87
Median TTFT (ms):                        68.16
P99 TTFT (ms):                           1194.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.47
Median TPOT (ms):                        10.10
P99 TPOT (ms):                           20.00
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.96
Median ITL (ms):                         9.10
P95 ITL (ms):                            9.87
P99 ITL (ms):                            51.39
Max ITL (ms):                            888.63
==================================================
```

##### High Concurrency

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model mistralai/Ministral-3-14B-Instruct-2512 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  88.75
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    252547
Request throughput (req/s):              5.63
Input token throughput (tok/s):          2815.01
Output token throughput (tok/s):         2846.91
Peak output token throughput (tok/s):    4271.00
Peak concurrent requests:                110
Total token throughput (tok/s):          5661.93
Concurrency:                             93.04
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16514.45
Median E2E Latency (ms):                 15834.45
---------------Time to First Token----------------
Mean TTFT (ms):                          148.57
Median TTFT (ms):                        99.15
P99 TTFT (ms):                           455.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.93
Median TPOT (ms):                        34.73
P99 TPOT (ms):                           38.05
---------------Inter-Token Latency----------------
Mean ITL (ms):                           32.45
Median ITL (ms):                         27.30
P95 ITL (ms):                            71.73
P99 ITL (ms):                            73.45
Max ITL (ms):                            328.10
==================================================
```

### 5.2 Accuracy Benchmark

Document model accuracy on standard benchmarks:

#### 5.2.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319 \
  --num-threads 1316
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.959
Invalid: 0.000
Latency: 29.185 s
Output throughput: 4854.672 token/s
```


# Mistral Medium 3.5
Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Mistral-Medium-3.5



## 1. Model Introduction

**Mistral Medium 3.5** is Mistral AI's first flagship **merged model** — a single dense 128B checkpoint that handles instruction following, reasoning, and coding in one set of weights. It replaces Mistral Medium 3.1 and Magistral in Le Chat, and replaces Devstral 2 in the Vibe coding agent. Reasoning effort is configurable per request, so the same model can answer a quick chat reply or work through a deep agentic run. The vision encoder was trained from scratch to handle variable image sizes and aspect ratios.

**Key Features:**

* **Dense 128B parameters** — no MoE, no MLA, plain GQA (96 heads, 8 KV heads, head\_dim=128)
* **256K context window** — YARN RoPE scaling on top of the original 4K base
* **Hybrid Reasoning**: Toggle between instant reply and deep reasoning per request via `reasoning_effort` (`"none"` or `"high"`)
* **Vision**: Accepts text + image input; from-scratch encoder that handles variable image sizes/aspect ratios
* **Function Calling**: Native tool calling and JSON output
* **FP8 Native**: Released with FP8 e4m3 static-tensor quantization built in
* **Multilingual**: 24 supported languages including English, French, German, Spanish, Portuguese, Italian, Japanese, Korean, Russian, Chinese, Arabic, Persian, Indonesian, Malay, Nepali, Polish, Romanian, Serbian, Swedish, Turkish, Ukrainian, Vietnamese, Hindi, and Bengali
* **License**: Modified MIT (open for commercial and non-commercial use except for companies with large revenue)

**Architecture:**

* Mistral 3 backbone with YARN RoPE for 256K context
* Dense (no MoE), 128B parameters
* Standard GQA attention (not MLA)
* Pixtral-style vision encoder (48 layers, patch\_size=14, spatial\_merge=2, image\_size=1540) trained from scratch
* Multimodal input: text + image

**Models:**

* **[mistralai/Mistral-Medium-3.5-128B](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B)** (FP8)

The HuggingFace repo ships both the mistral native layout (`params.json` + `consolidated-*.safetensors`) and the HF layout (`config.json` + `model-*.safetensors`). SGLang auto-detects the format — the HF layout is preferred when both are present.

***

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install).

**Docker Image:** `lmsysorg/sglang:latest` covers all the GPUs in this cookbook (H100 / H200 / B200 / B300).

***

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Medium 3.5.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Tensor Parallelism**: Mistral Medium 3.5 FP8 (\~130 GB) requires `--tp 4` on Hopper (H100/H200) and `--tp 2` on Blackwell (B200/B300).
* **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
* **Recommended temperature**: `0.7` when `reasoning_effort="high"`. Anywhere from `0.0` to `0.7` when `reasoning_effort="none"`, depending on the task — lower for to-the-point answers, higher for creative output.
* **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
* **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
* **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
* **System prompt**: The model ships with a recommended system prompt in `chat_template.jinja` and `SYSTEM_PROMPT.txt`. If you do not pass a system message yourself, the chat template injects Mistral's default (model identity, current date, tool-use guidelines). For full fidelity with Mistral's reference setup, load `SYSTEM_PROMPT.txt` from the HF repo and substitute `{name}`, `{today}`, `{yesterday}` (see Section 4.6).

### 3.3 Speculative Decoding (EAGLE)

Mistral ships an EAGLE draft head, [`mistralai/Mistral-Medium-3.5-128B-EAGLE`](https://huggingface.co/mistralai/Mistral-Medium-3.5-128B-EAGLE), that lets you run speculative decoding on top of the dense 128B target. The draft is a 2-layer GQA body sharing the target's vocab/head, FP8-quantized like the target (\~4 GB), and is meant for low-concurrency latency-bound serving.

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path mistralai/Mistral-Medium-3.5-128B \
  --tp 4 \
  --dtype bfloat16 \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-algorithm EAGLE \
  --speculative-draft-model-path mistralai/Mistral-Medium-3.5-128B-EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --port 30000
```

* **`--dtype bfloat16` is required.** The draft `params.json` does not carry a `dtype` field, so `--dtype auto` falls back to fp32 and downcasts to fp16, which conflicts with the bf16 target when the embed/head are shared. Setting bf16 explicitly keeps both sides aligned (this is a no-op for the target — it already loads as bf16).
* The draft uses the same vocab and lm\_head as the target. Memory overhead on top of the base model is \~4 GB per TP shard.
* `(num-steps, eagle-topk, num-draft-tokens) = (3, 1, 4)` is the recommended starting point. Tune for your workload — wider trees (higher `eagle-topk` / `num-draft-tokens`) help high-acceptance (templated) outputs, narrower trees keep latency tight on more diverse text.
* EAGLE shines at low concurrency. At high concurrency, throughput is dominated by the target's batched forward pass and the draft's contribution shrinks; consider running without EAGLE for batch-serving workloads.

***

## 4. Model Invocation

### 4.1 Thinking Mode

Mistral Medium 3.5 is a hybrid reasoning model. By default it does not produce a reasoning trace — pass `reasoning_effort="high"` to switch on the deep-reasoning path. Mistral recommends `temperature=0.7` for reasoning mode.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
    ],
    temperature=0.7,
    extra_body={"reasoning_effort": "high"},
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("Answer:", response.choices[0].message.content)
```

**Output:**

```text Output theme={null}
Reasoning: I need to follow the order of operations (PEMDAS/BODMAS): multiplication and
division before addition, evaluated left to right.

17 × 23: I'll break it as 17 × (20 + 3) = 340 + 51 = 391.
144 / 12 = 12.
Finally, 391 + 12 = 403.

Answer: **17 × 23 + 144 / 12 = 403**

Step by step:
1. 17 × 23 = 391
2. 144 / 12 = 12
3. 391 + 12 = 403
```

### 4.2 Instruct Mode (Reasoning Off)

To skip the reasoning trace and get a fast direct response, set `reasoning_effort="none"`. For instruct mode, Mistral recommends temperature in the `0.0`–`0.7` range depending on how creative the task is:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {"role": "user", "content": "What is the capital of France?"},
    ],
    temperature=0.1,
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)
```

**Output:**

```text Output theme={null}
The capital of France is **Paris**. It is one of the most famous and visited cities in
the world, known for its rich history, art, culture, and landmarks like the Eiffel Tower,
Louvre Museum, and Notre-Dame Cathedral.
```

### 4.3 Streaming with Reasoning

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {"role": "user", "content": "Explain the difference between async and threading in Python."},
    ],
    temperature=0.7,
    extra_body={"reasoning_effort": "high"},
    stream=True,
)

print("=== Reasoning ===")
for chunk in stream:
    delta = chunk.choices[0].delta
    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
        print(delta.reasoning_content, end="", flush=True)
    elif delta.content:
        print("\n=== Response ===")
        print(delta.content, end="", flush=True)
print()
```

### 4.4 Tool Calling

Mistral Medium 3.5 supports native function calling. Enable with `--tool-call-parser mistral`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)

tool_calls = response.choices[0].message.tool_calls
for tc in tool_calls:
    print(f"Tool: {tc.function.name}")
    print(f"Args: {tc.function.arguments}")
```

**Output:**

```text Output theme={null}
Tool: get_weather
Args: {"location": "Paris"}
```

### 4.5 Vision (Image Input)

Mistral Medium 3.5 accepts image inputs alongside text. The vision encoder was retrained from scratch to handle variable image sizes and aspect ratios:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Medium-3.5-128B",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see in this image."},
                {
                    "type": "image_url",
                    "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
                },
            ],
        }
    ],
    temperature=0.7,
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)
```

**Output:**

```text Output theme={null}
The image features a stylized representation of the acronym "SGL." The letters
are large, bold, and orange with a brown outline, giving them a three-dimensional
effect. To the left of the letters, there is a graphic that resembles a neuron
or a node with connections, also in a similar orange and brown color scheme. The
node has a code symbol (</>) inside a square, suggesting a connection to
programming or technology.
```

### 4.6 Loading the Reference System Prompt

Mistral ships a `SYSTEM_PROMPT.txt` alongside the weights. The reference setup loads it from the HF repo and substitutes `{name}`, `{today}`, and `{yesterday}` at runtime so the model knows its identity and the current date. SGLang's chat template will inject a default system prompt if you omit one, but for full parity with Mistral's reference, load it explicitly:

```python Example theme={null}
from datetime import datetime, timedelta
from huggingface_hub import hf_hub_download
from openai import OpenAI

MODEL = "mistralai/Mistral-Medium-3.5-128B"

def load_system_prompt(repo_id: str, filename: str = "SYSTEM_PROMPT.txt") -> str:
    path = hf_hub_download(repo_id=repo_id, filename=filename)
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    name = repo_id.split("/")[-1]
    with open(path) as f:
        return f.read().format(name=name, today=today, yesterday=yesterday)

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": load_system_prompt(MODEL)},
        {"role": "user", "content": "Write me a sentence where every word starts with the next letter in the alphabet — start with 'a' and end with 'z'."},
    ],
    temperature=0.1,
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)
```

***

## 5. Benchmarks

Validation runs on 4× H200 with `--tp 4`, served via the `/v1/chat/completions` endpoint.

### 5.1 Accuracy Benchmarks

#### GSM8K

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Results:**

```text Output theme={null}
Accuracy: 0.945
Invalid: 0.000
Latency: 13.594 s
Output throughput: 1560.660 token/s
```

#### MMMU

```bash Command theme={null}
python3 benchmark/mmmu/bench_sglang.py --port 30000
```

**Results:**

```text Output theme={null}
Overall accuracy: 0.586
```

### 5.2 Speed Benchmarks

#### Latency (Low Concurrency)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 10 \
  --max-concurrency 1 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Successful requests:                     10
Benchmark duration (s):                  38.86
Total input tokens:                      6101
Total generated tokens:                  2684
Output token throughput (tok/s):         69.07
Mean E2E Latency (ms):                   3883.80
Median TTFT (ms):                        95.90
Median TPOT (ms):                        14.19
==================================================
```

#### Throughput (High Concurrency)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Successful requests:                     1000
Benchmark duration (s):                  117.28
Total input tokens:                      512842
Total generated tokens:                  262023
Output token throughput (tok/s):         2234.18
Total token throughput (tok/s):          6607.01
Mean E2E Latency (ms):                   11303.79
Median TTFT (ms):                        152.95
Median TPOT (ms):                        42.53
==================================================
```

### 5.3 EAGLE Speculative Decoding (Latency)

Same 4× H200 setup, EAGLE configuration from [Section 3.3](#3-3-speculative-decoding-eagle). Single-stream latency benchmark (`--max-concurrency 1`).

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 10 \
  --max-concurrency 1 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Successful requests:                     10
Benchmark duration (s):                  27.64
Total input tokens:                      6101
Total generated tokens:                  2684
Output token throughput (tok/s):         97.10
Mean E2E Latency (ms):                   2762.99
Median TTFT (ms):                        90.69
Median TPOT (ms):                        9.73
Accept length:                           1.72
==================================================
```

EAGLE delivers **\~1.41× output throughput and \~29% lower E2E latency** vs. the baseline in [Section 5.2](#5-2-speed-benchmarks) on the same workload. Acceptance length of 1.72 means each draft cycle averages roughly 1.7 accepted tokens.


# Mistral Small 4
Source: https://docs.sglang.io/cookbook/autoregressive/Mistral/Mistral-Small-4



## 1. Model Introduction

**Mistral Small 4** is a powerful hybrid model from Mistral AI that unifies the capabilities of three different model families — **Instruct**, **Reasoning** (formerly called Magistral), and **Agentic (formerly called Devstral)** — into a single, unified model.

With its multimodal capabilities, efficient MoE architecture, and flexible mode switching, Mistral Small 4 is a versatile general-purpose model for virtually any task. In a latency-optimized setup, it achieves a 40% reduction in end-to-end completion time; in a throughput-optimized setup, it delivers 3× more requests per second compared to Mistral Small 3.

**Key Features:**

* **Hybrid Reasoning**: Switch between instant reply mode and deep reasoning/thinking mode — reasoning effort is configurable per request
* **Vision**: Accepts both text and image inputs, providing insights based on visual content
* **Function Calling**: Native tool calling and JSON output support with best-in-class agentic capabilities
* **Multilingual**: Supports dozens of languages including English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more
* **Context Window**: 256K context window
* **Efficient MoE**: 119B total parameters, 128 experts, 4 active per token (6.5B activated parameters)
* **Apache 2.0 License**: Open-source, usable and modifiable for commercial and non-commercial purposes
* Reasoning effort supported are only **"none" and "high"**

**Architecture:**

* Same general architecture as Mistral 3
* MoE: 128 experts, 4 active per token
* 119B total parameters, 6.5B activated per token
* Multimodal input: text + image

**Models:**

* **[mistralai/Mistral-Small-4-119B-2603](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603)** (FP8)
* **[mistralai/Mistral-Small-4-119B-2603-NVFP4](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-NVFP4)**
* **[mistralai/Leanstral-2603](https://huggingface.co/mistralai/Leanstral-2603)** — same architecture, use the same launch commands as Mistral-Small-4-119B-2603
* **[mistralai/Mistral-Small-4-119B-2603-eagle](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle)** — EAGLE speculative decoding weights for faster inference

***

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

> [!NOTE]
>
> Mistral Small 4 support landed in [sgl-project/sglang#20708](https://github.com/sgl-project/sglang/pull/20708) and has been merged into `main`. A model-specific Docker image is no longer required. Use the standard SGLang installation methods from the [official installation guide](../../../docs/get-started/install).

***

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to generate a launch command for Mistral Small 4.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Tensor Parallelism**: Mistral Small 4 FP8 (\~119 GB) requires tp=2 on Hopper (H100/H200), tp=1 on Blackwell (B200/B300). NVFP4 (\~60 GB, Blackwell only) runs with tp=1.
* **Reasoning effort**: Reasoning depth is configurable per request via `reasoning_effort` (`"none"`, `"high"`). No restart required — toggle per call.
* **Context length vs memory**: The model has a 256K context window. If you are memory-constrained, lower `--context-length` (e.g. `32768`) and increase once things are stable.
* **Tool calling**: Enable `--tool-call-parser mistral` to activate native function calling support.
* **Reasoning parser**: Enable `--reasoning-parser mistral` to separate `reasoning_content` from the main response content.
* **Speculative decoding (EAGLE)**: Enable with `--speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle` using the [EAGLE weights](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603-eagle) for lower latency.

***

## 4. Model Invocation

### 4.1 Thinking Mode

Mistral Small 4 is a hybrid reasoning model. By default, it does not produce a default reasoning response. Use `--reasoning_effort high` to toggle reasoning on.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Solve step by step: what is 17 × 23 + 144 / 12?"},
    ],
    extra_body={"reasoning_effort": "high"},
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("Answer:", response.choices[0].message.content)
```

**Output:**

```text Output theme={null}
Reasoning: First, I'll break down the problem into two parts: the multiplication and
the division. According to the order of operations (PEMDAS/BODMAS), multiplication and
division are performed from left to right before addition.

17 × 23 = 17 × (20 + 3) = (17 × 20) + (17 × 3) = 340 + 51 = 391
144 / 12 = 12

Finally, add the results: 391 + 12 = 403

Answer: The solution to the problem is as follows:

1. First, perform the multiplication: 17 × 23.
   - 17 × 20 = 340
   - 17 × 3 = 51
   - 340 + 51 = 391

2. Then, perform the division: 144 / 12 = 12.

3. Finally, add the results:
   - 391 + 12 = 403

**Answer:** \boxed{403}
```

### 4.2 Instruct Mode (Reasoning Off)

To skip the reasoning trace and get a fast direct response, set `reasoning_effort` to `"none"`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Write a Python function to reverse a string."},
    ],
    extra_body={"reasoning_effort": "none"},
)

print(response.choices[0].message.content)
```

**Output:**

````text Output theme={null}
# Python Function to Reverse a String

Here are several ways to write a Python function to reverse a string:

## Method 1: Using String Slicing (Most Pythonic)
```python
def reverse_string(s):
    """Reverse a string using slicing."""
    return s[::-1]
```

## Method 2: Using a Loop
```python Example
def reverse_string(s):
    """Reverse a string using a loop."""
    reversed_str = ""
    for char in s:
        reversed_str = char + reversed_str
    return reversed_str
```

## Method 3: Using reversed() function
```python Example
def reverse_string(s):
    """Reverse a string using reversed() function."""
    return ''.join(reversed(s))
```

The first method using string slicing (`s[::-1]`) is generally the most efficient and
recommended approach in Python.

Example usage:
```python Example
original = "Hello, World!"
reversed_str = reverse_string(original)
print(reversed_str)  # Output: "!dlroW ,olleH"
```
````

### 4.3 Streaming with Reasoning

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {"role": "user", "content": "Explain the difference between async and threading in Python."},
    ],
    extra_body={"reasoning_effort": "high"},
    stream=True,
)

print("=== Reasoning ===")
for chunk in stream:
    delta = chunk.choices[0].delta
    if hasattr(delta, "reasoning_content") and delta.reasoning_content:
        print(delta.reasoning_content, end="", flush=True)
    elif delta.content:
        print("\n=== Response ===")
        print(delta.content, end="", flush=True)
print()
```

**Output:**

```text Output theme={null}
=== Reasoning ===
Okay, the user is asking about the difference between async and threading in Python.
I need to break this down clearly, covering the key aspects of both, like their
purposes, performance characteristics, and use cases...
=== Response ===
In Python, **`async`/`asyncio`** and **`threading`** are two different concurrency
models, each suited for specific use cases. Here's a breakdown of their key differences:

### 1. Model of Concurrency
- **Threading**: Based on preemptive multitasking using OS threads.
- **Async** (`asyncio`): Based on cooperative multitasking. Tasks voluntarily yield...
```

### 4.4 Tool Calling

Mistral Small 4 supports native function calling. Enable with `--tool-call-parser mistral`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[{"role": "user", "content": "What's the weather in Paris?"}],
    tools=tools,
    tool_choice="auto",
)

tool_calls = response.choices[0].message.tool_calls
for tc in tool_calls:
    print(f"Tool: {tc.function.name}")
    print(f"Args: {tc.function.arguments}")
```

**Output:**

```text Output theme={null}
Tool: get_weather
Args: {"location": "Paris"}
```

### 4.5 Vision (Image Input)

Mistral Small 4 accepts image inputs alongside text:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Small-4-119B-2603",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see in this image."},
                {
                    "type": "image_url",
                    "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"},
                },
            ],
        }
    ],
)

print(response.choices[0].message.content)
```

**Output:**

```text Output theme={null}
The image is a copyright symbol, represented by a stylized version of the lowercase
letter "c" inside a circle. The "c" is depicted in a white or light-colored font, and
the circle is orange. The design is simple yet striking, using oval and elliptical
shapes to create a distinct symbol which signifies copyright protection.
```

***

## 5. Benchmarks

### 5.1 Accuracy Benchmarks

#### GSM8K

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

**Results:**

```text Output theme={null}
TODO
```

#### MMLU

```bash Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

**Results:**

```text Output theme={null}
TODO
```

### 5.2 Speed Benchmarks

#### Latency (Low Concurrency)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompts 10 \
  --max-concurrency 1 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output theme={null}
TODO
```

#### Throughput (High Concurrency)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --random-input-len 1024 \
  --random-output-len 512 \
  --port 30000
```

**Results:**

```text Output theme={null}
TODO
```


# Kimi-K2
Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2



## 1. Model Introduction

[Kimi-K2](https://moonshotai.github.io/Kimi-K2/) is a state-of-the-art MoE language model by Moonshot AI with 32B activated parameters and 1T total parameters.

**Model Variants:**

* **[Kimi-K2-Instruct](https://huggingface.co/moonshotai/Kimi-K2-Instruct)**: Post-trained model optimized for general-purpose chat and agentic tasks. Compatible with vLLM, SGLang, KTransformers, and TensorRT-LLM.
* **[Kimi-K2-Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)**: Advanced thinking model with step-by-step reasoning and tool calling. Native INT4 quantization with 256k context window. Ideal for complex reasoning and multi-step tool use.
* **ROCm Support**: Compatible with AMD MI300X GPUs via SGLang (verified).

For details, see [official documentation](https://github.com/MoonshotAI/Kimi-K2) and [technical report](https://www.arxiv.org/abs/2507.20534).

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory.
* **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism).
* **Data Parallel (DP)**: Enable with `--attn-dp-size 4` for production throughput.
* **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+).
* **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content.
* **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
* **AMD GPU**: Set `SGLANG_ROCM_FUSED_DECODE_MLA=0` before launching AMD GPU.

## 4. Model Invocation

### 4.1 Basic Usage

See [Basic API Usage](../../../docs/get-started/quickstart).

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

Enable reasoning parser for Kimi-K2-Thinking:

```shell Command theme={null}
python -m sglang.launch_server \
  --model moonshotai/Kimi-K2-Thinking \
  --reasoning-parser kimi_k2 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2-Thinking",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.6,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
  The user asks: "What is 15% of 240?" This is a straightforward percentage calculation problem. I need to solve it step by step.

Step 1: Understand what "percent" means.
- "Percent" means "per hundred". So 15% means 15 per 100, or 15/100, or 0.15.

Step 2: Convert the percentage to a decimal.
- 15% = 15 / 100 = 0.15

Step 3: Multiply the decimal by the number.
- 0.15 * 240

Step 4: Perform the multiplication.
- 0.15 * 240 = (15/100) * 240
- = 15 * 240 / 100
- = 3600 / 100
- = 36

Alternatively, I can calculate it directly:
- 0.15 * 240
- 15 * 240 = 3600
- 3600 / 100 = 36

Or, break it down:
- 10% of 240 = 24
- 5% of 240 = half of 10% = 12
- 15% of 240 = 10% + 5% = 24 + 12 = 36

I should present the solution clearly with steps. The most standard method is converting to decimal and multiplying.

Let me structure the answer:
1. Convert the percentage to a decimal.
2. Multiply the decimal by the number.
3. Show the calculation.
4. State the final answer.

This is simple and easy to follow.
=============== Content =================
 Here is the step-by-step solution:

**Step 1: Convert the percentage to a decimal**
15% means 15 per 100, which is 15 ÷ 100 = **0.15**

**Step 2: Multiply the decimal by the number**
0.15 × 240

**Step 3: Calculate the result**
0.15 × 240 = **36**

**Answer:** 15% of 240 is **36**.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.2 Tool Calling

Kimi-K2-Instruct and Kimi-K2-Thinking support tool calling capabilities. Enable the tool call parser during deployment:

**Deployment Command:**

```shell Command theme={null}
python -m sglang.launch_server \
  --model moonshotai/Kimi-K2-Instruct \
  --tool-call-parser kimi_k2 \
  --tp 8 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 8000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2-Thinking",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
  The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information. Beijing is a major city in China, so I should be able to get weather data for it. The location parameter is required, but the unit parameter is optional. Since the user didn't specify a temperature unit, I can just provide the location and let the function use its default. I'll check the weather in Beijing for you.
=============== Content =================

  🔧 Tool Call: get_weather
   Arguments: {"location":"Beijing"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="moonshotai/Kimi-K2-Thinking",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8x)
* Model: Kimi-K2-Instruct
* sglang version: 0.5.6.post1

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
    --model-path moonshotai/Kimi-K2-Instruct \
    --tp 8 \
    --attn-dp-size 4 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 8000 \
  --model moonshotai/Kimi-K2-Instruct\
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  44.93
Total input tokens:                      1951
Total input text tokens:                 1951
Total input vision tokens:               0
Total generated tokens:                  2755
Total generated tokens (retokenized):    2748
Request throughput (req/s):              0.22
Input token throughput (tok/s):          43.42
Output token throughput (tok/s):         61.32
Peak output token throughput (tok/s):    64.00
Peak concurrent requests:                3
Total token throughput (tok/s):          104.74
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4489.56
Median E2E Latency (ms):                 4994.53
---------------Time to First Token----------------
Mean TTFT (ms):                          141.22
Median TTFT (ms):                        158.28
P99 TTFT (ms):                           166.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          18.40
Median TPOT (ms):                        15.63
P99 TPOT (ms):                           39.88
---------------Inter-Token Latency----------------
Mean ITL (ms):                           15.78
Median ITL (ms):                         15.76
P95 ITL (ms):                            16.36
P99 ITL (ms):                            16.59
Max ITL (ms):                            19.94
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
    --model-path moonshotai/Kimi-K2-Instruct \
    --tp 8 \
    --attn-dp-size 4 \
    --ep 4 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 8000 \
  --model moonshotai/Kimi-K2-Instruct\
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results**:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  174.11
Total input tokens:                      296642
Total input text tokens:                 296642
Total input vision tokens:               0
Total generated tokens:                  193831
Total generated tokens (retokenized):    168687
Request throughput (req/s):              5.74
Input token throughput (tok/s):          1703.73
Output token throughput (tok/s):         1113.25
Peak output token throughput (tok/s):    2383.00
Peak concurrent requests:                112
Total token throughput (tok/s):          2816.97
Concurrency:                             89.60
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   15601.09
Median E2E Latency (ms):                 10780.52
---------------Time to First Token----------------
Mean TTFT (ms):                          457.42
Median TTFT (ms):                        221.62
P99 TTFT (ms):                           2475.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          97.23
Median TPOT (ms):                        85.61
P99 TPOT (ms):                           435.95
---------------Inter-Token Latency----------------
Mean ITL (ms):                           78.61
Median ITL (ms):                         43.66
P95 ITL (ms):                            169.53
P99 ITL (ms):                            260.91
Max ITL (ms):                            1703.21
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* Server Command

```shell Command theme={null}
python3 -m sglang.launch_server \
    --model-path moonshotai/Kimi-K2-Instruct \
    --tp 8 \
    --dp 4 \
    --trust-remote-code  \
    --host 0.0.0.0 \
    --port 8000
```

* Benchmark Command

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:8000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Result**:

```text Output theme={null}
Accuracy: 0.960
Invalid: 0.000
Latency: 15.956 s
Output throughput: 1231.699 token/s
```


# Kimi-K2.5
Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.5



## 1. Model Introduction

[Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) is an open-source, native multimodal agentic model by Moonshot AI, built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes.

**Key Features:**

* **Native Multimodality**: Pre-trained on vision-language tokens, K2.5 excels in visual knowledge, cross-modal reasoning, and agentic tool use grounded in visual inputs.
* **Coding with Vision**: K2.5 generates code from visual specifications (UI designs, video workflows) and autonomously orchestrates tools for visual data processing.
* **Agent Swarm**: K2.5 transitions from single-agent scaling to a self-directed, coordinated swarm-like execution scheme. It decomposes complex tasks into parallel sub-tasks executed by dynamically instantiated, domain-specific agents.
* **Speculative Decoding**: EAGLE-based speculative decoding support for lower latency.

**Available Models**:

* INT4 (Initial Released): [moonshotai/Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5)
* NVFP4 (4-bit quantized): [nvidia/Kimi-K2.5-NVFP4](https://huggingface.co/nvidia/Kimi-K2.5-NVFP4)

For details, see [official documentation](https://huggingface.co/moonshotai/Kimi-K2.5) and [deployment guidance](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/docs/deploy_guidance.md).

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory**: Requires GPUs with >=140GB each. Supported platforms: H200 (8x, TP=8), B300 (8x, TP=8), GB300 (4x, TP=4), MI300X/MI325X (4x, TP=4), MI350X/MI355X (4x, TP=4). Use `--context-length 128000` to conserve memory.
* **AMD GPU TP Constraint**: On AMD GPUs, TP must be \<= 4 (not 8). Kimi-K2.5 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
* **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. The ROCm 7.2 images (`rocm720`) have an AITER compatibility issue.
* **DP Attention**: Enable with `--attn-dp-size <N>` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required.
* **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
* **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.

## 4. Model Invocation

### 4.1 Basic Usage

See [Basic API Usage](../../../docs/basic_usage/send_request).

### 4.2 Advanced Usage

#### 4.2.1 Multimodal (Vision + Text) Input

Kimi-K2.5 supports native multimodal input with images:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "What is in this image? Describe it in detail."
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
This image shows a **receipt from Auntie Anne's** (a pretzel franchise restaurant).

## Key Details:

**Item Purchased:**
- **CINNAMON SUGAR** - 1 unit x 17,000 = **17,000**

**Payment Summary:**
- **SUB TOTAL:** 17,000
- **GRAND TOTAL:** 17,000
- **CASH IDR:** 20,000 (Indonesian Rupiah)
- **CHANGE DUE:** 3,000

## Context:
The receipt indicates a transaction in **Indonesian Rupiah (IDR)**. A customer purchased one Cinnamon Sugar pretzel for 17,000 IDR, paid with a 20,000 IDR note, and received 3,000 IDR in change.

The top of the receipt shows the Auntie Anne's logo (a heart-shaped pretzel with a halo), and some text appears blurred for privacy, likely obscuring the store location, date, and transaction number. The receipt is printed on white thermal paper.
```

#### 4.2.2 Reasoning Output

Kimi-K2.5 supports both thinking mode (default) and instant mode.

**Thinking Mode (default)** -- reasoning content is automatically separated:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=[
        {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
    ]
)

print("====== Reasoning Content (Thinking Mode) ======")
print(response.choices[0].message.reasoning_content)
print("====== Response (Thinking Mode) ======")
print(response.choices[0].message.content)
```

**Instant Mode (thinking off)** -- disable thinking for faster responses:

```python Example theme={null}
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=[
        {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
    ],
    extra_body={"chat_template_kwargs": {"thinking": False}}
)

print("====== Response (Instant Mode) ======")
print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
====== Reasoning Content (Thinking Mode) ======
The user is asking which number is bigger: 9.11 or 9.9.

At first glance, someone might think 9.11 is bigger because 11 > 9, but that's incorrect because we're dealing with decimal numbers, not whole numbers.

Let me compare them properly:
- 9.9 = 9.90
- 9.11

When comparing decimals, we look at each place value from left to right:
- Units place: 9 = 9 (tie)
- Tenths place: 9 vs 1

Since 9 > 1, we have 9.9 > 9.11.

Alternatively, we can think of it as:
- 9.9 = 9 + 9/10 = 9 + 0.9 = 9.90
- 9.11 = 9 + 11/100 = 9 + 0.11

Since 0.90 > 0.11, then 9.9 > 9.11.

So the answer is clearly 9.9 is bigger.

The "think carefully" hint suggests the user is trying to catch the common error where people compare 11 and 9 as whole numbers rather than understanding decimal place value (tenths vs hundredths).

I should explain this clearly to avoid confusion.
====== Response (Thinking Mode) ======
**9.9 is bigger.**

Here's why this can be tricky: Many people instinctively compare 11 and 9 and think "11 is bigger than 9," but that's comparing the wrong place values.

When comparing decimals, align them by place value:
- 9.9 = 9.**90**
- 9.11 = 9.**11**

After the decimal point:
- The first digit (tenths place): **9** vs **1**
- Since 9 > 1, we stop there. **9.9 is larger.**

Think of it as money:
- $9.90 (nine dollars and ninety cents)
- $9.11 (nine dollars and eleven cents)

$9.90 is clearly more than $9.11.
====== Response (Instant Mode) ======
 Let me think through this carefully.

**9.9 is bigger than 9.11**

Here's why: When comparing decimals, we need to align them by their decimal places:

- 9.9 = 9.90
- 9.11 = 9.11

Now comparing:
- The whole number parts are equal (9 = 9)
- Comparing tenths: **9 > 1**

So 9.90 > 9.11

A common mistake is thinking 11 hundredths is larger than 9 tenths, but 9 tenths = 90 hundredths, which is clearly larger than 11 hundredths.
```

#### 4.2.3 Tool Calling

Kimi-K2.5 supports tool calling capabilities for agentic tasks:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

# Process streaming response
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {'name': None, 'arguments': ''}
                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        if delta.content:
            print(delta.content, end="", flush=True)

for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"  Arguments: {tool_call['arguments']}")
```

**Output Example:**

```text Output theme={null}
Tool Call: get_weather
  Arguments: {"location": "Beijing"}
```

**Handling Tool Call Results:**

```python Example theme={null}
# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": "The weather in Beijing is 22°C and sunny."
    }
]

final_response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=messages
)

print(final_response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
The weather in Beijing is **22°C and sunny**. ☀️

It's a nice day there with comfortable temperatures and clear skies!
```

#### 4.2.4 Multimodal + Tool Calling (Agentic Vision)

Combine vision understanding with tool calling for advanced agentic tasks:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_product",
            "description": "Search for a product by name or description",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The product name or description to search for"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.5",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "Can you identify this product and search for similar items?"
                }
            ]
        }
    ],
    tools=tools
)

msg = response.choices[0].message

# Print reasoning process
if msg.reasoning_content:
    print("=== Reasoning ===")
    print(msg.reasoning_content)

# Print response content
if msg.content:
    print("=== Content ===")
    print(msg.content)

# Print tool calls
if msg.tool_calls:
    print("=== Tool Calls ===")
    for tc in msg.tool_calls:
        print(f"  Function: {tc.function.name}")
        print(f"  Arguments: {tc.function.arguments}")
```

**Output Example:**

```text Output theme={null}
=== Reasoning ===
The user is asking me to identify a product from a receipt and search for similar items.
Looking at the receipt, I can see:

 1. The store is "Auntie Anne's" - which is a popular pretzel chain
 2. The product purchased is "CINNAMON SUGAR"
 3. Price is 17,000 (likely Indonesian Rupiah based on "CASH IDR")
 4. Quantity is 1

So the product is a Cinnamon Sugar pretzel from Auntie Anne's.
Now I need to search for this product or similar items using the search_product function.
=== Content ===
I can see from the receipt that the product is a **Cinnamon Sugar** item from **Auntie Anne's** (the famous pretzel chain). This appears to be a Cinnamon Sugar Pretzel purchased for 17,000 IDR (Indonesian Rupiah).

Let me search for this product and similar items:
=== Tool Calls ===
  Function: search_product
  Arguments: {"query": "Auntie Anne's Cinnamon Sugar Pretzel"}
```

#### 4.2.5 Speculative Decoding

**Nvidia**

Deploy Kimi-K2.5 with the following command (H200/B300, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path moonshotai/Kimi-K2.5 \
  --tp 8 \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --speculative-algorithm=EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

Deploy Kimi-K2.5-NVFP4 with the following command (B300, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path nvidia/Kimi-K2.5-NVFP4 \
  --tp 8 \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --kv-cache-dtype fp8_e4m3 \
  --speculative-algorithm=EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

For GB300, use `--tp 4`.

## 5. Benchmark

### 5.1 Accuracy Benchmark

#### 5.1.1 MMMU Benchmark

You can evaluate the model's accuracy using the MMMU benchmark, which tests multimodal understanding and reasoning across various subjects:

* **Benchmark Command:**

```shell Command theme={null}
python3 benchmark/mmmu/bench_sglang.py \
    --response-answer-regex "(?i)(?:answer|ans)[:\s]*(?:\*\*)?[\(\[]?([A-Za-z])[\)\]]?(?:\*\*)?" \
    --port 30000 \
    --concurrency 64
```

* **Result:**

```text Output theme={null}
Benchmark time: 2785.4322692090645
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.667, 'num': 30},
 'Agriculture': {'acc': 0.567, 'num': 30},
 'Architecture_and_Engineering': {'acc': 0.733, 'num': 30},
 'Art': {'acc': 0.833, 'num': 30},
 'Art_Theory': {'acc': 0.8, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.833, 'num': 30},
 'Biology': {'acc': 0.6, 'num': 30},
 'Chemistry': {'acc': 0.633, 'num': 30},
 'Clinical_Medicine': {'acc': 0.733, 'num': 30},
 'Computer_Science': {'acc': 0.667, 'num': 30},
 'Design': {'acc': 0.7, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.5, 'num': 30},
 'Economics': {'acc': 0.867, 'num': 30},
 'Electronics': {'acc': 0.3, 'num': 30},
 'Energy_and_Power': {'acc': 0.767, 'num': 30},
 'Finance': {'acc': 0.833, 'num': 30},
 'Geography': {'acc': 0.667, 'num': 30},
 'History': {'acc': 0.767, 'num': 30},
 'Literature': {'acc': 0.767, 'num': 30},
 'Manage': {'acc': 0.733, 'num': 30},
 'Marketing': {'acc': 0.833, 'num': 30},
 'Materials': {'acc': 0.567, 'num': 30},
 'Math': {'acc': 0.633, 'num': 30},
 'Mechanical_Engineering': {'acc': 0.567, 'num': 30},
 'Music': {'acc': 0.5, 'num': 30},
 'Overall': {'acc': 0.698, 'num': 900},
 'Overall-Art and Design': {'acc': 0.708, 'num': 120},
 'Overall-Business': {'acc': 0.787, 'num': 150},
 'Overall-Health and Medicine': {'acc': 0.74, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.75, 'num': 120},
 'Overall-Science': {'acc': 0.66, 'num': 150},
 'Overall-Tech and Engineering': {'acc': 0.595, 'num': 210},
 'Pharmacy': {'acc': 0.767, 'num': 30},
 'Physics': {'acc': 0.767, 'num': 30},
 'Psychology': {'acc': 0.667, 'num': 30},
 'Public_Health': {'acc': 0.867, 'num': 30},
 'Sociology': {'acc': 0.8, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.698
```

### 5.2 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA H200 GPU (8x)
* Model: Kimi-K2.5
* Tensor Parallelism: 8
* SGLang Version: 0.5.6.post2

We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation.

#### 5.2.1 Latency Benchmark

* **Model Deployment:**

```bash Command theme={null}
sglang serve \
  --model-path moonshotai/Kimi-K2.5 \
  --tp 8 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

* **Benchmark Command:**

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  39.77
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4221
Request throughput (req/s):              0.25
Input token throughput (tok/s):          153.40
Output token throughput (tok/s):         106.10
Peak output token throughput (tok/s):    156.00
Peak concurrent requests:                2
Total token throughput (tok/s):          259.50
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3972.87
Median E2E Latency (ms):                 4044.55
P90 E2E Latency (ms):                    7046.30
P99 E2E Latency (ms):                    7441.13
---------------Time to First Token----------------
Mean TTFT (ms):                          176.89
Median TTFT (ms):                        154.24
P99 TTFT (ms):                           285.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.22
Median TPOT (ms):                        9.32
P99 TPOT (ms):                           12.72
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.02
Median ITL (ms):                         8.80
P95 ITL (ms):                            13.23
P99 ITL (ms):                            14.17
Max ITL (ms):                            29.38
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  158.05
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40775
Request throughput (req/s):              0.51
Input token throughput (tok/s):          250.99
Output token throughput (tok/s):         258.18
Peak output token throughput (tok/s):    1103.00
Peak concurrent requests:                19
Total token throughput (tok/s):          509.17
Concurrency:                             14.09
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   27837.05
Median E2E Latency (ms):                 23508.00
P90 E2E Latency (ms):                    57126.31
P99 E2E Latency (ms):                    66044.35
---------------Time to First Token----------------
Mean TTFT (ms):                          374.30
Median TTFT (ms):                        375.51
P99 TTFT (ms):                           695.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          53.25
Median TPOT (ms):                        57.93
P99 TPOT (ms):                           85.45
---------------Inter-Token Latency----------------
Mean ITL (ms):                           53.95
Median ITL (ms):                         53.97
P95 ITL (ms):                            84.74
P99 ITL (ms):                            244.84
Max ITL (ms):                            655.61
==================================================
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* **Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  996.64
Total input tokens:                      249831
Total input text tokens:                 249831
Total generated tokens:                  252662
Total generated tokens (retokenized):    252588
Request throughput (req/s):              0.50
Input token throughput (tok/s):          250.67
Output token throughput (tok/s):         253.51
Peak output token throughput (tok/s):    1199.00
Peak concurrent requests:                104
Total token throughput (tok/s):          504.18
Concurrency:                             92.70
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   184773.75
Median E2E Latency (ms):                 174183.65
P90 E2E Latency (ms):                    343625.28
P99 E2E Latency (ms):                    404284.53
---------------Time to First Token----------------
Mean TTFT (ms):                          1289.59
Median TTFT (ms):                        1313.35
P99 TTFT (ms):                           2346.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          364.70
Median TPOT (ms):                        403.32
P99 TPOT (ms):                           452.34
---------------Inter-Token Latency----------------
Mean ITL (ms):                           363.82
Median ITL (ms):                         316.21
P95 ITL (ms):                            745.91
P99 ITL (ms):                            1345.88
Max ITL (ms):                            3118.59
==================================================
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  680.26
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  44462
Total generated tokens (retokenized):    44455
Request throughput (req/s):              0.01
Input token throughput (tok/s):          8.97
Output token throughput (tok/s):         65.36
Peak output token throughput (tok/s):    151.00
Peak concurrent requests:                2
Total token throughput (tok/s):          74.33
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   68019.29
Median E2E Latency (ms):                 70568.85
P90 E2E Latency (ms):                    113237.40
P99 E2E Latency (ms):                    121682.34
---------------Time to First Token----------------
Mean TTFT (ms):                          206.17
Median TTFT (ms):                        177.28
P99 TTFT (ms):                           445.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.36
Median TPOT (ms):                        15.89
P99 TPOT (ms):                           16.43
---------------Inter-Token Latency----------------
Mean ITL (ms):                           15.26
Median ITL (ms):                         15.85
P95 ITL (ms):                            17.50
P99 ITL (ms):                            23.21
Max ITL (ms):                            45.22
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  2475.98
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  318306
Total generated tokens (retokenized):    318166
Request throughput (req/s):              0.03
Input token throughput (tok/s):          16.02
Output token throughput (tok/s):         128.56
Peak output token throughput (tok/s):    847.00
Peak concurrent requests:                18
Total token throughput (tok/s):          144.58
Concurrency:                             14.62
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   452592.46
Median E2E Latency (ms):                 486002.05
P90 E2E Latency (ms):                    833197.57
P99 E2E Latency (ms):                    957399.48
---------------Time to First Token----------------
Mean TTFT (ms):                          359.38
Median TTFT (ms):                        350.78
P99 TTFT (ms):                           500.36
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          111.18
Median TPOT (ms):                        122.76
P99 TPOT (ms):                           145.90
---------------Inter-Token Latency----------------
Mean ITL (ms):                           113.69
Median ITL (ms):                         122.81
P95 ITL (ms):                            147.87
P99 ITL (ms):                            151.03
Max ITL (ms):                            272.05
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
Waiting for completion...
```

**Scenario 3: Summarization (8K/1K)**

* Low Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  120.73
Total input tokens:                      41941
Total input text tokens:                 41941
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.08
Input token throughput (tok/s):          347.41
Output token throughput (tok/s):         34.96
Peak output token throughput (tok/s):    73.00
Peak concurrent requests:                2
Total token throughput (tok/s):          382.36
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   12068.56
Median E2E Latency (ms):                 10211.36
P90 E2E Latency (ms):                    23203.32
P99 E2E Latency (ms):                    30677.66
---------------Time to First Token----------------
Mean TTFT (ms):                          1625.64
Median TTFT (ms):                        1526.63
P99 TTFT (ms):                           3743.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.95
Median TPOT (ms):                        23.95
P99 TPOT (ms):                           35.40
---------------Inter-Token Latency----------------
Mean ITL (ms):                           24.80
Median ITL (ms):                         21.73
P95 ITL (ms):                            59.56
P99 ITL (ms):                            61.10
Max ITL (ms):                            62.70
==================================================
```

* Medium Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  389.96
Total input tokens:                      300020
Total input text tokens:                 300020
Total generated tokens:                  41669
Total generated tokens (retokenized):    41670
Request throughput (req/s):              0.21
Input token throughput (tok/s):          769.36
Output token throughput (tok/s):         106.86
Peak output token throughput (tok/s):    304.00
Peak concurrent requests:                19
Total token throughput (tok/s):          876.22
Concurrency:                             14.95
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   72870.97
Median E2E Latency (ms):                 70495.88
P90 E2E Latency (ms):                    121820.46
P99 E2E Latency (ms):                    148933.09
---------------Time to First Token----------------
Mean TTFT (ms):                          2460.45
Median TTFT (ms):                        1976.29
P99 TTFT (ms):                           7305.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          140.57
Median TPOT (ms):                        142.31
P99 TPOT (ms):                           273.40
---------------Inter-Token Latency----------------
Mean ITL (ms):                           135.44
Median ITL (ms):                         95.96
P95 ITL (ms):                            152.93
P99 ITL (ms):                            1488.37
Max ITL (ms):                            6540.24
==================================================
```

* High Concurrency

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  1279.50
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total generated tokens:                  170000
Total generated tokens (retokenized):    169981
Request throughput (req/s):              0.25
Input token throughput (tok/s):          995.62
Output token throughput (tok/s):         132.86
Peak output token throughput (tok/s):    703.00
Peak concurrent requests:                67
Total token throughput (tok/s):          1128.49
Concurrency:                             60.12
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   240385.63
Median E2E Latency (ms):                 236266.30
P90 E2E Latency (ms):                    429882.12
P99 E2E Latency (ms):                    515158.36
---------------Time to First Token----------------
Mean TTFT (ms):                          2710.44
Median TTFT (ms):                        2345.63
P99 TTFT (ms):                           7144.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          443.84
Median TPOT (ms):                        493.29
P99 TPOT (ms):                           606.19
---------------Inter-Token Latency----------------
Mean ITL (ms):                           448.23
Median ITL (ms):                         296.17
P95 ITL (ms):                            1869.15
P99 ITL (ms):                            2708.95
Max ITL (ms):                            7778.47
==================================================
```

#### 5.2.2 Speculative Decoding Benchmark

* **Model Deployment:**

```bash Command theme={null}
sglang serve \
  --model-path moonshotai/Kimi-K2.5 \
  --tp 8 \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --speculative-algorithm=EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

* **Benchmark Command:**

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Results:**

```text Output theme={null}
Pending update...
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
Pending update...
```

* High Concurrency (Throughput-Optimized)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
Pending update...
```

### 5.3 Speed Benchmark (AMD MI350X)

**Test Environment:**

* Hardware: AMD Instinct MI350X GPU (4x)
* Model: Kimi-K2.5 (BF16)
* Tensor Parallelism: 4
* SGLang Version: 0.5.9
* Docker Image: `lmsysorg/sglang:v0.5.9-rocm700-mi35x`
* ROCm: 7.0

We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation.

:::info AMD GPU TP Constraint
Kimi-K2.5 requires TP \<= 4 on AMD GPUs. The model has 64 attention heads, and the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
:::

#### 5.3.1 Latency Benchmark

* **Model Deployment:**

```bash Command theme={null}
SGLANG_USE_AITER=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 \
sglang serve \
  --model-path moonshotai/Kimi-K2.5 \
  --tp 4 \
  --mem-fraction-static 0.8 \
  --trust-remote-code \
  --reasoning-parser kimi_k2 \
  --host 0.0.0.0 \
  --port 30000
```

* **Benchmark Command:**

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  155.81
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4222
Request throughput (req/s):              0.06
Input token throughput (tok/s):          39.16
Output token throughput (tok/s):         27.09
Peak output token throughput (tok/s):    29.00
Peak concurrent requests:                2
Total token throughput (tok/s):          66.24
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   15576.22
Median E2E Latency (ms):                 12539.80
P90 E2E Latency (ms):                    28150.56
P99 E2E Latency (ms):                    34873.51
---------------Time to First Token----------------
Mean TTFT (ms):                          563.50
Median TTFT (ms):                        594.92
P99 TTFT (ms):                           830.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          35.61
Median TPOT (ms):                        35.66
P99 TPOT (ms):                           35.77
---------------Inter-Token Latency----------------
Mean ITL (ms):                           35.66
Median ITL (ms):                         35.69
P95 ITL (ms):                            35.96
P99 ITL (ms):                            36.13
Max ITL (ms):                            36.92
==================================================
```

* Medium Concurrency (Balanced)

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.5 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  526.66
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40798
Request throughput (req/s):              0.15
Input token throughput (tok/s):          75.32
Output token throughput (tok/s):         77.48
Peak output token throughput (tok/s):    96.00
Peak concurrent requests:                18
Total token throughput (tok/s):          152.80
Concurrency:                             14.59
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   96023.27
Median E2E Latency (ms):                 93940.20
P90 E2E Latency (ms):                    159449.54
P99 E2E Latency (ms):                    194706.61
---------------Time to First Token----------------
Mean TTFT (ms):                          989.08
Median TTFT (ms):                        886.42
P99 TTFT (ms):                           1543.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          191.04
Median TPOT (ms):                        195.20
P99 TPOT (ms):                           238.84
---------------Inter-Token Latency----------------
Mean ITL (ms):                           186.68
Median ITL (ms):                         183.82
P95 ITL (ms):                            189.90
P99 ITL (ms):                            673.64
Max ITL (ms):                            1633.20
==================================================
```


# Kimi-K2.6
Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.6



## 1. Model Introduction

[Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) is an open-source, native multimodal agentic model by Moonshot AI, delivering industry-leading coding, long-horizon execution, and agent swarm capabilities. It matches or surpasses GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro across key benchmarks.

**Key Features:**

* **Long-Horizon Coding**: Excels at complex, end-to-end coding tasks with 13+ hours of continuous execution and 4,000+ lines of code modification, generalizing across languages (Rust, Go, Python) and tasks (frontend, devops, performance optimization).
* **Coding-Driven Design**: Transforms prompts and visual inputs into production-ready interfaces with motion-rich elements including WebGL shaders, GSAP + Framer Motion, and Three.js 3D.
* **Agent Swarms Elevated**: Scales to 300 parallel sub-agents executing 4,000 coordinated steps per run. One prompt, 100+ files.
* **Proactive Agents**: Powers OpenClaw, Hermes Agent, and other autonomous frameworks for 5-day continuous operation.
* **Native Multimodality**: Pre-trained on vision–language tokens with MoonViT (400M parameters) for visual understanding, cross-modal reasoning, and agentic tool use grounded in visual inputs.

**Benchmarks (Open-Source SOTA):**

<table>
  <thead>
    <tr>
      <th>Benchmark</th>
      <th>Score</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>HLE w/ tools</td>
      <td>54.0</td>
    </tr>

    <tr>
      <td>SWE-Bench Pro</td>
      <td>58.6</td>
    </tr>

    <tr>
      <td>SWE-bench Multilingual</td>
      <td>76.7</td>
    </tr>

    <tr>
      <td>BrowseComp</td>
      <td>83.2</td>
    </tr>

    <tr>
      <td>Toolathlon</td>
      <td>50.0</td>
    </tr>

    <tr>
      <td>AIME 2026</td>
      <td>96.4</td>
    </tr>

    <tr>
      <td>GPQA-Diamond</td>
      <td>90.5</td>
    </tr>

    <tr>
      <td>LiveCodeBench</td>
      <td>89.6</td>
    </tr>
  </tbody>
</table>

**Recommended Generation Parameters:**

* Thinking Mode: `temperature=1.0`, `top_p=0.95`
* Instant Mode: `temperature=0.6`, `top_p=0.95`

**Available Models:**

* **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6)
* **NVFP4 (4-bit quantized, NVIDIA Blackwell)**: [nvidia/Kimi-K2.6-NVFP4](https://huggingface.co/nvidia/Kimi-K2.6-NVFP4)

**License:** Modified MIT for the native checkpoint. The NVIDIA NVFP4 checkpoint is governed by the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).

For details, see [official documentation](https://huggingface.co/moonshotai/Kimi-K2.6) and [tech blog](https://kimi.com/blog/kimi-k2-6).

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory.
* **NVFP4**: Use `nvidia/Kimi-K2.6-NVFP4` with `--quantization modelopt_fp4` on NVIDIA Blackwell. Use `tp=8` on B300 and `tp=4` on GB300.
* **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.6 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
* **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X.
* **DP Attention**: Enable with `--attn-dp-size <N>` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required.
* **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
* **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
* **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.

## 4. Model Invocation

### 4.1 Basic Usage

See [Basic API Usage](../../../docs/basic_usage/send_request).

### 4.2 Advanced Usage

#### 4.2.1 Multimodal (Vision + Text) Input

Kimi-K2.6 supports native multimodal input with images:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "What is in this image? Describe it in detail."
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown:

## Header
- At the top left is the Auntie Anne's logo (a pretzel with a halo)
- The store name "**Auntie Anne's**" is printed prominently at the top
- Some text below the store name appears blurred/redacted (likely store location, address, or transaction details)

## Purchase Details
- **Item**: CINNAMON SUGAR
- **Quantity & Price**: 1 × 17,000
- **Item Total**: 17,000

## Financial Summary
- **SUB TOTAL**: 17,000
- **GRAND TOTAL**: 17,000
- **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah)
- **CHANGE DUE**: 3,000

## Physical Description
- The receipt is printed on white thermal paper
- Some information in the middle section and toward the bottom is intentionally blurred/obscured
- The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter)

The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash.
```

#### 4.2.2 Reasoning Output

Kimi-K2.6 supports both thinking mode (default) and instant mode.

**Thinking Mode (default)** — reasoning content is automatically separated:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=[
        {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
    ]
)

print("====== Reasoning Content (Thinking Mode) ======")
print(response.choices[0].message.reasoning_content)
print("====== Response (Thinking Mode) ======")
print(response.choices[0].message.content)
```

**Instant Mode (thinking off)** — disable thinking for faster responses:

```python Example theme={null}
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=[
        {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
    ],
    extra_body={"chat_template_kwargs": {"thinking": False}}
)

print("====== Response (Instant Mode) ======")
print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
====== Reasoning Content (Thinking Mode) ======
The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion.

Let me think carefully:
- 9.11 means 9 + 11/100 = 9.11
- 9.9 means 9 + 9/10 = 9.90

So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11.

The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9).

So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79.

I should explain this clearly and address the common misconception.
====== Response (Thinking Mode) ======
Mathematically, **9.9 is bigger**.

Here's why:

**9.9 = 9.90**

When comparing decimals, you need to look at the same place values:
- 9.11 = 9 ones, 1 tenth, and 1 hundredth
- 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90)

Since **0.90 > 0.11**, it follows that **9.9 > 9.11**.

The difference is:
9.9 - 9.11 = 0.79

**Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger.
====== Response (Instant Mode) ======
I need to compare 9.11 and 9.9.

Let me think carefully by aligning the decimal places:

- 9.11 = 9 and 11/100 = 9.11
- 9.9 = 9 and 9/10 = 9.90

Since 0.90 > 0.11

**9.9 is bigger.**

This is a common trick question because people sometimes mistakenly compare 11 and 9 as whole numbers after the decimal point, forgetting that 9.9 = 9.90, which is greater than 9.11.
```

#### 4.2.3 Tool Calling

Kimi-K2.6 supports tool calling capabilities for agentic tasks:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

# Process streaming response
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {'name': None, 'arguments': ''}
                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        if delta.content:
            print(delta.content, end="", flush=True)

for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"  Arguments: {tool_call['arguments']}")
```

**Output Example:**

```text Output theme={null}
Tool Call: get_weather
  Arguments: {"location": "Beijing"}
```

**Handling Tool Call Results:**

```python Example theme={null}
# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": "The weather in Beijing is 22°C and sunny."
    }
]

final_response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=messages
)

print(final_response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
The weather in Beijing is currently **22°C and sunny**. ☀️

It's a nice, warm day there—great for being outdoors!
```

#### 4.2.4 Multimodal + Tool Calling (Agentic Vision)

Combine vision understanding with tool calling for advanced agentic tasks:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_product",
            "description": "Search for a product by name or description",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The product name or description to search for"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "Can you identify this product and search for similar items?"
                }
            ]
        }
    ],
    tools=tools
)

msg = response.choices[0].message

# Print reasoning process
if msg.reasoning_content:
    print("=== Reasoning ===")
    print(msg.reasoning_content)

# Print response content
if msg.content:
    print("=== Content ===")
    print(msg.content)

# Print tool calls
if msg.tool_calls:
    print("=== Tool Calls ===")
    for tc in msg.tool_calls:
        print(f"  Function: {tc.function.name}")
        print(f"  Arguments: {tc.function.arguments}")
```

**Output Example:**

```text Output theme={null}
=== Reasoning ===
The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain.

I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items.
=== Content ===
Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR).

Let me search for this product and similar items for you.
=== Tool Calls ===
  Function: search_product
  Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"}
```

#### 4.2.5 Speculative Decoding

**NVIDIA**

Deploy Kimi-K2.6 with the following command (H200/B300, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path moonshotai/Kimi-K2.6 \
  --tp 8 \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --speculative-algorithm EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path lightseekorg/kimi-k2.6-eagle3.1-mla \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

Deploy Kimi-K2.6-NVFP4 with the following command (B300, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path nvidia/Kimi-K2.6-NVFP4 \
  --tp 8 \
  --quantization modelopt_fp4 \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --speculative-algorithm EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path lightseekorg/kimi-k2.6-eagle3.1-mla \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

For GB300, use `--tp 4`.

## 5. Benchmark

### 5.1 Accuracy Benchmark

**Test Environment:**

* Hardware: 8× NVIDIA H200
* Model: moonshotai/Kimi-K2.6 (INT4)
* Tensor Parallelism: 8
* SGLang version: 0.5.9
* Reasoning Parser: `kimi_k2`
* Tool Call Parser: `kimi_k2`

#### NVIDIA NVFP4 Accuracy Reference

NVIDIA reports the following accuracy results for `nvidia/Kimi-K2.6-NVFP4` compared with the native INT4 baseline, using `temperature=1.0`, `top_p=0.95`, and max tokens 128,000:

<table>
  <thead>
    <tr>
      <th>Precision</th>
      <th>GPQA Diamond</th>
      <th>SciCode</th>
      <th>τ²-Bench Telecom</th>
      <th>MMMU Pro</th>
      <th>AA-LCR</th>
      <th>IFBench</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Baseline (INT4)</td>
      <td>90.9</td>
      <td>52.6</td>
      <td>98.2</td>
      <td>75.6</td>
      <td>71.0</td>
      <td>73.9</td>
    </tr>

    <tr>
      <td>NVFP4</td>
      <td>90.4</td>
      <td>54.4</td>
      <td>98.0</td>
      <td>76.5</td>
      <td>71.8</td>
      <td>73.9</td>
    </tr>
  </tbody>
</table>

#### 5.1.1 K2-Vendor-Verifier (Tool Calling)

* Dataset: [K2-Vendor-Verifier](https://github.com/MoonshotAI/K2-Vendor-Verifier) tool-calls dataset (2,000 requests)
* Evaluation Tool: K2-Vendor-Verifier `tool_calls_eval.py`
* Settings: temperature=1.0, max\_tokens=64,000, concurrency=256

**Evaluation Command:**

```shell Command theme={null}
cd K2-Vendor-Verifier

python tool_calls_eval.py tool-calls/samples.jsonl \
  --model "moonshotai/Kimi-K2.6" \
  --base-url "http://localhost:30000/v1" \
  --api-key "placeholder" \
  --concurrency 256 \
  --temperature 1.0 \
  --max-tokens 64000 \
  --output kimi-k26-results.jsonl
```

**Results:**

<table>
  <thead>
    <tr>
      <th>Metric</th>
      <th>Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Success Rate</td>
      <td>99.95% (1999/2000)</td>
    </tr>

    <tr>
      <td>Tool Call Triggered</td>
      <td>970</td>
    </tr>

    <tr>
      <td>Tool Call Valid</td>
      <td>89.6% (869/970)</td>
    </tr>

    <tr>
      <td>Tool Call Invalid (schema error)</td>
      <td>10.4% (101/970)</td>
    </tr>
  </tbody>
</table>

#### 5.1.2 AIME 2025

* Dataset: [AIME 2025](https://huggingface.co/datasets/nvidia/aime25) (30 problems)
* Evaluation Tool: [sgl-eval](https://github.com/sgl-project/sgl-eval)
* Prompt: `eval/matharena/aime` (MathArena format with `\boxed{}` answers)
* Settings: temperature=1.0, top\_p=0.95, max\_tokens=131,072, 32 repeats

**Evaluation Command:**

```shell Command theme={null}
pip install sgl-eval

# The dataset ships with sgl-eval; --n-repeats runs all 32 samples per problem.
sgl-eval run aime25 \
  --base-url http://localhost:30000/v1 \
  --model moonshotai/Kimi-K2.6 \
  --n-repeats 32 \
  --temperature 1.0 \
  --top-p 0.95 \
  --max-tokens 131072 \
  --prompt matharena-aime \
  --num-threads 512
```

**Results:**

<table>
  <thead>
    <tr>
      <th>Evaluation Mode</th>
      <th>Accuracy</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>pass\@1 (avg-of-32)</td>
      <td>98.9% (29.7/30)</td>
    </tr>

    <tr>
      <td>**majority\@32**</td>
      <td>**100.0% (30/30)**</td>
    </tr>

    <tr>
      <td>pass\@32</td>
      <td>100.0%</td>
    </tr>
  </tbody>
</table>

> 22 out of 32 seeds achieved a perfect score of 30/30. The remaining 10 seeds each missed exactly 1 problem (29/30).

#### 5.1.3 GPQA Diamond

* Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions, 4-choice multiple choice)
* Evaluation Tool: [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) with `inspect_evals/gpqa_diamond`
* Settings: temperature=1.0, top\_p=0.95, max\_tokens=131,072, 4 epochs, cot=True

**Evaluation Command:**

```shell Command theme={null}
OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \
inspect eval inspect_evals/gpqa_diamond \
  --model openai/moonshotai/Kimi-K2.6 \
  --max-tokens 131072 \
  --temperature 1.0 \
  --top-p 0.95 \
  --max-connections 128 \
  -T cot=True
```

**Results (partial — 553/792 samples across 4 epochs):**

<table>
  <thead>
    <tr>
      <th>Evaluation Mode</th>
      <th>Accuracy</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>pass\@1 (avg across epochs)</td>
      <td>**96.9%**</td>
    </tr>
  </tbody>
</table>

<table>
  <thead>
    <tr>
      <th>Epoch</th>
      <th>Accuracy</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>1</td>
      <td>96.4% (160/166)</td>
    </tr>

    <tr>
      <td>2</td>
      <td>96.9% (156/161)</td>
    </tr>

    <tr>
      <td>3</td>
      <td>96.9% (155/160)</td>
    </tr>

    <tr>
      <td>4</td>
      <td>98.5% (65/66)</td>
    </tr>
  </tbody>
</table>

#### 5.1.4 OCRBench

* Dataset: [OCRBench](https://huggingface.co/datasets/echo840/OCRBench) (1,000 questions with images)
* Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based)
* Settings: max\_tokens=4,096, thinking mode enabled (opensource)

**Evaluation Command:**

```shell Command theme={null}
cd Kimi-Vendor-Verifier

OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \
python3 eval.py ocrbench \
  --model openai/moonshotai/Kimi-K2.6 \
  --max-tokens 4096 \
  --think-mode opensource \
  --thinking \
  --max-connections 256
```

**Results:**

<table>
  <thead>
    <tr>
      <th>Evaluation Mode</th>
      <th>Accuracy</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>pass\@1</td>
      <td>**90.8%**</td>
    </tr>
  </tbody>
</table>

#### 5.1.5 MMMU Pro Vision

* Dataset: [MMMU Pro](https://huggingface.co/datasets/MMMU/MMMU_Pro) standard 10-option subset (1,730 questions with images)
* Evaluation Tool: [Kimi-Vendor-Verifier](https://github.com/MoonshotAI/Kimi-Vendor-Verifier) (inspect-ai based)
* Settings: max\_tokens=32,768, thinking mode (default), max\_connections=256

> **Important**: Kimi-K2.6 is a reasoning model. Setting `max_tokens` too low (e.g., 4096) causes the thinking process to consume the entire token budget, leaving no tokens for the final answer. Use `max_tokens=32768` or higher.

**Evaluation Command:**

```shell Command theme={null}
cd Kimi-Vendor-Verifier

OPENAI_BASE_URL=http://localhost:30000/v1 OPENAI_API_KEY=placeholder \
python3 eval.py mmmu \
  --model openai/moonshotai/Kimi-K2.6 \
  --max-tokens 32768 \
  --think-mode none \
  --max-connections 256
```

**Results (1,481/1,730 samples completed):**

<table>
  <thead>
    <tr>
      <th>Evaluation Mode</th>
      <th>Accuracy</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>pass\@1</td>
      <td>**82.2%**</td>
    </tr>
  </tbody>
</table>

### 5.2 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA H200 GPU (8x)
* Model: Kimi-K2.6
* Tensor Parallelism: 8
* SGLang Version: 0.5.9

> [!NOTE]
>
> Kimi-K2.6 shares the same architecture as K2.5. Speed benchmarks are expected to be equivalent. The results below are measured with K2.5 and serve as a reference.

We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation.

#### 5.2.1 Latency Benchmark

* **Model Deployment:**

```shell Command theme={null}
sglang serve \
  --model-path moonshotai/Kimi-K2.6 \
  --tp 8 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

**Scenario 1: Chat (1K/1K)**

* Low Concurrency

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  39.77
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4221
Request throughput (req/s):              0.25
Input token throughput (tok/s):          153.40
Output token throughput (tok/s):         106.10
Peak output token throughput (tok/s):    156.00
Peak concurrent requests:                2
Total token throughput (tok/s):          259.50
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3972.87
Median E2E Latency (ms):                 4044.55
P90 E2E Latency (ms):                    7046.30
P99 E2E Latency (ms):                    7441.13
---------------Time to First Token----------------
Mean TTFT (ms):                          176.89
Median TTFT (ms):                        154.24
P99 TTFT (ms):                           285.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.22
Median TPOT (ms):                        9.32
P99 TPOT (ms):                           12.72
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.02
Median ITL (ms):                         8.80
P95 ITL (ms):                            13.23
P99 ITL (ms):                            14.17
Max ITL (ms):                            29.38
==================================================
```

* Medium Concurrency (Balanced)

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  158.05
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40775
Request throughput (req/s):              0.51
Input token throughput (tok/s):          250.99
Output token throughput (tok/s):         258.18
Peak output token throughput (tok/s):    1103.00
Peak concurrent requests:                19
Total token throughput (tok/s):          509.17
Concurrency:                             14.09
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   27837.05
Median E2E Latency (ms):                 23508.00
P90 E2E Latency (ms):                    57126.31
P99 E2E Latency (ms):                    66044.35
---------------Time to First Token----------------
Mean TTFT (ms):                          374.30
Median TTFT (ms):                        375.51
P99 TTFT (ms):                           695.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          53.25
Median TPOT (ms):                        57.93
P99 TPOT (ms):                           85.45
---------------Inter-Token Latency----------------
Mean ITL (ms):                           53.95
Median ITL (ms):                         53.97
P95 ITL (ms):                            84.74
P99 ITL (ms):                            244.84
Max ITL (ms):                            655.61
==================================================
```

* High Concurrency (Throughput-Optimized)

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  996.64
Total input tokens:                      249831
Total input text tokens:                 249831
Total generated tokens:                  252662
Total generated tokens (retokenized):    252588
Request throughput (req/s):              0.50
Input token throughput (tok/s):          250.67
Output token throughput (tok/s):         253.51
Peak output token throughput (tok/s):    1199.00
Peak concurrent requests:                104
Total token throughput (tok/s):          504.18
Concurrency:                             92.70
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   184773.75
Median E2E Latency (ms):                 174183.65
P90 E2E Latency (ms):                    343625.28
P99 E2E Latency (ms):                    404284.53
---------------Time to First Token----------------
Mean TTFT (ms):                          1289.59
Median TTFT (ms):                        1313.35
P99 TTFT (ms):                           2346.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          364.70
Median TPOT (ms):                        403.32
P99 TPOT (ms):                           452.34
---------------Inter-Token Latency----------------
Mean ITL (ms):                           363.82
Median ITL (ms):                         316.21
P95 ITL (ms):                            745.91
P99 ITL (ms):                            1345.88
Max ITL (ms):                            3118.59
==================================================
```

**Scenario 2: Reasoning (1K/8K)**

* Low Concurrency

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  680.26
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  44462
Total generated tokens (retokenized):    44455
Request throughput (req/s):              0.01
Input token throughput (tok/s):          8.97
Output token throughput (tok/s):         65.36
Peak output token throughput (tok/s):    151.00
Peak concurrent requests:                2
Total token throughput (tok/s):          74.33
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   68019.29
Median E2E Latency (ms):                 70568.85
P90 E2E Latency (ms):                    113237.40
P99 E2E Latency (ms):                    121682.34
---------------Time to First Token----------------
Mean TTFT (ms):                          206.17
Median TTFT (ms):                        177.28
P99 TTFT (ms):                           445.37
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.36
Median TPOT (ms):                        15.89
P99 TPOT (ms):                           16.43
---------------Inter-Token Latency----------------
Mean ITL (ms):                           15.26
Median ITL (ms):                         15.85
P95 ITL (ms):                            17.50
P99 ITL (ms):                            23.21
Max ITL (ms):                            45.22
==================================================
```

* Medium Concurrency

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  2475.98
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  318306
Total generated tokens (retokenized):    318166
Request throughput (req/s):              0.03
Input token throughput (tok/s):          16.02
Output token throughput (tok/s):         128.56
Peak output token throughput (tok/s):    847.00
Peak concurrent requests:                18
Total token throughput (tok/s):          144.58
Concurrency:                             14.62
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   452592.46
Median E2E Latency (ms):                 486002.05
P90 E2E Latency (ms):                    833197.57
P99 E2E Latency (ms):                    957399.48
---------------Time to First Token----------------
Mean TTFT (ms):                          359.38
Median TTFT (ms):                        350.78
P99 TTFT (ms):                           500.36
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          111.18
Median TPOT (ms):                        122.76
P99 TPOT (ms):                           145.90
---------------Inter-Token Latency----------------
Mean ITL (ms):                           113.69
Median ITL (ms):                         122.81
P95 ITL (ms):                            147.87
P99 ITL (ms):                            151.03
Max ITL (ms):                            272.05
==================================================
```

**Scenario 3: Summarization (8K/1K)**

* Low Concurrency

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  120.73
Total input tokens:                      41941
Total input text tokens:                 41941
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.08
Input token throughput (tok/s):          347.41
Output token throughput (tok/s):         34.96
Peak output token throughput (tok/s):    73.00
Peak concurrent requests:                2
Total token throughput (tok/s):          382.36
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   12068.56
Median E2E Latency (ms):                 10211.36
P90 E2E Latency (ms):                    23203.32
P99 E2E Latency (ms):                    30677.66
---------------Time to First Token----------------
Mean TTFT (ms):                          1625.64
Median TTFT (ms):                        1526.63
P99 TTFT (ms):                           3743.51
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.95
Median TPOT (ms):                        23.95
P99 TPOT (ms):                           35.40
---------------Inter-Token Latency----------------
Mean ITL (ms):                           24.80
Median ITL (ms):                         21.73
P95 ITL (ms):                            59.56
P99 ITL (ms):                            61.10
Max ITL (ms):                            62.70
==================================================
```

* Medium Concurrency

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  389.96
Total input tokens:                      300020
Total input text tokens:                 300020
Total generated tokens:                  41669
Total generated tokens (retokenized):    41670
Request throughput (req/s):              0.21
Input token throughput (tok/s):          769.36
Output token throughput (tok/s):         106.86
Peak output token throughput (tok/s):    304.00
Peak concurrent requests:                19
Total token throughput (tok/s):          876.22
Concurrency:                             14.95
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   72870.97
Median E2E Latency (ms):                 70495.88
P90 E2E Latency (ms):                    121820.46
P99 E2E Latency (ms):                    148933.09
---------------Time to First Token----------------
Mean TTFT (ms):                          2460.45
Median TTFT (ms):                        1976.29
P99 TTFT (ms):                           7305.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          140.57
Median TPOT (ms):                        142.31
P99 TPOT (ms):                           273.40
---------------Inter-Token Latency----------------
Mean ITL (ms):                           135.44
Median ITL (ms):                         95.96
P95 ITL (ms):                            152.93
P99 ITL (ms):                            1488.37
Max ITL (ms):                            6540.24
==================================================
```

* High Concurrency

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  1279.50
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total generated tokens:                  170000
Total generated tokens (retokenized):    169981
Request throughput (req/s):              0.25
Input token throughput (tok/s):          995.62
Output token throughput (tok/s):         132.86
Peak output token throughput (tok/s):    703.00
Peak concurrent requests:                67
Total token throughput (tok/s):          1128.49
Concurrency:                             60.12
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   240385.63
Median E2E Latency (ms):                 236266.30
P90 E2E Latency (ms):                    429882.12
P99 E2E Latency (ms):                    515158.36
---------------Time to First Token----------------
Mean TTFT (ms):                          2710.44
Median TTFT (ms):                        2345.63
P99 TTFT (ms):                           7144.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          443.84
Median TPOT (ms):                        493.29
P99 TPOT (ms):                           606.19
---------------Inter-Token Latency----------------
Mean ITL (ms):                           448.23
Median ITL (ms):                         296.17
P95 ITL (ms):                            1869.15
P99 ITL (ms):                            2708.95
Max ITL (ms):                            7778.47
==================================================
```

### 5.3 Speed Benchmark (AMD MI350X)

**Test Environment:**

* Hardware: AMD Instinct MI350X GPU (4x)
* Model: Kimi-K2.6 (INT4)
* Tensor Parallelism: 4
* SGLang Version: 0.5.9
* Docker Image: `lmsysorg/sglang:v0.5.9-rocm700-mi35x`
* ROCm: 7.0

We use SGLang's built-in benchmarking tool with the `random` dataset for standardized performance evaluation.

> [!NOTE]
>
> **AMD GPU TP Constraint**: Kimi-K2.6 requires TP ≤ 4 on AMD GPUs. The model has 64 attention heads, and the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).

#### 5.3.1 Latency Benchmark

* **Model Deployment:**

```shell Command theme={null}
SGLANG_USE_AITER=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 \
sglang serve \
  --model-path moonshotai/Kimi-K2.6 \
  --tp 4 \
  --mem-fraction-static 0.8 \
  --trust-remote-code \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --kv-cache-dtype fp8_e4m3 \
  --host 0.0.0.0 \
  --port 30000
```

* **Benchmark Command:**

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* **Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  155.81
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4222
Request throughput (req/s):              0.06
Input token throughput (tok/s):          39.16
Output token throughput (tok/s):         27.09
Peak output token throughput (tok/s):    29.00
Peak concurrent requests:                2
Total token throughput (tok/s):          66.24
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   15576.22
Median E2E Latency (ms):                 12539.80
P90 E2E Latency (ms):                    28150.56
P99 E2E Latency (ms):                    34873.51
---------------Time to First Token----------------
Mean TTFT (ms):                          563.50
Median TTFT (ms):                        594.92
P99 TTFT (ms):                           830.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          35.61
Median TPOT (ms):                        35.66
P99 TPOT (ms):                           35.77
---------------Inter-Token Latency----------------
Mean ITL (ms):                           35.66
Median ITL (ms):                         35.69
P95 ITL (ms):                            35.96
P99 ITL (ms):                            36.13
Max ITL (ms):                            36.92
==================================================
```

* Medium Concurrency (Balanced)

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-K2.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  526.66
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40798
Request throughput (req/s):              0.15
Input token throughput (tok/s):          75.32
Output token throughput (tok/s):         77.48
Peak output token throughput (tok/s):    96.00
Peak concurrent requests:                18
Total token throughput (tok/s):          152.80
Concurrency:                             14.59
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   96023.27
Median E2E Latency (ms):                 93940.20
P90 E2E Latency (ms):                    159449.54
P99 E2E Latency (ms):                    194706.61
---------------Time to First Token----------------
Mean TTFT (ms):                          989.08
Median TTFT (ms):                        886.42
P99 TTFT (ms):                           1543.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          191.04
Median TPOT (ms):                        195.20
P99 TPOT (ms):                           238.84
---------------Inter-Token Latency----------------
Mean ITL (ms):                           186.68
Median ITL (ms):                         183.82
P95 ITL (ms):                            189.90
P99 ITL (ms):                            673.64
Max ITL (ms):                            1633.20
==================================================
```


# Kimi-K2.7-Code
Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K2.7-Code

Deploy Kimi-K2.7-Code with SGLang for coding-focused agentic workflows, thinking output, tool calling, and multimodal input.

## 1. Model Introduction

[Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code) is a coding-focused agentic model by Moonshot AI, built on top of Kimi-K2.6. It improves real-world long-horizon coding task completion while reducing thinking-token usage by approximately 30% compared with Kimi-K2.6.

**Key Features:**

* **Coding-Focused Agentic Model**: Optimized for end-to-end coding workflows and complex software engineering tasks.
* **Token Efficiency**: Reduces thinking-token usage by approximately 30% versus Kimi-K2.6.
* **K2.6-Compatible Deployment**: Shares the same architecture as Kimi-K2.5/Kimi-K2.6, so the SGLang deployment method can be reused with the new model ID.
* **Native Multimodality**: Shares Kimi-K2.6's native multimodal architecture with a MoonViT vision encoder (400M parameters) and supports image and video (experimental) input.

**Benchmarks:**

<table>
  <thead>
    <tr>
      <th>Benchmark</th>
      <th>Kimi-K2.6</th>
      <th>Kimi-K2.7-Code</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Kimi Code Bench v2</td>
      <td>50.9</td>
      <td>62.0</td>
    </tr>

    <tr>
      <td>Program Bench</td>
      <td>48.3</td>
      <td>53.6</td>
    </tr>

    <tr>
      <td>MLS Bench Lite</td>
      <td>26.7</td>
      <td>35.1</td>
    </tr>

    <tr>
      <td>Kimi Claw 24/7 Bench</td>
      <td>42.9</td>
      <td>46.9</td>
    </tr>

    <tr>
      <td>MCP Atlas</td>
      <td>69.4</td>
      <td>76.0</td>
    </tr>

    <tr>
      <td>MCP Mark Verified</td>
      <td>72.8</td>
      <td>81.1</td>
    </tr>
  </tbody>
</table>

**Recommended Generation Parameters:**

* Thinking Mode: `temperature=1.0`, `top_p=0.95`
* Kimi-K2.7-Code forces thinking and preserve-thinking behavior; instant mode is not supported.

**Available Models:**

* **INT4 (native checkpoint)**: [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code)
* **MXFP4**: [amd/Kimi-K2.7-Code-MXFP4](https://huggingface.co/amd/Kimi-K2.7-Code-MXFP4) — validated on AMD MI350X/MI355X.

**License:** Modified MIT for the native checkpoint.

For details, see the [official model card](https://huggingface.co/moonshotai/Kimi-K2.7-Code).

## 2. SGLang Installation

Refer to the [official SGLang installation guide](/docs/get-started/install).

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, deployment strategy, and capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory**: Requires GPUs with ≥140GB each. The native INT4 checkpoint supports H200 (8×, TP=8), B300 (8×, TP=8), GB300 (4×, TP=4), MI300X/MI325X (4×, TP=4), and MI350X/MI355X (4×, TP=4). Use `--context-length 128000` to conserve memory.
* **Context Length**: The model supports a 256K context length. Use a shorter `--context-length` when you need to reserve memory for larger batches.
* **Transformers Version**: The model card requires `transformers>=4.57.1,<5.0.0`.
* **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.7-Code has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
* **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X.
* **DP Attention**: Enable with `--attn-dp-size <N>` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required.
* **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
* **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
* **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.
* **MXFP4 Checkpoint**: The [MXFP4 checkpoint](https://huggingface.co/amd/Kimi-K2.7-Code-MXFP4) is validated on MI350X/MI355X (TP=4) with the `rocm/sgl-dev:v0.5.17-rocm720-mi35x-20260812` image (ROCm 7.2). Select `MXFP4` in the Quantization control to target `amd/Kimi-K2.7-Code-MXFP4`. The MXFP4 quantization is auto-detected from the checkpoint, so no `--quantization` flag is required; the generator emits the AITER/ROCm environment block plus `--attention-backend aiter`, `--mem-fraction-static 0.90`, FP8 KV cache, `--disable-radix-cache`, and `--enable-aiter-allreduce-fusion`. The reasoning and tool-call parsers are auto-detected, so they are not passed explicitly.

## 4. Model Invocation

### 4.1 Basic Usage

See [Basic API Usage](/docs/basic_usage/send_request).

### 4.2 Advanced Usage

#### 4.2.1 Multimodal (Vision + Text) Input

Kimi-K2.7-Code supports native multimodal input with images:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "What is in this image? Describe it in detail."
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
This image shows a **paper receipt from Auntie Anne's**, the pretzel chain restaurant. Here's a detailed breakdown:

## Header
- At the top left is the Auntie Anne's logo (a pretzel with a halo)
- The store name "**Auntie Anne's**" is printed prominently at the top
- Some text below the store name appears blurred/redacted (likely store location, address, or transaction details)

## Purchase Details
- **Item**: CINNAMON SUGAR
- **Quantity & Price**: 1 × 17,000
- **Item Total**: 17,000

## Financial Summary
- **SUB TOTAL**: 17,000
- **GRAND TOTAL**: 17,000
- **CASH IDR**: 20,000 (customer paid 20,000 Indonesian Rupiah)
- **CHANGE DUE**: 3,000

## Physical Description
- The receipt is printed on white thermal paper
- Some information in the middle section and toward the bottom is intentionally blurred/obscured
- The paper appears slightly curved/wrinkled and is placed on a dark brown surface (likely a table or counter)

The transaction is in **Indonesian Rupiah (IDR)**, indicating this purchase was made at an Auntie Anne's location in Indonesia. The customer bought one Cinnamon Sugar pretzel for 17,000 IDR and received 3,000 IDR in change after paying with 20,000 IDR cash.
```

#### 4.2.2 Reasoning Output

Kimi-K2.7-Code forces thinking mode and preserve-thinking behavior.

**Thinking Mode (default)** — reasoning content is automatically separated:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {"role": "user", "content": "Which one is bigger, 9.11 or 9.9? Think carefully."}
    ]
)

print("====== Reasoning Content (Thinking Mode) ======")
print(response.choices[0].message.reasoning_content)
print("====== Response (Thinking Mode) ======")
print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
====== Reasoning Content (Thinking Mode) ======
The user is asking which number is bigger: 9.11 or 9.9. This seems straightforward, but there's a viral internet debate about this due to decimal confusion.

Let me think carefully:
- 9.11 means 9 + 11/100 = 9.11
- 9.9 means 9 + 9/10 = 9.90

So 9.9 = 9.90, and 9.90 > 9.11 because 0.90 > 0.11.

The confusion often comes from people thinking of software versioning (where 9.11 comes after 9.9) or comparing the numbers after the decimal as whole numbers (11 vs 9, thinking 11 > 9).

So mathematically, 9.9 is clearly bigger. 9.9 - 9.11 = 0.79.

I should explain this clearly and address the common misconception.
====== Response (Thinking Mode) ======
Mathematically, **9.9 is bigger**.

Here's why:

**9.9 = 9.90**

When comparing decimals, you need to look at the same place values:
- 9.11 = 9 ones, 1 tenth, and 1 hundredth
- 9.9 = 9 ones, 9 tenths, and 0 hundredths (9.90)

Since **0.90 > 0.11**, it follows that **9.9 > 9.11**.

The difference is:
9.9 - 9.11 = 0.79

**Why people get confused:** Many mistakenly treat the decimals like whole numbers (thinking "11 is bigger than 9") or confuse this with software version numbering (where version 9.11 comes after version 9.9). But in standard mathematics, 9.9 is definitively larger.
```

#### 4.2.3 Preserve Thinking

Kimi-K2.7-Code keeps reasoning content across multi-turn interactions. This behavior is enabled by default and cannot be disabled.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

messages = [
    {
        "role": "user",
        "content": "Tell me three random numbers."
    },
    {
        "role": "assistant",
        "reasoning_content": "I'll start by listing five numbers: 473, 921, 235, 215, 222, and I'll tell you the first three.",
        "content": "473, 921, 235"
    },
    {
        "role": "user",
        "content": "What are the other two numbers you have in mind?"
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=messages,
    stream=False,
    max_tokens=4096,
)

print(response.choices[0].message.content)
```

Some OpenAI-compatible deployments use `reasoning` instead of `reasoning_content` in assistant messages. Use the field your serving stack exposes.

#### 4.2.4 Tool Calling

Kimi-K2.7-Code supports tool calling capabilities for agentic tasks:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

# Process streaming response
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {'name': None, 'arguments': ''}
                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        if delta.content:
            print(delta.content, end="", flush=True)

for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"  Arguments: {tool_call['arguments']}")
```

**Output Example:**

```text Output theme={null}
Tool Call: get_weather
  Arguments: {"location": "Beijing"}
```

**Handling Tool Call Results:**

```python Example theme={null}
# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": "The weather in Beijing is 22°C and sunny."
    }
]

final_response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=messages
)

print(final_response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
The weather in Beijing is currently **22°C and sunny**. ☀️

It's a nice, warm day there—great for being outdoors!
```

#### 4.2.5 Multimodal + Tool Calling (Agentic Vision)

Combine vision understanding with tool calling for advanced agentic tasks:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_product",
            "description": "Search for a product by name or description",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The product name or description to search for"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.7-Code",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                    }
                },
                {
                    "type": "text",
                    "text": "Can you identify this product and search for similar items?"
                }
            ]
        }
    ],
    tools=tools
)

msg = response.choices[0].message

# Print reasoning process
if msg.reasoning_content:
    print("=== Reasoning ===")
    print(msg.reasoning_content)

# Print response content
if msg.content:
    print("=== Content ===")
    print(msg.content)

# Print tool calls
if msg.tool_calls:
    print("=== Tool Calls ===")
    for tc in msg.tool_calls:
        print(f"  Function: {tc.function.name}")
        print(f"  Arguments: {tc.function.arguments}")
```

**Output Example:**

```text Output theme={null}
=== Reasoning ===
The user wants me to identify the product from the receipt and search for similar items. Looking at the receipt, it's from Auntie Anne's and the item purchased is "CINNAMON SUGAR" for 17,000 IDR. This is likely a Cinnamon Sugar Pretzel from Auntie Anne's, which is a popular pretzel chain.

I should search for this product using the search_product function. The query should be something like "Auntie Anne's Cinnamon Sugar Pretzel" or just "Cinnamon Sugar Pretzel" to find similar items.
=== Content ===
Based on the receipt, the product is a **Cinnamon Sugar Pretzel** from **Auntie Anne's** (a popular pretzel bakery chain). The receipt shows it was purchased for 17,000 Indonesian Rupiah (IDR).

Let me search for this product and similar items for you.
=== Tool Calls ===
  Function: search_product
  Arguments: {"query":"Auntie Anne's Cinnamon Sugar Pretzel"}
```

#### 4.2.6 Deployment Command Example

Deploy Kimi-K2.7-Code with the following command (H200/B300, reasoning and tool parsing enabled):

```shell Command theme={null}
sglang serve \
  --model-path moonshotai/Kimi-K2.7-Code \
  --tp 8 \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --trust-remote-code \
  --host 0.0.0.0 \
  --port 30000
```

For GB300, use `--tp 4`.

Deploy the MXFP4 checkpoint on AMD MI350X/MI355X (reasoning and tool-call parsers are auto-detected from the checkpoint):

```shell Command theme={null}
SGLANG_USE_AITER=1 \
HIP_FORCE_DEV_KERNARG=1 \
SGLANG_EXPERT_PARALLEL_SIZE=1 \
TORCH_BLAS_PREFER_HIPBLASLT=1 \
TENSILE_STREAMK_DYNAMIC_GRID=6 \
AITER_QUICK_REDUCE_QUANTIZATION=INT4 \
AITER_USE_FLYDSL_MOE_SORTING=1 \
AITER_AR_1STAGE_MAX_KB=512 \
AITER_MXFP4_INTERMEDIATE=1 \
ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
sglang serve \
  --model-path amd/Kimi-K2.7-Code-MXFP4 \
  --tp 4 \
  --trust-remote-code \
  --attention-backend aiter \
  --mem-fraction-static 0.90 \
  --kv-cache-dtype fp8_e4m3 \
  --disable-radix-cache \
  --enable-aiter-allreduce-fusion \
  --host 0.0.0.0 \
  --port 30000
```

## 5. Benchmark

### 5.1 Model Card Benchmarks

The following results are from the official Kimi-K2.7-Code model card. They were evaluated with thinking mode enabled through Kimi Code CLI at `temperature=1.0`, `top_p=0.95`, and a 262,144-token context length unless otherwise stated.

<table>
  <thead>
    <tr>
      <th>Category</th>
      <th>Benchmark</th>
      <th>Kimi-K2.6</th>
      <th>Kimi-K2.7-Code</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Coding</td>
      <td>Kimi Code Bench v2</td>
      <td>50.9</td>
      <td>62.0</td>
    </tr>

    <tr>
      <td>Coding</td>
      <td>Program Bench</td>
      <td>48.3</td>
      <td>53.6</td>
    </tr>

    <tr>
      <td>Coding</td>
      <td>MLS Bench Lite</td>
      <td>26.7</td>
      <td>35.1</td>
    </tr>

    <tr>
      <td>Agentic</td>
      <td>Kimi Claw 24/7 Bench</td>
      <td>42.9</td>
      <td>46.9</td>
    </tr>

    <tr>
      <td>Agentic</td>
      <td>MCP Atlas</td>
      <td>69.4</td>
      <td>76.0</td>
    </tr>

    <tr>
      <td>Agentic</td>
      <td>MCP Mark Verified</td>
      <td>72.8</td>
      <td>81.1</td>
    </tr>
  </tbody>
</table>

### 5.2 MXFP4 Benchmark

**Test Environment:**

* Hardware: 4× AMD MI355X
* Model: `amd/Kimi-K2.7-Code-MXFP4` (MXFP4)
* Tensor Parallelism: 4
* Docker Image: `rocm/sgl-dev:v0.5.17-rocm720-mi35x-20260812` (ROCm 7.2)

#### 5.2.1 Accuracy Benchmark

Evaluate GSM8K on the deployed MXFP4 checkpoint across client parallelism levels (all 1319 test questions via sgl-eval):

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319 \
  --num-threads '<PARALLEL>'
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

<table>
  <thead>
    <tr>
      <th>Parallel</th>
      <th>GSM8K Accuracy</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>100</td>
      <td>0.951</td>
    </tr>

    <tr>
      <td>300</td>
      <td>0.953</td>
    </tr>

    <tr>
      <td>600</td>
      <td>0.942</td>
    </tr>

    <tr>
      <td>1200</td>
      <td>0.950</td>
    </tr>
  </tbody>
</table>

#### 5.2.2 Latency Benchmark

We use SGLang's built-in `bench_serving` tool with the `random` dataset (input 8192, output 1024, range ratio 0.8). For each concurrency `C`, `--num-prompts` is set to `C × 5`.

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model amd/Kimi-K2.7-Code-MXFP4 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --random-range-ratio 0.8 \
  --max-concurrency <C> \
  --num-prompts <C x 5> \
  --host 0.0.0.0 \
  --port 30000
```

<table>
  <thead>
    <tr>
      <th>Concurrency</th>
      <th>Output Throughput (tok/s)</th>
      <th>Total Throughput (tok/s)</th>
      <th>Mean TTFT (ms)</th>
      <th>Median TTFT (ms)</th>
      <th>Mean TPOT (ms)</th>
      <th>Mean E2E Latency (ms)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>4</td>
      <td>416.22</td>
      <td>3747.97</td>
      <td>260.35</td>
      <td>194.14</td>
      <td>9.03</td>
      <td>8720.79</td>
    </tr>

    <tr>
      <td>8</td>
      <td>670.88</td>
      <td>6086.74</td>
      <td>364.15</td>
      <td>199.00</td>
      <td>11.19</td>
      <td>10687.52</td>
    </tr>

    <tr>
      <td>16</td>
      <td>963.47</td>
      <td>8766.11</td>
      <td>490.72</td>
      <td>203.12</td>
      <td>15.49</td>
      <td>14680.68</td>
    </tr>

    <tr>
      <td>32</td>
      <td>1329.55</td>
      <td>11857.47</td>
      <td>775.37</td>
      <td>215.45</td>
      <td>22.61</td>
      <td>21831.61</td>
    </tr>

    <tr>
      <td>64</td>
      <td>1740.46</td>
      <td>15759.18</td>
      <td>1343.98</td>
      <td>300.81</td>
      <td>34.41</td>
      <td>33068.86</td>
    </tr>
  </tbody>
</table>


# Kimi-K3
Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3

Deploy Moonshot AI's Kimi-K3 with SGLang — a 2.8T-parameter hybrid Mixture-of-Experts vision-language model (Kimi Delta Attention + MLA, 16/896 active experts) with NVIDIA, AMD, and NPU recipes.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest                                  # NVIDIA (CUDA)
> docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260916     # AMD MI350X / MI355X (ROCm)
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.
>
> **NPU**
>
> ```bash Command theme={null}
> docker pull quay.io/ascend/sglang:main-cann9.1.0-a3   # Ascend A3 Series
> docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-B070   # Ascend 950PR/DT Series
> ```
>
> For host and platform setup, see the
> [NPU installation guide](../../../docs/hardware-platforms/ascend-npus/getting-started/installation) and the
> [quick start guide](../../../docs/hardware-platforms/ascend-npus/getting-started/quick_start).
>
> **Weights (NPU):** [Kimi-K3](https://modelscope.cn/models/moonshotai/Kimi-K3) (950PR/DT Series, MXFP4) ·
> [Kimi-K3-W4A8](https://www.modelscope.cn/models/sgl-npu/Kimi-K3-W4A8) (A3 Series, W4A8, 1.49 TB) ·
> [Kimi-K3-DSpark](https://www.modelscope.cn/models/RadixArk/Kimi-K3-DSpark) (DSPARK draft, 4.5 GB)

Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8, Ascend A3 Series 4×8 — 32 cards / 64 dies, Ascend 950PR/DT Series 4×8), so it is not a separate choice. If you serve the NVFP4 checkpoint (`nvidia/Kimi-K3-NVFP4`, the **Quantization** row in the panel below), use the `lmsysorg/sglang:dev-dev-kimi-k3-nvfp4` image.

**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)).

**Ascend NPU PD** — the A3 Series splits over the Ascend memory fabric, with `Default` as the only prefill strategy: an asymmetric **2P4D** (TP16 × PP2 × DP2 prefill, NOSPEC; TP64 × DP4 decode). Pool layout and the fabric env: [On Ascend NPUs](#npu-pd).

**Strategy** — the operating point within that shape:

* **Low-Latency** — no DCP, so the MLA KV stays TP-replicated. For chat. B200 splits its two nodes into PP2 × TP8; every other platform is flat TP.
* **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8/DCP8 ROCm/AITER on MI35x.
* **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). The cell itself is Balanced, except on H100 (plus `extra_buffer_lazy`) and H200 (widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`).

`Long-Context` appears only under the `Prefill` PD mode; for long-context unified serving on B200, start from High-Throughput and raise `--context-length`.

**Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1` on the shipped recipes, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH uses the [Kimi-K3-DFlash](https://huggingface.co/modal-labs/Kimi-K3-DFlash) draft at block size 8; it doesn't support pipeline parallelism or DP attention yet, so it is unavailable on the pipelined B200 Unified recipes. It has only been validated on Blackwell, so the Hopper and AMD recipes don't offer it yet. The win is largest on short interactive traffic and fades as the prompt grows.

> [!NOTE]
>
> `--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. (The Ascend NPU recipes don't use the calculator and set no ratio at all — the Ascend path sizes the KDA state pool itself: the A3 Series Unified cell pins `--max-mamba-cache-size` and its PD roles size the pool from `--max-running-requests` instead, while the A5 recipe caps the paged KV pool with `--max-total-tokens` and serves with the radix cache off.)

> [!NOTE]
>
> B300 1×8 Unified speed numbers are measured on `v0.5.18 @ 71de97b2` with `--random-range-ratio 1.0`,
> `--warmup-requests 64`, `--flush-cache`, at ISL 8192 / OSL 1024. DSPARK cells pin the acceptance
> length via the serve env `SGLANG_SIMULATE_ACC_LEN=4.5` — they report what block size 7 delivers at
> that acceptance, not a measured acceptance rate for this workload. Balanced DSPARK adds
> `--max-running-requests 256`; without it speculation resets the cap to 48. The KDA state pool still
> clamps admission below that (101 / 68 / 91 / 60 concurrent requests for MXFP4 NOSPEC / MXFP4 DSPARK /
> NVFP4 NOSPEC / NVFP4 DSPARK), which is why no point past concurrency 64 is published for Balanced.

### AMD AITER with DCP8

The MI350X/MI355X unified Balanced recipe uses TP8/DCP8 with AITER prefill and
decode attention. DCP shards the target MLA KV cache; RadixArk DSPARK's draft KV
remains replicated. The pinned `v0.5.19-rocm720-mi35x-20260916` image records
SGLang revision `e7f7447333`, which includes
[AITER DCP support (#34432)](https://github.com/sgl-project/sglang/pull/34432) and
the [DCP KV-free fix (#38941)](https://github.com/sgl-project/sglang/pull/38941).
No source overlay is required for DCP.

Keep `SGLANG_ROCM_K3_KDA_FUSED_BACKEND` unset with this image. The separate fused-KDA
opt-in requires the [deferred-gate fix (#39066)](https://github.com/sgl-project/sglang/pull/39066),
which is not included in this image. This updated recipe remains **Final
Verification In Progress**; the recorded speed numbers use their original
configurations and do not validate the new image or DCP8 recipe.

### Mamba ratio calculator

> **How --mamba-full-memory-ratio is calculated**
>
> `--mamba-full-memory-ratio` is the ratio between the KDA state pool and the MLA KV pool. Every parameter below except `L` is read live from the Deploy panel and Playground selection; the balanced value is the per-request cost ratio:
>
> ```text theme={null}
> ratio = (S + D) x state_bytes / (L x (mla_kv_bytes / DCP + draft_kv_bytes))
> ```
>
> * `S` — KDA state slots per request: `extra_buffer=5`, `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK` frees one slot on the extra-buffer strategies; with the overlap scheduler off (or `pp > 1`, which disables it) the track buffer costs one slot instead of two.
> * `D` — verify intermediate states under speculative decoding: `0` when disabled, otherwise DSPARK block size + 1 (`8` at the default 7). ReplaySSM (`--enable-linear-replayssm-spec`) folds them into a per-slot ring, returning `D` to `0`.
> * `state_bytes` — one state slot's bytes, from K3's fixed geometry, the attention-TP width, and the SSM dtype.
> * `mla_kv_bytes` — one token's MLA latent KV bytes (KV-dtype dependent); DCP shards it across its ranks. The DSPARK draft model's KV (\~1.4 KB per token) is replicated on every rank, so it enters flat — negligible without DCP, the same order as the sharded MLA share under DCP8.
> * `L` — average total request length in tokens: input + output.

## Advanced Features Playground

The Playground is where you experiment with **SGLang features beyond the deployment matrix**. The Deploy panel above emits the recipes the SGLang team is converging on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

**Kimi-K3** is Moonshot AI's flagship hybrid MoE vision-language model: **2.8 trillion parameters**, **16 of 896 experts** active per token, roughly **2.5× the scaling efficiency of Kimi-K2**. The backbone interleaves **Kimi Delta Attention (KDA)** with MLA across 93 layers (plus Attention Residuals and Stable LatentMoE); serving supports image input and a **1M-token** window with prefix caching. Weights ship in **MXFP4**: the FlashInfer MXFP4 (trtllm-gen SiTU) runner serves them on Blackwell, Marlin (W4A16) elsewhere, MegaMoE for short-context batch throughput.

K3 **always runs with thinking enabled**, with reasoning depth controlled by `reasoning_effort` (`low` / `high` / `max`; default `max`).

> [!NOTE]
>
> Kimi-K3 is Moonshot AI's first open-source model in the trillion-plus class; **full model weights
> are scheduled to release by July 27, 2026**. The recipes on this page were validated on the public
> [`sgl-project/sglang` `kimi-k3` branch](https://github.com/sgl-project/sglang/tree/kimi-k3) — the HuggingFace
> repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be
> available at launch.
>
> The B300 1×8 `Unified` Low-Latency and Balanced cells are **Verified** — a speed round on the final
> weights is published below. Every other cell is still marked **Final Verification In Progress**: the
> recipe runs, but its serving round on the final weights and current code is still open. Accuracy has
> not been re-measured on any cell — re-measure before you rely on one.

**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code).

**Resources:** [HuggingFace](https://huggingface.co/moonshotai/Kimi-K3) · [Kimi-K3 Quickstart](https://platform.kimi.ai/docs/guide/kimi-k3-quickstart).

## 2. Configuration Tips

**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. (On the Ascend NPU recipes there is no calculator and no ratio flag: the Ascend path sizes the KDA state pool itself — the A3 Series pins `--max-mamba-cache-size`, and the A5 recipe caps the paged KV pool with `--max-total-tokens` and runs the radix cache off.) After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side).

Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload:

| Lever | Effect |
| - | - |
| `--mamba-radix-cache-strategy extra_buffer_lazy` | 4 state slots per request instead of 5 |
| `--mamba-ssm-dtype bfloat16` | \~halves state bytes; with spec on, KDA verification falls back from the fused kernel to Triton |
| `--kv-cache-dtype fp8_e4m3` | halves KV bytes per token; under PD both roles must match at connect |
| `--mem-fraction-static` 0.90–0.92 | cheapest first win when the boot log shows a large idle `avail mem` |
| `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` | frees one more slot per request (experimental, under validation) |

Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise).

**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, official trtllm-gen SiTU kernels) is selected; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on.

**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode.

**Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV.

**DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. The published B300 DSPARK numbers pin the acceptance length with `SGLANG_SIMULATE_ACC_LEN`, so no measured acceptance rate exists for a real workload yet — measure against the same recipe running NOSPEC before adopting.

**Per-platform notes:**

| Platform | Topology | Notes |
| - | - | - |
| B300 1×8 | TP8 (+DCP8) | accuracy-first defaults on Low-Latency and Balanced |
| GB300 2×4 | TP8/DCP8 | MNNVL transport and cuMem auto-detected |
| B200 2×8 | PP2 × TP8 on Low-Latency, PP2 × DCPEP8 on Balanced and High-Throughput. DSPARK re-lays the same 16 GPUs as TP16 / TP16+DCP16+EP16. PD prefill is TP1 × PP16 | Unified serves all three operating points; `Long-Context` is a `Prefill`-only strategy |
| GB200 4×4 | TP16/DCP16 | MNNVL auto-detected |
| H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` |
| H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) |
| MI350X/MI355X 1×8 | TP8/DCP8 ROCm/AITER (Unified Balanced) | AITER A8W4 FlyDSL MoE, AITER prefill/decode attention with sharded target MLA KV, graph bs up to 256, fp8 kvcache; DSPARK supported. Activation-quant and fused-KDA-decode knobs: [AMD ROCm/AITER environment](#amd-env) |
| Ascend A3 Series 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | `Unified` plus an asymmetric 2P4D PD split — 2-node prefill (TP16/PP2/DP2, NOSPEC, no CUDA graph) + 4-node decode (TP64/DP4, DSPARK); pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node |
| Ascend 950PR/DT Series 4×8 | TP32/dp1 + DeepEP | `Unified` only; DSPARK baked in; shared experts / dense MLP shard over attention-TP (`--shared-experts-tp-size 4`); radix cache off; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node |

**Blackwell DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles:

* DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it.
* Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300).
* No `--enable-symm-mem` under DCP (force-disabled for decode-graph correctness).
* Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context.
* Don't use EP with an a2a backend: a2a buffers reclaim the KV that DCP buys. Compose only to measure. a2a backend is set when `--moe-a2a-backend` is set.

Outside the two verified B300 1×8 `Unified` cells, no cell has a serving round in this exact shape — treat those as starting points to verify.

**AMD ROCm/AITER environment (MI350X / MI355X).** The MI35x cell emits `SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1` — the first three turn on the AITER ROCm path, its K3-specific fused kernels, and the FlyDSL MoE kernels; the rest of this table is what you can change on top. Everything here is gfx950/ROCm-only and inert elsewhere. These knobs first ship in the `20260903` daily ROCm image pinned above — on an older image they are simply unread — and the two SiTU rows also need an AITER revision at or past [ROCm/aiter#4534](https://github.com/ROCm/aiter/pull/4534) (FlyDSL 0.3.0).

| Env var | Default | Effect |
| - | - | - |
| `AITER_SITUV2_A8W4=1` | unset | SiTU v2 MoE with A8W4 activation quantization, on AITER's GU-interleaved preshuffled weight layout. The performance default the cell ships. |
| `AITER_SITUV2_A4W4=1` | unset | A4W4 instead, on the generic separated shuffle layout. Numerically correct but slower than A8W4 (530.8 vs 537.3 tok/s median output on 8×MI35x). Setting **both** gives A8W4 precedence — SGLang follows AITER and keeps the GU-interleaved layout. |
| `SGLANG_ROCM_K3_KDA_FUSED_BACKEND=aiter` | unset | Opt in to the fused ROCm KDA decode boundary: the `f_b` projection is deferred into the gfx950 FlyDSL kernel so decode fuses `f_b` + convolution + recurrent state update + gated RMSNorm. Any other value (or unset) keeps the unfused KDA path. |
| `SGLANG_K3_FLYDSL_SOURCE` | `auto` | Which FlyDSL implementation backs the fused decode: `auto` prefers SGLang's vendored kernels and falls back to the AITER module, `sglang` / `aiter` pin one. Leave it alone unless you are bisecting the two. |

The fused KDA backend is fail-closed at two levels: it arms during model init only when the flag is exactly `aiter` **and** the gfx950 FlyDSL kernels are importable, and each decode step re-validates shapes, dtypes, strides, state indices, and output buffers before dispatch — anything unexpected falls back to the stock KDA implementation rather than erroring. Batch size 2 automatically picks a separately validated kernel schedule; every other batch keeps the original build options. Measured on a 69-layer graph the fused boundary is 9.20 → 8.38 µs/layer (−8.9%), with GSM8K 1319 at 0.950.

## 3. Advanced Usage

### 3.1 Reasoning

K3 always thinks; the `kimi_k3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) separates that thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Control the reasoning depth with `reasoning_effort` (`low` / `high` / `max`; default `max`).

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="moonshotai/Kimi-K3",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     reasoning_effort="high",  # "low" | "high" | "max" (default max)
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update...
> ```

### 3.2 Tool Calling

Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. (Not yet supported on the Ascend NPU recipes — A3 and A5.)

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city",
>         "parameters": {
>             "type": "object",
>             "properties": {"city": {"type": "string"}},
>             "required": ["city"],
>         },
>     },
> }]
> resp = client.chat.completions.create(
>     model="moonshotai/Kimi-K3",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update...
> ```

### 3.3 HiCache (Hierarchical KV Caching)

K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads.

* On the DCP recipes (Blackwell Balanced / High-Throughput, `Unified` and `Decode` roles), **L1+L2 keeps DCP**, with Spec Decode off or on DSPARK. On B200 the DSPARK pipeline collapse scales it too (PP2 × DCPEP8 → DCPEP16).
* **L3 drops the DCP flags** on those recipes (storage keys are not dcp\_rank-aware yet; the command hints call it out, and per-request KV capacity shrinks). Only DCP goes: the MLA KV reverts to TP-replicated while the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP and B200 Unified keeps its `--pp-size 2` / `--ep-size`.
* Low-Latency and the Hopper recipes take all tiers unchanged.

### 3.4 PD Disaggregation

PD splits prefill and decode into separate server groups; because K3 is hybrid, the transfer moves **both** the paged MLA KV and the KDA recurrent state.

* **Transfer**: the CUDA cells emit **NiXL** (RDMA); Mooncake stays selectable in the Playground. The Ascend NPU cells emit `--disaggregation-transfer-backend ascend` — the KV/KDA transfer rides the Ascend memory fabric instead (see [below](#npu-pd)).
* **Ports**: prefill `30000`, decode `30100` (derived ZMQ/dist ranges must not collide on a shared host). The positional `8998` after `--prefill` must match `--disaggregation-bootstrap-port`, or only the decode worker registers.
* **Decode state pool**: radix cache disabled — one slot per request; `--mamba-radix-cache-strategy` is inert. Keep `--disaggregation-decode-extra-slots` pinned: unpinned it defaults to twice the batch below 32 requests and **zero** above.

#### On Ascend NPUs (A3 Series)

The A3 Series disaggregates over the **Ascend memory fabric** (`--disaggregation-transfer-backend ascend`) in an asymmetric **2P4D** pool layout:

| Pool layout | A3 Series (2P4D) |
| - | - |
| Prefill pool | 2 nodes, TP16 × PP2 × DP2, NOSPEC, no CUDA graph |
| Decode pool | 4 nodes, TP64 × DP4, DSPARK, decode graph at bs 16 |
| Total | 6 nodes |

* Every rank of **both** pools registers its transfer buffers against a store served from the **prefill head node** (`ASCEND_MF_STORE_URL`) — set the prefill head's IP on the decode nodes too, not just the prefill pool.
* The A3 prefill role is **pipelined** (PP2) and runs NOSPEC — the Spec Decode overlay ships DSPARK on the flat recipes here; newer releases admit it behind a PD prefill ([sgl-project/sglang#40045](https://github.com/sgl-project/sglang/pull/40045)), but no measured A3 command uses it yet.
* Each pool is its own `--dist-init-addr` group, so fill `NODE0_IP` per role command (the A3 split's prefill head and decode head are different hosts) — pointing both pools at one address merges the rendezvous, and the mismatched `--nnodes` stalls the launch.
* The router can run on any host that reaches both pools.

#### Deep PP for prefill

Deep PP is `--tp-size 1` with one pipeline stage per GPU — `--pp-size 8` on B300/GB300, `--pp-size 16` on B200/GB200. Pipeline P2P overlaps the next microbatch's compute, unlike TP/EP collectives, and each stage owns whole layers (a clean slice of KV and state). `--tp-size 1` is also what buys context: above TP1 the MLA KV is replicated across the TP ranks, so TP2 × PP8 holds roughly half the tokens of TP1 × PP16 for the same memory.

* Use one stage per GPU; a shallow split still pays the in-stage all-reduce and can lose to flat TP.
* Pays only with several requests in flight. On the 8-GPU platforms that is why `Default` stays TP8; on the 16-GPU platforms deep PP wins at the Default operating point too, so both strategies use it — measured on GB200 at ISL 8192 / concurrency 32, PP16 × TP1 reached 4550 prefill tok/s/GPU vs 3596 (PP8 × TP2), 2407 (TEP16), and 1652 (TP16). Below concurrency \~8 the pipeline cannot fill and TEP16 leads instead (1947 vs 1227) — use `--tp-size 16 --ep-size 16` there.
* DSPARK off on the pipelined recipes — on B200/GB200 that applies to `Default` as well.
* Fan one prefill role out to several decode roles; budget for in-transfer KV on the decode side.

> **Router**
>
> ```bash Command theme={null}
> python3 -m sglang_router.launch_router \
>   --pd-disaggregation \
>   --prefill http://<prefill-host>:30000 8998 \
>   --decode http://<decode-host>:30100 \
>   --host 0.0.0.0 --port 8000 \
>   --disable-circuit-breaker \
>   --health-check-interval-secs 999999
> ```

Clients then send requests to the router (`:8000`) instead of an individual role server.

### 3.5 VLM Serving Profiles

The open-source K3 serving contract currently supports **image input only** — its
processor rejects video and audio input.

#### VLM feature transport

Use **VLM Transport** in the command picker. `Auto` is a topology-aware starting
point, not a claim that one configuration is fastest for every workload.

| Picker selection | Processor-to-scheduler feature path |
| - | - |
| Auto · single-node Unified CUDA | CUDA IPC |
| Auto · Unified GB200/GB300 | CUDA VMM when IMEX is available; CPU otherwise |
| Auto · PD or other topologies | CPU |
| CPU | CPU, with no GPU feature pool |

CUDA IPC and CUDA VMM reserve up to `SGLANG_MM_FEATURE_CACHE_MB` (1 GiB by
default) on the base GPU and fall back to CPU per tensor when full. This setting
does not control EPD encoder output or PD KV/KDA transfer. K3 already defaults to
2 processor workers and 16 I/O workers; leave those flags unset unless tuning.

#### VLM compatibility

| Feature | K3 behavior |
| - | - |
| PD | Supported. Image processing and ViT run on prefill; the PD transfer then moves both paged MLA KV and KDA recurrent state as described in [PD disaggregation](#pd-disaggregation). |
| EPD | Supported on the public `kimi-k3` branch. Use an `--encoder-only` vision role and a `--language-only` prefill role; add the normal decode role for full EPD. See the [EPD guide](../../../docs/advanced_features/epd_disaggregation). |
| MM encoder DP | Built in. K3 shards complete images across TP ranks, so leave `--mm-enable-dp-encoder` unset in unified, PD-prefill, and encoder-only roles. |
| MM feature transport | Processor-to-scheduler features only. EPD encoder output and PD KV/KDA transfer use their own backends. |
| ViT BCG | Compatible with unified and encoder-only roles, but recommended only for repeated encoder shapes after measuring the HBM trade-off below. |

#### Should ViT BCG be enabled?

Keep ViT BCG **off** for general serving; enable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` only for ViT-only / EPD encoder workloads with recurring image shapes and spare HBM.

* The win is confined to the encoder — no reliable end-to-end TTFT/TPOT gain in full-model serving.
* Each captured graph retains HBM (graph + per-entry metadata); measure on your own shapes.
* The default cache captures after two hits and falls back to eager above 6,144 tokens; do not enlarge it without measuring.

#### Low-HBM VLM

Use this profile when keeping HBM headroom matters more than peak concurrency.
It removes the GPU feature pool, keeps ViT BCG disabled, halves the context
window, caps concurrency, and lowers the static-memory target:

```bash Command theme={null}
sglang serve \
  --trust-remote-code \
  --model-path moonshotai/Kimi-K3 \
  --tp-size 8 \
  --context-length 65536 \
  --enable-symm-mem \
  --mem-fraction-static 0.82 \
  --mm-feature-transport cpu \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --host 0.0.0.0 \
  --port 30000
```

`--mem-fraction-static 0.82` is a conservative B300 starting point, not a portable minimum: raise it toward `0.85` if startup reports insufficient memory; if HBM must go back to other workloads, reduce context/concurrency first. The precision levers (`fp8_e4m3` KV, `bfloat16` SSM state) save far more but stay accuracy-gated.

### 3.6 Large-Scale Serving Presets (16–64 GPUs, Blackwell)

**The KDA state pool is the concurrency ceiling** — DP, EP, and DCP do not shard it; only attention-TP width, SSM dtype, and cache strategy change the per-GPU bill. The MLA KV is cheap to shrink (fp8) or deduplicate (DCP).

Two presets come out of this, at `N = 8k` GPUs:

| Preset | What it trades | Pick it for |
| - | - | - |
| **Peak Throughput** — `dp = k`, attention-TP 8 | State shards 8-way. The per-step KDA all-reduce stays within one 8-GPU B200/B300 node, or spans two 4-GPU GB200/GB300 nodes over MNNVL. `--kv-cache-dtype fp8_e4m3` is load-bearing — bf16 KV does not fit 128 requests per replica. | Maximum sustained TPS — the default large-scale shape. |
| **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, \~1.8× ITL. | Context ≥ \~16K, or per-replica concurrency past 128. |

* **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5.
* The fully data-parallel extreme (`--attn-dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's \~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape.

The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`:

```bash Command theme={null}
SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=20480 \
sglang serve \
  --trust-remote-code \
  --model-path moonshotai/Kimi-K3 \
  --tp-size 32 --ep-size 32 \
  --attn-dp-size 4 --enable-dp-lm-head \
  --nnodes 4 --node-rank <rank> --dist-init-addr <node0-ip>:20000 \
  --moe-a2a-backend megamoe --moe-runner-backend deep_gemm \
  --kv-cache-dtype fp8_e4m3 \
  --mamba-ssm-dtype bfloat16 \
  --mamba-radix-cache-strategy extra_buffer_lazy \
  --mem-fraction-static 0.92 \
  --reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
  --host 0.0.0.0 --port 30000
```

Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation:

| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--attn-dp-size` |
| - | - | - | - | - |
| 16 | 2×8 | 4×4 | 16 | 2 |
| 32 | 4×8 | 8×4 | 32 | 4 |
| 64 | 8×8 | 16×4 | 64 | 8 |

For Peak Capacity, add `--dcp-size 8` and re-derive the pool split with the [Mamba ratio calculator](#mamba-ratio-calculator).

Both presets are one click away in the [Playground above](#playground): pick a **Cluster Size** and a **Large-Scale Preset** and the full command composes onto whichever cell is showing.

Decisions the preset already makes:

* **MegaMoE + `deep_gemm`** — the fused DeepGEMM all-to-all/MoE path used by these large-scale DP/EP throughput presets, with K3's SiTU activation.
* **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not.
* **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales.

> [!NOTE]
>
> No preset has a full serving round on final weights; the constants derive from measured single- and dual-node rounds plus a 64-GPU sweep. Validate throughput and accuracy on your workload before committing a fleet.


# Kimi-Linear
Source: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-Linear



## AMD GPU Support

## 1. Model Introduction

Kimi Linear is a hybrid linear attention architecture that outperforms traditional full attention methods across various contexts, including short, long, and reinforcement learning (RL) scaling regimes. At its core is Kimi Delta Attention (KDA)—a refined version of Gated DeltaNet that introduces a more efficient gating mechanism to optimize the use of finite-state RNN memory.

This generation delivers comprehensive upgrades across the board:

Kimi Delta Attention (KDA): A linear attention mechanism that refines the gated delta rule with finegrained gating.
Hybrid Architecture: A 3:1 KDA-to-global MLA ratio reduces memory usage while maintaining or surpassing the quality of full attention.
Superior Performance: Outperforms full attention in a variety of tasks, including long-context and RL-style benchmarks on 1.4T token training runs with fair comparisons.
High Throughput: Achieves up to 6× faster decoding and significantly reduces time per output token (TPOT).

For more details, please refer to the \[official Kimi Linear GitHub Repository]: [https://github.com/MoonshotAI/Kimi-Linear](https://github.com/MoonshotAI/Kimi-Linear)

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model variant, deployment strategy, and thinking capabilities.

Run this Command:

<pre />

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Launch the docker

```shell Command theme={null}
docker pull lmsysorg/sglang:v0.5.7-rocm700-mi30x
```

```shell Command theme={null}
docker run -d -it --ipc=host --network=host --privileged \
  --cap-add=CAP_SYS_ADMIN \
  --device=/dev/kfd --device=/dev/dri --device=/dev/mem \
  --group-add video --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v /:/work \
  -e SHELL=/bin/bash \
  --name Kimi-linear \
  lmsysorg/sglang:v0.5.7-rocm700-mi30x \
  /bin/bash
```

#### 4.2.2 pre-installation steps inside the docker

```shell Command theme={null}
pip install sentencepiece tiktoken
```

#### 4.2.3 Launch the server

```shell Command theme={null}
export SGLANG_ROCM_FUSED_DECODE_MLA=0

SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
  --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --tokenizer-path  moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --tp 4 \
  --trust-remote-code
```

## 5. Benchmark

### 5.1 Speed Benchmark

Test Environment:

Hardware: AMD MI300X GPU

Model: Kimi-Linear-48B-A3B-Instruct

Tensor Parallelism: 4

sglang version: 0.5.7

* **Model Deployment**

```bash Command theme={null}
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
  --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --tokenizer-path  moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --tp 4 \
  --trust-remote-code
```

### 5.1.1 Low Concurrency (Latency-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  23.86
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4220
Total generated tokens (retokenized):    4001
Request throughput (req/s):              0.42
Input token throughput (tok/s):          255.70
Output token throughput (tok/s):         176.86
Peak output token throughput (tok/s):    190.00
Peak concurrent requests:                2
Total token throughput (tok/s):          432.56
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2383.93
Median E2E Latency (ms):                 1911.63
---------------Time to First Token----------------
Mean TTFT (ms):                          141.33
Median TTFT (ms):                        126.27
P99 TTFT (ms):                           294.76
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.32
Median TPOT (ms):                        5.33
P99 TPOT (ms):                           5.36
---------------Inter-Token Latency----------------
Mean ITL (ms):                           5.33
Median ITL (ms):                         5.32
P95 ITL (ms):                            5.44
P99 ITL (ms):                            5.58
Max ITL (ms):                            11.46
==================================================
```

### 5.1.2 Medium Concurrency (Balanced)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  31.38
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40805
Total generated tokens (retokenized):    39667
Request throughput (req/s):              2.55
Input token throughput (tok/s):          1264.13
Output token throughput (tok/s):         1300.37
Peak output token throughput (tok/s):    1801.00
Peak concurrent requests:                21
Total token throughput (tok/s):          2564.50
Concurrency:                             14.13
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5543.18
Median E2E Latency (ms):                 5755.31
---------------Time to First Token----------------
Mean TTFT (ms):                          175.25
Median TTFT (ms):                        137.87
P99 TTFT (ms):                           292.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.75
Median TPOT (ms):                        10.87
P99 TPOT (ms):                           16.74
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.54
Median ITL (ms):                         7.95
P95 ITL (ms):                            13.68
P99 ITL (ms):                            116.80
Max ITL (ms):                            299.89
==================================================

```

### 5.1.3 High Concurrency (Throughput-Optimized)

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100 \
  --request-rate inf
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  79.71
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252662
Total generated tokens (retokenized):    228448
Request throughput (req/s):              6.27
Input token throughput (tok/s):          3134.20
Output token throughput (tok/s):         3169.72
Peak output token throughput (tok/s):    6109.00
Peak concurrent requests:                110
Total token throughput (tok/s):          6303.92
Concurrency:                             94.80
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   15113.92
Median E2E Latency (ms):                 13851.52
---------------Time to First Token----------------
Mean TTFT (ms):                          564.46
Median TTFT (ms):                        226.04
P99 TTFT (ms):                           2683.14
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.63
Median TPOT (ms):                        31.28
P99 TPOT (ms):                           38.84
---------------Inter-Token Latency----------------
Mean ITL (ms):                           28.85
Median ITL (ms):                         16.29
P95 ITL (ms):                            123.42
P99 ITL (ms):                            157.80
Max ITL (ms):                            2481.11
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* Server Command

```shell Command theme={null}
SGLANG_ROCM_FUSED_DECODE_MLA=0 python3 -m sglang.launch_server \
  --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --tokenizer-path  moonshotai/Kimi-Linear-48B-A3B-Instruct \
  --tp 4 \
  --trust-remote-code
```

* Benchmark Command

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Result**:

```text Output theme={null}
Accuracy: 0.705
Invalid: 0.000
Latency: 11.855 s
Output throughput: 3224.982 token/s
```


# Nemotron3-Nano
Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Nano



## 1. Model Introduction

`NVIDIA Nemotron3-Nano` is a 30B-parameter hybrid LLM that mixes Mixture-of-Experts (MoE) feed-forward layers, Mamba2 sequence-modeling layers, and standard self-attention layers in a single stack rather than classic “attention + MLP” transformer blocks.

The BF16 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`) is designed as a high-fidelity reference model. For optimized inference performance on modern NVIDIA GPUs, the FP8 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8`) and the NVFP4 variant (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4`) are supported.

At a high level:

* **Hybrid layer stack (Mamba2 + MoE + attention):** The network is composed of interleaved layers that are *either* Mamba2, *or* MoE feed-forward, *or* attention-only.
* **Non-uniform layer ordering:** The order and mix of these specialized layers is not a simple, rigid pattern, enabling the model to trade off sequence modeling, routing capacity, and expressivity across depth.
* **Deployment-friendly precision:** Use BF16 for accuracy-sensitive and evaluation workloads; use FP8 for latency- and throughput-critical serving on recent NVIDIA GPUs.

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install), or install nightly wheel through:

```bash Command theme={null}
uv pip install --prerelease=allow sglang==0.5.6.post3.dev1278+gad1b4e472 --extra-index-url https://sgl-project.github.io/whl/nightly/
```

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance tuning.

### 3.1 Basic Configuration

The Nemotron3-Nano series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)).

**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Attention backend**:

  **H200**: Use flash attention 3 backend by default.
  **B200**: Use flashinfer backend by default.

* **TP support**:

  To set tp size, use `--tp <1|2|4|8>`.

* **FP8 KV cache**:

  To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.

## 4. Model Invocation

### 4.1 Basic Usage (OpenAI-Compatible API)

SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Summarize what MoE models are in 5 bullets."},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(resp.choices[0].message.content)

```

Streaming chat completion

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "What are the first 5 prime numbers?"}
    ],
    temperature=0.7,
    max_tokens=1024,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        print(delta.content, end="", flush=True)
```

### 4.2 Reasoning

To enable reasoning, `--reasoning-parser nemotron_3` should be appended to the launching command. The model supports two modes - Reasoning ON (default) vs OFF. This can be toggled by setting enable\_thinking to False, as shown below.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

# Reasoning on (default)
print("Reasoning on")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about GPUs."}
    ],
    temperature=0.7,
    max_tokens=512,
)
print(resp.choices[0].message.reasoning_content)

# Reasoning off
print("Reasoning off")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about GPUs."}
    ],
    temperature=0.6,
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
print(resp.choices[0].message.reasoning_content)

```

### 4.3 Tool calling

To enable reasoning, `--tool-call-parser qwen3_coder` should be appended to the launching command. Call functions using the OpenAI Tools schema and inspect returned tool\_calls.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

# Tool calling via OpenAI tools schema
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "calculate_tip",
            "parameters": {
                "type": "object",
                "properties": {
                    "bill_total": {
                        "type": "integer",
                        "description": "The total amount of the bill"
                    },
                    "tip_percentage": {
                        "type": "integer",
                        "description": "The percentage of tip to be applied"
                    }
                },
                "required": ["bill_total", "tip_percentage"]
            }
        }
    }
]

completion = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8",
    messages=[
        {"role": "system", "content": ""},
        {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
    ],
    tools=TOOLS,
    temperature=0.6,
    top_p=0.95,
    max_tokens=512,
    stream=False
)

print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.tool_calls)
```

***

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU

**FP8 variant**

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --trust-remote-code \
  --max-running-requests 1024 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 4096 \
  --max-concurrency 256
```

* **Serving benchmark results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 256
Successful requests:                     4096
Benchmark duration (s):                  183.18
Total input tokens:                      2081726
Total input text tokens:                 2081726
Total input vision tokens:               0
Total generated tokens:                  2116125
Total generated tokens (retokenized):    1076256
Request throughput (req/s):              22.36
Input token throughput (tok/s):          11364.25
Output token throughput (tok/s):         11552.04
Peak output token throughput (tok/s):    24692.00
Peak concurrent requests:                294
Total token throughput (tok/s):          22916.30
Concurrency:                             251.19
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   11233.74
Median E2E Latency (ms):                 11142.97
---------------Time to First Token----------------
Mean TTFT (ms):                          172.99
Median TTFT (ms):                        116.57
P99 TTFT (ms):                           1193.68
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          21.74
Median TPOT (ms):                        21.14
P99 TPOT (ms):                           41.12
---------------Inter-Token Latency----------------
Mean ITL (ms):                           21.45
Median ITL (ms):                         9.06
P95 ITL (ms):                            62.59
P99 ITL (ms):                            110.83
Max ITL (ms):                            5368.19
==================================================
```

**BF16 variant**

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
  --trust-remote-code \
  --max-running-requests 1024 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 4096 \
  --max-concurrency 256
```

* **Serving benchmark results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 256
Successful requests:                     4096
Benchmark duration (s):                  360.22
Total input tokens:                      2081726
Total input text tokens:                 2081726
Total input vision tokens:               0
Total generated tokens:                  2087288
Total generated tokens (retokenized):    1940652
Request throughput (req/s):              11.37
Input token throughput (tok/s):          5779.10
Output token throughput (tok/s):         5794.55
Peak output token throughput (tok/s):    9169.00
Peak concurrent requests:                276
Total token throughput (tok/s):          11573.65
Concurrency:                             249.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   21965.10
Median E2E Latency (ms):                 21706.35
---------------Time to First Token----------------
Mean TTFT (ms):                          211.54
Median TTFT (ms):                        93.06
P99 TTFT (ms):                           2637.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          43.27
Median TPOT (ms):                        43.04
P99 TPOT (ms):                           61.15
---------------Inter-Token Latency----------------
Mean ITL (ms):                           42.77
Median ITL (ms):                         28.46
P95 ITL (ms):                            71.85
P99 ITL (ms):                            113.20
Max ITL (ms):                            5237.28
==================================================

```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

**Environment**

* Hardware: NVIDIA B200 GPU
* Model: BF16 checkpoint

**Launch Model**

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
  --trust-remote-code \
  --reasoning-parser nemotron_3
```

**Run Benchmark with sgl-eval**

```bash Command theme={null}
pip install sgl-eval

sgl-eval run gsm8k --base-url http://127.0.0.1:30000/v1 --model nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --num-threads 4 --max-tokens 16384 --thinking
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

**Historical lm-eval results (not measurements of the sgl-eval command above):**

```text Output theme={null}
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.5603|±  |0.0137|
|     |       |strict-match    |     5|exact_match|↑  |0.8453|±  |0.0100|
```


# Nemotron 3 Nano Omni
Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Nano-Omni



## 1. Model Introduction

`NVIDIA Nemotron 3 Nano Omni` is a 30B-parameter hybrid MoE multimodal model that activates only 3B parameters per forward pass, combining vision and audio encoders into a unified architecture. Part of the Nemotron 3 family, it is designed to power multimodal sub-agents that perceive and reason across vision, audio, and language in a single inference loop — eliminating the fragmented stacks of separate models for each modality.

Architecture and key features:

* **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture for efficient routing and sequence modeling.
* **30B total / 3B active parameters:** Delivers strong multimodal accuracy at a fraction of the cost of dense models.
* **1M token context window:** Sustains coherent agent state across extended multimodal workflows — screen history, document content, and audio context remain in view without re-ingestion.
* **Unified vision and audio encoders:** One model replaces fragmented multimodal stacks; vision and audio perception happen in the same forward pass.
* **3D Convolution (Conv3D):** Efficient temporal-spatial processing for video inputs.
* **Efficient Video Sampling (EVS):** Enables longer video processing at the same compute budget via temporal-aware perception and adaptive frame sampling.
* **FP8 and NVFP4 quantization:** FP8 supports deployment from workstation (RTX 6000, DGX Spark) to cloud (H100, H200, B200, A100, L40S); NVFP4 requires Blackwell hardware.
* **9x higher throughput** than other open omni models at the same interactivity level.
* **\~20% higher multimodal intelligence** compared to the best open alternative.
* **Post-trained with multi-environment reinforcement learning** via NVIDIA NeMo RL and NeMo Gym across text, image, audio, and video environments, improving instruction following and convergence to correct multimodal answers.

**Modalities:** Input: text, image, video, audio — Output: text

**Supported GPUs:** NVIDIA B200, H100, H200, A100, L40S, DGX Spark, RTX 6000

Available model variants on HuggingFace:

* [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16)
* [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8)
* [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4)

**Agentic workloads this model enables:**

* **Computer Use Agent:** Perception loop for agents navigating GUIs — reads screens, understands UI state over time, validates outcomes. Collapses vision and reasoning into a single loop.
* **Document Intelligence:** Interprets documents, charts, tables, screenshots, and mixed media inputs for enterprise analysis and compliance workflows.
* **Audio & Video Understanding Agents:** Maintains continuous audio-video context for customer service, research, and monitoring workflows, tying what was said, shown, and documented into a single reasoning stream.

## 2. SGLang Installation

Install SGLang via pip or from source:

```shell Command theme={null}
# Install via pip
pip install sglang

# Or install from source
uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'

# Or use Docker
docker pull lmsysorg/sglang:latest
```

For the full Docker setup and other installation methods, refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance tuning.

### 3.1 Basic Configuration

**Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Attention backend:**

  **H100/H200:** Use flash attention 3 backend by default.
  **B200:** Use flashinfer backend by default.

* **TP support:**

  To set tensor parallelism, use `--tp <1|2|4|8>`. A 4×H100 setup is recommended for the BF16 variant.

* **FP8 KV cache:**

  To enable FP8 KV cache, append `--kv-cache-dtype fp8_e4m3`. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.

* **Reasoning parser:**

  Append `--reasoning-parser deepseek-r1` to enable structured reasoning traces (`reasoning_content` field in the response).

* **Tool calling:**

  Append `--tool-call-parser qwen3_coder` to enable tool calling support.

## 4. Model Invocation

The command below launches the server for a 4×H100 setup with reasoning and tool calling enabled. See [Section 4.8](#4-8-fp8-and-nvfp4-deployment) for FP8 and NVFP4 variants.

```shell Command theme={null}
sglang serve \
  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --host 0.0.0.0 \
  --port 30000 \
  --tp 4 \
  --trust-remote-code \
  --tool-call-parser qwen3_coder \
  --reasoning-parser deepseek-r1
```

### 4.1 Basic Usage (Text)

SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:

```python Example theme={null}
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
    ],
    temperature=0.6,
    max_tokens=512,
)
print(resp.choices[0].message.reasoning_content, resp.choices[0].message.content)
```

Output:

```text Output theme={null}
Reasoning: SGLang is a serving framework I know from my training data. Let me recall the key features...

Content:
- **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn and few-shot workloads.
- **OpenAI-compatible API** — Drop-in replacement for the OpenAI Python client; no application code changes required to serve a locally-hosted model.
- **High-throughput serving** — Continuous batching, chunked prefill, and optimized CUDA kernels deliver state-of-the-art throughput on NVIDIA GPUs across A100, H100, and B200.
```

Streaming chat completion:

```python Example theme={null}
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

stream = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "What are the first 5 prime numbers?"},
    ],
    temperature=0.6,
    max_tokens=512,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        print(delta.content, end="", flush=True)
```

### 4.2 Image Understanding

Pass image inputs using the OpenAI vision format. Supports both URLs and base64-encoded images:

```python Example theme={null}
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

# From URL
resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"},
                },
                {"type": "text", "text": "Describe this image in detail."},
            ],
        }
    ],
    temperature=0.6,
    max_tokens=512,
)
print(resp.choices[0].message.reasoning_content)
print(resp.choices[0].message.content)
```

For local images, encode as base64:

```python Example theme={null}
import base64
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

with open("screenshot.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image_b64}"},
                },
                {"type": "text", "text": "What UI elements are visible on this screen? What action would you take next?"},
            ],
        }
    ],
    temperature=0.6,
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### 4.3 Video Understanding

Nemotron 3 Nano Omni uses Conv3D layers and Efficient Video Sampling (EVS) for temporal-spatial video reasoning, processing longer videos at the same compute budget:

```python Example theme={null}
import base64
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

with open("video.mp4", "rb") as f:
    video_b64 = base64.b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "video_url",
                    "video_url": {"url": f"data:video/mp4;base64,{video_b64}"},
                },
                {"type": "text", "text": "Summarize what happens in this video step by step."},
            ],
        }
    ],
    temperature=0.6,
    max_tokens=1024,
)
print(resp.choices[0].message.reasoning_content)
print(resp.choices[0].message.content)
```

### 4.4 Audio Understanding

Pass audio inputs as base64-encoded WAV or MP3 data:

```python Example theme={null}
import base64
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

with open("audio.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_audio",
                    "input_audio": {"data": audio_b64, "format": "wav"},
                },
                {"type": "text", "text": "Transcribe and summarize what was said in this audio."},
            ],
        }
    ],
    temperature=0.6,
    max_tokens=512,
)
print(resp.choices[0].message.content)
```

### 4.5 Mixed Multimodal Input

Combine modalities in a single request. For example, an image alongside an audio question about it:

```python Example theme={null}
import base64
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

with open("chart.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode("utf-8")

resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{image_b64}"},
                },
                {"type": "text", "text": "Analyze this chart. What are the key trends and what conclusion does the data support?"},
            ],
        }
    ],
    temperature=0.6,
    max_tokens=1024,
)
print(resp.choices[0].message.reasoning_content)
print(resp.choices[0].message.content)
```

### 4.6 Reasoning

The model supports two modes — Reasoning ON (default) vs OFF. Toggle per-request by setting `enable_thinking` to `False`:

```python Example theme={null}
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

# Reasoning ON (default)
print("Reasoning on")
resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the derivative of x^3 sin(x)?"},
    ],
    temperature=0.6,
    max_tokens=1024,
)
print(f"Reasoning:\n{resp.choices[0].message.reasoning_content[:300]}...\nContent:\n{resp.choices[0].message.content}")
print("\n")

# Reasoning OFF
print("Reasoning off")
resp = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is 15% of 200?"},
    ],
    temperature=0.6,
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(f"Content:\n{resp.choices[0].message.content}")
```

Output:

```text Output theme={null}
Reasoning on
Reasoning:
The user wants the derivative of x^3 sin(x). I'll apply the product rule: d/dx[u·v] = u'v + uv'. Here u = x^3, v = sin(x). So u' = 3x^2, v' = cos(x). The result is 3x^2·sin(x) + x^3·cos(x)...
Content:
Using the product rule: d/dx[x³ sin(x)] = 3x² sin(x) + x³ cos(x)


Reasoning off
Content:
15% of 200 is **30**.
```

### 4.7 Tool Calling

Call functions using the OpenAI Tools schema. The server must be launched with `--tool-call-parser qwen3_coder`:

```python Example theme={null}
from openai import OpenAI

SERVED_MODEL_NAME = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16"
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City and state, e.g. San Francisco, CA",
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["location"],
            },
        },
    }
]

completion = client.chat.completions.create(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the weather like in Santa Clara, CA?"},
    ],
    tools=TOOLS,
    temperature=0.6,
    top_p=0.95,
    max_tokens=512,
    stream=False,
)
print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.tool_calls)
```

Output:

```text Output theme={null}
The user is asking about weather in Santa Clara, CA. I have a get_weather function that takes a location and optional unit. I should call it with location="Santa Clara, CA".

[ChatCompletionMessageFunctionToolCall(id='call_abc123', function=Function(arguments='{"location": "Santa Clara, CA", "unit": "fahrenheit"}', name='get_weather'), type='function', index=0)]
```

### 4.8 FP8 and NVFP4 Deployment

**FP8 variant** (recommended for throughput-critical serving on H100/H200/B200):

```shell Command theme={null}
sglang serve \
  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --tp 4 \
  --trust-remote-code \
  --tool-call-parser qwen3_coder \
  --reasoning-parser deepseek-r1
```

**NVFP4 variant** (maximum efficiency on Blackwell B200):

```shell Command theme={null}
sglang serve \
  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 \
  --host 0.0.0.0 \
  --port 30000 \
  --tp 4 \
  --trust-remote-code \
  --tool-call-parser qwen3_coder \
  --reasoning-parser deepseek-r1
```

***

## 5. Benchmark

### 5.1 Efficiency Benchmark

Nemotron 3 Nano Omni achieves **9x higher throughput** than other open omni models at the same interactivity level, delivering lower cost and better scalability without sacrificing responsiveness. It also achieves **\~20% higher multimodal intelligence** compared to the best open alternative across image, video, and audio reasoning tasks.

### 5.2 Speed Benchmark

**Test Environment:**

* Hardware: B200 (8×)
* Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning
* Tensor Parallelism: 4
* SGLang Version: main branch

Model Deployment Command:

```shell Command theme={null}
sglang serve \
  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --trust-remote-code \
  --tp 4 \
  --max-running-requests 1024 \
  --host 0.0.0.0 \
  --attention-backend flashinfer \
  --port 30000
```

Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 4096 \
  --max-concurrency 256
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 256
Successful requests:                     4096
Benchmark duration (s):                  206.52
Total input tokens:                      2081726
Total input text tokens:                 2081726
Total generated tokens:                  2087288
Total generated tokens (retokenized):    1945477
Request throughput (req/s):              19.83
Input token throughput (tok/s):          10080.25
Output token throughput (tok/s):         10107.18
Peak output token throughput (tok/s):    20199.00
Peak concurrent requests:                291
Total token throughput (tok/s):          20187.44
Concurrency:                             250.83
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   12646.47
Median E2E Latency (ms):                 12371.84
P90 E2E Latency (ms):                    22889.81
P99 E2E Latency (ms):                    26528.70
---------------Time to First Token----------------
Mean TTFT (ms):                          220.66
Median TTFT (ms):                        97.67
P99 TTFT (ms):                           2068.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.98
Median TPOT (ms):                        24.36
P99 TPOT (ms):                           44.97
---------------Inter-Token Latency----------------
Mean ITL (ms):                           24.43
Median ITL (ms):                         10.91
P95 ITL (ms):                            62.68
P99 ITL (ms):                            100.60
Max ITL (ms):                            2171.93
==================================================
```

### 5.3 Accuracy Benchmark

**Environment**

* Hardware: B200 (8×)
* Model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning
* Tensor Parallelism: 4
* SGLang Version: main branch

**Launch Model**

```shell Command theme={null}
sglang serve \
  --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 \
  --trust-remote-code \
  --tp 4 \
  --attention-backend flashinfer \
  --reasoning-parser deepseek-r1
```

#### 5.3.1 GSM8K Benchmark

**Run Benchmark**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.830
Invalid: 0.000
Latency: 13.970 s
Output throughput: 1611.623 token/s
```

#### 5.3.2 MMLU Benchmark

**Run Benchmark**

```shell Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
subject: abstract_algebra, #q:100, acc: 0.510
subject: anatomy, #q:135, acc: 0.711
subject: astronomy, #q:152, acc: 0.829
subject: business_ethics, #q:100, acc: 0.760
subject: clinical_knowledge, #q:265, acc: 0.781
subject: college_biology, #q:144, acc: 0.854
subject: college_chemistry, #q:100, acc: 0.560
subject: college_computer_science, #q:100, acc: 0.700
subject: college_mathematics, #q:100, acc: 0.590
subject: college_medicine, #q:173, acc: 0.775
subject: college_physics, #q:102, acc: 0.559
subject: computer_security, #q:100, acc: 0.750
subject: conceptual_physics, #q:235, acc: 0.821
subject: econometrics, #q:114, acc: 0.605
subject: electrical_engineering, #q:145, acc: 0.759
subject: elementary_mathematics, #q:378, acc: 0.638
subject: formal_logic, #q:126, acc: 0.524
subject: global_facts, #q:100, acc: 0.400
subject: high_school_biology, #q:310, acc: 0.906
subject: high_school_chemistry, #q:203, acc: 0.759
subject: high_school_computer_science, #q:100, acc: 0.860
subject: high_school_european_history, #q:165, acc: 0.812
subject: high_school_geography, #q:198, acc: 0.889
subject: high_school_government_and_politics, #q:193, acc: 0.933
subject: high_school_macroeconomics, #q:390, acc: 0.785
subject: high_school_mathematics, #q:270, acc: 0.496
subject: high_school_microeconomics, #q:238, acc: 0.887
subject: high_school_physics, #q:151, acc: 0.675
subject: high_school_psychology, #q:545, acc: 0.895
subject: high_school_statistics, #q:216, acc: 0.731
subject: high_school_us_history, #q:204, acc: 0.858
subject: high_school_world_history, #q:237, acc: 0.873
subject: human_aging, #q:223, acc: 0.740
subject: human_sexuality, #q:131, acc: 0.855
subject: international_law, #q:121, acc: 0.851
subject: jurisprudence, #q:108, acc: 0.815
subject: logical_fallacies, #q:163, acc: 0.847
subject: machine_learning, #q:112, acc: 0.598
subject: management, #q:103, acc: 0.864
subject: marketing, #q:234, acc: 0.910
subject: medical_genetics, #q:100, acc: 0.880
subject: miscellaneous, #q:783, acc: 0.881
subject: moral_disputes, #q:346, acc: 0.780
subject: moral_scenarios, #q:895, acc: 0.543
subject: nutrition, #q:306, acc: 0.814
subject: philosophy, #q:311, acc: 0.733
subject: prehistory, #q:324, acc: 0.852
subject: professional_accounting, #q:282, acc: 0.553
subject: professional_law, #q:1534, acc: 0.565
subject: professional_medicine, #q:272, acc: 0.779
subject: professional_psychology, #q:612, acc: 0.760
subject: public_relations, #q:110, acc: 0.709
subject: security_studies, #q:245, acc: 0.759
subject: sociology, #q:201, acc: 0.831
subject: us_foreign_policy, #q:100, acc: 0.910
subject: virology, #q:166, acc: 0.560
subject: world_religions, #q:171, acc: 0.807
Total latency: 67.512
Average accuracy: 0.737
```


# NVIDIA Nemotron3-Super
Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Super



## 1. Model Introduction

`NVIDIA Nemotron3-Super` is a leading open model in the Nemotron 3 family, built for running many collaborating agents together. It is optimized for agentic systems that chain planning, reasoning, and tool use workloads that generate far more tokens than single turn chat and require strong reasoning at every step.

Nemotron 3 Super is a 120B parameter hybrid MoE model that activates only 12B parameters per forward pass, delivering strong accuracy for coding, tool calling, and instruction following at a fraction of the cost. It also supports a 1M token context window so agents can keep conversation history and plan state in view across long workflows.

Architecture and key features:

* **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
* **Highest throughput efficiency in its size category:** Delivers up to 5x higher throughput compared to the previous Nemotron Super model (Llama Nemotron Super 1.5).
* **Multi-Token Prediction (MTP):** By predicting several future tokens simultaneously in a single forward pass, MTP drastically accelerates the generation of long-form text.
* **Thinking Budget support:** Supports Thinking Budget for optimal accuracy with minimum reasoning token generation.

## 2. SGLang Installation

SGLang from the main branch is required for Nemotron3-Super. You can install from source and with a nightly docker.

```bash Command theme={null}
# Install from source
uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'

# Or use Docker
docker pull lmsysorg/sglang:latest
```

For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance tuning.

### 3.1 Basic Configuration

**Interactive Command Generator**: select hardware, tensor parallelism, and common knobs to generate a launch command.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Attention backend**:

  **H200**: Use flash attention 3 backend by default.
  **B200**: Use flashinfer backend by default.

* **TP support**:

  To set tp size, use `--tp <2|4|8>`.

* **FP8 KV cache**:

  To enable fp8 kv cache, please append `--kv-cache-dtype fp8_e4m3`.

## 4. Model Invocation

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
  --host 0.0.0.0 \
  --port 5000 \
  --trust-remote-code \
  --tp 4 \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_3
```

### 4.1 Basic Usage (OpenAI-Compatible API)

SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
    ],
    temperature=0.6,
    max_tokens=1024,
)
print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
print("\n")
```

Output:

```text Output theme={null}
Reasoning: Okay, the user is asking for 3 bullet points about SGLang. Let me recall what I know about SGLang. It's a framework for serving large language models, right? Developed by the team at UC Berkeley and others.

First, I should verify the key features. SGLang is known for its high-performance serving capabilities, especially with features like Radix Attention and chunked prefill. Those are important points to mention...(more tokens)

Content: - SGLang introduces **Radix Attention**, an innovative attention mechanism that significantly reduces KV cache memory usage and improves computational efficiency during LLM serving by reusing intermediate states across tokens.
- It features **chunked prefill** for handling long prompts efficiently, breaking input sequences into manageable chunks to minimize latency and memory pressure while maintaining high throughput.
- Designed for **high-performance LLM serving**, SGLang achieves superior throughput and lower latency compared to traditional systems (like vLLM or TensorRT-LLM) through optimized kernel fusion, dynamic batching, and seamless integration with Hugging Face Transformers.
```

Streaming chat completion:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "What are the first 5 prime numbers?"}
    ],
    temperature=0.7,
    max_tokens=1024,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        print(delta.content, end="", flush=True)
```

Output:

```text Output theme={null}
The first 5 prime numbers are:
**2, 3, 5, 7, 11**.

### Explanation:
- A **prime number** is a natural number greater than 1 that has no positive divisors other than 1 and itself.
- **2** is the smallest and only even prime number.
- **3** is prime (divisible only by 1 and 3).
- **4** is not prime (divisible by 2).
- **5** is prime.
- **6** is not prime (divisible by 2 and 3).
- **7** is prime.
- **8, 9, 10** are not prime.
- **11** is prime (the fifth in the sequence).

Note: **1 is not considered a prime number** by definition, as it has only one positive divisor.
This list is universally accepted in mathematics. Let me know if you'd like to explore more primes or related concepts! 😊
```

### 4.2 Reasoning

The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5000/v1",
    api_key="EMPTY",
)

# Reasoning on (default)
print("Reasoning on")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about GPUs. Please make thinking process short."}
    ],
    temperature=1,
    max_tokens=1024,
)
print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
print("\n")
# Reasoning off
print("Reasoning off")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Give me 3 facts about SGLang."}
    ],
    temperature=0,
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
print(f"Content: \n{resp.choices[0].message.reasoning_content[:200]}...")
```

Output:

```text Output theme={null}
Reasoning on
Reasoning:
We need to output a haiku about GPUs, with short thinking process. Probably we just need to produce the haiku. No extra commentary needed. Provide a haiku: 5-7-5 syllable lines about GPUs.

Let's deci...
Content:
Silicon hearts beat
Paint vivid worlds with bright light
GPU dreams rise...

Reasoning off
Content:
Certainly! Here are three accurate and informative facts about **SGLang**:

1. **SGLang is a high-performance serving system for large language models (LLMs)**
   Developed by researchers at UC Berk...
```

### 4.3 Tool Calling

Call functions using the OpenAI Tools schema and inspect returned `tool_calls`.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:5000/v1",
    api_key="EMPTY",
)

# Tool calling via OpenAI tools schema
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "calculate_tip",
            "parameters": {
                "type": "object",
                "properties": {
                    "bill_total": {
                        "type": "integer",
                        "description": "The total amount of the bill"
                    },
                    "tip_percentage": {
                        "type": "integer",
                        "description": "The percentage of tip to be applied"
                    }
                },
                "required": ["bill_total", "tip_percentage"]
            }
        }
    }
]

completion = client.chat.completions.create(
    model="nemotron",
    messages=[
        {"role": "system", "content": ""},
        {"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}
    ],
    tools=TOOLS,
    temperature=0.6,
    top_p=0.95,
    max_tokens=512,
    stream=False
)

print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.tool_calls)
```

Output:

```text Output theme={null}
The user wants to calculate a 15% tip on a $50 bill. I have a function called calculate_tip that takes bill_total and tip_percentage as parameters. The bill_total is $50, and tip_percentage is 15. I need to call the function with these values. Let me do that.

[ChatCompletionMessageFunctionToolCall(id='call_ced9a83a3baa448e9d587aaf', function=Function(arguments='{"bill_total": 50, "tip_percentage": 15}', name='calculate_tip'), type='function', index=0)]
```

### 4.4 Controlling Reasoning Budget

The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.

If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.

```python Example theme={null}
from typing import Any, Dict, List
import openai
from transformers import AutoTokenizer

class ThinkingBudgetClient:
    def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
        self.base_url = base_url
        self.api_key = api_key
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
        self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)

    def chat_completion(
        self,
        model: str,
        messages: List[Dict[str, Any]],
        reasoning_budget: int = 512,
        max_tokens: int = 1024,
        **kwargs,
    ) -> Dict[str, Any]:
        assert (
            max_tokens > reasoning_budget
        ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"

        # 1. first call chat completion to get reasoning content
        response = self.client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=reasoning_budget,
            **kwargs
        )

        reasoning_content = response.choices[0].message.reasoning_content or ""

        if "</think>" not in reasoning_content:
            # reasoning content is too long, closed with a period (.)
            reasoning_content = f"{reasoning_content}.\n</think>\n\n"

        reasoning_tokens_used = len(
            self.tokenizer.encode(reasoning_content, add_special_tokens=False)
        )
        remaining_tokens = max_tokens - reasoning_tokens_used

        assert (
            remaining_tokens > 0
        ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."

        # 2. append reasoning content to messages and call completion
        messages.append({"role": "assistant", "content": reasoning_content})
        prompt = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            continue_final_message=True,
        )

        response = self.client.completions.create(
            model=model,
            prompt=prompt,
            max_tokens=remaining_tokens,
            **kwargs
        )

        response_data = {
            "reasoning_content": reasoning_content.strip().strip("</think>").strip(),
            "content": response.choices[0].text,
            "finish_reason": response.choices[0].finish_reason,
        }
        return response_data
```

Usage example with `reasoning_budget=128`:

```python Example theme={null}
SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"

# Client
client = ThinkingBudgetClient(
    base_url="http://127.0.0.1:5000/v1",
    api_key="null",
    tokenizer_name_or_path=SERVED_MODEL_NAME
)

resp = client.chat_completion(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write a haiku about GPUs."}
    ],
    temperature=1,
    max_tokens=512,
    reasoning_budget=128
)
print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
```

Output:

```text Output theme={null}
Reasoning: Okay, the user wants a haiku about GPUs. Let me recall what a haiku is: a traditional Japanese poem with three lines, 5-7-5 syllable structure. So I need to make sure the syllable count is exact.

First, I should think about what makes GPUs interesting. They're used for graphics rendering, parallel processing, AI, gaming, etc. Maybe focus on their speed, power, or how they handle many tasks at once.

Let me brainstorm some words and phrases related to GPUs: silicon, cores, transistors, parallel, rendering, pixels, frames per second, CUDA, tensor.
Content:

Silicon minds awaken,
Thousands of cores hum in unison—
Lightning paints the void.
```

***

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: H200 (4x)

* Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16

* Tensor Parallelism: 4

* SGLang Version: main branch

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
  --trust-remote-code \
  --tp 4 \
  --max-running-requests 1024 \
  --host 0.0.0.0 \
  --port 5000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 5000 \
  --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 4096 \
  --max-concurrency 256
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 256
Successful requests:                     4096
Benchmark duration (s):                  623.49
Total input tokens:                      2081726
Total input text tokens:                 2081726
Total generated tokens:                  2087288
Total generated tokens (retokenized):    2044666
Request throughput (req/s):              6.57
Input token throughput (tok/s):          3338.85
Output token throughput (tok/s):         3347.77
Peak output token throughput (tok/s):    6349.00
Peak concurrent requests:                270
Total token throughput (tok/s):          6686.62
Concurrency:                             250.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   38108.46
Median E2E Latency (ms):                 37186.80
P90 E2E Latency (ms):                    69325.24
P99 E2E Latency (ms):                    77776.90
---------------Time to First Token----------------
Mean TTFT (ms):                          436.49
Median TTFT (ms):                        114.90
P99 TTFT (ms):                           6938.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          75.02
Median TPOT (ms):                        76.02
P99 TPOT (ms):                           92.27
---------------Inter-Token Latency----------------
Mean ITL (ms):                           74.07
Median ITL (ms):                         38.45
P95 ITL (ms):                            230.42
P99 ITL (ms):                            242.70
Max ITL (ms):                            7181.72
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

**Environment**

* Hardware: H200 (4x)
* Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16
* Tensor Parallelism: 4
* SGLang Version: main branch

**Launch Model**

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \
  --trust-remote-code \
  --tp 4 \
  --reasoning-parser nemotron_3
```

**Run Benchmark**

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.950
Invalid: 0.000
Latency: 21.442 s
Output throughput: 996.815 token/s
```

#### 5.2.2 MMLU Benchmark

**Run Benchmark**

```bash Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
subject: abstract_algebra, #q:100, acc: 0.730
subject: anatomy, #q:135, acc: 0.830
subject: astronomy, #q:152, acc: 0.934
subject: business_ethics, #q:100, acc: 0.830
subject: clinical_knowledge, #q:265, acc: 0.879
subject: college_biology, #q:144, acc: 0.931
subject: college_chemistry, #q:100, acc: 0.620
subject: college_computer_science, #q:100, acc: 0.840
subject: college_mathematics, #q:100, acc: 0.820
subject: college_medicine, #q:173, acc: 0.821
subject: college_physics, #q:102, acc: 0.794
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.919
subject: econometrics, #q:114, acc: 0.746
subject: electrical_engineering, #q:145, acc: 0.828
subject: elementary_mathematics, #q:378, acc: 0.926
subject: formal_logic, #q:126, acc: 0.857
subject: global_facts, #q:100, acc: 0.570
subject: high_school_biology, #q:310, acc: 0.952
subject: high_school_chemistry, #q:203, acc: 0.828
subject: high_school_computer_science, #q:100, acc: 0.940
subject: high_school_european_history, #q:165, acc: 0.861
subject: high_school_geography, #q:198, acc: 0.939
subject: high_school_government_and_politics, #q:193, acc: 0.990
subject: high_school_macroeconomics, #q:390, acc: 0.928
subject: high_school_mathematics, #q:270, acc: 0.700
subject: high_school_microeconomics, #q:238, acc: 0.966
subject: high_school_physics, #q:151, acc: 0.834
subject: high_school_psychology, #q:545, acc: 0.960
subject: high_school_statistics, #q:216, acc: 0.852
subject: high_school_us_history, #q:204, acc: 0.926
subject: high_school_world_history, #q:237, acc: 0.937
subject: human_aging, #q:223, acc: 0.879
subject: human_sexuality, #q:131, acc: 0.939
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.898
subject: logical_fallacies, #q:163, acc: 0.914
subject: machine_learning, #q:112, acc: 0.821
subject: management, #q:103, acc: 0.903
subject: marketing, #q:234, acc: 0.944
subject: medical_genetics, #q:100, acc: 0.980
subject: miscellaneous, #q:783, acc: 0.945
subject: moral_disputes, #q:346, acc: 0.861
subject: moral_scenarios, #q:895, acc: 0.542
subject: nutrition, #q:306, acc: 0.902
subject: philosophy, #q:311, acc: 0.884
subject: prehistory, #q:324, acc: 0.920
subject: professional_accounting, #q:282, acc: 0.805
subject: professional_law, #q:1534, acc: 0.681
subject: professional_medicine, #q:272, acc: 0.923
subject: professional_psychology, #q:612, acc: 0.889
subject: public_relations, #q:110, acc: 0.800
subject: security_studies, #q:245, acc: 0.837
subject: sociology, #q:201, acc: 0.960
subject: us_foreign_policy, #q:100, acc: 0.920
subject: virology, #q:166, acc: 0.590
subject: world_religions, #q:171, acc: 0.906
Total latency: 150.267
Average accuracy: 0.841
```


# NVIDIA Nemotron3-Ultra
Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra

Deploy NVIDIA Nemotron3-Ultra with SGLang - 550B hybrid MoE model (55B active) with 1M context window, BF16/NVFP4 support, built for long-running autonomous agents.

## 1. Model Introduction

`NVIDIA Nemotron3-Ultra` is an open frontier reasoning model in the Nemotron 3 family, built for long-running autonomous agents. It is optimized for complex orchestration across coding, deep research, enterprise workflows, and EDA use cases where agents must sustain reasoning across many steps and large context windows.

Nemotron 3 Ultra is a 550B parameter hybrid MoE model that activates only 55B parameters per forward pass, delivering frontier reasoning accuracy with high-throughput inference. It supports a 1M token context window so agents can keep conversation history, tool outputs, and plan state in view across persistent workflows.

Architecture and key features:

* **Hybrid Transformer-Mamba Architecture (MoE):** Combines Mixture of Experts with a hybrid Transformer-Mamba architecture, enabling efficient routing and sequence modeling in a single stack.
* **Long-horizon agentic reasoning:** Tuned for agents that plan, call tools, inspect results, recover from failures, and continue working across long task horizons — coding, deep research, enterprise automation, and EDA.
* **1M token context window:** Sustains coherent agent state across extended workflows without re-ingestion.
* **BF16 and NVFP4 quantization:** Deployable from multi-node H100 down to a single Blackwell node with NVFP4.
* **Multi-environment RL post-training:** Post-trained with reinforcement learning across multiple environments for robust reasoning and reliable agentic behavior.
* **Open weights, open data, open recipes:** Customizable for domain-specific agents and deployable across your own infrastructure.

**Modalities:** Input: text — Output: text

**Supported GPUs:**

* **BF16:** 16×H100, 16×H200, 8×B200/B300
* **NVFP4:** 4/8×B200/B300, 4×GB200/GB300

Available model variants on HuggingFace:

* [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16)
* [`nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4)

## 2. SGLang Installation

Nemotron3-Ultra support is included in the latest stable release.

```bash Command theme={null}
docker pull lmsysorg/sglang:latest
```

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance tuning.

### 3.1 Basic Configuration

**Interactive Command Generator**: select model precision, hardware, tensor parallelism, and common knobs to generate a launch command.

The generator only emits a runnable command for combinations that NVIDIA / SGLang have validated. Selecting an unverified tuple (e.g. NVFP4 on H100/H200, BF16 with TP=4 on H100, …) is **blocked** — the command pane shows an explicit error and the verified support matrix instead of a launch line, so unvalidated commands can't be copied by accident.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Attention backend**:

  **H100/H200**: Use flash attention 3 backend by default.
  **B200/GB200/B300/GB300**: Set `--attention-backend trtllm_mha`. The flashinfer default breaks the overlap scheduler on Blackwell, so `trtllm_mha` is required there.

* **Mamba scheduler strategy**:

  Always launch with `--mamba-radix-cache-strategy extra_buffer`. This hybrid Transformer-Mamba model requires the `extra_buffer` strategy for correct scheduling of its Mamba state.

* **Mamba backend**:

  The Mamba layers use the Triton SSM kernels by default. For better performance, set `--mamba-backend flashinfer` to use the FlashInfer Mamba kernels instead.

* **Mamba SSM precision**:

  The SSM state dtype defaults to the model config value. Set `--mamba-ssm-dtype float16` to store the Mamba states in FP16, which reduces mamba cache memory without significant accuracy loss.

* **Mamba SSM stochastic rounding**:

  When storing the Mamba states in FP16, add `--enable-mamba-cache-stochastic-rounding` to round SSM cache writes stochastically and reduce accumulation bias. It requires `--mamba-ssm-dtype float16` and CUDA; with the default `--mamba-backend triton` it additionally requires SM100. Use `--mamba-cache-philox-rounds` to control the number of Philox rounds (`0` uses the backend default).

* **TP support**:

  To set tp size, use `--tp <4|8|16>`. Recommended pairings:

  * BF16: `--tp 16` on H100/H200, `--tp 8` on B200/B300
  * NVFP4: `--tp 4` or `--tp 8` on B200/B300, `--tp 4` on GB200/GB300

* **Multi-node BF16 on H100**:

  The 16×H100 BF16 setup spans two nodes. Use `--dist-init-addr <head-node-ip>:5000 --nnodes 2 --node-rank <0|1>` on each node and keep `--tp 16`.

* **DP attention**:

  By default the attention layers are tensor-parallel (sharded across all TP ranks). Enabling DP attention (the toggle above, or `--attn-dp-size <N>`) instead runs attention as `N` data-parallel groups: each DP rank serves its own slice of the requests with its own KV cache. `--attn-dp-size` must divide `--tp`.

* **Expert parallel (EP)**:

  This MoE only supports `ep_size == 1` (off) or `ep_size == tp_size`. To enable expert parallelism, set `--ep <tp>` with the same value as `--tp`.

* **Multi-token prediction (MTP)**:

  Enable MTP speculative decoding (the toggle above) for low latency.

* **FP8 KV cache**:

  To enable fp8 kv cache, set `--kv-cache-dtype fp8_e4m3`. This is enabled by default on the NVFP4 checkpoint.

* **Reasoning parser**:

  Set `--reasoning-parser nemotron_3` to enable structured reasoning traces (`reasoning_content` field in the response).

* **Tool calling**:

  Set `--tool-call-parser qwen3_coder` to enable tool calling support.

## 4. Model Invocation

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \
  --trust-remote-code \
  --tp 8 \
  --mamba-radix-cache-strategy extra_buffer \
  --attention-backend trtllm_mha \
  --tool-call-parser qwen3_coder \
  --reasoning-parser nemotron_3
```

### 4.1 Basic Usage (OpenAI-Compatible API)

SGLang provides an OpenAI-compatible endpoint. Example with the OpenAI Python client:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "Give me 3 bullet points about SGLang."},
    ],
    temperature=0.6,
    max_tokens=1024,
)
print("Reasoning:", resp.choices[0].message.reasoning_content, "\nContent:", resp.choices[0].message.content)
print("\n")
```

Output:

```text Output theme={null}
Reasoning: The user wants 3 bullet points about SGLang. Let me recall what I know about SGLang — it's a high-performance serving framework for large language models with a focus on structured generation and efficient KV cache reuse...(more tokens)

Content: - **Radix Attention** — SGLang reuses KV cache across requests sharing a common prefix, dramatically reducing memory and compute for multi-turn agent loops and few-shot workloads.
- **OpenAI-compatible API and structured generation** — Drop-in replacement for the OpenAI client, with first-class support for constrained decoding (JSON schema, regex) and OpenAI-style tool calling.
- **High-throughput serving on NVIDIA GPUs** — Continuous batching, chunked prefill, FP8/NVFP4 quantization, and optimized CUDA kernels deliver state-of-the-art throughput across H100, H200, B200, and GB200.
```

Streaming chat completion:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

stream = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant."},
        {"role": "user", "content": "What are the first 5 prime numbers?"}
    ],
    temperature=0.7,
    max_tokens=1024,
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        print(delta.content, end="", flush=True)
```

Output:

```text Output theme={null}
The first 5 prime numbers are:
**2, 3, 5, 7, 11**.

### Explanation:
- A **prime number** is a natural number greater than 1 whose only positive divisors are 1 and itself.
- **2** is the smallest prime and the only even prime.
- **3, 5, 7, 11** are each divisible only by 1 and themselves.
- **1** is not prime by definition (it has only one positive divisor).
- **4, 6, 8, 9, 10** are composite.
```

### 4.2 Reasoning

The model supports two modes — Reasoning ON (default) vs OFF. This can be toggled by setting `enable_thinking` to `False`, as shown below.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

# Reasoning on (default)
print("Reasoning on")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Plan a 3-step approach to debug a flaky integration test. Keep the thinking process short."}
    ],
    temperature=1,
    max_tokens=1024,
)
print(f"Reasoning: \n{resp.choices[0].message.reasoning_content[:200]}... \nContent: \n{resp.choices[0].message.content[:200]}...")
print("\n")
# Reasoning off
print("Reasoning off")
resp = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Give me 3 facts about SGLang."}
    ],
    temperature=0,
    max_tokens=256,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
print(f"Content: \n{resp.choices[0].message.content[:200]}...")
```

Output:

```text Output theme={null}
Reasoning on
Reasoning:
The user wants a short reasoning chain plus a 3-step debug plan for a flaky integration test. I'll think briefly about common causes (timing/race, shared state, external service variance) and pick a t...
Content:
1. **Reproduce deterministically** — run the test in a loop (e.g. 50–100x) with logging at the suspected race points to confirm the failure rate and surface ordering.
2. **Isolate state** — re-run with...

Reasoning off
Content:
Here are 3 facts about SGLang:

1. **High-performance LLM serving system** developed at UC Berkeley with contributions from a broad open-source community, focused on throughput and latency at scale.
...
```

### 4.3 Tool Calling

Call functions using the OpenAI Tools schema and inspect returned `tool_calls`. The server must be launched with `--tool-call-parser qwen3_coder`.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

# Tool calling via OpenAI tools schema
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search_codebase",
            "description": "Search the project codebase for a symbol or pattern.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "The symbol, function name, or regex to search for"
                    },
                    "path": {
                        "type": "string",
                        "description": "Optional sub-path to restrict the search to"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

completion = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16",
    messages=[
        {"role": "system", "content": "You are a coding agent. Use tools to inspect the repo before answering."},
        {"role": "user", "content": "Where is the `RadixCache` class defined?"}
    ],
    tools=TOOLS,
    temperature=0.6,
    top_p=0.95,
    max_tokens=512,
    stream=False
)

print(completion.choices[0].message.reasoning_content)
print(completion.choices[0].message.tool_calls)
```

Output:

```text Output theme={null}
The user is asking where the RadixCache class is defined. I should search the codebase for the symbol "RadixCache" to find the file and line. I'll call search_codebase with that query.

[ChatCompletionMessageFunctionToolCall(id='call_8a7f2c4e1b9d4a3e8c2f1d6b', function=Function(arguments='{"query": "class RadixCache"}', name='search_codebase'), type='function', index=0)]
```

### 4.4 Controlling Reasoning Budget

The `reasoning_budget` parameter allows you to limit the length of the model's reasoning trace. When the reasoning output reaches the specified token budget, the model will attempt to gracefully end the reasoning at the next newline character.

If no newline is encountered within 500 tokens after reaching the budget threshold, the reasoning trace will be forcibly terminated at `reasoning_budget + 500` tokens.

```python Example theme={null}
from typing import Any, Dict, List
import openai
from transformers import AutoTokenizer

class ThinkingBudgetClient:
    def __init__(self, base_url: str, api_key: str, tokenizer_name_or_path: str):
        self.base_url = base_url
        self.api_key = api_key
        self.tokenizer = AutoTokenizer.from_pretrained(tokenizer_name_or_path)
        self.client = openai.OpenAI(base_url=self.base_url, api_key=self.api_key)

    def chat_completion(
        self,
        model: str,
        messages: List[Dict[str, Any]],
        reasoning_budget: int = 512,
        max_tokens: int = 1024,
        **kwargs,
    ) -> Dict[str, Any]:
        assert (
            max_tokens > reasoning_budget
        ), f"reasoning_budget must be smaller than max_tokens. Given {max_tokens=} and {reasoning_budget=}"

        # 1. first call chat completion to get reasoning content
        response = self.client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=reasoning_budget,
            **kwargs
        )

        reasoning_content = response.choices[0].message.reasoning_content or ""

        if "</think>" not in reasoning_content:
            # reasoning content is too long, closed with a period (.)
            reasoning_content = f"{reasoning_content}.\n</think>\n\n"

        reasoning_tokens_used = len(
            self.tokenizer.encode(reasoning_content, add_special_tokens=False)
        )
        remaining_tokens = max_tokens - reasoning_tokens_used

        assert (
            remaining_tokens > 0
        ), f"remaining tokens must be positive. Given {remaining_tokens=}. Increase max_tokens or lower reasoning_budget."

        # 2. append reasoning content to messages and call completion
        messages.append({"role": "assistant", "content": reasoning_content})
        prompt = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            continue_final_message=True,
        )

        response = self.client.completions.create(
            model=model,
            prompt=prompt,
            max_tokens=remaining_tokens,
            **kwargs
        )

        response_data = {
            "reasoning_content": reasoning_content.strip().strip("</think>").strip(),
            "content": response.choices[0].text,
            "finish_reason": response.choices[0].finish_reason,
        }
        return response_data
```

Usage example with `reasoning_budget=256`:

```python Example theme={null}
SERVED_MODEL_NAME = "nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16"

# Client
client = ThinkingBudgetClient(
    base_url="http://127.0.0.1:30000/v1",
    api_key="null",
    tokenizer_name_or_path=SERVED_MODEL_NAME
)

resp = client.chat_completion(
    model=SERVED_MODEL_NAME,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Outline a research plan to evaluate the throughput of two MoE serving strategies."}
    ],
    temperature=1,
    max_tokens=1024,
    reasoning_budget=256
)
print("Reasoning:", resp["reasoning_content"], "\nContent:", resp["content"])
```

Output:

```text Output theme={null}
Reasoning: The user wants a research plan to compare throughput of two MoE serving strategies. I should outline goals, baselines, datasets, metrics (tokens/s, TTFT, ITL, MFU), variables to sweep (TP, batch size, sequence length, concurrency), and statistical handling. Keep it concise since reasoning_budget is 256...
Content:

**Research plan**

1. **Define goal & metrics** — peak token throughput (input+output), TTFT, P99 ITL, MFU; measured at fixed accuracy.
2. **Choose baselines** — Strategy A (TP-only) vs Strategy B (TP + expert-parallel). Hold model checkpoint, precision, and KV-cache dtype constant.
3. **Sweep** — `{batch ∈ 1,4,16,64, concurrency ∈ 16,64,256, seq_len ∈ 1k,8k,32k}` per strategy.
4. **Workload** — `sglang.bench_serving --dataset-name random` with matched input/output budgets.
5. **Analysis** — per-config throughput table + roofline overlay; bootstrap CIs over 3 reruns to bound noise.
```

***

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: GB200 (4x)

* Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4

* Tensor Parallelism: 4

* SGLang Version: main branch

* Model Deployment Command:

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
  --trust-remote-code \
  --tp 4 \
  --mamba-radix-cache-strategy extra_buffer \
  --attention-backend trtllm_mha \
  --max-running-requests 1024
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 4096 \
  --max-concurrency 256
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 256
Successful requests:                     4096
Benchmark duration (s):                  1184.58
Total input tokens:                      2081726
Total input text tokens:                 2081726
Total generated tokens:                  2087288
Total generated tokens (retokenized):    1990224
Request throughput (req/s):              3.46
Input token throughput (tok/s):          1757.35
Output token throughput (tok/s):         1762.05
Peak output token throughput (tok/s):    3150.00
Peak concurrent requests:                266
Total token throughput (tok/s):          3519.40
Concurrency:                             249.55
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   72169.95
Median E2E Latency (ms):                 71994.47
P90 E2E Latency (ms):                    99898.56
P99 E2E Latency (ms):                    107119.61
---------------Time to First Token----------------
Mean TTFT (ms):                          40057.33
Median TTFT (ms):                        41375.93
P99 TTFT (ms):                           46377.89
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          63.15
Median TPOT (ms):                        63.65
P99 TPOT (ms):                           78.16
---------------Inter-Token Latency----------------
Mean ITL (ms):                           63.14
Median ITL (ms):                         35.92
P95 ITL (ms):                            178.10
P99 ITL (ms):                            182.10
Max ITL (ms):                            2466.36
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

**Environment**

* Hardware: GB200 (4x)
* Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
* Tensor Parallelism: 4
* SGLang Version: main branch

**Launch Model**

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \
  --trust-remote-code \
  --tp 4 \
  --mamba-radix-cache-strategy extra_buffer \
  --attention-backend trtllm_mha \
  --reasoning-parser nemotron_3
```

**Run Benchmark**

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

**Test Results:**

```text Output theme={null}
Accuracy: 0.970
Invalid: 0.000
Latency: 29.129 s
Output throughput: 745.333 token/s
```

#### 5.2.2 MMLU Benchmark

**Run Benchmark**

```bash Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

**Test Results:**

```text Output theme={null}
TBD
```

***


# Nemotron3.5-Lightning
Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3.5-Lightning

Deploy NVIDIA Nemotron 3.5 Lightning with SGLang — NVFP4 serving with MTP, DFlash, and DSpark speculative decoding, reasoning, and tool calling.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> SGLANG_BUILD_RUST_EXTS=none uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-nemotron3-5-lightning
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware and recipe to generate the launch command. Every platform publishes four operating points: **Balanced** (no speculation) plus three speculative decoders — **MTP**, **DFlash**, and **DSpark**. Use the Playground below to explore knobs beyond them.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

**NVIDIA Nemotron 3.5 Lightning** is a 30B-A3B hybrid reasoning LLM. See the Hugging Face model cards below for architecture and evaluation details.

<table>
  <thead>
    <tr>
      <th>Checkpoint</th>
      <th>Precision</th>
      <th>Use</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4)**</td>
      <td>NVFP4</td>
      <td>Serving — the checkpoint this page deploys</td>
    </tr>

    <tr>
      <td>**[NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16)**</td>
      <td>BF16</td>
      <td>Full-precision reference</td>
    </tr>

    <tr>
      <td>**[NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash)**</td>
      <td>W4A16</td>
      <td>DFlash speculative draft model</td>
    </tr>

    <tr>
      <td>**[NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark)**</td>
      <td>W4A16</td>
      <td>DSpark speculative draft model</td>
    </tr>
  </tbody>
</table>

MTP needs no separate download — the draft head is embedded in the target checkpoint.

## 2. Usage

The server speaks the OpenAI API. With `--reasoning-parser nemotron_3` enabled, the thinking trace lands in `message.reasoning_content` and the answer in `message.content`.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="null",
)

response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Briefly explain: what is SGLang?"},
    ],
    temperature=1.0,
    top_p=0.95,
    max_tokens=1024,
)
choice = response.choices[0]
print("Reasoning:", choice.message.reasoning_content)
print("Content:", choice.message.content)
```

### 2.1 Tool Calling

With `--tool-call-parser qwen3_coder` enabled, structured tool calls are returned in `message.tool_calls`.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="null",
)

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "calculate_tip",
            "parameters": {
                "type": "object",
                "properties": {
                    "bill_total": {"type": "integer", "description": "The total amount of the bill"},
                    "tip_percentage": {"type": "integer", "description": "The percentage of tip to be applied"},
                },
                "required": ["bill_total", "tip_percentage"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
    messages=[{"role": "user", "content": "My bill is $50. What will be the amount for 15% tip?"}],
    tools=TOOLS,
    max_tokens=1024,
)
choice = response.choices[0]
print("Content:", choice.message.content)
print("Tool calls:", choice.message.tool_calls)
```


# NVIDIA Nemotron 3.5 Super VL
Source: https://docs.sglang.io/cookbook/autoregressive/NVIDIA/Nemotron3.5-Super

Deploy Nemotron 3.5 Super VL 120B-A12B with SGLang on B200, H200, RTX PRO 6000, and DGX Spark, with NVFP4 or BF16 and optional MTP.

## Deployment

The **Docker** commands use `lmsysorg/sglang:nemotron3-5-super`. The command panel selects either `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4` or `nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16`. Set `HF_TOKEN` in the panel's Env control if the checkpoint requires access.

Choose **B200, H200, RTX PRO 6000, or DGX Spark** on one node. B200 offers NVFP4 and BF16. H200 offers BF16; NVFP4 is greyed out. RTX PRO 6000 and DGX Spark offer NVFP4; BF16 is greyed out. Choose `bfloat16` or `float32` in **Mamba SSM DType**; `bfloat16` is the default. **High Throughput** leaves MTP off. **Low Latency** adds EAGLE MTP with top-k 1, three steps, and four draft tokens. These names identify configuration choices; no throughput or latency comparison was measured. Both command tabs use `sglang serve`.

## Playground

The Playground inherits the selected recipe and lets you toggle the reasoning and tool-call parsers. Parser changes are separate experiments.

## 1. Model introduction

**NVIDIA Nemotron 3.5 Super VL 120B-A12B** is a vision-language checkpoint in the Nemotron-H hybrid MoE family. The early-access NGC checkpoint configuration identifies `NemotronH_Omni_Reasoning_V3` and includes a vision encoder. The command panel uses the [NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4) and [BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16) Hugging Face IDs; the source runs used the [NGC model](https://registry.ngc.nvidia.com/orgs/0767305323357365/teams/super3-5-vl-ea-0911/models/nemotron-3.5-super-vl-120b-a12b-ea20261008). Check the checkpoint pages for access terms and model details.

The SGLang model entry point is [`nano_nemotron_vl.py` on the `nemotron3-super-dev-20261015` branch](https://github.com/sgl-project/sglang/blob/nemotron3-super-dev-20261015/python/sglang/srt/models/nano_nemotron_vl.py), with its supporting kernels and runtime code on the same branch.

The validation used text-only GSM8K prompts. It does **not** establish image or video request correctness, long-context behavior, or tool-call accuracy.

## 2. Configuration tips and validation

* **B200, NVFP4 or BF16:** both strategies use one eight-GPU TP8 node with FP8 E4M3 KV cache and 16,384-token configured context. Select the Mamba state dtype in the wizard; SGLang selects the attention backend and memory limits by default.
* **H200, BF16:** both strategies use one eight-GPU TP8 node with FA3 attention, Triton MoE, and `--disable-flashinfer-autotune`.
* **RTX PRO 6000, NVFP4:** both strategies served at TP1 on GPU 1 of an eight-GPU G4 machine. The configured context was 262,144 tokens; only short prompts were tested.
* **DGX Spark, NVFP4:** both strategies completed 300-question text-only GSM8K runs at TP1 with `bfloat16` Mamba state. The `float32` choice has no completed accuracy run.
* **KV precision:** these recipes deliberately use `--kv-cache-dtype fp8_e4m3` to match the validated runs. KV precision can affect output quality; no BF16-KV comparison was measured.

Server information confirmed `speculative_algorithm: null` for High Throughput and `EAGLE` for Low Latency. The runs used installed early-access NGC weights, not the Hugging Face checkpoint IDs in the command panel or the public Docker image.

The Deployment panel’s Benchmark card shows text-only GSM8K accuracy for measured selections and a pending state for unmeasured selections. The existing green Verified status marks a recipe with a completed source-run. These results do not validate the public checkpoint IDs or Docker image.

## 3. Reasoning output

The base recipes enable `--reasoning-parser nemotron_3`. In the tested chat completions, the thinking trace appeared in `message.reasoning_content` and the final answer in `message.content`. The tool-call parser is configured as `qwen3_coder`, but structured tool calls were not part of this accuracy check.

> **Reasoning example (Python client)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
> model = client.models.list().data[0].id
> response = client.chat.completions.create(
>     model=model,
>     messages=[{
>         "role": "user",
>         "content": "Claire makes a 3 egg omelet every morning for breakfast.  How many dozens of eggs will she eat in 4 weeks?\nSolve the problem and put the final numeric answer after ####.",
>     }],
> )
> message = response.choices[0].message
> print("Reasoning:", message.reasoning_content)
> print("Content:", message.content)
> ```

> **Example output from the B200 NVFP4 smoke run**
>
> ```text Output theme={null}
> Reasoning: The user wants me to solve a math problem.
>
> Claire makes a 3 egg omelet every morning.
> She eats 3 eggs per day.
> 4 weeks = 4 × 7 = 28 days.
> Total eggs = 3 × 28 = 84 eggs.
> Dozens of eggs = 84 / 12 = 7 dozens.
>
> Final answer: 7.
>
> I need to put the final numeric answer after ####.
> Content: Claire eats 3 eggs per day.
> In 4 weeks, there are 4 × 7 = 28 days.
> Total eggs eaten: 3 × 28 = 84 eggs.
> Since 1 dozen = 12 eggs: 84 ÷ 12 = 7 dozens.
>
> #### 7
> ```


# GPT-OSS
Source: https://docs.sglang.io/cookbook/autoregressive/OpenAI/GPT-OSS



## 1.Model Introduction

[GPT-OSS](https://huggingface.co/openai/gpt-oss-20b) is an advanced large language model developed by OpenAI designed for power reasoning, agentic tasks, and versatile developer use cases. It has versions with two model sizes.

* **gpt-oss-120b** — for production, general purpose, high reasoning use cases that fit into a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) (117B parameters with 5.1B active parameters)
* **gpt-oss-20b** — for lower latency, and local or specialized use cases (21B parameters with 3.6B active parameters)

GPT-OSS introduces several groundbreaking innovations:

* **Configurable reasoning effort**: Easily adjust the reasoning effort (low, medium, high) based on your specific use case and latency needs.
* **Full chain-of-thought**: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs. It’s not intended to be shown to end users.
* **Fine-tunable**: Fully customize models to your specific use case through parameter fine-tuning.
* **Agentic capabilities**: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs.
* **MXFP4 quantization**: The models were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU (like NVIDIA H100 80GB or AMD MI300X 192GB) and the gpt-oss-20b model run within 16GB of memory. All evals were performed with the same MXFP4 quantization.

## 2.SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3.Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The GPT-OSS series comes in two sizes. Recommended starting configurations vary depending on hardware.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Native web search:** Set `EXA_API_KEY` in the SGLang server environment to enable built-in web search (Exa). No `--tool-server` is required, and requests are tagged with `x-exa-integration: sglang`.
* **Web search defaults:** `numResults=10`, search `type="auto"`, and `contents.highlights=true`. Override with `SGLANG_EXA_NUM_RESULTS`, `SGLANG_EXA_SEARCH_TYPE`, and `SGLANG_EXA_INCLUDE_HIGHLIGHTS`.
* **Python tool:** Add `--tool-server demo` to enable the Python interpreter. Runs in a Docker sandbox by default; set `PYTHON_EXECUTION_BACKEND=UV` to run on the host (model-generated code executes locally — use with care).
* **MCP tool servers:** For production, point SGLang at external MCP SSE servers with `--tool-server ip-1:port-1,ip-2:port-2`.
* **Responses API:** GPT-OSS supports OpenAI's Responses API (`client.responses.create`) in addition to the standard Chat Completions API (see section 4.2.4).
* **Use Python 3.12** when running the demo Python tool.
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4.Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

GPT-OSS supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --reasoning-parser gpt-oss \
  --tp 8
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user asks: "Solve this problem step by step: What is 15% of 240?" So we need to provide step-by-step solution. Compute 15% of 240: 0.15 * 240 = 36. Provide steps: convert percent to decimal, multiply, maybe use fraction. Provide answer.
=============== Content =================
**Step‑by‑step solution**

1. **Understand what “percent” means**
   “15 %” means 15 out of every 100 parts, i.e. the fraction \(\displaystyle \frac{15}{100}\).

2. **Convert the percent to a decimal (or fraction)**
   \[
   \frac{15}{100}=0.15
   \]

3. **Set up the multiplication**
   To find 15 % of 240 we multiply 240 by the decimal 0.15:
   \[
   240 \times 0.15
   \]

4. **Do the multiplication**
   One convenient way is to break it into two easier parts:
   \[
   240 \times 0.15 = 240 \times \left(\frac{15}{100}\right)
                = \frac{240 \times 15}{100}
   \]

   - First compute \(240 \times 15\):
     \[
     240 \times 15 = 240 \times (10 + 5) = 2400 + 1200 = 3600
     \]

   - Then divide by 100:
     \[
     \frac{3600}{100} = 36
     \]

5. **Write the result**
   \[
   15\% \text{ of } 240 = 36
   \]

---

**Answer:** \(36\)
```

#### 4.2.2 Tool Calling

GPT-OSS supports tool calling capabilities. Enable the tool call parser:

**Python Example (without Thinking Process):**

Start sglang server:

```shell Command theme={null}
python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --tool-call-parser gpt-oss \
  --tp 8
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Python Example (with Thinking Process):**

Start sglang server:

```shell Command theme={null}
python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --reasoning-parser gpt-oss \
  --tool-call-parser gpt-oss \
  --tp 8
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
User asks: "What's the weather in Beijing?" We need to get current weather. Use function get_weather with location "Beijing". No unit specified; default? Probably use default (maybe Celsius). We can specify unit as "celsius". We'll call function.
=============== Content =================
🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The current weather in Beijing is 22 °C and sunny. Let me know if you’d like a forecast for the next few days or any other details!"
```

#### 4.2.3 EAGLE3 Speculative Decoding

SGLang supports speculative decoding for GPT-OSS models using the EAGLE3 algorithm. This can significantly improve decoding speed, especially for small batch sizes.

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path openai/gpt-oss-120b \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path lmsys/EAGLE3-gpt-oss-120b-bf16 \
  --tp 2
```

> [!TIP]
>
> The spec-v2 overlap scheduler is enabled by default. It improves performance by overlapping draft and verification stages. Pass `--disable-overlap-schedule` to disable.

#### 4.2.4 Responses API and Built-in Tools

GPT-OSS supports the OpenAI Responses API with built-in tool use (web search and Python interpreter). Set `EXA_API_KEY` to enable native web search; add `--tool-server demo` only when you also want the Python tool:

```shell Command theme={null}
export EXA_API_KEY=YOUR_EXA_KEY
# Optional: server-side Exa tuning (defaults shown)
export SGLANG_EXA_NUM_RESULTS=10
export SGLANG_EXA_SEARCH_TYPE=auto
export SGLANG_EXA_INCLUDE_HIGHLIGHTS=true
# Optional: run Python tool on host instead of Docker (model code executes locally)
export PYTHON_EXECUTION_BACKEND=UV

python3 -m sglang.launch_server \
  --model-path openai/gpt-oss-120b \
  --tp 2
```

For production, use external MCP SSE servers instead of `demo`:

```shell Command theme={null}
mcp run -t sse browser_server.py:mcp
mcp run -t sse python_server.py:mcp

python -m sglang.launch_server \
  --model-path openai/gpt-oss-120b \
  --tool-server ip-1:port-1,ip-2:port-2 \
  --tp 2
```

**Example using Responses API:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="sk-123456")

search_tools = [{"type": "web_search"}]
python_tools = [{"type": "code_interpreter"}]

# Configurable reasoning effort: "high", "medium", or "low"
response = client.responses.create(
    model="openai/gpt-oss-120b",
    instructions="You are a helpful assistant.",
    reasoning_effort="high",
    input="In one sentence, explain the transformer architecture.",
)
print(response.output_text)

# Web search (requires EXA_API_KEY on the SGLang server)
response = client.responses.create(
    model="openai/gpt-oss-120b",
    instructions="You are a helpful assistant, you can search the web when needed.",
    input="Search the web for the latest news about Nvidia stock price",
    tools=search_tools,
)
print(response.output_text)

# Python tool (requires launching SGLang with --tool-server demo)
response = client.responses.create(
    model="openai/gpt-oss-120b",
    instructions="You are a helpful assistant, you could use python tool to execute code.",
    input="Use python tool to calculate the sum of 29138749187 and 29138749187",
    tools=python_tools,
)
print(response.output_text)
# Output: The sum is 58,277,498,374.
```

## 5.Benchmark

### 5.1 Speed Benchmark

* Hardware: NVIDIA B200 GPU (8x)
* Tensor Parallelism: 8
* Model: openai/gpt-oss-120b
* sglang version: 0.5.6

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.

#### 5.1.1 Latency-Sensitive Benchmark

* Server Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --tp 8
```

* Test Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 100 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     100
Benchmark duration (s):                  52.35
Total input tokens:                      33178
Total input text tokens:                 33178
Total input vision tokens:               0
Total generated tokens:                  21251
Total generated tokens (retokenized):    20868
Request throughput (req/s):              1.91
Input token throughput (tok/s):          633.76
Output token throughput (tok/s):         405.93
Peak output token throughput (tok/s):    433.00
Peak concurrent requests:                8
Total token throughput (tok/s):          1039.69
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   523.30
Median E2E Latency (ms):                 389.91
---------------Time to First Token----------------
Mean TTFT (ms):                          33.71
Median TTFT (ms):                        31.79
P99 TTFT (ms):                           108.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          2.31
Median TPOT (ms):                        2.31
P99 TPOT (ms):                           2.39
---------------Inter-Token Latency----------------
Mean ITL (ms):                           2.31
Median ITL (ms):                         2.31
P95 ITL (ms):                            2.35
P99 ITL (ms):                            2.38
Max ITL (ms):                            3.54
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Server Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model openai/gpt-oss-120b \
  --tp 8
```

* Test Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 1000 \
  --max-concurrency 100
```

**Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  24.76
Total input tokens:                      297156
Total input text tokens:                 297156
Total input vision tokens:               0
Total generated tokens:                  192432
Total generated tokens (retokenized):    187145
Request throughput (req/s):              40.39
Input token throughput (tok/s):          12003.57
Output token throughput (tok/s):         7773.26
Peak output token throughput (tok/s):    13780.00
Peak concurrent requests:                156
Total token throughput (tok/s):          19776.83
Concurrency:                             89.23
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2208.97
Median E2E Latency (ms):                 1591.11
---------------Time to First Token----------------
Mean TTFT (ms):                          102.94
Median TTFT (ms):                        31.53
P99 TTFT (ms):                           674.32
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.31
Median TPOT (ms):                        11.00
P99 TPOT (ms):                           91.28
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.00
Median ITL (ms):                         5.75
P95 ITL (ms):                            25.35
P99 ITL (ms):                            43.18
Max ITL (ms):                            621.42
==================================================
```

### 5.2 Accuracy Benchmark

### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:8000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Results**:

  * GPT-OSS-120b

    ```text Output theme={null}
    Accuracy: 0.880
    Invalid: 0.005
    Latency: 5.262 s
    Output throughput: 12143.675 token/s
    ```

  * GPT-OSS-20b

    ```text Output theme={null}
    Accuracy: 0.535
    Invalid: 0.165
    Latency: 4.157 s
    Output throughput: 19589.165 token/s
    ```


# MiniCPM-V 4.6
Source: https://docs.sglang.io/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6



## 1. Model Introduction

MiniCPM-V 4.6 is the next-generation multimodal model from [OpenBMB](https://huggingface.co/openbmb), the team behind the MiniCPM-V series. The model combines a **Qwen3.5-style hybrid LLM backbone** (Gated Delta Net + full attention) with a **NaViT-packed vision encoder** that handles arbitrary aspect ratios and high-resolution slicing natively, plus end-to-end video support.

OpenBMB ships two variants on HuggingFace:

* [`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) — base instruct model. Use this for general multimodal serving; thinking mode is still available per-request via `chat_template_kwargs.enable_thinking=true`.
* [`openbmb/MiniCPM-V-4.6-Thinking`](https://huggingface.co/openbmb/MiniCPM-V-4.6-Thinking) — thinking-tuned variant with stronger chain-of-thought behavior. Pair with the same `--reasoning-parser qwen3` flag.

**Key Features:**

* **Hybrid LLM backbone**: Qwen3.5-style mix of Gated Delta Net (linear-attention) layers and full-attention layers, providing long-context efficiency without giving up modeling power.
* **Native variable-resolution vision**: NaViT-packed vision encoder with mid-ViT merger and per-image window attention. Images of any aspect ratio are processed without forced letterboxing.
* **High-resolution slicing**: Source image plus a configurable grid of slice tiles (up to 9 tiles in the open test variant) lets the model reason over fine detail in 1280×720+ images.
* **Video**: Frame-by-frame multi-modal data items routed through the same vision encoder; any number of frames per request.
* **Reasoning Parser**: switchable thinking mode (Qwen3.5 lineage), exposed via `chat_template_kwargs.enable_thinking` per request and SGLang's `--reasoning-parser qwen3` on the server side.
* **Tool Calling**: Qwen3.5-style `<tool_call><function=…><parameter=…>…</parameter></function></tool_call>` XML format, surfaced as OpenAI-compatible `message.tool_calls` via SGLang's `--tool-call-parser qwen3_coder`. Composes with thinking mode and with image / video inputs.

**License:** [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## 2. SGLang Installation

Pull the nightly Docker image (rolling tag, tracks `main`):

```bash theme={null}
# CUDA 13 (Hopper / Blackwell, default)
docker pull lmsysorg/sglang:dev

# CUDA 12 (Ampere or older drivers). Final build: the CUDA 12 lane is retired,
# so this tag no longer tracks main.
docker pull lmsysorg/sglang:dev-cu12
```

For the general SGLang installation guide (PyPI, source, Docker) see the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to generate the appropriate deployment command. The `Variant` toggle switches between `openbmb/MiniCPM-V-4.6` (base) and `openbmb/MiniCPM-V-4.6-Thinking`. The `Reasoning Parser` and `Tool Call Parser` toggles add `--reasoning-parser qwen3` and `--tool-call-parser qwen3_coder` respectively; see §4.4 for usage details.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
  * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs.
  * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
* The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
* Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
* To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command.
* **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
* **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300.
* For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
* Multi-image and high-resolution images: the image processor produces one source patch plus per-slice tile patches; each is its own `MultimodalDataItem`. No special server-side flag needed.
* Video: decoded frame-by-frame through the same image-style slicer. No extra flag needed; pass `video_url` in the OpenAI chat completion request.
* **Chunked Prefill**: For high-concurrency vision benchmarking with many large/sliced images, pass `--chunked-prefill-size -1` to disable prefill chunking. The default chunked-prefill path can mis-split a request across an image boundary in `mm_utils.embed_mm_inputs` and crash the server; disabling chunking sidesteps this at the cost of higher TTFT under concurrency. For interactive serving leave the default on.

## 4. Model Invocation

Deploy the model on an H200:

```bash Command theme={null}
sglang serve --model-path openbmb/MiniCPM-V-4.6 \
  --trust-remote-code \
  --dtype bfloat16 \
  --mem-fraction-static 0.15 \
  --mamba-radix-cache-strategy extra_buffer \
  --page-size 64 \
  --host 0.0.0.0 --port 30000
```

### 4.1 Basic Usage (Image)

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="openbmb/MiniCPM-V-4.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://www.ilankelman.org/stopsigns/australia.jpg",
                    },
                },
                {"type": "text", "text": "Describe this image in one sentence."},
            ],
        }
    ],
    max_tokens=200,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
A black SUV drives past a Chinese-style gate with a red stop sign and traditional architecture, while storefronts and street signs line the sidewalk.
```

### 4.2 High-Resolution / Sliced Images

The image processor automatically picks a slice grid (up to 9 tiles) for high-resolution inputs. A 1280×720 source produces grid `[2, 3]`

* 7 patches with `tgt_sizes=[(24, 44), 6×(28, 36)]`, byte-for-byte matching the HF reference implementation.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openbmb/MiniCPM-V-4.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/idefics-few-shot.jpg",
                    },
                },
                {"type": "text", "text": "Describe this image in one sentence."},
            ],
        }
    ],
    max_tokens=200,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

print(response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
The Statue of Liberty stands tall against a cloudy sky, holding a torch aloft and a document in her left hand, symbolizing freedom and enlightenment.
```

### 4.3 Video Input

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openbmb/MiniCPM-V-4.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "video_url",
                    "video_url": {"url": "<your-video-url-or-file-path>"},
                },
                {"type": "text", "text": "Describe what happens in this video in one sentence."},
            ],
        }
    ],
    max_tokens=200,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

print(response.choices[0].message.content)
```

**Output Example** (run against an 8-frame synthetic test mp4 of shifting colored squares):

```text Output theme={null}
The video shows a grid of colored squares moving in a random pattern.
```

### 4.4 Advanced Usage

#### 4.4.1 Reasoning Parser

Pass `--reasoning-parser qwen3` to the server (toggle "Reasoning Parser" on in §3.1, default) so SGLang splits each response on the `<think>` / `</think>` boundaries: the pre-`</think>` block goes to `reasoning_content`, the post-`</think>` text to `content`. Per-request, the chat template's `enable_thinking` flag toggles whether the model actually emits reasoning.

* **Thinking mode** (default, `enable_thinking=true`): assistant prompt ends with `<think>\n`; the model writes reasoning, closes with `</think>`, then the answer. `reasoning_content` and `content` are both populated.
* **Instruct mode** (`enable_thinking=false`): the chat template injects an empty `<think></think>` placeholder so the model emits no thinking tokens; `reasoning_content` ends up empty.

```python Example (thinking mode) theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="openbmb/MiniCPM-V-4.6",
    messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}],
    max_tokens=200,
)

msg = response.choices[0].message
print("reasoning_content:", msg.reasoning_content)
print("content          :", msg.content)
```

```text Output theme={null}
reasoning_content: Got it, let's see. The user wants a reply with "hi" and no explanation. So I need to just say "hi" as the response. ...
content          : hi
```

```python Example (instruct mode) theme={null}
response = client.chat.completions.create(
    model="openbmb/MiniCPM-V-4.6",
    messages=[{"role": "user", "content": "Reply with the single word 'hi'. No explanation."}],
    max_tokens=200,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

msg = response.choices[0].message
print("reasoning_content:", msg.reasoning_content)
print("content          :", msg.content)
```

```text Output theme={null}
reasoning_content:
content          : hi
```

#### 4.4.2 Tool Calling

Pass `--tool-call-parser qwen3_coder` to the server (toggle "Tool Call Parser" on in §3.1) so SGLang extracts `<tool_call>` blocks from the model output into the OpenAI-style `message.tool_calls` field (with `finish_reason="tool_calls"`). The model speaks the Qwen3.5 XML tool-call format (`<tool_call><function=name><parameter=k>v</parameter></function></tool_call>`); the `qwen3_coder` parser is the right one. Tool calls compose with both reasoning modes and with image / video inputs.

> [!WARNING]
>
> Do **not** use `--tool-call-parser qwen` for MiniCPM-V 4.6 — that parser expects the older Qwen2.5 JSON format `<tool_call>{"name":..., "arguments":...}</tool_call>`, but both public 4.6 variants emit the Qwen3.5-style XML format with nested `<function=…>` and `<parameter=…>` tags. With `qwen` the outer `<tool_call>` markers match but the inner JSON parse fails, so `tool_calls` returns empty and the raw markup is left in `content`.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    },
]

response = client.chat.completions.create(
    model="openbmb/MiniCPM-V-4.6",
    messages=[{"role": "user", "content": "What is the weather in San Francisco? Use the tool."}],
    tools=tools,
    max_tokens=200,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

choice = response.choices[0]
print("finish_reason:", choice.finish_reason)
for tc in choice.message.tool_calls or []:
    print(f"  {tc.function.name}({tc.function.arguments})")
```

```text Output theme={null}
finish_reason: tool_calls
  get_weather({"location": "San Francisco", "unit": "celsius"})
```

To get the final natural-language answer, feed the tool's result back as a `tool` role message and call the API again with the same `tools` list — the model emits `finish_reason="stop"` with the answer in `content`.

## 5. Benchmark

**Common Test Environment (all benchmarks below):**

* Hardware: 1× NVIDIA H200 (141 GB), single GPU (no TP / DP)
* Docker Image: `lmsysorg/sglang:dev` (transformers 5.6.0, sgl-kernel 0.4.2.post1)
* Precision: BF16

**Common Server Launch Command:**

```bash Command theme={null}
CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \
  --model-path openbmb/MiniCPM-V-4.6 \
  --trust-remote-code \
  --dtype bfloat16 \
  --mem-fraction-static 0.5 \
  --mamba-radix-cache-strategy extra_buffer \
  --chunked-prefill-size -1 \
  --host 0.0.0.0 --port 30000
```

(`--chunked-prefill-size -1` is required for the vision throughput run; see §3.2.)

### 5.1 Accuracy Benchmark

#### 5.1.1 MMMU Benchmark

* Benchmark Command

```bash Command theme={null}
python3 benchmark/mmmu/bench_sglang.py --port 30000 --concurrency 48 --max-new-tokens 2048
```

* Test Result

```
{'Accounting': {'acc': 0.767, 'num': 30},
 'Agriculture': {'acc': 0.533, 'num': 30},
 'Architecture_and_Engineering': {'acc': 0.4, 'num': 30},
 'Art': {'acc': 0.6, 'num': 30},
 'Art_Theory': {'acc': 0.667, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.533, 'num': 30},
 'Biology': {'acc': 0.333, 'num': 30},
 'Chemistry': {'acc': 0.333, 'num': 30},
 'Clinical_Medicine': {'acc': 0.467, 'num': 30},
 'Computer_Science': {'acc': 0.333, 'num': 30},
 'Design': {'acc': 0.533, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.333, 'num': 30},
 'Economics': {'acc': 0.633, 'num': 30},
 'Electronics': {'acc': 0.5, 'num': 30},
 'Energy_and_Power': {'acc': 0.633, 'num': 30},
 'Finance': {'acc': 0.533, 'num': 30},
 'Geography': {'acc': 0.367, 'num': 30},
 'History': {'acc': 0.533, 'num': 30},
 'Literature': {'acc': 0.7, 'num': 30},
 'Manage': {'acc': 0.367, 'num': 30},
 'Marketing': {'acc': 0.733, 'num': 30},
 'Materials': {'acc': 0.367, 'num': 30},
 'Math': {'acc': 0.567, 'num': 30},
 'Mechanical_Engineering': {'acc': 0.333, 'num': 30},
 'Music': {'acc': 0.267, 'num': 30},
 'Overall': {'acc': 0.527, 'num': 900},
 'Overall-Art and Design': {'acc': 0.517, 'num': 120},
 'Overall-Business': {'acc': 0.607, 'num': 150},
 'Overall-Health and Medicine': {'acc': 0.553, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.617, 'num': 120},
 'Overall-Science': {'acc': 0.473, 'num': 150},
 'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210},
 'Pharmacy': {'acc': 0.667, 'num': 30},
 'Physics': {'acc': 0.767, 'num': 30},
 'Psychology': {'acc': 0.567, 'num': 30},
 'Public_Health': {'acc': 0.767, 'num': 30},
 'Sociology': {'acc': 0.667, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.527
```

### 5.2 Speed Benchmark

We use SGLang's built-in `bench_serving` tool with random text prompts (1000 input / 1000 output tokens) to characterize text-only serving performance.

#### 5.2.1 Latency Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model openbmb/MiniCPM-V-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  7.47
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    3554
Request throughput (req/s):              1.34
Input token throughput (tok/s):          816.44
Output token throughput (tok/s):         564.73
Peak output token throughput (tok/s):    690.00
Peak concurrent requests:                4
Total token throughput (tok/s):          1381.17
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   746.20
Median E2E Latency (ms):                 590.05
P90 E2E Latency (ms):                    1446.13
P99 E2E Latency (ms):                    1709.38
---------------Time to First Token----------------
Mean TTFT (ms):                          138.12
Median TTFT (ms):                        103.70
P99 TTFT (ms):                           330.79
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.44
Median TPOT (ms):                        1.44
P99 TPOT (ms):                           1.45
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.44
Median ITL (ms):                         1.45
P95 ITL (ms):                            1.49
P99 ITL (ms):                            1.57
Max ITL (ms):                            5.79
==================================================
```

#### 5.2.2 Throughput Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model openbmb/MiniCPM-V-4.6 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  47.07
Total input tokens:                      502493
Total input text tokens:                 502493
Total generated tokens:                  500251
Total generated tokens (retokenized):    469844
Request throughput (req/s):              21.24
Input token throughput (tok/s):          10675.32
Output token throughput (tok/s):         10627.69
Peak output token throughput (tok/s):    25911.00
Peak concurrent requests:                130
Total token throughput (tok/s):          21303.01
Concurrency:                             97.24
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4576.94
Median E2E Latency (ms):                 4331.97
P90 E2E Latency (ms):                    8634.07
P99 E2E Latency (ms):                    9636.44
---------------Time to First Token----------------
Mean TTFT (ms):                          206.50
Median TTFT (ms):                        184.72
P99 TTFT (ms):                           624.23
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.73
Median TPOT (ms):                        9.16
P99 TPOT (ms):                           13.63
---------------Inter-Token Latency----------------
Mean ITL (ms):                           8.75
Median ITL (ms):                         0.05
P95 ITL (ms):                            29.95
P99 ITL (ms):                            108.91
Max ITL (ms):                            448.40
==================================================
```

### 5.3 Vision Speed Benchmark

We use SGLang's built-in `bench_serving` tool with random images. Each request has 128 input text tokens, one 720p image, and 1024 output tokens.

#### 5.3.1 Latency Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model openbmb/MiniCPM-V-4.6 \
  --dataset-name image \
  --image-count 1 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  10.26
Total input tokens:                      767
Total input text tokens:                 750
Total input vision tokens:               17
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.97
Input token throughput (tok/s):          74.77
Output token throughput (tok/s):         411.39
Peak output token throughput (tok/s):    654.00
Peak concurrent requests:                2
Total token throughput (tok/s):          486.16
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1024.04
Median E2E Latency (ms):                 897.99
P90 E2E Latency (ms):                    1584.25
P99 E2E Latency (ms):                    1781.78
---------------Time to First Token----------------
Mean TTFT (ms):                          416.94
Median TTFT (ms):                        403.18
P99 TTFT (ms):                           477.49
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.44
Median TPOT (ms):                        1.44
P99 TPOT (ms):                           1.45
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.44
Median ITL (ms):                         1.44
P95 ITL (ms):                            1.48
P99 ITL (ms):                            1.56
Max ITL (ms):                            2.89
==================================================
```

#### 5.3.2 Throughput Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model openbmb/MiniCPM-V-4.6 \
  --dataset-name image \
  --image-count 1 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  360.01
Total input tokens:                      79925
Total input text tokens:                 78283
Total input vision tokens:               1642
Total generated tokens:                  510855
Total generated tokens (retokenized):    430289
Request throughput (req/s):              2.78
Input token throughput (tok/s):          222.01
Output token throughput (tok/s):         1419.01
Peak output token throughput (tok/s):    19620.00
Peak concurrent requests:                105
Total token throughput (tok/s):          1641.02
Concurrency:                             99.69
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   35888.57
Median E2E Latency (ms):                 35321.48
P90 E2E Latency (ms):                    41017.37
P99 E2E Latency (ms):                    60343.22
---------------Time to First Token----------------
Mean TTFT (ms):                          35096.32
Median TTFT (ms):                        34301.37
P99 TTFT (ms):                           59966.25
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          1.63
Median TPOT (ms):                        1.45
P99 TPOT (ms):                           10.15
---------------Inter-Token Latency----------------
Mean ITL (ms):                           1.58
Median ITL (ms):                         0.12
P95 ITL (ms):                            0.23
P99 ITL (ms):                            0.77
Max ITL (ms):                            2086.12
==================================================
```


# MiniCPM-V 4.7
Source: https://docs.sglang.io/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_7

Deploy OpenBMB MiniCPM-V 4.7 with SGLang — 1B dense and 35B-A3B Mixture-of-Experts vision-language models on a hybrid Gated DeltaNet backbone, in BF16, FP8, or INT4, with DSpark speculative decoding on a single GB300 or H200.

## Deployment

> **Install SGLang**
>
> MiniCPM-V 4.7 support comes from [sgl-project/sglang#41993](https://github.com/sgl-project/sglang/pull/41993); both paths below use that PR's code. For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (from source)**
>
> ```bash Command theme={null}
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> git fetch origin pull/41993/head:minicpm-v-4.7
> git checkout minicpm-v-4.7
> pip install --upgrade pip
> pip install -e "python"
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-minicpm-v-4.7
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick the hardware, variant, and quantization to generate the launch command. Every recipe runs on a single GPU and is verified on GB300 and H200; the H100 recipes (1B, FP8, INT4) are unverified. The optional **DSPARK** row adds speculative decoding; see §2 for where it helps.

## Playground

The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's validated launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

MiniCPM-V 4.7 is the latest model in OpenBMB's MiniCPM-V series, covering single-image, multi-image, and video understanding. It ships in two sizes: a 1B dense model and a 35B-A3B Mixture-of-Experts model, the series' first MoE.

Both sizes use a Qwen3.5-style hybrid language backbone in which every fourth layer is full attention and the rest are Gated DeltaNet linear attention: 24 layers for 1B, and 40 layers with 256 experts (8 active per token) for 35B-A3B. Native context length is 262,144 tokens. The vision encoder compresses visual tokens early inside the ViT, slices high-resolution images into up to 9 tiles, and supports mixed 4x/16x visual token compression (see §3.3). Thinking is on by default.

**License:** [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0).

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Checkpoint</th>
      <th>Precision</th>
      <th>Role on this page</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[openbmb/MiniCPM-V-4.7-1B](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B)</td>
      <td>BF16</td>
      <td>1B dense base (1.55B parameters including the vision encoder).</td>
    </tr>

    <tr>
      <td>[openbmb/MiniCPM-V-4.7-35B-A3B](https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B)</td>
      <td>BF16</td>
      <td>35B-A3B MoE base (35.2B total parameters, about 3B active).</td>
    </tr>

    <tr>
      <td>[openbmb/MiniCPM-V-4.7-35B-A3B-FP8](https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-FP8)</td>
      <td>FP8 W8A8 (compressed-tensors)</td>
      <td>35B-A3B FP8.</td>
    </tr>

    <tr>
      <td>[openbmb/MiniCPM-V-4.7-35B-A3B-int4](https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-int4)</td>
      <td>INT4 W4A16, group size 128 (compressed-tensors)</td>
      <td>35B-A3B INT4.</td>
    </tr>

    <tr>
      <td>[openbmb/MiniCPM-V-4.7-1B-DSpark](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-DSpark)</td>
      <td>BF16 draft</td>
      <td>Draft model for the 1B **DSPARK** row; not served on its own.</td>
    </tr>

    <tr>
      <td>[openbmb/MiniCPM-V-4.7-35B-A3B-DSpark](https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-DSpark)</td>
      <td>BF16 draft</td>
      <td>Draft model for the 35B-A3B **DSPARK** row; not served on its own.</td>
    </tr>
  </tbody>
</table>

OpenBMB also publishes AWQ, GPTQ, and GGUF exports and a 1B int4 checkpoint. They are not part of the matrix above.

**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20`, from the 35B-A3B checkpoints' `generation_config.json`. SGLang applies these defaults, so do not hardcode them in client code.

**Resources:** [Hugging Face (1B)](https://huggingface.co/openbmb/MiniCPM-V-4.7-1B) · [Hugging Face (35B-A3B)](https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B) · [OpenBMB documentation](https://github.com/OpenBMB/MiniCPM-V/blob/main/docs/minicpm_v4dot7_en.md) · [SGLang model support (#41993)](https://github.com/sgl-project/sglang/pull/41993).

## 2. Configuration Tips

* Every generated command carries `--mamba-radix-cache-strategy extra_buffer`, the radix-cache strategy for the Gated DeltaNet hybrid backbone. The FP8 and INT4 commands also carry `--tokenizer-path openbmb/MiniCPM-V-4.7-35B-A3B` (see the note below).
* Every recipe runs on one GPU. On H100 the 35B-A3B BF16 checkpoint (about 66 GB of weights) does not fit the default static memory fraction of an 80 GB card; use FP8 or INT4 there.
* Pick the quantization by platform. On H200, quantization is a speed win: at concurrency 16, INT4 and FP8 reach 15,808 and 15,280 tokens/s per GPU against 12,103 for BF16, and INT4 has the fastest 35B-A3B decode (TPOT 3.47 ms at concurrency 1). On GB300, BF16 and FP8 decode at about the same speed at concurrency 1 (TPOT 4.36 vs 4.44 ms), and FP8 pulls ahead at concurrency 16 (16,059 vs 14,094 tokens/s per GPU). INT4 is slower on GB300 at every concurrency (TPOT 8.56 ms at concurrency 1), so use it there only to save memory.
* SGLang detects FP8 and INT4 from the checkpoint's `quantization_config`; no `--quantization` flag is needed.
* The **DSPARK** row adds `--speculative-algorithm DSPARK` and `--speculative-draft-model-path` pointing at the variant's DSpark checkpoint. On GB300, turn it on only for 35B-A3B BF16 and FP8 at low concurrency; on H200, turn it on for 35B-A3B BF16 at both measured concurrencies and for FP8 and INT4 at low concurrency; leave it off for 1B. The matched DSPARK on/off cells in the benchmark cards above back this.
* Benchmark speculative decoding on real text. Random-token prompts collapse draft acceptance, so they say nothing about DSPARK. For a real-workload reference, GSM8K accept length is about 4.5 on 1B (measured at the final PR head); in the sharegpt benchmark, accept length is about 2.5 on 1B and 3.2-3.3 on the 35B-A3B variants.
* Every command carries `--reasoning-parser auto --tool-call-parser auto`. SGLang resolves both from the chat template, to `qwen3` for reasoning and `qwen3_coder` for tool calls, so the thinking trace lands in `message.reasoning_content` and tool calls in `message.tool_calls` (see §3.1 and §3.2). To turn either off, toggle it in the **Parsers** card of the [Playground](#playground).
* Thinking is on by default. To get a direct answer for one request, pass `"chat_template_kwargs": {"enable_thinking": false}` (in the OpenAI Python client, `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`); `reasoning_content` then comes back empty.
* Image and video requests need no extra server flags.

> [!NOTE]
>
> The public FP8 and INT4 repos' `tokenizer_config.json` declares `auto_map.AutoProcessor = processing_minicpmv4_7.MiniCPMV4_7Processor`, but the repos do not ship that file, so SGLang cannot load their processor. The FP8 and INT4 commands therefore load the tokenizer and processor from the base repo with `--tokenizer-path openbmb/MiniCPM-V-4.7-35B-A3B`, which has the same `tokenizer.json` and chat template. Keep the flag until OpenBMB updates the FP8 and INT4 repos to ship the file or drop the `auto_map` entry.

## 3. Advanced Usage

### 3.1 Reasoning

MiniCPM-V 4.7 thinks by default. The reasoning parser in every generated command returns the thinking trace in `message.reasoning_content` and the final answer in `message.content`.

> **Reasoning (Python)**
>
> ```python Example theme={null}
> import requests
>
> resp = requests.post(
>     "http://localhost:30000/v1/chat/completions",
>     json={
>         "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
>         "messages": [{"role": "user", "content": "What is 23*17?"}],
>     },
> ).json()
> choice = resp["choices"][0]
> print("finish_reason:", choice["finish_reason"])
> print("--- reasoning_content ---")
> print(choice["message"]["reasoning_content"])
> print("--- content ---")
> print(choice["message"]["content"])
> ```

> **Example Output**
>
> ```text Output theme={null}
> finish_reason: stop
> --- reasoning_content ---
> The user is asking for the product of 23 and 17. I need to compute 23 * 17.
>
> Let me calculate:
> 23 * 17
> I can break it down:
> 23 * 10 = 230
> 23 * 7 = 161
> Now add: 230 + 161 = 391
>
> Wait, let me double-check:
> 23 * 17
> 23 * (10 + 7) = 23*10 + 23*7
> 23*10 = 230
> 23*7: 20*7=140, 3*7=21, so 140+21=161
> Now 230 + 161 = 391
>
> So the product is 391.
>
> --- content ---
>
>
> To find 23 × 17, I'll calculate the multiplication:
>
> 23 × 17 = 391
>
> Answer: 391
> ```

> [!NOTE]
>
> On GSM8K-style prompts that ask for the answer in `\boxed{}`, the model can write the final answer inside the thinking block and stop without emitting `</think>`, so the reasoning parser leaves `content` empty. Clients that read only `content`, such as accuracy-eval harnesses, should either serve without `--reasoning-parser` or also read `reasoning_content`.

### 3.2 Tool calling

The tool-call parser in every generated command returns calls in `message.tool_calls`, with the arguments as a JSON string. Because thinking is on, print `reasoning_content` and `content` alongside the calls.

> **Tool calling (Python)**
>
> ```python Example theme={null}
> import json
>
> import requests
>
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city.",
>         "parameters": {
>             "type": "object",
>             "properties": {"city": {"type": "string", "description": "City name"}},
>             "required": ["city"],
>         },
>     },
> }]
>
> resp = requests.post(
>     "http://localhost:30000/v1/chat/completions",
>     json={
>         "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
>         "messages": [{"role": "user", "content": "What is the weather in Beijing today?"}],
>         "tools": tools,
>     },
> ).json()
> choice = resp["choices"][0]
> msg = choice["message"]
> print("finish_reason:", choice["finish_reason"])
> print("--- reasoning_content ---")
> print(msg["reasoning_content"])
> print("--- content ---")
> print(repr(msg["content"]))
> print("--- tool_calls ---")
> print(json.dumps(msg["tool_calls"], indent=1))
> ```

> **Example Output**
>
> ```text Output theme={null}
> finish_reason: tool_calls
> --- reasoning_content ---
> The user is asking about the weather in Beijing. I need to call the get_weather function with the city parameter set to Beijing.
>
> --- content ---
> '\n\n'
> --- tool_calls ---
> [
>  {
>   "id": "call_592d3b19eca4432eaf1a89f1",
>   "index": 0,
>   "type": "function",
>   "function": {
>    "name": "get_weather",
>    "arguments": "{\"city\": \"Beijing\"}"
>   }
>  }
> ]
> ```

To get the final answer, append the assistant message and a `tool` role message with the tool's result, then call the API again with the same `tools` list.

### 3.3 Per-request image processing

OpenAI-compatible `/v1/chat/completions` and `/v1/completions` requests accept `mm_processor_kwargs`, which overrides the image processor for that request only:

* `downsample_mode`: `"16x"` (default) or `"4x"`. `"4x"` keeps more visual tokens per tile.
* `max_slice_nums`: the maximum number of high-resolution slices. `1` sends the image without slicing.

> **Image token budget (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> IMAGE_URL = "https://www.ilankelman.org/stopsigns/australia.jpg"
>
> for overrides in ({}, {"downsample_mode": "4x"}, {"max_slice_nums": 1}):
>     resp = client.chat.completions.create(
>         model="openbmb/MiniCPM-V-4.7-35B-A3B",
>         messages=[{
>             "role": "user",
>             "content": [
>                 {"type": "image_url", "image_url": {"url": IMAGE_URL}},
>                 {"type": "text", "text": "Describe this image in one sentence."},
>             ],
>         }],
>         extra_body={"mm_processor_kwargs": overrides} if overrides else None,
>     )
>     print(overrides or "default", "prompt_tokens =", resp.usage.prompt_tokens)
> ```

Prompt token counts from the validation probes. The counts depend on the image sizes and the text prompt, so the example above, which uses a different image, returns different values. The first two rows send the same image and text prompt and differ only in `downsample_mode`.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Input</th>
      <th>`mm_processor_kwargs`</th>
      <th>Prompt tokens</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>One 1568×448 image</td>
      <td>none (`"16x"`)</td>
      <td>355</td>
    </tr>

    <tr>
      <td>One 1568×448 image</td>

      <td>
        `{"downsample_mode": "4x"}`
      </td>

      <td>1291</td>
    </tr>

    <tr>
      <td>Two images (1568×448 and 768×384) in one request</td>
      <td>none (`"16x"`)</td>
      <td>556</td>
    </tr>

    <tr>
      <td>One 768×384 image</td>

      <td>
        `{"max_slice_nums": 1}`
      </td>

      <td>93</td>
    </tr>
  </tbody>
</table>

Overrides do not leak between requests: alternating `"16x"` and `"4x"` on the same image returns the matching token count and a correct answer each time.

For more API examples, see the [SGLang Basic Usage Guide](/docs/basic_usage/send_request).


# MiniCPM5-2B
Source: https://docs.sglang.io/cookbook/autoregressive/OpenBMB/MiniCPM5-2B

Deploy OpenBMB MiniCPM5-2B with SGLang — a 2.5B dense Llama-architecture on-device model with 131K context, thinking mode, XML tool calling and DSpark speculative decoding on H200, RTX PRO 6000, RTX 5090 and DGX Spark.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow "sglang>=0.5.12"
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your card to generate the launch command. MiniCPM5-2B is a 2.5B dense model and runs single-GPU at TP=1 on every supported card, so the page ships one operating point per card plus an optional **DSpark** speculative-decoding row.

> [!NOTE]
>
> Speed numbers exist for the RTX 5090 and DGX Spark cells — a single card at
> random 1024/1024, recorded as Mean. The H200 and RTX PRO 6000 cells are
> pending measurement, and no accuracy numbers have been taken on any platform
> yet. The DSpark overlay carries no speed numbers on any card.

## Playground

The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's documented launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

**MiniCPM5-2B** is the second model in OpenBMB's MiniCPM5 series, following MiniCPM5-1B. It is a dense 2B-class Transformer built for on-device assistants, local deployment, coding agents and tool-use workflows — scenarios where a compact model with a small deployment footprint is preferred. OpenBMB reports 2B-class open-source SOTA within its comparison set (average 53.9), with its clearest advantages in code reasoning, math reasoning, long-context understanding, tool use and agentic tasks.

The checkpoint uses the standard `LlamaForCausalLM` architecture — no custom kernels and no model-code fork — so SGLang loads it through the stock Llama path. It is 2,516,756,480 parameters (1,981,982,720 non-embedding) over 42 layers with GQA (16 query heads, 2 KV heads), and a native context length of 131,072 tokens. Post-training runs SFT → RL → On-Policy Distillation, which merges 16 RL expert models into the single released checkpoint. Weights are released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) license.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Checkpoint</th>
      <th>Precision</th>
      <th>Role on this page</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B)</td>
      <td>BF16</td>
      <td>The served model in every cell above (final release, post-trained with RL + OPD).</td>
    </tr>

    <tr>
      <td>[openbmb/MiniCPM5-2B-DSpark](https://huggingface.co/openbmb/MiniCPM5-2B-DSpark)</td>
      <td>BF16 draft</td>
      <td>Draft model loaded by the **DSPARK** row of the Deploy panel; not served on its own.</td>
    </tr>
  </tbody>
</table>

OpenBMB also publishes SFT-only, mid-training and base checkpoints, plus GGUF, MLX and GPTQ-Int4 exports for llama.cpp / Ollama / LM Studio / Apple Silicon. Those target other runtimes and are not part of the SGLang matrix above.

**Recommended generation:** `temperature=1.0`, `top_p=0.95` (informational — SGLang reads the checkpoint's `generation_config.json`; do not hardcode these in client code).

**Resources:** [HuggingFace](https://huggingface.co/openbmb/MiniCPM5-2B) · [ModelScope](https://www.modelscope.cn/models/OpenBMB/MiniCPM5-2B) · [GitHub](https://github.com/OpenBMB/MiniCPM) · [Tech report](https://arxiv.org/pdf/2506.07900).

## 2. Advanced Usage

### 2.1 Thinking Mode

Thinking is controlled by the chat template's `enable_thinking` flag, passed per request through `chat_template_kwargs`. The `qwen3` reasoning parser — already in every generated command, and toggleable from the **Parsers** card in the [Playground above](#playground) — splits the `<think>` segment into `message.reasoning_content` and leaves the final answer in `message.content`. Drop the flag and the thinking text stays inline in `content`, closing tag and all.

> **Thinking On / Off (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> # Thinking on (chat-template default).
> resp = client.chat.completions.create(
>     model="openbmb/MiniCPM5-2B",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     extra_body={"chat_template_kwargs": {"enable_thinking": True}},
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
>
> # Thinking off — a direct answer, no deliberation segment.
> resp = client.chat.completions.create(
>     model="openbmb/MiniCPM5-2B",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     extra_body={"chat_template_kwargs": {"enable_thinking": False}},
> )
> print("Without thinking:", resp.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: We are asked: "What is 15% of 240?" This is a simple percentage calculation. To find 15% of 240, we multiply 240 by 15% (which is 0.15). So: 240 × 0.15 = 36. Alternatively, we can think of it as (15/100) × 240 = (15 × 240)/100 = 3600/100 = 36. So the answer is 36.
>
> We need to respond in a helpful way. The user might be testing or seeking quick answer. Let's provide a clear response with explanation if needed, but since it's straightforward, we can just give the answer directly or briefly explain.
>
> Answer:
>
> 15% of 240 is **36**.
>
> To calculate:
> \( 240 \times 0.15 = 36 \)
> Reasoning: None
> Answer: To find 15% of 240, multiply 240 by 15% (which is 0.15):
>
> \[
> 240 \times 0.15 = 36
> \]
>
> So, 15% of 240 is **36**.
> ```

### 2.2 Tool Calling

MiniCPM5-2B emits XML-style tool calls (`<function name="..."><param name="...">...</param></function>`), and SGLang's built-in `minicpm5` detector converts them to OpenAI-compatible `tool_calls`, parallel calls included. The parser is already in every command the Deploy panel generates; the **Tool Call Parser** chip in the Playground's **Parsers** card is an opt-out, not an opt-in.

> **Tool Calling (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city.",
>         "parameters": {
>             "type": "object",
>             "properties": {
>                 "city": {"type": "string", "description": "City name"},
>                 "date": {"type": "string", "description": "YYYY-MM-DD"},
>             },
>             "required": ["city"],
>         },
>     },
> }]
>
> resp = client.chat.completions.create(
>     model="openbmb/MiniCPM5-2B",
>     messages=[{"role": "user", "content": "What is the weather in Beijing today?"}],
>     tools=tools,
> )
>
> msg = resp.choices[0].message
> # With the qwen3 reasoning parser on, the thinking segment lands in
> # `reasoning_content` and `content` may be empty on a tool-call turn.
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> for call in (msg.tool_calls or []):
>     print("Tool:", call.function.name, call.function.arguments)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning:
> The user is asking about the weather in Beijing today. I have access to a get_weather function that can help with this. Let me call it to get the current weather for Beijing.
>
> Content:
>
>
> Tool: get_weather {"city": "Beijing"}
> ```


# PPLX-Decider-v1-27B
Source: https://docs.sglang.io/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B

Deploy PPLX-Decider-v1-27B with SGLang. Perplexity's decision model, fine-tuned from Qwen3.8-27B, returns calibrated probabilities for choice, yes or no, and score questions on /v1/systemone, on one NVIDIA H200 or GB300 in BF16.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. PPLX-Decider support landed after v0.5.21, so until a release includes it, install a [nightly build](/docs/get-started/install#nightly-builds).
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> The image is multi-arch, so the same tag runs on x86 H200 hosts and Arm GB300 hosts. For how to launch it, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your GPU to generate the launch command. The model runs on one GPU in BF16, and the decision support needs no flag: SGLang reads the checkpoint's `decision_config.json`, loads its decision readout, and answers on `/v1/systemone` with the prompt, answer codes, and calibrated temperature the model was trained with. The GB300 recipe adds `--disable-radix-cache` for speed, as explained in [Configuration Tips](#2-configuration-tips).

> [!NOTE]
>
> The H200 recipe is the original one, validated against the checkpoint's own reference implementation on one H200 when support landed ([#42183](https://github.com/sgl-project/sglang/pull/42183)), and it has no speed numbers yet. The GB300 recipe adds `--disable-radix-cache` (see [Configuration Tips](#2-configuration-tips)) and was measured end to end on one GB300 at SGLang commit `70f0b7351e`: four request shapes from 1 to 256 concurrent requests, Belebele and WinoGrande accuracy, and per-question parity with the reference implementation. [§3](#3-benchmarks) has the method and the full tables.

## Playground

The Playground layers SGLang features on top of the recipe above. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end. The [Configuration Tips](#2-configuration-tips) say which overrides were measured on GB300 and what they did. Tensor parallelism times data-parallel replicas must fit the GPUs on one node.

## 1. Model Introduction

**PPLX-Decider-v1-27B** is a decision model from Perplexity, fine-tuned from [Qwen3.8-27B](/cookbook/autoregressive/Qwen/Qwen3.8-27B) under the Apache-2.0 license. It replaces the language model head with a readout over 255 answer codes and answers choice, yes or no (`noul`), and score questions about a text state and optional images with a probability for every option, without generating text. Each question is one prefill pass, so the cost of a decision is the cost of reading its prompt. Its successor, [PPLX-Decider-v1.1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B), adds noncausal full attention and more training data.

The backbone is the Qwen3.8-27B hybrid: 64 layers in which 48 Gated DeltaNet (linear attention) layers alternate with 16 gated full-attention layers, a 5120-wide hidden state, and the Qwen3.8 vision encoder, for 26.1B parameters (52 GB in BF16). The checkpoint ships a bare `Qwen3_5Model` backbone, the 255-row readout in `readout.safetensors`, and `decision_config.json` with the answer-code token ids and a calibrated temperature of 2.2076. SGLang serves it as `Qwen3_5ForConditionalGeneration` and writes the readout into the language model head rows of the code tokens, so the probabilities come out of the regular scoring path.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Property</th>
      <th>Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Checkpoint</td>
      <td>[perplexity-ai/pplx-decider-v1-27b](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b) (BF16)</td>
    </tr>

    <tr>
      <td>Base model</td>
      <td>Qwen3.8-27B, one epoch of supervised fine-tuning on decision data</td>
    </tr>

    <tr>
      <td>Question types</td>
      <td>`choice` (up to 255 options), `noul` (yes or no), `score` (up to 10 levels)</td>
    </tr>

    <tr>
      <td>Inputs</td>
      <td>Text or JSON state, optional images</td>
    </tr>

    <tr>
      <td>Prompt length</td>
      <td>Trained on question prompts of up to 8,192 tokens, and the backbone accepts 262,144</td>
    </tr>

    <tr>
      <td>License</td>
      <td>Apache-2.0</td>
    </tr>
  </tbody>
</table>

Perplexity reports 85.71% overall accuracy across 11 public benchmarks for this checkpoint, against 74.76% for Qwen3.8-27B, measured through the Perplexity API. The per-benchmark table is on the model card. [§3](#3-benchmarks) measures accuracy and speed on SGLang.

**Resources:** [HuggingFace](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b), [reference inference code](https://huggingface.co/perplexity-ai/pplx-decider-v1-27b/blob/main/inference.py), [Decision models in SGLang](/docs/supported-models/decision_models).

## 2. Configuration Tips

* **Route.** Send requests to `/v1/systemone`, the System One API the checkpoint was trained for. `/v1/decisions` refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head.
* **Clients.** Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See [System One compatible API](/docs/supported-models/decision_models#system-one-compatible-api) for the request and response reference.
* **Images.** Pass each image in `images` as base64 bytes, a base64 data URL, or an `http(s)` URL. They precede the text in every question, as in the checkpoint's reference code. The checkpoint's processor resizes every image to between 65,536 and 262,144 pixels, so one image costs 64 to 256 prompt tokens: a 1024×768 screenshot is 234 tokens, and a 4K one is no more expensive than 1080p (252). Crop to the region that matters rather than sending a larger image.
* **What a decision costs.** Each question is its own prompt: an \~84-token wrapper (system message, chat template, answer instructions), the state, the question, and its options, read in one prefill pass. Several questions in one request are scored together, but each re-reads the whole state, so the cost of a request is roughly *questions × state tokens*.
* **Prefix cache.** On this hybrid model the prefix cache resumes from where an earlier prompt ended, because that is where the Gated DeltaNet state is saved, not from any shared prefix. An identical request measured 215 ms cold and 56 ms cached on GB300, but a different question about the same state, and the other questions in the same request, reused none of the state. The GB300 recipe therefore turns the cache off with `--disable-radix-cache`: unique-state traffic loses no reuse, each request holds one state slot instead of five, and the per-request state bookkeeping goes away, which took a short question from 59 to 51 ms and an image request from 126 to 87 ms with the attention kernel held fixed. Turn it back on only if your traffic repeats identical requests, the same state and the same question, often enough to matter.
* **Prompt length.** The model was fine-tuned on question prompts of up to 8,192 tokens, and the reference code refuses longer ones. SGLang serves the backbone's full 262,144-token context, so keep states within the trained length. A longer prompt is answered, but its accuracy is untested.
* **Throughput is compute-bound.** A decision is pure prefill, and one GB300 reads about 27,000 prompt tokens per second in BF16 once batches are full: about 71 requests per second for a 382-token question, and 3.3 for an 8K-token state. Past that point, more concurrency only adds queueing. The tables in [§3](#3-benchmarks) map concurrency to latency, so pick the highest concurrency that fits your latency budget.
* **Memory.** The weights take 52 GB, and the default `--mem-fraction-static` hands the rest of the GPU to the KV and Gated DeltaNet state pools. The startup log line `max_running_requests is capped to ... by the mamba state cache` is expected: on GB300, each request reserves five state slots with the prefix cache on (a cap of 118) and one with it off (593). Even the lower cap only binds for questions shorter than about 140 tokens, because a prefill batch fills its 16,384-token chunk first. `--mamba-ssm-dtype bfloat16` halves the state pool but measured no faster, so keep the checkpoint's FP32 state.
* **GB300 kernels.** With the prefix cache off, SGLang picks the TRT-LLM attention kernel with 64-token pages on GB300 (Triton with 1-token pages while the cache is on), and that kernel is where the long-state gain comes from: an 8K-token state drops from 377 to 325 ms and saturates at 3.30 instead of 2.87 requests per second, the same as forcing `--attention-backend trtllm_mha --page-size 64` with the cache on. FlashInfer's Gated DeltaNet prefill (`--linear-attn-prefill-backend flashinfer`) adds another 4-8% on text, taking the 8K-token state to 300 ms and 3.56 requests per second, but it made image requests slower (101 vs 87 ms alone, 9% less throughput), so the recipe leaves it to the Playground for text-only traffic. A larger prefill chunk (`--chunked-prefill-size 32768`) did not help: one 16,384-token chunk already keeps the GPU busy.
* **More GPUs.** The model fits one GPU, so add throughput with data-parallel replicas: `--dp-size 2` on two GB300s served 139 short questions per second (1.96× one GPU) and 6.5 8K-token states per second, and because the questions of one request are spread over the replicas, it also cut four questions on a 1K-token state from 179 to 104 ms. The extra dispatch hop added about 15 ms to a lone short question. Tensor parallelism only pays off for latency on long states: `--tp 2` took an 8K-token state from 325 to 208 ms, but delivered less throughput than two replicas and made a short question slower (68 ms), because the all-reduce dominates small batches. SGLang Model Gateway does not route `/v1/systemone` yet, so to spread load over several servers instead of `--dp-size`, use a plain HTTP load balancer.
* **FP8.** `--quantization fp8` quantizes the BF16 weights at load time and measured about 20% more throughput on GB300 (84 vs 70 requests per second for short questions), with the same Belebele and WinoGrande scores. But its probabilities moved by 0.010 on average and up to 0.20 against the reference implementation, four times the drift of any BF16 configuration, and 0.9% of answers changed their top option. The calibrated probabilities are this model's output, so the recipes stay in BF16, and the Playground offers FP8 for workloads that only use the top choice.
* **Reproducibility.** Two launches of the same recipe returned bit-identical probabilities for the same requests at the same concurrency. Against the checkpoint's reference implementation, the largest probability difference per question averages 0.003 (see [§3](#3-benchmarks)), because batch composition and kernels differ, and near-ties at 0.500 can flip.

## 3. Benchmarks

All numbers in this section come from one NVIDIA GB300 (288 GB) at SGLang commit `70f0b7351e`, with checkpoint revision `5117a6c`, served with the GB300 recipe above unless a row says otherwise.

**Method.** A closed-loop client keeps a fixed number of `/v1/systemone` requests in flight. Every request carries a unique random state, so requests share nothing but the system message, and each concurrency level starts after 8 warmup requests and a cache flush. Latency is the median end-to-end time of a request, and prompt tokens come from each response's `usage.input_tokens`. Four request shapes cover the common cases: one four-option choice question about a 256-token state, four questions (two choice, one yes or no, one score) about a 1,024-token state, one question about an 8,192-token state, and one question about a 1024×768 JPEG with a 64-token state.

### 3.1 Speed

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Request</th>
      <th>Concurrent</th>
      <th>Median latency</th>
      <th>Requests/s</th>
      <th>Prompt tokens/s</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Short question, 382 tokens</td>
      <td>1</td>
      <td>51 ms</td>
      <td>19.4</td>
      <td>7,413</td>
    </tr>

    <tr>
      <td />

      <td>4</td>
      <td>103 ms</td>
      <td>32.8</td>
      <td>12,525</td>
    </tr>

    <tr>
      <td />

      <td>16</td>
      <td>267 ms</td>
      <td>59.0</td>
      <td>22,531</td>
    </tr>

    <tr>
      <td />

      <td>64</td>
      <td>882 ms</td>
      <td>70.8</td>
      <td>27,039</td>
    </tr>

    <tr>
      <td />

      <td>256</td>
      <td>3,521 ms</td>
      <td>71.1</td>
      <td>27,164</td>
    </tr>

    <tr>
      <td>Four questions on a 1K-token state, 4,540 tokens</td>
      <td>1</td>
      <td>179 ms</td>
      <td>5.6 (22 decisions/s)</td>
      <td>25,291</td>
    </tr>

    <tr>
      <td />

      <td>8</td>
      <td>1,321 ms</td>
      <td>6.0 (24 decisions/s)</td>
      <td>27,332</td>
    </tr>

    <tr>
      <td />

      <td>32</td>
      <td>5,206 ms</td>
      <td>6.0 (24 decisions/s)</td>
      <td>27,208</td>
    </tr>

    <tr>
      <td>8K-token state, 8,224 tokens</td>
      <td>1</td>
      <td>325 ms</td>
      <td>3.08</td>
      <td>25,297</td>
    </tr>

    <tr>
      <td />

      <td>4</td>
      <td>1,220 ms</td>
      <td>3.26</td>
      <td>26,785</td>
    </tr>

    <tr>
      <td />

      <td>16</td>
      <td>4,796 ms</td>
      <td>3.30</td>
      <td>27,175</td>
    </tr>

    <tr>
      <td>One 1024×768 image, 429 tokens</td>
      <td>1</td>
      <td>87 ms</td>
      <td>11.5</td>
      <td>4,945</td>
    </tr>

    <tr>
      <td />

      <td>16</td>
      <td>414 ms</td>
      <td>37.5</td>
      <td>16,078</td>
    </tr>

    <tr>
      <td />

      <td>64</td>
      <td>1,170 ms</td>
      <td>52.6</td>
      <td>22,572</td>
    </tr>
  </tbody>
</table>

One GB300 tops out at about 27,000 prompt tokens per second for every text shape, so requests per second follow from prompt length. Latency rises with concurrency once that ceiling is reached, so choose the concurrency from the latency you can accept. A decision has no output tokens, so in the benchmark card above TTFT is the whole request latency, and TPOT and interactivity do not apply.

### 3.2 Accuracy and parity

Both sets go through `/v1/systemone` as one `choice` question per item. Belebele sends the passage as the state, the question as the instructions, and the four answers as options `1` to `4`. WinoGrande sends the sentence as the state, asks which option fills the blank, and offers the two candidates. The reference row runs the same items through the checkpoint's own `DecisionModel` one at a time.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Implementation</th>
      <th>Belebele (eng\_Latn, 900)</th>
      <th>WinoGrande (xl dev, 1,267)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>SGLang, GB300 recipe</td>
      <td>**96.67%**</td>
      <td>**84.21%**</td>
    </tr>

    <tr>
      <td>SGLang, no extra flag</td>
      <td>96.67%</td>
      <td>84.37%</td>
    </tr>

    <tr>
      <td>Checkpoint's reference `DecisionModel` (Transformers, BF16, GB300)</td>
      <td>96.67%</td>
      <td>84.37%</td>
    </tr>

    <tr>
      <td>Perplexity, model card (Perplexity API, its own prompt conversion)</td>
      <td>94.00%</td>
      <td>83.30%</td>
    </tr>
  </tbody>
</table>

Against the reference implementation, the recipe picks the same option on 2,161 of 2,167 items, and each of the six differences is a near-tie that one side scores 0.500 against 0.500 and the other 0.486 against 0.514. Per item, the largest probability difference averages 0.003 (99th percentile 0.024, maximum 0.054). Perplexity's figures come from a different prompt conversion, so they are context rather than a target.

### 3.3 What else was measured

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Change from the GB300 recipe</th>
      <th>Measured effect</th>
      <th>Verdict</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Prefix cache on (no extra flag)</td>
      <td>Short question 58 ms alone, 8K-token state 377 ms alone and 2.87 requests/s saturated (recipe: 51 ms, 325 ms, 3.30)</td>
      <td>Only for traffic that repeats identical requests</td>
    </tr>

    <tr>
      <td>`--linear-attn-prefill-backend flashinfer`</td>
      <td>Text 4-8% faster (8K-token state 300 ms, 3.56 requests/s), images slower (101 ms alone, 9% less throughput)</td>
      <td>Text-only traffic</td>
    </tr>

    <tr>
      <td>`--dp-size 2` (2 GPUs)</td>
      <td>139 short questions/s and 6.5 8K-token states/s, four questions on a 1K-token state in 104 ms, a lone short question 15 ms slower</td>
      <td>Scale-out</td>
    </tr>

    <tr>
      <td>`--tp 2` (2 GPUs)</td>
      <td>8K-token state 208 ms alone, 124 short questions/s saturated, a lone short question in 68 ms</td>
      <td>Long states with a latency target</td>
    </tr>

    <tr>
      <td>`--quantization fp8`</td>
      <td>About 20% more throughput, same accuracy, probabilities shifted by 0.010 on average and up to 0.20</td>
      <td>Only when you use the top choice alone</td>
    </tr>

    <tr>
      <td>`--chunked-prefill-size 32768 --max-prefill-tokens 32768`</td>
      <td>No gain (within 4%)</td>
      <td>Keep the default</td>
    </tr>

    <tr>
      <td>`--mamba-ssm-dtype bfloat16`</td>
      <td>No gain, though it halves the state pool</td>
      <td>Keep the default</td>
    </tr>
  </tbody>
</table>

> **Check accuracy on your deployment (Python)**
>
> ```python Example theme={null}
> import asyncio
>
> import aiohttp
> from datasets import load_dataset
>
> URL = "http://localhost:30000/v1/systemone"
>
>
> async def main():
>     rows = load_dataset("facebook/belebele", "eng_Latn", split="test")
>     limit = asyncio.Semaphore(32)
>
>     async def ask(session, row):
>         body = {
>             "model": "perplexity-ai/pplx-decider-v1-27b",
>             "state": row["flores_passage"],
>             "questions": {
>                 "answer": {
>                     "type": "choice",
>                     "instructions": row["question"],
>                     "criteria": {str(k): row[f"mc_answer{k}"] for k in range(1, 5)},
>                 }
>             },
>         }
>         async with limit, session.post(URL, json=body) as response:
>             response.raise_for_status()
>             answer = (await response.json())["answers"]["answer"]
>         return answer["choice"] == row["correct_answer_num"]
>
>     async with aiohttp.ClientSession() as session:
>         correct = await asyncio.gather(*(ask(session, row) for row in rows))
>     print(f"Belebele eng_Latn: {sum(correct)}/{len(correct)} = {100 * sum(correct) / len(correct):.2f}%")
>
>
> asyncio.run(main())
> ```

> **Example Output**
>
> ```text Output theme={null}
> Belebele eng_Latn: 870/900 = 96.67%
> ```

> **Measure speed on your deployment (Python)**
>
> Run it as `python bench.py <state words> <concurrency> <requests>`. With 270 words, a request is 378 tokens, close to the short shape above.
>
> ```python Example theme={null}
> import asyncio
> import random
> import statistics
> import sys
> import time
>
> import aiohttp
>
> BASE = "http://localhost:30000"
> STATE_WORDS, CONCURRENCY, REQUESTS = int(sys.argv[1]), int(sys.argv[2]), int(sys.argv[3])
> WORDS = ("customer payment refund invoice order delivery account login error update server outage "
>          "contract price plan report urgent today please help the a and with for from to of in on").split()
> QUESTION = {
>     "type": "choice",
>     "instructions": "Which team should handle this request?",
>     "criteria": {"billing": "Charges and refunds", "technical_support": "Integration errors",
>                  "sales": "Questions about buying a product", "legal": "Contracts and liability"},
> }
>
>
> def body(rng):
>     # A unique random state per request, so no request reuses another's cache.
>     state = " ".join(rng.choices(WORDS, k=STATE_WORDS))
>     return {"model": "perplexity-ai/pplx-decider-v1-27b", "state": state, "questions": {"q": QUESTION}}
>
>
> async def main():
>     rng = random.Random(0)
>     bodies = [body(rng) for _ in range(REQUESTS)]
>     latencies, tokens = [], 0
>     async with aiohttp.ClientSession() as session:
>         for item in [body(rng) for _ in range(8)]:  # warmup
>             async with session.post(f"{BASE}/v1/systemone", json=item) as response:
>                 response.raise_for_status()
>         async with session.post(f"{BASE}/flush_cache") as response:
>             await response.read()
>         queue = iter(bodies)
>
>         async def worker():
>             nonlocal tokens
>             for item in queue:
>                 start = time.perf_counter()
>                 async with session.post(f"{BASE}/v1/systemone", json=item) as response:
>                     response.raise_for_status()
>                     tokens += (await response.json())["usage"]["input_tokens"]
>                 latencies.append(time.perf_counter() - start)
>
>         start = time.perf_counter()
>         await asyncio.gather(*(worker() for _ in range(CONCURRENCY)))
>         elapsed = time.perf_counter() - start
>     print(f"{REQUESTS / elapsed:.1f} req/s, {tokens / elapsed:.0f} prompt tok/s, "
>           f"P50 {1000 * statistics.median(latencies):.1f} ms, {tokens / REQUESTS:.0f} tokens/request")
>
>
> asyncio.run(main())
> ```

> **Example Output**
>
> ```text Output theme={null}
> $ python bench.py 270 1 64
> 19.6 req/s, 7393 prompt tok/s, P50 51.0 ms, 378 tokens/request
> $ python bench.py 270 64 512
> 68.4 req/s, 25865 prompt tok/s, P50 899.0 ms, 378 tokens/request
> ```

## 4. Advanced Usage

### 4.1 Text Decisions

> **Text Decision Example (Python)**
>
> ```python Example theme={null}
> import requests
>
> response = requests.post(
>     "http://localhost:30000/v1/systemone",
>     json={
>         "model": "perplexity-ai/pplx-decider-v1-27b",
>         "state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.",
>         "questions": {
>             "routing": {
>                 "type": "choice",
>                 "instructions": "Which team should handle this request?",
>                 "criteria": {
>                     "billing": "Charges and refunds",
>                     "technical_support": "Integration errors",
>                     "sales": "Questions about buying a product",
>                 },
>             },
>             "urgency": {"type": "noul", "instructions": "Does this message express urgency?"},
>         },
>     },
>     timeout=60,
> )
> response.raise_for_status()
> for name, answer in response.json()["answers"].items():
>     print(name, answer)
> ```

> **Example Output**
>
> ```text Output theme={null}
> routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.987854396908662, 'probabilities': {'billing': 0.004840538662952211, 'technical_support': 0.9919029312724412, 'sales': 0.0032565300646064375}}
> urgency {'type': 'noul', 'noul': 0.9929980993021188}
> ```

### 4.2 Image Decisions

The example output below came from a mostly red test image.

> **Image Decision Example (Python)**
>
> ```python Example theme={null}
> import base64
> from pathlib import Path
>
> import requests
>
> image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")
> response = requests.post(
>     "http://localhost:30000/v1/systemone",
>     json={
>         "model": "perplexity-ai/pplx-decider-v1-27b",
>         "state": "Look at the supplied image.",
>         "images": [image],
>         "questions": {
>             "color": {
>                 "type": "choice",
>                 "instructions": "What is the dominant color?",
>                 "criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"},
>             }
>         },
>     },
>     timeout=60,
> )
> response.raise_for_status()
> print(response.json()["answers"]["color"])
> ```

> **Example Output**
>
> ```text Output theme={null}
> {'type': 'choice', 'choice': 'red', 'confidence': 0.9498445161931509, 'probabilities': {'red': 0.9623833871448632, 'green': 0.009006826727044857, 'blue': 0.00926546931819573, 'other': 0.01934431680989631}}
> ```


# PPLX-Decider-v1.1-27B
Source: https://docs.sglang.io/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B

Deploy PPLX-Decider-v1.1-27B with SGLang. Perplexity's updated decision model reads the whole prompt in its full-attention layers and returns calibrated probabilities for choice, yes or no, and score questions on /v1/systemone, on one NVIDIA GB300 or H200 in BF16.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. PPLX-Decider-v1.1 support lands in [#42645](https://github.com/sgl-project/sglang/pull/42645), after v0.5.21, so until a release includes it, install a [nightly build](/docs/get-started/install#nightly-builds) built after that merge.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```
>
> The image is multi-arch, so the same tag runs on x86 H200 hosts and Arm GB300 hosts. For how to launch it, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your GPU to generate the launch command. The model runs on one GPU in BF16 and needs no flag: SGLang reads the checkpoint's `decision_config.json`, runs the full-attention layers over the whole prompt as the model was trained, and turns off the radix cache and chunked prefill, which would show those layers only part of the prompt.

> [!NOTE]
>
> The GB300 recipe was measured end to end on one GB300 at [#42645](https://github.com/sgl-project/sglang/pull/42645): four request shapes from 1 to 256 concurrent requests, Belebele and WinoGrande accuracy, and per-question parity with the checkpoint's own reference implementation. The H200 command is the same and has not been run yet. [§3](#3-benchmarks) has the method and the full tables.

## Playground

The Playground layers SGLang features on top of the recipe above. Any override changes the badge to **Not Verified** until that exact configuration is tested end to end. The [Configuration Tips](#2-configuration-tips) say which overrides were measured on GB300. Tensor parallelism times data-parallel replicas must fit the GPUs on one node.

## 1. Model Introduction

**PPLX-Decider-v1.1-27B** is the successor to [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B), Perplexity's decision model fine-tuned from [Qwen3.8-27B](/cookbook/autoregressive/Qwen/Qwen3.8-27B) under the Apache-2.0 license. Like v1, it replaces the language model head with a readout over 255 answer codes and answers choice, yes or no (`noul`), and score questions about a text state and optional images with a probability for every option, without generating text.

Two things changed. The 16 full-attention layers no longer use a causal mask, so every token attends to the whole prompt, while the 48 Gated DeltaNet layers stay causal. And training grew from 73,000 to 626,033 rows, most of them from [tasksource](https://github.com/sileod/tasksource). The prompt format, answer codes, and readout layout are the same as v1's, but the calibrated temperature is 1.0087 instead of 2.2076, so probabilities from the two versions are not interchangeable.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Property</th>
      <th>Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Checkpoint</td>
      <td>[perplexity-ai/pplx-decider-v1.1-27b](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b) (BF16)</td>
    </tr>

    <tr>
      <td>Base model</td>
      <td>Qwen3.8-27B, one epoch of supervised fine-tuning on 626,033 decision rows (530,103 from tasksource)</td>
    </tr>

    <tr>
      <td>Attention</td>
      <td>Noncausal in the 16 full-attention layers, causal in the 48 Gated DeltaNet layers</td>
    </tr>

    <tr>
      <td>Question types</td>
      <td>`choice` (up to 255 options), `noul` (yes or no), `score` (up to 10 levels)</td>
    </tr>

    <tr>
      <td>Inputs</td>
      <td>Text or JSON state, optional images</td>
    </tr>

    <tr>
      <td>Prompt length</td>
      <td>Trained on question prompts of up to 8,192 tokens, and the backbone accepts 262,144</td>
    </tr>

    <tr>
      <td>License</td>
      <td>Apache-2.0</td>
    </tr>
  </tbody>
</table>

Perplexity reports a Decision Index of 61.56 for v1.1, up from 56.4 for v1:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Decision Index category</th>
      <th>Jev</th>
      <th>PPLX-Decider-v1-27B</th>
      <th>PPLX-Decider-v1.1-27B</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Knowledge</td>
      <td>**51.4**</td>
      <td>40.9</td>
      <td>48.18</td>
    </tr>

    <tr>
      <td>Language</td>
      <td>62.0</td>
      <td>63.5</td>
      <td>**69.45**</td>
    </tr>

    <tr>
      <td>Retrieval</td>
      <td>55.4</td>
      <td>54.9</td>
      <td>**61.26**</td>
    </tr>

    <tr>
      <td>Tools</td>
      <td>75.1</td>
      <td>**79.3**</td>
      <td>78.88</td>
    </tr>

    <tr>
      <td>Arts</td>
      <td>37.7</td>
      <td>39.4</td>
      <td>**44.66**</td>
    </tr>

    <tr>
      <td>**Overall**</td>
      <td>57.9</td>
      <td>56.4</td>
      <td>**61.56**</td>
    </tr>
  </tbody>
</table>

**Resources:** [HuggingFace](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b), [reference model code](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b/blob/main/source/src/autojev/model.py), [Decision models in SGLang](/docs/supported-models/decision_models).

## 2. Configuration Tips

* **Route.** Send requests to `/v1/systemone`, the System One API the checkpoint was trained for. `/v1/decisions` refuses this checkpoint because its labels and prompt are not the ones the readout learned, and chat or generation requests are not meaningful without a language model head.
* **Clients.** Clients of the System One API, including the TypeSafe SDKs, work by pointing their base URL at the server. See [System One compatible API](/docs/supported-models/decision_models#system-one-compatible-api) for the request and response reference.
* **Images.** Pass each image in `images` as base64 bytes, a base64 data URL, or an `http(s)` URL. They precede the text in every question. The processor resizes every image to between 65,536 and 262,144 pixels, so one image costs 64 to 256 prompt tokens.
* **Noncausal attention is automatic.** SGLang reads `attention_mode: noncausal_full_attention` from `decision_config.json` and runs the full-attention layers as encoder layers, which the TRT-LLM, FlashInfer, and Triton attention backends support. It also turns off the radix cache and chunked prefill, logging `Radix cache and chunked prefill are disabled for a decision checkpoint with noncausal full attention`: a cached prefix or a prefill chunk would see only part of a prompt whose every position depends on the rest. These two settings override any flag you pass. A prompt longer than the 16,384-token prefill budget is still read in one pass: an 18,585-token prompt matched the reference implementation to four decimal places on GB300.
* **No prefix reuse.** Because every layer output depends on the whole prompt, nothing carries over between requests, not even an identical one. The cost of a request is roughly *questions × prompt tokens*, as on v1.
* **Prompt length.** Like v1, the model was trained on question prompts of up to 8,192 tokens, and the reference code refuses longer ones. SGLang serves longer prompts in one pass, but their accuracy is untested.
* **Attention backend.** On GB300, Auto resolves to the TRT-LLM kernel with 64-token pages. `--attention-backend flashinfer` and `--attention-backend triton` matched the reference implementation as closely, with the same top option on 99.8% of the accuracy items. The H200 path has not been run yet.
* **Speed.** One GB300 reads about 27,000 prompt tokens per second once batches are full, the same as v1. Measured back to back on the same GPU against v1's recipe, v1.1 was within 2% on short questions and four-question requests, and an 8K-token state saturated 6% lower (3.13 against 3.32 requests per second), because noncausal attention computes the full attention matrix in its 16 full-attention layers instead of half of it.
* **More GPUs.** The backbone and prompt sizes are v1's, so the [v1 page's scaling measurements](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B#2-configuration-tips) are the reference: data-parallel replicas (`--dp-size N`) for throughput, tensor parallelism only for latency on long prompts. They were not repeated for v1.1.
* **Upgrading from v1.** Re-check any thresholds you tuned on v1 probabilities: v1.1 has a different temperature and changed its top answer on 8% of the Belebele and WinoGrande items.

## 3. Benchmarks

All numbers in this section come from one NVIDIA GB300 (288 GB) at [#42645](https://github.com/sgl-project/sglang/pull/42645), with checkpoint revision `3b45dea`, served with the GB300 recipe above unless a row says otherwise. The method and request shapes are those of the [v1 page](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B#3-benchmarks), whose speed script works here unchanged.

### 3.1 Speed

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Request</th>
      <th>Concurrent</th>
      <th>Median latency</th>
      <th>Requests/s</th>
      <th>Prompt tokens/s</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Short question, 382 tokens</td>
      <td>1</td>
      <td>53 ms</td>
      <td>18.9</td>
      <td>7,203</td>
    </tr>

    <tr>
      <td />

      <td>4</td>
      <td>106 ms</td>
      <td>37.6</td>
      <td>14,362</td>
    </tr>

    <tr>
      <td />

      <td>16</td>
      <td>275 ms</td>
      <td>57.4</td>
      <td>21,909</td>
    </tr>

    <tr>
      <td />

      <td>64</td>
      <td>931 ms</td>
      <td>67.5</td>
      <td>25,770</td>
    </tr>

    <tr>
      <td />

      <td>256</td>
      <td>3,489 ms</td>
      <td>71.1</td>
      <td>27,139</td>
    </tr>

    <tr>
      <td>Four questions on a 1K-token state, 4,540 tokens</td>
      <td>1</td>
      <td>183 ms</td>
      <td>5.5 (22 decisions/s)</td>
      <td>24,782</td>
    </tr>

    <tr>
      <td />

      <td>8</td>
      <td>1,315 ms</td>
      <td>6.0 (24 decisions/s)</td>
      <td>27,398</td>
    </tr>

    <tr>
      <td />

      <td>32</td>
      <td>4,882 ms</td>
      <td>6.1 (24 decisions/s)</td>
      <td>27,649</td>
    </tr>

    <tr>
      <td>8K-token state, 8,224 tokens</td>
      <td>1</td>
      <td>334 ms</td>
      <td>2.99</td>
      <td>24,620</td>
    </tr>

    <tr>
      <td />

      <td>4</td>
      <td>1,280 ms</td>
      <td>3.12</td>
      <td>25,682</td>
    </tr>

    <tr>
      <td />

      <td>16</td>
      <td>5,101 ms</td>
      <td>3.13</td>
      <td>25,726</td>
    </tr>

    <tr>
      <td>One 1024×768 image, 429 tokens</td>
      <td>1</td>
      <td>118 ms</td>
      <td>8.3</td>
      <td>3,541</td>
    </tr>

    <tr>
      <td />

      <td>16</td>
      <td>466 ms</td>
      <td>33.6</td>
      <td>14,398</td>
    </tr>

    <tr>
      <td />

      <td>64</td>
      <td>1,197 ms</td>
      <td>49.9</td>
      <td>21,416</td>
    </tr>
  </tbody>
</table>

### 3.2 Accuracy and parity

Both sets go through `/v1/systemone` as one `choice` question per item, with the same conversion as the v1 page. The reference rows run the same items through the checkpoint's own `DecisionModel` one at a time.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Implementation</th>
      <th>Belebele (eng\_Latn, 900)</th>
      <th>WinoGrande (xl dev, 1,267)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>SGLang, GB300 recipe (TRT-LLM attention)</td>
      <td>**97.67%**</td>
      <td>**92.42%**</td>
    </tr>

    <tr>
      <td>SGLang, `--attention-backend flashinfer`</td>
      <td>97.67%</td>
      <td>92.58%</td>
    </tr>

    <tr>
      <td>SGLang, `--attention-backend triton`</td>
      <td>97.67%</td>
      <td>92.42%</td>
    </tr>

    <tr>
      <td>Checkpoint's reference `DecisionModel` (Transformers, BF16, GB300)</td>
      <td>97.67%</td>
      <td>92.50%</td>
    </tr>

    <tr>
      <td>v1.1 served with causal attention, as SGLang did before #42645</td>
      <td>97.00%</td>
      <td>88.87%</td>
    </tr>

    <tr>
      <td>PPLX-Decider-v1-27B on SGLang, for comparison</td>
      <td>96.67%</td>
      <td>84.21%</td>
    </tr>
  </tbody>
</table>

Against the reference implementation, the recipe picks the same option on 2,164 of 2,167 items, and each of the three differences is a near-tie within 0.031 of an even split. Per item, the largest probability difference averages 0.002 (99th percentile 0.030, maximum 0.089). Prompts of 11,099 and 18,585 tokens, an image question, and a score question also matched the reference, the largest difference being 0.015 on the score question. Served with causal attention instead, v1.1 changed its top answer on 101 of the 2,167 items and lost 3.6 points on WinoGrande.

> **Check accuracy on your deployment (Python)**
>
> ```python Example theme={null}
> import asyncio
>
> import aiohttp
> from datasets import load_dataset
>
> URL = "http://localhost:30000/v1/systemone"
>
>
> async def main():
>     rows = load_dataset("facebook/belebele", "eng_Latn", split="test")
>     limit = asyncio.Semaphore(32)
>
>     async def ask(session, row):
>         body = {
>             "model": "perplexity-ai/pplx-decider-v1.1-27b",
>             "state": row["flores_passage"],
>             "questions": {
>                 "answer": {
>                     "type": "choice",
>                     "instructions": row["question"],
>                     "criteria": {str(k): row[f"mc_answer{k}"] for k in range(1, 5)},
>                 }
>             },
>         }
>         async with limit, session.post(URL, json=body) as response:
>             response.raise_for_status()
>             answer = (await response.json())["answers"]["answer"]
>         return answer["choice"] == row["correct_answer_num"]
>
>     async with aiohttp.ClientSession() as session:
>         correct = await asyncio.gather(*(ask(session, row) for row in rows))
>     print(f"Belebele eng_Latn: {sum(correct)}/{len(correct)} = {100 * sum(correct) / len(correct):.2f}%")
>
>
> asyncio.run(main())
> ```

> **Example Output**
>
> ```text Output theme={null}
> Belebele eng_Latn: 879/900 = 97.67%
> ```

## 4. Advanced Usage

### 4.1 Text Decisions

> **Text Decision Example (Python)**
>
> ```python Example theme={null}
> import requests
>
> response = requests.post(
>     "http://localhost:30000/v1/systemone",
>     json={
>         "model": "perplexity-ai/pplx-decider-v1.1-27b",
>         "state": "My Stripe integration keeps failing. I'm losing sales. Please help ASAP.",
>         "questions": {
>             "routing": {
>                 "type": "choice",
>                 "instructions": "Which team should handle this request?",
>                 "criteria": {
>                     "billing": "Charges and refunds",
>                     "technical_support": "Integration errors",
>                     "sales": "Questions about buying a product",
>                 },
>             },
>             "urgency": {"type": "noul", "instructions": "Does this message express urgency?"},
>         },
>     },
>     timeout=60,
> )
> response.raise_for_status()
> for name, answer in response.json()["answers"].items():
>     print(name, answer)
> ```

> **Example Output**
>
> ```text Output theme={null}
> routing {'type': 'choice', 'choice': 'technical_support', 'confidence': 0.9977658531751561, 'probabilities': {'billing': 0.0014030901008239556, 'technical_support': 0.9985105687834377, 'sales': 8.634111573847184e-05}}
> urgency {'type': 'noul', 'noul': 0.9995671977710632}
> ```

### 4.2 Image Decisions

The example output below came from a 512×512 solid red test image.

> **Image Decision Example (Python)**
>
> ```python Example theme={null}
> import base64
> from pathlib import Path
>
> import requests
>
> image = base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")
> response = requests.post(
>     "http://localhost:30000/v1/systemone",
>     json={
>         "model": "perplexity-ai/pplx-decider-v1.1-27b",
>         "state": "Look at the supplied image.",
>         "images": [image],
>         "questions": {
>             "color": {
>                 "type": "choice",
>                 "instructions": "What is the dominant color?",
>                 "criteria": {"red": "Red", "green": "Green", "blue": "Blue", "other": "Another color"},
>             }
>         },
>     },
>     timeout=60,
> )
> response.raise_for_status()
> print(response.json()["answers"]["color"])
> ```

> **Example Output**
>
> ```text Output theme={null}
> {'type': 'choice', 'choice': 'red', 'confidence': 0.9989117512916561, 'probabilities': {'red': 0.9991838134687421, 'green': 6.743375714875418e-05, 'blue': 8.12085765657198e-05, 'other': 0.0006675441975434462}}
> ```


# Laguna-M.1
Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-M.1

Deploy poolside's Laguna-M.1 — a 225B-parameter Mixture-of-Experts model (23B active) for agentic coding — with SGLang on NVIDIA H200, B200, B300, GB200, and GB300, across BF16, FP8, and NVFP4.

## Deployment

> **Install SGLang**
>
> Laguna-M.1 support is already on SGLang `main` — **softplus per-element attention-output gating** ([PR #28400](https://github.com/sgl-project/sglang/pull/28400)) and a **global-attention fix** ([PR #28604](https://github.com/sgl-project/sglang/pull/28604), since M.1 is full-attention `sliding_window: 0`) — but not yet in a tagged release. The two paths below match the **Python / Docker** toggle in the command panel: install from `main` (Python tab), or use the **Docker** image, which bundles the same build (CUDA 13, covers H200 + all Blackwell). The model ships custom config code on the Hub, so `--trust-remote-code` is required (it is included in the launch commands).
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install -U uv
> uv venv --python 3.12 && source .venv/bin/activate
>
> # Laguna-M.1 support is on SGLang main (PRs #28400 + #28604, plus #28649 for FP8), not yet in a
> # tagged release — install from main. The serving runtime is in the base dependencies, no extra needed:
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> uv pip install --prerelease=allow -e python
> ```
>
> Then run the **Python** output of the command panel below in that environment. The **Docker** tab is simpler — `lmsysorg/sglang:latest` bundles the CUDA-13 runtime and the M.1 code.
>
> **Docker**
>
> ```bash Command theme={null}
> # CUDA 13 — covers H200 + all Blackwell:
> docker pull lmsysorg/sglang:latest
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + quantization to generate the launch command. Laguna-M.1 ships a single **Balanced** recipe per cell — poolside's recommended operating point, a good speed/throughput trade-off for typical multi-user serving. The 8-GPU HGX platforms (H200 / B200 / B300) use `--tp 8`; the 4-GPU Grace-Blackwell single nodes (GB200 / GB300) use `--tp 4`.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs (parsers, DP-Attention, DeepEP / EP) on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

[Laguna-M.1](https://huggingface.co/poolside/Laguna-M.1) is an open-weight, **225B-parameter** Mixture-of-Experts model (**23B activated per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software-engineering work. It is released under Apache 2.0.

**Key Features:**

* **Large sparse MoE**: 70-layer transformer — the first 3 layers are dense SwiGLU, the remaining 67 are sparse MoE with **256 experts, top-16 routing** (+1 shared expert) and auxiliary-loss-free load balancing.
* **Global attention with output gating**: global attention across all layers, 64 Q-heads / 8 KV-heads (head dim 128), with **softplus attention output gating** (requires [PR #28400](https://github.com/sgl-project/sglang/pull/28400)).
* **Long context**: 262,144 tokens, RoPE with YaRN.
* **Agentic coding**: competitive on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0.
* **Native reasoning**: interleaved thinking between tool calls, toggled per request via `chat_template_kwargs={"enable_thinking": ...}`.

**Available Quantizations:**

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Quantization</th>
      <th>Hugging Face path</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BF16**</td>
      <td>[`poolside/Laguna-M.1`](https://huggingface.co/poolside/Laguna-M.1)</td>
    </tr>

    <tr>
      <td>**FP8**</td>
      <td>[`poolside/Laguna-M.1-FP8`](https://huggingface.co/poolside/Laguna-M.1-FP8)</td>
    </tr>

    <tr>
      <td>**NVFP4**</td>
      <td>[`poolside/Laguna-M.1-NVFP4`](https://huggingface.co/poolside/Laguna-M.1-NVFP4)</td>
    </tr>
  </tbody>
</table>

**License:** Apache 2.0

**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-M.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai).

## 2. Configuration Tips

* **Trust remote code** (`--trust-remote-code`): Laguna-M.1 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model.
* **Long-context memory**: M.1 is global-attention (no sliding-window), so the 262,144-token KV cache is large. If you hit OOM at full context, lower `--mem-fraction-static` or cap `--context-length`.
* **FP8**: On **Blackwell** the recipe adds `--fp8-gemm-backend triton` — the compressed-tensors block-FP8 weight scales aren't UE8M0-packed, so the default DeepGEMM path emits garbage on Blackwell (sm\_100); the Triton backend is correct (\~19% slower). Temporary workaround pending [PR #28662](https://github.com/sgl-project/sglang/pull/28662) (which fixes the scales and restores the DeepGEMM fast path). On **Hopper (H200)** FP8 uses DeepGEMM with no extra flag — pre-warm its multi-session JIT with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-M.1-FP8` to avoid paying it on each restart.
* **Parsers** (`poolside_v1`): for agentic / tool-using deployments enable the **Reasoning Parser** and **Tool Call Parser** in the Playground above — they emit `--reasoning-parser poolside_v1` (thinking → `reasoning_content`) and `--tool-call-parser poolside_v1` (structured `tool_calls`).
* **Thinking default**: thinking is **off by default**; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
* **Served model id**: the server registers the model under whatever you pass to `--model-path`, so a client's `model` field must match it — `poolside/Laguna-M.1` (BF16) or `poolside/Laguna-M.1-FP8` / `-NVFP4` for the quantized cells. The §3 examples use the BF16 id; swap in the id you launched.
* **Recommended sampling**: poolside benchmarks M.1 at `temperature=1.0`, `top_k=20` with thinking enabled. These are per-request sampling params (not launch flags) — e.g. `temperature=1.0, extra_body={"top_k": 20}` on the OpenAI client.

## 3. Advanced Usage

### 3.1 Reasoning

Launch with `--reasoning-parser poolside_v1` (or toggle **Reasoning Parser** in the **Parsers** card of the Playground above). Reasoning is **opt-in**: the Laguna chat template gates it on `enable_thinking=True` (passed via `chat_template_kwargs`) — the generic `thinking` key is ignored. The `<think>` trace then lands in `message.reasoning_content`, separate from the final answer in `message.content` — no client-side tag stripping needed.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="poolside/Laguna-M.1",
>     messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
>     max_tokens=2048,
>     extra_body={"chat_template_kwargs": {"enable_thinking": True}},
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(message.reasoning_content)
> print("=============== Answer ==================")
> print(message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> =============== Reasoning ===============
> Okay, so I need to find out what 15% of 240 is. Hmm, percentages can sometimes be
> tricky, but let me think. I remember that "percent" means per hundred, right? So 15%
> is the same as 15 per 100 or 15/100. Maybe I can convert that percentage into a decimal
> first? ... 15 divided by 100 is 0.15. ... Now, to find 15% of 240, I just need to
> multiply 240 by 0.15. ... 240 times 0.1 is 24 (10% of 240), and 240 times 0.05 is 12
> (half of that), so 24 + 12 = 36.
> [… verifies the same result several more ways: 15/100 × 240, 240 × 15 ÷ 100,
> 1% × 15, and the fraction 3/20 × 240 — all give 36 …]
> So ... all methods are pointing to 36. I'm pretty confident that 15% of 240 is 36.
> =============== Answer ==================
> To find 15% of 240, convert the percentage to a decimal (0.15) and multiply by 240:
> **240 × 0.15 = 36**.
>
> **Step-by-Step Explanation:**
> 1. **Convert 15% to a decimal:** 15% = 15/100 = 0.15.
> 2. **Multiply by 240:**
>    - Break it down:
>      - 10% of 240 = 24 (since 240 × 0.1 = 24).
>      - 5% of 240 = 12 (half of 24).
>    - Add them: 24 + 12 = **36**.
>
> **Answer:** 15% of 240 is **36**.
> ```

> [!NOTE]
>
> Laguna-M.1's reasoning traces are long — the model explores and re-verifies an answer
> multiple ways. Give it a generous `max_tokens` for harder problems (reasoning regularly
> exceeds 3k tokens). The trace above is abbreviated; the model emits it in full.

### 3.2 Tool Calling

Launch with `--tool-call-parser poolside_v1` (or toggle **Tool Call Parser** in the **Parsers** card of the Playground above). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (`enable_thinking=False`, the default).

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="poolside/Laguna-M.1",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
>
> message = response.choices[0].message
> if message.tool_calls:
>     for call in message.tool_calls:
>         print(f"Tool: {call.function.name}")
>         print(f"Args: {call.function.arguments}")
> ```

> **Example Output**
>
> ```text Output theme={null}
> Tool: get_weather
> Args: {"location": "Beijing"}
> ```

### 3.3 Prefill-Decode (PD) Disaggregation

[PD disaggregation](../../../docs/advanced_features/pd_disaggregation) runs prefill and decode on **separate** SGLang servers linked by an RDMA KV-transfer fabric (mooncake or NIXL), fronted by the PD router. Laguna-M.1 is **global-attention with a standard KV cache** (no sliding window, no sparse "index" side-buffer), so its KV pages transfer with **no model-specific flags** — just the `--disaggregation-*` knobs. Both roles auto-select the same attention backend (FlashAttention-3) and page size because they share the model and flags, so the KV layout lines up for transfer.

**Supported / validated topology:**

* **Equal tensor parallelism** — prefill and decode run the same `--tp`.
* **Single pipeline stage** — PP = 1 (the default).
* **mooncake or NIXL** transfer backend over RDMA / InfiniBand.
* Validated on **2 × 8×H200** (TP8 prefill + TP8 decode, BF16), one node each, over an 8× 400 Gb/s NDR InfiniBand fabric.

Launch the prefill server, then the decode server — the same recipe with `--disaggregation-mode decode` and no bootstrap port. Point `--disaggregation-ib-device` at your RDMA NIC(s).

```bash Prefill server (node A) theme={null}
sglang serve \
  --model-path poolside/Laguna-M.1 \
  --trust-remote-code \
  --reasoning-parser poolside_v1 \
  --tool-call-parser poolside_v1 \
  --tp 8 \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake \
  --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  --host 0.0.0.0 --port 30000 \
  --disaggregation-bootstrap-port 8998
```

```bash Decode server (node B) theme={null}
sglang serve \
  --model-path poolside/Laguna-M.1 \
  --trust-remote-code \
  --reasoning-parser poolside_v1 \
  --tool-call-parser poolside_v1 \
  --tp 8 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mooncake \
  --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7 \
  --host 0.0.0.0 --port 30001
```

Then start the PD router, pointing it at the prefill bootstrap (URL plus its `--disaggregation-bootstrap-port`) and the decode endpoint:

```bash PD router theme={null}
python3 -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://<prefill-host>:30000 8998 \
  --decode http://<decode-host>:30001 \
  --policy round_robin \
  --host 0.0.0.0 --port 8000
```

Clients hit the router exactly like a single server — it splits each request across the two stages transparently:

> **PD Client Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://<router-host>:8000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="poolside/Laguna-M.1",
>     messages=[{"role": "user", "content": "What is 2 + 2?"}],
>     max_tokens=64,
> )
> print(response.choices[0].message.content)
> ```
>
> **Output Example:**
>
> ```text Output theme={null}
> 2 + 2 = 4
> ```

**Transfer backend — mooncake (recommended).** mooncake honors `--disaggregation-ib-device` and establishes its RDMA connection at registration, so the **first request is already fast** (no cold start). It works with a single NIC or all eight; using **all 8 NICs lowers TTFT** (more aggregate bandwidth for the KV payload — the gap widens at longer context). On 8×H200 (random isl=512 / osl=256, 16 concurrent) it served ≈ **717 tok/s** output (≈ 2.2k tok/s total), mean **TTFT 244 ms**, mean **TPOT 17.7 ms**; with a single `mlx5_0` NIC, ≈ 697 tok/s and TTFT 287 ms (TPOT unchanged — decode is compute-bound).

**Transfer backend — NIXL (works, with two caveats).**

> [!WARNING]
>
> The NIXL path **ignores `--disaggregation-ib-device`** — that flag is mooncake-only. NIXL uses its UCX backend, whose NIC is selected by the **`UCX_NET_DEVICES`** environment variable. **Set it** (e.g. `export UCX_NET_DEVICES=mlx5_0:1`) on both servers; without it UCX cannot establish a working cross-node path and every KV transfer hangs until it hits the 300 s timeout (`Request … timed out … in KVPoll.WaitingForInput`) and returns a 500.

With `UCX_NET_DEVICES` pinned, NIXL matches mooncake on quality and steady-state speed (≈ 720 tok/s, TTFT 230 ms, TPOT 17.7 ms). One difference: the **first request after launch pays a \~38 s one-time UCX connection cold-start** (a single port or all eight behave the same). Warm the path with one throwaway request after startup, or raise `SGLANG_DISAGGREGATION_WAITING_TIMEOUT` (default 300 s) so the first real request isn't dropped while UCX connects.

**Validation.** PD disaggregation preserves output quality — disaggregated output matches non-disaggregated serving, and GSM8K (no-thinking, 200-question subset via the router) scored **0.945** (mooncake, 8 NICs) / **0.940** (NIXL) / **0.950** (mooncake, 1 NIC), all with 100% stop-rate and 0% errors — in line with single-node BF16 (≈ 0.93 on the full split). Logs confirm the split: the prefill node logs `Prefill batch` (CUDA graph off), the decode node logs `Decode batch` (CUDA graph on).


# Laguna-S-2.1
Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-S-2.1

Deploy poolside's Laguna-S-2.1 — a 118B hybrid-SWA Mixture-of-Experts model (8B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4.

## Deployment

> **Install SGLang**
>
> Laguna-S-2.1 uses the same `laguna` model architecture as [Laguna-XS-2.1](./Laguna-XS-2.1), which is fully supported in SGLang `main`. The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install -U uv
> uv venv --python 3.12 && source .venv/bin/activate
>
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> uv pip install --prerelease=allow -e python
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:

* **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
* **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.

On the 8-GPU HGX platforms (H200 / B300) all quantizations run `--tp 8`. The 4-GPU GB300 node runs `--tp 4` throughout. NVFP4 is Blackwell-only (B300 / GB300 only).

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

[Laguna-S-2.1](https://huggingface.co/poolside/Laguna-S-2.1) is an open-weight **118B-parameter** hybrid sliding-window-attention MoE model (**\~8B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering. It sits between [Laguna XS 2.1](./Laguna-XS-2.1) (33B/3B active) and Laguna M.1 (222B/23B active) in the Laguna family.

**Key Features:**

* **Sparse MoE**: 48 layers, 256 routed experts, top-10 routing, plus 1 shared expert.
* **Hybrid attention**: 36 sliding-window layers (window 512) interleaved with 12 full-attention layers (1:3 global-to-SWA ratio); 8 KV heads, head dim 128; per-head sigmoid output gating with per-layer-type rotary scales.
* **Long context**: 1,048,576 tokens.
* **DFlash drafts**: matched draft models ship per quantization for low-latency serving.
* **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.

**Available quantizations:**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Precision</th>
      <th>Target model</th>
      <th>Draft model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BF16**</td>
      <td>[`poolside/Laguna-S-2.1`](https://huggingface.co/poolside/Laguna-S-2.1)</td>
      <td>[`poolside/Laguna-S-2.1-DFlash`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)</td>
    </tr>

    <tr>
      <td>**FP8**</td>
      <td>[`poolside/Laguna-S-2.1-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-FP8)</td>
      <td>[`poolside/Laguna-S-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-FP8)</td>
    </tr>

    <tr>
      <td>**NVFP4**</td>
      <td>[`poolside/Laguna-S-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-NVFP4)</td>
      <td>[`poolside/Laguna-S-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-NVFP4)</td>
    </tr>

    <tr>
      <td>**INT4**</td>
      <td>[`poolside/Laguna-S-2.1-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-INT4)</td>
      <td>[`poolside/Laguna-S-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-S-2.1-DFlash-INT4)</td>
    </tr>
  </tbody>
</table>

The drafts are small BF16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).

**License:** [OpenMDW-1.1](https://openmdw.ai/)

**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-S-2.1) · [Technical report](https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf) · [API platform](https://platform.poolside.ai)

## 2. Configuration Tips

**Attention backend**

Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (reproduced on Laguna-XS-2.1, greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Other attention backend choices have not been fully validated on Laguna; keep the default.

**BF16 memory on H200**

BF16 on H200 leaves less headroom for CUDA-graph capture and NCCL allocations than FP8/INT4. The High-throughput BF16 command carries `--mem-fraction-static 0.80`. FP8, INT4, and all B300/GB300 cells use the default heuristic.

**FP8 shared expert**

`SGLANG_SHARED_EXPERT_TP1=1` is required for FP8 cells on **all hardware** — confirmed on both H200 (TP=8) and GB300 (TP=4). The FP8 checkpoint block-quantizes the shared expert (128×128 scales), which cannot TP-shard cleanly at either TP degree on S-2.1. This env var replicates the shared expert instead of sharding it. INT4 keeps the shared expert in BF16 (no flag needed); BF16 is unquantized. Note: this differs from Laguna-XS-2.1 where TP=4 does not require the flag — the constraint is architecture-specific.

**FP8 and NVFP4 DFlash drafts**

Fixed upstream on 2026-07-21: all DFlash draft configs now use a flat top-level `rope_theta` (the `rope_parameters` block was removed). If a server crashes at draft-model load with `KeyError: 'rope_theta'`, you are serving a draft checkpoint cached before 2026-07-21 — re-download it (e.g. `hf download poolside/Laguna-S-2.1-DFlash-FP8`) to pick up the corrected config.

**DFlash memory**

Low-latency cells carry `--mem-fraction-static 0.7` (sufficient even for BF16 on H200). Dense cells use the default heuristic (except BF16 on H200 — see above).

**BF16 reasoning length**

BF16 reasons approximately 2× longer than FP8/INT4 on AIME25 (median 34.8 k vs 16.9 k tokens), consistently truncating at `max_tokens=64000`. FP8/INT4 truncate at ≈ 2%. For a valid BF16 AIME25 score, serve with `max_tokens ≥ 131072` (the model supports a 1 M context window).

**Chat template**

On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ \~5.8) pass `--chat-template <model-dir>/chat_template.jinja` explicitly.

**Thinking**

Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.

**Served model id**

The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-S-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).

## 3. Advanced Usage

### 3.1 DFlash Speculative Decoding

DFlash is a block-wise speculative decoder: the draft proposes a block of tokens and the target verifies the whole block in one forward pass — output quality is the target's by construction. The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step.

Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput. The generated commands always pair the draft calibrated for the selected target precision.

### 3.2 Reasoning

Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="poolside/Laguna-S-2.1",
>     messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
>     max_tokens=4096,
>     extra_body={"chat_template_kwargs": {"enable_thinking": True}},
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(message.reasoning_content)
> print("=============== Answer ==================")
> print(message.content)
> ```

> [!NOTE]
>
> Give generous `max_tokens` when thinking is enabled — hard problems regularly reason
> for thousands of tokens. Keep thinking off for short-form tasks.

### 3.3 Tool Calling

Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="poolside/Laguna-S-2.1",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
>
> message = response.choices[0].message
> if message.tool_calls:
>     for call in message.tool_calls:
>         print(f"Tool: {call.function.name}")
>         print(f"Args: {call.function.arguments}")
> ```


# Laguna-XS-2.1
Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-XS-2.1

Deploy poolside's Laguna-XS-2.1 — a 33B hybrid-SWA Mixture-of-Experts model (3B active) for agentic coding — with SGLang on NVIDIA H200, B300, and GB300 in BF16, FP8, NVFP4, and INT4.

## Deployment

> **Install SGLang**
>
> Laguna-XS-2.1 support is fully merged to SGLang `main` ([PR #29446](https://github.com/sgl-project/sglang/pull/29446): DFlash speculative decoding + shared-expert fix; [PR #29761](https://github.com/sgl-project/sglang/pull/29761): INT4 loader fix). Any build at or past their merge covers every cell below.
>
> The model ships custom config code on the Hub, so `--trust-remote-code` is required (included in the launch commands).
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install -U uv
> uv venv --python 3.12 && source .venv/bin/activate
>
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> uv pip install --prerelease=allow -e python
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + quantization + strategy to generate the launch command. The two serving strategies cover the common operating points:

* **Low-latency** — DFlash speculative decoding with a matched draft model. Pick for chat and interactive agents.
* **High-throughput** — plain serving. Best for batch workloads, where speculation's draft + rejection overhead costs more than it saves.

On the 8-GPU HGX platforms (H200 / B300), BF16 and NVFP4 run plain `--tp 8`; FP8 and INT4 run `--tp 8 --ep-size 8` because their quantization scales cannot shard the MoE 8-way (see [Configuration Tips](#2-configuration-tips)). The 4-GPU GB300 node runs plain `--tp 4` throughout.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs (TP degree, parsers) on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

[Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) is an open-weight **33B-parameter** hybrid sliding-window-attention MoE model (**\~3B active per token**) from [poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering — the extra-small sibling of [Laguna-M.1](./Laguna-M.1).

**Key Features:**

* **Sparse MoE**: 40 layers, 256 routed experts, top-8 routing.
* **Hybrid attention**: 30 sliding-window layers (window 512) interleaved with 10 full-attention layers; 48 Q / 8 KV heads.
* **Long context**: 262,144 tokens (RoPE + YaRN on the full-attention layers).
* **DFlash drafts**: matched draft models (5-layer, \~0.9 GB) ship per quantization for low-latency serving.
* **Hybrid reasoning**: `<think>…</think>` toggled per request via `chat_template_kwargs={"enable_thinking": …}`.

**Available quantizations:**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Precision</th>
      <th>Target model</th>
      <th>Draft model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BF16**</td>
      <td>[`poolside/Laguna-XS-2.1`](https://huggingface.co/poolside/Laguna-XS-2.1)</td>
      <td>[`poolside/Laguna-XS-2.1-DFlash`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash)</td>
    </tr>

    <tr>
      <td>**FP8**</td>
      <td>[`poolside/Laguna-XS-2.1-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-FP8)</td>
      <td>[`poolside/Laguna-XS-2.1-DFlash-FP8`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-FP8)</td>
    </tr>

    <tr>
      <td>**NVFP4**</td>
      <td>[`poolside/Laguna-XS-2.1-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-NVFP4)</td>
      <td>[`poolside/Laguna-XS-2.1-DFlash-NVFP4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-NVFP4)</td>
    </tr>

    <tr>
      <td>**INT4**</td>
      <td>[`poolside/Laguna-XS-2.1-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-INT4)</td>
      <td>[`poolside/Laguna-XS-2.1-DFlash-INT4`](https://huggingface.co/poolside/Laguna-XS-2.1-DFlash-INT4)</td>
    </tr>
  </tbody>
</table>

The drafts themselves are small bf16 models, each *calibrated against its quantized target* — always pair a target with its matched draft (mixing precisions degrades accept-length).

**License:** Apache 2.0

**Resources:** [Hugging Face](https://huggingface.co/poolside/Laguna-XS-2.1) · [Release blog post](https://poolside.ai/blog/laguna-a-deeper-dive) · [API platform](https://platform.poolside.ai).

## 2. Configuration Tips

**Attention backend**

Leave `--attention-backend` unset for High-throughput cells — auto-select is correct (`fa3` on Hopper, `trtllm_mha` on Blackwell). With DFlash active, auto-select instead falls back to `flashinfer`, which breaks this hybrid-SWA model at `tp ≥ 4` on Blackwell (greedy GSM8K 76% → 28%), so the Low-latency commands pin the target backend explicitly. Leave `--speculative-draft-attention-backend` unset. Never use `triton` attention with Laguna (GSM8K 13%).

**Quantized checkpoints cap plain TP at 4**

`moe_intermediate_size=512` with FP8 block `[128,128]` / INT4 `group_size=128` scales cannot shard 8-way (512/8 = 64 \< 128 granularity): FP8 fails at weight creation, INT4 crashes in the Marlin kernel, on any hardware. The generated 8-GPU FP8/INT4 commands therefore use `--tp 8 --ep-size 8` — expert parallelism keeps whole experts per rank, using all 8 GPUs on one instance. FP8 additionally needs `SGLANG_SHARED_EXPERT_TP1=1` (its shared expert is also block-quantized; INT4's stays bf16). Alternatives: plain `--tp 4`, or `--tp 4 --dp-size 2`. Accuracy is parallelism-independent within eval noise (verified tp1 ≡ tp4 on GB300 and tp4 ≡ tp8+ep8 on H200).

**DFlash memory**

Low-latency cells carry `--mem-fraction-static 0.7`: the default fraction OOMs in the draft vocab all-gather at `tp 4` on GB300. Dense cells use the default heuristic.

**INT4 is mixed-precision**

The INT4 checkpoint quantizes MoE layers in mixed 4-bit / 8-bit config groups. Builds older than [PR #29761](https://github.com/sgl-project/sglang/pull/29761) crash at load with `KeyError: 'Linear'`.

**Chat template**

On transformers ≥ 5.10 the standalone `chat_template.jinja` auto-loads — no flag needed (the server logs `Auto-detected template features: reasoning_parser=poolside_v1, ...`). On older transformers (≤ \~5.8) the `{% include %}` stub in `tokenizer_config.json` cannot resolve and the server silently falls back to a generic template — pass `--chat-template <model-dir>/chat_template.jinja` explicitly there.

**Thinking**

Off by default; opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`. The template gates on `enable_thinking` — the generic `thinking` key is ignored.

**Served model id**

The server registers the model under whatever you pass to `--model-path`; a client's `model` field must match it (`poolside/Laguna-XS-2.1`, or the `-FP8` / `-NVFP4` / `-INT4` id).

## 3. Advanced Usage

### 3.1 DFlash Speculative Decoding

DFlash is a block-wise speculative decoder: the 5-layer draft proposes a block of tokens and the target verifies the whole block in one forward pass, so only target-approved tokens are emitted — output quality is the target's by construction (GSM8K matches dense within noise on every quantization). The speedup lever is **accept-length**, the number of draft tokens surviving verification per target step:

* Measured \~6 tokens/step at `tp 1`, \~4 at `tp 4` (greedy GSM8K, matched-precision pairs; \~3 under mixed reasoning-heavy traffic; FP8 reached 6.75 at `tp 8 + ep 8` on H200) — versus 1 token/step dense.
* Best for interactive / few-stream serving. Under batch-saturated load prefer High-throughput: once the GPU is compute-bound, draft + rejected-token overhead costs aggregate throughput.
* The generated commands always pair the draft calibrated for the selected target precision.

### 3.2 Reasoning

Launch with `--reasoning-parser poolside_v1` (baked into every generated command). Reasoning is opt-in via `enable_thinking=True`; the `<think>` trace lands in `message.reasoning_content`, separate from the final answer in `message.content`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="poolside/Laguna-XS-2.1",
>     messages=[{"role": "user", "content": "What is 15% of 240? Explain briefly."}],
>     max_tokens=2048,
>     extra_body={"chat_template_kwargs": {"enable_thinking": True}},
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(message.reasoning_content)
> print("=============== Answer ==================")
> print(message.content)
> ```

> [!NOTE]
>
> XS-2.1 is an extra-small model — give it generous `max_tokens` when thinking is enabled
> (hard problems regularly reason for thousands of tokens), and keep thinking off for
> short-form tasks.

### 3.3 Tool Calling

Launch with `--tool-call-parser poolside_v1` (baked into every generated command). The parser converts Laguna's `<tool_call>` output into the standard OpenAI `tool_calls` structure. Tool calling works with reasoning off (the default).

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="poolside/Laguna-XS-2.1",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
>
> message = response.choices[0].message
> if message.tool_calls:
>     for call in message.tool_calls:
>         print(f"Tool: {call.function.name}")
>         print(f"Args: {call.function.arguments}")
> ```


# Laguna-XS.2
Source: https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-XS.2



## 1. Model Introduction

[Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) is an open-source hybrid sliding-window-attention MoE model from [Poolside](https://poolside.ai), built for agentic coding and long-horizon software engineering work.

**Key Features:**

* **MoE**: 33.4B total parameters, 3.0B active per token, 256 routed experts (top-8) plus 1 shared.
* **Long context**: 131,072 tokens.
* **Agentic coding**: Tuned for tool-using software engineering agents and long-horizon execution.
* **Hybrid reasoning**: `<think>...</think>` segments toggled per request via `chat_template_kwargs={"enable_thinking": ...}`.

**Available Quantizations:**

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variant</th>
      <th>Hugging Face path</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BF16**</td>
      <td>[`poolside/Laguna-XS.2`](https://huggingface.co/poolside/Laguna-XS.2)</td>
    </tr>

    <tr>
      <td>**FP8**</td>
      <td>[`poolside/Laguna-XS.2-FP8`](https://huggingface.co/poolside/Laguna-XS.2-FP8)</td>
    </tr>

    <tr>
      <td>**NVFP4**</td>
      <td>[`poolside/Laguna-XS.2-NVFP4`](https://huggingface.co/poolside/Laguna-XS.2-NVFP4)</td>
    </tr>
  </tbody>
</table>

**License:** Apache 2.0

For details, see the [Hugging Face model card](https://huggingface.co/poolside/Laguna-XS.2) and the [Laguna deeper-dive blog post](https://poolside.ai/blog/laguna-a-deeper-dive).

## 2. SGLang Installation

Laguna-XS.2 support is on `main` but not yet in a tagged release; install from the SGLang nightly wheel index, or pull a pre-built Docker image:

```bash Command theme={null}
# Install SGLang via pip (CUDA 13) — requires Python 3.10 (this nightly wheel is cp310 only)
python3 -m pip install --upgrade pip
python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu130 \
  "sglang[all]==0.5.12.dev20260509+g096ad02b0"

# CUDA 12: swap to the cu129 index
python3 -m pip install --extra-index-url https://docs.sglang.ai/whl/cu129 \
  "sglang[all]==0.5.12.dev20260509+g096ad02b0"

# Or use Docker (multi-arch amd64/arm64; CUDA 13, H200 / B200)
docker pull lmsysorg/sglang:latest
```

For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to generate a launch command for your hardware.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Trust remote code** (`--trust-remote-code`): Laguna-XS.2 ships custom modeling/config code on the Hugging Face Hub, so this flag is required for the server to load the model.
* **Quantization**: NVFP4 requires Blackwell (B200 / B300); BF16 and FP8 run on either H200 or B200. FP8's first launch triggers a multi-session DeepGEMM JIT pre-compile (\~10-20 min); pre-warm with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-XS.2-FP8` to avoid that cost on every restart.
* **Reasoning parser** (`--reasoning-parser poolside_v1`): Splits `<think>...</think>` segments into `reasoning_content` so `content` holds only the final answer. Disable only if you want the raw `<think>` tags in `content`.
* **Tool call parser** (`--tool-call-parser poolside_v1`): Required for OpenAI-compatible tool-call streaming. Disable only for chat-only deployments.
* **DP attention**: For higher-throughput deployments, enable the DP-Attention toggle — it emits `--attn-dp-size <N>` with `--attn-dp-size` matching `--tp` (tune independently if needed).
* **Thinking default**: Thinking is **off by default** at the model level. Opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.

## 4. Model Invocation

The samples below assume the server is reachable at `http://localhost:30000/v1`.

### 4.1 Basic Chat

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="poolside/Laguna-XS.2",
    messages=[
        {"role": "user", "content": "What is the difference between TCP and UDP?"}
    ],
    max_tokens=1024,
)
print(resp.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are two core protocols of the Internet Protocol (IP) suite, both used for network communication but with key differences:

## Connection Handling
- **TCP**: Connection-oriented protocol that establishes a connection before data transfer (like a phone call)
- **UDP**: Connectionless protocol that sends data without establishing a connection (like sending a letter)

## Reliability
- **TCP**: Guaranteed delivery with error checking, retransmission of lost packets, and flow control
- **UDP**: No guarantee of delivery; packets may be lost, duplicated, or arrive out of order

## Speed & Overhead
- **TCP**: Slower due to connection setup, acknowledgment overhead, and error correction mechanisms
- **UDP**: Faster with minimal overhead since it doesn't wait for acknowledgments or retransmit lost data

## Use Cases
- **TCP**: Web browsing (HTTP/HTTPS), email (SMTP), file transfers (FTP), database connections
- **UDP**: Video streaming, online gaming, VoIP calls, DNS queries, live broadcasts

In essence, TCP prioritizes reliability over speed, while UDP prioritizes speed over reliability.
```

### 4.2 Reasoning (Thinking Mode)

Laguna-XS.2 emits reasoning between `<think>...</think>` tags. The `--reasoning-parser poolside_v1` flag separates the thinking text into `reasoning_content` so `content` holds only the final answer. Thinking is opt-in per request:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

resp = client.chat.completions.create(
    model="poolside/Laguna-XS.2",
    messages=[
        {"role": "user", "content": "If a train travels at 60 km/h for 2.5 hours, how far does it go?"}
    ],
    max_tokens=4096,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)

print("====== Reasoning Content ======")
print(resp.choices[0].message.reasoning_content)
print("====== Answer ======")
print(resp.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
====== Reasoning Content ======
The user is asking a straightforward math problem about distance, speed, and time. I need to calculate the distance using the formula:

Distance = Speed × Time

Given:
- Speed = 60 km/h
- Time = 2.5 hours

So the calculation would be:
Distance = 60 × 2.5 = 150 km

This is a simple multiplication problem. I should provide a clear, direct answer and maybe explain the calculation briefly.

====== Answer ======
To find the distance, use the formula:

Distance = Speed × Time
Distance = 60 km/h × 2.5 h = 150 km

The train travels **150 kilometers**.
```

To disable thinking, omit `extra_body` (off by default) or pass `chat_template_kwargs={"enable_thinking": False}` explicitly.

### 4.3 Tool Calling

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "The city name"},
                },
                "required": ["location"],
            },
        },
    }
]

resp = client.chat.completions.create(
    model="poolside/Laguna-XS.2",
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}],
    tools=tools,
)

msg = resp.choices[0].message
print("====== Reasoning Content ======")
print(msg.reasoning_content)
print("====== Content ======")
print(msg.content)
print("====== Tool Calls ======")
for tc in msg.tool_calls or []:
    print(f"  Function: {tc.function.name}")
    print(f"  Arguments: {tc.function.arguments}")
```

**Output Example:**

```text Output theme={null}
====== Reasoning Content ======
None
====== Content ======

I'll check the current weather in Tokyo for you.

====== Tool Calls ======
  Function: get_weather
  Arguments: {"location": "Tokyo"}
```

`reasoning_content` is `None` because thinking is off by default; `content` carries the brief assistant message that precedes the tool call. Add `extra_body={"chat_template_kwargs": {"enable_thinking": True}}` if you want interleaved reasoning before the tool call.

## 5. Benchmark

### 5.1 Accuracy Benchmark

**Test Environment:**

* Hardware: NVIDIA H200 (4×H200)
* Model: `poolside/Laguna-XS.2` (BF16)
* Tensor Parallelism: 4
* SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs)
* Reasoning Parser: `poolside_v1`
* Tool Call Parser: `poolside_v1`
* Sampling: `temperature=0.6`, `max_tokens=16384`, `chat_template_kwargs={"enable_thinking": true}`, `n_repeats=1`
* Grader: NeMo-Skills `math_verify` (math) and `eval_mcq` (multichoice)

**Results (from [PR #24204](https://github.com/sgl-project/sglang/pull/24204)):**

| Eval | Accuracy |
| - | -: |
| GPQA Diamond | 0.5556 |
| AIME 25 | 0.5667 |
| MMLU | 0.836 |
| SWE-Bench Verified | 0.6540 |

### 5.2 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA H200 (1×H200 for TP=1, 4×H200 for TP=4)
* Model: `poolside/Laguna-XS.2` (BF16)
* SGLang Version: `0.5.12.dev20260509+g096ad02b0` (nightly wheel containing the #24204 merge commit; same code path as the original PR runs)
* Workload: `sglang.bench_serving --backend sglang --dataset-name random` (defaults: `--random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.0`)
* Server flags identical to the accuracy runs above.

#### 5.2.1 Latency Benchmark (10 prompts, concurrency = 1)

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 10 --max-concurrency 1
```

| Metric | TP=1 | TP=4 |
| - | -: | -: |
| Successful requests | 10 | 10 |
| Output token throughput (tok/s) | 193.10 | 238.88 |
| Total token throughput (tok/s) | 471.82 | 583.68 |
| Mean TTFT (ms) | 35.32 | 24.17 |
| Mean TPOT (ms) | 5.10 | 4.13 |
| Median ITL (ms) | 5.14 | 4.14 |

#### 5.2.2 Throughput Benchmark (1000 prompts, concurrency = 100)

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang \
  --host 0.0.0.0 --port 30000 \
  --dataset-name random --num-prompts 1000 --max-concurrency 100
```

| Metric | TP=1 | TP=4 |
| - | -: | -: |
| Successful requests | 1000 | 1000 |
| Request throughput (req/s) | 7.32 | 14.61 |
| Output token throughput (tok/s) | 3739.30 | 7465.18 |
| Peak output token throughput (tok/s) | 4718.00 | 10133.00 |
| Total token throughput (tok/s) | 7485.82 | 14944.81 |
| Mean TTFT (ms) | 115.17 | 68.36 |
| Mean TPOT (ms) | 25.51 | 12.71 |
| Median ITL (ms) | 21.31 | 10.64 |

TP=4 delivers roughly 2.0× total-token throughput and \~1.7× lower mean TTFT compared to TP=1 on the `cc=100` random workload.


# Qwen2.5-VL
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen2.5-VL



## 1. Model Introduction

**[Qwen2.5-VL](https://huggingface.co/collections/Qwen/qwen25-vl)** is a vision-language model series from the Qwen team, offering significant improvements over its predecessor in understanding, reasoning, and multi-modal processing.

**Key Features:**

* **Understand things visually**: Proficient in recognizing common objects such as flowers, birds, fish, and insects, and it is highly capable of analyzing texts, charts, icons, graphics, and layouts within images.
* **More Agentic**: Play as a visual agent that can reason and dynamically direct tools, which is capable of computer use and phone use.
* **Understanding long videos and capturing events**: Supports comprehending videos of over 1 hour, and this time it has a new ability of capturing event by pinpointing the relevant video segments.
* **Capable of visual localization in different formats**: Accurately localize objects in an image by generating bounding boxes or points, and it can provide stable JSON outputs for coordinates and attributes.
* **Generating structured outputs**: Supports structured outputs of the contents, benefiting usages in finance, commerce, etc for data like scans of invoices, forms, tables, etc.
* **Dynamic Resolution and Frame Rate Training for Video Understanding**: Extend dynamic resolution to the temporal dimension by adopting dynamic FPS sampling, enabling the model to comprehend videos at various sampling rates. Accordingly, we update mRoPE in the time dimension with IDs and absolute time alignment, enabling the model to learn temporal sequence and speed, and ultimately acquire the ability to pinpoint specific moments.
* **Multiple Sizes**: Available in 3B, 7B, 32B, and 72B variants to suit different deployment needs.
* **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).

For more details, please refer to the [official Qwen2.5-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

## 3. Model Deployment

This section provides deployment configurations optimized for AMD MI300X, MI325X and MI355X as well as Intel Xeon CPU hardware platforms and different use cases.

### 3.1 Basic Configuration

The Qwen2.5-VL series offers models in various sizes. The following configurations have been verified on AMD MI300X, MI325X and MI355X GPUs as well as Intel Xeon CPUs.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model size.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory Management**: For the 72B model on MI300X/MI325X/MI355X, we have verified successful deployment with `--context-length 128000`. Smaller context lengths can be used to reduce memory usage if needed.
* **Multi-GPU Deployment**: Use Tensor Parallelism (`--tp`) to scale across multiple GPUs. For example, use `--tp 8` for the 72B model and `--tp 2` for the 32B model on MI300X/MI325X/MI355X.
* **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Multi-Modal Inputs

Qwen2.5-VL supports image inputs. Here's a basic example with single image input:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 2.31s
Generated text: Auntie Anne's

CINNAMON SUGAR
1 x 17,000
SUB TOTAL
17,000

GRAND TOTAL
17,000

CASH IDR
20,000

CHANGE DUE
3,000
```

**Multi-Image Input Example:**

Qwen2.5-VL can process multiple images in a single request for comparison or analysis:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
                }
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
                }
            },
            {
                "type": "text",
                "text": "Compare these two images and describe the differences in 100 words or less."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 13.79s
Generated text: The first image shows a single red taxi driving on a street with a few other taxis in the background. The second image shows a large number of taxis parked in a lot, with some appearing to be in various states of repair. The first image has a single taxi with a visible license plate, while the second image has multiple taxis with different license plates. The first image has a clear view of the street and surrounding area, while the second image is taken from an elevated perspective, showing a wider view of the parking lot and the surrounding area.
```

**Note:**

* You can also provide local file paths using `file://` protocol.
* For larger images, you may need more memory, adjust `--mem-fraction-static` accordingly.

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: AMD MI300X GPU (8x)
* Model: Qwen2.5-VL-72B-Instruct
* Tensor Parallelism: 8
* SGLang Version: 0.5.6

We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen2.5-VL-72B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen2.5-VL-72B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen2.5-VL-72B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  37.99
Total input tokens:                      24781
Total input text tokens:                 821
Total input vision tokens:               23960
Total generated tokens:                  4220
Total generated tokens (retokenized):    2365
Request throughput (req/s):              0.26
Input token throughput (tok/s):          652.26
Output token throughput (tok/s):         111.07
Peak output token throughput (tok/s):    128.00
Peak concurrent requests:                2
Total token throughput (tok/s):          763.34
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3797.61
Median E2E Latency (ms):                 3140.90
P90 E2E Latency (ms):                    6545.54
P99 E2E Latency (ms):                    7939.56
---------------Time to First Token----------------
Mean TTFT (ms):                          504.45
Median TTFT (ms):                        510.93
P99 TTFT (ms):                           521.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.82
Median TPOT (ms):                        7.82
P99 TPOT (ms):                           7.84
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.07
Median ITL (ms):                         7.90
P95 ITL (ms):                            15.79
P99 ITL (ms):                            15.93
Max ITL (ms):                            23.60
==================================================
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen2.5-VL-72B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  454.68
Total input tokens:                      2481865
Total input text tokens:                 85865
Total input vision tokens:               2396000
Total generated tokens:                  510855
Total generated tokens (retokenized):    296466
Request throughput (req/s):              2.20
Input token throughput (tok/s):          5458.50
Output token throughput (tok/s):         1123.55
Peak output token throughput (tok/s):    5004.00
Peak concurrent requests:                106
Total token throughput (tok/s):          6582.05
Concurrency:                             98.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   44844.92
Median E2E Latency (ms):                 42866.15
P90 E2E Latency (ms):                    82798.20
P99 E2E Latency (ms):                    106306.30
---------------Time to First Token----------------
Mean TTFT (ms):                          4507.79
Median TTFT (ms):                        1180.83
P99 TTFT (ms):                           39975.22
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          80.26
Median TPOT (ms):                        82.38
P99 TPOT (ms):                           152.89
---------------Inter-Token Latency----------------
Mean ITL (ms):                           100.66
Median ITL (ms):                         13.26
P95 ITL (ms):                            428.45
P99 ITL (ms):                            1393.35
Max ITL (ms):                            31943.26
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 MMMU Benchmark

You can evaluate the model's accuracy using the MMMU dataset:

* Benchmark Command:

```shell Command theme={null}
python3 benchmark/mmmu/bench_sglang.py \
    --port 30000 \
    --concurrency 64
```

```text Output theme={null}
Benchmark time: 97.75084622902796
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.633, 'num': 30},
 'Agriculture': {'acc': 0.5, 'num': 30},
 'Architecture_and_Engineering': {'acc': 0.367, 'num': 30},
 'Art': {'acc': 0.767, 'num': 30},
 'Art_Theory': {'acc': 0.9, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.7, 'num': 30},
 'Biology': {'acc': 0.467, 'num': 30},
 'Chemistry': {'acc': 0.433, 'num': 30},
 'Clinical_Medicine': {'acc': 0.733, 'num': 30},
 'Computer_Science': {'acc': 0.567, 'num': 30},
 'Design': {'acc': 0.833, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30},
 'Economics': {'acc': 0.767, 'num': 30},
 'Electronics': {'acc': 0.433, 'num': 30},
 'Energy_and_Power': {'acc': 0.467, 'num': 30},
 'Finance': {'acc': 0.533, 'num': 30},
 'Geography': {'acc': 0.633, 'num': 30},
 'History': {'acc': 0.7, 'num': 30},
 'Literature': {'acc': 0.867, 'num': 30},
 'Manage': {'acc': 0.633, 'num': 30},
 'Marketing': {'acc': 0.733, 'num': 30},
 'Materials': {'acc': 0.333, 'num': 30},
 'Math': {'acc': 0.533, 'num': 30},
 'Mechanical_Engineering': {'acc': 0.433, 'num': 30},
 'Music': {'acc': 0.367, 'num': 30},
 'Overall': {'acc': 0.62, 'num': 900},
 'Overall-Art and Design': {'acc': 0.717, 'num': 120},
 'Overall-Business': {'acc': 0.66, 'num': 150},
 'Overall-Health and Medicine': {'acc': 0.693, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.775, 'num': 120},
 'Overall-Science': {'acc': 0.553, 'num': 150},
 'Overall-Tech and Engineering': {'acc': 0.443, 'num': 210},
 'Pharmacy': {'acc': 0.833, 'num': 30},
 'Physics': {'acc': 0.7, 'num': 30},
 'Psychology': {'acc': 0.767, 'num': 30},
 'Public_Health': {'acc': 0.733, 'num': 30},
 'Sociology': {'acc': 0.767, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.62
```


# Qwen3
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3



## 1. Model Introduction

[Qwen3 series](https://github.com/QwenLM/Qwen3) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.

This generation delivers comprehensive upgrades across the board:

* **Stronger general intelligence**: Significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage.
* **Broader multilingual knowledge**: Substantial gains in long-tail knowledge coverage across multiple languages.
* **More helpful & aligned responses**: Markedly better alignment with user preferences in subjective and open-ended tasks, enabling higher-quality, more useful text generation.
* **Extended context length**: Enhanced capabilities in understanding and reasoning over 256K-token long contexts.
* **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
* **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.

For more details, please refer to the [official Qwen3 GitHub Repository](https://github.com/QwenLM/Qwen3).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory Management:** Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
* **Expert Parallelism:** SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
* **Kernel Tuning:** For MoE Triton kernel tuning on your specific hardware, refer to [fused\_moe\_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).
* **Qwen3-4B and Qwen3-8B on H200:** For concurrent requests with long prompts, benchmark `--enable-mixed-chunk` to mix decode requests into chunked-prefill batches. This can reduce median time to first token (TTFT), but may increase time per output token (TPOT) and end-to-end tail latency. Compare these metrics and throughput on your workload before enabling it.
* **Speculative Decoding:** Using Speculative Decoding for latency-sensitive scenarios.
  * `--speculative-algorithm EAGLE3`: Speculative decoding algorithm
  * `--speculative-num-steps 3`: Number of speculative verification rounds
  * `--speculative-eagle-topk 1`: Top-k sampling for draft tokens
  * `--speculative-num-draft-tokens 4`: Number of draft tokens per step
  * `--speculative-draft-model-path`: The path of the draft model weights. This can be a local folder or a Hugging Face repo ID such as [`lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan`](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan).
* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

Qwen3-235B-A22B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-235B-A22B-Thinking-2507 \
  --reasoning-parser qwen3 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="Qwen/Qwen3-235B-A22B-Thinking-2507",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================

Okay, so I need to figure out what 15% of 240 is. Hmm, percentages can sometimes trip me up, but I think I remember some basics. Let me start by recalling that "percent" means "per hundred," so 15% is the same as 15 per 100, or 15/100. So, maybe I can convert 15% into a decimal first? Yeah, I think that's a common method.
...
So conclusion: The answer is 36.

=============== Content =================


To determine what 15% of 240 is, we can follow a systematic approach that involves converting the percentage to a decimal and then performing multiplication. Here's a step-by-step breakdown of the solution:

....

### Final Answer:

$$
\boxed{36}
$$

Thus, 15% of 240 is **36**.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.3 Tool Calling

Qwen3 supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-235B-A22B-Thinking-2507 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen25 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-235B-A22B-Thinking-2507",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================

Okay, the user is asking for the weather in Beijing. Let me check the tools available. There's a function called get_weather that takes location and unit parameters. The location is required, so I need to specify Beijing as the location. The unit is optional and can be either celsius or fahrenheit. Since the user didn't specify the unit, maybe I should default to a common one. In China, they usually use celsius, so I'll set unit to celsius. I'll call the get_weather function with location: Beijing and unit: celsius. That should get the current weather for them.



=============== Content =================

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="Qwen/Qwen3-235B-A22B-Thinking-2507",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The current weather in Beijing is **22°C** and **sunny**. A perfect day to enjoy outdoor activities! 🌞"
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8x)
* Model: Qwen3-235B-A22B-Instruct-2507
* Tensor Parallelism: 8
* sglang version: 0.5.6

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.

#### 5.1.1 Standard Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --tp 8
```

##### 5.1.1.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  43.56
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  4210
Total generated tokens (retokenized):    4206
Request throughput (req/s):              0.23
Input token throughput (tok/s):          140.07
Output token throughput (tok/s):         96.65
Peak output token throughput (tok/s):    100.00
Peak concurrent requests:                2
Total token throughput (tok/s):          236.72
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4353.63
Median E2E Latency (ms):                 3475.79
---------------Time to First Token----------------
Mean TTFT (ms):                          99.03
Median TTFT (ms):                        92.18
P99 TTFT (ms):                           166.05
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.12
Median TPOT (ms):                        10.12
P99 TPOT (ms):                           10.15
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.13
Median ITL (ms):                         10.12
P95 ITL (ms):                            10.49
P99 ITL (ms):                            10.70
Max ITL (ms):                            13.45
==================================================
```

##### 5.1.1.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  48.95
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  40725
Total generated tokens (retokenized):    40716
Request throughput (req/s):              1.63
Input token throughput (tok/s):          810.44
Output token throughput (tok/s):         832.04
Peak output token throughput (tok/s):    1151.00
Peak concurrent requests:                21
Total token throughput (tok/s):          1642.48
Concurrency:                             13.61
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8326.72
Median E2E Latency (ms):                 8827.86
---------------Time to First Token----------------
Mean TTFT (ms):                          215.70
Median TTFT (ms):                        88.82
P99 TTFT (ms):                           727.08
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.36
Median TPOT (ms):                        16.12
P99 TPOT (ms):                           24.09
---------------Inter-Token Latency----------------
Mean ITL (ms):                           15.96
Median ITL (ms):                         14.52
P95 ITL (ms):                            16.04
P99 ITL (ms):                            67.69
Max ITL (ms):                            457.52
==================================================
```

##### 5.1.1.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  92.07
Total input tokens:                      249831
Total input text tokens:                 249831
Total input vision tokens:               0
Total generated tokens:                  252162
Total generated tokens (retokenized):    251124
Request throughput (req/s):              5.43
Input token throughput (tok/s):          2713.46
Output token throughput (tok/s):         2738.78
Peak output token throughput (tok/s):    4400.00
Peak concurrent requests:                110
Total token throughput (tok/s):          5452.24
Concurrency:                             90.50
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16665.09
Median E2E Latency (ms):                 16060.10
---------------Time to First Token----------------
Mean TTFT (ms):                          260.55
Median TTFT (ms):                        122.68
P99 TTFT (ms):                           863.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.94
Median TPOT (ms):                        34.04
P99 TPOT (ms):                           41.19
---------------Inter-Token Latency----------------
Mean ITL (ms):                           32.59
Median ITL (ms):                         23.54
P95 ITL (ms):                            69.79
P99 ITL (ms):                            119.09
Max ITL (ms):                            577.70
==================================================
```

#### 5.1.2 Reasoning Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --tp 8
```

##### 5.1.2.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  457.45
Total input tokens:                      6101
Total input text tokens:                 6101
Total input vision tokens:               0
Total generated tokens:                  44452
Total generated tokens (retokenized):    44059
Request throughput (req/s):              0.02
Input token throughput (tok/s):          13.34
Output token throughput (tok/s):         97.17
Peak output token throughput (tok/s):    100.00
Peak concurrent requests:                2
Total token throughput (tok/s):          110.51
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   45742.42
Median E2E Latency (ms):                 49266.87
---------------Time to First Token----------------
Mean TTFT (ms):                          110.60
Median TTFT (ms):                        109.36
P99 TTFT (ms):                           167.43
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.23
Median TPOT (ms):                        10.24
P99 TPOT (ms):                           10.32
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.27
Median ITL (ms):                         10.26
P95 ITL (ms):                            10.71
P99 ITL (ms):                            10.97
Max ITL (ms):                            15.79
==================================================
```

##### 5.1.2.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  340.17
Total input tokens:                      39668
Total input text tokens:                 39668
Total input vision tokens:               0
Total generated tokens:                  318226
Total generated tokens (retokenized):    318104
Request throughput (req/s):              0.24
Input token throughput (tok/s):          116.61
Output token throughput (tok/s):         935.49
Peak output token throughput (tok/s):    1120.00
Peak concurrent requests:                19
Total token throughput (tok/s):          1052.10
Concurrency:                             13.85
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   58885.30
Median E2E Latency (ms):                 59238.70
---------------Time to First Token----------------
Mean TTFT (ms):                          169.71
Median TTFT (ms):                        101.61
P99 TTFT (ms):                           455.71
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.82
Median TPOT (ms):                        14.91
P99 TPOT (ms):                           15.20
---------------Inter-Token Latency----------------
Mean ITL (ms):                           14.76
Median ITL (ms):                         14.63
P95 ITL (ms):                            15.46
P99 ITL (ms):                            16.62
Max ITL (ms):                            104.94
==================================================
```

##### 5.1.2.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  544.83
Total input tokens:                      158939
Total input text tokens:                 158939
Total input vision tokens:               0
Total generated tokens:                  1300705
Total generated tokens (retokenized):    1293015
Request throughput (req/s):              0.59
Input token throughput (tok/s):          291.72
Output token throughput (tok/s):         2387.34
Peak output token throughput (tok/s):    3008.00
Peak concurrent requests:                68
Total token throughput (tok/s):          2679.06
Concurrency:                             56.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   95937.70
Median E2E Latency (ms):                 99362.32
---------------Time to First Token----------------
Mean TTFT (ms):                          265.03
Median TTFT (ms):                        129.11
P99 TTFT (ms):                           823.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.66
Median TPOT (ms):                        24.07
P99 TPOT (ms):                           24.97
---------------Inter-Token Latency----------------
Mean ITL (ms):                           23.54
Median ITL (ms):                         23.07
P95 ITL (ms):                            25.92
P99 ITL (ms):                            63.87
Max ITL (ms):                            408.30
==================================================
```

#### 5.1.3 Summarization Scenario Benchmark

##### 5.1.3.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  44.82
Total input tokens:                      41941
Total input text tokens:                 41941
Total input vision tokens:               0
Total generated tokens:                  4210
Total generated tokens (retokenized):    4210
Request throughput (req/s):              0.22
Input token throughput (tok/s):          935.86
Output token throughput (tok/s):         93.94
Peak output token throughput (tok/s):    99.00
Peak concurrent requests:                2
Total token throughput (tok/s):          1029.80
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4479.60
Median E2E Latency (ms):                 3622.99
---------------Time to First Token----------------
Mean TTFT (ms):                          139.90
Median TTFT (ms):                        114.85
P99 TTFT (ms):                           225.17
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.31
Median TPOT (ms):                        10.33
P99 TPOT (ms):                           10.51
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.33
Median ITL (ms):                         10.33
P95 ITL (ms):                            10.73
P99 ITL (ms):                            10.93
Max ITL (ms):                            14.48
==================================================
```

##### 5.1.3.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  50.68
Total input tokens:                      300020
Total input text tokens:                 300020
Total input vision tokens:               0
Total generated tokens:                  41589
Total generated tokens (retokenized):    41578
Request throughput (req/s):              1.58
Input token throughput (tok/s):          5920.41
Output token throughput (tok/s):         820.69
Peak output token throughput (tok/s):    1200.00
Peak concurrent requests:                20
Total token throughput (tok/s):          6741.10
Concurrency:                             13.90
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8805.54
Median E2E Latency (ms):                 9368.79
---------------Time to First Token----------------
Mean TTFT (ms):                          284.29
Median TTFT (ms):                        168.48
P99 TTFT (ms):                           1027.21
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.81
Median TPOT (ms):                        16.66
P99 TPOT (ms):                           27.18
---------------Inter-Token Latency----------------
Mean ITL (ms):                           16.42
Median ITL (ms):                         13.68
P95 ITL (ms):                            17.23
P99 ITL (ms):                            90.75
Max ITL (ms):                            574.64
==================================================
```

##### 5.1.3.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-235B-A22B-Instruct-2507 \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  94.77
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total input vision tokens:               0
Total generated tokens:                  169680
Total generated tokens (retokenized):    169640
Request throughput (req/s):              3.38
Input token throughput (tok/s):          13441.86
Output token throughput (tok/s):         1790.43
Peak output token throughput (tok/s):    2687.00
Peak concurrent requests:                70
Total token throughput (tok/s):          15232.28
Concurrency:                             58.63
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   17364.14
Median E2E Latency (ms):                 17495.95
---------------Time to First Token----------------
Mean TTFT (ms):                          238.22
Median TTFT (ms):                        203.27
P99 TTFT (ms):                           510.48
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          32.50
Median TPOT (ms):                        34.27
P99 TPOT (ms):                           40.59
---------------Inter-Token Latency----------------
Mean ITL (ms):                           32.36
Median ITL (ms):                         22.50
P95 ITL (ms):                            97.81
P99 ITL (ms):                            151.55
Max ITL (ms):                            352.79
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Results**:

  * Qwen/Qwen3-235B-A22B-Instruct-2507
    ```text Output theme={null}
    Accuracy: 0.945
    Invalid: 0.000
    Latency: 11.980 s
    Output throughput: 2358.105 token/s
    ```


# Qwen3-Coder
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Coder



## 1. Model Introduction

[Qwen3-Coder](https://huggingface.co/collections/Qwen/qwen3-coder) is the latest code-focused large language model series from the Qwen team. Built on the foundation of Qwen3, Qwen3-Coder delivers exceptional performance in code generation, understanding, and reasoning tasks.

**Key Features:**

* **State-of-the-art Coding Performance**: Achieves top-tier results on HumanEval, MBPP, LiveCodeBench, and other major coding benchmarks.
* **Tool Calling Support**: Native support for function calling and tool use, enabling seamless integration with external APIs and services.
* **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
* **Multilingual Code Support**: Proficient in Python, JavaScript, TypeScript, Java, C++, Go, Rust, and many other programming languages.
* **MoE Architecture**: Efficient Mixture-of-Experts design for optimal performance-to-cost ratio.
* **ROCm Support**: Compatible with AMD MI300X, MI325X and MI355X GPUs via SGLang (verified).
* **NVIDIA GPU Support**: Compatible with NVIDIA GB200 and B200 GPUs via SGLang (verified).

For more details, please refer to the [official Qwen3-Coder GitHub Repository](https://github.com/QwenLM/Qwen3-Coder).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

## 3. Model Deployment

This section provides deployment configurations verified on AMD MI300X, MI325X, MI355X, NVIDIA B200, GB200, and Intel Xeon CPU hardware platforms.

### 3.1 Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, and quantization method.

Run this Command:

<pre />

### 3.2 Configuration Tips

**AMD (MI300X/MI325X/MI355X):**

* **Memory Management**: We have verified successful deployment on MI300X/MI325X/MI355X with `--context-length 8192`. Larger context lengths may be supported but require additional memory.
* **Expert Parallelism**: For 480B-A35B with FP8 quantization, `--ep 2` is required to satisfy the dimension alignment requirement.
* **Page Size**: `--page-size 32` is recommended for MoE models to optimize memory usage.
* **Environment Variable**: If you encounter aiter-related issues, try setting `SGLANG_USE_AITER=0`.

**NVIDIA (B200/GB200):**

* **GB200 Parallelism**: Use `--tp 4 --ep 4` on GB200. B200 uses the default NVIDIA settings generated above.
* **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`).
* **DP Attention**: NVFP4 configuration supports DP attention (`--attn-dp-size <N>`) for improved throughput.

**Intel Xeon CPU:**

* Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

**General:**

* **Tool Use**: To enable tool calling capabilities, add `--tool-call-parser qwen3_coder` to the launch command.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Code Generation Example

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": "Write a Python function that implements binary search on a sorted list. Include docstring and type hints."
    }
]

response = client.chat.completions.create(
    model="Qwen/Qwen3-Coder-480B-A35B-Instruct",
    messages=messages,
    max_tokens=2048,
    temperature=0.7
)

print(response.choices[0].message.content)
```

**Example Output:**

````text Output theme={null}
```python
from typing import List, Optional, TypeVar

T = TypeVar('T')

def binary_search(arr: List[T], target: T) -> Optional[int]:
    """
    Perform binary search on a sorted list to find the index of a target element.

    This function implements the binary search algorithm, which efficiently finds
    a target value in a sorted array by repeatedly dividing the search interval
    in half.

    Args:
        arr (List[T]): A sorted list of elements to search through.
        target (T): The element to search for in the list.

    Returns:
        Optional[int]: The index of the target element if found, None otherwise.

    Time Complexity:
        O(log n) where n is the number of elements in the array.

    Space Complexity:
        O(1) - iterative implementation uses constant extra space.

    Examples:
        >>> binary_search([1, 2, 3, 4, 5], 3)
        2
        >>> binary_search([1, 2, 3, 4, 5], 6)
        None
        >>> binary_search(['a', 'b', 'c', 'd'], 'b')
        1
        >>> binary_search([], 1)
        None
    """
    if not arr:
        return None

    left: int = 0
    right: int = len(arr) - 1

    while left <= right:
        mid: int = (left + right) // 2

        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return None

# Alternative recursive implementation
def binary_search_recursive(arr: List[T], target: T, left: int = 0, right: Optional[int] = None) -> Optional[int]:
    """
    Perform binary search recursively on a sorted list to find the index of a target element.

    Args:
        arr (List[T]): A sorted list of elements to search through.
        target (T): The element to search for in the list.
        left (int): Left boundary of the search range (inclusive).
        right (Optional[int]): Right boundary of the search range (inclusive).

    Returns:
        Optional[int]: The index of the target element if found, None otherwise.

    Time Complexity:
        O(log n) where n is the number of elements in the array.

    Space Complexity:
        O(log n) due to recursive call stack.

    Examples:
        >>> binary_search_recursive([1, 2, 3, 4, 5], 3)
        2
        >>> binary_search_recursive([1, 2, 3, 4, 5], 6)
        None
    """
    if not arr:
        return None

    if right is None:
        right = len(arr) - 1

    if left > right:
        return None

    mid: int = (left + right) // 2

    if arr[mid] == target:
        return mid
    elif arr[mid] < target:
        return binary_search_recursive(arr, target, mid + 1, right)
    else:
        return binary_search_recursive(arr, target, left, mid - 1)
```

This implementation provides:

1. **Main function** (`binary_search`): An iterative implementation that's more memory-efficient
2. **Alternative function** (`binary_search_recursive`): A recursive implementation for educational purposes
3. **Type hints**: Using generics (`TypeVar`) to work with any comparable type
4. **Comprehensive docstring**: Including description, parameters, return value, complexity analysis, and examples
5. **Edge case handling**: Empty lists, elements not found, etc.
6. **Clear variable names**: Self-documenting code
7. **Examples**: Doctest-style examples in the docstring

The function works with any sorted list of comparable elements (integers, strings, etc.) and returns the index of the target element if found, or `None` if not found.
````

#### 4.2.2 Tool Calling Example

Qwen3-Coder supports tool calling capabilities. Enable the tool call parser during deployment. The following example uses 30B-A3B model:

```shell Command theme={null}
SGLANG_USE_AITER=0 python -m sglang.launch_server \
  --model Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --tp 1 \
  --context-length 8192 \
  --page-size 32 \
  --tool-call-parser qwen3_coder
```

**Python Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "execute_code",
            "description": "Execute Python code and return the result",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {
                        "type": "string",
                        "description": "The Python code to execute"
                    }
                },
                "required": ["code"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="Qwen/Qwen3-Coder-30B-A3B-Instruct",
    messages=[
        {"role": "user", "content": "Calculate the factorial of 10 using Python"}
    ],
    tools=tools,
    temperature=0.7
)

# Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]
    print(f"Tool: {tool_call.function.name}")
    print(f"Arguments: {tool_call.function.arguments}")
else:
    # Model may return tool call in content format
    print(response.choices[0].message.content)
```

**Example Output:**

```text Output theme={null}
Tool: execute_code
Arguments: {"code": "def factorial(n):\n    if n == 0 or n == 1:\n        return 1\n    else:\n        return n * factorial(n-1)\n\nresult = factorial(10)\nresult"}
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: AMD MI300X GPU (8x)
* Model: Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
* Tensor Parallelism: 8
* Expert Parallelism: 2
* sglang version: 0.5.7

We use SGLang's built-in benchmarking tool to conduct performance evaluation with random dataset.

#### 5.1.1 AMD Standard Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
SGLANG_USE_AITER=0 python -m sglang.launch_server \
  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --tp 8 \
  --ep 2 \
  --context-length 8192 \
  --page-size 32 \
  --trust-remote-code
```

##### 5.1.1.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  73.79
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4104
Request throughput (req/s):              0.14
Input token throughput (tok/s):          82.68
Output token throughput (tok/s):         57.19
Peak output token throughput (tok/s):    59.00
Peak concurrent requests:                2
Total token throughput (tok/s):          139.86
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7376.26
Median E2E Latency (ms):                 5851.51
P90 E2E Latency (ms):                    13351.89
P99 E2E Latency (ms):                    16908.32
---------------Time to First Token----------------
Mean TTFT (ms):                          191.93
Median TTFT (ms):                        126.06
P99 TTFT (ms):                           662.15
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          17.06
Median TPOT (ms):                        17.07
P99 TPOT (ms):                           17.08
---------------Inter-Token Latency----------------
Mean ITL (ms):                           17.06
Median ITL (ms):                         17.06
P95 ITL (ms):                            17.14
P99 ITL (ms):                            17.19
Max ITL (ms):                            18.53
==================================================
```

##### 5.1.1.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  87.04
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40364
Request throughput (req/s):              0.92
Input token throughput (tok/s):          455.77
Output token throughput (tok/s):         468.83
Peak output token throughput (tok/s):    608.00
Peak concurrent requests:                20
Total token throughput (tok/s):          924.59
Concurrency:                             13.76
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   14966.88
Median E2E Latency (ms):                 15871.93
P90 E2E Latency (ms):                    24983.41
P99 E2E Latency (ms):                    29504.85
---------------Time to First Token----------------
Mean TTFT (ms):                          388.94
Median TTFT (ms):                        157.49
P99 TTFT (ms):                           1318.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.41
Median TPOT (ms):                        29.22
P99 TPOT (ms):                           43.48
---------------Inter-Token Latency----------------
Mean ITL (ms):                           28.64
Median ITL (ms):                         26.42
P95 ITL (ms):                            27.51
P99 ITL (ms):                            131.63
Max ITL (ms):                            995.11
==================================================
```

##### 5.1.1.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  177.82
Total input tokens:                      158939
Total input text tokens:                 158939
Total generated tokens:                  170134
Total generated tokens (retokenized):    168387
Request throughput (req/s):              1.80
Input token throughput (tok/s):          893.84
Output token throughput (tok/s):         956.80
Peak output token throughput (tok/s):    1728.00
Peak concurrent requests:                70
Total token throughput (tok/s):          1850.64
Concurrency:                             58.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   32716.53
Median E2E Latency (ms):                 30896.37
P90 E2E Latency (ms):                    65605.24
P99 E2E Latency (ms):                    80970.63
---------------Time to First Token----------------
Mean TTFT (ms):                          372.97
Median TTFT (ms):                        181.67
P99 TTFT (ms):                           529.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          62.98
Median TPOT (ms):                        50.44
P99 TPOT (ms):                           204.24
---------------Inter-Token Latency----------------
Mean ITL (ms):                           60.95
Median ITL (ms):                         37.87
P95 ITL (ms):                            143.98
P99 ITL (ms):                            148.02
Max ITL (ms):                            36863.32
==================================================
```

#### 5.1.2 NVIDIA (B200/GB200) Standard Scenario Benchmark

The following runs use the same random dataset benchmark client commands as the AMD section. On B200, launch the server with the following command:

````bash theme={null}
sglang serve --model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 --tp 8 --ep 8 --context-length 8192 --page-size 32 --trust-remote-code

##### 5.1.2.1 FP8 Model

- Low Concurrency:

```text Output
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  42.68
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4204
Request throughput (req/s):              0.23
Input token throughput (tok/s):          142.95
Output token throughput (tok/s):         98.88
Peak output token throughput (tok/s):    102.00
Peak concurrent requests:                2
Total token throughput (tok/s):          241.83
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4266.06
Median E2E Latency (ms):                 3420.24
P90 E2E Latency (ms):                    7717.19
P99 E2E Latency (ms):                    9504.50
---------------Time to First Token----------------
Mean TTFT (ms):                          112.03
Median TTFT (ms):                        112.70
P99 TTFT (ms):                           115.35
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.87
Median TPOT (ms):                        9.86
P99 TPOT (ms):                           9.92
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.87
Median ITL (ms):                         9.87
P95 ITL (ms):                            10.06
P99 ITL (ms):                            10.18
Max ITL (ms):                            14.80
==================================================
````

* Medium Concurrency:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  60.80
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40543
Request throughput (req/s):              1.32
Input token throughput (tok/s):          652.43
Output token throughput (tok/s):         671.13
Peak output token throughput (tok/s):    864.00
Peak concurrent requests:                20
Total token throughput (tok/s):          1323.57
Concurrency:                             13.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   10587.26
Median E2E Latency (ms):                 11486.18
P90 E2E Latency (ms):                    17374.75
P99 E2E Latency (ms):                    21107.18
---------------Time to First Token----------------
Mean TTFT (ms):                          155.27
Median TTFT (ms):                        121.57
P99 TTFT (ms):                           294.31
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          20.77
Median TPOT (ms):                        21.13
P99 TPOT (ms):                           23.62
---------------Inter-Token Latency----------------
Mean ITL (ms):                           20.49
Median ITL (ms):                         18.73
P95 ITL (ms):                            19.65
P99 ITL (ms):                            98.85
Max ITL (ms):                            536.87
==================================================
```

* High Concurrency:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  100.07
Total input tokens:                      158939
Total input text tokens:                 158939
Total generated tokens:                  170134
Total generated tokens (retokenized):    169119
Request throughput (req/s):              3.20
Input token throughput (tok/s):          1588.32
Output token throughput (tok/s):         1700.19
Peak output token throughput (tok/s):    2303.00
Peak concurrent requests:                71
Total token throughput (tok/s):          3288.51
Concurrency:                             57.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   18114.01
Median E2E Latency (ms):                 18279.15
P90 E2E Latency (ms):                    30557.22
P99 E2E Latency (ms):                    35889.84
---------------Time to First Token----------------
Mean TTFT (ms):                          346.40
Median TTFT (ms):                        129.75
P99 TTFT (ms):                           1370.20
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.76
Median TPOT (ms):                        34.62
P99 TPOT (ms):                           39.97
---------------Inter-Token Latency----------------
Mean ITL (ms):                           33.48
Median ITL (ms):                         25.70
P95 ITL (ms):                            99.36
P99 ITL (ms):                            132.30
Max ITL (ms):                            1132.39
==================================================
```

##### 5.1.2.2 NVFP4 Model

* Low Concurrency:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  34.49
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4218
Request throughput (req/s):              0.29
Input token throughput (tok/s):          176.87
Output token throughput (tok/s):         122.34
Peak output token throughput (tok/s):    127.00
Peak concurrent requests:                2
Total token throughput (tok/s):          299.21
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3448.01
Median E2E Latency (ms):                 2768.11
P90 E2E Latency (ms):                    6225.73
P99 E2E Latency (ms):                    7668.26
---------------Time to First Token----------------
Mean TTFT (ms):                          104.55
Median TTFT (ms):                        105.38
P99 TTFT (ms):                           105.63
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.94
Median TPOT (ms):                        7.95
P99 TPOT (ms):                           7.97
---------------Inter-Token Latency----------------
Mean ITL (ms):                           7.94
Median ITL (ms):                         7.94
P95 ITL (ms):                            8.05
P99 ITL (ms):                            8.11
Max ITL (ms):                            24.64
==================================================
```

* Medium Concurrency:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  43.30
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    39975
Request throughput (req/s):              1.85
Input token throughput (tok/s):          916.16
Output token throughput (tok/s):         942.42
Peak output token throughput (tok/s):    1264.00
Peak concurrent requests:                21
Total token throughput (tok/s):          1858.57
Concurrency:                             13.90
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7521.95
Median E2E Latency (ms):                 8246.89
P90 E2E Latency (ms):                    12370.93
P99 E2E Latency (ms):                    15023.96
---------------Time to First Token----------------
Mean TTFT (ms):                          137.27
Median TTFT (ms):                        109.59
P99 TTFT (ms):                           208.78
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.69
Median TPOT (ms):                        14.87
P99 TPOT (ms):                           17.63
---------------Inter-Token Latency----------------
Mean ITL (ms):                           14.51
Median ITL (ms):                         12.75
P95 ITL (ms):                            13.33
P99 ITL (ms):                            92.85
Max ITL (ms):                            113.70
==================================================
```

* High Concurrency:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  73.93
Total input tokens:                      158939
Total input text tokens:                 158939
Total generated tokens:                  170134
Total generated tokens (retokenized):    168841
Request throughput (req/s):              4.33
Input token throughput (tok/s):          2149.98
Output token throughput (tok/s):         2301.42
Peak output token throughput (tok/s):    3497.00
Peak concurrent requests:                71
Total token throughput (tok/s):          4451.40
Concurrency:                             58.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   13463.58
Median E2E Latency (ms):                 13498.74
P90 E2E Latency (ms):                    22957.10
P99 E2E Latency (ms):                    26656.95
---------------Time to First Token----------------
Mean TTFT (ms):                          239.00
Median TTFT (ms):                        113.42
P99 TTFT (ms):                           713.87
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.13
Median TPOT (ms):                        26.02
P99 TPOT (ms):                           30.90
---------------Inter-Token Latency----------------
Mean ITL (ms):                           24.92
Median ITL (ms):                         16.68
P95 ITL (ms):                            93.33
P99 ITL (ms):                            119.26
Max ITL (ms):                            548.82
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

##### AMD (MI300X/MI325X/MI355X)

* **Results**:

  * Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
    ```
    Accuracy: 0.965
    Invalid: 0.000
    Latency: 23.084 s
    Output throughput: 1148.425 token/s
    ```

##### NVIDIA (B200/GB200)

For deployment commands, see [Section 3.1](#3-1-configuration).

* Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
  ```
  Accuracy: 0.965
  Invalid: 0.000
  Latency: 14.870 s
  Output throughput: 1777.726 token/s
  ```

* nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP (NVFP4)
  ```
  Accuracy: 0.960
  Invalid: 0.000
  Latency: 13.948 s
  Output throughput: 1988.548 token/s
  ```


# Qwen3-Coder-Next
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Coder-Next



## 1. Model Introduction

[Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) is a cost-efficient code-focused language model from the Qwen team (Alibaba). With 80B total parameters but only 3B activated parameters, it achieves performance comparable to models with 10–20x more active parameters through its innovative hybrid architecture.

**Key Features:**

* **Hybrid Architecture**: Uses a 48-layer hybrid layout combining Gated DeltaNet and Gated Attention with Mixture-of-Experts (512 total experts, 10 activated, 1 shared), enabling exceptional efficiency.
* **Tool Calling Support**: Advanced agentic capabilities with native support for function calling and tool use via the `qwen3_coder` parser.
* **Extended Context Length**: Supports up to 256K tokens for processing large codebases and long documents.
* **Cost-Efficient Inference**: Only 3B parameters activated per token, making it ideal for local development and cost-effective deployment at scale.
* **IDE Integration**: Compatible with Claude Code, Qwen Code, Cline, and other IDE platforms.

For more details, please refer to the [Qwen3-Coder-Next model card](https://huggingface.co/Qwen/Qwen3-Coder-Next).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

**Note:** Qwen3-Coder-Next requires SGLang v0.5.8 or later.

## 3. Model Deployment

This section provides a progressive guide from quick deployment to performance optimization, suitable for users at different levels.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and deployment options.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`.
* **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag.
* **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed.
* **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
  * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
  * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
* **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

**Deployment Command:**

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Coder-Next \
  --tp 2 \
  --tool-call-parser qwen3_coder \
  --host 0.0.0.0 \
  --port 30000
```

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Code Generation Example

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-Coder-Next",
    messages=[
        {"role": "user", "content": "Write a Python function that implements binary search on a sorted list. Include type hints."}
    ],
    max_tokens=2048
)

print(response.choices[0].message.content)
```

**Example Output:**

````text Output theme={null}
Here's a Python function implementing binary search on a sorted list, with comprehensive type hints:

```python
from typing import Sequence, TypeVar, Optional

T = TypeVar('T')

def binary_search(sorted_list: Sequence[T], target: T) -> Optional[int]:
    """
    Perform binary search on a sorted list to find the index of a target element.

    Args:
        sorted_list: A sequence (e.g., list, tuple) sorted in ascending order.
        target: The element to search for in the list.

    Returns:
        The index of the target element if found, or None if not found.

    Time Complexity: O(log n)
    Space Complexity: O(1)

    Note:
        The function assumes the list is sorted in ascending order.
        If the list contains duplicate elements, it returns the index of one of them.
    """
    left = 0
    right = len(sorted_list) - 1

    while left <= right:
        mid = (left + right) // 2
        mid_val = sorted_list[mid]

        if mid_val == target:
            return mid
        elif mid_val < target:
            left = mid + 1
        else:
            right = mid - 1

    return None
```

### Example usage:

```python
# Example 1: Finding an existing element
numbers = [1, 3, 5, 7, 9, 11]
print(binary_search(numbers, 7))  # Output: 3

# Example 2: Element not in the list
print(binary_search(numbers, 4))  # Output: None

# Example 3: Empty list
print(binary_search([], 5))  # Output: None

# Example 4: Single element
print(binary_search([1], 1))  # Output: 0
print(binary_search([1], 2))  # Output: None
```

### Key features:
- Uses `TypeVar` to support generic types (as long as comparison operations are defined)
- Returns `Optional[int]` to indicate either the index or no match found
- Uses `Sequence[T]` to accept any sequence type (list, tuple, etc.)
- Includes comprehensive docstring with time/space complexity
- Implements standard iterative binary search for O(1) space complexity
````

#### 4.2.2 Streaming Example

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-Coder-Next",
    messages=[
        {"role": "user", "content": "Explain the difference between a stack and a queue in 3 sentences."}
    ],
    max_tokens=512,
    stream=True
)

for chunk in response:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
print()
```

**Example Output:**

```text Output theme={null}
A **stack** follows the **Last In, First Out (LIFO)** principle, meaning the last element added is the first one removed—operations like `push` (add) and `pop` (remove) occur at the same end, called the *top*. In contrast, a **queue** follows the **First In, First Out (FIFO)** principle, where elements are added at the *back* (enqueue) and removed from the *front* (dequeue), preserving the order of insertion. This structural difference makes stacks ideal for tasks like function call management and expression evaluation, while queues suit scheduling, buffering, and breadth-first traversal.
```

#### 4.2.3 Tool Calling Example

Qwen3-Coder-Next supports tool calling capabilities. Make sure `--tool-call-parser qwen3_coder` is included in the deployment command above.

**Python Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "execute_code",
            "description": "Execute Python code and return the result",
            "parameters": {
                "type": "object",
                "properties": {
                    "code": {
                        "type": "string",
                        "description": "The Python code to execute"
                    }
                },
                "required": ["code"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="Qwen/Qwen3-Coder-Next",
    messages=[
        {"role": "user", "content": "Calculate the factorial of 10 using Python"}
    ],
    tools=tools
)

# Check if the model wants to call a tool
if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]
    print(f"Tool: {tool_call.function.name}")
    print(f"Arguments: {tool_call.function.arguments}")
else:
    print(response.choices[0].message.content)
```

**Example Output:**

```text Output theme={null}
Tool: execute_code
Arguments: {"code": "import math\nmath.factorial(10)"}
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (2x)
* Model: Qwen/Qwen3-Coder-Next
* Tensor Parallelism: 2
* sglang version: 0.5.8+

#### 5.1.1 Standard Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Coder-Next \
  --tp 2 \
  --host 0.0.0.0 \
  --port 30000
```

##### 5.1.1.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  27.86
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4218
Request throughput (req/s):              0.36
Input token throughput (tok/s):          219.00
Output token throughput (tok/s):         151.48
Peak output token throughput (tok/s):    166.00
Peak concurrent requests:                2
Total token throughput (tok/s):          370.48
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2784.14
Median E2E Latency (ms):                 2258.08
P90 E2E Latency (ms):                    5044.43
P99 E2E Latency (ms):                    6130.52
---------------Time to First Token----------------
Mean TTFT (ms):                          161.68
Median TTFT (ms):                        168.09
P99 TTFT (ms):                           183.26
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.19
Median TPOT (ms):                        6.23
P99 TPOT (ms):                           6.32
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.23
Median ITL (ms):                         6.23
P95 ITL (ms):                            6.51
P99 ITL (ms):                            6.64
Max ITL (ms):                            13.45
==================================================
```

##### 5.1.1.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  39.06
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  40805
Total generated tokens (retokenized):    40789
Request throughput (req/s):              2.05
Input token throughput (tok/s):          1015.62
Output token throughput (tok/s):         1044.73
Peak output token throughput (tok/s):    1664.00
Peak concurrent requests:                21
Total token throughput (tok/s):          2060.34
Concurrency:                             14.16
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6910.97
Median E2E Latency (ms):                 7248.27
P90 E2E Latency (ms):                    11612.63
P99 E2E Latency (ms):                    13933.91
---------------Time to First Token----------------
Mean TTFT (ms):                          183.48
Median TTFT (ms):                        156.50
P99 TTFT (ms):                           311.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          13.61
Median TPOT (ms):                        13.59
P99 TPOT (ms):                           21.11
---------------Inter-Token Latency----------------
Mean ITL (ms):                           13.22
Median ITL (ms):                         9.76
P95 ITL (ms):                            10.43
P99 ITL (ms):                            158.04
Max ITL (ms):                            394.39
==================================================
```

##### 5.1.1.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  102.81
Total input tokens:                      249831
Total input text tokens:                 249831
Total generated tokens:                  252662
Total generated tokens (retokenized):    252536
Request throughput (req/s):              4.86
Input token throughput (tok/s):          2429.99
Output token throughput (tok/s):         2457.53
Peak output token throughput (tok/s):    5299.00
Peak concurrent requests:                109
Total token throughput (tok/s):          4887.52
Concurrency:                             94.28
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   19385.20
Median E2E Latency (ms):                 17584.09
P90 E2E Latency (ms):                    36762.15
P99 E2E Latency (ms):                    42518.35
---------------Time to First Token----------------
Mean TTFT (ms):                          270.62
Median TTFT (ms):                        159.65
P99 TTFT (ms):                           938.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          38.57
Median TPOT (ms):                        41.78
P99 TPOT (ms):                           53.28
---------------Inter-Token Latency----------------
Mean ITL (ms):                           37.90
Median ITL (ms):                         18.26
P95 ITL (ms):                            167.82
P99 ITL (ms):                            311.45
Max ITL (ms):                            993.20
==================================================
```

#### 5.1.2 Reasoning Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Coder-Next \
  --tp 2 \
  --host 0.0.0.0 \
  --port 30000
```

##### 5.1.2.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  285.02
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  44462
Total generated tokens (retokenized):    44432
Request throughput (req/s):              0.04
Input token throughput (tok/s):          21.41
Output token throughput (tok/s):         156.00
Peak output token throughput (tok/s):    173.00
Peak concurrent requests:                2
Total token throughput (tok/s):          177.40
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   28499.54
Median E2E Latency (ms):                 30424.65
P90 E2E Latency (ms):                    49132.26
P99 E2E Latency (ms):                    51075.28
---------------Time to First Token----------------
Mean TTFT (ms):                          95.51
Median TTFT (ms):                        93.86
P99 TTFT (ms):                           112.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.24
Median TPOT (ms):                        6.30
P99 TPOT (ms):                           6.60
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.39
Median ITL (ms):                         6.34
P95 ITL (ms):                            7.16
P99 ITL (ms):                            7.42
Max ITL (ms):                            12.48
==================================================
```

##### 5.1.2.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  237.77
Total input tokens:                      39668
Total input text tokens:                 39668
Total generated tokens:                  318306
Total generated tokens (retokenized):    315646
Request throughput (req/s):              0.34
Input token throughput (tok/s):          166.83
Output token throughput (tok/s):         1338.72
Peak output token throughput (tok/s):    1727.00
Peak concurrent requests:                19
Total token throughput (tok/s):          1505.55
Concurrency:                             13.88
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   41266.21
Median E2E Latency (ms):                 41010.10
P90 E2E Latency (ms):                    77574.22
P99 E2E Latency (ms):                    82688.04
---------------Time to First Token----------------
Mean TTFT (ms):                          140.73
Median TTFT (ms):                        84.52
P99 TTFT (ms):                           365.86
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.32
Median TPOT (ms):                        10.38
P99 TPOT (ms):                           10.87
---------------Inter-Token Latency----------------
Mean ITL (ms):                           10.34
Median ITL (ms):                         10.19
P95 ITL (ms):                            10.75
P99 ITL (ms):                            11.18
Max ITL (ms):                            206.79
==================================================
```

##### 5.1.2.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 8000 \
  --num-prompts 320 \
  --max-concurrency 64
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  384.82
Total input tokens:                      158939
Total input text tokens:                 158939
Total generated tokens:                  1301025
Total generated tokens (retokenized):    1299908
Request throughput (req/s):              0.83
Input token throughput (tok/s):          413.02
Output token throughput (tok/s):         3380.83
Peak output token throughput (tok/s):    4317.00
Peak concurrent requests:                69
Total token throughput (tok/s):          3793.85
Concurrency:                             56.42
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   67847.54
Median E2E Latency (ms):                 70724.38
P90 E2E Latency (ms):                    120888.83
P99 E2E Latency (ms):                    133234.48
---------------Time to First Token----------------
Mean TTFT (ms):                          212.24
Median TTFT (ms):                        115.96
P99 TTFT (ms):                           652.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          16.76
Median TPOT (ms):                        16.99
P99 TPOT (ms):                           18.18
---------------Inter-Token Latency----------------
Mean ITL (ms):                           16.64
Median ITL (ms):                         15.83
P95 ITL (ms):                            31.64
P99 ITL (ms):                            90.85
Max ITL (ms):                            576.60
==================================================
```

#### 5.1.3 Summarization Scenario Benchmark

##### 5.1.3.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  29.42
Total input tokens:                      41941
Total input text tokens:                 41941
Total generated tokens:                  4220
Total generated tokens (retokenized):    4220
Request throughput (req/s):              0.34
Input token throughput (tok/s):          1425.35
Output token throughput (tok/s):         143.42
Peak output token throughput (tok/s):    169.00
Peak concurrent requests:                3
Total token throughput (tok/s):          1568.77
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2941.19
Median E2E Latency (ms):                 2411.84
P90 E2E Latency (ms):                    5661.26
P99 E2E Latency (ms):                    6497.45
---------------Time to First Token----------------
Mean TTFT (ms):                          139.46
Median TTFT (ms):                        160.33
P99 TTFT (ms):                           184.30
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.56
Median TPOT (ms):                        6.65
P99 TPOT (ms):                           7.29
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.65
Median ITL (ms):                         6.68
P95 ITL (ms):                            7.39
P99 ITL (ms):                            7.51
Max ITL (ms):                            16.34
==================================================
```

##### 5.1.3.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  41.62
Total input tokens:                      300020
Total input text tokens:                 300020
Total generated tokens:                  41669
Total generated tokens (retokenized):    41664
Request throughput (req/s):              1.92
Input token throughput (tok/s):          7208.67
Output token throughput (tok/s):         1001.19
Peak output token throughput (tok/s):    1536.00
Peak concurrent requests:                21
Total token throughput (tok/s):          8209.86
Concurrency:                             14.27
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7421.29
Median E2E Latency (ms):                 7985.77
P90 E2E Latency (ms):                    12122.09
P99 E2E Latency (ms):                    14595.05
---------------Time to First Token----------------
Mean TTFT (ms):                          248.49
Median TTFT (ms):                        179.25
P99 TTFT (ms):                           915.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          14.13
Median TPOT (ms):                        14.28
P99 TPOT (ms):                           24.02
---------------Inter-Token Latency----------------
Mean ITL (ms):                           13.80
Median ITL (ms):                         10.46
P95 ITL (ms):                            11.00
P99 ITL (ms):                            173.14
Max ITL (ms):                            823.32
==================================================
```

##### 5.1.3.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-Coder-Next \
  --dataset-name random \
  --random-input-len 8000 \
  --random-output-len 1000 \
  --num-prompts 320 \
  --max-concurrency 64
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 64
Successful requests:                     320
Benchmark duration (s):                  85.74
Total input tokens:                      1273893
Total input text tokens:                 1273893
Total generated tokens:                  170000
Total generated tokens (retokenized):    169983
Request throughput (req/s):              3.73
Input token throughput (tok/s):          14858.12
Output token throughput (tok/s):         1982.80
Peak output token throughput (tok/s):    3734.00
Peak concurrent requests:                70
Total token throughput (tok/s):          16840.92
Concurrency:                             59.75
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   16008.12
Median E2E Latency (ms):                 15460.65
P90 E2E Latency (ms):                    27705.81
P99 E2E Latency (ms):                    32874.74
---------------Time to First Token----------------
Mean TTFT (ms):                          476.99
Median TTFT (ms):                        177.50
P99 TTFT (ms):                           3014.39
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.81
Median TPOT (ms):                        31.19
P99 TPOT (ms):                           45.53
---------------Inter-Token Latency----------------
Mean ITL (ms):                           29.29
Median ITL (ms):                         15.75
P95 ITL (ms):                            173.94
P99 ITL (ms):                            202.00
Max ITL (ms):                            2783.23
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* **Test Results:**

```text Output theme={null}
Accuracy: 0.965
Invalid: 0.000
Latency: 26.407 s
Output throughput: 929.132 token/s
```

#### 5.2.2 MMLU Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* **Test Results:**

```text Output theme={null}
subject: abstract_algebra, #q:100, acc: 0.780
subject: anatomy, #q:135, acc: 0.807
subject: astronomy, #q:152, acc: 0.921
subject: business_ethics, #q:100, acc: 0.820
subject: clinical_knowledge, #q:265, acc: 0.860
subject: college_biology, #q:144, acc: 0.944
subject: college_chemistry, #q:100, acc: 0.590
subject: college_computer_science, #q:100, acc: 0.820
subject: college_mathematics, #q:100, acc: 0.800
subject: college_medicine, #q:173, acc: 0.803
subject: college_physics, #q:102, acc: 0.775
subject: computer_security, #q:100, acc: 0.880
subject: conceptual_physics, #q:235, acc: 0.936
subject: econometrics, #q:114, acc: 0.807
subject: electrical_engineering, #q:145, acc: 0.834
subject: elementary_mathematics, #q:378, acc: 0.854
subject: formal_logic, #q:126, acc: 0.802
subject: global_facts, #q:100, acc: 0.610
subject: high_school_biology, #q:310, acc: 0.971
subject: high_school_chemistry, #q:203, acc: 0.803
subject: high_school_computer_science, #q:100, acc: 0.920
subject: high_school_european_history, #q:165, acc: 0.891
subject: high_school_geography, #q:198, acc: 0.929
subject: high_school_government_and_politics, #q:193, acc: 0.969
subject: high_school_macroeconomics, #q:390, acc: 0.903
subject: high_school_mathematics, #q:270, acc: 0.689
subject: high_school_microeconomics, #q:238, acc: 0.962
subject: high_school_physics, #q:151, acc: 0.854
subject: high_school_psychology, #q:545, acc: 0.947
subject: high_school_statistics, #q:216, acc: 0.815
subject: high_school_us_history, #q:204, acc: 0.907
subject: high_school_world_history, #q:237, acc: 0.937
subject: human_aging, #q:223, acc: 0.821
subject: human_sexuality, #q:131, acc: 0.840
subject: international_law, #q:121, acc: 0.934
subject: jurisprudence, #q:108, acc: 0.870
subject: logical_fallacies, #q:163, acc: 0.847
subject: machine_learning, #q:112, acc: 0.812
subject: management, #q:103, acc: 0.922
subject: marketing, #q:234, acc: 0.923
subject: medical_genetics, #q:100, acc: 0.970
subject: miscellaneous, #q:783, acc: 0.941
subject: moral_disputes, #q:346, acc: 0.850
subject: moral_scenarios, #q:895, acc: 0.726
subject: nutrition, #q:306, acc: 0.915
subject: philosophy, #q:311, acc: 0.859
subject: prehistory, #q:324, acc: 0.889
subject: professional_accounting, #q:282, acc: 0.723
subject: professional_law, #q:1534, acc: 0.648
subject: professional_medicine, #q:272, acc: 0.923
subject: professional_psychology, #q:612, acc: 0.845
subject: public_relations, #q:110, acc: 0.782
subject: security_studies, #q:245, acc: 0.796
subject: sociology, #q:201, acc: 0.925
subject: us_foreign_policy, #q:100, acc: 0.950
subject: virology, #q:166, acc: 0.572
subject: world_religions, #q:171, acc: 0.883
Total latency: 208.985
Average accuracy: 0.834
```


# Qwen3-Next
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-Next



## 1. Model Introduction

[Qwen3-Next](https://huggingface.co/collections/Qwen/qwen3-next) is an advanced large language model architecture developed by Alibaba's Qwen team, designed to enhance efficiency and performance in handling extensive contexts and large-scale parameters. It features advanced capabilities in reasoning, function calling, and multilingual understanding.

Qwen3-Next introduces several groundbreaking innovations:

* **Hybrid Attention Mechanism**: Replaces standard attention with a combination of **Gated DeltaNet** (linear attention) and **Full Attention**, enabling efficient processing of context lengths up to 262,144 tokens. This hybrid approach makes it ideal for analyzing lengthy documents such as entire books or contracts.

* **Highly Sparse Mixture-of-Experts (MoE)**: Features an 80-billion parameter architecture where only 3 billion parameters are active during inference. This design reduces computational costs by up to 90% while maintaining high performance, drastically reducing FLOPs per token without compromising model capacity.

* **Multi-Token Prediction (MTP)**: Enables generation of multiple tokens per inference step, significantly reducing latency and enhancing user experience in real-time applications. This innovation boosts both pretraining performance and inference speed.

* **Multilingual Support**: Natively supports 119 languages, facilitating seamless cross-lingual tasks and making it versatile for global applications.

* **Enterprise-Ready Deployment**: Released under the Apache 2.0 license, offering flexible deployment options including on-premises, virtual private cloud (VPC), and private cloud environments, ensuring security and compliance for enterprise use.

* **Advanced Reasoning & Stability**: Demonstrates clear improvement in reasoning performance with support for tool use during inference. Includes stability optimizations such as **zero-centered** and **weight-decayed layernorm** for robust pre-training and post-training.

For more details, please refer to the [official Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd\&from=research.latest-advancements-list).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The Qwen3-Next series comes in only one size but offers different thinking modes. Recommended starting configurations vary depending on hardware.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* `--max-mamba-cache-size`: Adjust `--max-mamba-cache-size` to increase mamba cache space and max running requests capability. It will decrease KV cache space as a trade-off. You can adjust it according to workload.

* `--mamba-ssm-dtype`: `bfloat16` or `float32`, use `bfloat16` to save mamba cache size and `float32` to get more accurate results. The default setting is `float32`.

* `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`.

* **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
  * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
  * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).

* **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

1. **Streaming with Thinking Process:**

   Qwen3-Next-80B-A3B-Thinking only supports thinking mode. Enable the reasoning parser during deployment to separate the thinking and the content sections.

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Thinking \
  --reasoning-parser qwen3 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
Okay, let's see. I need to find 15% of 240. Hmm, percentages. Right, "percent" means per hundred, so 15% is 15 per 100, or 15/100. To find a percentage of a number, I think you multiply the number by the percentage as a decimal. So first, maybe convert 15% to a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Then, multiply that by 240. Let me check that. So 0.15 times 240. Let's calculate that. Maybe break it down. 10% of 240 is 24, because 10% is just moving the decimal one place left, so 240 becomes 24. Then 5% would be half of 10%, so half of 24 is 12. So 10% + 5% = 15%, so 24 + 12 = 36. Oh, that's another way to do it. Let me verify with the multiplication. 0.15 * 240. Let's do 240 * 0.1 = 24, 240 * 0.05 = 12, so 24 + 12 = 36. Yep, that works. Alternatively, 240 * 15 = 3600, then divide by 100, which is 36. Because 15% of 240 is (15/100)*240 = (15*240)/100. 15*240: 10*240=2400, 5*240=1200, so 2400+1200=3600. Then 3600/100=36. So that's 36. So the answer should be 36. Let me make sure. 15% of 240. If I take 240 and multiply by 0.15, 240*0.15. Let's compute 240*0.1=24, 240*0.05=12, so 24+12=36. Yep, that's right. So 15% of 240 is 36.

=============== Content =================

To find **15% of 240**, follow these steps:

---

### **Step 1: Understand what "percent" means**
- "Percent" means **per hundred**, so **15% = 15/100 = 0.15** in decimal form.

---

### **Step 2: Multiply the number by the decimal**
- To find 15% of 240, multiply:
  $$
  240 \times 0.15
  $$

---

### **Step 3: Break it down for clarity (optional but helpful)**
- **10% of 240** = $ 240 \times 0.1 = 24 $
- **5% of 240** = $ 240 \times 0.05 = 12 $
- Add them together:
  $$
  24 + 12 = 36
  $$

---

### **Step 4: Confirm with direct multiplication**
- $ 240 \times 0.15 = 36 $

---

### ✅ Final Answer:
$$
\boxed{36}
$$
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

2. **Turn off Thinking:**

   Qwen3-Next-80B-A3B-Instruct only supports instruct (non-thinking) mode.

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Turn off thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Instruct",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
To find **15% of 240**, follow these steps:

---

### **Step 1: Understand what percentage means**
"Percent" means "per hundred," so **15%** is the same as **15 per 100**, or the fraction:

$$
\frac{15}{100}
$$

---

### **Step 2: Multiply the fraction by the number**
To find 15% of 240, multiply:

$$
\frac{15}{100} \times 240
$$

---

### **Step 3: Simplify the multiplication**
You can simplify this in a couple of ways.

#### **Option A: Multiply first, then divide**
$$
15 \times 240 = 3600
$$
Then divide by 100:
$$
\frac{3600}{100} = 36
$$

#### **Option B: Simplify the fraction first**
$$
\frac{15}{100} = \frac{3}{20} \quad \text{(divided numerator and denominator by 5)}
$$
Now multiply:
$$
\frac{3}{20} \times 240 = \frac{3 \times 240}{20} = \frac{720}{20} = 36
$$

---

### **Step 4: Final Answer**
$$
\boxed{36}
$$

So, **15% of 240 is 36**.
```

#### 4.2.2 Tool Calling

Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen/Qwen3-Next-80B-A3B-Thinking both support tool calling capabilities. Enable the tool call parser:

**Python Example (without Thinking Process):**

Start sglang server:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tool-call-parser qwen \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Instruct",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>
```

**Python Example (with Thinking Process):**

Start sglang server:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Thinking \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --tp 8 \
  --host 0.0.0.0 \
  --port 8000
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"🔧 Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
Okay, the user is asking for the weather in Beijing. Let me check the available tools. There's a get_weather function that requires location and optionally unit. The location is needed, so I need to provide Beijing as the location. The unit is optional, but the user didn't specify Celsius or Fahrenheit. Since the default might be Celsius, but maybe I should check if the parameters require unit. Wait, the required field is only location, so unit is optional. So I can just call get_weather with location "Beijing" and not include the unit. Let me confirm the parameters. The parameters for get_weather have location as required, and unit is an enum with celsius or fahrenheit, but not required. So the correct call is to send location as Beijing, and omit unit. So the tool call should be {"name": "get_weather", "arguments": {"location": "Beijing"}}.

<tool_call>
{"name": "get_weather", "arguments": {"location": "Beijing"}}
</tool_call>
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="Qwen/Qwen3-Next-80B-A3B-Thinking",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

#### 4.2.3 Processing Ultra-Long Texts

Qwen3-Next natively supports context lengths of up to 262,144 tokens. For conversations where the total length (including both input and output) significantly exceeds this limit, we recommend using RoPE scaling techniques to handle long texts effectively. We have validated the model's performance on context lengths of up to 1 million tokens using the YaRN method.

**Qwen3-Next-80B-A3B-Instruct**

```shell Command theme={null}
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Instruct    --tp 8   --host 0.0.0.0   --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000

```

**Qwen3-Next-80B-A3B-Thinking**

```shell Command theme={null}
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python -m sglang.launch_server --model Qwen/Qwen3-Next-80B-A3B-Thinking   --reasoning-parser qwen3     --tp 8   --host 0.0.0.0   --port 8000 --json-model-override-args '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144}}' --context-length 1010000

```

#### 4.2.4 Multi-Token Prediction (NEXTN Speculative Decoding)

Qwen3-Next ships built-in Multi-Token Prediction (MTP) layers and supports [EAGLE-style speculative decoding](../../../docs/advanced_features/speculative_decoding#eagle-decoding) through the `NEXTN` algorithm. The MTP weights are bundled in the main checkpoint, so no separate draft model is required.

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --tp 4
```

Tune `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens` for your workload with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). See [PR #10233](https://github.com/sgl-project/sglang/pull/10233) for implementation details.

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8x)
* Tensor Parallelism: 8
* Model: Qwen/Qwen3-Next-80B-A3B-Instruct
* sglang version: 0.5.6

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.

#### 5.1.1 Latency-Sensitive Benchmark

* Server Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tp 8
```

* Test Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 100 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     100
Benchmark duration (s):                  146.52
Total input tokens:                      33839
Total input text tokens:                 33839
Total input vision tokens:               0
Total generated tokens:                  21640
Total generated tokens (retokenized):    21619
Request throughput (req/s):              0.68
Input token throughput (tok/s):          230.95
Output token throughput (tok/s):         147.70
Peak output token throughput (tok/s):    164.00
Peak concurrent requests:                6
Total token throughput (tok/s):          378.65
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1464.81
Median E2E Latency (ms):                 1077.48
---------------Time to First Token----------------
Mean TTFT (ms):                          127.88
Median TTFT (ms):                        132.88
P99 TTFT (ms):                           212.85
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          6.19
Median TPOT (ms):                        6.17
P99 TPOT (ms):                           6.64
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.21
Median ITL (ms):                         6.16
P95 ITL (ms):                            6.51
P99 ITL (ms):                            6.71
Max ITL (ms):                            10.07
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Server Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-Next-80B-A3B-Instruct \
  --tp 8 \
```

* Test Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --num-prompt 1000 \
  --max-concurrency 100
```

**Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  100.32
Total input tokens:                      302118
Total input text tokens:                 302118
Total input vision tokens:               0
Total generated tokens:                  195775
Total generated tokens (retokenized):    195016
Request throughput (req/s):              9.97
Input token throughput (tok/s):          3011.69
Output token throughput (tok/s):         1951.60
Peak output token throughput (tok/s):    5909.00
Peak concurrent requests:                120
Total token throughput (tok/s):          4963.29
Concurrency:                             93.05
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   9333.98
Median E2E Latency (ms):                 6054.12
---------------Time to First Token----------------
Mean TTFT (ms):                          161.77
Median TTFT (ms):                        137.94
P99 TTFT (ms):                           503.29
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          50.87
Median TPOT (ms):                        50.28
P99 TPOT (ms):                           122.87
---------------Inter-Token Latency----------------
Mean ITL (ms):                           47.11
Median ITL (ms):                         13.84
P95 ITL (ms):                            195.33
P99 ITL (ms):                            289.56
Max ITL (ms):                            486.38
==================================================
```

### 5.2 Accuracy Benchmark

### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:8000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Results**:

  * Qwen3-Next-80B-A3B-Instruct

    ```
    Accuracy: 0.960
    Invalid: 0.000
    Latency: 12.673 s
    Output throughput: 2538.255 token/s
    ```

  * Qwen3-Next-80B-A3B-Thinking
    ```
    Accuracy: 0.935
    Invalid: 0.000
    Latency: 9.912 s
    Output throughput: 3288.737 token/s
    ```

### 5.2.2 MMLU Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

The command evaluates all 57 subjects. Historical results below use the removed 10-subject harness and are not measurements of the sgl-eval command above.

* **Results**:

  * Qwen3-Next-80B-A3B-Instruct

    ```
    subject: abstract_algebra, #q:100, acc: 0.800
    subject: anatomy, #q:135, acc: 0.807
    subject: astronomy, #q:152, acc: 0.947
    subject: business_ethics, #q:100, acc: 0.810
    subject: clinical_knowledge, #q:265, acc: 0.894
    subject: college_biology, #q:144, acc: 0.972
    subject: college_chemistry, #q:100, acc: 0.680
    subject: college_computer_science, #q:100, acc: 0.860
    subject: college_mathematics, #q:100, acc: 0.780
    subject: college_medicine, #q:173, acc: 0.861
    Total latency: 10.098
    Average accuracy: 0.856
    ```

  * Qwen3-Next-80B-A3B-Thinking
    ```
    subject: abstract_algebra, #q:100, acc: 0.780
    subject: anatomy, #q:135, acc: 0.815
    subject: astronomy, #q:152, acc: 0.941
    subject: business_ethics, #q:100, acc: 0.870
    subject: clinical_knowledge, #q:265, acc: 0.894
    subject: college_biology, #q:144, acc: 0.965
    subject: college_chemistry, #q:100, acc: 0.670
    subject: college_computer_science, #q:100, acc: 0.840
    subject: college_mathematics, #q:100, acc: 0.770
    subject: college_medicine, #q:173, acc: 0.861
    Total latency: 10.236
    Average accuracy: 0.855
    ```


# Qwen3-VL
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3-VL



## 1. Model Introduction

[Qwen3-VL series](https://github.com/QwenLM/Qwen3-VL) are the most powerful vision-language models in the Qwen series to date, featuring advanced capabilities in multi-modal understanding, reasoning, and agentic applications.

This generation delivers comprehensive upgrades across the board:

* **Superior text understanding & generation**: Qwen3-VL-235B-A22B-Instruct was ranked as the [#1 open model for text on lmarena.ai](https://x.com/arena/status/1973151703563460942)
* **Deeper visual perception & reasoning**: Enhanced image and video understanding capabilities.
* **Extended context length**: Supports up to 262K tokens for processing long documents and videos.
* **Enhanced spatial and video dynamics comprehension**: Better understanding of spatial relationships and temporal dynamics.
* **Stronger agent interaction capabilities**: Improved tool use and search-based agent performance.
* **Flexible deployment options**: Available in Dense and MoE architectures that scale from edge to cloud, with Instruct and reasoning-enhanced Thinking editions.

For more details, please refer to the [official Qwen3-VL GitHub Repository](https://github.com/QwenLM/Qwen3-VL).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The Qwen3-VL series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA and AMD GPUs, as well as Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Multimodal attention backend** : Usually, `--mm-attention-backend` is default to `fa3` on H100/H200/A100 for better performance, but it is default to `triton_attn` on B200 for compatibility.
* **TTFT Optimization** : Set `SGLANG_USE_CUDA_IPC_TRANSPORT=1` to use CUDA IPC for transferring multimodal features, which significantly improves TTFT. This consumes additional memory and may require adjusting `--mem-fraction-static` and/or `--max-running-requests`. (additional memory is proportional to image size \* number of images in current running requests.)
* **Memory Management** : Set lower `--context-length` to conserve memory. A value of `128000` is sufficient for most scenarios, down from the default 262K.
* **Expert Parallelism** : SGLang supports Expert Parallelism (EP) via `--ep`, allowing experts in MoE models to be deployed on separate GPUs for better throughput. One thing to note is that, for quantized models, you need to set `--ep` to a value that satisfies the requirement: `(moe_intermediate_size / moe_tp_size) % weight_block_size_n == 0, where moe_tp_size is equal to tp_size divided by ep_size.` Note that EP may perform worse in low concurrency scenarios due to additional communication overhead. Check out [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism) for more details.
* **Kernel Tuning** : For MoE Triton kernel tuning on your specific hardware, refer to [fused\_moe\_triton](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton).

**Hardware-specific notes:**

* **H100 (FP8):** Use the `Qwen/Qwen3-VL-235B-A22B-Instruct-FP8` checkpoint for best memory efficiency.
* **A100 / H100 (BF16):** Use standard multimodal parameters to manage throughput and GPU memory usage.
* **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing.

**Additional multimodal server parameters:**

* `--mm-feature-transport=cuda_ipc`: Keep multimodal feature tensors in a bounded GPU pool shared over CUDA IPC to avoid device-to-host memory copies, improving performance for high-frequency inference.

**Example with full multimodal optimizations:**

```bash Command theme={null}
SGLANG_USE_CUDA_IPC_TRANSPORT=1 \
SGLANG_VLM_CACHE_SIZE_MB=0 \
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-235B-A22B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tp-size 8 \
  --enable-cache-report \
  --log-level info \
  --max-running-requests 64 \
  --mem-fraction-static 0.65 \
  --chunked-prefill-size 8192 \
  --attention-backend fa3 \
  --mm-attention-backend fa3 \
  --enable-metrics
```

* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Multi-Modal Inputs

Qwen3-VL supports both image and video inputs. Here's a basic example with image input:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 3.37s
Generated text: Auntie Anne's

CINNAMON SUGAR
1 x 17,000                    17,000

SUB TOTAL                    17,000

GRAND TOTAL                  17,000

CASH IDR                     20,000

CHANGE DUE                  3,000
```

**Multi-Image Input Example:**

Qwen3-VL can process multiple images in a single request for comparison or analysis:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
                }
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
                }
            },
            {
                "type": "text",
                "text": "Compare these two images and describe the differences in 100 words or less. Focus on the key visual elements, colors, textures, and any notable contrasts between the two scenes. Be specific about what you see in each image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 10.18s
Generated text: The two images present starkly different portrayals of Hong Kong’s iconic red taxis, contrasting a dynamic street-level moment with a static, large-scale gathering.

The first image is a close-up, eye-level shot capturing a single red Toyota Crown taxi (license plate RX 5004) in motion or paused at an urban intersection. Its glossy red paint gleams under daylight, reflecting the vibrant, cluttered backdrop of a Hong Kong street — neon signs, glass-fronted shops displaying sunglasses, and Chinese characters. The taxi’s chrome grille, clear headlights, and black trim provide visual contrast. A green “4 SEATS” sticker and a “的士 TAXI” sign on the side reinforce its identity. The composition is intimate, focusing on the vehicle’s details — the texture of its paint, the slight reflections on the windows, and the crispness of its license plate. Other red taxis flank it, suggesting a bustling city rhythm, but the central taxi dominates the frame, conveying movement and immediacy.

In contrast, the second image is an elevated, wide-angle shot of dozens of red taxis — along with a few green ones — parked in neat, grid-like rows on what appears to be a highway or staging area. The scene is static, almost ceremonial. Many taxis have their hoods open, suggesting maintenance, inspection, or protest. People are scattered among the vehicles, some inspecting engines, others conversing — adding a human, documentary element. The dominant color remains red, but the repetition creates a visual pattern rather than individual focus. The green taxis offer a subtle color contrast, hinting at different service zones (green for New Territories, red for urban areas). The setting is more utilitarian — concrete barriers, metal railings, and sparse vegetation — with an overpass looming in the background. The texture here is less about polished paint and more about the collective mass of vehicles, the asphalt, and the functional layout.

Key contrasts emerge: the first image is kinetic and personal, emphasizing the taxi as a working vehicle in the city’s daily flow; the second is static and collective, portraying the taxis as a fleet, possibly for logistical or political purposes. The lighting in both is bright daylight, but the first has richer color saturation and depth due to its proximity and urban backdrop, while the second feels flatter, more documentary in tone. The first image invites you into the city’s pulse; the second invites you to observe a system — organized, perhaps even paused — from a distance.

In essence, the first image celebrates the individual taxi in its natural habitat; the second reveals the scale and structure behind the fleet, transforming the familiar red icon into a symbol of coordination, maintenance, or collective action. Both are quintessentially Hong Kong, yet they offer vastly different narratives — one of motion and commerce, the other of assembly and purpose.
```

**Video Input Example:**

Qwen3-VL supports video understanding by processing video URLs:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video_url",
                "video_url": {
                    "url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"
                }
            },
            {
                "type": "text",
                "text": "Describe what happens in this video."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Instruct",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Note:**

* For video processing, ensure you have sufficient context length configured (up to 262K tokens)
* Video processing may require more memory; adjust `--mem-fraction-static` accordingly
* You can also provide local file paths using `file://` protocol

**Example Output:**

```text Output theme={null}
Response costs: 3.89s
Generated text: A person wearing blue gloves is using a microscope. They are adjusting the focus knob with one hand while holding a pipette with the other, suggesting they are preparing or examining a sample on the slide beneath the objective lens. The microscope's 40x objective lens is positioned over the slide, indicating a high-magnification observation. The person carefully manipulates the slide and the microscope controls, likely to achieve a clear view of the specimen.
```

#### 4.2.2 Reasoning Parser

Qwen3-VL-Thinking supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Thinking \
  --reasoning-parser qwen3 \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
To solve this problem, I need to calculate 15% of 240.
Step 1: Convert 15% to decimal: 15% = 0.15
Step 2: Multiply 240 by 0.15
Step 3: 240 × 0.15 = 36
=============== Content =================

The answer is 36. To find 15% of 240, we multiply 240 by 0.15, which equals 36.
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.3 Tool Calling

Qwen3-VL supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Thinking \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"🔧 Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I need to use the get_weather function to retrieve this information.
I should call the function with location="Beijing".
=============== Content =================

🔧 Tool Call: get_weather
   Arguments: {"location": "Beijing", "unit": "celsius"}
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="Qwen/Qwen3-VL-235B-A22B-Thinking",
    messages=messages,
    temperature=0.7
)

print(final_response.choices[0].message.content)
# Output: "The weather in Beijing is currently 22°C and sunny."
```

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8x)
* Model: Qwen3-VL-235B-A22B-Instruct
* Tensor Parallelism: 8
* sglang version: 0.5.6

We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. To simulate real-world usage, you can specify different input and output lengths for each request. For example, each request can have 128 input tokens, two 720p images, and 1024 output tokens.

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  45.97
Total input tokens:                      18348
Total input text tokens:                 708
Total input vision tokens:               17640
Total generated tokens:                  4220
Total generated tokens (retokenized):    3423
Request throughput (req/s):              0.22
Input token throughput (tok/s):          399.17
Output token throughput (tok/s):         91.81
Peak output token throughput (tok/s):    96.00
Peak concurrent requests:                2
Total token throughput (tok/s):          490.98
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   4594.52
Median E2E Latency (ms):                 3725.04
---------------Time to First Token----------------
Mean TTFT (ms):                          193.35
Median TTFT (ms):                        196.32
P99 TTFT (ms):                           222.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.44
Median TPOT (ms):                        10.44
P99 TPOT (ms):                           10.47
---------------Inter-Token Latency----------------
Mean ITL (ms):                           11.78
Median ITL (ms):                         10.48
P95 ITL (ms):                            21.01
P99 ITL (ms):                            31.40
Max ITL (ms):                            31.92
==================================================
```

**Optimized Results (with CUDA IPC Transport):**

For further TTFT optimization, enable CUDA IPC Transport for multimodal features by setting `SGLANG_USE_CUDA_IPC_TRANSPORT=1`. This significantly reduces TTFT by using CUDA IPC for transferring multimodal features.

* Model Deployment Command:

```shell Command theme={null}
SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 100 \
  --max-concurrency 1
```

* **Test Results:**

  With `SGLANG_USE_CUDA_IPC_TRANSPORT=1`, TTFT improves significantly:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     100
Benchmark duration (s):                  566.84
Total input tokens:                      183667
Total input text tokens:                 7267
Total input vision tokens:               176400
Total generated tokens:                  52444
Total generated tokens (retokenized):    28702
Request throughput (req/s):              0.18
Input token throughput (tok/s):          324.02
Output token throughput (tok/s):         92.52
Peak output token throughput (tok/s):    96.00
Peak concurrent requests:                3
Total token throughput (tok/s):          416.54
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   5667.50
Median E2E Latency (ms):                 5830.00
---------------Time to First Token----------------
Mean TTFT (ms):                          191.16
Median TTFT (ms):                        182.58
P99 TTFT (ms):                           244.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.46
Median TPOT (ms):                        10.46
P99 TPOT (ms):                           10.48
---------------Inter-Token Latency----------------
Mean ITL (ms):                           13.91
Median ITL (ms):                         10.56
P95 ITL (ms):                            21.35
P99 ITL (ms):                            31.55
Max ITL (ms):                            42.36
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --tp 8 \
  --host 0.0.0.0 \
  --port 30000
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3-VL-235B-A22B-Instruct \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  584.65
Total input tokens:                      1839015
Total input text tokens:                 75015
Total input vision tokens:               1764000
Total generated tokens:                  510855
Total generated tokens (retokenized):    284284
Request throughput (req/s):              1.71
Input token throughput (tok/s):          3145.50
Output token throughput (tok/s):         873.78
Peak output token throughput (tok/s):    2855.00
Peak concurrent requests:                107
Total token throughput (tok/s):          4019.29
Concurrency:                             98.35
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   57502.05
Median E2E Latency (ms):                 54301.08
---------------Time to First Token----------------
Mean TTFT (ms):                          5802.23
Median TTFT (ms):                        1444.75
P99 TTFT (ms):                           46675.92
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          100.22
Median TPOT (ms):                        105.43
P99 TPOT (ms):                           144.37
---------------Inter-Token Latency----------------
Mean ITL (ms):                           134.20
Median ITL (ms):                         25.57
P95 ITL (ms):                            558.14
P99 ITL (ms):                            1449.01
Max ITL (ms):                            33453.23
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 MMMU Benchmark

You can evaluate the model's accuracy using the MMMU dataset with `lmms_eval`:

* Benchmark Command:

```shell Command theme={null}
uv pip install lmms_eval

python3 -m lmms_eval \
  --model openai_compatible \
  --model_args "model=Qwen/Qwen3-VL-235B-A22B-Instruct,api_key=EMPTY,base_url=http://127.0.0.1:30000/v1/" \
  --tasks mmmu_val \
  --batch_size 128 \
  --log_samples \
  --log_samples_suffix "openai_compatible" \
  --output_path ./logs \
  --gen_kwargs "max_new_tokens=4096"
```

* **Test Results:**

```text Output theme={null}
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
  <colgroup>
    <col style={{width: "12%"}} />
    <col style={{width: "11%"}} />
    <col style={{width: "11%"}} />
    <col style={{width: "11%"}} />
    <col style={{width: "11%"}} />
    <col style={{width: "11%"}} />
    <col style={{width: "11%"}} />
    <col style={{width: "11%"}} />
    <col style={{width: "11%"}} />
  </colgroup>
  <thead>
    <tr style={{borderBottom: "2px solid #d55816"}}>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Tasks</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Filter</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>n-shot</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Metric</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Value</th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}></th>
      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Stderr</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_val</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>none</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>mmmu_acc</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>↑</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0.6567</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>±</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>N/A</td>
    </tr>
  </tbody>
</table>
```


# Qwen3.5
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.5



## 1. Model Introduction

[Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) is the latest flagship model in the Qwen series developed by Alibaba, representing a significant leap forward with unified vision-language foundation, efficient hybrid architecture, and scalable reinforcement learning.

Qwen3.5 features a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated), delivering high-throughput inference with minimal latency. It supports multimodal inputs (text, image, video) and natively handles context lengths of up to 262,144 tokens, extensible to over 1M tokens.

**Architecture details:**

* **Hybrid Attention:** Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer — linear layers provide low-cost long-context processing while periodic full attention ensures high associative recall.
* **MoE routing:** Top-10 active out of 512 routed experts plus a dedicated shared expert for universal features, keeping 17B parameters active from 397B total.
* **Native multimodal:** DeepStack Vision Transformer with Conv3d temporal encoding for image and video understanding without separate visual encoders.

**Key Features:**

* **Unified Vision-Language Foundation**: Early fusion training on multimodal tokens achieves cross-generational parity with Qwen3 and outperforms Qwen3-VL models
* **Efficient Hybrid Architecture**: Gated Delta Networks + sparse MoE (397B total / 17B active) for high-throughput inference
* **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
* **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser
* **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency
* **201 Language Support**: Expanded multilingual coverage across 201 languages and dialects

**Available Models:**

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>BF16 (Full precision)</th>
      <th>FP8 (8-bit Quantized)</th>
      <th>FP4 (4-bit Quantized)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen3.5-397B-A17B</td>
      <td>[Qwen/Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B)</td>
      <td>[Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8)</td>
      <td>NVIDIA NVFP4: [nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)\
      AMD MXFP4: [amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2)</td>
    </tr>

    <tr>
      <td>Qwen3.5-122B-A10B</td>
      <td>[Qwen/Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)</td>
      <td>[Qwen/Qwen3.5-122B-A10B-FP8](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8)</td>
      <td>-</td>
    </tr>

    <tr>
      <td>Qwen3.5-35B-A3B</td>
      <td>[Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)</td>
      <td>[Qwen/Qwen3.5-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)</td>
      <td>-</td>
    </tr>

    <tr>
      <td>Qwen3.5-27B</td>
      <td>[Qwen/Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B)</td>
      <td>[Qwen/Qwen3.5-27B-FP8](https://huggingface.co/Qwen/Qwen3.5-27B-FP8)</td>
      <td>-</td>
    </tr>

    <tr>
      <td>Qwen3.5-9B</td>
      <td>[Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)</td>
      <td>-</td>
      <td>-</td>
    </tr>

    <tr>
      <td>Qwen3.5-4B</td>
      <td>[Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B)</td>
      <td>-</td>
      <td>-</td>
    </tr>

    <tr>
      <td>Qwen3.5-2B</td>
      <td>[Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B)</td>
      <td>-</td>
      <td>-</td>
    </tr>

    <tr>
      <td>Qwen3.5-0.8B</td>
      <td>[Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B)</td>
      <td>-</td>
      <td>-</td>
    </tr>
  </tbody>
</table>

**License:** Apache 2.0

## 2. SGLang Installation

SGLang from the main branch is required for Qwen3.5. You can install from source or use a Docker image:

```bash Command theme={null}
# Install from source
uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'

# Or use Docker (NVIDIA GPUs)
docker pull lmsysorg/sglang:latest

# Or use Docker (AMD MI300X/MI325X)
docker pull lmsysorg/sglang-rocm:v0.5.22-rocm720-mi30x-20261020

# Or use Docker (AMD MI355X)
docker pull lmsysorg/sglang-rocm:v0.5.22-rocm720-mi35x-20261020
```

For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
* **H100 / B200 / B300 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. It only helps when tp > 1, so on B200/B300 it applies to the 397B-A17B FP8 recipe (tp=4); the 122B-A10B and 35B-A3B FP8 recipes are single-GPU there.
* **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32). For the **MXFP4 checkpoint on MI355X**, set `ROCM_QUICK_REDUCE_QUANTIZATION=INT4` to route multi-GPU collectives through INT4-quantized ROCm quick all-reduce, and drop `--enable-aiter-allreduce-fusion` (the two are mutually exclusive; quick all-reduce is preferred for this recipe).
* **KV cache offloading (MXFP4 on MI355X):** Selecting **Host DRAM (HiCache)** adds a host-memory tier below the device KV cache, which lets long-context agentic workloads keep more prefix cached and reach higher concurrency than the device KV pool alone allows. It emits `--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first`. HiCache extends the radix cache, so the generated command drops `--disable-radix-cache` — the two options are mutually exclusive and SGLang rejects them together at startup. `--hicache-ratio 1.5` sizes the host tier at 1.5x the device KV cache, so make sure the node has that much free CPU DRAM per server on top of the model weights. Leave this **Disabled** for fixed-length throughput serving, where the extra host traffic buys nothing.
* **KV cache offloading (397B FP8 on MI355X):** A separate recipe from the MXFP4 one above, for long-context agentic serving of the FP8 checkpoint. Selecting **Host DRAM (HiCache)** emits `--tp 4` instead of the tp=2 that merely fits the weights, `--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first`, `--kv-cache-dtype fp8_e4m3` to halve device KV bytes, and the scheduler and tokenizer tuning `--chunked-prefill-size 16384 --max-prefill-tokens 16384 --scheduler-recv-interval 60 --stream-interval 50 --tokenizer-worker-num 6`. `write_through` mirrors every page to the host tier as it is written. `--hicache-ratio 1.5` is the default sizing; the benchmarked high-concurrency points instead pin `--hicache-size` per rank (253 GB at concurrency 32-56, 374 GB at 64-80), which overrides the ratio — add it if you want a fixed host tier. This arm keeps `--enable-aiter-allreduce-fusion` and does not set `ROCM_QUICK_REDUCE_QUANTIZATION`; the MXFP4 recipe does the opposite. Read the FP8 KV cache warning below before using it for reasoning-heavy work.
* **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time.
* **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
  * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs.
  * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64).
* The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
* Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
* To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command.
* **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
* **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300.
* **B200 (FP8)**: Add `--enable-flashinfer-allreduce-fusion` for optimized throughput on Blackwell.
* **B200 / B300 (FP8 MoE)**: Add `--moe-runner-backend flashinfer_trtllm` to route the fused MoE through the trtllm-gen kernels on Blackwell. This is not picked up automatically — SGLang's auto-selection of `flashinfer_trtllm` is limited to the DeepSeek architecture family, so an FP8 Qwen3.5 MoE run otherwise falls back to the Triton MoE runner. It applies to the MoE sizes (397B-A17B / 122B-A10B / 35B-A3B); the dense sizes have no MoE layers. The NVFP4 recipes already set it.
* **B200 / B300 (FP8)**: Add `--linear-attn-prefill-backend flashinfer` to run GDN (linear attention) prefill on the FlashInfer CuTe-DSL kernel. SGLang auto-selects it only inside a narrow validated domain (a chunked prefill size of at most 8192 and a bfloat16 mamba state pool, among other conditions), which these FP8 recipes fall outside of, so the override has to be explicit. Requires CUDA 13+ and flashinfer >= 0.6.14 — SGLang raises at startup on SM100 with an older CUDA. Applies to every Qwen3.5 size, since all of them are hybrid GDN models.
* For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
* Hardware requirements:
  * **BF16**: \~397B parameters require \~800GB of GPU memory for weights.
    * **H100 (80GB)** requires tp=16 (2 nodes) since each rank needs \~100GB at tp=8.
    * **H200 (141GB)** runs with tp=8.
    * **B200 (183GB)** runs with tp=8.
    * **B300 (275GB)** runs with tp=4.
    * **MI300X (192GB)** runs with tp=8.
    * **MI325X (256GB)** runs with tp=4.
    * **MI355X (288GB)** runs with tp=4.
  * **FP8**: The FP8 quantized model requires \~400GB for weights, cutting memory in half.
    * **H100 (80GB)** runs with tp=8.
    * **H200 (141GB)** runs with tp=4.
    * **B200 (183GB)** runs with tp=4.
    * **B300 (275GB)** runs with tp=2.
    * **MI300X (192GB)** runs with tp=4.
    * **MI325X (256GB)** runs with tp=2.
    * **MI355X (288GB)** runs with tp=2.
  * **FP4**: The FP4 quantized model requires \~250GB for weights, cutting memory by almost 4x. NVFP4 ([nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)) requires B200/B300 (Blackwell architecture); AMD provides an MXFP4 checkpoint with FP8 attention ([amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2)) for MI355X.
    * **B200 (183GB)** runs with tp=4 (tp=2 with expert parallelism 2 when MTP is enabled). (NVFP4)
    * **B300 (275GB)** runs with tp=2. (NVFP4)
    * **MI355X (288GB)** runs with tp=2 (use tp=4 for low concurrency). (MXFP4)

<table>
  <thead>
    <tr>
      <th>Hardware</th>
      <th>Memory</th>
      <th>BF16 TP</th>
      <th>FP8 TP</th>
      <th>FP4 TP</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>H100</td>
      <td>80GB</td>
      <td>16</td>
      <td>8</td>
      <td>N/A</td>
    </tr>

    <tr>
      <td>H200</td>
      <td>141GB</td>
      <td>8</td>
      <td>4</td>
      <td>N/A</td>
    </tr>

    <tr>
      <td>B200</td>
      <td>183GB</td>
      <td>8</td>
      <td>4</td>
      <td>4 / 2 + EP2 (MTP)</td>
    </tr>

    <tr>
      <td>B300</td>
      <td>275GB</td>
      <td>4</td>
      <td>2</td>
      <td>2</td>
    </tr>

    <tr>
      <td>MI300X</td>
      <td>192GB</td>
      <td>8</td>
      <td>4</td>
      <td>N/A</td>
    </tr>

    <tr>
      <td>MI325X</td>
      <td>256GB</td>
      <td>4</td>
      <td>2</td>
      <td>N/A</td>
    </tr>

    <tr>
      <td>MI355X</td>
      <td>288GB</td>
      <td>4</td>
      <td>2 / 4 (agentic)</td>
      <td>2 / 4 (MXFP4)</td>
    </tr>
  </tbody>
</table>

> [!WARNING]
>
> **FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is emitted for the MXFP4-on-MI355X recipe and for the 397B FP8-on-MI355X agentic recipe (Host DRAM / HiCache selected), where it is part of the tuned configuration; for every other combination the generated commands leave it out, so add it manually only if memory constraints require the trade-off.

* **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

### 3.3 Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments of Qwen3.5-397B-A17B, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

* **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
* **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
* **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

## 4. Model Invocation

**NVIDIA:**

Deploy Qwen3.5-397B-A17B with the following command (H200, all features enabled):

```shell Command theme={null}
sglang serve \
  --model-path Qwen/Qwen3.5-397B-A17B \
  --tp 8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algo NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.8 \
  --host 0.0.0.0 \
  --port 30000
```

**AMD:**

Deploy Qwen3.5-397B-A17B with the following command (MI300X/MI325X/MI355X):

```shell Command theme={null}
SGLANG_USE_AITER=1 \
SGLANG_USE_AITER_UNIFIED_ATTN=1 \
AITER_FLYDSL_FORCE=1 \
SGLANG_MAMBA_SSM_DTYPE=bfloat16 \
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3.5-397B-A17B \
  --tp 8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --trust-remote-code \
  --page-size 16 \
  --mem-fraction-static 0.8 \
  --attention-backend aiter \
  --enable-aiter-allreduce-fusion \
  --model-loader-extra-config '{"enable_multithread_load": true}' \
  --watchdog-timeout 1200 \
  --host 0.0.0.0 \
  --port 30000
```

> **Note:** To enable MTP (speculative decoding), append `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`.

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Vision Input

Qwen3.5 supports image and video inputs as a unified vision-language model. Here is an example with an image:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.5-397B-A17B",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
                    }
                },
                {
                    "type": "text",
                    "text": "Describe this image in detail."
                }
            ]
        }
    ],
    max_tokens=2048,
    stream=True
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user wants a detailed description of the provided image.

1.  **Identify the main components:**
    *   There is a central square.
    *   There are four circles of varying sizes.

2.  **Analyze the arrangement and relationships:**
    *   **The Square:** It's in the middle, oriented upright (sides are vertical and horizontal). It's drawn with a thin black line.
    *   **The Circles:**
        *   **Top Right:** A large circle. It overlaps the top right corner of the square. A significant portion of the circle is outside the square, but a chunk of it is inside.
        *   **Bottom Left:** A large circle, roughly the same size as the top right one. It overlaps the bottom left corner of the square. Again, a chunk is inside, most is outside.
        *   **Top Left:** A smaller circle. It is positioned near the top left corner of the square. It overlaps the corner slightly.
        *   **Bottom Right:** A smaller circle, roughly the same size as the top left one. It is positioned near the bottom right corner of the square. It overlaps the corner slightly.

3.  **Synthesize the description:**
    *   Start with a general overview: A geometric line drawing.
    *   Describe the central shape: A square.
    *   Describe the surrounding shapes: Four circles.
    *   Detail the positioning:
        *   Two large circles are diagonally opposite (bottom-left and top-right).
        *   Two smaller circles are diagonally opposite (top-left and bottom-right).
    *   Describe the interaction/overlap: The shapes are not just touching; they intersect. The lines cross over each other. The square acts as a central anchor. The circles seem to be "orbiting" or clustered around the corners of the square.
    *   Mention the style: Black lines on a white background. Minimalist. No shading or color.

4.  **Refine the details:**
    *   Let's look closer at the overlaps.
    *   **Top Left Circle:** It sits mostly above and to the left of the square's top-left corner. The bottom-right part of this small circle cuts into the square.
    *   **Top Right Circle:** This is the largest circle (or tied for largest). Its bottom-left quadrant overlaps significantly with the top-right quadrant of the square.
    *   **Bottom Left Circle:** This is also a large circle. Its top-right quadrant overlaps significantly with the bottom-left quadrant of the square.
    *   **Bottom Right Circle:** This is a small circle. Its top-left part overlaps with the bottom-right corner of the square.

5.  **Final Polish:** Structure the description logically. Start with the big picture, then break down each element's position relative to the center.

    *   *Drafting the description:* The image shows a black and white line drawing of geometric shapes. In the center is a square. Surrounding it are four circles of two different sizes.
    *   *Adding detail:* There are two large circles and two small circles.
    *   *Positioning:*
        *   A large circle is in the bottom left. It overlaps the bottom-left corner of the square.
        *   A large circle is in the top right. It overlaps the top-right corner of the square.
        *   A small circle is in the top left. It overlaps the top-left corner of the square.
        *   A small circle is in the bottom right. It overlaps the bottom-right corner of the square.
    *   *Visual style:* The lines are thin and black. The background is white. The shapes are transparent (wireframe style), so you can see the lines of the square through the circles and vice versa where they overlap.

6.  **Final Review against the image:** Does this match? Yes. The description is accurate. The "transparency" point is important because it's a line drawing, not solid shapes. The lines cross.

    *   Wait, let me look really closely at the "Top Left" circle. It actually looks like it's touching the top edge and left edge. It overlaps the corner.
    *   Let me look at the "Bottom Right" circle. It overlaps the bottom right corner.
    *   The arrangement creates a sort of diagonal symmetry. Large-Large on one diagonal, Small-Small on the other.

    Let's write the final output based on these observations.
=============== Content =================


Based on the image provided, here is a detailed description:

**Overview**
The image is a minimalist, black-and-white line drawing featuring geometric shapes. It consists of a central square surrounded by four circles of varying sizes. The lines are thin and black against a plain white background. The shapes are drawn in a "wireframe" style, meaning they are transparent outlines; where shapes overlap, the lines cross over each other rather than one blocking the other.

**Detailed Breakdown**

1.  **The Central Square:**
    *   There is a single square positioned in the center of the composition. It is oriented upright with vertical and horizontal sides.

2.  **The Circles:**
    *   There are four circles arranged around the corners of the square. They appear in two distinct sizes: two large circles and two smaller circles.
    *   **Top Right:** A large circle is positioned at the top right. It overlaps significantly with the top-right corner of the square. A portion of the circle's interior is inside the square's boundary.
    *   **Bottom Left:** Another large circle (roughly the same size as the top right one) is positioned at the bottom left. It overlaps significantly with the bottom-left corner of the square.
    *   **Top Left:** A smaller circle is positioned near the top left corner. It overlaps slightly with the top-left corner of the square.
    *   **Bottom Right:** A smaller circle (roughly the same size as the top left one) is positioned near the bottom right corner. It overlaps slightly with the bottom-right corner of the square.

**Composition**
The arrangement creates a diagonal symmetry. The two largest circles are on a diagonal from bottom-left to top-right, while the two smallest circles are on a diagonal from top-left to bottom-right. The intersecting lines create a complex web of curves and angles in the center of the image.
```

### 4.3 Advanced Usage

#### 4.3.1 Reasoning Parser

Qwen3.5 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.

To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:

* **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
* **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.

**Example 1: Thinking Mode (Default)**

Thinking mode is enabled by default. The model will reason step-by-step before answering, and the thinking process is returned via `reasoning_content`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Thinking mode is enabled by default, no extra parameters needed
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-397B-A17B",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
Thinking Process:

1.  **Analyze the Request:** The user wants to solve a math problem: "What is 15% of 240?" and expects a step-by-step explanation.

2.  **Identify the Core Operation:** The problem asks for a percentage of a number. The formula is: $\text{Percentage} \times \text{Number} = \text{Result}$.
    *   Percentage: $15\%$
    *   Number: $240$

3.  **Determine the Steps:**
    *   Step 1: Convert the percentage to a decimal or fraction.
    *   Step 2: Multiply the converted value by the number.
    *   Step 3: Calculate the final result.
    *   Alternative Step (Mental Math): Break down 15% into 10% + 5%.

4.  **Draft the Explanation (Method 1: Decimal Conversion):**
    *   Convert $15\%$ to $0.15$.
    *   Multiply $0.15 \times 240$.
    *   Calculation: $15 \times 24 = 360$, adjust decimal places $\rightarrow 36.0$.

5.  **Draft the Explanation (Method 2: Fraction Conversion):**
    *   Convert $15\%$ to $\frac{15}{100}$ or $\frac{3}{20}$.
    *   Multiply $\frac{3}{20} \times 240$.
    *   $240 / 20 = 12$.
    *   $3 \times 12 = 36$.

6.  **Draft the Explanation (Method 3: Breaking it down - often easier for step-by-step):**
    *   Find 10% of 240.
    *   Find 5% of 240 (which is half of 10%).
    *   Add them together.
    *   10% of 240 = 24.
    *   5% of 240 = 12.
    *   $24 + 12 = 36$.

7.  **Select the Best Presentation:** Method 3 is very intuitive for "step-by-step" requests as it shows logical reasoning without heavy multiplication. However, the standard decimal multiplication (Method 1) is the most formal. I will present the standard method clearly, perhaps mentioning the breakdown method as a check or alternative way to understand it. Let's stick to the standard multiplication method for clarity but explain the conversion clearly. Actually, providing the breakdown method is often more helpful for understanding *why*. I'll provide the standard calculation method as the primary steps.

    *   Step 1: Convert percentage to decimal.
    *   Step 2: Multiply.
    *   Step 3: State answer.

    Let's refine the calculation for the final output.
    $15\% = 0.15$
    $0.15 \times 240$
    $= \frac{15}{100} \times 240$
    $= 15 \times 2.4$
    $= 36$

    Or:
    $10\% = 24$
    $5\% = 12$
    $15\% = 36$

    I will write out the solution using the decimal/fraction multiplication method as it's universally applicable, but I'll make the steps very clear.

8.  **Final Review:** Does this answer the prompt accurately and step-by-step? Yes.

    *   Step 1: Convert 15% to a decimal.
    *   Step 2: Multiply by 240.
    *   Step 3: Calculate.

    Let's write it out.

    Step 1: Convert the percentage to a decimal.
    $15\% = 0.15$

    Step 2: Multiply the decimal by the number.
    $0.15 \times 240$

    Step 3: Perform the multiplication.
    $15 \times 240 = 3600$
    Adjust decimal (2 places): $36.00$
    Result: 36.

    Wait, $0.15 \times 240$.
    $15 \times 24 = 360$.
    One zero from 240 cancels one decimal place?
    $0.15 \times 240 = 15 \times 2.4 = 36$.
    Or $\frac{15}{100} \times 240 = \frac{15 \times 240}{100} = \frac{3600}{100} = 36$.
    This looks correct.

    I will also include the "10% + 5%" method as an alternative way to think about it, as it's often easier for mental math.

    Plan:
    1. Define the formula.
    2. Convert percent to decimal/fraction.
    3. Multiply.
    4. Provide the alternative mental math method for completeness.
    5. State final answer clearly.

    Actually, keeping it simple is better. Just one clear path. I'll use the fraction/decimal multiplication path.

    Step 1: Convert 15% to a decimal.
    Step 2: Multiply 240 by 0.15.
    Step 3: Final result.

    Let's go.cw
=============== Content =================


Here is the step-by-step solution to find 15% of 240:

**Step 1: Convert the percentage to a decimal.**
To convert a percentage to a decimal, divide by 100.
$$15\% = \frac{15}{100} = 0.15$$

**Step 2: Multiply the decimal by the number.**
Now, multiply 0.15 by 240.
$$0.15 \times 240$$

**Step 3: Calculate the result.**
You can think of this as:
$$15 \times 240 = 3600$$
Since there are two decimal places in 0.15, move the decimal point in the result two places to the left:
$$3600 \rightarrow 36.00$$

**Alternative Method (Mental Math):**
*   Find 10% of 240: $240 \div 10 = 24$
*   Find 5% of 240 (half of 10%): $24 \div 2 = 12$
*   Add them together (10% + 5% = 15%): $24 + 12 = 36$

**Answer:**
15% of 240 is **36**.
```

**Example 2: Instruct Mode (Thinking Off)**

To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Disable thinking mode via chat_template_kwargs
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-397B-A17B",
    messages=[
        {"role": "user", "content": "What is 15% of 240?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    max_tokens=2048,
    stream=True
)

# In Instruct mode, the model responds directly without reasoning_content
for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
To find 15% of 240, you can follow these steps:

### Step-by-Step Deduction

1.  **Convert the percentage to a decimal
**:
    To convert a percentage to a decimal, divide by 100.
    $$15\% = \frac{15}{100} = 0.15$$

2.  **Multiply the decimal by the number**:
    Multiply $0.15$ by $240$.
    $$0.15 \times 240$$

    *Alternative Method (Mental Math)*:
    - Find 10% of 240: $240 \times 0.10 = 24$
    - Find 5% of 240 (which is half of 10%): $24 / 2 = 12$
    - Add them together ($10\% + 5\% = 15\%$): $24 + 12 = 36$

3.  **Calculation**:
    $$240 \times 0.15 = 36$$

### Final Conclusion
15% of 240 is **36**.
```

#### 4.3.2 Tool Calling

Qwen3.5 supports tool calling capabilities. Enable the tool call parser during deployment. Thinking mode is on by default; to disable it for tool calling requests, pass `extra_body={"chat_template_kwargs": {"enable_thinking": False}}`.

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="Qwen/Qwen3.5-397B-A17B",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

# Process streaming response
thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I have access to a get_weather function that can provide current weather information for a location. Let me check the parameters:

- location (required): "Beijing" - this is provided by the user
- unit (optional): The user didn't specify a temperature unit, so I won't include this optional parameter

I should call the get_weather function with Beijing as the location.


=============== Content =================
Tool Call: get_weather
   Arguments:
Tool Call: None
   Arguments: {
Tool Call: None
   Arguments: "location": "Beijing"
Tool Call: None
   Arguments: }
```

#### 4.3.3 Semantic Decisions

Qwen3.5-35B-A3B can answer typed questions about an input with a probability for every option through `/v1/decisions`, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains `/v1/decisions`, install a [nightly build](/docs/get-started/install#nightly-builds). With a server started by `sglang serve --model-path Qwen/Qwen3.5-35B-A3B --host 127.0.0.1 --port 30000`:

```python Example theme={null}
import requests

response = requests.post(
    "http://127.0.0.1:30000/v1/decisions",
    json={
        "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.",
        "questions": [
            {
                "id": "team",
                "type": "choice",
                "question": "Which team should handle this ticket?",
                "options": [{"name": "billing"}, {"name": "technical"}, {"name": "sales"}],
            },
            {"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."},
        ],
    },
    timeout=60,
)
response.raise_for_status()
answers = response.json()["answers"]
print(answers["team"]["choice"], answers["team"]["probabilities"])
print(answers["urgent"]["probabilities"]["yes"])
```

See [Decision models](/docs/supported-models/decision_models) for the request reference, how the probabilities are computed, and the limits.

## 5. Benchmark

### 5.1 Accuracy Benchmark

#### 5.1.1 GSM8K Benchmark

* Benchmark Command

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

Historical results below use the removed harness and are not measurements of the sgl-eval command above.

* Test Result

```text Output theme={null}
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:31<00:00,  6.43it/s]
Accuracy: 0.975
Invalid: 0.005
Latency: 31.784 s
Output throughput: 998.166 token/s
```

#### 5.1.2 Full GSM8K with sgl-eval

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://localhost:30000/v1 \
  --model Qwen/Qwen3.5-397B-A17B \
  --num-examples 1319 --num-threads 256 --max-tokens 2048 --thinking
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

#### 5.1.3 MMMU Benchmark

* Benchmark Command

```bash Command theme={null}
python3 benchmark/mmmu/bench_sglang.py --concurrency 128 --port 30000 --max-new-tokens 512
```

* Test Result

```text Output theme={null}
{'Accounting': {'acc': 1.0, 'num': 3},
 'Agriculture': {'acc': 1.0, 'num': 4},
 'Art': {'acc': 1.0, 'num': 9},
 'Art_Theory': {'acc': 1.0, 'num': 5},
 'Basic_Medical_Science': {'acc': 1.0, 'num': 2},
 'Biology': {'acc': 1.0, 'num': 1},
 'Chemistry': {'acc': 1.0, 'num': 1},
 'Computer_Science': {'acc': 1.0, 'num': 1},
 'Design': {'acc': 0.909, 'num': 11},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 1.0, 'num': 1},
 'Economics': {'acc': 1.0, 'num': 5},
 'Finance': {'acc': 1.0, 'num': 2},
 'Geography': {'acc': 1.0, 'num': 3},
 'History': {'acc': 1.0, 'num': 3},
 'Literature': {'acc': 0.938, 'num': 16},
 'Manage': {'acc': 1.0, 'num': 2},
 'Marketing': {'acc': 1.0, 'num': 5},
 'Math': {'acc': 1.0, 'num': 1},
 'Overall': {'acc': 0.978, 'num': 91},
 'Overall-Art and Design': {'acc': 0.96, 'num': 25},
 'Overall-Business': {'acc': 1.0, 'num': 17},
 'Overall-Health and Medicine': {'acc': 1.0, 'num': 7},
 'Overall-Humanities and Social Science': {'acc': 0.966, 'num': 29},
 'Overall-Science': {'acc': 1.0, 'num': 8},
 'Overall-Tech and Engineering': {'acc': 1.0, 'num': 5},
 'Pharmacy': {'acc': 1.0, 'num': 2},
 'Physics': {'acc': 1.0, 'num': 2},
 'Psychology': {'acc': 1.0, 'num': 4},
 'Public_Health': {'acc': 1.0, 'num': 2},
 'Sociology': {'acc': 1.0, 'num': 6}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.978
```

### 5.2 Speed Benchmark

**Test Environment:**

* Hardware: H200 (8x)
* Model: Qwen3.5-397B-A17B
* Tensor Parallelism: 8
* SGLang Version: main branch

Server Launch Command:

```bash Command theme={null}
SGLANG_USE_CUDA_IPC_TRANSPORT=1 python -m sglang.launch_server \
  --model Qwen/Qwen3.5-397B-A17B \
  --tp 8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algo NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.8 \
  --host 0.0.0.0 \
  --port 30000
```

#### 5.2.1 Latency Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3.5-397B-A17B \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  18.94
Total input tokens:                      6101
Total input text tokens:                 6101
Total generated tokens:                  4220
Total generated tokens (retokenized):    4211
Request throughput (req/s):              0.53
Input token throughput (tok/s):          322.16
Output token throughput (tok/s):         222.84
Peak output token throughput (tok/s):    289.00
Peak concurrent requests:                3
Total token throughput (tok/s):          545.00
Concurrency:                             1.00
Accept length:                           3.12
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1892.35
Median E2E Latency (ms):                 1410.85
P90 E2E Latency (ms):                    3749.34
P99 E2E Latency (ms):                    4216.52
---------------Time to First Token----------------
Mean TTFT (ms):                          190.40
Median TTFT (ms):                        208.46
P99 TTFT (ms):                           261.27
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          3.96
Median TPOT (ms):                        3.79
P99 TPOT (ms):                           4.96
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.04
Median ITL (ms):                         3.15
P95 ITL (ms):                            6.65
P99 ITL (ms):                            12.60
Max ITL (ms):                            58.03
==================================================
```

#### 5.2.2 Throughput Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model Qwen/Qwen3.5-397B-A17B \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  283.04
Total input tokens:                      502493
Total input text tokens:                 502493
Total generated tokens:                  500251
Total generated tokens (retokenized):    498222
Request throughput (req/s):              3.53
Input token throughput (tok/s):          1775.37
Output token throughput (tok/s):         1767.45
Peak output token throughput (tok/s):    3630.00
Peak concurrent requests:                108
Total token throughput (tok/s):          3542.82
Concurrency:                             96.71
Accept length:                           3.31
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   27372.05
Median E2E Latency (ms):                 26660.21
P90 E2E Latency (ms):                    39951.91
P99 E2E Latency (ms):                    48405.51
---------------Time to First Token----------------
Mean TTFT (ms):                          14247.21
Median TTFT (ms):                        14932.44
P99 TTFT (ms):                           20998.45
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          26.16
Median TPOT (ms):                        26.13
P99 TPOT (ms):                           41.33
---------------Inter-Token Latency----------------
Mean ITL (ms):                           26.29
Median ITL (ms):                         11.38
P95 ITL (ms):                            72.10
P99 ITL (ms):                            149.57
Max ITL (ms):                            1220.68
==================================================
```

### 5.3 Vision Speed Benchmark

We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images. Each request has 128 input tokens, two 720p images, and 1024 output tokens.

#### 5.3.1 Latency Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3.5-397B-A17B \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1 \
  --request-rate inf
```

```text Output theme={null}
TODO
```

#### 5.3.2 Throughput Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model Qwen/Qwen3.5-397B-A17B \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100 \
  --request-rate inf
```

```text Output theme={null}
TODO
```


# Qwen3.6
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.6



## 1. Model Introduction

The Qwen3.6 series is developed by Alibaba. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, delivering substantial upgrades in agentic coding and thinking preservation. Two size/sparsity variants are released:

* [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — **Sparse MoE** (35B total, 3B active) on a Gated Delta Networks backbone.
* [Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) — **Dense** hybrid GDN; smaller weights footprint, single-GPU friendly.

Both variants share the same hybrid reasoning, tool-calling, and multimodal interface and natively handle context lengths of up to 262,144 tokens, extensible to over 1M tokens.

**Key Features:**

* **Agentic Coding**: Handles frontend workflows and repository-level reasoning with greater fluency and precision
* **Thinking Preservation**: New option to retain reasoning context from historical messages, streamlining iterative development
* **Efficient Hybrid Architecture**: Gated Delta Networks backbone; sparse MoE (35B / 3B active) or dense 27B variant
* **Hybrid Reasoning**: Thinking mode enabled by default with step-by-step reasoning, can be disabled for direct responses
* **Tool Calling**: Built-in tool calling support with `qwen3_coder` parser
* **Multi-Token Prediction (MTP)**: Speculative decoding support for lower latency; both MoE and Dense variants ship `mtp.safetensors`
* **Multimodal**: Unified vision-language model supporting text, image, and video inputs

**Available Models:**

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Architecture</th>
      <th>Weights</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen3.6-35B-A3B (BF16)</td>
      <td>MoE 35B / 3B active</td>
      <td>[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B (FP8)</td>
      <td>MoE 35B / 3B active</td>
      <td>[Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B (NVFP4)</td>
      <td>MoE 35B / 3B active (Blackwell)</td>
      <td>[nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4)</td>
    </tr>

    <tr>
      <td>Qwen3.6-27B (BF16)</td>
      <td>Dense 27B</td>
      <td>[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)</td>
    </tr>

    <tr>
      <td>Qwen3.6-27B (FP8)</td>
      <td>Dense 27B</td>
      <td>[Qwen/Qwen3.6-27B-FP8](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)</td>
    </tr>

    <tr>
      <td>Qwen3.6-27B (NVFP4)</td>
      <td>Dense 27B (Blackwell)</td>
      <td>[nvidia/Qwen3.6-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)</td>
    </tr>
  </tbody>
</table>

**License:** Apache 2.0

## 2. SGLang Installation

SGLang `>=0.5.10` is required for Qwen3.6. You can install from PyPI, from source, or use a Docker image:

```bash Command theme={null}
# Install from PyPI
uv pip install --prerelease=allow sglang

# Or install from source
uv pip install --prerelease=allow 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'

# Or use Docker (NVIDIA GPUs; also serves the NVFP4 variants)
docker pull lmsysorg/sglang:latest
```

For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install).

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

On Ascend NPUs (Atlas 800I A3), follow [SGLang installation with NPU support](/docs/hardware-platforms/ascend-npus/getting-started/installation) and the [Ascend NPU quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) to select a compatible container image and host driver. The A3 recipes below require the Ascend software stack; the CUDA image above does not provide it.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and capabilities, including Ascend NPU (Atlas 800I A3) scenarios.

Run this Command:

<pre />

### RTX Spark: 35B-A3B NVFP4

Select **RTX Spark** in the generator above to use the single-node, single-GPU recipe for
[nvidia/Qwen3.6-35B-A3B-NVFP4](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4)
on [NVIDIA RTX Spark](https://www.nvidia.com/en-us/products/rtx-spark/), which
pairs a Blackwell RTX GPU with a Grace CPU and offers up to 128GB of unified
memory. The contributor tested this command on SGLang `main`. The recipe uses
the FlashInfer CUTLASS MoE runner,
`extra_buffer` Mamba radix caching, and a static memory fraction of `0.8`.
Speculative decoding is not enabled.

The contributor reported **75 output tokens/s for a single request** on SGLang
`main`. The exact commit, OS/runtime, memory configuration, input/output lengths,
and timing method were not provided. This is a reported generation speed, not
an input-plus-output throughput measurement or a measured P50 TPOT. The Docker
selector uses the nightly `dev` fallback; that image was not the reported test
environment.

### 3.2 Configuration Tips

* Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
* **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`:
  * **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage.
  * **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput.
* The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload.
* Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities.
* **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`.
* **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300.
* For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors.
* **Ascend NPU (Atlas 800I A3):** Each card has 2 dies, so `--tp-size` is twice the card count (1 card → `--tp-size 2`). Add `--attention-backend ascend --device npu` and `--mamba-ssm-dtype bfloat16`, run speculative decoding with the built-in MTP weights via `--speculative-algorithm NEXTN`. See [Deploying on Ascend NPUs](#ascend-npu-deployment) below.
* Hardware requirements for the GPU recipes:
  * **35B-A3B BF16**: \~70GB for weights. TP=1 fits on all supported hardware.
  * **35B-A3B FP8**: \~35GB for weights. TP=1 fits on all supported hardware.
  * **35B-A3B NVFP4**: \~23GB for weights. TP=1 fits on B200/B300 and RTX Spark.
  * **27B BF16**: \~54GB for weights. TP=1 fits on all supported hardware.
  * **27B FP8**: \~27GB for weights. TP=1 fits on all supported hardware.
  * **27B NVFP4**: \~22GB for weights. TP=1 fits on B200/B300.

All Qwen3.6 variants (MoE 35B-A3B and Dense 27B) fit on a single supported GPU. The table below covers B200/B300 NVFP4; the 35B-A3B RTX Spark recipe is provided above:

<table>
  <thead>
    <tr>
      <th>Hardware</th>
      <th>Memory</th>
      <th>BF16 TP</th>
      <th>FP8 TP</th>
      <th>NVFP4 TP</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>H100</td>
      <td>80GB</td>
      <td>1</td>
      <td>1</td>
      <td>—</td>
    </tr>

    <tr>
      <td>H200</td>
      <td>141GB</td>
      <td>1</td>
      <td>1</td>
      <td>—</td>
    </tr>

    <tr>
      <td>B200</td>
      <td>183GB</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>

    <tr>
      <td>B300</td>
      <td>275GB</td>
      <td>1</td>
      <td>1</td>
      <td>1</td>
    </tr>
  </tbody>
</table>

* **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

Referenced by the configuration tips above. Verify before removing.

### 3.3 Deploying on Ascend NPUs (Atlas 800I A3)

The Ascend model tutorials document support for [Qwen3.6-27B](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b) and [Qwen3.6-35B-A3B](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b) starting with SGLang v0.5.16. The generator provides a baseline for single-node deployment with prefill and decode on the same node (PD mixed mode). Use the complete scenario recipes below for workload-specific tuning.

Launch a compatible container using the [Ascend NPU quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start), including its driver, firmware, and runtime mounts, then run the generated `sglang serve` command inside it. Select the device IDs for your cards: one A3 card exposes two dies. For example, a 1-card deployment using dies 0 and 1 needs `--device /dev/davinci0 --device /dev/davinci1`, together with the management devices and mounts in the quickstart.

**Weights:** BF16 checkpoints are available from [Qwen](https://huggingface.co/Qwen) and mirrored on ModelScope. To download from ModelScope, install the `modelscope` Python package and set `SGLANG_USE_MODELSCOPE=1`. Alternatively, pre-download the weights and set `--model-path` to their local directory.

The generator offers the following card counts. Weight sizes exclude KV cache, Mamba state, and runtime buffers; choose capacity and concurrency for your workload.

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Checkpoint</th>
      <th>Weights</th>
      <th>1 card (TP 2)</th>
      <th>2 cards (TP 4)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen3.6-27B</td>
      <td>[Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B)</td>
      <td>\~54 GB (BF16)</td>
      <td>Available</td>
      <td>Available</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)</td>
      <td>\~70 GB (BF16)</td>
      <td>Available</td>
      <td>Available</td>
    </tr>
  </tbody>
</table>

**Published scenario references (PD mixed, single node):**

The TPOT values below are reproduced from the linked Ascend best-practice pages. They describe those pages' full configurations and workloads, not measurements of the generator's baseline command. Follow each scenario's host tuning, environment variables, launch flags, and benchmark command when reproducing it.

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Cards</th>
      <th>Quantization</th>
      <th>Dataset</th>
      <th>Source TPOT</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen3.6-27B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[1024x1024 (30)+1024](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-27B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[1080p\_30+256](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#qwen3-6-27b-1p-in1080p-30-out256-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-27B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[64k+1k (90% prefix cache hit rate)](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[254k+1k](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in254k-out1k-16-1ms)</td>
      <td>16.1ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[3.5k+1.5k](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[64k+1k](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in64k-out1k-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[64k+1k (90% prefix cache hit rate)](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[128k+1k](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in128k-out1k-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[128k+1k (90% prefix cache hit rate)](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[1024x1024 (30)+1024](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>1</td>
      <td>BF16</td>
      <td>[1080p\_30+256](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms)</td>
      <td>50ms</td>
    </tr>

    <tr>
      <td>Qwen3.6-35B-A3B</td>
      <td>2</td>
      <td>BF16</td>
      <td>[984k+1k](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#qwen3-6-35b-a3b-2p-in984k-out1k-40-91ms)</td>
      <td>40.91ms</td>
    </tr>
  </tbody>
</table>

See the [27B best practices](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b) and [35B-A3B best practices](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b) for the full scenario lists.

> [!NOTE]
>
> * Multimodal serving on A3 additionally enables `--enable-multimodal --mm-attention-backend ascend_attn`; see the image-input best-practice scenarios for the full flag set.
> * The generator leaves the NPU Mamba cache strategy to the installed SGLang version. Ascend prefix-cache recipes use `--mamba-radix-cache-strategy extra_buffer`; copy the complete matching recipe when enabling prefix caching.
> * Use a version-matched Ascend stack and check output quality with the [accuracy evaluation guide](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation) before [performance testing](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing). Successful checkpoint loading alone does not establish generation accuracy.

## 4. Model Invocation

Deploy Qwen3.6 with the following command (H200, all features enabled). Swap `--model-path` to `Qwen/Qwen3.6-27B-FP8` for the dense 27B variant — all other flags carry over:

```shell Command theme={null}
sglang serve \
  --model-path Qwen/Qwen3.6-35B-A3B-FP8 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --mem-fraction-static 0.8 \
  --host 0.0.0.0 \
  --port 30000
```

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Vision Input

Qwen3.6 supports image and video inputs as a unified vision-language model.

**Image Input Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/CI_Demo/mathv-1327.jpg"
                    }
                },
                {
                    "type": "text",
                    "text": "Describe this image in detail."
                }
            ]
        }
    ],
    max_tokens=2048,
    stream=True
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Video Input Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.5/demo/video/N1cdUjctpG8.mp4"
                    }
                },
                {
                    "type": "text",
                    "text": "Describe what happens in this video."
                }
            ]
        }
    ],
    max_tokens=2048,
    stream=True
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

### 4.3 Advanced Usage

#### 4.3.1 Reasoning Parser

Qwen3.6 supports Thinking mode **by default**. Enable the reasoning parser during deployment to separate the thinking and content sections. The thinking process is returned via `reasoning_content` in the streaming response.

To disable thinking and use Instruct mode, pass `chat_template_kwargs` at request time:

* **Thinking mode** (default): The model performs step-by-step reasoning before answering. No extra parameters needed.
* **Instruct mode** (`{"enable_thinking": false}`): The model responds directly without a thinking process.

**Example 1: Thinking Mode (Default)**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    max_tokens=2048,
    stream=True
)

has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Example 2: Instruct Mode (Thinking Off)**

To disable thinking and get a direct response, pass `{"enable_thinking": false}` via `chat_template_kwargs`:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "What is 15% of 240?"}
    ],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    max_tokens=2048,
    stream=True
)

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta
        if delta.content:
            print(delta.content, end="", flush=True)

print()
```

#### 4.3.2 Thinking Preservation

Qwen3.6 has been trained to preserve and leverage thinking traces from historical messages. Enable this for agent scenarios where maintaining full reasoning context improves decision consistency:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "Help me plan a web app architecture."}
    ],
    extra_body={"chat_template_kwargs": {"preserve_thinking": True}},
    max_tokens=2048,
    stream=True
)

thinking_started = False
has_thinking = False
has_answer = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if delta.content:
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

#### 4.3.3 Tool Calling

Qwen3.6 supports tool calling capabilities. Enable the tool call parser during deployment.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="Qwen/Qwen3.6-35B-A3B-FP8",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    stream=True
)

thinking_started = False
has_thinking = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            if has_thinking and thinking_started:
                print("\n=============== Content =================", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                if tool_call.function:
                    print(f"Tool Call: {tool_call.function.name}")
                    print(f"   Arguments: {tool_call.function.arguments}")

        if delta.content:
            print(delta.content, end="", flush=True)

print()
```


# Qwen3.8
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8

Deploy Qwen3.8 with SGLang — day-0 recipes for Qwen's 2.4T-parameter (95B active) hybrid GDN/GQA Mixture-of-Experts model on NVIDIA and AMD.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> **NVIDIA GPUs** (H200 / B200 / B300 / GB300) — the launch image, since this is a day-0 model with no release cut yet:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:qwen38
> ```
>
> **AMD GPUs** — pinned `v0.5.17` builds. The two are **not** interchangeable: they target different GPU architectures *and* different ROCm versions, so pick the one matching your hardware or AITER's kernels won't load.
>
> MI350X / MI355X (CDNA4, gfx950 — ROCm 7.20):
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260812
> ```
>
> MI300X (CDNA3, gfx942 — ROCm 7.00):
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang-rocm:v0.5.17-rocm700-mi30x-20260813
> ```
>
> For how to launch either image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with whatever the command generator below produces.

Pick your hardware + quantization to generate the launch command. Three of the strategies are operating points on the throughput/latency curve; the fourth swaps the speculative decoder:

* **Low Latency** — fastest reply for a single user. Pick for chat.
* **Balanced** — good speed with several users at once. Use for typical multi-user serving.
* **High Throughput** — most tokens per second across many users. Best for batch jobs.
* **DSpark** — the trained [DSpark draft model](#3-3-dspark-and-replayssm-speculative-decoding) instead of the checkpoint's built-in MTP head. Only offered where DSpark's constraints allow it (see [3.3](#3-3-dspark-and-replayssm-speculative-decoding)).

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

The knobs come in two flavors:

* **Built-in SGLang features** — TP / DP-Attention, MoE backend + EP (including WideEP), reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
* **Qwen3.8 specific features** — the **DSpark** speculative-decoding preset and its **ReplaySSM** opt-out, plus the GDN radix-cache-strategy and KV-cache-precision knobs (see [Configuration Tips](#2-configuration-tips) and [3.3 DSpark and ReplaySSM](#3-3-dspark-and-replayssm-speculative-decoding) below).

## 1. Model Introduction

**Qwen3.8** (`Qwen3.8-2.4T-A95B`) is Qwen's largest open-weight model to date: **2.4T total parameters, 95B active per token**. It continues the hybrid-attention design of the Qwen3.5 / Qwen3.6 series, scaled up to 92 layers.

* **Hybrid Attention** — 23 repeats of `3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE)`, so 69 linear-attention layers to 23 full-attention ones. Gated Attention runs 64 query heads over 4 KV heads at head dimension 256. This balances linear computational complexity against long-context modeling quality.
* **GDN (Gated Delta Network)** — the linear-attention layers pair a State Space Model with causal convolution (CausalConv1d), 128 V heads and 16 QK heads at head dimension 128. A fixed-size recurrent state replaces the growing KV cache, so memory is `O(1)` per layer while compute stays `O(N)`.
* **Sparse Mixture-of-Experts** — 512 experts, 10 routed plus 1 shared active per token, expert intermediate dimension 2048. Hidden dimension 8192, vocabulary 248,320.
* **MTP** — the checkpoint ships multi-token-prediction weights trained with multiple steps. That is what the NEXTN speculative recipes on this page decode against.

**License:** [Qwen3.8-2.4T-A95B](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/blob/main/LICENSE). **Context length:** 262,144 native, extensible to 1,010,000 tokens.

**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=0.0`, `repetition_penalty=1.0`. Raising `presence_penalty` toward 2 curbs runaway repetition at some risk of language mixing. For agentic work Qwen suggests allowing 262,144 tokens of reasoning and 131,072 for the final response.

**Resources:** Each precision is its own repo — [BF16](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B) · [FP8](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8) · [NVFP4, NVIDIA Blackwell (RadixArk)](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-NVFP4) · [MXFP4, AMD CDNA4 (Qwen)](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4). Speculative-decoding draft model: [`RadixArk/Qwen3.8-2.4T-A95B-DSpark`](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark) (see [3.3](#3-3-dspark-and-replayssm-speculative-decoding)).

## 2. Configuration Tips

Four cells are marked **Not Verified** — **GB300 BF16** and the three GB300 **DSpark** strategies. They have launch recipes but no completed validation run; every other cell on the page has been run, including B300 NVFP4 DSpark.

**Weight size decides the topology.** At 2.4T parameters BF16 is ≈4.8TB, FP8 ≈2.4TB, NVFP4 ≈1.2TB. FP8 fits no single node here — not even B300, whose 8 × 288GB = 2.30TB misses by a hair — so every FP8 recipe is multi-node. Single-node means FP4: NVFP4 on B300, MXFP4 on MI355X/MI350X. B200 NVFP4 would fit one node but pipelines two, because \~25GB per GPU after weights is too little to serve against. BF16 does not fit 16 GPUs either, so its one recipe is TP32 across 8 GB300 nodes — the only platform where a flat TP32 stays on rack-scale NVLink.

**Two distinct FP4 checkpoints.** [NVFP4](https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-NVFP4) is Blackwell-only; [MXFP4](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8-MXFP4) is MI350X/MI355X-only and hybrid (MXFP4 experts, FP8 attention/dense). MI300X is CDNA3 with no hardware MX matmul, so it serves FP8. Leave `--moe-runner-backend` unset on both and the runner resolves from the checkpoint's own `quant_method` — except the NVFP4 wide-EP tier, which pairs `flashinfer_trtllm_routed` with `--moe-a2a-backend flashinfer` by hand because auto cannot resolve that combination.

### GDN state is the scarce resource, not KV

Two thirds of the layers are Gated DeltaNet, and their recurrent state lives in its own pool. That pool, not KV, is usually what caps concurrency — and a request's cost depends on the caching strategy:

<table>
  <thead>
    <tr>
      <th>Strategy</th>
      <th>State slots per request</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--disable-radix-cache`</td>
      <td>1</td>
    </tr>

    <tr>
      <td>`no_buffer`</td>
      <td>3</td>
    </tr>

    <tr>
      <td>`extra_buffer` (this model's `auto`)</td>
      <td>5, or 4 where PP disables the overlap scheduler</td>
    </tr>
  </tbody>
</table>

So `--max-mamba-cache-size` has to match the ratio in force, or it silently clamps `max_running_requests` to a fraction of the target — every cell leaves the pool to `--mamba-full-memory-ratio` except GB300 FP8 Balanced and Low Latency, whose pins are part of tuned capacity sets — Low Latency's `--max-mamba-cache-size 80` is exactly its 16 concurrent requests × 5 slots. And `extra_buffer` needs radix caching on: `mamba_extra_buffer_of()` requires `disable_radix_cache` false, so adding `--disable-radix-cache` makes the strategy inert and drops the budget to one slot.

### NEXTN caps concurrency at 48

MTP weights ship inside the checkpoint, so NEXTN needs no draft model and the 3/1/4 preset fills in automatically. But a speculative cell with no `--max-running-requests` gets **48** from the speculative hook rather than a memory-derived ceiling — pin it explicitly to serve more. `pp_size > 1` rules speculative decoding out entirely in aggregated serving, which is why the H200, B200/B300 FP8, B200 NVFP4 and MI300X recipes carry no MTP.

### Linear-attention backends differ by GPU generation

`--mamba-ssm-dtype bfloat16` is load-bearing on SM100: the flashinfer GDN decode default is gated on it, and without it decode silently falls back to Triton. On SM90 the GDN default is Triton for *both* halves, which is why H200 is the one cell pinning `--linear-attn-decode-backend flashinfer` too. The flashinfer GDN prefill default only covers chunk sizes up to 8192, so any cell with a larger `--chunked-prefill-size` must state `--linear-attn-prefill-backend flashinfer` itself.

`--attention-backend trtllm_mha` is SM100-only. On Blackwell cells that leave it unset, the model hook picks it together with `--page-size 64` — and returns early when the backend *is* named, so an explicit backend also drops that paired page size and nothing then depends on `--speculative-eagle-topk` to keep the backend off Triton.

### GB300 tiers

The FP8 ladder spans the whole curve on 4 nodes × 4 GPUs:

<table>
  <thead>
    <tr>
      <th>Tier</th>
      <th>Shape</th>
      <th>Spec</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Low Latency</td>
      <td>TP16 narrow EP</td>
      <td>NEXTN 3+1</td>
    </tr>

    <tr>
      <td>Balanced</td>
      <td>DP4×TP4 + EP16</td>
      <td>NEXTN 3+1</td>
    </tr>

    <tr>
      <td>High Throughput</td>
      <td>DP4×TP4 + EP16</td>
      <td>off</td>
    </tr>
  </tbody>
</table>

Balanced and High Throughput share one shape and differ only in capacity. Low Latency is the odd one out: narrow EP wins at low concurrency, where spending ranks on expert parallelism costs more than it returns. MTP is off at saturation because draft-plus-verify overhead outweighs the speedup. NVFP4 has only two tiers — its recipes span different GPU counts (8 vs 16) and the wide-EP one holds capacity fixed across its whole concurrency list, so no third operating point exists.

Extra build requirements: FP8 Balanced and High Throughput need the **DeepEP v2 wheel** (`2.1.0+01dc3aa`), since their `--moe-a2a-backend deepep_v2` flags do not exist upstream. NVFP4 High Throughput needs `nvfp4_agg_wideep_dep16_flashinfer_setup.sh` run first, and its `SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8192` is not optional — unset it falls back to 1024 and startup raises once `1024 × ep_size` no longer covers the largest CuteDSL MoE forward.

### GB300 PD disaggregation layouts

The PD role selector adds the role and transfer flags to the selected base recipe. It does not resize the prefill and decode workers. Use separate workers with the layouts below for the measured GB300 operating points:

| Checkpoint and operating point | Prefill workers | Decode worker | Capacity setting |
| - | - | - | - |
| FP8, high throughput | 2 × TP1 / PP16 | DP4-attention / TP4 / EP16, DeepEP v2 + EPLB | Keep frontend concurrency above the decode MRR so prefill remains queued |
| NVFP4, high throughput | 2 × TP1 / PP6 | DP2-attention / TP4 / EP8, FlashInfer one-sided A2A | Prefill MRR 128 per worker; decode MRR 512; frontend concurrency 1536 |
| NVFP4, low latency | TP4 / PP2 | TP16 | Decode MRR 1 and frontend concurrency 1 at the latency endpoint |

For NVFP4 with NEXTN, use the 3/1/4 settings on both roles so the prefill worker transfers the draft state. Enable ReplaySSM on the decode role. The generated router command sets the main policy to `round_robin`, which applies to prefill, and keeps the decode policy explicit; the server-side `--load-balance-method` does not configure router worker selection. Round robin is the measured choice for the fixed-shape throughput runs above. For agentic workloads with substantial repeated-prefix reuse, consider cache-aware routing, especially on the prefill side, and validate the cache-locality versus load-balance tradeoff on the target workload.

### AllReduce fusion

The four cells running one flat TP group — GB300 FP8 Low Latency, GB300 NVFP4 Low Latency, GB300 BF16, B300 NVFP4 — set `SGLANG_FLASHINFER_MNNVL_CUTEDSL_AR_FUSION`, the Qwen3.5 CuteDSL path whose single workspace fuses AllReduce + Residual + RMSNorm with the MoE finalize. Worth 6–13% over the legacy path. Don't pass `--flashinfer-allreduce-fusion-backend` alongside it — the env suppresses the flag with a warning.

Nothing else can use it: the fusion needs DP-attention off and the built-in TP MoE, so the wide-EP tiers are out, and the pipelined cells put their cross-node traffic on IB rather than NVLink. The three GB300 cells also carry `NCCL_NVLS_ENABLE=1`, because SGLang forces NVLS collectives off when that variable is unset.

### AMD

Recommended: **MI355X + MXFP4, single node, TP8**. MI350X emits the identical command (same gfx950, same 288GB, same `mi35x` image); MI300X is CDNA3, takes the `mi30x` image, and needs two nodes for the FP8 weights.

* **`--mem-fraction-static` looks aggressive on purpose.** With the aiter backend above 8192 context SGLang multiplies it by **0.85** before allocating, so MI355X's `0.9` lands at ≈0.765 and MI300X's `1.0` at ≈0.85. Don't "fix" these downward. MI300X must stay at 1.0 or the weights stop fitting.
* **`--disable-custom-all-reduce` belongs on every MI300X rank** — SGLang resolves it per process, so setting it on one node would leave the two pipeline stages reducing through different code paths.
* MI300X runs `--kv-cache-dtype fp8_e4m3` with `--page-size 16`: at 8 × 192GB per node the shape is memory-bound.

### ReplaySSM

A GDN layer's recurrent state overwrites itself every token, so speculative verify has to be rewindable. Snapshotting the whole K×V state per draft step costs 64 KiB per request, layer and head at K=V=128, times γ+1 steps — scratch taken out of the same budget as the persistent state pool.

[ReplaySSM](https://tridao.me/blog/2026/replayssm/) stores each draft step's raw inputs `Sᵢ = (vᵢ, kᵢ, gᵢ, βᵢ)` instead, a few hundred bytes written by the verify kernel on its way through. Once the sampler fixes the accepted length, one fold kernel replays the accepted prefix from the committed checkpoint and advances it in place. The fold is a verbatim clone of the verify recurrence, so the rebuilt state is bit-identical to the recurrent baseline; draft-step scratch shrinks by roughly two orders of magnitude and is never allocated.

Folding on every commit is what lets it compose with radix prefix caching over the mutable GDN state: every `--mamba-track-interval` tokens the state is handed to the radix tree, and under `extra_buffer` it goes to a second slot so the running request keeps mutating its own. It is off by default — see [3.3](#3-3-dspark-and-replayssm-speculative-decoding).

## 3. Advanced Usage

> [!NOTE]
>
> The `model` argument in the examples below is the BF16 repo id. Every precision is a **separate repo**, so `model` has to be the checkpoint the server was actually launched with — `…-A95B-FP8`, `…-A95B-NVFP4`, or `…-A95B-FP8-MXFP4`. The Deploy panel's cURL snippet always shows the right id for the cell you have selected.

### 3.1 Reasoning

Qwen3.8 **always** reasons — thinking cannot be turned off, and every response opens with a `<think>…</think>` block. The `qwen3` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) splits that block into `reasoning_content`, leaving `content` as the answer alone.

Depth is tunable per request with `reasoning_effort` — `xhigh` (the default), `medium`, or `low`. `preserve_thinking` carries reasoning from earlier turns into context and is on by default.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="Qwen/Qwen3.8-2.4T-A95B",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     reasoning_effort="xhigh",  # xhigh (default) | medium | low
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update — a sample transcript will be added here.
> ```

### 3.2 Tool Calling

Enable the `qwen3_coder` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> resp = client.chat.completions.create(
>     model="Qwen/Qwen3.8-2.4T-A95B",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update — a sample transcript will be added here.
> ```

### 3.3 DSpark and ReplaySSM (Speculative Decoding)

We trained a **DSpark** draft model for Qwen3.8 with SpecForge. Turn it on with the **DSpark** chip in the **Speculative Decoding** card of the [Playground above](#playground) — it emits:

```bash Command theme={null}
--speculative-algorithm DSPARK \
--speculative-draft-model-path RadixArk/Qwen3.8-2.4T-A95B-DSpark
```

**ReplaySSM is a separate opt-in.** `--enable-linear-replayssm-spec` defaults off and DSpark does not turn it on, so add it with the **ReplaySSM (spec)** row in the flag-select list. With it on (see [Configuration Tips](#2-configuration-tips) above for how it works), the verify kernel stores each draft step's raw inputs instead of snapshotting the full K×V GDN state, and a single fold kernel replays the accepted prefix from the last committed checkpoint. It's a pure side channel behind a ring buffer — the verify output is bitwise unchanged — so there's no accuracy tradeoff, only a memory one.

**DSpark does not compose with every cell.** `_handle_dspark` rejects the run outright rather than degrading, so check these before turning the chip on:

* **`--pp-size` must be 1.** The H200, B200, B300 (FP8) and MI300X cells are all pipelined, so DSpark is unavailable on them.
* **With DP-Attention it additionally requires `--enable-dp-lm-head`**, the built-in TP MoE (`--moe-a2a-backend none`), and no context parallel. That rules out the GB300 wide-EP tiers, which run DP attention over DeepEP v2 or FlashInfer A2A.
* `--speculative-num-steps` is forced to 1, and an omitted `--speculative-draft-model-path` only works if the target checkpoint bundles the draft weights.

The Playground greys the DSpark chip out on the combinations above.

The **DSpark** strategy chip in the Deploy panel emits this substitution on the four hw × quantization combinations that clear those constraints: GB300 FP8 (on the low-latency shape — the balanced tier's DeepEP v2 a2a rules DSpark out), GB300 NVFP4, GB300 BF16 and B300 NVFP4. Everything else on the page is either pipelined or wide-EP.

Because the draft model needs its own weights and KV, those cells run tighter than their NEXTN counterparts — the validated B300 recipe drops `--mem-fraction-static` to 0.80 and trims `--context-length` to 200000 to buy the room back.

ReplaySSM composes with radix prefix caching, overlap scheduling, and PD decode, so DSpark speculative decoding runs alongside the rest of the stack (WideEP, PD disaggregation, HiCache) rather than requiring any of them to be turned off.


# Qwen3.8-27B
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B

Deploy Qwen3.8-27B with SGLang — dense hybrid GDN vision-language model with BF16/FP8/NVFP4 W4A4 checkpoints and in-checkpoint MTP, single-GPU on H200, RTX PRO 6000, RTX 5090 and DGX Spark.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install uv
>
> uv pip install --prerelease=allow sglang
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your card + checkpoint precision to generate the launch command. The model runs single-GPU on every supported card — H200, RTX PRO 6000, RTX 5090 and DGX Spark — and ships one operating point.

> [!NOTE]
>
> `--mamba-full-memory-ratio` is the one sizing flag that matters for throughput
> on hybrid GDN models: the default (0.9) over-provisions the KV pool and silently
> clamps concurrency. Set your average request length in the
> [Mamba ratio calculator](#mamba-ratio-calculator) below; everything else follows
> the panels, and the computed value is pinned into the command.

> [!NOTE]
>
> Every cell above — RTX 5090, RTX PRO 6000 and DGX Spark, across all five
> checkpoints and every Speculative Decoding / Serving Strategy / SSM dtype
> combination — is measured on **v0.5.19**. That is 202 cells, each one served
> and scored on the full 1319-question GSM8K (93.18-95.15%). The serving
> envelope behind the pins is ISL 8192 / OSL 1024 at concurrency 1; throughput
> and acceptance-length numbers were not re-taken in that sweep.

### Mamba ratio calculator

Average request length

Input + output tokens — the ratio's only free parameter

Target concurrency

Requests in flight — sizes the explicit pin, not the ratio

Serving configuration (follows the Deploy panel and Playground)

state/slot  MB · KV/token
KB · the ratio prices
state slots per request; the pin counts S = ,
so  concurrent requests pin
slots.

> **How --mamba-full-memory-ratio is calculated**
>
> Hybrid GDN models split post-weight memory into a worst-case-reserved **GDN
> state pool** (sets the concurrency ceiling) and a paged **attention KV pool**,
> divided by `--mamba-full-memory-ratio`. Every parameter below except `L` and the
> target concurrency is read live from the Deploy panel and Playground selection;
> the balanced value is the per-request cost ratio:
>
> ```text Formula theme={null}
> ratio = (S + D) x state_bytes / (L x kv_bytes_per_token)
> ```
>
> * `S` — state slots per running request: `extra_buffer=5` (default),
>   `extra_buffer_lazy=4`, `no_buffer=3`, disabled radix cache `=1`. For the two
>   `extra_buffer` strategies, `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` frees one
>   slot, and `extra_buffer` frees one more with the overlap scheduler off; the
>   calculator reads both knobs.
> * `D` — verify intermediate states under speculative decoding:
>   `--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4)
>   and for DFLASH (8, DFlash2's block size);
>   `--speculative-dspark-block-size + 1` for DSPARK, where the block size falls
>   back to the draft checkpoint's `block_size` when the flag is omitted (7 for
>   `RadixArk/Qwen3.8-27B-DSpark`, so `D = 8`); 0 with speculation off or with
>   `--enable-linear-replayssm-spec`, which keeps the verify intermediates on a
>   fixed ring instead of per-request slots.
> * `state_bytes` — one state slot, from the fixed geometry
>   (48 GDN layers x 48 heads x 128 x 128 at `--mamba-ssm-dtype`, plus bf16 conv
>   state): 153.9 MB at fp32, 78.4 MB at bf16.
> * `kv_bytes_per_token` — 16 attention layers x GQA 4 x 256 x K+V:
>   32.8 KB at fp8, 65.5 KB at bf16.
> * `L` — average total request length in tokens: input + output.
>
> `--max-mamba-cache-size = target_concurrency x S` is the equivalent explicit
> pin and overrides the ratio; the calculator emits it alongside. `D` is not a
> term here: the engine divides the state pool by `S` alone and sizes the
> speculative verify buffer separately, so folding `D` into the pin would
> over-provision the pool. After boot, verify with the `max_running_requests`
> line in the server log — it should not be capped below your target concurrency.

## Playground

The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's documented launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

**Qwen3.8-27B** is a dense hybrid Gated Delta Networks (GDN) **vision-language**
model: a 27B causal language model paired with a vision encoder, with native
image and video understanding alongside text. SGLang serves it through the
Qwen3-VL path, so the vision tower is live on the recipes below.

The language model is 64 layers, laid out as 16 repeats of *3 × (Gated DeltaNet
→ FFN)* followed by *1 × (Gated Attention → FFN)* — 48 linear-attention layers
to 16 full-attention ones. Gated DeltaNet runs 48 value heads and 16 QK heads at
head\_dim 128; Gated Attention is GQA 24/4 at head\_dim 256 with a 64-dim rotary
slice. Hidden size is 5120 over a 17,408-dim FFN, and the checkpoint ships an
MTP head trained with multiple steps. Context is 262,144 tokens natively,
extensible to 1,000,000. The serving-relevant architecture is identical to
Qwen3.6-27B.

Thinking mode is on by default and can be disabled per request; reasoning depth
is tunable with `reasoning_effort`, and `preserve_thinking` retains reasoning
context from earlier messages.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Quantization</th>
      <th>Weights</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen3.8-27B</td>
      <td>BF16</td>
      <td>[Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B)</td>
    </tr>

    <tr>
      <td>Qwen3.8-27B-FP8</td>
      <td>FP8 (blockwise)</td>
      <td>[Qwen/Qwen3.8-27B-FP8](https://huggingface.co/Qwen/Qwen3.8-27B-FP8)</td>
    </tr>

    <tr>
      <td>Qwen3.8-27B-NVFP4 (FP4 head)</td>
      <td>NVFP4 W4A4 + FP8 projections, `lm_head` packed to FP4</td>
      <td>[RadixArk/Qwen3.8-27B-NVFP4](https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4)</td>
    </tr>

    <tr>
      <td>Qwen3.8-27B-NVFP4 (BF16 head)</td>
      <td>Same body, `lm_head` left dense in BF16</td>
      <td>[RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead](https://huggingface.co/RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead)</td>
    </tr>

    <tr>
      <td>Qwen3.8-27B-NVFP4 (NVIDIA)</td>
      <td>NVIDIA's ModelOpt export of the same W4A4 body, `lm_head` packed to FP4</td>
      <td>[nvidia/Qwen3.8-27B-NVFP4](https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4)</td>
    </tr>
  </tbody>
</table>

The two RadixArk NVFP4 exports differ only in the `lm_head`: one packs it to
FP4, the other leaves it dense in BF16. The dense head is \~1.7 GB larger on disk
and \~3.2 GB larger at runtime, so it is the harder of the two to fit — every
recipe on this page was measured against it, and the FP4-head cells reuse
those pins unchanged.

NVIDIA's own export is that same W4A4 body with that same FP4 head: identical
quantized-layer map (FP8 attention and GDN projections, NVFP4 MLPs), identical
tensor set, identical 21.9 GB on disk. On GB300, RTX PRO 6000 and DGX Spark its
cells reuse the FP4-head pins unchanged, and both SM12x grids have been
re-measured against this export on v0.5.19: all 16 overlay combinations per
card serve and score 94.01-95.00% (RTX PRO 6000) and 94.16-95.07% (DGX Spark)
on the full 1319-question GSM8K.

The RTX 5090 is measured too — all 15 overlay combinations it offers serve and
score 93.93-94.92% — and every winning launch command there is identical to the
FP4-head export's, which is the strongest form of the claim above. What the
32GB card does need is the draft-model rows pinning their own pools: those
recipes pin `--max-running-requests 1`, but nothing caps the pools to match, so
the KV pool sizes itself for 127,332 tokens against the 9,216 one
8192-in/1024-out request needs, and the engine's default split then leaves the
GDN state pool far short of the slots it needs once the draft model's weights
are counted against `--mem-fraction-static`. The DSPARK row therefore pins
`--max-total-tokens` and a measured `--mamba-full-memory-ratio`, as do DFLASH2
and MTP on the dense-lm\_head export at float32 state. Those pins override the
calculator's live value for the selections that carry them. The no-speculation
row needs none of it and runs at the pins shown.

The two RadixArk checkpoints declare `kv_cache_quant_algo: FP8`, so SGLang's
default `--kv-cache-dtype auto` already puts their KV pool in `fp8_e4m3`. The
NVIDIA export ships no `kv_cache_scheme`, so `auto` would leave its pool in
BF16 instead. Every recipe on this page pins `--kv-cache-dtype fp8_e4m3`
explicitly, so all three run the same `fp8_e4m3` pool regardless; the
difference only shows up if you switch the Playground's **KV Cache Precision**
row back to Auto.

## 2. Configuration Tips

* **SM120/SM121 (RTX PRO 6000 Blackwell, RTX 5090, DGX Spark)**: use `--attention-backend
  flashinfer`; `trtllm_mha` is SM100-only. MTP with the FlashInfer backend
  requires a FlashInfer build whose prefill `plan` accepts `uniform_q_len`
  (newer than 0.6.15.post1); otherwise run spec with `--attention-backend triton`.
  On DGX Spark the 128GB is unified memory shared with the host CPU, so all
  three checkpoints fit, and its cells reuse the RTX PRO 6000 recipe at
  `--mem-fraction-static 0.80` rather than a separate operating point. The one
  lower pin is the unified pool pricing the host's memory too: 0.85 of 128GB
  leaves \~8GB for the OS — exactly DGX OS earlyoom's SIGTERM threshold — and
  the first long prefill or boot-time graph capture dips under it and gets the
  scheduler killed with `exit code -15` and no traceback (`journalctl -u
  earlyoom` shows the kill). At 0.85, 15 of the 48 cells were killed that way,
  and which cells is margin noise; at 0.80 every cell served on every attempt.
  **Validated on SM121 / aarch64**: all 80 configurations (5 checkpoints x
  Speculative Decoding x Serving Strategy x Mamba SSM Dtype, DFLASH2 included)
  served on GB10 on `v0.5.19` at ISL 8192 / OSL 1024, concurrency 1, and each
  scored the full 1319-question GSM8K (93.18-95.15%); the float32 and bfloat16
  halves ran on two separate GB10 boxes. No throughput or acceptance-length
  numbers were re-taken. The sweep exercises the FlashInfer `plan` /
  `uniform_q_len` path above, which raised no arity error on that build. Three host quirks when reproducing
  on GB10: docker GPU access is CDI-only (`--device nvidia.com/gpu=all`, as no
  `nvidia` runtime is registered); `nvidia-smi` reports `Not Supported` for
  memory because it is unified with the CPU — gate a relaunch on `MemAvailable`
  in `/proc/meminfo` instead; and the BF16 checkpoint takes \~6.5 minutes just
  to load its 18 shards from NVMe, so budget \~10 minutes to READY before
  calling a boot hung.
* **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so an
  NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path,
  and all three NVFP4 cells are greyed out. The H200 recipes use 32768-token
  prefill chunks (SM90 prefill is fast enough that a big chunk barely stalls
  decode, unlike the SM120 guidance below), and the FlashInfer GDN prefill
  backend engages by default under them. `--attention-backend fa3` is a valid
  alternative, measured slightly faster at bs=1.
* **MTP**: `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` uses the
  in-checkpoint MTP head. (This recipe was originally documented with `NEXTN`,
  an alias of `EAGLE` — same algorithm.)
* **DSpark**: the trained draft model is a separate checkpoint — add
  `--speculative-algorithm DSPARK --speculative-draft-model-path
  RadixArk/Qwen3.8-27B-DSpark` (the Playground's Speculative Decoding card
  emits this pair). DSpark does **not** take
  `--speculative-num-draft-tokens`: its verify window is
  `--speculative-dspark-block-size` (gamma) **+ 1**, and gamma is auto-inferred
  from the draft checkpoint when the flag is omitted (7 for this checkpoint, so
  D = 8). That `D` is a term in the balanced ratio —
  `r = (S + D) x token_equiv / L`, where `token_equiv` is the state slot
  expressed in KV tokens, `state_bytes / kv_bytes_per_token` (4698 at fp32
  state / 2394 at bf16, over fp8 KV) — so DSpark needs a materially higher
  `--mamba-full-memory-ratio` than no-spec at the same `S`, and pinning a
  different gamma changes the ratio with it. MTP is the opposite case: with
  `--enable-linear-replayssm-spec` its draft intermediates move onto a fixed
  ring, so `D = 0` and the ratio returns to the no-spec value. The
  [calculator](#mamba-ratio-calculator) applies both rules.
* **DFlash2**: a trained block-diffusion draft in a separate checkpoint — add
  `--speculative-algorithm DFLASH --speculative-draft-model-path
  incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the
  draft's block size, and it is the `D` term in the ratio, same value as
  DSpark's). Runs on Ascend NPUs as well
  ([#35629](https://github.com/sgl-project/sglang/pull/35629)): the selector
  verify falls back to argmax there, matching what EAGLE and first-generation
  DFlash drafts (such as `z-lab/Qwen3-8B-DFlash-b16`) already do on NPU.
  Therefore, NPU currently guarantees lossless verification only for greedy
  requests; use `temperature=0` and `top_k=1`. Non-greedy requests log a warning;
  both draft proposal and target verification fall back to greedy, so their
  requested sampling distribution is not preserved. The selector projects
  candidates through the target `lm_head`,
  including quantized heads, so it runs on the NVFP4 checkpoint (whose head
  is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
  The Ascend comparison in #35629 used an A3 Series device with BF16 target weights,
  `--tp-size 2 --attention-backend ascend --mamba-ssm-dtype bfloat16 --mamba-radix-cache-strategy extra_buffer`, and disabled RadixCache for both
  baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2
  run added the three flags shown above.
  That comparison's accuracy used zero-shot GSM8K with greedy sampling,
  `max_new_tokens=2048`, 128 examples, and concurrency levels 1, 2, 4, 8
  and 16 — a different protocol from this page's own sweep below.
  Validation: every SM12x cell on this page is measured end to end on
  v0.5.19 — 202 cells over the five checkpoints, four speculative options, two
  serving tiers and two GDN state dtypes, full 1319-question GSM8K on each,
  93.18-95.15%. The RTX PRO 6000 and DGX Spark recipes need no changes. On the
  32GB RTX 5090 the panel applies the measured pins automatically: DFlash2 at
  `--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` — at 0.91 the
  pools fit but a 2048-token chunk's activations do not — DSpark at 0.88
  (bfloat16), 0.91 (float32) and 0.92 on the dense-lm\_head export, all three
  with their pools pinned and the last two also cutting the prefill chunk to
  1024 and 512, EAGLE at 0.93 (bfloat16) and 0.94 (float32), and
  no-speculation at 0.90.
  Whether float32 is available with a draft model depends on the `lm_head`: on
  the BF16-head export it is greyed out for both DSpark and DFlash2, since the
  dense head's \~3.2 GB leave no fp32 state pool that also clears prefill graph
  capture. The FP4-head export frees that headroom back — DSpark serves at 0.91
  and DFlash2 High-Throughput at 0.895 with `--mamba-full-memory-ratio 10`
  overriding the balanced value — and only DFlash2 Low-Latency stays out of
  reach, where five fp32 slots and a full request's KV never coexist. bfloat16
  remains the faster choice regardless: DFlash2 posts 4.92 ms median TPOT at an
  accept length of 4.29, the best result on this card.
* **Hardware fit**: FP8 weights \~28.5GB (not serviceable beyond bs≤2 on
  32GB cards); NVFP4 weights \~16.5GB (recommended for RTX 5090-class GPUs).
* `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per
  request from 5 slots to 4 at no accuracy cost. On small-VRAM cards (RTX 5090
  32GB) the state pool bounds concurrency long before KV does — prefer lowering
  `S` (lazy strategy, or `--disable-radix-cache` for S=1); the
  [calculator](#mamba-ratio-calculator) re-derives the ratio for the new `S`.
  The balanced ratio itself is VRAM-independent.
* `--mamba-ssm-dtype`: the GDN state slot is **153.9 MB at `float32`** (the
  checkpoint's declared precision) and **78.4 MB at `bfloat16`**, so bf16 roughly
  halves the state pool and hands the difference to KV — measured on an RTX 5090
  with no speculation, 97,280 KV tokens at bf16 against 68,588 at fp32. On 32GB
  cards it also decides whether a config fits at all: EAGLE needs
  `--mem-fraction-static 0.94` at fp32 but 0.92 at bf16. Speed is **not** a
  one-way trade — with speculative decoding fp32 sometimes wins (NVFP4 + EAGLE:
  152.9 vs 144.5 tok/s/user) and sometimes loses (FP8 + EAGLE: 106.3 vs 116.1);
  measure both for your quantization. Treat
  `bfloat16` as an accuracy gate and validate it for your workload. On SM120
  both precisions run the Triton linear-attn prefill path — the FlashInfer GDN
  prefill fast path gates on SM100, where its validated domain is in fact a
  bf16 state pool — so no dtype forces an extra flag here. One interaction to
  know: `--enable-linear-replayssm-spec` auto-selects fp32 state when
  `--mamba-ssm-dtype` is unset, and an explicit non-fp32 value logs a
  state-drift warning at boot. The SSM dtype row always emits the flag
  explicitly, so the bf16 + EAGLE cells run with that warning — accounted for
  in their validation.
* `--chunked-prefill-size 2048`: decode steps stall behind each prefill chunk
  on hybrid GDN models, and 8192-token chunks stall them \~600ms at a time.
  2048 keeps decode inter-token latency smooth under mixed load and also
  improves single-wave TTFT.

## 3. Agent Harnesses

Agent harnesses drive the model through the OpenAI-compatible endpoint — or, for
Claude Code, through SGLang's Anthropic-compatible one — so any of them works
once three things line up.

**The parsers ship in the command.** Every recipe above carries
`--reasoning-parser qwen3 --tool-call-parser qwen3_coder`, because without them a
harness receives tool calls as raw text instead of structured `tool_calls`. The
**Parsers** card in the [Playground](#playground) is therefore an opt-out — both
chips start on, and turning one off strips its flag.

`qwen3_coder` is the right tool-call parser for this checkpoint: its chat
template instructs the model to reply with an inner `<function=…>` /
`<parameter=…>` block nested in `<tool_call></tool_call>`, which is exactly what
that parser decodes. The Hermes parser (`--tool-call-parser hermes`) reads a
*different* payload — bare JSON inside `<tool_call>` — so pointing a Hermes-format
harness at this model without switching the flag yields tool calls that never
parse. `--reasoning-parser qwen3` matches the template's `enable_thinking`
toggle, which defaults to on.

**Endpoint and model id.** The base URL is `http://<host>:30000/v1`. The `model`
string a harness sends must equal the server's `--model-path` — the OpenAI
`/v1/models` name defaults to it — unless you override it with
`--served-model-name`, which is usually worth doing to keep harness configs short.

SGLang also serves an Anthropic-compatible `/v1/messages`, which is what
[§3.3](#3-3-claude-code) uses. It converts each request to the OpenAI shape,
hands it to the same chat-serving path, and converts the response back — so the
parser flags above apply there identically.

**Auth.** `--api-key` is unset by default, so the server accepts unauthenticated
requests. Harnesses that insist on a key can send any placeholder; set
`--api-key` on the server if the endpoint is reachable beyond localhost.

### 3.1 OpenCode

[OpenCode](https://opencode.ai/docs/providers/) reaches a self-hosted endpoint
through a provider entry in `opencode.json`.

> **Register SGLang as an OpenCode provider**
>
> Store the credential first — pick **Other**, give the provider an id, and enter
> any placeholder when the server has no `--api-key`:
>
> ```bash Command theme={null}
> opencode
> /connect
> ```
>
> Then declare the provider in `opencode.json`:
>
> ```json Config theme={null}
> {
>   "$schema": "https://opencode.ai/config.json",
>   "provider": {
>     "sglang": {
>       "npm": "@ai-sdk/openai-compatible",
>       "name": "SGLang (Qwen3.8-27B)",
>       "options": {
>         "baseURL": "http://localhost:30000/v1"
>       },
>       "models": {
>         "RadixArk/Qwen3.8-27B-NVFP4": {
>           "name": "Qwen3.8-27B NVFP4"
>         }
>       }
>     }
>   }
> }
> ```
>
> `npm` selects the transport — `@ai-sdk/openai-compatible` is the one for a plain
> OpenAI-shaped endpoint. `apiKey` is optional and takes a `"{env:VAR_NAME}"`
> reference rather than a literal. The `models` keys are the ids sent on the wire,
> so they must match the served model name. Confirm with `/models`.

### 3.2 Pi

[Pi](https://pi.dev/docs/latest/custom-provider)
(`@earendil-works/pi-coding-agent`) registers providers from an extension rather
than a config file.

> **Register SGLang as a Pi provider**
>
> ```javascript Extension theme={null}
> pi.registerProvider("sglang", {
>   baseUrl: "http://localhost:30000/v1",
>   api: "openai-completions",
>   apiKey: "$SGLANG_API_KEY",
>   models: [
>     {
>       id: "RadixArk/Qwen3.8-27B-NVFP4",
>       name: "Qwen3.8-27B",
>       reasoning: true,
>       input: ["text", "image"],
>       cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
>       contextWindow: 262144,
>       maxTokens: 32768,
>     },
>   ],
> });
> ```
>
> `api: "openai-completions"` is what selects the OpenAI-compatible transport, and
> `apiKey` takes a `$ENV_VAR` reference rather than a literal. `contextWindow` is
> the checkpoint's native 262,144; set `maxTokens` to whatever output cap you want
> per turn. Confirm registration with `pi --list-models`.

### 3.3 Claude Code

Claude Code speaks the Anthropic API, so it points at SGLang's `/v1/messages`
rather than the OpenAI endpoint.

> [!WARNING]
>
> Anthropic documents that routing Claude Code to non-Claude models through a
> gateway is **not supported**. The wiring below works because SGLang implements
> the Anthropic message format, but it sits outside what Claude Code is tested
> against — expect newer Claude Code features to degrade or fail.

> **Point Claude Code at SGLang**
>
> `ANTHROPIC_BASE_URL` is the server origin — Claude Code appends `/v1/messages`
> itself, so leave the `/v1` suffix off:
>
> ```bash Command theme={null}
> export ANTHROPIC_BASE_URL=http://localhost:30000
> export ANTHROPIC_AUTH_TOKEN=placeholder
> ```
>
> The two credential variables travel in different headers:
> `ANTHROPIC_AUTH_TOKEN` goes out as `Authorization: Bearer`, `ANTHROPIC_API_KEY`
> as `x-api-key`. Either satisfies a server started without `--api-key`; with
> `--api-key` set, pick the variable matching the header your server reads. A
> credential variable also takes precedence over a saved claude.ai login for that
> session.
>
> The same pair can live in a settings file instead, which persists across shells
> and wins over a shell export:
>
> ```json Config theme={null}
> {
>   "env": {
>     "ANTHROPIC_BASE_URL": "http://localhost:30000",
>     "ANTHROPIC_AUTH_TOKEN": "placeholder"
>   }
> }
> ```
>
> Run `/status` in Claude Code to confirm which base URL and credential source the
> session picked up.

### 3.4 Hermes Agent

[Hermes Agent](https://github.com/NousResearch/hermes-agent) (Nous Research, MIT)
selects a self-hosted endpoint through its setup wizard or its config file.

> **Point Hermes Agent at SGLang**
>
> ```bash Command theme={null}
> hermes model
> # choose "Custom endpoint (self-hosted / VLLM / etc.)", then enter the
> # base URL, an API key (blank for a local server) and the model name
> ```
>
> Equivalently, in `~/.hermes/config.yaml`:
>
> ```yaml Config theme={null}
> model:
>   default: RadixArk/Qwen3.8-27B-NVFP4
>   provider: custom
>   base_url: http://localhost:30000/v1
>   api_key: ""
>   context_length: 262144
> ```
>
> For several endpoints at once, declare them under `providers:` and switch with
> `/model custom:<name>` mid-session:
>
> ```yaml Config theme={null}
> providers:
>   workstation:
>     api: http://localhost:30000/v1
>   server:
>     api: https://gpu-host.internal:30000/v1
>     key_env: SGLANG_API_KEY
> ```

## 4. Semantic Decisions

Qwen3.8-27B can answer typed questions about an input with a probability for every option through `/v1/decisions`, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains `/v1/decisions`, install a [nightly build](/docs/get-started/install#nightly-builds). With a server started by `sglang serve --model-path Qwen/Qwen3.8-27B --host 127.0.0.1 --port 30000`:

```python Example theme={null}
import requests

response = requests.post(
    "http://127.0.0.1:30000/v1/decisions",
    json={
        "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.",
        "questions": [
            {
                "id": "team",
                "type": "choice",
                "question": "Which team should handle this ticket?",
                "options": [{"name": "billing"}, {"name": "technical"}, {"name": "sales"}],
            },
            {"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."},
        ],
    },
    timeout=60,
)
response.raise_for_status()
answers = response.json()["answers"]
print(answers["team"]["choice"], answers["team"]["probabilities"])
print(answers["urgent"]["probabilities"]["yes"])
```

See [Decision models](/docs/supported-models/decision_models) for the request reference, how the probabilities are computed, and the limits.


# Qwen3.8-Flash-Next
Source: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next

Deploy Qwen3.8-Flash-Next with SGLang — day-0 recipes for Qwen's 176B-parameter (6B active) GDN + QSA hybrid Mixture-of-Experts preview of the Qwen4 architecture, on NVIDIA and AMD.

## Deployment

> **Install SGLang**
>
> Use an SGLang build that includes Qwen3.8-Flash-Next support (v0.5.22 or later).
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:latest
> ```
>
> The deployment panel can render a complete `docker run` command for the selected hardware and options. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup.

Pick your hardware + quantization to generate the launch command.

> **DGX Spark notes (1x GB10 with the N-gram table on NVMe, or 2x GB10 TP=2)**
>
> The NVFP4 checkpoint is 126 GiB (78 GiB of experts and dense weights plus a 47.7 GiB FP8 N-gram table), so it does not fit one DGX Spark's 128 GB of unified memory, and `--ple-offload-embedding` does not help there: on GB10 the "offloaded" pinned-host table comes out of the same pool as the GPU weights. The two-node shape is TP=2 across two Sparks over the ConnectX-7 200GbE link, with `--no-ple-offload-embedding` keeping the table GPU-resident and sharded (\~65 GB of weights per node). A single Spark works only with the table file-backed on NVMe (see the Single Spark bullet below).
>
> * **Launch order.** Start rank 1 first, then rank 0 within a few seconds. When re-launching, stop both ranks and confirm nothing listens on the rendezvous port before starting again: a new rank 1 attaching to a stale rank 0 store fails with a gloo "Connection reset by peer".
> * **NCCL.** The cross-node decode CUDA-graph deadlock seen on an earlier DGX Spark stack was on NCCL 2.28.x; both builds these images load have been verified here for TP=2 across two Sparks: `dev-qwen38-next-local` runs its pip NCCL 2.29.7 (a system 2.28.3 is also present but not loaded), `qwen38flashnext` its 2.30.7. Confirm with the startup log line `sglang is using nccl==…`, which reports the library actually loaded. If your Sparks also have a slower management NIC, pin `NCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` to the 200GbE interface; with `--dist-init-addr` on the 200GbE address the verified runs picked it without pinning.
> * **Memory.** Both cells run `--mem-fraction-static 0.85`, which leaves \~8–12 GiB of host memory free per node under load (measured through GSM8K at full concurrency and a 100k-token prefill). Two precautions for long-context work beyond that: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` stops variable-shape chunked-prefill buffers from fragmenting the caching allocator at 200k+ contexts, and a host-side watchdog that kills the server when `MemAvailable` drops below a few GiB, because a unified-memory exhaustion can take the whole box down and needs a power cycle to recover.
> * **Concurrency.** The hybrid GDN/QSA model reserves mamba state slots per running request (5 with the default `extra_buffer` radix strategy, 4 with `extra_buffer_lazy`), and the scheduler caps `--max-running-requests` to what the mamba pool admits; read the effective value from the startup log, not from `/get_server_info`. The cells pin `--max-mamba-cache-size` to requests × slots (24 × 5 = 120 for low latency, 96 × 4 = 384 for high throughput); raising concurrency further takes memory from the KV pool one-for-one.
> * **Flags.** The cells are the model card's TP=2 recipe without `--mamba-track-interval 64`: the default of 256 tokens satisfies the constraints (a multiple of the 64-token page, at least the 4 draft tokens) and leaves a \~40% larger KV pool (1.48M tokens at 24 concurrent with MTP, 1.07M at 96 without), at the cost of coarser prefix-cache reuse of the recurrent state. `--trust-remote-code` is not needed; the architecture is native to SGLang.
> * **Measured.** 100k-token prefill at 2,400–2,840 tok/s; MTP accept length 3.5–3.7 of 4 draft tokens on non-thinking output (lower, \~2.5, on thinking output).
> * **Single Spark (file-backed PLE table).** One GB10 holds the checkpoint only if the 47.7 GiB FP8 N-gram table leaves memory entirely: with **PLE Offload = On (NVMe file)** (`--ple-offload-embedding --ple-offload-backend file`, from [#37068](https://github.com/sgl-project/sglang/pull/37068), merged into `qwen4-main-squashed`) SGLang creates a sparse 47.7 GiB file under `$SGLANG_CACHE_DIR/ple/<model>` (relocate with `--ple-offload-dir`; put it on the local NVMe and mount that directory into the container), fills it on boot, and the gather kernel reads rows through the host page tables; the table's resident set stays at 0 while serving, with hot pages in page cache (capped at 8 GiB by `SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB`). The 78.3 GiB of experts and dense weights stay resident (the server logs \~80 GB after load, which also counts the CUDA context and allocator overhead) and `--mem-fraction-static 0.85` leaves \~12–18 GB for the pools, so concurrency is pinned low: **8 requests with MTP** (40 fp32 mamba slots of \~113 MB each) or **24 without** (96 slots on `extra_buffer_lazy`). Measured on `4ccff141db`: MTP cell 27.5 tok/s single-stream (TPOT 33.6 ms) and 71.7 tok/s output at 8; no-MTP cell 15.9 tok/s single-stream and 83 tok/s output at 24; host memory never below 10 GiB. **Boot-time caveat:** every boot rewrites the whole table through the mapping; on an already-populated file that is a read-modify-write per 4 KiB page with readahead disabled (`MADV_RANDOM`), \~17 MB/s and \~55 minutes; on a fresh sparse file it fills at GB/s and the boot takes \~10 minutes. Until that is fixed upstream, delete the previous `ple_table_*.bin` before each boot.
> * **NVIDIA export (NVFP4 (NVDA)).** `nvidia/Qwen3.8-Flash-Next-NVFP4` is a ModelOpt MIXED\_PRECISION checkpoint (NVFP4 experts, FP8 N-gram table, FP8 block-scaled MTP experts) and needs the loader from [#38121](https://github.com/sgl-project/sglang/pull/38121), merged into `qwen4-main-squashed` (the branch the Python install path builds and the `lmsysorg/sglang:dev-qwen38-next-local` image ships); the `qwen38flashnext` image predates it and cannot load this export. Do not pass `--quantization` for it (it resolves to `modelopt_mixed`), and pass `--moe-runner-backend flashinfer_cutlass` explicitly: the mixed-precision auto-default picks `flashinfer_trtllm` on GB10, which the NVFP4 MoE method rejects at autotune. Its MTP experts are FP8 block-scaled with 128-wide blocks and cannot be split across two ranks (640 / 2 = 320), so the two-node low-latency cell reads the MTP draft from the RadixArk export, the same trained head kept in BF16 there. On a single Spark (TP=1, file-backed table) nothing is sharded, so the export's own MTP head loads directly. The single-Spark NVDA cells use the same pins as the RDXA ones (8 requests with MTP, 24 without); the smaller fp8 draft leaves a 174k-token KV pool with MTP, against 93k for the RadixArk cell. Measured on the `qwen4-main-squashed` tip `4ccff141db` (#38121 merged) at TP=2: 47.4 tok/s single-stream with MTP, 253 tok/s output at 96 concurrent without, the same as the RadixArk export.

> **RTX PRO 6000 notes (1x 96 GB, TP=1)**
>
> The NVFP4 checkpoint fits a single 96 GB RTX PRO 6000 Blackwell (SM120) only with the 47.7 GiB FP8 N-gram table in **CPU pinned memory** (`--ple-offload-embedding`, the forced setting of the PLE Offload row on this hardware): the other 78 GiB of the checkpoint loads onto the card. After the loader's temporaries are collected, 74.7 GiB stays resident without speculative decoding and 81.8 GiB with it (the draft head is 0.5 GiB; the rest is memory the loader still holds), leaving 19.4 GiB and 12.3 GiB of the 94.2 GiB the process can use. `--mem-fraction-static` keeps (1 − fraction) × 94.2 GiB of that as runtime slack and the pools take the rest: 6.6 GiB of slack and 12.8 GiB of pools at 0.93, 3.8 GiB and 8.3 GiB at 0.96. (The 81 GiB "mem usage" in the load log is the delta before that collection.) The host needs ≥ 64 GB of free RAM for the locked table (plus page cache for the checkpoint) and Docker needs `--ulimit memlock=-1`, or the pinned allocation fails.
>
> * **Concurrency.** The hybrid model reserves mamba state slots per running request, and the scheduler caps `--max-running-requests` to what the state pool admits: with the default `extra_buffer` strategy (5 fp32 slots per request at 0.109 GiB) that is 3 requests with MTP and 12 without on this card. The cells keep prefix caching on and get to 16 / 64 with three levers: `--mamba-radix-cache-strategy extra_buffer_lazy` (4 slots per request), `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` (3; a running request's prefix state is no longer pinned in the radix tree during decode, so it can be evicted, which trades cache retention, not numerics), and `--mamba-ssm-dtype bfloat16` (0.055 GiB per slot). `--max-mamba-cache-size` is pinned to requests × 3 (48 / 192). The ceilings from the pool arithmetic are \~20 requests with MTP (each request carries 4 intermediate draft states) and \~70 without; turning prefix caching off (`--disable-radix-cache`, 1 slot per request) reaches 24 / 96 on this card and was verified too, but re-prefills every prompt.
>
> * **Linear-attention kernels.** Left on auto. On SM120 the server resolves decode, prefill and verify to triton — the same GDN kernels the fp32 runs and the DGX Spark cells use; the bf16-state FlashInfer GDN auto-default applies only to SM100. An explicit `--linear-attn-decode-backend flashinfer` also runs on this card and measured the same TPOT (within 0.3 ms) and accuracy, so there is nothing to gain from pinning either.
>
> * **Memory headroom.** With the state pool pinned, the KV pool absorbs the rest of the static budget, so `--mem-fraction-static` is what sets the activation headroom. 4096-token prefill chunks of ShareGPT-length prompts peak 1.5–2.6 GB above the post-graph-capture level, and cells left with 2.4 GB free OOMed in the GDN short-conv during prefill. The cells keep ≥ 4 GB free after graph capture and ≥ 2.3 GB at the measured peak: 0.96 with MTP (78k-token KV pool, \~4.9k per request at 16) and 0.93 without (98k tokens, \~1.5k per request at 64; 0.94 gives 138k tokens with 2.3 GB at peak). `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is set on both.
>
> * **Accuracy.** Full GSM8K, 1,319 questions, on the `lmsysorg/sglang:dev-qwen38-next-local` image, in two protocols. Chat completions API with thinking off, greedy, an 8,192-token budget, answer parsed from a final "The answer is N" line (the protocol of the DGX Spark rows, and the figure on the benchmark card): 96.9% for the MTP cell and 96.9% for the no-MTP cell (RadixArk export); 97.3% and 97.0% for the NVIDIA export. `python -m sglang.test.run_eval --eval-name gsm8k --num-examples 1319 --max-tokens 16384` (5-shot, greedy, last-number scorer, chat template with thinking on): 97.72% / 97.79% (RadixArk), 97.41% / 97.72% (NVIDIA). Measured on the `4ccff141db` build, which carries the #36811 and #38290 router fixes.
>
> * **Measured (1024-in / 256-out random prompts, `ignore_eos`).** MTP vs no-MTP at 1 request: TTFT 115 vs 116 ms median, TPOT 5.9 vs 11.4 ms, 148 vs 83 tok/s. At 16: TPOT 19.3 vs 25.6 ms, 613 vs 524 tok/s. No-MTP at 64: TPOT 55 ms, 861 tok/s, 3.4 req/s. ShareGPT chat (thinking off, ≤ 512 output tokens): 785 tok/s at 16-way with MTP, 1,258 tok/s at 64-way without. MTP accept length 3.3 of 4 on GSM8K / random prompts, 2.9 on long-form ShareGPT answers. On the `dev-qwen38-next-local` image (`4ccff141db`): 6.0 / 11.44 ms TPOT at 1 request (MTP / no MTP), 685 / 561 tok/s at 16, 909 tok/s at 64 without MTP, accept length 3.16.
>
> * **NVIDIA export (NVFP4 (NVDA)).** `nvidia/Qwen3.8-Flash-Next-NVFP4` runs on this card with the same shape, pools and flags as the RadixArk cells on the `lmsysorg/sglang:dev-qwen38-next-local` image (it carries the loader from [#38121](https://github.com/sgl-project/sglang/pull/38121); `qwen38flashnext` cannot load this export). Do not pass `--quantization`: the checkpoint resolves to `modelopt_mixed`. Low latency keeps the in-checkpoint MTP head: at TP=1 its fp8 block-scaled experts need no sharding, and #38121 runs them on triton under the `flashinfer_cutlass` pin. The RadixArk BF16 draft measured the same here (accept 3.33 vs 3.31, TPOT 18.5 vs 19.1 ms at 16), so the cell stays single-checkpoint. Verified on that image (`4ccff141db`): full GSM8K in the Accuracy bullet above; 6.02 ms TPOT at 1 request with MTP, 675 output tok/s at 16; 906 output tok/s at 64 without, the same as the RadixArk export. The smaller fp8 draft leaves a \~170k-token KV pool at 16 concurrent.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

* **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
* **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
* **Container image.** Run the commands inside the image listed for each configuration. Several are SGLang nightly images, so flag names follow that image.

## 1. Model Introduction

**Qwen3.8-Flash-Next** is a multimodal Mixture-of-Experts model released as an early preview of the architecture Qwen4 is being built on — the same role Qwen3-Next played for Qwen3.5, whose hybrid Gated DeltaNet + Gated Attention design then carried through the Qwen3.5, Qwen3.6, Qwen3.7 and Qwen3.8 series. Qwen is publishing the architectural changes ahead of the full Qwen4 family so the community can evaluate them independently.

It has **176B total parameters — 51B of which is an N-gram embedding table — and 6B active per token**. Against Qwen3.7-Plus it cuts both training and inference cost substantially (training takes roughly 1/9 as much) while holding comparable overall quality. It takes text and images in, and the hosted production variant is served as `qwen3.8-flash` on QwenCloud.

The upgrades span four axes:

* **Attention — GDN + QSA hybrid.** Three of every four layers use Gated DeltaNet, which compresses history into a fixed-size recurrent state; the fourth is global attention running **Qwen Sparse Attention (QSA)**. A lightweight indexer aggregates the sequence into micro-blocks, scores importance at block level, and selects the relevant regions — so the indexing overhead shrinks along with the attention itself. Unlike approaches that reuse indices across layers, QSA compresses independently per layer, which suits an architecture that interleaves GDN and attention. Qwen measures up to 10.2× prefill and 6.6× decode speedups for the QSA attention kernel at 1M tokens.
* **Residual — Gated Residual (GR).** The single residual stream widens into four parallel branches, with an element-wise dynamic gate controlling how much each layer reads from and writes back to each branch. Qwen reports one branch naturally becoming a long-range bus. The gate also suppresses activation outliers, and the residual state can be held in FP8.
* **Embedding — N-gram Embedding.** Lookups keyed on the local context (current token plus a few preceding ones) rather than a single token, adding 51B parameters at almost no extra per-token compute. Because lookup addresses are known in advance, the table can live in host memory and be prefetched asynchronously alongside model compute. The final model uses a single such layer near the start of the network.
* **Optimization — Muon.** Muon for the genuine 2-D linear maps (attention, GDN and MoE expert weights), AdamW for embeddings, the MoE router and GR's low-rank parameters, with fused QKV / SwiGLU / GDN projections split before orthogonalization. The scaling law was refit for the new architecture, and batch-size warmup was dropped — it cost 18.8% more optimizer steps without improving the result.

Carried over from Qwen3-Next and refined through the Qwen3.5–Qwen3.8 series: an ultra-sparse MoE (large expert pool, few routed experts per token plus one shared expert) with global load balancing; a **multi-step-trained MTP module**, whose own full-attention layers are QSA as well, which is what keeps speculative acceptance high in practice; and the training-stability set of zero-centered RMSNorm with weight decay on norm weights, attention output gating, and normalized MoE router initialization.

**Context length:** 262,144 native, extensible to 1,000,000 tokens with YaRN. **License:** see [the model card's LICENSE](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE).

**Recommended generation:** Qwen has not published sampling recommendations for this preview. SGLang applies the checkpoint's own `generation_config.json`, so leave `temperature` / `top_p` unset unless you have a measured reason not to.

Each precision is its own repository:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Precision</th>
      <th>Repository</th>
      <th>Where it runs</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BF16**</td>
      <td>[Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)</td>
      <td>H200, B200, B300, GB300, MI350X, MI355X</td>
    </tr>

    <tr>
      <td>**FP8**</td>
      <td>[Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8)</td>
      <td>H200, B200, B300, GB300, MI350X, MI355X</td>
    </tr>

    <tr>
      <td>**NVFP4 (RDXA)**</td>
      <td>[RadixArk/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4)</td>
      <td>B200, B300, GB300, RTX PRO 6000, 1x or 2x DGX Spark (Blackwell only)</td>
    </tr>

    <tr>
      <td>**NVFP4 (NVDA)**</td>
      <td>[nvidia/Qwen3.8-Flash-Next-NVFP4](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4) (ModelOpt MIXED\_PRECISION)</td>
      <td>RTX PRO 6000, 1x or 2x DGX Spark (needs [#38121](https://github.com/sgl-project/sglang/pull/38121): the `dev-qwen38-next-local` image or the Python install path)</td>
    </tr>
  </tbody>
</table>

**Resources:** [Qwen's announcement](https://qwen.ai/blog?id=qwen3.8-flash-next).

## 2. Advanced Usage

> [!NOTE]
>
> The `model` argument in the examples below is the BF16 repo id. Every precision is a **separate repo**, so `model` has to be the checkpoint the server was actually launched with — `…-Flash-Next-FP8` or `…-Flash-Next-NVFP4`. The Deploy panel's cURL snippet always shows the right id for the cell you have selected.

### 2.1 Reasoning

Qwen3.8-Flash-Next **always** reasons — thinking cannot be turned off. `--reasoning-parser auto` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) lets SGLang pick the matching parser from the checkpoint's chat template, and splits the thinking into `reasoning_content`, leaving `content` as the answer alone. The resolved name is logged at startup if you want to pin it explicitly later.

Depth is requested with `reasoning_effort`. Qwen documents `xhigh` (the default), `medium` and `low` for the hosted model; SGLang forwards whatever you pass into the checkpoint's chat template.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="Qwen/Qwen3.8-Flash-Next",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     reasoning_effort="xhigh",  # xhigh (default) | medium | low
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update — a sample transcript will be added here once the weights are public.
> ```

### 2.2 Tool Calling

Add `--tool-call-parser auto` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. As with the reasoning parser, SGLang resolves the concrete detector from the chat template at startup. No Deploy cell sets it, so this is an opt-in: add the flag to the generated command, or flip the chip in the Playground.

Because this model always thinks, the final assistant turn can put text in `reasoning_content` rather than `content` — print both so a bare `None` doesn't mislead you.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "location": {"type": "string", "description": "The city name"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> resp = client.chat.completions.create(
>     model="Qwen/Qwen3.8-Flash-Next",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update — a sample transcript will be added here once the weights are public.
> ```


# Flint
Source: https://docs.sglang.io/cookbook/autoregressive/Recursive/Flint

Deploy Recursive's Flint with SGLang — a 40B Mixture-of-Experts model for fast, local agentic coding with a 262K context window — on NVIDIA H200, B200, B300, GB300, and DGX Spark.

## Deployment

> **Install SGLang**
>
> Flint runs on the `lmsysorg/sglang:dev-flint` Docker image:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-flint
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). The **Docker** toggle in the command panel below produces the full `docker run` command; the **Python** toggle produces the inner `sglang serve ...` to run inside the container.

Pick your hardware to generate the launch command.

## 1. Model Introduction

**[Flint](https://huggingface.co/RecursiveSI/flint)** is a new open-weight **40B Mixture-of-Experts** model from Recursive, built for fast, local AI coding. Flint was developed using Recursive's Eureka Machine for automated AI research and is released under Apache 2.0.

<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Total params</th>
      <th>Context</th>
      <th>Use</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[Flint](https://huggingface.co/RecursiveSI/flint)**</td>
      <td>40B (MoE)</td>
      <td>262,144</td>
      <td>Fast, local agentic coding</td>
    </tr>
  </tbody>
</table>

**License:** Apache 2.0

**Resources:** [Hugging Face](https://huggingface.co/RecursiveSI/flint).

## 2. Advanced Usage

### 2.1 Reasoning

Launch with `--reasoning-parser flint` (included in the commands above). Thinking is separated into `message.reasoning_content`, and the final answer is returned in `message.content`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="flint",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(message.reasoning_content)
> print("=============== Answer ===============")
> print(message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> =============== Reasoning ===============
> We need answer simple. Need maybe show calculation. User asks "What is 15% of 240?" We can answer. Need be careful maybe just 36. Let's craft concise.
> =============== Answer ===============
> 15% of 240 is **36**.
> ```

### 2.2 Tool Calling

Launch with `--tool-call-parser flint` (included in the commands above) to surface structured tool calls via `message.tool_calls`.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> import json
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"city": {"type": "string"}},
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="flint",
>     messages=[{"role": "user", "content": "What's the weather in Paris?"}],
>     tools=tools,
> )
>
> message = response.choices[0].message
> print("=============== Reasoning ===============")
> print(getattr(message, "reasoning_content", None))
> print("=============== Content ===============")
> print(message.content)
> print("=============== Tool Calls ===============")
> for call in message.tool_calls or []:
>     print(call.function.name, json.loads(call.function.arguments))
> ```

> **Example Output**
>
> ```text Output theme={null}
> =============== Reasoning ===============
> The user asked for the weather in Paris, and I have a get_weather tool available that takes a city parameter. I should call it with city 'Paris' to get the current weather.
> =============== Content ===============
> I'll get the current weather for Paris for you.
> =============== Tool Calls ===============
> get_weather {'city': 'Paris'}
> ```


# Dots3-Note
Source: https://docs.sglang.io/cookbook/autoregressive/RedNote/Dots3-Note

Deploy RedNote dots3.note with SGLang — a native multimodal omni model (MoE ViT + Whisper-derived audio encoder + native video flattening) on the dots3 hybrid MLA/SWA language model, with DSA and full-sharing MTP speculative decoding.

## Deployment

> **Install SGLang**
>
> dots3.note support is in [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829). Until that PR is included in a tagged SGLang release, install from a build that contains the PR.
>
> **Python (pip / uv)**
>
> ```bash Command theme={null}
> pip install -U uv
> uv venv --python 3.12 && source .venv/bin/activate
>
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> git fetch origin pull/33829/head && git checkout FETCH_HEAD
> uv pip install --prerelease=allow -e python
> ```
>
> Then run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-dots3-note
> ```
>
> This image packages SGLang with the dots3.note support from PR #33829 and is the recommended way to deploy until the PR lands in a tagged SGLang release — it saves you from building the branch yourself.
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick the hardware and the checkpoint precision. The recipe runs on a single 8-GPU Hopper node with DP8 attention × TP8 × EP8 and DeepEP as the MoE all-to-all transport. Blackwell is not supported yet.

**Precision** — selects the MoE path, not just the weights. The BF16 cells pin `--moe-runner-backend deep_gemm` with BF16 DeepEP dispatch output (JIT DeepGEMM is enabled via `SGLANG_ENABLE_JIT_DEEPGEMM=1`). The FP8 cells leave both at `auto` and let SGLang resolve the runner from the checkpoint's quantization config.

**Spec Decode** — NEXTN is on in every cell: 3 draft steps, 4 draft tokens per step, and the draft model path pointing at the target checkpoint itself. dots3's MTP layer is full-sharing — it carries the dots3 sliding-window attention geometry and reuses the target LM head — so no separate draft checkpoint is needed. Target verification and draft extension run on the paged, absorbed SWA-MLA FA3 path.

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations signed off on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

dots3.note is RedNote's native multimodal omni model, built on the dots3 language model. It accepts text, image, audio, and native video input.

* **Native multimodality** — a custom MoE vision transformer and a Whisper-derived audio encoder run in-process with the language model, loaded from the same checkpoint directory. Image and audio placeholders are expanded by a model-specific processor.
* **Native video pipeline** — the server jointly samples and interleaves frames, timestamps, and audio segments under a token budget, reproducing the training-time flattening algorithm. A generic uniform-frame video processor would silently change the modality ordering and token allocation (inference/training mismatch), so the pipeline is vendored into the serving path.
* **Hybrid attention** — dots3 combines MLA with full-attention and sliding-window layers of different geometry, attention gates, and optional DSA indexing on full-attention layers.
* **MTP speculative decoding** — a full-sharing MTP/NextN architecture exposes one recursively shared, SWA-shaped MTP layer and shares the target LM head.

**Available checkpoints:**

* **BF16**: [dots-studio/dots3-note-prev](https://huggingface.co/dots-studio/dots3-note-prev)
* **FP8**: [dots-studio/dots3-note-prev-fp8](https://huggingface.co/dots-studio/dots3-note-prev-fp8)

**Resources:** [Hugging Face (BF16)](https://huggingface.co/dots-studio/dots3-note-prev) · [Hugging Face (FP8)](https://huggingface.co/dots-studio/dots3-note-prev-fp8) · [SGLang PR #33829](https://github.com/sgl-project/sglang/pull/33829)

## 2. Configuration Tips

**Hybrid KV pool.** dots3 mixes full-attention and sliding-window layers, and its MTP draft layer is an ordinary SWA layer — not a full-attention one. SGLang sizes the pool accordingly, with `--swa-full-tokens-ratio 0.03` setting the ratio of SWA-layer KV tokens to full-layer KV tokens (`swa_tokens ≈ full_tokens × ratio`). Lower it when long full-attention contexts dominate and the full pool fills first; raise it when the SWA pool is the bottleneck.

**MoE runner.** Leave the runner at the cell default: `deep_gemm` for BF16 checkpoints, `auto` for quantized ones. DeepEP is the all-to-all transport in every cell (`--moe-a2a-backend deepep`, dispatch tokens per rank tuned via `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128`).

**Attention backend.** FA3 across the board: prefill, decode, and draft (`--prefill-attention-backend fa3 --decode-attention-backend fa3 --speculative-draft-attention-backend fa3`) with `--page-size 64`. MTP target verification uses FA3's absorbed SWA-MLA fallback, which consumes the same paged latent KV view as decode.

**DSA.** DSA indexing on full-attention layers is on by default. To disable it, add `--json-model-override-args '{"index_topk":null}'`.

**CUDA graphs.** The cells enable decode-side CUDA graphs only (`--cuda-graph-backend-decode full --cuda-graph-backend-prefill disabled`, max batch size 32) and are sized for GPUs with at least 120 GiB of memory. On smaller GPUs, switch to `--cuda-graph-backend-decode disabled` (and expect `--deepep-mode normal` to be the better fit).

**Context length.** `--context-length 524288` is the model's window. Like other SGLang models, it bounds the longest accepted request; it does not size the KV pool.

**Language-only mode.** Add `--language-only` to skip constructing the vision and audio towers entirely — the freed memory goes to the language model. This is also the language role of an encoder/LLM-disaggregated (EPD) deployment; see [EPD](#epd-disaggregation) below.

## 3. Advanced Usage

### 3.1 Native video input

dots3.note accepts a native `video_url`. The server decodes the remote video in memory and applies the training-consistent flattening pipeline — interleaving timestamps, frames, and audio under a token budget, with a deterministic seed derived from the video and the question.

> **Video Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="dots3.note",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {
>                     "type": "video_url",
>                     "video_url": {"url": "https://example.com/sample.mp4"},
>                 },
>                 {"type": "text", "text": "Summarize what happens in this video."},
>             ],
>         }
>     ],
>     extra_body={
>         "video_config": {
>             "seq": 131072,
>             "audio_cap": 0.5,
>             "audio_sr": 16000,
>             "k_mode": "eval_ek",
>         },
>     },
> )
>
> print(response.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update...
> ```

Per-request video preprocessing controls are grouped under `video_config` in
`extra_body`:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Default</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`seq`</td>
      <td>`131072`</td>
      <td>Total sequence budget used by the video flattener.</td>
    </tr>

    <tr>
      <td>`audio_cap`</td>
      <td>`1.0`</td>
      <td>Maximum fraction of the input budget assigned to audio; `0` disables audio processing.</td>
    </tr>

    <tr>
      <td>`audio_sr`</td>
      <td>`16000`</td>
      <td>Audio sample rate.</td>
    </tr>

    <tr>
      <td>`k_mode`</td>
      <td>`eval_ek`</td>
      <td>Deterministic evaluation/sampling mode of the flattener.</td>
    </tr>
  </tbody>
</table>

These controls are request-scoped so that evaluation jobs with different context budgets can share one server. For example: `extra_body={"video_config": {"seq": 131072, "audio_cap": 0.5}}`. The flattener reserves room for `max_new_tokens` inside the budget and falls back to visual-only processing if audio would exceed the configured token budget.

A request may carry several videos, and videos can be mixed with image and audio parts. Each video is flattened independently under the same per-request budget, and the flattened frames and audio segments are spliced back at the position of their `video_url` part, so the modality ordering of the prompt is preserved.

### 3.2 Image and audio input

Outside the native-video path, images and audio clips use the standard OpenAI multimodal message format. `--enable-multimodal` is in every cell; the vision and audio towers run in-process, so no extra server is needed.

> **Image Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="dots3.note",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {
>                     "type": "image_url",
>                     "image_url": {"url": "https://example.com/sample.jpg"},
>                 },
>                 {"type": "text", "text": "Describe this image."},
>             ],
>         }
>     ],
> )
>
> print(response.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update...
> ```

### 3.3 Tool Calling

Toggle **Tool Call Parser** (`--tool-call-parser dots`) and **Reasoning Parser** (`--reasoning-parser dots`) in the **Parsers** card of the [Playground above](#playground). Structured tool calls then surface via `message.tool_calls`.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city",
>         "parameters": {
>             "type": "object",
>             "properties": {"city": {"type": "string"}},
>             "required": ["city"],
>         },
>     },
> }]
> resp = client.chat.completions.create(
>     model="dots3.note",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
> print(resp.choices[0].message.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update...
> ```

### 3.4 Encoder/LLM Disaggregation (EPD)

`Dots3NoteForCausalLM` supports both roles of an encoder/LLM-disaggregated deployment:

* **Encoder role** — serve with `--encoder-only`; the instance runs only the vision and audio towers.
* **Language role** — serve with `--language-only`; the instance skips tower construction, leaving the memory to the language model.

See the [EPD guide](../../../docs/advanced_features/epd_disaggregation) for how to wire the roles together.


# Beam 1.0
Source: https://docs.sglang.io/cookbook/autoregressive/Reflection/Beam-1_0

Deploy Reflection Beam 1.0 with SGLang on B300, GB300, and H200: NVFP4, FP8, and BF16 recipes for low latency and high throughput.

## Deployment

> **Install SGLang**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:beam
> ```
>
> For host setup, see [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker). The command panel below has a **Python / Docker** toggle, with Python selected by default. The Python output is the bare `sglang serve` command: run it inside a container started from this image. The Docker output wraps the same command in `docker run`.

Select your hardware, strategy, and quantization to generate the launch command.

* **Low-Latency** uses TP with the [DFlash2 drafter](https://huggingface.co/reflection/Beam-1.0-501B-A23B-DFlash2) at block size 8.
* **High-Throughput** disables DFlash2. H200 BF16 uses TP8 with an automatic static memory fraction; the other recipes use TP + DPA. Blackwell NVFP4 also enables EP and FlashInfer MegaMoE.

Each GSM8K score covers all 1,319 test problems; the benchmark notes identify the evaluation harness.

NVFP4 requires B300 or GB300. **GB300 BF16 uses two four-GPU nodes**: set the head-node address and node rank in **Env**, then run the command on both nodes with rank 0 on the head and rank 1 on the peer.

## TPU Support

Beam 1.0 has day-0 TPU support through SGLang-TorchTPU, a native PyTorch backend for LLM, VLM, and diffusion model inference on TPUs. It combines TPU acceleration with SGLang features such as RadixAttention and HiCache to enable cost-efficient, production-ready serving for agentic workloads.

TPU recipes for Beam 1.0 and access to SGLang-TorchTPU will be available soon. Sign up through this [Google Form](https://docs.google.com/forms/d/e/1FAIpQLSe2Y8hzyU3KGB2StA-0huulrVsq79A0tSxcBQwjj8vlhLhYuQ/viewform?usp=publish-editor) to receive the latest updates.

## Usage

All recipes serve the model as `beam` on port 30000. For GB300 BF16, connect to the head node.

> **Chat completion (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> response = client.chat.completions.create(
>     model="beam",
>     messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
> )
> message = response.choices[0].message
> print("Reasoning:", message.reasoning_content)
> print("Answer:", message.content)
> ```


# Step-3.7-Flash
Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step-3.7-Flash



## 1. Model Introduction

[Step-3.7-Flash](https://huggingface.co/stepfun-ai/Step-3.7-Flash) is a 198B-parameter Mixture-of-Experts (MoE) vision-language model that combines a 196B-parameter language backbone with a 1.8B-parameter vision encoder for native image understanding. Engineered for high-frequency production workloads, it activates approximately 11B parameters per token and supports a 256k context window with three selectable reasoning levels (low, medium, and high). The model is available in multiple quantization formats (BF16, FP8, NVFP4).

Step-3.7-Flash is built for developers who need to scale agentic workflows that combine perception, search, and reasoning — from parsing massive financial reports in one pass, to running multi-step search loops with cross-source verification, to operating concurrent coding agents in high-throughput pipelines.

## 2. SGLang Installation

Step-3.7-Flash is currently available in SGLang via Docker image install.

### Docker (NVIDIA)

```bash Command theme={null}
# Pull the docker image
docker pull lmsysorg/sglang:latest

# Launch the container
docker run -it --gpus all \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  lmsysorg/sglang:latest bash
```

## 3. Model Deployment

This section provides deployment configurations optimized for different use cases.

### 3.1 Basic Configuration

The Step-3.7-Flash series comes in one size with multiple quantization options. Recommended starting configurations vary depending on hardware.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4x, TP=4), B200/B300 (4x, TP=4), GB200/GB300 (4x, TP=4).
* **NVFP4 Quantization**: NVFP4 provides the smallest memory footprint. Requires `--quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 --moe-runner-backend flashinfer_trtllm`.
* **Trust Remote Code**: All Step-3.7-Flash variants require `--trust-remote-code` due to the custom model architecture.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Multi-Modal Inputs

Step-3.7-Flash supports image inputs alongside text. Here's a basic example:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="stepfun-ai/Step-3.7-Flash",
    messages=messages,
    max_tokens=2048,
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Multi-Image Input Example:**

Step-3.7-Flash can process multiple images in a single request for comparison or analysis:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
                }
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
                }
            },
            {
                "type": "text",
                "text": "Compare these two images and describe the differences in 100 words or less."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="stepfun-ai/Step-3.7-Flash",
    messages=messages,
    max_tokens=2048,
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

#### 4.2.2 Reasoning Parser

Step-3.7-Flash supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
sglang serve \
  --model-path stepfun-ai/Step-3.7-Flash \
  --tp 4 \
  --trust-remote-code \
  --reasoning-parser step3p5
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="stepfun-ai/Step-3.7-Flash",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

#### 4.2.3 Tool Calling

Step-3.7-Flash supports tool calling capabilities. Enable the tool call parser:

**Start sglang server:**

```shell Command theme={null}
sglang serve \
  --model-path stepfun-ai/Step-3.7-Flash \
  --tp 4 \
  --trust-remote-code \
  --reasoning-parser step3p5 \
  --tool-call-parser step3p5
```

```python Example theme={null}
from openai import OpenAI
import json

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# 1. define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "The city name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"}
                },
                "required": ["location"]
            }
        }
    }
]

# 2. tool run
def get_weather(location, unit="celsius"):
    return f"The weather in {location} is 22 {unit[0].upper()} and sunny."

# 3. send first request
print("--- Sending first request ---")
response = client.chat.completions.create(
    model="stepfun-ai/Step-3.7-Flash",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=1.0,
    stream=False
)

message = response.choices[0].message

# 4. Handle Reasoning Content
reasoning = getattr(message, 'reasoning_content', None)
if reasoning:
    print("=============== Thinking =================")
    print(reasoning)
    print("==========================================")

# 5. Handle Tool Calls
if message.tool_calls:
    print("\nTool Calls detected:")
    history_messages = [
        {"role": "user", "content": "What's the weather in Beijing?"},
        message
    ]

    for tool_call in message.tool_calls:
        print(f"   Tool: {tool_call.function.name}")
        print(f"   Args: {tool_call.function.arguments}")

        args = json.loads(tool_call.function.arguments)
        tool_result = get_weather(args.get("location"), args.get("unit", "celsius"))

        history_messages.append({
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": tool_result
        })

    print("\n--- Sending tool results ---")
    final_response = client.chat.completions.create(
        model="stepfun-ai/Step-3.7-Flash",
        messages=history_messages,
        temperature=1.0,
        stream=False
    )

    print("=============== Final Content =================")
    print(final_response.choices[0].message.content)

else:
    if message.content:
        print("=============== Content =================")
        print(message.content)
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

## 5. Benchmark

*Benchmark results will be added soon.*


# Step 5 Preview
Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step-5-Preview

Deploy StepFun's Step 5 Preview with SGLang — a 600B-parameter (27B active) Mixture-of-Experts model for agentic work with region-sparse attention, three-layer MTP and 1M context, on NVIDIA Blackwell and Hopper.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> Step 5 Preview support has **not shipped in an SGLang release yet**, so a stock `pip install sglang` cannot serve it. Use the preview Docker image.
>
> **Docker**
>
> Every recipe on this page uses the preview image `lmsysorg/sglang:step-5-preview` (B300 and H200); the command panel picks it automatically. The image already includes the FlashInfer patches Step 5 Preview needs.
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:step-5-preview
>
> docker run --gpus all \
>     --shm-size 32g \
>     -p 30000:30000 \
>     -v ~/.cache/huggingface:/root/.cache/huggingface \
>     --env "HF_TOKEN=<your-hf-token>" \
>     --ipc=host \
>     lmsysorg/sglang:step-5-preview \
>     sglang serve <use args below>
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker).

Pick your hardware and parallelism, then the model, MoE backend and serving options. Recommended starting points:

* **Lowest latency** (Blackwell) — TP4 with the FlashInfer TRT-LLM MoE on four B300 GPUs. Fastest reply for a single user.
* **Multi-user serving** (Blackwell) — TP4 with MegaMoE, which speeds up prefill and decode once several requests share a batch.
* **Highest throughput** (Blackwell) — ATTN DP2: TP8 with DP attention 2 and EP8 on eight B300 GPUs, with either MoE backend. Most tokens per second across many users.
* **Hopper** — EP8: TP8 + EP8 on eight H200 GPUs with FA3 attention and Triton MoE. The FP8 weights do not fit four H200 GPUs. ATTN DP2 is also available on Hopper (TP8 with DP attention 2 and EP8, `--mem-fraction-static 0.85`).

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

## 1. Model Introduction

**Step 5 Preview** is StepFun's flagship model for agentic work, with particular strength in software engineering, professional knowledge work and finance. It is a sparse Mixture-of-Experts model with 600B total parameters and 27B active parameters. It supports a 1M-token context, takes image input through a built-in vision encoder, and ships three MTP layers for speculative decoding.

**Architecture.** The language model has 92 layers. In each MoE layer, a token is routed to 8 of 352 experts, alongside one shared expert. Attention repeats a four-layer pattern: three sliding-window layers that attend to the most recent 512 tokens, then one sparse-attention layer.

**Sparse attention.** The sparse layers split the KV history into 8-token regions. A lightweight indexer scores a summary of each region and selects up to 512 historical regions; attention then reads the token-level keys and values of those regions plus the current region, with causal masking.

**Precision.** Routed-expert weights are FP8 (E4M3, 128×128 block scales) except in the last three MoE layers, which stay BF16. SGLang runs every layer at its released precision.

**Reasoning.** Step 5 Preview always thinks before answering. `reasoning_effort` sets the depth: `low`, `medium` or `high`; when unset, it defaults to `high`.

**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=0` (StepFun's recommended settings; informational — do not hardcode them in application code).

**License:** [StepFun Model License 1.0](https://huggingface.co/stepfun-ai/Step-5-Preview/blob/main/LICENSE.txt).

**Resources:** [HuggingFace (FP8)](https://huggingface.co/stepfun-ai/Step-5-Preview) · [HuggingFace (NVFP4)](https://huggingface.co/stepfun-ai/Step-5-Preview-NVFP4) · [Model page](https://www.stepfun.com/step-5-preview) · [Documentation](https://platform.stepfun.ai/docs/en/guides/models/step-5-preview).

## 2. Configuration Tips

**MoE runner.** On B300, the default runner is the FlashInfer TRT-LLM MoE. Picking **MegaMoE** in the **MoE** row adds `--moe-a2a-backend megamoe`, which runs the FP8 routed experts on MegaMoE; the BF16 routed experts in the last three MoE layers stay on TRT-LLM. MegaMoE also works with ATTN DP2. On H200, the recipe uses the Triton MoE for all expert layers, including the three BF16 ones, because the TRT-LLM MoE kernels are Blackwell-only.

**NVFP4 checkpoint.** Pick **NVFP4** in the **Model** row to serve `stepfun-ai/Step-5-Preview-NVFP4` on Blackwell. Its routed experts are NVFP4 (the last three MoE layers stay BF16) and run on the FlashInfer FP4 MoE kernels; the smaller weights leave more GPU memory for the KV cache. It runs with TP4, EP4 and ATTN DP2, does not support MegaMoE, and is not available on H200.

**Attention backend.** On Blackwell, Step 5 Preview selects FA4 automatically. On H200, keep `--attention-backend fa3` from the recipe.

**KV cache.** Every recipe uses BF16 KV cache. On both Blackwell and Hopper, pick **FP8** in the **KV Cache** row of the command panel for `--kv-cache-dtype fp8_e4m3` (full-attention layers only): it increases KV capacity but is lossy, so re-verify accuracy before using it.

**Prefill CUDA graph.** Step 5 Preview runs prefill on Breakable CUDA Graph by default, including with ATTN DP2, so the commands do not set it. ATTN DP2 runs TP8 with DP attention 2 and EP8; with DP attention on, the vision encoder and projector use replicated weights.

**FlashInfer patches.** The TRT-LLM MoE path needs Step 5 Preview's FlashInfer patches. They are already in `lmsysorg/sglang:step-5-preview`; for a custom build, see the README under `docker/patches/step5` in the SGLang repository.

**First start.** The first launch compiles FlashInfer modules (about 15 minutes on B300); later launches take about 6 minutes. Persist `FLASHINFER_WORKSPACE_BASE` and `~/.cache/sglang` across restarts.

**HiCache.** Turn it on in the **HiCache** row to keep evicted prefixes in host memory. Size `--hicache-ratio` to the host memory available per rank; larger ratios need proportionally more host RAM.

**Per-platform notes:**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Platform</th>
      <th>Topology</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>B300 1×4</td>
      <td>TP4, or TP4 + EP4</td>
      <td>TRT-LLM MoE or MegaMoE; FA4 attention</td>
    </tr>

    <tr>
      <td>B300 1×8</td>
      <td>TP8 + DP attention 2 + EP8</td>
      <td>TRT-LLM MoE or MegaMoE; FA4 attention</td>
    </tr>

    <tr>
      <td>H200 1×8</td>
      <td>TP8 + EP8, or TP8 + DP attention 2 + EP8</td>
      <td>FA3 attention, Triton MoE; BF16 or FP8 KV cache</td>
    </tr>
  </tbody>
</table>

## 3. Advanced Usage

### 3.1 Reasoning

The `step5` reasoning parser (already in every recipe) returns the thinking in `reasoning_content` and the answer in `content`. Pass `reasoning_effort` (`low`, `medium` or `high`) per request to trade depth for latency.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="stepfun-ai/Step-5-Preview",
>     messages=[{"role": "user", "content": "What is 15% of 240?"}],
>     reasoning_effort="low",
> )
> msg = resp.choices[0].message
> print("Reasoning:", msg.reasoning_content)
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: We need answer simple. Need maybe compute 36. Need provide perhaps concise. Use reasoning low but final. 15% of 240 = 0.15*240=36. final.
> Answer: 36
> ```

### 3.2 Tool Calling

The `step5` tool-call parser (already in every recipe) returns structured tool calls in `message.tool_calls`.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> import json
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> tools = [{
>     "type": "function",
>     "function": {
>         "name": "get_weather",
>         "description": "Get the current weather for a city",
>         "parameters": {
>             "type": "object",
>             "properties": {"city": {"type": "string", "description": "City name"}},
>             "required": ["city"],
>         },
>     },
> }]
> messages = [{"role": "user", "content": "What's the weather like in Beijing right now?"}]
>
> resp = client.chat.completions.create(
>     model="stepfun-ai/Step-5-Preview", messages=messages, tools=tools,
> )
> msg = resp.choices[0].message
> for call in msg.tool_calls:
>     print("Tool call:", call.function.name, call.function.arguments)
>
> call = msg.tool_calls[0]
> messages.append(msg)
> messages.append({
>     "role": "tool",
>     "tool_call_id": call.id,
>     "content": json.dumps({"city": "Beijing", "temperature_c": 18, "condition": "Sunny"}),
> })
> final = client.chat.completions.create(
>     model="stepfun-ai/Step-5-Preview", messages=messages, tools=tools,
> )
> print("Reasoning:", final.choices[0].message.reasoning_content)
> print("Answer:", final.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Tool call: get_weather {"city": "Beijing"}
> Reasoning: The weather in Beijing is 18°C and sunny.
> Answer: The weather in Beijing right now is **sunny** with a temperature of **18°C** — a pleasant day! ☀️
> ```

### 3.3 Image Input

> **Image Input Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
> resp = client.chat.completions.create(
>     model="stepfun-ai/Step-5-Preview",
>     messages=[{
>         "role": "user",
>         "content": [
>             {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
>             {"type": "text", "text": "Describe this image in one sentence."},
>         ],
>     }],
> )
> print(resp.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> A man in a yellow hoodie irons a blue shirt on an ironing board attached to the back of a yellow SUV on a city street, with a yellow taxi behind him.
> ```

### 3.4 Speculative Decoding (MTP)

Every recipe enables multi-layer MTP with the checkpoint's three MTP layers (`--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --enable-multi-layer-eagle`). Each verification round can advance generation by up to four tokens: three accepted draft tokens plus one target-model token. To turn MTP off, pick **Off** in the **Spec Decode** row of the command panel.

### 3.5 HiCache (Hierarchical KV Caching)

For long multi-turn sessions, turn on **HiCache** in the command panel to keep evicted prefixes in host memory (L2). To add an L3 store such as Mooncake, pick it in the **Hierarchical KV Cache** card of the [Playground above](#playground). See the HiCache note in [Configuration Tips](#2-configuration-tips) for sizing.

### 3.6 PD Disaggregation

Step 5 Preview supports prefill/decode disaggregation over Mooncake, including the MTP draft KV cache. Pick the **Prefill role** and **Decode role** in the **PD Disaggregation** card of the [Playground above](#playground) to generate both commands and the router command.


# Step3-VL-10B
Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step3-VL-10B



## 1. Model Introduction

[Step3-VL-10B](https://huggingface.co/stepfun-ai/Step3-VL-10B) is a lightweight open-source multimodal model developed by StepFun, designed to redefine the trade-off between compact efficiency and frontier-level multimodal intelligence. Despite its compact 10B parameter footprint, Step3-VL-10B excels in visual perception, complex reasoning, and human-centric alignment.

Key highlights of Step3-VL-10B include:

* **STEM Reasoning**: Achieves 94.43% on AIME 2025 and 75.95% on MathVision (with PaCoRe), demonstrating exceptional complex reasoning capabilities that outperform models 10×–20× larger.
* **Visual Perception**: Records 92.05% on MMBench and 80.11% on MMMU, establishing strong general visual understanding and multimodal reasoning.
* **GUI & OCR**: Delivers state-of-the-art performance on ScreenSpot-V2 (92.61%), ScreenSpot-Pro (51.55%), and OCRBench (86.75%), optimized for agentic and document understanding tasks.
* **Spatial Understanding**: Demonstrates emergent spatial awareness with 66.79% on BLINK and 57.21% on All-Angles-Bench, establishing strong potential for embodied intelligence applications.

For more details, please refer to the [Step3-VL-10B model card on Hugging Face](https://huggingface.co/stepfun-ai/Step3-VL-10B).

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

Step3-VL-10B is a compact 10B dense model that can run on a single GPU. Recommended starting configurations vary depending on hardware.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and quantization method. SGLang supports serving Step3-VL-10B on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Single GPU Deployment**: Step3-VL-10B fits comfortably on a single GPU with BF16 precision, no tensor parallelism required.
* **Memory Management**: Set lower `--context-length` to conserve memory if needed. A value of `32768` is sufficient for most scenarios.
* **FP8 Quantization**: Use FP8 quantization to further reduce memory usage while maintaining quality.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)
* [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision)

### 4.2 Advanced Usage

#### 4.2.1 Multi-Modal Inputs

Step3-VL-10B supports image inputs. Here's a basic example with image input:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="stepfun-ai/Step3-VL-10B",
    messages=messages,
    max_tokens=2048,
    extra_body={"top_k": -1}
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example output:**

```text Output theme={null}
Response costs: 5.89s
Generated text: Auntie Anne's

CINNAMON SUGAR
1 × 17,000               17,000

SUB TOTAL                    17,000

GRAND TOTAL                 17,000

CASH IDR                    20,000

CHANGE DUE                 3,000
```

**Multi-Image Input Example:**

Step3-VL-10B can process multiple images in a single request for comparison or analysis:

```python Example theme={null}
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:30000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://www.civitatis.com/f/china/hong-kong/guia/taxi.jpg"
                }
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://cdn.cheapoguides.com/wp-content/uploads/sites/7/2025/05/GettyImages-509614603-1280x600.jpg"
                }
            },
            {
                "type": "text",
                "text": "Compare these two images and describe the differences in 100 words or less."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="stepfun-ai/Step3-VL-10B",
    messages=messages,
    max_tokens=2048,
    extra_body={"top_k": -1}
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
```

**Example Output:**

```text Output theme={null}
Response costs: 3.24s
Generated text: First image: Single red Hong Kong taxi close - up, clear license plate (RX 5004), “4 SEATS” sticker, urban street with shops behind. Second image: Aerial view of many taxis (red, green) on a highway with a viaduct, some hoods open, dense arrangement. Differences: Scale (single vs many), perspective (close - up vs aerial), context (street shops vs highway), and taxi conditions (normal vs some open hoods).
```

#### 4.2.2 Reasoning Parser

Step3-VL-10B supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
python -m sglang.launch_server \
  --model stepfun-ai/Step3-VL-10B \
  --reasoning-parser deepseek-r1 \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code
```

**Streaming with Thinking Process:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="stepfun-ai/Step3-VL-10B",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True,
    extra_body={"top_k": -1}
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Example Output:**

```text Output theme={null}
=============== Thinking =================
Okay, let's see. The problem is asking for 15% of 240. Hmm, I need to remember how to calculate percentages. So, percentage means "per hundred," right? So, 15% is the same as 15 per 100, or 15/100. To find a percentage of a number, I think you convert the percentage to a decimal and then multiply it by the number. Let me check that.

First, 15% as a decimal. To convert a percentage to a decimal, you divide by 100. So 15 divided by 100 is 0.15. Yeah, that's right. So 15% is 0.15 in decimal form. Then, to find 15% of 24
0, I need to multiply 0.15 by 240. Let me do that calculation.

Let me write it out: 0.15 * 240. Let's compute that. Maybe break it down. 0.1 is 10%, and 0.05 is 5%, so 10% of 240 is 24, and 5% of 240 is 12. Then 10% + 5% is 15%, so 24 + 12 is 36. Oh,
 that's a good way to check. So 15% is 10% plus 5%, which adds up to 36. Let me verify with the decimal method. 0.15 * 240. Let's multiply 240 by 0.1 first: 24. Then 240 by 0.05: 12. Adding them gives 36. Yep, that matches. Alternatively, 240 * 15 = 3600, and then divide by 100 (since it's per hundred), so 3600 / 100 = 36. That's another way. So 15% of 240 is 36. Let me make sure I didn't make a mistake. Let's check with another method. 10% of 240 is 24, 20% would be 48, so 15% is halfway between 10% and 20%, which is (24 + 48)/2 = 36. Yep, that works too
. So all methods point to 36. I think that's the answer.

=============== Content =================

To solve the problem "What is 15% of 240?" step by step:

---

### **Step 1: Understand the concept of percentage**
A percentage represents a portion of a whole. Specifically, "percent" means "per hundred." So, 15% means **15 out of 100**, or **15/100**.

---

### **Step 2: Convert the percentage to a decimal**
To use percentages in calculations, convert them to decimals by dividing by 100:
$$
15\% = \frac{15}{100} = 0.15
$$

---

### **Step 3: Multiply the decimal by the given number**
Now, multiply 0.15 (the decimal form of 15%) by 240:
$$
0.15 \times 240 = 36
$$

---

### **Alternative Verification Methods**

#### **Method A: Break into parts**
- 10% of 240 = $ 0.10 \times 240 = 24 $
- 5% of 240 = $ 0.05 \times 240 = 12 $
- Add them: $ 24 + 12 = 36 $

#### **Method B: Use direct multiplication**
- $ 15\% \text{ of } 240 = \frac{15}{100} \times 240 = \frac{3600}{100} = 36 $

#### **Method C: Estimate using known percentages**
- 20% of 240 = $ 0.20 \times 240 = 48 $
- 10% of 240 = $ 0.10 \times 240 = 24 $
- 15% is halfway between 10% and 20%: $ \frac{24 + 48}{2} = 36 $

---

### **Final Answer**
$$
\boxed{36}
$$
```

**Note:** The reasoning parser captures the model's step-by-step thinking process, allowing you to see how the model arrives at its conclusions.

#### 4.2.3 Tool Calling

Step3-VL-10B supports tool calling capabilities. Enable the tool call parser:

```shell Command theme={null}
python -m sglang.launch_server \
  --model stepfun-ai/Step3-VL-10B \
  --reasoning-parser deepseek-r1 \
  --tool-call-parser hermes \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code
```

**Python Example (with Thinking Process):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Define available tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city name"
                    },
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                        "description": "Temperature unit"
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make request with streaming to see thinking process
response = client.chat.completions.create(
    model="stepfun-ai/Step3-VL-10B",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=0.7,
    stream=True,
    extra_body={"top_k": -1}
)

# Process streaming response
thinking_started = False
has_thinking = False
tool_calls_accumulator = {}

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Accumulate tool calls
        if hasattr(delta, 'tool_calls') and delta.tool_calls:
            # Close thinking section if needed
            if has_thinking and thinking_started:
                print("\n=============== Content =================\n", flush=True)
                thinking_started = False

            for tool_call in delta.tool_calls:
                index = tool_call.index
                if index not in tool_calls_accumulator:
                    tool_calls_accumulator[index] = {
                        'name': None,
                        'arguments': ''
                    }

                if tool_call.function:
                    if tool_call.function.name:
                        tool_calls_accumulator[index]['name'] = tool_call.function.name
                    if tool_call.function.arguments:
                        tool_calls_accumulator[index]['arguments'] += tool_call.function.arguments

        # Print content
        if delta.content:
            print(delta.content, end="", flush=True)

# Print accumulated tool calls
for index, tool_call in sorted(tool_calls_accumulator.items()):
    print(f"Tool Call: {tool_call['name']}")
    print(f"   Arguments: {tool_call['arguments']}")

print()
```

**Example Output:**

```text Output theme={null}
=============== Thinking =================
The user is asking about the weather in Beijing. I have a function called "get_weather" that can provide weather information for a location. Let me check the parameters:

- location: required (string) - "Beijing"
- unit: optional (string, enum: ["celsius", "fahrenheit"]) - not specified by the user, so I won't include it

I should call the function with location="Beijing".

<tool_calls>

=============== Content =================

</tool_calls>Tool Call: get_weather
   Arguments: {"location": "Beijing"}
```

**Handling Tool Call Results:**

```python Example theme={null}
# After getting the tool call, execute the function
def get_weather(location, unit="celsius"):
    # Your actual weather API call here
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# Send tool result back to the model
messages = [
    {"role": "user", "content": "What's the weather in Beijing?"},
    {
        "role": "assistant",
        "content": None,
        "tool_calls": [{
            "id": "call_123",
            "type": "function",
            "function": {
                "name": "get_weather",
                "arguments": '{"location": "Beijing", "unit": "celsius"}'
            }
        }]
    },
    {
        "role": "tool",
        "tool_call_id": "call_123",
        "content": get_weather("Beijing", "celsius")
    }
]

final_response = client.chat.completions.create(
    model="stepfun-ai/Step3-VL-10B",
    messages=messages,
    temperature=0.7,
    extra_body={"top_k": -1}
)

print(final_response.choices[0].message.content)
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA B200 GPU (1x)
* Model: stepfun-ai/Step3-VL-10B
* Tensor Parallelism: 1
* sglang version: 0.5.8+

We use SGLang's built-in benchmarking tool to conduct performance evaluation with random images.

#### 5.1.1 Latency-Sensitive Benchmark

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model stepfun-ai/Step3-VL-10B \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code
```

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model stepfun-ai/Step3-VL-10B \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  30.85
Total input tokens:                      14120
Total input text tokens:                 720
Total input vision tokens:               13400
Total generated tokens:                  4220
Total generated tokens (retokenized):    4217
Request throughput (req/s):              0.32
Input token throughput (tok/s):          457.71
Output token throughput (tok/s):         136.79
Peak output token throughput (tok/s):    240.00
Peak concurrent requests:                2
Total token throughput (tok/s):          594.50
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3083.40
Median E2E Latency (ms):                 2747.00
P90 E2E Latency (ms):                    4574.50
P99 E2E Latency (ms):                    5462.49
---------------Time to First Token----------------
Mean TTFT (ms):                          1327.69
Median TTFT (ms):                        1341.01
P99 TTFT (ms):                           1486.11
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.16
Median TPOT (ms):                        4.17
P99 TPOT (ms):                           4.18
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.17
Median ITL (ms):                         4.18
P95 ITL (ms):                            4.30
P99 ITL (ms):                            4.38
Max ITL (ms):                            8.24
==================================================
```

#### 5.1.2 Throughput-Sensitive Benchmark

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model stepfun-ai/Step3-VL-10B \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* Result:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  976.52
Total input tokens:                      1416949
Total input text tokens:                 76949
Total input vision tokens:               1340000
Total generated tokens:                  510855
Total generated tokens (retokenized):    510526
Request throughput (req/s):              1.02
Input token throughput (tok/s):          1451.02
Output token throughput (tok/s):         523.14
Peak output token throughput (tok/s):    20429.00
Peak concurrent requests:                103
Total token throughput (tok/s):          1974.16
Concurrency:                             99.81
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   97463.22
Median E2E Latency (ms):                 91872.75
P90 E2E Latency (ms):                    118553.42
P99 E2E Latency (ms):                    198445.56
---------------Time to First Token----------------
Mean TTFT (ms):                          94379.07
Median TTFT (ms):                        87163.09
P99 TTFT (ms):                           194871.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.89
Median TPOT (ms):                        5.72
P99 TPOT (ms):                           23.58
---------------Inter-Token Latency----------------
Mean ITL (ms):                           6.05
Median ITL (ms):                         0.13
P95 ITL (ms):                            0.56
P99 ITL (ms):                            3.99
Max ITL (ms):                            97551.06
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 MMMU Benchmark

You can evaluate the model's accuracy using the MMMU dataset:

* Model Deployment Command:

```shell Command theme={null}
python -m sglang.launch_server \
  --model stepfun-ai/Step3-VL-10B \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code
```

* Benchmark Command:

```shell Command theme={null}
python3 benchmark/mmmu/bench_sglang.py \
    --port 30000 \
    --concurrency 64
```

* Result:

```text Output theme={null}
Benchmark time: 934.6179109360091
answers saved to: ./answer_sglang.json
Evaluating...
answers saved to: ./answer_sglang.json
{'Accounting': {'acc': 0.667, 'num': 30},
 'Agriculture': {'acc': 0.367, 'num': 30},
 'Architecture_and_Engineering': {'acc': 0.4, 'num': 30},
 'Art': {'acc': 0.467, 'num': 30},
 'Art_Theory': {'acc': 0.5, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.367, 'num': 30},
 'Biology': {'acc': 0.3, 'num': 30},
 'Chemistry': {'acc': 0.467, 'num': 30},
 'Clinical_Medicine': {'acc': 0.567, 'num': 30},
 'Computer_Science': {'acc': 0.467, 'num': 30},
 'Design': {'acc': 0.567, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.3, 'num': 30},
 'Economics': {'acc': 0.6, 'num': 30},
 'Electronics': {'acc': 0.567, 'num': 30},
 'Energy_and_Power': {'acc': 0.633, 'num': 30},
 'Finance': {'acc': 0.733, 'num': 30},
 'Geography': {'acc': 0.333, 'num': 30},
 'History': {'acc': 0.533, 'num': 30},
 'Literature': {'acc': 0.533, 'num': 30},
 'Manage': {'acc': 0.6, 'num': 30},
 'Marketing': {'acc': 0.767, 'num': 30},
 'Materials': {'acc': 0.6, 'num': 30},
 'Math': {'acc': 0.7, 'num': 30},
 'Mechanical_Engineering': {'acc': 0.333, 'num': 30},
 'Music': {'acc': 0.4, 'num': 30},
 'Overall': {'acc': 0.523, 'num': 900},
 'Overall-Art and Design': {'acc': 0.483, 'num': 120},
 'Overall-Business': {'acc': 0.673, 'num': 150},
 'Overall-Health and Medicine': {'acc': 0.513, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.492, 'num': 120},
 'Overall-Science': {'acc': 0.5, 'num': 150},
 'Overall-Tech and Engineering': {'acc': 0.481, 'num': 210},
 'Pharmacy': {'acc': 0.6, 'num': 30},
 'Physics': {'acc': 0.7, 'num': 30},
 'Psychology': {'acc': 0.467, 'num': 30},
 'Public_Health': {'acc': 0.733, 'num': 30},
 'Sociology': {'acc': 0.433, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.523
```


# Step-3.5-Flash
Source: https://docs.sglang.io/cookbook/autoregressive/StepFun/Step3.5



## 1. Model Introduction

[Step-3.5-Flash](https://huggingface.co/stepfun-ai/Step-3.5-Flash) is StepFun's production-grade reasoning engine built to decouple elite intelligence from heavy compute, and cuts attention cost for low-latency, cost-effective long-context inference—purpose-built for autonomous agents in real-world workflows. The model is available in multiple quantization formats optimized for different hardware platforms.

This generation delivers comprehensive upgrades across the board:

* **Hybrid Attention Architecture**:  Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 3:1 ratio and an aggressive 128-token window. This hybrid approach ensures consistent performance across massive datasets or long codebases while significantly reducing the computational overhead typical of standard long-context models.
* **Sparse Mixture-of-Experts**: Only 11B active parameters out of 196B parameters.
* **Multi-Layer Multi-Token Prediction (MTP)**: Equipped with a  3-way Multi-Token Prediction (MTP-3). This allows for complex, multi-step reasoning chains with immediate responsiveness.

## 2.SGLang Installation

Step-3.5-Flash is currently available in SGLang via Docker image install.

### Docker (NVIDIA)

```bash Command theme={null}
# Pull the docker image
docker pull lmsysorg/sglang:latest

# Launch the container
docker run -it --gpus all \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  lmsysorg/sglang:latest bash
```

### Docker (AMD ROCm)

```bash Command theme={null}
# For MI300X/MI325X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi30x

# For MI350X/MI355X
docker pull lmsysorg/sglang:v0.5.9-rocm700-mi35x

docker run -it \
  --device=/dev/kfd --device=/dev/dri \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  --group-add video --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  lmsysorg/sglang:v0.5.9-rocm700-mi30x bash  # or mi35x for MI350X/MI355X
```

## 3.Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The Step-3.5-Flash series comes in only one sizes. Recommended starting configurations vary depending on hardware.

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

* **Memory**: Requires GPUs with high VRAM capacity. Supported platforms: H200 (4×, TP=4), MI300X/MI325X/MI350X/MI355X (4×, TP=4 EP=4).
* **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X and `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X.
* **AMD Expert Parallelism Required**: On AMD GPUs, always use `--ep 4` with `--tp 4`. Both BF16 and FP8 models require expert parallelism. Without EP, the MoE intermediate dimension is split across GPUs (N=320), which triggers an AITER CK GEMM incompatibility. With EP=4, each GPU handles 72 full experts (N=1280), which works correctly with cuda graph enabled.
* **AITER JIT Compilation**: First inference on AMD may take 30-40 seconds for AITER kernel JIT compilation. Subsequent requests use cached kernels.

## 4.Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser

Step-3.5-Flash only supports reasoning mode. Enable the reasoning parser during deployment to separate the thinking and content sections:

```shell Command theme={null}
sglang serve \
  --model-path stepfun-ai/Step-3.5-Flash \
  --tp 4 \
  --ep 4 \
  --reasoning-parser step3p5
```

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# Enable streaming to see the thinking process in real-time
response = client.chat.completions.create(
    model="stepfun-ai/Step-3.5-Flash",
    messages=[
        {"role": "user", "content": "Solve this problem step by step: What is 15% of 240?"}
    ],
    temperature=0.7,
    max_tokens=2048,
    stream=True
)

# Process the stream
has_thinking = False
has_answer = False
thinking_started = False

for chunk in response:
    if chunk.choices and len(chunk.choices) > 0:
        delta = chunk.choices[0].delta

        # Print thinking process
        if hasattr(delta, 'reasoning_content') and delta.reasoning_content:
            if not thinking_started:
                print("=============== Thinking =================", flush=True)
                thinking_started = True
            has_thinking = True
            print(delta.reasoning_content, end="", flush=True)

        # Print answer content
        if delta.content:
            # Close thinking section and add content header
            if has_thinking and not has_answer:
                print("\n=============== Content =================", flush=True)
                has_answer = True
            print(delta.content, end="", flush=True)

print()
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
We are asked: "What is 15% of 240?" We need to solve step by step.

Step 1: Understand that "15% of 240" means we need to calculate 15 percent of 240. In mathematical terms, it is (15/100) * 240.

Step 2: Simplify the calculation. We can compute 15% of 240 by first finding 10% of 240 and then 5% of 240, and adding them. Alternatively, we can multiply directly.

Method 1:
10% of 240 = 240 * 0.10 = 24.
5% is half of 10%, so 5% of 240 = 24 / 2 = 12.
Then 15% = 10% + 5% = 24 + 12 = 36.

Method 2: Direct multiplication: 15% = 15/100 = 0.15, so 0.15 * 240 = 36.

We can also compute fractionally: (15/100)*240 = (15*240)/100. 15*240 = 3600, divided by 100 gives 36.

Thus, the answer is 36.

We'll present the solution step by step.

=============== Content =================

To find 15% of 240, follow these steps:

1. **Convert the percentage to a decimal**:
   \( 15\% = \frac{15}{100} = 0.15 \)

2. **Multiply by the number**:
   \( 0.15 \times 240 = 36 \)

Alternatively, break it down:
- \( 10\% \text{ of } 240 = 240 \times 0.10 = 24 \)
- \( 5\% \text{ of } 240 = \frac{24}{2} = 12 \) (since 5% is half of 10%)
- \( 15\% = 10\% + 5\% = 24 + 12 = 36 \)

**Answer:** 36
```

#### 4.2.2 Tool Calling

Step-3.5 supports tool calling capabilities. Enable the tool call parser:

**Python Example:**

Start sglang server:

```shell Command theme={null}
sglang serve \
  --model-path stepfun-ai/Step-3.5-Flash \
  --tp 4 \
  --ep 4 \
  --reasoning-parser step3p5 \
  --tool-call-parser step3p5
```

```python Example theme={null}
from openai import OpenAI
import json

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

# 1. define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "The city name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Temperature unit"}
                },
                "required": ["location"]
            }
        }
    }
]

# 2. tool run
def get_weather(location, unit="celsius"):
    return f"The weather in {location} is 22°{unit[0].upper()} and sunny."

# 3. send first request
print("--- Sending first request ---")
response = client.chat.completions.create(
    model="stepfun-ai/Step-3.5-Flash",
    messages=[
        {"role": "user", "content": "What's the weather in Beijing?"}
    ],
    tools=tools,
    temperature=1.0,
    stream=False
)

message = response.choices[0].message

# 4. Handle Reasoning Content
reasoning = getattr(message, 'reasoning_content', None)
if reasoning:
    print("=============== Thinking =================")
    print(reasoning)
    print("==========================================")

# 5. Handle Tool Calls
if message.tool_calls:
    print("\n🔧 Tool Calls detected:")
    history_messages = [
        {"role": "user", "content": "What's the weather in Beijing?"},
        message
    ]

    for tool_call in message.tool_calls:
        print(f"   Tool: {tool_call.function.name}")
        print(f"   Args: {tool_call.function.arguments}")

        args = json.loads(tool_call.function.arguments)
        tool_result = get_weather(args.get("location"), args.get("unit", "celsius"))

        history_messages.append({
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": tool_result
        })

    print("\n--- Sending tool results ---")
    final_response = client.chat.completions.create(
        model="stepfun-ai/Step-3.5-Flash",
        messages=history_messages,
        temperature=1.0,
        stream=False
    )

    print("=============== Final Content =================")
    print(final_response.choices[0].message.content)

else:
    if message.content:
        print("=============== Content =================")
        print(message.content)
```

**Output Example:**

```text Output theme={null}
--- Sending first request ---
=============== Thinking =================
The user is asking for the weather in Beijing. I should use the get_weather function with location="Beijing". The unit parameter is optional and the user didn't specify a preference, so I'll leave it out (the default should be fine).

==========================================

🔧 Tool Calls detected:
   Tool: get_weather
   Args: {"location": "Beijing"}

--- Sending tool results ---
=============== Final Content =================
The weather in Beijing is 22°C and sunny.
```

**Note:**

* The reasoning parser shows how the model decides to use a tool
* Tool calls are clearly marked with the function name and arguments
* You can then execute the function and send the result back to continue the conversation

## 5. Benchmark

### 5.1 Speed Benchmark

**Test Environment:**

* Hardware: NVIDIA H200 GPU (4x)
* Model: Step-3.5-Flash
* Tensor Parallelism: 4
* Expert Parallelism: 4
* sglang version: 0.5.8

We use SGLang's built-in benchmarking tool to conduct performance evaluation on the [ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered) dataset. This dataset contains real conversation data and can better reflect performance in actual use scenarios.

#### 5.1.1 Standard Scenario Benchmark

* Model Deployment Command:

```shell Command theme={null}
sglang serve \
  --model-path stepfun-ai/Step-3.5-Flash \
  --tp 4 \
  --ep 4
```

##### 5.1.1.1 Low Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model stepfun-ai/Step-3.5-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  35.30
Total input tokens:                      6091
Total input text tokens:                 6091
Total generated tokens:                  4220
Total generated tokens (retokenized):    4212
Request throughput (req/s):              0.28
Input token throughput (tok/s):          172.57
Output token throughput (tok/s):         119.56
Peak output token throughput (tok/s):    124.00
Peak concurrent requests:                2
Total token throughput (tok/s):          292.14
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   3527.94
Median E2E Latency (ms):                 2884.72
P90 E2E Latency (ms):                    6350.38
P99 E2E Latency (ms):                    7858.53
---------------Time to First Token----------------
Mean TTFT (ms):                          107.53
Median TTFT (ms):                        80.93
P99 TTFT (ms):                           269.52
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          8.12
Median TPOT (ms):                        8.13
P99 TPOT (ms):                           8.14
---------------Inter-Token Latency----------------
Mean ITL (ms):                           8.12
Median ITL (ms):                         8.11
P95 ITL (ms):                            8.61
P99 ITL (ms):                            8.91
Max ITL (ms):                            20.77
==================================================
```

##### 5.1.1.2 Medium Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model stepfun-ai/Step-3.5-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 80 \
  --max-concurrency 16
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     80
Benchmark duration (s):                  54.06
Total input tokens:                      39588
Total input text tokens:                 39588
Total generated tokens:                  40805
Total generated tokens (retokenized):    40479
Request throughput (req/s):              1.48
Input token throughput (tok/s):          732.33
Output token throughput (tok/s):         754.84
Peak output token throughput (tok/s):    928.00
Peak concurrent requests:                21
Total token throughput (tok/s):          1487.17
Concurrency:                             14.06
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   9501.23
Median E2E Latency (ms):                 10010.71
P90 E2E Latency (ms):                    15655.09
P99 E2E Latency (ms):                    18803.63
---------------Time to First Token----------------
Mean TTFT (ms):                          198.34
Median TTFT (ms):                        89.50
P99 TTFT (ms):                           984.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          18.97
Median TPOT (ms):                        18.80
P99 TPOT (ms):                           35.67
---------------Inter-Token Latency----------------
Mean ITL (ms):                           18.27
Median ITL (ms):                         17.48
P95 ITL (ms):                            18.44
P99 ITL (ms):                            62.47
Max ITL (ms):                            460.85
==================================================
```

##### 5.1.1.3 High Concurrency

* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model stepfun-ai/Step-3.5-Flash \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Test Results:

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     500
Benchmark duration (s):                  125.88
Total input tokens:                      249331
Total input text tokens:                 249331
Total generated tokens:                  252662
Total generated tokens (retokenized):    251323
Request throughput (req/s):              3.97
Input token throughput (tok/s):          1980.77
Output token throughput (tok/s):         2007.23
Peak output token throughput (tok/s):    2500.00
Peak concurrent requests:                109
Total token throughput (tok/s):          3987.99
Concurrency:                             92.25
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   23223.31
Median E2E Latency (ms):                 22631.90
P90 E2E Latency (ms):                    42269.38
P99 E2E Latency (ms):                    47637.53
---------------Time to First Token----------------
Mean TTFT (ms):                          372.13
Median TTFT (ms):                        127.26
P99 TTFT (ms):                           1880.42
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          46.06
Median TPOT (ms):                        47.61
P99 TPOT (ms):                           51.34
---------------Inter-Token Latency----------------
Mean ITL (ms):                           45.31
Median ITL (ms):                         39.86
P95 ITL (ms):                            72.49
P99 ITL (ms):                            117.05
Max ITL (ms):                            1359.81
==================================================
```

### 5.2 Accuracy Benchmark

#### 5.2.1 GSM8K Benchmark

* **Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* **Results**:

  * Step-3.5-Flash
    ```
    Accuracy: 0.885
    Invalid: 0.005
    Latency: 9.986 s
    Output throughput: 1972.911 token/s
    ```


# Hy3 preview
Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hunyuan3-Preview



## 1. Model Introduction

Hy3 preview is Tencent's preview of its third-generation flagship MoE language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving.

**Key Features:**

* **MoE Architecture**: 192 routed experts + 1 shared expert, 8 experts activated per token. \~276B total parameters with \~20B active, delivering dense-model quality at MoE inference cost.
* **Hybrid Thinking**: Reasoning modes (`high`, `medium`, `low`, `none`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning.
* **Native Tool Calling**: Trained on structured `<tool_call>` / `<arg_key>` / `<arg_value>` grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output.
* **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
* **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box.

**Available Models:**

* [tencent/Hy3-preview](https://huggingface.co/tencent/Hy3-preview) — BF16 instruct
* [tencent/Hy3-preview-Base](https://huggingface.co/tencent/Hy3-preview-Base) — BF16 base

**Recommended Generation Parameters:**

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`temperature`</td>
      <td>0.7</td>
    </tr>

    <tr>
      <td>`top_p`</td>
      <td>0.9</td>
    </tr>

    <tr>
      <td>`reasoning_effort`</td>
      <td>`high` / `medium` / `low` (thinking) or `none` (instant)</td>
    </tr>
  </tbody>
</table>

**License:** TODO — verify on HuggingFace model card.

## 2. SGLang Installation

SGLang offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements.

Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions.

**Docker Images by Hardware Platform:**

<table>
  <thead>
    <tr>
      <th>Hardware Platform</th>
      <th>Docker Image</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>NVIDIA H200 / B200 / B300 / GB300</td>
      <td>`lmsysorg/sglang:latest`</td>
    </tr>
  </tbody>
</table>

`lmsysorg/sglang:latest` bundles the HYV3 model code, the `hunyuan` tool-call / reasoning parsers, and the MTP draft-module runtime.

For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation).

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization, and feature capabilities.

Run this Command:

<pre />

### 3.2 Configuration Tips

**Key Parameters:**

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Recommended Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--tool-call-parser`</td>
      <td>Tool call parser for function-calling support</td>
      <td>`hunyuan`</td>
    </tr>

    <tr>
      <td>`--reasoning-parser`</td>
      <td>Reasoning parser for hybrid thinking modes</td>
      <td>`hunyuan`</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>Required for Hunyuan model loading</td>
      <td>Always enabled</td>
    </tr>

    <tr>
      <td>`--mem-fraction-static`</td>
      <td>Static memory fraction (KV + activations)</td>
      <td>`0.9`</td>
    </tr>

    <tr>
      <td>`--tp`</td>
      <td>Tensor parallelism size</td>
      <td>`2` / `4` / `8` depending on hardware</td>
    </tr>

    <tr>
      <td>`--attention-backend`</td>
      <td>Attention backend (Blackwell only)</td>
      <td>`trtllm_mha`</td>
    </tr>

    <tr>
      <td>`--speculative-algorithm`</td>
      <td>Speculative decoding via the bundled MTP draft</td>
      <td>`EAGLE` + `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`</td>
    </tr>
  </tbody>
</table>

**Hardware Requirements: NVIDIA BF16 (`Hy3-preview`, \~552GB weights)**

* **H200 (141GB) / B200 (180GB)**: TP=8 (minimum for BF16 to fit single-node).
* **B300 (275GB) / GB300**: TP=4.
* **A100 / H100 (80GB)**: not supported single-node — BF16 requires multi-node TP=16+ on 80GB-class GPUs.

**Blackwell (B200 / B300 / GB300):** Auto-selected attention backend can mis-route for HYV3 on Blackwell. Always pass `--attention-backend trtllm_mha` explicitly on Blackwell hardware (the config generator above enforces this).

**Multi-Token Prediction (MTP):** The `Hy3-preview` release bundles an MTP draft module. SGLang runs it via its EAGLE speculative-decoding path — the draft module auto-loads from the same `--model-path`. Enable with the standard MTP flags:

```bash Command theme={null}
sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --reasoning-parser hunyuan \
  --tool-call-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.85
```

Toggle the "Speculative Decoding (MTP)" option in the generator above to add these flags automatically. Tune `num-steps` / `num-draft-tokens` based on acceptance rate in your workload.

**Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.

## 4. Model Invocation

### 4.1 Basic Usage

For basic API usage and request examples, please refer to:

* [SGLang Basic Usage Guide](../../../docs/basic_usage/send_request)

**Deployment Command (H200 × 8, BF16 default):**

```bash Command theme={null}
sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --reasoning-parser hunyuan \
  --tool-call-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.9
```

**Testing Deployment:**

After startup, you can test the SGLang OpenAI-compatible API with the following command:

```bash Command theme={null}
curl http://localhost:30000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "tencent/Hy3-preview",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
    }'
```

**Simple Completion Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"}
    ],
    max_tokens=1024
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("Content:  ", response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
Reasoning: None
Content:   The Los Angeles Dodgers won the 2020 World Series. They defeated the Tampa Bay Rays in six games (4-2). This was the Dodgers' first World Series championship since 1988. The series was notable for being played in a neutral-site bubble at Globe Life Field in Arlington, Texas, due to the COVID-19 pandemic.
```

When `reasoning_effort` is not set, the server defaults to instant mode (no thinking, `reasoning_content=None`). To opt into thinking, pass `reasoning_effort="high" / "medium" / "low"` on the request — see the Hybrid Thinking section below.

### 4.2 Advanced Usage

#### 4.2.1 Reasoning Parser (Hybrid Thinking)

Hy3-preview is a hybrid-thinking model. Control the thinking budget via the OpenAI-standard `reasoning_effort`:

* `high` / `medium` / `low` — increasing amounts of chain-of-thought in `reasoning_content`
* `none` — skip thinking entirely (instant responses, content-only)

Enable the reasoning parser during deployment so that the thinking section (`<think>...</think>`) is separated into `reasoning_content`:

```bash Command theme={null}
sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --reasoning-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.9
```

**Thinking Mode — High Effort:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
    reasoning_effort="high",
    max_tokens=2048,
)

msg = response.choices[0].message
print("=============== Thinking =================")
print(msg.reasoning_content)
print("=============== Content =================")
print(msg.content)
```

**Output Example:**

```text Output theme={null}
=============== Thinking =================
We need to solve: "What is 15% of 240?" Step by step. So we need to compute 15% of 240. The process: 15% means 15 per hundred, i.e., 15/100 = 0.15. Multiply 0.15 by 240. Or we can do: 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. Or do multiplication: 15/100 * 240 = (15*240)/100 = (3600)/100 = 36. So answer is 36.

We need to produce step-by-step explanation. The instruction: "Solve step by step: What is 15% of 240?" So we should provide a clear solution with steps. The final answer: 36. Also maybe include units? No units.

We'll output the solution in a clear manner.
=============== Content =================
To find 15% of 240, follow these steps:

1. **Understand that percent means "per hundred."**
   So, 15% = 15/100 or 0.15.

2. **Multiply the number (240) by the percentage in decimal form.**
   0.15 × 240.

   Alternatively, you can use fractions:
   (15/100) × 240.

3. **Perform the multiplication.**
   0.15 × 240 = 36.
   Or:
   (15 × 240) / 100 = 3600 / 100 = 36.

4. **Check using an alternative method:**
   - 10% of 240 = 24.
   - 5% of 240 = half of 10% = 12.
   - 15% = 10% + 5% = 24 + 12 = 36.

Thus, **15% of 240 is 36**.
```

**Instant Mode — No Thinking:**

```python Example theme={null}
response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
    reasoning_effort="none",
    max_tokens=256,
)

print("Content:", response.choices[0].message.content)
```

**Output Example:**

```text Output theme={null}
Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.
```

#### 4.2.2 Tool Calling

Hy3-preview supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips thinking tokens before the tool-call parser runs:

```bash Command theme={null}
sglang serve \
  --model-path tencent/Hy3-preview \
  --tp 8 \
  --reasoning-parser hunyuan \
  --tool-call-parser hunyuan \
  --trust-remote-code \
  --mem-fraction-static 0.9
```

**Non-Streaming Example:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
)

msg = response.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Content:  ", msg.content)
for tc in msg.tool_calls or []:
    print(f"Tool Call: {tc.function.name}")
    print(f"  Arguments: {tc.function.arguments}")
```

**Output Example:**

```text Output theme={null}
Reasoning: None
Content:   I'll get the current weather for Beijing in Fahrenheit for you.
Tool Call: get_weather
  Arguments: {"city": "Beijing", "unit": "fahrenheit"}
```

**Streaming Example (incremental argument deltas):**

Hy3-preview's `hunyuan` tool-call parser emits tool names first, then argument JSON in incremental fragments — matching the OpenAI streaming contract:

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

stream = client.chat.completions.create(
    model="tencent/Hy3-preview",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
    stream=True,
)

tool_buffer = {}
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)
    for tc in delta.tool_calls or []:
        buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
        if tc.function and tc.function.name:
            buf["name"] += tc.function.name
        if tc.function and tc.function.arguments:
            buf["args"] += tc.function.arguments

for idx, buf in tool_buffer.items():
    print(f"\nTool[{idx}] {buf['name']}({buf['args']})")
```

**Output Example:**

```text Output theme={null}
I'll check the current weather in Beijing for you using Fahrenheit.
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})
```

## 5. Benchmark

### 5.1 Accuracy Benchmark

**Test Environment:**

* Hardware: 8× NVIDIA H200 (141GB)
* Docker Image: `lmsysorg/sglang:hy3-preview`
* Model: `tencent/Hy3-preview` (BF16)
* Tensor Parallelism: 8
* SGLang version: latest `main`

#### 5.1.1 GSM8K

* Benchmark Method: GSM8K on 200 questions using sgl-eval
* Benchmark Command:

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200 \
  --num-threads 64
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

* Test Results:

```text Output theme={null}
TODO — replace with real GSM8K accuracy after benchmark run on Hy3-preview (BF16).
```

#### 5.1.2 MMLU

* Benchmark Method: zero-shot Chat API evaluation via sgl-eval, all 57 subjects
* Benchmark Command:

```bash Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-threads 64 --max-tokens 2048
```

* Test Results:

```text Output theme={null}
TODO — replace with real MMLU accuracy after benchmark run on Hy3-preview (BF16).
```

#### 5.1.3 Tool-Call Accuracy (MiniMax-Provider-Verifier)

* Benchmark Tool: [MiniMax-Provider-Verifier](https://github.com/MiniMax-AI/MiniMax-Provider-Verifier)
* Metric: function-call schema validity, argument match, and end-to-end response correctness
* Test Results:

```text Output theme={null}
TODO — replace with real tool-call accuracy after benchmark run on Hy3-preview (BF16).
```

### 5.2 Speed Benchmark

#### 5.2.1 Low Concurrency

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model tencent/Hy3-preview \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 10 \
  --max-concurrency 1
```

* Test Results:

```text Output theme={null}
TODO — replace with real low-concurrency output on Hy3-preview (BF16).
```

#### 5.2.2 High Concurrency

* Benchmark Command:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model tencent/Hy3-preview \
  --dataset-name random \
  --random-input-len 1000 \
  --random-output-len 1000 \
  --num-prompts 500 \
  --max-concurrency 100
```

* Test Results:

```text Output theme={null}
TODO — replace with real high-concurrency output on Hy3-preview (BF16).
```


# Hy3
Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy3

Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel.
>
> **Python (pip / uv)**
>
> ```bash theme={null}
> pip install -U uv
> uv venv --python 3.12 && source .venv/bin/activate
>
> # Install from source (main carries the suffix-aware `hunyuan` parser + the
> # HYV3 model code). Once a tagged release picks it up,
> # `uv pip install --prerelease=allow sglang` is enough.
> git clone https://github.com/sgl-project/sglang.git
> cd sglang
> uv pip install --prerelease=allow -e python
> ```
>
> Run the **Python** output of the command panel below in that environment.
>
> **Docker**
>
> ```bash Command theme={null}
> # The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser.
> docker pull lmsysorg/sglang:dev
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces.
>
> > [!NOTE]
> >
> > The `dev` image bundles the HYV3 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding.

Pick your hardware + recipe to generate the launch command.

* **Low-Latency** — fastest reply for a single user. Pick for chat.
* **Balanced** — good speed with several users at once. Use for typical multi-user serving.

**Panel controls** (top of the command box):

<ul>
  <li>**Python / Docker** — bare `sglang serve …` for an existing SGLang env, or a `docker run … sglang serve …` wrap against the per-hardware image from the [Install SGLang](#install) panel above.</li>
  <li>**⧉ Copy** — copies the current command (with whichever framing is active) to your clipboard.</li>
  <li>**\$ cURL** — a sample request against `localhost:30000` to confirm the server is up.</li>
  <li>**⚙ Env** — edits the placeholders (`HOST_IP`, `PORT`, `HF_TOKEN`, `NODE_RANK`, `NODE0_IP`) the command and cURL share. Persists in localStorage across cookbooks.</li>
  <li>**Verified / Not Verified** badge — green when the `(hw, variant, quant, strategy, nodes)` combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
</ul>

## Playground

The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change.

The knobs come in two flavors:

* **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
* **Hy3 specific** — `--tool-call-parser auto` / `--reasoning-parser auto` (auto-detect Hy3's suffix-aware `hunyuan` parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime).

Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base's **Verified** badge; any actual change flips it to **Not Verified** until the new configuration is run end-to-end and submitted back.

**Panel controls** reuse **Python / Docker** · **⧉ Copy** · **\$ cURL** · **⚙ Env** from the Deploy panel, plus one extra:

<ul>
  <li>**Submit ↗** — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says **Not Verified**; click it once you've actually run the command on your hardware and confirmed it works.</li>
</ul>

## 1. Model Introduction

**Hy3** is Tencent's third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving.

**Key Features:**

* **MoE Architecture**: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost.
* **Hybrid Thinking**: Reasoning modes (`high`, `low`, `no_think`) controllable via OpenAI-standard `reasoning_effort`, allowing the same weights to trade off latency and depth of reasoning.
* **Native Tool Calling**: Trained on a structured grammar. Pairs with SGLang's `hunyuan` tool-call parser for streaming OpenAI-compatible function-calling output.
* **Long Context**: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
* **Multi-Token Prediction (MTP)**: Ships with a built-in MTP draft module enabling speculative decoding out of the box.

**Available Model:**

* [tencent/Hy3](https://huggingface.co/tencent/Hy3) — BF16 instruct
* [tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8) — FP8

**Recommended Generation Parameters:**

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`temperature`</td>
      <td>0.9</td>
    </tr>

    <tr>
      <td>`top_p`</td>
      <td>1.0</td>
    </tr>

    <tr>
      <td>`reasoning_effort`</td>
      <td>`high` / `low` (thinking) or `no_think` (instant)</td>
    </tr>
  </tbody>
</table>

**Special tokens.** The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. `<tool_calls:TAG>` instead of the bare `<tool_calls>`). SGLang's `hunyuan` parsers resolve the real token strings from the tokenizer vocab at runtime ([PR #29920](https://github.com/sgl-project/sglang/pull/29920)), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why `--reasoning-parser hunyuan` / `--tool-call-parser hunyuan` work out of the box on the shipping model.

## 2. Configuration Tips

**Hardware requirements (BF16, \~590GB weights):**

<table>
  <thead>
    <tr>
      <th>GPU</th>
      <th>VRAM</th>
      <th>TP</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>H200</td>
      <td>141GB</td>
      <td>8</td>
      <td>minimum single-node for BF16</td>
    </tr>

    <tr>
      <td>B200</td>
      <td>192GB</td>
      <td>4</td>
      <td>BF16 590GB → 148GB/GPU</td>
    </tr>

    <tr>
      <td>B300 / GB300</td>
      <td>288GB</td>
      <td>4</td>
      <td>BF16 590GB → 148GB/GPU; ample KV headroom</td>
    </tr>

    <tr>
      <td>GB200</td>
      <td>192GB</td>
      <td>4</td>
      <td>single-node 4×192GB = 768GB fits BF16 590GB</td>
    </tr>
  </tbody>
</table>

**Blackwell attention backend.** On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the `trtllm_mha` attention backend for HYV3's MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason.

**MTP (Multi-Token Prediction, EAGLE).**

* `low-latency`: steps=3, draft-tokens=4 → largest win at bs=1.
* `balanced`: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient.

**`reasoning_effort` vs `thinking`.** The Hy3 chat template is driven by `reasoning_effort` (`high` / `low` / `no_think`), NOT by the `thinking` flag that some other families use. The default is `no_think` (instant). To opt into thinking, pass `reasoning_effort="high"` on the request (the OpenAI-standard field; sglang forwards it to the template). `reasoning_effort: max` is rejected by sglang — use `high`. For eval, sgl-eval's `--thinking` flag translates to `reasoning_effort="high"` for Hy3, so the benchmark commands below use it as-is.

## 3. Advanced Usage

### 3.1 Reasoning (Hybrid Thinking)

Hy3 is a hybrid-thinking model. Control the thinking budget via `reasoning_effort`:

* `high` / `low` — increasing amounts of chain-of-thought in `reasoning_content`
* `no_think` — skip thinking entirely (instant responses, content-only)

Enable the reasoning parser during deployment so the thinking section is separated into `reasoning_content`:

> **Deploy with reasoning parser**
>
> ```bash Command theme={null}
> sglang serve \
>   --model-path tencent/Hy3 \
>   --tp 8 \
>   --reasoning-parser auto \
>   --tool-call-parser auto
> ```

> **Example: thinking (reasoning\_effort=high)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy3",
>     messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
>     reasoning_effort="high",
>     max_tokens=2048,
> )
>
> msg = response.choices[0].message
> print("=============== Thinking =================")
> print(msg.reasoning_content)
> print("=============== Content =================")
> print(msg.content)
> ```
>
> ```text Output theme={null}
> =============== Thinking =================
> We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240.
> 10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36.
> =============== Content =================
> To find 15% of 240, follow these steps:
>
> 1. 15% = 15/100 or 0.15.
> 2. Multiply 240 by 0.15: 0.15 × 240 = 36.
> 3. Check: 10% of 240 = 24, 5% = 12, 15% = 36.
>
> Thus, 15% of 240 is 36.
> ```

> **Example: instant mode (reasoning\_effort=no\_think)**
>
> ```python Example theme={null}
> response = client.chat.completions.create(
>     model="tencent/Hy3",
>     messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
>     reasoning_effort="no_think",
>     max_tokens=256,
> )
>
> print("Content:", response.choices[0].message.content)
> ```
>
> ```text Output theme={null}
> Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.
> ```

### 3.2 Tool Calling

Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs:

> **Deploy with tool-call parser**
>
> ```bash Command theme={null}
> sglang serve \
>   --model-path tencent/Hy3 \
>   --tp 8 \
>   --reasoning-parser auto \
>   --tool-call-parser auto
> ```

> **Example: non-streaming tool call**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city.",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "city": {"type": "string"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="tencent/Hy3",
>     messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
>     tools=tools,
> )
>
> msg = response.choices[0].message
> print("Reasoning:", msg.reasoning_content)
> print("Content:  ", msg.content)
> for tc in msg.tool_calls or []:
>     print(f"Tool Call: {tc.function.name}")
>     print(f"  Arguments: {tc.function.arguments}")
> ```
>
> ```text Output theme={null}
> Reasoning: None
> Content:   I'll get the current weather for Beijing in Fahrenheit for you.
> Tool Call: get_weather
>   Arguments: {"city": "Beijing", "unit": "fahrenheit"}
> ```

> **Example: streaming tool call (incremental argument deltas)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> stream = client.chat.completions.create(
>     model="tencent/Hy3",
>     messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
>     tools=tools,
>     stream=True,
> )
>
> tool_buffer = {}
> for chunk in stream:
>     delta = chunk.choices[0].delta
>     if delta.content:
>         print(delta.content, end="", flush=True)
>     for tc in delta.tool_calls or []:
>         buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
>         if tc.function and tc.function.name:
>             buf["name"] += tc.function.name
>         if tc.function and tc.function.arguments:
>             buf["args"] += tc.function.arguments
>
> for idx, buf in tool_buffer.items():
>     print(f"\nTool[{idx}] {buf['name']}({buf['args']})")
> ```
>
> ```text Output theme={null}
> I'll check the current weather in Beijing for you using Fahrenheit.
> Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})
> ```


# Hy4 preview
Source: https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy4-Preview

Deploy Tencent Hy4 preview with SGLang — launch recipes for the 770B-parameter Mixture-of-Experts model with MLA, DeepSeek Sparse Attention (DSA), and MTP speculative decoding, in BF16 on H200/B200/B300/GB300 and MXFP8 on Blackwell GPUs.

## Deployment

> **Install SGLang**
>
> For all methods and hardware platforms, see the [official SGLang installation guide](/docs/get-started/install). The Docker path below matches the **Docker** framing in the command panel.
>
> **Docker**
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:hy4-preview
> ```
>
> > [!NOTE]
> >
> > The `hy4-preview` image bundles the HYV4 model code, the suffix-aware `hunyuan` reasoning/tool-call parsers, and the NEXTN MTP runtime. Switch to `:latest` once a tagged release picks them up.
>
> For how to launch the image, see [Install → Method 3: Using Docker](/docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware + recipe to generate the launch command. The two serving strategies map to whether the MTP (NextN) draft layer is on:

* **Low-Latency** — MTP speculative decoding on (steps=3, draft-tokens=4). Fastest reply for a single user; pick for chat.
* **High-Throughput** — MTP off. At saturation the draft+verify overhead outweighs the speedup; best for batch jobs.

**Panel controls** (top of the command box):

<ul>
  <li>**Python / Docker** — bare `sglang serve …` for an existing SGLang env, or a `docker run … sglang serve …` wrap against the per-hardware image from the [Install SGLang](#install) panel above.</li>
  <li>**⧉ Copy** — copies the current command (with whichever framing is active) to your clipboard.</li>
  <li>**\$ cURL** — a sample request against `localhost:30000` to confirm the server is up.</li>
  <li>**⚙ Env** — edits the placeholders (`HOST_IP`, `PORT`, `HF_TOKEN`, `NODE_RANK`, `NODE0_IP`) the command and cURL share. Persists in localStorage across cookbooks.</li>
  <li>**Badge** — single-node recipes show green **Verified** (run end-to-end on real hardware); the 2-node BF16 recipes show **In Progress** and flip to **Verified** as their multi-node verification lands.</li>
</ul>

## Playground

The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change.

The knobs come in two flavors:

* **Built-in SGLang features** — parallelism overrides (TP / DP-Attention), the MoE a2a backend (DeepEP, which sets EP = TP), reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
* **Hy4 specific** — `--reasoning-parser auto` / `--tool-call-parser auto` resolve to the suffix-aware Hunyuan parsers and read the real structural-token strings from the tokenizer vocab at runtime.

Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the base recipe but stripped by an override. Parallelism combinations beyond the listed recipes (DP-Attention, DeepEP, other TP degrees) are experimentation territory — any override flips the badge to **Not Verified** until the configuration is run end-to-end and submitted back.

**Panel controls** reuse **Python / Docker** · **⧉ Copy** · **\$ cURL** · **⚙ Env** from the Deploy panel, plus one extra:

<ul>
  <li>**Submit ↗** — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says **Not Verified**; click it once you've actually run the command on your hardware and confirmed it works.</li>
</ul>

## 1. Model Introduction

**Hy4 preview** is Tencent's next-generation flagship Mixture-of-Experts language model: 770B total parameters with 49B active per token, pairing a DeepSeek-style MLA + sparse-attention stack with Hunyuan's own residual control, MoE routing, and attention gating. It is a **text-only** model (vocab 120,832; the endpoint rejects image input by design), released under **Apache-2.0**.

**Key architecture:**

* **MoE**: 78 layers — layer 0 is a dense MLP, the remaining 77 are sparse MoE with 256 routed experts + 1 shared expert, top-8 sigmoid-scored routing (routed scaling 2.827), expert intermediate size 2048, and bounded SwiGLU (clamp 10.0).
* **MLA + DSA on every layer**: Multi-head Latent Attention (`q_lora_rank` 2048, `kv_lora_rank` 512, 192 nope + 64 rope head dims, `v_head_dim` 256) under DeepSeek Sparse Attention — indexer top-k 2048 with 32 index heads, indexers alternating full/shared (one full indexer every 4 layers), and an FP8 index cache.
* **iHC residual control**: Hunyuan's own inter-layer residual gating (`enable_ihc`, `hc_mult` 4) with pre- and post-residual gate groups. Semantically distinct from DeepSeek-V4's mHC (no combination step, no Sinkhorn) — the implementations are not interchangeable.
* **Gated MLA + attention sink**: element-wise attention output gating evaluated in fp32, plus a learnable per-head attention sink propagated through all attention backends.
* **MTP (NextN)**: one built-in multi-token-prediction draft layer (`model.mtp_layers.0`, 10B parameters with \~0.7B active, present in both checkpoints) enabling speculative decoding out of the box.
* **Long context**: 1M max positions (RoPE theta 1e7). Size `--context-length` to your KV budget — the sizing table in §2 suggests 262,144 (131,072 on H200).

**Available models:**

* [tencent/Hy4-preview](https://huggingface.co/tencent/Hy4-preview) — BF16 (\~1.5TB weights)
* [tencent/Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8) — MXFP8 (ModelOpt recipe, UE8M0 group-32 weight scales, dynamic activations; \~760GB weights)

**Recommended generation:** `temperature=0.9`, `top_p=1.0` (informational — SGLang applies the checkpoint's `generation_config.json` defaults; don't hardcode sampling parameters in client code). Thinking depth is controlled per request via `reasoning_effort` (defaults to `high`; `no_think` for direct responses — see §3.1).

**Resources:** [HuggingFace](https://huggingface.co/tencent/Hy4-preview) · [GitHub](https://github.com/Tencent-Hunyuan/Hy4-preview).

**Special tokens.** The Hy4 preview tokenizer's structural tokens are suffix-bearing (`<think:opensource>`, `<tool_calls:opensource>`, `<tool_call:opensource>`, `<arg_key:opensource>`, `<arg_value:opensource>`). SGLang's Hunyuan reasoning/tool-call parsers resolve the real token strings from the tokenizer vocab at runtime, so `--reasoning-parser auto --tool-call-parser auto` work out of the box.

## 2. Configuration Tips

**Hardware sizing.** BF16 weights are \~1.5TB and MXFP8 \~760GB; the MLA KV cache (compressed `kv_lora` 512 + rope 64, plus the DSA FP8 indexer cache, ≈95KB/token) is replicated per TP rank, so the per-rank pool left after weights sets the context ceiling:

<table>
  <thead>
    <tr>
      <th>GPU</th>
      <th>VRAM</th>
      <th>MXFP8 (\~760GB)</th>
      <th>BF16 (\~1.5TB)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>H200</td>
      <td>141GB</td>
      <td>Not supported — the MXFP8 kernel path requires SM100 (Blackwell)</td>
      <td>TP16, 2×8 nodes · 131K context</td>
    </tr>

    <tr>
      <td>B200</td>
      <td>192GB</td>
      <td>TP8, single node · 262K context</td>
      <td>TP16, 2×8 nodes · 262K context (8×192GB ≈ the weights alone)</td>
    </tr>

    <tr>
      <td>B300</td>
      <td>288GB</td>
      <td>TP4, single node · 262K context (\~190GB/rank)</td>
      <td>TP8, single 8-GPU node · 262K context</td>
    </tr>

    <tr>
      <td>GB300</td>
      <td>288GB</td>
      <td>TP4, single node · 262K context</td>
      <td>TP8, 2×4 nodes · 262K context (GB300 hosts carry 4 GPUs)</td>
    </tr>
  </tbody>
</table>

On H200 (BF16, TP16 → \~95GB of weights per rank) the per-rank pool holds roughly 260K tokens of KV — set `--context-length 131072` there (the model default is 1M positions, far beyond the pool); the 192GB+ parts comfortably support 262144.

**DSA attention backend.** Every layer runs DeepSeek Sparse Attention; SGLang auto-selects the DSA backend for HYV4 (`--attention-backend dsa` with `flashmla_sparse` prefill/decode and an FP8 indexer cache over a bf16 KV pool), so the recipes don't pass attention flags. Override only with a kernel-specific reason.

**MXFP8 kernel stack.** The MXFP8 checkpoint self-describes via its ModelOpt `hf_quant_config` (dynamic activations, UE8M0 group-32 weight scales) — no `--quantization` flag needed; the recipes pin `--moe-runner-backend deep_gemm --fp8-gemm-backend deep_gemm`, the validated HYV4 MXFP8 path (the runtime also defaults both to `deep_gemm` for HYV4 when the flags are left unset). The MXFP8 kernel path requires SM100+ (Blackwell); H200 (SM90) cannot serve the MXFP8 checkpoint — use BF16 there.

**CUDA graph decode vs eager.** Decode CUDA-graph capture is on by default; long-duration soak validation of the graph path on Hy4 is still in progress. If you hit instability under long mixed agentic workloads, pass `--disable-cuda-graph` to fall back to eager decode (a restart recovers cleanly either way).

**MTP (NextN) speculative decoding.** Both checkpoints ship one draft layer; the preset is `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-num-draft-tokens 4` (top-k 1). Speculative decoding reserves 4 draft-token slots per request, so the effective request budget is `prompt_tokens + max_tokens + 4 ≤ context length` — requests at the exact context boundary are rejected with the reservation accounted for.

**Fail-fast guardrails.** The model rejects pipeline parallelism and `--enable-prefill-cp` before allocation. The recipes run pure TP; DP-Attention, DeepEP (which sets EP = TP at runtime), and other TP degrees are Playground experimentation territory.

**Multi-node BF16.** The BF16 weights don't fit a single H200/B200/GB300 host, so those cells are 2-node TP recipes — run the generated command on every node (the panel injects `--nnodes 2 --node-rank --dist-init-addr`) and keep the weights on storage shared across ranks.

**Text-only.** Image input is rejected with HTTP 400 by design — don't route vision traffic to this endpoint.

**Large prefills under concurrency.** First-prefill latency on very large prompts can exceed 30 s under high concurrency; use a client timeout of 300 s (and moderate concurrency) for long-context agentic workloads instead of the common 30 s default.

## 3. Advanced Usage

### 3.1 Reasoning (`reasoning_effort`)

Hy4 preview is a hybrid-thinking model: reasoning defaults to `high` (deep chain-of-thought — suits math, coding, and agentic tasks). The top-level OpenAI `reasoning_effort` field accepts the standard tiers (e.g. `high`); for direct responses, pass the model-specific `no_think` through `extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}`. The Deploy recipes enable the reasoning parser (`--reasoning-parser auto`) so thinking is separated into `reasoning_content` and the final answer into `content`:

> **Example: thinking (reasoning\_effort=high) (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-preview-FP8",
>     messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
>     reasoning_effort="high",
>     max_tokens=2048,
> )
>
> msg = response.choices[0].message
> print("=============== Thinking =================")
> print(msg.reasoning_content)
> print("=============== Content =================")
> print(msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update — will be captured verbatim from a live Hy4 preview server.
> ```

> **Example: instant mode (reasoning\_effort=no\_think) (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-preview-FP8",
>     messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
>     extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
>     max_tokens=256,
> )
>
> print("Content:", response.choices[0].message.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update — will be captured verbatim from a live Hy4 preview server.
> ```

### 3.2 Tool Calling

Hy4 preview emits tool calls through suffixed structural tokens with an `arg_key` / `arg_value` argument format; SGLang's Hunyuan tool-call parser reassembles them into OpenAI-compatible `message.tool_calls` with schema-aware type coercion, for both streaming and non-streaming requests. The Deploy recipes enable both parsers together (`--reasoning-parser auto --tool-call-parser auto`) — the reasoning parser strips thinking tokens before the tool-call parser runs.

> [!NOTE]
>
> Tool-call output is parsed, not grammar-constrained: `tool_choice: "required"` / named-function forcing is not enforced with structural-tag guided decoding on the current implementation.

> **Example: non-streaming tool call (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a city.",
>             "parameters": {
>                 "type": "object",
>                 "properties": {
>                     "city": {"type": "string"},
>                     "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
>                 },
>                 "required": ["city"],
>             },
>         },
>     }
> ]
>
> response = client.chat.completions.create(
>     model="tencent/Hy4-preview-FP8",
>     messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
>     tools=tools,
> )
>
> msg = response.choices[0].message
> print("Reasoning:", msg.reasoning_content)
> print("Content:  ", msg.content)
> for tc in msg.tool_calls or []:
>     print(f"Tool Call: {tc.function.name}")
>     print(f"  Arguments: {tc.function.arguments}")
> ```

> **Example Output**
>
> ```text Output theme={null}
> Pending update — will be captured verbatim from a live Hy4 preview server.
> ```


# Inkling
Source: https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling

Deploy Inkling with SGLang — verified launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' 975B Mixture-of-Experts model with 1M-token context.

## Deployment

> **Install SGLang**
>
> For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
>
> **Python (pip / uv)**
>
> Inkling has merged to `main` but isn't in a `pip` release yet — install from source:
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
> ```
>
> Then run the **Python** output of the command panel below.
>
> **Docker**
>
> > [!NOTE]
> >
> > The Inkling images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status.
>
> There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-inkling-dspark      # CUDA 13
> docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired)
> docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark  # AMD MI350X / MI355X
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).

**Panel controls** (top of the command box):

<ul>
  <li>**⧉ Copy** — copies the current command to your clipboard.</li>
  <li>**\$ cURL** — a sample request against `localhost:30000` to confirm the server is up.</li>
  <li>**⚙ Env** — edits the placeholders (`HOST_IP`, `PORT`, `NODE_RANK`, `NODE0_IP`) the command and cURL share.</li>
  <li>**Verified / Not Verified** badge — green when the `(hw, variant, quant, strategy, nodes)` combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
</ul>

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.

Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.

## 1. Model Introduction

**Inkling** is a Mixture-of-Experts model from Thinking Machines — **975B** total parameters, **41B** active per token, with a **1M-token** context window and **open weights** (BF16 and NVFP4 checkpoints below). It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache).

**Resources:** HuggingFace — [Inkling](https://huggingface.co/thinkingmachines/Inkling) (BF16) · [Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4).

## 2. Configuration Tips

**Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving.

**Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage.

**MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage.

**Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):

<table>
  <thead>
    <tr>
      <th>reasoning\_effort</th>
      <th>value</th>
    </tr>
  </thead>

  <tbody>
    <tr><td>`none`</td><td>0.0</td></tr>
    <tr><td>`minimal`</td><td>0.1</td></tr>
    <tr><td>`low`</td><td>0.2</td></tr>
    <tr><td>`medium`</td><td>0.7</td></tr>
    <tr><td>`high`</td><td>0.9</td></tr>
    <tr><td>`xhigh`</td><td>0.99</td></tr>
    <tr><td>`max`</td><td>0.99</td></tr>
  </tbody>
</table>

## 3. Advanced Usage

### 3.1 Reasoning

Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-NVFP4",
>     messages=[{"role": "user", "content": "What is 17 times 24?"}],
>     extra_body={"chat_template_kwargs": {"thinking": True}},
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user is asking for the product of 17 and 24. Let me calculate that.
>
> 17 × 24
>
> I can break this down:
> 17 × 20 = 340
> 17 × 4 = 68
> 340 + 68 = 408
>
> Alternatively:
> 24 × 10 = 240
> 24 × 7 = 168
> 240 + 168 = 408
>
> So the answer is 408.
> Answer: 17 times 24 is **408**.
>
> Here's a quick breakdown:
> - 17 × 20 = 340
> - 17 × 4 = 68
> - 340 + 68 = **408**
> ```

### 3.2 Tool Calling

Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"location": {"type": "string", "description": "The city name"}},
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-NVFP4",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location.
> Content:
> Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
> ```

### 3.3 Multimodal Input (Image + Audio)

Inkling is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. Audio can also be sent as OpenAI's `input_audio` part, carrying the base64 bytes in `data` alongside a `format` of `wav` or `mp3`. The server must be started with `--enable-multimodal` (already included in every recipe above).

> **Image + Audio Example (Python)**
>
> ```python Example theme={null}
> import base64
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> with open("image.png", "rb") as f:
>     image_b64 = base64.b64encode(f.read()).decode()
> with open("audio.wav", "rb") as f:
>     audio_b64 = base64.b64encode(f.read()).decode()
>
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-NVFP4",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
>                 {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
>                 {"type": "text", "text": "Describe the image, then transcribe the audio."},
>             ],
>         }
>     ],
>     max_tokens=1024,
> )
> print(resp.choices[0].message.content)
> ```

> [!NOTE]
>
> Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows.

### 3.4 LoRA (Serving Adapters)

The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name.

Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`:

> **LoRA Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> # Option A (recommended): "<model>:<adapter-name>" in the model field
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-NVFP4:lora0",
>     messages=[{"role": "user", "content": "Summarize the changelog."}],
> )
>
> # Option B: explicit lora_path via extra_body
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-NVFP4",
>     messages=[{"role": "user", "content": "Summarize the changelog."}],
>     extra_body={"lora_path": "lora0"},
> )
>
> print(resp.choices[0].message.content)
> ```

> [!NOTE]
>
> One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence.

### 3.5 HiCache (Hierarchical KV Caching)

Inkling serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads.

To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`.

### 3.6 Long Context (MXFP8 KV)

The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound.

**Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200).

The tradeoff is not just a \~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality.

To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300.

### 3.7 DSpark (Speculative Decoding)

The **DSpark** deploy strategy is the second speculative-decoding path for Inkling. Unlike **MTP**, which drives Inkling's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-DSpark-Preview` — served unquantized alongside the NVFP4 target.

DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).


# Inkling-Small
Source: https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling-Small

Deploy Inkling-Small with SGLang — launch commands, tuning, and multimodal / reasoning / tool-calling usage for Thinking Machines' Inkling-Small Mixture-of-Experts model.

## Deployment

> **Install SGLang**
>
> For all install methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install).
>
> **Python (pip / uv)**
>
> Inkling-Small has merged to `main` but isn't in a `pip` release yet — install from source:
>
> ```bash Command theme={null}
> pip install --upgrade pip
> pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python'
> ```
>
> Then run the **Python** output of the command panel below.
>
> **Docker**
>
> > [!NOTE]
> >
> > The Inkling-Small images are being published to [`lmsysorg/sglang`](https://hub.docker.com/r/lmsysorg/sglang/tags) — watch the tag list for status.
>
> There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the CUDA build by your CUDA version, not your GPU. DGX Spark (GB10) uses a dedicated arm64 CUDA 13 image:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev-inkling-dspark      # CUDA 13
> docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired)
> docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121)
> docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark  # AMD MI350X / MI355X
> ```
>
> For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces.

Pick your hardware to generate the launch command. Each platform ships a **Balanced** recipe plus **MTP** and **DSpark** (speculative decoding) tiers and a **Long Context (MXFP8 KV)** tier where validated; the **LoRA** variant serves adapters on top of the frozen base model. Set `MAX_LORAS` to the number of distinct adapters you serve (1 is fastest for single-adapter serving).

**Panel controls** (top of the command box):

<ul>
  <li>**⧉ Copy** — copies the current command to your clipboard.</li>
  <li>**\$ cURL** — a sample request against `localhost:30000` to confirm the server is up.</li>
  <li>**⚙ Env** — edits the placeholders (`HOST_IP`, `PORT`, `NODE_RANK`, `NODE0_IP`) the command and cURL share.</li>
  <li>**Verified / Not Verified** badge — green when the `(hw, variant, quant, strategy, nodes)` combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.</li>
</ul>

## Playground

The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations that have been signed off; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. The base is read live from your Deploy selection — only your overrides change.

Lines highlighted **green** are added by your overrides; lines with **red strikethrough** were in the verified base but stripped by an override. Any change flips the badge to **Not Verified** until the new configuration is run end-to-end.

## 1. Model Introduction

**Inkling-Small** is a Mixture-of-Experts model from Thinking Machines with **open weights** (BF16 and NVFP4 checkpoints below), in the same architecture family as Inkling. It handles text, image, and audio inputs natively, and exposes a **variable reasoning-effort** control to trade latency and cost against answer quality. This page covers serving Inkling-Small on SGLang, including its **MTP** speculative-decoding path and long-context prefix caching (unified radix cache + HiCache).

**Resources:** HuggingFace — [Inkling-Small](https://huggingface.co/thinkingmachines/Inkling-Small) (BF16) · [Inkling-Small-NVFP4](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4).

## 2. Configuration Tips

**Multimodal.** The recipes pass `--enable-multimodal` so the server accepts image and audio inputs alongside text — drop it for text-only serving.

**DGX Spark (2× GB10).** The verified cell runs NVFP4 with TP=2 across two Sparks over ConnectX-7 (1 GPU per node). Use the `dev-inkling-small-dgx-spark` image, Triton attention + Marlin FP4/MoE, and `--disable-prefill-cuda-graph`. The Docker command already carries the ConnectX-7 flags `--ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`.

**Memory pool ratios.** `--swa-full-tokens-ratio` and `--mamba-full-memory-ratio` (both default `0.1`) size the SWA and Mamba/sconv state pools; tune them to your workload's usage.

**MTP needs `--enable-multi-layer-eagle`.** The MTP recipe drives Inkling-Small's multi-layer draft head; without this flag the standard EAGLE worker runs against it and outputs garbage.

**Reasoning effort.** Pass `reasoning_effort` as one of the named levels below; requests that omit it default to `high`, and `max` is the strongest. Each level maps to an internal effort value (max at `0.99`):

<table>
  <thead>
    <tr>
      <th>reasoning\_effort</th>
      <th>value</th>
    </tr>
  </thead>

  <tbody>
    <tr><td>`none`</td><td>0.0</td></tr>
    <tr><td>`minimal`</td><td>0.1</td></tr>
    <tr><td>`low`</td><td>0.2</td></tr>
    <tr><td>`medium`</td><td>0.7</td></tr>
    <tr><td>`high`</td><td>0.9</td></tr>
    <tr><td>`xhigh`</td><td>0.99</td></tr>
    <tr><td>`max`</td><td>0.99</td></tr>
  </tbody>
</table>

## 3. Advanced Usage

### 3.1 Reasoning

Enable the `inkling` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer into `reasoning_content` vs `content`.

> **Reasoning Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-Small-NVFP4",
>     messages=[{"role": "user", "content": "What is 17 times 24?"}],
>     extra_body={"chat_template_kwargs": {"thinking": True}},
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Answer:", msg.content)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user is asking for the product of 17 and 24. Let me calculate that.
>
> 17 × 24
>
> I can break this down:
> 17 × 20 = 340
> 17 × 4 = 68
> 340 + 68 = 408
>
> Alternatively:
> 24 × 10 = 240
> 24 × 7 = 168
> 240 + 168 = 408
>
> So the answer is 408.
> Answer: 17 times 24 is **408**.
>
> Here's a quick breakdown:
> - 17 × 20 = 340
> - 17 × 4 = 68
> - 340 + 68 = **408**
> ```

### 3.2 Tool Calling

Enable the `inkling` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`.

> **Tool Calling Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> tools = [
>     {
>         "type": "function",
>         "function": {
>             "name": "get_weather",
>             "description": "Get the current weather for a location",
>             "parameters": {
>                 "type": "object",
>                 "properties": {"location": {"type": "string", "description": "The city name"}},
>                 "required": ["location"],
>             },
>         },
>     }
> ]
>
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-Small-NVFP4",
>     messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
>     tools=tools,
> )
> msg = resp.choices[0].message
> print("Reasoning:", getattr(msg, "reasoning_content", None))
> print("Content:", msg.content)
> print("Tool calls:", msg.tool_calls)
> ```

> **Example Output**
>
> ```text Output theme={null}
> Reasoning: The user is asking for the weather in Beijing. I have a tool called `get_weather` that can get the current weather for a location. Let me call it with "Beijing" as the location.
> Content:
> Tool calls: [ChatCompletionMessageFunctionToolCall(id='call_98f772f3a0044f45b80c5ba5', function=Function(arguments='{"location": "Beijing"}', name='get_weather'), type='function', index=0)]
> ```

### 3.3 Multimodal Input (Image + Audio)

Inkling-Small is multimodal: a single user message can mix **text**, **images**, and **audio**. Pass each media item as its own content part — `image_url` for images, `audio_url` for audio — with the `url` set to either an HTTP(S) link or a base64 `data:` URI. Audio can also be sent as OpenAI's `input_audio` part, carrying the base64 bytes in `data` alongside a `format` of `wav` or `mp3`. The server must be started with `--enable-multimodal` (already included in every recipe above).

> **Image + Audio Example (Python)**
>
> ```python Example theme={null}
> import base64
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> with open("image.png", "rb") as f:
>     image_b64 = base64.b64encode(f.read()).decode()
> with open("audio.wav", "rb") as f:
>     audio_b64 = base64.b64encode(f.read()).decode()
>
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-Small-NVFP4",
>     messages=[
>         {
>             "role": "user",
>             "content": [
>                 {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
>                 {"type": "audio_url", "audio_url": {"url": f"data:audio/wav;base64,{audio_b64}"}},
>                 {"type": "text", "text": "Describe the image, then transcribe the audio."},
>             ],
>         }
>     ],
>     max_tokens=1024,
> )
> print(resp.choices[0].message.content)
> ```

> [!NOTE]
>
> Images and audio can be sent as public HTTP(S) URLs instead of base64 — e.g. `{"type": "image_url", "image_url": {"url": "https://.../photo.jpg"}}`. Use one content part per media item; mix as many as the context budget allows.

### 3.4 LoRA (Serving Adapters)

The **LoRA** deploy variant serves adapters on top of the frozen base model. Its launch command adds `--enable-lora --lora-paths lora0={{ADAPTER_PATH}} --max-loras-per-batch {{MAX_LORAS}}` — each adapter is registered under the **name** to the left of `=` (here `lora0`). Adapters can also be added/removed at runtime via the `POST /load_lora_adapter` endpoint. To serve several adapters, pass multiple `--lora-paths name=path` at launch and reference each by its name.

Pick the adapter per request by that name — either in the `model` field with `base-model:adapter` syntax (recommended), or explicitly via `lora_path` in `extra_body`:

> **LoRA Example (Python)**
>
> ```python Example theme={null}
> from openai import OpenAI
>
> client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
>
> # Option A (recommended): "<model>:<adapter-name>" in the model field
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-Small-NVFP4:lora0",
>     messages=[{"role": "user", "content": "Summarize the changelog."}],
> )
>
> # Option B: explicit lora_path via extra_body
> resp = client.chat.completions.create(
>     model="thinkingmachines/Inkling-Small-NVFP4",
>     messages=[{"role": "user", "content": "Summarize the changelog."}],
>     extra_body={"lora_path": "lora0"},
> )
>
> print(resp.choices[0].message.content)
> ```

> [!NOTE]
>
> One adapter per request — omit the `:adapter` suffix (and `lora_path`) to hit the base model. Different requests **in the same batch** may use different adapters; the number of *distinct* adapters co-resident in a batch is capped by `--max-loras-per-batch` (the `MAX_LORAS` field, default `1`). If both `model:adapter` and `lora_path` are supplied, the `model` suffix takes precedence.

### 3.5 HiCache (Hierarchical KV Caching)

Inkling-Small serves on SGLang's **unified radix cache**: the historically separate full-attention, SWA, and Mamba/sconv caches are combined into one radix tree with typed components, and native HiCache offloads cold prefix pages across tiers (GPU HBM → host DRAM → disk / remote). This expands effective prefix-cache capacity for multi-turn and long-context workloads.

To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**, then pick a storage backend (`file` / `mooncake` / `nixl`) for the L3 tier. The Write policy defaults to `write_through`.

### 3.6 Long Context (MXFP8 KV)

The **Long Context** deploy strategy adds `--kv-cache-dtype mxfp8` on top of the Balanced recipe. KV entries are stored as block-scaled MXFP8 instead of BF16, so the SWA + Mamba/sconv memory pool holds roughly 2x as many tokens on the same GPU. Use it when you're context-bound or concurrency-bound.

**Blackwell only.** MXFP8 KV cache requires Blackwell (B200 / B300 / GB200 / GB300), it's not offered on Hopper (H200).

The tradeoff is not just a \~5% decode latency penalty from the extra quantize/dequantize work versus BF16 KV — storing KV in MXFP8 also introduces some accuracy loss at long context lengths. Treat it as a capacity lever, not a speed one — stay on **Balanced** if you have headroom in the memory pool and just want lower latency or maximum output quality.

To try it, select the **Long Context** strategy in the Deploy panel above for any NVFP4 cell; the panel regenerates the launch command with `--kv-cache-dtype mxfp8` inserted. Verified end-to-end on B200, B300, and GB300.

### 3.7 DSpark (Speculative Decoding)

The **DSpark** deploy strategy is the second speculative-decoding path for Inkling-Small. Unlike **MTP**, which drives Inkling-Small's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-Small-DSpark` — served unquantized alongside the NVFP4 target.

DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4).


# MiMo-V2-Flash
Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash



## Introduction

XiaomiMiMo/MiMo-V2-Flash, with 309B total parameters and 15B activated parameters, is a new inference-centric model designed to maximize decoding efficiency created by XiaomiMiMo Team explicitly co-designed for real-world serving workloads, enabling flexible tradeoffs between throughput and latency on different hardware.

This model creates a new balance between long-context modeling capability and inference efficiency. Key features include:

* **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) with a 5:1 ratio and an aggressive 128-token window. This reduces KV-cache storage by nearly 6x while maintaining long-context performance via learnable attention sink bias.
* **Multi-Token Prediction (MTP)**: Equipped with a lightweight MTP module (0.33B params/block) using dense FFNs. This triples output speed during inference and will be good to accelerates rollout in RL training.
* **Efficient Pre-Training**: Trained on 27T tokens using FP8 mixed precision and native 32k seq length. The context window supports up to 256k length.
* **Agentic Capabilities**: Post-training utilizes Multi-Teacher On-Policy Distillation (MOPD) and large-scale agentic RL, achieving superior performance on SWE-Bench and complex reasoning tasks.

## Installation

MiMo-V2-Flash is currently available in SGLang via Docker image and pip install.

### Docker

```bash Command theme={null}
# Pull the docker image
docker pull lmsysorg/sglang:latest

# Launch the container
docker run -it --gpus all \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  lmsysorg/sglang:latest bash
```

### Pip Installation

```bash Command theme={null}
# On a machine with SGLang dependencies installed or inside a SGLang nightly container
# Start an SGLang nightly container
docker run -it --gpus all \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  lmsysorg/sglang:latest bash

# If you already have SGLang installed, uninstall the current SGLang version
pip uninstall sglang -y

# Install the PyPI Package
pip install sglang==0.5.6.post2.dev8005+pr.15207.g39d5bd57a \
  --extra-index-url https://sgl-project.github.io/whl/pr/
```

## Model Deployment

Use the configuration selector below to automatically generate the appropriate deployment command.

Run this Command:

<pre />

MI355X (ROCm) is validated in the selector above with `--tp-size 4`, Triton attention, and `--disable-custom-all-reduce`. `--tp-size 8` hit a QKV sharding error during validation. EAGLE speculative decoding is still WIP on MI355X.

## Testing the deployment

Once the server is running, test it with a chat completion request in another terminal:

```bash Command theme={null}
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "XiaomiMiMo/MiMo-V2-Flash",
    "messages": [
      {"role": "user", "content": "Hello! What can you help me with?"}
    ],
    "temperature": 0.7,
    "max_tokens": 100
  }'
```

**Expected response:**

```json Config theme={null}
{
  "id": "...",
  "object": "chat.completion",
  "model": "XiaomiMiMo/MiMo-V2-Flash",
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "Hello! I can help you with..."
    }
  }]
}
```

## Troubleshooting

**DeepGEMM Timeout Error**

Occasionally DeepGEMM timeout errors occur during first launch. Simply rerun the server command in the same container - the compiled kernels are cached and subsequent launches will be fast.

**ROCm MI355X Attention Backend**

If you see an error such as `AiterAttnBackend.forward_decode() got an unexpected keyword argument 'sinks'` on MI355X, use the `MI355X` + `Performance Optimizations` command from the selector above, which switches to Triton attention and keeps `--disable-custom-all-reduce`.


# MiMo-V2.5
Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5



## 1. Model Introduction

[MiMo-V2.5-Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro) and [MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5) are next-generation Mixture-of-Experts models from the XiaomiMiMo Team.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variant</th>
      <th>Total params</th>
      <th>Active (MoE)</th>
      <th>Modalities</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**[MiMo-V2.5-Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro)**</td>
      <td>**1.02T**</td>
      <td>42B</td>
      <td>Text (multimodal planned)</td>
    </tr>

    <tr>
      <td>**[MiMo-V2.5](https://huggingface.co/XiaomiMiMo/MiMo-V2.5)**</td>
      <td>**310B**</td>
      <td>15B</td>
      <td>Text, Image, Video, Audio</td>
    </tr>
  </tbody>
</table>

**Key Features:**

* **Hybrid Attention Architecture**: Interleaves Sliding Window Attention (SWA) and Global Attention (GA) for reduced KV cache while preserving long-context capability.
* **Multi-Token Prediction (MTP)**: 3-layer MTP module accelerates decoding. Both variants support EAGLE speculative decoding with MTP weights.
* **1M-Token Context**: Both variants support up to 1 million token context windows.
* **Agentic Capabilities**: Post-training with large-scale agentic RL achieves strong performance on coding, reasoning, and tool-use benchmarks.
* **MiMo-V2.5 Multimodal** (V2.5 only): Native omnimodal architecture with a 729M-param ViT Vision Encoder (28 layers: 24 SWA + 4 Full) and a 261M-param Audio Transformer (24 layers: 12 SWA + 12 Full); supports image, video, and audio understanding via standard OpenAI-compatible multimodal API.

**License:** Apache 2.0

## 2. SGLang Installation

Refer to the [official SGLang installation guide](../../../docs/get-started/install).

**Docker Image:** All variants (MiMo-V2.5 310B and MiMo-V2.5-Pro 1.02T) use `lmsysorg/sglang:latest`, which ships CUDA 13.0 and runs on both Hopper (H100 / H200) and Blackwell (B200 / GB300).

**TPU (sgl-jax):** MiMo-V2.5-Pro can also be served on TPU via the JAX-based [sgl-jax](https://github.com/sgl-project/sglang-jax) runtime. The container image and `pip install` steps are listed in [§3.3 TPU Deployment](#3-3-tpu-deployment-mimo-v2-5-pro-sgl-jax).

## 3. Model Deployment

### 3.1 Basic Configuration

Use the selector below to generate the deployment command for your variant and hardware.

Run this Command:

<pre />

### 3.2 Configuration Tips

**MiMo-V2.5-Pro (1.02T):**

* **B200**: single node, TP=8 (verified). Uses `--attention-backend fa4` + `--moe-runner-backend flashinfer_trtllm` + `--mem-fraction-static 0.8`. Set `--swa-full-tokens-ratio 0.1` to keep KV-cache footprint within 192 GB HBM.
* **GB300**: 2 nodes, TP=8 (verified). Same Blackwell stack as B200; multi-node interconnect requires `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1`. Default SWA ratio is fine.
* **H100/H200**: 2 nodes × 8 GPUs (TP=16, not yet verified). Uses the Hopper stack (`fa3` + DeepEP + EAGLE multi-layer); fits with `--mem-fraction-static 0.7` and `--swa-full-tokens-ratio 0.3`. DeepEP dispatch tuning: `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` avoids memory spikes during prefill.
* EAGLE speculative decoding (3 steps, topk=1) typically yields a 2–3× decode speedup. Requires `--enable-multi-layer-eagle` (both Hopper and Blackwell). See §5.4 for acceptance-rate behavior on natural text vs random prompts.

**MiMo-V2.5 (310B):**

* The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. Use `--attn-dp-size = TP / 4`; for TP > 4 this means DP-attention is on. Total GPUs must be a multiple of 4. A bare `--tp 8` without `--attn-dp-size 2` will fail to load with `MiMoV2 fused qkv_proj checkpoint is TP=4-interleaved; got attention tp_size=8`.
* Single-node deployments: H100/H200 8× GPUs (`--tp 8 --attn-dp-size 2`), B200 4× GPUs (`--tp 4`, attention DP size 1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization.
* On Blackwell, pass `--attention-backend fa4`: MiMoV2's asymmetric KV (`head_dim` 192 / `v_head_dim` 128) fails on the SM100 default `trtllm_mha`, which requires equal K/V widths.
* On Blackwell, pass `--mm-attention-backend fa4` for the V2.5 vision encoder. The checkpoint config requests FlashAttention-3 internally, but SGLang rejects FA3 on Blackwell and expects FA4 for multimodal attention.
* On Blackwell, pass `--moe-runner-backend flashinfer_trtllm`; the default `auto` falls through to the triton fused-MoE runner, \~12% slower at bs=1 decode.
* `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--attn-dp-size` is greater than 1, to keep LM head and encoder sharding consistent.
* EAGLE MTP uses the checkpoint's MTP weights. Enable with `--speculative-algorithm EAGLE` and `--enable-multi-layer-eagle` (both Hopper and Blackwell).
* **Multimodal**: Supports image, video, and audio understanding; see Section 4.3 for invocation examples.

**DeepEP (optional toggle, Hopper-only):**

* DeepEP replaces the default MoE all-to-all dispatch with a fused [DeepEP](https://github.com/deepseek-ai/DeepEP) backend; it lowers expert dispatch latency and memory traffic, so it pays off under **high concurrency / throughput-bound workloads** on H100/H200. Under concurrency=1 / latency-bound workloads the gain is negligible — leave it off.
* Enabling adds `--moe-a2a-backend deepep` + `--moe-dense-tp-size 1` (and `--ep <tp>` for Pro) plus `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256` env to cap the dispatch buffer. Requires `pip install deep_ep` (not part of the default sglang install).
* On Blackwell (B200, GB300) the verified MoE backend is `flashinfer_trtllm`; the DeepEP toggle is a no-op there.

### 3.3 TPU Deployment (MiMo-V2.5-Pro, sgl-jax)

MiMo-V2.5-Pro can also be served on TPU via [sgl-jax](https://github.com/sgl-project/sglang-jax). The runtime is a separate JAX-based stack (`sgl_jax.launch_server`); pick **TPU v7x** or **TPU v6e** in the panel above to generate the launch command. Verified topologies:

| TPU Type | Topology | Chips/Node | Nodes | Total Chips | JAX Devices/Chip | Total JAX Devices (= `--tp-size`) |
| - | - | - | - | - | - | - |
| **v7x** | 2×2×4 | 4 | 4 | 16 | 2 | 32 |
| **v6e** | 4×4×4 | 4 | 16 | 64 | 1 | 64 |

> v7x exposes **2 logical JAX devices per chip**, so `--tp-size = 16 chips × 2 = 32`. v6e exposes 1 device per chip, so `--tp-size = 64`. Always set `--tp-size` to the total JAX device count across all nodes, not the chip count.

All nodes must sit in the same TPU slice and reach each other on the JAX init port (`20000`) and the TPU process port (`8471`).

**Step 1 — Launch the JAX TPU container on every node:**

```shell Command theme={null}
docker run -it --privileged \
  --shm-size=32g \
  --ipc=host \
  --network=host \
  -v /dev:/dev \
  us-docker.pkg.dev/cloud-tpu-images/jax-ai-image/tpu:jax0.8.1-rev1 bash
```

> The image is pinned to `jax0.8.1-rev1` to keep the JAX runtime aligned with sgl-jax's TPU extras.

**Step 2 — Clone and install sgl-jax (inside the container):**

```shell Command theme={null}
git clone https://github.com/sgl-project/sglang-jax.git
cd sglang-jax
pip install -e "python[tpu]"
```

## 4. Model Invocation

### 4.1 Basic Usage

See [Basic API Usage](../../../docs/basic_usage/send_request).

### 4.2 Reasoning Output

Both variants support hybrid thinking mode. Thinking content is separated via the reasoning parser.

**Thinking Mode (default):**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[
        {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."}
    ]
)

print("====== Reasoning ======")
print(response.choices[0].message.reasoning_content)
print("====== Answer ======")
print(response.choices[0].message.content)
```

**Output Example (MiMo-V2.5):**

```text theme={null}
====== Reasoning ======
Comparing 9.11 and 9.9.

The integer parts are both 9. Now compare the decimal parts: 0.11 vs 0.9.

0.9 = 0.90, which is greater than 0.11.

So 9.9 > 9.11.
====== Answer ======
**9.9 is larger than 9.11.**

Here's the reasoning: When comparing decimals, line them up to the same number of decimal places:

- 9.11
- 9.90

Both have a **9** in the ones place, but in the tenths place, **9 > 1**, so 9.90 > 0.11.

**9.9 > 9.11**
```

**Thinking Off (instant mode):**

```python Example theme={null}
response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[
        {"role": "user", "content": "Which is larger, 9.11 or 9.9? Think carefully."}
    ],
    extra_body={"chat_template_kwargs": {"thinking": False}}
)

print(response.choices[0].message.content)
```

**Output Example (MiMo-V2.5):**

```text theme={null}
## Comparing 9.11 and 9.9

**9.9 is larger.**

The key is to compare them place by place. It helps to write them with the same number of decimal places:

- **9.11** → 9.11
- **9.9** → 9.90

Both have **9** in the ones place, but in the tenths place: **9** (in 9.90) is greater than **1** (in 9.11).

So **9.90 > 9.11**.
```

### 4.3 Multimodal Invocation (V2.5 only)

**Image Understanding:**

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/images/man_ironing_on_back_of_suv.png"}},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }]
)

print(response.choices[0].message.content)
```

**Output Example:**

```text theme={null}
Based on the image provided, here is a detailed description:

The image captures a whimsical or surreal scene set on a busy city street, likely in New York City given the iconic yellow cabs. In the center foreground, a man is sitting on a folding chair, casually crossing his legs. He is wearing a bright yellow hoodie with a graphic on the front and blue jeans. He is intently focused on ironing a white dress shirt that rests on an ironing board set up directly on the asphalt.

Behind him, a yellow SUV taxi cab is stopped or moving slowly, angled slightly away from the camera. To his left, another yellow taxi sedan is captured in motion blur, indicating it is driving past him. The background features tall city buildings with glass windows and storefronts. There are banners hanging from streetlights, and some greenery is visible in the distance. The overall impression is one of incongruity—performing a domestic chore like ironing in the middle of a chaotic urban environment.
```

**Video Understanding:**

```python Example theme={null}
response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": "https://videos.pexels.com/video-files/4114797/4114797-uhd_3840_2160_25fps.mp4"}},
            {"type": "text", "text": "Summarize what happens in this video."}
        ]
    }]
)

print(response.choices[0].message.content)
```

**Output Example:**

```text theme={null}
A person wearing blue protective gloves is shown operating a microscope in a close-up shot. The individual is adjusting a knob on the side of the microscope, which moves the stage holding a glass slide, likely focusing the lens on the specimen.
```

> Video decoding requires `decord` (`pip install decord`); SGLang's MiMo-V2.5 multimodal processor uses `decord.VideoReader` for frame extraction.

**Audio Understanding:**

```python Example theme={null}
response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{
        "role": "user",
        "content": [
            {"type": "audio_url", "audio_url": {"url": "https://raw.githubusercontent.com/sgl-project/sgl-test-files/refs/heads/main/audios/Trump_WEF_2018_10s.mp3"}},
            {"type": "text", "text": "Transcribe and summarize this audio."}
        ]
    }]
)

print(response.choices[0].message.content)
```

**Output Example:**

```text theme={null}
**Transcript:**
"Thank you Klaus very much. It's a privilege to be here at this forum where leaders in business, science, art, diplomacy and world affairs have gathered for..."

**Summary:**
The speaker thanks Klaus for the introduction and expresses their honor at attending a forum. They highlight that the event has brought together high-level leaders from various sectors, including business, science, art, and diplomacy.
```

### 4.4 Tool Calling

```python Example theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY"
)

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="XiaomiMiMo/MiMo-V2.5",
    messages=[{"role": "user", "content": "What's the weather in Beijing?"}],
    tools=tools
)

msg = response.choices[0].message
if msg.reasoning_content:
    print("=== Reasoning ===")
    print(msg.reasoning_content)
if msg.tool_calls:
    print("=== Tool Calls ===")
    for tc in msg.tool_calls:
        print(f"  Function: {tc.function.name}")
        print(f"  Arguments: {tc.function.arguments}")
```

**Output Example (MiMo-V2.5):**

```text theme={null}
=== Reasoning ===
The user wants to know the weather in Beijing. I have a function available called "get_weather" that can retrieve current weather for a location. Let me call that function with Beijing as the location.
=== Tool Calls ===
  Function: get_weather
  Arguments: {"location": "Beijing"}
```

## 5. Benchmark

Accuracy numbers come from `sglang.test.run_eval` (GSM8K standard 5-shot, MMMU validation split). Speed numbers come from `sglang.bench_serving` with generated random prompts; text runs use 1024 input tokens and 1024 output tokens per request, and the image run uses 2 random 720p images per request.

### 5.1 Accuracy Benchmark

#### 5.1.1 GSM8K

Standard 5-shot, `temperature=0`, `max_tokens=4096`, model defaults to thinking-on (responses contain `<think>...</think>` and the eval extracts the trailing number via regex). Server launch: see [Section 3](#3-model-deployment).

**Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --num-examples 200 \
  --num-threads 8 \
  --max-tokens 4096 \
  --temperature 0.0
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

> `run_eval.py` automatically appends `/v1` to `--base-url`; pass the bare `host:port` URL (without trailing `/v1`), otherwise requests resolve to `/v1/v1/chat/completions` and 404.

* **Test Results:**
  * MiMo-V2.5-Pro (FP8, 8× B200)
    ```
    Score:             0.965  (193 / 200)
    Latency:           253.90 s
    Output throughput: 461.78 tok/s
    ```
  * MiMo-V2.5 (FP8, 8× H200)
    ```
    Score:             0.980  (196 / 200)
    Latency:           477.52 s
    Output throughput: 88.9 tok/s
    ```

#### 5.1.2 MMMU (V2.5 only)

`MMMU/MMMU` validation split (multi-discipline multimodal), `concurrency=16`, default sampling.

* **Benchmark Command:**

```shell Command theme={null}
python3 benchmark/mmmu/bench_sglang.py \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --concurrency 16
```

* **Test Results:**
  * MiMo-V2.5 (FP8)
    ```
    Pending update
    ```

### 5.2 Speed Benchmark — MiMo-V2.5-Pro

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8×)
* Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8)
* Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`)
* Recipe: Blackwell verified baseline (EAGLE off for this benchmark — see note below)
* sglang version: 0.5.11

> The numbers in §5.2 are the **no-EAGLE baseline** on `random 1024/1024`. On uniform-random token streams the MiMo-V2.5-Pro 3-layer MTP draft has very low accept-rate (\~0.13–0.27 vs \~0.75 on natural-text prompts, see §5.4) — there's no token-co-occurrence signal for the draft to model — so EAGLE here adds verify overhead without recovering enough draft tokens to be a net win on this workload. EAGLE MTP itself works on B200 + `--enable-multi-layer-eagle` (see §3 deployment command and §5.4 for an acceptance profile on natural text).

#### 5.2.1 Latency-Sensitive Benchmark

* **Model Deployment Command:** see the [command panel above](#3-model-deployment).
* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  27.59
Total input tokens:                      1997
Total input text tokens:                 1997
Total generated tokens:                  2798
Total generated tokens (retokenized):    2794
Request throughput (req/s):              0.36
Input token throughput (tok/s):          72.38
Output token throughput (tok/s):         101.41
Peak output token throughput (tok/s):    110.00
Peak concurrent requests:                3
Total token throughput (tok/s):          173.79
Concurrency:                             1.00
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2757.26
Median E2E Latency (ms):                 3319.10
P90 E2E Latency (ms):                    4157.47
P99 E2E Latency (ms):                    4869.32
---------------Time to First Token----------------
Mean TTFT (ms):                          162.17
Median TTFT (ms):                        68.11
P99 TTFT (ms):                           929.58
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.19
Median TPOT (ms):                        9.33
P99 TPOT (ms):                           9.39
---------------Inter-Token Latency----------------
Mean ITL (ms):                           9.31
Median ITL (ms):                         9.35
P95 ITL (ms):                            9.44
P99 ITL (ms):                            9.77
Max ITL (ms):                            19.80
==================================================
```

#### 5.2.2 Throughput-Sensitive Benchmark

* **Model Deployment Command:** see the [command panel above](#3-model-deployment).
* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  112.78
Total input tokens:                      302118
Total input text tokens:                 302118
Total generated tokens:                  195775
Total generated tokens (retokenized):    191069
Request throughput (req/s):              8.87
Input token throughput (tok/s):          2678.83
Output token throughput (tok/s):         1735.90
Peak output token throughput (tok/s):    3040.00
Peak concurrent requests:                121
Total token throughput (tok/s):          4414.73
Concurrency:                             87.80
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   9901.96
Median E2E Latency (ms):                 6525.54
P90 E2E Latency (ms):                    23567.98
P99 E2E Latency (ms):                    42109.22
---------------Time to First Token----------------
Mean TTFT (ms):                          223.69
Median TTFT (ms):                        139.45
P99 TTFT (ms):                           1082.02
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          50.63
Median TPOT (ms):                        51.66
P99 TPOT (ms):                           91.41
---------------Inter-Token Latency----------------
Mean ITL (ms):                           49.79
Median ITL (ms):                         33.69
P95 ITL (ms):                            103.37
P99 ITL (ms):                            151.34
Max ITL (ms):                            1600.00
==================================================
```

### 5.3 Speed Benchmark — MiMo-V2.5

**Test Environment:**

* Hardware: NVIDIA H200 GPU (8×)
* Model: `XiaomiMiMo/MiMo-V2.5` (FP8)
* Tensor Parallelism: 8 (DP-attention with `--attn-dp-size 2`)
* Recipe: Balanced (DP-attn + EAGLE MTP)
* sglang version: `0.0.0.dev1+g7d99af439` (`lmsysorg/sglang:dev-mimo-v2.5`)

#### 5.3.1 Latency-Sensitive Benchmark

* **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  14.72
Total input tokens:                      1997
Total input text tokens:                 1997
Total generated tokens:                  2798
Total generated tokens (retokenized):    2697
Request throughput (req/s):              0.68
Input token throughput (tok/s):          135.67
Output token throughput (tok/s):         190.09
Peak output token throughput (tok/s):    245.00
Peak concurrent requests:                3
Total token throughput (tok/s):          325.77
Concurrency:                             1.00
Accept length:                           3.08
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   1469.98
Median E2E Latency (ms):                 1652.84
P90 E2E Latency (ms):                    2210.80
P99 E2E Latency (ms):                    2823.86
---------------Time to First Token----------------
Mean TTFT (ms):                          143.89
Median TTFT (ms):                        99.25
P99 TTFT (ms):                           481.01
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          4.87
Median TPOT (ms):                        4.30
P99 TPOT (ms):                           6.64
---------------Inter-Token Latency----------------
Mean ITL (ms):                           4.76
Median ITL (ms):                         3.46
P95 ITL (ms):                            13.52
P99 ITL (ms):                            13.84
Max ITL (ms):                            74.37
==================================================
```

#### 5.3.2 Throughput-Sensitive Benchmark

* **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --random-input-len 1024 \
  --random-output-len 1024 \
  --num-prompts 1000 \
  --max-concurrency 100
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 100
Successful requests:                     1000
Benchmark duration (s):                  93.41
Total input tokens:                      302118
Total input text tokens:                 302118
Total generated tokens:                  195775
Total generated tokens (retokenized):    188139
Request throughput (req/s):              10.71
Input token throughput (tok/s):          3234.48
Output token throughput (tok/s):         2095.97
Peak output token throughput (tok/s):    3019.00
Peak concurrent requests:                121
Total token throughput (tok/s):          5330.45
Concurrency:                             91.04
Accept length:                           2.95
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   8503.45
Median E2E Latency (ms):                 7491.96
P90 E2E Latency (ms):                    13706.99
P99 E2E Latency (ms):                    20474.33
---------------Time to First Token----------------
Mean TTFT (ms):                          4399.20
Median TTFT (ms):                        4333.35
P99 TTFT (ms):                           8004.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          58.23
Median TPOT (ms):                        21.78
P99 TPOT (ms):                           747.79
---------------Inter-Token Latency----------------
Mean ITL (ms):                           20.06
Median ITL (ms):                         15.28
P95 ITL (ms):                            48.36
P99 ITL (ms):                            96.99
Max ITL (ms):                            969.61
==================================================
```

#### 5.3.3 Multimodal (Image) Benchmark

* **Model Deployment Command:** select MiMo-V2.5, H200, and EAGLE MTP in the [command panel above](#3-model-deployment).
* Benchmark Command:

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang-oai-chat \
  --host 127.0.0.1 \
  --port 30000 \
  --model XiaomiMiMo/MiMo-V2.5 \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 \
  --random-output-len 1024 \
  --num-prompts 10 \
  --max-concurrency 1
```

* **Test Results:**

```text Output theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    inf
Max request concurrency:                 1
Successful requests:                     10
Benchmark duration (s):                  25.73
Total input tokens:                      661
Total input text tokens:                 631
Total input vision tokens:               30
Total generated tokens:                  4220
Total generated tokens (retokenized):    0
Request throughput (req/s):              0.39
Input token throughput (tok/s):          25.69
Output token throughput (tok/s):         164.03
Peak output token throughput (tok/s):    1.00
Peak concurrent requests:                2
Total token throughput (tok/s):          189.73
Concurrency:                             1.00
Accept length:                           2.94
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   2570.74
Median E2E Latency (ms):                 2411.92
P90 E2E Latency (ms):                    3711.62
P99 E2E Latency (ms):                    4949.74
---------------Time to First Token----------------
Mean TTFT (ms):                          0.00
Median TTFT (ms):                        0.00
P99 TTFT (ms):                           0.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          7.31
Median TPOT (ms):                        6.17
P99 TPOT (ms):                           17.18
---------------Inter-Token Latency----------------
Mean ITL (ms):                           0.00
Median ITL (ms):                         0.00
P95 ITL (ms):                            0.00
P99 ITL (ms):                            0.00
Max ITL (ms):                            0.00
==================================================
```

### 5.4 Multi-Layer EAGLE Acceptance Profile — MiMo-V2.5-Pro

Pro's 3-layer MTP behaves very differently on natural text vs uniform-random token streams. The §5.2 benchmarks use `random 1024/1024`, which collapses accept-rate; this section measures the same server on GSM8K so the acceptance number is comparable to real workloads.

**Test Environment:**

* Hardware: NVIDIA B200 GPU (8×)
* Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8)
* Tensor Parallelism: 8 (single-node, `--moe-runner-backend flashinfer_trtllm`, `--attention-backend fa4`, `--mem-fraction-static 0.8`, `--swa-full-tokens-ratio 0.1`)
* Recipe: 3-layer EAGLE — `--enable-multi-layer-eagle --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` (top-1, max accept length 4)

**Benchmark Command:**

```shell Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --num-examples 200 \
  --num-threads 4
```

The command above uses sgl-eval. Historical scores from other harnesses are not directly comparable; rerun this command for a matching result.

The `accept_rate` and `accept_length` rows below are not part of `run_eval`'s own output — they were aggregated from the server-side `Decode batch ... accept rate: X accept len: Y` log lines emitted during the GSM8K run (307 batches total).

| Workload | accept\_rate | accept\_length (max = 4) |
| - | - | - |
| GSM8K (natural text) | **0.755** | **3.27** |
| `random 1024/1024` (reference) | 0.13–0.27 | \~1.x |

GSM8K Score: **0.97** (194 / 200), output throughput ≈ 635 tok/s end-to-end on this single-server run.

The accept-rate gap is intrinsic to MTP-style speculative decoding: the draft model is trained on natural-language token distributions and has no useful signal on uniform-random byte sequences. Workloads with structure (chat, code, reasoning traces) should expect the GSM8K-class number; the random-prompt baseline in §5.2 is a worst case for draft acceptance.

### 5.5 Long-Context Prefill & MTP Decode — MiMo-V2.5-Pro (Reference)

Reference numbers from the [day0 enablement PR](https://github.com/sgl-project/sglang/pull/23808), collected on a 2-node Hopper deployment with the **EP=16, DP=2, TP=16** recipe (`--moe-a2a-backend deepep`, `--attention-backend fa3`, `--enable-multi-layer-eagle`). The setup, parallelism, and benchmark methodology all differ from §5.2 (Blackwell TP=8 with `random 1024/1024`), so treat these as a separate operating point — long-context prefill scaling and the MTP decode speedup — rather than a comparison against §5.2.

**Test Environment:**

* Hardware: NVIDIA Hopper GPU (2 nodes × 8 GPUs, GPU SKU intentionally not disclosed)
* Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8)
* Parallelism: `--tp 16 --attn-dp-size 2 --ep 16 --moe-dense-tp-size 1`
* Recipe: Hopper EP16 (DeepEP + EAGLE multi-layer MTP)

#### 5.5.1 Long-Context Prefill Throughput

Test setting: `chunked_prefill_size=32K`, `random_output_len=1`, cache flushed before every run. For input lengths ≥ 512K the workload was split into two requests routed to distinct DP ranks and the per-node throughput was read from `bench_serving` output.

* **Benchmark Command:**

```shell Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --model XiaomiMiMo/MiMo-V2.5-Pro \
  --host 0.0.0.0 \
  --port 30000 \
  --dataset-name random \
  --random-input-len <INPUT_LEN> \
  --random-output-len 1 \
  --random-range-ratio 1.0 \
  --flush-cache \
  --seed 12345 \
  --num-prompts 10000
```

* **Test Results** — single-node prefill throughput, cache-miss:

| Input length | Output length | Single-node prefill throughput |
| - | - | - |
| 4K | 1 | 30.80K tok/s |
| 8K | 1 | 30.65K tok/s |
| 16K | 1 | 29.85K tok/s |
| 32K | 1 | 28.60K tok/s |
| 64K | 1 | 26.65K tok/s |
| 128K | 1 | 23.00K tok/s |
| 256K | 1 | 17.90K tok/s |
| 512K | 1 | 11.30K tok/s |
| 768K | 1 | 9.40K tok/s |
| 1M | 1 | 7.30K tok/s |

Prefill throughput stays within \~10% of peak from 4K up to 32K and degrades gracefully past 128K, confirming the hybrid SWA+GA attention works correctly at 1M context.

#### 5.5.2 Decode Throughput — MTP Speedup

Test setting: fixed **16K input / 1K output**, varying batch size per DP rank, with and without the 3-layer MTP module. `MTP accept length` is the average number of draft tokens accepted per step under EAGLE speculative decoding. **TPS** below is per-request output tokens/sec (i.e. single-user perceived speed); the rightmost column is aggregated single-node decode throughput (= TPS × batch size).

* **Test Results** — single-node decode throughput:

| BS per DP rank | MTP | MTP accept length | Per-request TPS | Single-node decode throughput |
| - | - | - | - | - |
| 64 | disabled | - | 29.3 | 1875 tok/s |
| 64 | 3-layer | 3 | 60.5 | 3873 tok/s |
| 64 | 3-layer | 4 | 79.7 | 5103 tok/s |
| 96 | disabled | - | 26.7 | 2564 tok/s |
| 96 | 3-layer | 3 | 50.4 | 4840 tok/s |
| 96 | 3-layer | 4 | 64.8 | 6225 tok/s |

**Summary — MTP on / off:**

| BS per DP rank | Without MTP | 3-layer MTP, accept=3 | 3-layer MTP, accept=4 |
| - | - | - | - |
| 64 | 1875 tok/s | 3873 tok/s (2.07×) | 5103 tok/s (2.72×) |
| 96 | 2564 tok/s | 4840 tok/s (1.89×) | 6225 tok/s (2.43×) |

The 3-layer MTP module delivers \~2× decode throughput at accept length 3 and \~2.5–2.7× at accept length 4 — the same order of magnitude as the "2–3× decode speedup" guidance in §3.2.


# MiMo-V2.6
Source: https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.6

Deploy MiMo-V2.6-Flash and Pro with SGLang on NVIDIA H200 and B300 GPUs, using MXFP4 MoE weights, BF16 routing, DFlash decoding, and a 1M-token context window.

## Deployment

The recipes below target **MiMo-V2.6-Flash on 4× H200 or B300** and **MiMo-V2.6-Pro on 8× H200 or B300**, each on a single node. Both use MXFP4 MoE weights, a BF16 MoE router, and DFlash speculative decoding.

> **Install SGLang with MiMo-V2.6 support**
>
> BF16 MoE routing and MXFP4 expert loading for MiMo V2.6 landed in [SGLang PR #40448](https://github.com/sgl-project/sglang/pull/40448) and are on `main`, so a recent nightly already carries them. Prepare an NVIDIA CUDA environment with FlashAttention 4, DeepGEMM, and DeepEP available. See the [installation guide](/docs/get-started/install) for platform prerequisites.
>
> **Python (source)**
>
> Install from source. The commit below is the revision these recipes were captured at; build from `main` instead if you want later fixes.
>
> ```bash Command theme={null}
> git clone https://github.com/sgl-project/sglang.git sglang-mimo-v2.6
> cd sglang-mimo-v2.6
> git checkout 983e643854f15cf9ef4370a49dfd74b6af54c3e3
> python3 -m pip install -e ./python
> ```
>
> Then run the **Python** command from the panel below. Checkpoint paths refer to directories on this host.
>
> **Docker (official image)**
>
> Pull the official nightly image, which already includes the MiMo-V2.6 support:
>
> ```bash Command theme={null}
> docker pull lmsysorg/sglang:dev
> ```

Select **Flash** or **Pro** to generate its command. Each variant has one base recipe; **Balanced** identifies that recipe and does not imply a measured throughput or latency optimum. The `/model/...` checkpoint and DFlash paths in the generated command are editable defaults, not download locations — set them to your own paths under **⚙ Env** before launching. Use the panel's cURL example to send a request to port **30000** after the server is ready.

## Playground

Experiment with reasoning and tool-call parsers, or disable speculative decoding for comparison. These overrides inherit the selected Flash or Pro recipe; they require separate validation. Parallelism and attention backends remain at the PR settings.

## 1. Model introduction

**MiMo-V2.6-Flash** is an open-source foundation model developed by Xiaomi. Its Mixture-of-Experts architecture has **309B total parameters and 15B activated per token**, with hybrid attention for computational efficiency. It supports a **1M-token context window** and **native multimodal capabilities**, and is designed for coding, visual understanding, general-purpose assistance, and research in agentic workflows, including complex tasks that require many steps.

**MiMo-V2.6-Pro** is Xiaomi's flagship foundation model, with **1.02T total parameters and 42B activated per token**, designed for demanding workloads. It also supports a **1M-token context window** and **native multimodal capabilities**, with an emphasis on coding, visual understanding, general-purpose assistance, research, and long-horizon agentic tasks.

<table>
  <thead>
    <tr>
      <th>Variant</th>
      <th>Total parameters</th>
      <th>Context window</th>
      <th>Recipe (H200 / B300)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>MiMo-V2.6-Flash</td>
      <td>309B (15B active)</td>
      <td>1M tokens</td>
      <td>4 GPUs, TP=EP=4</td>
    </tr>

    <tr>
      <td>MiMo-V2.6-Pro</td>
      <td>1.02T (42B active)</td>
      <td>1M tokens</td>
      <td>8 GPUs, TP=EP=8</td>
    </tr>
  </tbody>
</table>

**License:** MIT, continuing the MiMo-V2.5 licensing as confirmed by the model team.

**Architecture references:** The [MiMo-V2.6-Flash-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) and [MiMo-V2.6-Pro-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL) model cards document this generation directly — the hybrid sliding-window/global attention backbone, the omnimodal encoders, and the published evaluation scores — and each repository also ships the MiMo-V2.6 technical report. Both checkpoints declare `store_dtype: mxfp4` alongside `moe_router_dtype: bfloat16` in `config.json`, which is the expert-only MXFP4 quantization and BF16 routing these recipes depend on, and both carry the paired DFlash drafter in a `dflash/` subdirectory.

## 2. Configuration tips

### Parallelism and precision

* **Flash:** `--tp 4 --ep 4`; **Pro:** `--tp 8 --ep 8`. Both keep `--dp 1 --pp-size 1`. TP and EP use the same GPU ranks; do not multiply them to determine the GPU count.
* **MoE compute and communication:** Keep `--moe-runner-backend deep_gemm` on B300 and `--moe-runner-backend marlin` on H200, with `--moe-a2a-backend deepep --deepep-mode auto`, together with `--moe-dense-tp-size 1` and `--enable-dp-lm-head`, as in the PR.
* **Precision:** MXFP4 describes the MoE expert weights, not the entire model. BF16 router weights are selected by the checkpoint's `moe_router_dtype`; the support commit computes router output logits in FP32. The PR relies on checkpoint quantization metadata and does not add a `--quantization` override.
* **Attention:** Keep `--attention-backend fa4` and `--mm-attention-backend fa4` on both H200 and B300. The recipe also enables `--mm-enable-dp-encoder` for multimodal encoding.

### Context length and memory

Both commands set `--context-length 1048576`. This is the configured per-request context limit; it does not establish that 64 simultaneous 1M-token requests fit in memory.

Flash uses `--mem-fraction-static 0.6 --swa-full-tokens-ratio 0.03`; Pro uses `--mem-fraction-static 0.7 --swa-full-tokens-ratio 0.08`. The memory fraction budgets weights and KV cache, while the SWA ratio controls sliding-window versus full-attention KV allocation. Preserve these model-specific values for the base recipe.

The shared prefill settings are `--chunked-prefill-size 49152 --max-prefill-tokens 65536`. Both commands cap running requests and decode CUDA-graph batch size at **64**, disable prefill CUDA graphs, and use page size **1**. `--enable-cache-report` and `--log-level-http warning` retain the PR's reporting settings.

### DFlash and caching

The base recipe uses `--speculative-algorithm DFLASH --speculative-num-draft-tokens 8`. Supply the DFlash checkpoint paired with the selected target model; the draft path is independently editable and does not have to be a subdirectory of the target checkpoint.

At the pinned commit, DFlash on NVIDIA GPUs requires PP=1 and rejects DP-attention. Do not set `--attn-dp-size` above 1 or copy EAGLE-specific settings from the V2.5 cookbook.

## 3. Reasoning and tool calling

Both launch commands enable `--reasoning-parser mimo --tool-call-parser mimo`. The MiMo reasoning parser uses the request's `chat_template_kwargs.enable_thinking` setting: set it to `true` for reasoning or `false` for a direct answer. The command panel's cURL example enables reasoning explicitly.

For OpenAI-compatible clients, read reasoning from `choices[0].message.reasoning_content`, the final answer from `choices[0].message.content`, and structured tool requests from `choices[0].message.tool_calls`. Preserve both reasoning and answer fields when inspecting responses. Use the checkpoint's generation defaults unless your application needs explicit sampling settings.

Runtime throughput, latency, accuracy, and example responses for V2.6 will be added when measurements from these checkpoints are available.


# Overview
Source: https://docs.sglang.io/cookbook/autoregressive/intro

Practical guides for deploying and using large language models and vision language models with SGLang.

> **[Reflection](/cookbook/autoregressive/Reflection/Beam-1_0)**

> **[IQuestLab](/cookbook/autoregressive/IQuestLab/IQuest-Q1)**

> **[IFM](/cookbook/autoregressive/IFM/K2-Horizon)**

> **[Kimi (Moonshot AI)](/cookbook/autoregressive/Moonshotai/Kimi-K3)**

> **[Thinking Machines](/cookbook/autoregressive/ThinkingMachines/Inkling)**

> **[GigaChat](/cookbook/autoregressive/GigaChat/GigaChat3.5)**

> **[GLM](/cookbook/autoregressive/GLM/GLM-5.3-Flash)**

> **[Qwen](/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next)**

> **[DeepSeek](/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1)**

> **[DeepReinforce](/cookbook/autoregressive/DeepReinforce/Ornith-1.5)**

> **[Meta](/cookbook/autoregressive/Meta/MuseSparkOpen)**

> **[Meituan](/cookbook/autoregressive/Meituan/LongCat-2.0)**

> **[Google](/cookbook/autoregressive/Google/Gemma4)**

> **[IBM](/cookbook/autoregressive/IBM/Granite-4.2)**

> **[LiquidAI](/cookbook/autoregressive/LiquidAI/LFM2.5)**

> **[OpenAI](/cookbook/autoregressive/OpenAI/GPT-OSS)**

> **[MiniMax](/cookbook/autoregressive/MiniMax/MiniMax-M3.1)**

> **[NVIDIA](/cookbook/autoregressive/NVIDIA/Nemotron3.5-Super)**

> **[Baidu](/cookbook/autoregressive/Baidu/PaddleOCR-VL)**

> **[Ernie](/cookbook/autoregressive/Ernie/Ernie4.5)**

> **[StepFun](/cookbook/autoregressive/StepFun/Step-5-Preview)**

> **[InclusionAI](/cookbook/autoregressive/InclusionAI/Ling-3.1-flash)**

> **[InternLM](/cookbook/autoregressive/InternLM/Intern-S2-Mobius)**

> **[InternVL](/cookbook/autoregressive/InternVL/InternVL3.5)**

> **[OpenBMB](/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_7)**

> **[Jina AI](/cookbook/autoregressive/Jina/Jina-reranker-m0)**

> **[Mistral](/cookbook/autoregressive/Mistral/Ministral-3)**

> **[Xiaomi](/cookbook/autoregressive/Xiaomi/MiMo-V2.6)**

> **[FlashLabs](/cookbook/autoregressive/FlashLabs/Chroma1.0)**

> **[Tencent](/cookbook/autoregressive/Tencent/Hy4-Preview)**

> **[Recursive](/cookbook/autoregressive/Recursive/Flint)**

> **[Poolside](/cookbook/autoregressive/Poolside/Laguna-S-2.1)**

> **[Perplexity](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B)**

> **[Cloudflare](/cookbook/autoregressive/Cloudflare/clef)**

> **[Kev](/cookbook/autoregressive/Kev/kev)**


# Autoregressive Model Benchmark Documentation
Source: https://docs.sglang.io/cookbook/base/benchmarks/autoregressive_model_benchmark



`sglang.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of Large Language Models (LLMs) and Vision Language Models(VLMs). It supports various backends (`SGLang`, `vLLM`, etc.) and offers flexible configurations for request rates, dataset types, and profiling.

## 1. Quick Start

### Basic Usage (Random Data)

Run a benchmark using randomly generated prompts with a local SGLang server.

```bash Command theme={null}
python -m sglang.bench_serving --backend sglang --port 30000 --dataset-name random --num-prompts 100
```

### Real-World Data (ShareGPT)

Run a benchmark using the ShareGPT dataset with a specific request rate.

```shell Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --dataset-name sharegpt \
  --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
  --num-prompts 1000 \
  --request-rate 10
```

## 2. Parameter Reference

### 2.1 Backend & Server Configuration

These parameters define the target server and the inference engine being used.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--backend`</td>
      <td>**Required.** Specifies the backend engine. Options: `sglang`, `sglang-native`, `sglang-oai`, `sglang-oai-chat`, `vllm`, `vllm-chat`, `lmdeploy`, `lmdeploy-chat`, `trt`, `gserver`, `truss`.</td>
    </tr>

    <tr>
      <td>`--base-url`</td>
      <td>The API base URL (if not using specific host/port flags).</td>
    </tr>

    <tr>
      <td>`--host`</td>
      <td>Server hostname. Default: `0.0.0.0`.</td>
    </tr>

    <tr>
      <td>`--port`</td>
      <td>Server port. If not set, it defaults to the specific backend's standard port.</td>
    </tr>

    <tr>
      <td>`--model`</td>
      <td>Model name or path. If unset, it queries `/v1/models` for configuration.</td>
    </tr>

    <tr>
      <td>`--served-model-name`</td>
      <td>The model name used in the API request body. Defaults to the value of `--model`.</td>
    </tr>

    <tr>
      <td>`--tokenizer`</td>
      <td>Path or name of the tokenizer. Defaults to the model configuration.</td>
    </tr>
  </tbody>
</table>

### 2.2 Dataset Configuration

Controls the source of the prompts used for benchmarking.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dataset-name`</td>
      <td>The type of dataset. Options: `sharegpt`, `custom`, `random`, `random-ids`, `generated-shared-prefix`, `mmmu`, `image`, `mooncake`.</td>
    </tr>

    <tr>
      <td>`--dataset-path`</td>
      <td>File path to the dataset (e.g., local JSON file for ShareGPT).</td>
    </tr>

    <tr>
      <td>`--num-prompts`</td>
      <td>Total number of prompts to process. Default: `1000`.</td>
    </tr>

    <tr>
      <td>`--seed`</td>
      <td>Random seed for reproducibility.</td>
    </tr>

    <tr>
      <td>`--tokenize-prompt`</td>
      <td>Uses integer IDs instead of strings for inputs. Useful for precise length control.</td>
    </tr>
  </tbody>
</table>

### 2.3 Input/Output Length Control

Parameters to control the shape of requests (context length and generation length).

#### For Random/Image Datasets:

* `--random-input-len`: Number of input tokens per request.
* `--random-output-len`: Number of output tokens per request.
* `--random-range-ratio`: Range ratio for sampling input/output lengths.

#### For ShareGPT Dataset:

* `--sharegpt-output-len`: Overrides the output length defined in the dataset for each request.
* `--sharegpt-context-len`: Max context length. Requests exceeding this are dropped.

#### General Request Modifiers:

* `--extra-request-body`: Appends a JSON object to the request payload (e.g., \{"key": "value"}). Useful for passing sampling parameters.
* `--prompt-suffix`: A string suffix appended to all user prompts.
* `--disable-ignore-eos`: If set, the model will stop generation upon hitting the EOS token (benchmarks usually ignore EOS to force max generation length).
* `--apply-chat-template`: Applies the model's chat template to the input.

### 2.4 Traffic & Concurrency

Controls how fast requests are sent to the server.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--request-rate`</td>
      <td>Requests per second (RPS). If `inf` (default), all requests are sent immediately (burst). Otherwise, arrival times follow a Poisson process.</td>
    </tr>

    <tr>
      <td>`--max-concurrency`</td>
      <td>The maximum number of active requests allowed at once. Even if `request-rate` is high, the client will hold back requests if this limit is reached.</td>
    </tr>

    <tr>
      <td>`--warmup-requests`</td>
      <td>Number of requests to run before the actual measurement begins to warm up the server.</td>
    </tr>

    <tr>
      <td>`--flush-cache`</td>
      <td>Flushes the server cache before starting the benchmark.</td>
    </tr>
  </tbody>
</table>

### 2.5 Output & Logging

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--output-file`</td>
      <td>Path to save the results in JSONL format.</td>
    </tr>

    <tr>
      <td>`--output-details`</td>
      <td>Includes detailed metrics in the output.</td>
    </tr>

    <tr>
      <td>`--print-requests`</td>
      <td>Prints requests to stdout as they are sent (useful for debugging).</td>
    </tr>

    <tr>
      <td>`--disable-tqdm`</td>
      <td>Hides the progress bar.</td>
    </tr>

    <tr>
      <td>`--disable-stream`</td>
      <td>Disables streaming mode (waits for full response).</td>
    </tr>

    <tr>
      <td>`--return-logprob`</td>
      <td>Requests logprobs from the server.</td>
    </tr>

    <tr>
      <td>`--tag`</td>
      <td>An arbitrary string tag added to the output file for identification.</td>
    </tr>
  </tbody>
</table>

### 2.6 Advanced

#### 2.6.1 Image / Multi-modal

Only applicable when --dataset-name is set to image.

* `--image-count`: Number of images per request.
* `--image-resolution`: Resolution (e.g., 1080p, 4k, or custom 1080x1920).
* `--image-format`: jpeg or png.
* `--image-content`: random (noise) or blank.

#### 2.6.2 LoRA Benchmarking

Used to simulate multi-LoRA serving scenarios.

* `--lora-name`: A list of LoRA adapter names (e.g., `--lora-name` adapter1 adapter2).
* `--lora-request-distribution`: How requests are assigned to adapters:
  * `uniform`: Equal probability.
  * `distinct`: New adapter for every request.
  * `skewed`: Follows a Zipf distribution (simulating hot/cold adapters).
* `--lora-zipf-alpha`: The alpha parameter for the Zipf distribution (if `skewed` is used).

#### 2.6.3 Profiling

Tools for deep performance analysis.

* `--profile`: Enables Torch Profiler (Requires `SGLANG_TORCH_PROFILER_DIR` env var on server).
* `--plot-throughput`: Generates throughput/concurrency plots (requires `termplotlib` and `gnuplot`).
* `--profile-activities`: Activities to profile (CPU, GPU, CUDA\_PROFILER).
* `--profile-num-steps`: Number of steps to profile.
* `--profile-by-stage` / `--profile-stages`: Profile specific processing stages.

#### 2.6.4 PD Disaggregation

For benchmarking Prefill-Decode (PD) separated architectures.

* `--pd-separated`: Enable PD disaggregation benchmarking.
* `--profile-prefill-url`: URL(s) of prefill workers for profiling.
* `--profile-decode-url`: URL(s) of decode workers for profiling.

Note: In PD mode, `prefill` and `decode` must be profiled separately.

### 2.7 Specialized Datasets

#### 2.7.1 Generated Shared Prefix (GSP):

Designed to test system prompt caching/prefix sharing performance.

* `--gsp-num-groups`: Number of unique system prompts.
* `--gsp-prompts-per-group`: How many user questions share the same system prompt.
* `--gsp-system-prompt-len`: Length of the shared prefix.
* `--gsp-fast-prepare`: Skips some statistics calculation for faster startup.

#### 2.7.2 Mooncake

Designed for trace replay.

* `--mooncake-slowdown-factor`: Slows down the trace replay (e.g., 2.0 = 2x slower).
* `--mooncake-num-rounds`: Number of conversation rounds (supports multi-turn).
* `--use-trace-timestamps`: Schedules requests based on timestamps found in the trace file.

## 3. Metrics

After running the benchmark, the tool generally reports:

* `E2E` (End-to-End Latency): The total time from sending the request to receiving the final token.
* `TTFT` (Time To First Token): The time between sending the request and seeing the first word appear. This represents the Prefill time (processing the image and text prompt).
* `TPOT` (Time per Output Token): The average time it takes to generate one token (excluding the first one). This is calculated per request.
* `ITL` (Inter-Token Latency): The time gap between two distinct streaming packets. While TPOT is an average, ITL measures the "jitter" or smoothness of the stream.


# Diffusion Models Benchmark Documentation
Source: https://docs.sglang.io/cookbook/base/benchmarks/diffusion_model_benchmark



`sglang.multimodal_gen.benchmarks.bench_serving` is a command-line tool designed to benchmark the online serving throughput and latency of diffusion models. It selects the image or video API from the requested task and offers flexible configurations for request rates, dataset types, and profiling.

## 1. Quick Start

### 1.1 Benchmarking in Low Concurrency

Run a benchmark on a local server (port 30000) generating 1 videos/images from the `vbench` dataset.

```bash Command theme={null}
# For text to video: such as Wan2.2-T2V-A14B-Diffusers
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1

# For image to video: such as Wan2.2-I2V-A14B-Diffusers
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1

# For image-text to video: such as Wan2.2-TI2V-5B-Diffusers
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-video --num-prompts 1 --max-concurrency 1

# For text to image: such as Qwen-Image
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1

# For image-text to image: such as Qwen-Image-Edit
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1
```

### 1.2 Benchmarking in High Concurrency

Run a benchmark on a local server (port 30000) generating 20 videos/images from the `vbench` dataset.

```bash Command theme={null}
# For text to video: such as Wan2.2-T2V-A14B-Diffusers
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20

# For image to video: such as Wan2.2-I2V-A14B-Diffusers
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20

# For image-text to video: such as Wan2.2-TI2V-5B-Diffusers
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-video --num-prompts 20 --max-concurrency 20

# For text to image: such as Qwen-Image
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20

# For image-text to image: such as Qwen-Image-Edit
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20
```

## 2. Parameter Reference

### 2.1 Connection Settings

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--base-url`</td>
      <td>`None`</td>
      <td>Base URL of the server (e.g., `http://localhost:30000`). If specified, this overrides `--host` and `--port`.</td>
    </tr>

    <tr>
      <td>`--host`</td>
      <td>`None`</td>
      <td>The server host (e.g., `127.0.0.1`).</td>
    </tr>

    <tr>
      <td>`--port`</td>
      <td>`None`</td>
      <td>The server port.</td>
    </tr>

    <tr>
      <td>`--model`</td>
      <td>`None`</td>
      <td>Model name or path.</td>
    </tr>
  </tbody>
</table>

### 2.2 Workload & Task Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Choices</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--task`</td>
      <td>`text-to-video`, `image-to-video`, `text-to-image`, `image-to-image`, `video-to-video`</td>
      <td>Defines the generation task when it cannot be inferred from the model metadata.</td>
    </tr>

    <tr>
      <td>`--dataset`</td>
      <td>`vbench`, `random`</td>
      <td>The source of prompts/inputs.</td>
    </tr>

    <tr>
      <td>`--dataset-path`</td>
      <td>`None`</td>
      <td>(Optional) Path to a local dataset file if not using built-in presets.</td>
    </tr>

    <tr>
      <td>`--num-prompts`</td>
      <td>`None`</td>
      <td>The total number of prompts/requests to execute during the benchmark.</td>
    </tr>
  </tbody>
</table>

### 2.3 Generation Parameters

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--width`</td>
      <td>The target width for the generated image or video.</td>
    </tr>

    <tr>
      <td>`--height`</td>
      <td>The target height for the generated image or video.</td>
    </tr>

    <tr>
      <td>`--num-frames`</td>
      <td>Number of frames to generate (Specific to Video backends).</td>
    </tr>

    <tr>
      <td>`--fps`</td>
      <td>Frames Per Second configuration (Specific to Video backends).</td>
    </tr>
  </tbody>
</table>

### 2.4 Concurrency & Load Control

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--request-rate`</td>
      <td>The number of requests initiated per second. If set to `inf`, all requests are sent immediately (burst). If set to a number, request arrival times follow a Poisson process.</td>
    </tr>

    <tr>
      <td>`--max-concurrency`</td>
      <td>The maximum number of requests allowed to execute simultaneously. This simulates a semaphore or upstream limit. Even if `request-rate` is high, the actual processing rate is capped by this value.</td>
    </tr>
  </tbody>
</table>

### 2.5 Logging & Output

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--output-file`</td>
      <td>Path to save the benchmark metrics (JSON format).</td>
    </tr>

    <tr>
      <td>`--disable-tqdm`</td>
      <td>If set, disables the progress bar in the console.</td>
    </tr>
  </tbody>
</table>

## 3. Metrics

* `Request Throughput` (req/s), Output Throughput (tok/s)
* `Latency Mean` (ms): Time to Per Step
* `Peak Memory Max` (ms): Max Memory Usage during running


# Server Arguments
Source: https://docs.sglang.io/cookbook/base/reference/server_arguments



This guide explains the parallelism configuration fields used in SGLang model configurations and how they map to SGLang server command-line arguments.

## Quick Reference

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Config Field</th>
      <th>SGLang CLI Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`tp`</td>
      <td>`--tp-size`, `--tensor-parallel-size`</td>
      <td>Tensor Parallelism - splits model across GPUs</td>
    </tr>

    <tr>
      <td>`dp`</td>
      <td>`--dp-size`, `--data-parallel-size`</td>
      <td>Data Parallelism - runs multiple model replicas</td>
    </tr>

    <tr>
      <td>`ep`</td>
      <td>`--ep-size`, `--expert-parallel-size`, `--ep`</td>
      <td>Expert Parallelism - distributes MoE experts</td>
    </tr>

    <tr>
      <td>`attn_dp_size`</td>
      <td>`--attn-dp-size`, `--attention-data-parallel-size`</td>
      <td>Attention Data Parallelism - data-parallel attention groups inside the TP group, TP for FFN (hybrid)</td>
    </tr>

    <tr>
      <td>`enable_dp_attention`</td>
      <td>`--enable-dp-attention`</td>
      <td>**Deprecated.** Use `--attn-dp-size <dp-size>` instead of `--dp-size <dp-size> --enable-dp-attention`.</td>
    </tr>
  </tbody>
</table>


# Anima
Source: https://docs.sglang.io/cookbook/diffusion/CircleStone/Anima

Deploy Anima Base v1.0 with SGLang Diffusion for anime and illustration generation, using native components and single- or multi-GPU execution.

## 1. Quick start

On Linux with NVIDIA CUDA, use a nightly Docker image containing this integration, or install from source, following the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Run the commands below inside that environment.

Use the official `circlestone-labs/Anima-Base-v1.0-Diffusers` checkpoint. SGLang reads
its `modular_model_index.json` directly; no checkpoint conversion or newer Diffusers
runtime is required.

Start with **Resident + Compiled** for repeated generation on RTX 5090, DGX Spark,
or H200. Select **Eager** for shorter startup or frequent shape changes. Spark's
128 GB is unified memory shared with the CPU, not dedicated VRAM. See the
[measurements and tradeoffs](#5-measured-tuning) below.

## 2. Model capabilities

[Anima](https://huggingface.co/circlestone-labs/Anima) is CircleStone Labs and Comfy
Org's text-to-image model for anime and illustration. It combines a Cosmos Predict2
transformer with Qwen3 text encoding, a learned T5-token conditioner, and a Qwen-Image
VAE. Prompts can combine tags with natural-language descriptions.

This integration targets **Base v1.0 text-to-image**. It does not claim support for
the separate Aesthetic or Turbo checkpoints, single-file ComfyUI weights, or image
editing. The checkpoint uses the CircleStone Labs Non-Commercial License; review
the [official license](https://huggingface.co/circlestone-labs/Anima/blob/main/LICENSE.md)
before deployment.

## 3. Sampling

Defaults are 1024 x 1024, 30 steps, guidance scale 4, and an empty negative prompt.
Width and height should be divisible by 16. `max_sequence_length` defaults to 512
and accepts 1 through 4096. The text conditioner pads short sequences to 512 tokens;
these padding positions remain part of the transformer's cross-attention, matching
the official implementation.

For offline generation:

```bash Generate theme={null}
sglang generate \
  --model-path circlestone-labs/Anima-Base-v1.0-Diffusers \
  --prompt "masterpiece, best quality, safe, watercolor landscape, a quiet seaside village at sunset" \
  --seed 42 \
  --save-output
```

Use the same seed, generator device, dimensions, scheduler settings, and precision
when comparing runtimes. Latents and scheduler updates remain FP32; the transformer,
text components, and VAE default to BF16. Different attention kernels can produce
small floating-point differences that accumulate during denoising.

## 4. Runtime features

The pipeline reuses SGLang's native Qwen3 encoder, Qwen-Image VAE, component loaders,
denoising loop, and residency management. Anima's transformer and text conditioner
are native modules, not wrappers around Diffusers models.

For multi-GPU execution, choose TP to shard transformer weights or Ulysses/Ring to
shard image tokens. The transformer has 16 heads, so `tp_size * ulysses_degree`
must divide 16. CFG parallelism additionally splits conditional and unconditional
denoising and requires guidance greater than 1. The total GPU count must match the
selected parallel topology.

CPU and layerwise offload trade memory for transfers. The additional
`text_conditioner` component accepts the same residency controls as other native
components. Cache-DiT and quantized attention are approximate optimizations; assess
image quality for your prompts before enabling them. See
[performance optimization](/docs/sglang-diffusion/performance-optimization) for the
shared controls. H200 functional checks cover TP, Ulysses, Ring, CFG parallelism,
encoder folding, parallel tiled VAE decode, layerwise offload, FlashAttention,
Torch SDPA, SageAttention, Cache-DiT, and breakable CUDA graphs. These checks are
not a quality guarantee for approximate optimizations. Combined TP and SP,
multi-node execution, non-NVIDIA devices, and third-party LoRAs remain unverified.

Breakable CUDA graphs reuse the conditioner's actual sequence length, without
additional text-bucket padding. Requests with uncaptured shapes fall back to eager
execution. Standard short prompts use the same 512-token conditioning shape.

## 5. Measured tuning

For repeated requests, compilation was faster than eager execution on all three
tested platforms. The picker keeps eager available because initial compilation
and recompilation for new shapes can take minutes and use additional CPU memory.
No CPU offload, quantized attention, or Cache-DiT is enabled in the recommended
recipes.

The following are warm, sequential HTTP request medians: 1024 x 1024, 30 steps,
CFG 4, one image, CPU generator, and PNG/base64 output. Startup is excluded.

| GPU | Recommended recipe | Original eager | Recommended | Peak allocated |
| - | - | - | - | - |
| RTX 5090, 32 GB | cuDNN SDPA + compile, tiled VAE | 6.99 s | 6.10 s | 5.7 GiB |
| DGX Spark, 128 GB unified | Torch SDPA + compile, tiled VAE | 26.03 s | 17.41 s | 5.6 GiB |
| H200, 141 GB | FlashAttention + compile, untiled VAE | 3.22 s | 2.22 s | 10.2 GiB |

Measurements used five timed requests after warmup, PyTorch
2.13.0+cu130, and the native Anima implementation at `214c9a48bb1` (Spark used
`7f3f903f039`, with identical runtime code). Results are workload-specific, not
throughput-at-saturation measurements. The original eager baseline uses Torch
SDPA on 5090/Spark and FlashAttention on H200, with VAE tiling enabled. Peak
allocated memory excludes the CUDA context, reserved pool, and compilation CPU
memory; it is not total device usage.

On **two NVLink-connected H200s**, Auto selects CFG parallelism instead of TP:
the compiled, untiled recipe measured **1.28 s**. Untiled decoding trades about
4-5 GiB more allocated memory for lower latency and is verified at 1024 x 1024,
one output. The tiled recipes additionally passed repeated 512 x 512 and
1536 x 1536 requests, plus two outputs at 1024 x 1024. Use **Tiled** for that
verified scope. At 512 x 512, two-GPU CFG eager was faster than compiled;
more GPUs or compilation are not universally better.

The recommended recipes change floating-point kernels and, on H200, VAE tiling.
They are **not pixel-identical to eager**: across three fixed prompt/seed pairs,
the single-GPU recipes above measured PSNR 21.95-35.44 dB and SSIM 0.851-0.969
against their same-device eager outputs.
These measure output differences, not perceptual quality guarantees. Use eager
with the same backend and tiling settings when reproducing an eager reference.

Breakable CUDA graphs were near parity at 1024 x 1024 while consuming about
1.2 GiB more allocated GPU memory for a single captured shape. They are not the
default. Cache-DiT can accelerate further, but changes the denoising computation;
validate it separately against your quality requirements.


# Cosmos3
Source: https://docs.sglang.io/cookbook/diffusion/Cosmos/Cosmos3

Serve NVIDIA Cosmos3 image, video, sound, and action generation with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

Choose a [generation or policy request](#4-openai-compatible-requests) matching the checkpoint; the Request tab queries model metadata.

> **Installation and prerequisites**
>
> Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Docker is recommended for Linux GPU deployments; run the commands below inside the container or your installed Python environment.
>
> Cosmos3 guardrails are enabled by default when the package is available:
>
> ```bash Command theme={null}
> pip install "cosmos-guardrail==0.3.1"
> ```
>
> `cosmos-guardrail` downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set `SGLANG_DISABLE_COSMOS3_GUARDRAILS=1` before starting the server.
>
> There may be problems loading the Cosmos-1.0-Guardrail weights on Ascend NPU. If the *\_pickle.UnpicklingError* error occurs during startup, you should change `weight_only=True` to `weights_only=False` parameter in *cosmos\_guardrail/cosmos\_utils.py*:
>
> ```
> #!/usr/bin/env bash
> COSMOS_GUARDRAIL_DIR="$(dirname "$(python -c 'import cosmos_guardrail; print(cosmos_guardrail.__file__)')")"
> sed -i 's/weights_only=True/weights_only=False/g' "$COSMOS_GUARDRAIL_DIR/cosmos_utils.py"
> ```

## 2. Model capabilities

[NVIDIA Cosmos3](https://huggingface.co/collections/nvidia/cosmos3) is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler.

Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository.

| Model | Status | Notes |
| - | - | - |
| `nvidia/Cosmos3-Nano` | Supported | T2I, T2V, I2V, V2V, joint sound, and action |
| `nvidia/Cosmos3-Super` | Supported | T2I, T2V, I2V, and V2V; use multi-GPU for the 64B checkpoint |
| `nvidia/Cosmos3-Super-Text2Image` | Supported | T2I-specialized checkpoint |
| `nvidia/Cosmos3-Super-Image2Video` | Supported | I2V-specialized checkpoint |
| `nvidia/Cosmos3-Nano-Policy-DROID` | Supported | DROID policy action generation |
| `nvidia/Cosmos3-Edge` | Supported | 4B dense model for T2I, T2V, I2V, V2V, and action generation |
| `nvidia/Cosmos3-Edge-Policy-DROID` | Supported | 4B DROID policy action generation |
| `nvidia/Cosmos3-Super-Text2Image-4Step` | Supported | 64B T2I checkpoint distilled to a fixed 4-step schedule |
| `nvidia/Cosmos3-Super-Image2Video-4Step` | Supported | 64B I2V checkpoint distilled to a fixed 4-step schedule |

Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native `FlowUniPCMultistepScheduler`; distilled checkpoints use the fixed sigma schedule stored in the checkpoint.

The default `flow_shift` is `3.0` for T2I, `10.0` for non-Edge video and all action modes, and `3.0` for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level `flow_shift`.

## 3. Serve Cosmos3

For Intel Xeon CPU deployment, use `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` with the selected Cosmos3 checkpoint.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path nvidia/Cosmos3-Nano --port 30010
```

Serve `Cosmos3-Nano` directly from the Hugging Face model ID:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --num-gpus 1
```

With `--performance-mode auto`, Cosmos3 Nano keeps its DiT and VAE resident
when every selected GPU has at least 90 GiB available at startup. Other
Cosmos3 checkpoints use a 120 GiB threshold. Below the applicable threshold,
auto mode retains the conservative DiT component-offload policy. Cosmos3 runs
one DiT per pipeline, so component offload above the threshold only pays to
copy the weights out to host memory and back on every request. Serve
`Cosmos3-Super` across multiple GPUs as shown below so each rank holds a shard
of the weights.

For `Cosmos3-Super`, split the model across multiple GPUs:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Super \
  --num-gpus 4
```

The server also accepts the specialized `nvidia/Cosmos3-Super-Text2Image` and `nvidia/Cosmos3-Super-Image2Video` checkpoint IDs.

### Edge checkpoints

`Cosmos3-Edge` is a 4B dense model and can be served on one GPU:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Edge \
  --num-gpus 1
```

Edge is trained for 256p and 480p generation. Its default video configuration is `832x480` with `guidance_scale=5.0`; its default image configuration is `640x640` with `guidance_scale=7.0`. Supported sizes are `832x480`, `480x832`, `640x480`, `480x640`, `480x480`, `640x640`, `448x256`, `256x448`, and `256x256`.

Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with `nvidia/Cosmos3-Edge-Policy-DROID`.

### Distilled checkpoints

The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Super-Text2Image-4Step \
  --num-gpus 4
```

For distilled I2V, replace the model path with `nvidia/Cosmos3-Super-Image2Video-4Step`. SGLang detects both checkpoints from `scheduler/scheduler_config.json`, uses the checkpoint's fixed four-step sigma schedule, and forces `guidance_scale=1.0`. Do not tune `num_inference_steps` or `flow_shift` for these checkpoints.

### Simulation and multiview checkpoints

These checkpoints use native pipelines with checkpoint-specific conditioning:

| Checkpoint | Inputs | Rollout |
| - | - | - |
| `nvidia/Cosmos3-Nano-Sim-Bimanual` | Image, prompt, camera or robot actions | Causal chunks with committed KV history |
| `nvidia/Cosmos3-Nano-Sim-Transfer` | Control video and prompt | Causal control-video transfer |
| `nvidia/Cosmos3-Nano-Transfer-Auto` | Per-camera WSM clips and captions; optional LiDAR | Bidirectional multiview generation |
| `nvidia/Cosmos3-Nano-Sim-Auto` | Per-camera WSM clips and captions; optional LiDAR | Causal multiview generation |

For example, start the Bimanual server:

```bash theme={null}
sglang serve --model-path nvidia/Cosmos3-Nano-Sim-Bimanual
```

Submit image-conditioned video requests through `/v1/videos`. Supply `action`
rows according to the checkpoint's embodiment contract, selected by `domain_name`
or `domain_id`. Native contracts accept raw rows with `action_normalization="contract"`;
unified contracts require normalized 59-dimensional rows and default to
`action_normalization="none"`. Without actions, Bimanual uses null-action conditioning.
These pipelines predict video from actions, not actions from video; use the policy
checkpoints above for action prediction.

For realtime Bimanual sessions, add
`--pipeline-class-name CosmosDreamsRealtimePipeline` and use
[`/v1/realtime_video`](/docs/sglang-diffusion/realtime_models).
Each tick consumes one action block and preserves causal KV history.

Transfer-Auto and Sim-Auto take a `multiview` object with `views`, each containing
`camera_key`, `control_path`, optional `vision_path`, and optional `prompt`. Joint checkpoints
also accept `lidar`; set `lidar.decode=true` to return decoded range maps and previews.
Outputs pack all frames of each camera consecutively, in request camera order.

LiDAR uses built-in compiled FlexAttention by default. The optional NATTEN path
requires a wheel matching your PyTorch/CUDA versions and
`SGLANG_DIFFUSION_COSMOS3_LIDAR_USE_NATTEN=1`; it is not installed by default.

Bimanual and Sim-Transfer support TP, but not SP or CFG parallelism. Transfer-Auto
supports Ulysses without TP or Ring; Sim-Auto supports TP and Ulysses subject to
the checkpoint's query/KV head divisibility, but not Ring or CFG parallelism.
Causal pipelines use the exported fixed-step schedule and require guidance `1.0`.

Use `--component-residency` for memory-constrained deployments. Joint checkpoints
register `lidar_encoder` and `lidar_decoder` separately from the camera `vae`.
Bimanual overlaps causal VAE decode only when the VAE is resident; offloaded VAEs
use the final serial decode stage. History-window reduction and frustum retrieval
change the attention context and are not lossless acceleration settings.

## 4. OpenAI-Compatible Requests

### Text to image

Cosmos3 text-to-image uses `/v1/images/generations`. The default Cosmos3 image response is `b64_json`, matching vLLM-Omni's examples.

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "1280x720",
    "n": 1,
    "num_inference_steps": 35,
    "guidance_scale": 6.0,
    "flow_shift": 3.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'
```

With a server running `nvidia/Cosmos3-Super-Text2Image-4Step`, omit the scheduler controls and use `guidance_scale=1.0`:

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "640x640",
    "n": 1,
    "guidance_scale": 1.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'
```

### Text to video with sound

Use `/v1/videos` to create an asynchronous job, then poll the job and download the completed MP4. Set `generate_sound=true` to generate and mux a stereo 48 kHz audio track; omit it for a silent video.

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=81" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=4.0" \
  --form-string "flow_shift=10.0" \
  --form-string "generate_sound=true" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
    | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \
  -o cosmos3_t2v.mp4
```

### Image to video

This mirrors the official `nvidia/Cosmos3-Nano` Hugging Face image-to-video example:

```python Python theme={null}
import json
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

base_url = "http://127.0.0.1:30010"
model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano"))
asset_dir = model_dir / "assets"

prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text()))
negative_prompt = json.dumps(
    json.loads((asset_dir / "negative_prompt.json").read_text())
)

data = {
    "prompt": prompt,
    "negative_prompt": negative_prompt,
    "size": "1280x720",
    "num_frames": "189",
    "fps": "24",
    "num_inference_steps": "35",
    "guidance_scale": "6.0",
    "max_sequence_length": "4096",
    "flow_shift": "10.0",
    "seed": "1111",
    "extra_params": json.dumps(
        {
            "use_resolution_template": False,
            "use_duration_template": False,
            "guardrails": True,
        }
    ),
}

with (asset_dir / "example_i2v_input.jpg").open("rb") as image:
    response = requests.post(
        f"{base_url}/v1/videos",
        data=data,
        files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")},
        timeout=60,
    )
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300)
response.raise_for_status()
Path("cosmos3_i2v.mp4").write_bytes(response.content)
```

For the distilled I2V checkpoint, use the same API with a server running `nvidia/Cosmos3-Super-Image2Video-4Step`. The recommended request is 480p and does not specify scheduler controls:

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \
  --form "input_reference=@first_frame.png;type=image/png" \
  --form-string "size=832x480" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "guidance_scale=1.0" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
```

Poll and download this job with the same status and content endpoints used by the T2V example.

### Video to video

Upload a source video with `video_reference`. Cosmos3 keeps latent frames `[0, 1]` by default and generates the remaining frames. Use `condition_frame_indexes` to select different latent frames, and `condition_video_keep` to take conditioning frames from the start or end of the source.

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \
  --form "video_reference=@robot_pouring.mp4;type=video/mp4" \
  --form-string "size=1280x704" \
  --form-string "num_frames=45" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string 'condition_frame_indexes=[0,1]' \
  --form-string "condition_video_keep=first" \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')
```

Poll and download this job with the same status and content endpoints used by the T2V example.

### Action generation

For DROID policy generation, start a single-GPU server with either the Nano or Edge policy checkpoint. Cosmos3 action generation does not currently support CFG or sequence parallelism.

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1
```

Use `nvidia/Cosmos3-Edge-Policy-DROID` in the same command to serve the smaller 4B policy checkpoint.

`policy` and `inverse_dynamics` return actions, so their canonical API is the synchronous `/v1/actions/generations` endpoint. The following request predicts a 16-step action chunk from one observation image. `action_horizon=16` maps to the model's `num_frames=17` convention.

```python Python theme={null}
import base64
from pathlib import Path

import requests

image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode()
response = requests.post(
    "http://127.0.0.1:30010/v1/actions/generations",
    json={
        "input": {
            "task": "Put the pot to the left of the purple item.",
            "observation": {
                "image": {"b64_json": image_b64},
            },
        },
        "parameters": {
            "action_mode": "policy",
            "action_horizon": 16,
            "domain_name": "droid_lerobot",
            "height": 480,
            "width": 832,
            "fps": 5,
            "num_inference_steps": 30,
            "guidance_scale": 1.0,
            "seed": 42,
        },
    },
    timeout=300,
)
response.raise_for_status()
action = response.json()["data"][0]["action"]
print(action["shape"], action["values"])
```

Use `GET /v1/actions/metadata` to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the `/v1/actions/realtime` websocket use the same action envelope.

To batch policy observations inside one request, opt in with a bounded batch size:

```bash Command theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1 \
  --batching-max-size 4
```

Send one image per observation as a list or `[B, H, W, C]` uint8 array in `input.input_reference`, and either one prompt per image or one scalar prompt to broadcast across the batch. Batched prompts must currently tokenize to the same length because Cosmos3 GEN cross-attention does not mask padded text K/V. All items in one request share the domain, resolution, action horizon, and denoise settings. The standard action envelope returns one `data[i]` item per input, each with action shape `[H, D]`. For a compact msgpack response containing one `[B, H, D]` array, set `runtime.response_format="raw"` and read the top-level `actions` field.

For JSON, `input_reference` can be a list of base64 image payloads. For msgpack, it can be a packed uint8 numpy array directly:

```json JSON theme={null}
{
  "input": {
    "prompt": ["pick up the block", "close the drawer"],
    "input_reference": [
      {"b64_json": "<first-image-base64>"},
      {"b64_json": "<second-image-base64>"}
    ]
  },
  "parameters": {
    "action_mode": "policy",
    "domain_name": "droid_lerobot"
  }
}
```

The batch size cannot exceed `--batching-max-size`; this keeps one request from bypassing the server's configured memory limit. Batching applies to `action_mode="policy"` only. A request seed controls the random stream for the whole batch, so a batched result is deterministic for that request but is not expected to be bit-exact with separately seeded B=1 requests.

`inverse_dynamics` also uses `/v1/actions/generations`; set `action_mode="inverse_dynamics"` and pass an observation video URL or server-local path as `input.observation.video`. Select the embodiment head with `domain_name` or `domain_id`; set `raw_action_dim` explicitly when it cannot be inferred from the domain name.

`forward_dynamics` is intentionally different: it consumes an action array and predicts video, so it remains on `/v1/videos`. Action-producing modes submitted to `/v1/videos` return HTTP 400 with the canonical action endpoint in the error message.

## 5. Cosmos3 Parameters

Cosmos3 supports the standard SGLang video and image fields such as `size`, `num_frames`, `fps`, `num_inference_steps`, `guidance_scale`, `negative_prompt`, and `seed`. For distilled checkpoints, SGLang replaces `num_inference_steps` with the checkpoint's fixed four-step schedule and forces `guidance_scale=1.0`; negative-prompt CFG and request-level `flow_shift` do not apply.

Top-level Cosmos3 request fields:

* `max_sequence_length`: maximum text token length used by the Cosmos3 tokenizer.
* `flow_shift`: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses `--flow-shift`, then the mode default (`3.0` for T2I, `10.0` for non-Edge video and all action modes, or `3.0` for Edge video).
* `guidance_interval`: optional `[start, end]` noise interval for CFG. Non-distilled T2I defaults to `[400, 1000]`; video modes guide at every step.

Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields:

* `generate_sound`: generate a sound track whose duration follows `num_frames / fps`.
* `sound_duration`: explicit sound duration in seconds; takes precedence over the derived duration.
* `condition_frame_indexes`: V2V latent-frame indexes to keep from the source video; defaults to `[0, 1]`.
* `condition_video_keep`: use the `first` or `last` source frames for V2V conditioning.
* `action_mode`: `policy`, `forward_dynamics`, or `inverse_dynamics`.
* `domain_name` / `domain_id`: select the action embodiment head.
* `raw_action_dim`: number of active action dimensions; inferred for known domain names.
* `action`: action array with shape `[T, D]`, required by `forward_dynamics`.
* `action_fps`: action-token frame rate for temporal mRoPE; defaults to the video FPS.
* `action_view_point`: viewpoint used in the structured action caption.
* `action_normalization`: dataset normalization mode, such as `quantile`, `meanstd`, or `minmax`.

Pass model-specific controls through `extra_body` with the OpenAI Python SDK.
Raw JSON may keep them at the top level; multipart video requests should put
them in the `extra_params` JSON object. The legacy image `extra_args` container
remains accepted for compatibility, but new clients should use `extra_body`:

* `use_duration_template`: whether to append SGLang's generated duration suffix to video prompts.
* `use_resolution_template`: accepted for vLLM-Omni request compatibility.
* `use_system_prompt`: whether to add the Cosmos3 system prompt to the chat template.
* `guardrails` or `use_guardrails`: per-request guardrail toggle when the server started with guardrails enabled.

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# ERNIE-Image
Source: https://docs.sglang.io/cookbook/diffusion/Ernie-Image/Ernie-Image

Deploy ERNIE-Image and ERNIE-Image-Turbo with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[ERNIE-Image](https://huggingface.co/baidu/ERNIE-Image) is Baidu's text-to-image family with separate standard and Turbo checkpoints. The standard model is the quality-oriented path; Turbo is the lower-latency choice, and both load through SGLang's native `ErnieImagePipeline`.

This integration currently targets text-only image generation rather than image editing or reference conditioning. Choose it for a straightforward single-GPU T2I deployment; use a dedicated editing model when preserving source-image structure is part of the task.

| Model | Hugging Face model ID | Notes |
| - | - | - |
| ERNIE-Image | `baidu/ERNIE-Image` | Regular text-to-image checkpoint |
| ERNIE-Image-Turbo | `baidu/ERNIE-Image-Turbo` | Turbo text-to-image checkpoint |

## 3. Serve the model

The commands below target a single supported NVIDIA CUDA or AMD ROCm GPU. Start with `--performance-mode auto`; use `speed` only when the full pipeline fits comfortably on the selected GPU(s), and use `memory` when you need lower peak GPU memory.

Serve ERNIE-Image:

```bash Command theme={null}
sglang serve \
  --model-path baidu/ERNIE-Image \
  --num-gpus 1 \
  --performance-mode auto \
  --port 30010
```

Serve ERNIE-Image-Turbo:

```bash Command theme={null}
sglang serve \
  --model-path baidu/ERNIE-Image-Turbo \
  --num-gpus 1 \
  --performance-mode auto \
  --port 30010
```

Intel Xeon CPU deployment is supported by prefixing the server command with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` and omitting GPU-only flags.

Serve ERNIE-Image:

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path baidu/ERNIE-Image --port 30010
```

Serve ERNIE-Image-Turbo:

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path baidu/ERNIE-Image-Turbo --port 30010
```

## 4. Generate an image

Use the OpenAI-compatible image generation API after the server starts:

```python Python theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://127.0.0.1:30010/v1")

response = client.images.generate(
    model="baidu/ERNIE-Image-Turbo",
    prompt="A cinematic photo of a quiet lakeside cabin at sunrise",
    n=1,
    response_format="b64_json",
)

image_bytes = base64.b64decode(response.data[0].b64_json)
with open("ernie_image.png", "wb") as f:
    f.write(image_bytes)
```

## 5. Configuration tips

* ERNIE-Image is a text-to-image pipeline; do not pass `--image-path`.
* `--performance-mode auto` keeps conservative defaults while preserving explicit user flags.
* If the checkpoint includes a PE component, SGLang loads it automatically with the native Ministral3 runtime. Use `--layerwise-offload-components pe` when the local PE decoder needs to trade latency for lower GPU memory usage.
* Treat FSDP, SP/Ulysses/Ring, and TP as explicit benchmark knobs. Measure the target resolution, step count, and GPU type before making them production defaults.

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# FLUX
Source: https://docs.sglang.io/cookbook/diffusion/FLUX/FLUX

Deploy FLUX diffusion models with SGLang - 12B/32B rectified flow transformers for high-quality text-to-image generation.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[FLUX](https://blackforestlabs.ai/) is Black Forest Labs' rectified-flow image model family. [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) is the smaller 12B text-to-image checkpoint; [FLUX.2-dev](https://huggingface.co/black-forest-labs/FLUX.2-dev) is a 32B model that adds instruction-based editing plus single- and multi-reference composition.

FLUX is a strong default when prompt adherence, polished image quality, or reference consistency matters. The tradeoff is deployment weight: FLUX.2 needs substantially more memory than FLUX.1, and the dev checkpoints use the FLUX non-commercial license, so review the model license before production use.

| Checkpoint | Best fit | Main limitation |
| - | - | - |
| `black-forest-labs/FLUX.1-dev` | High-quality text-to-image with the lighter FLUX deployment | No native multi-reference editing path |
| `black-forest-labs/FLUX.2-dev` | Text-to-image, editing, and reference-guided composition in one model | 32B model with a larger memory footprint |

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements.

* `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
* `--num-gpus`: Number of GPUs to use
* `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
* `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
* `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
* `--ring-degree`: The degree of ring attention-style SP in USP

### 3.3 Breakable CUDA graph for FLUX.1-dev

For repeated `quality=lossless` requests, FLUX.1-dev supports breakable CUDA graph (BCG) execution with the native backend. BCG captures DiT segments during startup and replays them for the warmed resolutions. It can reduce recurring host launch overhead without enabling `torch.compile`.

The following configuration was validated on two NVIDIA H200 GPUs with BF16 weights, PyTorch 2.13, and CUDA 13.0. Set `HF_TOKEN` to a Hugging Face token with access to the checkpoint before downloading it.

```bash theme={null}
CUDA_VISIBLE_DEVICES=0,1 sglang generate \
  --model-path black-forest-labs/FLUX.1-dev \
  --backend sglang \
  --num-gpus 2 --tp-size 2 \
  --component-residency dit=resident \
  --enable-torch-compile false \
  --enable-breakable-cuda-graph \
  --warmup-resolutions 1024x1024 \
  --quality exact \
  --width 1024 --height 1024 \
  --num-inference-steps 50 --guidance-scale 3.5 --seed 42 \
  --prompt "A futuristic cyberpunk city at night, neon lights reflecting on wet streets"
```

Confirm `[Diffusion BCG] captured` in the log, then compare warmed request latency with eager execution on the same GPUs. Capture time and graph memory are additional startup costs. Add other served resolutions to `--warmup-resolutions`; a request with an uncaptured signature falls back to eager.

FLUX.1-dev uses a fixed 512-token T5 conditioning sequence, so changing `--bcg-text-buckets` does not create additional prompt-length graphs. Its request-gated DiT fusions at `quality=high` and `lossless` cannot be combined with BCG: the runtime rejects those requests because the captured graph uses the lossless branches. FLUX.2 and quantized transformer overrides require separate validation.

## 4. API Usage

For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api).

### 4.1 Generate an Image

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1")

response = client.images.generate(
    model="black-forest-labs/FLUX.1-dev",
    prompt="A cat holding a sign that says hello world",
    size="1024x1024",
    n=1,
    response_format="b64_json",
)

# Save the generated image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
    f.write(image_bytes)
```

### 4.2 Advanced Usage

#### 4.2.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit).

**Basic Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path black-forest-labs/FLUX.1-dev
```

**Advanced Usage**

* DBCache Parameters: DBCache controls block-level caching behavior:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

Combined Configuration Example:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path black-forest-labs/FLUX.1-dev
```

#### 4.2.2 CPU Offload

* `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
* `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
* `--vae-cpu-offload`: Use CPU offload for VAE.
* `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".

#### 4.2.3 Known LoRA examples

Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known FLUX examples include:

* [`dvyio/flux-lora-simple-illustration`](https://huggingface.co/dvyio/flux-lora-simple-illustration)
* [`XLabs-AI/flux-furry-lora`](https://huggingface.co/XLabs-AI/flux-furry-lora)
* [`XLabs-AI/flux-RealismLora`](https://huggingface.co/XLabs-AI/flux-RealismLora)

#### 4.2.4 Negative prompts (true CFG)

FLUX.1-dev is guidance-distilled: its guidance embedding comes from the server's `--embedded-cfg-scale` (3.5 for FLUX.1), and the requests above run one DiT forward per step. A negative prompt turns on classifier-free guidance, which adds an unconditional forward per step:

* `--true-cfg-scale` sets the CFG weight, as in diffusers `FluxPipeline`.
* Without `--true-cfg-scale`, the negative prompt uses `--guidance-scale` as the CFG weight.

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/FLUX.1-dev \
  --prompt "A red bicycle leaning against a stone wall beside a quiet lake at sunset" \
  --negative-prompt "blurry, distorted, low quality" \
  --true-cfg-scale 2.0 \
  --width 1024 --height 1024 --num-inference-steps 50 --seed 42
```

Keep the weight moderate. On top of the distilled guidance, CFG over-exposes FLUX.1-dev in diffusers as well as in SGLang: at 1024x1024 and 28 steps, about 45% of pixels clip to 0 or 255 at a weight of 4, and 8-15% at 1.5-2.

## 5. Benchmark

### 5.1 Speedup Benchmark

#### 5.1.1 Generate a image

Test Environment:

* Hardware: NVIDIA B200 GPU (1x)
* Model: black-forest-labs/FLUX.1-dev
* sglang diffusion version: 0.5.6.post2

**NVIDIA B200**

**Server Command**:

```shell Command theme={null}
sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000
```

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Model:                                   black-forest-labs/FLUX.1-dev
Dataset:                                 vbench
Task:                                    text-to-image
--------------------------------------------------
Benchmark duration (s):                  50.97
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.02
Latency Mean (s):                        50.9681
Latency Median (s):                      50.9681
Latency P99 (s):                         50.9681
--------------------------------------------------
Peak Memory Max (MB):                    27905.19
Peak Memory Mean (MB):                   27905.19
Peak Memory Median (MB):                 27905.19
============================================================
```

**Ascend A3 Series**

**Server Command**:

```shell Command theme={null}
#One A3 Series card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2
```

**Benchmark Command**:

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        black-forest-labs/FLUX.1-dev
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       16.30
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.06
Output throughput (outputs/s):                0.06
Latency Mean (s):                             16.30
Latency Median (s):                           16.30
Latency P90 (s):                              16.30
Latency P95 (s):                              16.30
Latency P99 (s):                              16.30
--------------------------------------------------
Peak Memory Max (MB):                         19972.00
Peak Memory Mean (MB):                        19972.00
Peak Memory Median (MB):                      19972.00
------------------------------------------------------------
```

#### 5.1.2 Generate images with high concurrency

**NVIDIA B200**

**Server Command** :

```shell Command theme={null}
sglang serve --model-path black-forest-labs/FLUX.1-dev --port 30000
```

**Benchmark Command** :

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
```

**Result** :

```text Output theme={null}
================= Serving Benchmark Result =================
Model:                                   black-forest-labs/FLUX.1-dev
Dataset:                                 vbench
Task:                                    text-to-image
--------------------------------------------------
Benchmark duration (s):                  111.79
Request rate:                            inf
Max request concurrency:                 20
Successful requests:                     20/20
--------------------------------------------------
Request throughput (req/s):              0.18
Latency Mean (s):                        67.0646
Latency Median (s):                      66.9691
Latency P99 (s):                         110.8949
--------------------------------------------------
Peak Memory Max (MB):                    27917.19
Peak Memory Mean (MB):                   27916.59
Peak Memory Median (MB):                 27917.19
============================================================
```

**Ascend A3 Series**

**Server Command** :

```shell Command theme={null}
#One A3 Series card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2
```

**Benchmark Command** :

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
```

**Result** :

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        black-forest-labs/FLUX.1-dev
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       300.85
Request rate:                                 inf
Max request concurrency:                      20
Successful requests:                          18/20
Completed outputs:                            18
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.06
Output throughput (outputs/s):                0.06
Latency Mean (s):                             155.16
Latency Median (s):                           155.11
Latency P90 (s):                              266.30
Latency P95 (s):                              280.15
Latency P99 (s):                              291.23
--------------------------------------------------
Peak Memory Max (MB):                         19972.00
Peak Memory Mean (MB):                        19972.00
Peak Memory Median (MB):                      19972.00
------------------------------------------------------------
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — ComfyUI's own sampler, CLIP, and
VAE drive the loop while SGLang replaces the model forward. Load the
checkpoint with `SGLDiffusion UNET Loader` and set
`model_type` to `flux` on the
`SGLDiffusion Options` node.

A reference workflow is included at
`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/flux_sgld_sp.json`.


# FLUX.3
Source: https://docs.sglang.io/cookbook/diffusion/FLUX/FLUX.3

Serve FLUX.3 Image Klein text-to-image generation and image editing with SGLang Diffusion.

## 1. Quick start

Use a [SGLang Diffusion Docker environment](/docs/sglang-diffusion/installation).
Until a released image contains this integration, install SGLang from `main`
inside it:

```bash theme={null}
pip install 'sglang[diffusion] @ git+https://github.com/sgl-project/sglang.git#subdirectory=python'
```

> [!NOTE]
>
> The picker selects `black-forest-labs/flux-3-image-klein-base` or
> `black-forest-labs/flux-3-image-klein`, each a self-contained repository. Access
> to the selected checkpoint is required. Keep `--model-path` at its repository
> root; `--model-variant` selects only the quantized DiT. Both release BF16
> checkpoints have single-GPU generation checks; see the validation boundaries
> below. Unverified hardware or resource shapes remain selectable but are not
> measured recommendations.

The picker defaults to resident weights for its 1 MP, batch-1 workload. Its
recommended execution uses BCG only where a matched E2E comparison showed a
material gain: currently single-GB300 distilled NVFP4 T2I. Other combinations
start eager; this graph recipe covers the 80-token text bucket. Changing
precision never happens automatically. Recommendations are best among the
tested settings, not an exhaustive optimum.

> [!WARNING]
>
> FLUX.3 Image Klein is released under the
> [FLUX Kommunity License v1.0](https://huggingface.co/black-forest-labs/flux-3-image-klein/blob/main/LICENSE.md):
> non-commercial use, plus commercial output generation for Qualifying Users with
> less than US\$5M in gross annualized revenue, subject to its terms. The
> Qwen3-VL-4B-Instruct text encoder is Apache-2.0. Review the license before
> production or commercial use.

For platform-specific installation, see the
[SGLang Diffusion installation guide](/docs/sglang-diffusion/installation).

## 2. Model capabilities

[FLUX.3 Image Klein](https://huggingface.co/collections/black-forest-labs/flux-3-image-klein)
is Black Forest Labs' 5.6B open-weights model for text-to-image generation and
image editing. It takes a prompt and up to 10 reference images and returns one
image; text-to-image, single-reference edits and multi-reference compositions
run through the same DiT, with Qwen3-VL-4B-Instruct as the text encoder. The
5.6B count covers the image transformer only.

Prompts can place elements with bounding boxes on a 0 to 1000 grid, then
recolor, move, replace or remove them; boxes guide placement and scale rather
than masking pixels. The model also renders text inside images. One SGLang
server serves text-to-image and editing, chosen per request. This image
pipeline does not serve the separate FLUX.3 Action robot policy.

| Checkpoint | Training | Steps | Guidance | Built for |
| - | - | - | - | - |
| [Klein](https://huggingface.co/black-forest-labs/flux-3-image-klein) | step-distilled flow map | 16 | none | generation and editing, self-hosting; the checkpoint behind the BFL API |
| [Klein Base](https://huggingface.co/black-forest-labs/flux-3-image-klein-base) | undistilled | 50 | 3.5 | fine-tuning, LoRA training, research |

Each repository ships its BF16 DiT with FP8 (`fp8r`) and NVFP4 variants,
together with the text encoder and image autoencoder. Quantization and caching
are separate fidelity/latency tradeoffs, not lossless substitutes for the
selected checkpoint.

| Task | Endpoint | Conditioning |
| - | - | - |
| Text to image | `POST /v1/images/generations` | Text prompt |
| Image to image | `POST /v1/images/edits` | Text prompt and 1 to 10 `image` uploads |

Start the server:

```bash theme={null}
sglang serve \
  --model-path black-forest-labs/flux-3-image-klein-base \
  --host 0.0.0.0 \
  --port 30010
```

Sampling is set per request (`num_inference_steps`, `guidance_scale`, `size`,
`seed`, `n`); unset fields take the checkpoint's defaults from its
`config.json`. SGLang expands `n` into independently seeded requests and can
pack compatible requests into one DiT forward.

References are prepared as in the
[BFL reference pipeline](https://github.com/black-forest-labs/flux-3-image-klein).
A single reference is
fitted under 4 MP, reflect-padded to the 16-pixel grid and, when you set no
output size, sets the output size; the padding is cropped from the result.
With several references each is fitted under 1 MP and centre-cropped, and the
output takes the first one's size.

### Prompting

The caption controls composition; reference images anchor content but do not
lock structure.

* Write full scene captions covering framing, subject, background and lighting.
  A short prompt gives generic output, and a one-line edit instruction ("make it
  blue") reframes the shot. The [prompt upsampler](#7-prompt-upsampler) expands
  a short request into a full caption.
* For edits, describe the target image as a complete caption of the reference
  with the edit folded in.
* For layout, append a JSON list of elements to the caption. Each box is
  `[top, left, bottom, right]` on a 0 to 1000 grid, and the caption tags each
  element as `<id>`. To edit by box, refer to the n-th reference as
  `<ref_image_n>` and give each element `from`, `src_bbox` and `tgt_bbox`; a
  `null` target removes it. See BFL's
  [bounding box guide](https://docs.bfl.ai/flux_3/flux3_image_bounding_boxes)
  for the full format.

## 3. Checkpoints and precision

| Repository | `--model-variant` | Sampling |
| - | - | - |
| `black-forest-labs/flux-3-image-klein-base` | Omit / `fp8r` / `nvfp4` | Base CFG; defaults from the selected configuration |
| `black-forest-labs/flux-3-image-klein` | Omit / `fp8r` / `nvfp4` | Step-distilled flow map; no CFG |

Both repositories use `flux-3-image-klein` in configuration and safetensors
metadata. This shared identifier does not determine the sampling mode. The
release repositories do not need the pre-release `sd-*` variant names.

The distilled model is a step-distilled flow-map DiT: each step jumps straight
to the next time of the schedule, so it samples without CFG. SGLang tells a
flow map from its weights. It rejects `--enable-cfg-parallel`, a request
`guidance_scale` other than 1, and a non-empty `negative_prompt`. Tensor
parallelism, Ulysses SP and breakable CUDA graphs work as for the base model.

The Base model defaults to CFG 3.5 against an empty negative prompt. Both
variants reuse the package's text encoder and VAE. Keep the checkpoint revision,
precision, attention backend, guidance and sampling parameters fixed when
comparing outputs.
Explicit `guidance_scale=1` uses scale 1, not the package default;
`true_cfg_scale`, when provided, takes precedence over `guidance_scale`.

### Quantized checkpoints

Each repo ships quantized DiTs under `variants/`: select one with
`--model-variant fp8r` or `--model-variant nvfp4` and keep `--model-path`
pointing at the repo root. Text encoder and VAE precision follow their own
package settings; quantizing the DiT does not quantize them. The quantization
comes from the variant's `hf_quant_config.json`, so do not also pass
`--quantization`.

```bash theme={null}
sglang serve \
  --model-path black-forest-labs/flux-3-image-klein-base \
  --model-variant fp8r \
  --host 0.0.0.0 \
  --port 30010
```

**FP8R** stores rowwise-FP8 weights. On Blackwell (SM10x) the FP8 projections
run FlashInfer PrimsTS GEMMs with fused QK-norm/RoPE and SwiGLU epilogues
(FlashInfer >= 0.7.1); set `SGLANG_ENABLE_FLUX3_PRIMS_TS_AUTOTUNE=1` to
autotune them during warmup. Other GPUs use the generic rowwise-FP8 GEMM.

**NVFP4** stores FP4 (E2M1) weights with FP8 block scales and needs a Blackwell
GPU and a compatible GEMM implementation. SM10x and consumer SM120 use
different kernel paths; a B200 recipe does not establish RTX 5090 performance.
Both quantization modes can change output quality and need workload-specific
comparison against BF16.

These flags remain separately configurable; see the validation boundaries below
before combining quantization with other optimizations.

### Breakable CUDA graphs

BCG reserves additional device memory and captures specific request shapes.
The picker generates `--warmup-resolutions 1024x1024` and matching
`--warmup-sampling-params` for the selected quality. For editing it also sets
`flux3_mode="i2i"` and a server-local reference path: mount a 1024x1024 image
there before starting the server. This path can differ from the client's upload
path. Longer prompt token buckets, other reference dimensions/counts, batches
or quality settings may run eager instead; check for a serving-signature miss
in the server log.

Changing the picker workload also changes its graph warmup command. Restart
with that command to capture the new workload. Keep eager for variable-shape
traffic, or explicitly validate every shape you intend to capture.

### LoRA adapters

Use the standard LoRA API with native DiT keys, optionally prefixed by `dit.` or
`transformer.`. BF16 dynamic and merged adapters have a synthetic rank-4 generation
check, including repeatability and exact restoration after unloading; this does
not validate every community adapter. For quantized deployments requiring LoRA,
pass `--lora-path` at startup to select the generic quantized layout. Attaching
LoRA after PrimsTS weight preparation is rejected before replacing any layers.
FP8R and NVFP4 startup adapters also have synthetic rank-4 generation checks
on GB300, including repeatability and exact restoration after unloading. These
do not establish compatibility with arbitrary quantized community adapters.

## 4. Multi-GPU and memory

FLUX.3 shares its DiT with FLUX.3 Action, so the image pipeline supports
tensor parallelism, Ulysses sequence parallelism, CFG parallelism, layerwise
offload and breakable CUDA graphs. The picker covers RTX 5090, RTX PRO 6000,
H100/H200, and B200/B300/GB200/GB300 with one to eight GPUs on one node.
TP times Ulysses degree must equal the GPU count and evenly partition the
24 attention heads. Ring is not admitted by this image recipe.

For example, two-GPU TP uses `--num-gpus 2 --tp-size 2`; two-GPU Ulysses
uses `--num-gpus 2 --tp-size 1 --ulysses-degree 2`. Base also permits CFG
parallelism; distilled checkpoints have no CFG branch to parallelize.
More GPUs do not guarantee lower latency, particularly over PCIe.

Ulysses inherits all-to-all streaming through the shared attention layer.
Leave its pipeline-group policy automatic; short sequences need not benefit.
TP-only deployments do not use this path. Encoding-while-decoding overlaps
video-frame decoding with video-file encoding, so it does not apply to this
still-image pipeline.

> [!NOTE]
>
> Under sequence parallelism every rank holds the full DiT, so combining it with
> layerwise offload streams the whole model to each GPU every step. TP also
> partitions weights, so it is the lower-memory starting point when offloading
> on several GPUs. Preserve headroom for the longest prompt, references and output
> size you intend to serve; a successful small request is not a capacity guarantee.

For FP32 VAE attention above 1 MP, `quality="lossless"` can use dense-layout,
full-context memory-efficient SDPA; `quality="exact"` retains the original path.
It does not prune attention or reduce precision, but reduction order can change
pixels. In a Base BF16 test with a 2048x2048 reference and 1024x1024 output,
GB300 peak allocated memory fell from 54.3 to 29.5 GiB; latency was effectively
unchanged. GB300/H200 image SSIM was 0.9951-0.9992 against the exact path across
four samples. This is allocated memory, not total device usage: a 32 GB card
still needs offload or quantization headroom for that reference size.

### Disaggregated serving

You can deploy the prompt encoder, reference-image VAE encoder, DiT, and VAE
decoder as separate worker pools behind the same image API. Both T2I and i2i
are supported. Use the [split encoding deployment guide](/docs/sglang-diffusion/disaggregation#flux3-split-encoding)
for the four-role commands, or keep the existing three-role topology with a
combined encoder. This separates resources and scaling; it adds transfer hops
and does not imply lower single-request latency.
T2I skips the image encoder; editing keeps reference padding/crop semantics.
Each pool can have a different number of workers. Keep the package variant and
numerical settings consistent across workers; the guide includes the required
transfer dependency, network ports, and same-node validation scope.

## 5. Cache-DiT

[Cache-DiT](/docs/sglang-diffusion/cache_dit) skips the 28 single-stream blocks
on steps where their input barely changes and reuses the cached residual. Turn
it on for every request with `SGLANG_CACHE_DIT_ENABLED=true` on `sglang serve`,
or per request with `"enable_cache_dit": true`. A request can also tune it, for
example `"cache_dit_params": {"residual_diff_threshold": 0.12, "enable_taylorseer": true}`.

For image editing, send `-F "enable_cache_dit=true"` and a JSON-encoded
`cache_dit_params` form field; generation uses JSON request fields.

Cache-DiT is approximate and may change composition; measure quality and latency
on representative prompts rather than treating its threshold as a fidelity guarantee.
It combines with FP8R and NVFP4, tensor parallelism, CFG parallelism and
layerwise offload. It is skipped, with a warning, under Ulysses sequence
parallelism and breakable CUDA graphs, and for a request whose CFG pair has to
run as two forwards to fit in device memory.

## 6. Request examples

**Text to image**

```bash theme={null}
curl -sS -X POST http://localhost:30010/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "A red fox standing in fresh snow", "size": "1024x1024", "seed": 0, "response_format": "b64_json"}' \
  -o flux3_t2i.json
```

**Image to image**

```bash theme={null}
curl -sS -X POST http://localhost:30010/v1/images/edits \
  -F "image=@/data/flux3/reference.png" \
  -F "prompt=Render the reference as a winter wildlife photograph" \
  -F "seed=0" \
  -F "response_format=b64_json" \
  -o flux3_i2i.json
```

**Offline generation**

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/flux-3-image-klein-base \
  --height 1024 --width 1024 \
  --num-inference-steps 50 \
  --guidance-scale 3.5 \
  --prompt "A red fox standing in fresh snow"
```

Pass `--image-path` (one or more images) to edit references instead.

## 7. Prompt upsampler

[black-forest-labs/flux-3-image-klein-pup](https://huggingface.co/black-forest-labs/flux-3-image-klein-pup)
is a Qwen3.5-9B vision-language fine-tune that expands a short prompt into the detailed
caption FLUX.3 works best with. The BFL reference code runs it in front of the image model
by default. It is a standard `Qwen3_5ForConditionalGeneration` checkpoint, so it is served
as a regular LLM, separately from the image pipeline. The system prompt (`system_prompts.json`
in the repository) selects the mode; `t2i` is the text-to-image upsampler.

Download the package and start the upsampler on its own port:

```bash theme={null}
PUP_PATH="<local-path>"
hf download black-forest-labs/flux-3-image-klein-pup --local-dir ${PUP_PATH}

CUDA_VISIBLE_DEVICES=1 sglang serve ${PUP_PATH} \
      --model-type llm \
      --served-model-name flux-3-image-klein-pup \
      --host 0.0.0.0 --port 8001 \
      --context-length 16384 \
      --mem-fraction-static 0.90 \
      --reasoning-parser qwen3
```

Upsample a text prompt through the OpenAI-compatible chat endpoint. The `t2i` system
prompt is read from the downloaded package; thinking is disabled and decoding follows the
model card (greedy, presence penalty 1.5):

```bash theme={null}
jq -n --slurpfile p ${PUP_PATH}/system_prompts.json '{
    model: "flux-3-image-klein-pup",
    messages: [
      {role: "system", content: $p[0].t2i},
      {role: "user",   content: "A man walking with his dog, afternoon sun"}
    ],
    max_tokens: 3584,
    temperature: 0,
    presence_penalty: 1.5,
    chat_template_kwargs: {enable_thinking: false}
  }' | curl -s http://0.0.0.0:8001/v1/chat/completions \
         -H 'Content-Type: application/json' -d @- \
     | jq -r '.choices[0].message.content'
```

You can then pass the returned caption as the `--prompt` of a `t2i` request to the Flux3 pipeline.

To have the image server call the upsampler itself, point it at this endpoint with the
built-in `flux3-klein-pup` protocol. It reads `system_prompts.json` from the upsampler
package (`protocol_source`, a local directory or the Hugging Face repo, which is the
default) and follows BFL's reference code: the `t2i` prompt for generations, the `i2i`
prompt with the references first (resized as the reference code does) for edits, greedy
decoding with presence penalty 1.5 for `t2i` and 0 for `i2i`, and thinking disabled.

```bash theme={null}
cat > flux3_pup.json <<'JSON'
{
  "base_url": "http://127.0.0.1:8001/v1",
  "model": "flux-3-image-klein-pup",
  "protocol": "flux3-klein-pup",
  "protocol_source": "<local-path>"
}
JSON

sglang serve --model-path black-forest-labs/flux-3-image-klein-base \
  --prompt-enhancer-config flux3_pup.json --port 30010
```

A request opts in with `enhance_prompt`; it is off by default, so a full caption is sent
as typed:

```bash theme={null}
curl -sS -X POST http://localhost:30010/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt": "A man walking with his dog, afternoon sun", "enhance_prompt": true, "size": "1024x1024", "response_format": "b64_json"}'
```

Greedy decoding is not by itself reproducible here: the caption changes with the
upsampler's prefix cache and batch. Start the upsampler with
`--enable-deterministic-inference --disable-radix-cache` when the same request must give
the same caption.

The layout prompts (`t2i_bbox`, `i2i_bbox`) are not wired in yet; write the box table
into the prompt yourself as described in [Model capabilities](#2-model-capabilities).

## 8. Dynamic batching

Enable a small batching window to collect concurrent requests:

```bash Serve concurrent requests theme={null}
sglang serve --model-path black-forest-labs/flux-3-image-klein-base \
  --batching-max-size 2 --batching-delay-ms 5
```

The existing scheduler groups requests with matching sampling/runtime settings.
Each keeps its own prompt, seed and reference images. Text encoding groups unique
prompts by token-length bucket; DiT attention remains isolated per packed sample,
including each CFG branch. VAE decoding groups outputs by canvas size before
restoring request order and each reference crop.

Larger batches need more activation memory and may increase individual request
latency. Start with batch 2, compare throughput and tail latency under your actual
QPS, and keep batch 1 for low-latency or low-VRAM workloads. Batching does not imply
bitwise equality with separate forwards: GEMM and attention reduction choices can
change with shape. Check output fidelity with fixed prompts and seeds.

### Pre-release batching measurements

These measurements used pre-release weights, not the release repositories
above. They do not establish release-checkpoint performance.

The following are warmed HTTP measurements on one GB300 (about 277 GiB device
memory), Base BF16, Torch SDPA, eager execution and all components resident.
Each row compares two different prompt/seed requests served serially with the
same two requests served together. Times include local HTTP transfer and JPEG
output; medians cover three rounds after warming both shapes and reference
encoding. Software: PyTorch 2.14.1+cu130, Transformers 5.19.0. Checkpoint revision:
`5de4fe32f17aee3dbc34375a94935f09206dba2c`.

| Workload | Serial pair | Batch-2 pair | Throughput ratio |
| - | - | - | - |
| T2I, 512x512, 8 steps | 0.649 s | 0.324 s | 2.00x |
| I2I, 512x512, 8 steps | 0.737 s | 0.546 s | 1.35x |
| T2I, 1024x1024, 50 steps | 6.248 s | 6.050 s | 1.03x |
| I2I, 1024x1024, 50 steps | 13.171 s | 12.996 s | 1.01x |

Eight steps exercise batching, not the recommended Base image-quality setting.
The standard 50-step workload shows little batching benefit;
FlashAttention gives a similarly small gain in this
setup. Batch 1 remains the default. Do not extrapolate the small-request speedup
to full-resolution generation or other GPUs.

For these 50-step requests, batch-1 versus batch-2 JPEG outputs had SSIM
0.9920-0.9977. A separate four-image PNG comparison at 512x512/8 steps had SSIM
0.9885-0.9955; JPEG can amplify small rounding differences. These are observed
samples, not a universal quality guarantee. Identical single requests before
and after batching matched exactly. Preserve additional memory headroom for
batching rather than assuming a batch-1 recipe fits on a 32 GB card at batch 2.

## 9. Validation boundaries

The release BF16 and FP8R checkpoints have single-GB300/H200 generation checks;
NVFP4 has GB300 and RTX 5090 checks. RTX 5090 also has Base BF16 and both
checkpoints' FP8R and Cache-DiT checks. Single-GPU measurements use eager
execution, all components resident, batch 1, 1024x1024 T2I and I2I, two
prompts/seeds and repeated requests. Base uses 50 steps and distilled uses 16.
Repeated requests matched pixel-for-pixel within each tested configuration.
This does not establish official-reference parity or validate every combination
of hardware, parallelism, batching and optional features in the picker.

### Multi-GPU measurements

Base BF16, 50 steps, CFG 3.5, batch 1 and `quality="exact"`. These are warmed
localhost HTTP medians through PNG decoding, excluding loading and warmup;
two prompts/seeds and two timed rounds per prompt. Software: PyTorch
2.14.1+cu130, Transformers 5.19.0, FlashInfer 0.7.1. Both runs use checkpoint
`d8acde427a698bdb98b5c1c941c60fa4c97e9f28`. The H200 run uses FA3 and
NVLink; the RTX 4090 run uses Torch SDPA over PCIe without CUDA peer access.

| GPUs | Topology | T2I 1024x1024 | I2I, 1 MP reference/output |
| - | - | - | - |
| 2x H200 | TP2 | 4.27 s | Not measured |
| 2x H200 | Ulysses 2, streaming auto | 4.68 s | Not measured |
| 4x RTX 4090 | TP4 | 16.97 s | 32.61 s |
| 4x RTX 4090 | TP2 x Ulysses 2 | 16.28 s | 32.63 s |

The picker keeps TP as its multi-GPU starting point, not a guarantee that TP
always wins. A hybrid topology can help on PCIe; validate it on your own
interconnect. H100, B200 and other unmeasured resource shapes are extrapolated
starting points and are not labelled verified.

On the same two H200s, Ulysses streaming reduced 2048x2048 T2I from 24.00 to
22.87 s and editing a 2048x2048 reference into a 1024x1024 output from 31.01
to 29.54 s. Streamed and non-streamed outputs matched pixel-for-pixel in these
samples, with no quantization or caching. At 1024x1024, automatic dispatch
selected no streaming groups and gave no speedup. TP and SP themselves are
not bit-identical: sample SSIM was 0.9890-0.9985. None of these comparisons
establishes which path is closer to an official reference.

### Single-GPU measurements

Warmed local HTTP medians, including PNG encoding and response transfer, from
two rounds per prompt. PyTorch 2.14.1+cu130, Transformers 5.19.0, FP32 VAE,
`quality="exact"`; FlashAttention on GB300/H200 and Torch SDPA on RTX 5090.
The Base checkpoint revision is `d8acde427a698bdb98b5c1c941c60fa4c97e9f28`.
SSIM compares each quantized result with BF16 on the same GPU, prompt, seed and
reference, and reports the lowest value across two samples, not a quality bound.

| GPU | Base DiT | T2I | I2I | Minimum SSIM, T2I / I2I |
| - | - | - | - | - |
| GB300, 277 GiB | BF16 | 3.21 s | 6.31 s | Reference |
| GB300, 277 GiB | FP8R | 2.18 s | 4.49 s | 0.958 / 0.981 |
| GB300, 277 GiB | NVFP4 | 2.14 s | 4.31 s | 0.919 / 0.958 |
| H200, 140 GiB | BF16 | 7.09 s | 14.52 s | Reference |
| H200, 140 GiB | FP8R | 5.15 s | 11.24 s | 0.955 / 0.984 |
| RTX 5090, 32 GB | BF16 | 20.39 s | 43.4 s | Reference |

For the distilled checkpoint, quantization's T2I benefit was small on GB300;
do not assume Base's speedup transfers to a 16-step workload. Cache-DiT at
threshold 0.12 reduced Base I2I from 6.31 to 3.08 s on GB300, with sample SSIM
0.9786-0.9975. It remains opt-in and approximate.

### Eager versus CUDA graphs

Matched ABBA comparisons use `quality="lossless"`, resident weights, batch 1,
1024x1024 output/reference, and the same software versions above with stable
FlashInfer 0.7.1. Base uses 50 steps/CFG 3.5; distilled uses 16 steps/no CFG.
Times are warmed localhost HTTP medians through JPEG decoding: two prompts/seeds
and four timed samples per arm, excluding loading and graph capture. These
JPEG timings are not directly comparable with the earlier PNG table.

| GPU | Checkpoint | Workload | Eager | BCG | Recommended |
| - | - | - | - | - | - |
| GB300 | Distilled NVFP4, FA4 | T2I | 0.563 s | 0.436 s | BCG |
| GB300 | Distilled NVFP4, FA4 | I2I | 0.882 s | 0.886 s | Eager |
| GB300 | Base BF16, FA4 | T2I | 3.103 s | 3.130 s | Eager |
| GB300 | Base BF16, FA4 | I2I | 6.236 s | 6.733 s | Eager |
| H200 | Base BF16, FA3 | T2I | 6.845 s | 6.916 s | Eager |
| H200 | Base BF16, FA3 | I2I | 14.183 s | 15.252 s | Eager |
| 2x H200, TP2 | Base BF16, FA3 | T2I | 4.000 s | 4.199 s | Eager |
| RTX 5090 | Distilled NVFP4, SDPA | T2I | 1.676 s | 1.678 s | Eager |
| RTX 5090 | Distilled NVFP4, SDPA | I2I | 4.092 s | 4.104 s | Eager |
| RTX 5090 | Base NVFP4, SDPA | T2I | 9.449 s | 9.583 s | Eager |

GB300 distilled NVFP4 T2I improves by 22.5% with FA4 + BCG. Peak allocated
memory increases from 16.63 to 17.46 GiB; peak reserved rises from 18.75 to
19.96 GiB. These are per-process allocator maxima across warmup/runtime,
not loading or total-device peaks. Other workloads also need headroom; memory
capacity alone is not a reason to enable graphs when they do not improve latency.

All 15 fixed-backend eager/BCG workload pairs passed repeated-request checks
and pixel-identical PNG comparisons across the two prompts. This is not an
official-reference accuracy guarantee or evidence that quantization preserves
BF16 output. GB300 measurements use Base revision `17333191c36bae1b61b0a247e33ba284f1df07fa`
and distilled `875076bf83d4364b68dccdbf8307f4f01475f68d`; H200/5090 use
`d8acde427a698bdb98b5c1c941c60fa4c97e9f28` and
`4d02a6a4dc4d5c7a0f3fa767187483b2b70f2384`. TP2 uses tensor-identical Base
DiT weights with a different safetensors header.

GB300 distilled NVFP4 T2I also ran at 0.424 s with SDPA + BCG (versus 0.589 s
SDPA eager). It is selectable but not the default backend: SDPA versus FA4
outputs had sample SSIM 0.978-0.987 and MAD 1.74-1.75/255. Neither path is
established as closer to the official output. Keep the backend fixed when
evaluating the graph speedup.

The native DiT's packed CFG forward has focused comparison tests against the
shared FLUX 3 transformer. Such component tests do not replace full-checkpoint
E2E or establish parity with every upstream release. Keep unverified hardware,
topologies and optional optimizations separate from measured deployment recipes.

The opt-in full-checkpoint HTTP regression covers concurrent prompts, image
editing, multiple outputs and a repeated single request:

```bash theme={null}
SGLANG_TEST_FLUX3_IMAGE=1 pytest -q \
  python/sglang/multimodal_gen/test/server/test_server_flux3_image.py
```

Set `SGLANG_TEST_FLUX3_IMAGE_MODEL` to a local package path when using an
offline checkpoint. This test needs checkpoint access; it does not substitute
for an official-reference golden-image comparison.

## 10. ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.

FLUX.3 has no integrated KSampler executor. Use server mode; the plugin-specific FLUX.3 workflow has not been validated.


# HY-Nomas
Source: https://docs.sglang.io/cookbook/diffusion/HY-Nomas/HY-Nomas

Serve the HY-Nomas event-conditioned video world model with SGLang-diffusion, as offline video jobs or a realtime WebSocket session.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) on Linux with NVIDIA CUDA.

One `sglang serve` process answers both HTTP video jobs and realtime WebSocket sessions. On first load, SGLang downloads the official checkpoints and assembles a local pipeline; see [Weights](#weights).

## 2. Model capabilities

[HY-Nomas](https://huggingface.co/tencent/Hy-Nomas) is Tencent Hunyuan's video world model. A video is described as a sequence of events: each event is a structured JSON caption with a frame count and optional camera poses. The model generates one event at a time while attending to the cached state of earlier events, so a scene can be extended or redirected without regenerating what came before.

Choose it for long, steerable scene continuation, either as one offline request with all events planned up front or as a live session that receives events one by one. It generates from noise: there is no image-to-video input. Captions must follow the official event JSON schema, which the official prompt enhancer (Nomas-PE) produces from free text; SGLang does not host that enhancer.

The code and weights are released under Apache-2.0. See the [GitHub repository](https://github.com/Tencent-Hunyuan/Hy-Nomas) for the event schema and training details.

| Component | Implementation |
| - | - |
| Text encoder | Qwen2.5-VL-7B-Instruct, 1280 tokens per event |
| DiT | Distilled segment-autoregressive DiT, 8 Euler steps, no CFG |
| Decoder | PixelUMM pixel decoder, windowed over 10 latents with 5-latent overlap |
| Camera | Optional normalized extrinsics, one matrix per latent-aligned frame |

## 3. Deployment

```bash Command theme={null}
sglang serve --model-path tencent/Hy-Nomas --num-gpus 1 --port 30000
```

For two GPUs, use Ulysses sequence parallelism. Its output matched the single-GPU output pixel for pixel:

```bash Command theme={null}
sglang serve --model-path tencent/Hy-Nomas --num-gpus 2 --sp-degree 2 --port 30000
```

Tensor parallelism (`--tp-size 2`) also runs, but its trained-weight output differs from one GPU, so prefer SP. `tp_size * sp_degree` must divide the 16 attention heads, and TP and SP can be combined. Ring attention, CFG parallelism and data parallelism are rejected. Both layouts shard only the DiT computation; the text encoder and decoder are not placed on a separate GPU.

To reduce DiT memory, stream its blocks from host memory:

```bash Command theme={null}
sglang serve --model-path tencent/Hy-Nomas --num-gpus 1 --dit-layerwise-offload true --port 30000
```

### Weights

`tencent/Hy-Nomas` is resolved through a pinned [model overlay](https://huggingface.co/Niehen6174/Hy-Nomas-overlay). It downloads the official distilled, non-quantized DiT, the PixelUMM decoder and the Qwen2.5-VL text encoder. It then caches a Diffusers-style pipeline under `SGLANG_DIFFUSION_CACHE_ROOT`. Weights keep their original values and dtypes. Later starts reuse the cache, and a local directory with the same layout can be passed to `--model-path` instead. The first load needs extra disk space for the decoder export.

The overlay does not include Nomas-PE, LoRA adapters, or FP8/INT8 variants.

## 4. Offline and HTTP generation

A request lists its events in `segments`. Each caption is a JSON string in the official event schema; later events usually describe only what changes. This two-event request makes 240 frames at 352×640 and saves it as `request.json`:

```python Python theme={null}
import json

first = {
    "Style": "Cinematic fantasy",
    "perspective_view": "Third-person view",
    "subject_list": [{"subject_id": "fox", "display_name": "a small orange fox",
                      "subject_attribute": "Orange fur, white chest, and a fluffy tail."}],
    "segments": [{
        "visual_description_long": "A small orange fox stands on a stone path in a luminous forest.",
        "contained_actions": [{"start_time": "00:00.000", "end_time": "00:05.000",
                               "action_description": "The fox walks slowly along the path."}],
    }],
}
second = {
    "segments": [{
        "description_change": "Glowing blue flowers open beside the path.",
        "contained_actions": [{"start_time": "00:00.000", "end_time": "00:05.000",
                               "action_description": "The fox passes the flowers and looks around."}],
    }],
}
request = {
    "prompt": json.dumps(first),
    "prompt_format": "json",
    "segments": [
        {"frames": 120, "caption": json.dumps(first)},
        {"frames": 120, "caption": json.dumps(second)},
    ],
    "num_frames": 240,
    "height": 352,
    "width": 640,
    "fps": 24,
    "num_inference_steps": 8,
    "guidance_scale": 1.0,
    "seed": 42,
}
with open("request.json", "w") as f:
    json.dump(request, f)
```

Send it to the HTTP video API, then poll `GET /v1/videos/{id}` and download from `GET /v1/videos/{id}/content`:

```bash Command theme={null}
curl -sS http://127.0.0.1:30000/v1/videos \
  -H 'Content-Type: application/json' \
  --data-binary @request.json
```

The same request runs without a server through the Python API:

```python Python theme={null}
from sglang.multimodal_gen import DiffGenerator

with DiffGenerator.from_pretrained(model_path="tencent/Hy-Nomas") as generator:
    result = generator.generate({**request, "save_output": True})
```

| Field | Meaning |
| - | - |
| `segments` | One entry per event: `frames` and a JSON `caption`, optionally `norm_extrinsics`. Event frames must sum to `num_frames`, and `prompt` repeats the first caption. |
| `prompt_format` | `json` applies the official event schema. `plain` accepts free text for smoke tests only and logs a warning. |
| `norm_extrinsics` | Camera matrices that are already normalized. The server does not normalize raw poses. |
| `num_inference_steps`, `guidance_scale` | The distilled checkpoint uses 8 steps and guidance 1. |
| `height`, `width` | Rounded to multiples of 32 as in the official code; 720×1280 becomes 704×1280. |

Event frame counts do not need to be multiples of four; the decoded video is cropped to each requested count. Sampling noise is always drawn on the CPU, so a fixed seed gives the same video through the Python API and HTTP.

## 5. Realtime session

Open `ws://HOST:PORT/v1/realtime_video/generate` and exchange MessagePack maps.

| Message | Direction | Content |
| - | - | - |
| `{"type": "init", "prompt": ..., "num_frames": ..., "size": "640x352", "num_inference_steps": 8, "condition_inputs": {"prompt_format": "json", "segments": [...]}}` | client | Starts the session. Without `segments`, `prompt` and `num_frames` form the first event. |
| `{"type": "event", "kind": "segment", "event_id": n, "payload": {"caption": ..., "frames": ..., "norm_extrinsics": ...}}` | client | Queues the next event. `norm_extrinsics` covers this event only. |
| `{"type": "event", "kind": "end"}` | client | No more events. The server sends the remaining frames and closes. |
| `frame_batch` / `frame_batch_header` | server | Raw RGB, WebP or JPEG frames for one chunk. A chunk may contain no frames. |

Each event is generated as soon as it arrives, against the session's cached history. The decoder streams in overlapping windows and sends a frame only when no later window can change it, so frames are never revised. Output therefore lags generation, and `end` releases the tail: a 120 + 120 frame session sends 84, 120 and then 36 frames.

The server keeps one session at a time and rejects a second connection. Disconnecting ends the session and frees its state; there is no resume. Because a session draws noise from a persistent bank, compare its frames with another session using the same seed, not with an offline request.

A minimal client sends the first event in `init`, the rest as `segment` events, then `end`, and collects raw RGB frames until the server closes the socket:

```python Python theme={null}
import msgspec
import numpy as np
import websocket

events = request["segments"]  # from the request above
ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate")
ws.send_binary(msgspec.msgpack.encode({
    "type": "init",
    "prompt": events[0]["caption"],
    "num_frames": events[0]["frames"],
    "size": "640x352",
    "fps": 24,
    "num_inference_steps": 8,
    "seed": 42,
    "realtime_output_format": "raw",
    "condition_inputs": {"prompt_format": "json", "segments": events[:1]},
}))
for event_id, event in enumerate(events[1:], start=1):
    ws.send_binary(msgspec.msgpack.encode(
        {"type": "event", "kind": "segment", "event_id": event_id, "payload": event}))
ws.send_binary(msgspec.msgpack.encode({"type": "event", "kind": "end"}))

frames = []
while True:
    try:
        raw = ws.recv()
    except websocket.WebSocketConnectionClosedException:
        break
    if not isinstance(raw, bytes):
        break
    message = msgspec.msgpack.decode(raw)
    if message.get("type") == "error":
        raise RuntimeError(message)
    if message.get("type") in ("frame_batch", "frame_batch_header"):
        # Large batches send their payload as the next binary message.
        payload = message["payload"] if message["type"] == "frame_batch" else ws.recv()
        if message["num_frames"]:
            shape = (message["num_frames"], message["height"], message["width"], message["channels"])
            frames.append(np.frombuffer(payload, dtype=np.uint8).reshape(shape))
video = np.concatenate(frames)  # 240 frames
```

To steer from free text, run Nomas-PE as a separate service. It turns the user's text and recent frames into the next event JSON, which the client then sends as the `caption`.

### KV cache budget

The session cache starts at 65,536 tokens and grows by 1.5× up to 32 GiB, or to free GPU memory minus 8 GiB, whichever is smaller. At that limit, the oldest event is evicted first, as in the official code. Setting `max_cache_segments` also limits the number of cached events. Positions keep counting across evictions. If a cache write fails after an eviction, the session must be restarted.

These limits are `NomasPipelineConfig` fields (`interactive_kv_cache_initial_tokens`, `interactive_kv_cache_growth_factor`, `max_cache_gib`, `cache_free_memory_reserve_gib`, `max_cache_segments`). At 1280×704, one 10-latent event uses about 4.5 GB of K/V on one GPU; TP and SP divide this across ranks.

## 6. Limitations

* No image-to-video, classifier-free guidance, or raw camera-pose normalization.
* Nomas-PE is not served by SGLang.
* One realtime session per server; Ring attention, KV CPU swap, INT8 KV and sliding-window history are not implemented.
* TP output is not yet equivalent to single-GPU output with trained weights.
* Validation was on H200 only.


# Ideogram 4
Source: https://docs.sglang.io/cookbook/diffusion/Ideogram/Ideogram4

Deploy Ideogram 4 with SGLang Diffusion for high-aesthetic text-to-image generation.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

Set `HF_TOKEN` with access to the gated Ideogram repositories. Install `bitsandbytes>=0.46.1` and `accelerate>=1.1.0` for NF4 and the shared NF4 text encoder.

> **Installation and prerequisites**
>
> * NVIDIA CUDA GPU.
> * SGLang installed with diffusion dependencies.
> * `bitsandbytes>=0.46.1` and `accelerate>=1.1.0` for the NF4 checkpoint and the fal variants' shared NF4 text encoder.
> * `HF_TOKEN` with access to the Ideogram 4 gated repositories.

## 2. Model capabilities

[Ideogram 4](https://huggingface.co/ideogram-ai/ideogram-4-nf4) is a text-to-image family focused on polished composition and typography-aware generation. SGLang supports the official NF4 and FP8 checkpoints, Comfy-Org's NVFP4 transformer, and fal's distilled Fast and Instant variants.

Choose NF4 when memory is the primary constraint, FP8/NVFP4 for supported accelerator deployments, Fast for a 20-step balance, and Instant for the 8-step latency target. The public repositories are gated, and the available releases are already quantized or distilled rather than a single full-precision baseline.

| Variant | Hugging Face model ID | Notes |
| - | - | - |
| NF4 | `ideogram-ai/ideogram-4-nf4` | Official bitsandbytes NF4 checkpoint. Use this path first for low-memory deployment. |
| FP8 | `ideogram-ai/ideogram-4-fp8` | Official FP8 checkpoint. |
| NVFP4 | `Comfy-Org/Ideogram-4` | Comfy-Org NVFP4 transformer weights. SGLang loads non-transformer components from `ideogram-ai/ideogram-4-fp8`. |
| Fast | `fal/ideogram-v4-fast` | 20-step, CFG-distilled, FP4-targeted floating checkpoint. Defaults to `V4_FAST_20`. |
| Instant | `fal/ideogram-v4-instant` | 8-step, CFG- and timestep-distilled BF16 checkpoint. Defaults to `V4_INSTANT_8`. |

The fal repositories contain only the transformer component. When either model ID is passed directly to `--model-path`, SGLang loads the text encoder, tokenizer, VAE, and scheduler from the `ideogram-ai/ideogram-4-nf4-diffusers` revision referenced by fal's model cards, then runs the distilled transformer without the unconditional branch.

## 3. Serve the model

NF4:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path ideogram-ai/ideogram-4-nf4 \
  --num-gpus 1 \
  --performance-mode auto \
  --port 30010
```

Intel Xeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path ideogram-ai/ideogram-4-nf4 --port 30010
```

FP8:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path ideogram-ai/ideogram-4-fp8 \
  --num-gpus 1 \
  --performance-mode auto \
  --port 30010
```

Intel Xeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path ideogram-ai/ideogram-4-fp8 --port 30010
```

Comfy-Org NVFP4:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path Comfy-Org/Ideogram-4 \
  --num-gpus 1 \
  --performance-mode auto \
  --port 30010
```

Use B200 or another Blackwell GPU for NVFP4.

fal Fast:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path fal/ideogram-v4-fast \
  --num-gpus 1 \
  --performance-mode auto \
  --port 30010
```

IntelXeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path fal/ideogram-v4-fast --port 30010
```

fal Instant:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path fal/ideogram-v4-instant \
  --num-gpus 1 \
  --performance-mode auto \
  --port 30010
```

Intel Xeon CPU deployment is supported for the fal checkpoints with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only flags.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu HF_TOKEN=$HF_TOKEN sglang serve --model-path fal/ideogram-v4-instant --port 30010
```

### Tensor and sequence parallelism

Both fal variants support native DiT tensor parallelism and Ulysses sequence parallelism. The following two-GPU layouts were validated at 1024×1024 on B200 GPUs for both `fal/ideogram-v4-fast` and `fal/ideogram-v4-instant`.

TP2 shards the distilled DiT weights. The shared bitsandbytes NF4 text encoder is replicated because 4-bit row-parallel quantization states cannot be safely sharded:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path fal/ideogram-v4-instant \
  --num-gpus 2 \
  --tp-size 2 \
  --port 30010
```

Ulysses/SP2 shards the image-token sequence while keeping the DiT weights replicated:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path fal/ideogram-v4-instant \
  --num-gpus 2 \
  --ulysses-degree 2 \
  --port 30010
```

Replace the model ID with `fal/ideogram-v4-fast` to use the same layouts for Fast. TP2 and Ulysses/SP2 are the validated configurations; Ring SP is not yet validated for this pipeline. Ideogram 4 has 18 attention heads, so any other TP degree must divide 18.

### Layerwise offload

If the distilled DiT does not fit in GPU memory, enable transformer layerwise offload. This keeps the shared NF4 text encoder on the GPU while streaming the DiT blocks from pinned CPU memory. It reduces peak VRAM at the cost of additional latency:

```bash Command theme={null}
HF_TOKEN=$HF_TOKEN sglang serve \
  --model-path fal/ideogram-v4-instant \
  --num-gpus 1 \
  --dit-layerwise-offload \
  --layerwise-offload-components transformer \
  --port 30010
```

The same option applies to Fast. Layerwise offload was validated with a real Instant generation; TP2 and Ulysses/SP2 were validated separately without offload.

The released Fast weights are stored in a floating-point pre-pack form and can be loaded directly, but fal trained them with QAD for an NVFP4 execution path. SGLang does not currently quantize dense NVIDIA DiTs to NVFP4 at load time, so direct Fast inference bypasses the intended quantization path and may be visibly degraded. Treat direct floating-point Fast inference as compatibility/testing support, not the production-quality path. The released Instant checkpoint is BF16 and is the recommended local checkpoint today.

## 4. Generate an image

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:30010/v1")

response = client.images.generate(
    model="ideogram-ai/ideogram-4-nf4",
    prompt="A cinematic poster of a quiet bookstore at dusk with elegant hand-lettered signage",
    size="1024x1024",
    n=1,
    response_format="b64_json",
    extra_body={"preset": "V4_QUALITY_48", "seed": 0},
)

image_bytes = base64.b64decode(response.data[0].b64_json)
with open("ideogram4.png", "wb") as f:
    f.write(image_bytes)
```

fal's hosted endpoint expands natural-language prompts, but these local checkpoints expect Ideogram 4's structured JSON caption format. Pass a complete caption as the API `prompt`, for example:

```python Example theme={null}
import json

prompt = json.dumps(
    {
        "high_level_description": (
            "A bold typographic poster centered on the exact words INSTANT BY FAL, "
            "printed in black and electric orange on warm white paper."
        ),
        "compositional_deconstruction": {
            "background": "Warm white textured paper with generous negative space.",
            "elements": [
                {
                    "type": "text",
                    "text": "INSTANT BY FAL",
                    "desc": "Large uppercase geometric sans-serif lettering, precisely centered.",
                }
            ],
        },
    },
    ensure_ascii=False,
    separators=(",", ":"),
)

response = client.images.generate(
    model="fal/ideogram-v4-instant",
    prompt=prompt,
    size="1024x1024",
    n=1,
    response_format="b64_json",
    extra_body={"seed": 42},
)
```

Base Ideogram 4 presets are `V4_DEFAULT_20`, `V4_QUALITY_48`, and `V4_TURBO_12`. The fal variants automatically select `V4_FAST_20` and `V4_INSTANT_8`, respectively. A preset controls both `num_inference_steps` and guidance, so do not set those fields directly.

You can also produce captions with a separate SRT LLM using
[prompt enhancement](/docs/sglang-diffusion/models_with_pe). Configure an
Ideogram-specific JSON caption template and schema; the generic prose template
is not appropriate for these checkpoints. This does not reproduce fal's
checkpoint-specific prompt expander or establish equivalent image quality.

## 5. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# JoyAI-Echo
Source: https://docs.sglang.io/cookbook/diffusion/JoyEcho/JoyEcho

Run JoyAI-Echo multi-shot audio–video generation with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

Prepare the [pinned Echo 1.0 checkpoint](#3-1-prepare-the-echo-1-0-checkpoint) and set `JOY_ECHO_MODEL_PATH` before starting the server.

## 2. Model capabilities

[JoyAI-Echo](https://huggingface.co/jdopensource/JoyAI-Echo) is an 8-step long-form audio-video model built on LTX-2. Its paired memory bank carries decoded visual context and audio latents across prompt changes, making it strongest for multi-shot, minute-scale sequences that need continuity in both picture and soundtrack.

Choose JoyEcho over a standard LTX pipeline when shots must share audiovisual memory. Its distilled 832×480 path prioritizes long-form continuity and throughput rather than the higher-resolution two-stage quality modes offered by LTX-2.3.

SGLang materializes the Echo 1.0 monolithic release through the built-in [JoyAI-Echo overlay](https://huggingface.co/Niehen6174/JoyAI-Echo-overlay). Prepare the pinned checkpoint below before running the examples.

| Aspect | Standard LTX-2.3 | JoyEcho |
| - | - | - |
| Pipeline | `LTX2Pipeline` / `LTX2TwoStageHQPipeline` | `JoyEchoPipeline` (default for this model) |
| Denoising | Multi-step flow matching + CFG | LTX-2 DMD distilled path (8 steps, `guidance_scale=1.0`) |
| Multi-shot | Not supported | Paired audio–video memory bank across shots |
| Sequence parallelism | LTX-2 SP (video/audio sharded) | Ulysses SP (`ulysses_degree=2`): single-shot and multi-shot + memory bank |
| Post-processing | Optional two-stage HQ upscaling | Per-shot mp4 output |

> [!WARNING]
>
> Review the model license on the [JoyAI-Echo Hugging Face page](https://huggingface.co/jdopensource/JoyAI-Echo) before production or commercial use. SGLang support does not grant additional model usage rights.

## 3. Model Deployment

### 3.1 Prepare the Echo 1.0 checkpoint

The native overlay requires `JoyAI-Echo-release.safetensors`. The upstream repository's Echo 1.5 revision does not contain that file. Download the [Echo 1.0 revision](https://huggingface.co/jdopensource/JoyAI-Echo/tree/4187f9a53c6eff3a76c51e79bd27f70d10f7591b) into the Hugging Face cache:

```bash theme={null}
JOY_ECHO_MODEL_PATH=$(python - <<'PY'
from huggingface_hub import snapshot_download

print(snapshot_download(
    repo_id="jdopensource/JoyAI-Echo",
    revision="4187f9a53c6eff3a76c51e79bd27f70d10f7591b",
    allow_patterns=["JoyAI-Echo-release.safetensors", "*.json", "*.md", "LICENSE"],
))
PY
)
```

Use the returned cache path as `--model-path` and keep `--model-id jdopensource/JoyAI-Echo` when using local weights. The model ID lets BCG select JoyEcho's support policy. Initial startup also downloads and materializes the overlay and its text encoder dependencies.

### 3.2 Serve the model

JoyEcho uses the default `JoyEchoPipeline` registered for `jdopensource/JoyAI-Echo`. A single high-VRAM GPU (for example H100 or H200) is enough for the common 832x480 / 121-frame / 8-step setting.

```bash theme={null}
sglang serve \
  --model-path "$JOY_ECHO_MODEL_PATH" \
  --model-id jdopensource/JoyAI-Echo
```

Optional environment variable for long runs:

```bash theme={null}
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
```

For multi-GPU serving, tensor parallelism (TP) and **Ulysses sequence parallelism (SP)** are supported. JoyEcho SP uses an **asymmetric layout**: video target latents are time-sharded across ranks, while audio (including memory tokens) is **replicated** on every rank so cross-attention stays temporally aligned. Multi-shot runs with `enable_memory_bank=true` are supported on SP.

```bash theme={null}
sglang serve \
  --model-path "$JOY_ECHO_MODEL_PATH" \
  --model-id jdopensource/JoyAI-Echo \
  --num-gpus 2 \
  --ulysses-degree 2
```

> [!NOTE]
>
> JoyEcho SP currently targets **Ulysses-only** parallelism (`ulysses_degree=2`, `ring_degree=1`). Ring SP is not validated for this pipeline. For `sglang generate`, add `--num-gpus 2 --ulysses-degree 2` to the commands in section 4.

## 4. Model Invocation

### 4.1 Default sampling

| Setting | Default |
| - | - |
| Resolution | 832x480 |
| Frames | 121 |
| FPS | 25 |
| Steps | 8 |
| Guidance scale | 1.0 |
| Seed | 12345 |

### 4.2 Single-shot text-to-video

```bash theme={null}
sglang generate \
  --model-path "$JOY_ECHO_MODEL_PATH" \
  --model-id jdopensource/JoyAI-Echo \
  --prompt "A curious raccoon walks through a sunlit forest path" \
  --height 480 --width 832 --num-frames 121 --fps 25 \
  --num-inference-steps 8 --seed 42 \
  --save-output
```

Disable the memory bank for standalone clips with a config file:

```bash theme={null}
cat > /tmp/joy_echo_single.json <<'EOF'
{
  "prompt": "A curious raccoon walks through a sunlit forest path",
  "enable_memory_bank": false,
  "seed": 42,
  "height": 480,
  "width": 832,
  "num_frames": 121,
  "fps": 25,
  "num_inference_steps": 8
}
EOF

sglang generate --config /tmp/joy_echo_single.json \
  --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \
  --save-output
```

### 4.3 Multi-shot generation

JoyEcho does **not** generate all shots in one forward pass. Each shot is one generation request. Continuity is carried by an in-process **memory bank** on the pipeline instance.

Typical workflow:

1. **Shot 0** — memory bank is empty; the model generates a standalone A/V clip.
2. **After decode** — decoded video frames and packed audio latents are committed to the memory bank (up to 7 slots by default).
3. **Shot 1+** — prior-shot frames are re-encoded and prepended as a memory prefix before denoising.
4. **Per-shot seeding** — official semantics use `prompt_seed = base_seed + shot_index`.

Pass multiple prompts as a list in a config file:

```bash theme={null}
cat > /tmp/joy_echo_4shot.json <<'EOF'
{
  "prompt": [
    "Shot 0: A raccoon wakes up in a cozy attic.",
    "Shot 1: The raccoon climbs down and opens the back door.",
    "Shot 2: It walks through a rainy alley under neon signs.",
    "Shot 3: The raccoon finds a warm bakery window and stops."
  ],
  "enable_memory_bank": true,
  "reset_memory_bank": true,
  "seed": 42,
  "height": 480,
  "width": 832,
  "num_frames": 121,
  "fps": 25,
  "num_inference_steps": 8
}
EOF

sglang generate --config /tmp/joy_echo_4shot.json \
  --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \
  --save-output
```

You can also pass prompts from a text file (one prompt per line) with `--prompt-path`:

```bash theme={null}
sglang generate \
  --model-path "$JOY_ECHO_MODEL_PATH" \
  --model-id jdopensource/JoyAI-Echo \
  --prompt-path /tmp/joy_echo_shots.txt \
  --seed 42 \
  --height 480 --width 832 --num-frames 121 --fps 25 \
  --num-inference-steps 8 \
  --save-output
```

### 4.4 Memory bank controls

| Parameter | Default | Meaning |
| - | - | - |
| `enable_memory_bank` | `true` | Read/write the paired A/V memory bank between shots. |
| `reset_memory_bank` | `true` | Clear the bank and shot counter at the start of a new session (`request_id` change or first shot). |

Set `enable_memory_bank=false` when you want independent shots without cross-shot continuity.

### 4.5 Measured two-H200 single-shot configuration

For short independent clips, keep the text and audio/video components on GPU with `--component-residency=all=resident`. Full-stage profiles showed that this removes repeated host-to-device weight copies between component uses.

The following configuration was measured on two H200s with Ulysses degree 2, TP1, PyTorch 2.11.0+cu130, 640x384, 33 frames, 8 steps and seed 42. It disables compilation and the memory bank:

```bash theme={null}
cat > joy_echo_h200.json <<'EOF'
{"enable_memory_bank": false}
EOF

CUDA_VISIBLE_DEVICES=0,1 sglang generate \
  --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \
  --config joy_echo_h200.json --prompt "A curious raccoon" \
  --width 640 --height 384 --num-frames 33 --num-inference-steps 8 --seed 42 \
  --num-gpus 2 --ulysses-degree 2 \
  --performance-mode manual --enable-torch-compile=false --quality exact \
  --component-residency=all=resident --warmup-mode request \
  --save-output --perf-dump-path joy_echo_h200.json.perf
```

For BCG, add `--enable-breakable-cuda-graph --warmup-resolutions 640x384 --warmup-num-frames 33`. Check for successful `[Diffusion BCG] captured` logs and absence of request signature misses. Keep the same model ID, resolution, frame count and quality as warmup.

Two fresh-process saved requests per configuration, after request warmup:

| Lossless mode | Auto residency E2E | All resident E2E | Reduction | Peak reserved per rank, auto → resident |
| - | -: | -: | -: | -: |
| Eager | 2.563 s | 2.366 s | 7.65% | 46.43 → 69.32 GiB |
| BCG | 1.229 s | 1.058 s | 13.88% | 48.77–48.88 → 69.32 GiB |

Loading, warmup and profiling are excluded from these timings. The paired eager profiles remove 71 pinned host-to-device copies (13.07 GB, 278.86 ms on the profiled rank), with the same 77,045 kernel launches. All eight lossless outputs have pixel-identical video frames; audio differences are comparable to baseline repeat variability. These results cover this compact single-shot workload, rather than the default 121-frame or multi-shot memory-bank workload.

Use `quality=lossless` for this recipe. High-mode output did not pass the separate quality comparison, and high + BCG is rejected by the runtime. If a larger request exceeds available memory, return to auto residency or keep only selected components resident.

## 5. Practical Tips

* Use `--num-inference-steps 8` and `--guidance-scale 1.0` to match the official JoyEcho DMD distilled path.
* Multi-shot prompts can be passed as a `prompt` list, via `prompt_path`, or as sequential API calls on the same server instance.
* The memory bank caps at **7 slots**; from shot 8 onward the oldest slots roll off.
* For **2-GPU latency**, try **Ulysses SP** (`--num-gpus 2 --ulysses-degree 2`) on both single-shot and multi-shot runs. Use **TP** when you need a different sharding strategy or more than two GPUs.
* Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for long multi-shot SP sessions.
* JoyEcho outputs per-shot mp4 files with synchronized audio. There is no built-in two-stage HQ upscaling path like LTX-2.3 HQ.

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# Kandinsky6
Source: https://docs.sglang.io/cookbook/diffusion/Kandinsky/Kandinsky6

Deploy Kandinsky 6.0 for joint text- and image-conditioned video and audio generation.

## 1. Quick start

Use Linux with NVIDIA CUDA. This model is not yet in a released package;
use a checkout containing this implementation, not an unmodified release image.
Your Hugging Face account must have access to the checkpoint; authenticate with
`hf auth login` or supply `HF_TOKEN` in the server environment. Install from the repo root:

```bash theme={null}
uv pip install -e "python[diffusion]" --prerelease=allow
```

Start with Pro-distill and resident weights on B200, GB300, or RTX PRO 6000 (96 GB).
The picker separates
verified lightweight HTTP requests from full-size offline generation and
unverified custom topologies. See the [installation guide](/docs/sglang-diffusion/installation)
for platform setup.

The request returns a video job ID, not the video bytes. Poll
`GET /v1/videos/{id}` until `status` is `completed`, then download
`GET /v1/videos/{id}/content` for the MP4 containing both video and audio.
Treat `failed` as an error; delete the job with `DELETE /v1/videos/{id}` when finished.
See [Download a completed video](#5-download-a-completed-video) for a complete polling example.

## 2. Model capabilities

Kandinsky 6.0 generates video and audio jointly from a text prompt, optionally
conditioned on a reference image. One DiT predicts both modalities at each
denoising step. A Reason1 encoder derived from Qwen2.5-VL supplies token features,
and CLIP supplies pooled text features. Both request modes use the same server;
image conditioning does not require switching checkpoints.

Choose Pro-distill for the 10-step path, or Pro for 50-step classifier-free guidance.
The recipes cover 768 x 512, 121-frame clips at 24 fps. They use native SGLang
components, without calling a third-party model pipeline. Validation compares
against the initial SGLang implementation; it does not establish parity with
the unpublished Kandinsky6 Diffusers implementation.

| Checkpoint | Scheduler | Default steps | Default guidance |
| - | - | -: | -: |
| `kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers` | PiFlow | 10 | 1.0 |
| `kandinskylab/Kandinsky-6.0-Pro-5s-Diffusers` | Flow-match Euler | 50 | 5.0 |

Pro-distill only accepts guidance 1.0 and does not encode a negative prompt.
The scheduler is loaded from the checkpoint. For a local directory with an
unrelated name, specify `--model-id Kandinsky-6.0-Pro-distill-5s-Diffusers` to
select its sampling defaults, or explicitly pass 10 steps and guidance 1.0.

## 3. Offline generation

The default request produces a muxed video and audio file:

```bash theme={null}
sglang generate \
  --model-path kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers \
  --performance-mode speed --attention-backend fa --warmup-mode off \
  --prompt "A woman chops vegetables in a sunlit kitchen, with soft jazz playing." \
  --seed 42 --save-output
```

Add `--image-path /path/to/reference.png` for image conditioning.
On RTX PRO 6000, use `--attention-backend torch_sdpa` in the CLI or
`attention_backend="torch_sdpa"` in Python.
For two B200 GPUs, add `--num-gpus 2 --tp-size 1 --ulysses-degree 2 --ring-degree 1`.
To use Pro, change the model path; its defaults select 50 steps and guidance 5.0.

### Python API

```python theme={null}
from sglang.multimodal_gen import DiffGenerator

generator = DiffGenerator.from_pretrained(
    model_path="kandinskylab/Kandinsky-6.0-Pro-distill-5s-Diffusers",
    num_gpus=1,
    performance_mode="speed",
    attention_backend="fa",
    warmup_mode="off",
)
try:
    result = generator.generate(sampling_params_kwargs={
        "prompt": "A woman chops vegetables in a sunlit kitchen, with soft jazz playing.",
        "seed": 42,
        "save_output": True,
        "output_path": "kandinsky6_samples",
        "return_file_paths_only": False,
        "return_frames": True,
    })
    print(result.output_file_path)
finally:
    generator.shutdown()
```

`result.frames` contains RGB frames and `result.audio` contains the decoded waveform.
Add `image_path` to the sampling parameters for a reference image. Omit the two
return-data flags when only the output file is needed.

## 4. Parallelism and memory

* **Ulysses**: full-size Pro-distill generation on two B200 or four GB300 GPUs
  matches single-GPU output on the same hardware, including every decoded RGB
  frame and the audio waveform.
  It shards the video stream; audio and text streams remain replicated. Video K/V
  is gathered for audio-to-video attention.
* **CFG parallel**: Pro can distribute its conditional and unconditional branches
  with `--num-gpus 2 --ulysses-degree 1 --enable-cfg-parallel`.
  This is a different topology from Ulysses. It does not accelerate Pro-distill,
  which has no guidance branch.
* **TP and Ring**: implemented, but full-model outputs differ from the single-GPU
  baseline. Do not treat them as verified bitwise-equivalent modes.
  Ring2, Ring4, and TP2 x Ring2 complete full-size Pro-distill generation on
  GB300 and repeat exactly within each topology. Ring2 serving also covers
  image conditioning, Cache-DiT, and dynamic/merged LoRA restoration. Use
  `--num-gpus 2 --ulysses-degree 1 --ring-degree 2` to select Ring2; validate
  both video and audio quality before deployment.
* **TP x SP**: four-GB300 TP2 x Ulysses2 full generation is tested. It reduces
  weight memory but was slower than Ulysses4 and produced different video and
  audio. Prefer Ulysses when memory permits; validate quality before deploying TP.
* **FSDP + Ulysses**: add `--use-fsdp-inference true` to shard DiT weights
  without sharding linear reductions. Four-GB300 Ulysses4 generation matches
  resident Ulysses4 in every decoded RGB frame and the audio waveform. This is
  a lower-memory alternative with additional weight-gather latency, not a
  faster default. Request-level Cache-DiT switching is also tested with FSDP.
* **Encoder parallelism**: `--encoder-parallel fold` shards Reason1 and CLIP
  across the DiT replica, including a Ulysses-only replica. Four-GPU generation
  is tested, but folding changes reductions and the generated video and audio.
  Keep `auto` for the baseline recipes and validate quality before forcing
  folding. See [Encoder parallelism](/docs/sglang-diffusion/encoder_parallel).
* **Offload**: keep components resident when memory allows. Use
  `--cpu-offload-components transformer text_encoder vae` for whole-component
  offload, or `--layerwise-offload-components all` for layerwise offload.
  Layerwise offload includes both DiT text towers and the audio VAE's encoder,
  decoder, and vocoder blocks. Use component-specific residency settings to
  keep selected components resident.
  Both require host RAM and add transfers; lower device memory does not imply
  lower total memory or lower latency.

The picker's **Server** tab exposes attention, weight placement and offload
components. With DiT layerwise offload, you can also tune prefetch depth and
resident layers using `--dit-offload-prefetch-size` and
`--dit-layerwise-resident-layers`. Increasing either uses more VRAM; these
non-default settings are unverified tuning choices, not faster recommendations.
Component offload moves whole components; layerwise offload streams supported
blocks. They are alternatives to FSDP in this picker, not additive presets.

### Deployment limits

Use monolithic serving. Encoder/denoiser/decoder disaggregation is not supported.
Breakable CUDA graphs are also not enabled for Kandinsky6: the server disables
`--enable-breakable-cuda-graph`. Its variable-length text path needs additional
handling to preserve the unpadded computation; same-shape graph experiments do
not establish support for arbitrary requests. Keep eager execution for the
recipes on this page.

### Choosing a topology

Pro-distill, batch 1, 768 x 512, 121 frames, 10 steps, BF16/FA, no offload or
compilation, on GB300 with 277.5 GiB per GPU:

| Topology | Hot E2E (s) | Output-rank peak reserved (GiB) | Matches single-GPU RGB/audio |
| - | -: | -: | - |
| Single GPU | 33.95 | 80.36 | Reference |
| Ulysses4 | 11.33 | 80.91 | Exact |
| Ulysses4 + FSDP | 12.13 | 42.54 | Exact |
| TP2 x Ulysses2 | 13.99 | 50.97 | No |
| TP4 | 17.72 | 35.91 | No |

Times are medians of three hot requests after one initial request, with startup
warmup disabled and default conditioning caching enabled. They include MP4
saving but exclude model loading. Memory is PyTorch runtime peak reserved on
the output rank, not loading peak or the maximum across all GPUs. These results
do not establish output quality for TP; TP also changes the text encoder's
numerical path.

For this workload, FSDP reduces measured memory by 47% versus resident Ulysses4
at a 7% latency cost. Add it when weight memory is the constraint; keep ordinary
Ulysses when latency is the priority.

### RTX PRO 6000 and SageAttention3

Pro-distill also runs with resident weights on one 96 GB RTX PRO 6000 Blackwell
Server Edition. Use `--attention-backend torch_sdpa` on this SM120 device:
the current platform resolves `fa` to SDPA, so requesting `fa` does not establish
a FlashAttention benchmark.

With [SageAttention3](https://github.com/thu-ml/SageAttention/tree/d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5/sageattention3_blackwell)
installed, `--component-attention-backends transformer=sage_attn_3` selects it
only for the DiT, keeping the encoders on SDPA. The tested upstream revision
requires SM120/121 and rejects B200 SM100. With PyTorch 2.14, its build required
`NVCC_APPEND_FLAGS=-std=c++20`; no kernel source or hardware guard was modified.

Same full-size workload and timing method as above, on one RTX PRO 6000 with
driver 595.84, PyTorch 2.14.1+cu130, Transformers 5.17.0, and Diffusers 0.37.0:

| DiT backend | Hot E2E (s) | Runtime peak reserved (GiB) |
| - | -: | -: |
| SDPA | 133.91 | 80.34 |
| SageAttention3 | 105.33 | 82.05 |

Sage3 reduced latency by 21.3% but changed the outputs substantially: all-frame
PSNR 11.97 dB, mean SSIM 0.375, and raw-audio waveform SNR -2.21 dB versus SDPA.
These are paired differences, not perceptual quality scores. Each backend was
repeatable across four requests, but Sage3 is **not lossless** and is not the
default recipe. Validate both video and audio before choosing it.

### VAE decode

The video VAE defaults to tiled decoding, including multi-GPU execution. Its
decode group is separate from the DiT's TP and CFG collectives. Explicit
`--vae-config.parallel-decode-mode spatial_shard` uses whole-clip decoding with
frame-causal attention computed without a quadratic whole-video mask.

Whole-clip decoding still needs substantial activation memory. A 768 x 512,
121-frame run on two B200 GPUs completed with the transformer, both text encoders,
and audio VAE offloaded, but reached 170.4 GiB output-rank peak reserved. This is
not a lower-memory replacement for tiled decode. Tiled and whole-clip decoding
use different normalization and attention contexts and are not interchangeable
numerical baselines; keep tiled decoding for the validated default output.

The audio VAE contains both the mel decoder and vocoder. Current checkpoints
include a complete vocoder copy inside `audio_vae`, so this pipeline does not
also load the redundant standalone `vocoder` component.

CUDA matrix RoPE uses a fused kernel that preserves the eager FP32 rounding
boundaries. CPU and autograd paths retain the eager implementation. No sampling
parameter or sparse-attention approximation is introduced.
The checkpoint's 128-wide Q/K normalization uses native PyTorch FP32 fusion
during CUDA inference without changing the stored parameter dtype or offload
policy. Other head widths retain the original normalization path.

### Cache-DiT

Cache-DiT is opt-in per request through `enable_cache_dit` and `cache_dit_params`
in the Python sampling parameters or the JSON `/v1/videos` body. Set
`enable_cache_dit: false` to restore uncached execution without restarting.
See [Cache-DiT configuration](/docs/sglang-diffusion/cache_dit)
for the available knobs.

The **Request** tab offers an opt-in Cache-DiT switch for both text and image
conditioning. It uses the runtime cache defaults, not a quality-calibrated
preset. A short request may finish before cache warmup ends and gain no speedup.

The adapter caches both evolving video and audio streams together. Its dynamic
decision uses the video residual, not an independent audio-quality metric.
Skipping blocks is approximate and can change both outputs substantially;
validate video and audio on your own prompts before enabling it. It is not part
of the lossless defaults, and should not be combined with breakable CUDA graphs.

### LoRA adapters

Use adapters trained for the same Kandinsky6 checkpoint architecture. The native
loader accepts PEFT A/B weights with Diffusers or SGLang DiT layer names, including
the `transformer.` prefix. Both video and audio branches belong to `transformer`;
an adapter can change both outputs.

Add `--lora-path /path/to/adapter.safetensors` to load an adapter at startup.
To switch an existing server to dynamic LoRA without restarting:

```bash theme={null}
curl --fail-with-body http://localhost:30000/v1/set_lora \
  -H "Content-Type: application/json" \
  -d '{
    "lora_nickname": "kandinsky6",
    "lora_path": "/path/to/adapter.safetensors",
    "target": "transformer",
    "strength": 1.0,
    "merge_mode": "dynamic"
  }'
```

The adapter path must be readable by the server. `dynamic` applies the low-rank
update during inference; `merge` updates the base weights before inference.
These modes may differ numerically. To disable either mode and restore the base
model:

```bash theme={null}
curl --fail-with-body http://localhost:30000/v1/unmerge_lora_weights \
  -H "Content-Type: application/json" -d '{"target": "transformer"}'
```

Adapter selection is server-wide, not isolated per request. Coordinate changes
with other clients. Merging offloaded weights retains CPU snapshots of the
modified layers for exact restoration; budget for this host memory.
Validation uses synthetic adapters to check loading,
strength changes, repeatability, and exact restoration of video and audio;
it does not establish the quality of a community adapter.

See the [model inventory](/docs/sglang-diffusion/compatibility_matrix) and the
[super-resolution recipe](/cookbook/diffusion/Kandinsky/Kandinsky6-SR) for the
separate video SR pipeline.

## 5. Download a completed video

Use the `id` returned by the picker's Request command. The same endpoints serve
joint generation and SR. Set the base URL to match your server port:

```python Download theme={null}
import json
import shutil
import time
import urllib.request

base_url = "http://localhost:30000"
video_id = "REPLACE_WITH_RETURNED_ID"
deadline = time.monotonic() + 1800

while time.monotonic() < deadline:
    with urllib.request.urlopen(f"{base_url}/v1/videos/{video_id}", timeout=30) as response:
        job = json.load(response)
    if job["status"] == "failed":
        raise RuntimeError(job)
    if job["status"] == "completed":
        with urllib.request.urlopen(
            f"{base_url}/v1/videos/{video_id}/content", timeout=120
        ) as response, open("output.mp4", "wb") as output:
            shutil.copyfileobj(response, output)
        break
    time.sleep(2)
else:
    raise TimeoutError("Video generation did not finish within 30 minutes")
```

After downloading, remove the server-side job when no longer needed:

```bash Cleanup theme={null}
curl --fail-with-body -X DELETE http://localhost:30000/v1/videos/REPLACE_WITH_RETURNED_ID
```

## 6. ComfyUI

There is no Kandinsky6 integrated DiT executor in the
[ComfyUI plugin](/docs/sglang-diffusion/comfyui). The generic server video node
has not been validated for this model's joint audio/video output. Use the HTTP
or offline recipes above until that integration is verified.


# Kandinsky6 SR
Source: https://docs.sglang.io/cookbook/diffusion/Kandinsky/Kandinsky6-SR

Deploy Kandinsky 6 video super-resolution with x2, x4 or x2.25 upscaling, source audio and explicit decode speed, memory and numerical tradeoffs.

## 1. Quick start

Use Linux with NVIDIA CUDA and a checkout containing this implementation; it is
not yet in a released package or unmodified release image. Authenticate an account
with checkpoint access using `hf auth login`, or supply `HF_TOKEN` to the server.
From the repo root:

```bash Install theme={null}
uv pip install -e "python[diffusion]" --prerelease=allow
```

The picker uses the distilled two-step checkpoint. Four-GB300 whole-tile decode
is the output-preserving choice measured on multi-tile clips; one GPU suffices
for a functional check when it has enough memory. The picker also includes
explicitly **extrapolated** Hopper and RTX 5090 starting points, not measured
speed or memory guarantees. HTTP verification is scoped separately from offline
measurements. See the [installation guide](/docs/sglang-diffusion/installation)
for platform setup.

The request uploads a local video and returns a job `id`. Poll
`GET /v1/videos/{id}`, then download `GET /v1/videos/{id}/content` when completed.
Use the shared [polling and download example](/cookbook/diffusion/Kandinsky/Kandinsky6#5-download-a-completed-video).
For JSON requests, `video_path` refers to a file on the **server**, not the client.

## 2. Model capabilities

Kandinsky 6 VSR upscales a source video without a prompt. It encodes the source
with a causal KVAE, denoises overlapping tiles and blends their decoded outputs.
Unlike the [generation model](/cookbook/diffusion/Kandinsky/Kandinsky6), it retains
the source soundtrack rather than generating audio. Audio is converted to mono
44.1 kHz and trimmed to the processed video duration.

Use `kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers` for x2, x4 or
x2.25 upscaling of up to 121 frames. Its checkpoint-selected PiFlow scheduler
uses two DiT calls per tile; changing `num_inference_steps` does not increase
quality or the number of evaluations. The scheduler and transformer must come
from the same Diffusers checkpoint; sampler settings are read from `scheduler/`.
The native path uses dense attention,
not the checkpoint's requested NABLA sparse attention, so this is not a claim
of numerical parity with the sparse reference.

> [!WARNING]
>
> The tested non-distilled `Kandinsky-6.0-VSR-5s-Diffusers` revision
> `d6ddfabc62b3920b6da6487e4f444d37c591d5f4` declares a flow-Euler scheduler but
> contains a 640-wide output head instead of the required 64. It is not supported
> as-is. Do not change the scheduler to bypass this error; use the distilled model.

## 3. Offline usage

No prompt is required. A single-GPU example:

```bash Generate theme={null}
sglang generate \
  --model-path kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers \
  --video-path /path/to/input.mp4 \
  --sr-resolution-scale 2 \
  --num-inference-steps 2 \
  --seed 42 --save-output --output-path ./sr_outputs
```

> **Python API**
>
> ```python Generate theme={null}
> from sglang.multimodal_gen import DiffGenerator
>
> generator = DiffGenerator.from_pretrained(
>     model_path="kandinskylab/Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers",
>     num_gpus=1,
> )
> try:
>     result = generator.generate(sampling_params_kwargs={
>         "video_path": "/path/to/input.mp4",
>         "sr_resolution_scale": 2,
>         "num_inference_steps": 2,
>         "seed": 42,
>         "output_path": "sr_outputs",
>         "save_output": True,
>     })
>     print(result.output_file_path)
> finally:
>     generator.shutdown()
> ```

### Request parameters

* `sr_resolution_scale`: `2`, `4` or `2.25` (default). The x2.25 path first
  enlarges pixels x1.125, then processes x2 tiles.
* `sr_tiles_batch_size`: default `1`. More tiles per DiT call need more activation
  memory. Each chunk is seeded by its first tile index, so changing this value
  changes the noise and output. It is **not a lossless batching optimization**.
* `sr_tile_min_overlap`: default `0.2`. Changing overlap changes tile geometry
  and blending, not just performance.
* `sr_target_resolution`: optional `hd`, `fullhd`, `2k` or `WxH`; only downscales
  the generated result. `sr_target_resize_mode` is `fit` (default) or `exact`.
* `seed`: default `42`. Compare identical seeds, tile batching, source frames and
  attention policy when evaluating a decode optimization.

The input must fit at least one tile. A tested landscape source is 384 x 256.
The default filename is `<input name>_sr<scale>x_<timestamp>.mp4`; override it
with `--output-file-name`.

## 4. Decode speed, memory and numerics

These are **VAE decode choices**, independent of DiT TP/Ulysses/Ring:

<table>
  <thead><tr><th>Decode</th><th>Hot E2E</th><th>Peak reserved</th><th>Output versus serial</th></tr></thead>

  <tbody>
    <tr><td>Serial</td><td>45.26 s</td><td>47.55 GiB</td><td>Reference</td></tr>
    <tr><td>Whole-tile parallel</td><td>21.73 s</td><td>47.55 GiB</td><td>RGB and audio exact in this test</td></tr>
    <tr><td>Spatial shard</td><td>18.01 s</td><td>24.37 GiB</td><td>RGB differs; audio exact</td></tr>
  </tbody>
</table>

Setup: four GB300 GPUs (277.5 GiB each, NV18 links), host RAM 955.7 GiB with an
860 GiB cgroup limit; BF16/FA, Ulysses4, resident weights, no compile or startup
warmup. PyTorch 2.14.1+cu130, Transformers 5.17.0, Diffusers 0.37.0. Distilled
checkpoint revision `3e3e10bcd23659eeeb23cac740258ee4fc14aa7d`; seed 42, tile
batch 1, two steps per tile. A generated kitchen clip, 768 x 512 x 121 frames at
24 fps, was upscaled x2 to 1536 x 1024. Times are medians of three hot requests
after one initial request, including MP4 saving and excluding model loading.
Memory is PyTorch request peak reserved on the output rank, not loading, NVML or
the maximum across ranks. Host memory is not claimed unchanged.

**Whole-tile parallel** assigns complete existing tiles to ranks and preserves
their temporal caches and blend order. It matched serial RGB and raw audio
exactly on the tested workloads **within the same topology**. Select
`--vae-config.use-parallel-tiling true`. A single tile falls back to serial.
Communication can outweigh savings for small clips or slow interconnects.

**Spatial shard is not bitwise lossless.** It splits each tile's height, using
halo exchanges and the original temporal cache. Changed convolution shapes
change rounding. In the experiment above, all-frame RGB PSNR was **57.51 dB**,
minimum SSIM over the first/middle/last frames **0.99921**, mean absolute
difference **0.1132/255**, and maximum difference **4/255**. Audio was exact.
These measurements are not a guarantee for other videos or perceptual-quality
certification. Select it only after validating the output for your workload:

```bash Spatial decode flags theme={null}
--vae-config.use-parallel-decode true --vae-config.parallel-decode-mode spatial_shard
```

Both modes are opt-in runtime features. The picker selects whole-tile parallel
for its multi-tile recipe; it does not change runtime defaults. If both flags
are enabled, spatial sharding takes precedence. One GPU or a tile too short to
split retains replicated decoding. Neither mode changes the tile geometry.

### RTX PRO 6000 over PCIe

| GPUs / placement | Decode | Hot E2E | Sampled GPU peak |
| - | - | - | - |
| 1 / resident | Serial | 134.09 s | 48.91 GiB |
| 2 / resident, Ulysses2 | Serial | 122.11 s | 48.81 GiB |
| 2 / resident, Ulysses2 | Whole-tile | 83.04 s | 48.81 GiB |
| 2 / resident, Ulysses2 | Spatial | 73.62 s | 37.45 GiB |
| 1 / all-layerwise offload | Serial | 133.60 s | 37.43 GiB |

Setup: two RTX PRO 6000 Blackwell Server Edition GPUs (96GB nominal), PCIe/SYS
across CPU sockets, **no NVLink**; 499.2 GiB host RAM, no cgroup cap. Driver
595.84, the same PyTorch/Transformers/Diffusers versions and checkpoint as above,
but **SDPA**, not FA. A separate generated 768 x 512 x 121-frame clip at 24 fps
was upscaled x2 with seed 42, tile batch 1 and two steps per tile. Each row uses
one initial request and three hot requests, no compile/startup warmup. E2E
includes MP4 saving and raw-frame return, excludes model loading. GPU memory is
the maximum **per-GPU NVML sample across ranks**, sampled every 0.5 seconds during
requests, not the sum or PyTorch reserved memory; brief peaks may be missed.
These are within-machine comparisons, not GB300-versus-RTX speed ratios.

Ulysses2 alone reduced latency by only 8.9%. Whole-tile decode reduced it by
38.1% versus one GPU, with **exact RGB/audio** in this test. Spatial reached
45.1% lower latency, but RGB differed: PSNR **57.54 dB**, minimum sampled SSIM
**0.99919**, maximum difference **4/255**; audio remained exact. All five
configurations reproduced their own outputs across repeated requests.
Layerwise offload reduced memory with essentially unchanged latency here; this
does not make offload free on other workloads or slower host links.

A separate **memory-budget probe**, still on RTX PRO 6000, completed the full
clip twice with all-layerwise offload and a **30 GiB PyTorch allocator cap**.
Sampled whole-GPU usage peaked at **31.03 GiB**, and RGB/audio matched the
resident baseline exactly. The cap excludes CUDA allocations outside PyTorch;
it is a test setting, not a deployment recommendation. This supports trying the
32GB recipe, but is **not a 5090 speed, memory-fit or quality measurement**.

The complete HTTP regression also passed on one-GPU resident serial, Ulysses2
resident whole-tile and one-GPU layerwise serial with the allocator cap:
six requests each covering JSON, uploads, x2/x4/x2.25, tile batch 2,
audio/silent inputs and repeated-request equality. Arbitrary videos still need
their own validation.

### Small workloads do not necessarily scale

On GB300, a 384 x 256 x 33-frame source upscaled x2 (one tile, serial decode,
otherwise the same BF16/FA/seed/step policy) took **1.77 s on one GPU,
1.72 s on Ulysses2 and 1.74 s on Ulysses4**. These are three-hot-request medians
including saving. The roughly 2% difference is not evidence of a useful speedup;
prefer one GPU for this workload. More tiles, not just more GPUs, create work
that whole-tile decoding can distribute.

## 5. Memory and parallelism

### Hardware and GPU counts

| Hardware | Starting point | Evidence boundary |
| - | - | - |
| GB300 | Resident, FA; Ulysses4 + whole-tile for multi-tile clips | Measured offline; smaller 1/2/4-GPU workloads also tested |
| RTX PRO 6000 96GB | Resident, SDPA; compare 1 GPU with Ulysses2 | PCIe measurements, not a substitute for 5090 memory validation |
| H100 80GB / H200 141GB | Resident, FA; start with 1 GPU, then Ulysses2/4 | Extrapolated from native CUDA support; no Hopper SR measurement |
| B200 | Resident, FA; start with 1 GPU, then Ulysses2/4 | Extrapolated SR recipe; generation-model tests do not validate SR |
| RTX 5090 32GB | SDPA, layerwise offload, tile batch 1; short 384 x 256 input first | Extrapolated from the RTX memory-budget probe; no 5090 execution claim |

Only compare GPU counts on the **same input, scale, tile batching and backend**.
For small single-tile clips, one GPU avoids communication overhead. For larger
multi-tile clips, try Ulysses2/4 with whole-tile decode; extra GPUs do not divide
replicated KVAE/latent-upscaler weights. Spatial decode can reduce activation
memory, but changes rounding. Offload needs sufficient host RAM as well as VRAM.
Do not extrapolate NVLink scaling to a PCIe system, especially across CPU sockets.

Eight-GPU FA configurations are selectable as **Ulysses4 x Ring2**, not Ulysses8:
28 heads are not divisible by 8. This is an unmeasured topology, not a speedup
recommendation. The current RTX SDPA path has no Ring support, so the picker
rejects that combination. Start with fewer GPUs before trying an eight-GPU run.

### Placement and topology

DiT Ulysses, TP, Ring and TP x SP are supported. For example, use
`--num-gpus 4 --tp-size 1 --ulysses-degree 4` for Ulysses4, or
`--num-gpus 4 --tp-size 2 --ulysses-degree 2` for TP2 x Ulysses2.
TP times Ulysses must divide the checkpoint's 28 attention heads.
**TP and Ring can change output numerics independently of decode mode**;
whole-tile decode being exact does not make a TP/Ring topology change lossless.

The decode group includes TP/SP ranks within each DP replica, not different
requests. KVAE encoding, latent upscaling and tile scheduling remain replicated.
For short two-step clips, extra GPUs can add communication without a speedup.

Keep components resident when memory permits. For less GPU weight memory, use
`--layerwise-offload-components all`, or select `vae latent_upscaler` instead.
Temporal caches and non-block weights remain on GPU. Offload is compatible with
both decode modes, but consumes host RAM and adds transfers for tiles and temporal
segments. The assembled output also occupies host memory until saved.

The **Server** tab also offers whole-component offload with
`--cpu-offload-components`, selectable offload components, and DiT prefetch and
resident-layer counts when streaming the DiT. Increasing those counts consumes
additional VRAM. Partial offload recipes, non-default layer counts and
FSDP (`--use-fsdp-inference true`) are **unverified options**, not measured
improvements. FSDP requires multiple GPUs; SDPA cannot be combined with Ring.
The default backend is FA on datacenter Blackwell/Hopper and SDPA on RTX.
Hardware starting points do not automatically adapt to every uploaded video's
memory demand. Request scale and tile batching do not require restarting the server.

## 6. Limits

* **Input:** at most 121 frames. Sources near 24 fps retain their frames; faster
  videos are subsampled, slower videos retain their native rate. This can change
  the source timing; SR is not a bit-preserving video/audio transcode.
* **Attention:** this native path is dense. Do not compare it with a NABLA sparse
  reference and attribute every difference to parallel decoding. The latent
  upscaler uses SDPA motion attention when `natten` is unavailable.
* **Deployment:** monolithic only. Disaggregated roles, `torch.compile`, LoRA,
  direct latent inputs and KVAE-bridge chaining are not supported.
* **Loading:** component config and weight mismatches fail explicitly. For a
  renamed local checkpoint, pass
  `--model-id Kandinsky-6.0-VSR-distilled2steps-5s-Diffusers` to retain SR routing.

See the [compatibility matrix](/docs/sglang-diffusion/compatibility_matrix) for
other model families. The opt-in server regression is
`python/sglang/multimodal_gen/test/server/test_server_kandinsky6_sr.py`; it covers
uploads, JSON, all three scales, source audio and repeated requests, not official
pipeline parity.


# Krea-2
Source: https://docs.sglang.io/cookbook/diffusion/Krea/Krea-2

Deploy Krea-2 with SGLang - fast, high-quality text-to-image generation.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[Krea-2](https://huggingface.co/krea/Krea-2-Turbo) is Krea's photorealistic text-to-image family, built as a single-stream MMDiT with a Qwen3-VL text encoder and Qwen-Image VAE. Both public variants use the same native SGLang pipeline and differ mainly in their sampling target.

Choose Turbo for interactive generation: it is distilled to 8 steps with `guidance_scale=1.0`. Choose Raw when maximum fidelity matters more than latency: it uses roughly 52 steps with classifier-free guidance. Neither checkpoint is an image-editing model; use the Qwen-Image-Edit or FLUX.2 path when an input image must be preserved or transformed.

| Variant | Model ID | Sampling profile |
| - | - | - |
| Turbo | `krea/Krea-2-Turbo` | 8 steps, no CFG; fastest path |
| Raw | `krea/Krea-2-Raw` | About 52 steps with CFG; higher-fidelity path |

## 3. Model Deployment

This section covers deploying Krea-2-Turbo for fast, high-quality image generation.

### 3.1 Basic Configuration

Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` and the selected Krea-2 checkpoint.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path krea/Krea-2-Turbo
```

Krea-2-Turbo generates high-quality images in only 8 inference steps. Launch the server with:

```bash Command theme={null}
sglang serve \
  --model-path krea/Krea-2-Turbo \
  --num-gpus 1 \
  --port 30000
```

The step count and guidance scale are **request-time** settings (see [API Usage](#4-api-usage)); Krea-2-Turbo defaults to 8 steps with `guidance_scale = 1.0`.

### 3.2 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements.

* `--num-gpus`: Number of GPUs to use.
* Multi-GPU (tensor and/or sequence parallelism): see [Section 3.3](#3-3-multi-gpu-tensor-and-sequence-parallelism).

### 3.3 Multi-GPU: tensor and sequence parallelism

Krea-2 supports two multi-GPU axes that can be combined; `--num-gpus` must equal
`tp_size × ulysses_degree`.

* **Tensor parallelism (`--tp-size N`)** shards the DiT weights across GPUs, lowering
  per-GPU VRAM. Krea-2's attention heads (48 query / 12 KV) and text heads (20) are
  divisible by a tp size of 1, 2, or 4.
* **Sequence parallelism / Ulysses (`--ulysses-degree N`)** shards the image-token
  sequence across GPUs while keeping the text prefix replicated. It does **not** shard
  weights (per-GPU VRAM is unchanged), but its output is **bitwise-identical** to
  single-GPU. It currently requires a single prompt per request (ragged/padded
  multi-prompt batches under SP are not supported — use `--tp-size` for those).

```bash Command theme={null}
# Tensor parallel (2 GPUs) — lowest per-GPU VRAM (DiT weights sharded)
sglang serve --model-path krea/Krea-2-Turbo --num-gpus 2 --tp-size 2 --port 30000

# Sequence parallel / Ulysses (2 GPUs) — output bitwise-identical to single-GPU
sglang serve --model-path krea/Krea-2-Turbo --num-gpus 2 --ulysses-degree 2 --port 30000

# Hybrid TP × SP (4 GPUs) — composes both axes
sglang serve --model-path krea/Krea-2-Turbo --num-gpus 4 --tp-size 2 --ulysses-degree 2 --port 30000
```

Measured on 2× H200 (Krea-2-Turbo, 8 steps, 1024×1024): `--tp-size 2` and
`--ulysses-degree 2` each give \~1.7× denoise speedup over single-GPU; the hybrid
TP=2 × SP=2 reaches \~2.8× on 4 GPUs. **Choosing:** on memory-constrained GPUs prefer
`--tp-size` (it shards the \~24 GB DiT, e.g. \~38 GB → \~27 GB per GPU on 2 GPUs); on
large-VRAM GPUs sequence parallelism is marginally faster and numerically exact, and
the two compose for the highest throughput.

## 4. API Usage

For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api).

### 4.1 Generate an Image

Generate an image with the OpenAI-compatible images API:

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")

response = client.images.generate(
    model="krea/Krea-2-Turbo",
    prompt="a red fox sitting in fresh snow, golden hour, photorealistic",
    n=1,
    response_format="b64_json",
)

# Save the generated image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
    f.write(image_bytes)
```

You can also generate a single image from the command line:

```bash Command theme={null}
sglang generate --model-path krea/Krea-2-Turbo \
    --prompt "a red fox sitting in fresh snow, golden hour, photorealistic" \
    --num-inference-steps 8 --height 1024 --width 1024 --save-output
```

### 4.2 Advanced Usage

#### 4.2.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to speed up inference with minimal quality loss. Enable it by setting `SGLANG_CACHE_DIT_ENABLED=true`. For more details, see the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit).

Cache-DiT works for **both** Krea-2 variants with no extra configuration: SGLang tracks each request's classifier-free-guidance mode, so Krea-2-Turbo (no CFG, `guidance_scale = 1.0`) and Krea-2-Raw (CFG, `guidance_scale ≈ 4.5`) both cache correctly and automatically.

**Basic Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve \
  --model-path krea/Krea-2-Turbo \
  --num-gpus 1 \
  --port 30000
```

Measured per-image denoise speedup with the default cache settings (NVIDIA H200, 1024x1024, seed 0):

| Variant | Inference steps | Denoise (no cache → cache) | Speedup |
| :- | :- | :- | :- |
| Krea-2-Turbo (no CFG) | 8 | 1.27s → 0.92s | \~1.4x |
| Krea-2-Raw (CFG 4.5) | 50 | 18.0s → 6.3s | \~2.9x |

Caching has the most headroom on Raw's longer schedule; the 8-step distilled Turbo has only a few cacheable steps after warmup.

**Advanced Usage**

* DBCache Parameters: DBCache controls block-level caching behavior:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion (best suited to the longer Raw schedule; not recommended for the 8-step Turbo):

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

Combined Configuration Example (Krea-2-Raw, default cache settings shown explicitly):

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=1 \
SGLANG_CACHE_DIT_BN=0 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.24 \
SGLANG_CACHE_DIT_MC=3 \
sglang serve --model-path krea/Krea-2-Raw
```

#### 4.2.2 Memory and Component Residency

Krea-2's DiT is \~24 GB in bf16 (the bulk of the model). On memory-constrained GPUs you can keep less of it resident:

* `--component-residency dit=layerwise-offload`: stream the DiT's transformer blocks layer-by-layer with async host-to-device prefetch overlap, so only a small working set stays on the GPU. This is the primary way to fit Krea-2 on a single consumer / 32 GB-class card, at a modest latency cost. Tune the memory/latency trade-off with `--dit-offload-prefetch-size` (`0.0` prefetches one layer for the lowest memory; larger values prefetch more layers -- faster but more memory).
* `--component-residency dit=component-offload`: keep the complete DiT on CPU between denoising uses. This and layerwise offload are distinct modes; do not combine them for the same component.
* `--component-residency text_encoder=component-offload`: offload the Qwen3-VL text encoder while it is idle during denoising.
* `--component-residency vae=component-offload`: offload the VAE between uses.
* `--pin-cpu-memory`: pin host memory for offload. Add only as a temporary workaround if you hit `CUDA error: invalid argument`.

The legacy `--dit-layerwise-offload`, `--dit-cpu-offload`, `--text-encoder-cpu-offload`, and `--vae-cpu-offload` forms remain accepted. If both legacy DiT offload flags are enabled, layerwise offload is the effective DiT mode.

On large-VRAM GPUs (e.g. H200), keep everything resident (offloads off) for the fastest latency.

## 5. Benchmark

Test Environment:

* Hardware: NVIDIA H200 GPU (1x)
* Model: krea/Krea-2-Turbo (8 inference steps)
* sglang diffusion version: 0.5.13

**Server Command** (used for both benchmarks below):

```shell Command theme={null}
sglang serve --model-path krea/Krea-2-Turbo --port 30000
```

### 5.1 Generate an image

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --model krea/Krea-2-Turbo --dataset vbench --task text-to-image \
    --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-image
Model:                                   krea/Krea-2-Turbo
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  1.56
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.64
Latency Mean (s):                        1.5600
Latency Median (s):                      1.5600
Latency P99 (s):                         1.5600
--------------------------------------------------
Peak Memory Max (MB):                    37466.00
Peak Memory Mean (MB):                   37466.00
Peak Memory Median (MB):                 37466.00
============================================================
```

### 5.2 Generate images with high concurrency

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --model krea/Krea-2-Turbo --dataset vbench --task text-to-image \
    --num-prompts 20 --max-concurrency 20
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-image
Model:                                   krea/Krea-2-Turbo
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  31.47
Request rate:                            inf
Max request concurrency:                 20
Successful requests:                     20/20
--------------------------------------------------
Request throughput (req/s):              0.64
Latency Mean (s):                        16.5000
Latency Median (s):                      16.5200
Latency P99 (s):                         31.1300
--------------------------------------------------
Peak Memory Max (MB):                    37468.00
Peak Memory Mean (MB):                   37466.40
Peak Memory Median (MB):                 37466.00
============================================================
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# LTX2 & LTX2.3
Source: https://docs.sglang.io/cookbook/diffusion/LTX/LTX2 & LTX2.3

Run LTX-2 and LTX-2.3 video generation pipelines with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[LTX-2](https://huggingface.co/Lightricks/LTX-2) and [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) are Lightricks video models for text-to-video and image-conditioned generation. Their defining deployment choice is the pipeline: one-stage minimizes latency, standard two-stage refines quality, and LTX-2.3 adds a 1920×1088 HQ path.

Choose LTX when one model family must span fast generation, higher-quality refinement, and LoRA adaptation. Two-stage quality costs substantially more memory and startup work because both DiTs and the spatial upsampler participate; use the one-stage path when latency or capacity matters more than final detail.

Use `Lightricks/LTX-2` or `Lightricks/LTX-2.3` as `--model-path`. SGLang loads the two-stage upsampler and distilled components from the selected snapshot.

> [!WARNING]
>
> **License notice:** LTX-2 and LTX-2.3 are released under the LTX-2 Community License Agreement, not Apache 2.0. The license includes commercial-use restrictions for some entities. Review the [official Lightricks license](https://huggingface.co/Lightricks/LTX-2.3/blob/main/LICENSE) before production or commercial use; SGLang support does not grant additional model usage rights.

## 3. Model Deployment

This section provides deployment configurations optimized for different LTX pipelines and hardware targets.

### 3.1 Basic Configuration

The LTX series supports one-stage and two-stage pipelines. LTX-2.3 also supports the HQ two-stage pipeline. The recommended launch configuration depends on whether the target GPU can keep both two-stage DiTs resident.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

Choose the pipeline class based on the quality and latency target:

| Use case | Pipeline class | Notes |
| - | - | - |
| One-stage generation | `LTX2Pipeline` | Fastest LTX native path. Supports T2V and TI2V. |
| Two-stage generation | `LTX2TwoStagePipeline` | Uses a base stage and a refinement stage. Supported by LTX-2 and LTX-2.3. |
| Two-stage High Quality (HQ) generation | `LTX2TwoStageHQPipeline` | LTX-2.3 HQ path; defaults to 1920x1088 unless you override `--width` and `--height`. |

Feature compatibility:

| Pipeline class | T2V | TI2V (`--image-path`) | LoRA (`--lora-path`) | Notes |
| - | - | - | - | - |
| `LTX2Pipeline` | Yes | Yes | Yes | One-stage path. Cannot be combined with HQ because HQ is a separate two-stage pipeline class. |
| `LTX2TwoStagePipeline` | Yes | Yes | Yes | Standard two-stage path for LTX-2 and LTX-2.3. |
| `LTX2TwoStageHQPipeline` | Yes | Yes | Yes | High Quality two-stage path for LTX-2.3. Use this instead of `LTX2Pipeline`; it is not a one-stage mode flag. |

For two-stage pipelines, `--ltx2-two-stage-device-mode` controls transformer residency:

| Mode | When to use it |
| - | - |
| `resident` | Best latency on high-VRAM GPUs because both DiTs can stay resident. |
| `original` | Closest to the original two-stage switching semantics. |

Other deployment flags:

* `--lora-path`: Preload a community LoRA adapter.
* `--lora-weight-name`: Select the exact safetensors file when the LoRA repository contains multiple weight files.

> [!NOTE]
>
> For LTX-2 and LTX-2.3 two-stage serving without a user LoRA, `resident` is the fastest high-VRAM path: stage 2 runs a second DiT with the distilled LoRA merged in, and it is the default on GPUs with at least 130 GiB. When you pass `--lora-path`, SGLang still applies the user LoRA during the two-stage switch, so use `resident` on H200-class GPUs for enough VRAM, but do not expect the same premerged-stage2 benefit as the no-user-LoRA path.

### 3.3 Fast multi-GPU presets

For latency-oriented LTX serving, prefer CFG parallel over sequence parallelism. CFG parallel splits guidance branches across GPUs, while SP/Ulysses is mainly a memory/long-sequence tool for LTX.

| Target | Recommended server flags | Notes |
| - | - | - |
| LTX-2.3, 1 high-VRAM GPU | `--ltx2-two-stage-device-mode resident` | Fastest two-stage setup when both DiTs fit. |
| LTX-2.3, 1 standard GPU | `--ltx2-two-stage-device-mode original` | Lower VRAM than `resident`; use this when H100-class memory is tight. |
| LTX-2, 2 GPUs | `--num-gpus 2 --enable-cfg-parallel` | Fastest verified 2-GPU setup; keep `--dit-layerwise-offload` disabled unless memory is tight. |
| LTX-2.3, 2 GPUs | `--num-gpus 2 --enable-cfg-parallel --ltx2-two-stage-device-mode resident` | Fastest common 2-GPU setup. |
| LTX-2.3, 4 GPUs | `--num-gpus 4 --tp-size 2 --enable-cfg-parallel --ltx2-two-stage-device-mode resident` | Fastest common 4-GPU layout: TP2 inside each CFG branch. |
| Official comparison | `--ltx2-two-stage-device-mode original` | Use this only when matching the original LTX-2.3 stage-switch semantics matters. |

Use `--enable-cfg-parallel` for degree-2 CFG parallel. Use `--cfg-parallel-size` only when you explicitly need a different CFG branch count. If `resident` exceeds available VRAM, keep the same parallelism preset and switch only the device mode to `original`.

On high-VRAM GPUs, add `--text-encoder-cpu-offload false` if text encoding latency matters and you have enough memory.

On GPUs with at least 130 GiB (H200, B200, B300), `LTX2TwoStagePipeline` serving turns on breakable CUDA graphs by default when the DiTs stay resident (LTX-2.3: `resident` mode). Warmup captures the default 768x512, 121-frame request; other resolutions or frame counts run eager, so pass `--warmup-resolutions` and `--warmup-num-frames` for the shape you serve (the OpenAI video API defaults to 4 seconds, which LTX rounds to 89 frames). Pass `--enable-breakable-cuda-graph false` to run eager.

#### 3.3.1 Two GPUs

```bash theme={null}
sglang serve \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --num-gpus 2 \
  --enable-cfg-parallel \
  --ltx2-two-stage-device-mode resident
```

#### 3.3.2 Four GPUs

```bash theme={null}
sglang serve \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --num-gpus 4 \
  --tp-size 2 \
  --enable-cfg-parallel \
  --ltx2-two-stage-device-mode resident
```

## 4. Model Invocation

### 4.1 Basic Usage

The examples below spell out the current SGLang sampling defaults for reproducibility:

| Model path | Default output | Default frames | Default steps |
| - | - | - | - |
| `Lightricks/LTX-2` | 768x512 | 121 | 40 |
| `Lightricks/LTX-2.3` | 768x512 | 121 | 30 |
| `Lightricks/LTX-2.3` with `LTX2TwoStageHQPipeline` | 1920x1088 | 121 | 15 |

#### 4.1.1 LTX-2 one-stage text-to-video

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2 \
  --pipeline-class-name LTX2Pipeline \
  --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \
  --save-output
```

#### 4.1.2 LTX-2.3 one-stage text-to-video

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2Pipeline \
  --prompt "A quiet coastal town at sunrise, fishing boats moving slowly through golden mist, cinematic camera movement" \
  --save-output
```

#### 4.1.3 LTX-2 two-stage text-to-video

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \
  --save-output
```

#### 4.1.4 LTX-2.3 two-stage text-to-video

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "A handheld shot follows a red tram crossing a rainy city square at night, reflections on the pavement, cinematic lighting" \
  --save-output
```

#### 4.1.5 LTX-2.3 HQ text-to-video

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStageHQPipeline \
  --prompt "A wide cinematic shot of alpine clouds rolling over a mountain ridge, soft morning light, slow aerial camera movement" \
  --save-output
```

#### 4.1.6 Image-to-video with one reference image

Pass one image to `--image-path` for image-conditioned generation:

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --image-path ./inputs/start.png \
  --prompt "The camera slowly pushes forward as the subject turns toward warm window light, subtle natural motion, cinematic" \
  --save-output
```

#### 4.1.7 First-to-last-frame transition with two reference images

Pass two images to `--image-path` for transition-style TI2V. The first image is used as the starting condition and the second image is used as the ending condition.

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --image-path ./inputs/start.png ./inputs/end.png \
  --prompt "A smooth cinematic transition from the first scene into the final scene, dynamic camera motion, motion blur, zhuanchang" \
  --save-output
```

### 4.2 Advanced Usage

#### 4.2.1 Use community LoRAs

Use `--lora-path` to load a LoRA adapter. If the Hugging Face repo contains multiple safetensors files, use `--lora-weight-name` to select the exact file. `--lora-scale` maps to the standard LoRA merge scale and defaults to `1.0`.

The following example uses [`valiantcat/LTX-2.3-Transition-LORA`](https://huggingface.co/valiantcat/LTX-2.3-Transition-LORA):

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --lora-path valiantcat/LTX-2.3-Transition-LORA \
  --lora-weight-name ltx2.3-transition.safetensors \
  --prompt "A low-angle tracking shot moves through a foggy forest road. The camera rises above the treetops and transitions into a clear view of a snowy mountain peak under bright sunlight, zhuanchang" \
  --save-output
```

You can combine the Transition LoRA with two reference images:

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --pipeline-class-name LTX2TwoStagePipeline \
  --image-path ./inputs/start.png ./inputs/end.png \
  --lora-path valiantcat/LTX-2.3-Transition-LORA \
  --lora-weight-name ltx2.3-transition.safetensors \
  --prompt "A fast cinematic transition from the first image to the second image, whip-pan motion, atmospheric lighting, zhuanchang" \
  --save-output
```

> [!NOTE]
>
> Some community LoRAs only include weights for transformer blocks. In that case, SGLang logs a concise coverage summary and leaves unmatched LoRA-capable layers on the base model weights. This is expected when the adapter format intentionally omits those layers.

## 5. Practical Tips

* Use `--pipeline-class-name LTX2TwoStagePipeline` as the default LTX two-stage quality path.
* Use `--pipeline-class-name LTX2TwoStageHQPipeline` when you want the HQ path and have enough VRAM for larger outputs.
* Use `--ltx2-two-stage-device-mode resident` on high-VRAM GPUs if latency matters more than memory usage.
* Use `--ltx2-two-stage-device-mode original` when comparing against official two-stage behavior.
* Keep `--width` and `--height` aligned with the target model resolution; for LTX models, these are output video dimensions.

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# LTX2.5
Source: https://docs.sglang.io/cookbook/diffusion/LTX/LTX2.5

Run LTX-2.5 video + audio generation with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

> **Installation and prerequisites**
>
> Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Docker is recommended for Linux GPU deployments; run the commands below inside the container or your installed Python environment.
>
> NATTEN is an optional extra, worth installing only if you plan to use the
> [diffusion decoder](#4-6-diffusion-decoder) — see that section for why.

To use diffusion decoding, load **Diffusion decoder weights** in Server, then
select **Diffusion** in Request. Duration and decoder selection are request-time
choices; loading the optional decoder requires a server restart.

## 2. Model capabilities

[LTX-2.5](https://huggingface.co/Lightricks/LTX-2.5) is an open world model from
Lightricks, built for local execution and fine-tuning. Its established use is
generating synchronized, high-fidelity video and audio from text, image and
video inputs.

It is a 22B DiT paired with a Gemma-4-12B text encoder, separate video and audio
VAEs, and a vocoder that outputs 48 kHz stereo. Video and audio are denoised
jointly in one pass rather than dubbed afterwards, so they stay in sync.

Use **`Lightricks/LTX-2.5-Diffusers`** as `--model-path`.

> [!WARNING]
>
> **License notice:** LTX-2.5 is released under the LTX-2.x Community License
> Agreement, not Apache 2.0. The license includes commercial-use restrictions for
> some entities. Review the [official Lightricks license](https://github.com/Lightricks/LTX-2/blob/main/LICENSE.md)
> before production or commercial use; SGLang support does not grant additional
> model usage rights.

### 2.1 New in LTX-2.5

Two capabilities have no equivalent in LTX-2 / LTX-2.3:

> **[Auto-duration](#4-3-auto-duration)**
>
> A duration head predicts how long the shot the caption implies should run,
> and picks the frame count for you. Pass `--auto-duration` instead of
> `--num-frames`.

> **[Diffusion decoder](#4-6-diffusion-decoder)**
>
> A diffusion model replaces the convolutional VAE decoder for the
> latent-to-pixel step. Enable with `--use-diffusion-decoder`.

Both are optional and off by default.

### 2.2 Components

| Path | Component | Used by |
| - | - | - |
| `transformer/` | Distilled DiT (the default) | always |
| `transformer_full/` | Full / SFT DiT | `--model-variant dev` |
| `vae/` | Convolutional video VAE | encode always; decode by default |
| `diffusion_decoder/` | Diffusion video decoder, decoder-only | `--use-diffusion-decoder` |
| `latent_upsampler/` | Spatial x2 latent upsampler | `LTX2TwoStagePipeline` |
| `duration_head/` | Predicts clip length from the caption | `--auto-duration` |
| `audio_vae/`, `vocoder/`, `connectors/`, `text_encoder/`, `tokenizer/`, `scheduler/` | Shared | always |

Encoding always uses `vae/`, and both decoders consume the same latents, so the
decoder choice does not change anything upstream of it.

## 3. Model Deployment

### 3.1 Basic Configuration

```bash theme={null}
sglang serve \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline
```

On a single high-VRAM GPU no extra flags are needed.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

Choose the pipeline class based on the quality and latency target:

| Use case | Pipeline class | Notes |
| - | - | - |
| One-stage generation | `LTX2Pipeline` | Fastest path. Supports T2V and TI2V, auto-duration and the diffusion decoder. |
| Two-stage generation | `LTX2TwoStagePipeline` | Half-resolution base stage, x2 latent upsample, then a short refinement. Pass the **final** resolution. |

There is no HQ pipeline class for LTX-2.5, and no `--distilled-lora-path` for
either weights variant: LTX-2.5 distils the weights themselves rather than
merging a LoRA per stage, so `--ltx2-two-stage-device-mode` (which governs that
swap) does not apply either.

Every feature on this page — text-to-video, image conditioning, auto-duration,
the diffusion decoder, and either weights variant — works with both pipeline
classes.

Selecting weights:

* `--model-variant dev` serves the full / SFT DiT from `transformer_full/`; the
  default is the distilled one. See [section 4.5](#4-5-the-dev-transformer).

On GPUs with at least 130 GiB, `LTX2TwoStagePipeline` serving turns on breakable
CUDA graphs by default, as for LTX-2 / LTX-2.3. Graphs are captured per shape at
warmup, so pass the shape you serve (for example `--warmup-resolutions 1920x1088 --warmup-num-frames 121`); other shapes run eager. Pass
`--enable-breakable-cuda-graph false` to run eager throughout.

### 3.3 Multi-GPU presets

| Target | Recommended server flags | Notes |
| - | - | - |
| 1 high-VRAM GPU | *(no extra flags)* | 960×544 fits comfortably on an H200. |
| 1 tight-VRAM GPU | `--quantization fp8` | Halves the DiT and cuts peak memory \~18 GB at unchanged speed. See [section 3.4](#3-4-fp8-quantization). |
| 1 very tight GPU | `--dit-layerwise-offload` | Cuts peak memory by roughly 10 GB, at about 4x the wall clock. |
| 1 32 GB GPU (e.g. RTX 5090) | `--layerwise-offload-components dit text_encoder` | Streams the DiT and the 22 GB Gemma-4 encoder layer by layer and keeps the rest resident. 960×544 / 121 frames / 8 steps: 15.2 s at 17 GB peak, vs 17.0 s at 31 GB with `--cpu-offload-components text_encoder ...`. Pins \~55 GB of host memory per GPU. |
| 2 GPUs, long sequences | `--num-gpus 2 --ulysses-degree 2` | Sequence parallel; the memory/long-sequence tool. |
| 2 32 GB GPUs | `--num-gpus 2 --ulysses-degree 2 --layerwise-offload-components dit text_encoder --vae-tiling false --vae-sp false --dit-layerwise-resident-layers 12` | Untiled VAE decode splits the frame height across the ranks (2.2 s to 1.1 s); 12 resident DiT layers cut the per-step host stream. RTX 5090 960×544 / 121 frames: 16.9 s at 26 GB peak per GPU, vs 21.8 s. Pins \~55 GB of host memory per GPU. |
| 2 GPUs, large DiT | `--num-gpus 2 --tp-size 2` | Tensor parallel across attention heads. |
| 2 GPUs, dev weights | `--num-gpus 2 --enable-cfg-parallel` | Splits the guided and unguided branches across GPUs. Measured 1.77x on denoising (15.1s to 8.5s, 960×544 / 57 frames / 30 steps). |
| 2 GPUs, diffusion decoder | `--num-gpus 2 --ulysses-degree 2` | The decoder's tiles are split across the ranks by default. See [section 4.6.1](#4-6-1-memory-and-multi-gpu). |

> [!WARNING]
>
> **CFG parallelism does not apply on the default (distilled) path.** That DiT
> runs unguided, so there is no negative branch to split across GPUs and
> `--enable-cfg-parallel` buys nothing — the CFG-parallel presets on the
> LTX-2 / LTX-2.3 page do not carry over. It *is* worth using with
> `--model-variant dev`, which runs with guidance.

### 3.4 fp8 quantization

`--quantization fp8` quantizes the DiT's linear layers as it loads them, so it
needs no pre-quantized checkpoint:

```bash theme={null}
sglang serve \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --quantization fp8
```

At 960×544 / 49 frames the transformer loads in 18.11 GB against 35.37 GB for
bf16, and the run peaks at 53.5 GB against 71.1 GB. Denoising time is
unchanged: the distilled 8-step path at this size is bound by memory traffic
rather than matmul throughput, so fp8 buys headroom rather than speed.

Expect a different sample for a given seed. Quantization nudges the denoising
trajectory and diffusion amplifies that, so the result differs from bf16
without being worse.

## 4. Model Invocation

### 4.1 Text-to-video with audio

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn, the camera tracking alongside, snow crunching underfoot." \
  --save-output
```

Defaults: 960×544, 121 frames, 24 fps. Video and audio are generated jointly and
muxed into one MP4.

The default DiT is distilled and runs off a fixed 8-sigma schedule rather than a
step count, so `--num-inference-steps` and `--guidance-scale` have no effect
here. Use [`--model-variant dev`](#4-5-the-dev-transformer) when you want
control over either.

### 4.2 Image-to-video

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --image-path ./inputs/start.png \
  --prompt "The camera pushes forward as the subject turns toward the light." \
  --save-output
```

The conditioning image is re-compressed to match the compression the model was
trained against — CRF 18 for LTX-2.5, where LTX-2 / 2.3 use 33. SGLang picks the
right one from the checkpoint, so nothing needs to be passed.

### 4.3 Auto-duration

LTX-2.5 ships a duration head — a small module that reads the encoded caption
and regresses the natural length of the shot it describes. Use it when the
prompt implies a duration ("a quick glance" vs "a slow pan across the valley")
and you would rather not guess a frame count:

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --prompt "A red fox walking through a snowy forest at dawn." \
  --auto-duration \
  --save-output
```

The prediction is clamped to `--auto-duration-min-seconds` /
`--auto-duration-max-seconds` (default 1–20 s) and snapped to the VAE's temporal
grid, so the result is always a valid frame count. It overrides `--num-frames`.

For an online server, pass the same LTX-2.5-only controls through `extra_body`:

```python Python theme={null}
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:30010/v1")
video = client.videos.create(
    model="Lightricks/LTX-2.5-Diffusers",
    prompt="A red fox walking through a snowy forest at dawn.",
    extra_body={
        "auto_duration": True,
        "auto_duration_min_seconds": 2.0,
        "auto_duration_max_seconds": 8.0,
    },
)
```

### 4.4 Two-stage (higher quality)

Stage 1 runs at half the requested resolution, the latents are upsampled 2x, and
a short sigma tail refines at full resolution. Pass the **final** size:

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2TwoStagePipeline \
  --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn." \
  --height 1088 --width 1920 \
  --save-output
```

Resolution must be divisible by 64. Unlike LTX-2.3, no `--distilled-lora-path`
is needed: the LTX-2.5 transformer is already distilled.

### 4.5 The dev transformer

LTX-2.5 ships two DiTs. `model_index.json` points at the distilled one; the
full / SFT weights live in `transformer_full/` and are deliberately left out of
the index. Select them with `--model-variant dev`:

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --model-variant dev \
  --prompt "A cinematic shot of a red fox walking through a snowy forest at dawn." \
  --num-inference-steps 30 --guidance-scale 3.0 \
  --save-output
```

The dev variant is not distilled, so SGLang automatically drops the pinned
distilled sigma schedule and re-enables the dynamic shifting that `scheduler/`
turns off for the distilled DiT. Unlike the distilled path it *is* driven by a
step count and *does* want CFG, so pass `--num-inference-steps` and
`--guidance-scale` yourself.

Note that `from_pretrained` only fetches what `model_index.json` lists, so a
partial snapshot download will not include `transformer_full/` (another 38 GB).

### 4.6 Diffusion decoder

LTX-2.5 adds a diffusion-based video decoder as an alternative to the
convolutional VAE decoder. Rather than deconvolving the latent it denoises
pixels conditioned on a context volume built from it, which recovers detail a
convolutional decoder tends to smooth away:

```bash theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --prompt "A red fox walking through a snowy forest at dawn." \
  --use-diffusion-decoder \
  --save-output
```

It is a diffusion model in its own right and decodes more slowly than the VAE
decoder, so it is off by default — matching upstream, where `LTX2Pipeline` also
decodes with the VAE. The offline `generate` command loads the optional decoder
automatically when `--use-diffusion-decoder` is present.

For an online server, opt into loading the decoder at startup, then select it per
request with `use_diffusion_decoder: true`:

```bash theme={null}
sglang serve \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --load-diffusion-decoder
```

```python Python theme={null}
video = client.videos.create(
    model="Lightricks/LTX-2.5-Diffusers",
    prompt="A red fox walking through a snowy forest at dawn.",
    extra_body={"use_diffusion_decoder": True},
)
```

This keeps the default server footprint unchanged while still allowing VAE and
diffusion-decoder requests to share one server. When GPU memory is constrained,
`--cpu-offload-components diffusion_decoder` keeps the optional decoder on CPU
between uses.

> [!TIP]
>
> **Install NATTEN for this decoder.** Its stages run 3D neighborhood attention,
> and SGLang uses NATTEN's fused `na3d` kernel for it when the package is present.
> NATTEN is *not* a dependency of `sglang[diffusion]`: without it the decoder
> falls back to a compiled FlexAttention block mask. The two agree to bf16
> rounding, but the fallback is roughly **5x slower** on the decoder's largest
> attention grid, and has to build the mask on top of that.
>
> NATTEN ships prebuilt wheels pinned to a specific torch and CUDA build, so
> install the one matching your environment rather than a bare version — check
> your combination at [natten.org](https://natten.org). For torch 2.11 / CUDA
> 13.0, for example:
>
> ```bash theme={null}
> uv pip install natten==0.21.6+torch2110cu130 -f https://whl.natten.org/
> ```
>
> Nothing else changes if you skip it: the decoder still produces the same video,
> just slower.

#### 4.6.1 Memory and multi-GPU

Two flags govern how the decode is executed. Both default to on, so the numbers
below are what you already get — they matter when you want to turn one off.

`--diffusion-decoder-tiling` runs the decoder's two expensive stages over
overlapping tiles instead of the whole volume. It is a **memory** control, not a
speed one: it costs wall clock and buys headroom.

`--diffusion-decoder-parallel-tiling` splits those tiles across the
decode-parallel ranks -- the TP, SP, PP and CFG ranks of one replica, since the
decoder is replicated over all of them. Without it every one of those ranks
decodes every tile and keeps its own identical copy. It only applies on the
tiled path, so it does nothing when tiling is off, and nothing at a single
rank.

Decoding stage on 2xH200 at 960×544, Ulysses degree 2. "Peak" is the whole
process, not the decode alone:

| Frames | Tiling | Parallel tiling | Decode | Peak |
| - | - | - | - | - |
| 121 | off | *(n/a)* | **3.38s** | 103.2 GB |
| 121 | on | off | 5.68s | 78.2 GB |
| 121 | on | on | 4.08s | 79.3 GB |
| 49 | off | *(n/a)* | **1.93s** | 84.0 GB |
| 49 | on | off | 2.79s | 78.2 GB |
| 49 | on | on | 2.12s | 78.2 GB |

Reading that:

* **Untiled is the fastest option** whenever it fits. Tiling exists for the
  \~25 GB it saves at 121 frames, which is the difference between fitting on an
  80 GB card and not.
* **Parallel tiling recovers most of tiling's cost** — 1.39x at 121 frames,
  1.32x at 49 — but does not beat an untiled decode. It also adds about 1 GB
  for the gather buffers.
* The gain grows with tile count, so it is larger at higher resolution: at
  1920×1088 / 49 frames the decode goes from 12.32s to 7.86s, **1.57x**.

Output is bitwise identical however many ranks the tiles are split over: every
rank draws the whole grid's noise in the same order, and only the decode is
shared out. Turning *tiling* on or off does change the result slightly near
tile borders, so pick one and stay with it if you need reproducible frames:

```bash theme={null}
# Fastest, if the untiled decode fits in VRAM
sglang serve \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --load-diffusion-decoder \
  --diffusion-decoder-tiling false

# Memory-bound: keep tiling, and split the tiles over both GPUs
sglang serve \
  --model-path Lightricks/LTX-2.5-Diffusers \
  --pipeline-class-name LTX2Pipeline \
  --load-diffusion-decoder \
  --num-gpus 2 --ulysses-degree 2
```


# LingBot Video MoE
Source: https://docs.sglang.io/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE

Serve the native LingBot Video MoE 30B-A3B text-to-video model with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[LingBot Video MoE 30B-A3B](https://huggingface.co/robbyant/lingbot-video-moe-30b-a3b)
is a text-to-video mixture-of-experts model served through a native SGLang
Diffusion pipeline. Its main advantage is the MoE economics: 30B total
parameters with 3B active per token, so a plain `sglang serve` invocation
covers the released 480×480, 81-frame profile without parallelism flags.

Prompting is the distinctive part of this checkpoint: it expects a structured
JSON caption rather than an unexpanded natural-language prompt, passed as the
request's `prompt` string. A compact 17-frame, 12-step smoke-test profile is
available for quick validation next to the released 40-step profile.

The public checkpoint served by the pipeline:

| Model ID | Task | Default output |
| - | - | - |
| `robbyant/lingbot-video-moe-30b-a3b` | Text to video | 480x480, 81 frames at 16 FPS |

The checkpoint expects a structured JSON caption rather than an unexpanded
natural-language prompt. The JSON is passed as the request's `prompt` string;
it is not an `extra_params` object.

## 3. Serve LingBot Video MoE

Start the server with the Hugging Face model ID:

```bash Command theme={null}
sglang serve \
  --model-path robbyant/lingbot-video-moe-30b-a3b \
  --port 30010
```

## 4. Generate a video

The following request uses the compact 17-frame, 12-step smoke-test profile.
Use the model defaults of 81 frames and 40 steps for the released generation
profile.

```python Python theme={null}
import json
import time
from pathlib import Path

import requests

base_url = "http://127.0.0.1:30010"
prompt = json.dumps(
    {
        "comprehensive_description": {
            "scene_content_description": (
                "A small silver robot arm on a white table slowly reaches "
                "toward a red cube. The background is a softly lit laboratory wall."
            ),
            "camera_movement_description": (
                "The camera is static at eye level in a medium shot."
            ),
        },
        "camera_info": {
            "color": "Neutral",
            "frame_size": "Medium",
            "shot_type_angle": "Eye level",
            "lens_size": "Medium",
            "composition": "Center",
            "lighting": "Soft light",
            "lighting_type": "Artificial light",
        },
        "world_knowledge": [],
        "prominent_elements": [
            {
                "name": "robot arm",
                "description": "A small silver robot arm with a two-finger gripper.",
                "actions": [
                    {
                        "timestamp": "[0.0s - 1.0s]",
                        "action": "reaches toward the red cube",
                    }
                ],
                "location": "center of the frame",
                "relative_size": "dominant",
                "shape_and_color": "articulated silver metal arm",
                "texture": "brushed metal",
                "appearance_details": "two-finger gripper and visible joints",
                "relationship": "reaching toward the red cube on the table",
                "orientation": "upright, base on the table",
                "pose": "reaching",
            }
        ],
    },
    separators=(",", ":"),
)

response = requests.post(
    f"{base_url}/v1/videos",
    json={
        "model": "robbyant/lingbot-video-moe-30b-a3b",
        "prompt": prompt,
        "size": "640x384",
        "num_frames": 17,
        "fps": 16,
        "num_inference_steps": 12,
        "guidance_scale": 6.0,
        "flow_shift": 3.0,
        "seed": 0,
    },
    timeout=60,
)
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

video = requests.get(
    f"{base_url}/v1/videos/{video_id}/content",
    timeout=300,
)
video.raise_for_status()
Path("lingbot_video_moe.mp4").write_bytes(video.content)
```

## 5. Request constraints

* `num_frames` must be `1` or `4n+1`; examples include 17 and 81.
* Width and height must both be multiples of 16.
* The native defaults are `guidance_scale=6.0`, `flow_shift=3.0`,
  `num_inference_steps=40`, and `fps=16`.
* Keep the prompt as serialized JSON. Raw free text is outside the
  checkpoint's expected caption format.


# LingBot World
Source: https://docs.sglang.io/cookbook/diffusion/LingBot-World/LingBot-World

Serve LingBot World realtime camera-controlled video world models with SGLang-diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

Use the [Realtime WebUI](#3-realtime-webui) to start a camera-controlled session. The Request tab checks model metadata; video generation uses WebSocket, not an HTTP POST.

## 2. Model capabilities

[LingBot World](https://huggingface.co/robbyant/lingbot-world-fast-diffusers) is a realtime camera-controlled video world model. It keeps a live causal session, applies prompt and camera events between chunks, reuses DiT/VAE state, and streams decoded frames instead of finishing a bounded clip before returning.

Choose it for interactive exploration and continuous control, not one-shot cinematic generation. The causal window makes long sessions practical but gives up the global bidirectional context available to offline Wan or LTX pipelines; session state and cache policy are therefore part of the serving contract.

## 3. Deployment

```bash Command theme={null}
sglang serve \
  --model-path robbyant/lingbot-world-fast-diffusers \
  --pipeline-class-name LingBotWorldCausalDMDPipeline \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false
```

Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only parallel and offload flags.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path robbyant/lingbot-world-fast-diffusers  --pipeline-class-name LingBotWorldCausalDMDPipeline
```

### Optional KV-Cache Compression

Long-running sessions can enable lossy int4 PRQ compression for completed
causal KV-cache chunks by installing `quant-videogen` as described in the
quantization guide and adding `--kv-cache-quant int4` to the server command.
The current and most recent completed chunks remain in BF16. See
[Causal KV-Cache Quantization](/docs/sglang-diffusion/quantization#causal-kv-cache-quantization)
for the algorithm, tuning options, measured memory-latency tradeoff, and
support limits.

## 4. Realtime WebUI

The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior.

```bash Command theme={null}
python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui
```

Open `http://127.0.0.1:18080` and use:

```text Example theme={null}
ws://127.0.0.1:30000/v1/realtime_video/generate
```

## 5. HTTP and WebSocket API

LingBot World uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running.

### Endpoints

| API | Method | Purpose | Notes |
| - | - | - | - |
| `/v1/models` | `GET` | Query the served model id before opening a session. | The WebUI uses this to fill the model field when the server exposes model metadata. |
| `/v1/realtime_video/generate` | `WebSocket` | Create one realtime LingBot session and stream generated video chunks. | The first client message must be an `init` message encoded with MessagePack. |

### `init` message

Send this MessagePack map immediately after the WebSocket opens.

| Parameter | Type | Required | Meaning |
| - | - | - | - |
| `type` | string | Yes | Must be `"init"`. |
| `model` | string | No | Model id. Leave empty to use the served model. |
| `prompt` | string | Yes | Text prompt for the initial scene and motion style. |
| `first_frame` | bytes or string | Yes | Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string. |
| `size` | string | Yes | Generation size as `WIDTHxHEIGHT`, for example `832x480`. |
| `fps` | number | Yes | Target playback FPS for the generated stream. |
| `num_frames` | integer | Yes | Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size. |
| `seed` | integer | No | Random seed for deterministic sampling. |
| `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. |
| `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. |
| `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. |
| `quality` | `"exact"`, `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `lossless` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. |
| `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. |
| `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. |
| `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. |
| `realtime_output_format` | `"webp"`, `"jpeg"`, `"raw"` | No | Preview/output transport. `webp` and `jpeg` send encoded preview frames; `raw` (the default when omitted) sends lossless raw RGB. |
| `output_compression` | integer | No | Preview quality for `webp` or `jpeg`, from `1` to `100`. |
| `enable_upscaling` | boolean | No | Enable server-side super resolution after frame decode. |
| `upscaling_scale` | integer | No | Super-resolution scale. Current default is `4` when upscaling is enabled. |
| `upscaling_model_path` | string | No | Optional Real-ESRGAN model path. |
| `enable_frame_interpolation` | boolean | No | Enable frame interpolation. Keep this disabled when measuring true generated FPS. |
| `frame_interpolation_exp` | integer | No | Interpolation multiplier exponent. `1` means 2x frames. |
| `frame_interpolation_scale` | number | No | RIFE internal scale for interpolation. |
| `frame_interpolation_model_path` | string | No | Optional RIFE model path. |
| `condition_inputs.camera_actions` | `list[list[string]]` | No | Initial scripted camera actions, one action list per frame. |

### Runtime `event` messages

After `init`, send MessagePack event maps to update the live session.

| Parameter | Type | Required | Meaning |
| - | - | - | - |
| `type` | string | Yes | Must be `"event"`. |
| `kind` | `"prompt"` or `"camera_actions"` | Yes | Runtime event kind. |
| `payload` | string or object/list | Yes | For `prompt`, a non-empty string. For `camera_actions`, either scripted `list[list[string]]` or state-mode payload. |
| `event_id` | integer | No | Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled. |

`camera_actions` supports two payload modes:

| Mode | Payload shape | Meaning |
| - | - | - |
| Script | `list[list[string]]` | A fixed sequence of per-frame actions consumed by upcoming chunks. |
| State | `{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }` | Live control state transitions from keyboard or UI controls. |

Supported LingBot action tokens include `w`, `a`, `s`, `d` for camera movement and `i`, `j`, `k`, `l` for look controls.

### Server messages

| Message | Payload | Meaning |
| - | - | - |
| `frame_batch` | MessagePack map with `payload` bytes | One batch of frames. The map includes `chunk_index`, `num_frames`, `content_type`, `encoding`, `width`, `height`, and frame-batch metadata. |
| `chunk_stats` | MessagePack map | Per-chunk timing and transport metrics, including `scheduler_forward_ms`, `raw_payload_build_ms`, `chunk_total_ms`, `num_frames`, and `ws_payload_bytes`. |
| `error` | MessagePack map | Server-side validation or generation error. |

### Minimal client sketch

```python Python theme={null}
import msgspec.msgpack
import websocket

ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate")
ws.send_binary(msgspec.msgpack.encode({
    "type": "init",
    "prompt": "A quiet rainy London alley, stable camera motion.",
    "first_frame": open("reference.jpg", "rb").read(),
    "size": "832x480",
    "fps": 25,
    "num_frames": 9,
    "num_inference_steps": 4,
    "guidance_scale": 1,
    "realtime_output_format": "webp",
    "output_compression": 95,
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "camera_actions",
    "event_id": 1,
    "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]},
}))
```

## 6. Consistency

LingBot World uses raw-frame websocket GT plus per-chunk latency guards for consistency checks.

## 7. Notes

* Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`.
* Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks.
* Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior.

## 8. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# LingBot World 2.0
Source: https://docs.sglang.io/cookbook/diffusion/LingBot-World/LingBot-World-2.0

Serve LingBot World 2.0 realtime camera-controlled video world models with SGLang-diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

Use the [Realtime WebUI](#3-realtime-webui) to start a camera-controlled session. The Request tab checks model metadata; video generation uses WebSocket, not an HTTP POST.

## 2. Model capabilities

[LingBot World 2.0](https://huggingface.co/robbyant/lingbot-world-v2-14b-causal-fast-diffusers) is the 14B realtime camera-controlled world model in the LingBot family. It generates continuously from a live causal session, accepts control updates between chunks, and reuses DiT and VAE state for incremental frame delivery.

Choose 2.0 when the larger model and interactive KV-window path justify an 8-GPU deployment. Like the earlier LingBot release, it prioritizes responsive control and bounded causal state over the full-clip global context of offline video diffusion.

## 3. Deployment

```bash Command theme={null}
export SGLANG_LINGBOT_LAZY_VAE_ENCODE_BLACK_FRAMES=60
export SGLANG_LINGBOT_ENABLE_INTERACTIVE_KV_WINDOW=true
sglang serve \
  --model-path robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
  --pipeline-class-name LingBotWorldCausalDMDPipeline \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false \
  --vae-config.use-parallel-decode true \
  --vae-config.parallel-decode-mode spatial \
  --enable-torch-compile false
```

Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu`; omit GPU-only parallel and offload flags.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path robbyant/lingbot-world-v2-14b-causal-fast-diffusers --pipeline-class-name LingBotWorldCausalDMDPipeline
```

### Optional KV-Cache Compression

Long-running sessions can enable lossy int4 PRQ compression for completed
causal KV-cache chunks by installing `quant-videogen` as described in the
quantization guide and adding `--kv-cache-quant int4` to the server command.
The current and most recent completed chunks remain in BF16. See
[Causal KV-Cache Quantization](/docs/sglang-diffusion/quantization#causal-kv-cache-quantization)
for the algorithm, tuning options, measured memory-latency tradeoff, and
support limits.

## 4. Realtime WebUI

The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior.

```bash Command theme={null}
python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui
```

Open `http://127.0.0.1:18080` and use:

```text Example theme={null}
ws://127.0.0.1:30000/v1/realtime_video/generate
```

## 5. HTTP and WebSocket API

LingBot World 2.0 uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running.

### Endpoints

| API | Method | Purpose | Notes |
| - | - | - | - |
| `/v1/models` | `GET` | Query the served model id before opening a session. | The WebUI uses this to fill the model field when the server exposes model metadata. |
| `/v1/realtime_video/generate` | `WebSocket` | Create one realtime LingBot session and stream generated video chunks. | The first client message must be an `init` message encoded with MessagePack. |

### `init` message

Send this MessagePack map immediately after the WebSocket opens.

| Parameter | Type | Required | Meaning |
| - | - | - | - |
| `type` | string | Yes | Must be `"init"`. |
| `model` | string | No | Model id. Leave empty to use the served model. |
| `prompt` | string | Yes | Text prompt for the initial scene and motion style. |
| `first_frame` | bytes or string | Yes | Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string. |
| `size` | string | Yes | Generation size as `WIDTHxHEIGHT`, for example `832x480`. |
| `fps` | number | Yes | Target playback FPS for the generated stream. |
| `num_frames` | integer | Yes | Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size. |
| `seed` | integer | No | Random seed for deterministic sampling. |
| `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. |
| `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. |
| `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. |
| `quality` | `"exact"`, `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `lossless` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. |
| `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. |
| `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. |
| `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. |
| `realtime_output_format` | `"webp"`, `"jpeg"`, `"raw"` | No | Preview/output transport. `webp` and `jpeg` send encoded preview frames; `raw` (the default when omitted) sends lossless raw RGB. |
| `output_compression` | integer | No | Preview quality for `webp` or `jpeg`, from `1` to `100`. |
| `enable_upscaling` | boolean | No | Enable server-side super resolution after frame decode. |
| `upscaling_scale` | integer | No | Super-resolution scale. Current default is `4` when upscaling is enabled. |
| `upscaling_model_path` | string | No | Optional Real-ESRGAN model path. |
| `enable_frame_interpolation` | boolean | No | Enable frame interpolation. Keep this disabled when measuring true generated FPS. |
| `frame_interpolation_exp` | integer | No | Interpolation multiplier exponent. `1` means 2x frames. |
| `frame_interpolation_scale` | number | No | RIFE internal scale for interpolation. |
| `frame_interpolation_model_path` | string | No | Optional RIFE model path. |
| `condition_inputs.camera_actions` | `list[list[string]]` | No | Initial scripted camera actions, one action list per frame. |

### Runtime `event` messages

After `init`, send MessagePack event maps to update the live session.

| Parameter | Type | Required | Meaning |
| - | - | - | - |
| `type` | string | Yes | Must be `"event"`. |
| `kind` | `"prompt"`, `"camera_actions"`, or `"composite_input"` | Yes | Runtime event kind. |
| `payload` | string, object, or list | Yes | For `prompt`, a non-empty string. For `camera_actions`, either scripted `list[list[string]]` or state-mode payload. For `composite_input`, a map containing `input_types` plus each named input. |
| `event_id` | integer | No | Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled. |

`camera_actions` supports two payload modes:

| Mode | Payload shape | Meaning |
| - | - | - |
| Script | `list[list[string]]` | A fixed sequence of per-frame actions consumed by upcoming chunks. |
| State | `{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }` | Live control state transitions from keyboard or UI controls. |

Supported LingBot action tokens include `w`, `a`, `s`, `d` for camera movement and `i`, `j`, `k`, `l` for look controls.

Use `composite_input` when multiple runtime inputs should be sampled together, such as updating the prompt and camera controls in one event.

### Server messages

| Message | Payload | Meaning |
| - | - | - |
| `frame_batch` | MessagePack map with `payload` bytes | One batch of frames. The map includes `chunk_index`, `num_frames`, `content_type`, `encoding`, `width`, `height`, and frame-batch metadata. |
| `chunk_stats` | MessagePack map | Per-chunk timing and transport metrics, including `scheduler_forward_ms`, `raw_payload_build_ms`, `chunk_total_ms`, `num_frames`, and `ws_payload_bytes`. |
| `error` | MessagePack map | Server-side validation or generation error. |

### Minimal client sketch

```python Python theme={null}
import msgspec.msgpack
import websocket

ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate")
ws.send_binary(msgspec.msgpack.encode({
    "type": "init",
    "prompt": "A quiet rainy London alley, stable camera motion.",
    "first_frame": open("reference.jpg", "rb").read(),
    "size": "832x480",
    "fps": 25,
    "num_frames": 9,
    "num_inference_steps": 4,
    "guidance_scale": 1,
    "realtime_output_format": "webp",
    "output_compression": 95,
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "camera_actions",
    "event_id": 1,
    "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]},
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "prompt",
    "event_id": 2,
    "payload": "A quiet snowy Tokyo alley, stable camera motion.",
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "composite_input",
    "event_id": 3,
    "payload": {
        "input_types": ["prompt", "camera_actions"],
        "prompt": "A quiet neon Shanghai alley, stable forward camera motion.",
        "camera_actions": [["w"], ["w"], []],
    },
}))
```

## 6. Consistency

LingBot World 2.0 uses raw-frame websocket GT plus per-chunk latency guards for consistency checks.

## 7. Notes

* Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`.
* Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks.
* Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior.

## 8. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# LongCat-Image
Source: https://docs.sglang.io/cookbook/diffusion/LongCat/LongCat-Image

Deploy LongCat-Image with SGLang Diffusion and its native in-process Qwen2.5-VL prompt rewriter.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[LongCat-Image](https://huggingface.co/meituan-longcat/LongCat-Image) is a
text-to-image model from Meituan. SGLang runs its Qwen2.5-VL prompt rewriter
in process with the native SGLang runtime before text encoding and denoising.

The native pipeline keeps prompt rewriting and diffusion behind one OpenAI-compatible
image endpoint. Rewriting is enabled by default for stronger prompt expansion, but
each request can disable it when lower latency matters more than the rewritten prompt.

## 3. Serve the model

```bash Command theme={null}
sglang serve \
  --model-path meituan-longcat/LongCat-Image \
  --performance-mode auto \
  --port 30010
```

Prompt rewriting is enabled by default for LongCat-Image. It adds an
autoregressive Qwen2.5-VL pass before diffusion; set
`enable_prompt_rewrite=false` on a request when lower latency is more important
than rewritten prompt quality.

## 4. Generate an image

```python Python theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://127.0.0.1:30010/v1")

response = client.images.generate(
    model="meituan-longcat/LongCat-Image",
    prompt="A quiet bookshop on a rainy evening, warm light in the windows",
    n=1,
    response_format="b64_json",
)

image_bytes = base64.b64decode(response.data[0].b64_json)
with open("longcat_image.png", "wb") as f:
    f.write(image_bytes)
```

To skip prompt rewriting with the OpenAI client, pass the model-specific request
field through `extra_body`:

```python Python theme={null}
response = client.images.generate(
    model="meituan-longcat/LongCat-Image",
    prompt="A quiet bookshop on a rainy evening",
    extra_body={"enable_prompt_rewrite": False},
)
```

## 5. Memory placement

Use the unified component-residency selector when the complete pipeline does
not fit on the accelerator. For example, keep the repeatedly used DiT resident
while moving auxiliary components to CPU between stages:

```bash Command theme={null}
sglang serve \
  --model-path meituan-longcat/LongCat-Image \
  --component-residency dit=resident text_encoder=component-offload vae=component-offload \
  --pin-cpu-memory \
  --port 30010
```

See [Component Residency](/docs/sglang-diffusion/api/cli#component-residency)
for mode semantics and compatibility with the existing CPU-offload flags.


# LongLive 2.0
Source: https://docs.sglang.io/cookbook/diffusion/LongLive/LongLive-2.0

Serve LongLive 2.0 distilled text-to-video and image-to-video models with SGLang-diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[LongLive 2.0](https://nvlabs.github.io/LongLive/LongLive2/) is NVIDIA's 4-step text/image-to-video model distilled from Wan2.2-TI2V-5B. Its main strength is extending few-step causal generation across prompt changes, so a single request can produce multi-shot sequences without paying a full diffusion schedule for every shot.

Choose it for low-step long or multi-shot generation rather than maximum one-shot fidelity. The SGLang path uses a Diffusers conversion of the official weights, and scene continuity still depends on prompt-block and sink settings; validate transitions on the target storyboard.

The model weights use the NVIDIA Open Model License. See the [paper](https://arxiv.org/abs/2605.18739) and [GitHub repository](https://github.com/NVlabs/LongLive) for training details.

## 3. Deployment

```bash Command theme={null}
sglang serve --model-path Rabinovich/LongLive-2.0-5B-Diffusers
```

In `auto` mode, GPUs with at least 60 GiB available keep the DiT, text encoder,
and VAE resident. This uses about 44 GiB on H200 for the 832x480 preset and
avoids moving encoder and decoder layers from host memory on every request.
Smaller GPUs retain the layerwise-offload defaults.

If the GPU runs out of memory, move the text encoder, VAE, and DiT to CPU between stages:

```bash Command theme={null}
sglang serve \
  --model-path Rabinovich/LongLive-2.0-5B-Diffusers \
  --dit-cpu-offload \
  --text-encoder-cpu-offload \
  --vae-cpu-offload
```

`Rabinovich/LongLive-2.0-5B-Diffusers` is the Diffusers-format conversion of the official `Efficient-Large-Model/LongLive-2.0-5B` weights.

Intel Xeon CPU deployment is supported with `SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu` and the selected LongLive checkpoint.

```bash Command theme={null}
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path Rabinovich/LongLive-2.0-5B-Diffusers
```

## 4. Generation

### 4.1 Single prompt

Generate one clip without starting a server:

```bash Command theme={null}
sglang generate \
  --model-path Rabinovich/LongLive-2.0-5B-Diffusers \
  --prompt "A quiet street at dusk" \
  --num-frames 61 \
  --save-output \
  --output-path outputs
```

61 frames is 16 latent frames, which is two causal blocks of 8.

### 4.2 Multi-shot long video

Multi-shot prompts are sampling parameters, so pass them through the Python API:

```python Python theme={null}
from sglang import DiffGenerator

gen = DiffGenerator.from_pretrained("Rabinovich/LongLive-2.0-5B-Diffusers")
result = gen.generate(sampling_params_kwargs={
    "shot_prompts": [
        "A husky walks down a sunlit hallway.",
        "The husky turns and looks at the camera.",
        "Two dogs play together on a carpet.",
    ],
    "chunks_per_shot": 4,
    "num_frames": 381,  # 3 shots x 4 chunks x 8 = 96 latent frames -> 381 frames
    "scene_cut_prefix": "The scene transitions. ",
    "multi_shot_sink": True,
    "multi_shot_rope_offset": 8.0,
    "save_output": True,
    "output_path": "outputs",
})
```

Each shot runs for `chunks_per_shot` causal blocks before the next prompt is used. The multi-shot defaults mirror the original LongLive prompt-block settings.

### 4.3 Key parameters

These are SGLang request parameters. Original LongLive configs use latent-frame `num_output_frames`; SGLang exposes output-video `num_frames`.

* `num_frames`: 61 in the examples. This maps to 16 latent frames, while the original release config defaults to 128 latent frames.
* `num_inference_steps`: 4, matching original `sampling_steps`.
* `guidance_scale`: 1.0, matching the original inference config.
* `height` / `width`: 704 / 1280 by default, matching original latent H/W 44 / 80 with 16x spatial compression.
* `shot_prompts`, `chunks_per_shot`, `scene_cut_prefix`, `multi_shot_sink`, and `multi_shot_rope_offset`: SGLang request fields for the original prompt-block and multi-shot behavior.

### 4.4 Image-to-video

Pass a first frame with `--image-path` to condition the clip on an image:

```bash Command theme={null}
sglang generate \
  --model-path Rabinovich/LongLive-2.0-5B-Diffusers \
  --prompt "A quiet street at dusk" \
  --image-path first_frame.png \
  --num-frames 61 \
  --save-output \
  --output-path outputs
```

The image is used as the first-frame condition.

## 5. Notes

* `num_frames` must map to a whole number of causal blocks. The latent frame count is `(num_frames - 1) / 4 + 1` and must be divisible by 8. For example, 61, 125, and 189 frames give 16, 32, and 48 latent frames.
* SGLang supports T2V sizes 1280x704, 704x1280, 832x480, and 480x832.
* I2V request images follow the Wan TI2V preprocessing path in SGLang. This is different from the original LongLive dataset resize path.
* For multi-shot runs, set `num_frames` to match `len(shot_prompts) * chunks_per_shot * 8` latent frames, that is `num_frames = (len(shot_prompts) * chunks_per_shot * 8 - 1) * 4 + 1`.

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# MOVA
Source: https://docs.sglang.io/cookbook/diffusion/MOVA/MOVA

Deploy MOVA with SGLang - simultaneous video and audio generation with asymmetric dual-tower architecture, precise lip-sync, and environment-aware sound effects.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

Set the output directory before starting the server:

```bash theme={null}
export SG_OUTPUT_DIR=/root/output_mova
mkdir -p "$SG_OUTPUT_DIR"
```

## 2. Model capabilities

[MOVA](https://github.com/OpenMOSS/MOVA) generates video and audio together with an asymmetric dual-tower model connected by bidirectional cross-attention. Its strongest use cases are speaking subjects, visible sound-producing events, and scenes where ambient audio must track the picture rather than be synthesized by a later cascade.

The public 360p and 720p checkpoints both generate up to 8 seconds. Choose 360p for the lighter deployment and 720p for output resolution; MOVA is less suitable when the task needs long-form continuity or the richer image/video/audio reference conditioning provided by H3.

| Checkpoint | Best fit | Output limit |
| - | - | - |
| `OpenMOSS-Team/MOVA-360p` | Faster, lower-memory joint audiovisual generation | Up to 8 seconds at 360p |
| `OpenMOSS-Team/MOVA-720p` | Higher-resolution lip-sync and environment audio | Up to 8 seconds at 720p |

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

MOVA supports both online serving and CLI generation modes. The recommended launch configurations vary by hardware and resolution.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements.

* `--num-gpus`: Number of GPUs to use
* `--tp`: Tensor parallelism size (should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
* `--ring-degree`: The degree of ring attention-style SP in USP
* `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
* `--adjust-frames`: Whether to adjust frames automatically (set to `false` for MOVA)
* `--enable-torch-compile`: Enable torch.compile for faster inference

## 4. API Usage

For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api).

### 4.1 CLI Generation (sglang generate)

```bash Command theme={null}
sglang generate \
  --model-path OpenMOSS-Team/MOVA-720p \
  --prompt "A man in a blue blazer and glasses speaks in a formal indoor setting, \
  framed by wooden furniture and a filled bookshelf. \
  Quiet room acoustics underscore his measured tone as he delivers his remarks. \
  At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \
  --image-path "<YOUR-IMAGE-PATH>" \
  --adjust-frames false \
  --num-gpus 8 \
  --ring-degree 2 \
  --ulysses-degree 4 \
  --num-frames 193 \
  --fps 24 \
  --seed 67 \
  --num-inference-steps 25 \
  --enable-torch-compile \
  --save-output
```

### 4.2 Generate a Video

```bash Command theme={null}
curl -X POST "http://0.0.0.0:30002/v1/videos" \
  -F "prompt=A man in a blue blazer and glasses speaks in a formal indoor setting, framed by wooden furniture and a filled bookshelf. Quiet room acoustics underscore his measured tone as he delivers his remarks. At one point, he says, \"I would also believe that this advance in AI recently wasn't unexpected.\"" \
  -F "input_reference=@<YOUR-IMAGE-PATH>" \
  -F "size=640x352" \
  -F "num_frames=193" \
  -F "fps=24" \
  -F "seed=67" \
  -F "guidance_scale=5.0" \
  -F "num_inference_steps=25" \
  -o create_video.json
```

### 4.3 Advanced Usage

#### 4.3.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit).

**Basic Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path OpenMOSS-Team/MOVA-720p
```

**Advanced Usage**

* DBCache Parameters: DBCache controls block-level caching behavior:

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

Combined Configuration Example:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path OpenMOSS-Team/MOVA-720p
```

#### 4.3.2 CPU Offload

* `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
* `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
* `--vae-cpu-offload`: Use CPU offload for VAE.
* `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".

## 5. Benchmark

### 5.1 Speedup Benchmark

#### 5.1.1 Generate a video

Test Environment:

* Hardware: NVIDIA H200 x 8
* git revision: 443b1a8
* Model: OpenMOSS-Team/MOVA-720p

**Server Command**:

```bash Command theme={null}
sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \
  --adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \
  --tp 1 --enable-torch-compile
```

**Benchmark Command**:

```bash Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --task image-to-video --dataset vbench --num-prompts 1 --max-concurrency 1 \
    --port 30002
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    image-to-video
Model:                                   OpenMOSS-Team/MOVA-720p
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  590.76
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.00
Latency Mean (s):                        590.7549
Latency Median (s):                      590.7549
Latency P99 (s):                         590.7549
--------------------------------------------------
Peak Memory Max (MB):                    74996.00
Peak Memory Mean (MB):                   74996.00
Peak Memory Median (MB):                 74996.00
============================================================
```

#### 5.1.2 Generate videos with high concurrency

**Server Command**:

```bash Command theme={null}
sglang serve --model-path OpenMOSS-Team/MOVA-720p --port 30002 \
  --adjust-frames false --num-gpus 8 --ring-degree 2 --ulysses-degree 4 \
  --tp 1 --enable-torch-compile
```

**Benchmark Command**:

```bash Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --task image-to-video --dataset vbench --num-prompts 20 --max-concurrency 20 \
    --port 30002
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# MiniMax-H3
Source: https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3

Run native MiniMax-H3 video-and-audio generation with SGLang Diffusion.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation),
then choose a verified recipe below. Docker is recommended where the picker offers it.
Setup changes the deployment; Server and Request expose
orthogonal startup and sampling choices.

> [!NOTE]
>
> The generated Server command already includes the recommended encoder policy.
> Change a Server option only for a deliberate trade-off; Request options do not
> reload the model.

The Docker form uses the image's bundled diffusion dependencies without reinstalling
them at startup. For conditioned requests, set **Host media directory**
under **Variables**; the builder mounts it read-only at `/data/minimax-h3`.
AMD and Grace Blackwell currently offer the Python form; the other NVIDIA
recipes also offer Docker.

To use ModelScope through the same normal `sglang serve` path, prefix the copied
command with `SGLANG_USE_MODELSCOPE=true` and replace the model path with
`MiniMax/MiniMax-H3`. Keep the selected variant and topology flags unchanged.

For platform-specific installation details, see the
[SGLang Diffusion installation guide](/docs/sglang-diffusion/installation).

## 2. Model capabilities

[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) is a native joint video-and-audio model for text-to-video-and-audio, first/last-frame control, and multimodal reference conditioning. Its main strength is producing the picture and stereo soundtrack together, so speech, music, ambient sound, and visible events can stay aligned without a separate audio-generation pass.

Choose H3 when synchronized audiovisual output or reference-driven generation matters more than a lightweight deployment. The released recipe targets a 768-pixel short edge at 24 fps for 4–15 seconds, and its capabilities are split across two checkpoint partitions; serving every mode therefore requires separate FL2VA and Ref2VA deployments.

| Task | `task` value | Conditioning |
| - | - | - |
| Text to video and audio | `t2va` | Text prompt only |
| First/last frame to video and audio | `fl2va` | First frame, last frame, or both |
| Reference to video and audio | `ref2va` | Image, video, and audio references, optionally combined with first/last keyframes |

Video-to-video (V2V) is a supported `ref2va` use case, not a fourth task
value. Run the `Ref2VA` partition and provide a video reference in
`conditions`. A hybrid `ref2va` request may also include the same ordered
first/last keyframes accepted by `fl2va`, but it must still contain at least one
reference condition.

Use the selected Hub's root model ID: `MiniMaxAI/MiniMax-H3` on Hugging Face
or `MiniMax/MiniMax-H3` on ModelScope. Select the checkpoint variant with
`--model-variant`: `fl2va` serves both `t2va` and `fl2va`, while `ref2va`
serves reference-conditioned requests. SGLang owns the checkpoint-directory
mapping; do not point `--model-path` at a manually downloaded subdirectory.
The same FL2VA / Ref2VA split applies in
[ComfyUI integrated mode](#comfyui-integrated-mode).

> [!WARNING]
>
> Review the license and usage terms in the MiniMax-H3 model card before production or commercial use. SGLang support does not grant additional model usage rights.

## 3. Deployment details

The builder accepts legal custom GPU counts and topologies, marking them
**Unverified** until the exact recipe has completed end-to-end validation.
Static H3 head or partition violations disable Copy before they reach
`sglang serve`.

### Grace Blackwell

Select **GB300** or **GB200** in the builder. The default is one Linux ARM64
host with four GPUs, Ulysses4, and resident weights. This follows the
[four-GPU compute-tray layout](https://docs.nvidia.com/dgx/dgxgb200-user-guide/hardware.html);
an NVL72 rack is not a single 72-GPU process host.

For two four-GPU hosts, set **Nodes = 2** and **GPUs / node = 4**. The builder
emits Ulysses4 inside each host, Ring2 across hosts, and replicated encoders.
Set the same reachable **Head address** on both hosts and use node ranks 0 and

1. Run the generated command once per host, with the same SGLang revision and
   dependencies. Let `sglang serve` resolve and download the selected weights.

GB200 is a **derived, unverified recipe**, not a measured GB300-equivalent
performance claim. Keep the native precision and platform-default attention
for the baseline; the B200/B300 quantization measurements and H200 Cache-DiT
quality audit do not establish coverage on Grace Blackwell.

GB300 end-to-end coverage currently includes single-host FL2VA text-to-video
with audio, native precision, eager execution, and `quality="exact"`.
Ref2VA and multi-host commands remain **Unverified**. See the
[GB300 measurements](#gb300-single-host) for the tested workload.

### Checkpoint and adapter formats

Start with the command emitted by the builder. Every row below is an overlay on
the same native SGLang pipeline; component repositories contribute their own
config and weights, while weight files retain the base component config. Storage
layout and inference behavior are separate contracts: for example, a PEFT file
may be either a normal style adapter or a timestep-distilled Turbo adapter.

| Scope | Format or variant | Add to the base command | Contract |
| - | - | - | - |
| Full model | Official mixed BF16/FP32, CFG-distilled | `--model-variant fl2va` or `--model-variant ref2va` | Lossless reference and consistency GT path. CFG distillation removes the negative branch; it is not the few-step timestep distillation used by Turbo releases. |
| DiT | Official Diffusers component layout | `--component-paths.transformer MiniMaxAI/MiniMax-H3/transformer` (`fl2va`) or `.../transformer_ref` (`ref2va`) | Loads the official component through the native SGLang graph; no Diffusers runtime fallback. |
| DiT | [AdaLN-pruned Diffusers component](https://huggingface.co/multimodalart/MiniMax-H3-Pruned) | `--component-paths.transformer multimodalart/MiniMax-H3-Pruned/transformer` or `.../transformer_ref` | Approximate curve-AdaLN architecture; its config and basis metadata are loaded natively. |
| DiT | [Full or AdaLN-pruned](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models), or [LoRA-merged/remixed](https://huggingface.co/FX-FeiHou/MiniMax-H3-Remix) BF16 safetensors | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Weight-only override for a native full/pruned H3 layout. Match the FL2VA/Ref2VA partition; pruned, merged, or dtype-converted exports are approximate, and any author-specific sampler remains a separate requirement. |
| DiT | FL/Ref hybrid fine-tunes, including [Singularity v1.3](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity) (full or pruned INT8) | Replace `--model-variant` with `hybrid`; add `--component-weights-paths.transformer OWNER/REPO/FILE.safetensors` | Explicit merged-weight deployment serving `t2va`, `fl2va`, and `ref2va` on one native pipeline. Quantization and pruning are detected from the file; INT8 requires `comfy-kitchen`. See [Singularity](#singularity-hybrid-weights). |
| DiT | Comfy FP8 or self-describing MXFP8 safetensors | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Per-layer metadata selects static/dynamic FP8 or MXFP8 automatically. |
| DiT | ConvRot INT8, W4A8, W4A4, or mixed W4A4+INT8 safetensors ([INT8](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models), [W4A8](https://huggingface.co/Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI), [W4A4](https://huggingface.co/Merserk/MiniMax-H3-INT4-ConvRot)) | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Auto-detected. INT8 layers run on SGLang's JIT-compiled ConvRot ops on CC 9.0, 10.0, 12.0 and 12.1 and on `comfy-kitchen` elsewhere; W4A8 and W4A4 layers require `comfy-kitchen`. TP must preserve each file's ConvRot group boundaries. |
| DiT | [NVFP4, optionally mixed with INT8 or FP8](https://huggingface.co/Abiray/Minimax-H3-nvfp4-INT4-INT8-Convrot) | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Auto-detected; NVFP4 execution requires NVIDIA compute capability 10.0+. |
| DiT | [AutoRound W4A16 component](https://huggingface.co/Ar4ikov/MiniMax-H3-transformer-W4A16-RTN) | `--component-paths.transformer Ar4ikov/MiniMax-H3-transformer-W4A16-RTN` | Self-describing Diffusers component; SGLang reuses the SRT GPTQ/Marlin backend. The linked export is FL2VA. |
| DiT | GGUF, full or AdaLN-pruned ([full](https://huggingface.co/leejet/MiniMax-H3-GGUF), [pruned](https://huggingface.co/unsloth/MiniMax-H3-GGUF)) | `--component-weights-paths.transformer OWNER/REPO/FILE.gguf` | CUDA capacity path; aligned TP and layerwise offload are supported, FSDP and LoRA are not. |
| Text encoder | Architecture-compatible Qwen3-VL BF16 finetune ([Heretic example](https://huggingface.co/llmfan46/Qwen3-VL-32B-Instruct-ultra-uncensored-heretic)) | `--component-paths.text_encoder OWNER/REPO` | Reuses the native H3 extractor, including its vision tower and layer-50 hidden-state selection. Finetuning changes conditioning, not the sampling schedule. |
| Text encoder | [Serialized FP8 component](https://huggingface.co/Qwen/Qwen3-VL-32B-Instruct-FP8) | `--component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8` | Only eligible language-model linears use FP8; embeddings, norms, and the vision tower keep their declared precision. |
| Text encoder | ConvRot INT8, W4A8, or W4A4 safetensors ([INT8](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders), [Heretic INT8](https://huggingface.co/ethanfel/Qwen3-VL-32B-Ultra-Heretic-H3-ComfyUI-INT8-ConvRot), [W4A8](https://huggingface.co/Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI), [W4A4](https://huggingface.co/Merserk/MiniMax-H3-INT4-ConvRot)) | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Auto-detected. INT8 layers run on SGLang's JIT-compiled ConvRot ops on CC 9.0, 10.0, 12.0 and 12.1 and on `comfy-kitchen` elsewhere; W4A8 and W4A4 layers require `comfy-kitchen`. INT8/W4A8 files may include a scalar-scale INT8 embedding, which requires `comfy-kitchen`. Unmarked tensors retain their declared precision. |
| Text encoder | Comfy NVFP4 with dynamic activation quantization ([Heretic example](https://huggingface.co/sakamakismile/Qwen3-VL-32B-Heretic-MiniMax-H3-NVFP4)) | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Requires `comfy-kitchen` and NVIDIA compute capability 10.0+. Per-layer metadata selects native NVFP4 matmul; scalar/row-wise INT8 embeddings remain packed. |
| Text encoder | [NVFP4-AWQ](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders) or [Quanto qint8](https://huggingface.co/DeepBeepMeep/MiniMax-H3/tree/main/Qwen3-VL-32B-Instruct) safetensors | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Memory-oriented formats: compressed storage is restored, then each active matrix uses BF16/FP16 compute. |
| Text encoder | [GGUF Qwen3-VL](https://huggingface.co/DeepBeepMeep/MiniMax-H3/tree/main/Qwen3-VL-32B-Instruct) | `--component-weights-paths.text_encoder OWNER/REPO/FILE.gguf` | CUDA capacity path with encoder TP/layerwise support; encoder FSDP is not supported. |
| Text encoder | [Compact Qwen3-VL 4B/8B + ClipProj](https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3) | `--component-paths.text_encoder ENCODER_REPO --component-paths.conditioning_projection PROJECTION.safetensors` | Approximate conditioning replacement. A separate weight-only override may quantize the selected small encoder. |
| DiT or adapter | Timestep-distilled Turbo, as merged weights or LoRA ([Larry](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora), [LightX2V](https://huggingface.co/lightx2v/Minimax-h3-Turbo), [merged Ref2VA INT8 example](https://huggingface.co/PulpCut/MiniMax-H3-Ref2VA-Turbo-INT8-ConvRot)) | Use `--component-weights-paths.transformer ...` for merged weights, or `--lora-path OWNER/REPO --lora-weight-name FILE --lora-merge-mode auto` for LoRA | Few-step semantic variant. Pin the exact FL2VA/Ref2VA file and its NFE/sigma schedule, scale, and alpha; storage-format detection does not infer sampling behavior. See [LoRA recipes](#5-lora-recipes). |
| Adapter | Style, subject, or behavior LoRA ([example](https://huggingface.co/fal/MiniMax-H3-Realism-People-LoRA)) | `--lora-path OWNER/REPO [--lora-weight-name FILE] --lora-merge-mode auto` | Native fused and Diffusers/PEFT layouts are normalized at load time. Keep the base schedule unless the author specifies another one, and preserve any trigger phrase, scale, and alpha metadata. |

The rows compose rather than enumerate every cross-product. A Turbo-merged INT8
ConvRot checkpoint, for example, must satisfy both the Turbo sampling contract
and the ConvRot storage/backend contract.

H3's Qwen3-VL text encoder also handles image understanding; there is no separate
`image_encoder` component. Select the conditioning checkpoint, not an optional
generation-tail file for prompt rewriting. The `uncensored` or `Heretic` label
describes a weight modification, not a separate loader or a guarantee of output quality.

For H3, the registered component names are `transformer`, `text_encoder`,
`video_vae`, and `audio_vae`. The shorter `--transformer-weights-path` and
`--text-encoder-path` aliases remain supported. `conditioning_projection` is an
H3 text-encoder sidecar key, not a standalone model component. Plain
video/audio VAE safetensors can use `--component-weights-paths.video_vae` or
`--component-weights-paths.audio_vae`, but SGLang does not currently advertise
a native quantized H3 VAE format.

Pre-quantized files are self-describing: do not combine those rows with
`--quantization` or `--component-quantizations.*`. Only byte-identical official
full weights—whether loaded from the original model, Diffusers component, or a
weight-only layout—belong to the consistency GT. Pruned, quantized,
compact-encoder, and LoRA routes are outside that baseline and the audited
`quality="high"` contract. Packed and per-layer mixed formats reject FSDP unless
their row says otherwise; see
[Quantization](/docs/sglang-diffusion/quantization) for backend-wide constraints.

LoRA tensors alone do not make an execution-coupled release portable. Sparse-
attention/SLA adapters and causal-streaming adapters such as RAVEN also require
their matching attention or streaming pipeline; they are not standard H3 LoRA
overlays in SGLang. Likewise, a remixed checkpoint that prescribes a custom
sampler is only covered when that sampler contract can be reproduced—the fact
that its safetensors layout loads is not sufficient.

For a four-card H200 host, keep the full BF16/FP32 model resident by default.
The model fits without FSDP, so this path avoids the per-block parameter
all-gathers of the memory-oriented FSDP profile:

```bash 4×H200 resident theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --encoder-parallel auto \
  --performance-mode speed \
  --port 30010
```

Pure Ulysses4 is also the faster measured topology on H200, not just a
capacity default. The 4×H100 TP2 + Ulysses2 recipe below fits on 141 GB H200
cards, but it replaces the Ulysses all-to-all exchange with two per-block
tensor-parallel all-reduces and measured slower end-to-end, at about 30 GB
lower peak memory per GPU. See the **H200 topology comparison** in the
Benchmarks section for the measured numbers; treat TP2 + Ulysses2 on H200 as
a deliberate memory trade, not a latency default.

For 4×H100 80 GB, balance the large packed activation with resident weight
sharding. TP2 + Ulysses2 was the fastest measured lossless topology while the
Qwen encoder still folds across all four GPUs:

```bash 4×H100 fastest theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --tp-size 2 \
  --ulysses-degree 2 \
  --encoder-parallel auto \
  --performance-mode speed \
  --port 30010
```

Pure Ulysses4 could not keep the full pipeline resident on 80 GB H100s. Use
`--tp-size 4 --ulysses-degree 1` when lower resident memory matters more than
the last few percent of latency. FSDP remains a verified capacity option, but
its per-block weight all-gathers do not make it the H100 speed default:

```bash 4×H100 FSDP capacity theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --encoder-parallel auto \
  --performance-mode speed \
  --use-fsdp-inference true \
  --port 30010
```

For a two-card RTX 5090 host, use TP2 and keep 20 DiT blocks
resident. Layerwise placement is lossless: it changes parameter placement and
transfer scheduling, not the BF16/FP32 denoising or VAE math. This is the
fastest measured 32 GB operating point:

```bash 2×RTX 5090 fastest lossless theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 2 \
  --tp-size 2 \
  --ulysses-degree 1 \
  --encoder-parallel auto \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder,vae \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 20 \
  --enable-torch-compile false \
  --port 30010
```

The DiT residency and prefetch knobs apply only to the repeatedly executed DiT
blocks. The text encoder and the video VAE decoder blocks use one-layer
prefetch with zero resident layers. The video VAE encoder stays resident
because its indexed down blocks cannot host executable layerwise hooks; the
roughly 577 MiB audio VAE also stays resident because offloading it only adds
transfer overhead. This exact recipe was validated on
2× RTX 5090 (32 GB each) and a 377 GiB host; use a 384 GiB-class machine. The
latency and memory comparison is collected in the benchmark section below.

For a single 24 GB consumer card (RTX 4090), stream the DiT and text encoder
and quantize DiT linear layers online with `convrot_int8`. Keep `vae` out of
`--layerwise-offload-components`: putting the VAE decoder in layerwise
offload re-streams about 9 GiB on each of 167 decode tiles. Default
attention stays `fa` (exact). Approximate backends are opt-in; see
[Attention Backends](/docs/sglang-diffusion/attention_backends#sage-then-sol-hybrid).
On the RTX 4090 (CC 8.9) `convrot_int8` runs on `comfy-kitchen`, so install it
first (`pip install comfy-kitchen`); on CC 9.0, 10.0, 12.0 and 12.1 it uses
SGLang's JIT-compiled ConvRot ops instead (built once per machine on first use).

```bash 1×RTX 4090 24GB theme={null}
sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --quantization convrot_int8 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output
```

The same flags work on `sglang serve`. Drop `--quantization` for the BF16
baseline; everything else stays identical. GPU peak stays about 18 GB
either way because streaming offload is set by the offload buffers and VAE
decode, not the weight dtype.

To skip DiT steps on 4090 while keeping `fa`, add Spectrum to the generate
command above. Do not combine it with `quality="high"` or Cache-DiT.
On this card, 50-step BF16+FA (**1008.6 s**) drops to **285.3 s (3.53×)**
with exact attention, up to **219.0 s (4.61×)** with Sage→Sol hybrid +
Spectrum. See the 4090 table in Benchmarks.

```bash theme={null}
  --enable-spectrum \
  --spectrum-warmup-steps 11 \
  --spectrum-window-size 5 \
  --spectrum-flex-window 1.0
```

The first launch resolves every selected source through the normal Hub path. If
a repository requires authentication, export a Hugging Face token in the
server environment; no manual pre-download is required.

For MiniMax-H3, `--performance-mode speed` deliberately keeps the DiT eager.
The current `torch.compile` path changes the model's numerical output, so no
recommended lossless preset enables it implicitly. An explicit
`--enable-torch-compile true` remains available for controlled experiments, but
do not use it to generate consistency ground truth.

### ComfyUI integrated mode

The [ComfyUI plugin](/docs/sglang-diffusion/comfyui) can run H3 as a per-step
DiT backend. ComfyUI keeps CLIP, VAE, and the sampler loop. SGLang loads only
the transformer under `--comfyui-mode`.

Copy `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion` into ComfyUI
`custom_nodes/`. Load a shipped graph and set `SGLDOptions.model_type` to
`minimax_h3`:

| Graph | Task | DiT filename |
| - | - | - |
| `minimax_h3_t2v_sgld.json` | T2V / I2VA / FL2VA | `minimax_h3_fl2va_*.safetensors` |
| `minimax_h3_r2v_sgld.json` | Reference-to-video | `minimax_h3_ref2va_*.safetensors` |
| `minimax_h3_t2v_sgld_upscaler.json` | Low-res then 3D ×2 | `fl2va` DiT |

Pruned BF16 / FP8 files from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models)
are the usual loader inputs. GGUF goes in `SGLDOptions.transformer_weights_path`.
On a 24 GB card put the 32B CLIP on CPU and enable DiT layerwise offload; use
`convrot_int8` or GGUF rather than the full unpruned BF16 DiT.

### Singularity hybrid weights

[Singularity](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity) is an
FL/Ref fusion fine-tune, not a complete pipeline repository. Keep the official
model ID for the encoders and VAEs, and select one exact transformer file:

```bash 2×B300 hybrid checkpoint theme={null}
pip install comfy-kitchen

WEIGHTS=WarmBloodAban/Minimax-h3_Singularity/Minimax-h3_Singularity_ref2va_Pruned_v1.3_int8.safetensors
sglang serve --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant hybrid \
  --component-weights-paths.transformer "$WEIGHTS" \
  --num-gpus 2 --ulysses-degree 2 --performance-mode speed \
  --port 30000
```

For the full INT8 checkpoint, replace the filename with
`Minimax-h3_Singularity_ref2va_v1.3_int8.safetensors`. SGLang downloads the
selected file automatically. Do not add an online quantization override.
`hybrid` requires an explicit weight override; it does not convert the official
Ref2VA weights into a multimode model. Request `task` and `conditions` retain
their [usual meanings](#4-generate-video-and-audio), including `conditions: []`
for `t2va` and keyframes for `fl2va`.

Start with `quality: "lossless"` and the standard 50-point schedule. Here
`exact` disables additional request-time approximations; it does not undo
INT8 quantization, pruning, or fine-tuning. The author's recommended Ref2V
Turbo LoRA is a separate optional adapter, not an automatically inferred
4-step schedule. Reference fidelity can differ from the official model; see
the [author's guidance](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity/discussions/1)
before applying the [LoRA sampling contract](#5-lora-recipes).

### Advanced: precomputed AdaLN cache

The [model card](https://huggingface.co/MiniMaxAI/MiniMax-H3) notes that about
13B H3 parameters are AdaLN branches whose outputs can be precomputed for
inference. The public base checkpoint contains the original branches, not a
ready-to-use cache. SGLang therefore keeps the standard path as the default.

> [!WARNING]
>
> This is an experimental deployment path. It is intentionally disabled unless
> you provide an explicitly generated cache; end-to-end numerical and peak-memory
> validation remains required before using it in production.

When an inference-only deployment has a fixed sampling schedule, build a cache
from the already materialized transformer directory on CUDA, then pass it to
the usual `sglang serve` command. This does not alter the denoising formula:
the cache stores the BF16 outputs of the original AdaLN linears.

```bash Command theme={null}
python -m sglang.multimodal_gen.tools.build_minimax_h3_adaln_cache \
  --transformer-path "$TRANSFORMER_PATH" \
  --model-variant fl2va \
  --mode t2va \
  --num-inference-steps 50 \
  --flow-shift 12 \
  --audio-flow-shift 3 \
  --output /models/minimax-h3-fl2va-adaln-50step.safetensors

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --minimax-h3-adaln-cache-path /models/minimax-h3-fl2va-adaln-50step.safetensors \
  --num-gpus 4 \
  --tp-size 2 \
  --ulysses-degree 2 \
  --port 30010
```

`$TRANSFORMER_PATH` is the `FL2VA/transformer` or `Ref2VA/transformer`
directory in the normal SGLang/Hugging Face snapshot; the builder never
downloads a second copy. A cache only covers the scheduler settings used to
create it, including its mode, step count, flow shifts, and condition noise
values. SGLang rejects a request outside that coverage instead of silently
changing conditioning. Cache mode supports the matching unquantized checkpoint
only.

### Advanced: online AdaLN rebuild with a host cache

`--minimax-h3-adaln-online true` needs no prebuilt artifact: the server drops
the 24.2 GiB of `adaln_proj` weights from the GPU and computes each request's
AdaLN outputs from the checkpoint on demand, bit-exact with the resident-weight
path. Because a rebuild pass streams the whole 24.2 GiB, the first request of
every new `(task shape, num_inference_steps, flow_shift, audio_flow_shift)`
combination pays several seconds; after that the plans are served from a
64-slot GPU slab (LRU per plan) backed by a pinned host cache
(`--minimax-h3-adaln-host-cache-gb`, LRU per schedule, default 8 GB per
rank), so mixed-schedule serving does not re-read the checkpoint. Plan sets
that exceed the host budget are simply recomputed on their next occurrence.
Expert escape hatches live in environment variables:
`SGLANG_DIFFUSION_MINIMAX_H3_ADALN_GPU_PLANS` resizes the GPU slab (needed
only beyond 65 inference steps) and `SGLANG_DIFFUSION_MINIMAX_H3_ADALN_FP32`
computes the one-time projections in fp32 (experimental; not bit-comparable
to resident weights, validate end-to-end before production). LoRA adapters
that modify `adaln_proj` are rejected in both cache modes rather than
silently ignored.

Both cache modes hold values derived from `adaln_proj`, so a runtime weight
update is only accepted when the cache can follow it: online mode takes a disk
update whose target directory carries native `adaln_proj` safetensors, and
rejects everything else (tensor updates, and directories without those
tensors) before a single weight is written. A sidecar is built offline and
cannot be regenerated in the server, so weight updates are rejected outright;
rebuild the sidecar against the new weights and restart.

### Serve MiniMax-H3 on Ascend NPUs

For Ascend NPU, follow the
[NPU installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation)
before starting the server.

The Ascend commands below explicitly enable the Cache-DiT configuration used
for the reported performance measurements. Remove these `SGLANG_CACHE_DIT_*`
variables to use lossless denoising. See the **Ascend NPU topology comparison**
in the Benchmarks section for the measured eight- and four-NPU latency.

The measured latency configuration also passes `--dit-cpu-offload false` to
keep the transformer resident on the NPUs. Omit this flag when lower device
memory usage is more important than avoiding CPU-to-NPU transfer latency.

For an eight-NPU host, the validated topology is TP2 + SP4 with Laser
Attention. Use Ascend Flash Attention by replacing `laser_attn` with `fa`.

```bash 8-NPU theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
HCCL_BUFFSIZE=256 sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-type diffusion \
  --model-variant fl2va \
  --dit-cpu-offload false \
  --num-gpus 8 \
  --tp-size 2 \
  --sp-degree 4 \
  --attention-backend laser_attn \
  --port 30088 \
  --component-residency text_encoder=layerwise-offload
```

For a four-NPU host, use TP2 + SP2:

```bash 4-NPU theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
HCCL_BUFFSIZE=256 sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-type diffusion \
  --model-variant fl2va \
  --dit-cpu-offload false \
  --num-gpus 4 \
  --tp-size 2 \
  --sp-degree 2 \
  --attention-backend laser_attn \
  --port 30088 \
  --component-residency text_encoder=layerwise-offload
```

## 4. Generate video and audio

`num_inference_steps: N` executes **N denoising updates** over N+1 sigma
points, including the terminal zero. Reference implementations that count
sigma points need N+1 to reproduce this schedule. Older SGLang versions ran
N-1 updates for the same request; use N-1 here to preserve their sampling
trajectory. Changing the step count changes the trajectory, not just latency.

MiniMax-H3 uses the asynchronous OpenAI-compatible video endpoint. Choose a
generation mode below, submit a job, poll its status, and then download the
completed MP4.

**T2VA**

MiniMax-H3 supports output durations from 4 through 15 seconds, inclusive. The
following request keeps the verified 5-second profile at a 768-pixel short
edge. MiniMax-H3 resolves the aligned output canvas and frame count from
`target`.

```bash Command theme={null}
video_id=$(
  curl -sS -X POST http://127.0.0.1:30010/v1/videos \
    -H "Content-Type: application/json" \
    -d '{
      "model": "MiniMaxAI/MiniMax-H3",
      "prompt": "At night, while their owner sleeps in a bedroom, three cats march in loudly playing tiny brass instruments, then abruptly file out.",
      "seconds": 5,
      "task": "t2va",
      "conditions": [],
      "target": {
        "short_edge": 768,
        "aspect_ratio": "16:9",
        "duration_seconds": 5.0
      },
      "num_outputs_per_prompt": 1,
      "num_inference_steps": 50,
      "flow_shift": 12.0,
      "audio_flow_shift": 3.0,
      "seed": 1101
    }' |
    jq -r '.id'
)

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${video_id}" | jq -r '.status')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${video_id}/content" \
  -o minimax-h3-t2va.mp4
```

The output contract is an MP4 containing H.264 video at 24 fps and one AAC stereo audio stream at 32 kHz.

**FL2VA**

For `fl2va`, provide one or two image conditions with role `keyframe`. The supported frame-index sets are `[0]`, `[-1]`, and `[0, -1]`.

The following request uses one server-local first frame. Use
`frame_index: -1` for a last frame, or include both entries for first-and-last
conditioning.

Choose FL2VA when the supplied image should be the actual first or last frame
of the generated clip. Use image-based Ref2VA instead when the image should
guide identity, style, or composition without being preserved as an endpoint;
Ref2VA may recompose or crop the reference.

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMaxAI/MiniMax-H3",
    "prompt": "The supplied frame continues with calm, natural motion and synchronized ambient sound.",
    "seconds": 5,
    "task": "fl2va",
    "conditions": [
      {
        "type": "image",
        "uri": "file:///data/minimax-h3/first-frame.png",
        "role": "keyframe",
        "frame_index": 0
      }
    ],
    "target": {
      "short_edge": 768,
      "aspect_ratio": "auto",
      "duration_seconds": 5.0
    },
    "num_outputs_per_prompt": 1,
    "num_inference_steps": 50,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "seed": 2101
  }'
```

**V2V**

V2V uses the reference-conditioning weights. Launch the server with
`--model-variant ref2va`, keep the request `task` set to `ref2va`, and provide a video
reference in `conditions`. There is no separate `v2v` task value.

Use `type: "video"` when the input may be silent. If the file has a soundtrack,
H3 also uses it as an audio reference. Use `type: "video_audio"` only when both
streams are required; that form rejects an input without audio. The prompt tag
for the visual stream is `<Video 1>`; an available soundtrack is exposed as
`<Audio 1>`.

> [!NOTE]
>
> Ref2VA treats the input video as reference material, not as a pixel-aligned
> edit source. It can resynthesize or reorder motion and cuts, and it does not
> expose a denoising-strength control. Do not rely on it to preserve every source
> frame or exact timing.

Set `conditions[].start_time_seconds` to select a segment from a longer source.
The default is `0`. SGLang seeks the visual stream and soundtrack to the same
offset, then decodes at most the requested target duration in one pass; the
source is not re-encoded into an intermediate clip.

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMaxAI/MiniMax-H3",
    "prompt": "Follow the motion and appearance of <Video 1>, changing the setting to a moonlit bedroom while preserving coherent timing.",
    "seconds": 5,
    "task": "ref2va",
    "conditions": [
      {
        "type": "video",
        "uri": "file:///data/minimax-h3/input.mp4",
        "role": "reference",
        "start_time_seconds": 35.0
      }
    ],
    "target": {
      "short_edge": 768,
      "aspect_ratio": "16:9",
      "duration_seconds": 5.0
    },
    "num_outputs_per_prompt": 1,
    "num_inference_steps": 50,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "seed": 4101
  }'
```

Use `conditions[].uri` for H3 V2V. The generic top-level `video_path`,
`video_url`, and `video_reference` upload fields are not lowered into H3
reference conditions.

**Multimodal Ref2VA**

For `ref2va`, first launch the reference-conditioning capability with
`--model-variant ref2va`, then provide conditions with role `reference`.
Image, video, and audio references can be combined. Material tags in the
prompt use the one-based order for each modality.

An image condition here is semantic reference material rather than a
pixel-aligned first frame. Use the FL2VA tab when animating a screenshot from
that exact starting composition.

Set `reference_resize_mode` on the request to choose how reference images are
resized before encoding:

* `diffusers` (default) scales the short edge to 2048 pixels, including upscaling
  small inputs. This preserves the original preprocessing behavior.
* `max` only downsizes references whose short edge exceeds 2048 pixels.
* `match` downsizes to the target canvas pixel area when needed, without
  upscaling smaller references. Use this with the
  [ModelTC Ref2VA Turbo LoRA](https://github.com/ModelTC/Minimax-H3-Turbo#note-on-reference-image-resizing),
  whose training uses this policy.

All modes scale without cropping, then round each dimension to the nearest
multiple of 32. This rounding can slightly change the aspect ratio. The option
affects Ref2VA image references, not FL2VA keyframes, video, or audio references.
Changing it changes the conditioning input, so keep it fixed for quality and
performance comparisons; the server does not infer it from the adapter name.

```bash Command theme={null}
curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMaxAI/MiniMax-H3",
    "prompt": "Use <Picture 1> as the visual subject and <Audio 1> as the sound reference, with coherent natural motion.",
    "seconds": 5,
    "task": "ref2va",
    "reference_resize_mode": "diffusers",
    "conditions": [
      {
        "type": "image",
        "uri": "file:///data/minimax-h3/reference.png",
        "role": "reference"
      },
      {
        "type": "audio",
        "uri": "file:///data/minimax-h3/reference.mp3",
        "role": "reference"
      }
    ],
    "target": {
      "short_edge": 768,
      "aspect_ratio": "auto",
      "duration_seconds": 5.0
    },
    "num_outputs_per_prompt": 1,
    "num_inference_steps": 50,
    "flow_shift": 12.0,
    "audio_flow_shift": 3.0,
    "seed": 3101
  }'
```

Poll and download any conditioned request with the same job-status and
content endpoints used in the T2VA example. Server-local `file://` URIs must
refer to files visible inside the SGLang server environment.

## 5. LoRA recipes

H3 accepts native fused adapters, standard Diffusers/PEFT adapters, and Kohya
`lora_unet_blocks_*` exports from `networks.lora_minimax_h3`. Native adapters
target modules such as `blocks.*.attn.qkv_proj`; PEFT adapters may instead
provide separate `to_q`, `to_k`, and `to_v` projections and the `default`
adapter namespace. SGLang normalizes all three layouts.

The following pinned FL2VA adapters have distinct purposes:

| Recipe | Repository and pinned file | Request setting | Prompt requirement |
| - | - | - | - |
| Recommended speed/quality balance | [`larryvrh/MiniMax-H3-Turbo-Lora`](https://huggingface.co/larryvrh/MiniMax-H3-Turbo-Lora), `minimax_h3_turbo_v4_step600_ema.safetensors` | `num_inference_steps: 8` (8 denoiser evaluations), `lora_scale: 1.0` | None |
| Most aggressive speed preset (standard PEFT layout) | [`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo), `minimax_h3_fl2v_turbo_4step_v0.1.safetensors` | `num_inference_steps: 4` (4 denoiser evaluations), `lora_scale: 1.0`, `lora_alpha: 8` | None |
| Realistic people style | [`fal/MiniMax-H3-Realism-People-LoRA`](https://huggingface.co/fal/MiniMax-H3-Realism-People-LoRA), `h3-realism-people-t2v-i2v-r2v.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.7` | Include `r34l1sm` in the prompt |
| Spatial and physics | [`Jojocodex/minimax-h3-spatial-physics-lora`](https://huggingface.co/Jojocodex/minimax-h3-spatial-physics-lora), `wushu_spatial_physics_clean_3000_pruned.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.4` | None. Describe collisions, stacking, and gravity-consistent motion in the prompt |
| Wushu action | [`Jojocodex/minimax-h3-wushu-action-lora`](https://huggingface.co/Jojocodex/minimax-h3-wushu-action-lora), `wushu_action_h3_lora_v5_3000_pruned.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.8` | Put `wushu_action` at the start of the prompt |
| Camera motion | [`Jojocodex/minimax-h3-yunjing-lora`](https://huggingface.co/Jojocodex/minimax-h3-yunjing-lora), `camera_motion_h3_lora_v1_3000_pruned.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 0.8` | Put `camera motion` at the start of the prompt |
| Looping sketch anime | [`Inner-Reflections/MiniMax-H3-Looping-Sketch-Anime`](https://huggingface.co/Inner-Reflections/MiniMax-H3-Looping-Sketch-Anime), `minimax_h3_looping_sketch_anime_v1.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 1.0` | Include the author's hand-drawn sketch style phrase in the prompt |
| 3D-to-real detail slider | [`siraxe/3d_to_real_detail_slider_H3`](https://huggingface.co/siraxe/3d_to_real_detail_slider_H3), `3d_to_real_H3_detail_slider_bf16.safetensors` | Keep the normal `num_inference_steps: 50` schedule; start with `lora_scale: 1.0` | None. Sweep the scale; this is a slider, not a fixed style |

The H3 request field controls the number of denoiser evaluations. An 8-step
adapter uses `8`, and a 4-step adapter uses `4`; the sigma schedule includes
one additional terminal zero.

Every recipe uses the same launch shape. Pinning the filename is required for
repositories that publish multiple revisions, and is also recommended for a
reproducible single-file recipe:

```bash Command theme={null}
LORA_REPO=larryvrh/MiniMax-H3-Turbo-Lora
LORA_FILE=minimax_h3_turbo_v4_step600_ema.safetensors
LORA_NAME=h3-turbo-v4
LORA_SCALE=1.0
LORA_ALPHA_ARGS=()
# LightX2V only: LORA_ALPHA_ARGS=(--lora-alpha 8)

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --performance-mode speed \
  --lora-path "$LORA_REPO" \
  --lora-weight-name "$LORA_FILE" \
  --lora-nickname "$LORA_NAME" \
  --lora-scale "$LORA_SCALE" \
  "${LORA_ALPHA_ARGS[@]}" \
  --lora-merge-mode auto \
  --port 30010
```

`auto` merges an adapter into ordinary resident weights to avoid per-step LoRA
matmuls. It keeps the dynamic path in two cases. One is FSDP-sharded weights,
where a full gather can increase peak memory. The other is an adapter whose
update would mostly round away in a BF16 merge. Distilled adapters such as the
Larry Turbo LoRA sit below a BF16 ulp of most weights, and a merge drops
77–97% of their update in typical layers. Use `dynamic` when one resident
server must switch repeatedly between base and LoRA output, and `merge` to
force a merge.

Use the filename, scale, and request schedule from the table together. The
4-evaluation LightX2V recipe is the more aggressive latency/quality tradeoff.
Its checkpoint has rank 128 but omits the training alpha from both the file and
repository metadata, so `--lora-alpha 8` is required to reproduce the author's
reference implementation. Start with the Larry 8-evaluation recipe when
preserving fine visual detail is more important than minimum latency.

The pinned files above were trained for the **FL2VA** partition and apply to
`t2va` or `fl2va` requests. Some repositories, including LightX2V, publish
separate files for Ref2VA/Ref2V; select one explicitly for a `ref2va` server
rather than reusing an FL2VA file. Those Ref2VA files are not yet a pinned,
validated recipe on this page. Also avoid stacking a distilled adapter with
`quality: "high"`: both alter denoising, and that combination has not been
quality-validated.

AdaLN-pruned Diffusers components that publish `adaln_basis` and `adaln_mean`
can also consume a LoRA trained against the released full-width AdaLN modules:
SGLang projects those adapter factors onto the pruned coordinates at load time.
A structurally modified checkpoint without that metadata still fails closed,
and packed GGUF weights remain incompatible with LoRA.

### PDD acceleration LoRAs

[Alibaba PAI's MiniMax-H3-Acc-LoRAs](https://huggingface.co/alibaba-pai/MiniMax-H3-Acc-LoRAs)
use Parallel Decoding Distillation (PDD) for eight denoiser evaluations. These
checkpoints contain backbone LoRA weights and interval-specific output heads.
Merge the backbone weights offline, then fuse the output heads before serving.
The `--lora-path` recipe above does not perform these preparation steps.

Match the adapter and original MiniMax-H3 transformer weights to the server's
model variant:

| Model variant | Adapter file | Tasks |
| - | - | - |
| `fl2va` | `MiniMax-H3-FL2VA-Acc-8Step.safetensors` | `t2va`, `fl2va` |
| `ref2va` | `MiniMax-H3-Ref2VA-Acc-8Step.safetensors` | `ref2va` |

`BASE_TRANSFORMER_DIR` is the directory containing the original model's
BF16/FP32 transformer weight files, not the model root or the LoRA directory.
If you downloaded `MiniMaxAI/MiniMax-H3` to `/path/to/MiniMax-H3`, use
`/path/to/MiniMax-H3/FL2VA/transformer` for T2VA/FL2VA or
`/path/to/MiniMax-H3/Ref2VA/transformer` for Ref2VA. This directory contains
`model-*.safetensors`, `model.safetensors.index.json`, and `config.json`.
The builder reads these files and writes the merged weights to
`$PDD_DIR/transformer`, leaving the original weights unchanged.

The following example prepares FL2VA
weights and serves them on eight NVIDIA CUDA GPUs with TP2 + Ulysses4:

```bash Command theme={null}
BASE_TRANSFORMER_DIR=/path/to/MiniMax-H3/FL2VA/transformer
LORA_DIR=/path/to/h3-acc-loras
PDD_DIR=/path/to/h3-fl2va-pdd

hf download alibaba-pai/MiniMax-H3-Acc-LoRAs \
  MiniMax-H3-FL2VA-Acc-8Step.safetensors --local-dir "$LORA_DIR"

python3 -m sglang.multimodal_gen.tools.build_minimax_h3_pdd_weights \
  "$BASE_TRANSFORMER_DIR" \
  "$LORA_DIR/MiniMax-H3-FL2VA-Acc-8Step.safetensors" \
  "$PDD_DIR"

python3 -m sglang.multimodal_gen.tools.fuse_minimax_h3_pdd_heads \
  "$PDD_DIR" --video-shift 12.0 --audio-shift 3.0

SGLANG_DIFFUSION_MINIMAX_H3_PDD_HEADS="$PDD_DIR/pdd_fused_heads.safetensors" \
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --transformer-weights-path "$PDD_DIR/transformer" \
  --num-gpus 8 \
  --tp-size 2 \
  --ulysses-degree 4 \
  --performance-mode speed \
  --port 30010
```

Keep `pdd_config.json` beside `pdd_fused_heads.safetensors`, outside the
`transformer/` directory. For Ref2VA, use its adapter and base transformer,
a separate output directory, and `--model-variant ref2va`.

Use the requests in [Generate video and audio](#4-generate-video-and-audio)
with `num_inference_steps: 8`, `flow_shift: 12.0`, and `audio_flow_shift: 3.0`.
The nine sigma grid points give eight denoiser evaluations. These settings
must match the fused heads; the server rejects a different step count or
shift. Use `quality: "lossless"` in requests.

## 6. FastH3: 8-step distilled

[FastVideo/FastVideo-FastH3-8-Step-V2](https://huggingface.co/FastVideo/FastVideo-FastH3-8-Step-V2)
is an 8-step DMD2 distillation of MiniMax-H3, trained data-free with Video
Sparse Attention (VSA) at 0.8 sparsity and 64-token tiles, with video / audio
scheduler shifts 10 / 3. Only the T2VA capability was distilled: requests must
use `task: "t2va"`, and `fl2va` / `ref2va` requests are rejected. The
checkpoint inherits the MiniMax-H3 Community License.

Pass the repository directly to `--model-path`. The flat native-Diffusers
upload is materialized into the base-H3 layout through a registered model
overlay; the only non-symlink step is a one-time re-serialization of the
roughly 10 GB video VAE on first launch.

```bash 4×GB300 VSA-H3 theme={null}
sglang serve \
  --model-path FastVideo/FastVideo-FastH3-8-Step-V2 \
  --num-gpus 4 \
  --component-attention-backends transformer=video_sparse_attn_h3 \
  --port 30010
```

Requests use the same asynchronous video endpoint as the base model, with
`task: "t2va"`, `conditions: []`, and a target such as
`{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 5.0}`. The
request default is `num_inference_steps: 8`: nine sigma grid points, i.e. the
eight distilled DiT evaluations on the checkpoint's trained rungs
`[999, 874, 749, 624, 500, 375, 250, 125]` (unshifted timesteps over 1000,
shifted once per modality, as FastVideo runs them). Any other step count is
rejected. The trained sparsity 0.8 is the default; `--attention-backend-config
'{"VSA_sparsity": ...}'` overrides it. FastH3 requests default to
`quality: "lossless"`, which fuses the VAE decoder's per-head QK RMSNorm and
RoPE and runs its attention on cuDNN SDPA; `quality: "exact"` keeps the
reference decoder kernels. Under `"lossless"` FastH3 decodes each GPU's
spatial VAE tiles in one call rather than one at a time (base MiniMax-H3
batches up to 64 per call), which changes GEMM shapes and so the output at
rounding level; `"exact"` decodes tile by tile on both.

`video_sparse_attn_h3` (VSA-H3) is the trained sparse policy: in-tree
block-sparse kernels (FastVideo's native tcgen05 forward on SM100 / SM103, a
Triton tile-64 kernel on SM90) over segment-pure prefix tiles and (4, 4, 4)
video tiles, driven by the checkpoint's trained
`to_gate_compress` compression branch. Select it for the transformer only, as
with cube sparse attention: the packed DiT blocks run sparse while the token
refiner falls back to dense FA inside the backend, and the text encoder and
VAEs keep their own backends. Ulysses sequence parallelism is supported. See
[Attention Backends](/docs/sglang-diffusion/attention_backends) for
`VSA_sparsity`, `vsa_mode`, `vsa_dense_first_n_steps`, and
`vsa_dense_layers`. On CUDA, FastH3 selects VSA-H3 for the transformer when no
transformer backend is given, unless torch.compile, the breakable CUDA graph or
ring attention is on (VSA-H3 serves none of them). Every dense backend that
runs on base H3 also runs on the FastH3 weights, without the compression
branch: select it with `--component-attention-backends transformer=fa` (or
`torch_sdpa`, ...). On 4× H200 dense FA is about 25% slower end to end than
VSA-H3 (12.6 s against 10.0 s). `sglang generate` takes the same flags as
`sglang serve`.

Measured latencies for the 4× GB300 recipe are in
[FastH3 on GB300](#fasth3-on-gb300).

FastH3 rejects deployment options that do not apply to the distilled
checkpoint instead of silently ignoring them: `--model-variant`,
`quality: "high"`, `fl2va` / `ref2va` requests, and, with VSA-H3,
`--ring-degree` greater than 1, `torch.compile`, and breakable CUDA graph
execution.

> [!NOTE]
>
> Quality gaps versus base H3 on hard motion and fine detail are properties of
> the released distillation, not of the SGLang port. Use base MiniMax-H3 when
> output quality matters more than latency.

### FastH3 Omni: Ref2VA, 8-step PDD

[FastVideo/FastVideo-FastH3-Omni-8-Step-V1](https://huggingface.co/FastVideo/FastVideo-FastH3-Omni-8-Step-V1)
is a Parallel Decoding Distillation (PDD) student of the MiniMax-H3 Ref2VA
partition: eight denoiser evaluations over a 32-interval grid, with VSA at 0.9
sparsity on 128-token (4, 4, 8) tiles and each reference video keeping 0.1 of
its tiles. Only `task: "ref2va"` is served, with up to 9 images, 3 videos and 3
audio clips as ordered references (12 in total), at least one of them visual.

Pass the repository directly to `--model-path`. Its registered model overlay
maps `transformer_ref/` into the base-H3 layout, fuses the 32 interval output
heads into 8 per-step heads (the math of `tools/fuse_minimax_h3_pdd_heads.py`)
and links the text encoder and VAEs from the `MiniMaxAI/MiniMax-H3` Ref2VA
partition pinned by the release. The served model is the native H3 pipeline:
the release's `pdd_fused_heads` load through the same path as the PDD LoRA
recipe above, and the PDD grid is the 8-step shifted grid scaled to the
release's `sigma_max` of 0.999 (FastGen's `max_t`).

```bash 2×H200 theme={null}
sglang serve \
  --model-path FastVideo/FastVideo-FastH3-Omni-8-Step-V1 \
  --num-gpus 2 --ulysses-degree 2 \
  --dit-cpu-offload true --text-encoder-cpu-offload true --vae-cpu-offload true \
  --port 30010
```

```json Request theme={null}
{
  "model": "FastVideo/FastVideo-FastH3-Omni-8-Step-V1",
  "prompt": "<Picture 1> is the appearance reference for <Subject 1>. [reference generation] ...",
  "task": "ref2va",
  "conditions": [{"type": "image", "role": "reference", "uri": "/path/to/subject.png"}],
  "target": {"short_edge": 480, "aspect_ratio": "16:9", "duration_seconds": 5.0},
  "seed": 0
}
```

Requests default to `num_inference_steps: 8` and clean references
(`imgvid_cond_noise_aug_for_inference` and `audio_cond_noise_aug_for_inference`
of 1.0, the release's target-only flow). On CUDA the transformer selects VSA-H3
when no backend is given; Ref2VA layouts tile every reference video as its own
region and take a per-region top-k, while text, reference images and audio
rows stay dense. `--attention-backend-config` keys `vsa_tile_size`,
`vsa_ref_keep_rate` and `VSA_sparsity` override the trained values.

Warm request latency on 2× H200 (Ulysses 2, DiT / text encoder / VAE CPU
offload; median of three requests after one full warmup, 124 frames at 24 FPS
with audio, until the MP4 is saved):

| References | Short edge | Latency |
| - | - | - |
| Image | 480 | 26.9 s |
| Image + audio | 480 | 27.8 s |
| Video with audio | 480 | 23.2 s |
| Image | 768 | 46.6 s |
| Video with audio | 768 | 37.1 s |

## 7. VDN-H3: hybrid attention, 8-step distill

[Video DeltaNet](https://openvdn.github.io/) (VDN) is a hybrid attention
architecture that approaches the speed of linear attention while maintaining
near-lossless quality. It splits the video-video attention into two
complementary branches: a sliding-window softmax branch for local frame pairs
(five consecutive latent frames form a chunk, and each chunk attends to
itself, the previous chunk, and the following chunk; the first and last
frames, text and audio stay dense) and a bidirectional linear branch, Video
Delta Attention, for long-range context.
[OpenVDN/vdn-minimax-h3](https://huggingface.co/OpenVDN/vdn-minimax-h3)
applies it to MiniMax-H3: the released `stage-dmd-step-250` checkpoint adds a
4.3 GB linear branch and OpenVDN's own 8-step DMD2 LoRA on the unchanged H3
backbone. It is the FL2VA partition, so it serves `t2va` and `fl2va` (first,
last, or both keyframes); `ref2va` is rejected. The weights inherit the
MiniMax-H3 Community License.

Pass the repository directly to `--model-path`. On first launch a registered
overlay prefuses both adapters into the transformer weights (a 62 GB write)
and attaches the linear branch; the Qwen3-VL conditioner and the VAEs are
hard-linked from `MiniMaxAI/MiniMax-H3`. Keep `SGLANG_DIFFUSION_CACHE_ROOT` on
the same filesystem as the Hugging Face cache with at least 90 GB free.

```bash 8×B200 theme={null}
sglang serve \
  --model-path OpenVDN/vdn-minimax-h3 \
  --num-gpus 8 \
  --attention-backend hybrid_window_attn_h3 \
  --performance-mode speed \
  --warmup-num-frames 345 \
  --warmup-resolutions 1344x768 \
  --port 30010
```

`--num-gpus 4`, `2` or `1` gives the smaller Ulysses recipes in the benchmark
table; `sglang generate` takes the same flags. `--warmup-num-frames` /
`--warmup-resolutions` warm up at the clip length and canvas you will serve;
without them the first long request pays 2 to 3 s of allocator growth and
kernel setup.

Requests use the same video endpoint as the base model with `task: "t2va"`
and `conditions: []`, or `task: "fl2va"` with the
[keyframe conditions](#4-generate-video-and-audio), and a target such as
`{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 14.375}` (the
345-frame paper workload). The default `num_inference_steps: 8` is the only
accepted value: nine sigma grid points, eight DiT forwards. Each keyframe adds
about 2,000 dense rows, so an `fl2va` forward costs a little more than `t2va`.

`hybrid_window_attn_h3` is required: a dense backend would silently skip the
linear branch and the gates and produce the wrong model. On Blackwell
(SM100 / SM103 / SM120) the transformer defaults to online MXFP8;
`--quantization bf16` opts out. Before SM100, `--quantization fp8` is base
H3's per-channel fp8 path, and SM80 runs the bf16 DiT (62 GB; budget for
`--layerwise-offload-components`). Ampere and Ada are enabled but not
benchmarked. Ulysses sequence parallelism is supported; `--model-variant`,
`quality: "high"`, `--ring-degree` greater than 1, `torch.compile`, and
breakable CUDA graph execution are rejected. The backend's
`--attention-backend-config` keys are listed in
[Attention Backends](/docs/sglang-diffusion/attention_backends).

Measured latencies for 1, 2, 4 and 8 B200s are in
[VDN-H3 on B200](#vdn-h3-on-b200); single-card and PCIe multi-card numbers
for the RTX PRO 6000 are in [VDN-H3 on RTX PRO 6000](#vdn-h3-on-rtx-pro-6000).

## 8. Sampling and output controls

MiniMax-H3 supports more than one output per prompt. The video API accepts
`num_outputs_per_prompt` (or OpenAI-compatible `n`) from 1 through 10. Offline
generation accepts `--num-outputs-per-prompt N`; `--num-outputs N` is the short
alias. A scalar seed is expanded deterministically as `seed + output_index`, so
the outputs do not reuse the same noise.

Same-prompt fan-out reuses text conditioning. On the verified 2× RTX 5090
recipe, a 5-step two-output request completed in 155.39 seconds versus 78.11
seconds for one output, while producing two distinct valid MP4 files. The
independent denoise and decode passes remain sequential on this 32 GB profile
to keep peak memory bounded; the grouped path adds essentially no orchestration
overhead. Use server replicas when lower wall-clock latency for many variants
matters more than per-server memory efficiency.

For example, set `"num_outputs_per_prompt": 2` in any request above. After the
job completes, download both outputs by selecting each zero-based variant:

```bash Command theme={null}
video_id="<completed-job-id>"
for variant in 0 1; do
  curl -sS -L \
    "http://127.0.0.1:30010/v1/videos/${video_id}/content?variant=${variant}" \
    -o "minimax-h3-${variant}.mp4"
done
```

### Choose the quality level

`quality` is a cumulative request-scoped optimization parameter with three
levels:

* `"exact"`: the exact reference path. Output is bit-exact
  against the reference implementation and the CI ground truth. On SM100+
  the video VAE's temporal blend and output denormalization run as single
  kernels that reproduce the reference arithmetic bit for bit and verify that
  against the eager chain on first use.
* `"lossless"` (default): includes the global fusion-only tier but does not enable
  Cache-DiT or another approximate optimization. Each DiT block runs its
  RMSNorm and AdaLN modulation as one kernel that rounds once instead of after
  every op, and VAE decode fuses the ViT decoder's per-head QK RMSNorm and
  RoPE into one launch, runs its attention on cuDNN SDPA and, on CUDA, decodes
  several spatial tiles per decoder forward (rounding-level differences only).
* `"high"`: the audited accelerated path. Quality is guaranteed (the audited
  Cache-DiT configuration measures SSIM 0.931 / PSNR 28.16 dB against
  `exact`), but output is no longer bit-identical to the reference.

One resident server serves all three levels; a `quality: "high"` request
mounts its audited Cache-DiT policy at the batch boundary, and a later
`quality: "exact"` or `quality: "lossless"` request removes the hooks
before denoising.

Start the validated server once:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 4 \
  --tp-size 1 \
  --sp-degree 4 \
  --ulysses-degree 4 \
  --ring-degree 1 \
  --encoder-parallel auto \
  --performance-mode speed \
  --use-fsdp-inference false \
  --enable-torch-compile false \
  --port 30010
```

Then choose a request level:

**exact**

Native denoising and VAE decode with no feature-cache approximation or
quality-gated VAE fusions.

```json Request field theme={null}
{
  "quality": "exact"
}
```

**lossless (default)**

The global fusion-only tier. It does not enable MiniMax-H3 Cache-DiT. It
differs from `exact` only in rounding: the DiT blocks' fused RMSNorm + AdaLN
modulation, and the VAE decoder's fused QK RMSNorm + RoPE and cuDNN SDPA
attention.
Omitting the field selects this level.

```json Request field theme={null}
{
  "quality": "lossless"
}
```

**high**

The audited accelerated path. Use it when you can trade bit-exactness for
latency while keeping output closest to the same-seed exact trajectory.

```json Request field theme={null}
{
  "quality": "high"
}
```

The measured trade-off is:

| `quality` | Mean inference latency | Speedup | SSIM vs exact | PSNR vs exact | Expected trade-off |
| - | -: | -: | -: | -: | - |
| `exact` | 75.10 s | 1.00× | 1.000 | exact | Native reference path |
| `lossless` | Not separately measured | — | Same H3 denoise path | Same H3 denoise path | Fusion-only tier; VAE decode fuses QK RMSNorm + RoPE and uses cuDNN SDPA (rounding-level differences) |
| `high` | 53.70 s | 1.40× | 0.931 | 28.16 dB | Smallest same-seed visual change |

These numbers use 1344×768, 124-frame, 24 fps T2VA with 50 inference steps,
video flow shift 12, audio flow shift 3, and three fixed prompt/seed pairs on
4×H200. The prompts cover a quiet detailed scene, fast multi-subject action,
and a moving close-up portrait. `inference_time_s` is averaged across the three
prompts; the quiet-scene point is itself the mean of two repeats.

SSIM and PSNR compare decoded, frame-aligned output with the `exact`
result for the same prompt and seed. They measure trajectory deviation, not
absolute perceptual quality: the `high` path can produce a different but
still plausible realization. It also changes the joint audio-video denoise
trajectory, while these two metrics cover video only.

`quality: "high"` currently accepts only the exact workload and 4×H200
deployment above; other hardware, task modes, request shapes, step counts, or
flow shifts fail before denoising. Offline generation uses the same level
name, for example `sglang generate --quality high`.

> [!NOTE]
>
> `quality` selects a model sampling level and can change generated content.
> `output_quality` controls only output-file compression; it is a separate field.

For manually tuned Cache-DiT experiments outside that validated path, omit
the request `quality` field and set `--enable-cache-dit` or the
process-wide `SGLANG_CACHE_DIT_*` defaults. Any explicit `quality`, including
`"exact"` and `"lossless"`, takes H3 off the generic Cache-DiT path. The
24 GB layerwise recipe above can use the same switch; skipped blocks are
not streamed.

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=1 \
SGLANG_CACHE_DIT_BN=0 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.12 \
SGLANG_CACHE_DIT_MC=2 \
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --encoder-parallel auto \
  --performance-mode speed \
  --port 30010
```

> [!WARNING]
>
> Cache-DiT skips selected block computation and is approximate. It cannot
> be combined with FSDP inference. DiT layerwise offload is compatible:
> skipped blocks are not streamed, and the first layer after a skip may
> sync-load. Breakable CUDA graph execution takes precedence and leaves
> Cache-DiT disabled. Tune the cache thresholds only after comparing both
> video and audio quality on the target task profile. A real B200 request
> has completed, but the `quality: "high"` path above remains fail-closed
> to the audited 4×H200 workload.

## 9. Feature contracts and advanced recipes

The generated command already contains the recommended topology and encoder
setting. Use the detailed reference below only when applying an optional
override or checking its installation, topology limits, and validation evidence.

**Lossless runtime**

The recommended `speed` launch already combines resident components with
Ulysses sequence parallelism. Validation status below applies only to the
listed hardware and topology; it is not inherited by a similar GPU family.

| Feature | Validation status | Notes |
| - | - | - |
| Ulysses sequence parallelism | Verified: 8× B200, 4× H200, 4× H100, and Ulysses1/2/4/8 on MI300X and MI355X | Use `--ulysses-degree`. Combine with Ring for cross-node scaling; see the next row. |
| Ring sequence parallelism (cross-node) | Verified: 2 nodes of 8× H200 each (Ulysses8 × Ring2) | Use `--ring-degree` together with `--nnodes`/`--node-rank`/`--dist-init-addr`. Ring shards the sequence across nodes while Ulysses shards heads within a node; H3's packed multi-segment attention only supports Ring across the node boundary, not within a single node's Ulysses group. Requires `--encoder-parallel replicate` — `auto`'s fold decision is not node-boundary aware. See the benchmark section below. |
| Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. |
| FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. |
| Resident components | Verified: B200, H200, 4×H100 with TP, and 1/2/4/8× MI300X and MI355X | This is the recommended single-request latency path when the complete workload fits. |
| CPU and layerwise offload | Verified: 2× RTX 5090 TP2; 1× RTX 4090 24 GB | The 5090 lossless recipe keeps 20 DiT blocks plus both VAE encoders resident, streams the remaining DiT blocks, text encoder, and video VAE decoder blocks, and leaves the small audio VAE resident. The 4090 recipe streams DiT and the text encoder with zero resident DiT layers and **omits `vae`** from `--layerwise-offload-components`. Compatible with Cache-DiT; skipped blocks are not streamed. |
| Breakable CUDA graph | Verified: B200 Ref2VA, opt-in | Matching eager output was observed for the captured signature, without a measured speedup. Re-capture for other shapes and reference sets. |
| `torch.compile` | Measured: H200, opt-in | Steady-state benefit was below measurement noise, while startup increased and numerical output changed. Do not use it for consistency ground truth. |

The verified parallel, placement, and matching-signature BCG paths keep the
BF16/FP32 weights and denoising math. `torch.compile` is the exception called
out above. Always use the eager BF16/FP32 launch when producing CI consistency
ground truth.

For the validated 1344×768 Ref2VA profile, use a 5504-row text bucket so both
the server warmup and reference-conditioned requests share the captured
signature:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --encoder-parallel auto \
  --performance-mode speed \
  --enable-breakable-cuda-graph true \
  --warmup-resolutions 1344x768 \
  --bcg-text-buckets 5504 \
  --port 30010
```

BCG is lossless for a matching captured signature, but capture reserves extra
GPU memory. Re-measure the live H3 text length before reusing this bucket for a
different task profile, reference set, resolution, or prompt template.

**Attention backends**

Leave `--attention-backend` unset for the platform default. Use
`--attention-backend fa` only for an explicit FlashAttention comparison.

SageAttention uses quantized attention math and is not a consistency mode. To
select H3's native packed-varlen Sage path, install the dependency and add
`--attention-backend sage_attn`. On Hopper, install the upstream SM90 binding
fix rather than the PyPI 2.2.0 build:

```bash Command theme={null}
pip install --force-reinstall \
  git+https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4 \
  --no-build-isolation
```

The backend is a server-wide default. Use
`--component-attention-backends` only when a measured component needs a
different kernel, and keep the platform default for every component not named
in the override.

**Online quantization**

On the verified 8× B200 topology, quantize the BF16 transformer at server load:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --encoder-parallel auto \
  --performance-mode speed \
  --quantization fp8 \
  --port 30010
```

H3 automatically keeps its video/audio patch projections, timestep MLP, and
final video/audio heads in FP32. All other linear layers have stable full
module prefixes, so additional layers can be kept unquantized:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --encoder-parallel auto \
  --quantization fp8 \
  --quantization-ignored-layers blocks.0.attn token_refiner \
  --port 30010
```

> [!WARNING]
>
> Online FP8 is approximate and is not a consistency ground-truth mode. It can
> be combined with Cache-DiT, but the two approximations compound. Validate
> visual quality, audio quality, memory use, and latency on the target workload.
> This recipe is limited to the resident B200 and B300 topologies used for real
> H3 validation runs.

On a single 24 GB card, use `convrot_int8` instead of FP8. It quantizes the
four GEMMs per DiT block online from the Hub BF16 weights (data-free, no
calibration) and runs them on `comfy_kitchen.int8_linear` on the 4090 (CC 8.9)
or on SGLang's JIT-compiled ConvRot ops on CC 9.0, 10.0, 12.0 and 12.1
(see [Quantization](/docs/sglang-diffusion/quantization#convrot-int8)).
Quantization happens after H3's grouped `qkv` reorder, so do not load an
externally pre-quantized INT8 checkpoint here.

```bash Command theme={null}
sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --quantization convrot_int8 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output
```

`fa` keeps exact attention. For a faster, approximate DiT path, use
`--attention-backend sol_attn` with
`--attention-backend-config dense_backend=sage_attn,dense_steps=10` and
`--component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn`.
See [Quantization](/docs/sglang-diffusion/quantization#convrot-int8)
and [Attention Backends](/docs/sglang-diffusion/attention_backends#sage-then-sol-hybrid).

> [!WARNING]
>
> `convrot_int8` changes Linear numerics. `sol_attn` / `sage_attn` also change
> the attention algorithm. Neither is a consistency ground-truth mode. The
> BF16 path never depends on `comfy-kitchen`; `convrot_int8` needs it only on
> GPUs the JIT ops do not cover (any CC other than 9.0, 10.0, 12.0 and 12.1) or
> when no CUDA toolchain is available to build them.

Pre-quantized and compact H3 text encoders are listed once in
[Checkpoint and adapter formats](#checkpoint-and-adapter-formats). They use
component-local paths and never inherit the DiT's `--quantization` setting.
SGLang reads their metadata before constructing the native Qwen3-VL encoder and
fails closed when the selected format, projection, or topology is incompatible.

**Cube sparse attention**

Cube sparse attention applies TopK sparsity only to H3's 3D visual streams.
Text, audio, standalone reference images, and the text-only token refiner stay
dense. It runs on pure PyTorch plus FlexAttention, so it has no third-party
kernel dependency.

Select it for the H3 transformer with
`--component-attention-backends transformer=cube_sparse_attn` and pass
`--attention-backend-config` with both `local_cube_size` and
`topk_ratio_list`. Scoping the backend leaves the text encoder on its native
backend:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --performance-mode speed \
  --component-attention-backends transformer=cube_sparse_attn \
  --attention-backend-config '{"local_cube_size": [4, 4, 4], "topk_ratio_list": [1.0, 1.0, 0.8, 0.7, 0.6, 0.5, 0.5]}' \
  --port 30010
```

* `local_cube_size` is the `(T, H, W)` cube that groups neighboring latent
  tokens into one attention block. It must have exactly three entries.
* `topk_ratio_list` sets the per-step keep ratio and must have exactly one
  entry per denoise step, each in `(0, 1]`. A ratio of `1.0` keeps a step
  on H3's native dense attention path; smaller values select the FlexAttention
  sparse path and drop more blocks. The example above matches a request with
  `num_inference_steps: 7`, whose endpoint-inclusive sigma schedule has seven
  denoise updates.

Cube labeling is coordinate-driven. FL2VA keyframes share the target video's
position grid, so a keyframe token and a target token at the same `(T, H, W)`
coordinate receive the same semantic cube label. Duplicate coordinates do not
extend the temporal grid; a semantic cube can therefore span multiple physical
attention blocks. In Ref2VA, standalone reference images remain dense, while
reference videos and the target video contribute to one global TopK candidate
pool rather than receiving separate per-stream quotas.

> [!WARNING]
>
> Cube sparse attention is an approximate backend and is not a consistency
> ground-truth mode. `topk_ratio_list` length must equal the denoise step count
> or the server rejects the request. Cube sparse attention does not support Ring
> parallelism; use `--ulysses-degree` without `--ring-degree`. FlexAttention's
> routing overhead can outweigh sparse-kernel savings on short sequences, so
> benchmark latency as well as visual and audio quality on the target workload.

**SubBlock sparse attention**

SubBlock sparse attention applies training-free block sparsity to MiniMax-H3's
long, non-causal DiT self-attention. The BF16 path supports CUDA SM90, SM100,
and SM120. Other compute capabilities, including SM103/B300, are rejected.
Ulysses sequence parallelism is supported; Ring parallelism is not.

The global backend selects the H3 DiT lazily, so keep the Qwen text encoder on
its compatible dense backend with a component override:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --attention-backend subblock_sparse_attn \
  --component-attention-backends text_encoder=fa \
  --attention-backend-config '{"sparsity": 0.75, "skip_first_steps": 10}' \
  --port 30010
```

Use `text_encoder=torch_sdpa` on SM120. The default configuration drops at
most 75% of key blocks after the first 10 denoise forwards; short sequences,
cross-attention, and unsupported shapes continue on the dense path. Tune
`sparsity` and `skip_first_steps` together with visual and audio quality checks.

On H100/H200 (SM90), `compute_mode=sage_fp8` switches the sparse kernel to
online INT8 Q/K and FP8 P/V compute. Install the optional kernel and add the
compute mode:

```bash Command theme={null}
pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation

sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --attention-backend subblock_sparse_attn \
  --component-attention-backends text_encoder=fa \
  --attention-backend-config '{"compute_mode": "sage_fp8", "sparsity": 0.75}' \
  --port 30010
```

> [!WARNING]
>
> Both SubBlock routing and `sage_fp8` are approximate. Use dense BF16 attention
> for consistency ground truth, and validate the selected sparsity on the target
> resolution, duration, task, and checkpoint.

**Encoder scheduling**

The picker explicitly writes `--encoder-parallel auto` in every single-node
recipe. At the default request batch size of one, H100/H200/B200/B300 servers
with peer-to-peer access fold the Qwen encoder across otherwise idle Ulysses
ranks. A pure-TP recipe keeps the encoder inside its TP group, while a
PCIe-only host can avoid an expensive world fold. Keep `auto` unless one of
the cases below applies.

Encoder DP is a throughput policy for compatible request batches. It requires
TP1 and DiT DP1, replicates the encoder weights, and does not improve a batch
of one:

```bash Overlay theme={null}
--encoder-parallel dp \
--batching-max-size 2
```

The cross-node picker recipe already uses replication because the automatic
fold decision is not node-boundary aware:

```bash Overlay theme={null}
--encoder-parallel replicate
```

## 10. Configuration notes

* MiniMax-H3 produces the canonical 24 fps output; request duration is expressed through `target.duration_seconds`.
* `target.duration_seconds` must be between 4 and 15 seconds, inclusive. The command picker defaults to the verified 5-second profile.
* Use a 768-pixel short edge for the released quality recipe. The aligned output dimensions are derived from `target.aspect_ratio`.
* `flow_shift` controls video diffusion and `audio_flow_shift` controls audio diffusion.
* V2V uses `task: "ref2va"` with a `video` or `video_audio` reference; it is served by the `Ref2VA` partition and is not a separate public task value.
* `conditions[].start_time_seconds` selects a non-negative offset for a video reference. Its visual and audio streams are always sought together.
* Ref2VA condition order is semantic and must match the one-based material tags in the prompt. For Ref2VA, `target.aspect_ratio: "auto"` resolves to the model's 16:9 fallback rather than inheriting a reference asset's geometry.
* The distilled pipeline uses a single denoising branch, so CFG parallelism does not apply. Keep it disabled: a CFG degree greater than 1 is accepted for request compatibility, but the extra ranks only recompute the same branch and do not accelerate the request. Explicitly disabling CFG, or setting its size to 1, remains a valid no-op.
* The released visual VAE quality recipe uses overlapping tiled decode. SGLang keeps that recipe by default and distributes complete tiles across the decode group; this changes scheduling, not the computation inside each tile.
* H3 rejects `--vae-config.parallel-decode-mode spatial` and `spatial_shard`: validation found output mismatches. Use the default released tiled recipe.
* On CUDA, `quality: "lossless"` (the default) decodes up to `vae_config.decoder_tile_batch` (64) spatial tiles and `vae_config.window_batch` (1) temporal windows per video VAE decoder forward, next to the fused Q/K norm+RoPE and cuDNN SDPA; `exact` decodes one tile and one window at a time, `--performance-mode memory` does so at every level, and other platforms keep one tile per forward. On SM100+ the temporal blend and output denormalization run as bit-exact kernels at every quality level.
* Keep the default `--encoder-parallel auto`. With the server’s default `batching_max_size` of 1, single-node H100/H200/B200/B300 recipes with peer-to-peer access fold the Qwen text encoder over otherwise idle Ulysses ranks. This is separate from DiT tensor parallelism. A pure-TP recipe already shards the encoder over its TP group and does not add a world fold.
* Single-node Ulysses recipes pipeline each block's all-to-all against dense attention (FlashAttention on Hopper, cuDNN SDPA on Blackwell) over the copy engine: one head group's q/k/v moves while the previous group attends, every copy (the merge into the output included) runs on the copy engine, and the output is bit-identical to the sequential exchange. It needs peer-to-peer access and stream memory operations on every rank and about 6 × (rows per rank) × heads × head\_dim extra activations per rank (2.1 GB for Ref2VA at Ulysses 2), one set shared by every resolution; when that does not fit on some rank, every rank keeps the sequential exchange. `SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS=0` restores the sequential exchange.
* For throughput-oriented serving, select **DP (batched throughput)**. The picker pairs `--encoder-parallel dp` with an editable `--batching-max-size` greater than 1. Encoder DP stays inside each DiT replica and composes with encoder TP: the H100 TP2 + Ulysses2 recipe has two TP-sharded encoder copies that can split a batch, while the RTX 5090 pure-TP2 recipe has one encoder copy and therefore no additional batch-DP degree. It provides no benefit for a batch of one and is not bitwise-identical to the unsplit deployment.
* Use explicit **Fold** to prioritize single-request latency and encoder memory on a measured high-bandwidth single-node topology. Use **Replicate** as the compatibility path when folding or encoder DP is unsuitable.
* `--use-fsdp-inference true` shards only the DiT. MiniMax-H3 preserves the original FP32 dtype of its patch, time, and output projections during FSDP all-gather, so this path does not trade numerical correctness for memory. On 4×H100, prefer TP2 + Ulysses2 for speed; use FSDP as an explicit capacity policy rather than assuming it is faster.
* `speed` keeps model components resident, while `auto` applies the model-aware 120 GiB residency threshold. `memory` prioritizes avoiding OOM and includes the executable VAE decoder in its default layerwise set. A measured recipe with sufficient headroom can opt into `--component-residency vae=resident`; the 2×H100 CI recipe does this because the VAE's 4.8 GiB/GPU cost avoids repeated decoder transfers during tiled decode. DiT residency and prefetch knobs remain scoped to the DiT. Use `speed` only after confirming that the complete target workload fits.
* Breakable CUDA graph execution is an explicit opt-in, not part of the recommended `speed` preset. It requires `--enable-breakable-cuda-graph`, every served size in `--warmup-resolutions`, and `--bcg-text-buckets` that cover the live H3 condition sequence. The validated 1344×768 Ref2VA recipe uses 5504; other task profiles and reference sets may need a different value. It preserves eager output for matching captured signatures, but graph capture consumes additional GPU memory and may provide little latency benefit when Ulysses attention and collectives dominate, so benchmark it on the target topology before enabling it.
* `--enable-spectrum` skips selected DiT steps. On H3 it cannot combine with Cache-DiT / `quality="high"` or SP > 1. 50-step 4090 recipe is in §3.

## 11. Benchmarks

The picker exposes resident and FSDP profiles on NVIDIA datacenter GPUs. GPU
counts are properties of the selected recipes, not a claim that every platform
requires that many GPUs. The detailed tables below report performance only for
the configurations with collected measurements:

| Hardware | Default resident recipe | Other profile or topology |
| - | - | - |
| B300 | 8× Ulysses8 resident | 8× FSDP + Ulysses8; the 8-GPU sweep is not a minimum-GPU claim. |
| B200 | 8× Ulysses8 resident | 4× FSDP + Ulysses4 |
| GB300 | 4× Ulysses4 resident (FL2VA T2VA) | 2 nodes × 4× Ulysses4×Ring2 is unverified. |
| GB200 | 4× Ulysses4 resident, derived/unverified | No performance measurements. |
| H200 | 4× Ulysses4 resident | 4× FSDP + Ulysses4; 4× TP2 + Ulysses2; 2 nodes × 8× Ulysses8×Ring2 cross-node |
| H100 | 4× TP2 + Ulysses2 resident | 4× TP4 + Ulysses1; 4× FSDP + Ulysses4 |
| Ascend NPU | 8 NPUs, TP2 + SP4, Laser Attention | 4 NPUs, TP2 + SP2, Laser Attention |
| MI300X / MI355X | 8× Ulysses8 resident | 1×, 2×, and 4× scaling runs |
| RTX 5090 | 1× layerwise offload, measured on a physical desktop | 2× TP2 + layerwise offload |
| RTX 4090 24 GB | 1× layerwise offload + `convrot_int8` | Spectrum and approximate attention backends are opt-in |

### GB300 single-host

Four requests on **1 host × 4 NVIDIA GB300**, using Linux ARM64, SGLang main
[`15d2cbcc90fc`](https://github.com/sgl-project/sglang/commit/15d2cbcc90fc66b4d08745994cc707a45b067ffc),
PyTorch 2.13.0+cu130, and NCCL 2.29.7. The source checkout was installed with
`python3 -m pip install -e "python[diffusion]" --upgrade` inside
`lmsysorg/sglang:latest`; this is a source-validation environment, not a claim
that the image's bundled release contains that revision.

The command matches the GB300 default in the builder: Ulysses4, resident
weights, automatic encoder folding, native BF16/FP32, platform-default
Dynamic cuDNN/FA attention, and eager execution. The model was downloaded by
`sglang serve` from `MiniMaxAI/MiniMax-H3`.

The workload uses the builder's cat-band prompt, FL2VA T2VA, a 5-second request
at a 768-pixel short edge, 50 inference steps, `quality="exact"`, one output,
and seed 1101. Every MP4 contained video and audio and passed a full decode check.

| First full request after server warmup | Following three requests | Warm median |
| -: | - | -: |
| 35.16 s | 33.11 / 33.10 / 33.10 s | **33.10 s** |

These are client-observed completion times, including up to one second of
status-polling delay. Download, model loading, and server warmup are excluded.
This is a deployment smoke/latency measurement, not a cross-framework quality
audit, a minimum-GPU claim, or an extrapolation to GB200/NVL72 scaling.

### Ascend NPU topology comparison

Both topologies used Laser Attention and the explicit Cache-DiT configuration
from the Ascend launch commands, with `--dit-cpu-offload false` keeping the DiT
resident. The measured workload was one 5-second T2VA request at 1344×768,
124 frames, 24 fps, and 50 inference steps.

| NPU count | Topology | End-to-end latency |
| -: | - | -: |
| 8 | TP2 + SP4 | **55.07 s** |
| 4 | TP2 + SP2 | **103.57 s** |

These are individual end-to-end measurements for each topology, not averages.
The eight-NPU topology had 46.8% lower end-to-end latency than the four-NPU
topology.

### B300 precision and encoder placement

A 12-configuration sweep on a single 8× B300 host, covering both checkpoint
partitions, both transformer precisions, and all three text-encoder
placements. It answers one question — *how long does one request take, and how
much memory does it need*.

### What was measured

**Hardware.** 8× NVIDIA B300 SXM6, single node.

**Model.** `MiniMaxAI/MiniMax-H3`, both released weight partitions.

**Serve command.** Exactly the recipe the picker emits for B300, plus the one
or two overlay flags under test:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --performance-mode speed \
  --host 0.0.0.0 \
  --port 30010
```

The swept axes are `--model-variant` (`fl2va` / `ref2va`), `--quantization`
(unset for BF16 / `fp8`), and `--encoder-parallel` (`auto` / `fold` /
`replicate`). Nothing else differs between the 12 servers.

This is a single-request latency sweep (`batching_max_size: 1`), so encoder DP
is intentionally excluded: it cannot distribute a batch of one. Use the
**DP for a request batch** setting above for a compatible multi-request
deployment; the table below does not claim a measured H3 DP speedup.

**Driver.**

```bash Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
  --host 127.0.0.1 --port 30010 \
  --model MiniMaxAI/MiniMax-H3 \
  --dataset vbench --task text-to-video \
  --num-prompts 1 --max-concurrency 1 \
  --warmup-requests 1 --warmup-inference-steps 50 \
  --extra-body '{"task":"t2va","conditions":[],"target":{"short_edge":768,"aspect_ratio":"16:9","duration_seconds":5.0},"seconds":5,"flow_shift":12.0,"audio_flow_shift":3.0}'
```

**Workload**

| Property | Value |
| - | - |
| Output duration | 5.167 s |
| Resolution | 1344×768 |
| Frames | 124 @ 24 fps |
| Denoising steps | 50 |
| `flow_shift` / `audio_flow_shift` | 12.0 / 3.0 |
| Requests in flight | 1 (`--max-concurrency 1`, server at `batching_max_size: 1`) |
| Requests measured | 1 per cell, after 1 warmup request |

### Results

| Weights | Precision | Encoder | Load | Warmup | Latency | Peak/GPU |
| - | - | - | -: | -: | -: | -: |
| FL2VA | BF16 | auto | 118.1 s | 29.65 s | **19.04 s** | 83,578 MB |
| FL2VA | BF16 | fold | 114.0 s | 28.72 s | **19.04 s** | 83,578 MB |
| FL2VA | BF16 | replicate | 116.0 s | 28.33 s | **19.04 s** | 124,158 MB |
| FL2VA | FP8 | auto | 116.0 s | 27.16 s | **18.03 s** | 51,926 MB |
| FL2VA | FP8 | fold | 116.0 s | 25.99 s | **18.04 s** | 51,926 MB |
| FL2VA | FP8 | replicate | 118.0 s | 27.97 s | **18.04 s** | 92,506 MB |
| Ref2VA | BF16 | auto | 114.0 s | 38.69 s | **29.12 s** | 83,968 MB |
| Ref2VA | BF16 | fold | 118.0 s | 36.58 s | **29.13 s** | 83,968 MB |
| Ref2VA | BF16 | replicate | 116.0 s | 35.17 s | **29.13 s** | 124,490 MB |
| Ref2VA | FP8 | auto | 124.0 s | 34.30 s | **27.12 s** | 52,816 MB |
| Ref2VA | FP8 | fold | 112.0 s | 34.44 s | **27.12 s** | 52,816 MB |
| Ref2VA | FP8 | replicate | 116.0 s | 33.42 s | **27.12 s** | 93,396 MB |

### FastH3 on GB300

A 4× GB300 host served [FastH3](#6-fasth3-8-step-distilled) with the VSA-H3
recipe above (1344×768 at 24 fps with audio, `task: "t2va"`,
eight DiT forwards, the default `quality: "extra-high"` and sparsity 0.8,
eager BF16, Ulysses4, `NCCL_NVLS_ENABLE=0`), measured with the MP4 encoded
while the VAE decodes
([sgl-project/sglang#41819](https://github.com/sgl-project/sglang/pull/41819)).
The workload is 72 requests at
concurrency 1 after one warm request per shape: six scene families, 1,000- and
10,000-token prompts, 5 / 10 / 15 s, two seeds, every prompt unique so no
conditioning cache hits. E2E is the client wall clock from submission to the
last byte of the MP4; the stage columns are the server timings of the same
requests. Each cell is the median of the 12 requests of that shape:

| Prompt / requested / aligned | Text encoder | Denoise (8 forwards) | Decode (video + audio) | E2E | Peak/GPU |
| - | -: | -: | -: | -: | -: |
| 1K / 5 s / 124 | 0.10 s | 3.84 s | 0.56 s | **5.20 s** | 94,468 MB |
| 1K / 10 s / 243 | 0.10 s | 8.45 s | 1.12 s | **10.74 s** | 97,628 MB |
| 1K / 15 s / 362 | 0.10 s | 14.85 s | 1.72 s | **18.24 s** | 100,246 MB |
| 10K / 5 s / 124 | 0.18 s | 6.31 s | 0.55 s | **7.90 s** | 94,510 MB |
| 10K / 10 s / 243 | 0.18 s | 12.35 s | 1.12 s | **14.98 s** | 98,128 MB |
| 10K / 15 s / 362 | 0.18 s | 20.28 s | 1.67 s | **24.01 s** | 100,408 MB |

Over all 72 requests the E2E p50 / p90 / p95 is 12.94 / 23.94 / 24.08 s, and
every MP4 carries 1344×768 H.264 at 24 fps with a 32 kHz stereo AAC track.

### VDN-H3 on B200

An 8× B200 (SM100, 183 GB) host served [VDN-H3](#7-vdn-h3-hybrid-attention-8-step-distill)
on the paper workload: 1344×768 at 24 fps with audio for 14.375 s (345 frames,
102 latent frames, about 104k packed rows), `task: "t2va"`,
nine sigma grid points (8 DiT forwards), seed 1000, `hybrid_window_attn_h3`,
eager, and the warmup run at the served clip shape
(`--warmup-num-frames 345 --warmup-resolutions 1344x768`), so every forward
of the served request is steady state; without those flags the first forward
pays 2 to 3 s of allocator growth and kernel setup. "Steady
s/NFE" is the mean of forwards 2 to 8. The OpenVDN reference rows ran the released
inference stack (`8nfe_tuned_fp8.yaml`, `infer_ulysses.py`) on the same host
and the same clip length, with its Ulysses layout (`parallel.softmax_ranks`)
swept at every GPU count rather than assumed. The SGLang rows were re-measured
in one campaign at `f1a2ee763d`, one arm at a time on an otherwise idle node,
each with the [8×B200 recipe](#7-vdn-h3-hybrid-attention-8-step-distill) and
only `--num-gpus` and `--quantization` changed; the OpenVDN reference rows come
from an earlier campaign on the same host type. A repeated arm moves by about
3% across nodes, so treat smaller differences as noise:

| Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU |
| - | -: | -: | -: | -: |
| OpenVDN published, FP8, 8× B200 Ulysses 5+3 ([openvdn.github.io](https://openvdn.github.io/)) | 1.40 | 11.2 s | — | — |
| OpenVDN reference, BF16 (`8nfe_tuned.yaml`), 1× B200 | 7.66 | 61.2 s | — | — |
| OpenVDN reference, FP8, 1× B200 | 6.25 | 50.0 s | — | — |
| OpenVDN reference, FP8, 2× B200 Ulysses (standard / 1+1 branch-parallel) | 4.87 / 5.04 | 39.0 / 40.3 s | — | — |
| OpenVDN reference, FP8, 4× B200 Ulysses (3+1 branch-parallel) | 2.59 | 20.7 s | — | — |
| OpenVDN reference, FP8, 8× B200 Ulysses (5+3 / 4+4 / standard) | 1.40 / 1.47 / 1.52 | 11.2 / 11.7 / 12.2 s | — | — |
| SGLang, `--quantization bf16`, 1× B200, `--performance-mode speed`, served-shape warmup | 7.27 | 58.2 s | 7.0 s | 156,298 MB |
| SGLang, `--quantization fp8` (→ online `mxfp8` on SM100+), 1× B200, `--performance-mode speed`, served-shape warmup | 5.93 | 47.4 s | 7.0 s | 128,420 MB |
| SGLang default (online `mxfp8`), 2× B200 Ulysses2, `--performance-mode speed`, served-shape warmup | **3.25** | **26.0 s** | 3.7 s | 102,664 MB |
| SGLang, `--quantization bf16`, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | 2.05 | 16.4 s | 2.1 s | 104,358 MB |
| SGLang default (online `mxfp8`), 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **1.67** | **13.3 s** | 2.1 s | 81,168 MB |
| SGLang default (online `mxfp8`), 8× B200 Ulysses8, `--performance-mode speed`, served-shape warmup | **0.85** | **6.8 s** | 1.4 s | 76,918 MB |
| SGLang, per-channel fp8 weight scales (the online `fp8` path of other models; on SM100+ `--quantization fp8` resolves to `mxfp8` and no flag selects the per-channel path, so this arm forces it), 8× B200 Ulysses8 | 0.91 | 7.3 s | 1.4 s | 76,450 MB |

On 8× B200 the SGLang Ulysses8 path runs the paper workload at 0.85 s/NFE
(6.8 GPU-seconds per NFE) against the published 1.40 s/NFE of OpenVDN's 5+3
branch-parallel layout; the whole request (text encoding, 8 forwards, joint
decode) completes in about 8.2 s after warmup. At this GPU count the step is
launch- and copy-bound in the linear branch rather than FLOP-bound, and the
transformer runs online `mxfp8` by default (e4m3 with one E8M0 scale per 32
elements into cuBLASLt's block-scaled GEMM, with the activation quant fused
into the adaLN and SwiGLU producers).

Against the published 8× B200 headline (1.40 s/NFE, 11.2
GPU-seconds per NFE), the SGLang 4× B200 run spends 6.7 GPU-seconds per
NFE, with half the GPUs and half the all-to-all fan-out. At equal GPU count
SGLang is 1.50× faster at two cards (3.25 vs 4.87 s/NFE), 1.56× at four
(1.67 vs 2.59) and 1.66× at eight (0.85 vs 1.40), while on one card the two
stacks are within 5% of each other (5.93 vs 6.25 s/NFE quantized, 7.27 vs 7.66
in BF16). The advantage is therefore parallel scaling rather than per-GPU
kernel work: SGLang holds 88-91% parallel efficiency from 2 to 8 cards against
the reference stack's 56-64%, and sweeping the reference stack's own layout
does not close it (its best split is 8% better than standard Ulysses at eight
cards, 1.40 against 1.52 s/NFE).

On one B200 both single-GPU rows run with the DiT resident under the auto
memory policy (128 GB peak quantized, 156 GB in BF16, of the card's 183 GB),
so no `--layerwise-offload-components` flag is needed at this clip length and
the single-GPU rows are compute-bound.

### VDN-H3 on RTX PRO 6000

The same paper workload (1344×768, 345 frames, 8 DiT forwards, seed 1000,
`hybrid_window_attn_h3`, `--performance-mode speed`, served-shape warmup) on
RTX PRO 6000 Blackwell Server Edition cards (SM120, 96 GB, PCIe, no NVLink).
The window runs on FA4's sm120 kernel and the transformer defaults to online
`mxfp8` (a cutlass sm120 block-scaled GEMM); `--quantization fp8` maps to it.
One card needs `--layerwise-offload-components text_encoder`: the fp8 DiT
(33 GB) and the bf16 text encoder (48 GB) do not both fit, and the whole-module
`--text-encoder-cpu-offload` re-homes the encoder in one piece and runs out of
memory when it is used. Multi-card Ulysses goes over PCIe; the all-to-all is
the bulk of the step there (54% of GPU time at 8 cards), so the per-card
efficiency drops with count. A 2-card run should use a pair on one PCIe
switch (`nvidia-smi topo -m` shows `PIX`): a pair across the CPU root complex
measured 15.7 to 17.0 s/NFE on a shared host.

| Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU |
| - | -: | -: | -: | -: |
| 1× RTX PRO 6000, `--layerwise-offload-components text_encoder` (default `mxfp8`) | **20.27** | **160.0 s** | 20.6 s | 95,790 MB |
| 1× RTX PRO 6000, same, per-channel `fp8` path (before this mapping) | 22.85 | 180.3 s | 20.7 s | 95,160 MB |
| 2× RTX PRO 6000 Ulysses2, one PCIe switch (default `mxfp8`) | 13.61 | 107.4 s | 10.6 s | 93,846 MB |
| 2× RTX PRO 6000 Ulysses2, one PCIe switch, per-channel `fp8` | 14.97 | 118.1 s | 10.6 s | 94,700 MB |
| 4× RTX PRO 6000 Ulysses4 (default `mxfp8`) | 7.98 | 63.0 s | 5.6 s | 82,178 MB |
| 4× RTX PRO 6000 Ulysses4, per-channel `fp8` | 8.23 | 65.1 s | 5.6 s | 82,422 MB |
| 8× RTX PRO 6000 Ulysses8 (default `mxfp8`) | 5.43 | 43.1 s | 3.5 s | 77,704 MB |
| 8× RTX PRO 6000 Ulysses8, per-channel `fp8` | 5.47 | 43.4 s | 3.5 s | 77,236 MB |

For comparison the same code on B200 measures 6.03 / 3.30 / 1.67 / 0.88 s/NFE
at 1 / 2 / 4 / 8 cards (NVLink, 86% parallel efficiency at 8).

### H200 topology comparison

The same four-card H200 host completed both lossless resident placements with
the standard 1344×768, 5-second, 50-step T2VA request (fixed prompt and seed,
eager BF16/FP32, back-to-back runs on an otherwise idle host). Latency is the
warmed-up request; the first pair uses the default warmup request, the second
pair adds `--warmup-resolutions 1344x768` so warmup already covers the served
resolution:

| Topology | Warmup | Denoise | Decode | E2E | Peak/GPU |
| - | - | -: | -: | -: | -: |
| Ulysses4 | default | 79.04 s | 3.77 s | **84.14 s** | 94,288 MB |
| TP2 + Ulysses2 | default | 81.17 s | 2.97 s | 85.51 s | 63,490 MB |
| Ulysses4 | `--warmup-resolutions 1344x768` | 71.73 s | 1.32 s | **74.38 s** | 94,290 MB |
| TP2 + Ulysses2 | `--warmup-resolutions 1344x768` | 75.52 s | 1.29 s | 78.33 s | 63,490 MB |

Ulysses4 stays the H200 latency default: 5.0 % faster end-to-end than
TP2 + Ulysses2 once warmup covers the served resolution (1.6 % with the
default warmup, where first-request cold start masks the topology gap).
TP2 + Ulysses2 shards the DiT weights and holds peak memory about 30 GB per
GPU lower, which is why it remains the 80 GB H100 recipe. Matching the warmup
request to the served resolution removes the cold first-request cost on both
topologies (about 10 s end-to-end on this workload).

### H200 cross-node scaling

Long references and long durations grow the packed sequence length, and
Ulysses alone cannot scale sequence parallelism past the GPU count of one
node without either violating head-count divisibility or exposing
all-to-all traffic across the slower inter-node link. H3 combines
node-local Ulysses with cross-node Ring: Ring's point-to-point KV rotation
is designed to overlap with attention compute, which fits a slower
cross-node link better than an all-to-all does.

**Hardware.** 2 nodes × 8× NVIDIA H200 SXM, same cluster, InfiniBand
between nodes.

**Serve command.** The cross-node cell the picker emits for H200, run
identically on both nodes with `--node-rank` set to 0 and 1:

```bash Command theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 16 \
  --nnodes 2 \
  --node-rank {{NODE_RANK}} \
  --dist-init-addr {{NODE0_IP}}:20000 \
  --sp-degree 16 \
  --ulysses-degree 8 \
  --ring-degree 2 \
  --encoder-parallel replicate \
  --performance-mode speed \
  --host 0.0.0.0 \
  --port 30010
```

**What was measured.** A controlled denoise-stage comparison on identical
hardware: 8× H200 single-node (Ulysses8, no Ring) versus the same 16-GPU
cross-node command above (Ulysses8 × Ring2), holding prompt, seed, and
step count fixed:

| Task | Single-node (Ulysses8) | Cross-node (Ulysses8 × Ring2) | Change |
| - | -: | -: | -: |
| T2VA denoise/step | 0.749 s | 0.477 s | −36.3% |
| Ref2VA/V2V denoise/step | 2.572 s | 1.494 s | −41.9% |

The gain grows with sequence length because Ring's per-hop communication
cost stays roughly constant while attention compute grows quadratically
with sequence length, so V2V's longer packed sequence benefits more than
T2VA's shorter one. With the point-to-point KV rotation pipelined against
attention compute, one V2V request's full denoise stage completed in
68.1–68.3 seconds versus 128.6 seconds on the single-node 8-GPU baseline
(−47.0%), with byte-identical output to the unpipelined cross-node path.

Cross-node determinism was confirmed separately: the same request run
twice against the same cross-node deployment produced byte-identical
output. A cross-node run's output is not expected to bit-match a
single-node run of the same prompt and seed — Ring's online-softmax merge
across hops accumulates floating-point operations in a different order
than single-node attention, which is an expected source of bit-level
difference, not a correctness regression.

> [!WARNING]
>
> `--encoder-parallel auto`'s fold decision is not yet node-boundary aware
> and attempts to fold the text encoder across nodes, which crashes the
> Ref2VA reference-conditioned encoder. Always pass
> `--encoder-parallel replicate` explicitly for cross-node H3 deployments.

### H100 topology comparison

The same four-card H100 host completed three lossless placements. TP2 with
Ulysses2 was the fastest; TP4 used the least memory:

| Topology | Pipeline latency | Peak/GPU |
| - | -: | -: |
| TP2 + Ulysses2 | 13.25 s | 66.04 GB |
| FSDP + Ulysses4 | 13.36 s | 57.01 GB |
| TP4 + Ulysses1 | 13.86 s | 49.80 GB |

### RTX 5090 capacity run (two cards)

The single-card recipe is a different one — see *RTX 5090 single card, physical
desktop* below; the twenty resident layers here are half a layer per GPU under
TP2 and do not transfer to one card. The verified two-card RTX 5090 host used TP2 with layerwise offload. The full
50-step, 1344×768, 5-second request completed in 559.67 seconds: 525.05
seconds of denoising and 33.61 seconds of decoding, with a 26.3 GiB sampled
peak per GPU.

| DiT settings | 5-step denoise | Inference | Peak/GPU | Result |
| - | -: | -: | -: | - |
| prefetch 1, resident 20 | 43.48 s | 78.11 s | 26.3 GiB | Selected recipe |
| prefetch 2, resident 20 | 43.37 s | 78.06 s | 27.5 GiB | No measurable gain |
| Ulysses2, prefetch 2, resident 10 | Did not reach warmup | — | — | Rejected |

### Consumer GPU tuning

On consumer hardware the binding question is not which card you have but how much
host RAM sits behind it. H3's weights are about 108 GB — 61.73 GB of DiT and
46.18 GB of text encoder — so no consumer configuration holds them all, and where
the shortfall lands decides the throughput.

**The command** — most consumer machines need exactly one flag beyond the model:

```bash consumer single GPU, lossless theme={null}
sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \
  --layerwise-offload-components dit,text_encoder,vae
```

With 16 GB of VRAM or more, add `--layerwise-resident-layers video_vae=36` for
the 13 s decode; with \~96 GB of host RAM and 16 GB+ of VRAM, add
`--dit-layerwise-resident-layers 4` for the 6 s step. That is the whole flag
surface. The [builder at the top of this page](#1-quick-start) has consumer
cards and a Host RAM selector: pick your budget and it emits this command with
your tier's measured expectations attached as comments. The table below is the
same data in one view.

**Unified-memory machines are the exception**: on a DGX Spark (GB10, 128 GB
shared between CPU and GPU) launch with **no flags at all**. The deployment
still exceeds the pool, automatic offload engages on its own, and its placement
beat the explicit recipe above by 2.1× on the denoise (12.1 vs 25.8 s/it
measured on the same box) — the VRAM/host split that justifies every flag in
this section does not exist there. Expect \~12 min of load, a \~5.5 min text
encoding stage per request (steady-state compute on this chip, not a stall),
and \~12 min per warm 480P request.

**Two budgets, and what each one buys**

| | 12 GB VRAM + 32 GB host | host free, VRAM 16 GB |
| - | -: | -: |
| Recipe | A | B |
| Peak VRAM | ≤ 12 GiB | ≤ 16 GiB (OOMs at 12) |
| Host anonymous (must fit) | 24.5 GiB | 116.7 GB pinned |
| Denoise, 864×480 / 124 frames / 20 NFE | 16.8 - 18.7 s/it | **6.01 s/it** |
| Runs at all | yes | yes |

The left column is one configuration measured twice, at 318.94 s and 356.37 s;
the 12% spread tracked host load on a shared machine, so treat smaller
differences than that as unresolved. The right column is 120.92 s at a 16 GiB
allocator cap. Four resident DiT layers is what Recipe B buys its speed with,
and it is also why 12 GiB is not enough for it.

Read the host row carefully, because the two numbers are not the same kind of
memory. *Anonymous* host memory — pinned buffers and pageable copies — has to fit,
and the kernel cannot reclaim it. Page cache backing a file mapping is
*droppable*, so it does not count against the budget even though it shows up in
`VmRSS`; use `RssAnon` from `/proc/<pid>/status` when checking. Likewise measure
VRAM with `torch.cuda.set_per_process_memory_fraction` and let the allocator fail,
rather than reading `nvidia-smi`, which reports the caching allocator's reserved
pool and overstates the requirement.

Inside 32 GB the weights cannot be pinned, so each denoise step copies about
60 GiB from the checkpoint mapping, and a mapped source is synchronous however
the copy is requested: the driver stages it through its own buffer, so the
transfer neither overlaps compute nor runs at pinned bandwidth. That is where
the step goes, and giving the host room to pin the weights instead is what takes
it to 6.01 s.

Two caveats on the constrained number, both from instrumenting the run rather
than from arithmetic. The machine it was measured on has 2 TB of host memory, so
the kernel kept all 107.7 GiB of mapped checkpoint pages resident: major faults
across a whole request were 6, and `read_bytes` was zero. Nothing was read from
disk. A real 32 GB host cannot cache 107.7 GiB, so it will fault and re-read,
and should be expected to be slower than the figures here rather than equal to
them — an NVMe is a requirement, not a recommendation. Measure your own machine
with major faults (`/proc/<pid>/stat`) on the worker process, not on the
launcher, which holds no weights.

**Recipe A — fits 12 GB VRAM + 32 GB host**

```bash 12 GB + 32 GB, lossless theme={null}
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder,vae \
  --layerwise-resident-layers video_vae=36
```

**Recipe B — host memory is free (the fast path)**

```bash unconstrained host, lossless theme={null}
sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder,vae \
  --dit-layerwise-resident-layers 4 \
  --layerwise-resident-layers video_vae=36
```

Recipe B pins \~112 GB of host memory (DiT 61.56 GB, text encoder 46.18 GB, VAE
\~4.5 GB in its decode dtype). Do not reach for it on a 32 GB machine.

**What not to change, and why**

* `video_vae=36` holds every decoder block for the decode only — residency
  arms at the decoder's first block and releases when it finishes, so the
  denoise still runs on an empty card. It fits 12 GB because decoder weights
  are held in their decode compute dtype (fp16) from load, which halves them
  to \~4.9 GiB; the rounding was already part of every output (the decode
  computes in fp16 autocast), so the result is bit-identical, and the decode
  drops from 60 s streamed (or 209 s on a busy host) to \~10 s. The
  `expandable_segments` line stays: the decode sits close enough to the cap
  that fragmentation otherwise tips it over.
* Leave `--enable-torch-compile` off, as elsewhere on this page. Layerwise offload
  rebinds `param.data` on every layer, so compiled graphs do not get the benefit
  they would on resident weights.
* Recipe A's flags are what the automatic policy should choose on its own. Until
  the model declares its own placement, `--performance-mode memory` plus the
  explicit component list is what makes it happen; pass them.

**Reading the startup log**

The server prints the memory decisions it made; checking three lines against
your budget catches a mis-set machine in the first minute instead of the first
request.

* `Layerwise offload: host memory available: N GiB` — what the runtime sees
  after loading, not your DIMM size. On a 32 GB host expect single digits here;
  a much larger number means another process's memory accounting (or a
  container limit) is in play.
* `leaving N GiB of weights on the checkpoint mapping` — the expected line on a
  32 GB host: the DiT streams from the checkpoint file. If instead the log
  reports pinned weights, the runtime decided your host has room — which is
  faster, and means the 32 GB figures above do not apply to you.
* `Loaded video_vae: ... host mmap` vs `host pageable` — where the VAE landed
  (decoder weights are \~4.9 GiB once held in their decode dtype).
  `Loaded <component>` lines carry the same buckets for every component.

If a request dies after the denoise finishes, it is the decode colliding with
the cap: keep the `expandable_segments` line, and if it persists drop to
`video_vae=24` and take the partially streamed decode.

**Against ComfyUI, on the same weights**

Same unpruned bf16 checkpoints, same card, same sampler settings (cfg 1.0,
euler\_ancestral, sigma shift 12.0/3.0, seed 1101), 864×480 / 124 frames / 20 NFE:

When host memory is free, the engines are close and sglang is ahead:

| | denoise | host anonymous | peak VRAM |
| - | -: | -: | -: |
| sglang, Recipe B | **6.01 s/it** | 116.7 GB pinned | ≤ 16 GiB |
| ComfyUI KSampler | 6.58–6.59 s/it | 116.5 GiB | 13048 MiB |

Inside 12 GB, both engines run these weights, and one measurement convention
matters on each side. ComfyUI's memory manager reads system RAM and adapts, so
the rows below patch `psutil` to a pretend host size — the same convention the
sglang rows use. Its `--reserve-vram` is also soft: told to keep 12 GiB free it
still peaked at 13.5 GiB, a figure a real 12 GB card cannot give it, so both
engines here run under the same hard allocator cap
(`set_per_process_memory_fraction`), where its peak stays at 12.1–12.3 GiB.
Under that cap, Recipe A wins the whole request at every host size:

| 12 GB VRAM, both engines hard-capped | sglang Recipe A (TE + denoise + decode) | ComfyUI, bf16 (warm) |
| - | -: | -: |
| 32 GB host | 12.4 + 212.8 + 9.4 ≈ **235 s** | 276–302 s |
| 48 GB host | 15.8 + 192.1 + 10.0 ≈ **218 s** | 246–267 s |
| 64 GB host | 7.5 + 162.4 + 10.3 ≈ **180 s** | 194–195 s |

Same GPU, same load window, unpruned bf16 checkpoints, outputs verified. All
figures are anchored at 480P — activations grow with the pixel count, so at
768P drop the resident DiT layers to 0 first, then `video_vae` to 24 if the
decode still collides.

One convention behind that table is the one a home machine does not share: the
host sizes above are `psutil` patches on a 2 TB lab machine, so the kernel kept
the whole checkpoint in page cache and nothing was read from disk. A physical
host of that size re-reads the streamed weights every step, and what that costs
is computable — `bytes per step = (50 - resident DiT layers - layers the host
pin budget covered) x 1.23 GiB`. The courier overlaps that read with compute, so
the drive stops being the denoise clock once its sequential rate exceeds those
bytes divided by the card's step time. A 32 GB host covers roughly 13 layers,
which puts a 24 GB card at \~38 GiB/step and a 12/16 GB card at \~46 GiB/step: a
PCIe 4.0 NVMe (6–7 GiB/s) clears both, a PCIe 3.0 drive (\~3.2 GiB/s) becomes the
clock, and a SATA SSD is not usable for this model. Resident DiT layers remove
1.23 GiB/step each, but only while the streamed set is larger than the pin
budget — once the read fits under the step's compute time, more of them buy
nothing and only cost VRAM.

An earlier revision of this page recorded 38 s/step on a physical 4090 with a
990 Pro, reading 52.9 GB per step — about 1.4 GB/s, and faulted sequentially, so
almost none of it showed in `majflt` (measure `read_bytes`, not major faults).
That rate was the old path, not the drive: the pinned-store pool rounded every
block up to a power of two, so the pin budget over-committed and the machine had
to run with pinning off, and the layers that stayed mapped faulted in 4 KiB at a
time. Exact-size pinning and O\_DIRECT reads through the courier removed both,
and the physical 5090 run below shows the drive fully hidden behind compute. A
4090 on a 32 GB host is derived to return to its compute wall (\~8.5 s/step) on a
PCIe 4.0 drive; that machine has not been re-measured. The
VRAM axis holds too: capped at 16 GiB the same recipe wins \~250 vs 292–301 s,
and at 24 GiB (with `--dit-layerwise-resident-layers 6` — measured at a
22 GiB cap so a desktop's own allocations fit; a headless card can raise it
to 10 for under 1% more) \~8.5 s/step vs ComfyUI's 249–260 s requests. Four changes carry it: the VAE staying on its checkpoint mapping (#35862, root fix
\#35946), per-layer pinning with net-cost accounting (#35867), the courier
thread that ships still-mapped layers through pinned slots (#35882), and
decoder weights held in their decode dtype from load (#35967) — which is what
lets `video_vae=36` fit and turns the decode from the slowest stage (54–96 s
streamed) into the fastest (\~10 s, faster than ComfyUI's own 15–25 s). Output
equivalence is bit-level: the fp16-held decode reproduced the fp32-store run's
video byte for byte, and the audio stream is bit-identical.

Stage by stage under the cap: text encoding is even (both stream the same
48 GB Qwen3VL), the denoise leads at 32–48 GB hosts and sits within
run-to-run variance of ComfyUI at 64 GB (162 vs 159 s), and the decode leads
everywhere. Two ComfyUI notes that still matter: `--fast-disk` measured no
faster than its default on this lab host — though on a physical 60 GB desktop
its default is OOM-killed while loading (55.6 GB of anonymous memory) and
`--fast-disk` is what makes it run at all — and stacking
`--novram --cache-none --disable-pinned-memory` made things strictly worse
(69.1 GiB anonymous, 750 s requests) — the adaptive default is the right
configuration on a small host.

The path ComfyUI ships for 12 GB cards uses
`minimax_h3_fl2va_pruned_int8_convrot` and `qwen3vl_32b_minimax_h3_nvfp4_awq`,
i.e. an int8 DiT and an NVFP4 text encoder, and its pruned bf16 file is 40.2 GB
against the unpruned 66.3 GB. Those are different weights, so it is not a
like-for-like comparison with the recipes above.

### RTX 5090 single card, physical desktop

Every consumer figure above this point was measured with at least one axis
simulated: a hard allocator cap standing in for VRAM, a `psutil` patch standing
in for host RAM, and a lab machine whose page cache held the whole checkpoint.
This run had none of that — an RTX 5090 (32 GB) in a Ryzen 9 9950X desktop with
60 GB of RAM, an 8 GB swap file and a Samsung 9100 PRO on PCIe 5.0, on Ubuntu
24.04. The workload matches the comparison above: FL2VA T2VA, 864×480, 124
frames, 20 steps, cfg 1.0, euler\_ancestral, sigma shift 12.0/3.0, same seeds and
keyframes on both engines, unpruned bf16 weights on both sides.

```bash 32 GB VRAM + 60 GB host, lossless theme={null}
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder,vae \
  --dit-layerwise-resident-layers 14 \
  --layerwise-resident-layers video_vae=36 \
  --warmup-resolutions 864x480 --warmup-num-frames 124
```

| | text encoding | denoise step | decode | request |
| - | -: | -: | -: | -: |
| sglang | **4.2 s** | 5.14–5.17 s | **6.4 s** | **112.2 / 112.1 s** |
| ComfyUI `--fast-disk --cache-none` | 11.3 s | 5.17–5.30 s | 7.2 s | 145.9 / 145.6 / 140.9 s |

Three things a capped run cannot settle:

**The denoise is at the card's compute wall, not the drive's.** The two engines
are within noise of each other per step, and prefetch depth 2 and cuDNN SDPA
each measured as no change. The 51.3 GiB the DiT reads from the drive per
request is fully overlapped — across a whole request the compute thread waited
0.9 s for the courier.

**Host RAM decides how much traffic there is to hide.** At 60 GB the pin budget
covers 34 of the 50 DiT layers, so only about two layers per step come off the
disk. The 46 GiB text encoder is never pinned — streamed layers take the budget
first — and is read whole, once per request, at the drive's sequential rate:
that is the 4.2 s stage, against 12.1–12.9 s when the same bytes go through the
page cache.

**Neither engine's default command survives this machine.** ComfyUI's is
OOM-killed during load (55.6 GB of anonymous memory) and needs `--fast-disk`;
sglang's was too, until pinned stores were allocated at their exact size instead
of the allocator's next power of two. Even now the automatic placement keeps no
resident DiT layers and streams the VAE on every decode, so the flags above are
what reach the numbers in the table.

Two caveats to carry into your own machine. The run finished with its 8 GB swap
file full, because the pin budget takes 95% of the memory available after
loading and there is no smaller setting yet — on a machine that is also your
desktop, expect that pressure and give the card more resident layers, since a
resident layer is one the host no longer has to pin. And these are 480P figures:
at 768P drop the resident DiT layers first, then `video_vae` to 24.

### RTX 4090 24 GB single-GPU run

One RTX 4090 D 24 GB completed the 1344×768, 107-frame, 20-NFE T2VA
workload (euler, `torch.compile` and step caching disabled) with DiT and
text-encoder layerwise offload. Same process: load → warmup (seed 0) →
timed (seed 42); only the timed pass is reported. GPU peak stayed about
18 GB.

| Config | Timed e2e | Denoise | vs BF16 | PSNR vs BF16 |
| - | -: | -: | -: | -: |
| BF16 + FlashAttention | 405.6 s | 370.2 s | 1.00× | — |
| `convrot_int8` + FA | 303.3 s | 273.7 s | 1.34× | 24.81 dB |
| `convrot_int8` + `sol_attn` | 223.9 s | 203.9 s | 1.81× | 24.44 dB |
| `convrot_int8` + `sage_attn` | 174.9 s | 154.2 s | 2.32× | 23.51 dB |
| `convrot_int8` + Sage→Sol hybrid | 163.8 s | 143.1 s | 2.48× | 23.04 dB |

`convrot_int8` + FA changes Linear numerics only. The `sol_attn` /
`sage_attn` / hybrid rows also change the attention algorithm, so speed
and pixel fidelity rank in opposite orders there. Default remains
`convrot_int8` + `fa`. Cache-DiT can share this layerwise recipe; omit
`quality` and see the quality-level section.

Same card, 50-step run. Speedup is vs BF16+FA. PSNR is vs same-tree INT8+FA (Spectrum off).

| Config | e2e | vs BF16+FA | PSNR mean / min |
| - | -: | -: | -: |
| BF16 + FlashAttention | 1008.6 s | 1.00× | — |
| `convrot_int8` + FA | 718.5 s | 1.40× | — |
| `convrot_int8` + FA + Spectrum `11 / 5 / 1.0` | **285.3 s** | **3.53×** | **27.4 / 25.3** |
| `convrot_int8` + `sage_attn` + Spectrum `15 / 5 / 0` | 245.3 s | 4.11× | 23.6 / 21.0 |
| `convrot_int8` + Sage→Sol hybrid + Spectrum `15 / 5 / 0.5` | **219.0 s** | **4.61×** | 20.6 / 17.8 |

Prefer FA + `11 / 5 / 1.0` when the bar is FA-like. Sage / hybrid + Spectrum is faster; quality then follows the attention backend, not the skip schedule.

### AMD Instinct task and scaling runs

The AMD recipes keep the released BF16/FP32 precision policy and use AITER
packed attention. The picker emits the fastest measured topology, 8 GPUs with
Ulysses degree 8. All runs below completed full H.264/AAC decoding and
representative-frame inspection.

| Hardware | Task | Denoise | Decode | Peak/GPU |
| - | - | -: | -: | -: |
| MI355X | T2VA | 55.2907 s | 9.5344 s | 97,444 MB |
| MI355X | FL2VA | 53.7978 s | 9.4477 s | 96,922 MB |
| MI355X | Ref2VA | 41.3812 s | 6.8247 s | 94,518 MB |
| MI300X | T2VA | 167.4878 s | 25.3244 s | 97,272 MB |
| MI300X | FL2VA | 150.2311 s | 12.5684 s | 96,750 MB |
| MI300X | Ref2VA | 107.6232 s | 11.3768 s | 94,268 MB |

The task matrix used 8 GPUs and 50 denoising steps. The scaling matrix uses
one 1344×768, 209-frame T2VA request and changes only the GPU count and
matching Ulysses degree:

| Hardware | GPUs | Denoise | Decode | Peak/GPU |
| - | -: | -: | -: | -: |
| MI355X | 8 | 55.2907 s | 9.5344 s | 97,444 MB |
| MI355X | 4 | 104.2294 s | 11.1824 s | 103,350 MB |
| MI355X | 2 | 223.0246 s | 15.5330 s | 115,250 MB |
| MI355X | 1 | 288.7968 s | 24.0472 s | 137,676 MB |
| MI300X | 8 | 167.4878 s | 25.3244 s | 97,272 MB |
| MI300X | 4 | 297.3727 s | 26.5067 s | 103,436 MB |
| MI300X | 2 | 585.5401 s | 29.4909 s | 115,010 MB |
| MI300X | 1 | 978.0886 s | 36.0142 s | 137,626 MB |

For a measured lower-count AMD deployment, set both `--num-gpus` and
`--ulysses-degree` to 4, 2, or 1. AITER packed attention matched segment-wise
BF16 SDPA at cosine similarity `0.9999991655` on MI355X and `0.9999991059` on
MI300X.

## 10. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate H3`.
This path has been run end to end against a live server.

**Integrated mode** — not available for this model
: H3 denoises a packed video-and-audio sequence in one pass and routes conditioning by task, while ComfyUI's KSampler drives a single latent tensor and has no audio branch
. Use server mode.


# MOSS-Image
Source: https://docs.sglang.io/cookbook/diffusion/OpenMOSS/MOSS-Image

Serve MOSS-Image with native Qwen3.5 conditioning, FLUX.2 decoding, and request-level guidance.

## 1. Quick start

Use a [SGLang Diffusion Docker environment](/docs/sglang-diffusion/installation).
Until your image includes this integration, install the model-support PR inside
the container:

```bash Install model support theme={null}
pip install 'sglang[diffusion] @ git+https://github.com/sgl-project/sglang.git@refs/pull/44325/head#subdirectory=python'
```

You need access to the selected Hugging Face checkpoint. The picker distinguishes
available implementation paths from hardware recipes that have not been measured.

## 2. Model capabilities

MOSS-Image is a text-to-image model using Qwen3.5 text conditioning and a FLUX.2
image decoder. Its native pipeline accepts natural-language prompts and structured
layout descriptions through the same Images API. Structured prompts are passed
to the text encoder as text, not interpreted as a separate rendering language.

Choose Base for reproducing the reference comparison. The SFT identifier resolves
to the same pipeline family, but its checkpoint has not been validated here.
Generation is supported; image editing is not implemented. Keep the scheduler,
guidance recipe, attention backend, and precision fixed when comparing outputs.

## 3. Guidance and precision

Defaults are 1024 x 1024, 50 steps, flow shift 5, CFG 3, and an empty negative
prompt. Width and height must be divisible by 16. DiT and text-encoder weights
use BF16; latents, CFG arithmetic, scheduler updates, and VAE decode use FP32.
Autocast is disabled by the pipeline configuration; no extra CLI flag is needed.

The default two-pass recipe skips unconditional layer 22 inside the
`slg_start=0.01`, `slg_stop=1.0` window. It leaves the first step at full CFG.
Set `uncond_skip_layers=[]` for ordinary CFG. Three-pass SLG adds a skipped
conditional prediction and changes the sampling algorithm; it is not a lossless
speed optimization.

```json Three-pass request overrides theme={null}
{
  "guidance_scale": 5,
  "uncond_skip_layers": [],
  "slg_layers": [22],
  "slg_scale": 3,
  "slg_start": 0.01,
  "slg_stop": 1.0
}
```

For the official pipeline's start-at-zero behavior, explicitly set
`slg_start=0.0` and `uncond_skip_layers=[22]`, including when using three-pass
SLG. Guidance fields are request-scoped and do not require a server restart.

`quality=exact` preserves the native rounding policy, but does not make different
attention backends identical. `quality=lossless` can additionally enable shared
VAE fast paths. Compare each configuration against its matched reference rather
than treating either quality name as a cross-backend guarantee.

## 4. Deployment features

Component and layerwise offload reuse the standard residency manager. Start with
Auto placement; use `--component-residency all=resident` only when weights and
request activations fit. Offload saves VRAM at the cost of host RAM and transfers.
The text encoder also supports layerwise offload through its native layer list.

Sequence parallelism uses the shared shard/pad/gather helpers and USP attention;
the text refiner stays replicated. Ulysses degree must divide the 24 DiT heads.
Ring requires a backend exposing softmax LSE, such as FlashAttention. CFG
parallelism uses the shared branch dispatcher, including the optional third SLG
branch. Multi-GPU checkpoint recipes remain experimental until measured on the
selected hardware and workload.

TP, Cache-DiT, quantization, LoRA, compilation, and breakable CUDA graphs do not
have validated recipes in this integration. Do not assume that a framework-wide
flag establishes model support. See [performance optimization](/docs/sglang-diffusion/performance-optimization)
for the shared feature contracts.

## 5. Batching

Dynamic batching uses the shared scheduler and is opt-in through the maximum
batch size. For example, on an H200:

```bash theme={null}
sglang serve --model-path OpenMOSS-Team/MOSS-Image-Base \
  --batching-max-size 4 --batching-delay-ms 20
```

Compatible requests may have different prompts and seeds; resolution, step count,
negative prompt, and guidance settings must match. The maximum size limits total
outputs, not just HTTP requests. A request can also use `n` to generate multiple
images from one prompt. Higher batch sizes need more activation VRAM and can
increase individual-request latency; the default remains one output at a time.

At `quality=lossless` (the default) or `high`, the DiT groups samples by their actual text length
and restores their original output order. Different lengths run in separate
groups without padding or switching attention backends. Empty negative prompts
can share one group even when positive prompts differ. The text refiner retains
per-sample GEMM shapes to reduce numerical drift. Batched DiT GEMM shapes can
change floating-point rounding, which denoising can amplify into visible changes;
cross-batch-size pixel identity is not guaranteed. `quality=exact` retains the
per-sample DiT path instead. Neither mode promises batch-invariant text encoding
or VAE decoding. Benchmark your prompts and batch sizes before deployment.

On one H200, four concurrent requests with different prompts and seeds 42-45 measured
the following warmed HTTP throughput. Both paths use BF16 DiT/text encoder,
FP32 VAE, default attention backends, `quality=lossless`, resident components,
maximum batch size 4, and a 20 ms batching delay. Loading is excluded; PNG encoding,
transfer, and client decoding are included. Results average three measured waves
at 512 pixels and two at 1024, after one warmup wave per resolution.
The environment used PyTorch 2.14.1+cu130, Transformers 5.19.0, and eager execution.

| Workload | Per-sample DiT | Grouped DiT | Throughput Change |
| - | - | - | - |
| 512 x 512, 8 steps | 2.51 images/s | 2.92 images/s | +16% |
| 1024 x 1024, 20 steps | 0.293 images/s | 0.296 images/s | Essentially unchanged |

At 512 pixels, agreement with the per-sample DiT path ranged from 28.3 to 46.2 dB
PSNR and 0.939 to 0.996 SSIM. All four 1024-pixel outputs were pixel-identical.
These measure output agreement, not semantic quality or a guarantee for other
prompts. Visible differences remain possible; use `quality=exact`
to retain per-sample DiT execution. Fixed-batch repeated requests are checked
separately from cross-batch agreement.

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# Qwen-Image
Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image

Deploy Qwen-Image with SGLang - community contribution guide for Qwen's image generation model.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[Qwen-Image](https://huggingface.co/Qwen/Qwen-Image) is a 20B text-to-image model built for strong prompt following and precise rendering of English and Chinese text. It is especially useful for posters, signs, diagrams, and dense layouts where typography and spatial relationships matter as much as general image quality.

This page covers generation rather than editing: use Qwen-Image-Edit when an existing image, subject identity, or untouched region must be preserved. The full checkpoint is memory-heavy, while the validated ModelOpt NVFP4 release provides a supported low-precision deployment option with an expected quality tradeoff.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs, Intel Xeon CPUs, and Ascend A2/A3 Series NPUs.

Use the [command builder](#1-quick-start) above.

For the validated ModelOpt NVFP4 checkpoint on Blackwell, load the published
Qwen-Image-2512 NVFP4 repo directly:

```bash Command theme={null}
sglang serve \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --ulysses-degree=1 \
  --ring-degree=1
```

For high-resolution B200 generations, the FlashInfer CUTLASS FP4 GEMM backend
can be faster than the default TensorRT-LLM FP4 GEMM backend:

```bash Command theme={null}
SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \
sglang generate \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --width 2048 --height 2048 \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output
```

### 3.2 Fixed-resolution latency on two H200 GPUs

For `Qwen/Qwen-Image-2512` at 1024x1024, use breakable CUDA graph (BCG) to
reduce launch overhead across graph-safe DiT segments while retaining explicit
breakpoints around unsupported operations. This recipe was validated on two
NVIDIA H200 GPUs with 50 denoising steps and no classifier-free guidance:

```bash Command theme={null}
sglang serve \
  --model-path Qwen/Qwen-Image-2512 \
  --model-type diffusion \
  --num-gpus 2 \
  --tp-size 2 \
  --performance-mode speed \
  --dit-layerwise-offload false \
  --enable-torch-compile false \
  --enable-breakable-cuda-graph \
  --warmup-mode server \
  --warmup-resolutions 1024x1024
```

Declare every production resolution in `--warmup-resolutions`. A request at an
uncaptured resolution runs eagerly, so omitting `1024x1024` removes the gain
from this recipe. Graph capture used about 5 GB more peak memory per GPU in the
validation run.

On CUDA, the TP path dispatches supported collectives through SRT
CustomAllReduceV2. At 1024x1024, Qwen-Image reduces 24 MiB row-parallel
outputs; the diffusion runtime reserves a 32 MiB V2 workspace so these
collectives do not fall back to NCCL. If profiling shows large NCCL all-reduce
kernels again, first confirm that V2 is enabled and the requested shape fits
the workspace.

BCG changed floating-point execution order but not the sampling algorithm. The
fixed-seed output measured 0.984 SSIM and 39.7 dB PSNR against eager output; use
eager execution when you require bit-exact output. Regional `torch.compile` was
also tested on this profile and did not improve steady-state latency.

### 3.3 Two 32 GB GPUs (RTX 5090)

With `--tp-size 2`, the adaLN modulation projections and the text-stream FFN
are sharded, so each rank holds about 19 GiB of DiT weights and the DiT stays
resident on 32 GB cards:

```bash Command theme={null}
sglang serve \
  --model-path Qwen/Qwen-Image \
  --num-gpus 2 \
  --tp-size 2 \
  --ulysses-degree 1 \
  --performance-mode manual \
  --text-encoder-cpu-offload true \
  --attention-backend torch_cudnn_sdpa
```

On 2x RTX 5090 (PCIe, no P2P) this measured 19.7 s at 1024x1024 with 50 steps
and no CFG. Add `--dit-cpu-offload true` to cut peak memory to about 22 GB per
GPU at 21.2 s.

### 3.4 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements.

* `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
* `--num-gpus`: Number of GPUs to use
* `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
* `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
* `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
* `--ring-degree`: The degree of ring attention-style SP in USP

**AMD ROCm Notes**: Requires SGLang >= v0.5.8.

## 4. API Usage

For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai_api).

### 4.1 Generate an Image

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")

response = client.images.generate(
    model="Qwen/Qwen-Image",
    prompt="A logo With Bold Large text: SGL Diffusion",
    n=1,
    response_format="b64_json",
)

# Save the generated image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
    f.write(image_bytes)
```

### 4.2 Advanced Usage

#### 4.2.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache_dit).

**Basic Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image
```

**Advanced Usage**

* DBCache Parameters: DBCache controls block-level caching behavior:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

Combined Configuration Example:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Qwen/Qwen-Image
```

#### 4.2.2 CPU Offload

* `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
* `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
* `--vae-cpu-offload`: Use CPU offload for VAE.
* `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".

#### 4.2.3 Known LoRA examples

Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Qwen-Image examples include:

* [`lightx2v/Qwen-Image-Lightning`](https://huggingface.co/lightx2v/Qwen-Image-Lightning)
* [`flymy-ai/qwen-image-realism-lora`](https://huggingface.co/flymy-ai/qwen-image-realism-lora)
* [`prithivMLmods/Qwen-Image-HeadshotX`](https://huggingface.co/prithivMLmods/Qwen-Image-HeadshotX)
* [`starsfriday/Qwen-Image-EVA-LoRA`](https://huggingface.co/starsfriday/Qwen-Image-EVA-LoRA)

## 5. Benchmark

Test Environment:

* Hardware: AMD Instinct MI300X GPU (1x)
* Model: Qwen/Qwen-Image
* Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x
* sglang diffusion version: 0.5.8

### 5.1 Speedup Benchmark

#### 5.1.1 Generate an image

**AMD MI300X**

**Server Command**:

```shell Command theme={null}
sglang serve --model-path Qwen/Qwen-Image \
    --ulysses-degree=1 --ring-degree=1 --port 30000
```

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-image
Model:                                   Qwen/Qwen-Image
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  29.04
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.03
Latency Mean (s):                        29.0378
Latency Median (s):                      29.0378
Latency P99 (s):                         29.0378
--------------------------------------------------
Peak Memory Max (MB):                    48018.83
Peak Memory Mean (MB):                   48018.83
Peak Memory Median (MB):                 48018.83
============================================================
```

**Ascend A3 Series**

**Server Command**:

```shell Command theme={null}
#One A3 Series card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path Qwen/Qwen-Image --num-gpus 2
```

**Benchmark Command**:

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        Qwen/Qwen-Image
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       36.26
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.03
Output throughput (outputs/s):                0.03
Latency Mean (s):                             36.26
Latency Median (s):                           36.26
Latency P90 (s):                              36.26
Latency P95 (s):                              36.26
Latency P99 (s):                              36.26
--------------------------------------------------
Peak Memory Max (MB):                         36984.00
Peak Memory Mean (MB):                        36984.00
Peak Memory Median (MB):                      36984.00
------------------------------------------------------------
```

#### 5.1.2 Generate images with high concurrency

**AMD MI300X**

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20 --port 30000
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-image
Model:                                   Qwen/Qwen-Image
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  300.79
Request rate:                            inf
Max request concurrency:                 20
Successful requests:                     14/20
--------------------------------------------------
Request throughput (req/s):              0.05
Latency Mean (s):                        154.5368
Latency Median (s):                      154.8363
Latency P99 (s):                         285.4603
--------------------------------------------------
Peak Memory Max (MB):                    48030.31
Peak Memory Mean (MB):                   48030.30
Peak Memory Median (MB):                 48030.29
============================================================
```

**Ascend A3 Series**

**Benchmark Command**:

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        Qwen/Qwen-Image
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       300.81
Request rate:                                 inf
Max request concurrency:                      20
Successful requests:                          8/20
Completed outputs:                            8
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.03
Output throughput (outputs/s):                0.03
Latency Mean (s):                             166.61
Latency Median (s):                           167.02
Latency P90 (s):                              270.80
Latency P95 (s):                              283.48
Latency P99 (s):                              293.64
--------------------------------------------------
Peak Memory Max (MB):                         36984.00
Peak Memory Mean (MB):                        36984.00
Peak Memory Median (MB):                      36984.00
------------------------------------------------------------
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — ComfyUI's own sampler, CLIP, and
VAE drive the loop while SGLang replaces the model forward. Load the
checkpoint with `SGLDiffusion UNET Loader` and set
`model_type` to `qwen_image` on the
`SGLDiffusion Options` node.

A reference workflow is included at
`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/qwen_image_sgld.json`.


# Qwen-Image 2.1 and Turbo
Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1

Run Qwen-Image 2.1 and its eight-step Turbo variant with SGLang Diffusion: text-to-image, reference-image editing, and transparent RGBA output.

## 1. Quick start

Use a nightly Docker image containing this integration, or install from source,
following the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation).
Run the commands below inside that environment. Select **Checkpoint weights** for
`Qwen/Qwen-Image-2.1` or `Qwen/Qwen-Image-2.1-Turbo`; you can also set a local
checkpoint directory under **Variables**. The recipes
target NVIDIA CUDA on Linux; the picker marks which workloads have
been verified with the full checkpoint.

Use **Setup** to select text-to-image, single-image editing, or multi-image
editing. **Server** controls placement, attention, encoder scheduling, VAE
tiling, graph execution, and request batching. **Request** controls the background, resolution,
steps, and output count. Set reference PNG paths under **Variables**; edits
upload files from the machine running cURL, so they need not exist on the server.

Hardware selection applies the recommended placement for that GPU. H200,
B200, RTX PRO 6000 96GB, and DGX Spark keep weights resident; RTX 5090 and RTX 4090
offload selected components to fit the full pipeline.
Untested topologies and feature combinations remain selectable and are labeled
**Unverified**. Invalid topology combinations disable Copy. This integration
currently uses the Python/source command; no published Docker image is verified.

Both request modes return base64 PNGs. To save all returned images, append
`> response.json` to the request command, then run:

```bash Command theme={null}
python - <<'PY'
import base64
import json
from pathlib import Path

for i, item in enumerate(json.loads(Path("response.json").read_text())["data"]):
    Path(f"output-{i}.png").write_bytes(base64.b64decode(item["b64_json"]))
PY
```

### Recommended hardware settings

The picker defaults to native BF16/FP32 precision, exact attention, eager
execution, and full-image VAE decoding.
Commands omit default values, including one GPU, encoder auto scheduling, and
batch size one. Explicit placement and attention overrides preserve each recipe.

| GPU | Placement / attention | Generation | Edit | Peak VRAM |
| - | - | - | - | - |
| H200 141GB | Resident / FlashAttention | 4.48 s | 5.29 s | 38.4 GiB |
| B200 192GB | Resident / FlashAttention | 2.46 s | 3.02 s | 38.5 GiB |
| RTX PRO 6000 96GB | Resident / Torch SDPA | 8.03 s | 9.63 s | 38.4 GiB |
| RTX 4090 24GB | DiT and VAE resident, encoder layerwise offload / FlashAttention | 18.68 s | 21.68 s | 22.7 GiB |
| DGX Spark 128GB unified | Resident / Torch SDPA | 35.36 s | 42.23 s | — (unified) |

Measured with the original checkpoint on 2026-09-20 at 1024×1024, 40 steps, CFG 1, and one RGBA PNG per
request. Times are median HTTP latency after warmup, including PNG serialization
and excluding startup; VRAM is the sampled request-phase peak. Prompts and
software versions affect both latency and memory use.

On one RTX 5090, native-precision, eager text-to-image or single-reference edits
with one output and batching off keep the DiT and VAE resident and stream the
encoder with Torch SDPA. Multiple reference images, multiple outputs, or request
batching select DiT layerwise offload instead; use the updated Server command
when changing these settings. The picker identifies which exact combinations
are verified. Both RTX 5090 and RTX PRO 6000 use SDPA when FlashAttention is
selected in this runtime. CPU offload requires host RAM.

### DGX Spark

Select **DGX Spark** for one GB10 GPU on Linux ARM64 with CUDA 13. Use the
source installation above. The recommended configuration keeps all components
resident, uses native BF16/FP32 precision, and lets the runtime select Torch SDPA:

```bash Command theme={null}
sglang serve \
  --model-path Qwen/Qwen-Image-2.1 \
  --performance-mode speed
```

The [128 GB unified memory](https://docs.nvidia.com/dgx/dgx-spark/hardware.html)
is shared by the CPU and GPU. CPU offload is unnecessary for the verified
single-image 1024×1024 workload. Keep full-image VAE decoding and eager execution.
Generation, editing, transparent generation, and transparent editing were
verified with PyTorch 2.13.0+cu130. Spark reports no separate VRAM usage in `nvidia-smi`.
This recipe covers one Spark; multi-node deployment and batching remain unverified.

### Batching

Keep **Request batching → Off** and **Outputs → 1** for interactive use.
Batching increases individual request latency and does not guarantee higher
throughput. Measure your workload before enabling it.

Cross-request batching merges compatible text-to-image requests. Image edits
run separately; **Outputs** controls multiple images within one request.
On RTX 4090, selecting multiple outputs or request batching switches to DiT
layerwise offload for memory headroom. Restart with the updated **Server** command.

Batching preserves native precision but can change floating-point rounding and
output pixels, even with the same seed. See
[Inference batching](/docs/sglang-diffusion/dynamic_batching) for admission rules
and metrics.

### Turbo sampling

Select **Qwen-Image 2.1 Turbo** under **Setup → Checkpoint weights** and restart
with the generated Server command. Requests use the checkpoint's preset eight-step
sigma grid automatically; omit `num_inference_steps`. Setting that field to 8
on the original checkpoint does not reproduce Turbo.

For offline generation:

```bash Command theme={null}
sglang generate --model-path Qwen/Qwen-Image-2.1-Turbo \
  --prompt "A ceramic teapot on a wooden table in soft morning light"
```

The preset comes from `sample_sigmas` in `model_index.json`, with scheduler
settings loaded from `scheduler/scheduler_config.json`. Preserve both when copying
or exporting the checkpoint. Changing only `num_inference_steps` does not override
the preset. Custom grids are experimental and can reduce quality; the picker keeps
Turbo on its published schedule. The default CFG scale is 1 for both checkpoints.
See the [Turbo model card](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo).

Turbo was verified at 1024×1024 with native precision and eager execution on
H200: single-GPU generation, editing, transparent generation and editing,
multi-image editing, and two outputs; TP×2 generation and editing also passed.
The picker distinguishes these from untested Turbo hardware and combinations.

The hardware latency table above describes the original 40-step checkpoint.
Turbo has the same architecture and weight-memory footprint; fewer denoising steps
do not eliminate encoder or VAE costs. Existing quantization and acceleration
results for the base checkpoint do not establish Turbo quality or speed.

## 2. Model capabilities

Qwen-Image 2.1 supports text-to-image generation, single- and multi-image editing,
and RGBA output. Both the original and eight-step Turbo checkpoints support all
these modes. Choose Turbo for shorter denoising runs; compare the two checkpoints
on your prompts when fidelity is the priority.

For multi-round editing, send the previous output as the next reference image.
The server does not retain conversation state. Repeated prompts and reference
images can reuse exact encoder outputs and VAE posteriors through the default
[conditioning cache](/docs/sglang-diffusion/caching-acceleration#conditioning-cache).
Changing the prompt or image recomputes their joint embedding; unchanged images
can still reuse vision features. Condition-prefix KV reuse stays within each
request. Use `--disable-conditioning-cache` to turn off cross-request reuse.

## 3. Checkpoint layout

The checkpoint directory must contain `model_index.json` and the `processor`,
`text_encoder`, `transformer`, `vae`, and `scheduler` subdirectories. The processor
includes the Qwen3-VL tokenizer assets; no separate tokenizer directory is needed.
Use `--model-id Qwen-Image-2.1` when your local checkpoint directory has another
name. Older Qwen-Image and Qwen-Image-Edit transformer/VAE weights are incompatible.

Keep SGLang's installed dependencies. Its native encoder preserves the
reference's Transformers 4.57.3 conditioning semantics without requiring a
runtime-wide downgrade.

### Transparent PNG output

Choose **Transparent / alpha** under Request and describe an isolated subject
on a transparent background in the prompt. The picker adds this instruction
and selects PNG. `background: "transparent"` alone does not change conditioning
or remove the background; JPEG cannot retain alpha.

PNG references retain their alpha channel during editing; RGB references
receive an opaque alpha channel. The model predicts continuous alpha values,
including partly transparent edges, without thresholding or background removal.

## 4. Offline requests

Defaults are 1024×1024, CFG 1, and seed 42; output saving is enabled.
The original checkpoint defaults to 40 steps; Turbo uses its preset 8-step grid.
For GPUs that need offload, also pass the placement flags from the picker.

### Text-to-image

```bash Command theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image-2.1 \
  --prompt "A capybara reading a book by candlelight"
```

### Image-conditioned editing

```bash Command theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image-2.1 \
  --image-path /path/to/input.png \
  --prompt "Move the scene to a snowy mountain at sunrise"
```

Height and width must be positive multiples of 32. Reference images preserve
their aspect ratio and are resized to approximately the requested output area;
the same resized image feeds the VLM and VAE. Image labels are deterministic
(`Picture 1`, `Picture 2`, and so on). Multiple outputs receive independent
noise seeds and independent prefix caches.

## 5. Runtime features

Both checkpoints use Euler flow matching with CFG disabled by default: 40 steps
for the original, or the preset 8-step grid for Turbo. For CFG, provide
`--negative-prompt` and `--guidance-scale` greater than one. The API requires a
text prompt; precomputed embeddings alone are insufficient.

* **Parallelism:** TP, Ulysses, Ring, CFG parallelism, and encoder folding are
  available in the picker. The target token count, `(height / 16) × (width / 16)`,
  must be divisible by the SP degree. Ring requires FlashAttention or SageAttention.
* **Memory:** use the hardware's recommended placement. **All components
  layerwise** also streams encoder and VAE blocks, trading transfers for lower
  device memory.
* **VAE:** full-image decoding is the default except on AMD gfx1151 (Strix Halo).
  Tiling can change pixels near boundaries. With two or more GPUs, **Spatial shard**
  distributes full-image decoding without enabling tiling; floating-point
  rounding can still differ.

On gfx1151, VAE tiling defaults to enabled to avoid reported full-frame decode
hangs at 896px and higher. Explicit CLI, pipeline config, and Python API settings
take precedence: `--vae-tiling false` disables tiling but may encounter the hang.
Defaults on NVIDIA CUDA and other platforms are unchanged.

See the [performance guide](/docs/sglang-diffusion/performance-optimization)
for shared runtime options.

### Validation boundaries

The picker marks exact tested HTTP deployments, not every combination of the
features below. Functional checks do not establish quality for lossy settings.
The feature checks below cover the original checkpoint; see
[Turbo sampling](#turbo-sampling) for the Turbo validation scope.

* **Parallelism and offload:** H200/B200 full-checkpoint checks cover TP,
  Ulysses, Ring with FlashAttention, CFG parallelism, encoder folding, and
  all-component layerwise offload. Spatial VAE sharding was checked on two B200
  GPUs; floating-point rounding can differ from full-image decoding.
* **Disaggregation:** encoder, denoiser, and decoder roles were checked on three
  B200 GPUs with same-host Mooncake TCP at 512px/4 steps. Multi-host RDMA and
  multi-rank roles remain unverified for this model.
* **Quantized exports:** serialized FP8, native-name Q4\_0 GGUF, and calibrated
  ModelOpt NVFP4 component exports were checked on B200, including TP2 with
  encoder folding and separate offload checks. A community ComfyUI-GGUF Q4\_K\_M
  DiT, which also quantizes the projections outside the blocks, was checked for
  text-to-image on one RTX 5090. These results do not establish support for
  arbitrary community exports, other GGUF types, or other hardware.
  The NVFP4 export used six calibration requests; assess image and alpha quality
  on your workload.
* **LoRA:** synthetic Diffusers-format adapters were checked on one B200 and
  TP2 with encoder folding. Trained-adapter quality remains unverified.

### Quantization

Native precision is the default. Quantization changes image and alpha values;
check quality on your own prompts and reference images. Set compatible component
paths under **Variables** when choosing an exported format. Adding quantization
metadata to native weights does not convert them.

For online FP8, use `--component-quantizations.transformer fp8`,
`--component-quantizations.text_encoder fp8`, or both.

### Serialized FP8 components

Select a **Serialized FP8** option and set the exported component directories.
Each directory needs its architecture `config.json`, weights, and quantization
metadata. Use `--component-paths.transformer` and/or
`--component-paths.text_encoder`; omit online quantization flags.
See the [quantization guide](/docs/sglang-diffusion/quantization) for formats.

### GGUF components

Select a **GGUF** option and set the `.gguf` files. The picker uses
`--component-weights-paths.transformer` and/or
`--component-weights-paths.text_encoder`, retaining architecture configs from the
base checkpoint. Each file must contain the entire component with native tensor
names. GGUF reduces weight storage but does not guarantee lower latency.
See the [GGUF guide](/docs/sglang-diffusion/quantization#gguf).

### NVFP4 components

NVFP4 requires Blackwell and compatible ModelOpt exports. Select the component
directories using `--component-paths.transformer` and/or
`--component-paths.text_encoder`. Keep the FlashInfer backend at `auto` on
RTX 5090, RTX PRO 6000, and DGX Spark: TensorRT-LLM FP4 GEMM does not support SM12.x.
These GPUs remain unverified for this model's NVFP4 exports. See the
[NVFP4 guide](/docs/sglang-diffusion/quantization#modelopt-nvfp4).

### LoRA and execution options

Use `--lora-path` and `--lora-merge-mode dynamic|merge` or the runtime adapter APIs.
Diffusers adapter keys prefixed with `transformer.` map to the native DiT.

Keep eager execution as the default. Breakable CUDA Graph replay requires
matching resolution and condition-prefix length; unseen shapes run eagerly.
Text buckets alone do not guarantee replay. SageAttention and Cache-DiT can
change numerical results and require quality checks for your workload.

### Cache-DiT

Enable `--enable-cache-dit true` or `SGLANG_CACHE_DIT_ENABLED=true`. 2.1 prefix
KV is per layer: each block slices caches by `_layer_id`. Cache-DiT wraps
`transformer_blocks` and forwards the same extras to every layer; without that
slice, later layers reuse layer 0 and the image collapses to color noise.
See the [Cache-DiT guide](/docs/sglang-diffusion/cache_dit).


# Qwen-Image-Edit-2511
Source: https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit

Deploy Qwen-Image-Edit-2511 with SGLang - 20B image editing model with text rendering, character consistency, and geometric reasoning.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[Qwen-Image-Edit-2511](https://huggingface.co/Qwen/Qwen-Image-Edit-2511) is the 20B editing counterpart to Qwen-Image. It is strongest at changing text, materials, lighting, viewpoint, or composition while reducing drift in regions that were not requested to change.

Choose it for identity-sensitive portrait edits, multi-person composition, typography replacement, and geometry-aware design work. It is substantially heavier than small specialist editors, and consistency is improved rather than guaranteed; evaluate untouched-region drift on the actual editing workload.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

Qwen-Image-Edit-2511 is a 20B parameter model optimized for image editing tasks. The recommended launch configurations vary by hardware.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements.

* `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
* `--num-gpus`: Number of GPUs to use
* `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
* `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
* `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
* `--ring-degree`: The degree of ring attention-style SP in USP

### 3.3 Decompose an image into layers on H200

`Qwen/Qwen-Image-Layered` returns separate RGBA images. For this model,
`--num-frames 4` requests four output layers. The CLI saves all four PNGs,
and `DiffGenerator.generate()` returns one result per layer.

On Linux with NVIDIA CUDA and two H200 GPUs, you can run the conditional and
unconditional branches on separate GPUs:

```bash Command theme={null}
CUDA_VISIBLE_DEVICES=0,1 sglang generate \
  --model-path Qwen/Qwen-Image-Layered \
  --num-gpus 2 \
  --cfg-parallel-size 2 \
  --tp-size 1 \
  --ulysses-degree 1 \
  --quality exact \
  --enable-torch-compile false \
  --warmup-mode request \
  --width 640 --height 640 --num-frames 4 \
  --num-inference-steps 50 --guidance-scale 4.0 --seed 42 \
  --image-path https://raw.githubusercontent.com/QwenLM/Qwen-Image-Layered/main/assets/test_images/4.png \
  --prompt "a high quality, cute halloween themed illustration, consistent style and lighting" \
  --output-path outputs/qwen-layered \
  --save-output
```

For a single H200, set `CUDA_VISIBLE_DEVICES=0`, `--num-gpus 1`, and
`--cfg-parallel-size 1`. Both configurations use eager execution. Layered
does not currently support breakable CUDA graph; enabling BCG falls back to
eager execution.

The Layered CFG policy gathers the branch predictions before applying the
single-GPU arithmetic order, preserving BF16 rounding and alpha values in the
validated fixed-seed example. Each GPU still holds a full DiT replica, so CFG
parallelism reduces request latency without reducing the model memory needed
on each GPU.

## 4. API Usage

For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api).

### 4.1 Edit an Image

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:3000/v1")

response = client.images.edit(
    model="Qwen/Qwen-Image-Edit-2511",
    image=open("input.png", "rb"),
    prompt="Change the color of the taxi to black.",
    n=1,
    response_format="b64_json",
)

# Save the edited image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
    f.write(image_bytes)
```

### 4.2 Advanced Usage

#### 4.2.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit).

**Basic Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Qwen/Qwen-Image-Edit-2511
```

**Advanced Usage**

* DBCache Parameters: DBCache controls block-level caching behavior:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

Combined Configuration Example:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Qwen/Qwen-Image-Edit-2511
```

#### 4.2.2 CPU Offload

* `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
* `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
* `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference.
* `--vae-cpu-offload`: Use CPU offload for VAE.
* `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".

#### 4.2.3 Known LoRA examples

Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Qwen-Image-Edit examples include:

* [`ostris/qwen_image_edit_inpainting`](https://huggingface.co/ostris/qwen_image_edit_inpainting)
* [`lightx2v/Qwen-Image-Edit-2511-Lightning`](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning)

## 5. Benchmark

Test Environment:

* Hardware: NVIDIA B200 GPU (1x)
* Model: Qwen/Qwen-Image-Edit-2511
* sglang diffusion version: 0.5.6.post2

### 5.1 Speedup Benchmark

#### 5.1.1 Edit a image

**Server Command**:

```shell Command theme={null}
sglang serve --model-path Qwen/Qwen-Image-Edit-2511 --port 30000
```

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-image --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Model:                                   Qwen/Qwen-Image-Edit-2511
Dataset:                                 vbench
Task:                                    image-to-image
--------------------------------------------------
Benchmark duration (s):                  35.31
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.03
Latency Mean (s):                        35.3053
Latency Median (s):                      35.3053
Latency P99 (s):                         35.3053
--------------------------------------------------
Peak Memory Max (MB):                    47959.35
Peak Memory Mean (MB):                   47959.35
Peak Memory Median (MB):                 47959.35
============================================================
```

#### 5.1.2 Edit a image with high concurrency

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task image-to-image --num-prompts 20 --max-concurrency 20
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Model:                                   Qwen/Qwen-Image-Edit-2511
Dataset:                                 vbench
Task:                                    image-to-image
--------------------------------------------------
Benchmark duration (s):                  286.11
Request rate:                            inf
Max request concurrency:                 20
Successful requests:                     20/20
--------------------------------------------------
Request throughput (req/s):              0.07
Latency Mean (s):                        150.0428
Latency Median (s):                      150.0600
Latency P99 (s):                         283.3843
--------------------------------------------------
Peak Memory Max (MB):                    47971.82
Peak Memory Mean (MB):                   47971.49
Peak Memory Median (MB):                 47971.29
============================================================
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — ComfyUI's own sampler, CLIP, and
VAE drive the loop while SGLang replaces the model forward. Load the
checkpoint with `SGLDiffusion UNET Loader` and set
`model_type` to `qwen_image_edit` on the
`SGLDiffusion Options` node.

Image editing through the integrated path is experimental.


# SANA-Video
Source: https://docs.sglang.io/cookbook/diffusion/SANA-Video/SANA-Video

Generate videos with native SANA-Video 2B and SANA-Video 2.0 5B pipelines.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[SANA-Video 2B 480p](https://huggingface.co/Efficient-Large-Model/SANA-Video_2B_480p_diffusers)
is a lightweight text-to-video model served through a native SGLang Diffusion
pipeline. Its main advantage is deployment cost: at 2B parameters it serves
480p video from a single GPU with a plain `sglang serve` invocation, no
parallelism flags required.

The released generation profile produces 832×480 output at 81 frames and
16 FPS over 50 inference steps; a compact 17-frame, 8-step profile is covered
by server CI for quick validation. Motion strength can be steered directly
from the prompt with an optional `motion score: N.` suffix.

| Model ID | Task | Default output |
| - | - | - |
| `Efficient-Large-Model/SANA-Video_2B_480p_diffusers` | Text to video | 832x480, 81 frames at 16 FPS |

## 3. Serve SANA-Video

```bash Command theme={null}
sglang serve \
  --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \
  --port 30010
```

## 4. Generate a video

The following request uses the compact 17-frame, 8-step profile covered by
server CI. Use the model defaults of 81 frames and 50 steps for the released
generation profile.

```python Python theme={null}
import time
from pathlib import Path

import requests

base_url = "http://127.0.0.1:30010"
response = requests.post(
    f"{base_url}/v1/videos",
    json={
        "model": "Efficient-Large-Model/SANA-Video_2B_480p_diffusers",
        "prompt": (
            "A red tram moves slowly through a sunlit city square while "
            "pedestrians cross behind it. motion score: 30."
        ),
        "size": "832x480",
        "num_frames": 17,
        "fps": 16,
        "num_inference_steps": 8,
        "guidance_scale": 6.0,
        "seed": 42,
    },
    timeout=60,
)
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

video = requests.get(
    f"{base_url}/v1/videos/{video_id}/content",
    timeout=300,
)
video.raise_for_status()
Path("sana_video.mp4").write_bytes(video.content)
```

## 5. Request constraints

* The default profile uses `832x480`, 81 frames, 50 inference steps, and 16 FPS.
* Frame counts are aligned to `4n+1`; for example, a request for 80 frames is
  adjusted to 77.
* Use width and height values divisible by 16.
* The prompt supports an optional `motion score: N.` suffix to express the
  desired amount of motion.

## 6. SANA-Video 2.0

The separate `Efficient-Large-Model/SANA-Video_2.0_5B_720p` checkpoint supports
text-to-video and first-frame image conditioning. Its default profile is
1280×736, 193 frames at 24 FPS, 50 steps, and guidance scale 8. This integration
loads the official checkpoint directly and uses Gemma 2 and the LTX 2.3 VAE.

On a single NVIDIA CUDA GPU, generate a short video with:

```bash Command theme={null}
sglang generate \
  --model-path Efficient-Large-Model/SANA-Video_2.0_5B_720p \
  --prompt "A red tram moves slowly through a sunlit city square." \
  --width 832 --height 480 --num-frames 17 --fps 24 \
  --num-inference-steps 50 --guidance-scale 8 --seed 42 \
  --output-file-path sana_video2.mp4
```

Add `--image-path first_frame.png` to condition the video on an image. Frame
counts align to `8n+1`; width and height must be divisible by 32. The 4-step
preview is a different checkpoint and is not supported by this pipeline.

For offline weights, keep the model directory name, such as
`/path/to/SANA-Video_2.0_5B_720p`, and pass it as `--model-path`. That directory
must contain `config.yaml` and `checkpoints/SANA_Video_2.0_5B_720p.pth`. Set
`--component-paths.text_encoder /path/to/gemma-2-2b-it` and
`--component-paths.vae /path/to/ltx-2.3-vae`. The latter directory may contain
the `vae` subdirectory or the VAE files directly. Without these overrides,
the component checkpoints are downloaded automatically.


# SANA-WM
Source: https://docs.sglang.io/cookbook/diffusion/SANA-WM/SANA-WM

Deploy SANA-WM with SGLang - a camera-controlled text+image-to-video world model with WASD/IJKL 6-DoF camera control, served three ways: dense bidirectional and chunk-causal batch streaming over /v1/videos (SanaWMTwoStagePipeline), and live over a realtime WebSocket API (SanaWMRealtimePipeline, /v1/realtime_video/generate).

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

For realtime mode, use the [WebSocket client](#7-realtime-websocket-api); the Request tab only checks model metadata in that mode.

> **Installation and prerequisites**
>
> SGLang-diffusion offers multiple installation methods depending on your hardware platform. Please refer to the [SGLang Diffusion installation guide](../../../docs/sglang-diffusion/installation).
>
> SANA-WM adds the `SanaWMTransformer3DModel` + GDN kernels, the `SanaWMTwoStagePipeline` (dense bidirectional + chunk-causal streaming), and the `SanaWMRealtimePipeline` with the `/v1/realtime_video` WebSocket router. Use `sglang serve` to launch the diffusion server.

## 2. Model capabilities

[SANA-WM](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) is NVLabs' 2.6B text-and-image-to-video world model for 720p, minute-scale generation with explicit per-frame 6-DoF camera control. Its hybrid recurrent/softmax attention keeps long causal histories bounded, while an LTX-2 refiner supplies the detail that the fast Stage-1 world model does not produce on its own.

Choose the dense checkpoint for the best bounded-clip quality and the streaming checkpoint for long-running or interactive control. Streaming and realtime trade global bidirectional context for bounded state and lower response latency; the realtime WebSocket path is not bit-identical to offline batch streaming.

| Serving mode | Checkpoint and API | Best fit | Tradeoff |
| - | - | - | - |
| Dense bidirectional | `SANA-WM_bidirectional` via `/v1/videos` | Highest single-clip quality | Bounded one-shot generation; full attention cost |
| Batch streaming | `SANA-WM_streaming` via `/v1/videos` | Long clips returned as one job | Chunk-causal context rather than global bidirectional context |
| Live realtime | `SANA-WM_streaming` via WebSocket | Interactive camera events and incremental output | Stateful session; not bit-identical to batch streaming |

**Architecture & components**

| Component | Value |
| - | - |
| Stage-1 DiT | 2.6B; 20 layers, hidden 2240, 20 heads (head\_dim 112); \~10 GB |
| Attention | frame-wise Gated DeltaNet + softmax every 4th block (hybrid linear) |
| Camera | dual-branch, UCPE + PRoPE (raymap + Plücker), 6-DoF |
| VAE | LTX-2 causal, strides (T, H, W) = (8, 32, 32); \~2 GB |
| Refiner | LTX-2 Stage-2 distilled; \~41 GB |
| Output | up to 720p (704×1280) @ 16 fps, minute-scale |

For architecture and training details, see the [SANA-WM paper](https://arxiv.org/abs/2605.15178) and [model card](https://huggingface.co/Efficient-Large-Model/SANA-WM_bidirectional) (Apache-2.0).

## 3. Model Setup

Both SANA-WM checkpoints are **public** (Apache-2.0, no gating, no token) and load **directly** — there is no manual assembly step. Pass the HuggingFace repo id to `--model-path` and SGLang downloads, materializes, validates, and loads it:

| Mode | `--model-path` |
| - | - |
| Dense bidirectional (§4) | `Efficient-Large-Model/SANA-WM_bidirectional` |
| Batch streaming (§5) / realtime (§6) | `Efficient-Large-Model/SANA-WM_streaming` |

Both repo ids are registered in SGLang's **built-in model-overlay registry**, so on first load the overlay transparently materializes the official release into a runnable Diffusers directory — for the streaming checkpoint this converts the DMD self-forcing checkpoint (`sana_dit/model.pt`) into a Diffusers `transformer/` and wires the LTX-2 causal VAE, the LTX-2 refiner, and the Gemma encoders. No environment variable or `build_model_dir.sh` step is needed. (You may also pass a local, already-materialized Diffusers directory.)

The materialized checkpoint is a Diffusers directory whose `model_index.json` declares the loadable components:

| Component (`model_index.json`) | Class |
| - | - |
| `transformer` (Stage-1 DiT) | `diffusers.SanaWMTransformer3DModel` |
| `vae` | `diffusers.AutoencoderKLCausalLTX2Video` |
| `text_encoder` | `transformers.Gemma2Model` |
| `tokenizer` | `transformers.GemmaTokenizer` |
| `scheduler` | `diffusers.FlowMatchEulerDiscreteScheduler` |

How loading works:

* The server resolves the checkpoint via `maybe_download_model(model_path, force_diffusers_model=True)` and verifies it contains a `model_index.json` plus the required component subdirectories (`transformer/`, `vae/`).
* If `text_encoder` / `tokenizer` are not provided as component paths, the pipeline falls back to the default Stage-1 text encoder **`Efficient-Large-Model/gemma-2-2b-it`** (`DEFAULT_SANA_WM_TEXT_ENCODER`).
* **Pick the path with `--pipeline-class-name`.** The checkpoint's `model_index.json` `_class_name` selects the default pipeline (`SanaWMTwoStagePipeline`). Pin it explicitly to choose: `--pipeline-class-name SanaWMTwoStagePipeline` for the `/v1/videos` paths (§4–5) or `--pipeline-class-name SanaWMRealtimePipeline` for live realtime (§6). Pinning is also required if you point `--model-path` at a bare safetensors file instead of a Diffusers directory.
* **The Stage-2 LTX-2 refiner** lives under `refiner/` in the checkpoint: `refiner/transformer` (`transformer_2`), `refiner/connectors` (`connectors`), and `refiner/text_encoder` (the Gemma-3 encoder for `text_encoder_2`, whose tokenizer also serves as `tokenizer_2`). The refiner is **optional**: it is skipped (Stage-1-only output) when the env flag `SGLANG_SANA_WM_SKIP_REFINER` (or a `skip_refiner` request extra) is set, or when no `refiner/` is present (`transformer_2` unloaded). On the batch path it runs chunk-wise with `--refiner-chunked` (the official streaming path, default on) or whole-clip without it; on the realtime path the pipeline builds a `SanaWMChunkedRefinerChainStage` only when a refiner is available, and otherwise streams Stage-1 frames.

> [!NOTE]
>
> Throughout this cookbook, `<checkpoint>` stands for the appropriate SANA-WM repo id from the table above (or a local materialized Diffusers directory).

## 4. Dense bidirectional (offline `/v1/videos`)

The **bidirectional** checkpoint generates the whole clip in **one shot** (full bidirectional attention, not chunked) followed by a dense LTX-2 refiner — the highest single-clip quality, matching the NVlabs dense reference.

Launch with the two-stage pipeline and **no** `--streaming` flag (dense is the default — `streaming` defaults to `False`):

```bash Command theme={null}
sglang serve \
  --model-path Efficient-Large-Model/SANA-WM_bidirectional \
  --pipeline-class-name SanaWMTwoStagePipeline \
  --host 127.0.0.1 --port 30000
```

Then POST to **`/v1/videos`** exactly as in §5, but pass the NVlabs dense sampling defaults for closest parity — the dense path is denser than the distilled streaming few-step schedule:

```bash Command theme={null}
curl -s http://127.0.0.1:30000/v1/videos \
  -H 'content-type: application/json' -d '{
    "prompt": "a camera moving forward and turning left",
    "input_reference": "/path/to/first_frame.png",
    "num_frames": 321,
    "seed": 42,
    "fps": 16,
    "num_inference_steps": 60,
    "guidance_scale": 5.0,
    "diffusers_kwargs": {
      "action": "w-80,wl-80,l-80,wj-80",
      "intrinsics": "/path/to/intrinsics.npy"
    }
  }'
```

* `num_inference_steps` / `guidance_scale` — the dense path uses CFG; NVlabs' reference defaults to **60 steps, guidance 5.0** (the `SanaWMSamplingParams` defaults are the lighter 20 / 4.5 — pass 60 / 5.0 explicitly for dense parity).
* The dense refiner drops the leading sink frame, so a `num_frames=321` request yields 320 output frames.

## 5. Batch streaming (offline `/v1/videos`)

The **streaming** checkpoint generates a **full camera-controlled clip in one request** — no websocket. This is SGLang's offline streaming path: the whole clip is generated chunk-by-chunk internally, refined, decoded, and returned as one video.

Launch with the two-stage pipeline + the streaming flags:

```bash Command theme={null}
sglang serve \
  --model-path Efficient-Large-Model/SANA-WM_streaming \
  --pipeline-class-name SanaWMTwoStagePipeline \
  --streaming --refiner-chunked \
  --host 127.0.0.1 --port 30000
```

* `--streaming` — chunk-causal `forward_long` Stage-1 (vs the dense one-shot path of §4).
* `--refiner-chunked` — chunk-wise streaming LTX-2 refiner (**on by default**). To use the whole-clip dense refiner instead (also valid, higher peak memory), pass `--refiner-chunked false` — simply omitting the flag keeps the default chunked refiner.
* `--num-frame-per-block N` — latent frames per chunk (default `3`).

Then POST to **`/v1/videos`** (JSON body shown below; multipart/form-data with an uploaded `input_reference` file also works). Camera control goes in `diffusers_kwargs` — the action-DSL string (§8) and the intrinsics:

```bash Command theme={null}
curl -s http://127.0.0.1:30000/v1/videos \
  -H 'content-type: application/json' -d '{
    "prompt": "a camera moving forward and turning left",
    "input_reference": "/path/to/first_frame.png",
    "num_frames": 321,
    "seed": 42,
    "fps": 16,
    "diffusers_kwargs": {
      "action": "w-80,wl-80,l-80,wj-80",
      "intrinsics": "/path/to/intrinsics.npy"
    }
  }'
```

| Field | Notes |
| - | - |
| `prompt` | text prompt |
| `input_reference` | first-frame image — a server-side path, or (multipart) an uploaded file. For an `http(s)://` URL in a JSON body, use the separate `reference_url` field (the server downloads it and assigns it to `input_reference`) |
| `num_frames` | total pixel frames (e.g. `321` → 41 latent frames, 13 chunks; output 704×1280) |
| `seed` | RNG seed (default `42`) |
| `fps` | output frame rate — **pass `16`** (SANA-WM's native rate). The generic `/v1/videos` default is `24`, which would encode the same frames at 24 fps and make the clip play \~33% shorter (16/24 of the duration) |
| `diffusers_kwargs.action` | camera action-DSL string (§8) |
| `diffusers_kwargs.intrinsics` | path to a camera-intrinsics `.npy` (per-frame `(T,3,3)`) or an inline 3×3 / `(T,3,3)` list |

The response is a `VideoResponse`; fetch the rendered MP4 via the returned reference or `GET /v1/videos/{id}/content`. The streaming hyperparameters (`num_frame_per_block`, `denoising_step_list`, `sink_size`, `num_cached_blocks`, `streaming_cfg_scale`) are **pipeline-config** defaults on `SanaWMPipelineConfig`, not request fields — see §9.

## 6. Launch the Realtime Server

Launch with the realtime pipeline **pinned** — the checkpoint defaults to `SanaWMTwoStagePipeline`, so realtime must be selected explicitly (see §3). The `/v1/realtime_video` router is always mounted and becomes functional once the realtime config is active, because `SanaWMRealtimeConfig` has a registered realtime adapter (`SanaWMRealtimeAdapter`).

```bash Command theme={null}
sglang serve \
  --model-path Efficient-Large-Model/SANA-WM_streaming \
  --pipeline-class-name SanaWMRealtimePipeline \
  --host 127.0.0.1 --port 30000
```

Common launch variants:

```bash Command theme={null}
# recommended multi-GPU realtime profile
sglang serve \
  --model-path Efficient-Large-Model/SANA-WM_streaming \
  --pipeline-class-name SanaWMRealtimePipeline \
  --num-gpus 8 --sp-degree 8 \
  --host 127.0.0.1 --port 30000

# single GPU
sglang serve \
  --model-path Efficient-Large-Model/SANA-WM_streaming \
  --pipeline-class-name SanaWMRealtimePipeline \
  --num-gpus 1 --host 127.0.0.1 --port 30000

# offload DiT + text encoder to CPU (tight VRAM)
sglang serve \
  --model-path Efficient-Large-Model/SANA-WM_streaming \
  --pipeline-class-name SanaWMRealtimePipeline \
  --host 127.0.0.1 --port 30000 \
  --dit-cpu-offload --text-encoder-cpu-offload
```

Notes on launch behavior:

* **Default endpoint** is `127.0.0.1:30000` (`--host` / `--port` override).
* **CPU offload flags are optional.** `--dit-cpu-offload`, `--text-encoder-cpu-offload`, and `--image-encoder-cpu-offload` are available; defaults are auto-adjusted from GPU memory (GPUs under 30 GB get more aggressive offloading).
* **Multi-GPU realtime.** Prefer explicit sequence parallelism (`--sp-degree` equal to the number of GPUs for a single session). Do not enable CFG parallel for the realtime profile: the default request uses `guidance_scale=1.0`, so the extra CFG ranks only recompute the same branch.
* **FSDP.** Use `--use-fsdp-inference` only when you specifically need weight sharding for memory. For the low-latency realtime profile, prefer keeping components resident and using SP first.
* **Warmup.** Server warmup is **automatically skipped** for the realtime pipeline — a synthetic warmup request has no WebSocket session, so the server detects the registered realtime adapter and skips it. No explicit `--warmup-mode` setting is needed.

Once up, the realtime WebSocket endpoint lives at `ws://127.0.0.1:30000/v1/realtime_video/generate` (use the Python client in §7 to connect — plain `curl` does not speak the `ws://` upgrade).

## 7. Realtime WebSocket API

The realtime API is a single WebSocket at **`/v1/realtime_video/generate`**. Requests and control messages use **msgpack** (`msgspec.msgpack.encode` / `decode`), not JSON. Large frame batches use a msgpack header followed by a separate binary payload.

The lifecycle is:

**1. Connect & send INIT**

The client opens the WebSocket and sends exactly one **init** message (`type: "init"`), carrying the prompt, the required `first_frame`, output/sampling options, and optional camera conditions in `condition_inputs`.

**2. Stream live EVENTs (optional)**

While generation runs, the client may push **event** messages (`type: "event"`) to steer the camera — either `kind: "camera_actions"` (frame-by-frame lists or state transitions) or `kind: "action"` (an action-DSL string).

**3. Receive frame batches**

The server streams **frame batches** back; `is_final_frame_batch: true` marks the end of a chunk. Clients must handle both output formats described below, as well as `chunk_stats` timing messages and `error` responses.

### INIT message

`RealtimeVideoGenerationsRequest` (`type` is the literal `"init"`). Key fields:

| Field | Type | Notes |
| - | - | - |
| `type` | `"init"` | Required literal |
| `prompt` | str | Text prompt |
| `first_frame` | bytes \| str | **Required by the SANA-WM adapter** (`on_init` raises if absent), though the generic request schema defines it as optional. Raw image bytes, a server-side path, or an `http(s)://` URL (downloaded & cached) |
| `condition_inputs` | dict | Camera/conditioning inputs (see below) |
| `num_frames` | int | Total frames to generate. **Omit it for an open-ended, continuous session** — the adapter leaves `num_frames` unset and flags an open-ended run (`condition_inputs["sana_wm_open_ended"] = True`), generating uniform chunks indefinitely (until `max_chunks` or the client disconnects). Provide an integer for a fixed-length clip |
| `seed` | int | RNG seed (default `42`) |
| `size` | str | `"WIDTHxHEIGHT"`; realtime requests default to `"832x480"` for latency. Pass `"1280x704"` for the native landscape resolution |
| `max_chunks` | int | Optional cap on total chunks generated |
| `num_inference_steps` | int | Default `4` for SANA-WM (realtime adapter) |
| `guidance_scale` | float | Default `1.0` |
| `realtime_output_format` | `"raw"` \| `"webp"` \| `"jpeg"` | Frame encoding for output (see below) |
| `realtime_causal_sink_size` | int | Optional override |
| `realtime_causal_kv_cache_num_frames` | int | Optional override |

`condition_inputs` accepts (all optional; pass **only one** of `action` / `camera_actions`):

| Key | Type | Meaning |
| - | - | - |
| `camera_actions` | `list[list[str]]` or `{mode: "state", transitions: [...]}` | Frame-by-frame camera actions, or state-based transitions |
| `action` | str | Action-DSL string, e.g. `"w-10,none-5,a-8"` (see §8) |
| `intrinsics_path` | str | Server-side path to a camera-intrinsics **`.npy`** file (loaded via `np.load`; shapes `(4,)`, `(3,3)`, or `(F,3,3)`) |
| `intrinsics` | list | Inline intrinsics with shape `(4,)`, `(3,3)`, `(F,4)`, or `(F,3,3)` |

If you omit both `intrinsics_path` and `intrinsics`, SGLang uses a centered heuristic intrinsic matrix derived from the first-frame size. Pass explicit intrinsics when you need closer camera parity with a prepared trajectory.

```json INIT (msgpack dict) — open-ended (omit num_frames) theme={null}
{
  "type": "init",
  "prompt": "beautiful landscape video",
  "first_frame": "<bytes or url>",
  "size": "832x480",
  "seed": 42,
  "max_chunks": 10,
  "realtime_output_format": "raw",
  "num_inference_steps": 4,
  "guidance_scale": 1.0,
  "condition_inputs": {
    "camera_actions": [["w"], [], ["a", "s"]],
    "intrinsics_path": "/path/to/intrinsics.npy"
  }
}
```

### Live EVENT messages

`RealtimeEvent` (`type: "event"`). Use `kind` + `payload` (optional `event_id` correlates the response back to this event).

```json EVENT - camera_actions (frame-by-frame list[list[str]]) theme={null}
{
  "type": "event",
  "kind": "camera_actions",
  "event_id": 1,
  "payload": [["w"], ["w"], ["a"], []]
}
```

```json EVENT - camera_actions (state-based transitions) theme={null}
{
  "type": "event",
  "kind": "camera_actions",
  "event_id": 2,
  "payload": {
    "mode": "state",
    "transitions": [
      {"actions": ["w"], "client_ts_ms": 1000},
      {"actions": ["a", "w"], "client_ts_ms": 1500}
    ]
  }
}
```

```json EVENT - action (DSL string) theme={null}
{
  "type": "event",
  "kind": "action",
  "event_id": 3,
  "payload": "w-10,none-5,a-8,d-10"
}
```

### Server frame output

The server streams **frame batches** in two formats:

* `frame_batch`: one msgpack message with header fields and an inline `payload` bytes field.
* `frame_batch_header`: a msgpack header followed immediately by one binary WebSocket message containing the payload. Do not msgpack-decode that second message; check its length against `total_size`.

The second format avoids copying large payloads into a msgpack envelope. Error and timing messages do not interrupt a header/payload pair. If the connection closes before the payload arrives, discard the incomplete batch.

Header fields:

| Field | Meaning |
| - | - |
| `type` | `"frame_batch"` or `"frame_batch_header"` |
| `request_id` | Generation id |
| `chunk_index` | Chunk index |
| `content_type` | `application/x-raw-rgb`, `image/webp`, or `image/jpeg` |
| `num_frames` | Frames in this batch |
| `total_size` | Payload size in bytes (`len(payload)`) |
| `width`, `height`, `channels` | Frame geometry (`channels: 3`) |
| `bytes_per_frame` | Bytes per raw frame (`width*height*3`) |
| `format` | `rgb24` for raw |
| `encoding` | `raw`, `webp`, or `jpeg` |
| `event_id` | Echoes the steering event id; **omitted** from the header for INIT-only chunks |
| `frame_batch_index`, `num_frame_batches` | Sequence multiple batches within a chunk |
| `is_final_frame_batch` | `true` ends the chunk |

```json Server output - frame_batch_header (msgpack dict) theme={null}
{
  "type": "frame_batch_header",
  "request_id": "uuid-string",
  "chunk_index": 0,
  "content_type": "application/x-raw-rgb",
  "num_frames": 3,
  "total_size": 8110080,
  "width": 1280,
  "height": 704,
  "channels": 3,
  "bytes_per_frame": 2703360,
  "format": "rgb24",
  "encoding": "raw",
  "event_id": 1,
  "frame_batch_index": 0,
  "num_frame_batches": 1,
  "is_final_frame_batch": true
}
```

For this header, the next binary message contains 8,110,080 bytes of raw RGB24 data.

**Encodings.** `application/x-raw-rgb` is uncompressed RGB24 (3 × uint8, `bytes_per_frame = width*height*3`), sent by default or with `realtime_output_format: "raw"`; `"webp"` / `"jpeg"` send preview-encoded frames.

### Minimal client example

```python Python theme={null}
import msgspec
import numpy as np
import websockets  # pip install websockets

WS_URL = "ws://127.0.0.1:30000/v1/realtime_video/generate"

async def run():
    async with websockets.connect(WS_URL, max_size=None) as ws:
        # 1) INIT — omit num_frames for an open-ended session; "raw" = uncompressed RGB24
        with open("first_frame.png", "rb") as f:
            first_frame = f.read()
        await ws.send(msgspec.msgpack.encode({
            "type": "init",
            "prompt": "a camera moving forward and turning right",
            "first_frame": first_frame,
            "size": "832x480",
            "seed": 42,
            "max_chunks": 10,
            "realtime_output_format": "raw",
            "num_inference_steps": 4,
            "guidance_scale": 1.0,
            "condition_inputs": {
                "action": "w-100,wd-50,d-30",
                "intrinsics_path": "/path/to/intrinsics.npy",  # optional; centered heuristic if omitted
            },
        }))

        # 2) optional: steer mid-stream
        await ws.send(msgspec.msgpack.encode({
            "type": "event",
            "kind": "camera_actions",
            "event_id": 1,
            "payload": [["w"], ["w"], ["a"], []],
        }))

        # 3) receive frame batches (raw RGB24)
        async for message in ws:
            msg = msgspec.msgpack.decode(message)
            if msg["type"] == "frame_batch_header":
                payload = await ws.recv()
            elif msg["type"] == "frame_batch":
                payload = msg["payload"]
            elif msg["type"] == "error":
                print("Server error:", msg["content"])
                continue
            else:
                continue  # skip chunk_stats etc.
            if not isinstance(payload, bytes) or len(payload) != msg["total_size"]:
                raise ValueError("Invalid frame payload")
            n, h, w, c = msg["num_frames"], msg["height"], msg["width"], msg["channels"]
            frames = np.frombuffer(payload, dtype=np.uint8).reshape(n, h, w, c)
            # ... display/save frames ...
            if msg.get("is_final_frame_batch") and msg.get("chunk_index", 0) >= 9:
                break

# asyncio.run(run())
```

## 8. Camera Action DSL

Camera trajectories are described by a compact string of comma-separated `<keys>-<frames>` segments, e.g. `"w-100,wd-50,d-30,none-10"`. This is the format accepted by `condition_inputs.action` at init and by `kind: "action"` events.

Parsing rules (`parse_action_string`):

* Each segment is `<keys>-<frames>`; `<frames>` must be a positive integer.
* `none` means no motion for that span: `none-10` = 10 static frames.
* Keys are case-insensitive; combined keys apply simultaneously (`wd` = forward + right strafe). Allowed keys are exactly `wasdijkl`.

| Key | Motion |
| - | - |
| `w` / `s` | move forward / backward |
| `a` / `d` | strafe left / right |
| `i` / `k` | look (pitch) up / down |
| `j` / `l` | look (yaw) left / right |

Pose generation (`action_string_to_c2w`):

* **Translation** (`w`/`s`/`a`/`d`) moves at `translation_speed` (default `0.04` world-units/frame).
* **Rotation** (`i`/`k` pitch, `j`/`l` yaw) turns at `rotation_speed_deg` (default `1.2`°/frame); pitch is clamped to ±85°.
* **Strafe-yaw coupling** (coefficient `0.4`): a `d` (right) strafe also nudges yaw right and `a` (left) nudges yaw left, so `wd` traces a curving arc rather than a pure sidestep.
* Produces `(F+1, 4, 4)` camera-to-world matrices; the realtime stage pads the trajectory to the requested frame count.

Example: `"w-100,wd-50,d-30,none-10"` = 100 frames forward → 50 frames forward + sweep right → 30 frames right strafe → 10 frames static.

## 9. Configuration Reference

SANA-WM's defaults live in three places: **request-time** sampling params, the **pipeline config** (streaming/refiner knobs), and the **realtime adapter** (init-time overrides).

### Request-time — `SanaWMSamplingParams` (`configs/sample/sana_wm.py`)

| Field | Default | Purpose |
| - | - | - |
| `height` | `704` | Output height |
| `width` | `1280` | Output width |
| `num_frames` | `49` | Total pixel frames (must satisfy `(num_frames - 1) % 8 == 0`) |
| `fps` | `16` | Output frame rate (overrides the base default of 24) |
| `num_inference_steps` | `20` | Stage-1 step count |
| `guidance_scale` | `4.5` | Dense-path CFG scale |
| `negative_prompt` | `""` | Negative prompt |
| `camera_to_world` | `None` | In-memory `(T,4,4)` c2w extrinsics (mutually exclusive with `action`) |
| `intrinsics` | `None` | In-memory `(T,3,3)` pinhole intrinsics |
| `action` | `None` | Action-DSL string (see §8) |
| `translation_speed` | `0.04` | World-units/frame for W/S/A/D |
| `rotation_speed_deg` | `1.2` | Degrees/frame for I/K/J/L |
| `pitch_limit_deg` | `85.0` | Pitch clamp |

`generator_device` is inherited from the base `SamplingParams` (default `None` = use the pipeline/model default). On the `/v1/videos` HTTP API the camera fields are passed inside `diffusers_kwargs` (`action` / `intrinsics`, as in §4–5).

### Pipeline config — `SanaWMPipelineConfig` (`configs/pipeline_configs/sana_wm.py`)

These are server-launch knobs (set via the `--streaming` / `--refiner-chunked` / `--num-frame-per-block` CLI flags or a pipeline-config override), **not** request fields:

| Field | Default | Purpose |
| - | - | - |
| `streaming` | `False` | Chunk-causal `forward_long` (§5) vs dense one-shot (§4) |
| `refiner_chunked` | `True` | Chunk-wise streaming refiner vs whole-clip dense refiner |
| `num_frame_per_block` | `3` | Latent frames per Stage-1 / refiner chunk |
| `num_cached_blocks` | `2` | Rolling KV-cache history window |
| `denoising_step_list` | `(1000, 960, 889, 727, 0)` | 4-step streaming self-forcing timesteps (must end in 0) |
| `streaming_cfg_scale` | `1.0` | CFG scale for the distilled streaming path (1.0 = off) |
| `sink_size` | `1` | Sink (unrefined context) frames |
| `refiner_block_size` | `3` | Refiner block size |
| `refiner_kv_max_frames` | `11` | Refiner sliding KV window |

### Realtime adapter init overrides — `SanaWMRealtimeAdapter`

At WebSocket `init` the realtime adapter fills SANA-WM defaults that differ from the request/sampling defaults above:

| Field | Realtime default | Note |
| - | - | - |
| `size` | `832x480` | Realtime request default; pass `1280x704` for native landscape output |
| `num_frames` | *(unset)* | Omitting → open-ended continuous session (§7) |
| `num_inference_steps` | `4` | Distilled few-step |
| `guidance_scale` | `1.0` | CFG off |
| `fps` | `16` | Native rate |

> [!NOTE]
>
> `guidance_scale` applies to the dense path (§4) only; the distilled streaming path uses `streaming_cfg_scale` (default `1.0`, i.e. no CFG) so a `guidance_scale` override never accidentally enables CFG on the streaming stage. `denoising_step_list = (1000, 960, 889, 727, 0)` is the official 4-step streaming schedule (it must end in 0).

## 10. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# SenseNova-U1.5-8B-MoT
Source: https://docs.sglang.io/cookbook/diffusion/SenseNova/SenseNova-U1.5-8B-MoT

Run SenseNova-U1.5-8B-MoT text-to-image generation with SGLang's native multimodal generation runtime.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[SenseNova-U1.5-8B-MoT](https://huggingface.co/sensenova/SenseNova-U1.5-8B-MoT) is a text-to-image model in the SenseNova-U1 family. It is useful for high-resolution poster, infographic, and enterprise visual-generation workloads where prompt following, layout structure, and Chinese text rendering are important.

Built on [NEO-Unify](https://huggingface.co/blog/sensenova/neo-unify), this release improves composition, color harmony, material rendering, and local visual detail for native high-resolution generation. It also focuses on clearer Chinese and English text rendering, stronger infographic layout structure, and more reliable execution of complex prompts with object counts, spatial relationships, styles, and multiple constraints.

## 3. Model Deployment

### 3.1 Offline Image Generation

SenseNova-U1.5-8B-MoT is a text-to-image model. You can run a single offline generation job with `sglang generate`:

```bash Command theme={null}
sglang generate \
  --model-path sensenova/SenseNova-U1.5-8B-MoT \
  --prompt "A cinematic mountain lake at sunrise, realistic photography." \
  --width 2048 \
  --height 2048 \
  --guidance-scale 4.0 \
  --num-inference-steps 50 \
  --seed 42 \
  --output-file-path output_sensenova_u15.png
```

### 3.2 Recommended Resolutions

SenseNova-U1 was trained around the 2K-pixel resolution tier. The SGLang pipeline only supports positive `--width` and `--height` values that are divisible by 32, and generation quality may degrade for untrained size combinations.

Recommended resolution tiers:

| Aspect ratio | Resolution |
| - | - |
| 1:1 | `2048 x 2048` |
| 16:9 / 9:16 | `2720 x 1536` / `1536 x 2720` |
| 3:2 / 2:3 | `2496 x 1664` / `1664 x 2496` |
| 4:3 / 3:4 | `2368 x 1760` / `1760 x 2368` |
| 2:1 / 1:2 | `2880 x 1440` / `1440 x 2880` |
| 3:1 / 1:3 | `3456 x 1152` / `1152 x 3456` |
| 9:16 4K | `2304 x 4096` |

## 4. API Usage

For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai_api).

### 4.1 Generate an Image

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")

response = client.images.generate(
    model="sensenova/SenseNova-U1.5-8B-MoT",
    prompt="A clean technology poster with structured layout and clear typography.",
    n=1,
    response_format="b64_json",
)

image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output_sensenova_u15.png", "wb") as f:
    f.write(image_bytes)
```

## 5. Advanced usage

### 5.1 Dynamic request batching

Start the server with dynamic batching enabled:

```bash Command theme={null}
sglang serve \
  --model-path sensenova/SenseNova-U1.5-8B-MoT \
  --port 30000 \
  --batching-max-size 2 \
  --batching-delay-ms 100
```

Submit requests concurrently so they arrive within the batching delay. Requests
in one batch must use the same resolution, inference steps, guidance settings,
and output options. Requests with multiple outputs or `think_mode=true` are
executed sequentially.

### 5.2 Ascend NPU optimizations

On Ascend NPU, SenseNova-U1 automatically uses fused inference attention,
RMSNorm, and SwiGLU MLP operators for supported inputs. FIA and SwiGLU fall back
when their requirements are not met. Run one complete warmup request before
benchmarking because the fused MLP packs its gate and up projection weights on
first use.

## 6. Benchmark

### 6.1 Speedup benchmark

#### 6.1.1 Single-run profile

**NVIDIA A800**

This is a single-run profile for smoke validation rather than a formal benchmark. Environment: NVIDIA A800; workload: vbench text-to-image, 1 prompt, concurrency 1, 1 output per prompt; warmup: none; repetitions: 1; dispersion: not reported; baseline: none; revisions: PR checkout and resolved model snapshot at run time; output validation: request completed and returned one image.

**Server Command**:

```shell Command theme={null}
sglang serve \
  --model-path sensenova/SenseNova-U1.5-8B-MoT \
  --port 30000
```

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
  --dataset vbench \
  --task text-to-image \
  --num-prompts 1 \
  --max-concurrency 1 \
  --port 30000
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        /workspace/models/ModelScope/SenseNova/SenseNova-U1.5-8B-MoT
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       13.10
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.08
Output throughput (outputs/s):                0.08
Latency Mean (s):                             13.10
Latency Median (s):                           13.10
Latency P90 (s):                              13.10
Latency P95 (s):                              13.10
Latency P99 (s):                              13.10
--------------------------------------------------
Peak Memory Max (MB):                         34326.00
Peak Memory Mean (MB):                        34326.00
Peak Memory Median (MB):                      34326.00
------------------------------------------------------------
```

**Ascend 910C**

Environment: one Ascend 910C; workload: four concurrent 2048 x 2048
text-to-image requests, 50 denoising steps, CFG 4, BF16.

| Batch size | NPU operators | Duration (s) | Throughput (images/s) | Mean latency (s) | Peak reserved memory (MB) |
| -: | - | -: | -: | -: | -: |
| 1 | Disabled | 234.28 | 0.01707 | 146.47 | 35698 |
| 2 | Disabled | 234.83 | 0.01703 | 176.06 | 37870 |
| 1 | FIA + RMSNorm + SwiGLU | 205.94 | 0.01942 | 128.80 | 35718 |
| 2 | FIA + RMSNorm + SwiGLU | 195.02 | 0.02051 | 146.33 | 37850 |

Optimized B1 improves throughput by 13.76% over unoptimized B1. Optimized
B2 improves throughput by 20.42% over unoptimized B2 and by 5.60% over
optimized B1.


# HunyuanImage-3.0-Instruct
Source: https://docs.sglang.io/cookbook/diffusion/Tencent/HunyuanImage3

Deploy HunyuanImage-3.0-Instruct for text-to-image generation and reference-image editing.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). The command builder below targets Linux/CUDA; use the [Ascend installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) for NPU environments. Exact deployment recipes remain unverified unless explicitly marked otherwise.

```bash theme={null}
uv pip install -e 'python[diffusion]'
```

## 2. Model capabilities

[HunyuanImage-3.0-Instruct](https://huggingface.co/tencent/HunyuanImage-3.0-Instruct) combines a text-and-image transformer with diffusion image generation. It supports text-to-image requests and instruction-guided editing with reference images. The native pipeline reuses the transformer's text prefix during denoising and uses a SigLIP2 encoder plus a convolutional VAE for image conditioning.

Choose it when one checkpoint must handle both generation and reference-based editing. Its 80B total parameters, despite approximately 13B active per token, make weight residency a major deployment constraint. This recipe covers the Instruct checkpoint, not the separate distilled release. Start with tensor parallelism and the default 50-step, guidance-2.5 sampling profile before testing acceleration options.

## 3. Reference-image editing

Use the same server with the images edit endpoint. Repeat `image[]` for multiple reference images. The file paths below are local to the client.

```bash theme={null}
curl -sS http://localhost:30010/v1/images/edits \
  -F model=tencent/HunyuanImage-3.0-Instruct \
  -F 'prompt=Change the background to a quiet seaside town; preserve the subject.' \
  -F 'image[]=@reference.png' \
  -F num_inference_steps=50 \
  -F guidance_scale=2.5 \
  -F seed=42 \
  -F response_format=b64_json
```

For offline generation, pass the same model and topology to `sglang generate`:

```bash theme={null}
sglang generate --model-path tencent/HunyuanImage-3.0-Instruct \
  --num-gpus 2 --tp-size 2 --ulysses-degree 1 \
  --prompt "Change the background to a quiet seaside town; preserve the subject." \
  --image-path reference.png --num-inference-steps 50 --guidance-scale 2.5 \
  --seed 42 --save-output
```

## 4. Request semantics

* **Resolution:** `width` and `height` select an aspect ratio by default, not an exact generation canvas. The processor selects a native bucket; decoding crops or pads it. Editing without an explicit size uses the first reference image's dimensions. Use `output_size_mode="exact_size"` when exact output dimensions matter; this resamples the decoded result rather than changing its native bucket.
* **Prompting:** `bot_task="image"` and `system_prompt="en_unified"` are the defaults. Other `bot_task` values change the response prefix. They do not run a separate autoregressive reasoning or recaptioning loop. `cot_text` accepts an already-generated reasoning/recaption prefix.
* **Multiple outputs:** `n` on the HTTP API or `--num-outputs-per-prompt` offline controls output count. More outputs require additional activation memory.

## 5. Deployment boundaries

For an Ascend environment with four visible NPU devices and sufficient aggregate memory, the equivalent TP4 launch is:

```bash theme={null}
sglang serve --model-path tencent/HunyuanImage-3.0-Instruct \
  --num-gpus 4 --tp-size 4 --ulysses-degree 1 --port 30010
```

Here `--num-gpus` counts visible accelerator devices, including NPUs. Follow the platform installation guide for compatible PyTorch/CANN packages; do not install the CUDA extra into an NPU environment.

Use TP to shard transformer weights; keep Ulysses and Ring at 1. This pipeline does not implement sequence-parallel attention or CFG-parallel execution. The vision encoder and VAE remain replicated, so increasing TP does not divide every component's memory usage. Approximately 160 GB of BF16 transformer weights is only a capacity estimate, not a peak-VRAM measurement: leave room for those components, activations, and loading buffers.

Compatible requests can be grouped with `--batching-max-size` and `--batching-delay-ms`. Begin at batch size 1, then validate peak memory at the largest reference-image count and output size you intend to serve. Different prompt/seed values can share a batch; incompatible conditioning and sampling settings cannot.

The pipeline has a model-specific Cache-DiT adapter. Caching reuses intermediate results and is not numerically lossless; measure quality and latency on your workload before enabling it. See [Cache-DiT](/docs/sglang-diffusion/cache_dit). SP, CFG parallel, quantization, and full-checkpoint CUDA performance are not claimed as verified by this recipe.


# Wan-Animate-2
Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan-Animate-2

Run Wan-Animate-2 character animation (reference image + reference video) with SGLang Diffusion, with the base or the distilled checkpoint.

## 1. Quick start

Follow the [installation guide](/docs/sglang-diffusion/installation) on Linux with
NVIDIA CUDA, then select a verified recipe. **Checkpoint weights** under **Setup**
switches between the base and the distilled checkpoint; the server flags and the
Request command follow that choice.

Provide both a reference image and a server-readable reference video. See
[HTTP invocation](#4-2-serve-and-request-over-http) for polling and downloading
the finished video.

## 2. Model capabilities

[Wan-Animate-2](https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Diffusers) is a
14B character-animation model in the Wan2.2 family. Instead of the text- or
image-only conditioning of the other Wan2.2 checkpoints, it animates a **reference
image** so that it follows the motion of a **reference video**, producing a new clip
of that subject performing the reference video's motion.

Two checkpoints are supported; select one under **Checkpoint weights** or pass it
as `--model-path`:

| Variant | `--model-path` | Steps | `guidance_scale` | `log_scale` | CFG parallel |
| - | - | -: | -: | -: | - |
| Base | `Wan-AI/Wan2.2-Animate-2-14B-Diffusers` | `40` | `3.0` | `0` | yes |
| Distilled | `Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers` | `10` | `1.0` | `-1.3` | rejected at startup |

The distilled checkpoint differs in its transformer weights, in the
attention-score bias `log_scale` (the official distillation value, applied by
default because the published `transformer/config.json` carries no such key) and
in the two defaults above. The distilled checkpoint is sampled with DPM-Solver++
2M on the official sigma grid, like the base checkpoint: that is what the model
authors' command-line implementation runs for both checkpoints. The
`FlowMatchEulerDiscreteScheduler` named in the distilled repo's
`scheduler/scheduler_config.json` is read only by the Diffusers pipeline (which
also omits the `log_scale` bias) and is not used here; on the same seed it lands
1.8 dB farther from the authors' output. `--scheduler-class-override` is ignored
for this checkpoint and logged. A local directory routes by the pipeline class in
its `modular_model_index.json` (`WanAnimate2DistilledModularPipeline` /
`WanAnimate2ModularPipeline`) or by a `distill` token in the directory name (in a
path that also contains `wan2.2-animate-2` or `wan_animate_2`); the detectors are
checked in registration order, the distilled one first, so either signal selects
the distilled entry. `--model-id Wan2.2-Animate-2-14B-Distilled-Diffusers` pins
the variant explicitly.

Unlike the A14B T2V/I2V checkpoints, Wan-Animate-2 is a **single-expert** model:
it has one `guidance_scale` (see the variant table), no low-noise expert
(`guidance_scale_2`) and no boundary switching. Diffusers >= 0.40.0
serves Wan-Animate-2 as a modular pipeline (`WanAnimate2ModularPipeline`); SGLang
implements the model natively and loads the same checkpoints. There is no diffusers
backend fallback.

> [!WARNING]
>
> Wan-Animate-2 is served through the native `WanAnimate2Pipeline`, selected
> automatically from the checkpoint's `model_index.json`. It takes **two** inputs, a
> reference image and a reference video, so it is invoked differently from the
> other Wan2.2 models. See [section 4](#4-model-invocation).

### 2.1 Inputs

| Input | Field | Passed via |
| - | - | - |
| Reference image (the subject to animate) | `image_path` / `input_reference` | `--config` YAML for `sglang generate`; the `input_reference` part of a `POST /v1/videos` request |
| Reference video (the motion to follow) | `video_path` | `--video-path` or `--config` YAML for `sglang generate`; the `video_path` field in the `/v1/videos` request body or `extra_body` |

Both inputs are the base `image_path` / `video_path` sampling fields shared with the
other video models; Wan-Animate-2 adds no model-specific input field. Offline runs
pass both through a `--config` file (or `--image-path` / `--video-path`), and the
online API forwards `video_path` into the pipeline while the reference image rides
`input_reference`. Local paths must be readable by the server process. HTTP(S)
video URLs use SGLang's shared media loader before decoding, without a disk cache.

### 2.2 Defaults

| Parameter | Default | Notes |
| - | - | - |
| Resolution (`width`×`height`, the API's `size`) | `640`×`800` | A pixel-area budget, not the output dimensions: the frame keeps the reference image's aspect ratio and is the largest 16-aligned canvas with at most `width × height` pixels (see **Output resolution** below). `720`×`1280` is the larger budget. |
| `clip_len` | `37` | Per-clip denoising chunk length; must be `4k+1` (VAE temporal stride 4), e.g. `37`, `65`, `81`. Other values are rounded to `4k+1` with the same `k` and the rounding is logged (`40` becomes `41`; `38` and `39` become `37`); values below `5` are rejected. |
| `fps` | `16` | The reference video is resampled to this rate and the output MP4 is written at it; duration is preserved. |
| `num_inference_steps` | `40` (`10` for the distilled checkpoint) | Fewer steps (e.g. `20`) run faster on the base checkpoint. |
| `guidance_scale` | `3.0` (`1.0` for the distilled checkpoint) | Single expert; there is no `guidance_scale_2`. The distilled checkpoint samples at `1.0` only (one branch per step); a request with any other value is rejected at admission (HTTP 400 online, `ValueError` offline), see [section 6](#6-troubleshooting). |
| `prompt_ref` | `人物动作的参考视频` ("reference video of the character's motion") | Text for the reference-video branch. The official demo conditions that branch on this fixed description of the reference video rather than on the scene prompt; leave it unchanged unless reproducing a different reference setup. |
| `enable_audio` | `true` | Keep the reference video's audio track on the output (see **Audio** below). Per request, like `clip_len`. |

The distilled checkpoint inherits every other default. The official distilled demo
defaults to `clip_len` 81, `size` 720×1280 and `fps` 24; the distilled rows in
[section 5.3](#5-3-distilled-checkpoint-on-gb200) were measured at `clip_len` 37
and `size` 640×800. `clip_len` 81 at the 640×800 budget runs on one 189 GB GPU at
about 230 s.

**Output length.** The output has as many frames as the (resampled) reference
video, not `clip_len`. The video is generated clip by clip with `clip_len` frames
per chunk; a larger `clip_len` improves temporal coherence at the cost of VRAM.
`clip_len=81` roughly doubles the attention tokens of the default and OOMs a
single 80 GB GPU. `num_frames` (and the API's `seconds`, which maps to it) does
not control the length: it is ignored and the server logs that it was, so leave
it unset.

**Output resolution.** `width`×`height` (the API's `size`) is a pixel-area budget,
not the output dimensions. The reference image's aspect ratio is kept, the frame is
the largest canvas with at most `width × height` pixels whose sides are multiples of
16, and the reference image is letterboxed onto it; the driving-video frames are
fitted the same way to their own canvas at the same budget. The output MP4 has
the canvas size. At the default `640`×`800` (512,000 pixels) a
9:16 reference gives `528`×`944`, a 1:1 reference gives `704`×`704`, and only a 4:5
reference gives exactly `640`×`800`. To obtain a specific output size, request a
budget with the same aspect ratio as the reference image (for a 9:16 image,
`576`×`1024` maps onto itself). This differs from the OpenAI Videos API and from the
other SGLang diffusion pipelines, where `size` is the literal output resolution.

**Audio.** The output MP4 carries the reference video's audio track, as the
official pipeline does. Set `enable_audio=false` on a request (a `--config` field
offline, a form field or `extra_body` entry online) for a silent output. If the
track cannot be decoded, the output is silent and the server logs one warning per
process; a reference video without an audio track also gives a silent output.

**Lossless conditioning reuse.** Reference-video keys are rotated once and cached per clip, then reused by all
denoising steps and both CFG branches. This removes repeated RoPE computation
without changing the attention mask, precision, or sampling schedule. The cache
uses the existing key storage and is released after the clip; it is not an
approximate timestep-skipping cache.
Clips containing only padding that would be removed from the final video are not
denoised. The padding and conditioning of every retained clip are unchanged.

Text encoding uses the native `TextEncodingStage` and its conditioning cache and
residency management. CFG-parallel dispatch shares the native CFG utilities while
retaining the single-GPU arithmetic order.
Each clip uses the shared denoising loop for step profiling, NVTX ranges, progress,
and DiT release. Reference K/V preparation and clip-specific prediction remain
model hooks; clip decoding still feeds the next clip's conditioning. This does
not enable cache-dit or breakable CUDA graphs for this model.
Clip decoding uses the shared `DecodingStage.decode_raw` path for VAE compilation,
tiling and OOM diagnostics. Raw decoded pixels are kept for the next clip: no
extra conversion to and from `[0, 1]` is introduced. The adapter preserves the
existing FP32 VAE execution and per-channel latent scaling.
QKV projections, cross-attention, FFN and the output head are inherited from the
native Wan implementation. Ulysses exchanges use the shared packed-QKV path;
the model-specific reference attention mask and K/V lifecycle are retained.
Text and image conditioning projections are reused within each clip and CFG
branch, rather than recomputed at every timestep. These projected tensors are
released after the clip and never reused across requests. Decoded output is
copied to CPU and clip-local intermediates are released; only the decoded
overlap frames needed by the next clip remain on the GPU.
Geometry caches retain at most four RoPE tables and two attention masks, rather
than accumulating GPU tensors for every request size. Evicted entries are
recomputed without changing the results; frequently alternating among more
geometries can add setup work.

Measured on one H200, with a 576×1024 budget, 8 steps, CFG 3, seed 42,
`clip_len=37`, text-encoder CPU offload, and MP4 plus raw-frame output, the initial
reuse changes reduced warm request latency from 42.30s to 41.34s (two requests
per variant). With `clip_len=17`, omitting a fully cropped trailing clip reduced
59.60s to 40.45s (one warm request per variant). Raw RGB frames were bit-identical
and peak allocated VRAM did not increase. These are workload-specific results,
not general speedup claims; both comparisons used PyTorch 2.13.0+cu130,
Transformers 5.12.1, and Diffusers 0.37.0. The baseline was the initial native
implementation, not the official Diffusers implementation.

## 3. Model Deployment

### 3.1 Recommended configurations

Start with **Auto**, or choose **Memory** for more GPU headroom. Memory usage
depends on the input video, pixel-area budget, and `clip_len`; see the
[hardware comparison](#5-1-hardware-comparison) for measured configurations.
For two B300 GPUs, set **GPUs / node** to **2** and **Memory policy** to **CFG parity**;
**Auto** CFG parallelism enables the measured two-GPU recipe. The RTX 4090 entry
selects its single-GPU memory recipe.

With **Checkpoint weights** set to the distilled checkpoint, CFG parallelism is
off (one branch per step), **Auto** topology resolves to Ulysses across the GPUs
and **Auto** text encoder to CPU offload, the policy of the measured GB200
recipes (1, 2 and 4 GPUs, resident or DiT layerwise offload, Ulysses or tensor
parallel); see the [distilled benchmark](#5-3-distilled-checkpoint-on-gb200).

Routing to the native Wan-Animate-2 pipeline is automatic from the checkpoint's
`model_index.json` (`_class_name: WanAnimate2Pipeline`); pass
`--pipeline-class-name WanAnimate2Pipeline` only for a local mirror that lacks
`model_index.json`.

### 3.2 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for
acceleration features and their runtime requirements.

* `--num-gpus {NUM_GPUS}`: Number of GPUs to use.
* `--tp-size {TP_SIZE}`: Tensor parallelism size.
* `--ulysses-degree {ULYSSES_DEGREE}`: DeepSpeed-Ulysses-style sequence parallelism (USP). The **ring degree must be 1** for this model; `--ring-degree` greater than 1 is rejected with `NotImplementedError`. Combines with `--tp-size`: Ulysses splits the TP-local attention heads, so `--tp-size` must divide the 40 heads and `--ulysses-degree` must divide the per-shard count (see [section 3.3](#3-3-multi-gpu-and-memory-presets)).
* `--enable-cfg-parallel`: Split the guided and unguided branches across GPUs. Base checkpoint only: the distilled checkpoint has one branch and rejects the flag at startup.
* `--text-encoder-cpu-offload`: CPU-offload the text encoder to save memory; it is moved to the GPU for the one encode per request.

### 3.3 Multi-GPU and memory presets

| Target | Recommended server flags | Notes |
| - | - | - |
| 1 GPU, ample VRAM | *(no extra flags)* | Automatic placement. Start here on H200 or B300. On H200, `speed` used more memory without improving measured latency. |
| 1 GPU, memory priority | `--performance-mode memory` | Layerwise offload for supported components. On the measured H200 workload, 31.7 GiB peak versus 68.9 GiB with auto, at nearly the same latency and identical RGB output. |
| 1x RTX 4090, 24 GB | `--performance-mode memory` with `PYTORCH_ALLOC_CONF=expandable_segments:True` | Short-video recipe with little spare VRAM. The default allocator OOMed on the measured workload; offload also requires sufficient host RAM. |
| 2+ GPUs, long sequences | `--num-gpus N --ulysses-degree N` | Sequence parallel across ranks (ring degree stays 1). |
| 2+ GPUs, large DiT | `--num-gpus N --tp-size N` | Tensor parallel across attention heads. With `--tp-size` smaller than `--num-gpus` (or than half of it with `--enable-cfg-parallel`), the leftover GPUs are auto-assigned to Ulysses sequence parallelism (last row). |
| 2 GPUs, split CFG | `--num-gpus 2 --enable-cfg-parallel` | Splits the guided/unguided branches. |
| 2x B300, recommended | **CFG parity** in the picker: `--num-gpus 2 --enable-cfg-parallel --dit-layerwise-offload --layerwise-offload-components transformer --encoder-parallel replicate --vae-config.use-parallel-decode false` with `SGLANG_DIFFUSION_VAE_CHANNELS_LAST_3D=1` | 54.03 s, 39.53 GiB peak per GPU, and RGB identical to the single-B300 baseline. Only 0.50 s slower than plain CFG, with substantially less GPU memory. |
| 4 GPUs | `--num-gpus 4 --enable-cfg-parallel --ulysses-degree 2` | CFG-parallel across two pairs of Ulysses ranks: the fastest 4-GPU layout; pass the flags explicitly to pin the layout instead of relying on the automatic assignment. Ulysses shards the attention sequence, so 4-GPU outputs match the single-GPU video to about 30 dB mean PSNR rather than bit for bit. On 80 GB cards add `--dit-layerwise-offload --layerwise-offload-components transformer` (about 35 GiB per GPU, same speed). |
| FSDP sharding | `--use-fsdp-inference` is **not supported** | Rejected at startup, before any weights load; use `--tp-size` to shard the DiT or `--dit-layerwise-offload` to cut peak memory. |
| 4+ GPUs, TP x Ulysses | `--num-gpus 4 --tp-size 2 --ulysses-degree 2` | Tensor parallel across attention heads, Ulysses across the TP-local heads: `--tp-size` must divide the 40 heads and `--ulysses-degree` the per-shard count (`40 / tp`). The memory-leaning 4-GPU layout: about 33 GiB peak per GPU (25 GiB torch reserved), the lowest 4-GPU peak after `--tp-size 4`, at about 13 percent slower than CFG-parallel x tensor parallel and 27 percent slower than the default CFG-parallel x Ulysses layout; use the default for speed and this layout for memory. CFG-parallel can be added on top with twice the GPUs (`--num-gpus 8 --enable-cfg-parallel --tp-size 2 --ulysses-degree 2`). |

The distilled checkpoint (`Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers`) runs
one branch per step, so the CFG-parallel rows above do not apply to it. The rows
below are the picker's distilled recipes, all measured with
`--text-encoder-cpu-offload` (its **Auto** text-encoder policy) on 189 GB GB200
GPUs; the 80 to 96 GB advice is inferred from their peaks. The measurements, with
the request-phase and whole-run peaks defined, are in
[section 5.3](#5-3-distilled-checkpoint-on-gb200).

| Target | Recommended server flags | Notes |
| - | - | - |
| Distilled, 1 GPU, no offload, needs about 96 GiB over the whole run (69.5 GiB during the request) | *(no extra flags)* | DiT resident, the fastest single-GPU form. The whole-run peak sets the 96 GiB figure: it includes the server warmup's synthetic requests, one of which runs at the largest supported budget, 720×1280. |
| Distilled, 1 GPU, 80 to 96 GB | `--dit-layerwise-offload --layerwise-offload-components transformer` | Streams the DiT layer by layer at the same speed, with the output identical to the resident DiT, at about 40 GiB peak per GPU during the request (65 GiB over the whole run) versus 69.5 GiB resident. |
| Distilled, 2 GPUs | `--num-gpus 2 --ulysses-degree 2` | The fastest 2-GPU layout, 1.73x one GPU; the automatic 2-GPU assignment resolves to it. On 80 GB cards add `--dit-layerwise-offload --layerwise-offload-components transformer` (same speed, identical output); `--tp-size 2` is the resident lower-memory alternative. |
| Distilled, 4 GPUs | `--num-gpus 4 --ulysses-degree 4` | The fastest 4-GPU layout, 2.69x one GPU; the automatic 4-GPU assignment resolves to it. Lower-memory alternatives, fastest first: layerwise offload on top (identical output), `--tp-size 2 --ulysses-degree 2` and `--tp-size 4`. |
| Distilled, CFG parallel | `--enable-cfg-parallel` / `--cfg-parallel-size 2` are **rejected** | The server refuses to start with `Wan_Animate_2_14B_Distilled_Config does not support CFG parallelism`: the distilled model runs one branch per step, so there is nothing to split; use `--ulysses-degree` or `--tp-size`. |

## 4. Model Invocation

### 4.1 Generate offline with a config file

The offline path passes both inputs (and the sampling parameters) through a
`--config` YAML:

```yaml wan_animate_2_run.yaml theme={null}
prompt: "a person dancing"
image_path: /path/to/reference.png
video_path: /path/to/reference_video.mp4
clip_len: 37
height: 800
width: 640
fps: 16
seed: 42
```

```bash Command theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-Animate-2-14B-Diffusers \
  --config wan_animate_2_run.yaml \
  --num-gpus 1 --text-encoder-cpu-offload --pin-cpu-memory \
  --save-output --output-path outputs --output-file-name wan_animate_2.mp4
```

Any value in the YAML can also be passed as a CLI flag (for example
`--image-path`, `--video-path`, `--num-inference-steps`, `--guidance-scale`,
`--seed`). The examples on this page set neither `num_inference_steps` nor
`guidance_scale`: each checkpoint takes the defaults of the variant table in
[section 2](#2-model-capabilities); set them only to override the defaults (the
distilled checkpoint accepts `guidance_scale` 1.0 only). The picker's Request
command sets both for the selected checkpoint.

### 4.2 Serve and request over HTTP

`POST /v1/videos` creates an asynchronous job. Send the **reference image** as the
`input_reference` part and the **reference video** as `video_path`, then poll
the job and download the finished MP4. `size` is the pixel-area budget described in
[section 2.2](#2-2-defaults); the output dimensions follow the reference image's
aspect ratio.

```bash Command theme={null}
job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=a person dancing" \
  --form-string "video_path=/path/to/reference_video.mp4" \
  --form-string "clip_len=37" \
  --form-string "size=640x800" \
  --form-string "fps=16" \
  --form-string "seed=42" \
  --form-string "enable_audio=false" \
  --form "input_reference=@/path/to/reference.png;type=image/png" \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
    | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \
  -o wan_animate_2.mp4
```

These examples produce silent video; use `enable_audio=true` to retain reference
audio. With the OpenAI Python client, pass the image as `input_reference` and
the server-readable `video_path` and `clip_len` through `extra_body`:

```python Python theme={null}
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://127.0.0.1:30010/v1")
video = client.videos.create(
    model="Wan-AI/Wan2.2-Animate-2-14B-Diffusers",
    prompt="a person dancing",
    input_reference=open("/path/to/reference.png", "rb"),
    extra_body={
        "video_path": "/path/to/reference_video.mp4",
        "clip_len": 37,
        "enable_audio": False,  # silent output; the reference video's audio is kept by default
    },
)
```

For more API usage and request examples, see the
[SGLang Diffusion OpenAI API](/docs/sglang-diffusion/api/openai_api) reference.

### 4.3 Reducing memory

Use memory mode to stream supported components layer by layer from host RAM:

```bash Command theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.2-Animate-2-14B-Diffusers \
  --performance-mode memory \
  --port 30010
```

For DiT-only offload, choose **DiT layerwise offload** in the picker instead.
On RTX 4090, prefix the command with `PYTORCH_ALLOC_CONF=expandable_segments:True`
as shown in its hardware preset; memory mode alone OOMed in the tested workload.
Lowering `clip_len` also reduces memory, but shortens each clip's temporal context.
The distilled checkpoint has the same DiT, so the same flags apply with the same
effect; its layerwise-offload rows are in
[section 5.3](#5-3-distilled-checkpoint-on-gb200).

## 5. Benchmark

### 5.1 Hardware comparison

Native inference at [SGLang `23d0a949a525`](https://github.com/sgl-project/sglang/commit/23d0a949a525006fa309c476f6f824c0644215af),
with PyTorch 2.14.1+cu130, Transformers 5.19.0, and Diffusers 0.37.0.
The [checkpoint revision](https://huggingface.co/Wan-AI/Wan2.2-Animate-2-14B-Diffusers/tree/7d48412d7b903ff3a89f4f5a960d99e1899605a1)
and [reference inputs](https://github.com/sgl-project/ci-data-diffusion/tree/31d8ad69f2434727aad518f0678fbd57823856fe/diffusion-ci/inputs/wan_animate_2)
are pinned. The workload uses the [CI prompt](https://github.com/sgl-project/sglang/blob/23d0a949a525006fa309c476f6f824c0644215af/python/sglang/multimodal_gen/test/server/testcase_configs.py#L900-L904)
and produces 24 frames at 576x1024: `clip_len=37`, 40 steps, CFG 3,
seed 42, and 16 FPS, without audio.

E2E is the median of three warm `DiffGenerator` requests, including MP4 saving
and raw-frame return; loading and the first request are excluded. Runs use
`--warmup-mode off` and `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1`. The same prompt
is repeated with the default conditioning cache enabled. GPU memory is the
maximum per-GPU device usage sampled at 10 Hz across loading and all four requests,
including allocator reservation. It is **not** a minimum VRAM requirement;
offload also uses host RAM.

| Hardware | Server flags | Warm E2E (s) | Peak GPU memory (GiB) |
| - | - | -: | -: |
| 1x RTX 4090 | `--performance-mode memory` with `PYTORCH_ALLOC_CONF=expandable_segments:True` | 634.70 | 23.78 |
| 1x H200 | Default (`auto`) | 192.27 | 68.88 |
| 1x H200 | `--performance-mode memory` | 193.08 | 31.69 |
| 1x H200 | `--dit-layerwise-offload --layerwise-offload-components transformer` | 193.28 | 36.18 |
| 1x H200 | `--performance-mode speed` | 192.58 | 90.65 |
| 1x B300 | Default (`auto`) | 101.99 | 67.67 |
| 1x B300 | `--performance-mode memory` | 104.07 | 30.71 |
| 2x B300 | `--num-gpus 2 --enable-cfg-parallel` | 53.52 | 64.93 |
| 2x B300 | **CFG parity** preset | 54.03 | 39.53 |
| 2x B300 | `--num-gpus 2 --ulysses-degree 2` | 56.17 | 58.42 |
| 2x B300 | `--num-gpus 2 --tp-size 2` | 62.97 | 38.88 |

RTX 4090 fits this short input only with the expandable allocator; both default
`auto` and `memory` without it OOMed. All four successful runs produced identical
RGB. This leaves little VRAM headroom and does not validate longer videos. The
test host had 1 TiB of RAM; smaller desktop RAM budgets were not validated.

On this H200 workload, `memory` reduced the GPU peak by 54% with less than 0.5%
additional latency; all 16 raw RGB outputs were identical. Start with `auto`
when memory is plentiful, or choose `memory` for headroom. `speed` offered no
measured latency benefit here.

On B300, **CFG parity** is the recommended two-GPU recipe: 1.89x faster than one
GPU with byte-identical RGB on this workload. Plain CFG is slightly faster but
uses more memory and changes the output. Memory mode on one B300 also preserves
RGB while reducing the GPU peak by 55%, with about 2% additional latency.

> **Output consistency on B300**
>
> Comparison against the single-B300 baseline, using all 24 raw RGB frames from
> the first warm request. SSIM is the per-frame mean; PSNR is computed from the
> global RGB mean squared error. This checks numerical consistency on one input,
> not perceptual quality across prompts or parity with the official implementation.
>
> | Recipe | Byte-identical | Mean / worst SSIM | Global / worst-frame PSNR (dB) |
> | - | - | - | - |
> | Memory; CFG parity | Yes | 1.0000 / 1.0000 | Infinite |
> | CFG | No | 0.9857 / 0.9574 | 35.36 / 27.65 |
> | Ulysses | No | 0.9895 / 0.9602 | 37.92 / 27.77 |
> | TP | No | 0.9896 / 0.9808 | 37.82 / 33.34 |

### 5.2 GB200 reference benchmark

This older benchmark uses a longer video and different settings; its numbers
are not directly comparable with the hardware comparison above.

Test Environment:

* Hardware: NVIDIA GB200 GPU (1x, 2x and 4x; 189 GB per GPU)
* Model: Wan-AI/Wan2.2-Animate-2-14B-Diffusers
* sglang diffusion version: 0.5.22
* Workload: `size` 640x800 (pixel-area budget) with a 16:9 reference image, so the frame is 960x528; a 237-frame reference video at `fps` 30 (7 clips of `clip_len` 37); 40 steps, CFG 3.0, seed 123; `--text-encoder-cpu-offload --warmup-mode request`. E2E is the request time after the warmup request, so it excludes the one-off flex-attention compile.

`bench_serving` cannot pass `video_path`, so the benchmark runs offline
with `sglang generate --config` (the YAML from [section 4.1](#4-1-generate-offline-with-a-config-file))
and reads timings from `--perf-dump-path`. `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1`
synchronizes the device so the per-stage timings are accurate.

**NVIDIA GB200**

**Benchmark Command** (1 GPU shown; add the flags from the table for the other rows):

```shell Command theme={null}
SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1 sglang generate \
  --model-path Wan-AI/Wan2.2-Animate-2-14B-Diffusers \
  --config wan_animate_2_run.yaml \
  --num-gpus 1 --text-encoder-cpu-offload --warmup-mode request \
  --save-output --perf-dump-path wan_animate_2_perf.json
```

**Result** (peak memory per GPU from `nvidia-smi`):

| Config | GPUs | Extra flags | E2E (s) | Peak per GPU (GiB) |
| - | -: | - | -: | -: |
| 1 GPU, DiT resident | 1 | none | 993 | 69.5 |
| 1 GPU, layerwise DiT offload | 1 | `--dit-layerwise-offload --layerwise-offload-components transformer` | 996 | 40.2 |
| 2 GPUs, CFG-parallel | 2 | `--enable-cfg-parallel` | 513 | 67.3 |
| 2 GPUs, CFG-parallel, replicated encoders, serial VAE decode | 2 | `--enable-cfg-parallel --encoder-parallel replicate --vae-config.use-parallel-decode false` | 515 | 71.0 |
| 2 GPUs, CFG-parallel, layerwise DiT offload, bit-identical to 1 GPU | 2 | `--enable-cfg-parallel --dit-layerwise-offload --layerwise-offload-components transformer --encoder-parallel replicate --vae-config.use-parallel-decode false` with `SGLANG_DIFFUSION_VAE_CHANNELS_LAST_3D=1` | 512 | 43.0 |
| 2 GPUs, Ulysses | 2 | `--ulysses-degree 2` | 547 | 58.1 |
| 2 GPUs, tensor parallel | 2 | `--tp-size 2` | 613 | 36.6 |
| 4 GPUs, framework default (CFG2 x SP2, KV-gather exchange) | 4 | none | 283 | 58.4 |
| 4 GPUs, CFG-parallel x Ulysses | 4 | `--enable-cfg-parallel --ulysses-degree 2` | 283 | 58.4 |
| 4 GPUs, CFG-parallel x Ulysses, layerwise DiT offload | 4 | `--enable-cfg-parallel --ulysses-degree 2 --dit-layerwise-offload --layerwise-offload-components transformer --encoder-parallel replicate --vae-config.use-parallel-decode false` | 291 | 35.2 |
| 4 GPUs, Ulysses | 4 | `--ulysses-degree 4` | 317 | 49.0 |
| 4 GPUs, CFG-parallel x tensor parallel | 4 | `--enable-cfg-parallel --tp-size 2` | 318 | 37.5 |
| 4 GPUs, tensor parallel x Ulysses | 4 | `--tp-size 2 --ulysses-degree 2` | 361 | 33.5 |
| 4 GPUs, tensor parallel | 4 | `--tp-size 4` | 379 | 27.3 |

Layerwise DiT offload cuts the 1-GPU peak by 29 GiB at the same end-to-end time.
The fastest 4-GPU layout is 3.5x faster than one GPU. The framework default and the explicit CFG-parallel x Ulysses
layout run at the same speed and produce the same output; the explicit flags pin the layout.
Tensor parallel x Ulysses is the memory-leaning 4-GPU layout: the lowest 4-GPU peak after `--tp-size 4`, about 13 percent slower than CFG-parallel x tensor parallel and 27 percent slower than the default.
Adding `--encoder-parallel replicate --vae-config.use-parallel-decode false` to the CFG-parallel x Ulysses row costs about 1 percent (287 s, 62.5 GiB) and does not make the output bit-identical to a single GPU: every Ulysses layout matches the single-GPU video to about 30 dB mean PSNR.
The layerwise rows are the configurations for 80 GB GPUs. With the replicated encoders, serial VAE decode and the
channels-last VAE layout (`SGLANG_DIFFUSION_VAE_CHANNELS_LAST_3D=1`) the 2-GPU CFG-parallel output is bit-identical to a single GPU.

### 5.3 Distilled checkpoint on GB200

`Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers` on the GB200 node of
[section 5.2](#5-2-gb200-reference-benchmark) with the same inputs at the distilled
defaults (10 steps, `guidance_scale` 1.0, seed 123, `clip_len` 37, `size` 640x800,
`fps` 30). The rows were measured through the server, not through
`sglang generate`, so the two GB200 tables are not like for like: the command
below plus the row's flags, then two identical `/v1/videos` requests per server
(the request of [section 4.2](#4-2-serve-and-request-over-http) with the benchmark
inputs). These rows are the picker's verified GB200 recipes.

```bash Command theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.2-Animate-2-14B-Distilled-Diffusers \
  --num-gpus 1 --text-encoder-cpu-offload --warmup-mode server \
  --port 30010
```

E2E is the server's `Pixel data generated` time for the steady state, the second
identical request on the same server. Peak is the `nvidia-smi` maximum over ranks
during the request.

**Result** (peak memory per GPU from `nvidia-smi`):

| Config | GPUs | Extra flags | E2E (s) | Peak per GPU (GiB) |
| - | -: | - | -: | -: |
| 1 GPU, DiT resident | 1 | none | 150.2 | 69.5 |
| 1 GPU, layerwise DiT offload | 1 | `--dit-layerwise-offload --layerwise-offload-components transformer` | 150.0 | 39.7 |
| 2 GPUs, Ulysses (= framework default) | 2 | `--ulysses-degree 2` | 87.0 | 60.1 |
| 2 GPUs, Ulysses, layerwise DiT offload | 2 | `--ulysses-degree 2 --dit-layerwise-offload --layerwise-offload-components transformer` | 86.3 | 31.4 |
| 2 GPUs, tensor parallel | 2 | `--tp-size 2` | 97.1 | 40.0 |
| 4 GPUs, Ulysses (= framework default) | 4 | `--ulysses-degree 4` | 55.8 | 51.0 |
| 4 GPUs, Ulysses, layerwise DiT offload | 4 | `--ulysses-degree 4 --dit-layerwise-offload --layerwise-offload-components transformer` | 58.0 | 21.5 |
| 4 GPUs, tensor parallel x Ulysses | 4 | `--tp-size 2 --ulysses-degree 2` | 59.3 | 35.3 |
| 4 GPUs, tensor parallel | 4 | `--tp-size 4` | 64.1 | 30.4 |

Whole-run peaks, which include the server warmup's synthetic requests (one at the
largest supported budget, 720x1280, to decide whether the DiT stays resident), are
higher: 95.5 / 64.8 / 72.3 / 41.9 / 47.3 / 57.4 / 27.3 / 38.9 / 37.8 GiB in row order.

The first request after the server warmup adds a one-time flex-attention autotune
when its frame geometry differs from the warmup shapes (about 16 s on 1 GPU, 12 to
13 s on 2, 9 s on 4). With one branch per step there is no CFG split to
parallelize, so Ulysses is the distilled
checkpoint's fastest layout: 1.73x faster than one GPU on 2 GPUs and 2.69x on 4.
The framework default for `--num-gpus 2` and `--num-gpus 4` resolves to the same
Ulysses layout and produces the same output; the explicit flag pins it. Layerwise
DiT offload costs about 2 s and leaves the output identical. Tensor parallel x
Ulysses is the memory-leaning 4-GPU layout, 6 percent slower than Ulysses 4 at 31
percent less memory and faster than `--tp-size 4`; tensor parallelism alone is 12
to 15 percent slower than Ulysses with the lowest peak. With the default encoder
and VAE parallel settings no multi-GPU layout is bit-identical to the single GPU
(Ulysses 2 and tensor parallel 4 are the closest, above 36 dB PSNR).
`--enable-torch-compile` is not faster than eager on this model and is not
recommended.

### 5.4 Native regression coverage

A separate regression used two B300 GPUs, CFG-parallel with replicated encoders
and serial VAE decode, a 576x1024 area budget, 40 steps, `clip_len=37`, CFG=3,
and seed 42. The driving input was looped to 64 frames to exercise two clips;
this is a bounded regression workload, not a long-video quality benchmark.
Both revisions (`4ee69b4cf9ea` and `04725c1c2900`) used PyTorch 2.14.1+cu130,
Transformers 5.19.0, Diffusers 0.37.0, and FlashAttention 4.0.0b34 on the same node.

Cold and warm raw RGB outputs were byte-identical to the pre-optimization native
implementation, both with resident DiT weights and with layerwise DiT offload.
The single warm samples measured 112.29 s before and 112.40 s after optimization;
offload measured 113.46 s. This does not establish a significant latency change.
Rank 0's request-stage peak allocated memory was 52.24, 52.23, and 23.93 GiB,
respectively; these are not loading peaks or whole-device physical memory peaks.

An H200 HTTP smoke test also covered image upload, asynchronous job polling and
download, request warmup, repeated requests, URL video input, and reference-audio
passthrough. Repeated decoded RGB outputs were identical.

These checks preserve the existing native output, not bitwise parity with the
newer Diffusers modular pipeline. The native sampler follows the original Wan
sigma grid starting at exactly 1.0 and retains its BF16 conditioning conversion;
the modular pipeline's sampling and preprocessing differ. Official-reference
accuracy must be validated separately with pinned versions and aligned settings.

## 6. Troubleshooting

* **OOM**: lower `clip_len` (for example `37` instead of `65`/`81`) or add `--dit-layerwise-offload --layerwise-offload-components transformer`.
* **`--ring-degree` greater than 1** is rejected at startup with `NotImplementedError`. Use `--ulysses-degree` for sequence parallelism and keep the ring degree at 1.
* **`--ulysses-degree` that does not divide the TP-local attention heads** (40 / `--tp-size`; with TP 1, degrees 3, 6, 16, ... are rejected) fails at startup with `ValueError`; with TP 1 use 2, 4, 5, 8, 10, 20 or 40. The same check covers leftover GPUs auto-assigned to sequence parallelism (for example `--num-gpus 8 --tp-size 2` runs TP 2 x Ulysses 4).
* **`--enable-cfg-parallel` (or `--cfg-parallel-size 2`) with the distilled checkpoint**: the server refuses to start with `Wan_Animate_2_14B_Distilled_Config does not support CFG parallelism`. The distilled model runs one branch per step, so there is nothing to split; use `--ulysses-degree` or `--tp-size` for multiple GPUs.
* **`guidance_scale` other than `1.0` with the distilled checkpoint**: the request is rejected at admission (HTTP 400 on `/v1/videos`, `ValueError` offline) with `guidance_scale=<value> is not supported by the distilled Wan-Animate-2 checkpoint: it samples at guidance_scale 1.0 only (the distilled recipe has no unconditional branch); omit the field or set it to 1.0.` The distilled checkpoint runs one branch per step; omit the field or set it to `1.0`.
* **`The expanded size of the tensor (N) must match the existing size (M)`** at the first denoising step (the request fails): the reference image and the reference video have different aspect ratios and at the requested `size` the video's 16-aligned grid has more tokens per frame than the image's. With this page's inputs it happens at the 720×1280 budget on both checkpoints; use the 640×800 budget.
* **FSDP** (`--use-fsdp-inference`) is not supported for this model and is rejected at startup; use `--tp-size` or DiT layerwise offload for memory.
* **`--attention-backend`** (or `--component-attention-backends transformer=...`) is rejected at startup with `ValueError`: the in-context self-attention runs on torch `flex_attention` only. The text encoder still accepts `--component-attention-backends text_encoder=<backend>`.
* **`enable_teacache` / `enable_spectrum`** request fields are rejected with `ValueError`: every block runs at every step, so the cache heuristics do not apply.
* **`SGLANG_DIFFUSION_ENABLE_MXFP8_ATTENTION`** is rejected at startup with `ValueError`: this model does not apply the offline Q/K rotation.
* **Silent output**: the reference video's audio track is kept only when it can be decoded; otherwise the output is silent and the server logs one warning per process naming the cause. A reference video without an audio track also gives a silent output (one info line). Pass `enable_audio=false` to skip audio extraction altogether.
* **Reference video not found**: `video_path` must be a path the server process can read. Pass it as `--video-path` or through `--config` offline, or in the request body online.


# Wan2.1
Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan2.1

Deploy Wan2.1 video generation models with SGLang - community contribution guide for Wan Video's diffusion models.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[Wan2.1](https://github.com/Wan-Video/Wan2.1) is a broad open video family covering text-to-video and image-to-video across 1.3B and 14B checkpoints. Its practical strengths are motion-rich generation, temporal consistency, and readable Chinese/English text, with 480p and 720p variants for different quality and memory targets.

Choose the 1.3B T2V model for consumer-GPU experiments and the 14B models when quality matters more than footprint. Wan2.1 is a dense DiT family; for timestep-specialized MoE capacity or the unified 5B TI2V path, use Wan2.2 instead.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs, Intel Xeon CPUs, and Ascend A2/A3 Series NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

Current supported optimization options are listed in the [SGLang diffusion support matrix](../../../docs/sglang-diffusion/attention_backends#platform-support-matrix).

* `--vae-path`: Path to a custom VAE model or HuggingFace model ID. If not specified, the VAE will be loaded from the main model path.
* `--num-gpus {NUM_GPUS}`: Number of GPUs to use.
* `--tp-size {TP_SIZE}`: Tensor parallelism size (for the encoder/DiT; keep (\leq 1) if relying heavily on CPU offload).
* `--sp-degree {SP_SIZE}`: Sequence parallelism degree.
* `--ulysses-degree {ULYSSES_DEGREE}`: Degree of DeepSpeed-Ulysses-style SP in USP.
* `--ring-degree {RING_DEGREE}`: Degree of ring attention-style SP in USP.
* `--text-encoder-cpu-offload`, `--dit-cpu-offload`, `--vae-cpu-offload`: Use CPU offload to reduce peak GPU memory when needed.

### 3.3 24 GB NVIDIA GPUs

The 14B DiTs do not fit on a 24 GB card in BF16 (the I2V-14B one is 30.5 GiB), so their weights have to stream from host memory layer by layer. On CUDA, `--performance-mode memory` streams the DiT, text encoder, image encoder and VAE:

```bash theme={null}
# 1 GPU
sglang serve --model-path Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --performance-mode memory

# 8 GPUs: CFG parallel x Ulysses 4
sglang serve --model-path Wan-AI/Wan2.1-I2V-14B-480P-Diffusers --num-gpus 8 \
  --enable-cfg-parallel --ulysses-degree 4 --performance-mode memory
```

Each GPU's worker keeps its own copy of the streamed weights in host memory, about 60 GiB, so 8 GPUs need close to 500 GiB of RAM.

Measured with SGLang main `c7be3e9` (2026-09-29) on RTX 4090 24 GB (PCIe, no P2P between GPUs), for Wan2.1-I2V-14B-480P image-to-video at 480x832, 81 frames, 40 steps, BF16:

| GPUs | Time per video | Time per step | Peak GPU memory | Host memory |
| - | -: | -: | -: | -: |
| 1 | 840 s | 20.7 s | 12.5 GiB | 59 GiB |
| 8 | 198 s | 4.8 s | 11.9 GiB | 483 GiB |

Component CPU offload (`--dit-cpu-offload`) does not help here: it moves the whole DiT onto the GPU around each use.

## 4. Model Invocation

### 4.1 Basic Usage

For more API usage and request examples, please refer to:
[SGLang Diffusion OpenAI API](../../../docs/sglang-diffusion/api/openai_api)

#### 4.1.1 Launch a server and then send requests

```bash Command theme={null}
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000

curl http://127.0.0.1:30000/v1/images/generations \
  -o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "Wan-AI/Wan2.1-T2V-14B-Diffusers",
    "prompt": "A cute baby sea otter",
    "n": 1,
    "size": "1024x1024",
    "response_format": "b64_json"
  }'
```

#### 4.1.2 Generate a video without launching a server

```bash Command theme={null}
SERVER_ARGS=(
  --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
  --text-encoder-cpu-offload
  --pin-cpu-memory
  --num-gpus 4
  --ulysses-degree=2
  --enable-cfg-parallel
)

SAMPLING_ARGS=(
  --prompt "A curious raccoon"
  --save-output
  --output-path outputs
  --output-file-name "A curious raccoon.mp4"
)

sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"
```

### 4.2 Advanced Usage

#### 4.2.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve significant inference speedups with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](../../../docs/sglang-diffusion/cache_dit).

**Basic Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
```

**Advanced Usage**

Combined Configuration Example:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
```

#### 4.2.2 GPU Optimization

* `--dit-cpu-offload`: Keep DiT weights on the CPU and move them onto the GPU whole around each use. This takes the DiT out of FSDP, which shards only resident components.
* `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
* `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference.
* `--vae-cpu-offload`: Use CPU offload for VAE.
* `--pin-cpu-memory`: Pin memory for CPU offload. Use as a workaround if you see "CUDA error: invalid argument".

#### 4.2.3 Supported LoRA Registry

SGLang supports applying Wan2.1 LoRA adapters on top of base models:

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>origin model</th>
      <th>supported LoRA</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B)</td>
      <td>[NIVEDAN/wan2.1-lora](https://huggingface.co/NIVEDAN/wan2.1-lora)</td>
    </tr>

    <tr>
      <td>[Wan-AI/Wan2.1-I2V-14B-720P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P)</td>
      <td>[valiantcat/Wan2.1-Fight-LoRA](https://huggingface.co/valiantcat/Wan2.1-Fight-LoRA)</td>
    </tr>
  </tbody>
</table>

**Example**:

```bash Command theme={null}
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers --port 30000 \
    --lora-path NIVEDAN/wan2.1-lora
```

## 5. Reference Benchmark

The following benchmark is a point-in-time reference for one model, hardware platform, SGLang image, and parameter set. It is not a complete hardware support matrix.

Test Environment:

* Hardware: AMD MI300X GPU (1x)
* Model: Wan-AI/Wan2.1-T2V-14B-Diffusers
* SGLang Docker Image Version: 0.5.9

### 5.1 How to Run Benchmarks with SGLang

You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 performance on your hardware.

#### 5.1.1 Generate a single video

**NVIDIA B200**

**Server Command**:

```bash Command theme={null}
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
```

**Benchmark Command**:

```bash Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-video
Model:                                   Wan-AI/Wan2.1-T2V-14B-Diffusers
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  1958.41
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.00
Latency Mean (s):                        1958.4059
Latency Median (s):                      1958.4059
Latency P99 (s):                         1958.4059
--------------------------------------------------
Peak Memory Max (MB):                    59662.00
Peak Memory Mean (MB):                   59662.00
Peak Memory Median (MB):                 59662.00
============================================================
```

**Ascend A3 Series**

**Server Command**:

```bash Command theme={null}
#One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards
sglang serve \
  --model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \
  --tp-size 2 \
  --sp-degree 2 \
  --num-gpus 4 \
  --attention-backend laser_attn
```

**Benchmark Command**:

```bash Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving \
  --dataset vbench \
  --task text-to-video \
  --num-prompts 1 \
  --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-video
Model:                                        Wan-AI/Wan2.1-T2V-14B-Diffusers/
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       1282.90
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.00
Output throughput (outputs/s):                0.00
Latency Mean (s):                             1282.90
Latency Median (s):                           1282.90
Latency P90 (s):                              1282.90
Latency P95 (s):                              1282.90
Latency P99 (s):                              1282.90
--------------------------------------------------
Peak Memory Max (MB):                         31938.00
Peak Memory Mean (MB):                        31938.00
Peak Memory Median (MB):                      31938.00
============================================================
```

#### 5.1.2 Generate videos with Cache-DiT acceleration

**NVIDIA B200**

**Server Command**:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers
```

**Benchmark Command**:

```bash Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-video
Model:                                   Wan-AI/Wan2.1-T2V-14B-Diffusers
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  556.99
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.00
Latency Mean (s):                        556.9885
Latency Median (s):                      556.9885
Latency P99 (s):                         556.9885
--------------------------------------------------
Peak Memory Max (MB):                    69306.00
Peak Memory Mean (MB):                   69306.00
Peak Memory Median (MB):                 69306.00
============================================================
```

**Ascend A3 Series**

**Server Command**:

```bash Command theme={null}
#One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
SGLANG_CACHE_DIT_ENABLED=true sglang serve \
  --model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \
  --tp-size 2 \
  --sp-degree 2 \
  --num-gpus 4 \
  --attention-backend laser_attn
```

**Benchmark Command**:

```bash Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving \
  --dataset vbench \
  --task text-to-video \
  --num-prompts 1 \
  --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-video
Model:                                        Wan-AI/Wan2.1-T2V-14B-Diffusers/
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       413.88
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.00
Output throughput (outputs/s):                0.00
Latency Mean (s):                             413.88
Latency Median (s):                           413.88
Latency P90 (s):                              413.88
Latency P95 (s):                              413.88
Latency P99 (s):                              413.88
--------------------------------------------------
Peak Memory Max (MB):                         32782.00
Peak Memory Mean (MB):                        32782.00
Peak Memory Median (MB):                      32782.00
============================================================
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# Wan2.2
Source: https://docs.sglang.io/cookbook/diffusion/Wan/Wan2.2

Deploy Wan2.2 video generation models with SGLang - MoE architecture, cinematic aesthetics, and efficient 720P@24fps generation.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[Wan2.2](https://github.com/Wan-Video/Wan2.2) extends the Wan family with timestep-specialized Mixture-of-Experts models and a unified 5B TI2V checkpoint. It targets cinematic composition, lighting, and complex motion while supporting text-to-video, image-to-video, and hybrid text/image conditioning.

Choose the A14B MoE checkpoints for maximum T2V or I2V capacity and the 5B TI2V model for a smaller unified 720p-at-24-fps path. MoE reduces active compute relative to total capacity but does not remove the memory cost of loading expert weights, so hardware selection still matters.

> [!WARNING]
>
> The Wan2.2 TI2V 5B checkpoint currently has known quality issues when it is used
> for image-to-video generation. Use `Wan-AI/Wan2.2-I2V-A14B-Diffusers` when I2V
> quality is the priority.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

The Wan2.2 series offers models in various sizes, architectures and input types, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements.

* `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
* `--num-gpus {NUM_GPUS}`: Number of GPUs to use
* `--tp-size {TP_SIZE}`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
* `--sp-degree {SP_SIZE}`: Sequence parallelism size (typically should match the number of GPUs)
* `--ulysses-degree {ULYSSES_DEGREE}`: The degree of DeepSpeed-Ulysses-style SP in USP
* `--ring-degree {RING_DEGREE}`: The degree of ring attention-style SP in USP

## 4. Model Invocation

### 4.1 Basic Usage

For more API usage and request examples, please refer to:
[SGLang Diffusion OpenAI API](/docs/sglang-diffusion/api/openai_api)

#### 4.1.1 Launch a server and then send requests

```shell Command theme={null}
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000

curl http://127.0.0.1:3000/v1/images/generations \
  -o >(jq -r '.data[0].b64_json' | base64 --decode > example.png) \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "Wan-AI/Wan2.2-T2V-A14B-Diffusers",
    "prompt": "A cute baby sea otter",
    "n": 1,
    "size": "1024x1024",
    "response_format": "b64_json"
  }'
```

#### 4.1.2 Generate a video without launching a server

```shell Command theme={null}
SERVER_ARGS=(
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
  --text-encoder-cpu-offload
  --pin-cpu-memory
  --num-gpus 4
  --ulysses-degree=2
  --enable-cfg-parallel
)

SAMPLING_ARGS=(
  --prompt "A curious raccoon"
  --save-output
  --output-path outputs
  --output-file-name "A curious raccoon.mp4"
)

sglang generate "${SERVER_ARGS[@]}" "${SAMPLING_ARGS[@]}"

```

### 4.2 Advanced Usage

#### 4.2.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit).

**Basic Usage**

```shell Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
```

**Advanced Usage**

* DBCache Parameters: DBCache controls block-level caching behavior:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

Combined Configuration Example:

```shell Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
```

#### 4.2.2 GPU Optimization

* `--dit-cpu-offload`: Keep DiT weights on the CPU and move them onto the GPU whole around each use. This takes the DiT out of FSDP, which shards only resident components.
* `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference. Enable if run out of memory with FSDP.
* `--image-encoder-cpu-offload`: Use CPU offload for image encoder inference. Enable if run out of memory with FSDP.
* `--vae-cpu-offload`: Use CPU offload for VAE. Enable if run out of memory.
* `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".

#### 4.2.3 Supported LoRA Registry

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>origin model</th>
      <th>supported LoRA</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[Wan-AI/Wan2.2-I2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B-Diffusers)</td>
      <td>[lightx2v/Wan2.2-Distill-Loras](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras)</td>
    </tr>

    <tr>
      <td>[Wan-AI/Wan2.2-T2V-A14B-Diffusers](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B-Diffusers)</td>
      <td>[Cseti/wan2.2-14B-Arcane\_Jinx-lora-v1](https://huggingface.co/Cseti/wan2.2-14B-Arcane_Jinx-lora-v1)</td>
    </tr>
  </tbody>
</table>

**Example**:

```shell Command theme={null}
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers --port 3000 \
    --lora-path Cseti/wan2.2-14B-Arcane_Jinx-lora-v1
```

## 5. Benchmark

Test Environment:

* Hardware: NVIDIA B200 GPU (1x)
* Model: Wan-AI/Wan2.2-T2V-A14B-Diffusers
* sglang diffusion version: 0.5.6.post2

### 5.1 Speedup Benchmark

### 5.1.1 Generate a video

**NVIDIA B200**

**Server Command**:

```shell Command theme={null}
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
```

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-video --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Model:                                   Wan-AI/Wan2.2-T2V-A14B-Diffusers
Dataset:                                 vbench
Task:                                    text-to-video
--------------------------------------------------
Benchmark duration (s):                  630.43
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.00
Latency Mean (s):                        630.4277
Latency Median (s):                      630.4277
Latency P99 (s):                         630.4277
--------------------------------------------------
Peak Memory Max (MB):                    62627.41
Peak Memory Mean (MB):                   62627.41
Peak Memory Median (MB):                 62627.41

============================================================
```

**Ascend A3 Series**

**Server Command**:

```shell Command theme={null}
#One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking
sglang serve \
  --model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \
  --tp-size 2 \
  --sp-degree 4 \
  --num-gpus 8 \
  --attention-backend laser_attn
```

**Benchmark Command**:

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving \
  --dataset vbench \
  --task text-to-video \
  --num-prompts 1 \
  --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-video
Model:                                        Wan-AI/Wan2.2-T2V-A14B-Diffusers/
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       214.50
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.00
Output throughput (outputs/s):                0.00
Latency Mean (s):                             214.50
Latency Median (s):                           214.50
Latency P90 (s):                              214.50
Latency P95 (s):                              214.50
Latency P99 (s):                              214.50
--------------------------------------------------
Peak Memory Max (MB):                         46692.00
Peak Memory Mean (MB):                        46692.00
Peak Memory Median (MB):                      46692.00
------------------------------------------------------------
```

#### 5.1.2 Generate videos with high concurrency

**NVIDIA B200**

**Server Command**:

```shell Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers
```

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-video --num-prompts 20 --max-concurrency 20
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Model:                                   Wan-AI/Wan2.2-T2V-A14B-Diffusers
Dataset:                                 vbench
Task:                                    text-to-video
--------------------------------------------------
Benchmark duration (s):                  5163.21
Request rate:                            inf
Max request concurrency:                 20
Successful requests:                     20/20
--------------------------------------------------
Request throughput (req/s):              0.00
Latency Mean (s):                        2739.7695
Latency Median (s):                      2742.0673
Latency P99 (s):                         5121.6331
--------------------------------------------------
Peak Memory Max (MB):                    72523.56
Peak Memory Mean (MB):                   70253.34
Peak Memory Median (MB):                 70824.46

============================================================
```

**Ascend A3 Series**

**Server Command**:

```shell Command theme={null}
#One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
SGLANG_CACHE_DIT_ENABLED=true sglang serve \
  --model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \
  --tp-size 2 \
  --sp-degree 4 \
  --num-gpus 8 \
  --attention-backend laser_attn
```

**Benchmark Command**:

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving \
  --dataset vbench \
  --task text-to-video \
  --num-prompts 20 \
  --max-concurrency 20
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-video
Model:                                        Wan-AI/Wan2.2-T2V-A14B-Diffusers/
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       4384.65
Request rate:                                 inf
Max request concurrency:                      20
Successful requests:                          20/20
Completed outputs:                            20
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.00
Output throughput (outputs/s):                0.00
Latency Mean (s):                             2304.17
Latency Median (s):                           2297.69
Latency P90 (s):                              3972.32
Latency P95 (s):                              4178.99
Latency P99 (s):                              4343.52
--------------------------------------------------
Peak Memory Max (MB):                         46692.00
Peak Memory Mean (MB):                        46691.90
Peak Memory Median (MB):                      46692.00
------------------------------------------------------------
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Video`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.


# Z-Image-Turbo
Source: https://docs.sglang.io/cookbook/diffusion/Z-Image/Z-Image-Turbo

Deploy Z-Image-Turbo with SGLang - community contribution guide for Z-Image's fast image generation model.

## 1. Quick start

Follow the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation) for your GPU platform.

## 2. Model capabilities

[Z-Image-Turbo](https://huggingface.co/Tongyi-MAI/Z-Image-Turbo) is a distilled 6B single-stream DiT for fast text-to-image generation. It reaches its intended operating point in 8 function evaluations and is particularly strong at photorealistic scenes, prompt adherence, and English/Chinese text rendering.

Choose it when latency and a relatively small deployment footprint matter more than the editability or maximum capacity of larger image models. It is a generation-only checkpoint; use Qwen-Image-Edit or FLUX.2 when the request includes source images or identity-preserving edits.

## 3. Model Deployment

This section provides deployment configurations optimized for different hardware platforms and use cases.

### 3.1 Basic Configuration

Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware.

Use the [command builder](#1-quick-start) above.

### 3.2 Configuration Tips

See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements.

* `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path.
* `--num-gpus`: Number of GPUs to use
* `--tp-size`: Tensor parallelism size (only for the encoder; should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster)
* `--sp-degree`: Sequence parallelism size (typically should match the number of GPUs)
* `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP
* `--ring-degree`: The degree of ring attention-style SP in USP

**AMD ROCm Notes**: Requires SGLang >= v0.5.8.

## 4. API Usage

For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api).

### 4.1 Generate an Image

```python Example theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1")

response = client.images.generate(
    model="Tongyi-MAI/Z-Image-Turbo",
    prompt="A logo With Bold Large text: SGL Diffusion",
    n=1,
    response_format="b64_json",
)

# Save the generated image
image_bytes = base64.b64decode(response.data[0].b64_json)
with open("output.png", "wb") as f:
    f.write(image_bytes)
```

### 4.2 Advanced Usage

Z-Image uses `positive + guidance_scale * (positive - negative)` guidance, unlike
the standard CFG convention. Any positive `guidance_scale` enables CFG; omitted
negative prompts use an empty string. Z-Image-Turbo defaults to `guidance_scale=0`
and does not run the negative branch. Enabling CFG adds a second DiT branch, so
expect additional inference cost; it is not automatically recommended for Turbo.

#### 4.2.1 Cache-DiT Acceleration

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to 7.4x inference speedup with minimal quality loss. You can set `SGLANG_CACHE_DIT_ENABLED=True` to enable it. For more details, please refer to the SGLang Cache-DiT [documentation](/docs/sglang-diffusion/cache_dit).

**Basic Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true sglang serve --model-path Tongyi-MAI/Z-Image-Turbo
```

**Advanced Usage**

* DBCache Parameters: DBCache controls block-level caching behavior:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

* TaylorSeer Configuration: TaylorSeer improves caching accuracy using Taylor expansion:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

Combined Configuration Example:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang serve --model-path Tongyi-MAI/Z-Image-Turbo
```

#### 4.2.2 CPU Offload

* `--dit-cpu-offload`: Use CPU offload for DiT inference. Enable if run out of memory.
* `--text-encoder-cpu-offload`: Use CPU offload for text encoder inference.
* `--vae-cpu-offload`: Use CPU offload for VAE.
* `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument".

#### 4.2.3 Known LoRA examples

Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Z-Image-Turbo examples include:

* [`tarn59/pixel_art_style_lora_z_image_turbo`](https://huggingface.co/tarn59/pixel_art_style_lora_z_image_turbo)
* [`wcde/Z-Image-Turbo-DeJPEG-Lora`](https://huggingface.co/wcde/Z-Image-Turbo-DeJPEG-Lora)

## 5. Benchmark

Test Environment:

* Hardware: AMD Instinct MI300X GPU (1x)
* Model: Tongyi-MAI/Z-Image-Turbo
* Docker Image: lmsysorg/sglang:v0.5.8-rocm700-mi30x
* sglang diffusion version: 0.5.8

### 5.1 Speedup Benchmark

#### 5.1.1 Generate an image

**AMD MI300X**

**Server Command**:

```shell Command theme={null}
sglang serve --model-path Tongyi-MAI/Z-Image-Turbo \
    --ulysses-degree=1 --ring-degree=1 --port 30000
```

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-image
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  1.84
Request rate:                            inf
Max request concurrency:                 1
Successful requests:                     1/1
--------------------------------------------------
Request throughput (req/s):              0.54
Latency Mean (s):                        1.8435
Latency Median (s):                      1.8435
Latency P99 (s):                         1.8435
--------------------------------------------------
Peak Memory Max (MB):                    30689.20
Peak Memory Mean (MB):                   30689.20
Peak Memory Median (MB):                 30689.20
============================================================
```

**Ascend A3 Series**

**Server Command**:

```shell Command theme={null}
#One A3 Series card has 2 npu chips
sglang serve --model-path Tongyi-MAI/Z-Image-Turbo --tp-size 2 --sp-degree 1 --num-gpus 2
```

**Benchmark Command**:

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 1 --max-concurrency 1
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        Tongyi-MAI/Z-Image-Turbo
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       2.43
Request rate:                                 inf
Max request concurrency:                      1
Successful requests:                          1/1
Completed outputs:                            1
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.41
Output throughput (outputs/s):                0.41
Latency Mean (s):                             2.43
Latency Median (s):                           2.43
Latency P90 (s):                              2.43
Latency P95 (s):                              2.43
Latency P99 (s):                              2.43
--------------------------------------------------
Peak Memory Max (MB):                         11052.00
Peak Memory Mean (MB):                        11052.00
Peak Memory Median (MB):                      11052.00
------------------------------------------------------------
```

#### 5.1.2 Generate images with high concurrency

**AMD MI300X**

**Benchmark Command**:

```shell Command theme={null}
python3 -m sglang.multimodal_gen.benchmarks.bench_serving \
    --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                    text-to-image
Model:                                   Tongyi-MAI/Z-Image-Turbo
Dataset:                                 vbench
--------------------------------------------------
Benchmark duration (s):                  35.32
Request rate:                            inf
Max request concurrency:                 20
Successful requests:                     20/20
--------------------------------------------------
Request throughput (req/s):              0.57
Latency Mean (s):                        18.5672
Latency Median (s):                      18.5573
Latency P99 (s):                         34.9880
--------------------------------------------------
Peak Memory Max (MB):                    30689.26
Peak Memory Mean (MB):                   30689.21
Peak Memory Median (MB):                 30689.21
============================================================
```

**Ascend A3 Series**

**Benchmark Command**:

```shell Command theme={null}
python -m sglang.multimodal_gen.benchmarks.bench_serving --dataset vbench --task text-to-image --num-prompts 20 --max-concurrency 20
```

**Result**:

```text Output theme={null}
================= Serving Benchmark Result =================
Task:                                         text-to-image
Model:                                        /models/Tongyi-MAI/Z-Image-Turbo/Z-Image-Turbo
Dataset:                                      vbench
--------------------------------------------------
Benchmark duration (s):                       49.08
Request rate:                                 inf
Max request concurrency:                      20
Successful requests:                          20/20
Completed outputs:                            20
Outputs per prompt:                           1
--------------------------------------------------
Request throughput (req/s):                   0.41
Output throughput (outputs/s):                0.41
Latency Mean (s):                             25.78
Latency Median (s):                           25.77
Latency P90 (s):                              44.42
Latency P95 (s):                              46.75
Latency P99 (s):                              48.61
--------------------------------------------------
Peak Memory Max (MB):                         11054.00
Peak Memory Mean (MB):                        11054.00
Peak Memory Median (MB):                      11054.00
------------------------------------------------------------
```

## 6. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — ComfyUI's own sampler, CLIP, and
VAE drive the loop while SGLang replaces the model forward. Load the
checkpoint with `SGLDiffusion UNET Loader` and set
`model_type` to `lumina2` on the
`SGLDiffusion Options` node.

A reference workflow is included at
`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/workflows/z-image_sgld.json`.


# Ming-Image
Source: https://docs.sglang.io/cookbook/diffusion/inclusionAI/Ming-Image

Deploy Ming-Image Design and Design-Layer with SGLang Diffusion for text-to-image, RGBA image editing, and ordered transparent layer decomposition.

## 1. Quick start

Use a nightly Docker image containing this integration, or install from source, following the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation). Run the commands below inside that environment.

Choose **Design** for generation or editing and **Design-Layer** for decomposition.
The two checkpoints require separate servers. Start with one H200 on Linux/CUDA;
the picker marks unverified hardware and feature combinations explicitly.

Requests return base64 PNGs, preserving all four RGBA channels. For decomposition,
each returned image is one ordered layer, not an independent generation. The
internal composite frame is omitted from the response. Save all entries in
`data`, rather than only the first image.
`n` selects independent generations (or sets of layers), processed sequentially.

## 2. Model capabilities

[Ming-Image Design](https://huggingface.co/inclusionAI/Ming-Image-0.1-Design)
generates images from text and edits a single reference image. It targets design
assets, layouts, and transparent RGBA output. Use the prompt prefix
`RGBA, 4-channel, transparent background` to request transparency; RGBA output
alone does not guarantee a transparent background.

[Design-Layer](https://huggingface.co/inclusionAI/Ming-Image-0.1-Design-Layer)
decomposes one reference into ordered transparent layers. Supply `num_layers`,
or specify a count with `into N layers` or `Number of layers: N` in the prompt;
the prompt count takes precedence. Defaults are 12 steps with guidance 1 for
Design and guidance 2 for Design-Layer. The model uses zero unconditional
embeddings and does not support custom negative prompts.

Both checkpoints use the native Bailing multimodal MoE encoder, query connector,
joint DiT, and four-channel Qwen-style VAE. No remote modeling code is executed.
The original checkpoint directory layout is accepted without conversion.

## 3. Offline generation

```bash Design theme={null}
sglang generate --model-path inclusionAI/Ming-Image-0.1-Design \
  --prompt "A minimalist exhibition poster with a red geometric sculpture" \
  --height 1024 --width 1024 --save-output
```

```bash Layer decomposition theme={null}
sglang generate --model-path inclusionAI/Ming-Image-0.1-Design-Layer \
  --image-path /path/to/input.png --num-layers 4 --save-output
```

Design defaults to 2048 by 2048; 1024 by 1024 reduces latency and activation
memory. Decomposition defaults to the official 1024-resolution aspect buckets;
use `--height 512 --width 512` for the 512 buckets. Reference-image requests
select the closest official bucket and restore the input aspect ratio in the
returned PNGs. Multiple reference images are not supported.

For a renamed local checkpoint directory, also provide
`--model-id inclusionAI/Ming-Image-0.1-Design` or the corresponding Layer ID.

## 4. Runtime features

DiT parallelism reuses the native TP and sequence-parallel attention layers.
The DiT has 30 attention heads: TP multiplied by Ulysses must divide 30.
Encoder folding has its own head constraints; two-way folding is compatible
with the 16-head language tower, 16-head vision tower, and 12-head connector.
Do not infer encoder compatibility from the DiT's head count.

Use the standard component residency controls for `transformer`, `text_encoder`,
and `vae`. Layerwise offload trades host memory and transfer time for lower VRAM.
The default eager path does not use approximate caching or quantized attention.
Cache-DiT and SageAttention require workload-specific quality validation.
VAE tiling is disabled by default to match the official inference path. Enable
`--vae-tiling true` when decode memory is constrained; tiling can change pixels.

Breakable CUDA graphs preserve Ming's checkpoint-specific padding and RoPE
offsets. They capture exact prompt shapes; an uncaptured shape runs eagerly.
Requests with different prompts are not merged into a dynamic batch. A fixed
seed does not imply bit-identical output across attention backends, GPU types,
or parallel topologies.

Full-checkpoint validation covers generation, editing, and layer decomposition
on H200, including repeated HTTP requests before and after warmup. Tested
resolutions include 2048-square generation and four-layer 1024-square
decomposition, plus repeated 512-square HTTP generation and editing with one or
two sequential generations per request.

The validated runtime scope is:

* One H200: DiT and encoder layerwise offload, opt-in VAE tiling, Cache-DiT,
  SageAttention, breakable CUDA graph replay, and dynamic LoRA loading/removal
  with a synthetic adapter.
* Two H200s: DiT TP2, Ulysses2, Ring2 with FlashAttention, CFG parallelism for
  Design-Layer, encoder folding, and spatial VAE decoding.

These are functional checks, not quality-equivalence claims for lossy
optimizations. Other GPU families and quantized checkpoints remain unverified.

See [performance optimization](/docs/sglang-diffusion/performance-optimization)
for shared runtime controls and the
[official repository](https://github.com/inclusionAI/Ming-Image) for model examples.

## 5. Run in ComfyUI

Run this model from ComfyUI with the
[SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion), which ships in the SGLang
repository at `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion`.

**Server mode** — SGLang runs the pipeline and ComfyUI
sends the request. Start a server as shown above, point the
`SGLDiffusion Server Model` node at it, then generate with
`SGLDiffusion Generate Image`.

**Integrated mode** — not available for this model
, which has no executor in the plugin
. Use server mode.

Use the generic image node for Design generation. The ComfyUI integration has
not been validated for ordered multi-layer output; use the HTTP API above for
Design-Layer decomposition.


# Overview
Source: https://docs.sglang.io/cookbook/diffusion/intro

Practical guides for deploying and using diffusion models with SGLang.

Choose a recipe by output modality. The sidebar stays organized by model family, while this overview separates image, video, and realtime/world workloads.

## Image Models

Image models generate one image request as a bounded denoising job, usually with bidirectional attention over the whole latent sequence.

> **[HunyuanImage-3.0-Instruct](/cookbook/diffusion/Tencent/HunyuanImage3)**

> **[FLUX](/cookbook/diffusion/FLUX/FLUX)**

> **[Ideogram 4](/cookbook/diffusion/Ideogram/Ideogram4)**

> **[Qwen-Image](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1)**

> **[SenseNova-U1.5](/cookbook/diffusion/SenseNova/SenseNova-U1.5-8B-MoT)**

> **[LongCat-Image](/cookbook/diffusion/LongCat/LongCat-Image)**

> **[Z-Image](/cookbook/diffusion/Z-Image/Z-Image-Turbo)**

> **[Ming-Image](/cookbook/diffusion/inclusionAI/Ming-Image)**

> **[Krea](/cookbook/diffusion/Krea/Krea-2)**

> **[ERNIE-Image](/cookbook/diffusion/Ernie-Image/Ernie-Image)**

## Video Models

Video models denoise a bounded latent video sequence for each request. Use these recipes for offline text-to-video, image-to-video, and video generation serving.

> **[FLUX.3](/cookbook/diffusion/FLUX/FLUX.3)**

> **[Cosmos](/cookbook/diffusion/Cosmos/Cosmos3)**

> **[Wan](/cookbook/diffusion/Wan/Wan2.2)**

> **[LongLive 2.0](/cookbook/diffusion/LongLive/LongLive-2.0)**

> **[LTX](</cookbook/diffusion/LTX/LTX2 & LTX2.3>)**

> **[SANA-Video](/cookbook/diffusion/SANA-Video/SANA-Video)**

> **[JoyAI-Echo](/cookbook/diffusion/JoyEcho/JoyEcho)**

> **[MOVA](/cookbook/diffusion/MOVA/MOVA)**

> **[MiniMax-H3](/cookbook/diffusion/MiniMax/MiniMax-H3)**

> **[MOSS-Image](/cookbook/diffusion/OpenMOSS/MOSS-Image)**

> **[LingBot Video MoE](/cookbook/diffusion/LingBot-Video/LingBot-Video-MoE)**

## Realtime / World Models

Realtime models keep a session alive and generate chunk by chunk with causal state, control signals, and cached video history.

> **[LingBot World](/cookbook/diffusion/LingBot-World/LingBot-World-2.0)**

> **[SANA-WM](/cookbook/diffusion/SANA-WM/SANA-WM)**

Use the sidebar group for LingBot World family variants. The overview links the newer LingBot World 2.0 recipe directly.


# SGLang Cookbook
Source: https://docs.sglang.io/cookbook/intro



A community-maintained repository of practical guides and recipes for deploying and using SGLang in production environments. Our mission is simple: answer the question **"How do I use SGLang (and related models) on hardware Y for task Z?"** with clear, actionable solutions.

## Guides

> **[Autoregressive Models](./autoregressive/intro)**

> **[Diffusion Models](./diffusion/intro)**

> **[VLA (Vision-Language-Action) Models](./vla/intro)**

## Benchmarks

> **[Autoregressive Model Benchmark](./base/benchmarks/autoregressive_model_benchmark)**

> **[Diffusion Model Benchmark](./base/benchmarks/diffusion_model_benchmark)**


# SpecBundle Usage
Source: https://docs.sglang.io/cookbook/specbundle/specbundle_usage



![specbundle logo](https://mintcdn.com/lmsysorg/iZdDMbLWP1BLEIzC/logo/logo.png?fit=max\&auto=format\&n=iZdDMbLWP1BLEIzC\&q=85\&s=b27d467947ee2167744cb07b69badf00)

## About SpecBundle

Speculative decoding, especially EAGLE3, offer strong theoretical guarantees alongside consistent empirical improvements in token acceptance rate and end-to-end inference speed. However, despite these advances, adoption of speculative decoding—especially EAGLE3—remains limited in the open-source ecosystem, due primarily to three key factors.

1. Lack of production-ready training infrastructure: Existing speculative decoding toolchains are largely research prototypes, offering limited system-level optimization and inadequate support for diverse architectures and large-scale models.
2. Scarcity of high-quality draft models: Effective speculative decoding depends on strong draft models, yet publicly available EAGLE3-compatible checkpoints are extremely limited, primarily originating from the original authors.
3. Insufficient training scale of existing drafts: Most available draft models are trained on small or curated datasets and fail to generalize to the large, diverse corpora used in modern LLM training, resulting in low token acceptance rates and diminished practical speedups.

**SpecBundle** is a direct response to these limitations. Jointly driven by the open-source community and industry partners including **Ant Group**, **Meituan**, **Nex-AGI** and **EigenAI**, **SpecBundle** represents the **first open initiative** aimed at democratizing speculative decoding by providing high-performance, production-grade EAGLE3 draft model weights for mainstream open-source LLMs. This initiative also serves to verify the robustness of the [**SpecForge**](https://github.com/sgl-project/SpecForge) framework through multiple scales and architectures.

## Installation

```bash Command theme={null}
git clone https://github.com/sgl-project/SpecForge.git
```

## Usage

### Launch SGLang Server with SpecBundle models

You can use the following command to launch the SGLang server with SpecBundle models. Please add `--tp`, `--ep` and `--mem-fraction-static` arguments when you encounter memory issues.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model <target-model-path> \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path <draft-model-path> \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4
```

For example:

```bash Command theme={null}
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \
    --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --tp 4
```

### Use SpecBundle to compare the performance of Speculative Decoding draft models

We provide a benchmark suite to evaluate the performance of SpecBundle draft models [here](https://github.com/sgl-project/SpecForge/tree/main/benchmarks).

#### Example:

1. Launch a SGLang Server

```bash Command theme={null}
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \
    --model Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --tp 4
```

2. Use the benchmark suite to evaluate the performance of SpecBundle draft models

`bench_eagle3.py` can help you launch a SGLang server process and a Benchmarking process concurrently. In this way, you don't have to launch the SGLang server manually, this script will manually handle the SGLang launch under different speculative decoding configurations. Some important arguments are:

* `--model-path`: the path to the target model.
* `--speculative-draft-model-path`: the path to the draft model.
* `--port`: the port to launch the SGLang server.
* `--trust-remote-code`: trust the remote code.
* `--mem-fraction-static`: the memory fraction for the static memory.
* `--tp-size`: the tensor parallelism size.
* `--attention-backend`: the attention backend.
* `--config-list`: the list of speculative decoding configuration to test, the format is `<batch-size>,<num-steps>,<topk>,<num-draft-tokens>`.
* `--benchmark-list`: the list of benchmarks to test, the format is `<benchmark-name>:<num-prompts>:<subset>`.

```bash Command theme={null}
cd SpecForge/benchmarks
python bench_eagle3.py \
    --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --port 30000 \
    --config-list 1,3,1,4 \
    --benchmark-list mtbench:5 gsm8k:100 \
    --skip-launch-server
```

**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate test command for your model and benchmark.

Generated Command

<pre />

It will generate a json file, content is listed below:

```json Config theme={null}
{
  "mtbench": [
    {
      "batch_size": 1,
      "steps": null,
      "topk": null,
      "num_draft_tokens": null,
      "metrics": [
        {
          "latency": 12.232808108034078,
          "output_throughput": 319.71399906382845,
          "accept_length": 2.170366259711432,
          "accuracy": null,
          "num_questions": 5,
          "num_valid_predictions": 0,
          "categorical_performance": null
        }
      ],
      "num_samples": 5
    }
  ],
  "gsm8k": [
    {
      "batch_size": 1,
      "steps": null,
      "topk": null,
      "num_draft_tokens": null,
      "metrics": [
        {
          "latency": 37.42077191895805,
          "output_throughput": 373.6160234823207,
          "accept_length": 2.643410852713178,
          "accuracy": 0.96,
          "num_questions": 100,
          "num_valid_predictions": 100,
          "categorical_performance": null
        }
      ],
      "num_samples": 100
    }
  ]
}
```

## Performance Scores

We evaluate the performance of SpecBundle draft models on various benchmarks, please visit the [Performance Dashboard](https://docs.sglang.io/SpecForge/SpecBundle/index.html) for more details.


# Supported Models
Source: https://docs.sglang.io/cookbook/specbundle/supported_models



## [Released Models](https://huggingface.co/collections/lmsys/specbundle)

We list the models released by the SpecForge and several industrial partners below. These models are released as part of the SpecBundle models, which are trained on large-scale multi-domain datasets and deliver exceptional performance on various benchmarks.

> We also include some of the models previously trained by the SpecForge team but not technically part of the SpecBundle release.
> We mark models trained on ShareGPT+Ultrachat datasets with a **\*** mark and models trained on Perfect-Blend datasets but released before SpecBundle with **+** mark.

### Llama Series

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target Model</th>
      <th>EAGLE3 Draft Model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>meta-llama/Llama-3.1-8B-Instruct</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-3.1-8B-Instruct-SpecForge)</td>
    </tr>

    <tr>
      <td>meta-llama/Llama-3.3-70B-Instruct</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-3.3-70B-Instruct-SpecForge)</td>
    </tr>

    <tr>
      <td>meta-llama/Llama-4-Scout-17B-16E-Instruct</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Llama-4-Scout-17B-16E-Instruct-SpecForge)</td>
    </tr>

    <tr>
      <td>meta-llama/Llama-4-Maverick-17B-128E-Instruct</td>
      <td>[🤗 Hugging Face \*](https://huggingface.co/lmsys/sglang-EAGLE3-Llama-4-Maverick-17B-128E-Instruct-v1)</td>
    </tr>
  </tbody>
</table>

### Qwen Series

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target Model</th>
      <th>EAGLE3 Draft Model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen/Qwen3-30B-A3B-Instruct-2507</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-30B-A3B-Instruct-2507-SpecForge-Nex)</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-235B-A22B-Instruct-2507</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-235B-A22B-Instruct-2507-SpecForge-Meituan)</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-Next-80B-A3B-Instruct-FP8</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Next-80B-A3B-Instruct-FP8-perfect-blend-regenerated)</td>
    </tr>
  </tbody>
</table>

### Qwen Coder Series

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target Model</th>
      <th>EAGLE3 Draft Model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen/Qwen3-Coder-30B-A3B-Instruct</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Coder-30B-A3B-Instruct-SpecForge)</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-Coder-480B-A35B-Instruct</td>
      <td>[🤗 Hugging Face](https://huggingface.co/lmsys/SGLang-EAGLE3-Qwen3-Coder-480B-A35B-Instruct-SpecForge-EigenAI)</td>
    </tr>
  </tbody>
</table>

### Ling Series

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target Model</th>
      <th>EAGLE3 Draft Model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>inclusionAI/Ling-flash-2.0</td>
      <td>[🤗 Hugging Face](https://huggingface.co/AQ-MedAI/Ling-Flash-2.0-eagle3)</td>
    </tr>
  </tbody>
</table>

### Kimi Series

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target Model</th>
      <th>EAGLE3 Draft Model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>moonshotai/Kimi-K2-Instruct</td>
      <td>[🤗 Hugging Face](https://huggingface.co/AQ-MedAI/Kimi-K2-Instruct-eagle3)</td>
    </tr>
  </tbody>
</table>

### GPT-OSS Series

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target Model</th>
      <th>EAGLE3 Draft Model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>openai/gpt-oss-20b</td>
      <td>[🤗 Hugging Face +](https://huggingface.co/zhuyksir/EAGLE3-gpt-oss-20b-bf16)</td>
    </tr>

    <tr>
      <td>openai/gpt-oss-120b</td>
      <td>[🤗 Hugging Face +](https://huggingface.co/lmsys/EAGLE3-gpt-oss-120b-bf16)</td>
    </tr>
  </tbody>
</table>

### Nex Series

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target Model</th>
      <th>EAGLE3 Draft Model</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>nex-agi/Qwen3-30B-A3B-Nex-N1</td>
      <td>[🤗 Hugging Face](https://huggingface.co/nex-agi/SGLANG-EAGLE3-Qwen3-30B-A3B-Nex-N1)</td>
    </tr>

    <tr>
      <td>nex-agi/Qwen3-32B-Nex-N1</td>
      <td>[🤗 Hugging Face](https://huggingface.co/nex-agi/SGLANG-EAGLE3-Qwen3-32B-Nex-N1)</td>
    </tr>
  </tbody>
</table>


# FLUX 3 Action
Source: https://docs.sglang.io/cookbook/vla/FLUX/FLUX-3-Action



## 1. Model Introduction

FLUX 3 Action is a robot policy from Black Forest Labs built on a FLUX 3 video diffusion transformer. From the current camera frames, the robot state and a language instruction, it denoises a chunk of future video latents **jointly** with a chunk of continuous actions; only the actions are returned.

The model has three parts:

* **DiT** (`JointSingleSeq`, about 6.6B parameters). Each stream (text, `video`, `video_cond`, the action and state streams) runs through its own 5 mode blocks. The text and all streams then share 28 joint single-stream blocks with per-stream modulation and a 4-axis `(t, h, w, l)` RoPE.
* **Text encoder**: Qwen3-VL-4B. Eight hidden layers are stacked into a 20480-dim context.
* **Video VAE**: a Swin3D neighborhood-attention VAE (NATTEN), 96 latent channels, 32x spatial compression.

The sampler is Cosmos UniPC (order 2). Classifier-free guidance is applied per stream (DROID: `4.0` on video, `1.0` on actions).

SGLang runs it in the native `multimodal_gen` runtime. Work that does not depend on the noised streams is computed once:

* The text context, once per caption, across requests.
* The observation and state streams, once per request.
* The mode blocks of the noised streams, once per step, shared by the conditional and unconditional passes.

Supported checkpoints (`black-forest-labs/flux-3-action-droid`; three 360x640 cameras `wrist`, `left`, `right`; state and action dim 8; 32-action chunks):

| `--model-variant` | Recipe | Weights | Latency on 1x RTX 5090 | Latency on 1x H200 |
| - | - | - | -: | -: |
| (default) / `base` | 4 steps, guidance 4.0 (video) / 1.0 (action) | BF16 | 1.14 s | 0.41 s |
| `fp8r` | same | FP8 rowwise | 0.93 s | 0.47 s |
| `gd` | guidance-distilled: 4 steps, no CFG | BF16 | 0.63 s | 0.24 s |
| `gd-fp8r` | same | FP8 rowwise | 0.52 s | 0.27 s |
| `sd` | step-distilled: 1 step | BF16 | 0.18 s | 0.09 s |
| `sd-fp8r` | same | FP8 rowwise | 0.16 s | 0.11 s |

Latency is the median server-side time per request (`server_timing.infer_ms`) over 50 sequential requests after 10 warmup requests, on one GPU with the default serve command, eager mode (no `torch.compile`). Requests go through the OpenPI WebSocket with msgpack numpy images, and the caption is cached. Measure it with `python -m sglang.multimodal_gen.benchmarks.bench_flux3_action --url ws://127.0.0.1:30000` against a running server. On H200 the FP8r packages are slower than BF16; they save memory. LayerNorm + modulation, SwiGLU and the gated residual run on bit-exact fused kernels; QK RMSNorm + RoPE runs on a fused kernel at bf16 rounding level (set `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0` to use the eager path). FP8r packages load their native E4M3 weights (one scale per output row) and run `torch._scaled_mm` with per-token activation scales. This needs an SM89+ GPU.

The frozen text encoder and video VAE are downloaded from the pinned revision of [`black-forest-labs/flux-3-action-base`](https://huggingface.co/black-forest-labs/flux-3-action-base) that the policy config references.

References:

* [FLUX Action](https://github.com/black-forest-labs/flux-action)
* [FLUX 3 Action collection](https://huggingface.co/collections/black-forest-labs/flux-3-action)

## 2. Installation

```bash Command theme={null}
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[diffusion]"
```

The video VAE runs its neighborhood attention on [NATTEN](https://natten.org) when it is installed. NATTEN is not a dependency of `sglang[diffusion]`; without it the VAE uses a compiled FlexAttention fallback with the same windows (actions stay at the bf16 rounding level). The fallback compiles on the first request (about 3 s on H200) and is then no slower than NATTEN for this single-frame encode (24 ms vs. 36 ms per request on H200). To use NATTEN, pick the wheel that matches your torch and CUDA versions from [whl.natten.org](https://whl.natten.org/).

## 3. Model Deployment

Serve the DROID policy:

```bash Command theme={null}
sglang serve black-forest-labs/flux-3-action-droid \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000
```

Serve a distilled variant:

```bash Command theme={null}
sglang serve black-forest-labs/flux-3-action-droid \
  --model-type diffusion \
  --model-variant gd \
  --port 30000
```

A local policy export (a directory containing `manifest.json`, `config.native.json` and `model.safetensors`) is detected automatically when passed as the model path. Pass `--revision` to pin a Hub revision. At startup the policy config and weights are checked against the SHA-256 hashes in `manifest.json`.

Peak GPU memory and per-request latency on one RTX 5090. The resident rows use the same median protocol as the table above. The layerwise-offload rows are from the earlier single-run measurement.

| Flags | Peak memory | Latency |
| - | -: | -: |
| (none) | 23.3 GiB | 1.14 s |
| `--dit-layerwise-offload` | 14.5 GB | 2.51 s |
| `--layerwise-offload-components all` | 9.8 GB | 2.56 s |
| `--model-variant fp8r` | 17.9 GiB | 0.93 s |
| `--model-variant fp8r --dit-layerwise-offload` | 13.2 GB | 1.29 s |

Layerwise offload streams the DiT blocks (and, with `all`, the text encoder and VAE blocks) from host memory, so use it only when the resident configuration does not fit.

### 3.1 Multi-GPU

Three layouts split the DiT across GPUs. They compose as `--num-gpus` = TP size x SP degree x (2 with CFG parallel):

| Flags | What is split | Actions vs 1 GPU |
| - | - | - |
| `--num-gpus 2 --enable-cfg-parallel` | The conditional and unconditional passes | Bit-identical |
| `--num-gpus 2 --tp-size 2` | DiT weights (heads and MLP channels), one all-reduce per block | bf16 rounding level |
| `--num-gpus 2 --sp-degree 2` | The joint sequence and the video mode blocks (K/V gather; add `--ulysses-degree 2` for Ulysses) | bf16 rounding level |

CFG parallel only helps recipes that use guidance (`base`, `fp8r`). The distilled `gd` and `sd` variants run one pass per step, so both GPUs compute the same pass. TP and SP accept the FP8r packages too. Ring attention is not supported.

```bash Command theme={null}
sglang serve black-forest-labs/flux-3-action-droid \
  --model-type diffusion \
  --num-gpus 2 \
  --enable-cfg-parallel \
  --port 30000
```

### 3.2 Action Request Schema

| Field | Type | Description |
| - | - | - |
| `input.task` | string | Language instruction. |
| `input.observation.images` | object | Camera name -> HWC RGB image: a base64 PNG or JPEG (`{"b64_json": ...}`), or a pixel array of uint8 or floats in `[0, 1]`. DROID: `wrist`, `left`, `right` (or the LeRobot names `wrist_image_left`, `exterior_image_1_left`, `exterior_image_2_left`). Alternatively send `composite`: the 540x640 image with the wrist camera on top and the two exterior cameras at half resolution below. |
| `input.observation.state` | array | Robot state in dataset units. DROID: 7 joint positions (rad) followed by the gripper position. |
| `parameters.num_inference_steps` | integer, optional | Defaults to the checkpoint recipe. |
| `parameters.guidance_scale` / `guidance_scale_action` | number, optional | Guidance scale on the video / action stream. Defaults to the checkpoint recipe. |
| `parameters.seed` | integer, optional | Noise seed. Defaults to the package's `inference_seed` (`0` for DROID), so a repeated observation returns the same actions. |
| `runtime.prefix_cache` | boolean, optional | `false` bypasses the per-caption text context cache for this request. Defaults to `true`. |
| `runtime.output_format` | `"list"` or `"numpy"`, optional | Use `"numpy"` with msgpack clients. |

The response returns absolute commands of shape `[32, 8]` in the dataset's conventions.

## 4. API Usage

### 4.1 Generic Action HTTP API

```python Example theme={null}
import base64
import io

import numpy as np
import requests
from PIL import Image


def png(image: np.ndarray) -> dict:
    buffer = io.BytesIO()
    Image.fromarray(image).save(buffer, format="PNG")
    return {"b64_json": base64.b64encode(buffer.getvalue()).decode()}


image = np.zeros((360, 640, 3), dtype=np.uint8)
payload = {
    "input": {
        "task": "put the marker in the cup",
        "observation": {
            "images": {"wrist": png(image), "left": png(image), "right": png(image)},
            "state": np.zeros(8, dtype=np.float32).tolist(),
        },
    },
}
response = requests.post("http://127.0.0.1:30000/v1/actions/generations", json=payload)
action = response.json()["data"][0]["action"]
print(action["shape"])  # [32, 8]
```

PNG is lossless. Pixel lists (`image.tolist()`) are accepted too, but three 360x640 cameras become about 11 MB of JSON, and encoding and parsing it adds about 0.3 s per request on top of the base64 PNG payload. For the lowest overhead, use the OpenPI WebSocket below with msgpack numpy arrays.

`GET /v1/actions/metadata` reports the camera keys, action shape and sampler defaults of the served policy.

### 4.2 OpenPI-Compatible WebSocket

`/openpi/policy` takes one observation per message, with camera images as
`observation.images.<camera>` (`wrist`, `left`, `right`, or the LeRobot names
`wrist_image_left`, `exterior_image_1_left`, `exterior_image_2_left`), the state
as `observation.state` and the instruction as `task` or `prompt`. The response
carries the `[32, 8]` chunk as `actions`. Use the msgpack helpers from the
[Pi0.5 page](/cookbook/vla/OpenPI/Pi0.5) to pack numpy arrays.

## 5. Accuracy

With the same observation and seed, SGLang matches the FLUX Action reference implementation to the bf16 rounding level:

* The DiT agrees to 1e-6 (relative) in fp32.
* The VAE latents and text contexts are bit-identical.
* Across the full 4-step, CFG 4.0 sampling loop, actions differ by at most 0.015 rad (mean 0.003 rad). The reference's own eager and prepared paths differ from each other by 0.015 rad.
* FP8r packages differ from the reference FP8r path by at most 0.025 rad. The reference FP8r path itself differs from BF16 by 0.055 rad.


# Pi0.5
Source: https://docs.sglang.io/cookbook/vla/OpenPI/Pi0.5



dVLA
OpenPI / LeRobot
flow matching action
robot edge

## 1. Model Introduction

Pi0.5 is an OpenPI / LeRobot diffusion Vision-Language-Action (dVLA) policy. It consumes camera images, a language instruction, and robot state, then returns a continuous action chunk for robot control.

SGLang serves Pi0.5 through the native `multimodal_gen` runtime. The implementation uses a SigLIP/PaliGemma prefix encoder and a Gemma action expert: the prefix is encoded once, then the action expert runs the flow-matching denoising loop. This is not a token decode workload, so the Pi0.5 path does not use the LLM sampler, logits processor, token streaming, paged decode KV cache, or a separate SRT serving engine.

The SigLIP vision tower, PaliGemma language stack, and action expert are all SGLang-native modules. Transformers is used for checkpoint configuration and tokenization, not for the runtime neural network.

The prefix encoder covers both stages of observation encoding: SigLIP turns resized camera pixels into continuous patch embeddings, then the PaliGemma transformer jointly encodes those patches with tokenized task/state inputs and produces per-layer prefix K/V. At flow timestep `t`, the action expert projects the noisy continuous action chunk `x_t` into action embeddings. Its queries attend to both the fixed prefix K/V and the current action K/V, while the timestep follows a separate sinusoidal-MLP path and conditions every action-expert layer through AdaRMSNorm gates.

Supported public checkpoints:

| Checkpoint | Cameras | State Dim | Output Action Dim | Action Horizon | Denoise Steps |
| - | - | -: | -: | -: | -: |
| `lerobot/pi05_base` | `base_0_rgb`, `left_wrist_0_rgb`, `right_wrist_0_rgb` | 32 | 32 | 50 | 10 |
| `lerobot/pi05_libero_base` | `image`, `image2`, one empty camera | 8 | 7 | 50 | 10 |

References:

* [OpenPI](https://github.com/Physical-Intelligence/openpi)
* [lerobot/pi05\_base](https://huggingface.co/lerobot/pi05_base)
* [LeRobot Pi0.5 docs](https://huggingface.co/docs/lerobot/en/pi05)

## 2. Installation

Install SGLang with the diffusion extra. Pi0.5 lives in `multimodal_gen`, and the extra includes the runtime dependencies used by the policy server.

```bash Command theme={null}
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[diffusion]"
```

For general environment setup, see the [SGLang Diffusion installation guide](/docs/sglang-diffusion/installation).

## 3. Model Deployment

Serve the base Pi0.5 policy:

```bash Command theme={null}
sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000
```

Serve the LIBERO checkpoint:

```bash Command theme={null}
sglang serve lerobot/pi05_libero_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000
```

These registered LeRobot checkpoints dispatch to the native `multimodal_gen` Pi0.5 pipeline automatically. The `--pipeline` / `--pipeline-class-name` flag is only an advanced override for local or private checkpoints that cannot be resolved from the model registry. Pi0.5 serving does not start a second SRT LLM serving engine.

### 3.1 Action Request Schema

| Field | Type | Description |
| - | - | - |
| `model` | string, optional | Served model name. When omitted, the server uses the currently loaded policy. |
| `input.task` or `input.prompt` | string | Language instruction for the policy. |
| `input.observation.images` | object | Map from camera name to RGB image. Images can be nested here or sent as OpenPI observation keys such as `observation.images.base_0_rgb` through the websocket adapter. |
| `input.observation.state` | array or tensor object | Robot state vector. Use the same normalization convention as the OpenPI / LeRobot checkpoint. |
| `input.observation.noise` | array or tensor object, optional | Initial action noise with shape `[action_horizon, action_dim]`. Use this for deterministic debugging. |
| `parameters.num_inference_steps` | integer, optional | Flow-matching denoise steps. Defaults to `10`. |
| `parameters.action_horizon` | integer, optional | Output action horizon. Defaults to the checkpoint config. |
| `parameters.action_dim` | integer, optional | Internal padded action dimension. Defaults to the checkpoint config. |
| `runtime.return_timing` | boolean, optional | Return stage timing fields. Defaults to `true`. |
| `runtime.prefix_cache` | boolean or `"auto"`, optional | Enable exact full-prefix lookup for this request when the server has `enable_global_prefix_cache=true`. Defaults to `"auto"`. |
| `runtime.cuda_graph` | boolean or `"auto"`, optional | Enable the available prefix and action-denoise CUDA graph paths for this request. Defaults to `"auto"`. |
| `runtime.output_format` | `"list"` or `"numpy"`, optional | Use `"list"` for JSON compatibility. Use `"numpy"` with msgpack or Python clients to avoid Python-list materialization. Defaults to `"list"`. |
| `runtime.response_format` | `"envelope"` or `"raw"`, optional | HTTP-only response shape. `"envelope"` returns the generic action envelope. `"raw"` returns the policy payload directly. Defaults to `"envelope"`. |

## 4. API Usage

### 4.1 Generic Action HTTP API

Use `/v1/actions/generations` for direct policy calls and debugging. Use `/v1/actions/metadata` to discover the camera keys, state size, action shape, defaults, and websocket capabilities of the currently served policy.

```python Example theme={null}
import numpy as np
import requests

image = np.zeros((224, 224, 3), dtype=np.uint8)
payload = {
    "model": "lerobot/pi05_base",
    "input": {
        "task": "pick up the block",
        "observation": {
            "images": {
                "base_0_rgb": image.tolist(),
                "left_wrist_0_rgb": image.tolist(),
                "right_wrist_0_rgb": image.tolist(),
            },
            "state": np.zeros(32, dtype=np.float32).tolist(),
        },
    },
    "runtime": {
        "return_timing": True,
        "prefix_cache": "auto",
        "cuda_graph": "auto",
    },
}

response = requests.post(
    "http://127.0.0.1:30000/v1/actions/generations",
    json=payload,
    timeout=60,
)
response.raise_for_status()
data = response.json()

actions = data["data"][0]["action"]["values"]
print(len(actions), len(actions[0]))
print(data.get("timings"))
```

The same `/v1/actions/generations` endpoint also accepts `Content-Type: application/msgpack` and can return msgpack when `Accept: application/msgpack` is set. For msgpack clients, send numpy arrays directly using the `pack_numpy_payload` helper from the websocket example below. Msgpack requests default to numpy action output on the server side; set `runtime.output_format` to `"list"` only when a client explicitly needs nested Python lists.

For the lowest-overhead generic HTTP path, use msgpack with the raw policy response:

```python Example theme={null}
payload["runtime"] = {
    "return_timing": True,
    "prefix_cache": False,
    "cuda_graph": "auto",
    "response_format": "raw",
}

response = requests.post(
    "http://127.0.0.1:30000/v1/actions/generations",
    data=packb(payload),
    headers={
        "Content-Type": "application/msgpack",
        "Accept": "application/msgpack",
    },
    timeout=60,
)
result = unpackb(response.content)
actions = result["actions"]
```

For `lerobot/pi05_libero_base`, use the LIBERO camera names and an 8-dimensional state vector:

```python Example theme={null}
payload = {
    "input": {
        "task": "pick up the object",
        "observation": {
            "images": {
                "image": image.tolist(),
                "image2": image.tolist(),
            },
            "state": np.zeros(8, dtype=np.float32).tolist(),
        },
    },
}
```

### 4.2 Generic Realtime WebSocket

`/v1/actions/realtime` is the generic msgpack websocket path. It sends `action.metadata` on connect and returns the same `action.generation` envelope as the HTTP API for each request.

### 4.3 OpenPI-Compatible WebSocket

Robot clients can use the OpenPI-compatible msgpack websocket endpoint at `/openpi/policy`. The server sends metadata immediately after connection, then each client message should contain one observation.

```python Example theme={null}
import asyncio

import msgpack
import numpy as np
import websockets


def pack_array(obj):
    if isinstance(obj, np.ndarray):
        return {
            b"__ndarray__": True,
            b"data": obj.tobytes(),
            b"dtype": obj.dtype.str,
            b"shape": obj.shape,
        }
    if isinstance(obj, np.generic):
        return {
            b"__npgeneric__": True,
            b"data": obj.item(),
            b"dtype": obj.dtype.str,
        }
    return obj


def unpack_array(obj):
    ndarray_marker = obj.get("__ndarray__") or obj.get(b"__ndarray__")
    npgeneric_marker = obj.get("__npgeneric__") or obj.get(b"__npgeneric__")
    data = obj.get("data", obj.get(b"data"))
    dtype = obj.get("dtype", obj.get(b"dtype"))
    shape = obj.get("shape", obj.get(b"shape"))
    if ndarray_marker:
        return np.ndarray(
            buffer=data,
            dtype=np.dtype(dtype),
            shape=shape,
        )
    if npgeneric_marker:
        return np.dtype(dtype).type(data)
    return obj


def packb(payload):
    return msgpack.packb(payload, default=pack_array, use_bin_type=True)


def unpackb(payload):
    return msgpack.unpackb(payload, object_hook=unpack_array, raw=False)


async def main():
    image = np.zeros((224, 224, 3), dtype=np.uint8)
    observation = {
        "task": "pick up the block",
        "observation.images.base_0_rgb": image,
        "observation.images.left_wrist_0_rgb": image,
        "observation.images.right_wrist_0_rgb": image,
        "observation.state": np.zeros(32, dtype=np.float32),
    }

    async with websockets.connect(
        "ws://127.0.0.1:30000/openpi/policy",
        max_size=None,
    ) as websocket:
        metadata = unpackb(await websocket.recv())
        print(metadata)

        await websocket.send(packb(observation))
        result = unpackb(await websocket.recv())
        actions = result["actions"]
        print(len(actions), len(actions[0]))
        print(result.get("server_timing"))


asyncio.run(main())
```

## 5. Configuration Tips

* Request-local `PrefixContext` is always reused across all denoise steps in one request. The prefix K/V is not cloned per step.
* The optional global prefix cache is a bounded exact-match LRU. It is disabled by default because changing robot frames rarely hit it and enabling it prevents unrelated misses from entering grouped prefix execution. Set `enable_global_prefix_cache=true` for repeated observations, retries, or multiple policy calls over the same camera/state sample; `runtime.prefix_cache` can then disable lookup per request.
* Partial-prefix reuse is not supported because Pi0.5 combines image and tokenized task/state inputs under full attention. Changing any input can change every deeper-layer prefix K/V tensor. The exact key hashes resized and normalized pixels before SigLIP, plus effective token IDs, token masks, camera masks, model revision, dtype, and parallel layout. Tensor content hashing reuses SRT's CPU/CUDA implementation; hashing the pre-SigLIP input lets an exact hit skip both the vision encoder and prefix transformer.
* CUDA graph capture targets single-request prefix encoding and one action-denoise step. The lossless default keeps exact prompt lengths and one resident prefix graph. Prefix and action graph residency are independently bounded by `prefix_cuda_graph_max_entries` (default `1`) and `action_cuda_graph_max_entries` (default `4`). Without prompt buckets, unseen signatures run eagerly after the cache reaches capacity, avoiding capture churn. The denoise signature includes whether prefix attention is full or masked, plus batch size, prefix length, action horizon, action dim, dtype, and parallel layout. Mask-aware action graphs rebuild position IDs from the current request mask inside the captured graph. With action SP enabled, the denoise bucket uses the local action shard length and rank-specific position offset.
* `prompt_token_buckets` optionally right-pads prompt tokens to fixed sizes such as `[32, 64, 128, 200]`, allowing nearby prompt lengths to reuse both graphs. In this mode the bounded graph caches use LRU replacement and reset evicted CUDA graphs; prompts beyond the largest configured bucket stay exact-length and eager. This is opt-in because padding changes GPU reduction shapes: an H200 sweep over lengths around all four boundaries was structurally correct and bounded to four prefix/action graphs, but non-boundary prompts differed from the exact-length path by up to `0.09589` in normalized action space over five denoise steps. Validate closed-loop policy quality before enabling it; empty buckets preserve the numerically lossless path.

An opt-in four-bucket serving config is:

```json File theme={null}
{
  "prompt_token_buckets": [32, 64, 128, 200],
  "prefix_cuda_graph_max_entries": 4,
  "action_cuda_graph_max_entries": 4
}
```

* Cache-DiT is not used in the default Pi0.5 path. The current robot policy target is numerically lossless inference, while Cache-DiT-style reuse is an image/video DiT approximation that needs separate policy-quality validation before it can be recommended for action control.
* Do not use CFG parallelism to split the 10 Euler steps. Use it only for independent branches such as multiple candidate actions or future conditional/unconditional branches.
* Prefix TP uses native SGLang parallel linear layers for the PaliGemma language prefix model when model parallel TP is initialized and the VLA split broadcast group is not active. The action expert does not share that TP layout. The v1 split prefix/action path instead uses the SP group: prefix root computes/broadcasts `PrefixContext`, while action ranks run the SP action path.
* The split path uses the SP group as the action group: prefix root computes/broadcasts `PrefixContext`, all action ranks broadcast the initial action noise once, shard the action horizon, and run the action expert through Ulysses attention when the prefix is full-attention, ring degree is one, heads are divisible by SP size, and the horizon is evenly shardable. Otherwise it falls back to action-root execution.
* `lerobot/pi05_libero_base` returns 7 action dimensions even though the internal padded action tensor uses 32 dimensions.

## 6. VRAM Tuning

For current public Pi0.5 checkpoints, a stable 16GB discrete GPU target is a reasonable v1 deployment bar for robot workstations. In practice, leave headroom for the driver, camera middleware, robot process, and allocator fragmentation. Jetson/Orin unified-memory devices need extra caution because system RAM and GPU memory share the same pool.

OpenPI inference is mixed precision, not full fp32: most weights and compute run in `bf16`, selected stability-sensitive weights stay in `fp32`, and returned actions are `float32`. SGLang mirrors that policy by default. The validated `pi05_aloha` OpenPI PyTorch checkpoint keeps `119,720,608` parameters in fp32; SGLang reports the same fp32 stability set, plus `3,233,713,264` bf16 runtime parameters after skipping unused LM heads for continuous action inference. The fp32 set includes SigLIP patch/position embeddings, Gemma layer norms/final norms, and the action/time projection heads. Keep `materialize_dtype` at `bf16` unless you are debugging numerical parity.

Current H100 pressure validation shows that the bf16 model path fits a 16GB-free budget without layerwise offload. The Pi0.5 path batches all camera frames for a grouped request into one SigLIP forward before splitting the embeddings back by camera, which is important for multi-camera robot workloads. The latency numbers below use the Python grouped API with global prefix cache disabled and CUDA graph enabled on unconstrained H100; rerun HTTP/OpenPI websocket on your target server before using the policy in a closed-loop robot.

| Mode | Command Shape | Steady VRAM Snapshot | Notes |
| - | - | - | - |
| Single GPU bf16 | `--num-gpus 1`, no offload | fit with `16383 MiB` free before load in H100 pressure test | Recommended first path for 16GB-class discrete GPUs. |
| ALOHA bf16 grouped | Python API, batch size 1 / 2 / 4 / 8 | unconstrained H100 | `52.4 ms` single; `65.1 ms / 2`; `91.9 ms / 4`; `147.4 ms / 8`. |
| ALOHA 16GB-free pressure | Python API, no offload | same 16GB-free pressure | Fit was validated after the bf16 correction; rerun latency on the target because older pressure latency was collected before the final OpenPI precision fix. |
| Global prefix cache disabled | `enable_global_prefix_cache=false` or request `runtime.prefix_cache=false` | avoids cache growth across changing frames | Recommended for tight edge budgets unless repeated exact frames are common. |
| Offload fallback | config in 6.2 | target-dependent | Use only if the default bf16 path OOMs on the target device. |

Use these knobs first because they do not change action numerics for a fixed input/noise:

### 6.1 16GB Edge Config

Use this single-GPU config first for 16GB-class robot workstations. It keeps parameters resident on GPU, keeps CUDA graph enabled, and disables global prefix cache growth.

```json File theme={null}
{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_prefix_cuda_graph": true,
  "prefix_cuda_graph_max_entries": 1,
  "enable_action_cuda_graph": true,
  "action_cuda_graph_max_entries": 1
}
```

Start a single-GPU server with the override:

```bash Command theme={null}
sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --pipeline-config-path pi05_edge_16gb.json \
  --num-gpus 1 \
  --warmup-mode off \
  --host 127.0.0.1 \
  --port 30000
```

Treat the H100 pressure result as a memory-budget validation, not a Jetson latency guarantee. Jetson/Orin devices use shared system memory and have much lower effective memory bandwidth than H100, so measure closed-loop control latency on the target device before deciding the action chunk cadence.

### 6.2 Offload Fallback

Use offload only when the default bf16 config still does not fit on the target. These modes are numerically lossless for fixed inputs/noise, but they move weights between CPU and GPU and can hurt latency substantially.

For a moderate fallback, offload cache growth and selected stage-resident modules first:

```json File theme={null}
{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_prefix_cuda_graph": false,
  "enable_action_cuda_graph": false,
  "offload_prefix_image_encoder_after_embed": true,
  "offload_prefix_token_embedding": true,
  "offload_prefix_language_layer_count_after_prefix": 2,
  "offload_action_expert_after_denoise": true,
  "empty_cache_after_prefix": true
}
```

If that still does not fit, full prefix layerwise CPU offload keeps every PaliGemma language layer on CPU and moves one layer at a time to GPU during prefix compute:

```json File theme={null}
{
  "materialize_dtype": "bf16",
  "enable_global_prefix_cache": false,
  "prefix_cache_max_entries": 0,
  "enable_prefix_cuda_graph": false,
  "enable_action_cuda_graph": false,
  "offload_prefix_image_encoder": true,
  "offload_prefix_token_embedding": true,
  "offload_prefix_language_layers": true,
  "offload_prefix_language_layers_empty_cache": true,
  "empty_cache_after_prefix": true
}
```

Offload validation should be repeated on the target hardware after any dtype or loader change. Earlier fp32-runtime offload numbers are not comparable to the current bf16 path.

### 6.3 Per-Request Controls

For HTTP calls, disable cache or both CUDA graph paths without restarting the server:

```python Example theme={null}
payload = {
    "input": {
        "task": "pick up the block",
        "observation": {
            "images": images,
            "state": state,
        },
    },
    "runtime": {
        "prefix_cache": False,
        "cuda_graph": False,
    },
}
```

For OpenPI websocket clients, include equivalent `enable_prefix_cache` and `enable_cuda_graph` fields in each raw msgpack observation if you need per-request compatibility controls.

### 6.4 Deployment Choices

* Keep batch size to one control stream unless grouped robot streams are explicitly validated. More concurrent observations increase activation and PrefixContext residency.
* Keep `materialize_dtype` at the default `bf16`. `fp32` is useful only for debugging numerical issues and will increase memory and latency.
* Use split prefix/action only when you have multiple GPUs and have validated the robot control latency on that topology. On two GPUs with `--sp-degree 2 --ulysses-degree 2`, the action horizon can be sequence-sharded while the prefix root still computes and broadcasts `PrefixContext`. A single 16GB-class GPU should try the default bf16 config first.
* Reducing `num_inference_steps` lowers latency but is not a pure memory fix and can change policy behavior. Validate closed-loop task success before using fewer than the checkpoint default.
* CPU offload is a compatibility fallback, not the preferred 16GB path. Quantization and deeper prefix/vision sharding are the next steps for sub-16GB devices.

### 6.5 Loader And Run:ai Model Streamer

Run:ai Model Streamer can improve cold-start and checkpoint loading by reading safetensors concurrently and streaming tensors toward GPU memory. It is useful for local SSD, object storage, and cloud deployments where startup time is dominated by model file IO.

The `python[diffusion]` extra includes `runai_model_streamer`. If the package is installed, SGLang enables it by default through `SGLANG_USE_RUNAI_MODEL_STREAMER=true`. Set it to `false` to force the plain safetensors loader:

```bash Command theme={null}
SGLANG_USE_RUNAI_MODEL_STREAMER=false \
sglang serve lerobot/pi05_base \
  --model-type diffusion \
  --host 127.0.0.1 \
  --port 30000
```

Pi0.5 uses the direct SSD-to-GPU Run:ai path when all load targets for the current process are GPU-resident. This check is rank-local: distributed action ranks can stream their action-expert subset directly to GPU, while any rank with CPU/offloaded target tensors stays on the CPU safetensors fallback. The single-GPU validation streamed `13.5 GiB` of safetensors directly to `cuda:0` in about `1.5 s`, then returned `[50, 32]` actions over both HTTP and OpenPI websocket.

For mixed CPU offload and low-VRAM 16GB-class modes, Pi0.5 still uses the header-filtered safe loader for ranks whose target tensors are not fully CUDA-resident. Direct GPU streaming can increase the exact VRAM pressure the low-memory path is trying to avoid. When debugging distributed startup, compare with `SGLANG_USE_RUNAI_MODEL_STREAMER=false` to separate streamer behavior from model execution behavior.

Run:ai Model Streamer does not reduce steady inference VRAM after parameters, caches, activations, CUDA contexts, and graph buffers are resident. For Pi0.5 low-VRAM work, prioritize component placement, prefix cache size, CUDA graph residency, and CPU/offload first. Model Streamer is a cold-start optimization after the steady-memory budget is correct.

## 7. OpenPI Comparison Benchmark

Use `bench_pi05_openpi.py` when you need a side-by-side latency and action-difference report against the OpenPI policy implementation. Start the SGLang Pi0.5 server first for HTTP or websocket modes, then run the benchmark from the SGLang repository root:

```bash Command theme={null}
python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \
  --profile aloha \
  --sglang-url http://127.0.0.1:30000 \
  --openpi-checkpoint gs://openpi-assets/checkpoints/pi05_base \
  --num-inference-steps 10 \
  --batch-size 4 \
  --repeats 20 \
  --warmup 3 \
  --deterministic-noise \
  --output pi05_openpi_compare.json
```

The benchmark reports:

* SGLang HTTP latency for `/v1/actions/generations`, including stage timings when the server returns them.
* SGLang msgpack HTTP latency when `--sglang-api http_msgpack` is set. This keeps the generic HTTP endpoint but avoids JSON image-array overhead. Use `--sglang-http-response-format raw` to benchmark the compact raw policy response.
* SGLang OpenPI-compatible websocket latency when `--sglang-api openpi_ws` is set. This uses persistent msgpack websocket connections and is closer to the robot client path than JSON-over-HTTP.
* SGLang Python in-process latency when `--sglang-api python` is set. This loads the native Pi0.5 pipeline in the benchmark process and avoids HTTP, websocket, scheduler, and serialization overhead. Use `--sglang-python-batch-mode grouped` to exercise the conservative native grouped-batch path. The Python path also reports actual SGLang module parameter dtype counts and example parameter names.
* OpenPI single-request latency through `Policy.infer`.
* Batch latency for grouped robot streams. SGLang uses concurrent HTTP requests in HTTP mode and persistent multi-connection msgpack calls in websocket mode. The Python backend can use true grouped model execution for fresh-prefix requests. OpenPI defaults to its internal direct model batch path because the public `Policy.infer` API is single-observation.
* Action difference in normalized model space from identical OpenPI-transformed model inputs and noise. The check requires `--deterministic-noise` and fails when either `--action-max-abs-diff` or `--action-mean-abs-diff` is exceeded. This mode isolates model parity from robot-specific normalization and action postprocessing. The LIBERO policy returns only 10 actions after policy postprocessing, but its flow-matching model still generates a 50-step chunk; the benchmark compares that model output with SGLang before OpenPI unnormalization and horizon slicing.

For one-sided 16GB-class checks, run each backend separately under the same VRAM pressure. The SGLang Python path accepts the same pipeline config override as serving:

```bash Command theme={null}
python python/sglang/multimodal_gen/benchmarks/bench_pi05_openpi.py \
  --profile aloha \
  --sglang-api python \
  --sglang-python-batch-mode grouped \
  --skip-openpi \
  --sglang-pipeline-config-path pi05_edge_16gb.json \
  --num-inference-steps 10 \
  --batch-size 4 \
  --repeats 5 \
  --warmup 2 \
  --deterministic-noise \
  --disable-prefix-cache \
  --disable-cuda-graph
```

Use `--skip-sglang --openpi-pytorch-compile-mode none` to measure an OpenPI eager baseline in the same environment. For a PyTorch-native OpenPI baseline, point `--openpi-checkpoint` at a checkpoint directory containing `model.safetensors` and the OpenPI `assets/` norm-stat tree. The `keep` mode preserves OpenPI's checkpoint default compile setting. The grouped Python path currently requires compatible fresh-prefix requests without split prefix/action workers or effective prefix-cache hits; other cases fall back to per-request execution.

## 8. Validation Notes

The following checks were run on H100 GPUs with the native SGLang Pi0.5 path:

| Check | Result |
| - | - |
| `lerobot/pi05_base` direct end-to-end | Prefix length `968`, output shape `[1, 50, 32]`, peak allocated memory `12.817 GiB`. |
| Official OpenPI parity | Against OpenPI PyTorch revision `15a9616`, with the same LeRobot checkpoint revision, observation, and noise: first-step velocity max/mean absolute difference `0.02677` / `0.00344`; production 10-step normalized action `0.00813` / `0.00092`. |
| Action denoise CUDA graph | Eager 10-step denoise `125.4 ms`; steady graph replay `50.8 ms`; max output difference `0`. |
| Prefix CUDA graph | On H200 with action graph already enabled, ALOHA batch=1 p50 improved from `48.04 ms` to `42.98 ms` (`1.118x`). Two observations at 5 and 10 steps were bit-exact with graph disabled. One prefix shape bucket added about `48.5 MiB`; batch=4 showed no benefit and stays eager. |
| Exact full-prefix cache | First prefix pass about `203 ms`; exact cache hit prefix stage about `0.2 ms`. |
| `lerobot/pi05_libero_base` direct end-to-end | Image keys `image`, `image2`, `empty_camera_0`; state dim `8`; output action dim `7`; output tensor shape `[1, 50, 32]`. |
| Python grouped execution | ALOHA batch=4 grouped path measured `91.9 ms / 4` on current mixed precision: prefix `18.4 ms`, action denoise `61.2 ms`, preprocess about `2.4 ms` per request. Sequential Python loop batch=4 measured `211.9 ms / 4`. |
| `sglang serve` HTTP | `/v1/actions/generations` returned action shape `[50, 32]`. JSON-over-HTTP remains compatible but image-array serialization dominates. Msgpack HTTP with prefix cache disabled measured `57.2 ms` single and `221.9 ms / 4` for the envelope response, and `56.4 ms` single and `219.7 ms / 4` for `runtime.response_format="raw"`. |
| OpenPI websocket | `/openpi/policy` returned action shape `[50, 32]`; with persistent msgpack connections and prefix cache disabled, ALOHA measured about `77.0 ms` single and `162.0 ms / 4`. |
| 2-GPU prefix/action split | Earlier split validation returned action shape `[50, 32]` and matched single-GPU HTTP with max absolute difference `0`. After the true action-SP change, rerun this check with `--num-gpus 2 --sp-degree 2 --ulysses-degree 2` and verify both action ranks enter denoise kernels. |
| OpenPI/SGLang precision | Official OpenPI JAX inference restores the public GCS checkpoint as bf16 with selected fp32 stability compute and returns float32 actions. The converted OpenPI PyTorch `pi05_aloha` checkpoint keeps `119,720,608` fp32 stability params; SGLang reports the same fp32 set and `3,233,713,264` bf16 runtime params after skipping unused LM heads. |
| Native attention dtype | Checkpoint source tensors may be fp32, but SGLang finalizes PiGemma and SigLIP compute dtype before native attention backend selection; backend logs showed `Using fa attention backend` for the PiGemma path in the prior run. |
| 16GB-free Python pressure | With an H100 artificially constrained to `16381 MiB` free before model load, single-GPU bf16 no-offload Python grouped path completed without OOM. Re-run latency after precision or loader changes before using pressure numbers for deployment sizing. |
| Low-VRAM switches | Disabling prefix cache prevents cache growth across changing robot frames. Prefix graph residency defaults to `1`, action graph residency defaults to `4`, and setting either limit to `0` disables that runner. One validated ALOHA prefix graph used about `48.5 MiB`; action graph residency is now bounded separately. Keep `prompt_token_buckets` empty for the lossless path. |
| Offload fallback | CPU/offload modes are retained as numerically lossless compatibility fallbacks, but earlier fp32-runtime offload latency numbers are stale after the bf16 dtype correction and should be revalidated before deployment decisions. |
| Run:ai direct loader | Single-GPU serve streamed `13.5 GiB` safetensors to `cuda:0` in about `1.5 s` and returned `[50, 32]` actions. Distributed direct streaming is now rank-local and should be revalidated on the target split topology; offload ranks with CPU targets still use the safe loader. |
| OpenPI comparison status | Official OpenPI GCS `pi05_base` is a JAX checkpoint; converted PyTorch eager was validated without `torch.compile`. On 80GB H100, ALOHA OpenPI PyTorch eager was about `125-130 ms` single and about `164 ms / 4` in the direct-model batch path. Current SGLang Python grouped measured `52.4 ms` single and `91.9 ms / 4`; JAX OpenPI was `53.0 ms` single and `59.5 ms / 2` in a short check. |

Run a full HTTP or websocket smoke test in your robot deployment environment before using the policy in a closed-loop controller.


# Overview
Source: https://docs.sglang.io/cookbook/vla/intro

Practical guides for deploying and using Vision-Language-Action policies with SGLang.

Vision-Language-Action policies map camera observations, language instructions, and robot state into continuous action chunks. They share some runtime machinery with diffusion pipelines, but the user-facing workload is robot action inference rather than image or video generation.

This section keeps VLA policies separate from the diffusion model cookbook so robot deployments can document their own input schemas, policy endpoints, cache behavior, and control-loop performance targets.

## OpenPI

> **[Pi0.5](/cookbook/vla/OpenPI/Pi0.5)**

## FLUX

> **[FLUX 3 Action](/cookbook/vla/FLUX/FLUX-3-Action)**


# Adaptive Speculative Decoding
Source: https://docs.sglang.io/docs/advanced_features/adaptive_speculative_decoding



Adaptive speculative decoding lets SGLang adjust `speculative_num_steps/speculative_num_draft_tokens` at runtime instead of keeping a single fixed value for the whole server lifetime.
It is designed for workloads whose accept length changes over time, where one static step count is rarely optimal.

## Current support

* Only `--speculative-algorithm EAGLE` or `EAGLE3`
* Only `--speculative-eagle-topk 1`
* If either condition is not met, SGLang falls back to static speculative settings

## Why adaptive steps help

`speculative_num_steps` controls how many draft-model autoregressive steps run in each speculative round. In practice, the best value depends on the current workload.

* If `num_steps` is too small, the draft model could have produced more accepted tokens, but the round stops too early.
* If `num_steps` is too large, the draft model produces many candidate tokens that the target model rejects, so extra draft work is wasted.
* At **high batch sizes**, the cost of each wasted draft step is multiplied across all sequences in the batch, so the optimal step count is often lower than at low batch sizes.

Real traffic often moves between high-acceptance and low-acceptance phases, and batch sizes vary continuously. Adaptive mode follows both signals at runtime instead of hard-coding a single global `num_steps`.

## Design overview

The adaptive mechanism has three pieces:

* `AdaptiveSpeculativeParams`: the EMA-based policy
* `SpecRuntimeState`: the per-tier runtime state bundle
* `AdaptiveController`: the coordinator that queries the policy for the current batch size and activates the matching runtime state

### Per-batch-size independent tracking

The controller maintains **independent EMA trackers for each batch size range**, so observations at small BS don't pollute the large BS signal. Each BS range can have its own candidate steps, hysteresis thresholds, and ceiling coefficient.

BS ranges are defined as lower bounds in the config file (e.g., keys `"1"` and `"8"` mean BS 1–7 uses one slot, BS 8+ uses another). `SpecRuntimeState` objects are shared across BS ranges with the same step count — each state owns CUDA graphs captured for the reachable padded batch sizes of that step.

```mermaid theme={null}
---
title: "SpecRuntimeState — speculative_num_steps / speculative_num_draft_tokens"
---
graph LR
  subgraph SR[" "]
    direction LR
    subgraph D["Draft stage"]
      direction TB
      d1[attn_backend]
      d2[cuda_graph]
    end
    subgraph V["Verify stage"]
      direction TB
      v1[attn_backend]
      v2[cuda_graph]
    end
    subgraph E["Extend stage"]
      direction TB
      e1[attn_backend]
      e2[cuda_graph]
    end
  end
```

This matters because `CudaGraphRunner` is shape-dependent. Each candidate tier owns its own graph and backend state, so runtime switching is a reference swap, not an online graph recapture.

## Runtime flow

The adaptive update happens in two places:

1. **Pre-draft**: query the optimal step for the current batch size and activate if different
2. **Post-verify**: update the matching BS slot's EMA with observed accept lengths

```mermaid theme={null}
---
title: "EAGLEWorker.forward_batch_generation() — decode path"
---
flowchart TD
  Z["⓪ activate_step_by_batch(batch_size)<br/>query optimal step for current BS range, activate if different"]
  A["① draft(batch)<br/>draft model multi-step generation with current tier"]
  B["② verify(batch, spec_info)<br/>target model tree verification → produces num_correct_drafts_per_req"]
  C["③ forward_draft_extend_after_decode(batch)<br/>draft model KV-cache catch-up"]
  D["④ adaptive_controller.on_verify_complete(num_correct_drafts_per_req, batch_size)<br/>update EMA for matching BS slot, apply warmup / interval / hysteresis gates<br/>if tier changed, select a pre-built state from pool"]
  E["worker.apply_runtime_state(state)"]
  Z --> A --> B --> C --> D --> E
```

> Tier switch happens after the current round completes. Backends and CUDA graphs are never swapped mid-round.

## How the policy decides

After each verify pass, SGLang reads the accepted draft length per request, computes the batch average, smooths it with an exponential moving average (EMA), and switches among the candidate tiers for the matching BS slot.

The decision logic is intentionally conservative:

* `warmup_batches` skips the first few batches
* `update_interval` avoids switching every batch
* `down_hysteresis` and `up_hysteresis` reduce oscillation
* `ceiling_coeff` — an optional EMA ceiling rule can cap `num_steps` proportionally to observed draft quality, preventing over-speculation at high BS

Conceptually, the policy probes one step beyond the observed acceptance:

```text theme={null}
target_steps ≈ clamp(round(ema_accept_len) + 1, min(candidate_steps), max(candidate_steps))
```

So if recent requests consistently accept more drafted tokens, the policy tends to move up. If they start rejecting earlier, it tends to move down.

## Usage

`--speculative-adaptive-config` is optional, but the speculative setup still needs to be valid for adaptive mode.

```bash theme={null}
python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-eagle-topk 1 \
    --speculative-num-steps 3 \
    --speculative-num-draft-tokens 4 \
    --speculative-adaptive
```

If you want to override the defaults, add `--speculative-adaptive-config /path/to/adaptive_spec.json`.

Example config:

```json theme={null}
{
  "ema_alpha": 0.2,
  "warmup_batches": 10,
  "update_interval": 5,
  "1": {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0},
  "8": {"candidate_steps": [1],       "up_hysteresis": 0.0, "down_hysteresis": 0.0,   "ceiling_coeff": 0}
}
```

Non-integer keys (`ema_alpha`, `warmup_batches`, `update_interval`) are global overrides applied to every BS slot. Integer keys (`"1"`, `"8"`) define per-BS slots.

## Config file reference

The config file is optional. When provided, each integer BS-slot key must specify `candidate_steps`; all other keys fall back to defaults.

### Per-BS slot parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Key</th>
      <th>Default</th>
      <th>Meaning</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`candidate_steps`</td>
      <td>*required*</td>
      <td>Candidate `speculative_num_steps` tiers for this BS range. Must be a non-empty list of positive ints; a slot that omits it raises a config error</td>
    </tr>

    <tr>
      <td>`down_hysteresis`</td>
      <td>`-0.25`</td>
      <td>Extra margin before moving to a smaller step</td>
    </tr>

    <tr>
      <td>`up_hysteresis`</td>
      <td>`0.0`</td>
      <td>Extra margin before moving to a larger step</td>
    </tr>

    <tr>
      <td>`ceiling_coeff`</td>
      <td>`0` (disabled)</td>
      <td>EMA ceiling coefficient; set > 0 to cap steps proportionally to draft quality</td>
    </tr>
  </tbody>
</table>

### Global parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Key</th>
      <th>Default</th>
      <th>Meaning</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`ema_alpha`</td>
      <td>`0.2`</td>
      <td>EMA smoothing factor for accepted draft length</td>
    </tr>

    <tr>
      <td>`update_interval`</td>
      <td>`5`</td>
      <td>Recompute interval, in verify batches, after warmup</td>
    </tr>

    <tr>
      <td>`warmup_batches`</td>
      <td>`10`</td>
      <td>Number of verify batches to observe before switching</td>
    </tr>
  </tbody>
</table>

## Monitoring

You can inspect the active tier and acceptance metric via `/server_info`:

```bash theme={null}
curl -s http://127.0.0.1:30000/server_info | jq '.internal_states[0] | {speculative_num_steps, avg_spec_accept_length}'
```

* `speculative_num_steps` is the current active tier
* `avg_spec_accept_length` helps explain whether the server is likely to move up or down

## Tuning tips

* Start with the built-in default (conservative) — it is safe for all draft model qualities
* For strong draft models, use the aggressive config with ceiling rule
* Use fewer candidate steps if you want lower startup GPU memory overhead
* Increase `ema_alpha` to react faster, or lower it for more stability
* Increase `warmup_batches` or `update_interval` if tier switching is too noisy
* At high batch sizes, narrower ladders (e.g., `[1, 2]` or `[1]`) often outperform wide ones
* If your workload is already stable and one static setting is well tuned, adaptive mode may not help much

## Recommended configs

The built-in default is conservative — safe for all draft models but may under-speculate for strong ones. Save one of these as a JSON file and pass via `--speculative-adaptive-config`.

### Conservative (default) — for weak draft models

This is the built-in default: BS 8–31 allows `[1, 3]`, and BS≥32 locks to `step=1` to avoid wasted compute. Best for models like MiniMax-M2.5, DSV4.

```json theme={null}
{
  "1":  {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0},
  "8":  {"candidate_steps": [1, 3],    "up_hysteresis": 0.0, "down_hysteresis": 0.0,   "ceiling_coeff": 0},
  "32": {"candidate_steps": [1],       "up_hysteresis": 0.0, "down_hysteresis": 0.0,   "ceiling_coeff": 0}
}
```

### Aggressive — for strong or high-variance draft models

Uses wider ladders with ceiling rule to cap speculation at high BS. Best for models like GLM-4.7-FP8.

```json theme={null}
{
  "1":   {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 0},
  "8":   {"candidate_steps": [1, 3, 7], "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 3.0},
  "64":  {"candidate_steps": [1, 3],    "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.67},
  "128": {"candidate_steps": [1, 3],    "up_hysteresis": 0.0, "down_hysteresis": -0.25, "ceiling_coeff": 1.2}
}
```

### Custom per-model config

For the best performance, benchmark your specific model across batch sizes with different static `num_steps` values, then build a per-BS config that matches each range's optimal step. A well-tuned per-model config might outperform the generic presets above.


# Attention Backend
Source: https://docs.sglang.io/docs/advanced_features/attention_backend



SGLang supports a large variety of attention backends. Each of them has different pros and cons.
You can test them according to your needs.

> [!WARNING]
>
> Selecting an optimal attention backend is crucial for maximizing your performance. Different backends excel in various scenarios, so choose based on your model, hardware, and use case. Not all backends are supported on all platforms and model architectures.
>
> If you don't specify `--attention-backend`, SGLang makes a best effort to automatically select the most performant backend based on your hardware and model architecture.

## Support Matrix

The support matrix is split into two parts: MHA (standard attention) and MLA (multi-head latent attention). For an explanation of the key differences between MHA and MLA, please see the [SGLang documentation on DeepSeek MLA](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-4-mla-optimizations) and the original [DeepSeek MLA paper](https://arxiv.org/pdf/2405.04434).

### MHA Backends

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>**Backend**</th>
      <th>**Page Size > 1 (native)**</th>
      <th>**FP8 KV Cache**</th>
      <th>**FP4 KV Cache**</th>
      <th>**Spec topk=1**</th>
      <th>**Spec topk>1**</th>
      <th>**Sliding Window**</th>
      <th>**MultiModal**</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**FlashInfer**</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**FA3 (FlashAttention 3)**</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**FA4 (FlashAttention 4)**</td>
      <td>128</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**Triton**</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**Torch Native (SDPA)**</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**FlexAttention (PyTorch)**</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**TRTLLM MHA**</td>
      <td>16, 32 or 64</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**Dual Chunk FlashAttention**</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**HPC-Ops**</td>
      <td>64</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**AITER (ROCm)**</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**Wave (ROCm)**</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**Ascend (NPU)**</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**Intel XPU**</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**Intel AMX (CPU)**</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
    </tr>
  </tbody>
</table>

### MLA Backends

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>**Backend**</th>
      <th>**Native Page Sizes**</th>
      <th>**FP8 KV Cache**</th>
      <th>**FP4 KV Cache**</th>
      <th>**Chunked Prefix Cache**</th>
      <th>**Spec topk=1**</th>
      <th>**Spec topk>1**</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**FlashInfer MLA**</td>
      <td>1</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**FlashMLA**</td>
      <td>64</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**Cutlass MLA**</td>
      <td>128</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**TRTLLM MLA (Blackwell)**</td>
      <td>32 or 64</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**CuteDSL MLA (Blackwell)**</td>
      <td>32 or 64</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**FA3 (FlashAttention 3)**</td>
      <td>n/a</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>⚠️ (page\_size=1 only)</td>
    </tr>

    <tr>
      <td>**Triton**</td>
      <td>n/a</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>⚠️ (page\_size=1 only)</td>
    </tr>

    <tr>
      <td>**FA4**</td>
      <td>1</td>
      <td>❌</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**Ascend MLA (NPU)**</td>
      <td>128</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
    </tr>
  </tbody>
</table>

> [!NOTE]
>
> Multimodal attention is selected by `--mm-attention-backend`. The "MultiModal" column indicates whether a corresponding multimodal implementation exists for that backend family.

> [!NOTE]
>
> * DSA is specifically designed for [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). See the [DSA Attention Backend](#dsa-attention-backend) section and [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) for details.

> [!NOTE]
>
> For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-backend to run. Except for trtllm\_mha being incompatible with FA4, all other decode backends behave as shown in the table.

> [!TIP]
>
> Speculative decoding topk: `topk` is the number of draft tokens sampled per step from the draft model. `topk = 1` follows classic EAGLE; `topk > 1` explores multiple branches and requires backend support in both draft and verification paths.

> [!TIP]
>
> Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching). Note that higher page sizes generally improve attention kernel performance, so prefer `page_size > 1` when prefix cache reuse is not critical.

Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), CuteDSL MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128), HPC-Ops (64).

MLA page-size constraints:

* FlashInfer MLA: page\_size = 1.
* FlashMLA: page\_size = 64.
* Cutlass MLA: page\_size = 128.
* TRTLLM MLA: page\_size ∈ \{32, 64}.
* CuteDSL MLA: page\_size ∈ \{32, 64} (decode-only; prefill falls back to `trtllm_mla` when unset).

### GDN Attention Backends

GDN (Gated Delta Network) is a linear attention mechanism with O(n) complexity, used in hybrid models that alternate GDN linear attention layers with standard full attention layers. GDN is **not** selected via `--attention-backend`; it is automatically activated when the model architecture requires it (e.g., Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM).

The GDN linear attention layers have their own kernel backends, selected via `--linear-attn-backend` (default: `triton`). You can override the kernel per phase with `--linear-attn-decode-backend` and `--linear-attn-prefill-backend`.

On SM100/SM103 with CUDA 13+, SGLang automatically selects FlashInfer for GDN prefill when the per-phase override is unset, the base linear-attention backend is Triton, recurrent state is BF16, key/value head dimensions are 128, dynamic chunking and page-major KV layout are disabled, and `--chunked-prefill-size` is between 1 and 8192. Radix caching may be disabled or use `no_buffer`, `extra_buffer`, or `extra_buffer_lazy`; the extra-buffer paths use state checkpoints.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Backend</th>
      <th>Decode</th>
      <th>Prefill / Extend</th>
      <th>Spec Decoding (Target Verify)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Triton (CUDA)**</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**Triton (AMD/ROCm)**</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>**Triton (NPU)**</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**Triton (CPU)**</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**CuTe DSL (CUDA only)**</td>
      <td>✅</td>
      <td>❌</td>
      <td>❌</td>
    </tr>

    <tr>
      <td>**FlashInfer (CUDA, SM90/SM100/SM103)**</td>
      <td>✅</td>
      <td>✅</td>
      <td>✅ linear chain; tree falls back to Triton</td>
    </tr>
  </tbody>
</table>

> [!WARNING]
>
> GDN models are hybrid: the full-attention layers still require a standard `--attention-backend`. Platform constraints for the full-attention backend on hybrid GDN models:
>
> * **Blackwell SM120 (e.g., RTX PRO 6000 Blackwell)**: `triton` or `flashinfer` for prefill/full attention; `trtllm_mha` is supported for `--decode-attention-backend` only.
> * **Other Blackwell variants (including SM100 B200/GB200)**: `triton`, `trtllm_mha`, or `fa4` only.
> * **NPU (Ascend)**: `ascend` only.
> * **AMD (ROCm)**: `triton` recommended.
> * **Other CUDA (Hopper, Ampere, etc.)**: auto-selection works; no special constraints.

### DSA Attention Backend

DSA (DeepSeek Sparse Attention) is a native sparse attention mechanism used by [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). It is activated automatically when the model architecture requires it and is selected via `--attention-backend dsa` (deprecated alias: `nsa`).

Internally, the DSA backend dispatches to different sub-backends for prefill and decode phases. You can override these with `--dsa-prefill-backend` and `--dsa-decode-backend`:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Sub-backend</th>
      <th>Prefill</th>
      <th>Decode</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**flashmla\_sparse**</td>
      <td>✅</td>
      <td>✅</td>
      <td>Default prefill on Hopper and Blackwell (BF16)</td>
    </tr>

    <tr>
      <td>**flashmla\_sparse\_q8**</td>
      <td>✅</td>
      <td>❌</td>
      <td>Native FP8 (q8×kv8) sparse prefill on Hopper (SM90); BF16 or FP8 KV cache</td>
    </tr>

    <tr>
      <td>**triton\_sparse\_mla**</td>
      <td>✅</td>
      <td>❌</td>
      <td>Fused Triton sparse prefill that works on the post-TP head count instead of padding heads to 64/128; opt-in. CUDA SM90+ (tiles tuned on SM90 and SM120; other architectures run an untuned tile and log a warning at startup), at most 32 query heads per rank, not with `index_kpool`. `--dsa-triton-union 2|4` lets adjacent query tokens share one gathered index set (exact; `num_q_heads * union` must be a power of two in \[16, 32])</td>
    </tr>

    <tr>
      <td>**flashmla\_kv**</td>
      <td>✅</td>
      <td>✅</td>
      <td>Default for FP8 on Hopper (prefill + decode)</td>
    </tr>

    <tr>
      <td>**flashmla\_auto**</td>
      <td>✅</td>
      <td>❌</td>
      <td>Picks flashmla\_sparse or flashmla\_kv by KV cache dtype</td>
    </tr>

    <tr>
      <td>**fa3**</td>
      <td>✅</td>
      <td>✅</td>
      <td>Default decode on Hopper (BF16)</td>
    </tr>

    <tr>
      <td>**trtllm**</td>
      <td>✅</td>
      <td>✅</td>
      <td>Default decode on Blackwell (BF16); default for FP8 on Blackwell (prefill + decode)</td>
    </tr>

    <tr>
      <td>**tilelang**</td>
      <td>✅</td>
      <td>✅</td>
      <td>Default on AMD (ROCm)</td>
    </tr>

    <tr>
      <td>**aiter**</td>
      <td>✅</td>
      <td>✅</td>
      <td>AMD-specific kernel library (requires aiter package)</td>
    </tr>
  </tbody>
</table>

#### Gemma 4 on `aiter` (ROCm)

Gemma 4's hybrid attention mixes sliding-window layers (`head_dim` 256) with
full-attention layers (`head_dim` 512) and shares KV across layers. The `aiter`
backend supports this and is allow-listed for Gemma 4, so you can serve it with
`--attention-backend aiter` on MI300X/MI350X. Under the hood `aiter` routes the
`head_dim`-512 prefill through the Triton `unified_attention` kernel (the CK
batch-prefill path caps `head_dim` at 256), honors each layer's softmax scale,
and resolves the sliding-window KV pool for speculative-decode draft workers.
Non–sliding-window models (`head_dim` ≤ 256) are unaffected and keep using the
CK path.

> [!WARNING]
>
> For Gemma 4 on `aiter`, use bf16 KV cache (`--kv-cache-dtype auto`). Do **not**
> use `--kv-cache-dtype fp8_e4m3`: batched decode currently returns incorrect
> output under fp8 KV on this path (a pre-existing aiter fp8-decode issue, tracked
> separately). bf16 KV is correct at every batch size.

For deployment examples, see the [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2).

### Hybrid attention (different backends for prefill vs decode) (Experimental)

> [!WARNING]
>
> Hybrid attention is an experimental feature.

You can mix-and-match attention backends for prefill and decode. This is useful when one backend excels at prefill and another excels at decode. For the implementation details, please see `python/sglang/srt/layers/attention/hybrid_attn_backend.py`.

```bash Command theme={null}
# Example: Prefill with FA4, Decode with TRTLLM MLA (Blackwell)
python3 -m sglang.launch_server \
  --model-path nvidia/DeepSeek-R1-FP4 \
  --tp 8 \
  --attention-backend trtllm_mla \
  --moe-runner-backend flashinfer_trtllm \
  --quantization modelopt_fp4 \
  --prefill-attention-backend fa4
```

#### Speculative decoding with hybrid attention

Hybrid attention also works with speculative decoding. The backend used for draft decoding and target verification depends on `--speculative-attention-mode`:

* `--speculative-attention-mode decode` (recommended): draft/verify use the decode backend.
* `--speculative-attention-mode prefill` (default): draft/verify use the prefill backend.

Constraints when combining hybrid attention with speculative decoding:

* If any attention backend is `trtllm_mha`, speculative decoding supports only `--speculative-eagle-topk 1`.
* For paged MHA backends with `--page-size > 1` and `--speculative-eagle-topk > 1`, only `flashinfer` is supported.
* CUDA Graph: the decode backend is always captured; the prefill backend is captured only when `--speculative-attention-mode prefill`.

> [!TIP]
>
> If you set only one of `--prefill-attention-backend` or `--decode-attention-backend`, the unspecified phase inherits `--attention-backend`.
> If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase.

## Attention Backend Selection Guide (CUDA)

If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture.

### Automatic Selection Logic

**1. MHA Models (e.g., Llama, Qwen)**

* **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported.
* **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`.
* **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`.

**2. MLA Models (e.g., DeepSeek V3)**

* **Hopper**: Defaults to `fa3` (requires CUDA 12.3+).
* **Blackwell**: Defaults to `flashinfer`; `trtllm_mla` is auto-selected for DeepSeek V3 models specifically.
* **Other Architectures**: Defaults to `triton`.

## User Guide

### Launch Command for Different Attention Backends

* FlashInfer (Default for Non-Hopper Machines, e.g., A100, A40)

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend flashinfer
python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-V3 \
  --attention-backend flashinfer \
  --trust-remote-code
```

* FlashAttention 3 (Default for Hopper Machines, e.g., H100, H200, H20)

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend fa3
python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-V3 \
  --trust-remote-code \
  --attention-backend fa3
```

* Triton

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend triton
python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-V3 \
  --attention-backend triton \
  --trust-remote-code
```

* FlashMLA

```bash Command theme={null}
python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-R1 \
  --attention-backend flashmla \
  --trust-remote-code
python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-R1 \
  --attention-backend flashmla \
  --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code
```

* TRTLLM MLA (Optimized for Blackwell Architecture, e.g., B200)

```bash Command theme={null}
python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-R1 \
  --attention-backend trtllm_mla \
  --trust-remote-code
```

* TRTLLM MLA with FP8 KV Cache (Higher concurrency, lower memory footprint)

```bash Command theme={null}
python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-R1 \
  --attention-backend trtllm_mla \
  --kv-cache-dtype fp8_e4m3 \
  --trust-remote-code
```

* TRTLLM MHA (Optimized for Blackwell Architecture, e.g., B200)

```bash Command theme={null}
python3 -m sglang.launch_server \
  --tp 4 \
  --model Qwen/Qwen3.5-35B-A3B-FP8 \
  --attention-backend trtllm_mha \
  --trust-remote-code
```

* TRTLLM MHA (XQA backend) (Optimized for SM90 and SM120, e.g., H20, H200, 5090)
  Note that TRTLLM XQA backend only works well for pagesize 64.

```bash Command theme={null}
python3 -m sglang.launch_server \
  --tp 4 \
  --model Qwen/Qwen3.5-35B-A3B-FP8 \
  --decode-attention-backend trtllm_mha \
  --trust-remote-code
```

* HPC-Ops (MHA kernels from [HPC-Ops](https://github.com/Tencent/hpc-ops) by the Tencent Hunyuan AI Infra team; Hopper (SM90) only, requires installing the `hpc` package from source, page size 64, bf16 or fp8\_e4m3 KV cache, head\_dim 128, q/kv head group 4 or 8)

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
  --attention-backend hpc_ops \
  --page-size 64 \
  --trust-remote-code

# FP8 models should also set --kv-cache-dtype fp8_e4m3 to run the FP8 attention
# kernels. This enables the fused QKNorm+RoPE+FP8-quant+StoreKV op, which is
# currently wired for Hunyuan V3 and requires per-rank (q_heads, kv_heads) of
# (64, 8) or (8, 1), e.g. --tp 1 or --tp 8 for Hy3.
python3 -m sglang.launch_server \
  --tp 8 \
  --model tencent/Hy3-FP8 \
  --attention-backend hpc_ops \
  --kv-cache-dtype fp8_e4m3 \
  --page-size 64 \
  --trust-remote-code
```

* FlashAttention 4 (MHA & MLA)

```bash Command theme={null}
# FA4 for both prefill and decode on SM90/SM100
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
  --attention-backend fa4 \
  --page-size 128 \
  --trust-remote-code

python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-R1 \
  --prefill-attention-backend fa4 \
  --trust-remote-code
```

* Ascend

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend ascend
```

* Intel XPU

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend intel_xpu
```

* Wave

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend wave
```

* FlexAttention

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend flex_attention
```

* Torch Native

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --attention-backend torch_native
```

## Steps to add a new attention backend

To add a new attention backend, you can learn from the existing backends
(`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`)
and follow the steps below.

> [!NOTE]
>
> Linear attention kernel backends (GDN, KDA) follow a different pattern. They implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/` and are dispatched by `GDNKernelDispatcher` / `KDAKernelDispatcher` rather than registered via `@register_attention_backend`.

1. Run without cuda graph. Support the two forward functions
   * forward\_extend
     * Will be used for prefill, prefill with KV cache, and target verification
     * It will be called once per layer
   * forward\_decode
     * Will be used for normal decode, and draft decode
     * It will be called once per layer
   * init\_forward\_metadata
     * Initialize the class and common metadata shared by all layers
     * Call the plan function for optimizations like split\_kv
     * It will be called once per forward
2. Run with cuda graph. It has two phases (capture and replay) and you need to implement three functions
   * init\_cuda\_graph\_state
     * It will be called once during life time
     * Create all common shared buffers
   * init\_forward\_metadata\_capture\_cuda\_graph
     * It will be called before capturing a cuda graph
     * It is similar to init\_forward\_metadata but write the medatada to some pre-defined buffers
   * init\_forward\_metadata\_replay\_cuda\_graph
     * It will be called before replaying a cuda graph
     * This function is in the critical path and needs to be fast


# Breakable CUDA Graph
Source: https://docs.sglang.io/docs/advanced_features/breakable_cuda_graph



## Motivation

Standard CUDA graphs capture an entire forward pass as a single, opaque graph. This is great for performance, but creates two problems:

1. **Debugging is hard.** When something goes wrong inside a captured graph (wrong outputs, numerical mismatches, crashes), there is no way to step through the operations or insert print statements because the graph replays as a monolithic unit.

2. **Some ops are incompatible.** Certain operations — dynamic control flow, host-device synchronization, JIT compilation, or ops that change behavior across iterations — cannot be captured into a CUDA graph at all. Today, the only workaround is to disable CUDA graphs entirely, which sacrifices the kernel launch overhead savings for the rest of the model.

**Breakable CUDA Graph** solves both problems by allowing graph breaks to be inserted at specific points. The computation is split into multiple captured graph segments with eager (non-graph) execution in between. This preserves most of the CUDA graph performance benefit while allowing targeted operations to run outside the graph.

## Usage

### Debug Mode: Run Everything Eagerly

The simplest use case is debugging. The `--debug-cuda-graph` flag wraps the entire decode forward pass in a graph break, so every operation runs eagerly while still going through the full CUDA graph capture/replay code path. This lets you debug CUDA graph issues without changing model code.

```bash theme={null}
python -m sglang.launch_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --debug-cuda-graph
```

This mode is intended for debugging only — it eliminates the performance benefit of CUDA graphs since every op runs eagerly.

### Selective Graph Breaks in Model Code

For production use, you can mark specific functions as "non-graphable" using the `@eager_on_graph` decorator. During CUDA graph capture, these functions run eagerly between captured graph segments. Outside of capture, they behave normally.

```python theme={null}
from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import eager_on_graph

@eager_on_graph
def my_dynamic_op(x):
    # This op is incompatible with CUDA graph capture
    return some_dynamic_operation(x)
```

You can also insert a bare graph break (no computation) using the `break_graph()` helper:

```python theme={null}
from sglang.srt.model_executor.runner_backend_utils.breakable_cuda_graph import break_graph

def forward(self, x):
    x = self.layer1(x)
    break_graph()  # force a segment split here
    x = self.layer2(x)
    return x
```

To enable breakable CUDA graph at the environment level (without debug mode), set the environment variable:

```bash theme={null}
export SGLANG_USE_BREAKABLE_CUDA_GRAPH=1
python -m sglang.launch_server \
    --model meta-llama/Llama-3.1-8B-Instruct
```

### Server Args

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--debug-cuda-graph`</td>
      <td>`False`</td>
      <td>Enable debug/eager mode. Wraps the entire forward pass in a graph break so every op runs eagerly through the capture/replay path.</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_BREAKABLE_CUDA_GRAPH`</td>
      <td>`0`</td>
      <td>Environment variable. Enables breakable CUDA graph without debug mode. Required for `@eager_on_graph` decorators to take effect.</td>
    </tr>
  </tbody>
</table>

## How It Works

### Capture

Breakable CUDA graph extends PyTorch's `torch.cuda.CUDAGraph` by splitting a single capture into multiple segments separated by graph breaks.

During capture, the flow is:

```
Begin capture (segment 1)
  ... graphable ops ...
  @eager_on_graph function encountered:
    1. End current capture segment
    2. Run the function eagerly (allocates output tensors)
    3. Record the function for later replay
    4. Begin new capture segment
  ... more graphable ops ...
End capture (segment N)
```

Each segment is independently instantiated as a CUDA graph executable. The non-graph functions and their argument references are stored for replay.

### Replay

During replay:

```
For each segment i:
  1. Launch CUDA graph segment i
  2. Run the recorded non-graph function i eagerly
Launch final CUDA graph segment
```

The non-graph functions are re-invoked with the same tensor references as capture time. Since these references point to the CUDA graph's static input/output buffers, they see updated values on each replay.

### Output Writeback

When a non-graph function produces output during replay, the result must be written back into the same tensor buffers that downstream graph segments reference. The mechanism handles:

* **Plain tensors**: In-place `copy_()` into the original buffer.
* **Structured outputs** (dataclasses, objects with tensor attributes): Tensor fields are copied in-place; non-tensor fields are replaced.
* **Dicts of tensors**: Tensor values are copied in-place; non-tensor values are replaced.

### Stream Fork/Join Tracking

Some models fork work onto secondary CUDA streams (e.g., for overlapped computation). Breakable CUDA graph hooks `torch.cuda.Stream.wait_stream` to track which streams are forked from the capture stream. When a graph break occurs, all forked streams are automatically joined back before ending the capture segment, and re-forked after beginning the next segment.

## Compatibility

* **CUDA and ROCm/HIP.** Breakable CUDA graph runs on both NVIDIA and AMD GPUs. Other platforms (NPU, CPU, MPS, XPU) are unsupported; there `--debug-cuda-graph` is automatically disabled with a warning.
* **Requires `cuda-python` on NVIDIA.** Stream-capture-status queries use the CUDA runtime via `cuda.bindings` (`pip install cuda-python`); the portable `torch.cuda.is_current_stream_capturing()` has proven unreliable on CUDA. On ROCm/HIP — where `cuda-python` is unavailable — the portable `torch.cuda` API (which maps to the HIP runtime) is used instead.
* **Not compatible with memory saver mode.** Cannot be used together with `SGLANG_MEMORY_SAVER_CUDA_GRAPH`.

## Performance

When no graph breaks are inserted, breakable CUDA graph has minimal overhead compared to standard CUDA graph — the capture/replay path is nearly identical.

Each graph break adds:

* One `cudaGraphLaunch` call (to replay the segment before the break)
* One eager Python function call
* One `cudaStreamBeginCapture` / `cudaStreamEndCapture` pair during capture

For typical use cases with a small number of graph breaks, the overhead is negligible compared to the saved kernel launch overhead from the captured segments.

## Code Reference

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>File</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py`</td>
      <td>Core implementation: `eager_on_graph`, `BreakableCUDAGraph`, `BreakableCUDAGraphCapture`</td>
    </tr>

    <tr>
      <td>`python/sglang/srt/model_executor/runner_backend_utils/breakable_cuda_graph/cuda_utils.py`</td>
      <td>CUDA runtime binding utilities (NVIDIA stream-capture queries)</td>
    </tr>

    <tr>
      <td>`python/sglang/srt/model_executor/runner_backend/breakable_cuda_graph_backend.py`</td>
      <td>Integration with CUDA graph runner backends</td>
    </tr>

    <tr>
      <td>`python/sglang/srt/server_args.py`</td>
      <td>`--debug-cuda-graph` flag and environment variable handling</td>
    </tr>

    <tr>
      <td>`python/sglang/srt/environ.py`</td>
      <td>`SGLANG_USE_BREAKABLE_CUDA_GRAPH` environment variable definition</td>
    </tr>
  </tbody>
</table>


# Checkpoint Engine Integration
Source: https://docs.sglang.io/docs/advanced_features/checkpoint_engine



The SGLang checkpoint engine integration provides an efficient way to load model weights using a distributed checkpoint loading system. This feature significantly reduces model loading time, especially for large models and multi-node setups, by parallelizing the weight loading process across multiple processes and nodes.

## Overview

The checkpoint engine integration allows SGLang to:

* Load model weights in parallel using multiple processes
* Distribute weight loading across multiple nodes to increase effective disk bandwidth
* Overlap weight loading with other initialization tasks like CUDA graph capture
* Support both single-node and multi-node deployments

## Installation

First, install the checkpoint engine package:

```bash Command theme={null}
pip install 'checkpoint-engine[p2p]'
```

## Architecture

The system consists of two main components:

1. **SGLang Server**: Runs with `--checkpoint-engine-wait-weights-before-ready` flag to wait for weights before becoming ready
2. **Checkpoint Engine Workers**: Separate processes (managed by torchrun) that load and distribute model weights

The checkpoint engine uses a parameter server architecture with support for:

* **Broadcast mode**: Weights are broadcast from loading processes to inference processes
* **P2P mode**: Direct peer-to-peer weight transfer between processes
* **All mode**: Combination of both broadcast and P2P methods

## Usage Examples

### Single Node Setup

**Terminal 1 - Launch SGLang Server:**

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --tp 8 \
    --load-format dummy \
    --checkpoint-engine-wait-weights-before-ready
```

**Terminal 2 - Run Checkpoint Engine:**

Using sglang entrypoint:

```bash Command theme={null}
python -m sglang.srt.checkpoint_engine.update \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 8
```

Using torchrun directly:

```bash Command theme={null}
torchrun --nproc-per-node 8 \
    examples/checkpoint_engine/update.py \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 8
```

### Multi-Node Setup (2 Nodes)

**Node 0:**

Launch SGLang server:

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --tp 8 \
    --load-format dummy \
    --checkpoint-engine-wait-weights-before-ready \
    --host [IP]
```

Run checkpoint engine:

Using sglang entrypoint (recommended):

```bash Command theme={null}
python -m sglang.srt.checkpoint_engine.update \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 8
```

Using torchrun directly:

```bash Command theme={null}
torchrun --nproc-per-node 8 \
    --nnodes 2 \
    --node-rank 0 \
    --master-addr [IP] \
    --master-port 29500 \
    examples/checkpoint_engine/update.py \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 8
```

**Node 1:**

Launch SGLang server:

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --tp 8 \
    --load-format dummy \
    --checkpoint-engine-wait-weights-before-ready \
    --host [IP]
```

Run checkpoint engine:

Using sglang entrypoint (recommended):

```bash Command theme={null}
python -m sglang.srt.checkpoint_engine.update \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 8
```

Using torchrun directly:

```bash Command theme={null}
torchrun --nproc-per-node 8 \
    --nnodes 2 \
    --node-rank 1 \
    --master-addr [IP] \
    --master-port 29500 \
    examples/checkpoint_engine/update.py \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 8
```

### Multi-Node Setup with Tensor Parallelism (TP=16)

**Node 0:**

Launch SGLang server:

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --tp 8 \
    --load-format dummy \
    --checkpoint-engine-wait-weights-before-ready \
    --host [IP] \
    --dist-init-addr [IP]:9120 \
    --nnodes 2 \
    --node-rank 0
```

Run checkpoint engine:

Using sglang entrypoint (recommended):

```bash Command theme={null}
python -m sglang.srt.checkpoint_engine.update \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 16
```

Using torchrun directly:

```bash Command theme={null}
torchrun --nproc-per-node 8 \
    --nnodes 2 \
    --node-rank 0 \
    --master-addr [IP] \
    --master-port 29500 \
    examples/checkpoint_engine/update.py \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 16
```

**Node 1:**

Launch SGLang server:

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --tp 8 \
    --load-format dummy \
    --checkpoint-engine-wait-weights-before-ready \
    --host [IP] \
    --dist-init-addr [IP]:9120 \
    --nnodes 2 \
    --node-rank 1
```

Run checkpoint engine:

Using sglang entrypoint (recommended):

```bash Command theme={null}
python -m sglang.srt.checkpoint_engine.update \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 16
```

Using torchrun directly:

```bash Command theme={null}
torchrun --nproc-per-node 8 \
    --nnodes 2 \
    --node-rank 1 \
    --master-addr [IP] \
    --master-port 29500 \
    examples/checkpoint_engine/update.py \
    --update-method broadcast \
    --checkpoint-path /path/to/Qwen/Qwen3-8B/ \
    --inference-parallel-size 16
```

## Configuration Options

### SGLang Server Options

* `--load-format dummy`: Use dummy format for initial loading (allows overlapping with other tasks)
* `--checkpoint-engine-wait-weights-before-ready`: Wait for checkpoint engine to provide weights before becoming ready
* `--host`: Host address for multi-node setups
* `--dist-init-addr`: Distributed initialization address for tensor parallelism

### Checkpoint Engine Options

* `--update-method`: Weight update method (`broadcast`, `p2p`, or `all`)
* `--checkpoint-path`: Path to model checkpoint directory
* `--inference-parallel-size`: Number of inference parallel processes
* `--endpoint`: SGLang server endpoint (default: `http://localhost:19730`)
* `--checkpoint-name`: Name for the checkpoint (default: `my-checkpoint-iter-0`)
* `--save-metas-file`: File to save checkpoint metadata
* `--load-metas-file`: File to load checkpoint metadata from
* `--uds`: Unix domain socket path for communication
* `--weight-version`: Version identifier for weights

## Performance Benefits

The checkpoint engine provides significant time savings in two main aspects:

1. **Multi-node Loading**: Each node only loads a portion of weights from disk, effectively increasing disk bandwidth. More participating nodes provide greater acceleration. Preliminary tests show 20-second acceleration when loading DeepSeek-R1 on H20-3e with two nodes.

2. **Single Process Optimization**: Using dummy format allows overlapping disk-to-CPU transfer with CUDA graph capture and other initialization tasks, providing additional time savings.

## Troubleshooting

* Ensure checkpoint engine package is installed: `pip install 'checkpoint-engine[p2p]'`
* Verify network connectivity between nodes in multi-node setups
* Check that the checkpoint path contains valid model files
* Monitor logs for connection errors between SGLang server and checkpoint engine
* Use `--sleep-time` parameter to add delays if needed for debugging

## References

* [Checkpoint Engine Repository](https://github.com/MoonshotAI/checkpoint-engine)


# Cuda Graph for Multi-Modal Encoder in SGLang
Source: https://docs.sglang.io/docs/advanced_features/cuda_graph_for_multi_modal_encoder



## Motivation

In multimodal reasoning services, the visual encoder (ViT / Vision Transformer) typically has a few characteristic traits:

Many layers, fragmented operators: Each layer includes LN, QKV projections, attention, MLP, residual connections, etc., resulting in extremely frequent kernel launches.

Server-side “small batch / low latency” is common: The batch size is very small (sometimes it looks like 1 after “flattening” the batch), so kernel launch overhead accounts for a large portion of end-to-end latency.

Input token count (number of patches) varies frequently: Different image/video resolutions and different batch composition lead to different sequence lengths
S — and this is precisely the biggest obstacle for CUDA Graph (unstable shapes).

The value of CUDA Graph: It captures a long sequence of GPU kernels with fixed shapes and fixed memory addresses into a graph; later, for the same shapes, it can replay the graph directly, dramatically reducing launch overhead and making GPU scheduling more compact.

This led us to seek a CUDA Graph enabled feature for ViT in order to improve ViT performance.

## Design and Restrictions

The new CUDA Graph enabled ViT logic is built on ViTCudaGraphRunner. This runner captures the "blocks + merger + deepstack merger (optional)" part of a vision transformer into a CUDA graph and replays it for identical shapes. See the following design consideration and restrictions for more details.

### Dynamic inputs to fit static constraints of CUDA Graph

Variable sequence length S is very common in ViT. While CUDA Graph requires fixed shapes. The solution is to build a graph cache by S(e.g., graph\_key = S). The first time create a new S, and then capture a graph; afterwards, replay it.

If there are many distinct S values, we need to increase VRAM usage which is graph-private memory pools for many graphs.

### Stable addresses

Everything "parameter-like" becomes a static buffer:

* block\_input / block\_ws / block\_output
* cu\_full\_len / cu\_window\_len and their kk variants
* sin\_cos\_ws

In this way to solve the underlying requirement: during replay, not allowed to swap tensors, can only modify tensor contents.

### Attention backend arguments

Attention backend arguments are fixed inside the graph:

TritonAttn expects \[cu\_seqlens, cu\_seqlens\_kk, max\_len]
FA3 expects \[cu\_seqlens, max\_len]

max\_len is frozen as an int constant.
cu\_seqlens is cached into a dict during create\_graph(), and its contents are not updated during subsequent replays.

For the same graph\_key = S, you not only require the input shape to match, but also require the segmentation pattern in cu\_seqlens (and window seqlens) to be identical. Otherwise, attention will segment the sequence incorrectly.

### Rotary buffer management

The feature reallocates a larger sin\_cos\_ws when seq\_len increases.
The max\_content\_len is used to make sure the maximum size of the allocated rotary buffer.

## Command Example

You can enable CUDA Graph for ViT by setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1`, for example:

```shell Command theme={null}
SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
python3 -m sglang.launch_server \
  --model Qwen/Qwen3-VL-8B-Instruct
```

To run the ViT graph together with Piecewise CUDA Graph over the decoder's
prefill, ask for the prefill backend explicitly. Qwen3-VL is on neither
multimodal allowlist, so the default backend resolves prefill to `disabled` for
it; naming a backend locks the choice and skips that rule:

```shell Command theme={null}
SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \
python3 -m sglang.launch_server \
  --model Qwen/Qwen3-VL-8B-Instruct \
  --cuda-graph-backend-prefill breakable \
  --cuda-graph-max-bs-prefill 4096 \
  --cuda-graph-tc-compiler eager
```

## Known supported models

* Qwen2.5-VL ([https://github.com/sgl-project/sglang/pull/14422](https://github.com/sgl-project/sglang/pull/14422))
* Qwen3-VL ([https://github.com/sgl-project/sglang/pull/15320](https://github.com/sgl-project/sglang/pull/15320))


# Decode Context Parallelism
Source: https://docs.sglang.io/docs/advanced_features/dcp



Decode context parallelism (DCP) stripes a request's MLA KV cache across ranks in an existing tensor-parallel group. With DCP size `c`, rank `r` owns position `p` when `p mod c = r`, so each rank stores and reads roughly `1/c` of the cache.

Local attention then sees only part of the context. The MLA path returns a partial output and log-sum-exp (LSE), exchanges both in one packed all-to-all, and merges them exactly. DCP complements TP and [attention data parallelism (DPA)](/docs/advanced_features/dp_dpa_smg_guide) rather than replacing them. The primary path is absorbed MLA decode and static target verification. `dcp_size=1` retains existing non-DCP behavior.

On Kimi K3, DCP applies only to MLA layers. Request-indexed KDA state is unchanged, so DCP grows long-context KV capacity but does not raise the KDA concurrency ceiling.

## Enable DCP

| Setting | Behavior |
| - | - |
| `--dcp-size N` | Enables DCP and widens virtual capacity and page size by `N`. Alias: `--decode-context-parallel-size`. |
| `--dcp-comm-backend` | Selects the decode partial-output merge (MLA on CUDA/ROCm, MHA on Intel XPU): `ag_rs`, `a2a`, or `fi_a2a`. Unset resolves to `fi_a2a` where the DCP group shares an MNNVL domain on Blackwell, otherwise `a2a` on CUDA/ROCm/XPU. |
| `--dcp-replicate-q-proj` | Removes the query all-gather for supported MLA weights. Use `--no-dcp-replicate-q-proj` to disable a model-specific default. |
| `--attn-dp-size N` | Composes only when DCP groups nest inside attention TP. |

A DCP group must fit inside one attention-TP group and one attention-DP replica:

```text Topology theme={null}
attn_tp_size = tp_size / attn_dp_size
attn_tp_size % dcp_size == 0
```

For example, `TP=64, attention DP=4, DCP=16` creates four valid 16-rank attention replicas. `DCP=32` would cross replica boundaries and is invalid.

> [!WARNING]
>
> Startup currently checks only `tp_size % dcp_size == 0`, not the stronger condition above. Until that check is fixed, DPA+DCP deployments must enforce containment when choosing the topology.

MLA decode on DeepSeek-V3.1 with DCP 8 inside TP 8:

```bash Command theme={null}
sglang serve \
  --model-path deepseek-ai/DeepSeek-V3.1 \
  --trust-remote-code \
  --tp-size 8 \
  --dcp-size 8 \
  --host 0.0.0.0 \
  --port 30000
```

Kimi K3 on 32 GPUs with DPA and DCP 8. The model override enables replicated Q by default; the communication backend resolves generically to `fi_a2a` on MNNVL systems, otherwise `a2a`:

```bash Command theme={null}
sglang serve \
  --trust-remote-code \
  --model-path moonshotai/Kimi-K3 \
  --tp-size 32 \
  --ep-size 32 \
  --attn-dp-size 4 \
  --dcp-size 8 \
  --host 0.0.0.0 \
  --port 30000
```

See the [Kimi K3 cookbook](/cookbook/autoregressive/Moonshotai/Kimi-K3) for large-scale presets that combine DCP with DPA and expert parallelism.

## Why decode context parallelism?

Absorbed MLA shares one latent KV representation across all query heads. TP can split the heads and weights, but not that cache, so every attention-TP rank stores and reads the same context.

```mermaid theme={null}
flowchart LR
    subgraph TP["TP only: every rank stores the full MLA context"]
        direction TB
        T0["Rank 0<br/>tokens 0 1 2 3 4 5 6 7"]
        T1["Rank 1<br/>tokens 0 1 2 3 4 5 6 7"]
        T2["Rank 2<br/>tokens 0 1 2 3 4 5 6 7"]
        T3["Rank 3<br/>tokens 0 1 2 3 4 5 6 7"]
    end

    TP -- "stripe by position" --> DCP

    subgraph DCP["DCP4: one interleaved shard per rank"]
        direction TB
        D0["Rank 0<br/>tokens 0 4"]
        D1["Rank 1<br/>tokens 1 5"]
        D2["Rank 2<br/>tokens 2 6"]
        D3["Rank 3<br/>tokens 3 7"]
    end
```

DCP stripes that cache round-robin by logical token position, keeps a shared virtual index on every rank, runs attention against the owner-local shard, and merges partial outputs by LSE into the usual TP-local head layout.

## Architecture

### Virtual KV layout

Let each rank have physical token capacity `C` and physical page size `P`. DCP exposes a shared virtual allocator:

```text Layout theme={null}
virtual capacity  = C * c
virtual page size = P * c
owner(v)          = v mod c
physical(v)       = floor(v / c)
```

Every rank sees the same request-to-token map. On write, it keeps its positions and compacts them into the local physical pool. One widened virtual page maps to one physical page on each rank, so the layout stays balanced to within one token as a sequence grows.

### MLA decode dataflow

```mermaid theme={null}
flowchart LR
    H["Hidden states"] --> Q["Full DCP-group query heads"]

    Q --> A0["Rank 0 local attention<br/>partial O0 + LSE0"]
    Q --> A1["Rank 1 local attention<br/>partial O1 + LSE1"]
    Q --> AN["Rank c-1 local attention<br/>partial O + LSE"]

    A0 --> A2A["Packed all-to-all"]
    A1 --> A2A
    AN --> A2A
    A2A --> MERGE["Exact LSE merge"]
    MERGE --> OUT["TP-local head shard"]
```

For context partition `r`, the kernel returns locally normalized output `o_r` and `lse_r`. The global result is `lse = logsumexp_r(lse_r)` and `o = sum_r(exp(lse_r - lse) * o_r)`. This is exact aside from floating-point reduction order. The merge uses the attention backend's LSE base, either base-e or base-2.

### Query-projection replication

Normally each layer all-gathers its absorbed and rotary query components. `--dcp-replicate-q-proj` gathers the query-projection and `w_kc` weights once at startup and computes the full DCP-group query locally. The trade is more weight memory and GEMM work for one fewer collective per layer. Only unquantized BF16/FP16 weights use this path; other layers fall back to the query all-gather.

### Communication backends

| Backend | DCP collectives per MLA layer | Notes |
| - | - | - |
| `ag_rs` | Query AG + LSE AG + FP32 output RS | Fallback for platforms without the a2a path; opt in with `--dcp-comm-backend ag_rs` |
| `a2a`, gathered Q | Query AG + packed NCCL A2A | Two collectives |
| `a2a`, replicated Q | One packed NCCL A2A | Canonical low-latency path |
| `fi_a2a`, replicated Q | One FlashInfer MNNVL A2A | Requires Blackwell and a DCP group inside one MNNVL domain (GB200/GB300 fabric, or a single node) |

When `--dcp-comm-backend` is unset, every DCP launch resolves it the same way: `fi_a2a` where the DCP group shares one MNNVL domain on Blackwell, otherwise `a2a` on CUDA/ROCm/XPU, and `ag_rs` only on platforms without the a2a path. Query replication stays model-resolved: Kimi K3 enables it by default on the a2a family, and its decode backend is `cutedsl_mla`.

DCP adds a context-independent decode collective while cutting context-dependent KV storage and reads by about `c`. Extend is outside that decode cost model: it may gather cached prefix shards, restore token order, and append new tokens, and that work grows with prefix length.

## Compositions

### DCP × speculative decoding

The draft KV cache is replicated: every DCP rank stores the full draft context. DCP only stripes the target MLA KV, so it saves target cache, not draft cache.

Target verify and decode use the DCP-aware MLA kernel `cutedsl_mla`. It:

1. Builds a rank-local page table from the cyclic token map (`owner(p) = p mod c`).
2. Runs attention against that shard, passing `cp_world`, `cp_rank`, and the global KV lengths so causality still sees the full sequence.
3. Returns a rank-local `(output, LSE)` that the usual DCP all-to-all merge combines.

Draft steps use the same backend against the replicated draft cache, so they skip DCP metadata. Kimi K3 DCP selects `cutedsl_mla` automatically.

### DCP × PD disaggregation

A dense prefill page cannot be copied directly into striped decode storage. The PD sender uses the decode rank's DCP metadata to select and pack the right rows. See [PD disaggregation](/docs/advanced_features/pd_disaggregation) for the transfer engines.

* Equal DCP sizes require matching DCP ranks and use the existing page path.
* DCP1 prefill to DCP decode uses token relayout for MLA and hybrid-MLA pools.
* All other DCP-size transitions are rejected.

The plan includes the exact token count, so stale rows do not leak out of a partial final page. Mooncake and NIXL use the same plan. KDA state keeps its attention-TP mapping and bypasses DCP filtering.

This composition requires Mooncake or NIXL, matching physical page size and KV dtype, and prefill attention CP `1`. Decode radix cache and HiCache are not supported here.

### DCP × HiCache L2

HiCache keeps one widened logical page space across L1 and L2. The controller sees `P * c` slots, while each GPU and host buffer stores only its `P` local rows. See [HiCache](/docs/advanced_features/hicache) for the cache hierarchy.

Before H2D or D2H, both index lists keep positions owned by this rank and map `i` to `floor(i / c)`. Transfers cover whole widened pages, and every rank moves its shard independently, with no DCP collective. For Kimi K3 this applies to MLA KV; KDA/Mamba state uses its own request-indexed host pool.

The current scope is MLA L1/L2. L3, LMCache, HiSparse, non-MLA KV host pools, speculative decoding, and PD decode are not supported in this combination.

### DCP × DSA (GLM-5.2, GLM-5.3)

DeepSeek Sparse Attention attends over a top-k slice of the context, so DCP keeps the target's latent KV striped by position while the **index-K cache stays replicated** on every rank: each rank scores the whole sequence, takes the global top-k, and attends over the selected slots it owns. Decode, target verify and extend all follow the same path, so the prefix KV is never gathered.

1. The index-K cache is addressed by the widened virtual slot and costs `c` times its usual share of each rank's KV budget (about 6% at DCP2 and 17% at DCP4 on GLM-5.3).
2. Each layer maps the global top-k to rank-local rows with one Triton kernel; layers that share the previous layer's top-k reuse the mapping.
3. The TRT-LLM sparse kernel returns the per-head LSE and the usual DCP merge combines the partial outputs.
4. The MTP draft (nextn) layer keeps a replicated KV cache over the widened slots and attends without the DCP group, so MTP index share feeds the draft directly.

Requirements: CUDA SM100/SM103 with the `trtllm` DSA backends (selected automatically), an `fp8_e4m3` or `bf16` KV cache, `attn_tp_size % dcp_size == 0`, and, with speculative decoding, the checkpoint's own MTP draft as a single EAGLE chain (`--speculative-eagle-topk 1`, draft tokens equal to steps + 1). HiSparse, prefill CP, HiCache, PD disaggregation, unified memory and `index_kpool > 1` are not supported in this combination.

```bash Command theme={null}
sglang serve \
  --model-path zai-org/GLM-5.3 \
  --trust-remote-code \
  --tp-size 8 \
  --dcp-size 2 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 6
```

## References

* [DCP and Helix Parallelism roadmap](https://github.com/sgl-project/sglang/issues/29736)
* [SGLang and Miles Add Day-0 Support for Kimi K3](https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support#decode-context-parallelism)
* [Original SGLang DCP design](https://docs.google.com/document/d/1pjBSID0palpuI7rx8Ck7Z0BcyzBXAgnkDrf9uQg0uqU/edit?tab=t.0)


# Deterministic Inference
Source: https://docs.sglang.io/docs/advanced_features/deterministic_inference



## Why Deterministic Inference Matters

Deterministic inference ensures consistent LLM outputs across runs, which is critical for:

* **Reinforcement Learning**: Ensures consistent logprobs across runs, reducing stochastic noise and making RL training more stable, reproducible, and debuggable.
* **Testing & Debugging**: Enables reproducible validation
* **Production**: Improves reliability and user experience

Even with `temperature=0`, standard LLM inference can produce different outputs due to dynamic batching and varying reduction orders in GPU kernels.

## The Root Cause of Non-Determinism

The main source is **varying batch sizes**. Different batch sizes cause GPU kernels to split reduction operations differently, leading to different addition orders. Due to floating-point non-associativity (`(a + b) + c ≠ a + (b + c)`), this produces different results even for identical inputs.

## SGLang's Solution

Building on [Thinking Machines Lab's batch-invariant operators](https://github.com/thinking-machines-lab/batch_invariant_ops), SGLang achieves fully deterministic inference while maintaining compatibility with chunked prefill, CUDA graphs, radix cache, and non-greedy sampling. The development roadmap for deterministic inference features can be found in this [issue](https://github.com/sgl-project/sglang/issues/10278).

### Supported Backends

Deterministic inference is only supported with the following three attention backends: **FlashInfer**, **FlashAttention 3 (FA3)**, and **Triton**.

The following table shows feature compatibility for deterministic inference across different attention backends:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Attention Backend</th>
      <th>CUDA Graph</th>
      <th>Chunked Prefill</th>
      <th>Radix Cache</th>
      <th>Non-greedy Sampling (Temp > 0)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**FlashInfer**</td>
      <td>✅ Yes</td>
      <td>✅ Yes</td>
      <td>❌ No</td>
      <td>✅ Yes</td>
    </tr>

    <tr>
      <td>**FlashAttention 3 (FA3)**</td>
      <td>✅ Yes</td>
      <td>✅ Yes</td>
      <td>✅ Yes</td>
      <td>✅ Yes</td>
    </tr>

    <tr>
      <td>**Triton**</td>
      <td>✅ Yes</td>
      <td>✅ Yes</td>
      <td>✅ Yes</td>
      <td>✅ Yes</td>
    </tr>
  </tbody>
</table>

## Usage

### Basic Usage

Enable deterministic inference by adding the `--enable-deterministic-inference` flag:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --attention-backend fa3 \
    --enable-deterministic-inference
```

### Server Arguments

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Type/Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-deterministic-inference`</td>
      <td>flag; default: disabled</td>
      <td>Enable deterministic inference with batch-invariant operations</td>
    </tr>

    <tr>
      <td>`--attention-backend`</td>
      <td>string; default: fa3</td>
      <td>Choose attention backend (flashinfer, fa3, or triton)</td>
    </tr>
  </tbody>
</table>

### Example Configurations

#### Qwen3-8B

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --attention-backend flashinfer \
    --enable-deterministic-inference
```

#### Llama Models

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --attention-backend fa3 \
    --enable-deterministic-inference
```

#### Qwen3-30B-A3B (MoE Model)

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B \
    --attention-backend fa3 \
    --enable-deterministic-inference
```

### Deterministic Inference with Non-Greedy Sampling (Temperature > 0)

SGLang supports deterministic inference even with non-greedy sampling by using sampling seeds. This is particularly useful for reinforcement learning scenarios like GRPO (Group Relative Policy Optimization) where you need multiple diverse but reproducible responses.

#### Default Behavior

By default, SGLang uses a sampling seed of `42` for reproducible sampling:

```python Example theme={null}
import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "Tell me a joke",
        "sampling_params": {
            "temperature": 0.8,  # Non-greedy sampling
            "max_new_tokens": 128,
        },
    },
)
print(response.json())
# This will always produce the same response across runs
```

#### Generating Multiple Reproducible Responses

To sample different responses from the same prompt while maintaining reproducibility (e.g., for GRPO training), provide different sampling seeds in your requests:

```python Example theme={null}
import requests

# Prepare a list of sampling seeds for different responses
sampling_seeds = [42, 43, 44, 45, 46]

responses = []
for seed in sampling_seeds:
    response = requests.post(
        "http://localhost:30000/generate",
        json={
            "text": "Tell me a joke",
            "sampling_params": {
                "temperature": 0.8,
                "max_new_tokens": 128,
                "sampling_seed": seed,  # Specify sampling seed
            },
        },
    )
    responses.append(response.json())

# Each seed will produce a different but reproducible response
# Using the same seed will always produce the same response
```

This approach ensures that:

* Different seeds produce diverse responses
* The same seed always produces the same response across different runs
* Results are reproducible for debugging and evaluation

## Verification

Run deterministic tests to verify consistent outputs:

```bash Command theme={null}
# Single test: same prompt, varying batch sizes
python3 -m sglang.test.test_deterministic --test-mode single --n-trials 50

# Prefix test: prompts with different prefix lengths
python3 -m sglang.test.test_deterministic --test-mode prefix --n-trials 50

# Radix Cache Consistency mode: test radix cache determinism (cached vs uncached prefill)
python3 -m sglang.test.test_deterministic --test-mode radix_cache
```

Expected result: All tests should show `Unique samples: 1` (perfectly deterministic).


# DP, DPA and SGLang DP Router
Source: https://docs.sglang.io/docs/advanced_features/dp_dpa_smg_guide



This guide explains the difference between Data Parallelism (DP) and Data Parallelism Attention (DPA), how to enable each mode correctly, and how to use the SGLang Model Gateway (SMG) for production-grade DP deployments.

## Data Parallelism (DP)

**Data Parallelism (DP)** is the most common parallelism strategy that replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. Each GPU set handles independent requests. With dedicated routing strategies, as we will introduce later, with those proper routing algorithms in SGLang Model Gateway, the throughput of your serving system could be multiplied nearly linearly.

### Key characteristics

* Each replica has a full copy of the model
* Requests are distributed/scattered across replicas
* No inter-replica communication during one request's inference (for simple DP)

## Data Parallelism Attention (DPA)

**Data Parallelism Attention (DPA)**, also known as DP Attention, is an advanced parallelism strategy. While DPA provides the most significant benefits for **Multi-Head Latent Attention (MLA)** models (such as DeepSeek, MiniMax, Kimi-K2), it also supports **standard attention models** like Qwen.

### The Problem with Tensor Parallelism for MLA Models

The most common parallelism strategy for inference is **Tensor Parallelism (TP)**. However, TP might not be the most efficient strategy for certain models. For example, DeepSeek models use MLA and only have **one KV head**. If we use tensor parallelism on 8 GPUs, it will lead to:

* **Duplicated KV cache** across all GPUs
* **Unwanted memory usage** that limits batch size
* **Reduced throughput** due to memory constraints

### How DPA Works

DPA addresses these limitations by applying **data parallelism specifically to the attention component**.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <tbody>
    <tr>
      <td>
        ![DPA + EP Architecture](https://mintcdn.com/lmsysorg/lCXzAEQmak7oHiXq/images/dpa.png?fit=max\&auto=format\&n=lCXzAEQmak7oHiXq\&q=85\&s=8a553be0ceb61b8d20241372e3b00370)
      </td>

      <td>
        **Each DP replica:**

        <ul>
          <li>Processes different batches independently (can be in different forward modes: prefill, decode, or idle)</li>
          <li>Maintains its own KV cache (no duplication)</li>
          <li>Enables significantly larger batch sizes due to memory savings</li>
        </ul>

        **Communication patterns in DPA + EP:**

        <ul>
          <li>**All2All (Dispatch)**: Routes tokens to expert sub-groups based on gating decisions</li>
          <li>**All2All (Combine)**: Gathers computed results from experts back to original token positions</li>
        </ul>
      </td>
    </tr>
  </tbody>
</table>

### Key benefits of DPA

1. **Significantly reduced KV cache memory**: Each DP replica only stores KV cache for its own batches
2. **Larger batch sizes**: Memory savings enable larger batch sizes
3. **Improved decoding throughput**: Significant throughput gains for MLA-based models
4. **Independent forward modes**: Each DP replica can be in different forward modes (prefill, decode, or idle) and handles its assigned batches independently during attention computation

### DPA with Expert Parallelism for MoE

For MoE models like DeepSeek, DPA is **often** paired with Expert Parallelism (EP) for best throughput at scale. However, **DPA does not require EP**: you can enable DPA without EP if your deployment does not need expert sharding.

* Distribute 256+ expert weights across GPUs (cannot fit on a single GPU)
* Enable efficient all-to-all token routing via DeepEP
* Scale to large clusters (up to 5x throughput improvement over vanilla TP)

### Recommended setup for DeepSeek

```bash theme={null}
python -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V3 \
    --tp 8 \
    --attn-dp-size 8 \
    --ep 8 \
    --moe-a2a-backend deepep \
    --moe-runner-backend deep_gemm
```

> **Note**: `--attn-dp-size` sets the number of attention data-parallel groups inside the TP group. If `attn_dp_size` is 1 (default), DPA is disabled.

For detailed EP configuration (DeepEP, Two-Batch Overlap, EPLB), see [Expert Parallelism](/docs/advanced_features/expert_parallelism).

### Target Models

DPA supports the following model architectures:

* **MLA (Multi-Head Latent Attention) models** - where DPA provides the most significant benefits:
  * DeepSeek family (DeepSeek-V2, DeepSeek-V3, DeepSeek-R1)
  * MiniMax models
  * Kimi-K2
  * Other models using MLA architecture

* **Standard attention models** - also supported:
  * Qwen models (see [PR #6121](https://github.com/sgl-project/sglang/pull/6121))

For models like Llama, with standard GQA, standard DP, or TP is typically recommended.

To enable DPA, add `--attn-dp-size N` (N > 1) to your server launch command.

### Activation Logic

DPA is enabled explicitly via server arguments (CLI or config) with `--attn-dp-size` (alias `--attention-data-parallel-size`). It creates N attention data-parallel groups inside the TP group, one scheduler each, while the FFN stays tensor parallel:

```bash theme={null}
python -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V3 \
    --tp 8 \
    --attn-dp-size 8
```

**Important**: `--attn-dp-size` must be greater than 1 for DPA to work. When `attn_dp_size == 1` (default), DPA is disabled. The constraint `tp_size % attn_dp_size == 0` must also be satisfied. `--dp-size` only sets the number of data-parallel replicas of the model; combining replicas (`--dp-size` > 1) with DPA is not supported.

> **Deprecated**: `--enable-dp-attention` still works: `--dp-size N --enable-dp-attention` resolves to `--attn-dp-size N --dp-size 1` and prints a deprecation warning. Use `--attn-dp-size N` instead. The flag will be removed after 2026-12-31.
>
> Under DeepSeek context parallelism (DSA, MLA, and DeepSeek-V4 prefill CP), `--dp-size N` alone still runs N attention-DP groups, and logs a deprecation warning; it will be rejected after 2026-12-31. Use `--attn-dp-size N` there too.

**Elastic EP scale joiners.** A rank that joins an existing DPA deployment through elastic EP scale-up runs DPA even when it is launched with `--attn-dp-size 1`: its own attention DP group is one rank wide.

**DP rank count.** The number of DP ranks (one scheduler each) is `dp_size × attn_dp_size`; one of the two is always 1.

* `/server_info` reports `dp_size` (replicas) and `attn_dp_size` separately, so under DPA its `dp_size` is 1. Until the flag is removed, its deprecated `enable_dp_attention` field still reports whether DPA runs; new clients should read `attn_dp_size`.
* Wire formats that predate `--attn-dp-size` keep the key `dp_size` for the DP rank count: KV-event descriptors, the PD prefill server info, and load snapshots.

### TP LMHead with all-to-all

`dp-lm-head` keeps full weight and computes the local batch, making the gemm inefficient when batchsize per-rank is small. The existing `tp-lm-head` improves the gemm efficiency but introduce heavy allgather communication on full vocabulary.

Therefore, all-to-all can be applied to eliminate the heavy allgather while retaining the gemm efficiency under small per-rank batchsize. Each rank will send and receive only the local rows with sharded vocab, which is more lightweight.

This optimization is **enabled by default for decode node and pure DPA**. Users can use `--no-enable-tp-lm-head-all-to-all` to opt out, or `--enable-tp-lm-head-all-to-all` for force enabling.

```bash theme={null}
python -m sglang.launch_server \
    --model-path MODEL_PATH \
    --disaggregation-mode decode \
    --tp 8 \
    --attn-dp-size 8
```

Prefill / colocated workers will fallback to TP LMHead with allgather, as well as other parallel strategies. More details can be found in [PR #32313](https://github.com/sgl-project/sglang/pull/32313).

### Standard DP for MLA models

Note that MLA models, of course, also support DP. Suppose you want to enable standard DP for MLA models. First, launch each MLA model's replica independently. You may launch these replicas one by one with DPA enabled. After launching each MLA model's replica, launch an SMG and connect all the replicas to the SMG. A detailed explanation of SMG is as follows.

## Modern Data Parallelism SGLang Model Gateway (SMG)

### Native DP Mode

Native DP (built-in Data Parallelism) in SGLang creates multiple worker processes within a single SGLang instance, under the control of `DataParallelController` with the launching parameter of `dp-size`.

```bash theme={null}
# Native DP mode
python -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dp-size 4
```

**Limitations:**

* Built-in in-process load balancing only (e.g., `round_robin`, `total_requests`, `total_tokens`)
* No cache-aware routing
* Limited observability and metrics
* No fault tolerance or circuit breakers
* Not suitable for production workloads

⚠️ Native DP is **highly not recommended for use right now**. It is only used in some ancient/outdated RL frameworks. You can use SGLang Model Gateway (SMG) to power up your data parallelism in any use case.

### SMG-Based DP (Recommended)

Starting from September 2024, SGLang Model Gateway, i.e., SMG, formerly named as SGLang DP Router, was built especially as a production-ready DP routing system with Rust. It starts from DP routing, but later we further expanded its scope to coordinate RL, PD Disaggregation, and other scenarios. This doc only discusses SMG's usage in DP routing. For other usage, please refer to [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).

> To achieve the best production-level routing performance and reduce the overhead to an extreme extent, we use Rust to build SMG, but not Python, since Python is never FAST enough.

**We strongly recommend using the SGLang Model Gateway (SMG) for production-grade Data Parallelism.** SMG provides significant advantages over native DP mode.

```bash theme={null}
# SMG-based DP mode (Recommended)
python -m sglang_router.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dp-size 4
```

⚠️ Note that **SMG and Naive DP share the same launching parameter, `--dp-size`**. But the entrypoint of Naive DP is `python -m sglang.launch_server`, and SMG's entrypoint is `python -m sglang_router.launch_server`.

**Advantages of SMG-Based DP:**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Feature</th>
      <th>Native DP</th>
      <th>SMG-Based DP</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Load Balancing**</td>
      <td>Built-in in-process methods</td>
      <td>Advanced policies (cache-aware, power-of-two, etc.)</td>
    </tr>

    <tr>
      <td>**Cache Awareness**</td>
      <td>❌ No</td>
      <td>✅ Yes - significantly higher cache hit rate</td>
    </tr>

    <tr>
      <td>**Throughput**</td>
      <td>Baseline</td>
      <td>Significant improvement</td>
    </tr>

    <tr>
      <td>**Multi-Node Support**</td>
      <td>Limited</td>
      <td>✅ Full support</td>
    </tr>

    <tr>
      <td>**Worker Health Monitoring**</td>
      <td>Basic</td>
      <td>✅ Circuit breakers, health checks</td>
    </tr>

    <tr>
      <td>**Reliability**</td>
      <td>Basic</td>
      <td>✅ Retries, rate limiting, queuing</td>
    </tr>

    <tr>
      <td>**Observability**</td>
      <td>Basic metrics</td>
      <td>✅ 40+ Prometheus metrics, OpenTelemetry</td>
    </tr>

    <tr>
      <td>**Hot Worker Add/Remove**</td>
      <td>❌ No</td>
      <td>✅ Yes</td>
    </tr>
  </tbody>
</table>

### SMG's Performance

The cache-aware routing policy in SMG significantly improves performance for workloads with shared prefixes:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Metric</th>
      <th>Without Cache-Aware</th>
      <th>With Cache-Aware SMG</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Throughput (token/s)</td>
      <td>82,665</td>
      <td>158,596 (+92%)</td>
    </tr>

    <tr>
      <td>Cache Hit Rate</td>
      <td>20%</td>
      <td>75% (+275%)</td>
    </tr>
  </tbody>
</table>

*Benchmark from [SGLang v0.4 blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), workload with multiple long prefix groups, 8x A100 80GB GPUs, dp-size=8*

### When to Use Each

**Use Native DP when:**

* ~~Never use Native/Naive DP~~
* Learning material of DP routing

**Use SMG-Based DP when:**

* In any case, when you think DP is needed
* Production deployments
* Multi-node distributed setups
* Workloads with shared prefixes (high cache reuse potential)
* You need high availability and reliability features
* You require detailed observability and metrics
* You want to have highly efficient RL rollout systems

Note that for RL rollout systems, **there are four crucial reasons that SMG-Based DP is far better than naive DP routing**. Details can be found at [Load Balancing Router in RL](/docs/advanced_features/sglang_for_rl#load-balancing-router).

### Quick Start For SMG

**Installation**

```bash theme={null}
pip install sglang-router
# or
pip install "sglang[all]"
```

**Option A: Co-launch Workers and SMG (Simplest)**

This is the easiest way to get started - SMG and workers are launched together:

```bash theme={null}
python -m sglang_router.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dp-size 4 \
    --host 0.0.0.0 \
    --port 30000
```

**Option B: Separate Launch (Multi-Node)**

For distributed deployments across multiple machines:

1. Launch workers on each node

```bash theme={null}
# Node 1
python -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --port 8000

# Node 2
python -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --port 8000
```

2. Launch SMG pointing to workers

```bash theme={null}
python -m sglang_router.launch_router \
    --worker-urls http://node1:8000 http://node2:8000 \
    --policy cache_aware \
    --host 0.0.0.0 \
    --port 30000
```

**Option C: Dynamic Worker Registration**

For elastic deployments where workers can be added/removed dynamically:

```bash theme={null}
# Launch SMG first
python -m sglang_router.launch_router \
    --policy cache_aware \
    --host 0.0.0.0 \
    --port 30000

# Register workers dynamically
curl -X POST http://localhost:30000/workers \
    -H "Content-Type: application/json" \
    -d '{"url": "http://worker1:8000"}'

curl -X POST http://localhost:30000/workers \
    -H "Content-Type: application/json" \
    -d '{"url": "http://worker2:8000"}'
```

### Load Balancing Policies

SMG supports multiple load balancing policies:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Policy</th>
      <th>Description</th>
      <th>Best For</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`cache_aware`</td>
      <td>Combines cache locality with load balancing</td>
      <td>**Recommended for most workloads**</td>
    </tr>

    <tr>
      <td>`round_robin`</td>
      <td>Cycles through workers in order</td>
      <td>Simple, predictable distribution</td>
    </tr>

    <tr>
      <td>`random`</td>
      <td>Random worker selection</td>
      <td>Baseline, testing</td>
    </tr>

    <tr>
      <td>`power_of_two`</td>
      <td>Samples two workers, picks lighter one</td>
      <td>Low latency requirements</td>
    </tr>
  </tbody>
</table>

**Cache-Aware Policy (Default, Recommended)**

The cache-aware policy provides the best performance for most workloads:

```bash theme={null}
python -m sglang_router.launch_router \
    --worker-urls http://worker1:8000 http://worker2:8000 \
    --policy cache_aware \
    --cache-threshold 0.5 \
    --balance-abs-threshold 32 \
    --balance-rel-threshold 1.5 \
    --eviction-interval-secs 120 \
    --max-tree-size 67108864
```

**How it works:**

1. Maintains an approximate radix tree for each worker based on request history
2. Routes requests to workers with the highest prefix match (cache hit)
3. Falls back to shortest-queue routing when load is imbalanced
4. Automatically evicts old entries to prevent memory overflow

### Best Practices

1. **Start with `cache_aware` policy** - It provides the best balance between cache locality and load distribution for most workloads
2. **Use SMG for production** - Prefer `sglang_router.launch_server` over `sglang.launch_server` for better reliability and observability
3. **Enable health checks** - Configure `--router-health-check-interval-secs` to detect and remove unhealthy workers automatically

**Recommended command with best practices applied:**

```bash theme={null}
python -m sglang_router.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --dp-size 4 \
    --router-policy cache_aware \
    --router-health-check-interval-secs 30 \
    --router-prometheus-port 10001 \
    --host 0.0.0.0 \
    --port 30000
```

For advanced configuration (circuit breakers, retries, Prometheus metrics, K8s integration), see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).

### Verifying Traffic Distribution

After launching SMG, verify that traffic is being distributed correctly:

**1. Check worker status:**

```bash theme={null}
curl http://localhost:30000/workers
```

**2. Check load distribution:**

```bash theme={null}
curl http://localhost:30000/get_loads
```

**3. Monitor metrics (if Prometheus enabled):**

```bash theme={null}
# Key metrics to check
smg_router_requests_total{model="..."}
smg_worker_requests_active{worker="..."}
sglang_cache_hit_rate{source="..."}
```

For detailed metrics and monitoring setup, see [SGLang Model Gateway Documentation](/docs/advanced_features/sgl_model_gateway).

## Reference

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Strategy</th>
      <th>Use Case</th>
      <th>Key Benefit</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Native DP** (`--dp-size`)</td>
      <td>Never</td>
      <td>Easy to understand, not rust based</td>
    </tr>

    <tr>
      <td>**SMG-Based DP**</td>
      <td>**Production (recommended)**</td>
      <td>Cache-aware routing, high availability</td>
    </tr>

    <tr>
      <td>**DPA** (`--attn-dp-size N`)</td>
      <td>DeepSeek/MLA models</td>
      <td>Eliminates KV cache duplication, improved throughput</td>
    </tr>

    <tr>
      <td>**DPA + EP**</td>
      <td>DeepSeek MoE models</td>
      <td>Significant throughput improvement vs vanilla TP</td>
    </tr>
  </tbody>
</table>

**Recommended production setup for DeepSeek:**

1. Enable **DPA** for attention layers (`--attn-dp-size 8`)
2. Enable **EP** for MoE layers (`--ep 8 --moe-a2a-backend deepep`)
3. Use **SMG** with **cache\_aware** policy

**Related documentation:**

* [Expert Parallelism](./expert_parallelism) - DeepEP, Two-Batch Overlap, EPLB
* [SGLang Model Gateway Documentation](./sgl_model_gateway) - SMG configuration & troubleshooting
* [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/) - 96 GPU deployment guide


# DP for Multi-Modal Encoder in SGLang
Source: https://docs.sglang.io/docs/advanced_features/dp_for_multi_modal_encoder



A typical VLM architecture involves two main components: an multi-modal encoder and a text decoder.

Most VLMs utilize a Vision Transformer (ViT) as their multi-modal encoder, it is responsible for processing visual data, extracting features (objects, colors, textures, etc.), and transforming them into a format that can be understood by the model.

The text deocoder is based on LLM. It processes textual data and generates output based on the encoded visual features.

However, since the size of ViT is very small compared to language decoders,
there is relatively little gain from TP. On the other hand, TP incurs significant communication
overhead because of all-reduce being performed after every layer.

Placing the ViT in data parallel while keeping the LLM in tensor parallel consistently lowers TTFT and boosts end-to-end throughput. In this hybrid layout, the vision front-end becomes parallel and lightweight, while scarce interconnect bandwidth and collective ops are reserved for the LLM.

Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel.

## Command Example

You can enable batch-level DP by setting `mm-enable-dp-encoder`, for example:

```shell Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen2.5-VL-7B-Instruct \
    --tp 2 \
    --mm-enable-dp-encoder
```

## Known supported models

* Qwen2.5-VL (\<[https://github.com/sgl-project/sglang/pull/13126](https://github.com/sgl-project/sglang/pull/13126)>)
* Qwen3-VL (\<[https://github.com/sgl-project/sglang/pull/13724](https://github.com/sgl-project/sglang/pull/13724)>)
* InternVL (\<[https://github.com/sgl-project/sglang/pull/13925](https://github.com/sgl-project/sglang/pull/13925)>)
* GLM-4.5V & GLM-4.6V (\<[https://github.com/sgl-project/sglang/pull/14097](https://github.com/sgl-project/sglang/pull/14097)>)


# EPD Disaggregation
Source: https://docs.sglang.io/docs/advanced_features/epd_disaggregation



## Why and What is EPD Disaggregation?

In modern Vision-Language Model (VLM) inference, request execution naturally decomposes into three distinct stages: Encoder, Prefill, and Decode.
The Encoder stage performs vision preprocessing and ViT-based image encoding, which is highly compute-intensive but only required during request initialization. The Prefill stage processes the full multimodal input sequence to initialize the language model’s Key-Value (KV) cache, while the Decode stage is dominated by memory bandwidth and KV cache access for autoregressive token generation.

Existing deployments typically colocate these stages within a unified execution engine, or at best apply Prefill–Decode (PD) disaggregation. However, such designs still tightly couple vision encoding with language prefill, leading to inefficient resource utilization, limited scalability for image-heavy workloads, and suboptimal scheduling under load.

To address these challenges, we introduce Encoder–Prefill–Decode (EPD) Disaggregation in SGLang. EPD further separates vision encoding from language processing, enabling independent horizontal scaling of encoder servers, improved load balancing for multimodal requests, and seamless integration with existing PD disaggregation to form a fully decoupled three-tier inference architecture.

### Usage

You can launch a language-only model using `--language-only`, or an encoder-only model using `--encoder-only`.
When launching a language-only model, you must additionally specify the encoder service endpoints via `--encoder-urls`.

We support multiple encoder transfer backends, including zmq\_to\_scheduler, zmq\_to\_tokenizer, and mooncake (the default is zmq\_to\_scheduler). The backend can be selected using `--encoder-transfer-backend`.

### Encoder transfer with Mooncake

`--encoder-transfer-backend mooncake` controls **how encoder outputs are transferred** between encoder and language/prefill services. It is an encoder transfer option and can be used independently of the global multimodal embedding cache.

Example:

```bash Command theme={null}
# encoder
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --encoder-only \
  --encoder-transfer-backend mooncake \
  --port 30000

# language-only server
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --language-only \
  --encoder-urls http://127.0.0.1:30000 \
  --encoder-transfer-backend mooncake \
  --port 30002
```

### Global multimodal embedding cache with Mooncake

SGLang also supports a Mooncake-backed **global multimodal embedding cache** for EPD workloads. When enabled on encoder servers, repeated image inputs can reuse previously computed ViT embeddings across instances instead of running the vision encoder again.

This feature is useful when:

* the deployment serves repeated or overlapping image inputs,
* encoder compute is the bottleneck, and
* Mooncake is already available in the cluster.

At a high level, the encoder checks whether the image embedding already exists in Mooncake. Cache hits are prefetched from the global store, while misses are encoded normally and inserted into the cache in the background.

To enable it:

* install and configure Mooncake in the same way as other SGLang Mooncake integrations,
* add `--enable-mm-global-cache` on the encoder server.

`--enable-mm-global-cache` controls **whether multimodal embeddings are looked up and stored in the global Mooncake cache**. It is separate from `--encoder-transfer-backend`, which only controls encoder output transport.

For Mooncake deployment and configuration details, see [HiCache best practices](./hicache_best_practices#deployment-with-mooncake) and the [Mooncake backend README](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md).

Example:

```bash Command theme={null}
# Shared Mooncake configuration
export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata"
export MOONCAKE_MASTER="127.0.0.1:50051"
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_GLOBAL_SEGMENT_SIZE="4gb"

# encoder with global multimodal cache enabled
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --encoder-only \
  --enable-mm-global-cache \
  --port 30000

# language-only server
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --language-only \
  --encoder-urls http://127.0.0.1:30000 \
  --port 30002
```

Notes:

* This cache is for **multimodal encoder embeddings**, not the language model KV cache.
* The feature currently uses Mooncake as the shared backing store.
* It can be enabled regardless of which `--encoder-transfer-backend` you use.
* It is most relevant for EPD or encoder-disaggregated VLM deployments where the same images are likely to appear across requests or instances.

#### Qwen VL

* EP Disaggregation

```bash Command theme={null}
# encoder 0
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --encoder-only \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30000
# encoder 1
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --encoder-only \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30001
# language-only server
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --language-only \
  --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30002
```

* EPD Disaggregation

```bash Command theme={null}
# encoder 0
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --encoder-only \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30000
# encoder 1
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --encoder-only \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30001
# prefill 0
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --disaggregation-mode prefill \
  --language-only \
  --encoder-urls http://127.0.0.1:30000 http://127.0.0.1:30001 \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30002
# decode 0
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --disaggregation-mode decode \
  --port 30003
# router
python -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://$PREFILL_HOST:30002 \
  --decode http://$DECODE_HOST:30003 \
  --port 8000

```

#### gRPC Encoder (EPD)

You can run the encoder as a gRPC server while keeping prefill/decode as HTTP.
When using gRPC encoders, set `SGLANG_ENCODER_MM_RECEIVER_MODE=grpc` for the
prefill process so it uses the gRPC receiver.

```bash Command theme={null}
# gRPC encoder
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --encoder-only \
  --grpc-mode \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30000

# prefill (HTTP) - tell it to use gRPC receiver
SGLANG_ENCODER_MM_RECEIVER_MODE=grpc \
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --disaggregation-mode prefill \
  --language-only \
  --encoder-urls grpc://127.0.0.1:30000 \
  --encoder-transfer-backend zmq_to_scheduler \
  --port 30002

# decode (HTTP)
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-8B-Instruct \
  --disaggregation-mode decode \
  --port 30003

# router
python -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://$PREFILL_HOST:30002 \
  --decode http://$DECODE_HOST:30003 \
  --port 8000
```


# Expert Parallelism
Source: https://docs.sglang.io/docs/advanced_features/expert_parallelism



Expert Parallelism (EP) in SGLang distributes expert weights across multiple devices in Mixture-of-Experts (MoE) models, addressing memory bottlenecks and enabling efficient scaling for high-performance inference. It is particularly vital for serving large-scale MoE models where tokens are dynamically routed to specialized experts across GPUs. By leveraging optimized all-to-all communication and grouped matrix multiplications (GEMMs), EP reduces latency, boosts throughput, and minimizes idle GPU time. SGLang's EP offers strong extensibility through its modular framework, allowing seamless integration of custom kernels, backends, and optimizations without refactoring core logic, supporting diverse hardware and quantization schemes.

## Supported Backends and Selection Guidance

SGLang's EP integrates diverse, highly efficient backends for different use cases, allowing fine-grained control over performance trade-offs. Users specify backends via command-line flags:

* `--moe-a2a-backend`: Selects the backend for all-to-all communication.
* `--moe-runner-backend`: Selects the backend for MoE computation.

### Backends for All-to-All Communication

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Backend</th>
      <th>Description</th>
      <th>Use Cases</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**`none` (default)**</td>
      <td>Disables all-to-all for EP. Uses All-Reduce or All-Gather for token dispatch.</td>
      <td>Hybrid EP and TP setups.</td>
    </tr>

    <tr>
      <td>`deepep`</td>
      <td>DeepEP, a communication library for efficient token shuffling in MoE models.</td>
      <td>Large-scale EP deployments.</td>
    </tr>

    <tr>
      <td>`mooncake`</td>
      <td>An extension of DeepEP for elastic inference, leveraging RDMA for high-performance data transfers.</td>
      <td>Elastic EP serving.</td>
    </tr>

    <tr>
      <td>`nixl`</td>
      <td>[NIXL-EP](https://github.com/ai-dynamo/nixl/tree/main/examples/device/ep), an elastic EP communication library built on NVIDIA's [NIXL](https://github.com/ai-dynamo/nixl) framework with native RDMA and NVLink support.</td>
      <td>Elastic EP serving with fault tolerance and dynamic scaling.</td>
    </tr>

    <tr>
      <td>`mori`</td>
      <td>MORI-EP, AMD's native all-to-all communication implementation optimized for ROCm.</td>
      <td>AMD GPU deployments.</td>
    </tr>

    <tr>
      <td>`flashinfer`</td>
      <td>Flashinfer implementation of all-to-all.</td>
      <td>Large-scale EP deployments.</td>
    </tr>

    <tr>
      <td>`ascend_fuseep`</td>
      <td>Ascend NPU native fused all-to-all communication.</td>
      <td>Ascend NPU deployments.</td>
    </tr>

    <tr>
      <td>`pplx`</td>
      <td>[pplx-kernels](https://github.com/perplexityai/pplx-kernels), Perplexity's NVSHMEM-based all-to-all dispatch/combine kernels. Low-latency (masked) only; targets FP8 (DeepGEMM) MoE models on Hopper. Requires NVSHMEM 3.2.5, `nvshmem4py`, `cuda-python`, and a prebuilt `libpplx_kernels.so` (`sm_90a`).</td>
      <td>Low-latency decode EP on Hopper.</td>
    </tr>
  </tbody>
</table>

DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). MORI backend only supports `normal` mode now. NIXL-EP and PPLX currently operate in low-latency mode with CUDA Graph support (PPLX reuses the DeepEP low-latency masked expert-compute path). Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes.

Currently, DeepEP, Mooncake, NIXL-EP, `ascend_fuseep`, `pplx` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported. Note that `pplx` additionally requires DP attention with at least 2 attention DP groups (i.e., `--attn-dp-size` > 1, so `tp_size / attention_tp_size > 1`); otherwise pplx-kernels' AllToAll cannot be constructed.

### Backends for MoE Computation

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Backend</th>
      <th>Description</th>
      <th>Use Cases</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**`auto` (default)**</td>
      <td>Automatically selects the optimal backend based on model architecture, hardware (e.g., NVIDIA architecture like Ampere, Hopper, Blackwell), quantization scheme (e.g., FP8, FP4), and runtime conditions.</td>
      <td>General-purpose deployments; ensures compatibility and performance without user intervention.</td>
    </tr>

    <tr>
      <td>`triton`</td>
      <td>Triton-based implementation for grouped GEMMs. To achieve higher performance, it's highly recommended to create [tuned configurations](https://github.com/sgl-project/sglang/blob/main/benchmark/kernels/fused_moe_triton/README.md).</td>
      <td>Custom kernel development or scenarios requiring high extensibility with Torch compilation support.</td>
    </tr>

    <tr>
      <td>`deep_gemm`</td>
      <td>DeepGEMM backend optimized for MoE matrix multiplications, supporting contiguous layouts for prefill and masked layouts for decode; often JIT-compiled for performance.</td>
      <td>Large-scale EP deployments with FP8 block-wise quantization.</td>
    </tr>

    <tr>
      <td>`cutlass`</td>
      <td>CUTLASS-based backend for efficient GEMMs.</td>
      <td>NVIDIA architectures with CUTLASS support.</td>
    </tr>

    <tr>
      <td>`flashinfer_trtllm`</td>
      <td>FlashInfer integrated with TensorRT-LLM for accelerated MoE computations, supporting FP4 communication operators and high-performance GEMMs.</td>
      <td>Blackwell with TRT-LLM.</td>
    </tr>

    <tr>
      <td>`flashinfer_trtllm_routed`</td>
      <td>FlashInfer integrated with TensorRT-LLM for accelerated routed MoE computations, consuming SGLang-computed top-k expert assignments and weights. Compatible with flashinfer all-to-all.</td>
      <td>Blackwell with TRT-LLM.</td>
    </tr>

    <tr>
      <td>`flashinfer_cutlass`</td>
      <td>FlashInfer combined with CUTLASS for high-performance grouped GEMMs in MoE layers, handling FP4/FP8 quantization efficiently.  Compatible with flashinfer all-to-all.</td>
      <td>Blackwell with FP4/FP8 models.</td>
    </tr>

    <tr>
      <td>`flashinfer_mxfp4`</td>
      <td>FlashInfer variant optimized for MXFP4 (mixed FP4) quantization in MoE runners, focusing on memory-efficient low-precision inference.</td>
      <td>Low-precision models with MXFP4.</td>
    </tr>

    <tr>
      <td>`flashinfer_cutedsl`</td>
      <td>FlashInfer with a custom DSL for flexible and efficient MoE kernel generation, integrated with ModelOpt FP4 quantization. Compatible with flashinfer all-to-all.</td>
      <td>Low-precision models with NVFP4.</td>
    </tr>
  </tbody>
</table>

### Examples

Launch with DeepEP and DeepGEMM for DeepSeek-V3:

```bash Command theme={null}
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 --moe-a2a-backend deepep --moe-runner-backend deep_gemm --tp 8 --ep 8
```

## Extensible EP Framework

SGLang's EP framework provides modular abstractions for easy integration of custom kernels, backends, and optimizations. It decouples the MoE forward pass into stages (dispatch → pre-permute → core runner → post-permute → combine), enabling seamless extensions without refactoring core logic. A fused function registered for a layer's all-to-all and runner backends replaces the three middle stages.

### Framework Overview

The framework centers on `FusedMoE` as the unified entry point for a single, extensible structure. Key components include:

* **Dispatcher**: Manages dispatch/combine for backends like DeepEP (implements `BaseDispatcher` subclasses).
* **MoeRunner**: Orchestrates grouped-GEMM execution via `MoeRunnerCore` implementations (e.g., `TritonRunnerCore`).
* **FusedOpPool** and **PermuteMethodPool**: Auto-register fused functions per all-to-all and runner backend pair (`register_fused_func`) and layout conversions (`register_pre_permute` and `register_post_permute`). `MoeRunner.run` calls the fused function when one is registered for the layer's pair and LoRA is disabled; otherwise it runs pre-permute, the runner core, and post-permute, or hands the dispatch output to a `DispatchMoeRunnerCore`.
* **TopK Router**: Backend-agnostic expert selection.

This design supports multiple backends via `--moe-a2a-backend` and `--moe-runner-backend`, with quantization integrated through a standardized `apply()` method. The computation flow ensures modularity:

```text Output theme={null}
[input_hidden_states]
          |
          v
     TopK.forward -> select_experts / triton_kernels.routing / bypass
          |
          v
     [TopKOutput]
          |
          v
   FusedMoE.forward -> Dispatcher.dispatch -> DeepEP / bypass
          |                     |
          |                     v
          |              [DispatchOutput]
          |                     |
          |                     v
          |             quant_method.apply -> MoeRunner.run
          |                     |              |
          |                     |              v
          |                     | fused func
          |                     | or DispatchMoeRunnerCore.run_from_dispatch
          |                     | or pre-permute + grouped_gemm + post-permute
          |                     |              |
          |                     |--------------
          |                     v
          |               [CombineInput]
          |                     |
          |                     v
          |            Dispatcher.combine -> DeepEP / bypass
          |                     |
          |---------------------
          v
[final_hidden_states]
```

For details, see the [MoE Refactor Roadmap](https://github.com/sgl-project/sglang/issues/8715).

### Implementing New Backends

To add a new backend:

1. For a new all-to-all dispatcher, implement a `BaseDispatcher` subclass with `dispatch` and `combine` methods.
2. For a new MoE runner backend, define a `MoeRunnerCore` subclass for core operations (e.g., grouped GEMMs).
3. Define new input/output formats for the dispatcher or model runner (e.g., `RunnerInput`, `RunnerOutput`).
4. Register permute/unpermute methods to ensure compatibility:

   * **Fused Mode** (static, torch.compile-compatible): Use `register_fused_func` for end-to-end operations.
   * **Permute Mode** (dynamic): Register `register_pre_permute` and `register_post_permute` for flexible layouts.

   When both are registered for a backend pair, the fused function takes precedence; the non-fused path still serves LoRA, which skips fused functions.

See the [MoE Refactor Implementation PR](https://github.com/sgl-project/sglang/pull/9269) for full changes, including type hints and config expansions.

### Examples

For an example implementation, see [moe\_runner/triton.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/moe_runner/triton.py), which demonstrates Triton-based grouped GEMMs with a fused function registered for the `none` all-to-all backend and permutation functions as the fallback path, used for example with LoRA.

## Computation and Communication Overlap

SGLang's EP employs advanced overlap techniques to hide communication latency behind computation, maximizing GPU utilization in MoE layers.

### Two-Batch Overlap (TBO)

TBO splits requests into micro-batches, interleaving attention computation with dispatch/combine operations. Yield points in the execution graph allow pausing for overlaps, increasing overall throughput without peak memory spikes:

```python Example theme={null}
operations = [
    self._forward_attn,
    YieldOperation(),  # Overlap with dispatch of prior micro-batch
    self._forward_dispatch,
    self._forward_mlp,
    YieldOperation(),  # Overlap with combine
    self._forward_combine,
]
```

Users need to specify `--enable-two-batch-overlap` to unlock up to 2x throughput. For details, see the [Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#two-batch-overlap).

### Single-Batch Overlap (SBO)

SGLang introduces a dispatcher-hook system for Single-Batch Overlap (SBO), enabling the overlap of operations within a single batch—such as shared experts computation with communication—while decentralizing logic to enhance modularity. These hooks execute before and after the `dispatch` and `combine` operations without modifying core MoE modules. This design simplifies interfaces, reduces coupling, and improves extensibility. For implementation details and an example of overlapping shared experts with DeepEP's combine operation, refer to [PR #13327](https://github.com/sgl-project/sglang/pull/13327). Users can set `--enable-single-batch-overlap` to enable this feature.

## Workload Balancer

SGLang integrates the [Expert Parallelism Load Balancer (EPLB)](https://github.com/deepseek-ai/EPLB) from DeepSeek to address routing imbalances in MoE models. By analyzing expert activation statistics, EPLB computes an optimal expert arrangement, strategically placing or replicating experts to minimize GPU utilization variance, reduce idle cycles, and enhance scalability.

To enable EPLB, use the flags `--enable-eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains.

For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb).

## Ascend NPU Guidance

### Guidance on SGLang configuration in Ascend NPU

* `--moe-a2a-backend` only supports `deepep` and `ascend_fuseep` backends,

  * `deepep`: The mechanism is consistent with the above description.

  * `ascend_fuseep`: Offer a large fused operator which integrates all operations between dispatch and combine to boost MoE computation. Only used for decode stage in PD Disaggregation Mode.

* `--moe-runner-backend` parameter does not need to be configured.

* `--deepep-mode`:

  * In PD mixed mode, please set `--deepep-mode auto`.

  * In PD Disaggregation Mode, prefill instance sets `--deepep-mode normal`, and decode instance sets `--deepep-mode low_latency`.

### DeepEP Ascend Introduction

DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP).
It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences.

Ant-moving Function can be enabled for both the dispatch and combine phases via the following environment variables:

* `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`: Enable ant-moving function in dispatch stage. Indicates the number of tokens transmitted per round on each rank, default 8192.

* `DEEPEP_NORMAL_LONG_SEQ_ROUND`: Enable ant-moving function in dispatch stage. Indicates the number of rounds transmitted on each rank, default 1.

* `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`: Enable ant-moving function in combine stage, default 0 (means disabled).

`DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * DEEPEP_NORMAL_LONG_SEQ_ROUND` means input sequence length. When the input sequence length exceeds 8192, it is recommended to enable the ant-moving function in both dispatch and combine phase.

The environment variable `HCCL_BUFFSIZE` is used to configure the buffer size (MB) actually allocated. Its calculation formula is as follows:

```text Output theme={null}
# Enable Ant-moving Function
HCCL_BUFFSIZE >= 2 * (102MB + 4MB + DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * (hidden_size + hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE

# Disable Ant-moving Function
HCCL_BUFFSIZE >= 2 * (102MB + 4MB + TOTAL_SEQ_LEN * (hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE
```

Wherein the parameters are described as follows:

* `hidden_size`: hidden size in model config.

* `topk`: The number of selected routing experts.

* `TOTAL_SEQ_LEN`: input sequence length.

* `PADDING_BUFFSIZE`: A value of 20 or greater is recommended.


# Hierarchical KV Caching (HiCache)
Source: https://docs.sglang.io/docs/advanced_features/hicache



* [Hicache Best Practices](./hicache_best_practices)
* [Hicache Design](./hicache_design)
* [Hicache Storage Runtime Attach Detach](./hicache_storage_runtime_attach_detach)


# SGLang HiCache Best Practices
Source: https://docs.sglang.io/docs/advanced_features/hicache_best_practices



## Why HiCache Matters

SGLang HiCache extends the traditional RadixAttention with a three-tier hierarchical KV caching system that dramatically improves performance for long-context and multi-turn conversation scenarios. By intelligently managing KV caches across GPU memory, host memory, and external storage backends, HiCache addresses the fundamental capacity bottleneck that limits cache hit rates in conventional systems.

> [!NOTE]
>
> L1 and L2 are private to a single inference instance; only L3 can be shared. Host memory cannot be pooled across instances or across hosts, not even for two instances on the same node. Raising `--hicache-ratio` or `--hicache-size` only enlarges the instance's own private L2. Cross-instance reuse is the job of L3, which needs `--hicache-storage-backend` and a backend configured for the scope you want: the `file` backend defaults to the node-local `/tmp/hicache`, whereas `mooncake`, `hf3fs`, `nixl` and `aibrix` reach cluster scope when every instance shares the same namespace. See [Tier Sharing Scope](/docs/advanced_features/hicache_design#tier-sharing-scope).

## Configuration Guidelines

## Core HiCache Parameters

```bash Command theme={null}
# Essential HiCache flags
--page-size 64                        # Page size for cache management
--enable-hierarchical-cache           # Enable HiCache
--hicache-ratio 2                     # Host memory ratio (2x GPU memory)
--hicache-size 100                    # Host memory size in GBs, will override the above ratio
--hicache-io-backend kernel           # The I/O backend of moving data between CPU and GPU
--hicache-write-policy write_through  # Cache write policy from GPU to CPU
--hicache-storage-backend             # Optional storage backend (e.g., hf3fs, mooncake, etc.)
```

Notes:

* Besides configuring `--hicache-storage-backend` at startup, SGLang also supports **runtime attach/detach** of the HiCache storage backend (no restart required) via HTTP admin endpoints. See [Runtime Attach/Detach HiCache Storage Backend](./hicache_storage_runtime_attach_detach).

## Key Configurations with Storage Backends Enabled

### Host Pool Sizing

```bash Command theme={null}
# Recommended: size the host pool as a multiple of the GPU KV pool
--hicache-ratio 2
--hicache-size 0
```

Guidelines:

* Storage (L3) hits are staged in the host pool before being loaded to the GPU, so the host pool needs free room at the time of the hit.
* With `write_through`, every token cached on the GPU also keeps a host copy. Once the GPU pool is full, at most `host pool tokens - GPU pool tokens` is left for storage hits.
* If the host pool is not larger than the GPU pool, storage hits are dropped and the prompt is recomputed. `sglang:storage_prefetch_unfulfilled_tokens_total{reason="host_capacity"}` grows.
* `--hicache-size` sets the host pool in GB and overrides `--hicache-ratio`. If you set it, check that the host pool is larger than the GPU pool in tokens, as reported in the startup log.

### Memory Layout Optimization

```bash Command theme={null}
# Page-first: Optimized for I/O efficiency with zero-copy (recommended with kernel backend)
--hicache-mem-layout page_first
# Page-first-direct: Optimized for direct I/O operations (Compatible with fa3 and same zero-copy performance as page_first)
--hicache-mem-layout page_first_direct
# Layer-first
--hicache-mem-layout layer_first
```

**Layout Compatibility:**

* `page_first`: Only compatible with `kernel` I/O backend, automatically switches to `layer_first` with `direct` backend
* `page_first_direct`: Specifically designed for `direct` I/O backend with optimized memory organization

### Heterogeneous TP Support (GQA/MHA models)

HiCache storage supports cross-cluster KV reuse when different deployments use different TP sizes (for example, `tp=4` and `tp=8`) and share the same storage backend namespace.

Use `tp_lcm_size` in `--hicache-storage-backend-extra-config`:

```bash Command theme={null}
# Example: heterogeneous TP = {4, 8}, so lcm = 8
--hicache-storage-backend-extra-config '{"tp_lcm_size": 8}'
```

Guidelines:

* Set `tp_lcm_size` to the least common multiple (LCM) of all TP sizes that will share the same HiCache storage.
* For MHA models with Mooncake and `page_head` layout, HiCache will split head shards based on `tp_lcm_size` to make keys reusable across heterogeneous TP deployments.
* If all clusters use the same TP size, this option is not needed.

### Prefetch Policies

```bash Command theme={null}
# Best-effort: Terminate prefetch when needed
--hicache-storage-prefetch-policy best_effort
# Wait-complete: Ensure complete prefetch, higher cache reuse
--hicache-storage-prefetch-policy wait_complete
# Timeout: Balance between completion and best-effort
--hicache-storage-prefetch-policy timeout
```

### Integration with PD Disaggregation

HiCache works seamlessly with PD Disaggregation. You can choose between two configurations:

1. **Prefill-only HiCache**: Enable HiCache only on Prefill nodes, allowing KV cache sharing among Prefill instances
2. **Full HiCache with async offloading**: Enable HiCache on Prefill nodes and async KV cache offloading on Decode nodes, allowing Prefill nodes to reuse KV caches from Decode nodes in multi-turn dialogue scenarios

```bash Command theme={null}
# Prefill node with HiCache enabled for cross-prefill sharing (ideal for SystemPrompt scenarios)
python3 -m sglang.launch_server \
  --model-path /xxx/DeepSeek-R1/ \
  --tp 8 \
  --host 0.0.0.0 \
  --port 10000 \
  --enable-metrics \
  --enable-cache-report \
  --mem-fraction-static 0.85 \
  --page-size 64 \
  --enable-hierarchical-cache \
  --hicache-ratio 2 \
  --hicache-size 0 \
  --hicache-mem-layout page_first_direct \
  --hicache-io-backend direct \
  --hicache-write-policy write_through \
  --hicache-storage-backend hf3fs \
  --hicache-storage-prefetch-policy wait_complete \
  --disaggregation-ib-device mlx5_0 \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend mooncake

# Decode node with async offloading enabled for KV cache reuse by Prefill (ideal for multi-turn conversations)
python3 -m sglang.launch_server \
  --model-path /xxx/DeepSeek-R1/ \
  --tp 8 \
  --host 0.0.0.0 \
  --port 10000 \
  --enable-metrics \
  --enable-cache-report \
  --page-size 64 \
  --hicache-ratio 2 \
  --hicache-size 0 \
  --hicache-mem-layout page_first_direct \
  --hicache-io-backend direct \
  --hicache-write-policy write_through \
  --hicache-storage-backend hf3fs \
  --hicache-storage-prefetch-policy wait_complete \
  --disaggregation-decode-enable-offload-kvcache \  # Enable async KV cache offloading in decode node
  --disaggregation-ib-device mlx5_0 \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mooncake
```

### Deployment with HF3FS

Here is an example of deploying DeepSeek-R1 with HiCache-HF3FS. For more details, see the [HF3FS Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/hf3fs/docs/README.md).

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path /xxx/DeepSeek-R1/ \
  --log-level info \
  --tp 8 \
  --host 0.0.0.0 \
  --port 10000 \
  --enable-metrics \
  --enable-cache-report \
  --page-size 64 \
  --mem-fraction-static 0.85 \
  --enable-hierarchical-cache \
  --hicache-ratio 2 \
  --hicache-size 0 \
  --hicache-mem-layout page_first_direct \
  --hicache-io-backend direct \
  --hicache-write-policy write_through \
  --hicache-storage-backend hf3fs \
  --hicache-storage-prefetch-policy wait_complete \
```

### Deployment with Mooncake

Here is an example of deploying Qwen3-235B-A22B-Instruct-2507 with Mooncake. For more details, see the [Mooncake Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md).

```bash Command theme={null}
# Set Mooncake environment variables
export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata"
export MOONCAKE_GLOBAL_SEGMENT_SIZE=816043786240
export MOONCAKE_PROTOCOL="rdma"
export MOONCAKE_DEVICE="$DEVICE_LIST"
export MOONCAKE_MASTER=127.0.0.1:50051

# Launch SGLang server with Mooncake backend
python3 -m sglang.launch_server \
  --model-path $MODEL_PATH \
  --tp 8 \
  --page-size 64 \
  --enable-hierarchical-cache \
  --hicache-ratio 2 \
  --hicache-mem-layout page_first_direct \
  --hicache-io-backend direct \
  --hicache-storage-backend mooncake \
  --hicache-write-policy write_through \
  --hicache-storage-prefetch-policy timeout
```

## Custom Storage Backend Integration

To integrate a new storage backend:

1. **Implement three core methods:**
   * `get(key)`: Retrieve value by key
   * `exists(key)`: Check key existence
   * `set(key, value)`: Store key-value pair

2. **Register your backend:** Add your storage backend to the HiCache [BackendFactory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/backend_factory.py#L188)

The HiCache controller handles all scheduling and synchronization automatically.

### Dynamic Backend Loading

Alternatively, you can use dynamic loading to avoid hard-coding your backend in the repository:

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path your-model \
  --enable-hierarchical-cache \
  --hicache-storage-backend dynamic \
  --hicache-storage-backend-extra-config '{"backend_name":"custom_backend_name", "module_path": "your_module_path", "class_name": "YourHiCacheClassName"}'
```

**Configuration Parameters:**

* `--hicache-storage-backend`: Set to `dynamic`
* `--hicache-storage-backend-extra-config`: JSON configuration with:
  * `backend_name`: Custom backend identifier
  * `module_path`: Python module path to your implementation
  * `class_name`: Your HiCache implementation class name
  * `interface_v1`: 0 (disable) or 1 (enable) to control usage of batch\_get\_v1 and batch\_set\_v1 methods

## Community and Support

* **GitHub Issues**: Report bugs and feature requests
* **Slack Channel**: Join community discussions in #sgl-kv-cache-store
* **Documentation**: Refer to storage backend-specific guides

***

*This document will be continuously updated based on community feedback and new features. Contributions and suggestions are welcome!*


# HiCache System Design and Optimization
Source: https://docs.sglang.io/docs/advanced_features/hicache_design



This document provides a comprehensive overview of SGLang HiCache, covering its system architecture, workflow and key components. It also details configuration parameters, optimization techniques, and integration with various L3 storage backends, serving as a complete reference for users and developers to understand and tune HiCache for efficient LLM inference.

## Why and What is HiCache?

In large language model inference, the prefill phase is often time-consuming: input sequences need to be first converted into Key-Value cache (KV cache) for subsequent decoding. When multiple requests share the same prefix, the KV cache for that prefix is identical. By caching and reusing these shared KV caches, redundant computation can be avoided. To address this, SGLang introduced RadixAttention, which leverages idle GPU memory to cache and reuse prefix KV caches, and **HiCache**, which extends this idea to host memory and distributed storage.

Inspired by the classic three-level cache design of modern CPUs, HiCache organizes GPU memory as L1, host memory as L2, and distributed storage as L3. This hierarchy enables HiCache to fully exploit the "idle" storage space of GPUs and CPUs, while integrating distributed cache systems such as Mooncake, 3FS, NIXL, and AIBrix KVCache for global KV cache storage and scheduling. As a result, HiCache significantly expands KV cache capacity while maintaining strong read performance—especially in workloads such as multi-QA and long-context inference, where KV cache reuse is frequent. For detailed benchmark results, see [this blog](https://lmsys.org/blog/2025-09-10-sglang-hicache/).

## System Design

### Overall Architecture

In many modern CPU architectures, the small but fast L1 and L2 caches are private to each core, enabling rapid access to the hottest data, while the larger L3 cache is shared across all cores to significantly reduce redundancy within the cache. Similarly, in HiCache, the L1 and L2 KV caches are private to each inference instance, whereas the L3 KV cache is shared among all inference instances within the cluster.

### Tier Sharing Scope

Which tier is private and which is shared decides where a cache hit can come from, so it is worth stating explicitly:

| Tier | Medium | Scope | Shared across instances? |
| - | - | - | - |
| **L1** | GPU memory (HBM) | One inference instance | No |
| **L2** | Host memory (CPU DRAM) | One inference instance, on its own node | **No** |
| **L3** | Storage backend (`file`, `mooncake`, `hf3fs`, `nixl`, `aibrix`, or a custom one) | Whatever the backend is configured to span | **Only if the backend is set up for it** |

```mermaid theme={null}
flowchart TB
    subgraph N0["Node A"]
        subgraph I0["Instance 0"]
            L1A["L1 - GPU memory<br/>private"]
            L2A["L2 - host memory<br/>private to this instance"]
            L1A --- L2A
        end
        subgraph I1["Instance 1"]
            L1B["L1 - GPU memory<br/>private"]
            L2B["L2 - host memory<br/>private to this instance"]
            L1B --- L2B
        end
    end

    subgraph N1["Node B"]
        subgraph I2["Instance 2"]
            L1C["L1 - GPU memory<br/>private"]
            L2C["L2 - host memory<br/>private to this instance"]
            L1C --- L2C
        end
    end

    L3["L3 - storage backend<br/>shared by the instances the backend is configured to span"]

    L2A --> L3
    L2B --> L3
    L2C --> L3
```

**L2 is node-local and instance-private.** It is host memory owned by one inference instance's process, so two instances never read each other's L2, not even two instances on the same node. A KV cache produced by instance 0 becomes visible to instance 1 only after it reaches L3.

This is the answer to a common question: HiCache cannot pool the host memory of several machines into one larger L2. Growing `--hicache-ratio` or `--hicache-size` only makes each instance's own private L2 larger. Cross-instance reuse is the job of L3, so it needs `--hicache-storage-backend`.

L3 is the tier that *can* be shared, but whether it actually is depends on the backend and how it is configured. The `file` backend writes to `/tmp/hicache` by default (`SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR` overrides it), which is node-local unless that path is a shared mount. Distributed backends such as `mooncake`, `hf3fs`, `nixl` and `aibrix` give cluster-wide scope, but only when every instance is pointed at the same namespace and configuration.

### HiRadixTree: Metadata Organization in HiCache

For KV cache data organization, HiCache builds upon the RadixTree structure introduced in RadixAttention and proposes HiRadixTree. In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory. A path from the root to a leaf node represents the prefix of a request, and shared prefixes across multiple requests can reuse the same nodes, thereby avoiding redundant storage.

HiRadixTree extends this idea: each node corresponds to the KV cache of a span of consecutive tokens and records where that KV cache is stored—whether in local GPU memory, CPU memory, L3 storage, or multiple of these tiers. If stored locally, HiRadixTree maintains precise metadata, including the exact storage address. However, to reduce overhead, HiRadixTree does not store or continuously synchronize metadata for L3 KV cache. Instead, when accessing L3 data, it queries the backend in real time to retrieve the necessary metadata, such as whether the data exists and on which server and location it resides.

### Overall Workflow

The workflow of HiCache mainly involves three key operations: **local match**, **prefetch** and **write-back**. When the system receives a new request, it first searches the local L1 and L2 caches for matching KV caches. For parts not found locally, it attempts to prefetch from L3. After prefetching, all required KV caches are loaded into the GPU for computation. Once the prefill computation is complete, the system considers storing the newly generated data into L2 or L3.

![HiCache Workflow](https://lmsys.org/images/blog/hicache/hicache_overview.png)

### Local Match

Local matching is the first step in HiCache's workflow, where incoming request tokens are matched against the HiRadixTree to locate cached KV data in local memory tiers (L1 GPU memory and L2 host memory).

The matching algorithm traverses the HiRadixTree from the root node, following child nodes that match the token sequence prefix. At each node, the incoming token sequence is compared with the node’s stored token sequence. When `page_size > 1`, matching is performed at the page granularity to optimize memory access patterns. If a match terminates within a node’s stored sequence, the node is automatically split to create an exact boundary, improving the efficiency of future matches.

The algorithm returns a continuous prefix of the request, with the first part residing in L1 and the latter part in L2.

Since the process only requires traversing the local HiRadixTree and does not involve any actual data copying, local matching is extremely fast.

### Prefetch from L3

Data prefetching is one of HiCache’s core optimization techniques, designed to proactively load KV caches from L3 storage into local L2 memory, thereby reducing access latency during subsequent operations.

**Prefetch Trigger Conditions**:
After local matching, for the parts not found in L1 or L2, the system queries L3 to retrieve metadata for the next continuous matching KV caches. If the length of hit cache in L3 exceeds a threshold (default: 256 tokens, configurable), a prefetch operation is triggered.

**Prefetch Strategies**: HiCache provides three different prefetch termination strategies to address different scenario needs:

* **best\_effort**: Terminates immediately when GPU can execute prefill computation, with no waiting time, suitable for scenarios extremely sensitive to latency.
* **wait\_complete**: Must wait for all prefetch operations to complete, suitable for scenarios requiring high cache hit rates.
* **timeout**: Terminates after specified time or when complete, balancing latency and cache hit rate needs.

After prefetching stops, the data already fetched is used together with the local data for the prefill computation.

For **timeout** strategy, HiCache introduces three configuration parameters to support fine-grained control over prefetch timeout conditions:

* `prefetch_timeout_base`: the base timeout, representing overhead unrelated to the number of tokens (e.g., scheduling and synchronization). Default: `2` seconds.
* `prefetch_timeout_per_ki_token`: the incremental timeout per thousand tokens. Default: `0.1` seconds per 1024 tokens.
* `prefetch_timeout_max`: the upper bound applied to the linear timeout, preventing very long prompts from waiting unboundedly. Default: `30` seconds.

The timeout is computed as:

```python Example theme={null}
timeout = min(
    prefetch_timeout_max,
    prefetch_timeout_base + prefetch_timeout_per_ki_token * num_token_to_fetch / 1024,
)
```

### Data Write-back

The write-back mechanism is responsible for moving frequently accessed KV caches from L1 to L2 and L3, enabling larger and longer-term storage as well as cache sharing across instances.

**Configurable Write-back Policies**: HiCache supports three write-back strategies:

* **write\_through**: Every access is immediately written back to the next level. When bandwidth is sufficient, this strategy provides the strongest caching benefit.
* **write\_through\_selective**: Data is written back only after the access frequency exceeds a threshold. This strategy backs up only hot data, reducing I/O overhead.
* **write\_back**: Data is written back to the next level only when it is evicted from the upper level. This strategy alleviates storage pressure and is suitable for scenarios where storage capacity is limited but memory utilization must be maximized.

**Cross-instance Sharing**: When data is written back from L2 to L3, only data not already present in L3 is transferred. KV caches stored in L3 can then be shared across all SGLang instances in the cluster (depending on the L3 backend implementation), significantly improving cache hit rates within the same memory budget.

### Multi-Rank Synchronization

During multi-GPU parallel computation, such as tensor parallelism (TP), HiCache must ensure consistent states across different ranks. Therefore, critical computation steps require the use of `all_reduce` for state synchronization.

For example, during prefetching, `all_reduce(op=min)` is used to ensure that all ranks obtain the same number of L3 hits, preventing inconsistent judgments about whether the prefetch threshold has been reached. Similarly, after prefetching completes or terminates, `all_reduce(op=min)` is again required to guarantee consensus among ranks on the prefix length of the successfully retrieved KV cache.

### Data Transfer Optimization

**Zero-Copy Data Transfers**: Both prefetching and write-back involve substantial data movement. Minimizing the number of data copies can significantly improve system performance. HiCache supports passing memory addresses and sizes directly when transferring data from L2 memory to an L3 backend.

**“Batch-Oriented” Data Organization**: The granularity of data reads and writes has a major impact on performance. To address this, HiCache L3 stores and transfers KV cache data at the granularity of **pages** and supports different data layouts beyond the existing `layer first` scheme, including `page first` and `page first direct`. Under the `page first` and `page first direct` layouts, all KV cache data belonging to the same page is placed in contiguous memory, allowing it to be passed as a single object to L3 using zero-copy transfers.

![HiCache L2 MEM layout](https://lmsys.org/images/blog/hicache/hicache_layout.png)

However, because GPU KV computation is naturally performed layer by layer, the GPU inherently operates in a `layer first` layout. When transferring `page first` data from L2 to the GPU, data must be transferred at the granularity of one token per layer. The `page first direct` layout mitigates this issue by grouping together all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level.

**CPU-to-GPU Transfer Optimizations**: In HiCache, moving data from CPU memory to GPU is as performance-critical as prefetching data from L3 to L2. HiCache employs several optimizations for this process:

* **Compute-Transfer Overlap**: During the prefill phase, when transferring data from CPU to GPU, HiCache overlaps layers by concurrently loading the KV cache of layer N+1 while computing layer N. This effectively hides data transfer latency.
* **GPU-assisted I/O Kernels**: On top of `cudaMemcpyAsync`, HiCache implements a set of GPU-assisted I/O kernels specifically optimized for KV cache transfers between CPU and GPU. Compared to the baseline approach, these kernels achieve up to 3x higher transfer speed.

**Write-back Optimization for MLA**: For MHA (Multi-Head Attention) models under multi-TP, each rank holds `1/tp_size` of a token’s KV data. In contrast, for MLA (Multi-Layer Attention) models, all ranks hold the complete and identical KV data for each token. HiCache includes a dedicated optimization for MLA: only one rank initiates the write-back operation, ensuring that data is not redundantly stored across ranks.

### Integration with PD-Disaggregation Deployment Mode

SGLang supports a PD (Prefill-Decode) disaggregation deployment mode through the Mooncake TransferEngine (for details, see [this doc](./pd_disaggregation)). In the PD-disaggregation deployment mode, HiCache can be enabled on both the prefill nodes and decode nodes to optimize prefill performance. If enabled on decode nodes, the decode output will also be written back to L3.

### Unified Interfaces and Rich L3 Storage Backends

HiCache encapsulates all read, write, and query operations on L3 backends within the `class HiCacheStorage(ABC)`, exposing a set of simple and consistent interfaces. This design supports a wide range of L3 storage backends and allows users to select the one that best fits their specific use cases.

When a backend is attached, HiCache hands it the host memory pools in a fixed order: `register_mem_pool_host` receives the anchor pool first, then every pool of the host pool group (Mamba state, indexer, SWA, draft sidecars, ...) arrives through `register_mem_host_pool_v2`, and `on_host_pools_registered` fires exactly once when this initial burst is over. Pools attached dynamically later still arrive through `register_mem_host_pool_v2` alone. Backends that talk to an external service whose instance specification must be complete up front (e.g. a remote cache manager) should finalize it inside `on_host_pools_registered`; raising there fails the attach.

* **Mooncake**: Mooncake is a high-performance caching system for LLM inference that leverages RDMA and multi-NIC resources to enable zero-copy, ultra-fast data transfers. Try Mooncake [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/mooncake_store).

* **DeepSeek 3FS (HF3FS)**: HF3FS is a Kubernetes-native distributed storage solution with operator-based deployment. Try HF3FS [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs).

* **NIXL**: NIXL provides a unified API for accessing various storage plugins, including but not limited to DeepSeek's 3FS, GPU Direct Storage (GDS) and Amazon S3-compatible object storage. Try NIXL [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/nixl).

* **AIBrix KVCache**: AIBrix KVCache is a production-ready KVCache Offloading Framework, which enables efficient memory tiering and low-overhead cross-engine reuse. Try AIBrix KVCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/aibrix_kvcache).

* **HiCacheFile**: A simple file-based storage backend for demonstration purposes.

Specifically, **LMCache**, an efficient KV cache layer for enterprise-scale LLM inference, provides an alternative solution to HiCache. Try LMCache [here](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/lmcache).

## Related Parameters

* **`--enable-hierarchical-cache`**: Enable hierarchical cache functionality. This is required to use HiCache.

* **`--hicache-ratio HICACHE_RATIO`**: The ratio of the size of host KV cache memory pool to the size of device pool. For example, a value of 2 means the host memory pool is twice as large as the device memory pool. The value of this parameter must be greater than 1, as the current implementation requires the host memory allocated for the KV cache to be larger than the device memory allocated for the KV cache.

* **`--hicache-size HICACHE_SIZE`**: The size of host KV cache memory pool in gigabytes. This parameter overrides `hicache-ratio` if set. For example, `--hicache-size 30` allocates 30GB (1GB = 1e9 bytes) for the host memory pool **for each rank**. If there are 8 ranks, then the total memory size is 240GB. Just like `hicache-ratio`, the value of this parameter must be larger than the size of device memory allocated for KV cache.

**Note**: `--hicache-ratio` and `--hicache-size` are two critical parameters. In general, a larger HiCache size leads to a higher cache hit rate, which improves prefill performance. However, the relationship between cache size and hit rate is not linear. Once most reusable KV data—especially hot tokens—are already cached, further increasing the size may yield only marginal performance gains. Users can set these parameters based on their workload characteristics and performance requirements.

* **`--page-size PAGE_SIZE`**: The number of tokens per page. This parameter determines the granularity of KV cache storage and retrieval. Larger page sizes reduce metadata overhead and improve I/O efficiency for storage backends, but may lower the cache hit rate when only part of a page matches the stored KV cache. For workloads with long common prefixes, larger pages can improve performance, while workloads with more diverse prefixes may benefit from smaller pages. See [Data Transfer Optimization](#data-transfer-optimization) for how page granularity affects I/O performance.

* **`--hicache-storage-prefetch-policy {best_effort,wait_complete,timeout}`**: Controls when prefetching from storage should stop. See [Prefetch from L3](#prefetch-from-l3) for details.
  * `best_effort`: Prefetch as much as possible without blocking
  * `wait_complete`: Wait for prefetch to complete before proceeding
  * `timeout`: Terminates after specified time or when complete (Recommended for production environments, as setting an appropriate timeout helps the system meet required SLOs)

* **`--hicache-write-policy {write_back,write_through,write_through_selective}`**: Controls how data is written from faster to slower memory tiers. See [Data Write-back](#data-write-back) for details.
  * `write_through`: Immediately writes data to all tiers (strongest caching benefits)
  * `write_through_selective`: Uses hit-count tracking to back up only frequently accessed data
  * `write_back`: Writes data back to slower tiers only when eviction is needed (reduces I/O load)

* **`--hicache-io-backend {direct,kernel}`**: Choose the I/O backend for KV cache transfer between CPU and GPU. See [Data Transfer Optimization](#data-transfer-optimization) for details.
  * `direct`: Standard CUDA memory copy operations
  * `kernel`: GPU-assisted I/O kernels (recommended for better performance)

* **`--hicache-mem-layout {layer_first,page_first,page_first_direct}`**: Memory layout for the host memory pool. See [Data Transfer Optimization](#data-transfer-optimization) for details.
  * `layer_first`: Compatible with GPU computation kernels (default for GPU memory)
  * `page_first`: Optimized for I/O efficiency
  * `page_first_direct`: Groups all tokens of a given layer within a page, allowing transfers from L2 to GPU to be aggregated at the page-layer level

* **`--hicache-storage-backend {file,mooncake,hf3fs,nixl,aibrix,dynamic}`**: Choose the storage backend for the L3 tier. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: `backend_name` (custom name), `module_path` (Python module path), `class_name` (backend class name). See [Unified Interfaces and Rich L3 Storage Backends](#unified-interfaces-and-rich-l3-storage-backends) for available backends.

* **`--enable-lmcache`**: Using LMCache as an alternative hierarchical cache solution.

* **`--lmcache-config-file`**: Path to the LMCache YAML configuration file.

* **`--hicache-storage-backend-extra-config HICACHE_STORAGE_BACKEND_EXTRA_CONFIG`**: the extra config can be either
  * a JSON string containing extra configuration for the storage backend, e.g., `--hicache-storage-backend-extra-config '{"prefetch_threshold":512, "prefetch_timeout_base": 0.5, "prefetch_timeout_per_ki_token": 0.25}' `, or
  * a TOML or JSON or YAML file specifying the extra configuration for the storage backend (to differentiate from the JSON string input, prepend a `@` in front of the file name), e.g., `--hicache-storage-backend-extra-config "@config.toml"` where `config.toml` is the config file containing the complex configurations. This can be useful when the configuration consists of many or complex key-value pairs (for instance, it is preferred to use a config file for NIXL backend as its configurations can be complex).


# Runtime Attach/Detach HiCache Storage Backend (No Restart)
Source: https://docs.sglang.io/docs/advanced_features/hicache_storage_runtime_attach_detach



This document explains how to **dynamically attach/detach the HiCache L3 storage backend at runtime** (e.g., `mooncake` / `hf3fs` / `nixl` / `file` / `aibrix` / `eic`) while **SGLang is already running and serving traffic**, without restarting the process.

For safety and consistency, the current implementation **strictly requires** these operations to happen only when the service is **idle**:

* **No running requests**
* **No waiting/queued requests**

If the idle condition is not met, the API will fail fast (HTTP 400) and **will not modify** the current service state.

***

## 1. Background and implementation overview

### 1.1 Architecture / control path

The control path is:

1. **HTTP Server** (`python/sglang/srt/entrypoints/http_server.py`)
   * Exposes `PUT /hicache/storage-backend`, `DELETE /hicache/storage-backend`, `GET /hicache/storage-backend`
2. **TokenizerManager** (`python/sglang/srt/managers/tokenizer_control_mixin.py`)
   * Sends the request to the Scheduler via `FanOutCommunicator`
3. **Scheduler** (`python/sglang/srt/managers/scheduler.py`)
   * Performs a **strict idle check**
   * Calls `tree_cache.attach_storage_backend(...)` / `detach_storage_backend(...)`
4. **UnifiedRadixCache** (`python/sglang/srt/mem_cache/unified_radix_cache.py`)
   * Parses `hicache_storage_backend_extra_config_json` (supports both backend config and prefetch knobs)
   * Calls `cache_controller.attach_storage_backend(...)` / `detach_storage_backend(...)`
5. **HiCacheController** (`python/sglang/srt/managers/cache_controller.py`)
   * Creates/destroys the storage backend instance (via `StorageBackendFactory`)
   * Starts/stops backend background threads at runtime (prefetch/backup)

***

## 2. Idle-state requirement (strict)

The Scheduler uses `is_fully_idle()` which checks:

* No running batches (including chunked prefill, overlap, pipeline-parallel, and disaggregation paths)
* No waiting requests in any queue (waiting, grammar, disagg bootstrap/prealloc/transfer/inflight)
* No DLLM staging requests

If the condition is not met, attach/detach returns an error like:

* `Reject attach: scheduler is not idle. #queue-req=... #running-req=...`

> [!TIP]
>
> before switching, drain upstream traffic and wait for the server to become idle, then call attach/detach.

### 2.1 DP (data parallel) semantics

When `dp_size > 1`, the tokenizer dispatches the request to **all DP scheduler instances** and aggregates their responses:

* The final `success` is **true only if all DP ranks return success**
* The final `message` concatenates messages from all DP ranks

This is intended to prevent “silent partial success”, but it also means you may see:

* Overall **failure** even though **some ranks already succeeded**

Currently there is **no automatic partial rollback** across DP ranks (see TODO in code). Operationally:

* Prefer to keep backend config identical across ranks
* If attach fails, immediately call detach (best-effort/idempotent), fix config, then retry attach

***

## 3. How to use (HTTP Admin API)

The examples below assume your SGLang HTTP server is at `http://127.0.0.1:30000`.

### 3.1 Query current storage backend status

```bash Command theme={null}
curl -s http://127.0.0.1:30000/hicache/storage-backend
```

Example response:

```json Config theme={null}
{
  "hicache_storage_backend": "mooncake",
  "hicache_storage_backend_extra_config": "{\"master_server_address\":\"127.0.0.1:50051\", ...}"
}
```

### 3.2 Attach (enable) a storage backend

```bash Command theme={null}
curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \
  -H 'Content-Type: application/json' \
  -d '{
    "hicache_storage_backend": "mooncake"
  }'
```

```bash Command theme={null}
curl -s -X PUT http://127.0.0.1:30000/hicache/storage-backend \
  -H 'Content-Type: application/json' \
  -d '{
    "hicache_storage_backend": "mooncake",
    "hicache_storage_backend_extra_config_json": "{\"master_server_address\":\"127.0.0.1:50051\",\"protocol\":\"tcp\",\"global_segment_size\":\"4gb\",\"prefetch_threshold\":256}",
    "hicache_storage_prefetch_policy": "timeout"
  }'
```

Notes:

* `hicache_storage_backend_extra_config_json` can include both:
  * **Backend configuration** (e.g., Mooncake master/metadata/protocol, etc.)
  * **Prefetch configuration** (`prefetch_threshold`, `prefetch_timeout_base`, `prefetch_timeout_per_ki_token`, `prefetch_timeout_max`, `hicache_storage_pass_prefix_keys`)

### 3.3 Detach (disable) the storage backend

```bash Command theme={null}
curl -s -X DELETE http://127.0.0.1:30000/hicache/storage-backend
```

Notes:

* Detach only makes SGLang **stop using** the L3 storage backend and stops prefetch/backup threads
* It **does not automatically delete** data stored in Mooncake/HF3FS (or other remote backends)

***

## 4. Behavior and caveats

* **No restart required**: attach/detach switches in-process at runtime
* **Must be idle**: otherwise the request is rejected to avoid consistency issues
* **Host KV layout constraints still apply**: for example, Mooncake still requires layouts like `page_first/page_first_direct/page_head`; if the server's HiCache host-memory layout does not satisfy the backend requirements, attach will fail with an error
* **Observability**:
  * After attach, `server_args.hicache_storage_backend*` is updated on both the tokenizer and scheduler sides
  * If metrics are enabled, attach will create a storage metrics collector in `UnifiedRadixCache` on demand


# HiSparse: Hierarchical Sparse Attention
Source: https://docs.sglang.io/docs/advanced_features/hisparse_guide



HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency.

> **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1), **DeepSeek V4**, and **MiniMax M3**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only.

## Why HiSparse?

In long-context LLM inference, each decoding request holds a full-length KV cache on GPU, limiting the number of concurrent requests a decode instance can serve. HiSparse addresses this by:

* **Reducing GPU memory per request**: Each request occupies only a fixed-size device buffer (e.g., 4KB tokens) instead of the full sequence length.
* **On-demand swap-in**: A CUDA kernel dynamically loads the top-k most relevant KV entries from host memory based on attention scores.
* **Transparent to prefill**: HiSparse is entirely a decode-side optimization; the prefill instance requires no changes.

## Design Overview

### Decode Workflow

Each decode step follows this flow:

1. **Forward decode** — generate the next token
2. **Top-k selection** — select the most relevant token positions via attention scores
3. **Swap-in** — the CUDA kernel loads top-k KV entries from host to device buffer:
   * *Short sequences* (`seq_len ≤ device_buffer_size`): fast path, all KV already in buffer
   * *Long sequences*: hit detection → LRU reordering → miss handling (host → device copy)
4. **Decode attention** — compute attention using the top-k device locations
5. **Eager backup** — asynchronously copy the previous token's KV from device to host

### PD Disaggregation Integration (Direct-to-Host)

In PD disaggregation mode, the prefill instance transfers KV cache directly into the decode instance's host pool via RDMA, bypassing the GPU entirely on the decode side. This eliminates the transient GPU memory spike during KV transfer and removes the staging DMA step.

```
Prefill GPU  ──RDMA──▶  Decode Host Pool (CPU pinned memory)
                              │
                              ▼
                     alloc device buffer (4KB)
                              │
                              ▼
                     swap-in kernel (on-demand top-k)
```

For DeepSeek V4, the direct-to-host path writes only C4 KV into the decode host pool. The c4\_indexer and C128 KV remain device-to-device transfers.

## Server Arguments

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Type / Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-hisparse`</td>
      <td>flag; default: disabled</td>
      <td>Enable HiSparse on the decode instance</td>
    </tr>

    <tr>
      <td>`--hisparse-config`</td>
      <td>JSON string</td>
      <td>Configuration for HiSparse (see below)</td>
    </tr>
  </tbody>
</table>

### HiSparse Config Parameters

Pass as a JSON string via `--hisparse-config`:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type / Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`top_k`</td>
      <td>int</td>
      <td>Number of topk entries</td>
    </tr>

    <tr>
      <td>`device_buffer_size`</td>
      <td>int</td>
      <td>Number of token slots in the per-request GPU device buffer</td>
    </tr>

    <tr>
      <td>`host_to_device_ratio`</td>
      <td>int</td>
      <td>Ratio of logical pool size to device pool size, determining host memory capacity</td>
    </tr>

    <tr>
      <td>`swap_in_block_size`</td>
      <td>int / 960</td>
      <td>CUDA thread-block size for the HiSparse swap-in kernel</td>
    </tr>
  </tbody>
</table>

Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'`

### Shared-index prefetch (automatic)

When a model reuses one anchor layer's top-k selection across a run of subsequent "skip" layers (DSA `index_topk_freq` / `index_topk_pattern`; native in GLM-5.2 as IndexShare), the working set of every skip layer is known the moment the anchor's index is computed. HiSparse exploits this automatically: the anchor's swap-in kernel records its miss plan (which host slots go to which device-buffer slots), and each skip layer replays that plan with a copy-only kernel issued ahead on a side stream, so the skip layers' host→device IO overlaps the intervening layers' compute instead of sitting on the decode critical path. The replay kernel uses a small fixed grid to keep its SM footprint low while overlapped.

The prefetch is enabled automatically for eligible models (no pipeline parallelism, no speculative decoding) and can be turned off for A/B comparison with `SGLANG_DISABLE_HISPARSE_PREFETCH=1`.

## Deployment

HiSparse currently requires **PD disaggregation mode** and is enabled only on the **decode instance**.

### Prefill Instance

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path /path/to/model \
    --trust-remote-code \
    --port 8000 --host 0.0.0.0 \
    --context-length 81920 \
    --chunked-prefill-size 65536 \
    --tp-size 8 --attn-dp-size 8 \
    --mem-fraction-static 0.85 \
    --disaggregation-mode prefill \
    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
    --nnodes 1 --node-rank 0
```

### Decode Instance (with HiSparse)

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path /path/to/model \
    --trust-remote-code \
    --port 8000 --host 0.0.0.0 \
    --context-length 81920 \
    --tp-size 8 --attn-dp-size 8 \
    --mem-fraction-static 0.85 \
    --disable-radix-cache \
    --disaggregation-mode decode \
    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
    --dist-init-addr 127.0.0.1:5757 \
    --nnodes 1 --node-rank 0 \
    --enable-hisparse \
    --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10, "swap_in_block_size": 960}'
```

> **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`), except for GLM DSA models on SM120/SM121 with `fp8_e4m3`, which use `flashinfer_sparse_mla`. DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend.

### MiniMax M3

Dense-layer K/V and index K stay on GPU; sparse-layer K/V use host memory plus a GPU working set.

* Use TP 4 or greater, with the same TP size and PP 1 on both PD instances.
* Use `--attention-backend triton`, `--mm-attention-backend triton_attn`, `--disable-prefill-cuda-graph`, and `--disable-radix-cache`.
* Set `device_buffer_size` to at least 2048 in `--hisparse-config`; `top_k` does not override the model's selection width.
* PD retraction backup is not supported. Use `--num-reserved-decode-tokens` to reserve capacity for the expected output length.

### Benchmark

```bash Command theme={null}
python3 -m sglang.bench_serving \
    --backend sglang \
    --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
    --dataset-name random \
    --random-input 40000 \
    --random-output 20000 \
    --num-prompts 200 \
    --max-concurrency 200 \
    --request-rate 40 \
    --random-range-ratio 1.0 \
    --host 127.0.0.1 \
    --port 20000 \
    --model /path/to/model \
    --flush-cache \
```

### Key Notes

* The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse.
* On the decode instance, `--enable-hisparse` and `--hisparse-config` are required for HiSparse.
* For DSA models, `--kv-cache-dtype bfloat16` uses `flashmla_sparse`, and `--kv-cache-dtype fp8_e4m3` uses `flashmla_kv`.
* On SM120/SM121 (e.g. RTX PRO 6000, RTX 5090) with GLM DSA models and `--kv-cache-dtype fp8_e4m3`, both DSA backends resolve to `flashinfer_sparse_mla`, which is the only DSA kernel available on that architecture. HiSparse accepts it there; no extra flag is needed.
* For DeepSeek V4, DSA backend flags are not applicable. DeepSeek V4 uses the `dsv4` attention backend and `fp8_e4m3` KV cache by default.
* `host_to_device_ratio` should be configured based on the host machine's available memory. For example:
  * **\~1 TB** host memory → `host_to_device_ratio: 5`
  * **\~2 TB** host memory → `host_to_device_ratio: 10`

## Acknowledgments

We would like to thank the SGLang team and community for the implementation and generous support, especially Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Shangming Cai, Teng Ma, and many others. We also thank the Alibaba Cloud TairKVCache team and the AntGroup SCT Inference team for their valuable contributions.


# Hyperparameter Tuning
Source: https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning



## Achieving high throughput for offline batch inference

Achieving a large batch size is the most important thing for attaining high throughput in offline batch inference.
When the server is running at full load in a steady state, look for the following in the log:

```text Output theme={null}
Decode batch. #running-req: 233, #token: 370959, token usage: 0.82, cuda graph: True, gen throughput (token/s): 4594.01, #queue-req: 317
```

### Adjust the request submission speed to control `#queue-req`

`#queue-req` indicates the number of requests in the queue.
If you frequently see `#queue-req: 0`, it suggests that your client code is submitting requests too slowly.
A healthy range for `#queue-req` is `100 - 2000`.
However, avoid making `#queue-req` too large, as this will increase the scheduling overhead on the server.

### Achieve a high `token usage`

`token usage` indicates the KV cache memory utilization of the server. `token usage > 0.9` means good utilization.

If you frequently see `token usage < 0.9` and `#queue-req > 0`, it means the server is too conservative about taking in new requests. You can decrease `--schedule-conservativeness` to a value like 0.3.
The case of a server being too conservative can happen when users send many requests with a large `max_new_tokens` but the requests stop very early due to EOS or stop strings.

On the other hand, if you see `token usage` very high and you frequently see warnings like
`KV cache pool is full. Retract requests. #retracted_reqs: 1, #new_token_ratio: 0.9998 -> 1.0000`, you can increase `--schedule-conservativeness` to a value like 1.3.
If you see `KV cache pool is full. Retract requests.` occasionally but not frequently (\~1 time per minute), it is okay.

### Tune `--mem-fraction-static` to increase KV cache pool capacity

SGLang allocates memory as follows:

Total memory usage = model weights + KV cache pool + CUDA graph buffers + activations

The `--mem-fraction-static` parameter determines how much memory is allocated to the first two components:

mem\_fraction\_static = (model weights + KV cache pool) / GPU memory capacity

To support higher concurrency, you should maximize the KV cache pool capacity by setting `--mem-fraction-static` as high as possible while still reserving enough memory for activations and CUDA graph buffers.

SGLang uses simple heuristics to set the default value of `--mem-fraction-static`, but you can optimize it for your use cases.
As a rule of thumb, reserving 5–8 GB of memory for activations is typically sufficient. You can check this by inspecting the logs just before the server is ready.
Look for log entries like this:

```text Output theme={null}
[2025-08-11 17:17:03] max_total_num_tokens=665690, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4096, context_len=65536, available_gpu_mem=13.50 GB
```

Check the `available_gpu_mem` value.

* If it is between 5–8 GB, the setting is good.
* If it is too high (e.g., 10 - 20 GB), increase `--mem-fraction-static` to allocate more memory to the KV cache.
* If it is too low, you risk out-of-memory (OOM) errors later, so decrease `--mem-fraction-static`.

Another straightforward approach is to increase `--mem-fraction-static` in increments of 0.01 until you encounter OOM errors for your workloads.

### Avoid out-of-memory errors by tuning `--chunked-prefill-size`, `--mem-fraction-static`, and `--max-running-requests`

If you encounter out-of-memory (OOM) errors, you can adjust the following parameters:

* If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts.
* If OOM occurs during decoding, try lowering `--max-running-requests`.
* You can also reduce `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.

### Tune `--cuda-graph-max-bs-decode`

By default, CUDA graph is enabled only for small batch sizes (e.g., less than 160 or 256).
However, for some models, especially at large tensor parallelism sizes, CUDA graph can be useful for batch sizes up to 512 or 768.
Therefore, it may be beneficial to increase `--cuda-graph-max-bs-decode` to a larger value.
Note that CUDA graph consumes more memory, so you may need to reduce `--mem-fraction-static` at the same time.

### Tune `--dp-size` and `--tp-size`

Data parallelism is better for throughput. When there is enough GPU memory, always favor data parallelism for throughput. Refer to [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for a better data parallelism rather than using `dp_size` parameter.

### Try other options

* `torch.compile` accelerates small models on small batch sizes. You can enable it with `--enable-torch-compile`.
* Try other quantization (e.g. FP8 quantization with `--quantization fp8`)
* Try other parallelism strategies (e.g. [expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) or DP attention for deepseek models (with `--attn-dp-size 8`).
* If the workload has many shared prefixes, try `--schedule-policy lpm`. Here, `lpm` stands for longest prefix match. It reorders requests to encourage more cache hits but introduces more scheduling overhead.


# llm-d
Source: https://docs.sglang.io/docs/advanced_features/llm-d



[llm-d](https://llm-d.ai/) is a Kubernetes-native distributed inference framework for serving large language models at scale across a fleet of inference servers. SGLang is a supported inference engine in llm-d: llm-d coordinates a fleet of SGLang instances across a cluster so that performance holds up under real production traffic, achieving the fastest "time to state-of-the-art (SOTA) performance" for key OSS models across most hardware accelerators.

llm-d is a [CNCF Sandbox project](https://www.cncf.io/blog/2026/03/24/welcome-llm-d-to-the-cncf-evolving-kubernetes-into-sota-ai-infrastructure/) founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.

## What llm-d adds to an SGLang deployment

A single SGLang server is fast, and RadixAttention already maximizes cache reuse within each replica. But at scale the picture changes: across many replicas, cache locality breaks under round-robin load balancing as related requests scatter and radix-cache hit rates collapse, long prompts inflate time-to-first-token, and accelerators sit underused. llm-d adds the cluster-level layer that the engine does not aim to provide on its own:

* **[Prefix-aware routing](https://llm-d.ai/docs/guides/precise-prefix-cache-aware).** Instead of round-robin, the llm-d Router scores each replica on prefix-cache locality and current load, routing each request to the replica most likely to already hold its prefix — raising RadixAttention hit rates on multi-turn and shared-prefix workloads while avoiding saturated servers.
* **[Distributed KV-cache management](https://llm-d.ai/docs/guides#advanced-kv-cache-management).** A global index tracks which token blocks live on which replica, and tiered offloading spills cache to CPU memory or local SSD, extending the working set beyond accelerator HBM.
* **[Prefill/decode disaggregation](https://llm-d.ai/docs/guides/pd-disaggregation).** Prompt processing and token generation run on separate workers, with KV-cache moved over high-speed interconnects, lowering TTFT and steadying per-token latency on long prompts.
* **[SLO-aware autoscaling](https://llm-d.ai/docs/guides/workload-autoscaling) and [flow control](https://llm-d.ai/docs/guides/flow-control).** Scale SGLang pools on real inference signals (queue depth, true demand) rather than raw GPU utilization, with multi-tenant fairness and priority dispatch.
* **One control plane for mixed fleets.** llm-d schedules across engines, so platform teams can serve SGLang and vLLM pools behind the same gateway, policies, and observability instead of running parallel stacks.

These capabilities are composable. Most teams start by adding prefix-aware routing over an existing SGLang pool, then layer in the rest as specific bottlenecks appear.

## Kubernetes-native gateway

llm-d builds on the Gateway API Inference Extension, so SGLang pools are managed through standard Kubernetes resources (Gateway, HTTPRoute, InferencePool) and work with supported gateway providers such as Istio, GKE Inference Gateway, and agentgateway, rather than a bespoke routing tier.

## Performance

llm-d publishes reproducible benchmarks from production-scale deployments on [Prism](https://prism.llm-d.ai/). One representative result: prefix-aware routing delivered **3x higher output throughput and 2x faster TTFT** than round-robin load balancing (Llama 3.1 70B). The mechanism carries over directly to SGLang, where RadixAttention makes the cluster-level cache hit rate a function of routing.

## Get started

* **Deploy the optimized baseline with the [Quickstart](https://llm-d.ai/docs/getting-started/quickstart), selecting SGLang as the inference server.** It stands up an intelligent router (the llm-d Router) over an SGLang pool on Kubernetes in a tested configuration.
* Browse the [well-lit path guides](https://llm-d.ai/docs/guides), each a tested recipe for one of the capabilities above, and add the optimization that fits your workload.
* Read the [Introduction](https://llm-d.ai/docs/getting-started) and [Architecture overview](https://llm-d.ai/docs/architecture) to see how the scheduler, gateway, and model servers wrap your SGLang deployment.

Questions and contributions are welcome on [GitHub](https://github.com/llm-d/llm-d) and [Slack](https://llm-d.ai/slack).

## Current scope

SGLang is supported across the well-lit paths — including intelligent inference scheduling, precise prefix-cache routing (SGLang publishes KV-cache events that the llm-d Router subscribes to), tiered KV-cache management, prefill/decode disaggregation, flow control, and autoscaling. The one current exception is Multi-Node Wide Expert Parallelism, which is vLLM-specific today. See the llm-d documentation for the latest per-engine support status.


# LoRA Serving
Source: https://docs.sglang.io/docs/advanced_features/lora



SGLang enables the use of [LoRA adapters](https://arxiv.org/abs/2106.09685) with a base model. By incorporating techniques from [S-LoRA](https://arxiv.org/pdf/2311.03285) and [Punica](https://arxiv.org/pdf/2310.18547), SGLang can efficiently support multiple LoRA adapters for different sequences within a single batch of inputs.

## Arguments for LoRA Serving

The following server arguments are relevant for multi-LoRA serving:

* `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.

* `enable_lora_overlap_loading`: Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.

* `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: \<PATH> | \<NAME>=\<PATH> | JSON with schema \{"lora\_name":str,"lora\_path":str,"pinned":bool}.

* `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.

* `max_loaded_loras`: If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `max-loras-per-batch`.

* `lora_eviction_policy`: LoRA adapter eviction policy when GPU memory pool is full. `lru`: Least Recently Used (default, better cache efficiency). `fifo`: First-In-First-Out.

* `lora_backend`: The backend of running GEMM kernels for Lora modules. Currently we support Triton LoRA backend (`triton`) and Chunked SGMV backend (`csgmv`). In the future, faster backend built upon Cutlass or Cuda kernels will be added.

* `max_lora_rank`: The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.

* `lora_target_modules`: The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of different target modules after server startup. You can also set it to `all` to enable LoRA for all supported modules. However, enabling LoRA on additional modules introduces a minor performance overhead. If your application is performance-sensitive, we recommend only specifying the modules for which you plan to load adapters.

* `max_lora_chunk_size`: Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance. Please tune this value based on your hardware and workload as needed. Defaults to 16.

* `lora_drain_wait_threshold`: When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default).

* `tp_size`: LoRA serving along with Tensor Parallelism is supported by SGLang. `tp_size` controls the number of GPUs for tensor parallelism. More details on the tensor sharding strategy can be found in [S-Lora](https://arxiv.org/pdf/2311.03285) paper.

From client side, the user needs to provide a list of strings as input batch, and a list of adaptor names that each input sequence corresponds to.

## Usage

### Serving Single Adaptor

**Note:** SGLang supports LoRA adapters through two APIs:

1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions#using-lora-adapters) for examples.

2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below).

```python Example theme={null}
import json
import requests

from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, terminate_process
```

```python Example theme={null}
server_process, port = launch_server_cmd(
    # Here we set max-loras-per-batch to 2: one slot for adaptor and another one for base model
    """
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
    --max-loras-per-batch 2 \
    --log-level warning \
"""
)

wait_for_server(f"http://localhost:{port}")
```

```python Example theme={null}
url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "List 3 countries and their capitals.",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses the base model
    "lora_path": ["lora0", None],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output 0: {response.json()[0]['text']}")
print(f"Output 1: {response.json()[1]['text']}")
```

```python Example theme={null}
terminate_process(server_process)
```

### Serving Multiple Adaptors

```python Example theme={null}
server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \
    --max-loras-per-batch 2 \
    --log-level warning \
"""
)

wait_for_server(f"http://localhost:{port}")
```

```python Example theme={null}
url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "List 3 countries and their capitals.",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses lora1
    "lora_path": ["lora0", "lora1"],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output 0: {response.json()[0]['text']}")
print(f"Output 1: {response.json()[1]['text']}")
```

```python Example theme={null}
terminate_process(server_process)
```

### Dynamic LoRA loading

Instead of specifying all adapters during server startup via `--lora-paths`. You can also load & unload LoRA adapters dynamically via the `/load_lora_adapter` and `/unload_lora_adapter` API.

When using dynamic LoRA loading, it's recommended to explicitly specify both `--max-lora-rank` and `--lora-target-modules` at startup. For backward compatibility, SGLang will infer these values from `--lora-paths` if they are not explicitly provided. However, in that case, you would have to ensure that all dynamically loaded adapters share the same shape (rank and target modules) as those in the initial `--lora-paths` or are strictly "smaller".

```python Example theme={null}
lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj
lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"  # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
lora0_new = "philschmid/code-llama-3-1-8b-text-to-sql-lora"  # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj


# The `--target-lora-modules` param below is technically not needed, as the server will infer it from lora0 which already has all the target modules specified.
# We are adding it here just to demonstrate usage.
server_process, port = launch_server_cmd(
    """
    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --cuda-graph-max-bs-decode 2 \
    --max-loras-per-batch 2 \
    --max-lora-rank 256
    --lora-target-modules all
    --log-level warning
    """
)

url = f"http://127.0.0.1:{port}"
wait_for_server(url)
```

Load adapter lora0

```python Example theme={null}
response = requests.post(
    url + "/load_lora_adapter",
    json={
        "lora_name": "lora0",
        "lora_path": lora0,
    },
)

if response.status_code == 200:
    print("LoRA adapter loaded successfully.", response.json())
else:
    print("Failed to load LoRA adapter.", response.json())
```

Load adapter lora1:

```python Example theme={null}
response = requests.post(
    url + "/load_lora_adapter",
    json={
        "lora_name": "lora1",
        "lora_path": lora1,
    },
)

if response.status_code == 200:
    print("LoRA adapter loaded successfully.", response.json())
else:
    print("Failed to load LoRA adapter.", response.json())
```

Check inference output:

```python Example theme={null}
url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "List 3 countries and their capitals.",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses lora1
    "lora_path": ["lora0", "lora1"],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output from lora0: \n{response.json()[0]['text']}\n")
print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n")
```

Unload lora0 and replace it with a different adapter:

```python Example theme={null}
response = requests.post(
    url + "/unload_lora_adapter",
    json={
        "lora_name": "lora0",
    },
)

response = requests.post(
    url + "/load_lora_adapter",
    json={
        "lora_name": "lora0",
        "lora_path": lora0_new,
    },
)

if response.status_code == 200:
    print("LoRA adapter loaded successfully.", response.json())
else:
    print("Failed to load LoRA adapter.", response.json())
```

Check output again:

```python Example theme={null}
url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "List 3 countries and their capitals.",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses lora1
    "lora_path": ["lora0", "lora1"],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output from lora0: \n{response.json()[0]['text']}\n")
print(f"Output from lora1 (updated): \n{response.json()[1]['text']}\n")
```

```python Example theme={null}
terminate_process(server_process)
```

### OpenAI-compatible API usage

You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai\_api\_completions](../basic_usage/openai_api_completions).

### LoRA GPU Pinning

Another advanced option is to specify adapters as `pinned` during loading. When an adapter is pinned, it is permanently assigned to one of the available GPU pool slots (as configured by `--max-loras-per-batch`) and will not be evicted from GPU memory during runtime. Instead, it remains resident until it is explicitly unloaded.

This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations.

In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported.

```python Example theme={null}
server_process, port = launch_server_cmd(
    """
    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --cuda-graph-max-bs-decode 8 \
    --max-loras-per-batch 3 \
    --max-lora-rank 256 \
    --lora-target-modules all \
    --lora-paths \
        {"lora_name":"lora0","lora_path":"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json","pinned":true} \
        {"lora_name":"lora1","lora_path":"algoprog/fact-generation-llama-3.1-8b-instruct-lora"} \
        lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora
    --log-level warning
    """
)


url = f"http://127.0.0.1:{port}"
wait_for_server(url)
```

You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter:

```python Example theme={null}
response = requests.post(
    url + "/unload_lora_adapter",
    json={
        "lora_name": "lora1",
    },
)

response = requests.post(
    url + "/load_lora_adapter",
    json={
        "lora_name": "lora1",
        "lora_path": "algoprog/fact-generation-llama-3.1-8b-instruct-lora",
        "pinned": True,  # Pin the adapter to GPU
    },
)
```

Verify that the results are expected:

```python Example theme={null}
url = f"http://127.0.0.1:{port}"
json_data = {
    "text": [
        "List 3 countries and their capitals.",
        "List 3 countries and their capitals.",
        "List 3 countries and their capitals.",
    ],
    "sampling_params": {"max_new_tokens": 32, "temperature": 0},
    # The first input uses lora0, and the second input uses lora1
    "lora_path": ["lora0", "lora1", "lora2"],
}
response = requests.post(
    url + "/generate",
    json=json_data,
)
print(f"Output from lora0 (pinned): \n{response.json()[0]['text']}\n")
print(f"Output from lora1 (pinned): \n{response.json()[1]['text']}\n")
print(f"Output from lora2 (not pinned): \n{response.json()[2]['text']}\n")
```

```python Example theme={null}
terminate_process(server_process)
```

## Choosing LoRA Backend

SGLang supports two LoRA backends that you can choose from using the `--lora-backend` argument:

* `triton`: Basic Triton-based backend.
* `csgmv`: Default chunked SGMV backend optimized for high concurrency scenarios.

The `csgmv` backend was recently introduced to improve performance especially at high-concurrency scenarios. Our benchmark shows that it achieves 20% to 80% latency improvements over the basic triton backend.

```python Example theme={null}
server_process, port = launch_server_cmd(
    """
    python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --lora-backend csgmv \
    --max-loras-per-batch 16 \
    --lora-paths lora1=path/to/lora1 lora2=path/to/lora2
    """
)
```

```python Example theme={null}
terminate_process(server_process)
```

## LoRA with DP attention

SGLang supports LoRA serving with [data parallel attention (DPA)](/docs/advanced_features/dp_dpa_smg_guide). The model architecture must support DPA, and you must currently select the Triton LoRA backend. Other built-in LoRA backends reject this configuration at startup.

The following NVIDIA CUDA example uses four attention-DP ranks:

```bash theme={null}
python3 -m sglang.launch_server \
    --model-path MODEL_PATH \
    --tp 4 \
    --attn-dp-size 4 \
    --enable-lora \
    --lora-backend triton \
    --lora-paths '{"lora_name":"ADAPTER_NAME","lora_path":"ADAPTER_PATH","pinned":true}'
```

Keep these constraints in mind:

* `--attn-dp-size` must equal `--tp-size`, so every attention DP rank owns its whole token batch. The server rejects other layouts at startup.
* All adapters must be GPU-pinned (`pinned: true`), including dynamically loaded adapters. You can pin at most `--max-loras-per-batch - 1` adapters; the remaining slot is reserved for the base model.
* Decode CUDA graphs are supported. LoRA prefill runs eagerly when DPA is enabled.
* `--max-loras-per-batch` limits the union of adapters active across all DP ranks in a synchronized forward, not the adapters on one rank alone.
* Dynamic load and unload requests are sent to every DP rank. An operation succeeds only if every rank reports success. After a partially successful load, retry unloading the same adapter name before loading it again.
* The built-in LoRA memory pool does not support `--enable-lora-overlap-loading` with DPA. The server rejects that combination rather than allowing DP ranks to assign different GPU slots.

## LoRA Overlap Loading

By using the `--enable-lora-overlap-loading` server argument, the SGLang engine is able to overlap the loading of LoRA weights with prefill and decode compute, essentially hiding the data movement for LoRA weights behind GPU computation. Our benchmarks show that under adversarial conditions, enabling this feature can result in a \~35% reduction in median TTFT - (see the [LoRA overlap loading PR](https://github.com/sgl-project/sglang/pull/15512) for detailed benchmarks).

```python Example theme={null}
lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json"
lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"
lora2 = "philschmid/code-llama-3-1-8b-text-to-sql-lora"


server_process, port = launch_server_cmd(
    """
    python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --enable-lora-overlap-loading \
    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \
    lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
    lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \
    --max-lora-rank 256 \
    --max-loras-per-batch 2 \
    --max-loaded-loras 4
    """
)

url = f"http://127.0.0.1:{port}"
wait_for_server(url)
```

```python Example theme={null}
json_data = {
    "text": [
        "Write a very long fairy-tale.",
        "List 3 countries and their capitals.",
        "List 3 countries and their capitals.",
    ],
    "sampling_params": [
        {"max_new_tokens": 1024, "temperature": 0},
        {"max_new_tokens": 64, "temperature": 0},
        {"max_new_tokens": 64, "temperature": 0},
    ],
    "lora_path": ["lora0", "lora1", "lora2"],
}

# lora0 and lora1 will be loaded into the memory pool first, and because max_loras_per_batch = 2, lora2's request will remain in the queue.
# lora1's request will likely finish first, and once it does, lora2 will be loaded. With --enable-lora-overlap-loading, this loading will
# occur asynchronously and thus decoding for lora0's request won't be blocked.
response = requests.post(
    url + "/generate",
    json=json_data,
)

for i in range(3):
    print(f"Output from lora{i}: \n{response.json()[i]['text']}\n")
```

```python Example theme={null}
terminate_process(server_process)
```

#### Limitations of LoRA Overlap Loading

However, LoRA overlap loading is not free and comes with two important caveats:

1. **Pinned CPU memory requirement**:
   Asynchronous H2D memory copies require LoRA weights to be pinned in CPU memory, which is a finite system resource. To mitigate excessive pinned-memory usage, SGLang currently restricts `max_loaded_loras` to be at most 2× `max_loras_per_batch` when LoRA overlap loading is enabled.

2. **Reduced multi-adapter prefill batching**:
   With overlap loading, adapters become available on the GPU at different times because each adapter is loaded asynchronously. This can reduce the scheduler’s ability to form multi-adapter prefill batches, since only requests whose adapters are currently loaded can be grouped together. As a result, requests for different adapters will be scheduled in separate (or smaller) prefill batches, which can increase TTFT when adapter load time is small compared to prefill compute time. This is why LoRA overlap loading is disabled by default: it should only be enabled when users have determined that LoRA weight loading is a bottleneck (EG high adapter churn, heavy adapter weights, or PCIe-bottlenecked workloads).

#### Example When Overlap Loading Results in Higher Latency

For instance, suppose we have four LoRA adapters: `lora0`, `lora1`, `lora2`, and `lora3`. Loading any adapter takes 2ms, while the prefill step for requests for that adapter takes 20ms.

1. **Baseline**:
   The engine loads all four adapters synchronously, then runs one combined prefill batch, giving us a total time of ≈ `2 * 4 + 20 = 28ms`

2. **With LoRA overlap loading enabled**:
   The engine begins loading `lora0` and, once it is ready, schedules a prefill batch containing only `lora0` while `lora1` loads in the background. Then it schedules `lora1`’s prefill while `lora2` loads, and so on. In the worst case where prefill cannot be batched across adapters, total time is ≈ `2 + 4 * 20 = 82ms`

In this scenario, overlap loading reduces adapter-load overhead, but the loss of multi-adapter prefill batching dominates and leads to higher TTFT.

## Future Works

The development roadmap for LoRA-related features can be found in this [issue](https://github.com/sgl-project/sglang/issues/2929). Other features, including Embedding Layer, Unified Paging, Cutlass backend are still under development.


# Model Loading
Source: https://docs.sglang.io/docs/advanced_features/model_loading

Control how SGLang loads model weights: load formats, model loader extra config, multithreaded loading, prefetching, and remote/streaming loaders.

`--model-path` selects the checkpoint to serve; `--load-format` and the weight-loading flags below control how those weights are read into memory. To stream weights from cloud object storage (S3/GCS/Azure), see [Loading Models from Object Storage](./object_storage).

## How loading works

SGLang picks a loader from `--load-format`, falling back to auto-detection from the checkpoint or model path. The default `auto` loader reads `safetensors` and falls back to PyTorch `.bin`.

```bash theme={null}
python -m sglang.launch_server \
  --model-path Qwen/Qwen3.6-35B-A3B \
  --load-format auto
```

Some formats are auto-detected and override `auto`:

* A Mistral native checkpoint is detected and loaded with `mistral`.
* A `.gguf` model path is detected and loaded with `gguf`.
* An object storage URI (`s3://`, `gs://`, `az://`) is loaded with `runai_streamer`.
* A remote URI is loaded with `remote`.

## Load formats

Set with `--load-format`:

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Format</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`auto`</td>
      <td>Default. Load `safetensors` if available, otherwise fall back to the PyTorch `.bin` format.</td>
    </tr>

    <tr>
      <td>`safetensors`</td>
      <td>Load weights in the safetensors format.</td>
    </tr>

    <tr>
      <td>`instanttensor`</td>
      <td>Load Safetensors weights quickly with InstantTensor's distributed loading and pipelined prefetching. Supports CUDA and ROCm; requires PyTorch's `cuda` device interface. Distributed loading may retain communicator memory for the server lifetime.</td>
    </tr>

    <tr>
      <td>`pt`</td>
      <td>Load weights in the PyTorch `.bin` format.</td>
    </tr>

    <tr>
      <td>`npcache`</td>
      <td>Load PyTorch-format weights and store a numpy cache to speed up subsequent loads. Only supports `.bin` checkpoints.</td>
    </tr>

    <tr>
      <td>`dummy`</td>
      <td>Initialize weights with random values, for profiling.</td>
    </tr>

    <tr>
      <td>`sharded_state`</td>
      <td>Each tensor-parallel worker reads only its own pre-sharded shard rather than the full checkpoint, giving a fast load path for large TP models. See `examples/runtime/engine/save_sharded_state.py` for creating a sharded checkpoint.</td>
    </tr>

    <tr>
      <td>`fastsafetensors`</td>
      <td>Load safetensors using the `fastsafetensors` iterator.</td>
    </tr>

    <tr>
      <td>`layered`</td>
      <td>Load weights layer by layer, so a layer can be quantized before the next is loaded, lowering the peak memory envelope.</td>
    </tr>

    <tr>
      <td>`gguf`</td>
      <td>Load weights in the GGUF format. Auto-detected from a `.gguf` model path.</td>
    </tr>

    <tr>
      <td>`bitsandbytes`</td>
      <td>Load weights using bitsandbytes quantization.</td>
    </tr>

    <tr>
      <td>`mistral`</td>
      <td>Load a Mistral native-format checkpoint. Auto-detected for such checkpoints.</td>
    </tr>

    <tr>
      <td>`flash_rl`</td>
      <td>Load a BF16/FP16 checkpoint with native SGLang FP8 quantization for RL training. Requires `--rl-quant-profile`.</td>
    </tr>

    <tr>
      <td>`runai_streamer`</td>
      <td>Stream weights from SSDs, shared filesystems, or object storage. See [Loading Models from Object Storage](./object_storage).</td>
    </tr>

    <tr>
      <td>`remote`</td>
      <td>Load tensors from a remote KV/filesystem connector. Auto-detected for remote URIs.</td>
    </tr>

    <tr>
      <td>`remote_instance`</td>
      <td>Pull weights over the network from another running SGLang instance (the "seed") rather than from disk. Configured with the `--remote-instance-weight-loader-*` flags.</td>
    </tr>
  </tbody>
</table>

## Model loader extra config

`--model-loader-extra-config` takes a JSON string passed to the loader selected by `--load-format`.

```bash theme={null}
python -m sglang.launch_server \
  --model-path Qwen/Qwen3.6-35B-A3B \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 16}'
```

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Load format</th>
      <th>Key</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`auto` / `safetensors` / `pt` / `npcache`</td>
      <td>`enable_multithread_load` (bool)</td>
      <td>Read weight shards with a thread pool instead of sequentially. Disabled by default when `--weight-loader-prefetch-checkpoints` is set (to avoid I/O oversubscription with the prefetch threads); set this to `true` to opt back in.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`auto` / `safetensors` / `pt` / `npcache`</td>
      <td>`num_threads` (int)</td>
      <td>Number of worker threads when multithreaded loading is enabled.</td>
      <td>8</td>
    </tr>

    <tr>
      <td>`sharded_state`</td>
      <td>`pattern` (str)</td>
      <td>Filename pattern for per-rank shards.</td>
      <td>`model-rank-{rank}-part-{part}.safetensors`</td>
    </tr>

    <tr>
      <td>`fastsafetensors`</td>
      <td>`enable_gds` (bool)</td>
      <td>Use GPU Direct Storage. Set to `false` when the host does not provide the NVIDIA GPUDirect Storage kernel driver, such as in a gVisor sandbox.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`instanttensor`</td>
      <td>`buffer_size` (int)</td>
      <td>GPU tensor buffer size in bytes. See the [InstantTensor GitHub homepage](https://github.com/scitix/InstantTensor) for the latest optional parameters.</td>
      <td>`null`</td>
    </tr>

    <tr>
      <td>`instanttensor`</td>
      <td>`chunk_size` (int)</td>
      <td>File I/O operation size in bytes.</td>
      <td>`null`</td>
    </tr>

    <tr>
      <td>`instanttensor`</td>
      <td>`io_depth` (int)</td>
      <td>Maximum number of rank-local I/O operations in flight.</td>
      <td>`null`</td>
    </tr>

    <tr>
      <td>`instanttensor`</td>
      <td>`backend` (str / list)</td>
      <td>A backend name or an ordered list of candidates: `AIO`, `AIO_BUFFERED`, `URING`, `URING_BUFFERED`, `CUFILE`, `MMAP`, or the `BUFFERED` policy.</td>
      <td>`null`</td>
    </tr>

    <tr>
      <td>`bitsandbytes`</td>
      <td>`qlora_adapter_name_or_path` (str)</td>
      <td>QLoRA adapter to apply on top of the bitsandbytes-quantized base weights.</td>
      <td>—</td>
    </tr>

    <tr>
      <td>`runai_streamer`</td>
      <td>`distributed`, `concurrency`, `memory_limit`</td>
      <td>Streaming controls. See [Loading Models from Object Storage](./object_storage).</td>
      <td>See linked page</td>
    </tr>
  </tbody>
</table>

## Weight-loading performance flags

Top-level arguments that tune how safetensors weights are read, independent of `--load-format`.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Flag</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--download-dir`</td>
      <td>Directory used to download and cache Hugging Face model files.</td>
      <td>HF default</td>
    </tr>

    <tr>
      <td>`--weight-loader-disable-mmap`</td>
      <td>Disable mmap while loading safetensors. Can help on filesystems where mmap is slow.</td>
      <td>off</td>
    </tr>

    <tr>
      <td>`--weight-loader-prefetch-checkpoints`</td>
      <td>Prefetch checkpoint files into the OS page cache before loading. Each rank prefetches a fraction of the shards, cutting total network I/O on shared filesystems (NFS/Lustre) from N×checkpoint to 1×checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set `enable_multithread_load=true` in `--model-loader-extra-config` to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op).</td>
      <td>off</td>
    </tr>

    <tr>
      <td>`--weight-loader-prefetch-num-threads`</td>
      <td>Threads per rank for checkpoint prefetching.</td>
      <td>4</td>
    </tr>

    <tr>
      <td>`--weight-loader-drop-cache-after-load`</td>
      <td>Call `posix_fadvise(DONTNEED)` after successfully loading each shard, freeing page cache. Supported by the standard safetensors and `fastsafetensors` loaders.</td>
      <td>off</td>
    </tr>

    <tr>
      <td>`--custom-weight-loader`</td>
      <td>Import path(s) of a custom weight-loading function, e.g. `my_package.weight_load_func`.</td>
      <td>—</td>
    </tr>
  </tbody>
</table>

## See also

* [Loading Models from Object Storage](./object_storage)
* [Quantization](./quantization)
* [Server Arguments](./server_arguments)


# Loading Models from Object Storage
Source: https://docs.sglang.io/docs/advanced_features/object_storage



SGLang can load models directly from object storage without a full local download. It uses the `runai_streamer` load format to stream model weights from cloud storage, reducing startup time and local storage requirements.

## Overview

When loading models from object storage, SGLang uses a two-phase approach:

1. **Metadata Download** (once, before process launch): Configuration files and tokenizer files are downloaded to a local cache
2. **Weight Streaming** (lazy, during model loading): Model weights are streamed directly from object storage as needed

## Supported Storage Backends

1. **Amazon S3**: `s3://bucket-name/path/to/model/`
2. **Google Cloud Storage**: `gs://bucket-name/path/to/model/`
3. **Azure Blob**: `az://some-azure-container/path/`
4. **S3 compatible**: `s3://bucket-name/path/to/model/`

## Quick Start

### Basic Usage

Simply provide an object storage URI as the model path:

```bash theme={null}
# S3
python -m sglang.launch_server \
  --model-path s3://my-bucket/models/llama-3-8b/ \
  --load-format runai_streamer

# Google Cloud Storage
python -m sglang.launch_server \
  --model-path gs://my-bucket/models/llama-3-8b/ \
  --load-format runai_streamer
```

**Note**: The `--load-format runai_streamer` is automatically detected when using object storage URIs, so you can omit it:

```bash theme={null}
python -m sglang.launch_server \
  --model-path s3://my-bucket/models/llama-3-8b/
```

### With Tensor Parallelism

```bash theme={null}
python -m sglang.launch_server \
  --model-path gs://my-bucket/models/llama-70b/ \
  --tp 4 \
  --model-loader-extra-config '{"distributed": true}'
```

## Configuration

### Load Format

The `runai_streamer` load format is designed for object storage, SSDs, and shared filesystems.

```bash theme={null}
python -m sglang.launch_server \
  --model-path s3://bucket/model/ \
  --load-format runai_streamer
```

### Extended Configuration Parameters

Use `--model-loader-extra-config` to pass additional configuration as a JSON string:

```bash theme={null}
python -m sglang.launch_server \
  --model-path s3://bucket/model/ \
  --model-loader-extra-config '{
    "distributed": true,
    "concurrency": 8,
    "memory_limit": 2147483648
  }'
```

#### Available Parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`distributed`</td>
      <td>bool</td>
      <td>Enable distributed streaming for multi-GPU setups. Automatically set to `true` for object storage paths on CUDA-like devices.</td>
      <td>Auto-detected</td>
    </tr>

    <tr>
      <td>`concurrency`</td>
      <td>int</td>
      <td>Number of concurrent download streams. Higher values can improve throughput for large models.</td>
      <td>4</td>
    </tr>

    <tr>
      <td>`memory_limit`</td>
      <td>int</td>
      <td>Memory limit (in bytes) for the streaming buffer.</td>
      <td>System-dependent</td>
    </tr>
  </tbody>
</table>

## Performance Considerations

### Distributed Streaming

For multi-GPU setups, enable distributed streaming to parallelize weight loading across processes:

```bash theme={null}
python -m sglang.launch_server \
  --model-path s3://bucket/model/ \
  --tp 8 \
  --model-loader-extra-config '{"distributed": true}'
```

## Limitations

* **Supported formats**: Only the `.safetensors` weight format is supported.
* **Supported devices**: Distributed streaming is supported on CUDA-like devices; otherwise it falls back to non-distributed streaming.

## See Also

* [Runai model streamer documentation](https://github.com/run-ai/runai-model-streamer)


# Observability
Source: https://docs.sglang.io/docs/advanced_features/observability



## Production Metrics

SGLang exposes the following metrics via Prometheus. You can enable them by adding `--enable-metrics` when launching the server.
You can query them by:

```bash Command theme={null}
curl http://localhost:30000/metrics
```

See [Production Metrics](../references/production_metrics) and [Production Request Tracing](../references/production_request_trace) for more details.

## Logging

By default, SGLang does not log any request contents. You can log them by using `--log-requests`.
You can control the verbosity by using `--log-request-level`.
See [Logging](./server_arguments#logging) for more details.

You can change verbosity at runtime:

```bash Command theme={null}
python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --log-level=debug
```

## Request Dump and Replay

You can dump all requests and replay them later for benchmarking or other purposes.

To start dumping, use the following command to send a request to a server:

```bash Command theme={null}
python3 -m sglang.srt.managers.configure_logging --url http://localhost:30000 --dump-requests-folder /tmp/sglang_request_dump --dump-requests-threshold 100
```

The server will dump the requests into a pickle file for every 100 requests.

To replay the request dump, use `scripts/playground/replay_request_dump.py`.

## Crash Dump and Replay

Sometimes the server might crash, and you may want to debug the cause of the crash.
SGLang can preserve recent request data for replay and collect CUDA device coredumps for low-level debugging.

Set the crash diagnostics folder with `--crash-dump-folder /tmp/crash_dump`.

When SGLang handles a crash, it writes completed requests retained by the crash-dump buffer plus in-flight requests to `/tmp/crash_dump/<hostname>/crash_dump_<timestamp>.pkl`. The file also contains the server arguments and launch command. Replay it with `scripts/playground/replay_request_dump.py`.

On NVIDIA CUDA, the option also sets default environment variables before CUDA initializes. These defaults enable device coredumps on CUDA exceptions and allow SGLang to trigger device coredumps for live scheduler processes when it handles a crash. CUDA device coredumps are written to `/tmp/crash_dump/<hostname>/core.cuda.<timestamp>.<pid>`. Explicitly configured CUDA coredump environment variables take precedence, including a custom `CUDA_COREDUMP_FILE` path. This option does not configure OS process core dumps.


# Advanced Features
Source: https://docs.sglang.io/docs/advanced_features/overview

Advanced configuration, optimization, and deployment features for SGLang.

* [Server Arguments](./server_arguments)
* [Session-Aware Radix Cache](./session_radix_cache)
* [Hyperparameter Tuning](./hyperparameter_tuning)
* [Attention Backend](./attention_backend)
* [Speculative Decoding](./speculative_decoding)
* [Structured Outputs](./structured_outputs)
* [Quantization](./quantization)
* [Expert Parallelism](./expert_parallelism)
* [Prefill Context Parallelism](./prefill_cp)
* [Decode Context Parallelism](./dcp)
* [LoRA](./lora)
* [PD Disaggregation](./pd_disaggregation)
* [Pipeline Parallelism](./pipeline_parallelism)
* [HiCache](./hicache_best_practices)
* [SGLang Simulator](./sglang_simulator)
* [Observability](./observability)
* [And more…](./server_arguments)


# Paged Experts
Source: https://docs.sglang.io/docs/advanced_features/paged_experts



Paged Experts serves a Mixture-of-Experts (MoE) model whose expert weights do not fit in GPU
memory. Each MoE layer keeps **K of its E routed experts in a K-slot table on the GPU** and holds
all E experts in pinned host memory; every step, the experts the step routes to that are not
resident are copied into slots before the expert GEMM runs.

## Why Paged Experts?

Expert weights are most of an MoE model's parameters, yet each token is routed to only `top_k`
of the `E` experts per layer, and consecutive tokens reuse many of the same experts. Paged
Experts keeps a working set of K experts per layer resident and pages the rest in from host
memory as routing requires them, so expert capacity is bounded by host RAM rather than VRAM.

**Computation stays on the GPU.** Only expert *storage* is host-backed: experts are copied to the
GPU and computed there by the model's own fused-MoE kernel. This differs from `--cpu-offload-gb`
and CPU expert offload, which move the computation to the CPU.

**Results are unchanged.** Routing is untouched (E-wide), and the paged layer produces the same
output as the unpaged one: bit-identical for unquantized and block-FP8 experts (with
`--enable-deterministic-inference`), and within the Marlin kernel's own run-to-run noise for
GPTQ. See [Correctness](#correctness).

## Usage

```bash theme={null}
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-30B-A3B-GPTQ-Int4 \
  --enable-paged-experts \
  --paged-experts-num-resident 92
```

See [Choosing K](#choosing-k) for the value. The default CUDA graphs stay on: decode graphs are captured up to
`K // top_k` concurrent requests, and breakable prefill graphs run the paged MoE layers as eager
breaks.

## Related Parameters

* **`--enable-paged-experts`**: Enable Paged Experts. Off by default.
* **`--paged-experts-num-resident K`**: Routed experts per MoE layer kept on the GPU. Required
  with `--enable-paged-experts`, at least the router's `top_k`; larger values page less but leave
  less memory for the KV cache.

## Supported Configurations and Limitations

* **Quantization:** unquantized (bf16/fp16), block-quantized FP8 (e.g. `weight_block_size`
  `[128, 128]`) on the triton MoE runner, and GPTQ int4/int8 on the Marlin runner. Other
  quantization methods are rejected at startup.
* **Single GPU:** tensor, expert, pipeline and data parallelism, EPLB, fused shared experts
  (use `--disable-shared-experts-fusion`), `--enable-fused-moe-sum-all-reduce` and
  `--load-format dummy` are rejected at startup.
* **Speculative decoding** is rejected at startup.
* **CUDA graphs:** decode graphs may use the `full` backend (the default) or be disabled;
  prefill graphs may use the `breakable` backend (the default on CUDA) or be disabled. Other
  combinations are rejected at startup.
* **Host memory:** the host store holds all E experts of every MoE layer and is page-locked, so
  the host must be able to pin that much memory. Under WSL2, pinned memory is capped at about
  half of system RAM.
* **Startup:** for GPTQ, the Marlin repack runs over the host store K experts at a time at load.

## How It Works

### The K-slot table and the host store

The paged method wraps the layer's own fused-MoE method. It creates the layer's expert tensors
with K rows instead of E, so the GPU holds only the K slots, and allocates a host store holding
all E experts of each per-expert tensor. Each tensor's weight loader is redirected to a one-row
view of the host store, so checkpoint weights are written by SGLang's native per-expert loader
(sharding, gate/up placement, scale layout) and the host rows have exactly the layout the GPU
slots would have.

After loading, the wrapped method's post-load step runs as it would without paging. For methods
that repack their experts (GPTQ's Marlin repack), it runs over the host store K experts at a
time, so the host store holds every expert in the repacked layout.

### A step: waves, page-in, merge

Every forward step of every MoE layer:

1. The distinct experts the step routes to are split into **waves** of at most K, the experts
   already resident first. Decode at
   moderate batch sizes needs one wave; a prefill chunk usually needs several.
2. Each wave makes its experts resident, evicting the least recently used experts outside the
   wave, and copies the missing ones from the host store into their slots.
3. The routed expert ids are remapped to slots and the wrapped fused-MoE kernel runs over the
   wave's own entries **without its final top-k sum**, returning each routed (token, expert)
   output already multiplied by its router weight.
4. Each entry is written into one merged buffer by the one wave that holds its expert, and the
   merged entries go through the same top-k sum the kernel would have performed, so the result
   does not depend on how the experts were split into waves.

### Captured decode

With decode CUDA graphs on, a step whose routed entries fit the K slots (`tokens × top_k ≤ K`,
i.e. decode up to `K // top_k` requests) is served by a single wave decided and paged on the
GPU, with no host synchronization, so the decode graph captures it:

* a JIT kernel makes the LRU decision on the device and writes the page-in plan (experts, slots,
  count) to GPU buffers;
* a second kernel copies the planned experts of every paged tensor from the pinned host store
  into their slots, reading the host memory directly through its unified address and the count
  on the device, so each graph replay moves exactly the experts it decided on.

Decode capture batch sizes are capped at `K // top_k` (logged at startup); larger batches run
without a graph. Prefill steps plan their waves on the host: under a breakable prefill graph the
paged MoE layer is an eager break between graph segments, so the rest of the layer stack stays
captured. Both paths share one LRU state, so residency carries over between captured and
uncaptured steps.

## Choosing K

The K-slot table counts as model weights in SGLang's memory accounting, so K and the KV cache
come out of the same static budget (`--mem-fraction-static`): whatever K does not take becomes KV
cache, as without paging. One resident expert per layer costs `per_expert_bytes × num_moe_layers`
of GPU memory, so K is roughly

```
K ≈ (free_memory × mem_fraction_static − non_expert_bytes − kv_cache_bytes) / (per_expert_bytes × num_moe_layers)
```

For example, Qwen3-30B-A3B-GPTQ-Int4 (2.43 MB per expert, 48 MoE layers, 117 MB per slot) fits
K = 92 of 128 on a 16 GB GPU with `--mem-fraction-static 0.88 --max-total-tokens 10240`.
If the server runs out of memory for the KV cache at startup, lower K; for more concurrency,
lower K or use `--kv-cache-dtype fp8`.

## Correctness

Prompt and greedy decode logprobs are compared with the unpaged server under
`--enable-deterministic-inference --attention-backend triton`, eager and with decode captured:

* **unquantized and block-FP8 experts (triton runner):** bit-identical;
* **GPTQ (Marlin runner):** Marlin's reduction depends on the batch shape, so a paged step (which
  runs the kernel on a different number of experts) differs from the unpaged one within Marlin's
  own batch-to-batch noise.


# PD Disaggregation
Source: https://docs.sglang.io/docs/advanced_features/pd_disaggregation



## Why and What is PD Disaggregation?

Large Language Model (LLM) inference comprises two distinct phases: **Prefill** and **Decode**. The Prefill phase is computation-intensive, processing the entire input sequence, while the Decode phase is memory-intensive, managing the Key-Value (KV) cache for token generation. Traditionally, these phases are handled within a unified engine, where combined scheduling of prefill and decode batches introduces inefficiencies. To address these challenges, we introduce **Prefill and Decoding (PD) Disaggregation** in SGLang.

### Issues with Unified Scheduling

The conventional unified engine, which processes prefill and decode batches together, results in two significant problems:

1. **Prefill Interruption**: Incoming prefill batches frequently interrupt ongoing decode batches, causing substantial delays in token generation.
2. **DP Attention Imbalance**: In data-parallel (DP) attention, one DP worker may process a prefill batch while another handles a decode batch simultaneously, leading to increased decode latency.

PD Disaggregation resolves these by separating the two stages, enabling tailored optimizations for each.

For the design details, please refer to [link](https://docs.google.com/document/d/1rQXJwKd5b9b1aOzLh98mnyMhBMhlxXA5ATZTHoQrwvc/edit?tab=t.0).

Currently, we support Mooncake and NIXL as the transfer engine.

## Decode allocation policy

By default, decode reserves each request's destination KV slots before prefill
finishes. Long prefills can hold these slots while decode waits for the KV data.
To delay this reservation, set the following flag on **both prefill and decode**:

```bash theme={null}
--disaggregation-decode-allocation-policy prefill_complete
```

With `prefill_complete`, prefill computes without waiting for destination slots,
then signals readiness after accepting its final prefill result. Decode reserves
slots and starts the existing KV transfer handshake only after that signal.
This changes per-request KV reservation, not the size or creation of the global
KV pool. The default `early` policy preserves the existing behavior.

Deferring allocation can help when long prefills compete with active decode
requests for destination KV capacity. It also retains KV on prefill longer and
removes overlap between chunked prefill and KV transfer. Compare both policies
for your workload; this option is not a general throughput improvement.

This initial implementation requires Mooncake (RDMA or `mooncake_tcp`), the Python
bootstrap registry, and attention TP/CP and pipeline sizes of one. Attention DP
is allowed when each DP rank has attention TP size one. Prefill automatically
resolves `--optimistic-prefill-attempts` from zero to one; configurations that
disable optimistic prefill after model validation are rejected.
Chunked prefill does not yield just because decode has not admitted the request.
Completed requests retain their KV in the existing optimistic-prefill inflight
queue until decode supplies destination indices; readiness alone never starts
a KV write.
After a successful transfer, prefill clears the room's readiness and transfer
state before notifying decode of success. Retraction recovery can then reuse
the same bootstrap room. This notification waits for worker and scheduler
cleanup; failure and cancellation records remain until their timeout expires.
Retries remain eligible for prefill after the optimistic attempt budget is
exhausted and remain subject to the existing cancellation and timeout handling.

On prefill, `SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT` starts after the final
prefill result, giving decode admission its own budget. On decode,
`SGLANG_DISAGGREGATION_WAITING_TIMEOUT` bounds the wait for prefill completion,
then starts afresh for admission and again when destination indices are sent
for transfer.

The option does not support staging buffers, runtime PD role switching, PD
multiplexing, encoder disaggregation, decoupled speculation, HiSparse, decode radix cache, decode
HiCache, or non-DSA auxiliary state pools. Prefill HiCache is limited to L2 with
`write_back` and no storage backend. Unsupported configurations fail validation.

All communicating workers must use the same revision and enable the same policy. This
version of the decode worker rejects policy or peer-layout mismatches before
destination registration. Older decode workers do not enforce this check. Keep
`early` on both sides until every worker has been upgraded, then enable the new
policy on both sides.

## Profiling in PD Disaggregation Mode

When you need to profile prefill or decode workers in PD disaggregation mode, please refer to the [Profile In PD Disaggregation Mode](../developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode) section in the Benchmark and Profiling guide. Due to torch profiler limitations, prefill and decode workers must be profiled separately using dedicated command-line options.

## Router Integration

For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](./sgl_model_gateway#prefill-decode-disaggregation).

### Responses API under HTTP PD

HTTP PD supports foreground generation through `POST /v1/responses`, including streaming generation with `stream=true` through the HTTP PD router. This requires both the router's Responses dispatch support and serving workers that accept Responses PD routing metadata.

Store-backed and stateful Responses features are **not supported under PD**:

* Response persistence (`store`) and response history.
* Conversation chaining with `previous_response_id`.
* Background Responses (`background=true`), including background streams.
* Retrieval by response ID (`GET /v1/responses/{id}`).
* Cancel-by-ID (`POST /v1/responses/{id}/cancel`).
* Other workflows requiring shared process-local Responses history.

Prefill and decode run in separate serving processes. Their process-local Responses stores cannot provide coherent shared state: both legs need the same conversation history to reconstruct the prompt, and prefill's stored response is not available to decode. Send explicit conversation history in foreground requests instead of relying on stored response IDs.

Built-in tools (`web_search`, `code_interpreter`) are also unsupported under PD: each tool call requires another generation, but the router dispatches exactly one prefill/decode pair per HTTP request.

Serving-side capability enforcement is tracked in [#39122](https://github.com/sgl-project/sglang/pull/39122). That change introduces opt-in standalone storage with `--enable-response-store`, disallows it under PD, and rejects stateful requests with HTTP 400 before generation. These checks require a serving version containing that change; older versions must not be assumed to reject every unsupported workflow at admission. The HTTP PD router also rejects detached background requests because it does not implement their retrieval/cancel lifecycle.

## Mooncake

### Requirements

```bash theme={null}
uv pip install mooncake-transfer-engine
```

### AWS EFA

#### EFA Docker images

Each published CUDA 13 image has an EFA variant for EFA-enabled AWS EC2 instances, built from the same SGLang commit with `docker/efa.Dockerfile`:

| Base image | EFA image |
| - | - |
| `latest`, `v<version>` | `latest-efa`, `v<version>-efa` |
| `latest-runtime`, `v<version>-runtime` | `latest-runtime-efa`, `v<version>-runtime-efa` |
| `nightly-dev-<date>-<sha>` | `nightly-dev-efa-<date>-<sha>`, plus the moving `dev-efa` |

Each EFA image matches its base and adds:

* The AWS EFA userspace stack: libfabric with the EFA provider, rdma-core, and the aws-ofi-nccl plugin. The host does not need its own EFA userspace install.
* On x86\_64, the EFA build of Mooncake with `MOONCAKE_PROTOCOL=efa`. On arm64 the standard Mooncake package is kept and `MOONCAKE_PROTOCOL=tcp` is set, because the EFA build is x86\_64-only.
* `SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC`, so NIXL transfers use EFA on both architectures.
* `NCCL_NET_PLUGIN=ofi` and `NCCL_TUNER_PLUGIN=ofi`, so NCCL uses aws-ofi-nccl.

To add EFA to another CUDA 13 SGLang image, build `docker/efa.Dockerfile` on top of it:

```bash theme={null}
docker build -f docker/efa.Dockerfile \
  --build-arg BASE_IMAGE=lmsysorg/sglang:latest-runtime \
  -t sglang:efa docker/
```

The host still needs the EFA kernel driver. Pass the EFA devices into the container and lift the locked-memory limit:

```bash theme={null}
HF_CACHE_DIR="/path/to/huggingface-cache"
docker run -it --gpus all \
  --network host \
  --ipc=host \
  $(for d in /dev/infiniband/uverbs*; do echo -n "--device $d "; done) \
  --ulimit memlock=-1 \
  -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
  --env HF_TOKEN \
  lmsysorg/sglang:latest-runtime-efa bash
```

Inside the container, `fi_info -p efa -t FI_EP_RDM` lists the EFA devices.

#### Install into your own environment

On an x86\_64 AWS EFA instance, replace the standard CUDA 13 Mooncake package with the EFA build. The two packages install the same `mooncake` Python module, so do not install them side by side.

The following commands replace the package in an SGLang CUDA 13 environment while preserving its existing Python dependencies:

```bash theme={null}
python3 -m pip uninstall -y \
  mooncake-transfer-engine-cuda13 \
  mooncake-transfer-engine-efa-cuda13
python3 -m pip install --no-deps \
  mooncake-transfer-engine-efa-cuda13==0.3.13.post1

export LD_LIBRARY_PATH="/opt/amazon/efa/lib:/opt/amazon/efa/lib64:${LD_LIBRARY_PATH:-}"
export MOONCAKE_PROTOCOL=efa

python3 -c 'from mooncake import engine; assert engine.SUPPORT_EFA'
fi_info -p efa -t FI_EP_RDM
```

The EFA wheel expects a compatible AWS EFA userspace installation on the host and does not bundle `libfabric` or `libefa`. The current wheel is available only for x86\_64. It is built without Mooncake EP/PG and the NVLink/MNNVL components, so use the standard Mooncake CUDA package for those features.

#### CUDA VMM limitations

These limitations apply whenever `MOONCAKE_PROTOCOL=efa`, including in the EFA Docker images.

The current libfabric EFA provider cannot transfer CUDA VMM-backed buffers. Until [libfabric issue #12853](https://github.com/ofiwg/libfabric/issues/12853) is resolved, apply these settings to both prefill and decode workers:

* Do not set `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` to `NVLINK` or `BAREX`.
  `INTRA_NODE_NVLINK` is supported because it uses the default CUDA allocator.
* Do not enable `expandable_segments:True` in `PYTORCH_CUDA_ALLOC_CONF` or `PYTORCH_ALLOC_CONF`.
* Post-capture KV sizing is automatically disabled for the EFA protocol because
  its KV cache uses CUDA VMM allocations.
* Do not set `MC_FORCE_MNNVL` or `NCCL_MNNVL_ENABLE`; EFA is not an MNNVL transport.

SGLang validates the first two constraints at startup to avoid a later `fi_read` or `fi_write` failure from libfabric.

### IB Device Configuration

`--disaggregation-ib-device` supports the following formats when using the Mooncake backend:

1. Shared device list for all GPUs: `mlx5_0` or `mlx5_0,mlx5_1`
2. Per-GPU JSON mapping: `{"0": "mlx5_0,mlx5_1", "1": "mlx5_2,mlx5_3"}`
3. Path to a JSON file containing the same per-GPU mapping

Each JSON value uses the same comma-separated device list format as the shared configuration.

### Usage

### Llama Single Node

```bash theme={null}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode prefill \
  --port 30000 \
  --disaggregation-ib-device mlx5_roce0
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode decode \
  --port 30001 \
  --base-gpu-id 1 \
  --disaggregation-ib-device mlx5_roce0
python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
```

### DeepSeek Multi-Node

```bash theme={null}
# prefill 0
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-ib-device ${device_name} \
  --disaggregation-mode prefill \
  --host ${local_ip} \
  --port 30000 \
  --trust-remote-code \
  --dist-init-addr ${prefill_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 0 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8
# prefill 1
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-ib-device ${device_name} \
  --disaggregation-mode prefill \
  --host ${local_ip} \
  --port 30000 \
  --trust-remote-code \
  --dist-init-addr ${prefill_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 1 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8
# decode 0
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-ib-device ${device_name} \
  --disaggregation-mode decode \
  --host ${local_ip} \
  --port 30001 \
  --trust-remote-code \
  --dist-init-addr ${decode_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 0 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8 \
  --max-running-requests 128
# decode 1
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-ib-device ${device_name} \
  --disaggregation-mode decode \
  --host ${local_ip} \
  --port 30001 \
  --trust-remote-code \
  --dist-init-addr ${decode_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 1 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8 \
  --max-running-requests 128
```

### Advanced Configuration

PD Disaggregation with Mooncake supports the following environment variables for fine-grained control over system behavior.

#### NVLink Transport Configuration

To enable NVLink transport for KV cache transfers with the mooncake backend (recommended for NVL72 deployments), set the following environment variables. Note that auxiliary data transfer will still use TCP as a temporary workaround.

```bash Command theme={null}
export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK
export MC_FORCE_MNNVL=True
```

To utilize Intra-Node NVLink for KV cache transfers with the Mooncake backend (recommended for A100, H20, H100, etc.), set the following environment variables. Please note that auxiliary data still needs to be transferred via TCP.

```bash theme={null}
export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK
export MC_INTRANODE_NVLINK=true
```

The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom memory pool. Supported values are `NVLINK` (or `True`), `BAREX`, and `INTRA_NODE_NVLINK`.

#### Prefill Server Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variable</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`**</td>
      <td>Controls the total number of worker threads for KVCache transfer operations per TP rank</td>
      <td>A dynamic value calculated by `int(0.75 * os.cpu_count()) // 8`, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions</td>
    </tr>

    <tr>
      <td>**`SGLANG_DISAGGREGATION_QUEUE_SIZE`**</td>
      <td>Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to `1`, then we transfer requests one by one according to fcfs strategy</td>
      <td>`4`</td>
    </tr>

    <tr>
      <td>**`SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES`**</td>
      <td>Opt-in limit for the number of KV cache indices represented by one synchronous all-layer Mooncake transfer batch. Set it to a positive value to slice larger index arrays into ordered sub-batches before contiguous address ranges are formed. The custom-memory-pool layerwise path is unchanged.</td>
      <td>`0` (disabled)</td>
    </tr>

    <tr>
      <td>**`SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT`**</td>
      <td>Timeout (seconds) for receiving destination KV indices during request initialization</td>
      <td>`300`</td>
    </tr>

    <tr>
      <td>**`SGLANG_DISAGGREGATION_BOOTSTRAP_ENTRY_CLEANUP_INTERVAL`**</td>
      <td>Interval (seconds) between cleanups of bootstrap entries</td>
      <td>`120`</td>
    </tr>
  </tbody>
</table>

If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600` (10 minutes) to relax the timeout condition.
Please be aware that this setting will cause prefill instances to take a longer time to clean up the affected memory resources when a running decode node loses connection.

#### Decode Server Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variable</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**`SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL`**</td>
      <td>Interval (seconds) between health checks to prefill bootstrap servers</td>
      <td>`5.0`</td>
    </tr>

    <tr>
      <td>**`SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE`**</td>
      <td>Consecutive heartbeat failures before marking prefill server offline</td>
      <td>`2`</td>
    </tr>

    <tr>
      <td>**`SGLANG_DISAGGREGATION_WAITING_TIMEOUT`**</td>
      <td>Timeout (seconds) for receiving KV Cache after request initialization</td>
      <td>`300`</td>
    </tr>
  </tbody>
</table>

If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600` (10 minutes) to relax the timeout condition.

## Heterogeneous TP with GPU Staging Buffer

When prefill and decode use different tensor parallelism (TP) sizes (e.g., prefill TP=4, decode DP attention with TP=1), the KV cache memory layout differs between the two sides. The **GPU staging buffer** solves this by gathering KV head slices into a contiguous buffer on the prefill side, performing bulk RDMA transfer, then scattering into the correct KV cache pages on the decode side. This provides **2–5x throughput improvement** over the default per-token slice approach at high concurrency and matches homogeneous TP baselines within \~5%.

Enable the staging buffer when prefill and decode use **different TP sizes** with the **Mooncake** transfer backend. When both sides use the same TP size, staging is automatically bypassed even if enabled.

> **Note:** The staging buffer is designed for non-MLA models (e.g. GQA, MHA). MLA models (e.g. DeepSeek-V2/V3) should not enable this flag.

> **Note:** The staging buffer also runs on Intel XPU, tested with prefill TP 1 and decode TP 2 on the Mooncake backend. On XPU, Mooncake needs a build with `-DUSE_XPU=ON -DUSE_TENT=ON` and runs with `MC_USE_TENT=1`.

> **Note:** You can use different attention TP sizes for hybrid sliding-window models (e.g. `openai/gpt-oss-120b`) with Mooncake or NIXL. For unequal TP, both backends transfer the sliding-window KV by token and KV-head slice. The staging buffer carries only full-attention KV. Use `--pp-size 1` and leave `--enable-unified-memory` disabled for this configuration. MXFP8 KV cache is not supported for unequal-TP SWA transfer.

### Environment Variables

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variable</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**`SGLANG_DISAGG_STAGING_BUFFER`**</td>
      <td>Enable GPU staging buffer for heterogeneous TP KV transfer</td>
      <td>`False`</td>
    </tr>

    <tr>
      <td>**`SGLANG_DISAGG_STAGING_POOL_SIZE_MB`**</td>
      <td>Decode-side ring buffer pool total size in MB</td>
      <td>`4096`</td>
    </tr>
  </tbody>
</table>

### Usage Example

```bash Command theme={null}
# Set staging buffer environment variables on BOTH prefill and decode
export SGLANG_DISAGG_STAGING_BUFFER=1
export SGLANG_DISAGG_STAGING_POOL_SIZE_MB=4096

# Prefill with TP=4
python -m sglang.launch_server \
  --model-path $MODEL_PATH \
  --disaggregation-mode prefill \
  --port 30000 \
  --tp 4 \
  --trust-remote-code \
  --disaggregation-ib-device mlx5_1,mlx5_2

# Decode with TP=1 (or DP attention with effective attention TP=1)
python -m sglang.launch_server \
  --model-path $MODEL_PATH \
  --disaggregation-mode decode \
  --port 30001 \
  --tp 4 \
  --attn-dp-size 4 \
  --trust-remote-code \
  --disaggregation-ib-device mlx5_3,mlx5_4

# Router
python -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://127.0.0.1:30000 \
  --decode http://127.0.0.1:30001 \
  --host 0.0.0.0 --port 8000
```

## NIXL

### Requirements

Install via pip.

```bash theme={null}
pip install nixl
```

Or build from source - may be required if you already have UCX installed.

```bash theme={null}
git clone https://github.com/ai-dynamo/nixl.git
cd nixl
pip install . --config-settings=setup-args="-Ducx_path=/path/to/ucx"
```

### Usage

### Llama Single Node

```bash theme={null}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode prefill \
  --port 30000 \
  --disaggregation-transfer-backend nixl
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode decode \
  --port 30001 \
  --base-gpu-id 1 \
  --disaggregation-transfer-backend nixl
python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000
```

### DeepSeek Multi-Node

```bash theme={null}
# prefill 0
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-transfer-backend nixl \
  --disaggregation-mode prefill \
  --host ${local_ip} \
  --port 30000 \
  --trust-remote-code \
  --dist-init-addr ${prefill_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 0 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8
# prefill 1
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-transfer-backend nixl \
  --disaggregation-mode prefill \
  --host ${local_ip} \
  --port 30000 \
  --trust-remote-code \
  --dist-init-addr ${prefill_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 1 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8
# decode 0
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-transfer-backend nixl \
  --disaggregation-mode decode \
  --host ${local_ip} \
  --port 30001 \
  --trust-remote-code \
  --dist-init-addr ${decode_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 0 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8 \
  --max-running-requests 128
# decode 1
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3-0324 \
  --disaggregation-transfer-backend nixl \
  --disaggregation-mode decode \
  --host ${local_ip} \
  --port 30001 \
  --trust-remote-code \
  --dist-init-addr ${decode_master_ip}:5000 \
  --nnodes 2 \
  --node-rank 1 \
  --tp-size 16 \
  --attn-dp-size 8 \
  --moe-a2a-backend deepep \
  --mem-fraction-static 0.8 \
  --max-running-requests 128
```

### Advanced Configuration

#### NIXL Backend Selection

By default, NIXL uses the **UCX** backend for KV cache transfers. You can select a different NIXL plugin backend depending on your infrastructure using the environment variable `SGLANG_DISAGGREGATION_NIXL_BACKEND`.

Example: `export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC`

**Available backends:** UCX (default), LIBFABRIC, or any installed NIXL plugin.

Example usage:

```bash theme={null}
export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend nixl \
  --port 30000
```

#### UCX device selection

For NVIDIA CUDA and AMD ROCm deployments using NIXL UCX, `--disaggregation-ib-device` accepts a shared HCA list or a per-GPU JSON/file mapping. Mapping keys are process-visible GPU IDs; use HCA names without port suffixes, such as `mlx5_0`.

A non-empty `UCX_NET_DEVICES` takes precedence over this CLI option. SGLang leaves the environment variable unchanged and skips the CLI mapping, including per-GPU mappings, with a warning. This preserves interfaces such as `lo` in an existing configuration like `rdma0:1,rdma4:1,lo`, which some RoCE hosts need for NIXL's self-endpoint.

To use per-GPU CLI selection, unset `UCX_NET_DEVICES`; keep it configured on hosts that require the loopback entry. An explicit `device_list` in `SGLANG_DISAGGREGATION_NIXL_BACKEND_PARAMS` is left unchanged and still overrides `UCX_NET_DEVICES`; do not set it if you rely on the environment's complete device list.

## ASCEND

### Usage

Use ascend backend with [memfabric\_hybrid](https://gitcode.com/Ascend/memfabric_hybrid) and ASCEND\_MF\_STORE\_URL being set

```bash Command theme={null}
pip install memfabric-hybrid==1.0.0
export ASCEND_MF_STORE_URL="tcp://xxx.xx.xxx.xxx:xxxx"
```

Use mooncake backend, more details can be found in mooncake section.

```bash theme={null}
export ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE=true
```

ASCEND\_NPU\_PHY\_ID need to be set in container env

```bash theme={null}
export ASCEND_NPU_PHY_ID=xxx
```

### MIMO Single Node

#### prefill

```bash theme={null}
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export STREAMS_PER_DEVICE=32
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export HCCL_BUFFSIZE=1600
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
export SGLANG_NPU_PROFILING=0
export SGLANG_NPU_PROFILING_STAGE="prefill"
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669"
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export SGLANG_DEEPEP_BF16_DISPATCH=0
export ASCEND_USE_FIA=1

python3 -m sglang.launch_server \
        --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \
        --attention-backend ascend \
        --device npu \
        --tp-size 8 --nnodes 1 --node-rank 0 \
        --chunked-prefill-size -1 \
        --trust-remote-code --port 10000 \
        --host 127.0.0.1 --max-running-requests 16 \
        --mem-fraction-static 0.8 \
        --disaggregation-mode prefill --disaggregation-transfer-backend ascend \
        --disaggregation-bootstrap-port 8996 \
        --base-gpu-id 0 \
        --disable-radix-cache \
        --disable-cuda-graph \
        --moe-a2a-backend deepep --deepep-mode normal \
        # 2>&1 | tee $SGLANG_LOG_PATH
```

#### decode

```bash theme={null}
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# export ASCEND_LAUNCH_BLOCKING=1

# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export STREAMS_PER_DEVICE=32
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256
export HCCL_BUFFSIZE=1600
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
export SGLANG_NPU_PROFILING=0
export SGLANG_NPU_PROFILING_STAGE="prefill"
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export ASCEND_MF_STORE_URL="tcp://127.0.0.1:24669"
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export SGLANG_DEEPEP_BF16_DISPATCH=0
export ASCEND_USE_FIA=1

python3 -m sglang.launch_server \
        --model-path  /path/to/MiMo-V2-Flash-w8a8-all-0512 \
        --attention-backend ascend \
        --device npu \
        --tp-size 8 --nnodes 1 --node-rank 0 \
        --trust-remote-code --port 10001 \
        --host 127.0.0.1 --max-running-requests 16 \
        --mem-fraction-static 0.8 \
        --disaggregation-mode decode --disaggregation-transfer-backend ascend \
        --disaggregation-bootstrap-port 8996 \
        --base-gpu-id 8 \
        --disable-radix-cache \
        --cuda-graph-bs-decode 1 2 4 8 10 12 14 16 \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
        --enable-multi-layer-eagle \
        --moe-a2a-backend deepep --deepep-mode low_latency \
        # 2>&1 | tee $SGLANG_LOG_PATH
```

#### router

```bash theme={null}
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://127.0.0.1:10000 \
    --decode  http://127.0.0.1:10001 \
    --host 127.0.0.1 \
    --port 9903 \
    --health-check-interval-secs 3600 \
    --mini-lb \
```

### DeepSeek Multi-Node

#### Environment

```bash theme={null}
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH}
export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
# IP set to p first node ip
export ASCEND_MF_STORE_URL="tcp://XXXXXX:24670"

# p node IP
P_IP=('XXXXX')
# D node IP
D_IP=('XXXXX')


# enable mlapo
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export ENABLE_MOE_NZ=1
#export SGLANG_NPU_USE_MULTI_STREAM=1

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
```

#### prefill

```bash theme={null}
MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export HCCL_BUFFSIZE=2600
        export HCCL_SOCKET_IFNAME=lo
        export GLOO_SOCKET_IFNAME=lo
        python -m sglang.launch_server --model-path ${MODEL_PATH}  --disaggregation-mode prefill --host ${P_IP[$i]} \
        --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
        --tp-size 16 --mem-fraction-static 0.7 --attention-backend ascend --device npu --quantization modelslim \
        --disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192  --disable-radix-cache \
        --chunked-prefill-size -1 --max-prefill-tokens 10240 --moe-a2a-backend deepep --deepep-mode normal \
        --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2  \
        --attn-dp-size 8 --disable-shared-experts-fusion --dtype bfloat16
        NODE_RANK=$i
        break
    fi
done
```

#### decode

```bash theme={null}
MODEL_PATH=/path/to/deepseekr1_w4a8_pertoken

for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export HCCL_BUFFSIZE=900
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112
        export TASK_QUEUE_ENABLE=1
        export HCCL_SOCKET_IFNAME=data0.3001
        export GLOO_SOCKET_IFNAME=data0.3001
        python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
        --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --attn-dp-size 16 \
        --mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \
        --moe-a2a-backend deepep --deepep-mode low_latency --enable-dp-lm-head \
        --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
       	--load-balance-method round_robin
        NODE_RANK=$i
        break
    fi
done
```

#### router

```bash theme={null}
python -m  sglang_router.launch_router --prefill ${P_IP}:8000 \
--decode ${D_IP}:8001 \
--host ${D_IP} --port 6688 \
--pd-disaggregation \
--health-check-interval-secs 3600 \
```


# Pipeline Parallelism for Long Context
Source: https://docs.sglang.io/docs/advanced_features/pipeline_parallelism



## Why Pipeline Parallelism?

As Large Language Models (LLMs) scale toward trillion-parameter architectures and "infinite" context windows, the underlying serving infrastructure must evolve toward more granular, cross-node parallelization strategies. While KV cache techniques effectively mitigate redundant computation, they cannot circumvent the prohibitive Time to First Token (TTFT) inherent in ultra-long sequences with extremely large initial Input Token Length (ITL). Although Tensor Parallelism (TP) remains the conventional approach for intra-node scaling, it frequently encounters communication bottlenecks during multi-node deployments. On the other hand, pipeline parallelism only requires cross-node communication at the boundaries of each pipeline stage, which can achieve better computation-communication overlap compared to a large TP. Therefore, it is also a promising parallelization strategy for improving throughput.

Detailed analysis can be found in this [blog](https://lmsys.org/blog/2026-01-15-chunked-pipeline/).

## Implementation Refactoring based on Async Communication

With Dynamic Chunked Prefill, pipeline parallelism has the potential to reduce the TTFT of long-context inputs. For each request, its input tokens can be partitioned into multiple chunks, each no longer than the chunked prefill size. Different chunks of the same request can be processed simultaneously by different nodes, thus parallelizing the processing and reducing TTFT. SGLang has supported Pipeline Parallelism (#5724) for some time and made it compatible with the PD Disaggregation feature (#8846), but the implementation was not perfect and had significant room for performance improvements.

To eliminate this performance hazard, SGLang implements a Micro-batching Event Loop with non-blocking asynchronous peer-to-peer (P2P) communication to overlap GPU computation with CPU metadata processing and PP communication. This ensures that while one micro-batch is being computed on the GPU, the next one is already being prepared and moved into position effectively, ensuring the pipeline remains as saturated as possible. This approach was first proposed in #7979 and has been redesigned and included in #11852.

The key mechanisms of the implementation include:

* **Decoupled Sync/Async Logic in the Event Loop:** The scheduler uses `async_send` in `_pp_send_pyobj_to_next_stage`. Instead of waiting for a transfer to complete, it returns a `P2PWork` handle. The actual synchronization (`P2PWork.work.wait()`) is deferred until `_pp_commit_comm_work` is called, allowing the CPU to perform other work—like scheduling the next batch or processing metadata—while data is in flight.
* **Multi-Stream Execution:** In addition to the main `default_stream`, which serves as the synchronization stream, SGLang utilizes dedicated `forward_stream` and `copy_stream` to execute forward pass GPU computation and Data-to-Host (D2H) memory transfers separately for better overlapping. While `_pp_launch_batch` is executing the current micro-batch on the GPU for the current stage, the CPU processes the previous micro-batch's results using `_pp_process_batch_result`.

## Guidance about Dynamic Chunking

### Why Dynamic Chunking

Chunked prefill with a fixed size can cause bubbles in the pipeline, especially when the pp size is large. The main reason behind this phenomenon is that the model has a non-uniform running time, even though each chunk size is identical (brought by the Transformer structure). The larger the prefix sequence length, the longer the running time of the chunk. And these bubbles will be propagated to the next stage, and will significantly degrade the scale efficiency of larger pp ranks.

To address this issue, SGLang introduces a dynamic chunking mechanism to predict the optimal size for the next chunk such that it satisfies this condition:

Runtime(L + Next Chunk Size) - Runtime(L) = Runtime(Initial Chunk Size)

where ***L*** denotes the Prefix Sequence Length. By profiling a series of requests with different ITLs, we model the cumulative runtime as a quadratic function of sequence length. Using this model, we solve the optimal next chunk size for any given prefix length ***L***. Since the computation complexity of the Attention mechanism scales with ***L***, the next chunk size will be progressively reduced as ***L*** grows to maintain an aligned chunk execution time across pipeline stages.

Based on this method, the scheduler can predict and dynamically reduce the chunk size during runtime to minimize the bubbles caused by the stage misalignment. To be noticed, the scheduler does not use the raw predicted value. To facilitate efficient KVCache memory management and ensure affinity with hardware execution efficiency, the value is aligned downward to the nearest multiple of max(`--page-size`, 64).

### Chunked Prefill Size and Smoothing Factor

When `--enable-dynamic-chunking` is enabled, each chunk size of a sequence is determined dynamically based on the quadratic model that predicts the next chunk size based on the estimated runtime of the initial chunk length. In this case, we use `--chunked-prefill-size` to set up the initial chunk size. When switching to the dynamic chunking mode, the initial chunk size (`--chunked-prefill-size`) should be set to a larger value comparable to the original chunked prefill size, so that there won't be too many chunks.

**`SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR`** is an environmental variable that controls the smoothing factor for the dynamic chunking algorithm, defaulting to 0.75. It determines how much the chunk size can change during the prefill phase. A larger value means a more aggressive chunk size change, which may lead to better performance but also to greater chunk size changes (the chunk size at the end may become very small, which could lead to performance degradation) and more total chunks. When it is set to 1, the chunk size will be adjusted strictly based on the aforementioned quadratic model that predicts the next chunk size. A smaller value means a more conservative chunk size change, which may lead to smaller chunk size changes and fewer total chunks. When it is set to 0, the chunk size will not be adjusted dynamically, so it is identical to the traditional way with a fixed chunked prefill size.

Due to the variation in hardware, models, and target workloads, a static configuration is seldom optimal across all scenarios. Consequently, achieving peak performance necessitates a degree of hyperparameter tuning when switching to the dynamic chunking mode.

**Tuning Guidance for Dynamic Chunked Prefill**

* **Step 1 - Iterate to find the optimal fixed chunked prefill size for the targeted PP size**: Different PP sizes for targeted ITL may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling.
* **Step 2 - Initial Chunk Size Selection for Dynamic Chunking**: Set the initial size to 2× or 3× the optimal fixed chunked prefill size. This reduces the total number of chunks and prevents "tail chunks" from underutilizing hardware. To maintain efficiency for extremely large Input Token Lengths (ITL), the dynamic predictor automatically ensures subsequent chunks are at least 1/4 of this initial size. In addition, it is recommended to use a larger initial chunk size (e.g., 4× the optimal fixed chunked prefill size) for such cases as well.
* **Step 3 - Smooth Factor Adjustment**: This factor controls how strictly the chunk size adjusts the prediction given by the quadratic performance fitting model.
  * 1.0: Follows the model strictly.
  * **0.6 – 0.85 (Recommended)**: Typical range for the best balance between dynamic scaling and hardware stability. Through experiments, we find that a range between 0.6 and 0.85 typically yields the best performance for dynamic chunking.
  * 0: Disables dynamic adjustment, reverting to traditional fixed-size chunking.
* **Another small optimization tip:** Put the larger partition in the higher PP rank when the layers are not evenly divisible across ranks. It can increase the GPU utilization when a larger PP rank is waiting for the previous stage’s result, hence reducing the bubbles on higher PP ranks. If we take DeepSeek-V3.1 as an example, `SGLANG_PP_LAYER_PARTITION=15,15,15,16` usually performs better than `16,15,15,15`.

## Best Practice for Long Context

### Tuning the Chunked Prefill Size

Optimizing the chunked prefill size is crucial for balancing pipeline efficiency and resource utilization. The ideal size depends on factors including model architecture, hardware configuration, and typical input lengths. We recommend starting with a small chunk size, such as 4K, and gradually increasing it until you find the optimal size for your specific use case (Different targeted ITL and PP Sizes may have different optimal chunked prefill sizes. Therefore, users should iterate to obtain the baseline according to the available resources for scaling). Alternatively, you can analyze the hardware capacity and determine the optimal chunk size based on the roofline model.

### Enable Dynamic Chunking and Adjust Smoothing Factor for Ultra-long ITL

SGLang also offers a dynamic chunking solution that could further improve performance. This feature is currently an experimental feature that requires a certain amount of tuning experimentation and may not be suitable for all workloads. In addition, fine-tuning the smoothing factor can help optimize performance for specific workloads and model characteristics.

### Case Study on NVIDIA H20

When evaluating pipeline parallelism with fixed chunked prefill sizes from 2K to 16K, experiment results show that a 4K chunk size delivered optimal prefill TTFT performance for the DeepSeek-V3.1, and a 6K chunk size delivered optimal prefill TTFT performance for the Qwen3-235B-A22B-FP8.

When enabling dynamic chunking, we first scale the optimal fixed chunked prefill size by a factor of 3 as the initial chunk size. Through experimentation, we found that a multiplier of 2-3 provides an appropriate balance—avoiding excessive initial pipeline bubbles while ensuring that subsequent chunks don't become too small as context length increases. With the default dynamic chunking smoothing factor of 0.75, we performed parameter tuning and determined that a value of 0.65 works optimally with the 12K initial chunk size for the DeepSeek-V3.1, while a value of 0.8 works optimally with the 18K initial chunk size for the Qwen3-235B-A22B-FP8.

#### DeepSeek-V3.1 with 128K Input Token Length

```bash Command theme={null}
# prefill node 0 (fixed chunked prefill size)
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \
  --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
  --disable-radix-cache --mem-fraction-static 0.8  \
  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
  --max-running-requests 128 --chunked-prefill-size 4096
```

```bash Command theme={null}
# prefill node 0 (with dynamic chunking)
export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.65
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \
  --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
  --disable-radix-cache --mem-fraction-static 0.8  \
  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
  --max-running-requests 128 --chunked-prefill-size 12288 --enable-dynamic-chunking
```

#### Qwen3-235B-A22B-FP8 with 128K Input Token Length

```bash Command theme={null}
# prefill node 0 (fixed chunked prefill size)
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \
  --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \
  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
  --disable-radix-cache --mem-fraction-static 0.8  \
  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
  --max-running-requests 128 --chunked-prefill-size 6144
```

```bash Command theme={null}
# prefill node 0 (with dynamic chunking)
export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.8
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-235B-A22B-FP8 --trust-remote-code \
  --nnodes 4 --node-rank 0 --tp 4 --pp-size 8 \
  --port 30000 --dist-init-addr <MASTER_NODE_IP> \
  --disable-radix-cache --mem-fraction-static 0.8  \
  --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
  --max-running-requests 128 --chunked-prefill-size 18432 --enable-dynamic-chunking
```

Note: `--disable-radix-cache` is enabled only for reproducible benchmarking purposes. It is not recommended to use it in production.

## Best Practice for Pipeline Parallelism with PD Disaggregation

To be added. Stay tuned for the latest updates on Pipeline Parallelism with PD Disaggregation.


# Prefill Context Parallelism
Source: https://docs.sglang.io/docs/advanced_features/prefill_cp



Prefill context parallelism (CP) distributes the tokens of a prompt across an attention CP group. Each rank computes attention and, for models with an indexer, indexer operations for its local queries, while all-gather on the KV cache makes the required keys and values available. This lets multiple GPUs share the work of processing a long prompt.

To compare prefill CP with other common parallelism strategy: Tensor parallelism (TP) splits attention heads and weights, [Decode context parallelism (DCP)](https://docs.sglang.io/docs/advanced_features/dcp) shards the KV cache and its attention workload during decode, while prefill CP (PCP) splits prompt tokens across GPUs. Note that PCP and DCP are independent optimizations for different phases of inference.

## Why prefill context parallelism?

Prefill CP offers three benefits:

* It partitions work that would otherwise be repeated across ranks, such as indexer computation in DSA models and the DeepSeek V4 series.
* It combines naturally with KV-cache sharding to reduce per-rank cache memory. Layer-wise sharding is available through DSA cache LayerSplit; additional sharding techniques are under development.
* Combined with expert parallelism (EP), it can reduce communication time by using all-to-all token dispatch and combine instead of TP all-reduce. The benefit depends on the workload and interconnect.

## Architecture

### Strategy interface

`ContextParallelStrategy` separates token-layout policy from model execution and attention kernels. Server argument resolution initializes one strategy per process. Per-forward state lives in `ForwardBatch.attn_cp_metadata`, rather than in the model's token-layout logic.

### Zigzag strategy

For a CP group of size `C`, split each request's newly extended tokens into `2C` contiguous blocks. Rank `r` receives block `r` and block `2C - 1 - r`. The split restarts for every request in a batch.

For example, with CP=4 and 16 new tokens, each block has two tokens:

```mermaid theme={null}
flowchart TB
    S["One request, original token order<br/>B0: 0–1 · B1: 2–3 · B2: 4–5 · B3: 6–7<br/>B4: 8–9 · B5: 10–11 · B6: 12–13 · B7: 14–15"]
    S --> R0["Rank 0<br/>B0 + B7<br/>tokens 0–1, 14–15"]
    S --> R1["Rank 1<br/>B1 + B6<br/>tokens 2–3, 12–13"]
    S --> R2["Rank 2<br/>B2 + B5<br/>tokens 4–5, 10–11"]
    S --> R3["Rank 3<br/>B3 + B4<br/>tokens 6–7, 8–9"]
```

Pairing an early, cheaper block with a late, more expensive block balances causal attention work while retaining contiguous query blocks. Lengths need not divide evenly: early blocks receive the remainder, and communication buffers are padded as needed.

The metadata records separate query and KV lengths for the early and late blocks, including any cached prefix. Each block therefore attends to the correct causal history, not just the other tokens assigned to its rank.

### Interleave strategy

For CP size `C`, rank `r` receives flattened token indices `r, r+C, r+2C, ...`. The index runs across the batch's newly extended tokens; it does not reset at request boundaries. Position IDs keep their original values.

With CP=4 and 16 new tokens:

```mermaid theme={null}
flowchart TB
    S["Flattened new tokens in batch order<br/>0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15"]
    S --> R0["Rank 0<br/>0, 4, 8, 12"]
    S --> R1["Rank 1<br/>1, 5, 9, 13"]
    S --> R2["Rank 2<br/>2, 6, 10, 14"]
    S --> R3["Rank 3<br/>3, 7, 11, 15"]
```

This gives each rank queries spread across the context. For a batch containing requests of lengths 5 and 3, rank 0 owns flattened indices 0 and 4, while rank 1 owns indices 1 and 5; index 5 is the first new token of the second request. Per-request sharding metadata preserves those boundaries and each query's causal extent.

### Attention backend integration

| Path | Integration |
| - | - |
| Zigzag with FlashAttention | Gather and reorder new KV; execute attention for the early and late query blocks with their respective sequence lengths. |
| Zigzag with `trtllm_mha` | Materialize KV, build zigzag page tables, and execute a combined variable-length attention call. |
| Interleave with DSA | Shard query/indexer metadata by request, gather indexer keys and MLA KV as required, and execute sparse attention in the backend. |
| DeepSeek V4 interleave | Uses model-specific sparse-attention integration; the CP regression suite includes DeepSeek-V4-Flash on B200. Do not infer its backend requirements from the dense zigzag path. |

The CP and attention-backend compatibility matrix is being expanded, with support for more combinations planned.

## Compositions

### Prefill CP × TP, DP, and EP

Attention TP splits heads within each CP partition; attention DP separates request batches. Dense FFNs and MoE layers can use a different layout, bridged by the layer boundaries between stages (see [Layer boundaries](/docs/developer_guide/layer_boundary)). The Qwen3 example in [Usage and Examples](#usage-and-examples) combines CP=2 with attention TP=2 and EP=4. For interleave DSA on DeepSeek V3.2 and the GLM-5 series, keep DP=1 and use the resolved attention CP=TP topology.

### Prefill CP × pipeline parallelism

The eager runner preserves CP-local hidden states between pipeline stages and gathers on the final stage. PP can place stages on different nodes while keeping each DSA attention CP group within one node; it does not remove the single-machine DSA CP restriction.

### Prefill CP × speculative decoding

Prefill CP can coexist with supported speculative-decoding configurations. The CP runner handles aligned speculative hidden-state inputs and gathers supported auxiliary outputs.

### Prefill CP × CUDA graphs

The shared CP breakable-prefill-graph path currently requires `zigzag`, the `trtllm_mha` prefill backend, PP=1, and attention CP size equal to TP size. It uses CP-local static input buffers and selects capture buckets large enough for each rank's padded input. Support for combining other attention backends with prefill CP and Breakable CUDA Graph is under development.

### Prefill CP × PD disaggregation and DSA cache LayerSplit

In supported DSA deployments, `--enable-dsa-cache-layer-split` distributes GPU KV/indexer cache layers across CP ranks to reduce per-rank cache memory. Each rank owns a range of layers and uses scratch storage for remote-layer data. This is layer-wise cache ownership, separate from the interleave token assignment used for computation, and is not applied to draft workers.

LayerSplit requires a PD prefill worker (`--disaggregation-mode prefill`), `--enable-prefill-cp --cp-strategy interleave`, and PP=1. It currently supports the `mooncake` and `mooncake_tcp` transfer backends. Do not enable LayerSplit on decode or non-PD workers; the decode side receives full cache shards through PD transfer. Use the [PD disaggregation guide](https://docs.sglang.io/docs/advanced_features/pd_disaggregation) and model-specific launch recipes to configure both sides of the transfer; enabling prefill CP alone does not configure a PD deployment.

## Usage and Examples

Use NVIDIA CUDA GPUs and select a strategy supported by your model and attention backend. The examples below target a single Linux host with Hopper GPUs and an SGLang version that includes the selected backend.

| Argument | Default | Description |
| - | - | - |
| `--enable-prefill-cp` | `False` | Enable context-parallel prefill. |
| `--cp-strategy` | Unset | Required when enabling prefill CP. Choose `zigzag` or `interleave`. |
| `--attn-cp-size` | `1` | Number of ranks in an attention CP group. Use a value greater than 1 to distribute work; model-specific argument resolution may adjust it. |

CP subdivides the existing TP world; it does not add another GPU multiplier. With attention DP enabled, the attention topology is:

```text theme={null}
TP world size = attention DP size × attention CP size × attention TP size

Example: --tp 4 --dp 1 --attn-cp-size 2
         → one attention DP group, CP=2, attention TP=2
```

The server requires `--tp` to be divisible by `--dp × --attn-cp-size`. The attention layout does not by itself specify the dense FFN or MoE layout; those use their own parallelism settings.

> [!WARNING]
>
> For DSA models, such as the GLM-5 series and DeepSeek V3.2, use `interleave`; `zigzag` is currently unavailable. Do not use a smaller `--attn-cp-size` to request hybrid attention TP/CP for DSA models. Broader compatibility is planned for future support.

### Zigzag example: GQA with attention TP and CP

This Qwen3 configuration uses four Hopper GPUs, with attention TP=2, CP=2, and EP=4 for the MoE layers. The model is a Hugging Face repository ID passed to `--model-path`.

```bash theme={null}
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-30B-A3B-FP8 \
  --tp 4 \
  --ep 4 \
  --attn-cp-size 2 \
  --enable-prefill-cp \
  --cp-strategy zigzag \
  --attention-backend fa3
```

### Interleave example: DSA

This GLM-5.2 configuration uses eight H200 GPUs with attention CP=8 and attention TP=1. The DSA backend is selected automatically for this model; the command makes that choice explicit.

```bash theme={null}
python -m sglang.launch_server \
  --model-path zai-org/GLM-5.2-FP8 \
  --tp 8 \
  --attn-cp-size 8 \
  --enable-prefill-cp \
  --cp-strategy interleave \
  --attention-backend dsa \
  --trust-remote-code \
  --mem-fraction-static 0.85
```

For model-specific deployment combinations, see [context parallelism in the GLM-5.3 cookbook](https://docs.sglang.io/cookbook/autoregressive/GLM/GLM-5.3#3-5-context-parallelism).

## References

* [CP strategies and metadata](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/layers/cp): strategy interface, zigzag/interleave layouts, padding, and graph helpers.
* [Prefill Context Parallelism Roadmap](https://github.com/sgl-project/sglang/issues/21788).
* [Prefill Context Parallelism Refactor Design](https://github.com/sgl-project/sglang/issues/27252).


# Quantization
Source: https://docs.sglang.io/docs/advanced_features/quantization



SGLang supports various quantization methods, including offline quantization and online dynamic quantization.

Offline quantization loads pre-quantized model weights directly during inference. This is required for quantization methods
such as GPTQ and AWQ, which collect and pre-compute various statistics from the original weights using the calibration dataset.

Online quantization dynamically computes scaling parameters—such as the maximum/minimum values of model weights—during runtime.
Like NVIDIA FP8 training's [delayed scaling](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/examples/fp8_primer.html#Mixed-precision-training-with-FP8) mechanism, online quantization calculates the appropriate scaling factors
on-the-fly to convert high-precision weights into a lower-precision format.

**Note: For better performance, usability and convenience, offline quantization is recommended over online quantization.**

If you use a pre-quantized model, **do not add `--quantization` to enable online quantization at the same time**.
For popular pre-quantized models, please visit [Unsloth](https://huggingface.co/unsloth), [NVIDIA ModelOpt](https://huggingface.co/collections/nvidia/inference-optimized-checkpoints-with-model-optimizer)
or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on HF for some
popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization
to guard against abnormal quantization loss regressions.

## Platform Compatibility

The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs.

<table>
  <thead>
    <tr>
      <th>Method</th>
      <th>NVIDIA GPUs</th>
      <th>AMD GPUs (MI300X/MI325X/MI350X)</th>
      <th>Ascend NPUs (A2/A3 Series, 950PR/DT Series)</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`fp8`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>WIP</td>
      <td>Aiter or Triton backend on AMD</td>
    </tr>

    <tr>
      <td>`mxfp4`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Yes (950PR/DT Series)</td>
      <td>On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (950PR/DT Series): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline `W4A4_MXFP4` dense and MoE checkpoints (single-level) are auto-detected via `modelslim`. On Intel GPUs (XPU): W4A16 MoE experts on Xe2/BMG via `sgl-kernel-xpu`, enabled automatically with `--device xpu` (see [Intel GPUs (XPU)](../hardware-platforms/xpu))</td>
    </tr>

    <tr>
      <td>`mxfp8`</td>
      <td>No</td>
      <td>No</td>
      <td>Yes (950PR/DT Series for Diffusion, LLM Dense Linear and LLM MoE)</td>
      <td>Ascend NPU only; online + offline MXFP8 for Diffusion models (e.g., Wan2.2), LLM Dense Linear, and LLM MoE (FusedMoE, e.g. Qwen3-30B-A3B) on 950PR/DT Series; uses CANN `npu_dynamic_mx_quant` / `npu_quant_matmul` (dense) and `npu_grouped_matmul_swiglu_quant_v2` / `npu_grouped_matmul` (MoE) kernels</td>
    </tr>

    <tr>
      <td>`mxfp_w4a8`</td>
      <td>No</td>
      <td>No</td>
      <td>Yes (950PR/DT Series)</td>
      <td>Ascend NPU only; online W4A8 for Qwen3 dense LLM (MXFP4 weights + MXFP8 activations) on 950PR/DT Series; offline `W4A8_MXFP` dense and MoE checkpoints are auto-detected via `modelslim`</td>
    </tr>

    <tr>
      <td>`blockwise_int8`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>No</td>
      <td>Triton-based, works on both platforms</td>
    </tr>

    <tr>
      <td>`w8a8_int8`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>No</td>

      <td />
    </tr>

    <tr>
      <td>`w8a8_fp8`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>No</td>
      <td>Aiter or Triton FP8 on AMD</td>
    </tr>

    <tr>
      <td>`awq`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Uses a JIT-compiled CUDA kernel on NVIDIA, Triton dequantize on AMD. Uses CANN kernels on Ascend</td>
    </tr>

    <tr>
      <td>`gptq`</td>
      <td>No</td>
      <td>No</td>
      <td>Yes</td>
      <td>Removed on NVIDIA and AMD GPUs — use `gptq_marlin` instead. Uses CANN kernels on Ascend. Still supported on Intel CPUs with AMX</td>
    </tr>

    <tr>
      <td>`compressed-tensors`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Partial</td>
      <td>Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, `FP8` not supported yet</td>
    </tr>

    <tr>
      <td>`quark`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>No</td>
      <td>AMD Quark quantization; Aiter GEMM paths on AMD</td>
    </tr>

    <tr>
      <td>`auto-round`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Partial</td>
      <td>Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend</td>
    </tr>

    <tr>
      <td>`quark_int4fp8_moe`</td>
      <td>No</td>
      <td>Yes</td>
      <td>No</td>
      <td>AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4)</td>
    </tr>

    <tr>
      <td>`awq_marlin`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>Marlin kernels are CUDA-only</td>
    </tr>

    <tr>
      <td>`gptq_marlin`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>Marlin kernels are CUDA-only</td>
    </tr>

    <tr>
      <td>`gguf`</td>
      <td>Yes</td>
      <td>No</td>
      <td>Yes</td>
      <td>CUDA kernels in sgl-kernel; Ascend uses CPU pre-dequantization at load time</td>
    </tr>

    <tr>
      <td>`modelopt` / `modelopt_fp8`</td>
      <td>Yes (Hopper/SM90+)</td>
      <td>No</td>
      <td>No</td>
      <td>[NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); requires NVIDIA hardware</td>
    </tr>

    <tr>
      <td>`modelopt_fp4`</td>
      <td>Yes (SM80-SM90 via Marlin; SM100+ native FP4)</td>
      <td>No</td>
      <td>No</td>
      <td>[NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); use Marlin W4A16 fallback on Ampere/Hopper and native FP4 backends on Blackwell; supports load-time BF16/FP16/FP8 MoE conversion with per-tensor FP32 activation scales</td>
    </tr>

    <tr>
      <td>`nvfp4_online`</td>
      <td>Yes (Blackwell/SM100 or SM103)</td>
      <td>No</td>
      <td>No</td>
      <td>Online MoE-only NVFP4 weight quantization with per-token FP32 activation scales for BF16/FP16/FP8 checkpoints; use `modelopt_fp4` for per-tensor FP32 activation scales; requires `flashinfer_trtllm` or `flashinfer_trtllm_routed`</td>
    </tr>

    <tr>
      <td>`petit_nvfp4`</td>
      <td>No</td>
      <td>Yes (MI250/MI300X/MI325X)</td>
      <td>No</td>
      <td>Enables NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel); use `modelopt_fp4` on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/) and [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html).</td>
    </tr>

    <tr>
      <td>`bitsandbytes`</td>
      <td>Yes</td>
      <td>Experimental</td>
      <td>No</td>
      <td>Depends on bitsandbytes ROCm support</td>
    </tr>

    <tr>
      <td>`modelslim`</td>
      <td>No</td>
      <td>No</td>
      <td>Yes</td>
      <td>Ascend quantization; Uses CANN kernels</td>
    </tr>
  </tbody>
</table>

On AMD, several of these methods use [Aiter](https://github.com/ROCm/aiter) for acceleration -- set `SGLANG_USE_AITER=1` where noted. See [AMD GPU setup](../hardware-platforms/amd_gpu) for installation and configuration details.

On Ascend, various layers quantization configurations are supported, see [Ascend NPU quantization](../hardware-platforms/ascend-npus/optimization/quantization) for details.

## GEMM Backends for FP4/FP8 Quantization

> [!NOTE]
>
> Backend selection applies to **blockwise FP8**, **MXFP8** (dense linear), and **NVFP4** GEMM. When running offline or online FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`.

### `--fp8-gemm-backend` (Blockwise FP8 GEMM)

<table>
  <thead>
    <tr>
      <th>Backend</th>
      <th>Hardware</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`auto`</td>
      <td>All</td>
      <td>Auto-selects based on hardware</td>
    </tr>

    <tr>
      <td>`deep_gemm`</td>
      <td>SM90, SM100</td>
      <td>JIT-compiled; enabled when DeepGEMM is installed</td>
    </tr>

    <tr>
      <td>`flashinfer_trtllm`</td>
      <td>SM100</td>
      <td>FlashInfer TensorRT-LLM backend; optimal for low-latency</td>
    </tr>

    <tr>
      <td>`flashinfer_cutlass`</td>
      <td>SM100/120</td>
      <td>FlashInfer CUTLASS groupwise FP8 GEMM</td>
    </tr>

    <tr>
      <td>`flashinfer_deepgemm`</td>
      <td>SM90</td>
      <td>Uses swapAB optimization for small M dimensions in decoding</td>
    </tr>

    <tr>
      <td>`cutlass`</td>
      <td>SM120</td>
      <td>sgl-kernel CUTLASS</td>
    </tr>

    <tr>
      <td>`triton`</td>
      <td>All</td>
      <td>Fallback; widely compatible</td>
    </tr>

    <tr>
      <td>`aiter`</td>
      <td>ROCm</td>
      <td>AMD AITER backend</td>
    </tr>
  </tbody>
</table>

**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM120); 4) AITER (AMD); 5) Triton (fallback).

**MXFP8 dense linear:** `auto` uses `flashinfer_cutlass` on SM100 (else `triton`). `flashinfer_cutlass` is fastest on most shapes; `flashinfer_trtllm` is faster only at small M.

### `--fp4-gemm-backend` (NVFP4 GEMM)

<table>
  <thead>
    <tr>
      <th>Backend</th>
      <th>Hardware</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`auto`</td>
      <td>SM80+</td>
      <td>Auto-selects: `flashinfer_cutedsl` on SM100; `marlin` on SM80-SM90; `flashinfer_cutlass` otherwise (including SM120)</td>
    </tr>

    <tr>
      <td>`flashinfer_cutlass`</td>
      <td>SM100/120</td>
      <td>FlashInfer CUTLASS backend</td>
    </tr>

    <tr>
      <td>`flashinfer_cudnn`</td>
      <td>SM100/120 (CUDA 13+, cuDNN 9.15+)</td>
      <td>FlashInfer cuDNN backend</td>
    </tr>

    <tr>
      <td>`flashinfer_cutedsl`</td>
      <td>SM100</td>
      <td>FlashInfer CuTe DSL backend</td>
    </tr>

    <tr>
      <td>`flashinfer_trtllm`</td>
      <td>SM100</td>
      <td>FlashInfer TensorRT-LLM backend</td>
    </tr>

    <tr>
      <td>`marlin`</td>
      <td>SM80-SM90</td>
      <td>Weight-only W4A16 fallback for NVFP4 checkpoints</td>
    </tr>
  </tbody>
</table>

On SM80-SM90, `auto` selects Marlin for NVFP4. NVFP4 GEMM requires FlashInfer to be installed.

## Offline Quantization

To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline,
there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the
downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.**

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
    --port 30000 --host 0.0.0.0
```

Take note, if your model is **per-channel quantized (INT8 or FP8) with per-token dynamic quantization activation**, you can opt to include `--quantization w8a8_int8` or `--quantization w8a8_fp8` to invoke the corresponding CUTLASS int8\_kernel or fp8\_kernel in sgl-kernel. This action will ignore the Hugging Face config's quantization settings. For instance, with `neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic`, if you execute with `--quantization w8a8_fp8`, the system will use the `W8A8Fp8Config` from SGLang to invoke the sgl-kernel, rather than the `CompressedTensorsConfig` for vLLM kernels.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8-dynamic \
    --quantization w8a8_fp8 \
    --port 30000 --host 0.0.0.0
```

### Examples of Offline Model Quantization

#### Using [Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide)

We strongly suggest the use of Unsloth to quantize and load the model. Please refer to [SGLang Deployment & Inference Guide with Unsloth](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide).

#### Using [auto-round](https://github.com/intel/auto-round)

```bash Command theme={null}
# Install
pip install auto-round
```

* LLM quantization

```py Example theme={null}
# for LLM
from auto_round import AutoRound
model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-autoround-4bit"
# Scheme examples: "W2A16", "W3A16", "W4A16", "W8A16", "NVFP4", "MXFP4" (no real kernels), "GGUF:Q4_K_M", etc.
scheme = "W4A16"
format = "auto_round"
autoround = AutoRound(model_id, scheme=scheme)
autoround.quantize_and_save(quant_path, format=format) # quantize and save

```

* VLM quantization

```py Example theme={null}
# for VLMs
from auto_round import AutoRoundMLLM
model_name = "Qwen/Qwen2-VL-2B-Instruct"
quant_path = "Qwen2-VL-2B-Instruct-autoround-4bit"
scheme = "W4A16"
format = "auto_round"
autoround = AutoRoundMLLM(model_name, scheme)
autoround.quantize_and_save(quant_path, format=format) # quantize and save

```

* Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)

```bash Command theme={null}
auto-round \
    --model meta-llama/Llama-3.2-1B-Instruct \
    --bits 4 \
    --group_size 128 \
    --format "auto_round" \
    --output_dir ./tmp_autoround
```

* SGlang API Usage (CPU/CUDA)

```python Example theme={null}
from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.model_loader.loader import get_model_loader
from sglang.srt.configs.device_config import DeviceConfig

# Configure model with inc quantization and saving
model_config = ModelConfig(
    model_path="meta-llama/Llama-3.2-3B-Instruct",
    quantization="auto-round-int8",
    trust_remote_code=True,
)

load_config = LoadConfig(
    inc_save_path="./quantized_model",
)
device_config = DeviceConfig(device="cpu")

# Load and quantize the model
model_loader = get_model_loader(load_config, model_config)
quantized_model = model_loader.load_model(
    model_config=model_config,
    device_config=device_config,
)
```

* known issues

Several limitations currently affect offline quantized model loading in sglang, These issues might be resolved in future updates of sglang. If you experience any problems, consider using Hugging Face Transformers as an alternative.

1. Mixed-bit Quantization Limitations

   Mixed-bit quantization is not fully supported. Due to vLLM's layer fusion (e.g., QKV fusion), applying different bit-widths to components within the same fused layer can lead to compatibility issues.

2. Limited Support for Quantized MoE Models

   Quantized MoE models may encounter inference issues due to kernel limitations (e.g., lack of support for mlp.gate layer quantization). please try to skip quantizing these layers to avoid such errors.

3. Limited Support for Quantized VLMs

   > **Details**
   >
   > Qwen2.5-VL-7B
   >
   > auto\_round:auto\_gptq format:  Accuracy is close to zero.
   >
   > GPTQ format:  Fails with:
   >
   > ```text Output theme={null}
   > The output size is not aligned with the quantized weight shape
   > ```
   >
   > auto\_round:auto\_awq and AWQ format:  These work as expected.

4. Limited Support for SGlang API Usage

   SGlang API Usage only supports `auto-round-int8` quantization method now, more quantization methods are on the way.

* CPU serving

  AutoRound INT4 checkpoints (both `auto_round:auto_gptq` and `auto_round:auto_awq` packing formats) can be served on Intel CPUs with AMX support:

  ```bash theme={null}
  SGLANG_USE_CPU_ENGINE=1 python3 -m sglang.launch_server \
      --model-path OPEA/Qwen2.5-0.5B-Instruct-int4-sym-inc \
      --quantization auto-round \
      --device cpu --trust-remote-code
  ```

  The current SGLang CPU backend supports only 4-bit AutoRound checkpoints on Intel AMX. Other AutoRound bit-widths and non-AMX CPU backends are not supported by this path.

#### Using [GPTQModel](https://github.com/ModelCloud/GPTQModel)

```bash Command theme={null}
# install
pip install gptqmodel --no-build-isolation -v
```

```py Example theme={null}
from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig

model_id = "meta-llama/Llama-3.2-1B-Instruct"
quant_path = "Llama-3.2-1B-Instruct-gptqmodel-4bit"

calibration_dataset = load_dataset(
    "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  ).select(range(1024))["text"]

quant_config = QuantizeConfig(bits=4, group_size=128) # quantization config
model = GPTQModel.load(model_id, quant_config) # load model

model.quantize(calibration_dataset, batch_size=2) # quantize
model.save(quant_path) # save model
```

#### Using [LLM Compressor](https://github.com/vllm-project/llm-compressor/)

```bash Command theme={null}
# install
pip install llmcompressor
```

Here, we take quantize `meta-llama/Meta-Llama-3-8B-Instruct` to `FP8` as an example to elaborate on how to do offline quantization.

```python Example theme={null}
from transformers import AutoTokenizer
from llmcompressor.transformers import SparseAutoModelForCausalLM
from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# Step 1: Load the original model.
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"

model = SparseAutoModelForCausalLM.from_pretrained(
  MODEL_ID, device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Step 2: Perform offline quantization.
# Step 2.1: Configure the simple PTQ quantization.
recipe = QuantizationModifier(
  targets="Linear", scheme="FP8_DYNAMIC", ignore=["lm_head"])

# Step 2.2: Apply the quantization algorithm.
oneshot(model=model, recipe=recipe)

# Step 3: Save the model.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
model.save_pretrained(SAVE_DIR)
tokenizer.save_pretrained(SAVE_DIR)
```

Then, you can directly use the quantized model with `SGLang`, by using the following command:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path $PWD/Meta-Llama-3-8B-Instruct-FP8-Dynamic \
    --port 30000 --host 0.0.0.0
```

#### Using [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)

NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware.

**Offline vs. Online Quantization:**

SGLang supports two modes for ModelOpt.

* **Offline Quantization (pre-quantized):**
  * **Usage:** Download a pre-quantized model from Hugging Face or run `hf_ptq.py` once to create a new quantized checkpoint. Then load this quantized checkpoint.
  * **Pros:** Fast server startup, quantization can be validated before deployment, efficient resource usage.
  * **Cons:** Requires an extra preparation step.

* **Online Quantization (quant and serve):**
  * **Usage:** Load a standard BF16/FP16 model and add a flag. The engine applies quantization *on startup*.
  * **Pros:** Convenient (no new checkpoint needed).
  * **Cons:** **High startup time**, increases VRAM usage during initialization (risk of OOM).

The following sections guide you through using the Offline path: loading pre-quantized models or creating your own checkpoints.

##### Using Pre-Quantized Checkpoints

If a model is already quantized (e.g., from Hugging Face), you can load it directly.

* **FP8 Models:**
  Use `--quantization modelopt_fp8`.
  ```bash Command theme={null}
  python3 -m sglang.launch_server \
      --model-path nvidia/Llama-3.1-8B-Instruct-FP8 \
      --quantization modelopt_fp8 \
      --port 30000
  ```

* **FP4 Models:**
  Use `--quantization modelopt_fp4`.
  ```bash Command theme={null}
  python3 -m sglang.launch_server \
      --model-path nvidia/Llama-3.3-70B-Instruct-NVFP4 \
      --quantization modelopt_fp4 \
      --port 30000
  ```

##### Creating Your Own Quantized Checkpoints

If a pre-quantized checkpoint is not available for your model, you can create one using NVIDIA Model Optimizer's `hf_ptq.py` script.

**Why quantize?**

* Reduce VRAM usage
* Higher throughput and lower latency
* More flexible deployment (on smaller GPUs)

**What can be quantized?**

* The entire model
* MLP layers only
* KV cache

**Key options in `hf_ptq.py`:**

`--qformat`: Quantization formats `fp8`, `nvfp4`, `nvfp4_mlp_only`

`--kv_cache_qformat`: KV cache quantization format (default: `fp8`)

**Note:** The default `kv_cache_qformat` may not be optimal for all use cases. Consider setting this explicitly.

**Hardware requirements:** Hopper and higher are recommended. Insufficient GPU memory may cause weight offloading, resulting in extremely long quantization time.

For detailed usage and supported model architectures, see [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq).

SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.

##### Installation

First, install ModelOpt:

```bash Command theme={null}
pip install nvidia-modelopt
```

##### Quantization and Export Workflow

SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow. Run from the SGLang repository root (see [modelopt\_quantize\_and\_export.py](https://github.com/sgl-project/sglang/blob/main/examples/usage/modelopt_quantize_and_export.py)):

```bash Command theme={null}
# Quantize and export a model using ModelOpt FP8 quantization
python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --export-dir ./quantized_tinyllama_fp8 \
    --quantization-method modelopt_fp8

# For FP4 quantization (requires Blackwell GPU)
python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --export-dir ./quantized_tinyllama_fp4 \
    --quantization-method modelopt_fp4
```

##### Available Quantization Methods

* `modelopt_fp8`: FP8 quantization with optimal performance on NVIDIA Hopper and Blackwell GPUs
* `modelopt_fp4`: FP4 quantization with optimal performance on Nvidia Blackwell GPUs

##### Python API Usage

You can also use ModelOpt quantization programmatically:

```python Example theme={null}
import sglang as sgl
from sglang.srt.configs.device_config import DeviceConfig
from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.model_loader.loader import get_model_loader

# Configure model with ModelOpt quantization and export
model_config = ModelConfig(
    model_path="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
    quantization="modelopt_fp8",  # or "modelopt_fp4"
    trust_remote_code=True,
)

load_config = LoadConfig(
    modelopt_export_path="./exported_model",
    modelopt_checkpoint_save_path="./checkpoint.pth",  # optional, fake quantized checkpoint
)
device_config = DeviceConfig(device="cuda")

# Load and quantize the model (export happens automatically)
model_loader = get_model_loader(load_config, model_config)
quantized_model = model_loader.load_model(
    model_config=model_config,
    device_config=device_config,
)
```

##### Deploying Quantized Models

After quantization and export, you can deploy the model with SGLang:

```bash Command theme={null}
# Deploy the exported quantized model
python -m sglang.launch_server \
    --model-path ./quantized_tinyllama_fp8 \
    --quantization modelopt \
    --port 30000 --host 0.0.0.0
```

Or using the Python API (use the same path as `modelopt_export_path` from the quantize step):

```python Example theme={null}
import sglang as sgl

def main():
    # Deploy exported ModelOpt quantized model
    # Path must match modelopt_export_path from quantize step (e.g., ./exported_model)
    llm = sgl.Engine(
        model_path="./exported_model",
        quantization="modelopt",
    )

    # Run inference
    prompts = [
        "Hello, how are you?",
        "What is the capital of France?",
    ]
    sampling_params = {
        "temperature": 0.8,
        "top_p": 0.95,
        "max_new_tokens": 100,
    }

    outputs = llm.generate(prompts, sampling_params)

    for i, output in enumerate(outputs):
        print(f"Prompt: {prompts[i]}")
        print(f"Output: {output['text']}")

if __name__ == "__main__":
    main()

```

##### Advanced Features

**Checkpoint Management**: Save and restore fake quantized checkpoints for reuse:

```bash Command theme={null}
# Save the fake quantized checkpoint during quantization
python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path meta-llama/Llama-3.2-1B-Instruct \
    --export-dir ./quantized_model \
    --quantization-method modelopt_fp8 \
    --checkpoint-save-path ./my_checkpoint.pth

# The checkpoint can be reused for future quantization runs and skip calibration
```

**Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly. See [LoadConfig](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/configs/load_config.py) for the full API:

```python Example theme={null}
from sglang.srt.configs.device_config import DeviceConfig
from sglang.srt.configs.load_config import LoadConfig
from sglang.srt.configs.model_config import ModelConfig
from sglang.srt.model_loader.loader import get_model_loader

model_config = ModelConfig(
    model_path="meta-llama/Llama-3.2-1B-Instruct",
    quantization="modelopt_fp8",
    trust_remote_code=True,
)

load_config = LoadConfig(
    modelopt_checkpoint_restore_path="./my_checkpoint.pth",
    modelopt_export_path="./exported_model",
)

# Load and export the model (DeviceConfig defaults to device="cuda")
model_loader = get_model_loader(load_config, model_config)
model_loader.load_model(model_config=model_config, device_config=DeviceConfig())
```

##### Benefits of ModelOpt

* **Hardware Optimization**: Specifically optimized for NVIDIA GPU architectures
* **Advanced Quantization**: Supports cutting-edge FP8 and FP4 quantization techniques
* **Seamless Integration**: Automatic export to HuggingFace format for easy deployment
* **Calibration-based**: Uses calibration datasets for optimal quantization quality
* **Production Ready**: Enterprise-grade quantization with NVIDIA support

#### Using [ModelSlim](https://gitcode.com/Ascend/msmodelslim)

MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.

* **Installation**

  ```bash Command theme={null}
  # Clone repo and install msmodelslim:
  git clone https://gitcode.com/Ascend/msmodelslim.git
  cd msmodelslim
  bash install.sh
  ```

* **LLM quantization**

  Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card).

  > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).

  *Traditional quantification methods require the preparation of calibration data files (`.jsonl` formats) for calibration in the quantification process.*

  ````bash Command theme={null}
  Qwen3-32B/      # floating-point model downloaded from official HF (or modelscope) repo
  msmodelslim/    # msmodelslim repo
    |----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones)
        |----- some file (such as laos_calib.jsonl)
    |----- lab_practice # best practice folder with configs for quantization
        |----- model folder (such as qwen3_5_moe folder) # folder with quantization configs
            |----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config
    |----- another folders
  output_folder/   # generated by below command
    |----- quant_model_weights-00001-of-0001.safetensors # quantized weights
    |----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.)
    |----- another files (such as config.json, tokenizer.json, etc.)
  ````

  Run quantization using one-click quantization (recommended):

  ```bash Command theme={null}
  msmodelslim quant \
  --model_path ${MODEL_PATH} \
  --save_path ${SAVE_PATH} \
  --device npu:0,1 \
  --model_type Qwen3-32B \
  --quant_type w8a8 \
  --trust_remote_code True
  ```

* **Usage Example**
  ```bash Command theme={null}
  python3 -m sglang.launch_server \
  --model-path $PWD/Qwen3-32B-w8a8 \
  --port 30000 --host 0.0.0.0
  ```

* **Available Quantization Methods**:
  * [x] &#x20;`W4A4_DYNAMIC` linear with online quantization of activations
  * [x] &#x20;`W8A8` linear with offline quantization of activations
  * [x] &#x20;`W8A8_DYNAMIC` linear with online quantization of activations
  * [x] &#x20;`W4A4_DYNAMIC` MOE with online quantization of activations
  * [x] &#x20;`W4A4_MXFP4` MOE with dynamic MXFP4 activation quantization
  * [x] &#x20;`W4A8_DYNAMIC` MOE with online quantization of activations
  * [x] &#x20;`W4A8_MXFP` MOE with dynamic MXFP8 activation quantization
  * [x] &#x20;`W8A8_DYNAMIC` MOE with online quantization of activations
  * [ ] &#x20;`W4A8` linear TBD
  * [ ] &#x20;`W4A16` linear TBD
  * [ ] &#x20;`W48A16` linear TBD
  * [ ] &#x20;`W4A16` MoE in progress
  * [ ] &#x20;`W8A16` MoE in progress
  * [ ] &#x20;`KV Cache` in progress
  * [ ] &#x20;`Attention` in progress

For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/README.md) section in ModelSLim repo.

## Online Quantization

To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --quantization fp8 \
    --port 30000 --host 0.0.0.0
```

Our team is working on supporting more online quantization methods. SGLang will soon support methods including but not limited to `["awq", "gptq", "marlin", "gptq_marlin", "awq_marlin", "bitsandbytes", "gguf"]`.

### `nvfp4_online` online quantization method

Use `--quantization nvfp4_online` to convert eligible BF16, FP16, or FP8 MoE expert weights to NVFP4 at load time with per-token FP32 activation scales. Use `modelopt_fp4` for serialized NVFP4 checkpoints or the same load-time conversion with per-tensor FP32 activation scales.

The design separates weight quantization from activation scaling:

* **Weights:** SGLang quantizes each eligible MoE expert weight tensor as it is loaded, using standard 2D NVFP4 weight quantization. The generated NVFP4 weights use static E4M3 block scales plus static per-tensor FP32 scales derived from the weight amax. For gated MoE experts, the w1/w3 pair shares one per-tensor FP32 scale.
* **Activations:** FlashInfer computes and propagates one FP32 scale per token at runtime. Backends that accept one per-tensor FP32 activation scale must use `modelopt_fp4`.
* **FP8 checkpoints:** If an eligible expert weight is stored as FP8, SGLang first dequantizes that tensor with the checkpoint scale and then requantizes it to NVFP4 during loading.
* **Other layers:** Dense linear layers stay in their source checkpoint precision or checkpoint quantization path.

Only `--moe-runner-backend flashinfer_trtllm` and `--moe-runner-backend flashinfer_trtllm_routed` are supported. If `--moe-runner-backend` is omitted, SGLang selects `flashinfer_trtllm`. Tensor parallelism is supported; activation per-token scales are computed locally on each TP rank, while online weight quantization still uses the loaded expert tensor's per-tensor amax-derived FP32 scale.

FlashInfer TRTLLM MoE backends disable shared-expert fusion, so online quantization applies to routed MoE experts while shared experts stay in the checkpoint precision. Both online modes honor `SGLANG_FP4_IGNORED_LAYERS`; for FP8 source checkpoints, listed experts remain FP8 instead of being converted to NVFP4.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
    --tp-size 2 \
    --ep-size 2 \
    --quantization nvfp4_online \
    --port 30000 --host 0.0.0.0
```

### `quark_int4fp8_moe` online quantization method

SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/v0.4.8/python/sglang/srt/layers/moe/fused_moe_triton/layer.py#L271) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.

Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly.

### `quark_mxfp4` online quantization method

SGLang running on AMD GPUs with hardware FP4 support (CDNA4 architecture, e.g. MI355x) supports the quantization method `--quantization quark_mxfp4`, that will quantize BF16 or NVFP4 model weights to MXFP4 at load time, use dynamic MXFP4 quantization for activations and MXFP4 GEMMs instead of BF16 GEMMs.

Example (BF16 to MXFP4 requantization):

```bash theme={null}
sglang serve --model-path Qwen/Qwen3-30B-A3B \
    --tensor-parallel-size 1 \
    --quantization quark_mxfp4
```

#### Online NVFP4 to MXFP4 Requantization

The option `--quantization quark_mxfp4` supports converting NVFP4 checkpoints (e.g. `nvidia/Kimi-K2.6-NVFP4`) to MXFP4 at load time to allow efficient inference using supported AMD hardware (gfx95x+):

* The quantization metadata of the source NVFP4 checkpoint is read from either `config.json` (`quantization_config`) or a standalone `hf_quant_config.json`;
* Producer-declared excluded modules will remain in higher precision;
* NVFP4 checkpoints with mixed precision (`"quant_algo": "MIXED_PRECISION"`, e.g. `nvidia/Qwen3.5-397B-A17B-NVFP4-V2`) are also supported.

Example (NVFP4 to MXFP4 requantization):

```bash theme={null}
sglang serve --model-path nvidia/Kimi-K2.6-NVFP4 \
    --tensor-parallel-size 4 \
    --quantization quark_mxfp4 \
```

#### Online FP8 to MXFP4 Requantization

The option `--quantization quark_mxfp4` supports converting FP8 dense and MOE models to MXFP4, following this logic:

1. Load an FP8 weight tensor,
2. Dequantize it to BF16,
3. Requantize it to MXFP4

progressively during weight loading.

Example (FP8 to MXFP4 requantization):

```bash theme={null}
sglang serve --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
    --tensor-parallel-size 1 \
    --quantization quark_mxfp4
```

### Intel® Neural Compressor online quantization method

SGLang supports quantization methods based on the advanced algorithm [auto-round](https://github.com/intel/auto-round) in [Intel® Neural Compressor](https://github.com/intel/neural-compressor). You can simply specify `--quantization auto-round-int8` to use this feature. It will quantize the model on the fly to target format. More online quantization methods are on the way.

##### Available Quantization Methods

| Quantization Method | Schemes | Validated Hardware Environment |
| :- | :- | :- |
| auto-round-int8 | INT8 per-channel quantized weight  INT8 per-token dynamic quantized activation | Intel Xeon Scalable processor  Nvidia A100 GPU |

## Diffusion Model Quantization on Ascend NPU

SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend 950PR/DT Series NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI.

**Requirements:** Ascend 950PR/DT Series, CANN ≥ 8.0.RC3

### Online MXFP8

Pass `--quantization mxfp8` to dynamically quantize FP16/BF16 transformer weights to MXFP8 at load time:

```bash theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --num-gpus 4
```

### Offline MXFP8 (ModelSlim)

Pre-quantize with [msModelSlim](https://gitcode.com/Ascend/msmodelslim) and load the checkpoint directly — the quantization scheme is auto-detected from `quant_model_description.json`:

```bash theme={null}
sglang generate \
  --model-path /path/to/wan2_2_mxfp8_diffusers \
  --prompt "a beautiful sunset" \
  --save-output
```

For the full quantization + format conversion workflow and a complete list of supported schemes, see [Diffusion Quantization on Ascend NPU](../hardware-platforms/ascend-npus/optimization/quantization#diffusion-model-quantization-on-ascend-npu) and [SGLang-Diffusion Quantization](../sglang-diffusion/quantization#modelslim).

## Reference

* [GPTQModel](https://github.com/ModelCloud/GPTQModel)
* [LLM Compressor](https://github.com/vllm-project/llm-compressor/)
* [NVIDIA Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer)
* [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq)
* [Petit: NVFP4 on ROCm](https://github.com/causalflow-ai/petit-kernel) — [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/), [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html)
* [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/)
* [auto-round](https://github.com/intel/auto-round)
* [ModelSlim](https://gitcode.com/Ascend/msmodelslim)


# Quantized KV Cache
Source: https://docs.sglang.io/docs/advanced_features/quantized_kv_cache



Quantized KV cache reduces the memory footprint of key-value cache storage by using lower-precision data types (FP8 or FP4) instead of the default model precision in BF16. During autoregressive generation, LLMs cache previously computed key-value pairs to avoid redundant calculations. The KV cache typically consumes a significant portion of GPU memory, especially for long sequences.

Quantized KV cache is a memory optimization technique that primarily benefits throughput by allowing more tokens to be cached, but may introduce minimal accuracy degradation depending on the quantization format used.

> [!WARNING]
>
> **Performance Warning**: When quantized KV cache must be dequantized before use in attention operations, performance can be extremely slow if dequantization is not fused with the attention kernel. Always verify that your chosen attention backend supports quantized KV cache. Backends without fused support may experience significant throughput degradation, potentially negating the memory benefits.
>
> **Backend Support**: Not all attention backends support quantized KV cache. Refer to [Attention Backend](./attention_backend) for which backends support it.

## Supported Formats

SGLang supports the following quantized KV cache formats:

### FP8 Format

[OCP (Open Compute Project)](https://www.opencompute.org) specifies two common 8-bit floating point formats:

* **E5M2** (5 exponent bits, 2 mantissa bits): Larger dynamic range (±57344.0), lower precision
* **E4M3** (4 exponent bits, 3 mantissa bits): Higher precision, smaller dynamic range (±240.0)

### FP4 Format

> [!WARNING]
>
> FP4 quantization is currently experimental.

[OCP (Open Compute Project)](https://www.opencompute.org) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format. SGLang exposes two experimental E2M1 KV-cache recipes:

* **`nvfp4`**: NVIDIA FP4 with 16-value blocks, E4M3 block scales, and a per-tensor global scale.
* **`fp4_mx_block16`**: An E2M1 block-size-16 compatibility recipe. It is distinct from the standard OCP MXFP4 block-size-32 format.

## Usage

### Enabling Quantized KV Cache

To enable quantized KV cache, use the `--kv-cache-dtype` argument when launching the server:

```bash Command theme={null}
# Enable FP8 E5M2 KV cache
python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-R1-0528 \
    --kv-cache-dtype fp8_e5m2 \

# Enable FP8 E4M3 KV cache
python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-R1-0528 \
    --kv-cache-dtype fp8_e4m3 \

# Enable NVFP4 FP4 E2M1 KV cache
python3 -m sglang.launch_server \
    --model-path nvidia/DeepSeek-R1-0528-NVFP4 \
    --kv-cache-dtype nvfp4 \

# Enable block-size-16 FP4 E2M1 KV cache
python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-R1-0528 \
    --kv-cache-dtype fp4_mx_block16 \
```

### SM100 native NVFP4 recipes

On SM100, prefill can either consume packed NVFP4 directly or dequantize it into an FP8 E4M3 workspace. Select the online dequantization dtype with `--prefill-kv-cache-dequant-dtype`; SGLang chooses the corresponding attention implementation.

For native NVFP4 prefill and decode:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3.5-35B-A3B-FP8 \
    --tp-size 1 \
    --kv-cache-dtype nvfp4 \
    --prefill-kv-cache-dequant-dtype nvfp4 \
    --page-size 16
```

For FP8 prefill backed by a temporary dequantization workspace, with native NVFP4 decode:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3.5-35B-A3B-FP8 \
    --tp-size 1 \
    --kv-cache-dtype nvfp4 \
    --prefill-kv-cache-dequant-dtype fp8_e4m3 \
    --page-size 16
```

`nvfp4` means that prefill consumes the packed native NVFP4 cache directly without any additional dequantization. The default value, `auto`, selects this native NVFP4 mode on SM100 and FP8 E4M3 dequantization on SM120. Decode consumes native NVFP4 in both recipes. The native recipe avoids the FP8 workspace and its token-linear scale copy; the FP8 recipe retains both the linear scales used during prefill and the physical scale layout used during decode. KV data remains stored as packed FP4 either way.

> [!NOTE]
>
> Native NVFP4 prefill requires SM100, a page size divisible by 4, and an attention head dimension divisible by 64. TRT-LLM GenMHA uses FP8 query and output buffers internally; SGLang converts the result back to the model activation dtype. Top-k-1 EAGLE/EAGLE3/NEXTN and breadth-1 NGRAM speculative decoding are supported: an EAGLE-family draft worker uses `trtllm_mha`, and target verification consumes the physical NVFP4 cache directly in eager execution and CUDA Graphs. Set `--speculative-ngram-max-bfs-breadth=1` for NGRAM. With the mixed FlashInfer-prefill/TRT-LLM-decode recipe, SGLang resolves `--speculative-attention-mode` to `decode` so verification does not depend on FlashInfer's transient dequantization workspace. Other speculative algorithms, PD disaggregation, hierarchical KV cache, and LMCache are not currently supported by the SM100 native NVFP4 path. SM120 XQA continues to use its architecture-specific linear scale layout and BF16 query/output path.

### Scaling Factors

FP8 quantization requires scaling factors to properly quantize and dequantize the KV cache.

> [!NOTE]
>
> Currently, only per-tensor (scalar) scaling factors are supported.

Scaling factors can be:

* **Loaded from checkpoints**: Pre-quantized models (e.g., ModelOpt) may include `k_scale` and `v_scale` parameters that are automatically loaded
* **Provided via JSON**: Supply scaling factors via `--quantization-param-path`.

The JSON file should follow this format:

```json Config theme={null}
{
  "kv_cache": {
    "dtype": "float8_e4m3fn",
    "scaling_factor": {
      "0": {
        "0": 1.0,
        "1": 1.0
      }
    }
  }
}
```

Where the outer keys in `scaling_factor` are tensor parallel ranks and inner keys are layer indices.

> [!WARNING]
>
> If scaling factors are not provided and not found in the checkpoint, it will default to 1.0, which may cause accuracy issues.

> [!TIP]
>
> **FP4 (MXFP4)**: Unlike FP8, FP4 quantization handles scaling factors automatically on-the-fly during quantization and dequantization. No pre-quantized models or external scaling factor files are required—the block-based scaling factors are computed dynamically as needed.

## Performance Considerations

### Memory Savings

Quantized KV cache provides significant memory savings:

* **BF16 → FP4**: Supports approximately 3.56× more tokens than BF16 (accounting for scaling factor overhead)

> [!NOTE]
>
> FP4 and FP8 quantization require additional memory for block-based scaling factors, which reduces the effective memory savings compared to the raw bit-width reduction. FP4 with block size 16 supports approximately 1.78× more tokens than FP8, and approximately 3.56× more tokens than BF16. The relative token capacity between FP8 and BF16 can be derived from these ratios.

This enables longer context lengths or more concurrent requests within the same memory budget.

For native `nvfp4`, each logical scalar costs 0.5 bytes of packed FP4 data plus 1/16 byte of block-scale storage, compared with 2 bytes for BF16. The resulting theoretical KV-token capacity multiplier is `2 / 0.5625 = 3.56×`. The mixed SM100 recipe owns a second scale layout and a shared one-layer FP8 prefill workspace, so its exact capacity depends on the number of full-attention layers. SGLang includes those auxiliary buffers in its KV-pool sizing calculation.

### Accuracy Impact

#### FP8 Accuracy

FP8 E4M3 quantization typically introduces minimal accuracy degradation. The impact depends on model architecture, sequence length, and quantization format (generally, E4M3 has better accuracy than E5M2).

#### FP4 Accuracy

FP4 (MXFP4) quantization provides significant memory savings with varying accuracy impact depending on model size and dataset complexity. Preliminary accuracy test results from [PR #10078](https://github.com/sgl-project/sglang/pull/10078) (MLA) and [PR #12612](https://github.com/sgl-project/sglang/pull/12612) (MHA) show:

**Large Models (e.g., Qwen3-235B-A22B, DeepSeek-R1-0528)**

On large-scale models, FP4 maintains accuracy close to FP8/BF16, especially on simpler datasets:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Dataset</th>
      <th>KV16</th>
      <th>KV8 (FP8 E4M3)</th>
      <th>KV4 (FP4 E2M1)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen3-235B-A22B</td>
      <td>gsm8k</td>
      <td>0.9168</td>
      <td>0.9181</td>
      <td>0.9186</td>
    </tr>

    <tr>
      <td>Qwen3-235B-A22B</td>
      <td>aime25</td>
      <td>0.7733</td>
      <td>0.7333</td>
      <td>0.6000</td>
    </tr>

    <tr>
      <td>Qwen3-235B-A22B</td>
      <td>gpqa\_diamond</td>
      <td>0.7010</td>
      <td>0.6899</td>
      <td>0.6778</td>
    </tr>

    <tr>
      <td>DeepSeek-R1-0528</td>
      <td>gsm8k</td>
      <td>0.9157</td>
      <td>0.9154</td>
      <td>0.9124</td>
    </tr>

    <tr>
      <td>DeepSeek-R1-0528</td>
      <td>aime25</td>
      <td>0.5067</td>
      <td>0.4934</td>
      <td>0.4000</td>
    </tr>

    <tr>
      <td>DeepSeek-R1-0528</td>
      <td>gpqa\_diamond</td>
      <td>0.7707</td>
      <td>0.7697</td>
      <td>0.7273</td>
    </tr>
  </tbody>
</table>

**Smaller Models (e.g., GPT-OSS-120B)**

On smaller models, FP4 shows more pronounced accuracy drops, particularly on challenging datasets:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Dataset</th>
      <th>KV16</th>
      <th>KV8 (FP8 E4M3)</th>
      <th>KV4 (FP4 E2M1)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>GPT-OSS-120B</td>
      <td>gsm8k</td>
      <td>0.9161</td>
      <td>0.9163</td>
      <td>0.9152</td>
    </tr>

    <tr>
      <td>GPT-OSS-120B</td>
      <td>aime25</td>
      <td>0.7533</td>
      <td>0.7667</td>
      <td>0.3533</td>
    </tr>

    <tr>
      <td>GPT-OSS-120B</td>
      <td>gpqa\_diamond</td>
      <td>0.5081</td>
      <td>0.5434</td>
      <td>0.3202</td>
    </tr>
  </tbody>
</table>

**Key Observations:**

* **Simple datasets (e.g., gsm8k)**: FP4 maintains accuracy close to FP8/BF16 across model sizes
* **Model size matters**: Large models (200B+ parameters) generally tolerate FP4 quantization better than smaller models
* **Context length**: Accuracy degradation may be more pronounced in long-context scenarios, as the accumulation of the quantization error may become significant.

> [!TIP]
>
> Evaluate FP4 accuracy on your specific model and workload. Large models on simpler tasks typically show minimal degradation, while smaller models or complex reasoning tasks may require FP8 or BF16 for acceptable accuracy.

## Best Practices

* **Use pre-quantized models**: Prefer models quantized offline with scaling factors included in the checkpoint.
* **Choose the right format**: Use `fp8_e4m3` for better accuracy (recommended), `fp8_e5m2` for larger dynamic range, or `nvfp4` / `fp4_mx_block16` for maximum memory savings (experimental)
* **Check backend compatibility**: Verify that your chosen attention backend supports quantized KV cache

> [!NOTE]
>
> See also:
>
> * [Quantization](./quantization)
> * [Attention Backend](./attention_backend)
> * [Server Arguments](./server_arguments)


# Radix Cache Eviction Policies
Source: https://docs.sglang.io/docs/advanced_features/radix_eviction_policy



When the KV cache pool is full, the radix cache reclaims space by evicting cached prefixes. The eviction policy decides which prefix goes first. The default, `lru`, is the right choice for most workloads; the others trade recency for hit frequency, reuse history, or request priority.

## How eviction picks a victim

Eviction only ever considers **evictable leaves**: nodes whose KV is present, unlocked (no in-flight request holds them), and not shadowed by a child that still holds KV. The root is never evictable.

The policy scores each candidate and the **lowest score is evicted first**. Once a leaf is evicted, its parent may become a leaf and re-enter the candidate set, so eviction walks a branch from its tip toward the root.

A policy only scores. It does not decide how much to free, and it cannot pin KV in memory — a node protected by the policy is still evicted if reclaiming everything else is not enough.

## Available policies

Select one with `--radix-eviction-policy`. All of them fall back to least-recently-used order within a tie.

| Policy | Evicts first | Use when |
| - | - | - |
| `lru` (default) | The prefix unused for longest. | General serving. Matches how prefix reuse decays with time. |
| `lfu` | The prefix with the fewest cache hits, then the least recent. | A small set of prompts is reused far more than the rest, and you want those to survive bursts of one-off traffic. |
| `slru` | Prefixes still in the probationary segment, then the least recent within a segment. | Like `lfu`, but you want a hard floor on how much a one-off prefix can displace a proven one. See [`slru` parameters](#slru-parameters). |
| `priority` | The prefix belonging to the lowest-priority request, then the least recent. | You run [priority scheduling](/docs/advanced_features/server_arguments) and want cache retention to follow the same ranking as admission. |
| `tlru` | The tail of a conversation beyond what its next prefill needs to meet the TTFT budget ("TEL-safe" tokens), then the least recent. | Agentic / multi-turn workloads where tail TTFT matters more than raw hit rate. See [`tlru` parameters](#tlru-parameters). Requires the unified radix cache (the default tree). |

Notes on the scoring inputs:

* **Hit count** (`lfu`, `slru`) counts how many times a node was matched by a later request. It is not incremented for chunked-prefill steps, for evicted nodes, or under write-back HiCache.
* **Request priority** (`priority`) is the `priority` field of the request that inserted the prefix; a node reached by several requests keeps the highest priority among them. Without priority scheduling every node is priority `0` and this policy is equivalent to `lru`.

`fifo`, `mru`, and `filo` also exist in the policy registry but are not offered on the command line. They are reachable only by out-of-tree code that extends the choice list, and exist for experiments rather than serving.

## Tuning a policy

Some policies take parameters. Pass them as a json object to `--radix-eviction-policy-config`; the keys are the policy's own, so they are only valid for the policy you selected.

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path MODEL_PATH \
  --radix-eviction-policy slru \
  --radix-eviction-policy-config '{"protected_threshold": 4}'
```

Omit the flag to accept every default. An unrecognized key fails at startup rather than being ignored:

```
TypeError: SLRUStrategy.__init__() got an unexpected keyword argument 'protected_treshold'
```

### Policy parameters

`slru` and `tlru` take parameters. `lru`, `lfu`, and `priority` take none, so `--radix-eviction-policy-config` has no effect with them and any key is an error.

#### `slru` parameters

`slru` splits the cache into a **probationary** segment and a **protected** segment. A prefix enters probationary, and is promoted to protected once it has been hit enough times. Everything probationary is evicted before anything protected.

| Key | Type | Default | Meaning |
| - | - | - | - |
| `protected_threshold` | int | `2` | Hit count at which a prefix is promoted to the protected segment. |

Raising it makes promotion harder, so the protected set stays small and closer to your genuinely hot prefixes; a prefix hit three times stays probationary at `4` but is protected at the default `2`. Lowering it to `1` promotes any prefix that is reused even once, which approaches `lru` with a one-hit grace period.

#### `tlru` parameters

`tlru` (Tail-Optimized LRU, [arXiv:2510.15152](https://arxiv.org/abs/2510.15152)) protects only the cached history a conversation's next prefill needs to stay under a TTFT budget; the rest of its tail is evicted first, and eviction continues in plain recency order once the TEL-safe tokens run out.

| Key | Type | Meaning |
| - | - | - |
| `threshold` | int | Tail-latency threshold ξ, in tokens: a conversation only keeps enough cache to hold its next prefill under ξ uncached tokens. Convert from a TTFT target by dividing it by the measured ms per uncached token. The paper states ξ in blocks, so multiply its values by `--page-size`. |
| `next_prompt_estimate` | int | Estimated tokens the next turn of a conversation will add; use the trace's empirical mean. |

Both keys are required in practice: `threshold` must be greater than `next_prompt_estimate` (only the difference affects behaviour), and values at or above it reduce T-LRU to plain LRU, which startup rejects.

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path MODEL_PATH \
  --radix-eviction-policy tlru \
  --radix-eviction-policy-config '{"threshold": 4096, "next_prompt_estimate": 512}'
```

## Which policy to pick

Start with `lru` and change it only against a measured cache hit rate — the counters are exposed under `--enable-metrics`. `lfu` and `slru` help when your traffic has a stable hot set that a recency-only policy keeps flushing; they hurt when prefix popularity shifts over time, because a prefix that earned a high hit count keeps its advantage after it stops being useful.


# Reasoning-Aware Compression
Source: https://docs.sglang.io/docs/advanced_features/reasoning_aware_compression



Pruning a reasoning model with a standard calibration set does more damage than pruning a
conventional LLM — and it can make the model *slower*. Reasoning-Aware Compression (RAC) fixes this
by changing what the pruning solver calibrates on: the model's own chain of thought, generated with
SGLang.

From [*Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction*](https://arxiv.org/abs/2509.12464)
(ICLR 2026).

## The problem

One-shot pruning methods such as SparseGPT and Wanda choose which weights to remove by minimizing a
layer-wise reconstruction error against a calibration activation matrix `X`:

```
min_{W'} || W X - W' X ||_F^2    s.t.  ||W'||_0 <= S
```

`X` is conventionally built from **prompt** tokens — a slice of C4, or a set of task prompts. That
is a fair proxy for a typical serving workload, where the prompt dominates the token count.

Reasoning models invert that ratio. They emit thousands of chain-of-thought tokens per query, so
almost every forward pass the pruned model will ever run is over a token it generated itself.
Calibrating only on prompts optimizes the pruned weights for a distribution the model barely visits.

The result is not a graceful accuracy decay. The pruned model starts to ramble: it produces longer
chains of thought *and* answers less accurately, so pruning increases end-to-end latency instead of
reducing it. At 50% sparsity on MATH-500, C4-calibrated DeepSeek-R1-Distill-Qwen-7B takes almost six
times as long to evaluate as the dense model it was meant to accelerate.

## The fix

RAC samples the dense model's own on-policy rollout during calibration and reconstructs the prompt
and decode activations jointly:

```
X_RAC = [ X_prompt , X_decode ]
```

The solver is untouched, so this is a drop-in change to any existing SparseGPT or Wanda workflow.

DeepSeek-R1-Distill-Qwen-7B, MATH-500, SparseGPT at 50% sparsity, 1M calibration tokens:

| Calibration set | acc\@1 | Eval wall clock |
| - | - | - |
| Dense (no pruning) | 0.936 | 23.3 min |
| C4 | 0.744 | 135.0 min |
| Task prompts only | 0.812 | 115.6 min |
| **RAC (prompts + on-policy CoT)** | **0.900** | **35.3 min** |

Across DeepSeek-R1-Distill-Qwen (1.5B–32B) and Qwen3 (1.7B–14B), the paper reports that RAC keeps up
to 95% of dense accuracy at 50% sparsity, improving on prompt-only calibration by up to 17 points.

## Using it

SGLang ships the recipe as a runnable example at
[`examples/usage/reasoning_aware_compression`](https://github.com/sgl-project/sglang/tree/main/examples/usage/reasoning_aware_compression),
in three phases:

| Phase | Script | What it does |
| - | - | - |
| I | `rac_collect_traces.py` | `sgl.Engine` samples on-policy CoT traces into a calibration set |
| II | `rac_prune.py` | `llm-compressor` runs SparseGPT/Wanda against those activations |
| III | `rac_serve_and_eval.py` | SGLang serves the sparse checkpoint and scores MATH-500 |

Phase I is the expensive step — the paper's budget is 1M on-policy CoT tokens — and is where SGLang's
batched generation does the work. Phase II delegates the pruning solver to
[`llm-compressor`](https://github.com/vllm-project/llm-compressor), which is **not** an SGLang
dependency; install it separately with `pip install "llmcompressor>=0.12.0"`.

```bash theme={null}
cd examples/usage/reasoning_aware_compression

python rac_collect_traces.py \
    --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --dataset open-r1/OpenR1-Math-220k --prompt-column problem \
    --target-tokens 1000000 --output-dir ./rac_traces_math

python rac_prune.py \
    --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --calibration ./rac_traces_math/traces.jsonl \
    --sparsity 0.5 --output-dir ./rac_pruned_50

python -m sglang.launch_server --model-path ./rac_pruned_50
```

The example README walks through building the paper's prompt-only baseline from the same prompts so
you can compare calibration strategies head to head.

## Evaluating a pruned reasoning model

Accuracy alone will hide the failure mode described above. Always report **mean completion length**
and **wall clock** alongside accuracy when comparing pruned reasoning checkpoints — a model that
scores two points lower while emitting three times the chain of thought is not a good trade.
`rac_serve_and_eval.py` reports all three.

## Related

* [Quantization](/docs/advanced_features/quantization) — the other axis of model compression, applied
  at serving time.


# Reasoning Parser
Source: https://docs.sglang.io/docs/advanced_features/separate_reasoning



SGLang supports parsing reasoning content out from "normal" content for reasoning models such as [DeepSeek R1](https://huggingface.co/deepseek-ai/DeepSeek-R1).

## Supported Models & Parsers

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Reasoning tags</th>
      <th>Parser</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[Apertus 2509 models](https://huggingface.co/swiss-ai/Apertus-8B-Instruct-2509)</td>
      <td>`<|inner_prefix|>` … `<|inner_suffix|>`</td>
      <td>`apertus2509`</td>
      <td>Supports `enable_thinking` parameter</td>
    </tr>

    <tr>
      <td>[DeepSeek‑R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d)</td>
      <td>`<think>` … `</think>`</td>
      <td>`deepseek-r1`</td>
      <td>Supports all variants (R1, R1-0528, R1-Distill)</td>
    </tr>

    <tr>
      <td>[DeepSeek‑V3 series](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)</td>
      <td>`<think>` … `</think>`</td>
      <td>`deepseek-v3`</td>
      <td>Including [DeepSeek‑V3.2](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp). Supports `thinking` parameter</td>
    </tr>

    <tr>
      <td>[Standard Qwen3 models](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f)</td>
      <td>`<think>` … `</think>`</td>
      <td>`qwen3`</td>
      <td>Supports `enable_thinking` parameter</td>
    </tr>

    <tr>
      <td>[Qwen3-Thinking models](https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507)</td>
      <td>`<think>` … `</think>`</td>
      <td>`qwen3` or `qwen3-thinking`</td>
      <td>Always generates thinking content</td>
    </tr>

    <tr>
      <td>[Kimi K2 Thinking](https://huggingface.co/moonshotai/Kimi-K2-Thinking)</td>
      <td>`◁think▷` … `◁/think▷`</td>
      <td>`kimi_k2`</td>
      <td>Uses special thinking delimiters. Also requires `--tool-call-parser kimi_k2` for tool use.</td>
    </tr>

    <tr>
      <td>[GPT OSS](https://huggingface.co/openai/gpt-oss-120b)</td>
      <td>`&lt;|channel|&gt;analysis&lt;|message|&gt;` … `&lt;|end|&gt;`</td>
      <td>`gpt-oss`</td>
      <td>N/A</td>
    </tr>
  </tbody>
</table>

### Model-Specific Behaviors

**Apertus 2509:**

* Uses `<|inner_prefix|>` and `<|inner_suffix|>` to delimit reasoning content. For agentic tool use, also specify `--tool-call-parser apertus2509`.

**DeepSeek-R1 Family:**

* DeepSeek-R1: No `<think>` start tag, jumps directly to thinking content
* DeepSeek-R1-0528: Generates both `<think>` start and `</think>` end tags
* Both are handled by the same `deepseek-r1` parser

**DeepSeek-V3 Family:**

* DeepSeek-V3.1/V3.2: Hybrid model supporting both thinking and non-thinking modes, use the `deepseek-v3` parser and `thinking` parameter (NOTE: not `enable_thinking`)

**Qwen3 Family:**

* Standard Qwen3 (e.g., Qwen3-2507): Use `qwen3` parser, supports `enable_thinking` in chat templates
* Qwen3-Thinking (e.g., Qwen3-235B-A22B-Thinking-2507): Use `qwen3` or `qwen3-thinking` parser, always thinks

**Kimi K2:**

* Kimi K2 Thinking: Uses special `◁think▷` and `◁/think▷` tags. For agentic tool use, also specify `--tool-call-parser kimi_k2`.

**GPT OSS:**

* GPT OSS: Uses special `<|channel|>analysis<|message|>` and `<|end|>` tags

## Usage

### Launching the Server

Specify the `--reasoning-parser` option.

```python Example theme={null}
import requests
from openai import OpenAI
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning"
)

wait_for_server(f"http://localhost:{port}")
```

Note that `--reasoning-parser` defines the parser used to interpret responses.

To cap how many tokens the model spends thinking, see [Thinking Budget](/docs/advanced_features/thinking_budget).

### OpenAI Compatible API

Using the OpenAI compatible API, the contract follows the [DeepSeek API design](https://api-docs.deepseek.com/guides/reasoning_model) established with the release of DeepSeek-R1:

* `reasoning_content`: The content of the CoT.
* `content`: The content of the final answer.

```python Example theme={null}
# Initialize OpenAI-like client
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1")
model_name = client.models.list().data[0].id

messages = [
    {
        "role": "user",
        "content": "What is 1+3?",
    }
]
```

#### Non-Streaming Request

```python Example theme={null}
response_non_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.6,
    top_p=0.95,
    stream=False,  # Non-streaming
    extra_body={"separate_reasoning": True},
)
print_highlight("==== Reasoning ====")
print_highlight(response_non_stream.choices[0].message.reasoning_content)

print_highlight("==== Text ====")
print_highlight(response_non_stream.choices[0].message.content)
```

#### Streaming Request

```python Example theme={null}
response_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.6,
    top_p=0.95,
    stream=True,  # Non-streaming
    extra_body={"separate_reasoning": True},
)

reasoning_content = ""
content = ""
for chunk in response_stream:
    if chunk.choices[0].delta.content:
        content += chunk.choices[0].delta.content
    if chunk.choices[0].delta.reasoning_content:
        reasoning_content += chunk.choices[0].delta.reasoning_content

print_highlight("==== Reasoning ====")
print_highlight(reasoning_content)

print_highlight("==== Text ====")
print_highlight(content)
```

Optionally, you can buffer the reasoning content to the last reasoning chunk (or the first chunk after the reasoning content).

```python Example theme={null}
response_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.6,
    top_p=0.95,
    stream=True,  # Non-streaming
    extra_body={"separate_reasoning": True, "stream_reasoning": False},
)

reasoning_content = ""
content = ""
for chunk in response_stream:
    if chunk.choices[0].delta.content:
        content += chunk.choices[0].delta.content
    if chunk.choices[0].delta.reasoning_content:
        reasoning_content += chunk.choices[0].delta.reasoning_content

print_highlight("==== Reasoning ====")
print_highlight(reasoning_content)

print_highlight("==== Text ====")
print_highlight(content)
```

The reasoning separation is enable by default when specify .
**To disable it, set the `separate_reasoning` option to `False` in request.**

```python Example theme={null}
response_non_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0.6,
    top_p=0.95,
    stream=False,  # Non-streaming
    extra_body={"separate_reasoning": False},
)

print_highlight("==== Original Output ====")
print_highlight(response_non_stream.choices[0].message.content)
```

### SGLang Native API

```python Example theme={null}
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
input = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)

gen_url = f"http://localhost:{port}/generate"
gen_data = {
    "text": input,
    "sampling_params": {
        "skip_special_tokens": False,
        "max_new_tokens": 1024,
        "temperature": 0.6,
        "top_p": 0.95,
    },
}
gen_response = requests.post(gen_url, json=gen_data).json()["text"]

print_highlight("==== Original Output ====")
print_highlight(gen_response)

parse_url = f"http://localhost:{port}/separate_reasoning"
separate_reasoning_data = {
    "text": gen_response,
    "reasoning_parser": "deepseek-r1",
}
separate_reasoning_response_json = requests.post(
    parse_url, json=separate_reasoning_data
).json()
print_highlight("==== Reasoning ====")
print_highlight(separate_reasoning_response_json["reasoning_text"])
print_highlight("==== Text ====")
print_highlight(separate_reasoning_response_json["text"])
```

```python Example theme={null}
terminate_process(server_process)
```

### Offline Engine API

```python Example theme={null}
import sglang as sgl
from sglang.srt.parser.reasoning_parser import ReasoningParser
from sglang.utils import print_highlight

llm = sgl.Engine(model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
input = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
sampling_params = {
    "max_new_tokens": 1024,
    "skip_special_tokens": False,
    "temperature": 0.6,
    "top_p": 0.95,
}
result = llm.generate(prompt=input, sampling_params=sampling_params)

generated_text = result["text"]  # Assume there is only one prompt

print_highlight("==== Original Output ====")
print_highlight(generated_text)

parser = ReasoningParser("deepseek-r1")
reasoning_text, text = parser.parse_non_stream(generated_text)
print_highlight("==== Reasoning ====")
print_highlight(reasoning_text)
print_highlight("==== Text ====")
print_highlight(text)
```

```python Example theme={null}
llm.shutdown()
```

## Supporting New Reasoning Model Schemas

For future reasoning models, you can implement the reasoning parser as a subclass of `BaseReasoningFormatDetector` in `python/sglang/srt/reasoning_parser.py` and specify the reasoning parser for new reasoning model schemas accordingly.


# Server Arguments
Source: https://docs.sglang.io/docs/advanced_features/server_arguments



This page provides a list of server arguments used in the command line to configure the behavior
and performance of the language model server during deployment. These arguments enable users to
customize key aspects of the server, including model selection, parallelism policies,
memory management, and optimization techniques.
You can find all arguments by `python3 -m sglang.launch_server --help`

## Common launch commands

* To use a configuration file, create a YAML file with your server arguments and specify it with `--config`. CLI arguments will override config file values.

  ```bash Command theme={null}
  # Create config.yaml
  cat > config.yaml << EOF
  model-path: meta-llama/Meta-Llama-3-8B-Instruct
  host: 0.0.0.0
  port: 30000
  tensor-parallel-size: 2
  enable-metrics: true
  log-requests: true
  EOF

  # Launch server with config file
  python -m sglang.launch_server --config config.yaml
  ```

* To enable multi-GPU tensor parallelism, add `--tp 2`. If it reports the error "peer access is not supported between these two devices", add `--enable-p2p-check` to the server launch command.

  ```bash Command theme={null}
  python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 2
  ```

* To enable multi-GPU data parallelism, add `--dp 2`. Data parallelism is better for throughput if there is enough memory. It can also be used together with tensor parallelism. The following command uses 4 GPUs in total. We recommend [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway) for data parallelism.

  ```bash Command theme={null}
  python -m sglang_router.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --dp 2 --tp 2
  ```

* If you see out-of-memory errors during serving, try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`.

  ```bash Command theme={null}
  python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --mem-fraction-static 0.7
  ```

* See [hyperparameter tuning](./hyperparameter_tuning) on tuning hyperparameters for better performance.

* For docker and Kubernetes runs, you need to set up shared memory which is used for communication between processes. See `--shm-size` for docker and `/dev/shm` size update for Kubernetes manifests.

* If you see out-of-memory errors during prefill for long prompts, try to set a smaller chunked prefill size.

  ```bash Command theme={null}
  python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --chunked-prefill-size 4096
  ```

* To enable fp8 weight quantization, add `--quantization fp8` on a fp16 checkpoint or directly load a fp8 checkpoint without specifying any arguments.

* To enable fp8 kv cache quantization, add `--kv-cache-dtype fp8_e4m3` or `--kv-cache-dtype fp8_e5m2`.

* To enable deterministic inference and batch invariant operations, add `--enable-deterministic-inference`. More details can be found in [deterministic inference document](./deterministic_inference).

* If a multimodal server accepts requests from untrusted clients, restrict remote image, video, and audio URLs with `--allowed-media-domains`. SGLang checks the initial URL and every redirect destination against the exact-hostname allowlist. Remote media downloads are limited to 64 MiB by default; adjust `--media-url-max-file-size-mb` when larger trusted media is required.

  ```bash Command theme={null}
  python -m sglang.launch_server \
    --model-path Qwen/Qwen2.5-VL-7B-Instruct \
    --allowed-media-domains upload.wikimedia.org raw.githubusercontent.com
  ```

  Without `--allowed-media-domains`, HTTP(S) media from any domain remains allowed for backward compatibility. Do not expose that configuration to untrusted users. Local paths and `data:` URLs are not governed by the domain allowlist.

* To enable decode context parallelism for MLA models, add `--dcp-size N`. See [Decode Context Parallelism](./dcp).

* If the model does not have a chat template in the Hugging Face tokenizer, you can specify a [custom chat template](../references/custom_chat_template). If the tokenizer has multiple named templates (e.g., 'default', 'tool\_use'), you can select one using `--hf-chat-template-name tool_use`.

* To run tensor parallelism on multiple nodes, add `--nnodes 2`. If you have two nodes with two GPUs on each node and want to run TP=4, let `sgl-dev-0` be the hostname of the first node and `50000` be an available port, you can use the following commands. If you meet deadlock, please try to add `--disable-cuda-graph`

* (Note: This feature is out of maintenance and might cause error) To enable `torch.compile` acceleration, add `--enable-torch-compile`. It accelerates small models on small batch sizes. By default, the cache path is located at `/tmp/torchinductor_root`, you can customize it using environment variable `TORCHINDUCTOR_CACHE_DIR`. For more details, please refer to [PyTorch official documentation](https://pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html) and [Enabling cache for torch.compile](../references/torch_compile_cache).
  ```bash Command theme={null}
  # Node 0
  python -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
    --tp 4 \
    --dist-init-addr sgl-dev-0:50000 \
    --nnodes 2 \
    --node-rank 0

  # Node 1
  python -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3-8B-Instruct \
    --tp 4 \
    --dist-init-addr sgl-dev-0:50000 \
    --nnodes 2 \
    --node-rank 1
  ```

Please consult the documentation below and [server\_args.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/server_args.py) to learn more about the arguments you may provide when launching a server.

## Model and tokenizer

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--model-path`\
      `--model`</td>
      <td>The path of the model weights. This can be a local folder or a Hugging Face repo ID.</td>
      <td>` None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--tokenizer-path`</td>
      <td>The path of the tokenizer.</td>
      <td>` None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--tokenizer-mode`</td>
      <td>Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer.</td>
      <td>`auto`</td>
      <td>`auto`, `slow`</td>
    </tr>

    <tr>
      <td>`--tokenizer-backend`</td>
      <td>Tokenizer backend. 'huggingface' uses the default HuggingFace tokenizers library; 'fastokens' uses the [fastokens](https://github.com/crusoecloud/fastokens) library for faster tokenization. Requires the `fastokens` package to be installed.</td>
      <td>`huggingface`</td>
      <td>`huggingface`, `fastokens`</td>
    </tr>

    <tr>
      <td>`--tokenizer-worker-num`</td>
      <td>The worker num of the tokenizer manager.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--skip-tokenizer-init`</td>
      <td>If set, skip init tokenizer and pass input\_ids in generate request.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--load-format`</td>
      <td>The format of the model weights to load. "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. "pt" will load the weights in the pytorch bin format. "safetensors" will load the weights in the safetensors format. "instanttensor" will load Safetensors weights using InstantTensor's high-performance distributed loader. "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. "dummy" will initialize the weights with random values, which is mainly for profiling."gguf" will load the weights in the gguf format. "bitsandbytes" will load the weights using bitsandbytes quantization."layered" loads weights layer by layer so that one can quantize a layer before loading another to make the peak memory envelope smaller. "flash\_rl" will load the weights in flash\_rl format. "fastsafetensors" and "private" are also supported. "runai\_streamer" enables direct model loading from object storage and shared file systems.</td>
      <td>`auto`</td>
      <td>`auto`, `pt`, `safetensors`, `instanttensor`, `npcache`, `dummy`, `sharded_state`, `gguf`, `bitsandbytes`, `mistral`, `layered`, `flash_rl`, `remote`, `remote_instance`, `fastsafetensors`, `private`, `runai_streamer`</td>
    </tr>

    <tr>
      <td>`--model-loader-extra-config`</td>
      <td>Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format.</td>
      <td>`{}`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>Whether or not to allow for custom models defined on the Hub in their own modeling files.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--context-length`</td>
      <td>The model's maximum context length. Defaults to None (will use the value from the model's config.json instead).</td>
      <td>` None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--is-embedding`</td>
      <td>Whether to use a CausalLM as an embedding model.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-multimodal`</td>
      <td>Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen.</td>
      <td>` None`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--revision`</td>
      <td>The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.</td>
      <td>` None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--model-impl`</td>
      <td>Which implementation of the model to use. "auto" will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. "sglang" will use the SGLang model implementation. "transformers" will use the Transformers model implementation.</td>
      <td>`auto`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--detokenizer-worker-num`</td>
      <td>The worker num of the detokenizer manager.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--prefill-only-disable-kv-cache`</td>
      <td>Skip the physical KV cache allocation for embedding-mode prefill-only workloads. Currently only valid with --is-embedding, --chunked-prefill-size=-1, --disable-radix-cache, an FA prefill backend, and non-FP4 KV cache so the fa\_skip\_kv\_cache path is active (no layer reads or writes the cache). Other prefill-only workloads such as scoring/MIS may benefit from this later once their attention paths stop using paged KV. Scheduler admission accounting is unchanged; per-layer K/V tensors are sized to (page\_size, head\_num, head\_dim) placeholders so GPU memory is not wasted.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--model-config-parser`</td>
      <td>Which model-config parser to use. "auto" picks "mistral" via the is\_mistral\_model name heuristic, else "hf" (AutoConfig over config.json). Plugins can register additional parsers via @register\_model\_config\_parser.</td>
      <td>`auto`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## HTTP server

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--host`</td>
      <td>The host of the HTTP server.</td>
      <td>`127.0.0.1`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--port`</td>
      <td>The port of the HTTP server.</td>
      <td>`30000`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--fastapi-root-path`</td>
      <td>App is behind a path based routing proxy.</td>
      <td>`""`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--grpc-mode`</td>
      <td>If set, use gRPC server instead of HTTP server.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--skip-server-warmup`</td>
      <td>If set, skip warmup.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--warmups`</td>
      <td>Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup\_name1,warmup\_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--nccl-port`</td>
      <td>The port for NCCL distributed environment setup. Defaults to a random port.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--checkpoint-engine-wait-weights-before-ready`</td>
      <td>If set, the server will wait for initial weights to be loaded via checkpoint-engine or other update methods before serving inference requests.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--ssl-keyfile`</td>
      <td>The file path to the SSL key file.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--ssl-certfile`</td>
      <td>The file path to the SSL certificate file.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--ssl-ca-certs`</td>
      <td>The CA certificates file.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--ssl-keyfile-password`</td>
      <td>The password to decrypt the SSL keyfile.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-ssl-refresh`</td>
      <td>Enable automatic SSL certificate hot-reloading when cert/key files change on disk. Requires --ssl-certfile and --ssl-keyfile.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-http-worker-reuseport`</td>
      <td>Give each Uvicorn HTTP worker its own SO\_REUSEPORT listener.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-http2`</td>
      <td>Use Granian instead of Uvicorn as the ASGI server, enabling HTTP/1.1 and HTTP/2 auto-negotiation. Clients may use h2c (cleartext HTTP/2) or plain HTTP/1.1. Requires 'pip install sglang\[http2]'.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--http2-max-concurrent-streams`</td>
      <td>Maximum number of concurrent streams advertised on each HTTP/2 connection (1 to 2^32 - 1). Only applies with --enable-http2.</td>
      <td>`200`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Quantization and data type

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dtype`</td>
      <td>Data type for model weights and activations. \* "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. \* "half" for FP16. Recommended for AWQ quantization. \* "float16" is the same as "half". \* "bfloat16" for a balance between precision and range. \* "float" is shorthand for FP32 precision. \* "float32" for FP32 precision.</td>
      <td>`auto`</td>
      <td>`auto`, `half`, `float16`, `bfloat16`, `float`, `float32`</td>
    </tr>

    <tr>
      <td>`--quantization`</td>
      <td>The quantization method.</td>
      <td>`None`</td>
      <td>`awq`, `fp8`, `mxfp8`, `gptq`, `marlin`, `gptq_marlin`, `awq_marlin`, `bitsandbytes`, `gguf`, `modelopt`, `modelopt_fp8`, `modelopt_fp4`, `nvfp4_online`, `modelopt_mixed`, `petit_nvfp4`, `w8a8_int8`, `w8a8_fp8`, `moe_wna16`, `w4afp8`, `mxfp4`, `auto-round`, `compressed-tensors`, `modelslim`, `quark`, `quark_int4fp8_moe`, `quark_mxfp4`, `mlx_q4`, `mlx_q8`, `unquant`</td>
    </tr>

    <tr>
      <td>`--quantization-param-path`</td>
      <td>Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues.</td>
      <td>`None`</td>
      <td>Type: Optional\[str]</td>
    </tr>

    <tr>
      <td>`--kv-cache-dtype`</td>
      <td>Data type for kv cache storage. "auto" will use model data type. "bf16" or "bfloat16" for BF16 KV cache. "fp8\_e5m2" and "fp8\_e4m3" are supported for CUDA 11.8+. "nvfp4" selects the NVFP4 FP4 E2M1 KV cache recipe; "fp4\_mx\_block16" selects the block-size-16 FP4 E2M1 KV cache recipe. Both require CUDA 12.8+ and PyTorch 2.8.0+</td>
      <td>`auto`</td>
      <td>`auto`, `fp8_e5m2`, `fp8_e4m3`, `bf16`, `bfloat16`, `nvfp4`, `fp4_mx_block16`</td>
    </tr>

    <tr>
      <td>`--enable-fp32-lm-head`</td>
      <td>If set, the LM head outputs (logits) are in FP32.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--modelopt-quant`</td>
      <td>The ModelOpt quantization configuration. Supported values: 'fp8', 'int4\_awq', 'w4a8\_awq', 'nvfp4', 'nvfp4\_awq'. This requires the NVIDIA Model Optimizer library to be installed: pip install nvidia-modelopt</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--modelopt-checkpoint-restore-path`</td>
      <td>Path to restore a previously saved ModelOpt quantized checkpoint. If provided, the quantization process will be skipped and the model will be loaded from this checkpoint.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--modelopt-checkpoint-save-path`</td>
      <td>Path to save the ModelOpt quantized checkpoint after quantization. This allows reusing the quantized model in future runs.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--modelopt-export-path`</td>
      <td>Path to export the quantized model in HuggingFace format after ModelOpt quantization. The exported model can then be used directly with SGLang for inference. If not provided, the model will not be exported.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--quantize-and-serve`</td>
      <td>Quantize the model with ModelOpt and immediately serve it without exporting. This is useful for development and prototyping. For production, it's recommended to use separate quantization and deployment steps.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--rl-quant-profile`</td>
      <td>Path to the FlashRL quantization profile. Required when using --load-format flash\_rl.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-quant-communications`</td>
      <td>Enable INT8 quantization of TP communications (Supported only for NPU for Qwen3 series).</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-tf32-matmul`</td>
      <td>Enable float32 matmuls to use TensorFloat32 precision for better performance (via torch.set\_float32\_matmul\_precision). CUDA only. Automatically enabled for MiniMax-M2 and GLM-4 models.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## Memory and scheduling

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--mem-fraction-static`</td>
      <td>The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors. When unset, it is computed as `(GPU memory - reserved memory) / GPU memory`, defaulting to `0.88` if GPU memory cannot be detected.</td>
      <td>`None`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--max-running-requests`</td>
      <td>The maximum number of running requests.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--max-queued-requests`</td>
      <td>The maximum number of queued requests. This option is ignored when using disaggregation-mode.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--max-total-tokens`</td>
      <td>The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction. This option is typically used for development and debugging purposes.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--chunked-prefill-size`</td>
      <td>The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--prefill-decode-interval`</td>
      <td>The number of decode rounds to run after a prefill batch before scheduling the next prefill. By default, this is disabled except for profiled Qwen3-VL serving configurations on Hopper. In data-parallel attention mode, the interval is synchronized across all DP ranks. Set to 0 to disable.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--prefill-decode-interval-min-bs`</td>
      <td>Apply `--prefill-decode-interval` only while the running batch holds at least this many requests. Below it, prefill is admitted every round so the batch fills quickly. In data-parallel attention mode, each rank compares its own running batch. Set to 0 to always apply the interval.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--prefill-max-requests`</td>
      <td>The maximum number of requests in a prefill batch. If not specified, there is no limit.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--enable-dynamic-chunking`</td>
      <td>Enable dynamic chunk size adjustment for pipeline parallelism. When enabled, chunk sizes are dynamically calculated based on fitted function to maintain consistent execution time across chunks.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--max-prefill-tokens`</td>
      <td>The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model's maximum context length.</td>
      <td>`16384`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--schedule-policy`</td>
      <td>The scheduling policy of the requests.</td>
      <td>`fcfs`</td>
      <td>`lpm`, `random`, `fcfs`, `dfs-weight`, `lof`, `priority`, `routing-key`</td>
    </tr>

    <tr>
      <td>`--enable-priority-scheduling`</td>
      <td>Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--abort-on-priority-when-disabled`</td>
      <td>If set, abort requests that specify a priority when priority scheduling is disabled.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--schedule-low-priority-values-first`</td>
      <td>If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--priority-scheduling-preemption-threshold`</td>
      <td>Minimum difference in priorities for an incoming request to have to preempt running request(s).</td>
      <td>`10`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--schedule-conservativeness`</td>
      <td>How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently.</td>
      <td>`1.0`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--retraction-policy`</td>
      <td>The decode retraction policy to use when the KV cache is full. 'length' preserves the existing behavior and retracts short-output, long-input requests first. 'priority' retracts lower-priority requests first, using the same priority direction as priority scheduling.</td>
      <td>`length`</td>
      <td>`length`, `priority`</td>
    </tr>

    <tr>
      <td>`--page-size`</td>
      <td>The number of tokens in a page.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--enable-page-major-kv-layout`</td>
      <td>Store the Mamba state and full/SWA KV caches as one entry per token slot, holding every layer's state, in a single page-granular buffer instead of one buffer per layer. Implied by `--enable-unified-memory`, and currently supported only with it.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-unified-memory`</td>
      <td>For hybrid Mamba/GDN and hybrid SWA models, replace the statically-partitioned pools (full-attention KV + SWA/Mamba conv/SSM state) with a single byte buffer split dynamically between the sub-pools, so KV-vs-state capacity flexes with the workload instead of being fixed at startup. Implies `--enable-page-major-kv-layout`. Requires stride-aware attention, linear-attention and Mamba backends; the allowed backends and the supported PD-disaggregation and speculative-decoding configurations are validated at startup.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--swa-full-tokens-ratio`</td>
      <td>The ratio of SWA layer KV tokens / full layer KV tokens, regardless of the number of swa:full layers. It should be between 0 and 1. E.g. 0.5 means if each swa layer has 50 tokens, then each full layer has 100 tokens.</td>
      <td>`0.8`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--disable-hybrid-swa-memory`</td>
      <td>Disable the hybrid SWA memory.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--radix-eviction-policy`</td>
      <td>The eviction policy of radix trees. 'lru' stands for Least Recently Used, 'lfu' stands for Least Frequently Used.</td>
      <td>`lru`</td>
      <td>`lru`, `lfu`, `slru`, `priority`</td>
    </tr>

    <tr>
      <td>`--enable-prefill-delayer`</td>
      <td>Enable prefill delayer for DP attention to reduce idle time.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-max-delay-passes`</td>
      <td>Maximum forward passes to delay prefill.</td>
      <td>`30`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-token-usage-low-watermark`</td>
      <td>Token usage low watermark for prefill delayer.</td>
      <td>`None`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-queue-min-ratio`</td>
      <td>Opt-in to the adaptive queue-based delay trigger (independent of the slot-based one). Defers prefill until the waiting queue reaches `min(running_req * ratio, max_prefill_bs)` so small fragments batch into a larger prefill. Unset keeps the original slot-only behavior. Typical: `0.1`–`0.5`.</td>
      <td>`None`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-max-delay-ms`</td>
      <td>Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, prefill is force-released to bound worst-case TTFT. Only consulted when `--prefill-delayer-queue-min-ratio` is set. Typical: `1000`–`5000`.</td>
      <td>`None`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-forward-passes-buckets`</td>
      <td>Custom buckets for prefill delayer forward passes histogram. 0 and max\_delay\_passes-1 will be auto-added.</td>
      <td>`None`</td>
      <td>List\[float]</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-wait-seconds-buckets`</td>
      <td>Custom buckets for prefill delayer wait seconds histogram. 0 will be auto-added.</td>
      <td>`None`</td>
      <td>List\[float]</td>
    </tr>

    <tr>
      <td>`--disable-priority-preemption`</td>
      <td>Disable priority scheduling preemption.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--default-priority-value`</td>
      <td>Default priority for requests without explicit priority.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Runtime options

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--device`</td>
      <td>The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu'). Defaults to auto-detection if not specified.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--tensor-parallel-size`\
      `--tp-size`</td>
      <td>The tensor parallelism size.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--pipeline-parallel-size`\
      `--pp-size`</td>
      <td>The pipeline parallelism size.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--attention-context-parallel-size`\
      `--attn-cp-size`</td>
      <td>The attention context parallelism size.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--dcp-size`\
      `--decode-context-parallel-size`</td>
      <td>The decode context parallelism size. See [Decode Context Parallelism](/docs/advanced_features/dcp).</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--dcp-comm-backend`</td>
      <td>Communication backend for the DCP attention reduction: AllGather + ReduceScatter, one All-to-All exchange, or FlashInfer MNNVL All-to-All. Unset resolves to `fi_a2a` where the DCP group shares an MNNVL domain on Blackwell, else `a2a` on CUDA/ROCm/XPU, else `ag_rs`.</td>
      <td>`None`</td>
      <td>`ag_rs`, `a2a`, `fi_a2a`</td>
    </tr>

    <tr>
      <td>`--dcp-replicate-q-proj`</td>
      <td>For MLA DCP with the a2a/fi\_a2a backend: replicate the Q projection so each DCP rank computes the full-head query locally. Use `--no-dcp-replicate-q-proj` to disable the model-specific default.</td>
      <td>`None`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--moe-data-parallel-size`\
      `--moe-dp-size`</td>
      <td>The moe data parallelism size.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--pp-max-micro-batch-size`</td>
      <td>The maximum micro batch size in pipeline parallelism.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--pp-async-batch-depth`</td>
      <td>The async batch depth of pipeline parallelism.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--stream-interval`</td>
      <td>The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--incremental-streaming-output`</td>
      <td>Whether to output as a sequence of disjoint segments.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--random-seed`</td>
      <td>The random seed.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--mlx-enable-sampling`</td>
      <td>MLX backend only: sample decode tokens (temperature / top-k / top-p / min-p) instead of greedy argmax. Sampling runs inside the lazy MLX graph, so it works with the overlap scheduler; first tokens from prefill/extend are sampled too. Greedy requests keep exact argmax behavior. Also enables on the MLX path: grammar vocab masks and custom logit processors (these break decode chaining per step; custom processors run on pure-decode steps only), logit\_bias, output logprobs (sampled token / top-k / token\_ids; prompt input logprobs are not computed), NaN sanitization (SGLANG\_SANITIZE\_NAN\_LOGITS), and per-request sampling\_seed under --enable-deterministic-inference (deterministic within MLX only). Penalties are not applied.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--constrained-json-whitespace-pattern`</td>
      <td>(outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model to generate consecutive whitespaces, set the pattern to \[\n\t ]\*</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--constrained-json-disable-any-whitespace`</td>
      <td>(xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--watchdog-timeout`</td>
      <td>Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging.</td>
      <td>`300`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--soft-watchdog-timeout`</td>
      <td>Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging.</td>
      <td>`None`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--dist-timeout`</td>
      <td>Set timeout for torch.distributed initialization.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--download-dir`</td>
      <td>Model download directory for huggingface.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--model-checksum`</td>
      <td>Model file integrity verification. If provided without value, uses model-path as HF repo ID. Otherwise, provide checksums JSON file path or HuggingFace repo ID.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--base-gpu-id`</td>
      <td>The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--gpu-id-step`</td>
      <td>The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,...</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--sleep-on-idle`</td>
      <td>Reduce CPU usage when sglang is idle.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--batch-notify-size`</td>
      <td>Number of streaming notifications to batch before yielding to the event loop. Reduces asyncio wakeup overhead under high concurrency.</td>
      <td>`16`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--stream-response-default-include-usage`</td>
      <td>Include usage in every streaming response (even when stream\_options is not specified).</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-streaming-session`</td>
      <td>Enable streaming session mode and StreamingSession wrapper.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--load-snapshot-publish-interval`</td>
      <td>Publish load snapshot to shared memory every N decode iterations. Prefill and idle always publish immediately.</td>
      <td>`15`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--use-ray`</td>
      <td>Use Ray actors for scheduler process management.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## Logging

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--log-level`</td>
      <td>The logging level of all loggers.</td>
      <td>`info`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--log-level-http`</td>
      <td>The logging level of HTTP server. If not set, reuse --log-level by default.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--log-requests`</td>
      <td>Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--log-requests-level`</td>
      <td>0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output.</td>
      <td>`2`</td>
      <td>`0`, `1`, `2`, `3`</td>
    </tr>

    <tr>
      <td>`--log-requests-format`</td>
      <td>Format for request logging: 'text' (human-readable) or 'json' (structured)</td>
      <td>`text`</td>
      <td>`text`, `json`</td>
    </tr>

    <tr>
      <td>`--log-requests-target`</td>
      <td>Target(s) for request logging: 'stdout' and/or directory path(s) for file output. Can specify multiple targets, e.g., '--log-requests-target stdout /my/path'.</td>
      <td>`None`</td>
      <td>List\[str]</td>
    </tr>

    <tr>
      <td>`--uvicorn-access-log-exclude-prefixes`</td>
      <td>Exclude uvicorn access logs whose request path starts with any of these prefixes. Defaults to empty (disabled).</td>
      <td>`[]`</td>
      <td>List\[str]</td>
    </tr>

    <tr>
      <td>`--crash-dump-folder`</td>
      <td>Folder for crash diagnostics. Stores completed requests retained by the crash-dump buffer plus in-flight requests and, on NVIDIA CUDA, configures device coredumps under this folder. Existing CUDA coredump environment variables take precedence. If not specified, this option does not enable request crash dumping or set CUDA coredump defaults.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--show-time-cost`</td>
      <td>Show time cost of custom marks.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-metrics`</td>
      <td>Enable log prometheus metrics.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-mfu-metrics`</td>
      <td>Enable estimated MFU-related prometheus metrics.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-metrics-for-all-schedulers`</td>
      <td>Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp\_attention is enabled, as otherwise all metrics appear to come from TP 0.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--tokenizer-metrics-custom-labels-header`</td>
      <td>Specify the HTTP header for passing custom labels for tokenizer metrics.</td>
      <td>`x-custom-labels`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--tokenizer-metrics-allowed-custom-labels`</td>
      <td>The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., \{'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set.</td>
      <td>`None`</td>
      <td>List\[str]</td>
    </tr>

    <tr>
      <td>`--bucket-time-to-first-token`</td>
      <td>The buckets of time to first token, specified as a list of floats.</td>
      <td>`None`</td>
      <td>List\[float]</td>
    </tr>

    <tr>
      <td>`--bucket-inter-token-latency`</td>
      <td>The buckets of inter-token latency, specified as a list of floats.</td>
      <td>`None`</td>
      <td>List\[float]</td>
    </tr>

    <tr>
      <td>`--bucket-e2e-request-latency`</td>
      <td>The buckets of end-to-end request latency, specified as a list of floats.</td>
      <td>`None`</td>
      <td>List\[float]</td>
    </tr>

    <tr>
      <td>`--prompt-tokens-buckets`</td>
      <td>The buckets rule of prompt tokens. Supports 3 rule types: 'default' uses predefined buckets; 'tse \<middle> \<base> \<count>' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets \[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom \<value1> \<value2> ...' uses custom bucket values (e.g., 'custom 10 50 100 500').</td>
      <td>`None`</td>
      <td>List\[str]</td>
    </tr>

    <tr>
      <td>`--generation-tokens-buckets`</td>
      <td>The buckets rule for generation tokens histogram. Supports 3 rule types: 'default' uses predefined buckets; 'tse \<middle> \<base> \<count>' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets \[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom \<value1> \<value2> ...' uses custom bucket values (e.g., 'custom 10 50 100 500').</td>
      <td>`None`</td>
      <td>List\[str]</td>
    </tr>

    <tr>
      <td>`--gc-warning-threshold-secs`</td>
      <td>The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable.</td>
      <td>`0.0`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--decode-log-interval`</td>
      <td>The log interval of decode batch.</td>
      <td>`40`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--enable-request-time-stats-logging`</td>
      <td>Enable per request time stats logging</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--kv-events-config`</td>
      <td>Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. Runtime-load publishing for load-aware routers is a separate opt-in; see `--load-publish-endpoint`. Set `replay_endpoint` (e.g. `tcp://*:5558`) to let routers re-fetch dropped batches; `/server_info` advertises its port.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--load-publish-endpoint`</td>
      <td>Opt in to the runtime-load PUB socket that load-aware routers subscribe to. Off by default (unset or `off`). Use `auto` to reserve the dp\_size ports packed after the `--kv-events-config` range, or a wildcard-host TCP address (e.g. `tcp://*:6000`) to place it explicitly; rank r binds port+r and `/server_info` advertises the base under the `kv_events` block. Requires `--kv-events-config`; startup fails if set without one, not bindable, or overlapping the KV range. `auto` reserves 2\*dp\_size ports from the KV base — space co-hosted engines accordingly. The router-facing update cadence follows `--load-snapshot-publish-interval`, so a large value there also staleness-caps this feed.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-trace`</td>
      <td>Enable opentelemetry trace</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--trace-modules`</td>
      <td>Select the components to trace. Available options are 'request' and 'mooncake'. Format: \<module1 name>,\<module2 name>,......</td>
      <td>`request`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--otlp-traces-endpoint`</td>
      <td>Config opentelemetry collector endpoint if --enable-trace is set. format: \<ip>:\<port></td>
      <td>`localhost:4317`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--grpc-http-sidecar-port`</td>
      <td>Port for the HTTP sidecar server in gRPC mode (--grpc-mode). Serves Prometheus metrics and profiling endpoints. Defaults to --port + 1. Not used in HTTP mode.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--extra-metric-labels`</td>
      <td>The custom labels for metrics. e.g. '\{"label1": "value1", "label2": "value2"}'</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-forward-pass-metrics`</td>
      <td>Enable per-iteration forward pass metrics via ZMQ IPC. External consumers (e.g. Dynamo planner) subscribe to the IPC endpoint exposed in server\_args.forward\_pass\_metrics\_ipc\_name.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--forward-pass-metrics-worker-id`</td>
      <td>—</td>
      <td>`""`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--forward-pass-metrics-ipc-name`</td>
      <td>—</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## RequestMetricsExporter configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--export-metrics-to-file`</td>
      <td>Export performance metrics for each request to local file (e.g. for forwarding to external systems).</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--export-metrics-to-file-dir`</td>
      <td>Directory path for writing performance metrics files (required when --export-metrics-to-file is enabled).</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## Responses storage

Responses storage is now disabled by default, including on standalone servers. Add `--enable-response-store` to retain responses for retrieval, `previous_response_id` chaining, and background requests. Storage is process-local and in-memory, has no TTL or size limit, and is lost on restart.

Foreground generation and streaming still accept the API's default `store: true` without retaining responses or message history when server storage is disabled. The response's `store` field echoes the client request; the server flag controls persistence. With storage enabled, `store: false` opts out for the current request while allowing it to read a stored predecessor.

Background requests, including streaming requests, require both `--enable-response-store` and `store: true`. Without the flag, chaining, background requests, retrieval, and cancellation return HTTP 400. Cancellation applies to detached background requests; active streams have no stored response until completion.

Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `decode` fails at startup. PD clients must send explicit conversation history in foreground requests. Built-in web search and code interpreter calls are also unsupported under PD.

## API related

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--api-key`</td>
      <td>Set API key of the server. It is also used in the OpenAI API compatible server.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--admin-api-key`</td>
      <td>Set **admin API key** for administrative/control endpoints (e.g., weights update, cache flush, `/server_info`). Endpoints marked as admin-only require `Authorization: Bearer <admin_api_key>` when this is set.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--served-model-name`</td>
      <td>Override the model name returned by the v1/models endpoint in OpenAI API server.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--weight-version`</td>
      <td>Version identifier for the model weights. Defaults to 'default' if not specified.</td>
      <td>`default`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--chat-template`</td>
      <td>The builtin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--hf-chat-template-name`</td>
      <td>When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool\_use', 'rag'), specify which named template to use. If not set, the first available template is used.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--default-chat-template-kwargs`</td>
      <td>A JSON object used as defaults for `chat_template_kwargs` by Chat Completions, Responses, Anthropic-compatible requests, and chat tokenization. Direct Ollama and embedding paths do not consume these defaults. For ordinary template keys, explicit per-request values take precedence. Keys are specific to the selected model's chat template; `reasoning_effort` is normalized separately.</td>
      <td>`None`</td>
      <td>Type: JSON / Dict</td>
    </tr>

    <tr>
      <td>`--completion-template`</td>
      <td>The builtin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--file-storage-path`</td>
      <td>The path of the file storage in backend.</td>
      <td>`sglang_storage`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-cache-report`</td>
      <td>Return number of cached tokens in usage.prompt\_tokens\_details for each openai request.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--reasoning-parser`</td>
      <td>Specify the parser for reasoning models. Use `auto` to detect the parser from the model's chat template.</td>
      <td>`None`</td>
      <td>`auto`, `apertus2509`, `deepseek-r1`, `deepseek-v3`, `deepseek-v4`, `dots`, `glm45`, `ling3`, `hunyuan`, `gpt-oss`, `k2_horizon`, `kimi`, `kimi_k2`, `kimi_k3`, `mimo`, `muse`, `poolside_v1`, `qwen3`, `qwen3-thinking`, `minimax`, `minimax-append-think`, `minimax-m3`, `step3`, `step3p5`, `mistral`, `nemotron_3`, `interns1`, `gemma4`, `gigachat35`, `inkling`, `cohere_command4`</td>
    </tr>

    <tr>
      <td>`--tool-call-parser`</td>
      <td>Specify the parser for handling tool-call interactions. Use `auto` to detect the parser from the model's chat template.</td>
      <td>`None`</td>
      <td>`auto`, `apertus2509`, `cohere_command4`, `deepseekv3`, `deepseekv31`, `deepseekv32`, `deepseekv4`, `dots`, `glm`, `glm45`, `glm47`, `gpt-oss`, `k2_horizon`, `kimi_k2`, `kimi_k3`, `lfm2`, `ling3`, `llama3`, `mimo`, `minicpm5`, `mistral`, `muse`, `poolside_v1`, `pythonic`, `qwen`, `qwen25`, `qwen3_coder`, `spark25`, `step3`, `step3p5`, `minimax-m2`, `minimax-m3`, `trinity`, `interns1`, `hermes`, `hunyuan`, `gigachat3`, `gigachat35`, `gemma4`, `inkling`</td>
    </tr>

    <tr>
      <td>`--tool-server`</td>
      <td>Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no external tool server will be used. Native GPT-OSS `web_search` can still be enabled with `EXA_API_KEY`.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--sampling-defaults`</td>
      <td>Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top\_p=1.0, etc.). 'model' uses the model's generation\_config.json to get the recommended sampling parameters if available. Default is 'model'.</td>
      <td>`model`</td>
      <td>`openai`, `model`</td>
    </tr>

    <tr>
      <td>`--strip-thinking-cache`</td>
      <td>Skip caching reasoning-model output (thinking + answer) in the radix tree on finish; keep only the prompt prefix. Opt-in: changes cache contents.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-strict-thinking`</td>
      <td>Enable strict token filtering during the thinking phase. Blocks model-specific excluded tokens (e.g., tool call markers) during reasoning. Requires a grammar backend that supports token filtering. See [Thinking Budget](/docs/advanced_features/thinking_budget) for per-request thinking caps.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--asr-max-buffer-seconds`</td>
      <td>Maximum seconds of PCM audio the streaming ASR WebSocket handler will accumulate before closing the session with a buffer\_overflow error. Guards against OOM when a client streams audio faster than inference can consume it. Default 60s.</td>
      <td>`60`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--asr-max-concurrent-sessions`</td>
      <td>Maximum number of concurrent realtime ASR WebSocket sessions served by /v1/realtime. New connections beyond this cap are accepted, sent an error\{code:too\_many\_sessions} frame, and closed. Default 32.</td>
      <td>`32`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Data parallelism

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--data-parallel-size`\
      `--dp-size`</td>
      <td>The number of data-parallel replicas of the model.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--attention-data-parallel-size`\
      `--attn-dp-size`</td>
      <td>The attention data parallelism size: the number of data-parallel attention groups inside the TP group, one scheduler each, while the FFN stays tensor parallel.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--load-balance-method`</td>
      <td>The load balancing strategy for data parallelism. The `total_tokens` algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers.</td>
      <td>`auto`</td>
      <td>`auto`, `round_robin`, `follow_bootstrap_room`, `total_requests`, `total_tokens`</td>
    </tr>
  </tbody>
</table>

## Multi-node distributed serving

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dist-init-addr`\
      `--nccl-init-addr`</td>
      <td>The host address for initializing distributed backend (e.g., `192.168.0.2:25000`).</td>
      <td>` None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--nnodes`</td>
      <td>The number of nodes.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--node-rank`</td>
      <td>The node rank.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Model override args

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--json-model-override-args`</td>
      <td>A dictionary in JSON string format used to override default model configurations.</td>
      <td>`{}`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--preferred-sampling-params`</td>
      <td>json-formatted sampling settings that will be returned in /get\_model\_info</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## LoRA

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-lora`</td>
      <td>Enable LoRA support for the model. This argument is automatically set to `True` if `--lora-paths` is provided for backward compatibility.</td>
      <td>`False`</td>
      <td>Bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-lora-overlap-loading`</td>
      <td>Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.</td>
      <td>`False`</td>
      <td>Bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--max-lora-rank`</td>
      <td>The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--lora-target-modules`</td>
      <td>The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. You can also set it to `all` to enable LoRA for all supported modules; note this may introduce minor performance overhead.</td>
      <td>`None`</td>
      <td>`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, `qkv_proj`, `gate_up_proj`, `all`</td>
    </tr>

    <tr>
      <td>`--lora-paths`</td>
      <td>The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: `<PATH>` | `<NAME>=<PATH>` | JSON with schema `{"lora_name": str, "lora_path": str, "pinned": bool}`.</td>
      <td>`None`</td>
      <td>Type: List\[str] / JSON objects</td>
    </tr>

    <tr>
      <td>`--max-loras-per-batch`</td>
      <td>Maximum number of adapters for a running batch, including base-only requests.</td>
      <td>`8`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--max-loaded-loras`</td>
      <td>If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ `--max-loras-per-batch`.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--lora-eviction-policy`</td>
      <td>LoRA adapter eviction policy when the GPU memory pool is full.</td>
      <td>`lru`</td>
      <td>`lru`, `fifo`</td>
    </tr>

    <tr>
      <td>`--lora-backend`</td>
      <td>Choose the kernel backend for multi-LoRA serving.</td>
      <td>`csgmv`</td>
      <td>`triton`, `csgmv`, `ascend`, `torch_native`</td>
    </tr>

    <tr>
      <td>`--max-lora-chunk-size`</td>
      <td>Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when `--lora-backend` is `csgmv`. Larger values may improve performance.</td>
      <td>`16`</td>
      <td>`16`, `32`, `64`, `128`</td>
    </tr>

    <tr>
      <td>`--lora-drain-wait-threshold`</td>
      <td>When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default).</td>
      <td>`0.0`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--experts-shared-outer-loras`</td>
      <td>Force shared outer LoRA mode for MoE models. When set, w1/w3 lora\_A and w2 lora\_B are shared across experts (expert\_dim=1). Use --no-experts-shared-outer-loras to force disable. By default this is auto-detected from adapter weights.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--lora-use-virtual-experts`</td>
      <td>Enable virtual expert computation for MoE models. When set, the model will use virtual expert computation.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--lora-strict-loading`</td>
      <td>Enable strict loading for LoRA adapters. When set, mismatched or missing keys in the adapter weights will raise an error.</td>
      <td>`False`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## Kernel Backends (Attention, Sampling, Grammar, GEMM)

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--attention-backend`</td>
      <td>Choose the kernels for attention layers.</td>
      <td>`None`</td>
      <td>`triton`, `torch_native`, `flex_attention`, `dsa`, `nsa`, `dsv4`, `compressed`, `fa3`, `fa4`, `flashinfer`, `flashmla`, `trtllm_mla`, `cutedsl_mla`, `tokenspeed_mla`, `trtllm_mha`, `aiter`, `wave`, `intel_amx`, `ascend`, `intel_xpu`</td>
    </tr>

    <tr>
      <td>`--prefill-attention-backend`</td>
      <td>Choose the kernels for prefill attention layers (have priority over --attention-backend).</td>
      <td>`None`</td>
      <td>`triton`, `torch_native`, `flex_attention`, `dsa`, `nsa`, `dsv4`, `compressed`, `fa3`, `fa4`, `flashinfer`, `flashmla`, `trtllm_mla`, `cutedsl_mla`, `tokenspeed_mla`, `trtllm_mha`, `aiter`, `wave`, `intel_amx`, `ascend`, `intel_xpu`</td>
    </tr>

    <tr>
      <td>`--decode-attention-backend`</td>
      <td>Choose the kernels for decode attention layers (have priority over --attention-backend).</td>
      <td>`None`</td>
      <td>`triton`, `torch_native`, `flex_attention`, `dsa`, `nsa`, `dsv4`, `compressed`, `fa3`, `fa4`, `flashinfer`, `flashmla`, `trtllm_mla`, `cutedsl_mla`, `tokenspeed_mla`, `trtllm_mha`, `aiter`, `wave`, `intel_amx`, `ascend`, `intel_xpu`</td>
    </tr>

    <tr>
      <td>`--enable-nvidia-megakernel`</td>
      <td>Enable the NVIDIA megakernel from the sgl-nvidia-mega package. Supported decode and verify steps use its fused kernels; other steps use the standard forward. An unsupported startup configuration raises an error listing the reasons. Experimental feature.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-amd-monokernel`</td>
      <td>Enable the AMD monokernels from the sgl-amd-mono package. Supported decode and verify steps use persistent kernels; other steps use the standard forward. An unsupported startup configuration raises an error listing the reasons. Experimental feature.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--monokernel-backend`</td>
      <td>**Deprecated.** Use `--enable-amd-monokernel` instead of `--monokernel-backend amd`.</td>
      <td>`None`</td>
      <td>`amd`</td>
    </tr>

    <tr>
      <td>`--sampling-backend`</td>
      <td>Choose the kernels for sampling layers.</td>
      <td>`None`</td>
      <td>`flashinfer`, `pytorch`, `ascend`</td>
    </tr>

    <tr>
      <td>`--grammar-backend`</td>
      <td>Choose the backend for grammar-guided decoding.</td>
      <td>`None`</td>
      <td>`xgrammar`, `outlines`, `llguidance`, `none`</td>
    </tr>

    <tr>
      <td>`--mm-attention-backend`</td>
      <td>Set multimodal attention backend.</td>
      <td>`None`</td>
      <td>`sdpa`, `fa3`, `fa4`, `triton_attn`, `ascend_attn`, `aiter_attn`, `flashinfer_cudnn`, `amx_attn`, `xpu_attn`</td>
    </tr>

    <tr>
      <td>`--dsa-prefill-backend`</td>
      <td>DSA backend for the prefill stage (overrides `--attention-backend` when running DeepSeek DSA-style attention).</td>
      <td>Auto (hardware-dependent)</td>
      <td>`flashmla_sparse`, `flashmla_sparse_q8`, `triton_sparse_mla`, `flashmla_kv`, `flashmla_auto`, `fa3`, `tilelang`, `aiter`, `trtllm`</td>
    </tr>

    <tr>
      <td>`--dsa-triton-union`</td>
      <td>Union group size for `--dsa-prefill-backend triton_sparse_mla`: adjacent query tokens share one gathered index set (exact; an ownership mask restores each token's own softmax). Requires `num_q_heads * union` to be a power of two in \[16, 32]; forced off under `--enable-deterministic-inference`.</td>
      <td>`0`</td>
      <td>`0`, `2`, `4`</td>
    </tr>

    <tr>
      <td>`--dsa-decode-backend`</td>
      <td>DSA backend for the decode stage when running DeepSeek DSA-style attention. Overrides `--attention-backend` for decoding.</td>
      <td>Auto (hardware-dependent)</td>
      <td>`flashmla_sparse`, `flashmla_kv`, `flashmla_auto`, `fa3`, `tilelang`, `aiter`, `trtllm`</td>
    </tr>

    <tr>
      <td>`--dsa-topk-backend`</td>
      <td>Choose the DSA indexer top-k backend for the target model. The `torch` backend currently requires `SGLANG_DSA_FUSE_TOPK=false`.</td>
      <td>`sgl-kernel`</td>
      <td>`sgl-kernel`, `torch`, `flashinfer`</td>
    </tr>

    <tr>
      <td>`--enable-deepseek-v4-fp4-indexer`</td>
      <td>Enable the experimental FP4 C4 indexer path for DeepSeek V4. When unset, SGLang keeps the default DeepSeek V4 indexer path. Requires SM100 GPUs with DeepGEMM FP4 indexer support.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--fp8-gemm-backend`</td>
      <td>Choose the runner backend for Blockwise FP8 GEMM operations. For MXFP8 dense GEMM, `auto` selects `flashinfer_cutedsl` when FlashInfer reports support (currently SM100/SM103), and otherwise selects `flashinfer_cutlass` on supported Blackwell GPUs. Options also include 'deep\_gemm' (JIT-compiled), 'flashinfer\_trtllm' (FlashInfer TRTLLM backend; SM100/SM103 only), 'flashinfer\_cutlass' (FlashInfer CUTLASS backend), 'flashinfer\_cutedsl' (FlashInfer CuTe DSL MXFP8 backend; SM100/SM103 only), 'flashinfer\_deepgemm' (Hopper SM90 only, uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for SM120 GPUs), 'triton' (fallback, widely compatible), and 'aiter' (ROCm only).</td>
      <td>`auto`</td>
      <td>`auto`, `deep_gemm`, `flashinfer_trtllm`, `flashinfer_cutlass`, `flashinfer_cutedsl`, `flashinfer_deepgemm`, `cutlass`, `triton`, `aiter`</td>
    </tr>

    <tr>
      <td>`--fp4-gemm-backend`</td>
      <td>Choose the runner backend for NVFP4 GEMM operations. Options: 'auto' (default; selects `flashinfer_cutedsl` on SM100, `marlin` on SM80-SM90, `flashinfer_cutlass` otherwise (including SM120)), 'flashinfer\_cutlass' (FlashInfer CUTLASS backend), 'flashinfer\_cudnn' (FlashInfer cuDNN backend, optimal on CUDA 13+ with cuDNN 9.15+), 'flashinfer\_cutedsl' (FlashInfer CuTe DSL backend), 'flashinfer\_trtllm' (FlashInfer TensorRT-LLM backend, requires different weight preparation with shuffling), 'marlin' (weight-only W4A16 fallback for SM80-SM90). Requires FlashInfer to be installed.</td>
      <td>`auto`</td>
      <td>`auto`, `flashinfer_cudnn`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `marlin`</td>
    </tr>

    <tr>
      <td>`--bf16-gemm-backend`</td>
      <td>Choose the backend for unquantized BF16 GEMM operations. Options: `auto` (default; selects `cutedsl` on SM10x GPUs, except deterministic inference selects `torch`; otherwise uses cuBLAS via `torch.nn.functional.linear`), `cutedsl` (SGLang JIT CuTe DSL TGV BF16 GEMM on SM10x; dispatches between the CuTe DSL kernel and cuBLAS), `torch` (always uses cuBLAS via `torch.nn.functional.linear`).</td>
      <td>`auto`</td>
      <td>`auto`, `cutedsl`, `torch`</td>
    </tr>

    <tr>
      <td>`--disable-flashinfer-autotune`</td>
      <td>Flashinfer autotune is enabled by default. Set this flag to disable the autotune.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--flashinfer-autotune-skip-ops`</td>
      <td>FlashInfer custom-op identifiers to skip during autotuning. See [FlashInfer's autotuning documentation](https://docs.flashinfer.ai/autotuning.html). Skipped ops use the heuristic fallback. SGLang temporarily skips `mxfp8_gemm` by default due to an IMA.</td>
      <td>`None`</td>
      <td>string</td>
    </tr>

    <tr>
      <td>`--radix-cache-backend`</td>
      <td>Name of a radix-cache backend previously registered via register\_radix\_cache\_backend. Omit this flag to use the built-in default cache selection chain.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## Speculative decoding

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-algorithm`</td>
      <td>Speculative algorithm.</td>
      <td>`None`</td>
      <td>`EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-path`\
      `--speculative-draft-model`</td>
      <td>The path of the draft model weights. This can be a local folder or a Hugging Face repo ID.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`speculative_boundary_reduction`\
      Server args field; no CLI flag</td>
      <td>Resolved FFN boundary reduction policy for the speculative draft. Argument processing derives this field from `boundary_reduction` and the draft model configuration; an independent value is overwritten by resolution. See [FFN boundary reduction](#ffn-boundary-reduction).</td>
      <td>`rs+rsv` before resolution; model-dependent afterward</td>
      <td>`ar`, `rs`, `rsv`, `rs+rsv`</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-revision`</td>
      <td>The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--speculative-draft-load-format`</td>
      <td>The format of the draft model weights to load. If not specified, will use the same format as `--load-format`. Use 'dummy' to initialize draft model weights with random values for profiling.</td>
      <td>`None`</td>
      <td>`auto`, `pt`, `safetensors`, `instanttensor`, `npcache`, `dummy`, `sharded_state`, `gguf`, `bitsandbytes`, `mistral`, `layered`, `flash_rl`, `remote`, `remote_instance`, `fastsafetensors`, `private`, `runai_streamer`</td>
    </tr>

    <tr>
      <td>`--speculative-num-steps`</td>
      <td>The number of steps sampled from draft model in Speculative Decoding.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-eagle-topk`</td>
      <td>The number of tokens sampled from the draft model in eagle2 each step.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>The number of tokens sampled from the draft model in Speculative Decoding.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-single`</td>
      <td>Accept a draft token if its probability in the target model is greater than this threshold.</td>
      <td>`1.0`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-acc`</td>
      <td>The accept probability of a draft token is raised from its target probability p to min(1, p / threshold\_acc).</td>
      <td>`1.0`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--speculative-use-block-verification`</td>
      <td>Use block verification for EAGLE/EAGLE3/NEXTN on CUDA or ROCm (requires topk=1).</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`--speculative-token-map`</td>
      <td>The path of the draft model's small vocab table.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--speculative-attention-mode`</td>
      <td>Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'.</td>
      <td>`prefill`</td>
      <td>`prefill`, `decode`</td>
    </tr>

    <tr>
      <td>`--speculative-draft-attention-backend`</td>
      <td>Attention backend for speculative decoding drafting.</td>
      <td>`None`</td>
      <td>Same as attention backend options</td>
    </tr>

    <tr>
      <td>`--speculative-dsa-topk-backend`</td>
      <td>Choose the DSA indexer top-k backend for speculative draft workers. The `torch` backend currently requires `SGLANG_DSA_FUSE_TOPK=false`.</td>
      <td>`sgl-kernel`</td>
      <td>`sgl-kernel`, `torch`, `flashinfer`</td>
    </tr>

    <tr>
      <td>`--speculative-moe-runner-backend`</td>
      <td>MOE backend for EAGLE speculative decoding, see `--moe-runner-backend` for options. Same as moe runner backend if unset.</td>
      <td>`None`</td>
      <td>`auto`, `deep_gemm`, `triton`, `triton_kernel`, `flashinfer_trtllm`, `experimental_sgl_trtllm`, `flashinfer_trtllm_routed`, `flashinfer_cutlass`, `flashinfer_mxfp4`, `flashinfer_cutedsl`, `cutlass`, `aiter`, `marlin`</td>
    </tr>

    <tr>
      <td>`--speculative-moe-a2a-backend`</td>
      <td>MOE A2A backend for EAGLE speculative decoding, see `--moe-a2a-backend` for options. Same as moe a2a backend if unset.</td>
      <td>`None`</td>
      <td>`none`, `deepep`, `mooncake`, `nixl`, `mori`, `ascend_fuseep`, `flashinfer`, `megamoe`, `pplx`</td>
    </tr>

    <tr>
      <td>`--speculative-enable-w4a4-mxfp4-megamoe`</td>
      <td>Whether the draft model's MXFP4 MegaMoE layers use the W4A4 `mxf4xmxf4` MMA type. Pass `--no-speculative-enable-w4a4-mxfp4-megamoe` to keep the draft on `fp8xfp4` (W4A8) while the target runs W4A4. Same as `--enable-w4a4-mxfp4-megamoe` if unset. A draft that runs MXFP4 MegaMoE with a different MMA type from the target may allocate an additional MegaMoE symmetric buffer after the KV pool is sized; lower `--mem-fraction-static` if CUDA-graph capture runs out of memory.</td>
      <td>`None`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-quantization`</td>
      <td>The quantization method for speculative model.</td>
      <td>`None`</td>
      <td>Same as `--quantization` options</td>
    </tr>

    <tr>
      <td>`--speculative-dflash-block-size`</td>
      <td>DFLASH only. Block size (verify window length). Alias of --speculative-num-draft-tokens for DFLASH.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-draft-window-size`</td>
      <td>Sliding window size for the draft model. Honored by Llama EAGLE-3 (`LlamaForCausalLMEagle3`) and DFLASH only; other EAGLE-3 backends (e.g. MLA-based drafters) silently ignore it. For Llama EAGLE-3, the drafter only attends to the most recent N keys (verifier hidden states + its own outputs); the verifier is unaffected. For DFLASH, the draft worker keeps a recent target-token window in its local KV cache (paged backends may retain up to one extra page on the left for alignment). Default is full attention/context.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Ngram speculative decoding

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-ngram-min-bfs-breadth`</td>
      <td>The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-bfs-breadth`</td>
      <td>The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding.</td>
      <td>`10`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-match-type`</td>
      <td>Ngram tree-building mode. `BFS` selects recency-based expansion and `PROB` selects frequency-based expansion. This setting is forwarded to the ngram cache implementation.</td>
      <td>`BFS`</td>
      <td>`BFS`, `PROB`</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-trie-depth`</td>
      <td>Maximum suffix length stored and matched by the ngram trie.</td>
      <td>`18`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-capacity`</td>
      <td>The cache capacity for ngram speculative decoding.</td>
      <td>`10000000`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-external-corpus-path`</td>
      <td>Path to an external JSONL corpus to pre-load into SAM at startup. Additional corpora can be added at runtime via POST /add\_external\_corpus.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-external-sam-budget`</td>
      <td>Number of draft nodes reserved for the external SAM subtree in ngram speculative decoding.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-external-corpus-max-tokens`</td>
      <td>Fail startup if the tokenized external ngram corpus exceeds this many tokens. Tune this based on your CPU memory budget.</td>
      <td>`10000000`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--speculative-adaptive`</td>
      <td>Enable adaptive speculative decoding that dynamically adjusts num\_steps based on acceptance rate.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--speculative-adaptive-config`</td>
      <td>Path to a JSON config file for adaptive speculative decoding tuning knobs.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## Multi-layer Eagle speculative decoding

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-multi-layer-eagle`</td>
      <td>Enable multi-layer Eagle speculative decoding.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## MoE

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--expert-parallel-size`\
      `--ep-size`\
      `--ep`</td>
      <td>The expert parallelism size.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--moe-a2a-backend`</td>
      <td>Select the backend for all-to-all communication for expert parallelism.</td>
      <td>`none`</td>
      <td>`none`, `deepep`, `mooncake`, `nixl`, `mori`, `ascend_fuseep`, `flashinfer`, `megamoe`, `pplx`</td>
    </tr>

    <tr>
      <td>`--moe-runner-backend`</td>
      <td>Choose the runner backend for MoE.</td>
      <td>`auto`</td>
      <td>`auto`, `deep_gemm`, `triton`, `triton_kernel`, `flashinfer_trtllm`, `experimental_sgl_trtllm`, `flashinfer_trtllm_routed`, `flashinfer_cutlass`, `flashinfer_mxfp4`, `flashinfer_cutedsl`, `cutlass`, `aiter`, `marlin`</td>
    </tr>

    <tr>
      <td>`--flashinfer-mxfp4-moe-precision`</td>
      <td>Choose the computation precision of flashinfer mxfp4 moe. On SM90, `fp8` selects the Humming-style MXFP4-weight x FP8-activation path introduced by FlashInfer #3738 and requires FlashInfer >= 0.6.18.</td>
      <td>`default`</td>
      <td>`default`, `bf16`, `fp8`</td>
    </tr>

    <tr>
      <td>`--enable-flashinfer-allreduce-fusion`</td>
      <td>Enable FlashInfer allreduce fusion with Residual RMSNorm.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-aiter-allreduce-fusion`</td>
      <td>Enable aiter allreduce fusion with Residual RMSNorm.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--deepep-mode`</td>
      <td>Select the mode when enable DeepEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch.</td>
      <td>`auto`</td>
      <td>`normal`, `low_latency`, `auto`</td>
    </tr>

    <tr>
      <td>`--deepep-dispatcher-output-dtype`</td>
      <td>Select DeepEP dispatcher output dtype, could be `bf16`, `fp8`, `int8` (only Ascend A2/A3 Series NPUs), `nvfp4` or `auto`. Default is `auto`, which follows a priority order (server argument → deprecated env var → input\_global\_scale check → dispatcher\_output\_dtype from quant\_config → flashinfer/cutlass backend → NPU BF16 default → GPU FP8 default).</td>
      <td>`auto`</td>
      <td>`auto`, `bf16`, `fp8`, `int8`, `nvfp4`</td>
    </tr>

    <tr>
      <td>`--flashinfer-a2a-dispatch-type`</td>
      <td>Select FlashInfer A2A dispatcher activation dtype. When omitted, it falls back to the `SGLANG_MOE_NVFP4_DISPATCH` environment variable. Explicit `auto` selects `mxfp8` for `--quantization mxfp8`, `nvfp4` for modelopt FP4 or hybrid NVFP4 MoE checkpoints, and `bf16` otherwise. The `SGLANG_MOE_NVFP4_DISPATCH` environment variable cannot be set with this argument.</td>
      <td>`None`</td>
      <td>`auto`, `bf16`, `nvfp4`, `mxfp8`</td>
    </tr>

    <tr>
      <td>`--ep-num-redundant-experts`</td>
      <td>Allocate this number of redundant experts in expert parallel.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--ep-dispatch-algorithm`</td>
      <td>The algorithm to choose ranks for redundant experts in expert parallel.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--init-expert-location`</td>
      <td>Initial location of EP experts.</td>
      <td>`trivial`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-eplb`</td>
      <td>Enable EPLB algorithm</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--eplb-algorithm`</td>
      <td>Chosen EPLB algorithm</td>
      <td>`auto`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--eplb-rebalance-num-iterations`</td>
      <td>Number of iterations to automatically trigger a EPLB re-balance.</td>
      <td>`1000`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--eplb-rebalance-layers-per-chunk`</td>
      <td>Number of layers to rebalance per forward pass.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--eplb-min-rebalancing-utilization-threshold`</td>
      <td>Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range \[0.0, 1.0].</td>
      <td>`1.0`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--expert-distribution-recorder-mode`</td>
      <td>Mode of expert distribution recorder.</td>
      <td>` None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--expert-distribution-recorder-buffer-size`</td>
      <td>Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--expert-balancedness-report-mode`</td>
      <td>Where to report expert balancedness.</td>
      <td>`off`</td>
      <td>`off`, `server_log`, `prometheus`, `both`</td>
    </tr>

    <tr>
      <td>`--deepep-config`</td>
      <td>Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path.</td>
      <td>` None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--moe-dense-tp-size`</td>
      <td>TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports.</td>
      <td>` none`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--elastic-ep-backend`</td>
      <td>Specify the collective communication backend for elastic EP. Currently supports 'mooncake'.</td>
      <td>` None`</td>
      <td>`none`, `mooncake`, `nixl`</td>
    </tr>

    <tr>
      <td>`--enable-elastic-expert-backup`</td>
      <td>Enable elastic EP backend to backup expert weights in DRAM feature. Currently supports 'mooncake'.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--mooncake-ib-device`</td>
      <td>The InfiniBand devices for Mooncake Backend transfer, accepts multiple comma-separated devices (e.g., --mooncake-ib-device mlx5\_0,mlx5\_1). Default is None, which triggers automatic device detection when Mooncake Backend is enabled.</td>
      <td>` None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-waterfill`</td>
      <td>Enable Waterfill: dispatch the fused shared expert as an extra routed expert slot to the least-loaded EP rank. Supports DeepEP and MegaMOE MoE A2A backends, implicitly enables shared-expert fusion, and supports `--deepep-mode auto`, `normal`, or `low_latency` when used with DeepEP. Use `auto` or `low_latency` for production DeepEP decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2. By default, Waterfill uses the static local-batch path; set `SGLANG_DISABLE_STATIC_WATERFILL=1` to force dynamic Waterfill with runtime EP all-reduce.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--flashinfer-allreduce-fusion-backend`</td>
      <td>Enable FlashInfer allreduce fusion and choose backend. Defaults to auto. 'auto': choose mnnvl on SM90 single-node systems and SM100/SM103 single-node or multi-node systems; choose trtllm otherwise. 'trtllm': available on single-node systems only. 'mnnvl': available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. Fuses allreduce with Residual + RMSNorm for supported MoE models.</td>
      <td>`None`</td>
      <td>`auto`, `trtllm`, `mnnvl`</td>
    </tr>

    <tr>
      <td>`--enforce-disable-flashinfer-allreduce-fusion`</td>
      <td>Enforce disable FlashInfer allreduce fusion.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## Mamba Cache

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--max-mamba-cache-size`</td>
      <td>The maximum size of the mamba cache.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--mamba-ssm-dtype`</td>
      <td>The data type of the SSM states in mamba cache. If not set, read from the model config.</td>
      <td>Auto (from model config)</td>
      <td>`float32`, `bfloat16`, `float16`</td>
    </tr>

    <tr>
      <td>`--enable-mamba-cache-stochastic-rounding`</td>
      <td>Enable stochastic rounding when writing FP16 Mamba SSM cache states. Requires `--mamba-ssm-dtype float16` and CUDA. With `--mamba-backend triton`, requires SM100.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`--mamba-cache-philox-rounds`</td>
      <td>Number of Philox rounds to use for stochastic rounding of FP16 Mamba SSM cache writes. Triton uses the Triton default when set to 0; FlashInfer uses 10 rounds when set to 0.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--mamba-full-memory-ratio`</td>
      <td>The ratio of mamba state memory to full kv cache memory.</td>
      <td>`0.9`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--mamba-radix-cache-strategy`</td>
      <td>The strategy to use for mamba scheduler. `auto` currently defaults to `no_buffer`. 1. `no_buffer` does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. `extra_buffer` supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes `2x` for non-spec; `1+(1/(2+speculative_num_draft_tokens))x` for spec dec (e.g. 1.16x if speculative\_num\_draft\_tokens==4)). 2a. `extra_buffer` is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support, currently only extra\_buffer supports branching. 3. `extra_buffer_lazy` lowers `extra_buffer`'s slot cost by allocating one track slot per request instead of two; the second slot is allocated on demand at track-interval boundaries (for speculative decoding it is reserved ahead of each verify window and committed only for accepted boundary crossings). Compatible with speculative decoding (EAGLE/NGRAM/DSPARK/DFLASH); not supported under PD disaggregation.</td>
      <td>`auto`</td>
      <td>`auto`, `no_buffer`, `extra_buffer`, `extra_buffer_lazy`</td>
    </tr>

    <tr>
      <td>`--mamba-track-interval`</td>
      <td>The interval (in tokens) to track the mamba state during decode. Only used when `--mamba-radix-cache-strategy` is `extra_buffer`. Must be divisible by page\_size if set, and must be >= speculative\_num\_draft\_tokens when using speculative decoding.</td>
      <td>`256`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--enable-int8-mamba-checkpoint`</td>
      <td>Store radix-cached linear-attn (mamba) states in int8 (separate checkpoint pool) for \~2x cached-prefix capacity at fixed memory.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--int8-mamba-ckpt-size`</td>
      <td>Number of int8 mamba checkpoint slots (default: 2x the active mamba pool size).</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--mamba-backend`</td>
      <td>Choose the kernel backend for Mamba SSM operations. Default is 'triton'. Options: 'triton' (default), 'flashinfer' (requires FlashInfer with Mamba support).</td>
      <td>`triton`</td>
      <td>`triton`, `flashinfer`</td>
    </tr>

    <tr>
      <td>`--linear-attn-backend`</td>
      <td>The default kernel backend for linear attention (GDN/KDA). Can be overridden per-mode by --linear-attn-decode-backend and --linear-attn-prefill-backend.</td>
      <td>`triton`</td>
      <td>`triton`, `cutedsl`, `flashinfer`</td>
    </tr>

    <tr>
      <td>`--linear-attn-decode-backend`</td>
      <td>Override the kernel backend for linear attention decode. If not set, uses --linear-attn-backend.</td>
      <td>`None`</td>
      <td>`triton`, `cutedsl`, `flashinfer`</td>
    </tr>

    <tr>
      <td>`--linear-attn-prefill-backend`</td>
      <td>Override the kernel backend for linear attention prefill/extend. If not set, uses --linear-attn-backend.</td>
      <td>`None`</td>
      <td>`triton`, `cutedsl`, `flashinfer`</td>
    </tr>
  </tbody>
</table>

## Hierarchical cache

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-hierarchical-cache`\
      `--enable-hicache`</td>
      <td>Enable hierarchical cache</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-unified-cache-external-linker`</td>
      <td>Link UnifiedRadixCache directly to an external KV store, with no host cache tier.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--unified-cache-external-linker-backend`</td>
      <td>Select the external linker backend. Use `dynamic` for an installed out-of-tree linker and specify its class and module in `--unified-cache-external-linker-config`.</td>
      <td>`mooncake`</td>
      <td>`mooncake`, `mori`, `dynamic`</td>
    </tr>

    <tr>
      <td>`--unified-cache-external-linker-config`</td>
      <td>JSON object configuring the selected backend. For `dynamic`, provide `linker` (class name), `linker_module_path` (Python module), and optional `linker_extra_config` (object). Built-in backends accept storage options directly; when supplied, this config replaces `--hicache-storage-backend-extra-config`, even when empty. You can supply config while the linker is disabled.</td>
      <td>`None`</td>
      <td>Type: JSON object</td>
    </tr>

    <tr>
      <td>`--hicache-ratio`</td>
      <td>The ratio of the size of host KV cache memory pool to the size of device pool.</td>
      <td>`2.0`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--hicache-size`</td>
      <td>The size of host KV cache memory pool in gigabytes, which will override the hicache\_ratio if set.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--hicache-write-policy`</td>
      <td>The write policy of hierarchical cache.</td>
      <td>`write_through`</td>
      <td>`write_back`, `write_through`, `write_through_selective`</td>
    </tr>

    <tr>
      <td>`--hicache-io-backend`</td>
      <td>The IO backend for KV cache transfer between CPU and GPU</td>
      <td>`kernel`</td>
      <td>`direct`, `kernel`, `kernel_ascend`</td>
    </tr>

    <tr>
      <td>`--hicache-mem-layout`</td>
      <td>The layout of host memory pool for hierarchical cache.</td>
      <td>`page_first`</td>
      <td>`layer_first`, `page_first`, `page_first_direct`, `page_first_kv_split`, `page_head`</td>
    </tr>

    <tr>
      <td>`--hicache-storage-backend`</td>
      <td>The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend\_name (custom name), module\_path (Python module path), class\_name (backend class name).</td>
      <td>`None`</td>
      <td>`file`, `mooncake`, `hf3fs`, `nixl`, `aibrix`, `dynamic`, `eic`, `simm`</td>
    </tr>

    <tr>
      <td>`--hicache-storage-prefetch-policy`</td>
      <td>Control when prefetching from the storage backend should stop.</td>
      <td>`timeout`</td>
      <td>`best_effort`, `wait_complete`, `timeout`</td>
    </tr>

    <tr>
      <td>`--hicache-storage-backend-extra-config`</td>
      <td>A dictionary in JSON string format, or a string starting with a `@` followed by a config file in JSON/YAML/TOML format, containing extra configuration for the storage backend.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## Hierarchical sparse attention

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--hisparse-config`\
      `--hierarchical-sparse-attention-extra-config`</td>
      <td>A dictionary in JSON string format for hierarchical sparse attention configuration. Required fields: `algorithm` (str), `backend` (str). All other fields are algorithm-specific and passed to the algorithm constructor.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-hisparse`</td>
      <td>Enable hierarchical sparse attention</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## LMCache

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-lmcache`</td>
      <td>Using LMCache as an alternative hierarchical cache solution</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--lmcache-config-file`</td>
      <td>Path to the LMCache YAML configuration file</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## Paged experts

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-paged-experts`</td>
      <td>Keep only `--paged-experts-num-resident` routed experts per MoE layer on the GPU and page the rest from pinned host memory on demand. Unquantized, block-FP8 or GPTQ experts, single GPU. Decode CUDA graphs (full backend) are captured up to K // top\_k requests; prefill graphs must be breakable or disabled.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--paged-experts-num-resident`</td>
      <td>Number of routed experts per MoE layer kept resident on the GPU. Required with `--enable-paged-experts`; at least the router's top-k.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Ktransformers

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--kt-weight-path`</td>
      <td>\[ktransformers parameter] The path of the quantized expert weights for amx kernel. A local folder.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--kt-method`</td>
      <td>\[ktransformers parameter] Quantization formats for CPU execution.</td>
      <td>`AMXINT4`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--kt-cpuinfer`</td>
      <td>\[ktransformers parameter] The number of CPUInfer threads.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--kt-threadpool-count`</td>
      <td>\[ktransformers parameter] One-to-one with the number of NUMA nodes (one thread pool per NUMA).</td>
      <td>`2`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--kt-num-gpu-experts`</td>
      <td>\[ktransformers parameter] The number of GPU experts.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--kt-max-deferred-experts-per-token`</td>
      <td>\[ktransformers parameter] Maximum number of experts deferred to CPU per token. All MoE layers except the final one use this value; the final layer always uses 0.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Diffusion LLM

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dllm-algorithm`</td>
      <td>The diffusion LLM algorithm, such as LowConfidence.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--dllm-algorithm-config`</td>
      <td>The diffusion LLM algorithm configurations. Must be a YAML file.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--dllm-fdfo`</td>
      <td>First-Done-First-Out (FDFO) scheduling lets completed requests leave the batch immediately instead of waiting for slower requests, eliminating head-of-line blocking. Enabled by default; pass `--no-dllm-fdfo` to fall back to synchronous lockstep scheduling. Works with any dLLM algorithm.</td>
      <td>`True`</td>
      <td>Type: bool</td>
    </tr>
  </tbody>
</table>

## Offloading

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--cpu-offload-gb`</td>
      <td>How many GBs of RAM to reserve for CPU offloading.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--offload-group-size`</td>
      <td>Number of layers per group in offloading.</td>
      <td>`-1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--ple-offload-embedding`</td>
      <td>Keep the Qwen4-Exp PLE n-gram embedding table in host memory instead of device memory; the gather kernel reads it from a host pointer. Enabled by default for BF16 Qwen4-Exp on CUDA; `--no-ple-offload-embedding` disables it.</td>
      <td>`None`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`--ple-offload-backend`</td>
      <td>Host storage for the offloaded PLE table: `pinned` (CPU pinned memory) or `file` (a sparse file-backed mmap under `--ple-offload-dir`, read directly by the gather kernel). Use `file` on unified-memory devices such as GB10 / DGX Spark, where pinned host memory comes out of the same pool as the weights; it requires a device that reports `cudaDevAttrPageableMemoryAccessUsesHostPageTables`.</td>
      <td>`pinned`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--ple-offload-dir`</td>
      <td>Directory for the file-backed PLE table when `--ple-offload-backend` is `file`. The file is sparse and rewritten on every startup; keep it on fast local storage (NVMe). Rewriting an existing table can be much slower than filling a fresh sparse file. Before restarting, stop all servers using that directory and remove its `ple_table_*.bin` files to avoid this startup cost.</td>
      <td>`$SGLANG_CACHE_DIR/ple/<model path>`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--offload-num-in-group`</td>
      <td>Number of layers to be offloaded within a group.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--offload-prefetch-step`</td>
      <td>Steps to prefetch in offloading.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--offload-mode`</td>
      <td>Mode of offloading.</td>
      <td>`cpu`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## Args for multi-item scoring

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-mis`</td>
      <td>Enable Multi-Item Scoring optimization. Combines query and multiple items into a single sequence for efficient batch processing. Requires --attention-backend flashinfer; auto-disables CUDA graph, radix cache, and chunked prefill.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## Optimization/debug options

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--disable-radix-cache`</td>
      <td>Disable RadixAttention for prefix caching.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--cuda-graph-config`</td>
      <td>Canonical per-phase CUDA graph settings as JSON, e.g. `{"decode":{"backend":"full","max_bs":256},"prefill":{"backend":"breakable"}}`. JSON wins over the per-phase `--cuda-graph-*` convenience flags and over the legacy flags. Allowed backends: `full`, `breakable`, `disabled`.</td>
      <td>`None`</td>
      <td>Type: JSON (dict-of-dicts)</td>
    </tr>

    <tr>
      <td>`--cuda-graph-backend-decode`</td>
      <td>Backend for the decode phase. Folds into `cuda_graph_config[decode].backend`.</td>
      <td>`None`</td>
      <td>`full`, `breakable`, `disabled`</td>
    </tr>

    <tr>
      <td>`--cuda-graph-backend-prefill`</td>
      <td>Backend for the prefill phase. Folds into `cuda_graph_config[prefill].backend`.</td>
      <td>`None`</td>
      <td>`breakable`, `disabled`</td>
    </tr>

    <tr>
      <td>`--cuda-graph-max-bs-decode`</td>
      <td>Maximum batch size captured for the decode CUDA graph.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--cuda-graph-max-bs-prefill`</td>
      <td>Maximum batch size captured for the prefill CUDA graph.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--cuda-graph-max-seq-len-prefill`</td>
      <td>Longest sequence a prefill CUDA graph replay admits; longer batches run eager prefill. Folds into `cuda_graph_config[prefill].max_seq_len`.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--cuda-graph-bs-decode`</td>
      <td>Explicit list of batch sizes to capture for the decode CUDA graph.</td>
      <td>`None`</td>
      <td>List\[int]</td>
    </tr>

    <tr>
      <td>`--cuda-graph-bs-prefill`</td>
      <td>Explicit list of batch sizes to capture for the prefill CUDA graph.</td>
      <td>`None`</td>
      <td>List\[int]</td>
    </tr>

    <tr>
      <td>`--disable-cuda-graph-padding`</td>
      <td>Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-profile-cuda-graph`</td>
      <td>Enable profiling of cuda graph capture.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--debug-cuda-graph`</td>
      <td>Eager-mode CUDA graph via the breakable backend: graph breaks let every op run eagerly while still going through the capture/replay path. Useful for debugging capture/replay issues.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-cudagraph-gc`</td>
      <td>Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-layerwise-nvtx-marker`</td>
      <td>Enable layerwise NVTX profiling annotations for the model. This adds NVTX markers to every layer for detailed per-layer performance analysis with Nsight Systems.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-nccl-nvls`</td>
      <td>Enable NCCL NVLS for prefill heavy requests when available.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-symm-mem`</td>
      <td>Enable NCCL symmetric memory for fast collectives.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-flashinfer-cutlass-moe-fp4-allgather`</td>
      <td>Disables quantize before all-gather for flashinfer cutlass moe.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-tokenizer-batch-encode`</td>
      <td>Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input\_ids, or input\_embeds.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-tokenizer-batch-decode`</td>
      <td>Disable batch decoding when decoding multiple completions.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-outlines-disk-cache`</td>
      <td>Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--boundary-reduction`</td>
      <td>Select the optional FFN output reduction strategy. Unsupported reduce-scatter paths fall back to all-reduce followed by token redistribution. See [FFN boundary reduction](#ffn-boundary-reduction).</td>
      <td>`auto`</td>
      <td>`auto`, `ar`, `rs`, `rsv`, `rs+rsv`</td>
    </tr>

    <tr>
      <td>`--disable-custom-all-reduce`</td>
      <td>Disable the custom all-reduce kernel and fall back to NCCL.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-mscclpp`</td>
      <td>Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-torch-symm-mem`</td>
      <td>Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM10 supports world size 6, 8.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-overlap-schedule`</td>
      <td>Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-mixed-chunk`</td>
      <td>Enabling mixing prefill and decode in a batch when using chunked prefill.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-dp-attention`</td>
      <td>**Deprecated.** Use `--attn-dp-size <dp-size>` instead of `--dp-size <dp-size> --enable-dp-attention`.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-dp-lm-head`</td>
      <td>Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-two-batch-overlap`</td>
      <td>Enabling two micro batches to overlap.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-single-batch-overlap`</td>
      <td>Let computation and communication overlap within one micro batch.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--tbo-token-distribution-threshold`</td>
      <td>The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap.</td>
      <td>`0.48`</td>
      <td>Type: float</td>
    </tr>

    <tr>
      <td>`--enable-torch-compile`</td>
      <td>Optimize the model with torch.compile. Experimental feature.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-torch-compile-debug-mode`</td>
      <td>Enable debug mode for torch compile.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--torch-compile-max-bs`</td>
      <td>Set the maximum batch size when using torch compile.</td>
      <td>`32`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--disable-cuda-graph`</td>
      <td>**Deprecated.** Use `--cuda-graph-backend-decode=disabled` and/or `--cuda-graph-backend-prefill=disabled`.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-prefill-cuda-graph`</td>
      <td>Disable the prefill-phase CUDA graph. Convenience for `--cuda-graph-backend-prefill=disabled`.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-decode-cuda-graph`</td>
      <td>Disable the decode-phase CUDA graph. Convenience for `--cuda-graph-backend-decode=disabled`.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-p2p-check`</td>
      <td>Enable P2P check for GPU access, otherwise the p2p access is allowed by default.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--triton-attention-reduce-in-fp32`</td>
      <td>Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16. This only affects Triton attention kernels.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--triton-attention-num-kv-splits`</td>
      <td>The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8.</td>
      <td>`8`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--triton-attention-split-tile-size`</td>
      <td>The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--num-continuous-decode-steps`</td>
      <td>Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--delete-ckpt-after-loading`</td>
      <td>Delete the model checkpoint after loading the model.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-memory-saver`</td>
      <td>Allow saving memory using release\_memory\_occupation and resume\_memory\_occupation</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-weights-cpu-backup`</td>
      <td>Save model weights to CPU memory during release\_weights\_occupation and resume\_weights\_occupation</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-draft-weights-cpu-backup`</td>
      <td>Save draft model weights to CPU memory during release\_weights\_occupation and resume\_weights\_occupation</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--allow-auto-truncate`</td>
      <td>Allow automatically truncating requests that exceed the maximum input length instead of returning an error.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-custom-logit-processor`</td>
      <td>Enable users to pass custom logit processors to the server (disabled by default for security)</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-watermark`</td>
      <td>Enable Aaronson-Gumbel text watermarking.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--watermark-config`</td>
      <td>Watermark settings as a JSON file path or an inline JSON object with optional key, key\_b, context\_window, mixing\_probability, max\_probability, default\_enabled, and enforce\_all fields.</td>
      <td>`None`</td>
      <td>Type: Optional\[str]</td>
    </tr>

    <tr>
      <td>`--flashinfer-mla-disable-ragged`</td>
      <td>Not using ragged prefill wrapper when running flashinfer mla</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-shared-experts-fusion`</td>
      <td>Disable shared experts fusion optimization for deepseek v3/r1.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-chunked-prefix-cache`</td>
      <td>Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--image-processor-backend`</td>
      <td>Image processor backend. `auto` lets Transformers select the best available backend.</td>
      <td>`auto`</td>
      <td>`auto`, `torchvision`, `pil`</td>
    </tr>

    <tr>
      <td>`--disable-fast-image-processor`</td>
      <td>Deprecated. Use `--image-processor-backend=pil` instead.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-return-hidden-states`</td>
      <td>Enable returning hidden states with responses.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--return-hidden-states-mode`</td>
      <td>Set the maximum hidden-state return mode supported by the server. `last` allows requests with return\_hidden\_states=False or `last`; `full` also allows return\_hidden\_states=True.</td>
      <td>`None`</td>
      <td>`last`, `full`</td>
    </tr>

    <tr>
      <td>`--enable-return-routed-experts`</td>
      <td>Enable returning routed experts of each layer with responses.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--scheduler-recv-interval`</td>
      <td>The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--numa-node`</td>
      <td>Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess.</td>
      <td>`None`</td>
      <td>List\[int]</td>
    </tr>

    <tr>
      <td>`--enable-deterministic-inference`</td>
      <td>Enable deterministic inference mode with batch invariant ops.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--rl-on-policy-target`</td>
      <td>The training system that SGLang needs to match for true on-policy.</td>
      <td>`None`</td>
      <td>`fsdp`</td>
    </tr>

    <tr>
      <td>`--enable-attn-tp-input-scattered`</td>
      <td>Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-layernorm-sp`</td>
      <td>Enable sequence parallelism for prefill LayerNorm and residual regions under pure tensor parallelism. Supports Qwen3 dense and Muse Spark. Muse Spark requires TP4/EP4 or TP8/EP8, NVLink/NVSwitch, and `--moe-runner-backend flashinfer_trtllm`. Use ModelOpt NVFP4 with `--moe-a2a-backend flashinfer`, or enable MegaMoE for NVFP4 or BF16. Attention DP and pipeline parallelism are unsupported; speculative decoding is supported only for Muse Spark DFLASH.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-megamoe-for-layernorm-sp`</td>
      <td>Use MegaMoE for Muse Spark LayerNorm SP prefill, including BF16 excluded layers. Requires `--enable-layernorm-sp` and DeepGEMM with post-expert normalization (release/v0.2.3). Prefill and decode share TRT-LLM expert weights; decode uses FlashInfer TRT-LLM and all-reduce. FlashInfer A2A is optional. Mixed chunking is unsupported.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-prefill-cp`</td>
      <td>Enable context parallelism for the prefill phase. Select the layout with `--cp-strategy`.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--cp-strategy`</td>
      <td>Sharding strategy for prefill CP. `zigzag` assigns each rank one early and one late sequence block; `interleave` assigns token indices modulo the CP size.</td>
      <td>`None`</td>
      <td>`zigzag`, `interleave`</td>
    </tr>

    <tr>
      <td>`--enable-kv-cache-sharding`</td>
      <td>Shard at-rest KV cache storage at page granularity across the group that currently replicates it (attention-CP ranks for GQA models under prefill CP; attention-TP ranks for MLA models), multiplying the group's unique KV capacity. Prefill workers of PD disaggregation only. With NCCL 2.31.2 or newer and supported NVIDIA CUDA hardware, eligible single-node shard groups request zero-SM copy-engine prefix all-gather on a dedicated NCCL communicator. Older NCCL versions keep the SM-kernel all-gather. This path needs cuMem-backed NCCL memory, so on single-node deployments the server sets `NCCL_CUMEM_ENABLE=1` for every NCCL communicator of all server processes unless `NCCL_CUMEM_ENABLE` is already set; it does not under `--enable-memory-saver` or `SGLANG_MOONCAKE_CUSTOM_MEM_POOL`, which keep the SM-kernel all-gather. Multi-node deployments keep NCCL's setting (MNNVL fabrics default it to 1); export `NCCL_CUMEM_ENABLE=1` there to use this path for shard groups within one node. Set `SGLANG_ENABLE_KV_SHARD_ZERO_SM_GATHER=0` to disable this path. Leave `NCCL_CTA_POLICY` unset: a global value overrides both the gather communicator's zero-SM policy and the default policy of forward collectives.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-fused-qk-norm-rope`</td>
      <td>Fuse q/k normalization and rope rotary embedding into one kernel. Enabled by default on XPU and disabled elsewhere; `--no-enable-fused-qk-norm-rope` disables it.</td>
      <td>`None`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`--enable-precise-embedding-interpolation`</td>
      <td>Enable corner alignment for resize of embeddings grid to ensure more accurate(but slower) evaluation of interpolated embedding values.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--kv-canary`</td>
      <td>KV cache canary mode. 'none' disables the canary (default). 'log' prints them while the server keeps running (production-safe). 'raise' fails the server on the first detected mismatch (CI lane).</td>
      <td>`none`</td>
      <td>`none`, `log`, `raise`</td>
    </tr>

    <tr>
      <td>`--kv-canary-real-data`</td>
      <td>Check the real KV-cache in the canary. 'none' (default) disables the feature. 'partial' checks the first 16 bytes of each real-KV slot. 'all' checks the full real-KV slot.</td>
      <td>`none`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--kv-canary-sweep-interval`</td>
      <td>Every N forward steps, run a full-pool sweep.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--pre-warm-nccl`</td>
      <td>Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL).</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-dp-attention-local-control-broadcast`</td>
      <td>With DP-attention, send control messages to every DP group leader and broadcast within attn\_tp\_group instead of the full tp\_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enforce-shared-experts-fusion`</td>
      <td>Enforce shared experts fusion even when it would normally be disabled (e.g. under DeepEP). Mutually exclusive with --disable-shared-experts-fusion.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-return-indexer-topk`</td>
      <td>Enable returning indexer topk indices of layers with indexer with responses.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--disable-attn-tp-gather`</td>
      <td>Disable scheduler-side attn\_tp\_gather (the upstream SP path that pads num\_tokens to attn\_tp\_size and pre-allocates a gathered buffer). Use for models that manage SP scatter/gather at the model level (e.g., perform their own all\_gather/reduce\_scatter inside attention) and do not consume the upstream gathered\_buffer. Without this, the cuda graph runner pads num\_tokens to attn\_tp\_size, which can cause kernel autotuners to select wrong-sized variants at small batches.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--enable-fused-moe-sum-all-reduce`</td>
      <td>Enable fused moe triton and sum all reduce.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--gc-threshold`</td>
      <td>Set the garbage collection thresholds (the collection frequency). Accepts 1 to 3 integers.</td>
      <td>—</td>
      <td>Type: int (one or more)</td>
    </tr>
  </tbody>
</table>

### FFN boundary reduction

Use `--boundary-reduction` to choose which optional FFN output reductions may combine the sum with token redistribution. The selected policy applies to the dense and MoE FFN stages of decoders built with independently constructed stage boundaries (`declare_ffn` / `append_stages`; see [Layer boundaries](/docs/developer_guide/layer_boundary)). Other models keep their built-in FFN output communication and ignore this option.

| Value | Behavior |
| - | - |
| `auto` | Resolve the model default during server-argument processing, before model construction. |
| `ar` | Use all-reduce followed by token redistribution where needed. |
| `rs` | Permit fixed-size reduce-scatter when the layout and backend support it; otherwise use `ar`. |
| `rsv` | Permit variable-size reduce-scatter for eligible attention-DP paths; otherwise use `ar`. |
| `rs+rsv` | Permit both, trying the eligible variable-size path first, then fixed-size, then `ar`. |

`auto` selects `ar` for dense Qwen3 language backbones (including Qwen3-VL wrappers), Moss-VL and MiniCPM-V 4.5; `rsv` for BailingMoELinear/BailingMoeV2\_5, LongCat Flash and Step3-VL; and `rs+rsv` otherwise. Qwen3-MoE, Qwen3-Next and Qwen3.5 are not matched by the dense Qwen3 rule and resolve to `rs+rsv`.

With `auto`, a draft loaded from a different path resolves its own model default; a draft on the target's model path (the MTP default) inherits the target policy. An explicit value applies to both target and draft. The result is recorded in the `speculative_boundary_reduction` server args field (`no_cli=True`); it has no separate CLI flag or independent override during resolution.

This option does not disable required attention or MoE collectives, including expert all-to-all, or select the all-reduce fusion backend. For example, `ar` can still use fused all-reduce plus residual normalization; use `--enforce-disable-flashinfer-allreduce-fusion` to disable that FlashInfer fusion.

## Dynamic batch tokenizer

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-dynamic-batch-tokenizer`</td>
      <td>Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--dynamic-batch-tokenizer-batch-size`</td>
      <td>\[Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer.</td>
      <td>`32`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--dynamic-batch-tokenizer-batch-timeout`</td>
      <td>\[Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests.</td>
      <td>`0.002`</td>
      <td>Type: float</td>
    </tr>
  </tbody>
</table>

## Debug tensor dumps

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--debug-tensor-dump-output-folder`</td>
      <td>The output folder for dumping tensors.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--debug-tensor-dump-layers`</td>
      <td>The layer ids to dump. Dump all layers if not specified.</td>
      <td>`None`</td>
      <td>Type: JSON list</td>
    </tr>

    <tr>
      <td>`--debug-tensor-dump-input-file`</td>
      <td>The input filename for dumping tensors</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## PD disaggregation

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--disaggregation-mode`</td>
      <td>Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated</td>
      <td>`null`</td>
      <td>`null`, `prefill`, `decode`</td>
    </tr>

    <tr>
      <td>`--disaggregation-transfer-backend`</td>
      <td>The backend for disaggregation transfer. Default is mooncake.</td>
      <td>`mooncake`</td>
      <td>`mooncake`, `nixl`, `ascend`, `fake`, `mori`, `mooncake_tcp`</td>
    </tr>

    <tr>
      <td>`--disaggregation-bootstrap-port`</td>
      <td>Bootstrap server port on the prefill server. Default is 8998.</td>
      <td>`8998`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--disaggregation-ib-device`</td>
      <td>The InfiniBand devices for disaggregation transfer, accepts single device (e.g., --disaggregation-ib-device mlx5\_0) or multiple comma-separated devices (e.g., --disaggregation-ib-device mlx5\_0,mlx5\_1). Default is None, which triggers automatic device detection when mooncake backend is enabled.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--disaggregation-decode-allocation-policy`</td>
      <td>When to reserve per-request decode KV slots. Set the same policy on prefill and decode. See [requirements and tradeoffs](/docs/advanced_features/pd_disaggregation#decode-allocation-policy).</td>
      <td>`early`</td>
      <td>`early`, `prefill_complete`</td>
    </tr>

    <tr>
      <td>`--disaggregation-decode-enable-offload-kvcache`</td>
      <td>Enable async KV cache offloading on decode server (PD mode).</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--num-reserved-decode-tokens`</td>
      <td>Number of decode tokens that will have memory reserved when adding new request to the running batch.</td>
      <td>`512`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--disaggregation-decode-polling-interval`</td>
      <td>The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this.</td>
      <td>`1`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--disaggregation-decode-enable-radix-cache`</td>
      <td>Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with --enable-hisparse, --disaggregation-transfer-backend fake, and speculative algorithms other than EAGLE, EAGLE3, and DSPARK.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--optimistic-prefill-attempts`</td>
      <td>Number of optimistic prefill forward passes that skip the bootstrap wait.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Encode prefill disaggregation

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--encoder-only`</td>
      <td>For MLLM with an encoder, launch an encoder-only server</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--language-only`</td>
      <td>For VLM, load weights for the language model only.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--language-model-only`</td>
      <td>Skip the multimodal encoder entirely: its weights are never loaded and the tower is never built, freeing that GPU memory for KV cache. Multimodal requests are rejected. Unlike --language-only this is a standalone mode, not part of encoder/decoder disaggregation.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--encoder-transfer-backend`</td>
      <td>The backend for encoder disaggregation transfer. Default is zmq\_to\_scheduler.</td>
      <td>`zmq_to_scheduler`</td>
      <td>`zmq_to_scheduler`, `zmq_to_tokenizer`, `mooncake`</td>
    </tr>

    <tr>
      <td>`--encoder-urls`</td>
      <td>List of encoder server urls.</td>
      <td>`[]`</td>
      <td>Type: JSON list</td>
    </tr>

    <tr>
      <td>`--encoder-bootstrap-port`</td>
      <td>Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here.</td>
      <td>`8997`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--encoder-register-urls`</td>
      <td>One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: --encoder-register-urls [http://prefill0:8997](http://prefill0:8997) [http://prefill1:8997](http://prefill1:8997). Used with --encoder-only servers.</td>
      <td>`[]`</td>
      <td>Type: str (one or more)</td>
    </tr>

    <tr>
      <td>`--enable-adaptive-dispatch-to-encoder`</td>
      <td>When enabled, adaptively dispatch: multi-image requests go to encoder in language\_only epd mode, single-image requests are processed locally.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## Custom weight loader

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--custom-weight-loader`</td>
      <td>The custom dataloader which used to update the model. Should be set with a valid import path, such as my\_package.weight\_load\_func</td>
      <td>`None`</td>
      <td>List\[str]</td>
    </tr>

    <tr>
      <td>`--weight-loader-disable-mmap`</td>
      <td>Disable mmap while loading weight using safetensors.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--weight-loader-prefetch-checkpoints`</td>
      <td>Prefetch checkpoint files into OS page cache before loading. Each rank prefetches a fraction of the shards in a background thread, reducing total network I/O on shared filesystems (NFS/Lustre) from N\*checkpoint to 1\*checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set `enable_multithread_load=true` in `--model-loader-extra-config` to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op).</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--weight-loader-prefetch-num-threads`</td>
      <td>Number of threads per rank for checkpoint prefetching.</td>
      <td>`4`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-seed-instance-ip`</td>
      <td>The ip of the seed instance for loading weights from remote instance.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-seed-instance-service-port`</td>
      <td>The service port of the seed instance for loading weights from remote instance.</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-send-weights-group-ports`</td>
      <td>The communication group ports for loading weights from remote instance.</td>
      <td>`None`</td>
      <td>Type: JSON list</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-backend`</td>
      <td>The backend for loading weights from remote instance. Can be 'transfer\_engine', 'nccl', or 'modelexpress'. Default is 'nccl'.</td>
      <td>`nccl`</td>
      <td>`transfer_engine`, `nccl`, `modelexpress`</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-start-seed-via-transfer-engine`</td>
      <td>Start seed server via transfer engine backend for remote instance weight loader.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--weight-loader-drop-cache-after-load`</td>
      <td>Call posix\_fadvise(DONTNEED) on each safetensors shard after loading it.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--engine-info-bootstrap-port`</td>
      <td>Port for the engine info bootstrap server. Default is 6789. Must be set explicitly when running multiple instances on the same node.</td>
      <td>`6789`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--modelexpress-config`</td>
      <td>JSON config for ModelExpress P2P weight loading. Keys: "url" (optional gRPC host:port override), "transport" ("nixl" or "transfer\_engine"). Example: '\{"url": "localhost:8001", "transport": "nixl"}'</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## For PD-Multiplexing

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-pdmux`</td>
      <td>Enable PD-Multiplexing, PD running on greenctx stream.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--pdmux-config-path`</td>
      <td>The path of the PD-Multiplexing config file.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--sm-group-num`</td>
      <td>Number of sm partition groups.</td>
      <td>`8`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

## Configuration file support

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--config`</td>
      <td>Read CLI options from a config file. Must be a YAML file with configuration options.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

## For Multi-Modal

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-broadcast-mm-inputs-process`</td>
      <td>Enable broadcast mm-inputs process in scheduler.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--mm-process-config`</td>
      <td>Multimodal preprocessing config, a json config contains keys: `image`, `video`, `audio`.</td>
      <td>`{}`</td>
      <td>Type: JSON / Dict</td>
    </tr>

    <tr>
      <td>`--allowed-media-domains`</td>
      <td>Restrict client-supplied HTTP(S) media URLs and redirect destinations to these exact hostnames.</td>
      <td>Unrestricted</td>
      <td>Space-separated hostnames</td>
    </tr>

    <tr>
      <td>`--media-url-max-file-size-mb`</td>
      <td>Maximum streamed size in MiB for one remote media download. Set to 0 to disable the limit.</td>
      <td>`64`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--mm-enable-dp-encoder`</td>
      <td>Enabling data parallelism for mm encoder. The dp size will be set to the tp size automatically.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--limit-mm-data-per-request`</td>
      <td>Limit the number of multimodal inputs per request. e.g. '\{"image": 1, "video": 1, "audio": 1}'</td>
      <td>`None`</td>
      <td>Type: JSON / Dict</td>
    </tr>

    <tr>
      <td>`--enable-mm-global-cache`</td>
      <td>Enable Mooncake-backed global multimodal embedding cache on encoder servers so repeated images can reuse cached ViT embeddings instead of recomputing them.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## For checkpoint decryption

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--decrypted-config-file`</td>
      <td>The path of the decrypted config file.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--decrypted-draft-config-file`</td>
      <td>The path of the decrypted draft config file.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--enable-prefix-mm-cache`</td>
      <td>Enable prefix multimodal cache. Currently only supports mm-only.</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>
  </tbody>
</table>

## Forward hooks

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--forward-hooks`</td>
      <td>JSON-formatted list of forward hook specifications. Each element must include `target_modules` (list of glob patterns matched against `model.named_modules()` names) and `hook_factory` (Python import path to a factory, e.g. `my_package.hooks:make_hook`). An optional `name` field is used for logging, and an optional `config` object is passed as a `dict` to the factory.</td>
      <td>`None`</td>
      <td>Type: JSON list</td>
    </tr>
  </tbody>
</table>

## For MindStudio-probe(msProbe) dump

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--msprobe-dump-config`</td>
      <td>The path of the JSON configuration file for msProbe. If specified, enables msProbe dump.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>


# Session-Aware Radix Cache
Source: https://docs.sglang.io/docs/advanced_features/session_radix_cache



Session-aware radix caching improves cache hits for long-lived, multi-turn workloads under memory pressure. It registers reusable KV to a session and evicts unreferenced KV before KV still referenced by an active session.

Session references are soft protection, not memory pins. Referenced KV can still be evicted when reclaiming unreferenced KV is insufficient.

## Enable the cache

This feature is implemented only by `UnifiedRadixCache`.

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path MODEL_PATH \
  --enable-session-radix-cache
```

## Pass and close the session

Your application must pass the same top-level `session_id` on every request in a session. The ID labels cache references only; it does not append or reconstruct conversation context, so each request must still contain the intended prompt.

```bash Command theme={null}
curl http://localhost:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "FULL_PROMPT_FOR_THIS_TURN",
    "sampling_params": {"max_new_tokens": 128},
    "session_id": "agent-42"
  }'
```

When a request finishes, SGLang automatically registers its reusable cache leaves under the `session_id`. This reference-only workflow does not require an `/open_session` call.

Call `/close_session` when the application session ends, including error and cancellation paths:

```bash Command theme={null}
curl -X POST http://localhost:30000/close_session \
  -H "Content-Type: application/json" \
  -d '{"session_id": "agent-42"}'
```

Closing removes the session's references but does not immediately free its KV. The KV remains reusable and returns to the normal eviction order.

## Eviction behavior

The cache tracks references independently for each UnifiedRadixCache component. Device and host eviction use the same session preference.

| Component | Referenced data | Eviction order |
| - | - | - |
| Full attention | The reusable prefix path from the registered leaf to the root | Unreferenced nodes first, then referenced nodes with fewer session references, then the configured policy such as LRU |
| Sliding-window attention (SWA) | The reusable tail covering the sliding window plus page-alignment allowance | Two LRU passes: unreferenced nodes first, then referenced nodes if more space is required |
| Mamba | The reusable state on the registered leaf | Two LRU passes: unreferenced nodes first, then referenced nodes if more space is required |

UnifiedRadixCache still applies component cascade rules. Evicting an internal Full node also evicts its SWA and Mamba data; evicting SWA also evicts Mamba data; evicting Mamba affects only Mamba. Evicting a leaf removes all component data on that leaf.


# SGLang Model Gateway
Source: https://docs.sglang.io/docs/advanced_features/sgl_model_gateway



SGLang Model Gateway is a high-performance model-routing gateway for large-scale LLM deployments. It centralizes worker lifecycle management, balances traffic across heterogeneous protocols (HTTP, gRPC, OpenAI-compatible), and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows. The gateway is deeply optimized for the SGLang serving runtime, but can route to any OpenAI-compatible backend.

***

## Table of Contents

1. [Overview](#overview)
2. [Architecture](#architecture)
   * [Control Plane](#control-plane)
   * [Data Plane](#data-plane)
   * [Storage and Privacy](#storage-and-privacy)
3. [Installation](#installation)
4. [Quick Start](#quick-start)
5. [Deployment Modes](#deployment-modes)
   * [Co-launch Router and Workers](#co-launch-router-and-workers)
   * [Separate Launch (HTTP)](#separate-launch-http)
   * [gRPC Launch](#grpc-launch)
   * [Prefill-Decode Disaggregation](#prefill-decode-disaggregation)
   * [OpenAI Backend Proxy](#openai-backend-proxy)
   * [Multi-Model Inference Gateway](#multi-model-inference-gateway)
6. [API Reference](#api-reference)
   * [Inference Endpoints](#inference-endpoints)
   * [Tokenization Endpoints](#tokenization-endpoints)
   * [Parser Endpoints](#parser-endpoints)
   * [Classification API](#classification-api)
   * [Conversation and Response APIs](#conversation-and-response-apis)
   * [Worker Management APIs](#worker-management-apis)
   * [Admin and Health Endpoints](#admin-and-health-endpoints)
7. [Load Balancing Policies](#load-balancing-policies)
8. [Reliability and Flow Control](#reliability-and-flow-control)
   * [Retries](#retries)
   * [Circuit Breaker](#circuit-breaker)
   * [Rate Limiting and Queuing](#rate-limiting-and-queuing)
   * [Health Checks](#health-checks)
9. [Reasoning Parser Integration](#reasoning-parser-integration)
10. [Tool Call Parsing](#tool-call-parsing)
11. [Tokenizer Management](#tokenizer-management)
12. [MCP Integration](#mcp-integration)
13. [Service Discovery (Kubernetes)](#service-discovery-kubernetes)
14. [History and Data Connectors](#history-and-data-connectors)
15. [WASM Middleware](#wasm-middleware)
16. [Language Bindings](#language-bindings)
17. [Security and Authentication](#security-and-authentication)
    * [TLS (HTTPS) for Gateway Server](#tls-https-for-gateway-server)
    * [mTLS for Worker Communication](#mtls-for-worker-communication)
18. [Observability](#observability)
    * [Prometheus Metrics](#prometheus-metrics)
    * [OpenTelemetry Tracing](#opentelemetry-tracing)
    * [Logging](#logging)
19. [Production Recommendations](#production-recommendations)
    * [Security Best Practices](#security-best-practices)
    * [High Availability](#high-availability)
    * [Performance](#performance)
    * [Kubernetes Deployment](#kubernetes-deployment)
    * [Monitoring with PromQL](#monitoring-with-promql)
20. [Configuration Reference](#configuration-reference)
21. [Troubleshooting](#troubleshooting)

***

## Overview

* **Unified control plane** for registering, monitoring, and orchestrating regular, prefill, and decode workers across heterogeneous model fleets.
* **Multi-protocol data plane** that routes traffic across HTTP, PD (prefill/decode), gRPC, and OpenAI-compatible backends with shared reliability primitives.
* **Industry-first gRPC pipeline** with native Rust tokenization, reasoning parsers, and tool-call execution for high-throughput, OpenAI-compatible serving; supports both single-stage and PD topologies.
* **Inference Gateway Mode (`--enable-igw`)** dynamically instantiates multiple router stacks (HTTP regular/PD, gRPC) and applies per-model policies for multi-tenant deployments.
* **Conversation & responses connectors** centralize chat history inside the router so the same context can be reused across models and MCP loops without leaking data to upstream vendors (memory, none, Oracle ATP, PostgreSQL).
* **Enterprise privacy**: agentic multi-turn `/v1/responses`, native MCP client (STDIO/HTTP/SSE/Streamable), and history storage all operate within the router boundary.
* **Reliability core**: retries with jitter, worker-scoped circuit breakers, token-bucket rate limiting with queuing, background health checks, and cache-aware load monitoring.
* **Comprehensive observability**: 40+ Prometheus metrics, OpenTelemetry distributed tracing, structured logging, and request ID propagation.

***

## Architecture

### Control Plane

* **Worker Manager** discovers capabilities (`/get_server_info`, `/get_model_info`), tracks load, and registers/removes workers in the shared registry.
* **Job Queue** serializes add/remove requests and exposes status (`/workers/{worker_id}`) so clients can track onboarding progress.
* **Load Monitor** feeds cache-aware and power-of-two policies with live worker load statistics.
* **Health Checker** continuously probes workers and updates readiness, circuit breaker state, and router metrics.
* **Tokenizer Registry** manages dynamically registered tokenizers with async loading from HuggingFace or local paths.

### Data Plane

* **HTTP routers** (regular & PD) implement `/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/embeddings`, `/v1/rerank`, `/v1/classify`, `/v1/tokenize`, `/v1/detokenize`, and associated admin endpoints.
* **gRPC router** streams tokenized requests directly to SRT gRPC workers, running fully in Rust—tokenizer, reasoning parser, and tool parser all reside in-process. Supports both single-stage and PD routing, including embeddings and classification.
* **OpenAI router** proxies OpenAI-compatible endpoints to external vendors (OpenAI, xAI, etc.) while keeping chat history and multi-turn orchestration local.

### Storage and Privacy

* Conversation and response history is stored at the router tier (memory, none, Oracle ATP, or PostgreSQL). The same history can power multiple models or MCP loops without sending data to upstream vendors.
* `/v1/responses` agentic flows, MCP sessions, and conversation APIs share the same storage layer, enabling compliance for regulated workloads.

***

## Installation

### Docker

Pre-built Docker images are available on Docker Hub with multi-architecture support (x86\_64 and ARM64):

```bash Command theme={null}
docker pull lmsysorg/sgl-model-gateway:latest
```

### Prerequisites

* **Rust and Cargo**
  ```bash Command theme={null}
  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
  source "$HOME/.cargo/env"
  rustc --version
  cargo --version
  ```
* **Python** with `pip` and virtualenv tooling available.

### Rust Binary

```bash Command theme={null}
cd sgl-model-gateway
cargo build --release
```

### Python Package

```bash Command theme={null}
pip install maturin

# Fast development mode
cd sgl-model-gateway/bindings/python
maturin develop

# Production build
maturin build --release --out dist --features vendored-openssl
pip install --force-reinstall dist/*.whl
```

***

## Quick Start

### Regular HTTP Routing

```bash Command theme={null}
# Rust binary
./target/release/sgl-model-gateway \
  --worker-urls http://worker1:8000 http://worker2:8000 \
  --policy cache_aware

# Python launcher
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 http://worker2:8000 \
  --policy cache_aware
```

### gRPC Routing

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls grpc://127.0.0.1:20000 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --reasoning-parser deepseek-r1 \
  --tool-call-parser json \
  --host 0.0.0.0 --port 8080
```

***

## Deployment Modes

### Co-launch Router and Workers

Launch the router and a fleet of SGLang workers in one process:

```bash Command theme={null}
python -m sglang_router.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --dp-size 4 \
  --host 0.0.0.0 \
  --port 30000
```

Comprehensive example with router arguments (prefixed with `--router-`):

```bash Command theme={null}
python -m sglang_router.launch_server \
  --host 0.0.0.0 \
  --port 8080 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tp-size 1 \
  --dp-size 8 \
  --grpc-mode \
  --log-level debug \
  --router-prometheus-port 10001 \
  --router-tool-call-parser llama \
  --router-model-path meta-llama/Llama-3.1-8B-Instruct \
  --router-policy round_robin \
  --router-log-level debug
```

### Separate Launch (HTTP)

Run workers independently and point the router at their HTTP endpoints:

```bash Command theme={null}
# Worker nodes
python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8000
python -m sglang.launch_server --model meta-llama/Meta-Llama-3.1-8B-Instruct --port 8001

# Router node
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 http://worker2:8001 \
  --policy cache_aware \
  --host 0.0.0.0 --port 30000
```

### gRPC Launch

Use SRT gRPC workers to unlock the highest throughput and access native reasoning/tool pipelines:

```bash Command theme={null}
# Workers expose gRPC endpoints
python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --grpc-mode \
  --port 20000

# Router
python -m sglang_router.launch_router \
  --worker-urls grpc://127.0.0.1:20000 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --reasoning-parser deepseek-r1 \
  --tool-call-parser json \
  --host 0.0.0.0 --port 8080
```

The gRPC router supports both regular HTTP-equivalent serving and PD (prefill/decode) serving. Provide `--tokenizer-path` or `--model-path` (HuggingFace ID or local directory) whenever connection mode resolves to gRPC.

### Prefill-Decode Disaggregation

Split prefill and decode workers for PD-aware caching and balancing:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --pd-disaggregation \
  --prefill http://prefill1:30001 9001 \
  --decode http://decode1:30011 \
  --prefill-policy cache_aware \
  --decode-policy power_of_two
```

Prefill entries accept an optional bootstrap port. PD mode merges prefill metadata with decode outputs and streams results back to the client.

### OpenAI Backend Proxy

Proxy OpenAI-compatible endpoints while keeping history and MCP sessions local:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --backend openai \
  --worker-urls https://api.openai.com \
  --history-backend memory
```

OpenAI backend mode expects exactly one `--worker-urls` entry per router instance.

### Multi-Model Inference Gateway

Enable IGW mode to route multiple models through a single router:

```bash Command theme={null}
./target/release/sgl-model-gateway \
  --enable-igw \
  --policy cache_aware \
  --max-concurrent-requests 512

# Register workers dynamically
curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{
        "url": "http://worker-a:8000",
        "model_id": "mistral",
        "priority": 10,
        "labels": {"tier": "gold"}
      }'
```

***

## API Reference

### Inference Endpoints

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Path</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`POST`</td>
      <td>`/generate`</td>
      <td>SGLang generate API</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/chat/completions`</td>
      <td>OpenAI-compatible chat completions (streaming/tool calls)</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/completions`</td>
      <td>OpenAI-compatible text completions</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/embeddings`</td>
      <td>Embedding generation (HTTP and gRPC)</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/rerank`, `/rerank`</td>
      <td>Reranking requests</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/classify`</td>
      <td>Text classification</td>
    </tr>
  </tbody>
</table>

### Tokenization Endpoints

The gateway provides HTTP endpoints for text tokenization with batch support, designed to mirror the SGLang Python tokenization API.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Path</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`POST`</td>
      <td>`/v1/tokenize`</td>
      <td>Tokenize text to token IDs (single or batch)</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/detokenize`</td>
      <td>Convert token IDs back to text (single or batch)</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/tokenizers`</td>
      <td>Register a new tokenizer (async, returns job status)</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/tokenizers`</td>
      <td>List all registered tokenizers</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/tokenizers/{id}`</td>
      <td>Get tokenizer info by UUID</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/tokenizers/{id}/status`</td>
      <td>Check async tokenizer loading status</td>
    </tr>

    <tr>
      <td>`DELETE`</td>
      <td>`/v1/tokenizers/{id}`</td>
      <td>Remove a tokenizer from the registry</td>
    </tr>
  </tbody>
</table>

#### Tokenize Request

```json Config theme={null}
{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "prompt": "Hello, world!"
}
```

#### Batch Tokenize Request

```json Config theme={null}
{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "prompt": ["Hello", "World", "How are you?"]
}
```

#### Tokenize Response

```json Config theme={null}
{
  "tokens": [15339, 11, 1917, 0],
  "count": 4,
  "char_count": 13
}
```

#### Detokenize Request

```json Config theme={null}
{
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "tokens": [15339, 11, 1917, 0],
  "skip_special_tokens": true
}
```

#### Detokenize Response

```json Config theme={null}
{
  "text": "Hello, world!"
}
```

#### Add Tokenizer (Async)

```bash Command theme={null}
curl -X POST http://localhost:30000/v1/tokenizers \
  -H "Content-Type: application/json" \
  -d '{"name": "llama3", "source": "meta-llama/Llama-3.1-8B-Instruct"}'
```

Response:

```json Config theme={null}
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "message": "Tokenizer registration queued"
}
```

Check status:

```bash Command theme={null}
curl http://localhost:30000/v1/tokenizers/550e8400-e29b-41d4-a716-446655440000/status
```

### Parser Endpoints

The gateway provides admin endpoints for parsing reasoning content and function calls from LLM outputs.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Path</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`POST`</td>
      <td>`/parse/reasoning`</td>
      <td>Separate reasoning (`&lt;think&gt;`) from normal text</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/parse/function_call`</td>
      <td>Parse function/tool calls from text</td>
    </tr>
  </tbody>
</table>

#### Separate Reasoning Request

```json Config theme={null}
{
  "text": "&lt;think&gt;Let me analyze this step by step...&lt;/think&gt;The answer is 42.",
  "parser": "deepseek-r1"
}
```

#### Response

```json Config theme={null}
{
  "normal_text": "The answer is 42.",
  "reasoning_text": "Let me analyze this step by step..."
}
```

#### Function Call Parsing

```json Config theme={null}
{
  "text": "{\"name\": \"get_weather\", \"arguments\": {\"city\": \"NYC\"}}",
  "parser": "json"
}
```

### Classification API

The `/v1/classify` endpoint provides text classification using sequence classification models (e.g., `Qwen2ForSequenceClassification`, `BertForSequenceClassification`).

#### Request

```bash Command theme={null}
curl http://localhost:30000/v1/classify \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": "I love this product!"
  }'
```

#### Response

```json Config theme={null}
{
  "id": "classify-a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "object": "list",
  "created": 1767034308,
  "model": "jason9693/Qwen2.5-1.5B-apeach",
  "data": [
    {
      "index": 0,
      "label": "positive",
      "probs": [0.12, 0.88],
      "num_classes": 2
    }
  ],
  "usage": {
    "prompt_tokens": 6,
    "completion_tokens": 0,
    "total_tokens": 6
  }
}
```

#### Response Fields

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`label`</td>
      <td>Predicted class label (from model's `id2label` config, or `LABEL_N` fallback)</td>
    </tr>

    <tr>
      <td>`probs`</td>
      <td>Probability distribution over all classes (softmax of logits)</td>
    </tr>

    <tr>
      <td>`num_classes`</td>
      <td>Number of classification classes</td>
    </tr>
  </tbody>
</table>

#### Notes

* Classification reuses the embedding backend—the scheduler returns logits which are converted to probabilities via softmax
* Labels come from the model's HuggingFace config (`id2label` field); models without this mapping use generic labels (`LABEL_0`, `LABEL_1`, etc.)
* Both HTTP and gRPC routers support classification

### Conversation and Response APIs

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Path</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`POST`</td>
      <td>`/v1/responses`</td>
      <td>Create background responses (agentic loops)</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/responses/{id}`</td>
      <td>Retrieve stored response</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/responses/{id}/cancel`</td>
      <td>Cancel background response</td>
    </tr>

    <tr>
      <td>`DELETE`</td>
      <td>`/v1/responses/{id}`</td>
      <td>Delete response</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/responses/{id}/input_items`</td>
      <td>List response input items</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/conversations`</td>
      <td>Create conversation</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/conversations/{id}`</td>
      <td>Get conversation</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/conversations/{id}`</td>
      <td>Update conversation</td>
    </tr>

    <tr>
      <td>`DELETE`</td>
      <td>`/v1/conversations/{id}`</td>
      <td>Delete conversation</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/conversations/{id}/items`</td>
      <td>List conversation items</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/v1/conversations/{id}/items`</td>
      <td>Add items to conversation</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/conversations/{id}/items/{item_id}`</td>
      <td>Get conversation item</td>
    </tr>

    <tr>
      <td>`DELETE`</td>
      <td>`/v1/conversations/{id}/items/{item_id}`</td>
      <td>Delete conversation item</td>
    </tr>
  </tbody>
</table>

### Worker Management APIs

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Path</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`POST`</td>
      <td>`/workers`</td>
      <td>Queue worker registration (returns 202 Accepted)</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/workers`</td>
      <td>List workers with health, load, and policy metadata</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/workers/{worker_id}`</td>
      <td>Inspect specific worker or job queue entry</td>
    </tr>

    <tr>
      <td>`PUT`</td>
      <td>`/workers/{worker_id}`</td>
      <td>Queue worker update</td>
    </tr>

    <tr>
      <td>`DELETE`</td>
      <td>`/workers/{worker_id}`</td>
      <td>Queue worker removal</td>
    </tr>
  </tbody>
</table>

#### Add Worker

```bash Command theme={null}
curl -X POST http://localhost:30000/workers \
  -H "Content-Type: application/json" \
  -d '{"url":"grpc://0.0.0.0:31000","worker_type":"regular"}'
```

#### List Workers

```bash Command theme={null}
curl http://localhost:30000/workers
```

Response:

```json Config theme={null}
{
  "workers": [
    {
      "id": "2f3a0c3e-3a7b-4c3f-8c70-1b7d4c3a6e1f",
      "url": "http://0.0.0.0:31378",
      "model_id": "mistral",
      "priority": 50,
      "cost": 1.0,
      "worker_type": "regular",
      "is_healthy": true,
      "load": 0,
      "connection_mode": "Http"
    }
  ],
  "total": 1,
  "stats": {
    "prefill_count": 0,
    "decode_count": 0,
    "regular_count": 1
  }
}
```

### Admin and Health Endpoints

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Path</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`GET`</td>
      <td>`/liveness`</td>
      <td>Health check (always returns OK)</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/readiness`</td>
      <td>Readiness check (checks healthy worker availability)</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/health`</td>
      <td>Alias for liveness</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/health_generate`</td>
      <td>Health generate test</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/engine_metrics`</td>
      <td>Engine-level metrics from workers</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/v1/models`</td>
      <td>List available models</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/get_model_info`</td>
      <td>Get model information</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/get_server_info`</td>
      <td>Get server information</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/flush_cache`</td>
      <td>Clear all caches</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/get_loads`</td>
      <td>Get all worker loads</td>
    </tr>

    <tr>
      <td>`POST`</td>
      <td>`/wasm`</td>
      <td>Upload WASM module</td>
    </tr>

    <tr>
      <td>`GET`</td>
      <td>`/wasm`</td>
      <td>List WASM modules</td>
    </tr>

    <tr>
      <td>`DELETE`</td>
      <td>`/wasm/{module_uuid}`</td>
      <td>Remove WASM module</td>
    </tr>
  </tbody>
</table>

***

## Load Balancing Policies

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Policy</th>
      <th>Description</th>
      <th>Usage</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`random`</td>
      <td>Uniform random selection</td>
      <td>`--policy random`</td>
    </tr>

    <tr>
      <td>`round_robin`</td>
      <td>Cycles through workers in order</td>
      <td>`--policy round_robin`</td>
    </tr>

    <tr>
      <td>`power_of_two`</td>
      <td>Samples two workers and picks the lighter one</td>
      <td>`--policy power_of_two`</td>
    </tr>

    <tr>
      <td>`cache_aware`</td>
      <td>Combines cache locality with load balancing (default)</td>
      <td>`--policy cache_aware`</td>
    </tr>

    <tr>
      <td>`bucket`</td>
      <td>Divides workers into load buckets with dynamic boundaries</td>
      <td>`--policy bucket`</td>
    </tr>
  </tbody>
</table>

### Cache-Aware Policy Tuning

```bash Command theme={null}
--cache-threshold 0.5 \
--balance-abs-threshold 32 \
--balance-rel-threshold 1.5 \
--eviction-interval-secs 120 \
--max-tree-size 67108864
```

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--cache-threshold`</td>
      <td>0.3</td>
      <td>Minimum prefix match ratio for cache hit</td>
    </tr>

    <tr>
      <td>`--balance-abs-threshold`</td>
      <td>64</td>
      <td>Absolute load difference before rebalancing</td>
    </tr>

    <tr>
      <td>`--balance-rel-threshold`</td>
      <td>1.5</td>
      <td>Relative load ratio before rebalancing</td>
    </tr>

    <tr>
      <td>`--eviction-interval-secs`</td>
      <td>120</td>
      <td>Cache eviction cadence in seconds</td>
    </tr>

    <tr>
      <td>`--max-tree-size`</td>
      <td>67108864</td>
      <td>Maximum nodes in cache tree</td>
    </tr>
  </tbody>
</table>

***

## Reliability and Flow Control

### HTTP Client

Configure upstream HTTP client connection settings:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--pool-idle-timeout-secs`</td>
      <td>50</td>
      <td>Idle timeout in seconds for pooled upstream HTTP connections. Can also be set with `SMG_POOL_IDLE_TIMEOUT_SECS`.</td>
    </tr>

    <tr>
      <td>`--connect-timeout-secs`</td>
      <td>10</td>
      <td>Timeout in seconds for new upstream HTTP connections. Can also be set with `SMG_CONNECT_TIMEOUT_SECS`.</td>
    </tr>

    <tr>
      <td>`--pool-max-idle-per-host`</td>
      <td>500</td>
      <td>Maximum idle upstream HTTP connections to keep per host. Can also be set with `SMG_POOL_MAX_IDLE_PER_HOST`.</td>
    </tr>

    <tr>
      <td>`--tcp-keepalive-secs`</td>
      <td>30</td>
      <td>TCP keepalive idle time in seconds for upstream HTTP connections. Can also be set with `SMG_TCP_KEEPALIVE_SECS`.</td>
    </tr>
  </tbody>
</table>

### Retries

Configure exponential backoff retries:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 http://worker2:8001 \
  --retry-max-retries 5 \
  --retry-initial-backoff-ms 50 \
  --retry-max-backoff-ms 30000 \
  --retry-backoff-multiplier 1.5 \
  --retry-jitter-factor 0.2
```

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--retry-max-retries`</td>
      <td>5</td>
      <td>Maximum retry attempts</td>
    </tr>

    <tr>
      <td>`--retry-initial-backoff-ms`</td>
      <td>50</td>
      <td>Initial backoff duration (ms)</td>
    </tr>

    <tr>
      <td>`--retry-max-backoff-ms`</td>
      <td>5000</td>
      <td>Maximum backoff duration (ms)</td>
    </tr>

    <tr>
      <td>`--retry-backoff-multiplier`</td>
      <td>2.0</td>
      <td>Exponential backoff multiplier</td>
    </tr>

    <tr>
      <td>`--retry-jitter-factor`</td>
      <td>0.1</td>
      <td>Random jitter factor (0.0-1.0)</td>
    </tr>

    <tr>
      <td>`--disable-retries`</td>
      <td>false</td>
      <td>Disable retries entirely</td>
    </tr>
  </tbody>
</table>

**Retryable Status Codes:** 408, 429, 500, 502, 503, 504

### Circuit Breaker

Per-worker circuit breakers prevent cascading failures:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 http://worker2:8001 \
  --cb-failure-threshold 5 \
  --cb-success-threshold 2 \
  --cb-timeout-duration-secs 30 \
  --cb-window-duration-secs 60
```

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--cb-failure-threshold`</td>
      <td>5</td>
      <td>Consecutive failures to open circuit</td>
    </tr>

    <tr>
      <td>`--cb-success-threshold`</td>
      <td>2</td>
      <td>Successes to close from half-open</td>
    </tr>

    <tr>
      <td>`--cb-timeout-duration-secs`</td>
      <td>30</td>
      <td>Time before half-open attempt</td>
    </tr>

    <tr>
      <td>`--cb-window-duration-secs`</td>
      <td>60</td>
      <td>Failure counting window</td>
    </tr>

    <tr>
      <td>`--disable-circuit-breaker`</td>
      <td>false</td>
      <td>Disable circuit breaker</td>
    </tr>
  </tbody>
</table>

**Circuit Breaker States:**

* **Closed**: Normal operation, requests allowed
* **Open**: Failing, requests rejected immediately
* **Half-Open**: Testing recovery, limited requests allowed

### Rate Limiting and Queuing

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 http://worker2:8001 \
  --max-concurrent-requests 256 \
  --rate-limit-tokens-per-second 512 \
  --queue-size 128 \
  --queue-timeout-secs 30
```

Requests beyond the concurrency limit wait in a FIFO queue. Returns:

* `429 Too Many Requests` when queue is full
* `408 Request Timeout` when queue timeout expires

### Health Checks

```bash Command theme={null}
--health-check-interval-secs 30 \
--health-check-timeout-secs 10 \
--health-success-threshold 2 \
--health-failure-threshold 3 \
--health-check-endpoint /health
```

***

## Reasoning Parser Integration

The gateway includes built-in reasoning parsers for models that use Chain-of-Thought (CoT) reasoning with explicit thinking blocks.

### Supported Parsers

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parser ID</th>
      <th>Model Family</th>
      <th>Think Tokens</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`deepseek-r1`</td>
      <td>DeepSeek-R1</td>
      <td>`&lt;think&gt;...&lt;/think&gt;` (initial reasoning)</td>
    </tr>

    <tr>
      <td>`qwen3`</td>
      <td>Qwen-3</td>
      <td>`&lt;think&gt;...&lt;/think&gt;`</td>
    </tr>

    <tr>
      <td>`qwen3-thinking`</td>
      <td>Qwen-3 Thinking</td>
      <td>`&lt;think&gt;...&lt;/think&gt;` (initial reasoning)</td>
    </tr>

    <tr>
      <td>`kimi`</td>
      <td>Kimi K2</td>
      <td>Unicode think tokens</td>
    </tr>

    <tr>
      <td>`glm45`</td>
      <td>GLM-4.5/4.6/4.7</td>
      <td>`&lt;think&gt;...&lt;/think&gt;`</td>
    </tr>

    <tr>
      <td>`step3`</td>
      <td>Step-3</td>
      <td>`&lt;think&gt;...&lt;/think&gt;`</td>
    </tr>

    <tr>
      <td>`minimax`</td>
      <td>MiniMax</td>
      <td>`&lt;think&gt;...&lt;/think&gt;`</td>
    </tr>
  </tbody>
</table>

### Usage

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls grpc://127.0.0.1:20000 \
  --model-path deepseek-ai/DeepSeek-R1 \
  --reasoning-parser deepseek-r1
```

The gRPC router automatically:

1. Detects reasoning blocks in streaming output
2. Separates reasoning content from normal text
3. Applies incremental streaming parsing with buffer management
4. Handles partial token detection for correct streaming behavior

***

## Tool Call Parsing

The gateway supports parsing function/tool calls from LLM outputs in multiple formats.

### Supported Formats

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parser</th>
      <th>Format</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`json`</td>
      <td>JSON</td>
      <td>Standard JSON tool calls</td>
    </tr>

    <tr>
      <td>`python`</td>
      <td>Pythonic</td>
      <td>Python function call syntax</td>
    </tr>

    <tr>
      <td>`xml`</td>
      <td>XML</td>
      <td>XML-formatted tool calls</td>
    </tr>
  </tbody>
</table>

### Usage

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls grpc://127.0.0.1:20000 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --tool-call-parser json
```

***

## Tokenizer Management

### Tokenizer Sources

The gateway supports multiple tokenizer backends:

* **HuggingFace**: Load from HuggingFace Hub by model ID
* **Local**: Load from local `tokenizer.json` or directory
* **Tiktoken**: Auto-detect OpenAI GPT models (gpt-4, davinci, etc.)

### Configuration

```bash Command theme={null}
# HuggingFace model
--model-path meta-llama/Llama-3.1-8B-Instruct

# Local tokenizer
--tokenizer-path /path/to/tokenizer.json

# With chat template override
--chat-template /path/to/template.jinja
```

### Tokenizer Caching

Two-level caching for optimal performance:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Cache</th>
      <th>Type</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>L0</td>
      <td>Exact match</td>
      <td>Whole-string caching for repeated prompts</td>
    </tr>

    <tr>
      <td>L1</td>
      <td>Prefix match</td>
      <td>Prefix boundary matching for incremental prompts</td>
    </tr>
  </tbody>
</table>

```bash Command theme={null}
--enable-l0-cache \
--l0-max-entries 10000 \
--enable-l1-cache \
--l1-max-memory 52428800  # 50MB
```

***

## MCP Integration

The gateway provides native Model Context Protocol (MCP) client integration for tool execution.

### Supported Transports

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Transport</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>STDIO</td>
      <td>Local process execution</td>
    </tr>

    <tr>
      <td>SSE</td>
      <td>Server-Sent Events (HTTP)</td>
    </tr>

    <tr>
      <td>Streamable</td>
      <td>Bidirectional streaming</td>
    </tr>
  </tbody>
</table>

### Configuration

```bash Command theme={null}
python -m sglang_router.launch_router \
  --mcp-config-path /path/to/mcp-config.yaml \
  --worker-urls http://worker1:8000
```

### MCP Configuration File

```yaml Config theme={null}
servers:
  - name: "filesystem"
    command: "npx"
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
    protocol: "stdio"
    required: false

  - name: "github"
    url: "https://api.github.com/mcp"
    token: "ghp_xxxxx"
    protocol: "sse"
    required: false

  - name: "custom-tools"
    url: "https://tools.example.com/mcp"
    protocol: "streamable"
    required: true

pool:
  max_connections: 100
  idle_timeout: 300

proxy:
  http: "http://proxy.internal:8080"
  https: "https://proxy.internal:8443"
  no_proxy: "localhost,127.0.0.1,*.internal"

inventory:
  enable_refresh: true
  tool_ttl: 300
  refresh_interval: 300
```

***

## Service Discovery (Kubernetes)

Enable automatic worker discovery via Kubernetes pod selectors:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --service-discovery \
  --selector app=sglang-worker role=inference \
  --service-discovery-namespace production \
  --service-discovery-port 8000
```

### PD Mode Discovery

```bash Command theme={null}
--pd-disaggregation \
--prefill-selector app=sglang component=prefill \
--decode-selector app=sglang component=decode \
--service-discovery
```

Prefill pods can expose bootstrap ports via the `sglang.ai/bootstrap-port` annotation. RBAC must allow `get`, `list`, and `watch` on pods.

***

## History and Data Connectors

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Backend</th>
      <th>Description</th>
      <th>Usage</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`memory`</td>
      <td>In-memory storage (default)</td>
      <td>`--history-backend memory`</td>
    </tr>

    <tr>
      <td>`none`</td>
      <td>No persistence</td>
      <td>`--history-backend none`</td>
    </tr>

    <tr>
      <td>`oracle`</td>
      <td>Oracle Autonomous Database</td>
      <td>`--history-backend oracle`</td>
    </tr>

    <tr>
      <td>`postgres`</td>
      <td>PostgreSQL Database</td>
      <td>`--history-backend postgres`</td>
    </tr>

    <tr>
      <td>`redis`</td>
      <td>Redis</td>
      <td>`--history-backend redis`</td>
    </tr>
  </tbody>
</table>

### Oracle Configuration

```bash Command theme={null}
# Connection descriptor
export ATP_DSN="(description=(address=(protocol=tcps)(port=1522)(host=adb.region.oraclecloud.com))(connect_data=(service_name=service_name)))"

# Or TNS alias (requires wallet)
export ATP_TNS_ALIAS="sglroutertestatp_high"
export ATP_WALLET_PATH="/path/to/wallet"

# Credentials
export ATP_USER="admin"
export ATP_PASSWORD="secret"
export ATP_POOL_MIN=4
export ATP_POOL_MAX=32

python -m sglang_router.launch_router \
  --backend openai \
  --worker-urls https://api.openai.com \
  --history-backend oracle
```

### PostgreSQL Configuration

```bash Command theme={null}
export POSTGRES_DB_URL="postgres://user:password@host:5432/dbname"

python -m sglang_router.launch_router \
  --backend openai \
  --worker-urls https://api.openai.com \
  --history-backend postgres
```

### Redis Configuration

```bash Command theme={null}
export REDIS_URL="redis://localhost:6379"
export REDIS_POOL_MAX=16
export REDIS_RETENTION_DAYS=30

python -m sglang_router.launch_router \
  --backend openai \
  --worker-urls https://api.openai.com \
  --history-backend redis \
  --redis-retention-days 30
```

Use `--redis-retention-days -1` for persistent storage (default is 30 days).

***

## WASM Middleware

The gateway supports WebAssembly (WASM) middleware modules for custom request/response processing. This enables organization-specific logic for authentication, rate limiting, billing, logging, and more—without modifying or recompiling the gateway.

### Overview

WASM middleware runs in a sandboxed environment with memory isolation, no network/filesystem access, and configurable resource limits.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Attach Point</th>
      <th>When Executed</th>
      <th>Use Cases</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`OnRequest`</td>
      <td>Before forwarding to workers</td>
      <td>Auth, rate limiting, request modification</td>
    </tr>

    <tr>
      <td>`OnResponse`</td>
      <td>After receiving worker response</td>
      <td>Logging, response modification, error handling</td>
    </tr>
  </tbody>
</table>

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Action</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`Continue`</td>
      <td>Proceed without modification</td>
    </tr>

    <tr>
      <td>`Reject(status)`</td>
      <td>Reject request with HTTP status code</td>
    </tr>

    <tr>
      <td>`Modify(...)`</td>
      <td>Modify headers, body, or status</td>
    </tr>
  </tbody>
</table>

### Examples

Complete working examples are available in `examples/wasm/`:

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Example</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`auth/`</td>
      <td>API key authentication for protected routes</td>
    </tr>

    <tr>
      <td>`rate_limit/`</td>
      <td>Per-client rate limiting (requests/minute)</td>
    </tr>

    <tr>
      <td>`logging/`</td>
      <td>Request tracking headers and response modification</td>
    </tr>
  </tbody>
</table>

The interface definition is located at `src/wasm/interface`.

### Building Modules

```bash Command theme={null}
# Prerequisites
rustup target add wasm32-wasip2
cargo install wasm-tools

# Build
cargo build --target wasm32-wasip2 --release

# Convert to component format
wasm-tools component new \
  target/wasm32-wasip2/release/my_middleware.wasm \
  -o my_middleware.component.wasm
```

### Deploying Modules

```bash Command theme={null}
# Enable WASM support
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 \
  --enable-wasm

# Upload module
curl -X POST http://localhost:30000/wasm \
  -H "Content-Type: application/json" \
  -d '{
    "modules": [{
      "name": "auth-middleware",
      "file_path": "/absolute/path/to/auth.component.wasm",
      "module_type": "Middleware",
      "attach_points": [{"Middleware": "OnRequest"}]
    }]
  }'

# List modules
curl http://localhost:30000/wasm

# Remove module
curl -X DELETE http://localhost:30000/wasm/{module_uuid}
```

### Runtime Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`max_memory_pages`</td>
      <td>1024 (64MB)</td>
      <td>Maximum WASM memory</td>
    </tr>

    <tr>
      <td>`max_execution_time_ms`</td>
      <td>1000</td>
      <td>Execution timeout</td>
    </tr>

    <tr>
      <td>`max_stack_size`</td>
      <td>1MB</td>
      <td>Stack size limit</td>
    </tr>

    <tr>
      <td>`module_cache_size`</td>
      <td>10</td>
      <td>Cached modules per worker</td>
    </tr>
  </tbody>
</table>

**Note:** Rate limiting state is per-worker thread and not shared across gateway replicas. For production, consider implementing rate limiting at a shared layer (e.g., Redis)

***

## Language Bindings

SGLang Model Gateway provides official language bindings for Python and Go, enabling integration with different technology stacks and organizational requirements.

### Python Bindings

The Python bindings provide a PyO3-based wrapper around the Rust gateway library. This is a straightforward binding that calls the gateway server startup from Python.

#### Installation

```bash Command theme={null}
# From PyPI
pip install sglang-router

# Development build
cd sgl-model-gateway/bindings/python
pip install maturin && maturin develop --features vendored-openssl
```

#### Usage

The Python bindings are used throughout this documentation. See the [Quick Start](#quick-start) and [Deployment Modes](#deployment-modes) sections for detailed examples.

Key components:

* `RouterArgs` dataclass with 50+ configuration options
* `Router.from_args()` for programmatic startup
* CLI commands: `smg launch`, `smg server`, `python -m sglang_router.launch_router`

### Go Bindings

The Go bindings provide a high-performance gRPC client library for organizations with Go-based infrastructure. This is ideal for:

* Integration with internal Go services and tooling
* High-performance client applications
* Building custom OpenAI-compatible proxy servers

#### Architecture

```text Output theme={null}
+-------------------------------------------+
|           High-Level Go API               |
|   (client.go - OpenAI-style interface)    |
+-------------------------------------------+
|              gRPC Layer                   |
+-------------------------------------------+
|           Rust FFI Layer                  |
|   (Tokenization, Parsing, Conversion)     |
+-------------------------------------------+
```

**Key Features:**

* Native Rust tokenization via FFI (thread-safe, lock-free)
* Full streaming support with context cancellation
* Configurable channel buffer sizes for high concurrency
* Built-in tool call parsing and chat template application

#### Installation

```bash Command theme={null}
# Build the FFI library first
cd sgl-model-gateway/bindings/golang
make build && make lib

# Then use in your Go project
go get github.com/sgl-project/sgl-go-sdk
```

**Requirements:** Go 1.24+, Rust toolchain

#### Examples

Complete working examples are available in `bindings/golang/examples/`:

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Example</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`simple/`</td>
      <td>Non-streaming chat completion</td>
    </tr>

    <tr>
      <td>`streaming/`</td>
      <td>Streaming chat completion with SSE</td>
    </tr>

    <tr>
      <td>`oai_server/`</td>
      <td>Full OpenAI-compatible HTTP server</td>
    </tr>
  </tbody>
</table>

```bash Command theme={null}
# Run examples
cd sgl-model-gateway/bindings/golang/examples/simple && ./run.sh
cd sgl-model-gateway/bindings/golang/examples/streaming && ./run.sh
cd sgl-model-gateway/bindings/golang/examples/oai_server && ./run.sh
```

#### Testing

```bash Command theme={null}
cd sgl-model-gateway/bindings/golang

# Unit tests
go test -v ./...

# Integration tests (requires running SGLang server)
export SGL_GRPC_ENDPOINT=grpc://localhost:20000
export SGL_TOKENIZER_PATH=/path/to/tokenizer
go test -tags=integration -v ./...
```

### Comparison

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Feature</th>
      <th>Python</th>
      <th>Go</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Primary Use**</td>
      <td>Gateway server launcher</td>
      <td>gRPC client library</td>
    </tr>

    <tr>
      <td>**CLI Support**</td>
      <td>Full CLI (smg, sglang-router)</td>
      <td>Library only</td>
    </tr>

    <tr>
      <td>**K8s Discovery**</td>
      <td>Native support</td>
      <td>N/A (client library)</td>
    </tr>

    <tr>
      <td>**PD Mode**</td>
      <td>Built-in</td>
      <td>N/A (client library)</td>
    </tr>
  </tbody>
</table>

**When to Use Python:** Launching and managing the gateway server, service discovery, PD disaggregation.

**When to Use Go:** Building custom client applications, integration with Go microservices, OpenAI-compatible proxy servers

***

## Security and Authentication

### Router API Key

```bash Command theme={null}
python -m sglang_router.launch_router \
  --api-key "your-router-api-key" \
  --worker-urls http://worker1:8000
```

Clients must supply `Authorization: Bearer <key>` for protected endpoints.

### Worker API Keys

```bash Command theme={null}
# Add worker with explicit key
curl -H "Authorization: Bearer router-key" \
  -X POST http://localhost:8080/workers \
  -H "Content-Type: application/json" \
  -d '{"url":"http://worker:8000","api_key":"worker-key"}'
```

### Security Configurations

1. **No Authentication** (default): Use only in trusted environments
2. **Router-only Authentication**: Clients authenticate to router
3. **Worker-only Authentication**: Router open, workers require keys
4. **Full Authentication**: Both router and workers protected

### TLS (HTTPS) for Gateway Server

Enable TLS to serve the gateway over HTTPS:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 \
  --tls-cert-path /path/to/server.crt \
  --tls-key-path /path/to/server.key
```

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--tls-cert-path`</td>
      <td>Path to server certificate (PEM format)</td>
    </tr>

    <tr>
      <td>`--tls-key-path`</td>
      <td>Path to server private key (PEM format)</td>
    </tr>
  </tbody>
</table>

Both parameters must be provided together. The gateway uses rustls with the ring crypto provider for TLS termination. If TLS is not configured, the gateway falls back to plain HTTP.

### mTLS for Worker Communication

Enable mutual TLS (mTLS) for secure communication with workers in HTTP mode:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls https://worker1:8443 https://worker2:8443 \
  --client-cert-path /path/to/client.crt \
  --client-key-path /path/to/client.key \
  --ca-cert-path /path/to/ca.crt
```

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--client-cert-path`</td>
      <td>Path to client certificate for mTLS (PEM format)</td>
    </tr>

    <tr>
      <td>`--client-key-path`</td>
      <td>Path to client private key for mTLS (PEM format)</td>
    </tr>

    <tr>
      <td>`--ca-cert-path`</td>
      <td>Path to CA certificate for verifying worker TLS (PEM format, repeatable)</td>
    </tr>
  </tbody>
</table>

**Key Points:**

* Client certificate and key must be provided together
* Multiple CA certificates can be added with multiple `--ca-cert-path` flags
* Uses rustls backend when TLS is configured
* Single HTTP client is created for all workers (assumes single security domain)
* TCP keepalive (30 seconds) is enabled for long-lived connections

### Full TLS Configuration Example

Gateway HTTPS + Worker mTLS + API Key authentication:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls https://worker1:8443 https://worker2:8443 \
  --tls-cert-path /etc/certs/server.crt \
  --tls-key-path /etc/certs/server.key \
  --client-cert-path /etc/certs/client.crt \
  --client-key-path /etc/certs/client.key \
  --ca-cert-path /etc/certs/ca.crt \
  --api-key "secure-api-key" \
  --policy cache_aware
```

***

## Observability

### Prometheus Metrics

Enable with `--prometheus-host`/`--prometheus-port` (defaults to `0.0.0.0:29000`).

#### Metric Categories (40+ metrics)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Layer</th>
      <th>Prefix</th>
      <th>Metrics</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>HTTP</td>
      <td>`smg_http_*`</td>
      <td>`requests_total`, `request_duration_seconds`, `responses_total`, `connections_active`, `rate_limit_total`</td>
    </tr>

    <tr>
      <td>Router</td>
      <td>`smg_router_*`</td>
      <td>`requests_total`, `request_duration_seconds`, `request_errors_total`, `stage_duration_seconds`, `upstream_responses_total`</td>
    </tr>

    <tr>
      <td>Inference</td>
      <td>`smg_router_*`</td>
      <td>`ttft_seconds`, `tpot_seconds`, `tokens_total`, `generation_duration_seconds`</td>
    </tr>

    <tr>
      <td>Worker</td>
      <td>`smg_worker_*`</td>
      <td>`pool_size`, `connections_active`, `requests_active`, `health_checks_total`, `selection_total`, `errors_total`</td>
    </tr>

    <tr>
      <td>Circuit Breaker</td>
      <td>`smg_worker_cb_*`</td>
      <td>`state`, `transitions_total`, `outcomes_total`, `consecutive_failures`, `consecutive_successes`</td>
    </tr>

    <tr>
      <td>Retry</td>
      <td>`smg_worker_*`</td>
      <td>`retries_total`, `retries_exhausted_total`, `retry_backoff_seconds`</td>
    </tr>

    <tr>
      <td>Discovery</td>
      <td>`smg_discovery_*`</td>
      <td>`registrations_total`, `deregistrations_total`, `sync_duration_seconds`, `workers_discovered`</td>
    </tr>

    <tr>
      <td>MCP</td>
      <td>`smg_mcp_*`</td>
      <td>`tool_calls_total`, `tool_duration_seconds`, `servers_active`, `tool_iterations_total`</td>
    </tr>

    <tr>
      <td>Database</td>
      <td>`smg_db_*`</td>
      <td>`operations_total`, `operation_duration_seconds`, `connections_active`, `items_stored`</td>
    </tr>
  </tbody>
</table>

#### Key Inference Metrics (gRPC mode)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Metric</th>
      <th>Type</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`smg_router_ttft_seconds`</td>
      <td>Histogram</td>
      <td>Time to first token</td>
    </tr>

    <tr>
      <td>`smg_router_tpot_seconds`</td>
      <td>Histogram</td>
      <td>Time per output token</td>
    </tr>

    <tr>
      <td>`smg_router_tokens_total`</td>
      <td>Counter</td>
      <td>Total tokens (input/output)</td>
    </tr>

    <tr>
      <td>`smg_router_generation_duration_seconds`</td>
      <td>Histogram</td>
      <td>End-to-end generation time</td>
    </tr>
  </tbody>
</table>

#### Duration Buckets

1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s, 10s, 15s, 30s, 45s, 60s, 90s, 120s, 180s, 240s

### OpenTelemetry Tracing

Enable distributed tracing with OTLP export:

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 \
  --enable-trace \
  --otlp-traces-endpoint localhost:4317
```

#### Features

* OTLP/gRPC exporter (default port 4317)
* W3C Trace Context propagation for HTTP and gRPC
* Batch span processing (500ms delay, 64 span batch size)
* Custom filtering to reduce noise
* Trace context injection into upstream worker requests
* Service name: `sgl-router`

### Logging

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 \
  --log-level debug \
  --log-dir ./router_logs
```

Structured tracing with optional file sink. Log levels: `debug`, `info`, `warn`, `error`.

### Request ID Propagation

```bash Command theme={null}
--request-id-headers x-request-id x-trace-id x-correlation-id
```

Responses include `x-request-id` header for correlation.

***

## Production Recommendations

This section provides guidance for deploying SGLang Model Gateway in production environments.

### Security Best Practices

**Always enable TLS in production:**

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls https://worker1:8443 https://worker2:8443 \
  --tls-cert-path /etc/certs/server.crt \
  --tls-key-path /etc/certs/server.key \
  --client-cert-path /etc/certs/client.crt \
  --client-key-path /etc/certs/client.key \
  --ca-cert-path /etc/certs/ca.crt \
  --api-key "${ROUTER_API_KEY}"
```

**Security Checklist:**

* Enable TLS for gateway HTTPS termination
* Enable mTLS for worker communication when workers are on untrusted networks
* Set `--api-key` to protect router endpoints
* Use Kubernetes Secrets or a secrets manager for credentials
* Rotate certificates and API keys periodically
* Restrict network access with firewalls or network policies

### High Availability

**Scaling Strategy:**

The gateway supports running multiple replicas behind a load balancer for high availability. However, there are important considerations:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Component</th>
      <th>Shared Across Replicas</th>
      <th>Impact</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Worker Registry</td>
      <td>No (independent)</td>
      <td>Each replica discovers workers independently</td>
    </tr>

    <tr>
      <td>Radix Cache Tree</td>
      <td>No (independent)</td>
      <td>Cache hits may decrease by 10-20%</td>
    </tr>

    <tr>
      <td>Circuit Breaker State</td>
      <td>No (independent)</td>
      <td>Each replica tracks failures independently</td>
    </tr>

    <tr>
      <td>Rate Limiting</td>
      <td>No (independent)</td>
      <td>Limits apply per-replica, not globally</td>
    </tr>
  </tbody>
</table>

**Recommendations:**

1. **Prefer horizontal scaling over vertical scaling**: Deploy multiple smaller gateway replicas rather than one large instance with excessive CPU and memory. This provides:
   * Better fault tolerance (single replica failure doesn't take down the gateway)
   * More predictable resource usage
   * Easier capacity planning

2. **Use Kubernetes Service Discovery**: Let the gateway automatically discover and manage workers:
   ```bash Command theme={null}
   python -m sglang_router.launch_router \
     --service-discovery \
     --selector app=sglang-worker \
     --service-discovery-namespace production
   ```

3. **Accept cache efficiency trade-off**: With multiple replicas, the cache-aware routing policy's radix tree is not synchronized across replicas. This means:
   * Each replica builds its own cache tree
   * Requests from the same user may hit different replicas
   * Expected cache hit rate reduction: **10-20%**
   * This is often acceptable given the HA benefits

4. **Configure session affinity (optional)**: If cache efficiency is critical, configure your load balancer for session affinity based on a consistent hash of the request (e.g., user ID or API key).

**Example HA Architecture:**

```text Output theme={null}
                    +-------------------+
                    |   Load Balancer   |
                    |     (L4/L7)       |
                    +---------+---------+
                              |
          +-------------------+-------------------+
          |                   |                   |
          v                   v                   v
    +-----------+       +-----------+       +-----------+
    |  Gateway  |       |  Gateway  |       |  Gateway  |
    | Replica 1 |       | Replica 2 |       | Replica 3 |
    +-----+-----+       +-----+-----+       +-----+-----+
          |                   |                   |
          +-------------------+-------------------+
                              |
          +-------------------+-------------------+
          |                   |                   |
          v                   v                   v
    +-----------+       +-----------+       +-----------+
    |  Worker   |       |  Worker   |       |  Worker   |
    |   Pod 1   |       |   Pod 2   |       |   Pod N   |
    +-----------+       +-----------+       +-----------+
```

### Performance

**Use gRPC mode for high throughput:**

gRPC mode provides the highest performance for SGLang workers:

```bash Command theme={null}
# Start workers in gRPC mode
python -m sglang.launch_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --grpc-mode \
  --port 20000

# Configure gateway for gRPC
python -m sglang_router.launch_router \
  --worker-urls grpc://worker1:20000 grpc://worker2:20000 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --policy cache_aware
```

**Performance Benefits of gRPC:**

* Native Rust tokenization (no Python overhead)
* Streaming with lower latency
* Built-in reasoning parser execution
* Tool call parsing in the gateway
* Reduced serialization overhead

**Tuning Recommendations:**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Recommendation</th>
      <th>Reason</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--policy`</td>
      <td>`cache_aware`</td>
      <td>Best for repeated prompts, \~30% latency reduction</td>
    </tr>

    <tr>
      <td>`--max-concurrent-requests`</td>
      <td>2-4x worker count</td>
      <td>Prevent overload while maximizing throughput</td>
    </tr>

    <tr>
      <td>`--queue-size`</td>
      <td>2x max-concurrent</td>
      <td>Buffer for burst traffic</td>
    </tr>

    <tr>
      <td>`--request-timeout-secs`</td>
      <td>Based on max generation length</td>
      <td>Prevent stuck requests</td>
    </tr>
  </tbody>
</table>

### Kubernetes Deployment

**Pod Labeling for Service Discovery:**

For the gateway to discover workers automatically, label your worker pods consistently:

```yaml Config theme={null}
# Worker Deployment (Regular Mode)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang-worker
  namespace: production
spec:
  replicas: 4
  selector:
    matchLabels:
      app: sglang-worker
      component: inference
  template:
    metadata:
      labels:
        app: sglang-worker
        component: inference
        model: llama-3-8b
    spec:
      containers:
      - name: worker
        image: lmsysorg/sglang:latest
        ports:
        - containerPort: 8000
          name: http
        - containerPort: 20000
          name: grpc
```

**Gateway configuration for discovery:**

```bash Command theme={null}
python -m sglang_router.launch_router \
  --service-discovery \
  --selector app=sglang-worker component=inference \
  --service-discovery-namespace production \
  --service-discovery-port 8000
```

**PD (Prefill/Decode) Mode Labeling:**

```yaml Config theme={null}
# Prefill Worker
metadata:
  labels:
    app: sglang-worker
    component: prefill
  annotations:
    sglang.ai/bootstrap-port: "9001"

# Decode Worker
metadata:
  labels:
    app: sglang-worker
    component: decode
```

**Gateway configuration for PD discovery:**

```bash Command theme={null}
python -m sglang_router.launch_router \
  --service-discovery \
  --pd-disaggregation \
  --prefill-selector app=sglang-worker component=prefill \
  --decode-selector app=sglang-worker component=decode \
  --service-discovery-namespace production
```

**RBAC Requirements:**

The gateway needs permissions to watch pods:

```yaml Config theme={null}
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sglang-gateway
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
***
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sglang-gateway
  namespace: production
subjects:
- kind: ServiceAccount
  name: sglang-gateway
  namespace: production
roleRef:
  kind: Role
  name: sglang-gateway
  apiGroup: rbac.authorization.k8s.io
```

### Monitoring with PromQL

Configure Prometheus to scrape the gateway metrics endpoint (default: `:29000/metrics`).

**Essential Dashboards:**

**1. Request Rate and Latency:**

```sql Example theme={null}
# Request rate by endpoint
sum(rate(smg_http_requests_total[5m])) by (path, method)

# P50 latency
histogram_quantile(0.50, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le))

# P99 latency
histogram_quantile(0.99, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le))

# Error rate
sum(rate(smg_http_responses_total{status=~"5.."}[5m])) / sum(rate(smg_http_responses_total[5m]))
```

**2. Worker Health:**

```sql Example theme={null}
# Healthy workers
sum(smg_worker_pool_size)

# Active connections per worker
smg_worker_connections_active

# Worker health check failures
sum(rate(smg_worker_health_checks_total{result="failure"}[5m])) by (worker_id)
```

**3. Circuit Breaker Status:**

```sql Example theme={null}
# Circuit breaker states (0=closed, 1=open, 2=half-open)
smg_worker_cb_state

# Circuit breaker transitions
sum(rate(smg_worker_cb_transitions_total[5m])) by (worker_id, from_state, to_state)

# Workers with open circuits
count(smg_worker_cb_state == 1)
```

**4. Inference Performance (gRPC mode):**

```sql Example theme={null}
# Time to first token (P50)
histogram_quantile(0.50, sum(rate(smg_router_ttft_seconds_bucket[5m])) by (le, model))

# Time per output token (P99)
histogram_quantile(0.99, sum(rate(smg_router_tpot_seconds_bucket[5m])) by (le, model))

# Token throughput
sum(rate(smg_router_tokens_total[5m])) by (model, direction)

# Generation duration P95
histogram_quantile(0.95, sum(rate(smg_router_generation_duration_seconds_bucket[5m])) by (le))
```

**5. Rate Limiting and Queuing:**

```sql Example theme={null}
# Rate limit rejections
sum(rate(smg_http_rate_limit_total{decision="rejected"}[5m]))

# Queue depth (if using concurrency limiting)
smg_worker_requests_active

# Retry attempts
sum(rate(smg_worker_retries_total[5m])) by (worker_id)

# Exhausted retries (failures after all retries)
sum(rate(smg_worker_retries_exhausted_total[5m]))
```

**6. MCP Tool Execution:**

```sql Example theme={null}
# Tool call rate
sum(rate(smg_mcp_tool_calls_total[5m])) by (server, tool)

# Tool latency P95
histogram_quantile(0.95, sum(rate(smg_mcp_tool_duration_seconds_bucket[5m])) by (le, tool))

# Active MCP server connections
smg_mcp_servers_active
```

**Alerting Rules Example:**

```yaml Config theme={null}
groups:
- name: sglang-gateway
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(smg_http_responses_total{status=~"5.."}[5m]))
      / sum(rate(smg_http_responses_total[5m])) > 0.05
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "High error rate on SGLang Gateway"

  - alert: CircuitBreakerOpen
    expr: count(smg_worker_cb_state == 1) > 0
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "Worker circuit breaker is open"

  - alert: HighLatency
    expr: |
      histogram_quantile(0.99, sum(rate(smg_http_request_duration_seconds_bucket[5m])) by (le)) > 30
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "P99 latency exceeds 30 seconds"

  - alert: NoHealthyWorkers
    expr: sum(smg_worker_pool_size) == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "No healthy workers available"
```

***

## Configuration Reference

### Core Settings

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--host`</td>
      <td>str</td>
      <td>127.0.0.1</td>
      <td>Router host</td>
    </tr>

    <tr>
      <td>`--port`</td>
      <td>int</td>
      <td>30000</td>
      <td>Router port</td>
    </tr>

    <tr>
      <td>`--worker-urls`</td>
      <td>list</td>
      <td>\[]</td>
      <td>Worker URLs (HTTP or gRPC)</td>
    </tr>

    <tr>
      <td>`--policy`</td>
      <td>str</td>
      <td>cache\_aware</td>
      <td>Routing policy</td>
    </tr>

    <tr>
      <td>`--max-concurrent-requests`</td>
      <td>int</td>
      <td>-1</td>
      <td>Concurrency limit (-1 disables)</td>
    </tr>

    <tr>
      <td>`--request-timeout-secs`</td>
      <td>int</td>
      <td>600</td>
      <td>Request timeout</td>
    </tr>

    <tr>
      <td>`--max-payload-size`</td>
      <td>int</td>
      <td>256MB</td>
      <td>Maximum request payload</td>
    </tr>
  </tbody>
</table>

### Prefill/Decode

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--pd-disaggregation`</td>
      <td>flag</td>
      <td>false</td>
      <td>Enable PD mode</td>
    </tr>

    <tr>
      <td>`--prefill`</td>
      <td>list</td>
      <td>\[]</td>
      <td>Prefill URLs + optional bootstrap ports</td>
    </tr>

    <tr>
      <td>`--decode`</td>
      <td>list</td>
      <td>\[]</td>
      <td>Decode URLs</td>
    </tr>

    <tr>
      <td>`--prefill-policy`</td>
      <td>str</td>
      <td>None</td>
      <td>Override policy for prefill nodes</td>
    </tr>

    <tr>
      <td>`--decode-policy`</td>
      <td>str</td>
      <td>None</td>
      <td>Override policy for decode nodes</td>
    </tr>

    <tr>
      <td>`--decode-dispatch-limit`</td>
      <td>int</td>
      <td>None</td>
      <td>Requests per prefill worker sent to decode before decode answers or prefill finishes; the rest pick their decode worker when sent. Needs prefill `--optimistic-prefill-attempts` above 0</td>
    </tr>

    <tr>
      <td>`--worker-startup-timeout-secs`</td>
      <td>int</td>
      <td>600</td>
      <td>Worker init timeout</td>
    </tr>
  </tbody>
</table>

### Kubernetes Discovery

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--service-discovery`</td>
      <td>flag</td>
      <td>Enable discovery</td>
    </tr>

    <tr>
      <td>`--selector`</td>
      <td>list</td>
      <td>Label selectors (key=value)</td>
    </tr>

    <tr>
      <td>`--prefill-selector` / `--decode-selector`</td>
      <td>list</td>
      <td>PD mode selectors</td>
    </tr>

    <tr>
      <td>`--service-discovery-namespace`</td>
      <td>str</td>
      <td>Namespace to watch</td>
    </tr>

    <tr>
      <td>`--service-discovery-port`</td>
      <td>int</td>
      <td>Worker port (default 80)</td>
    </tr>

    <tr>
      <td>`--bootstrap-port-annotation`</td>
      <td>str</td>
      <td>Annotation for bootstrap ports</td>
    </tr>
  </tbody>
</table>

### TLS Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--tls-cert-path`</td>
      <td>str</td>
      <td>Server certificate for gateway HTTPS (PEM)</td>
    </tr>

    <tr>
      <td>`--tls-key-path`</td>
      <td>str</td>
      <td>Server private key for gateway HTTPS (PEM)</td>
    </tr>

    <tr>
      <td>`--client-cert-path`</td>
      <td>str</td>
      <td>Client certificate for worker mTLS (PEM)</td>
    </tr>

    <tr>
      <td>`--client-key-path`</td>
      <td>str</td>
      <td>Client private key for worker mTLS (PEM)</td>
    </tr>

    <tr>
      <td>`--ca-cert-path`</td>
      <td>str</td>
      <td>CA certificate for verifying workers (PEM, repeatable)</td>
    </tr>
  </tbody>
</table>

***

## Troubleshooting

### Workers Never Ready

Increase `--worker-startup-timeout-secs` or ensure health probes respond before router startup.

### Load Imbalance / Hot Workers

Inspect `smg_router_requests_total` by worker and tune cache-aware thresholds (`--balance-*`, `--cache-threshold`).

### Circuit Breaker Flapping

Increase `--cb-failure-threshold` or extend the timeout/window durations. Consider temporarily disabling retries.

### Queue Overflow (429)

Increase `--queue-size` or reduce client concurrency. Ensure `--max-concurrent-requests` matches downstream capacity.

### Memory Growth

Reduce `--max-tree-size` or lower `--eviction-interval-secs` for more aggressive cache pruning.

### Debugging

```bash Command theme={null}
python -m sglang_router.launch_router \
  --worker-urls http://worker1:8000 \
  --log-level debug \
  --log-dir ./router_logs
```

### gRPC Connection Issues

Ensure workers are started with `--grpc-mode` and verify `--model-path` or `--tokenizer-path` is provided to the router.

### Tokenizer Loading Failures

Check HuggingFace Hub credentials (`HF_TOKEN` environment variable) for private models. Verify local paths are accessible.

***

SGLang Model Gateway continues to evolve alongside the SGLang runtime. Keep CLI flags, integrations, and documentation aligned when adopting new features or contributing improvements.


# SGLang for RL Systems
Source: https://docs.sglang.io/docs/advanced_features/sglang_for_rl



This document is a practical guide for infrastructure teams integrating SGLang into RL and post-training systems. It focuses on the operational pain points in the loop (rollout, evaluation, training, weight sync) and maps them to concrete SGLang APIs, flags, and integration patterns. The focus is on maximizing rollout efficiency, accuracy and stability while keeping rollout-serving behavior aligned in production environments.

## Why SGLang for RL Lifecycle?

Let's embrace a guiding principle from early DeepMind's RL engineering:

**Be a library, not a framework.**

This philosophy empowers innovation by providing SGLang as flexible tools, not rigid structures. Here are five reasons to use SGLang for your RL lifecycle:

* **Fine-Grained Engine Sleep and Wake Up**: facilitate maximum-powered rollout and training
* **Open-To-Use Refit Functionality**: diverse methods for co-location or disaggregation
* **Easy To Postpone Generation**: enable partial rollout and dedicated rollout control
* **Deterministic Inference**: achieve deterministic inference to enable zero training-inference mismatch
* **Load Balancing Router**: cache-aware load-balancing for high-throughput rollout

The following sections cover these aspects in detail.

## Fine-Grained Engine Sleep and Wake Up

Rollout and training are both memory-intensive, and co-locating them on the same GPUs often leads to memory pressure and slow handoffs. SGLang provides a memory-aware sleep/wake mechanism that releases KV cache and weights while keeping the server process alive, then resumes them for rollout without a full restart. This avoids repeated disk I/O and CUDA graph recapture during each RL step.

Under the hood, the RL team uses CUDA-graph-aware weight offload via [torch\_memory\_saver](https://github.com/fzyzcjy/torch_memory_saver) to preserve virtual memory addresses for graph replay. For details, see: [Efficient RL Training - Optimizing Memory Usage in verl](https://hebiao064.github.io/rl-memory-management).

### Server flag

Enable memory saver support when launching the server:

```text Output theme={null}
--enable-memory-saver
```

### Release Memory

**Endpoint:** `POST /release_memory_occupation`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`tags`</td>
      <td>Which memory regions to release. If omitted, all are released.</td>
      <td>`None`</td>
      <td>Type: list\[str], values: `kv_cache`, `weights`</td>
    </tr>
  </tbody>
</table>

**Behavior notes:**

* This call asserts there are no ongoing requests. Ensure the engine is idle before calling it.
* If `kv_cache` is released, SGLang flushes cache; subsequent requests will rebuild KV cache as needed.

### Resume Memory

**Endpoint:** `POST /resume_memory_occupation`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`tags`</td>
      <td>Which memory regions to resume. If omitted, all are resumed.</td>
      <td>`None`</td>
      <td>Type: list\[str], values: `kv_cache`, `weights`</td>
    </tr>
  </tbody>
</table>

## Open-To-Use Refit Functionality

After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang's weight update utilities, see [RL System Deep Thinking: Weight Update Mechanisms](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN.md).

**How to choose:**

* **From disk** is simplest and best for elastic rollout scaling and checkpointing.
* **From tensor** is best for co-located training/rollout when you can pass in-memory tensors.
* **From distributed** is best for disaggregated training/rollout with dedicated communication groups (NCCL/IB).

### Update Weights from Disk

**When to use:**

* Save checkpoint to disk and update weights from disk
* Dynamic scaling (new rollout instances can load from the same checkpoint)

**Why it works well:**

This path trades some I/O overhead for simplicity and flexibility. It integrates naturally with checkpointing and makes it trivial to add new rollout engines: point them at the same checkpoint and call the API. It is also the safest option for high availability because the checkpoint itself is the source of truth.

**Endpoint:** `POST /update_weights_from_disk`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`model_path`</td>
      <td>The model path with the new weights.</td>
      <td>Required</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`load_format`</td>
      <td>The format to load the weights.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`abort_all_requests`</td>
      <td>Abort all running requests before update.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`weight_version`</td>
      <td>Optional weight version label tracked by the server.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`is_async`</td>
      <td>Perform weight load asynchronously.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`torch_empty_cache`</td>
      <td>Empty torch cache.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`keep_pause`</td>
      <td>Keep scheduler paused after update.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`recapture_cuda_graph`</td>
      <td>Recapture CUDA graphs after update.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`token_step`</td>
      <td>Trainer step id for rollout bookkeeping.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`flush_cache`</td>
      <td>Flush KV cache after update.</td>
      <td>`True`</td>
      <td>Type: bool</td>
    </tr>
  </tbody>
</table>

**Response body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`success`</td>
      <td>Whether the update succeeded.</td>
      <td>-</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`message`</td>
      <td>Status / error message.</td>
      <td>-</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`num_paused_requests`</td>
      <td>Number of paused requests during update.</td>
      <td>`0`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

**Python Engine API:** `engine.update_weights_from_disk(model_path, load_format=None)`

**Diffusion engine (SGLang-Diffusion):** The diffusion engine exposes the same `POST /update_weights_from_disk` endpoint with the following behavior:

* **All-or-nothing with rollback:** if any module fails to load, all previously updated modules are rolled back to the original weights by reloading from the original model path. No partial updates are left behind. If rollback itself fails, the exception propagates so the caller knows the model is in an inconsistent state.
* **Offload-aware:** when layerwise offload (`--dit-layerwise-offload`) is enabled, the diffusion offload manager replaces GPU parameters with small `torch.empty((1,))` placeholders while real weights live in consolidated pinned CPU buffers. A naive `param.data.copy_()` would fail with a shape mismatch. Instead, the updater dynamically detects active offload managers and writes new weights directly into their CPU buffers, bypassing the placeholders entirely. For any layer that happens to be prefetched on GPU at update time, the live GPU tensor is also updated so the change takes effect immediately. This requires no extra GPU memory and does not disturb the offload state.
* **DTensor-aware:** parameters distributed via `torch.distributed.tensor` (tensor parallelism) are updated through `distribute_tensor` so that each shard is correctly placed on the right device mesh.

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`model_path`</td>
      <td>The model path with the new weights.</td>
      <td>Required</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`flush_cache`</td>
      <td>Flush TeaCache state after update.</td>
      <td>`True`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`target_modules`</td>
      <td>List of module names to update (e.g. `["transformer"]`). If omitted, all `nn.Module` components are updated.</td>
      <td>`None`</td>
      <td>Type: list\[str]</td>
    </tr>
  </tbody>
</table>

**Response body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`success`</td>
      <td>Whether the update succeeded.</td>
      <td>-</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`message`</td>
      <td>Status / error message.</td>
      <td>-</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

> **Note:** The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently.

### Weight Update Session

`update_weights_from_tensor` and `update_weights_from_distributed` must run inside a weight-update session. The update calls only write checkpoint-format values; `end_weight_update` runs every quantized layer's `process_weights_after_loading` once, turning them into kernel layout (Marlin repack, MXFP8 shuffles, ...), and runs `post_load_weights` if no weights were loaded through the engine during the session (P2P/RDMA writes). `begin_weight_update` first restores the few schemes whose kernel layout changes parameter shapes (today compressed-tensors W4A16 MoE on Marlin) so the checkpoint-format values fit. One session can span many update calls, so the finalize cost is paid once per refit rather than once per bucket. A call outside a session, a second `begin_weight_update`, or an `end_weight_update` without one is rejected with `success: false` (HTTP 400) and leaves the weights untouched.

**Endpoints:** `POST /begin_weight_update`, `POST /end_weight_update`

`begin_weight_update` accepts `selector`: `"target"`, `"draft"`, or `"all"` (default). Under speculative decoding the same selector on the update endpoints picks which model runners receive the weights; `end_weight_update` finalizes the runners the session opened.

```python theme={null}
engine.begin_weight_update()
for bucket in buckets:
    engine.update_weights_from_tensor(bucket)
engine.end_weight_update()
```

### Update Weights from Tensor

**When to use:**

* Co-located training and rollout, where training can provide tensors directly
* Fast in-memory updates

**Important constraints:**

This strategy requires the training process and rollout engine to share access to the tensors. Co-located setups must keep the model on GPU; moving tensors to CPU will break the update path. For high-performance MoE or specialized attention kernels, co-location may limit some optimizations compared to disaggregated rollouts.

**Endpoint:** `POST /update_weights_from_tensor`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`serialized_named_tensors`</td>
      <td>Per-TP serialized tensor payloads.</td>
      <td>Required</td>
      <td>Type: list\[str|bytes]</td>
    </tr>

    <tr>
      <td>`load_format`</td>
      <td>Optional load format selector.</td>
      <td>`None`</td>
      <td>`None`, `direct`, `flattened_bucket`, or a custom loader path string</td>
    </tr>

    <tr>
      <td>`flush_cache`</td>
      <td>Flush KV cache after update.</td>
      <td>`True`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`abort_all_requests`</td>
      <td>Abort all running requests before update.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`weight_version`</td>
      <td>Optional version label tracked by the server.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

**Note:** The serialized tensor payloads must be created with `MultiprocessingSerializer.serialize(...)` and should be base64-safe strings.

**Python Engine API:** `engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)`, inside `begin_weight_update()` / `end_weight_update()`.

### Update Weights from Distributed Group

**When to use:**

* Disaggregated training and rollout
* NCCL or IB-backed weight broadcast from training workers to rollout workers

**How it works:**

Training workers gather weights (typically on TP rank 0), broadcast them to the rollout group, and each rollout TP shard loads the parameters it needs. This avoids disk I/O and keeps training and rollout decoupled, at the cost of managing a dedicated communication group.

**Initialize weight update group**

**Endpoint:** `POST /init_weights_update_group`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`master_address`</td>
      <td>Group master address.</td>
      <td>Required</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`master_port`</td>
      <td>Group master port.</td>
      <td>Required</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`rank_offset`</td>
      <td>Offset for local rank mapping.</td>
      <td>Required</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`world_size`</td>
      <td>Total world size.</td>
      <td>Required</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`group_name`</td>
      <td>Group name.</td>
      <td>`weight_update_group`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`backend`</td>
      <td>Communication backend.</td>
      <td>`nccl`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

**Update weight**

**Endpoint:** `POST /update_weights_from_distributed`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`names`</td>
      <td>Parameter names to update.</td>
      <td>Required</td>
      <td>Type: list\[str]</td>
    </tr>

    <tr>
      <td>`dtypes`</td>
      <td>Dtype strings for each parameter.</td>
      <td>Required</td>
      <td>Type: list\[str]</td>
    </tr>

    <tr>
      <td>`shapes`</td>
      <td>Tensor shapes.</td>
      <td>Required</td>
      <td>Type: list\[list\[int]]</td>
    </tr>

    <tr>
      <td>`group_name`</td>
      <td>Group name.</td>
      <td>`weight_update_group`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`flush_cache`</td>
      <td>Flush KV cache after update.</td>
      <td>`True`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`abort_all_requests`</td>
      <td>Abort all running requests before update.</td>
      <td>`False`</td>
      <td>Type: bool</td>
    </tr>

    <tr>
      <td>`weight_version`</td>
      <td>Optional version label.</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`load_format`</td>
      <td>Optional format selector.</td>
      <td>`None`</td>
      <td>`None` or `flattened_bucket`</td>
    </tr>
  </tbody>
</table>

**Destroy weights update group**

**Endpoint:** `POST /destroy_weights_update_group`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`group_name`</td>
      <td>Group name.</td>
      <td>`weight_update_group`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>

**Python Engine APIs:**

* `engine.init_weights_update_group(...)`
* `engine.begin_weight_update(selector="all")`
* `engine.update_weights_from_distributed(names, dtypes, shapes, ...)`
* `engine.end_weight_update()`
* `engine.destroy_weights_update_group(group_name)`

### Per-Token Weight Version Attribution

A request can outlive a version change: the RL flow retracts it with `pause_generation`, refits, and continues it, or `POST /update_weight_version` relabels while it is still generating. `meta_info` then reports which tokens came from which weights:

```json Output theme={null}
"weight_version": "42",
"weight_versions": [
    {"version": "41", "start": 0, "end": 57},
    {"version": "42", "start": 57, "end": 128}
]
```

* Ranges are half-open `[start, end)` over output-token indices, prompt excluded. The usual case is a single span.

## Easy To Postpone Generation

Multi-turn RL rollouts often suffer from long-tail requests that block the entire batch. A small number of slow interactions can stall all GPUs, and the long-tail behavior makes profiling and monitoring difficult.

SGLang exposes explicit pause/resume APIs so you can pause slow requests and continue them later. This pattern matches systems like [APRIL](https://arxiv.org/abs/2509.18521), terminate once enough responses are collected, and recycle incomplete responses in the next step. The result is higher GPU utilization without discarding partial work.

`pause_generation` ---  update weights --- `continue_generation` is the correct execution flow when updating weights from training. An update can only happen when SGLang is not actively processing inference tasks.

### Pause Generation

**Endpoint:** `POST /pause_generation`

**Request body:**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Description</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`mode`</td>
      <td>Pause mode.</td>
      <td>`abort`</td>
      <td>`abort`, `retract`, `in_place`</td>
    </tr>
  </tbody>
</table>

**Modes:**

* `abort`: Default behavior, identical to `abort` endpoint with `abort_all` set. Pending requests from `waiting_queue` and `running_queue` will be returned immediately to the caller.
* `retract`: Put engine in "paused" state.  Move running requests back to waiting queue. KV cache can be flushed and recomputed later.
* `in_place`: Put engine in "paused" state without changing states of the requests. Running requests rely on availability of KV caches to continue, so any subsequent `flush_cache` call will be unsuccessful.

### Continue Generation

**Endpoint:** `POST /continue_generation`

## Deterministic Inference

In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem.

SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: [True On-Policy](https://github.com/radixark/miles/tree/main/examples/true_on_policy) and [True On-Policy for VLM](https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm). For additional context, see the blog post [Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md).

**Server flag:**

```text Output theme={null}
--enable-deterministic-inference
```

For more details, see [Deterministic Inference](./deterministic_inference)

## Load Balancing Router

SGLang Model Gateway is the recommended control plane for large‑scale RL rollouts. It provides async, non‑blocking request handling, cache‑aware load balancing, and fault‑tolerant routing across rollout and reward servers. This lets you keep GPUs saturated while avoiding long‑tail stalls and brittle, engine‑local concurrency logic. It has been deployed in the training of GLM 4.5+ models and proven to be highly efficient in production-level large-scale RL workloads.

Key benefits for RL infrastructure:

* **Async non-blocking efficiency**: SGLang’s native async server/router architecture (HTTPS/gRPC) manages concurrency automatically. This guarantees maximum GPU saturation and effective continuous batching without requiring complex, manual implementation by engineers.
* **Elasticity and fault tolerance**: By encapsulating the reward model and rollout as independent servers, SGLang decouples them logically and physically. This architecture provides robust disaster recovery for large-scale distributed training; if a server fails, the router automatically redirects traffic to healthy nodes, ensuring the training process continues without interruption.
* **Training–Inference alignment**: Using the SGLang Model Gateway for both training and inference ensures "What You See Is What You Get." This eliminates score discrepancies and the painful backend alignment issues often caused by using different engines for training versus deployment.
* **Dynamic load balancing and long-tail mitigation**: Unlike static partitioning, the SGLang Model Gateway enables request-level dynamic dispatching for multi-turn RL. It can distribute different turns of a conversation across different servers to balance workloads and eliminate long-tail latency caused by varying sequence lengths.

For deployment and configuration, see: [SGLang Model Gateway](./sgl_model_gateway)


# SGLang Simulator
Source: https://docs.sglang.io/docs/advanced_features/sglang_simulator



SGLang Simulator reuses SGLang's scheduler, request lifecycle, and KV-cache implementation while replacing model forward execution with a latency predictor. Use it to compare scheduling and cache configurations on timestamped or synthetic workloads without loading model weights.

## Supported scope

SGLang Simulator tracks the current `main` branch and recent SGLang releases. The current integration is validated with `v0.5.16`, `v0.5.17`, `v0.5.18`, and `main`.

The initial upstream scope uses one simulated worker with `tp_size=1`, `ep_size=1`, `dp_size=1`, and `pp_size=1`. A simulator configuration can describe a larger target system for latency prediction, but the SGLang runtime process topology remains single-worker.

The simulator supports:

* synthetic request rates, ShareGPT workloads, and timestamped Autobench traces;
* OFFLINE logical-time simulation and BLOCKING wall-clock replay;
* AIConfigurator, ML, and replay latency predictors;
* SGLang prefix caching and [HiCache](/docs/advanced_features/hicache); and
* serving-compatible TTFT, TPOT, ITL, throughput, and cache-hit metrics.

## Install from the SGLang repository

Use the simulator and SGLang source from the same monorepo checkout:

```bash theme={null}
python3 -m pip install -e tools/sglang-simulator
export PYTHONPATH="$PWD/tools/sglang-simulator/src:$PWD/python"
```

AIConfigurator is optional. Install the validated extra only when you use an AIConfigurator predictor:

```bash theme={null}
python3 -m pip install -e "tools/sglang-simulator[aic]"
```

## Start a simulator server

Choose a fresh output directory for every run. The server owns the simulation mode and writes metrics to this directory.

```bash theme={null}
export SGLANG_USE_CPU_ENGINE=1
export CUDA_VISIBLE_DEVICES=""
export SGLANG_SIMULATOR_OUTPUT_MODE=OFFLINE
export SGLANG_SIMULATOR_OUTPUT_DIR=/tmp/sglang-simulator-quickstart

python3 -m sglang_simulator.simulation.sglang.launch_server \
  --model-path tools/sglang-simulator/test/assets/qwen3-8b \
  --tokenizer-path tools/sglang-simulator/examples/assets/tokenizer \
  --sim-config-path tools/sglang-simulator/examples/sim_configs/replay.json \
  --port 30000
```

`OFFLINE` advances the simulator's logical clock without sleeping. `BLOCKING` also sleeps for predicted forward and cache-load latency, which is useful when a client must observe simulated wall-clock pacing.

## Send a workload

In another terminal, export the same output directory and run the simulator-aware serving benchmark from the repository root:

```bash theme={null}
export PYTHONPATH="$PWD/tools/sglang-simulator/src:$PWD/python"
export SGLANG_SIMULATOR_OUTPUT_DIR=/tmp/sglang-simulator-quickstart

python3 benchmark/simulator/bench_serving.py \
  --simulator-mode offline \
  --backend sglang \
  --base-url http://127.0.0.1:30000 \
  --model tools/sglang-simulator/test/assets/qwen3-8b \
  --tokenizer tools/sglang-simulator/examples/assets/tokenizer \
  --dataset-name sharegpt \
  --dataset-path tools/sglang-simulator/examples/workloads/sharegpt-example.json \
  --sharegpt-output-len 4 \
  --num-prompts 3 \
  --output-file /tmp/sglang-simulator-quickstart/benchmark.json
```

The benchmark injects logical arrival metadata into each request and displays the server-side simulator metrics. For timestamped traffic, use the simulator-owned Autobench JSONL format and add `--use-trace-timestamps`.

## Read the results

The output directory contains:

* `metrics.json`: aggregate latency, throughput, and cache metrics;
* `request.jsonl`: per-request timing and cache information; and
* `iteration.jsonl`: scheduler batch composition and predicted iteration latency.

Use a unique output directory for each run so metrics from separate experiments are not mixed. See the [SGLang Simulator source README](https://github.com/sgl-project/sglang/tree/main/tools/sglang-simulator) for simulator configuration fields, predictor examples, and maintained tests.


# Speculative Decoding
Source: https://docs.sglang.io/docs/advanced_features/speculative_decoding



SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, UNO, DFLASH, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.

## Summary

### Jump to sections

* [EAGLE Decoding](#eagle-decoding)
  * [EAGLE-2 Decoding](#eagle-2-decoding)
  * [EAGLE-2 Decoding with torch.compile](#eagle-2-decoding-with-torch-compile)
  * [EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling](#eagle-2-decoding-via-frequency-ranked-speculative-sampling)
  * [EAGLE-3 Decoding](#eagle-3-decoding)
* [Block verification](#block-verification)
* [Multi Token Prediction](#multi-token-prediction)
* [UNO decoding](#uno-decoding)
* [DFlash Decoding](#dflash-decoding)
* [Standalone Speculative Decoding (Small Draft Model)](#standalone-speculative-decoding-small-draft-model)
* [Speculative Decoding V2 (Overlap Scheduler)](#speculative-decoding-v2-overlap-scheduler)
* [Ngram Speculative Decoding](#ngram-speculative-decoding)
* [Full Parameter Reference](#full-parameter-reference)
* [OOM Troubleshooting](#oom-troubleshooting)
* [References](#references)

### Quick guidance

* **Best speed/quality (recommended)**: Use **EAGLE-3** with `--speculative-algorithm EAGLE3`.
* **Strong default / broad compatibility**: Use **EAGLE-2** with `--speculative-algorithm EAGLE`.
* **Workload acceptance changes over time**: Use [**Adaptive speculative decoding**](./adaptive_speculative_decoding) on top of **EAGLE** with `--speculative-eagle-topk 1`.
* **Lower `lm_head` overhead for EAGLE-2**: Enable **FR-Spec** with `--speculative-token-map`.
* **Model is MTP-enabled**: Use **MTP via speculative decoding** (often with small `speculative_num_steps/topk/num_draft_tokens`, see the example section).
* **You have a UNO adapter for the target model**: Use **UNO** with `--speculative-algorithm UNO` and `--uno-lora-path ...`.
* **You have a DFlash draft checkpoint**: Use **DFLASH** with `--speculative-algorithm DFLASH` and `--speculative-draft-model-path ...`.
* **You have a smaller draft LLM**: Use **STANDALONE** (`--speculative-algorithm STANDALONE`).
* **No extra model available**: Use **NGRAM** (`--speculative-algorithm NGRAM`, CUDA-only).

### Method comparison (mini table)

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Draft source</th>
      <th>Separate draft model?</th>
      <th>How to enable</th>
      <th>Notes / constraints</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>EAGLE-2</td>
      <td>EAGLE draft model (feature drafting + tree)</td>
      <td>Typically yes</td>
      <td>`--speculative-algorithm EAGLE` + `--speculative-draft-model-path ...`</td>
      <td>Tune `--speculative-num-steps`, `--speculative-eagle-topk`, `--speculative-num-draft-tokens`</td>
    </tr>

    <tr>
      <td>EAGLE-2 + `torch.compile`</td>
      <td>Same as EAGLE-2</td>
      <td>Typically yes</td>
      <td>Add `--enable-torch-compile` (optionally `--torch-compile-max-bs`)</td>
      <td>Benefit varies by hardware/model; benchmark to verify</td>
    </tr>

    <tr>
      <td>EAGLE-2 + FR-Spec</td>
      <td>Same as EAGLE-2 + token subset</td>
      <td>Typically yes</td>
      <td>Add `--speculative-token-map ...`</td>
      <td>Reduces `lm_head` overhead with high-frequency token vocab</td>
    </tr>

    <tr>
      <td>EAGLE-3</td>
      <td>EAGLE3 draft model</td>
      <td>Yes</td>
      <td>`--speculative-algorithm EAGLE3` + `--speculative-draft-model-path ...`</td>
      <td>Best throughput in the benchmark below</td>
    </tr>

    <tr>
      <td>MTP</td>
      <td>Built-in multi-token heads (model-specific)</td>
      <td>Often no</td>
      <td>See **Multi Token Prediction** section</td>
      <td>Uses speculative workflow; draft path may be auto-handled for some models</td>
    </tr>

    <tr>
      <td>UNO</td>
      <td>Target model with a trained UNO LoRA (linear chain or tree)</td>
      <td>No</td>
      <td>`--speculative-algorithm UNO` + `--uno-lora-path ...`</td>
      <td>CUDA and FA3; currently requires TP=PP=1</td>
    </tr>

    <tr>
      <td>DFLASH</td>
      <td>DFlash draft model (linear block verification)</td>
      <td>Yes</td>
      <td>`--speculative-algorithm DFLASH` + `--speculative-draft-model-path ...`</td>
      <td>CUDA DP attention requires `--enable-dp-lm-head` and `attn_cp_size == 1`; `pp_size == 1`</td>
    </tr>

    <tr>
      <td>STANDALONE</td>
      <td>Smaller draft LLM (token-level)</td>
      <td>Yes</td>
      <td>`--speculative-algorithm STANDALONE` + `--speculative-draft-model-path ...`</td>
      <td>Does **not** support DP attention (`--attn-dp-size` > 1)</td>
    </tr>

    <tr>
      <td>NGRAM</td>
      <td>Ngram cache from previous tokens</td>
      <td>No</td>
      <td>`--speculative-algorithm NGRAM`</td>
      <td>CUDA-only; no DP attention (`--attn-dp-size` > 1); disables overlap scheduler & mixed chunked prefill</td>
    </tr>
  </tbody>
</table>

### Performance Highlights

Please see below for the huge improvements on throughput for LLaMA-Instruct 3.1 8B tested on MT bench that can be achieved via EAGLE3 decoding.
For further details please see the [EAGLE3 paper](https://arxiv.org/pdf/2503.01840).

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Throughput (tokens/s)</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>SGLang (w/o speculative, 1x H100)</td>
      <td>158.34 tokens/s</td>
    </tr>

    <tr>
      <td>SGLang + EAGLE-2 (1x H100)</td>
      <td>244.10 tokens/s</td>
    </tr>

    <tr>
      <td>SGLang + EAGLE-3 (1x H100)</td>
      <td>373.25 tokens/s</td>
    </tr>
  </tbody>
</table>

***

## EAGLE Decoding

To enable EAGLE speculative decoding the following parameters are relevant:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-draft-model-path`</td>
      <td>Draft model path/weights. **Typically required** for EAGLE/EAGLE3 and STANDALONE. For some MTP-enabled models, this can be omitted.</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`--speculative-num-steps`</td>
      <td>Depth of autoregressive drafting. Increases speculation range but risks rejection cascades.</td>
      <td>Auto (`5` for Llama/Grok; `3` for many other models)</td>
    </tr>

    <tr>
      <td>`--speculative-eagle-topk`</td>
      <td>Branching factor per step. Improves candidate diversity and acceptance rate, but increases memory/compute consumption.</td>
      <td>Auto (`4` for Llama/Grok; `1` for many other models)</td>
    </tr>

    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>Maximum parallel verification capacity. Allows deeper tree evaluation but increases GPU memory usage.</td>
      <td>Auto (`8` for Llama/Grok; `4` for many other models). If `topk=1`, it is adjusted to `num_steps + 1`.</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-single`</td>
      <td>Acceptance threshold for single-token verification. Lower values accept more aggressively.</td>
      <td>`1.0`</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-acc`</td>
      <td>Accumulated acceptance threshold across steps.</td>
      <td>`1.0`</td>
    </tr>

    <tr>
      <td>`--speculative-attention-mode`</td>
      <td>Attention mode for speculative operations (`prefill` or `decode`), affecting both target verification and draft extension.</td>
      <td>`"prefill"`</td>
    </tr>

    <tr>
      <td>`--speculative-draft-attention-backend`</td>
      <td>Override attention backend for the draft model.</td>
      <td>`None` (same as target)</td>
    </tr>

    <tr>
      <td>`--speculative-dsa-topk-backend`</td>
      <td>Select the DSA indexer top-k backend for speculative draft workers independently of `--dsa-topk-backend`. Options are `sgl-kernel`, `torch`, and `flashinfer`; `torch` requires `SGLANG_DSA_FUSE_TOPK=false`.</td>
      <td>`sgl-kernel`</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-quantization`</td>
      <td>Quantization method for the draft model. Use `"unquant"` to force no quantization even when the target model is quantized.</td>
      <td>Same as target model</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-revision`</td>
      <td>Specific revision/commit of the draft model to load.</td>
      <td>`None` (auto-set to `"main"` when `--speculative-draft-model-path` is set and revision is omitted)</td>
    </tr>

    <tr>
      <td>`--speculative-draft-load-format`</td>
      <td>Load format for the draft model weights.</td>
      <td>`None`</td>
    </tr>
  </tbody>
</table>

These parameters are mostly the same for EAGLE-2 and EAGLE-3. `--speculative-token-map` is ignored for EAGLE-3 models.
For `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`: leave all three unset to use auto-tuning, or set all three explicitly when tuning.
If you use EAGLE with `--speculative-eagle-topk 1` and your acceptance rate varies across requests, see [Adaptive Speculative Decoding](./adaptive_speculative_decoding).

You can find the best combinations of these parameters with [bench\_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py).

### EAGLE-2 Decoding

You can enable EAGLE-2 Decoding by setting `--speculative-algorithm EAGLE` and choosing an appropriate model.

**Launch the server:**

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)
```

***

### EAGLE-2 Decoding with `torch.compile`

You can optionally enable `torch.compile` to apply kernel-level optimizations (operator fusion, autotune) to the draft model. The actual speedup depends on your hardware, model architecture, and batch size. In some configurations (e.g., small draft models on H100 where cuBLAS is already optimal and CUDA graphs are enabled), the benefit may be negligible. We recommend benchmarking with and without this flag on your specific setup to verify whether it helps.

To enable it, add `--enable-torch-compile` and optionally set `--torch-compile-max-bs`:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model meta-llama/Llama-2-7b-chat-hf \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --enable-torch-compile \
    --torch-compile-max-bs 8 \
    --log-level warning
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-chat-hf",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)
```

***

### EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling

By employing a truncated high-frequency token vocabulary in the draft model, EAGLE speculative decoding reduces `lm_head` computational overhead while accelerating the pipeline without quality degradation. For more details, check out [the paper](https://arxiv.org/pdf/2502.14856).

In our implementation, set `--speculative-token-map` to enable the optimization. You can get the high-frequency tokens in FR-Spec from [this model](https://huggingface.co/thunlp/LLaMA3-Instruct-8B-FR-Spec). Or you can obtain high-frequency tokens by directly downloading these tokens from [this repo](https://github.com/thunlp/FR-Spec/tree/main?tab=readme-ov-file#prepare-fr-spec-vocabulary-subset).

Thanks for the contribution from [Weilin Zhao](https://github.com/Achazwl) and [Zhousx](https://github.com/Zhou-sx).

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --dtype float16 \
    --log-level warning
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)
```

***

### EAGLE-3 Decoding

You can enable EAGLE-3 decoding by setting `--speculative-algorithm EAGLE3` and choosing an appropriate model.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model meta-llama/Meta-Llama-3.1-8B-Instruct \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 4 \
    --speculative-num-draft-tokens 16 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --dtype float16 \
    --log-level warning
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)
```

***

## Block verification

For stochastic EAGLE, EAGLE3, or NEXTN decoding on NVIDIA CUDA or AMD ROCm, add
`--speculative-use-block-verification`
with `--speculative-eagle-topk 1`. The draft model must produce a full target-vocabulary
proposal distribution, as required by the existing rejection sampler.

The same flag applies to the DFlash family (`DFLASH` and `DSPARK`) on CUDA or ROCm
when the draft samples its proposals: DSpark drafts, DFlash checkpoints with a
candidate selector, and [LiLiCorr](#lilicorr-reranked-dflash-drafts) drafts with
`SGLANG_ENABLE_LILICORR_SAMPLING=1`. These paths already publish their proposal
distribution, so `--speculative-use-rejection-sampling` is not needed. A DFlash draft
that proposes greedy tokens is a point mass, where block and token-wise verification
accept the same prefixes; the server warns that the flag has no effect.

[Block verification](https://arxiv.org/abs/2403.10444) checks every draft prefix and
keeps the longest accepted prefix, including when an earlier prefix was rejected.
It uses a corresponding correction distribution to preserve target sampling.
Use `--speculative-use-rejection-sampling` alone for token-wise verification, which
stops at the first rejected token.
Greedy requests keep their existing verification behavior.

Block verification adds vocabulary reductions during verification. Benchmark your
model and workload to measure the effect on latency and throughput.

## Multi Token Prediction

We support [MTP (Multi-Token Prediction)](https://arxiv.org/pdf/2404.19737) in SGLang by using speculative decoding. We use `XiaomiMiMo/MiMo-7B-RL` as an example here (for DeepSeek MTP usage, refer to [DeepSeek-V3.2 cookbook §4.2.3](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2#4-2-3-multi-token-prediction-eagle-speculative-decoding)).

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model XiaomiMiMo/MiMo-7B-RL \
    --host 0.0.0.0 \
    --trust-remote-code \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 1 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 2 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning
```

For DSA-based MTP, draft workers default to the `sgl-kernel` top-k backend. Use `--speculative-dsa-topk-backend` to override the draft independently of `--dsa-topk-backend` for the target model.

**Send a request:**

```python Example theme={null}
import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "XiaomiMiMo/MiMo-7B-RL",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
}

response = requests.post(url, json=data)
print(response.json())
```

***

## UNO decoding

UNO reuses the target transformer for both passes of each speculative decode cycle instead of loading a separate draft model. During the draft forward, the first row of each request uses the base weights and the remaining `B - 1` rows use a trained UNO LoRA. Target verification is entirely base-only.

UNO provides two sampling modes:

* **Linear** constructs and verifies one chain of draft tokens, similar to DFlash's proposal layout.
* **Tree** expands multiple candidates at each draft depth and uses SGLang's EAGLE tree-verification path.

For UNO modes, the three numbers are `B/K/V`: draft-forward width, candidates kept per expansion, and verification width. Thus, `Linear 8/1/8` uses an 8-token draft width, one candidate per expansion, and an 8-token verification width. UNO's `K` controls proposal-tree breadth and is unrelated to the request sampling parameter `top_k`. The command-line mapping depends on the mode:

* In linear mode, `B` and `V` are both `--speculative-num-draft-tokens`; set `--speculative-num-steps 1` and `--speculative-eagle-topk 1`.
* In tree mode, `B` is `--speculative-num-steps + 1`, `K` is `--speculative-eagle-topk`, and `V` is `--speculative-num-draft-tokens`.

Use an adapter trained for the exact target-model checkpoint. The current integration has been validated with `Qwen/Qwen3-8B`; unsupported LoRA target layers are rejected at startup. Set the adapter's local path or Hugging Face repository before starting the server:

### Linear 8/1/8

```bash Command theme={null}
export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B"

sglang serve \
    --model-path Qwen/Qwen3-8B \
    --speculative-algorithm UNO \
    --uno-lora-path "$UNO_LORA_PATH" \
    --speculative-num-steps 1 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 8 \
    --attention-backend fa3 \
    --tp 1 \
    --host 0.0.0.0 \
    --port 30000
```

### Tree 8/32/8

The following command selects tree mode because `K` is greater than one:

```bash Command theme={null}
export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B"

sglang serve \
    --model-path Qwen/Qwen3-8B \
    --speculative-algorithm UNO \
    --uno-lora-path "$UNO_LORA_PATH" \
    --speculative-num-steps 7 \
    --speculative-eagle-topk 32 \
    --speculative-num-draft-tokens 8 \
    --attention-backend fa3 \
    --tp 1 \
    --host 0.0.0.0 \
    --port 30000
```

Send requests through the standard OpenAI-compatible API:

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Explain speculative decoding."}],
    max_tokens=128,
)

print(response.choices[0].message.content)
```

### Key requirements and limitations

* UNO requires CUDA with FA3 for prefill and decode, tensor and pipeline parallel sizes of 1, and no DP attention or context parallelism.
* `--uno-lora-path` loads UNO's fixed internal adapter and cannot be combined with request-selectable Multi-LoRA serving.
* UNO manages its own stochastic verification; do not set `--speculative-use-rejection-sampling`.
* Grammar decoding, returned logprobs or hidden states, sampling penalties, `min_p`, logit bias, custom logit processors, strict thinking, and deterministic inference are not yet supported.
* In tree mode, both speculative acceptance thresholds must remain at 1.0, `V` must be at least `B` and at most 128, and `V * K` must not exceed 2048. SGLang validates the remaining tree-capacity and EAGLE parent-representation constraints at startup.
* Mixed chunked prefill is disabled for UNO.
* Ordinary overlap scheduling is supported. Tree mode does not yet support PDMux or the separate `--enable-two-batch-overlap` feature.

***

## DFlash Decoding

SGLang also supports **DFLASH** speculative decoding using a dedicated draft model checkpoint. Compared with EAGLE-style tree verification, DFLASH verifies a linear draft block and is configured around a block size / draft window. This path is useful when the target model has a matching DFlash draft checkpoint, such as `meta-llama/Llama-3.1-8B-Instruct` with `z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat`.

Relevant parameters:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-draft-model-path`</td>
      <td>Required DFlash draft model path/weights.</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>DFlash verify block size.</td>
      <td>Inferred from draft config, otherwise `16`</td>
    </tr>

    <tr>
      <td>`--speculative-dflash-block-size`</td>
      <td>Alias of `--speculative-num-draft-tokens` for DFlash.</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`--speculative-draft-window-size`</td>
      <td>Draft KV sliding-window size. Must be `>= speculative-num-draft-tokens` when set.</td>
      <td>`None`</td>
    </tr>
  </tbody>
</table>

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Write a quicksort implementation in Python."},
    ],
    temperature=0,
    max_tokens=128,
)

print(response.choices[0].message.content)
```

### DFlash and DFlash2 with DP attention

On NVIDIA CUDA, you can use attention DP with plain DFlash drafts and DFlash2
candidate-selector checkpoints. Both use `--speculative-algorithm DFLASH`.
Set `--enable-dp-lm-head` so the target head has the same vocabulary shards as
its local draft replica. Pipeline parallelism and DP combined with context
parallelism are not supported. Domino and LiLiCorr heads are not supported
with CUDA DP attention.

For example, on four NVIDIA GPUs, use TP4 with attention DP2 / TP2:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-4B \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/Qwen3-4B-DFlash-b16 \
    --tp-size 4 \
    --attn-dp-size 2 \
    --enable-dp-lm-head \
    --attention-backend flashinfer \
    --speculative-draft-attention-backend flashinfer \
    --cuda-graph-backend-prefill disabled
```

For DFlash2, use a matching target and selector checkpoint, such as
`Qwen/Qwen3.8-27B` and `incoai/Qwen3.8-27B-DFlash2`.
Each attention-DP group runs its draft within the local attention-TP group.
Draft execution stays eager; eligible target verify batches can use CUDA graphs.
The scheduler separates prefill and verify rounds across DP ranks. Keep the
experimental `SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION` setting disabled.

### LiLiCorr: reranked DFlash drafts

A DFlash drafter is trained on per-position marginals rather than on the joint block distribution, so its drafted tokens are individually plausible yet jointly incoherent. **LiLiCorr** ([arXiv:2608.20530](https://arxiv.org/abs/2608.20530)) keeps the top-`k` candidates per block position, scores the whole lattice jointly with a small two-layer transformer, and commits a path through it greedily instead of taking the per-slot argmax. Verify is unchanged, so outputs remain distributionally identical to the target model's and only acceptance length moves.

LiLiCorr is **not a separate `--speculative-algorithm`**: it is a property of the draft checkpoint, the same way the DFlash candidate selector is. Serve it with `--speculative-algorithm DFLASH` and a checkpoint whose config declares `architectures: ["LiLiCorrDraftModel"]` and carries the `lilicorr_*` geometry fields inside `dflash_config`. The head is then built, loaded and folded into the draft CUDA graph automatically.

No LiLiCorr drafter is published yet, so the draft path below is a placeholder.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model Qwen/Qwen3-8B \
    --attention-backend fa3 \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path <your-lilicorr-draft-model>
```

**Use a backend that opts out of the host `seq_lens` mirror.** `AttentionBackend.needs_cpu_seq_lens` defaults to `True`; `fa3`, `triton`, `trtllm_mha`, `trtllm_mla`, `flashmla` and `dsa` set it `False`, while `FlashInferAttnBackend` inherits the default. Under a backend that does not opt out the worker publishes `seq_lens_cpu` every step, which is a device-to-host sync per block. The flag is OR-ed over the target and draft runners, so both have to opt out. This is a property of DFlash decoding rather than of LiLiCorr, but it dominates the throughput of a 16-slot block. Backends also differ in attention numerics, so acceptance measured on different backends does not belong in one table.

A checkpoint carrying head weights but declaring `architectures: ["DFlashDraftModel"]` loads as plain DFlash and ignores the head, which reads as a small acceptance loss rather than an error. The reverse, declaring `LiLiCorrDraftModel` without complete head weights, is refused at load.

#### Sampling the committed path

Above temperature 0 the target samples, but a deterministic draft is paid `p(argmax)` per slot because verify is handed a point mass. `SGLANG_ENABLE_LILICORR_SAMPLING=1` draws each slot from `softmax(scores / T)` over that slot's candidates at the request's own temperature and publishes the proposal it drew from, so rejection sampling is paid the overlap of the two distributions instead. Verify stays exact. Greedy requests sharing the batch take the argmax and report a one-hot proposal.

Off by default; with it unset the captured body is `select` itself, so temperature-0 output is unchanged bit for bit. On Qwen3-8B it is worth about 6% acceptance and 6% end-to-end at temperature 1, at under 1% of the per-block wall.

**Only enable it on the folded path.** The eager fallback that serves the steps the draft CUDA graph cannot is correct but costs a large fraction of throughput, and because acceptance is identical either way that regression shows up only in tokens/s. `build_lilicorr_draft_sampler` warns with a reason when it declines to fold, and the worker warns once if a decode step lands on the eager head.

#### Drafters with grouped convolutions

LiLiCorr composes with the grouped dynamic convolutions that wrap each DFlash sublayer, and serving one needs no extra flag: the convolution belongs to the DFlash backbone, so `LiLiCorrDraftModel` inherits the builder and picks it up from `conv_kernel_size` and `conv_group_size` in `dflash_config`.

Both keys default to `0`, so a checkpoint whose config lost them builds no convolution modules and the backbone loader then drops every convolution tensor it cannot resolve. That is correct for rotary caches and the worst case here, because the draft would serve as its convolution-free parent at a lower but entirely believable acceptance length. The loader therefore refuses any checkpoint whose convolution tensors and built modules disagree, in either direction.

***

## Standalone Speculative Decoding (Small Draft Model)

Besides EAGLE/MTP, SGLang also supports **token-level speculative decoding** using a smaller **draft model**. Enable it with `--speculative-algorithm STANDALONE` and provide a draft model via `--speculative-draft-model-path`.

Relevant parameters:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-draft-model-path`</td>
      <td>Draft model weights (smaller than the target model).</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`--speculative-num-steps`</td>
      <td>Draft depth (how many steps the draft model runs autoregressively).</td>
      <td>`3` (auto default for STANDALONE)</td>
    </tr>

    <tr>
      <td>`--speculative-eagle-topk`</td>
      <td>Branching factor (token candidates per step).</td>
      <td>`1` (auto default for STANDALONE)</td>
    </tr>

    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>Verification capacity.</td>
      <td>`4` (auto default for STANDALONE)</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-quantization`</td>
      <td>Quantization for the draft model. Use `"unquant"` to disable quantization on the draft even when the target is quantized.</td>
      <td>Same as target</td>
    </tr>
  </tbody>
</table>

> **Note:** Standalone speculative decoding currently **does not support** DP attention (`--attn-dp-size` > 1).

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --speculative-algorithm STANDALONE \
    --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 2 \
    --speculative-num-draft-tokens 7 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)
```

***

## Speculative Decoding V2 (Overlap Scheduler)

Speculative decoding runs the V2 speculative workers (e.g. `StandaloneWorkerV2`, `EAGLEWorkerV2`) with the overlap scheduler enabled by default. Pass `--disable-overlap-schedule` to fall back to the synchronous (non-overlap) path.

Notes:

* The overlap scheduler currently only supports `--speculative-eagle-topk 1`; **set `--speculative-eagle-topk 1` explicitly**.
* If you explicitly set `--speculative-eagle-topk > 1`, the server will error.
* If you omit `--speculative-eagle-topk`, auto-tuning may pick `topk > 1` for some models (e.g. Llama). This is incompatible with the overlap scheduler and may not always trigger an immediate config error, so set `--speculative-eagle-topk 1` explicitly.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --speculative-algorithm STANDALONE \
    --speculative-draft-model-path Qwen/Qwen2.5-1.5B-Instruct \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)
```

***

## Ngram Speculative Decoding

SGLang also supports **ngram-based speculative decoding** (no separate draft model). It retrieves draft tokens from an ngram cache built from previously generated tokens, and then verifies them with the target model.

Enable it with:

* `--speculative-algorithm NGRAM`

### Ngram-specific parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>Number of draft tokens verified per step. If omitted, defaults to `min(--speculative-ngram-max-trie-depth, 12)`.</td>
      <td>`12` (with default ngram settings)</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-min-bfs-breadth`</td>
      <td>Minimum BFS breadth.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-bfs-breadth`</td>
      <td>Maximum BFS breadth.</td>
      <td>`10`</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-match-type`</td>
      <td>Ngram tree-building mode: `"BFS"` for recency-based expansion or `"PROB"` for frequency-based expansion.</td>
      <td>`"BFS"`</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-trie-depth`</td>
      <td>Maximum suffix length stored and matched by the ngram trie.</td>
      <td>`18`</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-capacity`</td>
      <td>Cache capacity (number of entries).</td>
      <td>`10,000,000`</td>
    </tr>
  </tbody>
</table>

Notes:

* Ngram speculative decoding **only supports CUDA**.
* It currently **does not support** DP attention (`--attn-dp-size` > 1).
* It disables the overlap scheduler and mixed chunked prefill.
* If `--speculative-ngram-max-bfs-breadth > 1` (thus `speculative_eagle_topk > 1`) and `page_size > 1`, use `--attention-backend flashinfer`; otherwise the server will error.
* Optional: set `SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True` to force greedy verification.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model Qwen/Qwen2.5-7B-Instruct \
    --speculative-algorithm NGRAM \
    --speculative-num-draft-tokens 16 \
    --speculative-ngram-max-bfs-breadth 10 \
    --mem-fraction-static 0.7 \
    --cuda-graph-max-bs-decode 8 \
    --log-level warning
```

**Send a request:**

```python Example theme={null}
import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print(response.choices[0].message.content)
```

***

## Full Parameter Reference

Below is a comprehensive list of all speculative decoding parameters available in SGLang:

### Core parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-algorithm`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>Algorithm to use: `UNO`, `DFLASH`, `EAGLE`, `EAGLE3`, `STANDALONE`, `NGRAM`, `NEXTN` (alias of `EAGLE`)</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-path`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>Path to the draft model weights</td>
    </tr>

    <tr>
      <td>`--uno-lora-path`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>UNO-only path or Hugging Face repository for the draft LoRA checkpoint</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-revision`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>Specific revision/commit of the draft model (`"main"` is auto-used when draft path is set and revision is omitted)</td>
    </tr>

    <tr>
      <td>`--speculative-draft-load-format`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>Load format for draft model weights</td>
    </tr>

    <tr>
      <td>`--speculative-num-steps`</td>
      <td>`int`</td>
      <td>`None` (auto-chosen when omitted)</td>
      <td>Autoregressive drafting depth. Some algorithms auto-tune it; tree UNO requires it and uses `B = speculative_num_steps + 1`.</td>
    </tr>

    <tr>
      <td>`--speculative-eagle-topk`</td>
      <td>`int`</td>
      <td>`None` (auto-chosen when omitted)</td>
      <td>Branching factor per drafting step. Some algorithms auto-tune it; UNO resolves an omitted value to `1`, and values greater than one select UNO tree mode.</td>
    </tr>

    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>`int`</td>
      <td>`None` (auto-chosen when omitted)</td>
      <td>Maximum number of draft tokens for verification. Some algorithms auto-tune it; UNO requires it and uses it as linear width or tree verification width `V`.</td>
    </tr>

    <tr>
      <td>`--speculative-dflash-block-size`</td>
      <td>`int`</td>
      <td>`None`</td>
      <td>DFlash-only alias of `--speculative-num-draft-tokens`</td>
    </tr>

    <tr>
      <td>`--speculative-draft-window-size`</td>
      <td>`int`</td>
      <td>`None`</td>
      <td>DFlash-only draft KV sliding-window size</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-single`</td>
      <td>`float`</td>
      <td>`1.0`</td>
      <td>Single-token acceptance threshold</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-acc`</td>
      <td>`float`</td>
      <td>`1.0`</td>
      <td>Accumulated acceptance threshold</td>
    </tr>

    <tr>
      <td>`--speculative-token-map`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>Path to FR-Spec high-frequency token map</td>
    </tr>

    <tr>
      <td>`--speculative-attention-mode`</td>
      <td>`str`</td>
      <td>`"prefill"`</td>
      <td>Attention mode for speculative operations (`"prefill"` or `"decode"`)</td>
    </tr>

    <tr>
      <td>`--speculative-draft-attention-backend`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>Override attention backend for the draft model</td>
    </tr>

    <tr>
      <td>`--speculative-dsa-topk-backend`</td>
      <td>`str`</td>
      <td>`sgl-kernel`</td>
      <td>DSA indexer top-k backend for speculative draft workers, independent of `--dsa-topk-backend` (`sgl-kernel`, `torch`, or `flashinfer`)</td>
    </tr>

    <tr>
      <td>`--speculative-moe-runner-backend`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>MoE runner backend for the draft model</td>
    </tr>

    <tr>
      <td>`--speculative-moe-a2a-backend`</td>
      <td>`str`</td>
      <td>`None`</td>
      <td>MoE all-to-all backend for the draft model</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-quantization`</td>
      <td>`str`</td>
      <td>Same as target</td>
      <td>Quantization for the draft model (`"unquant"` to disable)</td>
    </tr>
  </tbody>
</table>

### Ngram-specific parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-ngram-min-bfs-breadth`</td>
      <td>`int`</td>
      <td>`1`</td>
      <td>Minimum BFS breadth</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-bfs-breadth`</td>
      <td>`int`</td>
      <td>`10`</td>
      <td>Maximum BFS breadth</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-match-type`</td>
      <td>`str`</td>
      <td>`"BFS"`</td>
      <td>Ngram tree-building mode: `"BFS"` for recency-based expansion or `"PROB"` for frequency-based expansion</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-trie-depth`</td>
      <td>`int`</td>
      <td>`18`</td>
      <td>Maximum suffix length stored and matched by the ngram trie</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-capacity`</td>
      <td>`int`</td>
      <td>`10,000,000`</td>
      <td>Cache capacity</td>
    </tr>
  </tbody>
</table>

### Environment variables

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_NGRAM_FORCE_GREEDY_VERIFY`</td>
      <td>`False`</td>
      <td>Force greedy verification for ngram decoding</td>
    </tr>
  </tbody>
</table>

### Other related flags

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-multi-layer-eagle`</td>
      <td>Enable multi-layer EAGLE (auto-enabled for MiMoV2 and Step3p5 models)</td>
    </tr>

    <tr>
      <td>`--enable-torch-compile`</td>
      <td>Enable `torch.compile` for kernel-level optimizations</td>
    </tr>

    <tr>
      <td>`--torch-compile-max-bs`</td>
      <td>Maximum batch size for `torch.compile`</td>
    </tr>
  </tbody>
</table>

***

## OOM Troubleshooting

> [!WARNING]
>
> **Out of Memory (OOM)?** Speculative decoding may increase GPU memory usage because the draft tree, CUDA graphs, and verification-related buffers consume additional VRAM. If you encounter OOM errors, try the following adjustments.

### Step 1: Lower static memory fraction (most effective)

```bash Command theme={null}
--mem-fraction-static 0.5   # when omitted, this value is auto-computed
```

* `--mem-fraction-static` controls the memory budget for model weights + KV cache pool.
* Lowering it directly increases dynamic headroom for activations and CUDA graph buffers.
* If omitted, SGLang auto-estimates this value from other settings, and those auto settings can still be too aggressive for some workloads.

### Step 2: Reduce CUDA graph batch size

```bash Command theme={null}
# Fewer CUDA graph captures = less memory reserved
--cuda-graph-max-bs-decode 4   # or even 2 for tight memory situations
```

* If omitted, `--cuda-graph-max-bs-decode` is auto-selected based on GPU memory and TP size, and can be much larger on high-memory GPUs.

### Step 3: Reduce draft tree size

These three parameters directly control how much memory the draft tree consumes:

```bash Command theme={null}
# Before (aggressive, high memory)
--speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64

# After (conservative, lower memory)
--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
```

### Step 4: Limit concurrent requests

```bash Command theme={null}
# Fewer concurrent requests lowers in-flight load and can reduce OOM risk
--max-running-requests 4
```

### Quick OOM recovery recipe

If you're hitting OOM and just want something that works, start with this minimal configuration and scale up:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model <your-model> \
    --speculative-algorithm EAGLE \
    --speculative-draft-model-path <your-draft-model> \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --cuda-graph-max-bs-decode 2 \
    --mem-fraction-static 0.5 \
    --max-running-requests 4 \
    --log-level warning
```

Then gradually increase `--speculative-num-draft-tokens`, `--speculative-eagle-topk`, and `--cuda-graph-max-bs-decode`. Increase `--mem-fraction-static` last, only after the run is stable.

***

## References

EAGLE process is as follows:

* Within EAGLE the draft model predicts the next feature vector, i.e. the last hidden state of the original LLM, using the feature sequence $(f_1, ..., f_k)$ and the token sequence $(t_2, ..., t_&#123;k+1&#125;)$.
* The next token is then sampled from $p_&#123;k+2&#125;=\text&#123;LMHead&#125;(f_&#123;k+1&#125;)$. Afterwards, the two sequences are extended in a tree style—branching out multiple potential continuations, with the branching factor per step controlled by the `speculative_eagle_topk` parameter—to ensure a more coherent connection of context, and are given as input again.
* In SGLang's EAGLE-2 implementation, the draft tree is expanded for the configured steps and then reranked to select the top `speculative_num_draft_tokens` final nodes as draft tokens.
* EAGLE-3 removes the feature prediction objective, incorporates low and mid-layer features, and is trained in an on-policy manner.

This enhances drafting accuracy by operating on features instead of tokens for more regular inputs and by additionally passing tokens from the next timestep to reduce sampling randomness. For more details, see the [EAGLE-2](https://arxiv.org/abs/2406.16858) and [EAGLE-3](https://arxiv.org/abs/2503.01840) papers.

For guidance on how to train your own EAGLE model please see the [EAGLE repo](https://github.com/SafeAILab/EAGLE/tree/main?tab=readme-ov-file#train). For EAGLE-3 training specifically, check out [SpecForge](https://github.com/sgl-project/SpecForge), the SGLang team's training framework designed for EAGLE-3 speculative decoding models with seamless porting to SGLang serving. See the [SpecForge documentation](https://docs.sglang.ai/SpecForge/) and [blog post](https://lmsys.org/blog/2025-07-25-spec-forge) for details.


# Structured Outputs
Source: https://docs.sglang.io/docs/advanced_features/structured_outputs



You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.

SGLang supports three grammar backends:

* [XGrammar](https://github.com/mlc-ai/xgrammar)(default): Supports JSON schema, regular expression, and EBNF constraints.
* [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.
* [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints.

We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).

To use Outlines, simply add `--grammar-backend outlines` when launching the server.
To use llguidance, add `--grammar-backend llguidance`  when launching the server.
If no backend is specified, XGrammar will be used as the default.

For better output quality, **It's advisable to explicitly include instructions in the prompt to guide the model to generate the desired format.** For example, you can specify, 'Please generate the output in the following JSON format: ...'.

## OpenAI Compatible API

```python Example theme={null}
import openai
import os

from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

os.environ["TOKENIZERS_PARALLELISM"] = "false"


server_process, port = launch_server_cmd(
    "python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}")
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
```

### JSON

you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response.

**Using Pydantic**

```python Example theme={null}
from pydantic import BaseModel, Field


# Define the schema using Pydantic
class CapitalInfo(BaseModel):
    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
    population: int = Field(..., description="Population of the capital city")


response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Please generate the information of the capital of France in the JSON format.",
        },
    ],
    temperature=0,
    max_tokens=128,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "foo",
            # convert the pydantic model to json schema
            "schema": CapitalInfo.model_json_schema(),
        },
    },
)

response_content = response.choices[0].message.content
# validate the JSON response by the pydantic model
capital_info = CapitalInfo.model_validate_json(response_content)
print_highlight(f"Validated response: {capital_info.model_dump_json()}")
```

**JSON Schema Directly**

```python Example theme={null}
import json

json_schema = json.dumps(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "pattern": "^[\\w]+$"},
            "population": {"type": "integer"},
        },
        "required": ["name", "population"],
    }
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Give me the information of the capital of France in the JSON format.",
        },
    ],
    temperature=0,
    max_tokens=128,
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "foo", "schema": json.loads(json_schema)},
    },
)

print_highlight(response.choices[0].message.content)
```

### EBNF

```python Example theme={null}
ebnf_grammar = """
root ::= city | description
city ::= "London" | "Paris" | "Berlin" | "Rome"
description ::= city " is " status
status ::= "the capital of " country
country ::= "England" | "France" | "Germany" | "Italy"
"""

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful geography bot."},
        {
            "role": "user",
            "content": "Give me the information of the capital of France.",
        },
    ],
    temperature=0,
    max_tokens=32,
    extra_body={"ebnf": ebnf_grammar},
)

print_highlight(response.choices[0].message.content)
```

### Regular expression

```python Example theme={null}
response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "What is the capital of France?"},
    ],
    temperature=0,
    max_tokens=128,
    extra_body={"regex": "(Paris|London)"},
)

print_highlight(response.choices[0].message.content)
```

### Structural Tag

```python Example theme={null}
tool_get_current_weather = {
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {
                    "type": "string",
                    "description": "The city to find the weather for, e.g. 'San Francisco'",
                },
                "state": {
                    "type": "string",
                    "description": "the two-letter abbreviation for the state that the city is"
                    " in, e.g. 'CA' which would mean 'California'",
                },
                "unit": {
                    "type": "string",
                    "description": "The unit to fetch the temperature in",
                    "enum": ["celsius", "fahrenheit"],
                },
            },
            "required": ["city", "state", "unit"],
        },
    },
}

tool_get_current_date = {
    "type": "function",
    "function": {
        "name": "get_current_date",
        "description": "Get the current date and time for a given timezone",
        "parameters": {
            "type": "object",
            "properties": {
                "timezone": {
                    "type": "string",
                    "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'",
                }
            },
            "required": ["timezone"],
        },
    },
}

schema_get_current_weather = tool_get_current_weather["function"]["parameters"]
schema_get_current_date = tool_get_current_date["function"]["parameters"]


def get_messages():
    return [
        {
            "role": "system",
            "content": f"""
# Tool Instructions
- Always execute python code in messages that you share.
- When looking for real time information use relevant functions if available else fallback to brave_search
You have access to the following functions:
Use the function 'get_current_weather' to: Get the current weather in a given location
{tool_get_current_weather["function"]}
Use the function 'get_current_date' to: Get the current date and time for a given timezone
{tool_get_current_date["function"]}
If a you choose to call a function ONLY reply in the following format:
<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}
where
start_tag => `<function`
parameters => a JSON dict with the function argument name as key and function argument value as value.
end_tag => `</function>`
Here is an example,
<function=example_function_name>{{"example_name": "example_value"}}</function>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- Always add your sources when using search results to answer the user query
You are a helpful assistant.""",
        },
        {
            "role": "user",
            "content": "You are in New York. Please get the current date and time, and the weather.",
        },
    ]


messages = get_messages()

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=messages,
    response_format={
        "type": "structural_tag",
        "structures": [
            {
                "begin": "<function=get_current_weather>",
                "schema": schema_get_current_weather,
                "end": "</function>",
            },
            {
                "begin": "<function=get_current_date>",
                "schema": schema_get_current_date,
                "end": "</function>",
            },
        ],
        "triggers": ["<function="],
    },
)

print_highlight(response.choices[0].message.content)
```

```python Example theme={null}
# Support for XGrammar latest structural tag format
# https://xgrammar.mlc.ai/docs/api/python/structural_tag.html

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-8B-Instruct",
    messages=messages,
    response_format={
        "type": "structural_tag",
        "format": {
            "type": "triggered_tags",
            "triggers": ["<function="],
            "tags": [
                {
                    "begin": "<function=get_current_weather>",
                    "content": {
                        "type": "json_schema",
                        "json_schema": schema_get_current_weather,
                    },
                    "end": "</function>",
                },
                {
                    "begin": "<function=get_current_date>",
                    "content": {
                        "type": "json_schema",
                        "json_schema": schema_get_current_date,
                    },
                    "end": "</function>",
                },
            ],
            "at_least_one": False,
            "stop_after_first": False,
        },
    },
)

print_highlight(response.choices[0].message.content)
```

## Native API and SGLang Runtime (SRT)

### JSON

**Using Pydantic**

```python Example theme={null}
import requests
import json
from pydantic import BaseModel, Field

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")


# Define the schema using Pydantic
class CapitalInfo(BaseModel):
    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
    population: int = Field(..., description="Population of the capital city")


# Make API request
messages = [
    {
        "role": "user",
        "content": "Here is the information of the capital of France in the JSON format.\n",
    }
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": text,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 64,
            "json_schema": json.dumps(CapitalInfo.model_json_schema()),
        },
    },
)
print_highlight(response.json())


response_data = json.loads(response.json()["text"])
# validate the response by the pydantic model
capital_info = CapitalInfo.model_validate(response_data)
print_highlight(f"Validated response: {capital_info.model_dump_json()}")
```

**JSON Schema Directly**

```python Example theme={null}
json_schema = json.dumps(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "pattern": "^[\\w]+$"},
            "population": {"type": "integer"},
        },
        "required": ["name", "population"],
    }
)

# JSON
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": text,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 64,
            "json_schema": json_schema,
        },
    },
)

print_highlight(response.json())
```

### EBNF

```python Example theme={null}
messages = [
    {
        "role": "user",
        "content": "Give me the information of the capital of France.",
    }
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": text,
        "sampling_params": {
            "max_new_tokens": 128,
            "temperature": 0,
            "n": 3,
            "ebnf": (
                "root ::= city | description\n"
                'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
                'description ::= city " is " status\n'
                'status ::= "the capital of " country\n'
                'country ::= "England" | "France" | "Germany" | "Italy"'
            ),
        },
        "stream": False,
        "return_logprob": False,
    },
)

print_highlight(response.json())
```

### Regular expression

```python Example theme={null}
messages = [
    {
        "role": "user",
        "content": "Paris is the capital of",
    }
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": text,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 64,
            "regex": "(France|England)",
        },
    },
)
print_highlight(response.json())
```

### Structural Tag

```python Example theme={null}
from transformers import AutoTokenizer

# generate an answer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B-Instruct")

text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
payload = {
    "text": text,
    "sampling_params": {
        "structural_tag": json.dumps(
            {
                "type": "structural_tag",
                "structures": [
                    {
                        "begin": "<function=get_current_weather>",
                        "schema": schema_get_current_weather,
                        "end": "</function>",
                    },
                    {
                        "begin": "<function=get_current_date>",
                        "schema": schema_get_current_date,
                        "end": "</function>",
                    },
                ],
                "triggers": ["<function="],
            }
        )
    },
}


# Send POST request to the API endpoint
response = requests.post(f"http://localhost:{port}/generate", json=payload)
print_highlight(response.json())
```

```python Example theme={null}
# Support for XGrammar latest structural tag format
# https://xgrammar.mlc.ai/docs/api/python/structural_tag.html

payload = {
    "text": text,
    "sampling_params": {
        "structural_tag": json.dumps(
            {
                "type": "structural_tag",
                "format": {
                    "type": "triggered_tags",
                    "triggers": ["<function="],
                    "tags": [
                        {
                            "begin": "<function=get_current_weather>",
                            "content": {
                                "type": "json_schema",
                                "json_schema": schema_get_current_weather,
                            },
                            "end": "</function>",
                        },
                        {
                            "begin": "<function=get_current_date>",
                            "content": {
                                "type": "json_schema",
                                "json_schema": schema_get_current_date,
                            },
                            "end": "</function>",
                        },
                    ],
                    "at_least_one": False,
                    "stop_after_first": False,
                },
            }
        )
    },
}


# Send POST request to the API endpoint
response = requests.post(f"http://localhost:{port}/generate", json=payload)
print_highlight(response.json())
```

```python Example theme={null}
terminate_process(server_process)
```

## Offline Engine API

```python Example theme={null}
import sglang as sgl

llm = sgl.Engine(
    model_path="meta-llama/Meta-Llama-3.1-8B-Instruct", grammar_backend="xgrammar"
)
```

### JSON

**Using Pydantic**

```python Example theme={null}
import json
from pydantic import BaseModel, Field


prompts = [
    "Give me the information of the capital of China in the JSON format.",
    "Give me the information of the capital of France in the JSON format.",
    "Give me the information of the capital of Ireland in the JSON format.",
]


# Define the schema using Pydantic
class CapitalInfo(BaseModel):
    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
    population: int = Field(..., description="Population of the capital city")


sampling_params = {
    "temperature": 0.1,
    "top_p": 0.95,
    "json_schema": json.dumps(CapitalInfo.model_json_schema()),
}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print_highlight("===============================")
    print_highlight(f"Prompt: {prompt}")  # validate the output by the pydantic model
    capital_info = CapitalInfo.model_validate_json(output["text"])
    print_highlight(f"Validated output: {capital_info.model_dump_json()}")
```

**JSON Schema Directly**

```python Example theme={null}
prompts = [
    "Give me the information of the capital of China in the JSON format.",
    "Give me the information of the capital of France in the JSON format.",
    "Give me the information of the capital of Ireland in the JSON format.",
]

json_schema = json.dumps(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "pattern": "^[\\w]+$"},
            "population": {"type": "integer"},
        },
        "required": ["name", "population"],
    }
)

sampling_params = {"temperature": 0.1, "top_p": 0.95, "json_schema": json_schema}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print_highlight("===============================")
    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

### EBNF

```python Example theme={null}
prompts = [
    "Give me the information of the capital of France.",
    "Give me the information of the capital of Germany.",
    "Give me the information of the capital of Italy.",
]

sampling_params = {
    "temperature": 0.8,
    "top_p": 0.95,
    "ebnf": (
        "root ::= city | description\n"
        'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
        'description ::= city " is " status\n'
        'status ::= "the capital of " country\n'
        'country ::= "England" | "France" | "Germany" | "Italy"'
    ),
}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print_highlight("===============================")
    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

### Regular expression

```python Example theme={null}
prompts = [
    "Please provide information about London as a major global city:",
    "Please provide information about Paris as a major global city:",
]

sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print_highlight("===============================")
    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

### Structural Tag

```python Example theme={null}
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
prompts = [text]


sampling_params = {
    "temperature": 0.8,
    "top_p": 0.95,
    "structural_tag": json.dumps(
        {
            "type": "structural_tag",
            "structures": [
                {
                    "begin": "<function=get_current_weather>",
                    "schema": schema_get_current_weather,
                    "end": "</function>",
                },
                {
                    "begin": "<function=get_current_date>",
                    "schema": schema_get_current_date,
                    "end": "</function>",
                },
            ],
            "triggers": ["<function="],
        }
    ),
}


# Send POST request to the API endpoint
outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print_highlight("===============================")
    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

```python Example theme={null}
# Support for XGrammar latest structural tag format
# https://xgrammar.mlc.ai/docs/api/python/structural_tag.html

sampling_params = {
    "temperature": 0.8,
    "top_p": 0.95,
    "structural_tag": json.dumps(
        {
            "type": "structural_tag",
            "format": {
                "type": "triggered_tags",
                "triggers": ["<function="],
                "tags": [
                    {
                        "begin": "<function=get_current_weather>",
                        "content": {
                            "type": "json_schema",
                            "json_schema": schema_get_current_weather,
                        },
                        "end": "</function>",
                    },
                    {
                        "begin": "<function=get_current_date>",
                        "content": {
                            "type": "json_schema",
                            "json_schema": schema_get_current_date,
                        },
                        "end": "</function>",
                    },
                ],
                "at_least_one": False,
                "stop_after_first": False,
            },
        }
    ),
}


# Send POST request to the API endpoint
outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print_highlight("===============================")
    print_highlight(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

```python Example theme={null}
llm.shutdown()
```


# Structured Outputs For Reasoning Models
Source: https://docs.sglang.io/docs/advanced_features/structured_outputs_for_reasoning_models



When working with reasoning models that use special tokens like `<think>...</think>` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output.

SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output.

To enable this feature, use the `--reasoning-parser` flag which decide the think\_end\_token, such as `</think>`, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag.

## Supported Models

Currently, SGLang supports the following reasoning models:

* [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `<think>` and `</think>` tags.
* [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `<think>` and `</think>` tags.

## Usage

## OpenAI Compatible API

Specify the `--grammar-backend`, `--reasoning-parser` option.

```python Example theme={null}
import openai
import os

from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

os.environ["TOKENIZERS_PARALLELISM"] = "false"


server_process, port = launch_server_cmd(
    "python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning"
)

wait_for_server(f"http://localhost:{port}")
client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")
```

### JSON

you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response.

**Using Pydantic**

```python Example theme={null}
from pydantic import BaseModel, Field


# Define the schema using Pydantic
class CapitalInfo(BaseModel):
    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
    population: int = Field(..., description="Population of the capital city")


response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=[
        {
            "role": "assistant",
            "content": "Give me the information and population of the capital of France in the JSON format.",
        },
    ],
    temperature=0,
    max_tokens=2048,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "foo",
            # convert the pydantic model to json schema
            "schema": CapitalInfo.model_json_schema(),
        },
    },
)

print_highlight(
    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
)
```

**JSON Schema Directly**

```python Example theme={null}
import json

json_schema = json.dumps(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "pattern": "^[\\w]+$"},
            "population": {"type": "integer"},
        },
        "required": ["name", "population"],
    }
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=[
        {
            "role": "assistant",
            "content": "Give me the information and population of the capital of France in the JSON format.",
        },
    ],
    temperature=0,
    max_tokens=2048,
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "foo", "schema": json.loads(json_schema)},
    },
)

print_highlight(
    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
)
```

### EBNF

```python Example theme={null}
ebnf_grammar = """
root ::= city | description
city ::= "London" | "Paris" | "Berlin" | "Rome"
description ::= city " is " status
status ::= "the capital of " country
country ::= "England" | "France" | "Germany" | "Italy"
"""

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=[
        {"role": "system", "content": "You are a helpful geography bot."},
        {
            "role": "assistant",
            "content": "Give me the information and population of the capital of France in the JSON format.",
        },
    ],
    temperature=0,
    max_tokens=2048,
    extra_body={"ebnf": ebnf_grammar},
)

print_highlight(
    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
)
```

### Regular expression

```python Example theme={null}
response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=[
        {"role": "assistant", "content": "What is the capital of France?"},
    ],
    temperature=0,
    max_tokens=2048,
    extra_body={"regex": "(Paris|London)"},
)

print_highlight(
    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
)
```

### Structural Tag

```python Example theme={null}
tool_get_current_weather = {
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {
                    "type": "string",
                    "description": "The city to find the weather for, e.g. 'San Francisco'",
                },
                "state": {
                    "type": "string",
                    "description": "the two-letter abbreviation for the state that the city is"
                    " in, e.g. 'CA' which would mean 'California'",
                },
                "unit": {
                    "type": "string",
                    "description": "The unit to fetch the temperature in",
                    "enum": ["celsius", "fahrenheit"],
                },
            },
            "required": ["city", "state", "unit"],
        },
    },
}

tool_get_current_date = {
    "type": "function",
    "function": {
        "name": "get_current_date",
        "description": "Get the current date and time for a given timezone",
        "parameters": {
            "type": "object",
            "properties": {
                "timezone": {
                    "type": "string",
                    "description": "The timezone to fetch the current date and time for, e.g. 'America/New_York'",
                }
            },
            "required": ["timezone"],
        },
    },
}

schema_get_current_weather = tool_get_current_weather["function"]["parameters"]
schema_get_current_date = tool_get_current_date["function"]["parameters"]


def get_messages():
    return [
        {
            "role": "system",
            "content": f"""
# Tool Instructions
- Always execute python code in messages that you share.
- When looking for real time information use relevant functions if available else fallback to brave_search
You have access to the following functions:
Use the function 'get_current_weather' to: Get the current weather in a given location
{tool_get_current_weather["function"]}
Use the function 'get_current_date' to: Get the current date and time for a given timezone
{tool_get_current_date["function"]}
If a you choose to call a function ONLY reply in the following format:
<{{start_tag}}={{function_name}}>{{parameters}}{{end_tag}}
where
start_tag => `<function`
parameters => a JSON dict with the function argument name as key and function argument value as value.
end_tag => `</function>`
Here is an example,
<function=example_function_name>{{"example_name": "example_value"}}</function>
Reminder:
- Function calls MUST follow the specified format
- Required parameters MUST be specified
- Only call one function at a time
- Put the entire function call reply on one line
- Always add your sources when using search results to answer the user query
You are a helpful assistant.""",
        },
        {
            "role": "assistant",
            "content": "You are in New York. Please get the current date and time, and the weather.",
        },
    ]


messages = get_messages()

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    messages=messages,
    response_format={
        "type": "structural_tag",
        "max_new_tokens": 2048,
        "structures": [
            {
                "begin": "<function=get_current_weather>",
                "schema": schema_get_current_weather,
                "end": "</function>",
            },
            {
                "begin": "<function=get_current_date>",
                "schema": schema_get_current_date,
                "end": "</function>",
            },
        ],
        "triggers": ["<function="],
    },
)

print_highlight(
    f"reasoing_content: {response.choices[0].message.reasoning_content}\n\ncontent: {response.choices[0].message.content}"
)
```

## Native API and SGLang Runtime (SRT)

> Note: For native API, as a work-around, you need to set `require_reasoning` argument to `True` to ensure the model will think before generating the structured output. It's not required for chat-completion API.

### JSON

**Using Pydantic**

```python Example theme={null}
import requests
from pydantic import BaseModel, Field
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")


# Define the schema using Pydantic
class CapitalInfo(BaseModel):
    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
    population: int = Field(..., description="Population of the capital city")


messages = [
    {
        "role": "assistant",
        "content": "Give me the information and population of the capital of France in the JSON format.",
    },
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
# Make API request
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": text,
        "require_reasoning": True,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 2048,
            "json_schema": json.dumps(CapitalInfo.model_json_schema()),
        },
    },
)
print(response.json())


reasoing_content = response.json()["text"].split("</think>")[0]
content = response.json()["text"].split("</think>")[1]
print_highlight(f"reasoing_content: {reasoing_content}\n\ncontent: {content}")
```

**JSON Schema Directly**

```python Example theme={null}
json_schema = json.dumps(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "pattern": "^[\\w]+$"},
            "population": {"type": "integer"},
        },
        "required": ["name", "population"],
    }
)

# JSON
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": text,
        "require_reasoning": True,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 2048,
            "json_schema": json_schema,
        },
    },
)

print_highlight(response.json())
```

### EBNF

```python Example theme={null}
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": "Give me the information of the capital of France.",
        "require_reasoning": True,
        "sampling_params": {
            "max_new_tokens": 2048,
            "temperature": 0,
            "n": 3,
            "ebnf": (
                "root ::= city | description\n"
                'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
                'description ::= city " is " status\n'
                'status ::= "the capital of " country\n'
                'country ::= "England" | "France" | "Germany" | "Italy"'
            ),
        },
        "stream": False,
        "return_logprob": False,
    },
)

print(response.json())
```

### Regular expression

```python Example theme={null}
response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": "Paris is the capital of",
        "require_reasoning": True,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 2048,
            "regex": "(France|England)",
        },
    },
)
print(response.json())
```

### Structural Tag

```python Example theme={null}
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
payload = {
    "text": text,
    "require_reasoning": True,
    "sampling_params": {
        "max_new_tokens": 2048,
        "structural_tag": json.dumps(
            {
                "type": "structural_tag",
                "structures": [
                    {
                        "begin": "<function=get_current_weather>",
                        "schema": schema_get_current_weather,
                        "end": "</function>",
                    },
                    {
                        "begin": "<function=get_current_date>",
                        "schema": schema_get_current_date,
                        "end": "</function>",
                    },
                ],
                "triggers": ["<function="],
            }
        ),
    },
}


# Send POST request to the API endpoint
response = requests.post(f"http://localhost:{port}/generate", json=payload)
print_highlight(response.json())
```

```python Example theme={null}
terminate_process(server_process)
```

## Offline Engine API

```python Example theme={null}
import sglang as sgl

llm = sgl.Engine(
    model_path="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    reasoning_parser="deepseek-r1",
    grammar_backend="xgrammar",
)
```

### JSON

**Using Pydantic**

```python Example theme={null}
import json
from pydantic import BaseModel, Field


prompts = [
    "Give me the information of the capital of China in the JSON format.",
    "Give me the information of the capital of France in the JSON format.",
    "Give me the information of the capital of Ireland in the JSON format.",
]


# Define the schema using Pydantic
class CapitalInfo(BaseModel):
    name: str = Field(..., pattern=r"^\w+$", description="Name of the capital city")
    population: int = Field(..., description="Population of the capital city")


sampling_params = {
    "temperature": 0,
    "top_p": 0.95,
    "max_new_tokens": 2048,
    "json_schema": json.dumps(CapitalInfo.model_json_schema()),
}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print("===============================")
    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

**JSON Schema Directly**

```python Example theme={null}
prompts = [
    "Give me the information of the capital of China in the JSON format.",
    "Give me the information of the capital of France in the JSON format.",
    "Give me the information of the capital of Ireland in the JSON format.",
]

json_schema = json.dumps(
    {
        "type": "object",
        "properties": {
            "name": {"type": "string", "pattern": "^[\\w]+$"},
            "population": {"type": "integer"},
        },
        "required": ["name", "population"],
    }
)

sampling_params = {"temperature": 0, "max_new_tokens": 2048, "json_schema": json_schema}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print("===============================")
    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

### EBNF

```python Example theme={null}
prompts = [
    "Give me the information of the capital of France.",
    "Give me the information of the capital of Germany.",
    "Give me the information of the capital of Italy.",
]

sampling_params = {
    "temperature": 0.8,
    "top_p": 0.95,
    "ebnf": (
        "root ::= city | description\n"
        'city ::= "London" | "Paris" | "Berlin" | "Rome"\n'
        'description ::= city " is " status\n'
        'status ::= "the capital of " country\n'
        'country ::= "England" | "France" | "Germany" | "Italy"'
    ),
}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print("===============================")
    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

### Regular expression

```python Example theme={null}
prompts = [
    "Please provide information about London as a major global city:",
    "Please provide information about Paris as a major global city:",
]

sampling_params = {"temperature": 0.8, "top_p": 0.95, "regex": "(France|England)"}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print("===============================")
    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

```python Example theme={null}
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, return_dict=False
)
prompts = [text]


sampling_params = {
    "temperature": 0.8,
    "top_p": 0.95,
    "max_new_tokens": 2048,
    "structural_tag": json.dumps(
        {
            "type": "structural_tag",
            "structures": [
                {
                    "begin": "<function=get_current_weather>",
                    "schema": schema_get_current_weather,
                    "end": "</function>",
                },
                {
                    "begin": "<function=get_current_date>",
                    "schema": schema_get_current_date,
                    "end": "</function>",
                },
            ],
            "triggers": ["<function="],
        }
    ),
}


# Send POST request to the API endpoint
outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print("===============================")
    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

```python Example theme={null}
llm.shutdown()
```


# Text Watermarking
Source: https://docs.sglang.io/docs/advanced_features/text_watermarking



SGLang provides opt-in Aaronson-Gumbel watermarking for sampled text. The watermark replaces random token selection with a keyed, deterministic choice that follows the same output distribution under the ideal independent-uniform hash model, so anyone holding the secret key can later test whether a token sequence carries the watermark. Detection needs only the key and the token IDs; it does not need the model or a GPU.

This feature can support transparency controls such as the machine-readable marking and detectability described in [Article 50 of the EU AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj). It is one technical signal, not a complete compliance program or proof of authorship.

## How it works

### Generation

At each sampling step, SGLang first applies frequency, presence, and repetition penalties, logit bias, and any structured-output grammar mask, followed by the request's temperature and its top-k, top-p, and min-p truncation, then renormalizes. Let `p_i` be the resulting probability of candidate token `i`; grammar-masked and truncated tokens have `p_i = 0` and can never be selected. For each candidate, SGLang hashes the key, the preceding token context, and the token ID into a uniform value `u_i` in `(0, 1)` and selects

```text theme={null}
argmax over i with p_i > 0 of  log(u_i) / p_i
```

When the `u_i` are independent uniforms, this rule selects token `i` with probability exactly `p_i`, so a single step is distributed like ordinary sampling. Ties go to the lowest token ID.

A position is watermarked only when all of the following hold; otherwise the request samples that position normally:

* The request has watermarking enabled (see [Per-request control](#per-request-control)).
* The request is not greedy. `temperature=0` is normalized to `top_k=1`, and `top_k=1` positions are never watermarked.
* The context is not empty.
* The context has not already been watermarked earlier in the same request. Runtime identifies a context by its 32-bit context hash.
* The request has watermarked fewer than `min(4,096, per-request token-pool capacity)` contexts.
* The largest truncated probability is at most the server's `max_probability`. The default `1.0` disables this gate.

Only a watermarked position enters the request's context history. A position skipped by the probability gate therefore leaves its context eligible for a later, higher-entropy occurrence, and once a request reaches its context budget the remaining tokens use ordinary sampling. The repeated-context rule keeps a repeated context from forcing the same token indefinitely.

The context for a position is the last `h` tokens before it, where `h` is the request's context window, taken from the prompt followed by the tokens generated so far. The first generated token therefore uses the last `h` prompt tokens. A context is shorter than `h` only when fewer than `h` tokens precede the position.

### Hash contract

All arithmetic wraps to unsigned 32 bits. The primitives are the MurmurHash3 x86 32-bit body mix and finalizer:

```text theme={null}
mix(state, value):
  value = value * 0xcc9e2d51
  value = rotl32(value, 15)
  value = value * 0x1b873593
  state = rotl32(state XOR value, 13)
  return state * 5 + 0xe6546b64

fmix32(value):
  value = value XOR (value >> 16)
  value = value * 0x85ebca6b
  value = value XOR (value >> 13)
  value = value * 0xc2b2ae35
  return value XOR (value >> 16)
```

A key is 1 to 16 hexadecimal digits with an optional `0x` prefix, interpreted as an unsigned 64-bit integer; `key_lo` and `key_hi` are its low and high 32-bit words. For a context of `L` token IDs `c_1 ... c_L` and a candidate token `t`:

```text theme={null}
context_hash = fmix32(mix(...mix(mix(0, c_1), c_2)..., c_L) XOR (4 * L))
token_hash   = fmix32(mix(mix(mix(mix(0, key_lo), key_hi), context_hash), t) XOR 16)
u            = (token_hash + 0.5) / 2^32
```

The context hash covers only the context tokens; keys enter only through the token hash and, in dual-key mode, through a per-context coin that chooses which key produces the token hash:

```text theme={null}
coin      = fmix32(mix(mix(mix(mix(mix(0, a_lo), a_hi), b_lo), b_hi), context_hash) XOR 20)
threshold = floor(P * 2^32)
key       = key A if coin < threshold else key B
```

`a_lo`/`a_hi` and `b_lo`/`b_hi` are the words of key A and key B, and `P` is the server's `mixing_probability`.

### Detection

A detector holding the key scores each generated token `t` with `s = -log(1 - u)`, where `u` comes from the hash contract above using that token's context. Without the watermark, each `s` is an independent `Exponential(1)` sample. With the watermark, the generator preferred tokens with large `u`, so the scores are inflated.

Over `n` scored contexts, the score `S = sum(s)` follows `Gamma(shape=n, scale=1)` under the null hypothesis that the text is not watermarked with this key. The detector reports the upper-tail probability `P[Gamma(n, 1) >= S]` as the p-value, its natural logarithm (finite even when the p-value underflows to `0.0`), and the normal approximation `z = (S - n) / sqrt(n)`.

To match generation, a detector must follow these rules:

1. Use the exact prompt token IDs that preceded the completion and the request's context window `h`. With prompt IDs, the first completion token's context is the last `h` prompt tokens. Without prompt IDs, the first `h` completion tokens have unknown contexts and are skipped.
2. Score each distinct context once. The detector compares exact context token tuples, whereas generation compares 32-bit context hashes. After a hash collision, generation skips a later distinct context that the detector still scores. That extra score is an ordinary null sample, so the mismatch can only weaken detection, never create evidence.
3. Stop after the generation request's context budget and report a prefix test. The default detector budget is 4,096 distinct contexts. If the server's per-request token-pool capacity is lower, pass that lower value as `max_contexts`; scoring the unforced tail adds null samples that dilute the statistic without adding evidence. With a maximum-probability gate below `1.0`, the model-free detector cannot identify skipped positions, so this fixed prefix is conservative and may omit signal from later forced positions.

Positions that generation left unforced (greedy positions, and positions skipped by `max_probability`) are indistinguishable from ordinary text to the detector, which scores them as null samples; they dilute the statistic but cannot produce false evidence.

A low p-value is evidence that the text carries the keyed watermark. It is not a calibrated probability that a particular author or model produced the text. Applications must choose a decision threshold and measure false-positive and false-negative rates on their own traffic.

## Server configuration

`--enable-watermark` turns the feature on; every other watermark setting lives in one JSON object passed with `--watermark-config`, either as a file path or inline. Restrict the file to the server account and launch a CUDA server:

```json theme={null}
{
  "key": "0123456789abcdef",
  "key_b": "fedcba9876543210",
  "context_window": 4,
  "mixing_probability": 0.5,
  "max_probability": 1.0,
  "default_enabled": true,
  "enforce_all": false
}
```

```bash theme={null}
chmod 600 /run/secrets/sglang-watermark.json
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --enable-watermark \
  --watermark-config /run/secrets/sglang-watermark.json
```

| Field | Default | Behavior |
| - | - | - |
| `key` | unset | Server key (key A), 1 to 16 hex digits, for requests that enable watermarking without their own key. |
| `key_b` | unset | Enables dual-key mode with a second server key. Requires `key`. |
| `context_window` | `4` | Default context window and the maximum that a request may select, from 1 to 64 tokens. |
| `mixing_probability` | `0.5` | In dual-key mode, the probability of selecting key A for each context. Must be strictly between 0 and 1; a non-default value requires `key_b`. |
| `max_probability` | `1.0` | Skips watermarking when the largest truncated probability exceeds this value. Must be greater than 0 and at most 1; `1.0` disables the gate. |
| `default_enabled` | `false` | Watermarks requests that omit the `watermark` field with the server key. Requests may opt out. Requires `key`. |
| `enforce_all` | `false` | Watermarks requests that omit the `watermark` field and rejects `{"enabled": false}`. Requires `key`. |

Every field is optional and any other field is rejected. The config must be a JSON object of at most 4,096 bytes. A value that starts with `{` is parsed as inline JSON; any other value is read as a path to a regular file, and SGLang logs a warning when that file is readable by group or other users. Inline JSON is visible in the process command line, including `/proc/*/cmdline` on Linux, so pass keys through a file. `--watermark-config` requires `--enable-watermark`. A server key alone does not watermark every request; without `default_enabled` or `enforce_all`, requests must opt in.

SGLang redacts the keys and the `--watermark-config` value from the recorded launch command, startup logs, request logs, server information responses, and request crash-dump payloads. Process and GPU core dumps may contain in-memory keys. Protect those artifacts, the configuration file, and the transport used by clients that send per-request keys.

## Per-request control

The native `/generate` API and the OpenAI-compatible `/v1/completions` and `/v1/chat/completions` APIs accept a `watermark` object with three optional fields:

| Field | Type | Meaning |
| - | - | - |
| `enabled` | boolean | `true` enables watermarking with the server key; `false` opts out. |
| `key` | hex string | Watermarks with this key instead of the server key. A key implicitly enables watermarking. |
| `context_window` | integer | Context window for this request, from 1 to the server's `context_window`. |

On `/generate`, pass `watermark` either at the top level of the request or inside `sampling_params`, but not both; a batched request may pass one object per prompt as a list. On the OpenAI-compatible APIs, pass it as a top-level request field:

```bash theme={null}
curl http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Explain photosynthesis."}],
    "temperature": 1.0,
    "top_p": 0.95,
    "return_token_ids": true,
    "watermark": {"enabled": true, "context_window": 4}
  }'
```

The server resolves each request as follows:

| Server mode | No `watermark` field | `{"enabled": false}` | `{"enabled": true}` | `{"key": "HEX"}` |
| - | - | - | - | - |
| Watermarking disabled | Off | Off | HTTP 400 | HTTP 400 |
| `--enable-watermark`, no server key | Off | Off | HTTP 400 | Request key |
| `--enable-watermark` with a server key | Off | Off | Server key | Request key |
| Add `default_enabled` | Server key | Off | Server key | Request key |
| Add `enforce_all` | Server key | HTTP 400 | Server key | Request key |

The request also returns HTTP 400 when:

* the `watermark` object sets neither `enabled` nor `key`, or contains an unknown field;
* `key` is combined with `"enabled": false`;
* a key is not 1 to 16 hexadecimal digits;
* `context_window` is not a positive integer or exceeds the server maximum;
* a watermarked request uses beam search.

Request-level keys are credentials carried in the request body. Use an authenticated TLS endpoint whenever they cross a host boundary, and avoid recording request bodies in proxies or application logs.

## Dual-key mode

`key_b` in `--watermark-config` applies dual-key selection to every watermarked request. The request's resolved key, whether the server key or a request key, is key A, and the server's second key is key B. A request key therefore replaces key A while key B stays active. Each eligible context uses key A with probability `mixing_probability` and key B otherwise, according to the coin in the [hash contract](#hash-contract); truncation, repeated-context, and greedy rules are unchanged. Without key B, selection uses the single-key path.

A detector holding both keys splits the distinct contexts by the coin, scores each context with its selected key, and tests the union, which is the strongest statistic; it can also test each partition separately. A detector holding only one key can score every distinct context with that key, but the contexts selected for the other key contribute only null samples and dilute the signal.

## Using the detector

`WatermarkDetector` implements the [detection rules](#detection) above. Construct it with the same key, key B, mixing probability, and context window that generation used:

A minimal [reference detection server](https://github.com/sgl-project/sglang/tree/main/examples/watermark) exposes the in-tree detector over HTTP and reads keys only from `--watermark-config`.

```python theme={null}
WatermarkDetector(
    key,                      # hex key A
    *,
    key_b=None,               # hex key B for dual-key mode
    mixing_probability=0.5,   # must match the server's mixing_probability
    context_window=4,         # the request's effective window, 1 to 64
    max_contexts=4096,        # prefix-test length, 1 to 4096
)
```

* `detect_tokens(token_ids, *, prompt_token_ids=())` scores completion token IDs.
* `detect_text(text, tokenizer, *, prompt_token_ids=())` first encodes the text with `tokenizer.encode(text, add_special_tokens=False)`. Any object with that method works, including Hugging Face tokenizers.
* `detect(token_ids, *, key, key_b=None, prompt_token_ids=(), mixing_probability=0.5, context_window=4)` is a one-call shortcut.

Token IDs must be unsigned 32-bit integers; any iterable of Python or NumPy integers is accepted.

Every call returns a `WatermarkDetection`:

| Field | Meaning |
| - | - |
| `combined` | The test to report. Single-key: every distinct context scored with key A. Dual-key: each context scored with its coin-selected key. |
| `key_a_all_positions` | Every distinct context scored with key A. |
| `key_b_all_positions`, `key_a_partition`, `key_b_partition` | Dual-key only (`None` otherwise): every context scored with key B, and each coin partition scored with its own key. |
| `prefix_limit_reached` | `True` when scoring stopped at `max_contexts` distinct contexts. |
| `tokens_examined` | Completion tokens covered by the test, including repeated and skipped ones. |
| `repeated_contexts` | Positions skipped because their exact context was already scored. |
| `skipped_initial_tokens` | Leading completion tokens skipped because no prompt IDs were given. |

Each statistic is a `WatermarkStatistics` with `num_contexts`, `score`, `z_score`, `p_value`, and `log_p_value`.

Prefer the server's token IDs to re-tokenized text, because re-encoding decoded text does not always reproduce the generated IDs. With `"return_token_ids": true`, non-streaming chat completions return `prompt_token_ids` and `response_token_ids` on each choice, and completions return `prompt_token_ids` and `token_ids`.

### In an SGLang environment

```python theme={null}
import requests

from sglang.srt.sampling.watermarking import WatermarkDetector

response = requests.post(
    "http://127.0.0.1:30000/v1/chat/completions",
    json={
        "model": "Qwen/Qwen3-8B",
        "messages": [{"role": "user", "content": "Explain photosynthesis."}],
        "temperature": 1.0,
        "top_p": 0.95,
        "return_token_ids": True,
        "watermark": {"enabled": True},
    },
).json()
choice = response["choices"][0]

detector = WatermarkDetector("0123456789abcdef", context_window=4)
result = detector.detect_tokens(
    choice["response_token_ids"],
    prompt_token_ids=choice["prompt_token_ids"],
)
print(result.combined.num_contexts, result.combined.z_score, result.combined.log_p_value)
```

To score text, pass the model's tokenizer and, when available, the prompt IDs:

```python theme={null}
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
result = detector.detect_text(
    choice["message"]["content"],
    tokenizer,
    prompt_token_ids=choice["prompt_token_ids"],
)
```

For a dual-key server, pass both keys and the server's mixing probability:

```python theme={null}
detector = WatermarkDetector(
    "0123456789abcdef",
    key_b="fedcba9876543210",
    mixing_probability=0.5,
    context_window=4,
)
result = detector.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)
print(result.combined.p_value, result.key_a_partition.p_value, result.key_b_partition.p_value)
```

### Standalone, without SGLang or PyTorch

`detector.py` and `config.py` import only the Python standard library and `msgspec`, and the package's `__init__.py` imports only the detector. NumPy arrays are accepted as token-ID iterables, but NumPy is not required. To run detection in a separate application, copy the package directory and install `msgspec`:

```bash theme={null}
mkdir -p detector_app
cp -R python/sglang/srt/sampling/watermarking detector_app/watermarking
python -m pip install msgspec
```

The copy also contains `core.py`, the server runtime, which imports PyTorch; nothing on the detector import path loads it. Run the application with `detector_app` on the import path and import the detector from the copied package:

```python theme={null}
from watermarking.detector import WatermarkDetector

single = WatermarkDetector("0123456789abcdef", context_window=4)
result = single.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)

dual = WatermarkDetector("0123456789abcdef", key_b="fedcba9876543210")
result = dual.detect_tokens(completion_ids, prompt_token_ids=prompt_ids)
```

Text detection needs only an object with `encode(text, *, add_special_tokens)` that returns a list of IDs. For example, with the lightweight `tokenizers` package:

```python theme={null}
from tokenizers import Tokenizer

from watermarking.detector import WatermarkDetector


class TokenizerAdapter:
    def __init__(self, name):
        self._tokenizer = Tokenizer.from_pretrained(name)

    def encode(self, text, *, add_special_tokens):
        return self._tokenizer.encode(text, add_special_tokens=add_special_tokens).ids


detector = WatermarkDetector("0123456789abcdef", context_window=4)
result = detector.detect_text(text, TokenizerAdapter("Qwen/Qwen3-8B"), prompt_token_ids=prompt_ids)
```

A detector written from scratch instead of reusing this package must implement the [hash contract](#hash-contract) and all three [detection rules](#detection): prompt alignment, exact-tuple deduplication, and the 4,096-context prefix.

## Compatibility

| Feature | Status | Behavior and validation |
| - | - | - |
| CUDA graphs | Verified | Decode graphs, eager fallback, and fused state updates use the same selector contract. |
| Overlap scheduling | Verified | Prompt initialization is carried with the delayed sampling batch. |
| Tensor parallelism | Verified | TP2 and TP4 ordinary decode, plus NEXTN with TP2, produce detectable output without rank divergence. |
| Mixed enabled and disabled requests | Verified | Keys, context windows, histories, and enablement are resolved per request. |
| JSON schema and regex grammars | Verified | Grammar masking runs before watermark selection; strict outputs remain valid and detectable. |
| Streaming chat completions | Verified | Streaming and non-streaming requests use the same watermark state and token selection. |
| Parallel sampling (`n > 1`) | Verified | Each generated choice owns independent request state and history. |
| Output log probabilities | Verified | Ordinary and speculative decode report the pre-watermark target distribution. Logits are cloned only for requests that ask for log probabilities. |
| Top-k, top-p, and min-p | Verified | The selector reproduces SGLang's combined truncation support before keyed selection. |
| Maximum-probability gate | Verified | Positions above `max_probability` use ordinary sampling and do not consume the repeated-context budget. |
| Logit bias and frequency, presence, and repetition penalties | Verified | These transforms run before watermark selection. |
| Chunked prefill | Verified | Each prefill chunk rebuilds the prompt tail from the request's complete logical fill IDs. |
| Radix-cache prompt reuse | Verified | Prompt-tail initialization uses logical request IDs rather than only newly computed tokens. |
| Decode retraction | Verified | Re-prefill restores the distinct-context history before generation resumes. |
| Request abort and request-slot reuse | Verified | A later prefill resets ring-buffer and context-history state for the reused slot. |
| Multimodal text generation | Verified | Qwen3-VL text tokens use the same watermark path after multimodal prefill. |
| DP attention | Verified | DP2/TP2 normal and NEXTN decoding were verified with Qwen3-8B on two H200 GPUs, including idle-rank batches. |
| NGRAM speculative decoding | Verified | Draft-node contexts are reconstructed before verification sampling. |
| EAGLE and EAGLE3 speculative decoding | Verified | Tree nodes use their own ancestor-derived contexts. |
| NEXTN speculative decoding | Verified | NEXTN resolves to the EAGLE verification path. |
| Pipeline-parallel speculative decoding | Rejected | Startup validation fails closed. |
| Other speculative algorithms | Rejected | Startup validation fails closed. |
| Speculative rejection sampling | Rejected | Startup validation fails closed. |
| Custom logit processors | Rejected | Startup validation fails closed because processor ordering is not defined. |
| Token-oracle sampling backend | Rejected | Startup validation fails closed because oracle-selected tokens bypass the watermark selector. |
| Beam search | Rejected | Request validation rejects watermarked beam-search requests. |
| Diffusion LLM | Rejected | Startup validation fails closed. |
| PD disaggregation | Rejected | Startup validation fails closed because watermark state is not transferred between workers. |
| Embedded Rust API server | Rejected | Startup validation fails closed. |
| Non-CUDA devices | Rejected | Startup validation fails closed. |

In speculative decoding, verification clones the target logits before forcing, so target-model log-probability computation still sees the pre-watermark distribution. Each draft node gets its own context, rebuilt from its ancestors in the verify tree. Repeated-context admission is conservative across the whole tree: once a context hash appears in an earlier row, later rows with the same hash are not forced, even on sibling branches. This keeps duplicate entries out of the history, but reconvergent wide trees may carry less signal.

## Limitations and performance

* Greedy requests carry no signal, and very low-entropy distributions leave few meaningful alternatives for the selector.
* A `max_probability` value below 1 keeps ordinary selection at low-entropy positions and can improve speculative draft acceptance, but the unforced positions dilute the detector statistic.
* Short completions provide few distinct contexts and therefore weak statistical power.
* Paraphrasing, rewriting, translation, and re-tokenization can weaken or remove the signal.
* Detection needs the matching key or keys, mixing probability, context window, and tokenizer, and is strongest with the exact prompt token IDs.
* A request watermarks at most `min(4,096, per-request token-pool capacity)` distinct contexts; later tokens use ordinary sampling.
* The hashes are MurmurHash3, a keyed but non-cryptographic mix. There is no indistinguishability or unforgeability guarantee, and anyone holding the key can both detect and forge the watermark.
* Speculative decoding can accept fewer draft tokens when a draft proposal does not match the watermarked target token. The impact depends on tree width and workload.
* Mixed batches choose the selector path from the batch-wide maximum top-k. A transient unlimited-top-k request can keep that batch on the full-sort path until its sampling metadata is rebuilt.
* Retraction reconstructs a conservative distinct-context history. Repeated retractions can reduce later signal and add scheduler-side replay work.

On one single-GPU node of each type with public `Qwen/Qwen3-8B`, TP1, TRT-LLM MHA, ordinary decoding, temperature 1.0, top-p 0.95, unlimited top-k, and 800-token sampled outputs, the fused selector measured the following mean output throughput over three repetitions per cell:

| Node | Concurrency | Watermark off | Watermark on | Change |
| - | -: | -: | -: | -: |
| GB300 | 1 | 270.25 token/s | 260.44 token/s | -3.63% |
| GB300 | 8, per stream | 224.88 token/s | 212.84 token/s | -5.35% |
| B300 | 1 | 258.84 token/s | 247.69 token/s | -4.31% |
| B300 | 8, per stream | 196.26 token/s | 182.04 token/s | -7.24% |

The GB300 concurrency-eight row was remeasured after the score-precision fix on a default-off server by comparing omitted requests with request-level opt-in. These measurements describe one model, GPU, sampling configuration, and output length. Benchmark your deployment before setting capacity targets. Finite-top-k selector latency at batch size one is also hardware-dependent: in one microbenchmark shape (`top_k=20`), the fast path was 58.8% slower than a full sort on H200 with bfloat16 logits and 6.4% faster on GB300 with float32 logits, and at batch size eight it was faster on both.

On a 2x GB300 node with public `Qwen/Qwen3.5-9B`, TP2, NEXTN 3/1/4, concurrency 8, 1,024-token random inputs, 512-token outputs, temperature 0.8, top-p 0.95, and top-k 20, the fused speculative path measured the following means over three repetitions:

| Watermark | Mean TPOT | Aggregate output throughput | Mean accept length |
| - | -: | -: | -: |
| Off | 2.246 ms | 3,188.9 token/s | 2.759 |
| On | 2.451 ms | 2,922.2 token/s | 2.783 |
| Change | +9.17% | -8.36% | +0.85% |

Each speculative row still requires keyed selection, but tree-context reconstruction, repeated-context admission, accepted-context recording, and ring-buffer append run as fused GPU operations. The reported cost remains workload- and acceptance-dependent.

## See also

The fused Triton selector is adapted from Meta's Apache-2.0 [TextSeal](https://github.com/facebookresearch/textseal) implementation and re-implemented on the hash contract above, so detection follows this page rather than TextSeal's own tooling.

The runtime and reference detector live in `python/sglang/srt/sampling/watermarking/`. Registered coverage lives in `test/registered/kernels/ops/sampling/test_watermark_*.py` and `test/registered/e2e/openai_server/test_watermark.py`.


# Thinking Budget
Source: https://docs.sglang.io/docs/advanced_features/thinking_budget



Reasoning models can spend an unbounded number of tokens inside the thinking phase. SGLang gives you three ways to control that, from soft hints to hard per-request caps:

1. **Chat-template hints** (`enable_thinking`, `thinking_budget`, `reasoning_effort`): the model's own chat template decides what to do with them. No token guarantee.
2. **Strict thinking** (`--enable-strict-thinking`): a grammar-level token filter that enforces an exact per-request budget. Recommended when you need a guarantee.
3. **Custom logit processor** (`--enable-custom-logit-processor`): a logit processor that forces the thinking phase to end at the budget. Use when the grammar backend cannot be used.

The examples below use [Qwen3](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f), but every mechanism except the built-in logit processors is model-agnostic: strict thinking derives its token ids from the configured reasoning parser, so it works for any model whose parser is set.

## Chat-template hints

The OpenAI chat completions API passes `chat_template_kwargs` through to the model's chat template:

```python Chat-template kwargs theme={null}
response = client.chat.completions.create(
    model="Qwen/Qwen3-32B",
    messages=[{"role": "user", "content": "What is 1+3?"}],
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True, "thinking_budget": 512}
    },
)
```

Two things to know:

* `enable_thinking` toggles the thinking mode for hybrid models such as Qwen3. It is template-dependent.
* `thinking_budget` only does something if the model's own chat template defines that variable (the official Qwen3 template does). If the template does not define it, the kwarg is silently ignored.

The `reasoning_effort` request field is forwarded into the chat template the same way.

These are hints to the model, not token guarantees. The model can overshoot the budget.

## Hard budget with strict thinking

Launch the server with `--enable-strict-thinking` and a reasoning parser:

```bash Launch with strict thinking theme={null}
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-32B \
    --host 0.0.0.0 \
    --reasoning-parser qwen3 \
    --enable-strict-thinking
```

Strict thinking requires a grammar backend that supports token filtering. `xgrammar` is the default grammar backend, so no `--grammar-backend` flag is needed. Startup fails if the configured backend cannot filter tokens.

Enable `--reasoning-parser` so responses separate `reasoning_content` from the final answer (see [Reasoning Parser](/docs/advanced_features/separate_reasoning)). Strict thinking derives the thinking-end token ids from the parser's `think_end_token` through the tokenizer, so it is not tied to any hardcoded ids.

### Behavior

* When the budget is spent during thinking, the vocab mask allows only the `</think>` token sequence, so the cap is exact. The model then produces the final answer.
* With strict thinking enabled and no budget set, behavior is unchanged except that model-specific excluded tokens (for Qwen3, `<tool_call>`, `</tool_call>`, `<|im_end|>`, `<|endoftext|>`) are blocked during the thinking phase.
* To apply a server-wide cap to every request, set the environment variable `SGLANG_MAX_THINK_TOKENS` (default `-1`, no cap).

### Native `/generate` API

Set `max_thinking_tokens` per request, and always pair it with `require_reasoning: true`:

```python /generate with a thinking budget theme={null}
import requests

response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": prompt,
        "require_reasoning": True,
        "max_thinking_tokens": 512,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 2048,
        },
    },
)
```

> [!WARNING]
>
> Without `require_reasoning: true`, the grammar object starts in the generation state and `max_thinking_tokens` is silently ignored. Sending `max_thinking_tokens` to a server launched without `--enable-strict-thinking` returns a 400 error.

### OpenAI chat completions API

There is no `max_thinking_tokens` field on chat completions. Pass the budget through `custom_params` instead; the runtime picks up the `thinking_budget` key the same way. The chat path sets `require_reasoning` for you based on the thinking mode:

```python Chat completions with a thinking budget theme={null}
response = client.chat.completions.create(
    model="Qwen/Qwen3-32B",
    messages=[{"role": "user", "content": "What is 1+3?"}],
    extra_body={"custom_params": {"thinking_budget": 512}},
)

print(response.choices[0].message.reasoning_content)
print(response.choices[0].message.content)
```

## Hard budget with a custom logit processor

When the grammar backend cannot be used, a custom logit processor can enforce the budget instead. Launch the server with `--enable-custom-logit-processor`:

```bash Launch with custom logit processors theme={null}
python -m sglang.launch_server \
    --model-path Qwen/Qwen3-32B \
    --host 0.0.0.0 \
    --reasoning-parser qwen3 \
    --enable-custom-logit-processor
```

Pass a serialized processor class and the budget on each request. This works on both `/generate` and chat completions:

```python Logit processor with a thinking budget theme={null}
from sglang.srt.sampling.custom_logit_processor import (
    Qwen3ThinkingBudgetLogitProcessor,
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-32B",
    messages=[{"role": "user", "content": "What is 1+3?"}],
    extra_body={
        "custom_logit_processor": Qwen3ThinkingBudgetLogitProcessor.to_str(),
        "custom_params": {"thinking_budget": 512},
    },
)
```

The processor counts the tokens generated after the `<think>` start token. Once the budget is reached, it first forces a newline token, then forces `</think>`. The model may emit its own `</think>` right after the forced one; this is harmless.

### Built-in processors

SGLang ships processors in `python/sglang/srt/sampling/custom_logit_processor.py`, each with hardcoded thinking start / end / newline token ids:

| Processor | Model family |
| - | - |
| `Qwen3ThinkingBudgetLogitProcessor` | Qwen3 |
| `Glm4MoeThinkingBudgetLogitProcessor` | GLM-4.5 / GLM-4.6 |
| `DeepSeekR1ThinkingBudgetLogitProcessor` | DeepSeek-R1 |
| `InklingThinkingBudgetLogitProcessor` | Inkling |

These only work for models whose tokenizer produces those exact ids. For any other model, subclass `ThinkingBudgetLogitProcessor` with the model's own ids:

```python Custom thinking-budget processor theme={null}
from sglang.srt.sampling.custom_logit_processor import (
    ThinkingBudgetLogitProcessor,
)

# Find the ids with your model's tokenizer, e.g.:
# tokenizer.encode("</think>", add_special_tokens=False)
class MyModelThinkingBudgetLogitProcessor(ThinkingBudgetLogitProcessor):
    THINKING_START_TOKEN_ID = 12345
    THINKING_END_TOKEN_ID = 12346
    NEW_LINE_TOKEN_ID = 198
```

## Verify the budget applies

Budget knobs can be silently ignored, so confirm the cap with a deterministic comparison:

1. Run a prompt with `temperature=0` and no budget. Record the thinking length.
2. Run the same prompt with the budget set, again at `temperature=0`.

A working cap ends the thinking exactly at the budget, usually mid-sentence. An ignored knob reproduces the baseline output token for token. Common silent-failure modes:

* Missing `require_reasoning: true` on the native `/generate` API.
* `chat_template_kwargs.thinking_budget` on a model whose chat template does not define the variable.
* A built-in logit processor whose hardcoded token ids do not match the model's tokenizer.


# Tool Parser
Source: https://docs.sglang.io/docs/advanced_features/tool_parser



This guide demonstrates how to use SGLang’s [Function calling](https://platform.openai.com/docs/guides/function-calling) functionality.

## Currently supported parsers:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parser</th>
      <th>Supported Models</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`apertus2509`</td>
      <td>Apertus 2509 (e.g., `swiss-ai/Apertus-{8,70}B-Instruct-2509`)</td>
      <td>Tool calls are emitted as a JSON list of single-key objects: `<|tools_prefix|>[{"tool": {...}}]<|tools_suffix|>`.</td>
    </tr>

    <tr>
      <td>`deepseekv3`</td>
      <td>DeepSeek-v3 (e.g., `deepseek-ai/DeepSeek-V3-0324`)</td>
      <td>Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv3.jinja` to launch command.</td>
    </tr>

    <tr>
      <td>`deepseekv31`</td>
      <td>DeepSeek-V3.1 and DeepSeek-V3.2-Exp (e.g. `deepseek-ai/DeepSeek-V3.1`, `deepseek-ai/DeepSeek-V3.2-Exp`)</td>
      <td>Recommend adding `--chat-template ./examples/chat_template/tool_chat_template_deepseekv31.jinja` (Or ..deepseekv32.jinja for DeepSeek-V3.2) to launch command.</td>
    </tr>

    <tr>
      <td>`deepseekv32`</td>
      <td>DeepSeek-V3.2 (`deepseek-ai/DeepSeek-V3.2`)</td>

      <td />
    </tr>

    <tr>
      <td>`glm`</td>
      <td>GLM series (e.g. `zai-org/GLM-4.6`)</td>

      <td />
    </tr>

    <tr>
      <td>`gpt-oss`</td>
      <td>GPT-OSS (e.g., `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `lmsys/gpt-oss-120b-bf16`, `lmsys/gpt-oss-20b-bf16`)</td>
      <td>The gpt-oss tool parser filters out analysis channel events and only preserves normal text. This can cause the content to be empty when explanations are in the analysis channel. To work around this, complete the tool round by returning tool results as `role="tool"` messages, which enables the model to generate the final content.</td>
    </tr>

    <tr>
      <td>`kimi_k2`</td>
      <td>`moonshotai/Kimi-K2-Instruct`</td>

      <td />
    </tr>

    <tr>
      <td>`llama3`</td>
      <td>Llama 3.1 / 3.2 / 3.3 (e.g. `meta-llama/Llama-3.1-8B-Instruct`, `meta-llama/Llama-3.2-1B-Instruct`, `meta-llama/Llama-3.3-70B-Instruct`)</td>

      <td />
    </tr>

    <tr>
      <td>`llama4`</td>
      <td>Llama 4 (e.g. `meta-llama/Llama-4-Scout-17B-16E-Instruct`)</td>

      <td />
    </tr>

    <tr>
      <td>`mistral`</td>
      <td>Mistral (e.g. `mistralai/Mistral-7B-Instruct-v0.3`, `mistralai/Mistral-Nemo-Instruct-2407`, `mistralai/Mistral-7B-v0.3`)</td>

      <td />
    </tr>

    <tr>
      <td>`pythonic`</td>
      <td>Llama-3.2 / Llama-3.3 / Llama-4</td>
      <td>Model outputs function calls as Python code. Requires `--tool-call-parser pythonic` and is recommended to use with a specific chat template.</td>
    </tr>

    <tr>
      <td>`qwen`</td>
      <td>Qwen series (e.g. `Qwen/Qwen3-Next-80B-A3B-Instruct`, `Qwen/Qwen3-VL-30B-A3B-Thinking`) except Qwen3-Coder</td>

      <td />
    </tr>

    <tr>
      <td>`qwen3_coder`</td>
      <td>Qwen3-Coder (e.g. `Qwen/Qwen3-Coder-30B-A3B-Instruct`)</td>

      <td />
    </tr>

    <tr>
      <td>`step3`</td>
      <td>Step-3</td>

      <td />
    </tr>
  </tbody>
</table>

## OpenAI Compatible API

### Launching the Server

```python Example theme={null}
import json
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process
from openai import OpenAI

server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning"  # qwen25
)
wait_for_server(f"http://localhost:{port}")
```

Note that `--tool-call-parser` defines the parser used to interpret responses.

### Define Tools for Function Call

Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters.

```python Example theme={null}
# Define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "The city to find the weather for, e.g. 'San Francisco'",
                    },
                    "state": {
                        "type": "string",
                        "description": "the two-letter abbreviation for the state that the city is"
                        " in, e.g. 'CA' which would mean 'California'",
                    },
                    "unit": {
                        "type": "string",
                        "description": "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["city", "state", "unit"],
            },
        },
    }
]
```

### Define Messages

```python Example theme={null}
def get_messages():
    return [
        {
            "role": "user",
            "content": "What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.",
        }
    ]


messages = get_messages()
```

### Initialize the Client

```python Example theme={null}
# Initialize OpenAI-like client
client = OpenAI(api_key="None", base_url=f"http://0.0.0.0:{port}/v1")
model_name = client.models.list().data[0].id
```

### Non-Streaming Request

```python Example theme={null}
# Non-streaming mode test
response_non_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0,
    top_p=0.95,
    max_tokens=1024,
    stream=False,  # Non-streaming
    tools=tools,
)
print_highlight("Non-stream response:")
print_highlight(response_non_stream)
print_highlight("==== content ====")
print_highlight(response_non_stream.choices[0].message.content)
print_highlight("==== tool_calls ====")
print_highlight(response_non_stream.choices[0].message.tool_calls)
```

#### Handle Tools

When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly.

```python Example theme={null}
name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name
arguments_non_stream = (
    response_non_stream.choices[0].message.tool_calls[0].function.arguments
)

print_highlight(f"Final streamed function call name: {name_non_stream}")
print_highlight(f"Final streamed function call arguments: {arguments_non_stream}")
```

### Streaming Request

```python Example theme={null}
# Streaming mode test
print_highlight("Streaming response:")
response_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0,
    top_p=0.95,
    max_tokens=1024,
    stream=True,  # Enable streaming
    tools=tools,
)

texts = ""
tool_calls = []
name = ""
arguments = ""
for chunk in response_stream:
    if chunk.choices[0].delta.content:
        texts += chunk.choices[0].delta.content
    if chunk.choices[0].delta.tool_calls:
        tool_calls.append(chunk.choices[0].delta.tool_calls[0])
print_highlight("==== Text ====")
print_highlight(texts)

print_highlight("==== Tool Call ====")
for tool_call in tool_calls:
    print_highlight(tool_call)
```

#### Handle Tools

When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly.

```python Example theme={null}
# Parse and combine function call arguments
arguments = []
for tool_call in tool_calls:
    if tool_call.function.name:
        print_highlight(f"Streamed function call name: {tool_call.function.name}")

    if tool_call.function.arguments:
        arguments.append(tool_call.function.arguments)

# Combine all fragments into a single JSON string
full_arguments = "".join(arguments)
print_highlight(f"streamed function call arguments: {full_arguments}")
```

### Define a Tool Function

```python Example theme={null}
# This is a demonstration, define real function according to your usage.
def get_current_weather(city: str, state: str, unit: "str"):
    return (
        f"The weather in {city}, {state} is 85 degrees {unit}. It is "
        "partly cloudly, with highs in the 90's."
    )


available_tools = {"get_current_weather": get_current_weather}
```

### Execute the Tool

```python Example theme={null}
messages.append(response_non_stream.choices[0].message)

# Call the corresponding tool function
tool_call = messages[-1].tool_calls[0]
tool_name = tool_call.function.name
tool_to_call = available_tools[tool_name]
result = tool_to_call(**(json.loads(tool_call.function.arguments)))
print_highlight(f"Function call result: {result}")
# messages.append({"role": "tool", "content": result, "name": tool_name})
messages.append(
    {
        "role": "tool",
        "tool_call_id": tool_call.id,
        "content": str(result),
        "name": tool_name,
    }
)

print_highlight(f"Updated message history: {messages}")
```

### Send Results Back to Model

```python Example theme={null}
final_response = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0,
    top_p=0.95,
    stream=False,
    tools=tools,
)
print_highlight("Non-stream response:")
print_highlight(final_response)

print_highlight("==== Text ====")
print_highlight(final_response.choices[0].message.content)
```

## Native API and SGLang Runtime (SRT)

```python Example theme={null}
from transformers import AutoTokenizer
import requests

# generate an answer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

messages = get_messages()

input = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, tools=tools, return_dict=False
)

gen_url = f"http://localhost:{port}/generate"
gen_data = {
    "text": input,
    "sampling_params": {
        "skip_special_tokens": False,
        "max_new_tokens": 1024,
        "temperature": 0,
        "top_p": 0.95,
    },
}
gen_response = requests.post(gen_url, json=gen_data).json()["text"]
print_highlight("==== Response ====")
print_highlight(gen_response)

# parse the response
parse_url = f"http://localhost:{port}/parse_function_call"

function_call_input = {
    "text": gen_response,
    "tool_call_parser": "qwen25",
    "tools": tools,
}

function_call_response = requests.post(parse_url, json=function_call_input)
function_call_response_json = function_call_response.json()

print_highlight("==== Text ====")
print(function_call_response_json["normal_text"])
print_highlight("==== Calls ====")
print("function name: ", function_call_response_json["calls"][0]["name"])
print("function arguments: ", function_call_response_json["calls"][0]["parameters"])
```

```python Example theme={null}
terminate_process(server_process)
```

## Offline Engine API

```python Example theme={null}
import sglang as sgl
from sglang.srt.function_call.function_call_parser import FunctionCallParser
from sglang.srt.managers.io_struct import Tool, Function

llm = sgl.Engine(model_path="Qwen/Qwen2.5-7B-Instruct")
tokenizer = llm.tokenizer_manager.tokenizer
input_ids = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, tools=tools, return_dict=False
)

# Note that for gpt-oss tool parser, adding "no_stop_trim": True
# to make sure the tool call token <call> is not trimmed.

sampling_params = {
    "max_new_tokens": 1024,
    "temperature": 0,
    "top_p": 0.95,
    "skip_special_tokens": False,
}

# 1) Offline generation
result = llm.generate(input_ids=input_ids, sampling_params=sampling_params)
generated_text = result["text"]  # Assume there is only one prompt

print_highlight("=== Offline Engine Output Text ===")
print_highlight(generated_text)


# 2) Parse using FunctionCallParser
def convert_dict_to_tool(tool_dict: dict) -> Tool:
    function_dict = tool_dict.get("function", {})
    return Tool(
        type=tool_dict.get("type", "function"),
        function=Function(
            name=function_dict.get("name"),
            description=function_dict.get("description"),
            parameters=function_dict.get("parameters"),
        ),
    )


tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools]

parser = FunctionCallParser(tools=tools, tool_call_parser="qwen25")
normal_text, calls = parser.parse_non_stream(generated_text)

print_highlight("=== Parsing Result ===")
print("Normal text portion:", normal_text)
print_highlight("Function call portion:")
for call in calls:
    # call: ToolCallItem
    print_highlight(f"  - tool name: {call.name}")
    print_highlight(f"    parameters: {call.parameters}")

# 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc.
```

```python Example theme={null}
llm.shutdown()
```

## Tool Choice Mode

SGLang supports OpenAI's `tool_choice` parameter to control when and which tools the model should call. This feature is implemented using EBNF (Extended Backus-Naur Form) grammar to ensure reliable tool calling behavior.

### Supported Tool Choice Options

* **`tool_choice="required"`**: Forces the model to call at least one tool
* **`tool_choice={"type": "function", "function": {"name": "specific_function"}}`**: Forces the model to call a specific function

### Backend Compatibility

Tool choice is fully supported with the **Xgrammar backend**, which is the default grammar backend (`--grammar-backend xgrammar`). However, it may not be fully supported with other backends such as `outlines`.

### Example: Required Tool Choice

```python Example theme={null}
from openai import OpenAI
from sglang.utils import wait_for_server, print_highlight, terminate_process
from sglang.test.doc_patch import launch_server_cmd

# Start a new server session for tool choice examples
server_process_tool_choice, port_tool_choice = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0  --log-level warning"
)
wait_for_server(f"http://localhost:{port_tool_choice}")

# Initialize client for tool choice examples
client_tool_choice = OpenAI(
    api_key="None", base_url=f"http://0.0.0.0:{port_tool_choice}/v1"
)
model_name_tool_choice = client_tool_choice.models.list().data[0].id

# Example with tool_choice="required" - forces the model to call a tool
messages_required = [
    {"role": "user", "content": "Hello, what is the capital of France?"}
]

# Define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "The city to find the weather for, e.g. 'San Francisco'",
                    },
                    "unit": {
                        "type": "string",
                        "description": "The unit to fetch the temperature in",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["city", "unit"],
            },
        },
    }
]

response_required = client_tool_choice.chat.completions.create(
    model=model_name_tool_choice,
    messages=messages_required,
    temperature=0,
    max_tokens=1024,
    tools=tools,
    tool_choice="required",  # Force the model to call a tool
)

print_highlight("Response with tool_choice='required':")
print("Content:", response_required.choices[0].message.content)
print("Tool calls:", response_required.choices[0].message.tool_calls)
```

### Example: Specific Function Choice

```python Example theme={null}
# Example with specific function choice - forces the model to call a specific function
messages_specific = [
    {"role": "user", "content": "What are the most attactive places in France?"}
]

response_specific = client_tool_choice.chat.completions.create(
    model=model_name_tool_choice,
    messages=messages_specific,
    temperature=0,
    max_tokens=1024,
    tools=tools,
    tool_choice={
        "type": "function",
        "function": {"name": "get_current_weather"},
    },  # Force the model to call the specific get_current_weather function
)

print_highlight("Response with specific function choice:")
print("Content:", response_specific.choices[0].message.content)
print("Tool calls:", response_specific.choices[0].message.tool_calls)

if response_specific.choices[0].message.tool_calls:
    tool_call = response_specific.choices[0].message.tool_calls[0]
    print_highlight(f"Called function: {tool_call.function.name}")
    print_highlight(f"Arguments: {tool_call.function.arguments}")
```

```python Example theme={null}
terminate_process(server_process_tool_choice)
```

## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4)

Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a "pythonic" tool call format, where the model outputs function calls as Python code, e.g.:

```python Example theme={null}
[get_current_weather(city="San Francisco", state="CA", unit="celsius")]
```

* The output is a Python list of function calls, with arguments as Python literals (not JSON).
* Multiple tool calls can be returned in the same list:

```python Example theme={null}
[get_current_weather(city="San Francisco", state="CA", unit="celsius"),
 get_current_weather(city="New York", state="NY", unit="fahrenheit")]
```

For more information, refer to Meta’s documentation on  [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message).

Note that this feature is still under development on Blackwell.

### How to enable

* Launch the server with `--tool-call-parser pythonic`
* You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`).
  This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results.

#### Forcing Pythonic Tool Call Output Without a Chat Template

If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need:

```python Example theme={null}
import openai

server_process, port = launch_server_cmd(
    " python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1  --log-level warning"  # llama-3.2-1b-instruct
)
wait_for_server(f"http://localhost:{port}")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a given location.",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The name of the city or location.",
                    }
                },
                "required": ["location"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_tourist_attractions",
            "description": "Get a list of top tourist attractions for a given city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "The name of the city to find attractions for.",
                    }
                },
                "required": ["city"],
            },
        },
    },
]


def get_messages():
    return [
        {
            "role": "system",
            "content": (
                "You are a travel assistant. "
                "When asked to call functions, ALWAYS respond ONLY with a python list of function calls, "
                "using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. "
                "Do NOT use JSON, do NOT use variables, do NOT use any other format. "
                "Here is an example:\n"
                '[get_weather(location="Paris"), get_tourist_attractions(city="Paris")]'
            ),
        },
        {
            "role": "user",
            "content": (
                "I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? "
                "Propose parallel tool calls at once, using the python list of function calls format as shown above."
            ),
        },
    ]


messages = get_messages()

client = openai.Client(base_url=f"http://localhost:{port}/v1", api_key="xxxxxx")
model_name = client.models.list().data[0].id


response_non_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0,
    top_p=0.9,
    stream=False,  # Non-streaming
    tools=tools,
)
print_highlight("Non-stream response:")
print_highlight(response_non_stream)

response_stream = client.chat.completions.create(
    model=model_name,
    messages=messages,
    temperature=0,
    top_p=0.9,
    stream=True,
    tools=tools,
)
texts = ""
tool_calls = []
name = ""
arguments = ""

for chunk in response_stream:
    if chunk.choices[0].delta.content:
        texts += chunk.choices[0].delta.content
    if chunk.choices[0].delta.tool_calls:
        tool_calls.append(chunk.choices[0].delta.tool_calls[0])

print_highlight("Streaming Response:")
print_highlight("==== Text ====")
print_highlight(texts)

print_highlight("==== Tool Call ====")
for tool_call in tool_calls:
    print_highlight(tool_call)

terminate_process(server_process)
```

> **Note:**
> The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template.

## How to support a new model?

1. Update the TOOLS\_TAG\_LIST in sglang/srt/function\_call\_parser.py with the model’s tool tags. Currently supported tags include:

```text Output theme={null}
	TOOLS_TAG_LIST = [
	    “<|plugin|>“,
	    “<function=“,
	    “<tool_call>“,
	    “<|python_tag|>“,
	    “[TOOL_CALLS]”
	]
```

2. Create a new detector class in sglang/srt/function\_call\_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example:

```text Output theme={null}
    class NewModelDetector(BaseFormatDetector):
```

3. Add the new detector to the MultiFormatParser class that manages all the format detectors.


# Query VLM with Offline Engine
Source: https://docs.sglang.io/docs/advanced_features/vlm_query



This tutorial demonstrates how to use SGLang's **offline Engine API** to query VLMs. We will demonstrate usage with Qwen2.5-VL and Llama 4. This section demonstrates three different calling approaches:

1. **Basic Call**: Directly pass images and text.
2. **Processor Output**: Use HuggingFace processor for data preprocessing.
3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency.

## Understanding the Three Input Formats

SGLang supports three ways to pass visual data, each optimized for different scenarios:

### 1. **Raw Images** - Simplest approach

* Pass PIL Images, file paths, URLs, or base64 strings directly
* SGLang handles all preprocessing automatically
* Best for: Quick prototyping, simple applications

### 2. **Processor Output** - For custom preprocessing

* Pre-process images with HuggingFace processor
* Pass the complete processor output dict with `format: "processor_output"`
* Best for: Custom image transformations, integration with existing pipelines
* Requirement: Must use `input_ids` instead of text prompt

### 3. **Precomputed Embeddings** - For maximum performance

* Pre-calculate visual embeddings using the vision encoder
* Pass embeddings with `format: "precomputed_embedding"`
* Best for: Repeated queries on same images, caching, high-throughput serving
* Performance gain: Avoids redundant vision encoder computation (30-50% speedup)

**Key Rule**: Within a single request, use only one format for all images. Don't mix formats.

The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models.

## Querying Qwen2.5-VL Model

```python Example theme={null}
import nest_asyncio

nest_asyncio.apply()

import sglang.test.doc_patch  # noqa: F401

model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
chat_template = "qwen2-vl"
example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
```

```python Example theme={null}
from io import BytesIO
import requests
from PIL import Image

from sglang.srt.parser.conversation import chat_templates

image = Image.open(BytesIO(requests.get(example_image_url).content))

conv = chat_templates[chat_template].copy()
conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
conv.append_message(conv.roles[1], "")
conv.image_data = [image]

print("Generated prompt text:")
print(conv.get_prompt())
print(f"\nImage size: {image.size}")
image
```

### Basic Offline Engine API Call

```python Example theme={null}
from sglang import Engine

llm = Engine(model_path=model_path, chat_template=chat_template, log_level="warning")
```

```python Example theme={null}
out = llm.generate(prompt=conv.get_prompt(), image_data=[image])
print("Model response:")
print(out["text"])
```

### Call with Processor Output

Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`.

```python Example theme={null}
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained(model_path)
processor_output = processor(
    images=[image], text=conv.get_prompt(), return_tensors="pt"
)

out = llm.generate(
    input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
    image_data=[dict(processor_output, format="processor_output")],
)
print("Response using processor output:")
print(out["text"])
```

### Call with Precomputed Embeddings

You can pre-calculate image features to avoid repeated visual encoding processes.

```python Example theme={null}
from transformers import AutoProcessor
from transformers import Qwen2_5_VLForConditionalGeneration

processor = AutoProcessor.from_pretrained(model_path)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval()
vision = model.model.visual.cuda()
```

```python Example theme={null}
processor_output = processor(
    images=[image], text=conv.get_prompt(), return_tensors="pt"
)

input_ids = processor_output["input_ids"][0].detach().cpu().tolist()

precomputed_embeddings = vision(
    processor_output["pixel_values"].cuda(), processor_output["image_grid_thw"].cuda()
)
precomputed_embeddings = precomputed_embeddings.pooler_output

multi_modal_item = dict(
    processor_output,
    format="precomputed_embedding",
    feature=precomputed_embeddings,
)

out = llm.generate(input_ids=input_ids, image_data=[multi_modal_item])
print("Response using precomputed embeddings:")
print(out["text"])

llm.shutdown()
```

## Querying Llama 4 Vision Model

```python Example theme={null}
model_path = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
chat_template = "llama-4"

from io import BytesIO
import requests
from PIL import Image

from sglang.srt.parser.conversation import chat_templates

# Download the same example image
image = Image.open(BytesIO(requests.get(example_image_url).content))

conv = chat_templates[chat_template].copy()
conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
conv.append_message(conv.roles[1], "")
conv.image_data = [image]

print("Llama 4 generated prompt text:")
print(conv.get_prompt())
print(f"Image size: {image.size}")

image
```

### Llama 4 Basic Call

Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp\_size=4) and larger context length.

```python Example theme={null}
llm = Engine(
    model_path=model_path,
    enable_multimodal=True,
    attention_backend="fa3",
    tp_size=4,
    context_length=65536,
)

out = llm.generate(prompt=conv.get_prompt(), image_data=[image])
print("Llama 4 response:")
print(out["text"])
```

### Call with Processor Output

Using HuggingFace processor to preprocess data can reduce computational overhead during inference.

```python Example theme={null}
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained(model_path)
processor_output = processor(
    images=[image], text=conv.get_prompt(), return_tensors="pt"
)

out = llm.generate(
    input_ids=processor_output["input_ids"][0].detach().cpu().tolist(),
    image_data=[dict(processor_output, format="processor_output")],
)
print("Response using processor output:")
print(out)
```

### Call with Precomputed Embeddings

```python Example theme={null}
from transformers import AutoProcessor
from transformers import Llama4ForConditionalGeneration

processor = AutoProcessor.from_pretrained(model_path)
model = Llama4ForConditionalGeneration.from_pretrained(
    model_path, torch_dtype="auto"
).eval()

vision = model.vision_model.cuda()
multi_modal_projector = model.multi_modal_projector.cuda()

print(f'Image pixel values shape: {processor_output["pixel_values"].shape}')
input_ids = processor_output["input_ids"][0].detach().cpu().tolist()

# Process image through vision encoder
image_outputs = vision(
    processor_output["pixel_values"].to("cuda"),
    aspect_ratio_ids=processor_output["aspect_ratio_ids"].to("cuda"),
    aspect_ratio_mask=processor_output["aspect_ratio_mask"].to("cuda"),
    output_hidden_states=False
)
image_features = image_outputs.last_hidden_state

# Flatten image features and pass through multimodal projector
vision_flat = image_features.view(-1, image_features.size(-1))
precomputed_embeddings = multi_modal_projector(vision_flat)

# Build precomputed embedding data item
mm_item = dict(
    processor_output,
    format="precomputed_embedding",
    feature=precomputed_embeddings
)

# Use precomputed embeddings for efficient inference
out = llm.generate(input_ids=input_ids, image_data=[mm_item])
print("Llama 4 precomputed embedding response:")
print(out["text"])
```


# Anthropic-Compatible API
Source: https://docs.sglang.io/docs/basic_usage/anthropic_api

Use the Anthropic Messages API (/v1/messages) with SGLang, including Claude Code integration and prefix-cache tuning.

SGLang ships an Anthropic-compatible `/v1/messages` endpoint so any client built for the Anthropic
Messages API — including the Anthropic SDKs and agentic CLIs such as Claude Code — can talk to a
self-hosted SGLang server without changes. A complete reference for the API is available in the
[Anthropic API Reference](https://docs.anthropic.com/en/api/messages).

The endpoint is registered automatically on every SGLang server; no extra flag is required to enable it.
It reuses the same model, chat template, and reasoning / tool-call parsers as the OpenAI-compatible
endpoint, and supports both non-streaming and streaming responses, tool use, and a `count_tokens` route.

This tutorial covers:

* `POST /v1/messages` (non-streaming and streaming)
* `POST /v1/messages/count_tokens`
* Pointing **Claude Code** at the server, including the `CLAUDE_CODE_ATTRIBUTION_HEADER` setting that is
  required for good prefix-cache reuse.

## Launch A Server

Launch the server in your terminal and wait for it to initialize. The Anthropic `/v1/messages` endpoint
is registered automatically — no extra flag is required beyond the usual server launch. The example below
is a single-node GLM-5.2-FP8 config; see the
[GLM-5.2 cookbook](/cookbook/autoregressive/GLM/GLM-5.2) for verified commands
across hardware and quantizations.

```bash Command theme={null}
sglang serve \
    --model-path zai-org/GLM-5.2-FP8 \
    --tp 8 \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 5 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 6 \
    --reasoning-parser glm45 \
    --tool-call-parser glm47 \
    --host 0.0.0.0 \
    --port 30000
```

> [!NOTE]
>
> * **The endpoint is model-agnostic.** The `/v1/messages` route is on by default for any model; GLM-5.2 is
>   used here because its reasoning + tool-use output is where Claude Code integration shines, but any model
>   works.
> * **Model name and `[1m]`.** SGLang does not validate the request `model` field, so Claude Code can send
>   any name. The `[1m]` suffix is a **client-side hint**: Claude Code only enables its 1M-context beta when
>   the model name ends in `[1m]` — without it, context is capped. Set the same `glm-5.2[1m]` in the
>   `ANTHROPIC_DEFAULT_*_MODEL` env vars below.
> * **`--reasoning-parser` / `--tool-call-parser` are optional.** Add them when the model emits reasoning
>   content (GLM-5.2, Qwen3, DeepSeek-R1, …) or when you want tool calls parsed into structured `tool_use`
>   blocks. Without a tool-call parser, tool schemas are still accepted but the model's tool calls come back
>   as raw text, and Claude Code cannot execute them.
> * **Context length** defaults to the model's own (1M for GLM-5.2); pass `--context-length` only to cap it.

## Send A Message

### Non-Streaming

Use the Anthropic Python SDK pointed at the server. Unlike the OpenAI SDK, the Anthropic SDK appends
`/v1/messages` itself, so `base_url` is the server root **without** a `/v1` suffix.

```python Example theme={null}
from anthropic import Anthropic

client = Anthropic(
    base_url="http://127.0.0.1:30000",
    api_key="EMPTY",  # SGLang does not require a real key by default
)

message = client.messages.create(
    model="zai-org/GLM-5.2-FP8",
    max_tokens=512,
    messages=[{"role": "user", "content": "List 3 countries and their capitals."}],
)
# A reasoning model may emit a `thinking` block before the `text` block —
# pick the text block rather than assuming content[0].
print(next(b.text for b in message.content if b.type == "text"))
```

**Example Output:**

```text Output theme={null}
Here are 3 countries and their capitals:

1. **France** - Paris
2. **Japan** - Tokyo
3. **Brazil** - Brasília
```

### Streaming

Set `stream=True` to receive Server-Sent Events as they are produced.

```python Example theme={null}
with client.messages.stream(
    model="zai-org/GLM-5.2-FP8",
    max_tokens=512,
    messages=[{"role": "user", "content": "Say this is a test"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
```

**Example Output:**

```text Output theme={null}
This is a test.
```

### System Prompt

The top-level `system` field is accepted as a string or as a list of text blocks, matching the Anthropic
API shape:

```python Example theme={null}
message = client.messages.create(
    model="zai-org/GLM-5.2-FP8",
    max_tokens=512,
    system="You are a helpful assistant that answers concisely.",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(next(b.text for b in message.content if b.type == "text"))
```

**Example Output:**

```text Output theme={null}
The capital of France is Paris.
```

### Tool Use

Tool definitions follow the Anthropic `tools` schema. When the server is launched with a
`--tool-call-parser`, the model's tool calls are returned as `tool_use` content blocks:

```python Example theme={null}
message = client.messages.create(
    model="zai-org/GLM-5.2-FP8",
    max_tokens=512,
    tools=[
        {
            "name": "get_weather",
            "description": "Get the weather for a city",
            "input_schema": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        }
    ],
    messages=[{"role": "user", "content": "What is the weather in Paris?"}],
)
print(message.stop_reason)
print([b for b in message.content if b.type == "tool_use"])
```

**Example Output:**

```text Output theme={null}
tool_use
[ToolUseBlock(type='tool_use', id='toolu_01XXXX', name='get_weather', input={'city': 'Paris'})]
```

### Counting Tokens

`POST /v1/messages/count_tokens` returns the tokenized length of a request without generating a
response. It reuses the same request conversion as `/v1/messages`, so system prompts, tools, and
multi-turn history are all accounted for.

```python Example theme={null}
resp = client.messages.count_tokens(
    model="zai-org/GLM-5.2-FP8",
    messages=[{"role": "user", "content": "Hello, world"}],
)
print(resp.input_tokens)
```

**Example Output:**

```text Output theme={null}
15
```

## Using Claude Code

Claude Code can be pointed at an SGLang server by setting a few env vars in the shell that starts it.
With the server already running on `:30000`, export the full set and launch `claude`:

```bash Command theme={null}
export ANTHROPIC_BASE_URL="http://127.0.0.1:30000"
export ANTHROPIC_AUTH_TOKEN="dummy"                 # required by Claude Code; any non-empty string works
export API_TIMEOUT_MS="3000000"                     # long timeout — reasoning + 1M-context turns are slow
export CLAUDE_CODE_AUTO_COMPACT_WINDOW="1000000"     # let auto-compact use the full 1M window
export CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1   # drop autoupdater/telemetry/error-reporting noise
export CLAUDE_CODE_ATTRIBUTION_HEADER=0             # required for prefix-cache reuse — see below
export ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-5.2[1m]"    # [1m] suffix enables Claude Code's 1M-context beta
export ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5.2[1m]"  # [1m] suffix enables Claude Code's 1M-context beta
export ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.2[1m]"     # [1m] suffix enables Claude Code's 1M-context beta
claude
```

Each var matters:

* **`ANTHROPIC_BASE_URL`** — points Claude Code at your SGLang server instead of the Anthropic API.
* **`ANTHROPIC_AUTH_TOKEN`** — Claude Code requires a non-empty auth token; SGLang accepts any value
  when launched without `--api-key`.
* **`API_TIMEOUT_MS`** — raise it; reasoning models with long outputs and 1M-context turns routinely
  exceed the default timeout.
* **`ANTHROPIC_DEFAULT_{HAIKU,SONNET,OPUS}_MODEL`** — the model name Claude Code sends for each tier.
  SGLang does not validate this field, so any name works. Use `glm-5.2[1m]`: the `[1m]` suffix is a
  client-side hint that enables Claude Code's 1M-context beta (without it, context is capped).
* **`CLAUDE_CODE_AUTO_COMPACT_WINDOW`** — set to `1000000` so auto-compaction uses the full 1M window
  instead of the default, keeping long sessions alive.

> [!TIP]
>
> Instead of exporting these in every shell, persist them in `~/.claude/settings.json` under the `env` key
> — they apply to all Claude Code sessions:
>
> ```json theme={null}
> {
>   "env": {
>     "ANTHROPIC_BASE_URL": "http://127.0.0.1:30000",
>     "ANTHROPIC_AUTH_TOKEN": "dummy",
>     "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1",
>     "CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
>     "ANTHROPIC_DEFAULT_HAIKU_MODEL": "glm-5.2[1m]",
>     "ANTHROPIC_DEFAULT_SONNET_MODEL": "glm-5.2[1m]",
>     "ANTHROPIC_DEFAULT_OPUS_MODEL": "glm-5.2[1m]"
>   }
> }
> ```

### Required: `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for prefix-cache reuse

> [!NOTE]
>
> **Set this whenever Claude Code routes through SGLang (or any non-Anthropic gateway).** Without it,
> multi-turn conversations re-prefill the whole history every turn.

Claude Code prepends a per-request attribution block to the start of the system prompt, of the form
`x-anthropic-billing-header: cc_version=<ver>.<per-request-hash>; cc_entrypoint=...; cch=<hash>;`. The
per-request hash is the **first token to differ between turns**, so the radix prefix cache can only reuse
the short prefix before that hash and re-prefills the system prompt plus the entire conversation history
on every turn.

Setting `CLAUDE_CODE_ATTRIBUTION_HEADER=0` removes the whole attribution line from the system prompt.
This is a documented Claude Code env var whose explicit purpose is to "improve prompt-cache hit rates when
routing through an [LLM gateway](https://code.claude.com/docs/en/llm-gateway)" (see the [Claude Code env-vars reference](https://code.claude.com/docs/en/env-vars)).

> [!NOTE]
>
> `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` does **not** remove the attribution block — it only covers
> autoupdater/telemetry/error reporting. The attribution header is a separate code path; use
> `CLAUDE_CODE_ATTRIBUTION_HEADER=0` for it.

## Troubleshooting

**Connection refused / `fetch failed`** — Ensure the server is up and the port in `ANTHROPIC_BASE_URL`
matches `--port` (default 30000). If you set `ANTHROPIC_BASE_URL` to a remote host, confirm it's reachable
and not behind a proxy that blocks the connection.

**`Model not found` / 404 from the server** — SGLang does not validate the request `model` field and
serves whatever model was loaded at startup, so a 404 usually means the request did not reach the
`/v1/messages` route at all. Confirm `ANTHROPIC_BASE_URL` points at the server (not missing the port) and
that the server finished loading.

**Tool calls not working / returned as raw text** — Launch the server with the correct
`--tool-call-parser` for your model (e.g. `glm47`, `qwen3`). Without it the `tools` field is still accepted
but the model's tool calls come back as text instead of `tool_use` blocks, and Claude Code cannot execute
them.

**Slow / re-prefills the whole history every turn** — You are missing
`CLAUDE_CODE_ATTRIBUTION_HEADER=0`. Claude Code's per-request attribution hash in the system prompt
defeats radix prefix-cache reuse; see the section above.

**Context capped below 1M** — The model name must end in `[1m]` for Claude Code to enable its 1M-context
beta. Verify `ANTHROPIC_DEFAULT_*_MODEL` uses the `[1m]` suffix, and that the loaded model's native context
is 1M (GLM-5.2 is 1048576; pass `--context-length` only to cap it, not to extend).

## Parameters

The `/v1/messages` endpoint accepts the standard Anthropic Messages API parameters. Refer to the
[Anthropic Messages API reference](https://docs.anthropic.com/en/api/messages) for the full list.

Reasoning models are supported through the same `--reasoning-parser` mechanism as the OpenAI-compatible
endpoint; pass the model's reasoning kwarg via the request (e.g. `thinking` for DeepSeek-V3-style models,
`enable_thinking` for Qwen3-style models). See [OpenAI APIs - Completions](./openai_api_completions) for
the reasoning-parser / chat-template mapping.


# Amazon SageMaker AI
Source: https://docs.sglang.io/docs/basic_usage/aws_sagemaker

Deploy SGLang on Amazon SageMaker AI endpoints using the AWS Deep Learning Container.

Deploy SGLang on [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/) endpoints using the
[AWS Deep Learning Container (DLC)](https://aws.github.io/deep-learning-containers/sglang/) for SGLang.
The SageMaker image variant accepts model configuration via environment variables and serves on port 8080.

This guide uses the pre-built DLC image. To build and deploy your own container instead, see
[Method 7: Run on AWS SageMaker](/docs/get-started/install#aws-sagemaker) in the installation guide.

## Container image

AWS publishes pre-built, security-patched SGLang DLCs. The SageMaker GPU image is available from the
Amazon ECR registry (account `763104351884`) in each supported region. For example, in `us-west-2`:

```text theme={null}
763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0
```

For the full list of image tags, see the
[Available DLC Images](https://aws.github.io/deep-learning-containers/reference/available_images/) reference,
and for region-specific account IDs and supported regions, see
[Region Availability](https://aws.github.io/deep-learning-containers/reference/region_availability/).

## Specifying the model

The SageMaker image resolves the model in this order:

1. **`SM_SGLANG_MODEL_PATH` environment variable** — explicit Hugging Face ID or path.
2. **`/opt/ml/model`** — when SageMaker mounts model artifacts via `ModelDataUrl` or `ModelDataSource`,
   the entrypoint uses this path by default.

For gated models, also pass `HF_TOKEN`.

Any `SM_SGLANG_*` environment variable is converted to a `--<name>` SGLang server argument
(for example, `SM_SGLANG_CONTEXT_LENGTH=4096` becomes `--context-length 4096`).

## Deploy with the SageMaker Python SDK

```python theme={null}
from sagemaker.model import Model
from sagemaker.predictor import Predictor
from sagemaker.serializers import JSONSerializer

model = Model(
    image_uri="763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0",
    role="arn:aws:iam::<account_id>:role/<role_name>",
    predictor_cls=Predictor,
    env={"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"},
)

predictor = model.deploy(
    instance_type="ml.g5.2xlarge",
    initial_instance_count=1,
    inference_ami_version="al2023-ami-sagemaker-inference-gpu-4-1",
    serializer=JSONSerializer(),
)

response = predictor.predict({
    "model": "openai/gpt-oss-20b",
    "messages": [{"role": "user", "content": "What is deep learning?"}],
    "max_tokens": 256,
})
print(response)

# Cleanup
predictor.delete_model()
predictor.delete_endpoint(delete_endpoint_config=True)
```

## Deploy with Boto3

```python theme={null}
import json
import boto3

sm = boto3.client("sagemaker")
smrt = boto3.client("sagemaker-runtime")

sm.create_model(
    ModelName="sglang-model",
    PrimaryContainer={
        "Image": "763104351884.dkr.ecr.us-west-2.amazonaws.com/sglang:server-sagemaker-cuda-v1.0",
        "Environment": {"SM_SGLANG_MODEL_PATH": "openai/gpt-oss-20b"},
    },
    ExecutionRoleArn="arn:aws:iam::<account_id>:role/<role_name>",
)

sm.create_endpoint_config(
    EndpointConfigName="sglang-config",
    ProductionVariants=[{
        "VariantName": "default",
        "ModelName": "sglang-model",
        "InstanceType": "ml.g5.2xlarge",
        "InitialInstanceCount": 1,
        "InferenceAmiVersion": "al2023-ami-sagemaker-inference-gpu-4-1",
    }],
)

sm.create_endpoint(EndpointName="sglang-endpoint", EndpointConfigName="sglang-config")
sm.get_waiter("endpoint_in_service").wait(EndpointName="sglang-endpoint")

resp = smrt.invoke_endpoint(
    EndpointName="sglang-endpoint",
    ContentType="application/json",
    Body=json.dumps({
        "model": "openai/gpt-oss-20b",
        "messages": [{"role": "user", "content": "What is deep learning?"}],
        "max_tokens": 256,
    }),
)
print(json.loads(resp["Body"].read()))

# Cleanup
sm.delete_endpoint(EndpointName="sglang-endpoint")
sm.delete_endpoint_config(EndpointConfigName="sglang-config")
sm.delete_model(ModelName="sglang-model")
```

## Model artifacts

When `ModelDataUrl` (or `ModelDataSource`) points to a tarball or S3 prefix, SageMaker mounts the contents
at `/opt/ml/model`. The entrypoint defaults `--model-path` to that location, so `SM_SGLANG_MODEL_PATH`
can be omitted:

```text theme={null}
model.tar.gz
├── config.json              # standard model files (Hugging Face layout)
├── tokenizer.json
└── *.safetensors
```

## Notes

* GPU deployments require `inference_ami_version` — the default SageMaker host AMI has incompatible NVIDIA
  drivers for CUDA 13 images. See the
  [ProductionVariant API reference](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html)
  for valid values.
* The endpoint exposes an OpenAI-compatible API, so the request body matches the SGLang server's
  `/v1/chat/completions` schema.


# SGLang Native APIs
Source: https://docs.sglang.io/docs/basic_usage/native_api



Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce the following APIs:

* `/generate` (text generation model)
* `/get_model_info`
* `/server_info`
* `/health`
* `/health_generate`
* `/flush_cache`
* `/update_weights`
* `/encode`(embedding model)
* `/v1/rerank`(cross encoder rerank model)
* `/v1/score`(decoder-only scoring)
* `/v1/decisions`(typed decisions on a chat model)
* `/classify`(reward model)
* `/start_expert_distribution_record`
* `/stop_expert_distribution_record`
* `/dump_expert_distribution_record`
* `/tokenize`
* `/detokenize`
* A full list of these APIs can be found at [http\_server.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/http_server.py)

We mainly use `requests` to test these APIs in the following examples. You can also use `curl`.

## Launch A Server

```python Example theme={null}
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=server_process)
```

## Generate (text generation model)

Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params).

```python Example theme={null}
import requests

url = f"http://localhost:{port}/generate"
data = {"text": "What is the capital of France?"}

response = requests.post(url, json=data)
print_highlight(response.json())
```

## Get Model Info

Get the information of the model.

* `model_path`: The path/name of the model.
* `is_generation`: Whether the model is used as generation model or embedding model.
* `tokenizer_path`: The path/name of the tokenizer.
* `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args.
* `weight_version`: This field contains the version of the model weights. This is often used to track changes or updates to the model’s trained parameters.
* `has_image_understanding`: Whether the model has image-understanding capability.
* `has_audio_understanding`: Whether the model has audio-understanding capability.
* `model_type`: The model type from the HuggingFace config (e.g., "qwen2", "llama").
* `architectures`: The model architectures from the HuggingFace config (e.g., \["Qwen2ForCausalLM"]).
* `embedding`: The resolved embedding-serving plan. It includes pooling, normalization, execution and attention style, Matryoshka dimensions, cache policy, and effective BCG prefill settings. This field is available when the model configuration exposes an embedding capability contract.

```python Example theme={null}
url = f"http://localhost:{port}/get_model_info"

response = requests.get(url)
response_json = response.json()
print_highlight(response_json)
assert response_json["model_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["is_generation"] is True
assert response_json["tokenizer_path"] == "qwen/qwen2.5-0.5b-instruct"
assert response_json["preferred_sampling_params"] is None
assert response_json.keys() == {
    "model_path",
    "is_generation",
    "tokenizer_path",
    "preferred_sampling_params",
    "weight_version",
    "has_image_understanding",
    "has_audio_understanding",
    "model_type",
    "architectures",
    "embedding",
}
```

## Get Server Info

Gets the server information including CLI arguments, token limits, and memory pool sizes.

* Note: `get_server_info` merges the following deprecated endpoints:
  * `get_server_args`
  * `get_memory_pool_size`
  * `get_max_total_num_tokens`

```python Example theme={null}
url = f"http://localhost:{port}/server_info"

response = requests.get(url)
print_highlight(response.text)
```

## Health Check

* `/health`: Check the health of the server.
* `/health_generate`: Check the health of the server by generating one token.

```python Example theme={null}
url = f"http://localhost:{port}/health_generate"

response = requests.get(url)
print_highlight(response.text)
```

```python Example theme={null}
url = f"http://localhost:{port}/health"

response = requests.get(url)
print_highlight(response.text)
```

## Flush Cache

Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API.

Parameters:

* `timeout` (query, float, default `0`, unit: seconds): Wait time for idle state before flushing. `0` means fail fast if not idle. When HiCache async operations are in-flight, a non-zero timeout allows the server to wait until idle before flushing, avoiding unnecessary 400 errors.

```bash Command theme={null}
# With timeout (wait up to 30s for idle state)
curl -s -X POST "http://127.0.0.1:30000/flush_cache?timeout=30"
```

```python Example theme={null}
url = f"http://localhost:{port}/flush_cache"

response = requests.post(url)
print_highlight(response.text)
```

## Update Weights From Disk

Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.

SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).

```python Example theme={null}
# successful update with same architecture and size

url = f"http://localhost:{port}/update_weights_from_disk"
data = {"model_path": "qwen/qwen2.5-0.5b-instruct"}

response = requests.post(url, json=data)
print_highlight(response.text)
assert response.json()["success"] is True
assert response.json()["message"] == "Succeeded to update model weights."
```

```python Example theme={null}
# failed update with different parameter size or wrong name

url = f"http://localhost:{port}/update_weights_from_disk"
data = {"model_path": "qwen/qwen2.5-0.5b-instruct-wrong"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(response_json)
assert response_json["success"] is False
assert response_json["message"] == (
    "Failed to get weights iterator: "
    "qwen/qwen2.5-0.5b-instruct-wrong"
    " (repository not found)."
)
```

```python Example theme={null}
terminate_process(server_process)
```

## Encode (embedding model)

Encode text into embeddings. Note that this API is only available for [embedding models](./openai_api_embeddings) and will raise an error for generation models.
Therefore, we launch a new server to server an embedding model.

```python Example theme={null}
embedding_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
    --host 0.0.0.0 --is-embedding --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=embedding_process)
```

```python Example theme={null}
# successful encode for embedding model

url = f"http://localhost:{port}/encode"
data = {"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "text": "Once upon a time"}

response = requests.post(url, json=data)
response_json = response.json()
print_highlight(f"Text embedding (first 10): {response_json['embedding'][:10]}")
```

```python Example theme={null}
terminate_process(embedding_process)
```

## v1/rerank (cross encoder rerank model)

Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.

```python Example theme={null}
reranker_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \
    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=reranker_process)
```

```python Example theme={null}
# compute rerank scores for query and documents

url = f"http://localhost:{port}/v1/rerank"
data = {
    "model": "BAAI/bge-reranker-v2-m3",
    "query": "what is panda?",
    "documents": [
        "hi",
        "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.",
    ],
}

response = requests.post(url, json=data)
response_json = response.json()
for item in response_json:
    print_highlight(f"Score: {item['score']:.2f} - Document: '{item['document']}'")
```

```python Example theme={null}
terminate_process(reranker_process)
```

## v1/score (decoder-only scoring)

Compute token probabilities for specified tokens given a query and items. This is useful for classification tasks, scoring responses, or computing log-probabilities.

Parameters:

* `query`: Query text
* `items`: Item text(s) to score
* `label_token_ids`: Token IDs to compute probabilities for
* `apply_softmax`: Whether to apply softmax to get normalized probabilities (default: False)
* `item_first`: Whether items come first in concatenation order (default: False)
* `model`: Model name

`query` and `items` also accept token ids, with `query` set to `[]` when each item is a complete prompt. `label_token_ids` can be one list per item. `temperature` divides the label logits before the softmax and needs `apply_softmax`, and `return_token_logprobs` adds `token_logprobs`, the full-vocabulary log-probabilities of the labels.

The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`.

```python Example theme={null}
score_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
    --host 0.0.0.0 --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=score_process)
```

```python Example theme={null}
# Score the probability of different completions given a query
query = "The capital of France is"
items = ["Paris", "London", "Berlin"]

url = f"http://localhost:{port}/v1/score"
data = {
    "model": "qwen/qwen2.5-0.5b-instruct",
    "query": query,
    "items": items,
    "label_token_ids": [9454, 2753],  # e.g. "Yes" and "No" token ids
    "apply_softmax": True,  # Normalize probabilities to sum to 1
}

response = requests.post(url, json=data)
response_json = response.json()

# Display scores for each item
for item, scores in zip(items, response_json["scores"]):
    print_highlight(f"Item '{item}': probabilities = {[f'{s:.4f}' for s in scores]}")
```

```python Example theme={null}
terminate_process(score_process)
```

## v1/decisions (typed decisions on a chat model)

Answer typed questions about an input with per-option probabilities, without generating. The server renders each question with the model's chat template and thinking turned off, assigns one-token answer labels, and scores them through the same path as `v1/score`.

Parameters:

* `input`: Text, object, or array the questions are about
* `questions`: List of questions, each with a unique `id` and a `type` of `choice` (with `options`), `score` (with `levels`), or `yes_no`
* `temperature`: Divides the label logits before the softmax (default: 1)
* `chat_template_kwargs`: Extra chat template arguments
* `prompt_format_version`: Optional pin of the server-owned prompt wording
* `return_prompt_token_ids`: Return the scored prompt and label token ids for replay through `v1/score` (default: False)

The response contains `answers` keyed by question id, each with `type`, `probabilities`, and `label_mass`, plus `choice` for a choice question or `score` for a score question. A yes or no answer is read from `probabilities["yes"]`. See [Decision models](/docs/supported-models/decision_models) for examples, the full reference, and error cases.

## Classify (reward model)

SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations.

```python Example theme={null}
# Note that SGLang now treats embedding models and reward models as the same type of models.
# This will be updated in the future.

reward_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=reward_process)
```

```python Example theme={null}
from transformers import AutoTokenizer

PROMPT = (
    "What is the range of the numeric output of a sigmoid node in a neural network?"
)

RESPONSE1 = "The output of a sigmoid node is bounded between -1 and 1."
RESPONSE2 = "The output of a sigmoid node is bounded between 0 and 1."

CONVS = [
    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE1}],
    [{"role": "user", "content": PROMPT}, {"role": "assistant", "content": RESPONSE2}],
]

tokenizer = AutoTokenizer.from_pretrained("Skywork/Skywork-Reward-Llama-3.1-8B-v0.2")
prompts = tokenizer.apply_chat_template(CONVS, tokenize=False, return_dict=False)

url = f"http://localhost:{port}/classify"
data = {"model": "Skywork/Skywork-Reward-Llama-3.1-8B-v0.2", "text": prompts}

responses = requests.post(url, json=data).json()
for response in responses:
    print_highlight(f"reward: {response['embedding'][0]}")
```

```python Example theme={null}
terminate_process(reward_process)
```

## Capture expert selection distribution in MoE models

SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.

*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*

```python Example theme={null}
expert_record_server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=expert_record_server_process)
```

```python Example theme={null}
response = requests.post(f"http://localhost:{port}/start_expert_distribution_record")
print_highlight(response)

url = f"http://localhost:{port}/generate"
data = {"text": "What is the capital of France?"}

response = requests.post(url, json=data)
print_highlight(response.json())

response = requests.post(f"http://localhost:{port}/stop_expert_distribution_record")
print_highlight(response)

response = requests.post(f"http://localhost:{port}/dump_expert_distribution_record")
print_highlight(response)
```

```python Example theme={null}
terminate_process(expert_record_server_process)
```

## Tokenize/Detokenize Example (Round Trip)

This example demonstrates how to use the /tokenize and /detokenize endpoints together. We first tokenize a string, then detokenize the resulting IDs to reconstruct the original text. This workflow is useful when you need to handle tokenization externally but still leverage the server for detokenization.

```python Example theme={null}
tokenizer_free_server_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct
""")

wait_for_server(f"http://localhost:{port}", process=tokenizer_free_server_process)
```

```python Example theme={null}
import requests
from sglang.utils import print_highlight

base_url = f"http://localhost:{port}"
tokenize_url = f"{base_url}/tokenize"
detokenize_url = f"{base_url}/detokenize"

model_name = "qwen/qwen2.5-0.5b-instruct"
input_text = "SGLang provides efficient tokenization endpoints."
print_highlight(f"Original Input Text:\n'{input_text}'")

# --- tokenize the input text ---
tokenize_payload = {
    "model": model_name,
    "prompt": input_text,
    "add_special_tokens": False,
}
try:
    tokenize_response = requests.post(tokenize_url, json=tokenize_payload)
    tokenize_response.raise_for_status()
    tokenization_result = tokenize_response.json()
    token_ids = tokenization_result.get("tokens")

    if not token_ids:
        raise ValueError("Tokenization returned empty tokens.")

    print_highlight(f"\nTokenized Output (IDs):\n{token_ids}")
    print_highlight(f"Token Count: {tokenization_result.get('count')}")
    print_highlight(f"Max Model Length: {tokenization_result.get('max_model_len')}")

    # --- detokenize the obtained token IDs ---
    detokenize_payload = {
        "model": model_name,
        "tokens": token_ids,
        "skip_special_tokens": True,
    }

    detokenize_response = requests.post(detokenize_url, json=detokenize_payload)
    detokenize_response.raise_for_status()
    detokenization_result = detokenize_response.json()
    reconstructed_text = detokenization_result.get("text")

    print_highlight(f"\nDetokenized Output (Text):\n'{reconstructed_text}'")

    if input_text == reconstructed_text:
        print_highlight(
            "\nRound Trip Successful: Original and reconstructed text match."
        )
    else:
        print_highlight(
            "\nRound Trip Mismatch: Original and reconstructed text differ."
        )

except requests.exceptions.RequestException as e:
    print_highlight(f"\nHTTP Request Error: {e}")
except Exception as e:
    print_highlight(f"\nAn error occurred: {e}")
```

```python Example theme={null}
terminate_process(tokenizer_free_server_process)
```


# Offline Engine API
Source: https://docs.sglang.io/docs/basic_usage/offline_engine_api



SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:

* Offline Batch Inference
* Custom Server on Top of the Engine

This document focuses on the offline batch inference, demonstrating four different inference modes:

* Non-streaming synchronous generation
* Streaming synchronous generation
* Non-streaming asynchronous generation
* Streaming asynchronous generation

Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom\_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).

## Nest Asyncio

Note that if you want to use **Offline Engine** in ipython or some other nested loop code, you need to add the following code:

```python Example theme={null}
import nest_asyncio

nest_asyncio.apply()

```

## Advanced Usage

The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/tree/main/examples/runtime/hidden_states).

Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases.

## Offline Batch Inference

SGLang offline engine supports batch inference with efficient scheduling.

```python Example theme={null}
# launch the offline engine
import asyncio

import sglang as sgl
import sglang.test.doc_patch
from sglang.utils import async_stream_and_merge, stream_and_merge

llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
```

### Non-streaming Synchronous Generation

```python Example theme={null}
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

sampling_params = {"temperature": 0.8, "top_p": 0.95}

outputs = llm.generate(prompts, sampling_params)
for prompt, output in zip(prompts, outputs):
    print("===============================")
    print(f"Prompt: {prompt}\nGenerated text: {output['text']}")
```

### Streaming Synchronous Generation

```python Example theme={null}
prompts = [
    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
    "Provide a concise factual statement about France’s capital city. The capital of France is",
    "Explain possible future trends in artificial intelligence. The future of AI is",
]

sampling_params = {
    "temperature": 0.2,
    "top_p": 0.9,
}

print("\n=== Testing synchronous streaming generation with overlap removal ===\n")

for prompt in prompts:
    print(f"Prompt: {prompt}")
    merged_output = stream_and_merge(llm, prompt, sampling_params)
    print("Generated text:", merged_output)
    print()
```

### Non-streaming Asynchronous Generation

```python Example theme={null}
prompts = [
    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
    "Provide a concise factual statement about France’s capital city. The capital of France is",
    "Explain possible future trends in artificial intelligence. The future of AI is",
]

sampling_params = {"temperature": 0.8, "top_p": 0.95}

print("\n=== Testing asynchronous batch generation ===")

async def main():
    outputs = await llm.async_generate(prompts, sampling_params)

    for prompt, output in zip(prompts, outputs):
        print(f"\nPrompt: {prompt}")
        print(f"Generated text: {output['text']}")

asyncio.run(main())
```

### Streaming Asynchronous Generation

```python Example theme={null}
prompts = [
    "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
    "Provide a concise factual statement about France’s capital city. The capital of France is",
    "Explain possible future trends in artificial intelligence. The future of AI is",
]

sampling_params = {"temperature": 0.8, "top_p": 0.95}

print("\n=== Testing asynchronous streaming generation (no repeats) ===")

async def main():
    for prompt in prompts:
        print(f"\nPrompt: {prompt}")
        print("Generated text: ", end="", flush=True)

        # Replace direct calls to async_generate with our custom overlap-aware version
        async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):
            print(cleaned_chunk, end="", flush=True)

        print()  # New line after each prompt

asyncio.run(main())
```

```python Example theme={null}
llm.shutdown()
```


# Ollama-Compatible API
Source: https://docs.sglang.io/docs/basic_usage/ollama_api



SGLang provides Ollama API compatibility, allowing you to use the Ollama CLI and Python library with SGLang as the inference backend.

## Prerequisites

```bash Command theme={null}
# Install the Ollama Python library (for Python client usage)
pip install ollama
```

> [!NOTE]
>
> You don't need the Ollama server installed - SGLang acts as the backend. You only need the `ollama` CLI or Python library as the client.

## Endpoints

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Endpoint</th>
      <th>Method</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`/`</td>
      <td>GET, HEAD</td>
      <td>Health check for Ollama CLI</td>
    </tr>

    <tr>
      <td>`/api/tags`</td>
      <td>GET</td>
      <td>List available models</td>
    </tr>

    <tr>
      <td>`/api/chat`</td>
      <td>POST</td>
      <td>Chat completions (streaming & non-streaming)</td>
    </tr>

    <tr>
      <td>`/api/generate`</td>
      <td>POST</td>
      <td>Text generation (streaming & non-streaming)</td>
    </tr>

    <tr>
      <td>`/api/show`</td>
      <td>POST</td>
      <td>Model information</td>
    </tr>
  </tbody>
</table>

## Quick Start

### 1. Launch SGLang Server

```bash Command theme={null}
python -m sglang.launch_server \
    --model Qwen/Qwen2.5-1.5B-Instruct \
    --port 30001 \
    --host 0.0.0.0
```

> [!NOTE]
>
> The model name used with `ollama run` must match exactly what you passed to `--model`.

### 2. Use Ollama CLI

```bash Command theme={null}
# List available models
OLLAMA_HOST=http://localhost:30001 ollama list

# Interactive chat
OLLAMA_HOST=http://localhost:30001 ollama run "Qwen/Qwen2.5-1.5B-Instruct"
```

If connecting to a remote server behind a firewall:

```bash Command theme={null}
# SSH tunnel
ssh -L 30001:localhost:30001 user@gpu-server -N &

# Then use Ollama CLI as above
OLLAMA_HOST=http://localhost:30001 ollama list
```

### 3. Use Ollama Python Library

```python Example theme={null}
import ollama

client = ollama.Client(host='http://localhost:30001')

# Non-streaming
response = client.chat(
    model='Qwen/Qwen2.5-1.5B-Instruct',
    messages=[{'role': 'user', 'content': 'Hello!'}]
)
print(response['message']['content'])

# Streaming
stream = client.chat(
    model='Qwen/Qwen2.5-1.5B-Instruct',
    messages=[{'role': 'user', 'content': 'Tell me a story'}],
    stream=True
)
for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)
```

## Smart Router

For intelligent routing between local Ollama (fast) and remote SGLang (powerful) using an LLM judge, see the [Smart Router documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/ollama/README.md).

## Summary

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Component</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Ollama API**</td>
      <td>Familiar CLI/API that developers already know</td>
    </tr>

    <tr>
      <td>**SGLang Backend**</td>
      <td>High-performance inference engine</td>
    </tr>

    <tr>
      <td>**Smart Router**</td>
      <td>Intelligent routing - fast local for simple tasks, powerful remote for complex tasks</td>
    </tr>
  </tbody>
</table>


# OpenAI-Compatible APIs
Source: https://docs.sglang.io/docs/basic_usage/openai_api

Documentation for OpenAI-Compatible APIs

* [Openai Api Completions](./openai_api_completions)
* [Openai Api Vision](./openai_api_vision)
* [Openai Api Embeddings](./openai_api_embeddings)

For the Anthropic-compatible `/v1/messages` endpoint (including Claude Code integration), see [Anthropic-Compatible API](./anthropic_api).


# OpenAI APIs - Completions
Source: https://docs.sglang.io/docs/basic_usage/openai_api_completions



SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).

This tutorial covers the following popular APIs:

* `chat/completions`
* `completions`

Check out other tutorials to learn about [vision APIs](./openai_api_vision) for vision-language models and [embedding APIs](./openai_api_embeddings) for embedding models.

## Launch A Server

Launch the server in your terminal and wait for it to initialize.

```python Example theme={null}
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

server_process, port = launch_server_cmd(
    "python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}")
print(f"Server started on http://localhost:{port}")
```

## Chat Completions

### Usage

The server fully implements the OpenAI API.
It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.
You can also specify a custom chat template with `--chat-template` when launching the server.

```python Example theme={null}
import openai

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)

print_highlight(f"Response: {response}")
```

### Model Thinking/Reasoning Support

Some models support internal reasoning or thinking processes that can be exposed in the API response. SGLang provides unified support for various reasoning models through the `chat_template_kwargs` parameter and compatible reasoning parsers.

#### Supported Models and Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>Chat Template Parameter</th>
      <th>Reasoning Parser</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>DeepSeek-R1 (R1, R1-0528, R1-Distill)</td>
      <td>`enable_thinking`</td>
      <td>`--reasoning-parser deepseek-r1`</td>
      <td>Standard reasoning models</td>
    </tr>

    <tr>
      <td>DeepSeek-V3.1</td>
      <td>`thinking`</td>
      <td>`--reasoning-parser deepseek-v3`</td>
      <td>Hybrid model (thinking/non-thinking modes)</td>
    </tr>

    <tr>
      <td>Qwen3 (standard)</td>
      <td>`enable_thinking`</td>
      <td>`--reasoning-parser qwen3`</td>
      <td>Hybrid model (thinking/non-thinking modes)</td>
    </tr>

    <tr>
      <td>Qwen3-Thinking</td>
      <td>N/A (always enabled)</td>
      <td>`--reasoning-parser qwen3-thinking`</td>
      <td>Always generates reasoning</td>
    </tr>

    <tr>
      <td>Kimi</td>
      <td>N/A (always enabled)</td>
      <td>`--reasoning-parser kimi`</td>
      <td>Kimi thinking models</td>
    </tr>

    <tr>
      <td>Gpt-Oss</td>
      <td>N/A (always enabled)</td>
      <td>`--reasoning-parser gpt-oss`</td>
      <td>Gpt-Oss thinking models</td>
    </tr>
  </tbody>
</table>

#### Basic Usage

To use reasoning output:

1. Launch the server with the appropriate reasoning parser.
2. Use the model-specific parameter in `chat_template_kwargs` when you need to override the model's chat-template default.
3. Optionally use `separate_reasoning: False` to keep reasoning in the response content instead of returning it separately (defaults to `True`).

> [!NOTE]
>
> **Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content.

#### Example: Qwen3 Models

```python Example theme={null}
# Launch server:
# python3 -m sglang.launch_server --model Qwen/Qwen3-4B --reasoning-parser qwen3

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://127.0.0.1:30000/v1",
)

model = "Qwen/Qwen3-4B"
messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]

response = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {"enable_thinking": True},
        "separate_reasoning": True
    }
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("-"*100)
print("Answer:", response.choices[0].message.content)
```

**ExampleOutput:**

```text Output theme={null}
Reasoning: Okay, so the user is asking how many 'r's are in the word 'strawberry'. Let me think. First, I need to make sure I have the word spelled correctly. Strawberry... S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me break it down.

Starting with 'strawberry', let's write out the letters one by one. S, T, R, A, W, B, E, R, R, Y. Hmm, wait, that's 10 letters. Let me check again. S (1), T (2), R (3), A (4), W (5), B (6), E (7), R (8), R (9), Y (10). So the letters are S-T-R-A-W-B-E-R-R-Y.
...
Therefore, the answer should be three R's in 'strawberry'. But I need to make sure I'm not counting any other letters as R. Let me check again. S, T, R, A, W, B, E, R, R, Y. No other R's. So three in total. Yeah, that seems right.

----------------------------------------------------------------------------------------------------
Answer: The word "strawberry" contains **three** letters 'r'. Here's the breakdown:

1. **S-T-R-A-W-B-E-R-R-Y**
   - The **third letter** is 'R'.
   - The **eighth and ninth letters** are also 'R's.

Thus, the total count is **3**.

**Answer:** 3.
```

> [!NOTE]
>
> Setting `"enable_thinking": False` disables reasoning for Qwen3. Omitting the parameter follows the model's chat-template default; Qwen3 enables thinking by default. Qwen3-Thinking models always generate reasoning content and do not support the `enable_thinking` parameter.

#### Set a Default for OpenAI Chat Completions

If most OpenAI chat-completion requests should use the same chat-template mode, set a default when launching the server:

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-4B \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"enable_thinking": false}'
```

`--default-chat-template-kwargs` accepts a JSON object and merges it into each OpenAI chat-completion request's `chat_template_kwargs`. For ordinary template keys such as `enable_thinking`, the precedence is:

1. An explicit value in the request's `chat_template_kwargs`
2. The value in `--default-chat-template-kwargs`
3. The model's chat-template default

`reasoning_effort` follows a separate normalization path, so the precedence list above does not apply to it. Configure `reasoning_effort` in either the server defaults or individual requests, not both.

The accepted keys are model-specific. For example, Qwen3 uses `enable_thinking`, while other model families may use `thinking` or may not expose a reasoning toggle. The reasoning parser controls how reasoning tokens are separated from the response; it does not by itself enable reasoning in the chat template.

#### Logit Bias Support

SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where:

* **Positive values** (0 to 100) increase the likelihood of the token being selected
* **Negative values** (-100 to 0) decrease the likelihood of the token being selected
* **-100** effectively prevents the token from being generated

The `logit_bias` parameter accepts a dictionary where keys are token IDs (as strings) and values are the bias amounts (as floats).

#### Getting Token IDs

To use `logit_bias` effectively, you need to know the token IDs for the words you want to bias. Here's how to get token IDs:

```python Example theme={null}
# Get tokenizer to find token IDs
import tiktoken

# For OpenAI models, use the appropriate encoding
tokenizer = tiktoken.encoding_for_model("gpt-3.5-turbo")  # or your model

# Get token IDs for specific words
word = "sunny"
token_ids = tokenizer.encode(word)
print(f"Token IDs for '{word}': {token_ids}")

# For SGLang models, you can access the tokenizer through the client
# and get token IDs for bias
```

> [!TIP]
>
> **Important:** The `logit_bias` parameter uses token IDs as string keys, not the actual words.

#### Example: DeepSeek-V3 Models

DeepSeek-V3 models support thinking mode through the `thinking` parameter:

```python Example theme={null}
# Launch server:
# python3 -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.1 --tp 8  --reasoning-parser deepseek-v3

from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url=f"http://127.0.0.1:30000/v1",
)

model = "deepseek-ai/DeepSeek-V3.1"
messages = [{"role": "user", "content": "How many r's are in 'strawberry'?"}]

response = client.chat.completions.create(
    model=model,
    messages=messages,
    extra_body={
        "chat_template_kwargs": {"thinking": True},
        "separate_reasoning": True
    }
)

print("Reasoning:", response.choices[0].message.reasoning_content)
print("-"*100)
print("Answer:", response.choices[0].message.content)
```

**Example Output:**

```text Output theme={null}
Reasoning: First, the question is: "How many r's are in 'strawberry'?"

I need to count the number of times the letter 'r' appears in the word "strawberry".

Let me write out the word: S-T-R-A-W-B-E-R-R-Y.

Now, I'll go through each letter and count the 'r's.
...
So, I have three 'r's in "strawberry".

I should double-check. The word is spelled S-T-R-A-W-B-E-R-R-Y. The letters are at positions: 3, 8, and 9 are 'r's. Yes, that's correct.

Therefore, the answer should be 3.
----------------------------------------------------------------------------------------------------
Answer: The word "strawberry" contains **3** instances of the letter "r". Here's a breakdown for clarity:

- The word is spelled: S-T-R-A-W-B-E-R-R-Y
- The "r" appears at the 3rd, 8th, and 9th positions.
```

> [!NOTE]
>
> DeepSeek-V3 models use the `thinking` parameter (not `enable_thinking`) to control reasoning output.

```python Example theme={null}
# Example with logit_bias parameter
# Note: You need to get the actual token IDs from your tokenizer
# For demonstration, we'll use some example token IDs
response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[
        {"role": "user", "content": "Complete this sentence: The weather today is"}
    ],
    temperature=0.7,
    max_tokens=20,
    logit_bias={
        "12345": 50,  # Increase likelihood of token ID 12345
        "67890": -50,  # Decrease likelihood of token ID 67890
        "11111": 25,  # Slightly increase likelihood of token ID 11111
    },
)

print_highlight(f"Response with logit bias: {response.choices[0].message.content}")
```

### Parameters

The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.

SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor.

```python Example theme={null}
response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[
        {
            "role": "system",
            "content": "You are a knowledgeable historian who provides concise responses.",
        },
        {"role": "user", "content": "Tell me about ancient Rome"},
        {
            "role": "assistant",
            "content": "Ancient Rome was a civilization centered in Italy.",
        },
        {"role": "user", "content": "What were their major achievements?"},
    ],
    temperature=0.3,  # Lower temperature for more focused responses
    max_tokens=128,  # Reasonable length for a concise response
    top_p=0.95,  # Slightly higher for better fluency
    presence_penalty=0.2,  # Mild penalty to avoid repetition
    frequency_penalty=0.2,  # Mild penalty for more natural language
    n=1,  # Single response is usually more stable
    seed=42,  # Keep for reproducibility
)

print_highlight(response.choices[0].message.content)
```

Streaming mode is also supported.

#### Logit Bias Support

The completions API also supports the `logit_bias` parameter with the same functionality as described in the chat completions section above.

```python Example theme={null}
stream = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[{"role": "user", "content": "Say this is a test"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")
```

#### Returning Routed Experts (MoE Models)

For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. By default, `routed_experts` is returned in the response-level `sglext` object as base64-encoded int32 expert IDs in a flattened array with logical shape `[num_tokens, num_layers, top_k]`. For non-streaming chat responses with `return_meta_info: true`, `routed_experts` is returned only in each choice's `meta_info` object and omitted from response-level `sglext` to avoid duplicating the routing data. Other requested extension fields, such as `cached_tokens_details` and `spec_tokens_details`, remain in response-level `sglext`. By default, routed experts cover `[0, seqlen - 1)`, the full available sequence, because RL workflows need them for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that causes bottlenecks.

#### Returning Flat Output Top Logprobs

With `logprobs: true` and `top_logprobs: k`, set `return_flat_raw_output_top_logprobs: true` and `return_meta_info: true` in `extra_body` to receive the output top logprobs as flat arrays in each choice's `meta_info` instead of nested objects. This skips detokenizing every candidate and shrinks the response, which matters for large `k`. The arrays arrive with the finished response only, so `stream: true` is not supported. `choices[].logprobs` still carries the selected-token logprobs, but its `top_logprobs` lists are empty.

`meta_info` then carries `output_top_logprobs_shape` (`[rows, k]`) and `output_top_logprobs_null_prefix`; covered position `i`, rank `j` sits at flat index `(i - null_prefix) * k + j`. On chat completions the arrays are base64 by default: `output_top_logprobs_{val,idx}_flat_b64`, little-endian bytes whose dtypes are given by the `_dtype` fields (float32 and int32), so non-finite logprobs such as `-inf` survive. Set `return_flat_raw_top_logprobs_b64: false` for plain JSON lists in `output_top_logprobs_{val,idx}_flat`. The native `/generate` API takes the same fields, with base64 off by default.

```python theme={null}
import base64

import numpy as np

response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[{"role": "user", "content": "Say this is a test"}],
    max_tokens=16,
    logprobs=True,
    top_logprobs=5,
    extra_body={"return_meta_info": True, "return_flat_raw_output_top_logprobs": True},
)
meta = response.choices[0].meta_info
shape = meta["output_top_logprobs_shape"]
val = np.frombuffer(
    base64.b64decode(meta["output_top_logprobs_val_flat_b64"]),
    dtype=meta["output_top_logprobs_val_flat_b64_dtype"],
).reshape(shape)
idx = np.frombuffer(
    base64.b64decode(meta["output_top_logprobs_idx_flat_b64"]),
    dtype=meta["output_top_logprobs_idx_flat_b64_dtype"],
).reshape(shape)
```

```python Example theme={null}
# Example with logit_bias parameter for completions API
# Note: You need to get the actual token IDs from your tokenizer
# For demonstration, we'll use some example token IDs
response = client.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    prompt="The best programming language for AI is",
    temperature=0.7,
    max_tokens=20,
    logit_bias={
        "12345": 75,  # Strongly favor token ID 12345
        "67890": -100,  # Completely avoid token ID 67890
        "11111": -25,  # Slightly discourage token ID 11111
    },
)

print_highlight(f"Response with logit bias: {response.choices[0].text}")
```

## Completions

### Usage

Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates.

```python Example theme={null}
response = client.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    prompt="List 3 countries and their capitals.",
    temperature=0,
    max_tokens=64,
    n=1,
    stop=None,
)

print_highlight(f"Response: {response}")
```

### Parameters

The completions API accepts OpenAI Completions API's parameters.  Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.

Here is an example of a detailed completions request:

```python Example theme={null}
response = client.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    prompt="Write a short story about a space explorer.",
    temperature=0.7,  # Moderate temperature for creative writing
    max_tokens=150,  # Longer response for a story
    top_p=0.9,  # Balanced diversity in word choice
    stop=["\n\n", "THE END"],  # Multiple stop sequences
    presence_penalty=0.3,  # Encourage novel elements
    frequency_penalty=0.3,  # Reduce repetitive phrases
    n=1,  # Generate one completion
    seed=123,  # For reproducible results
)

print_highlight(f"Response: {response}")
```

#### Returning Routed Experts (MoE Models)

For MoE models, set `return_routed_experts: true` in `extra_body` to return expert routing data. Requires `--enable-return-routed-experts` server flag. The `routed_experts` field will be returned in the `sgl_ext` object on each choice, containing base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`. By default this returns `[0, seqlen - 1)`, the full available sequence, because RL workflows need routed experts for the full sequence. Set `routed_experts_start_len` in `extra_body` to an absolute prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks.

## Structured Outputs (JSON, Regex, EBNF)

For OpenAI compatible structured outputs API, refer to [Structured Outputs](../advanced_features/structured_outputs) for more details.

## Using LoRA Adapters

SGLang supports LoRA (Low-Rank Adaptation) adapters with OpenAI-compatible APIs. You can specify which adapter to use directly in the `model` parameter using the `base-model:adapter-name` syntax.

**Server Setup:**

```bash Command theme={null}
python -m sglang.launch_server \
    --model-path qwen/qwen2.5-0.5b-instruct \
    --enable-lora \
    --lora-paths adapter_a=/path/to/adapter_a adapter_b=/path/to/adapter_b
```

For more details on LoRA serving configuration, see the [LoRA documentation](../advanced_features/lora).

**API Call:**

(Recommended) Use the `model:adapter` syntax to specify which adapter to use:

```python Example theme={null}
response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct:adapter_a",  # ← base-model:adapter-name
    messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
    max_tokens=50,
)
```

**Backward Compatible: Using `extra_body`**

The old `extra_body` method is still supported for backward compatibility:

```python Example theme={null}
# Backward compatible method
response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
    extra_body={"lora_path": "adapter_a"},  # ← old method
    max_tokens=50,
)
```

**Note:** When both `model:adapter` and `extra_body["lora_path"]` are specified, the `model:adapter` syntax takes precedence.

```python Example theme={null}
terminate_process(server_process)
```


# OpenAI APIs - Embedding
Source: https://docs.sglang.io/docs/basic_usage/openai_api_embeddings



SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/embeddings).

This tutorial covers the embedding APIs for embedding models. For a list of the supported models see the [corresponding overview page](../supported-models)

## Launch A Server

Launch the server in your terminal and wait for it to initialize. Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models still require `--is-embedding`.

```python Example theme={null}
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

embedding_process, port = launch_server_cmd(
    """
sglang serve --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \
    --is-embedding --log-level warning
"""
)

wait_for_server(f"http://localhost:{port}")
```

## Using cURL

```python Example theme={null}
import subprocess, json

text = "Once upon a time"

curl_text = f"""curl -s http://localhost:{port}/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": "{text}"}}'"""

result = subprocess.check_output(curl_text, shell=True)

print(result)

text_embedding = json.loads(result)["data"][0]["embedding"]

print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
```

## Using Python Requests

```python Example theme={null}
import requests

text = "Once upon a time"

response = requests.post(
    f"http://localhost:{port}/v1/embeddings",
    json={"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": text},
)

text_embedding = response.json()["data"][0]["embedding"]

print_highlight(f"Text embedding (first 10): {text_embedding[:10]}")
```

## Using OpenAI Python Client

```python Example theme={null}
import openai

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

# Text embedding example
response = client.embeddings.create(
    model="Alibaba-NLP/gte-Qwen2-1.5B-instruct",
    input=text,
)

embedding = response.data[0].embedding[:10]
print_highlight(f"Text embedding (first 10): {embedding}")
```

## Using Input IDs

SGLang also supports `input_ids` as input to get the embedding.

```python Example theme={null}
import json
import os
from transformers import AutoTokenizer

os.environ["TOKENIZERS_PARALLELISM"] = "false"

tokenizer = AutoTokenizer.from_pretrained("Alibaba-NLP/gte-Qwen2-1.5B-instruct")
input_ids = tokenizer.encode(text)

curl_ids = f"""curl -s http://localhost:{port}/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{{"model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct", "input": {json.dumps(input_ids)}}}'"""

input_ids_embedding = json.loads(subprocess.check_output(curl_ids, shell=True))["data"][
    0
]["embedding"]

print_highlight(f"Input IDs embedding (first 10): {input_ids_embedding[:10]}")
```

## Compact Base64 Responses

Set `encoding_format` to `base64` when JSON arrays would dominate response size. The encoded value contains little-endian FP32 values and can be decoded by OpenAI-compatible clients.

```python Example theme={null}
response = requests.post(
    f"http://localhost:{port}/v1/embeddings",
    json={
        "model": "Alibaba-NLP/gte-Qwen2-1.5B-instruct",
        "input": text,
        "encoding_format": "base64",
    },
)

base64_embedding = response.json()["data"][0]["embedding"]
print_highlight(f"Base64 embedding: {base64_embedding[:20]}...")
```

```python Example theme={null}
terminate_process(embedding_process)
```

## Multi-Modal Embedding Model

Please refer to [Multi-Modal Embedding Model](../supported-models)


# OpenAI APIs - Vision
Source: https://docs.sglang.io/docs/basic_usage/openai_api_vision



SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.
A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/guides/vision).
This tutorial covers the vision APIs for vision language models.

SGLang supports various vision language models such as Llama 3.2, LLaVA-OneVision, Qwen2.5-VL, Gemma3 and [more](../supported-models/multimodal_language_models).

As an alternative to the OpenAI API, you can also use the [SGLang offline engine](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py).

## Launch A Server

Launch the server in your terminal and wait for it to initialize.

```python Example theme={null}
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, print_highlight, terminate_process

example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
logo_image_url = (
    "https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png"
)

vision_process, port = launch_server_cmd("""
python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --log-level warning
""")

wait_for_server(f"http://localhost:{port}", process=vision_process)
```

## Using cURL

Once the server is up, you can send test requests using curl or requests.

```python Example theme={null}
import subprocess

curl_command = f"""
curl -s http://localhost:{port}/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
      {{
        "role": "user",
        "content": [
          {{
            "type": "text",
            "text": "What’s in this image?"
          }},
          {{
            "type": "image_url",
            "image_url": {{
              "url": "{example_image_url}"
            }}
          }}
        ]
      }}
    ],
    "max_tokens": 300
  }}'
"""

response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)


response = subprocess.check_output(curl_command, shell=True).decode()
print_highlight(response)
```

## Using Python Requests

```python Example theme={null}
import requests

url = f"http://localhost:{port}/v1/chat/completions"

data = {
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": example_image_url},
                },
            ],
        }
    ],
    "max_tokens": 300,
}

response = requests.post(url, json=data)
print_highlight(response.text)
```

## Using OpenAI Python Client

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image?",
                },
                {
                    "type": "image_url",
                    "image_url": {"url": example_image_url},
                },
            ],
        }
    ],
    max_tokens=300,
)

print_highlight(response.choices[0].message.content)
```

## Multiple-Image Inputs

The server also supports multiple images and interleaved text and images if the model supports it.

```python Example theme={null}
from openai import OpenAI

client = OpenAI(base_url=f"http://localhost:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": example_image_url,
                    },
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": logo_image_url,
                    },
                },
                {
                    "type": "text",
                    "text": "I have two very different images. They are not related at all. "
                    "Please describe the first image in one sentence, and then describe the second image in another sentence.",
                },
            ],
        }
    ],
    temperature=0,
)

print_highlight(response.choices[0].message.content)
```

```python Example theme={null}
terminate_process(vision_process)
```


# Basic Usage
Source: https://docs.sglang.io/docs/basic_usage/overview

Core APIs and common usage patterns for SGLang.

* [OpenAI-Compatible APIs](./openai_api_completions) — Chat completions, vision, and embeddings
* [Anthropic-Compatible API](./anthropic_api) — `/v1/messages`, including Claude Code integration
* [Ollama API](./ollama_api)
* [Offline Engine API](./offline_engine_api)
* [Native API](./native_api)
* [Sampling Parameters](./sampling_params)
* [Popular Model Usage](/cookbook/autoregressive/intro) — DeepSeek, GLM, Qwen, Llama, and more


# Sampling Parameters
Source: https://docs.sglang.io/docs/basic_usage/sampling_params



This doc describes the sampling parameters of the SGLang Runtime. It is the low-level endpoint of the runtime.
If you want a high-level endpoint that can automatically handle chat templates, consider using the [OpenAI Compatible API](./openai_api_completions).

## `/generate` Endpoint

The `/generate` endpoint accepts the following parameters in JSON format. For detailed usage, see the [native API doc](./native_api). The object is defined at `io_struct.py::GenerateReqInput`. You can also read the source code to find more arguments and docs.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Type/Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>text</td>
      <td>`Optional[Union[List[str], str]] = None`</td>
      <td>The input prompt. Can be a single prompt or a batch of prompts.</td>
    </tr>

    <tr>
      <td>input\_ids</td>
      <td>`Optional[Union[List[List[int]], List[int]]] = None`</td>
      <td>The token IDs for text; one can specify either text or input\_ids.</td>
    </tr>

    <tr>
      <td>input\_embeds</td>
      <td>`Optional[Union[List[List[List[float]]], List[List[float]]]] = None`</td>
      <td>The embeddings for input\_ids; one can specify either text, input\_ids, or input\_embeds.</td>
    </tr>

    <tr>
      <td>image\_data</td>
      <td>`Optional[Union[List[List[ImageDataItem]], List[ImageDataItem], ImageDataItem]] = None`</td>
      <td>The image input. Supports three formats: (1) **Raw images**: PIL Image, file path, URL, or base64 string; (2) **Processor output**: Dict with `format: "processor_output"` containing HuggingFace processor outputs; (3) **Precomputed embeddings**: Dict with `format: "precomputed_embedding"` and `feature` containing pre-calculated visual embeddings. Can be a single image, list of images, or list of lists of images. See [Multimodal Input Formats](#multimodal) for details.</td>
    </tr>

    <tr>
      <td>audio\_data</td>
      <td>`Optional[Union[List[AudioDataItem], AudioDataItem]] = None`</td>
      <td>The audio input. Can be a file name, URL, or base64 encoded string.</td>
    </tr>

    <tr>
      <td>sampling\_params</td>
      <td>`Optional[Union[List[Dict], Dict]] = None`</td>
      <td>The sampling parameters as described in the sections below.</td>
    </tr>

    <tr>
      <td>rid</td>
      <td>`Optional[Union[List[str], str]] = None`</td>
      <td>The request ID.</td>
    </tr>

    <tr>
      <td>return\_logprob</td>
      <td>`Optional[Union[List[bool], bool]] = None`</td>
      <td>Whether to return log probabilities for tokens.</td>
    </tr>

    <tr>
      <td>logprob\_start\_len</td>
      <td>`Optional[Union[List[int], int]] = None`</td>
      <td>If return\_logprob, the start location in the prompt for returning logprobs. Default is "-1", which returns logprobs for output tokens only.</td>
    </tr>

    <tr>
      <td>top\_logprobs\_num</td>
      <td>`Optional[Union[List[int], int]] = None`</td>
      <td>If return\_logprob, the number of top logprobs to return at each position.</td>
    </tr>

    <tr>
      <td>token\_ids\_logprob</td>
      <td>`Optional[Union[List[List[int]], List[int]]] = None`</td>
      <td>If return\_logprob, the token IDs to return logprob for.</td>
    </tr>

    <tr>
      <td>return\_text\_in\_logprobs</td>
      <td>`bool = False`</td>
      <td>Whether to detokenize tokens in text in the returned logprobs.</td>
    </tr>

    <tr>
      <td>stream</td>
      <td>`bool = False`</td>
      <td>Whether to stream output.</td>
    </tr>

    <tr>
      <td>lora\_path</td>
      <td>`Optional[Union[List[Optional[str]], Optional[str]]] = None`</td>
      <td>The path to the LoRA.</td>
    </tr>

    <tr>
      <td>custom\_logit\_processor</td>
      <td>`Optional[Union[List[Optional[str]], str]] = None`</td>
      <td>Custom logit processor for advanced sampling control. Must be a serialized instance of `CustomLogitProcessor` using its `to_str()` method. For usage see below.</td>
    </tr>

    <tr>
      <td>return\_hidden\_states</td>
      <td>`Union[List[bool], bool] = False`</td>
      <td>Whether to return hidden states.</td>
    </tr>

    <tr>
      <td>return\_routed\_experts</td>
      <td>`bool = False`</td>
      <td>Whether to return routed experts for MoE models. Requires `--enable-return-routed-experts` server flag. With the default `routed_experts_start_len=0`, returns the full available sequence `[0, seqlen - 1)` because RL workflows need routed experts for the full sequence. The result is base64-encoded int32 expert IDs as a flattened array with logical shape `[num_tokens, num_layers, top_k]`.</td>
    </tr>

    <tr>
      <td>routed\_experts\_start\_len</td>
      <td>`int = 0`</td>
      <td>If `return_routed_experts`, the absolute start position for returned routed-experts rows. `0` preserves the default full sequence; set it to an accumulated prefix length to return only `[routed_experts_start_len, seqlen - 1)`. For example, in multi-turn RL rollouts, routed experts for tokens from previous turns have already been collected, so setting this value avoids unnecessary transfer that cause bottlenecks. Must be in `[0, prompt_tokens]`.</td>
    </tr>
  </tbody>
</table>

## Sampling parameters

The object is defined at `sampling_params.py::SamplingParams`. You can also read the source code to find more arguments and docs.

### Note on defaults

By default, SGLang initializes several sampling parameters from the model's `generation_config.json` (when the server is launched with `--sampling-defaults model`, which is the default). To use SGLang/OpenAI constant defaults instead, start the server with `--sampling-defaults openai`. You can always override any parameter per request via `sampling_params`.

```bash Command theme={null}
# Use model-provided defaults from generation_config.json (default behavior)
python -m sglang.launch_server --model-path <MODEL> --sampling-defaults model

# Use SGLang/OpenAI constant defaults instead
python -m sglang.launch_server --model-path <MODEL> --sampling-defaults openai
```

### Core parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Type/Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>max\_new\_tokens</td>
      <td>`int = 128`</td>
      <td>The maximum output length measured in tokens.</td>
    </tr>

    <tr>
      <td>stop</td>
      <td>`Optional[Union[str, List[str]]] = None`</td>
      <td>One or multiple [stop words](https://platform.openai.com/docs/api-reference/chat/create#chat-create-stop). Generation will stop if one of these words is sampled.</td>
    </tr>

    <tr>
      <td>stop\_token\_ids</td>
      <td>`Optional[List[int]] = None`</td>
      <td>Provide stop words in the form of token IDs. Generation will stop if one of these token IDs is sampled.</td>
    </tr>

    <tr>
      <td>stop\_regex</td>
      <td>`Optional[Union[str, List[str]]] = None`</td>
      <td>Stop when hitting any of the regex patterns in this list</td>
    </tr>

    <tr>
      <td>temperature</td>
      <td>`float (model default; fallback 1.0)`</td>
      <td>[Temperature](https://platform.openai.com/docs/api-reference/chat/create#chat-create-temperature) when sampling the next token. `temperature = 0` corresponds to greedy sampling, a higher temperature leads to more diversity.</td>
    </tr>

    <tr>
      <td>top\_p</td>
      <td>`float (model default; fallback 1.0)`</td>
      <td>[Top-p](https://platform.openai.com/docs/api-reference/chat/create#chat-create-top_p) selects tokens from the smallest sorted set whose cumulative probability exceeds `top_p`. When `top_p = 1`, this reduces to unrestricted sampling from all tokens.</td>
    </tr>

    <tr>
      <td>top\_k</td>
      <td>`int (model default; fallback -1)`</td>
      <td>[Top-k](https://developer.nvidia.com/blog/how-to-get-better-outputs-from-your-large-language-model/#predictability_vs_creativity) randomly selects from the `k` highest-probability tokens.</td>
    </tr>

    <tr>
      <td>min\_p</td>
      <td>`float (model default; fallback 0.0)`</td>
      <td>[Min-p](https://github.com/huggingface/transformers/issues/27670) samples from tokens with probability larger than `min_p * highest_token_probability`.</td>
    </tr>
  </tbody>
</table>

### Penalizers

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Type/Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>frequency\_penalty</td>
      <td>`float = 0.0`</td>
      <td>Penalizes tokens based on their frequency in generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of penalization grows linearly with each appearance of a token.</td>
    </tr>

    <tr>
      <td>presence\_penalty</td>
      <td>`float = 0.0`</td>
      <td>Penalizes tokens if they appeared in the generation so far. Must be between `-2` and `2` where negative numbers encourage repeatment of tokens and positive number encourages sampling of new tokens. The scaling of the penalization is constant if a token occurred.</td>
    </tr>

    <tr>
      <td>repetition\_penalty</td>
      <td>`float = 1.0`</td>
      <td>Scales the logits of previously generated tokens to discourage (values > 1) or encourage (values \< 1) repetition. Valid range is `(0, 2]`; `1.0` leaves probabilities unchanged.</td>
    </tr>

    <tr>
      <td>min\_new\_tokens</td>
      <td>`int = 0`</td>
      <td>Forces the model to generate at least `min_new_tokens` until a stop word or EOS token is sampled. Note that this might lead to unintended behavior, for example, if the distribution is highly skewed towards these tokens.</td>
    </tr>
  </tbody>
</table>

### Constrained decoding

Please refer to our dedicated guide on [constrained decoding](../advanced_features/structured_outputs) for the following parameters.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Type/Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>json\_schema</td>
      <td>`Optional[str] = None`</td>
      <td>JSON schema for structured outputs.</td>
    </tr>

    <tr>
      <td>regex</td>
      <td>`Optional[str] = None`</td>
      <td>Regex for structured outputs.</td>
    </tr>

    <tr>
      <td>ebnf</td>
      <td>`Optional[str] = None`</td>
      <td>EBNF for structured outputs.</td>
    </tr>

    <tr>
      <td>structural\_tag</td>
      <td>`Optional[str] = None`</td>
      <td>The structal tag for structured outputs.</td>
    </tr>
  </tbody>
</table>

### Other options

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Type/Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>n</td>
      <td>`int = 1`</td>
      <td>Specifies the number of output sequences to generate per request. (Generating multiple outputs in one request (n > 1) is discouraged; repeating the same prompts several times offers better control and efficiency.)</td>
    </tr>

    <tr>
      <td>ignore\_eos</td>
      <td>`bool = False`</td>
      <td>Don't stop generation when EOS token is sampled.</td>
    </tr>

    <tr>
      <td>skip\_special\_tokens</td>
      <td>`bool = True`</td>
      <td>Remove special tokens during decoding.</td>
    </tr>

    <tr>
      <td>spaces\_between\_special\_tokens</td>
      <td>`bool = True`</td>
      <td>Whether or not to add spaces between special tokens during detokenization.</td>
    </tr>

    <tr>
      <td>no\_stop\_trim</td>
      <td>`bool = False`</td>
      <td>Don't trim stop words or EOS token from the generated text.</td>
    </tr>

    <tr>
      <td>custom\_params</td>
      <td>`Optional[List[Optional[Dict[str, Any]]]] = None`</td>
      <td>Used when employing `CustomLogitProcessor`. For usage, see below.</td>
    </tr>
  </tbody>
</table>

## Examples

### Normal

Launch a server:

```bash Command theme={null}
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000
```

Send a request:

```python Example theme={null}
import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
    },
)
print(response.json())
```

Detailed example in [send request](./send_request).

### Streaming

Send a request and stream the output:

```python Example theme={null}
import requests, json

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
        "stream": True,
    },
    stream=True,
)

prev = 0
for chunk in response.iter_lines(decode_unicode=False):
    chunk = chunk.decode("utf-8")
    if chunk and chunk.startswith("data:"):
        if chunk == "data: [DONE]":
            break
        data = json.loads(chunk[5:].strip("\n"))
        output = data["text"].strip()
        print(output[prev:], end="", flush=True)
        prev = len(output)
print("")
```

Detailed example in [openai compatible api](./openai_api_completions).

### Multimodal

Launch a server:

```bash Command theme={null}
python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-7b-ov
```

Download an image:

```bash Command theme={null}
curl -o example_image.png -L https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true
```

Send a request:

```python Example theme={null}
import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
                "<|im_start|>user\n<image>\nDescribe this image in a very short sentence.<|im_end|>\n"
                "<|im_start|>assistant\n",
        "image_data": "example_image.png",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
    },
)
print(response.json())
```

The `image_data` can be a file name, a URL, or a base64 encoded string. See also `python/sglang/srt/utils.py:load_image`.

Streaming is supported in a similar manner as [above](#streaming).

Detailed example in [OpenAI API Vision](./openai_api_vision).

### Structured Outputs (JSON, Regex, EBNF)

You can specify a JSON schema, regular expression or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.

SGLang supports two grammar backends:

* [XGrammar](https://github.com/mlc-ai/xgrammar) (default): Supports JSON schema, regular expression, and EBNF constraints.
  * XGrammar currently uses the [GGML BNF format](https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md).
* [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.

If instead you want to initialize the Outlines backend, you can use `--grammar-backend outlines` flag:

```bash Command theme={null}
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--port 30000 --host 0.0.0.0 --grammar-backend [xgrammar|outlines] # xgrammar or outlines (default: xgrammar)
```

```python Example theme={null}
import json
import requests

json_schema = json.dumps({
    "type": "object",
    "properties": {
        "name": {"type": "string", "pattern": "^[\\w]+$"},
        "population": {"type": "integer"},
    },
    "required": ["name", "population"],
})

# JSON (works with both Outlines and XGrammar)
response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "Here is the information of the capital of France in the JSON format.\n",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 64,
            "json_schema": json_schema,
        },
    },
)
print(response.json())

# Regular expression (Outlines backend only)
response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "Paris is the capital of",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 64,
            "regex": "(France|England)",
        },
    },
)
print(response.json())

# EBNF (XGrammar backend only)
response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "Write a greeting.",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 64,
            "ebnf": 'root ::= "Hello" | "Hi" | "Hey"',
        },
    },
)
print(response.json())
```

Detailed example in [structured outputs](../advanced_features/structured_outputs).

### Custom logit processor

Launch a server with `--enable-custom-logit-processor` flag on.

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3-8B-Instruct \
  --port 30000 \
  --enable-custom-logit-processor
```

Define a custom logit processor that will always sample a specific token id.

```python Example theme={null}
from sglang.srt.sampling.custom_logit_processor import CustomLogitProcessor

class DeterministicLogitProcessor(CustomLogitProcessor):
    """A dummy logit processor that changes the logits to always
    sample the given token id.
    """

    def __call__(self, logits, custom_param_list):
        # Check that the number of logits matches the number of custom parameters
        assert logits.shape[0] == len(custom_param_list)
        key = "token_id"

        for i, param_dict in enumerate(custom_param_list):
            # Mask all other tokens
            logits[i, :] = -float("inf")
            # Assign highest probability to the specified token
            logits[i, param_dict[key]] = 0.0
        return logits
```

Send a request:

```python Example theme={null}
import requests

response = requests.post(
    "http://localhost:30000/generate",
    json={
        "text": "The capital of France is",
        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
        "sampling_params": {
            "temperature": 0.0,
            "max_new_tokens": 32,
            "custom_params": {"token_id": 5},
        },
    },
)
print(response.json())
```

Send an OpenAI chat completion request:

```python Example theme={null}
import openai
from sglang.utils import print_highlight

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0.0,
    max_tokens=32,
    extra_body={
        "custom_logit_processor": DeterministicLogitProcessor().to_str(),
        "custom_params": {"token_id": 5},
    },
)

print_highlight(f"Response: {response}")
```


# Tutorial: Sending a request
Source: https://docs.sglang.io/docs/basic_usage/send_request



This notebook provides a quick-start guide to use SGLang in chat completions after installation. Once your server is running, API documentation is available at `http://localhost:30000/docs` (Swagger UI), `http://localhost:30000/redoc` (ReDoc), or `http://localhost:30000/openapi.json` (OpenAPI spec, useful for AI agents). Replace `30000` with your port if using a different one.

* For Vision Language Models, see [OpenAI APIs - Vision](./openai_api_vision).
* For Embedding Models, see [OpenAI APIs - Embedding](./openai_api_embeddings) and [Encode (embedding model)](./native_api#encode-embedding-model).
* For Reward Models, see [Classify (reward model)](./native_api#classify-reward-model).

## Launch A Server

```python Example theme={null}
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import wait_for_server, terminate_process

# This is equivalent to running the following command in your terminal
# python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0

server_process, port = launch_server_cmd(
    """
python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct \
 --host 0.0.0.0 --log-level warning
"""
)

wait_for_server(f"http://localhost:{port}")
```

## Using cURL

```python Example theme={null}
import subprocess, json

curl_command = f"""
curl -s http://localhost:{port}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{{"model": "qwen/qwen2.5-0.5b-instruct", "messages": [{{"role": "user", "content": "What is the capital of France?"}}]}}'
"""

response = json.loads(subprocess.check_output(curl_command, shell=True))
print(response)
```

## Using Python Requests

```python Example theme={null}
import requests

url = f"http://localhost:{port}/v1/chat/completions"

data = {
    "model": "qwen/qwen2.5-0.5b-instruct",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
}

response = requests.post(url, json=data)
print(response.json())
```

## Using OpenAI Python Client

```python Example theme={null}
import openai

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print(response)
```

### Streaming

```python Example theme={null}
import openai

client = openai.Client(base_url=f"http://127.0.0.1:{port}/v1", api_key="None")

# Use stream=True for streaming responses
response = client.chat.completions.create(
    model="qwen/qwen2.5-0.5b-instruct",
    messages=[
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
    stream=True,
)

# Handle the streaming output
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

## Using Native Generation APIs

You can also use the native `/generate` endpoint with requests, which provides more flexibility. An API reference is available at [Sampling Parameters](./sampling_params).

```python Example theme={null}
import requests

response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
    },
)

print(response.json())
```

### Streaming

```python Example theme={null}
import requests, json

response = requests.post(
    f"http://localhost:{port}/generate",
    json={
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 32,
        },
        "stream": True,
    },
    stream=True,
)

prev = 0
for chunk in response.iter_lines(decode_unicode=False):
    chunk = chunk.decode("utf-8")
    if chunk and chunk.startswith("data:"):
        if chunk == "data: [DONE]":
            break
        data = json.loads(chunk[5:].strip("\n"))
        output = data["text"]
        print(output[prev:], end="", flush=True)
        prev = len(output)
```

```python Example theme={null}
terminate_process(server_process)
```


# Bench Serving Guide
Source: https://docs.sglang.io/docs/developer_guide/bench_serving



This guide explains how to benchmark online serving throughput and latency using `python -m sglang.bench_serving`. It supports multiple inference backends via OpenAI-compatible and native endpoints, and produces both console metrics and optional JSONL outputs.

### What it does

* Generates synthetic or dataset-driven prompts and submits them to a target serving endpoint
* Measures throughput, time-to-first-token (TTFT), inter-token latency (ITL), per-request end-to-end latency, and more
* Supports streaming or non-streaming modes, rate control, and concurrency limits

### Supported backends and endpoints

* `sglang` / `sglang-native`: `POST /generate`
* `sglang-oai`, `vllm`, `lmdeploy`: `POST /v1/completions`
* `sglang-oai-chat`, `vllm-chat`, `lmdeploy-chat`: `POST /v1/chat/completions`
* `sglang-embedding`, `vllm-embedding`: `POST /v1/embeddings`
* `trt` (TensorRT-LLM): `POST /v2/models/ensemble/generate_stream`
* `gserver`: Custom server (Not Implemented yet in this script)
* `truss`: `POST /v1/models/model:predict`

If `--base-url` is provided, requests are sent to it. Otherwise, `--host` and `--port` are used. When `--model` is not provided, the script will attempt to query `GET /v1/models` for an available model ID (OpenAI-compatible endpoints).

### Prerequisites

* Python 3.11+
* Dependencies typically used by this script: `aiohttp`, `numpy`, `requests`, `tqdm`, `transformers`, and for some datasets `datasets`, `pillow`, `pybase64`. Install as needed.
* An inference server running and reachable via the endpoints above
* If your server requires authentication, set environment variable `OPENAI_API_KEY` (used as `Authorization: Bearer <key>`)

### Quick start

Run a basic benchmark against an sglang server exposing `/generate`:

```bash Command theme={null}
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct
```

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --num-prompts 1000 \
  --model meta-llama/Llama-3.1-8B-Instruct
```

Or, using an OpenAI-compatible endpoint (completions):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --num-prompts 1000 \
  --model meta-llama/Llama-3.1-8B-Instruct
```

### Fair embedding comparison

Use the two embedding backends with the same model, tokenizer, input length, prompt count, and concurrency. The benchmark reports input-token throughput and end-to-end latency; embeddings have no decode-side TTFT or TPOT.

```bash Command theme={null}
# Start either server on the same hardware and precision, then run one at a time.
python3 -m sglang.bench_serving \
  --backend sglang-embedding \
  --model google/embeddinggemma-300m \
  --dataset-name random \
  --random-input-len 2048 \
  --num-prompts 300 \
  --max-concurrency 64 \
  --warmup-requests 3 \
  --flush-cache
```

```bash Command theme={null}
# vLLM's cache reset endpoint requires VLLM_SERVER_DEV_MODE=1 at server startup.
python3 -m sglang.bench_serving \
  --backend vllm-embedding \
  --model google/embeddinggemma-300m \
  --dataset-name random \
  --random-input-len 2048 \
  --num-prompts 300 \
  --max-concurrency 64 \
  --warmup-requests 3 \
  --flush-cache
```

`--flush-cache` calls `/flush_cache` for SGLang and `/reset_prefix_cache` for vLLM after warmup. For vLLM, start the server with `VLLM_SERVER_DEV_MODE=1`; without it the benchmark fails loudly rather than accidentally measuring warm-cache performance.

### Datasets

Select with `--dataset-name`:

* `sharegpt` (default): loads ShareGPT-style pairs; optionally restrict with `--sharegpt-context-len` and override outputs with `--sharegpt-output-len`
* `random`: random text lengths; sampled from ShareGPT token space
* `random-ids`: random token ids (can lead to gibberish)
* `image`: generates images and wraps them in chat messages; supports custom resolutions, multiple formats, and different content types
* `generated-shared-prefix`: synthetic dataset with shared long system prompts and short questions
* `mmmu`: samples from MMMU (Math split) and includes images
* `speed-bench`: [SPEED-Bench](https://huggingface.co/datasets/nvidia/SPEED-Bench) (**SPEculative Evaluation Dataset**) — a unified benchmark for evaluating [Speculative Decoding (SD)](https://arxiv.org/abs/2604.09557) algorithms. Uses the Throughput split, which provides fixed-length input sequences (1K–32K tokens) grouped into three output-entropy categories (`low_entropy`, `mixed`, `high_entropy`). Requires a pre-downloaded JSONL file passed via `--dataset-path`.
* `agentic-trace`: replays pre-built multi-turn agentic traces (e.g. OpenHands / SWE-smith). Each conversation is replayed round by round, feeding the server's real assistant reply back into the next round's history. Requires a chat backend (`--backend sglang-oai-chat`) and a trace JSON passed via `--dataset-path`.

Common dataset flags:

* `--num-prompts N`: number of requests

* `--random-input-len`, `--random-output-len`, `--random-range-ratio`: for random/random-ids/image

* `--image-count`: Number of images per request (for `image` dataset).

* `--apply-chat-template`: apply tokenizer chat template when constructing prompts

* `--dataset-path PATH`: file path for ShareGPT json; if blank and missing, it will be downloaded and cached

Generated Shared Prefix flags (for `generated-shared-prefix`):

* `--gsp-num-groups`
* `--gsp-prompts-per-group`
* `--gsp-system-prompt-len`
* `--gsp-question-len`
* `--gsp-output-len`
* `--gsp-group-distribution {uniform,zipf}`: per-request prefix-group sampling distribution (default: `uniform`). With `zipf`, each request's group is sampled by rank with `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)`; rank starts at 1 and group index 0 is the hottest. The on-disk dataset cache uses a distinct key per `(group_distribution, zipf_alpha)`, so uniform-mode caches are never mixed with zipf-mode caches.
* `--gsp-zipf-alpha FLOAT`: Zipf exponent for `--gsp-group-distribution=zipf`. Must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups. Required when the distribution is `zipf`; must be omitted otherwise.

Image dataset flags (for `image`):

* `--image-count`: Number of images per request
* `--image-resolution`: Image resolution; supports presets (4k, 1080p, 720p, 360p) or custom 'heightxwidth' format (e.g., 1080x1920, 512x768)
* `--image-format`: Image format (jpeg or png)
* `--image-content`: Image content type (random or blank)

Agentic trace flags (for `agentic-trace`):

* `--dataset-path`: path to the pre-built trace JSON
* `--sharegpt-output-len`: per-turn output length (default: 220)
* `--dataset-offset`: rotate the conversation list by this many entries before sampling, so successive sweep steps start on fresh conversations
* `--agentic-max-turns`: cap each conversation to at most this many turns (useful for small, fast profiling runs)

SPEED-Bench flags (for `speed-bench`):

* `--dataset-path PATH`: path to the pre-downloaded SPEED-Bench Throughput JSONL (e.g., `throughput_1k.jsonl`). Use the [SPEED-Bench measurement framework](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/specdec_bench) to generate it.
* `--speed-bench-category`: filter to one entropy category: `low_entropy`, `mixed`, or `high_entropy` (default: all)
* `--speed-bench-output-len`: fixed number of output tokens per request (default: 512)

### Examples

1. To benchmark image dataset with 3 images per request, 500 prompts, 512 input length, and 512 output length, you can run:

```bash Command theme={null}
python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-3B-Instruct --disable-radix-cache
```

```bash Command theme={null}
python -m sglang.bench_serving \
    --backend sglang-oai-chat \
    --dataset-name image \
    --num-prompts 500 \
    --image-count 3 \
    --image-resolution 720p \
    --random-input-len 512 \
    --random-output-len 512
```

2. To benchmark random dataset with 3000 prompts, 1024 input length, and 1024 output length, you can run:

```bash Command theme={null}
python -m sglang.launch_server --model-path Qwen/Qwen2.5-3B-Instruct
```

```bash Command theme={null}
python3 -m sglang.bench_serving \
    --backend sglang \
    --dataset-name random \
    --num-prompts 3000 \
    --random-input 1024 \
    --random-output 1024 \
    --random-range-ratio 0.5
```

3. To benchmark speculative decoding throughput using SPEED-Bench (mixed-entropy category, 1K ISL), you can run:

```bash Command theme={null}
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct \
    --speculative-algorithm EAGLE --speculative-draft-model-path <draft-model-path>
```

```bash Command theme={null}
python3 -m sglang.bench_serving \
    --backend sglang \
    --dataset-name speed-bench \
    --dataset-path /path/to/throughput_1k.jsonl \
    --speed-bench-category mixed \
    --speed-bench-output-len 512 \
    --num-prompts 512
```

### Choosing model and tokenizer

* `--model` is required unless the backend exposes `GET /v1/models`, in which case the first model ID is auto-selected.
* `--tokenizer` defaults to `--model`. Both can be HF model IDs or local paths.
* For ModelScope workflows, setting `SGLANG_USE_MODELSCOPE=true` enables fetching via ModelScope (weights are skipped for speed).
* If your tokenizer lacks a chat template, the script warns because token counting can be less robust for gibberish outputs.

### Rate, concurrency, and streaming

* `--request-rate`: requests per second. `inf` sends all immediately (burst). Non-infinite rate uses a Poisson process for arrival times.
* `--max-concurrency`: caps concurrent in-flight requests regardless of arrival rate.
* `--disable-stream`: switch to non-streaming mode when supported; TTFT then equals total latency for chat completions.

### Other key options

* `--output-file FILE.jsonl`: append JSONL results to file; auto-named if unspecified
* `--output-details`: include per-request arrays (generated texts, errors, ttfts, itls, input/output lens)
* `--extra-request-body '&#123;"top_p":0.9,"temperature":0.6&#125;'`: merged into payload (sampling params, etc.)
* `--disable-ignore-eos`: pass through EOS behavior (varies by backend)
* `--warmup-requests N`: run warmup requests with short output first (default 1)
* `--flush-cache`: call `/flush_cache` (sglang) before main run
* `--profile`: call `/start_profile` and `/stop_profile` (requires server to enable profiling, e.g., `SGLANG_TORCH_PROFILER_DIR`)
* `--lora-name name1 name2 ...`: randomly pick one per request and pass to backend (e.g., `lora_path` for sglang)
* `--tokenize-prompt`: send integer IDs instead of text (currently supports `--backend sglang` only)

### Authentication

If your target endpoint requires OpenAI-style auth, set:

```bash Command theme={null}
export OPENAI_API_KEY=sk-...yourkey...
```

The script will add `Authorization: Bearer $OPENAI_API_KEY` automatically for OpenAI-compatible routes.

### Metrics explained

Printed after each run:

* Request throughput (req/s)
* Input token throughput (tok/s) - includes both text and vision tokens
* Output token throughput (tok/s)
* Total token throughput (tok/s) - includes both text and vision tokens
* Total input text tokens and Total input vision tokens - per-modality breakdown
* Concurrency: aggregate time of all requests divided by wall time
* End-to-End Latency (ms): mean/median/std/p99 per-request total latency
* Time to First Token (TTFT, ms): mean/median/std/p99 for streaming mode
* Inter-Token Latency (ITL, ms): mean/median/std/p95/p99/max between tokens
* TPOT (ms): Token processing time after first token, i.e., `(latency - ttft)/(tokens-1)`
* Accept length (sglang-only, if available): speculative decoding accept length

The script also retokenizes generated text with the configured tokenizer and reports "retokenized" counts.

### Goodput

Throughput counts every completed request, including the ones that came back too late to be useful. Pass service level objectives with `--goodput` to count only the requests that met them:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang --host 127.0.0.1 --port 30000 \
  --num-prompts 1000 --request-rate 16 \
  --goodput ttft:500 tpot:50 e2el:20000
```

Each entry is `KEY:VALUE` in milliseconds, with `KEY` one of `ttft`, `tpot`, `e2el` (end-to-end latency). A request is good if it succeeded and met every listed SLO; a single-token response has a TPOT of 0. The run then also prints:

* Request goodput (req/s): good requests per second
* SLO attainment (%): good requests over all requests sent, so failed requests count as misses
* One attainment line per listed SLO, which shows which SLO is the binding one

Sweeping `--request-rate` and reading goodput gives the highest rate the deployment sustains within its SLOs: throughput keeps rising or flattens under overload, goodput peaks and then falls.

### JSONL output format

When `--output-file` is set, one JSON object is appended per run. Base fields:

* Arguments summary: backend, dataset, request\_rate, max\_concurrency, etc.
* Duration and totals: completed, total\_input\_tokens, total\_output\_tokens, retokenized totals
* Throughputs and latency statistics as printed in the console
* `accept_length` when available (sglang)
* `goodput_slos_ms`, `request_goodput`, `slo_attainment`, `slo_attainment_by_metric` when `--goodput` is given

With `--output-details`, an extended object also includes arrays:

* `input_lens`, `output_lens`
* `ttfts`, `itls` (per request: ITL arrays)
* `generated_texts`, `errors`

### End-to-end examples

1. sglang native `/generate` (streaming):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 1024 --random-range-ratio 0.5 \
  --num-prompts 2000 \
  --request-rate 100 \
  --max-concurrency 512 \
  --output-file sglang_random.jsonl --output-details
```

2. OpenAI-compatible Completions (e.g., vLLM):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend vllm \
  --base-url http://127.0.0.1:8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name sharegpt \
  --num-prompts 1000 \
  --sharegpt-output-len 256
```

3. OpenAI-compatible Chat Completions (streaming):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend vllm-chat \
  --base-url http://127.0.0.1:8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --num-prompts 500 \
  --apply-chat-template
```

4. Images (VLM) with chat template:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model your-vlm-model \
  --dataset-name image \
  --image-count 2 \
  --image-resolution 720p \
  --random-input-len 128 --random-output-len 256 \
  --num-prompts 200 \
  --apply-chat-template
```

4a) Images with custom resolution:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model your-vlm-model \
  --dataset-name image \
  --image-count 1 \
  --image-resolution 512x768 \
  --random-input-len 64 --random-output-len 128 \
  --num-prompts 100 \
  --apply-chat-template
```

4b) 1080p images with PNG format and blank content:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model your-vlm-model \
  --dataset-name image \
  --image-count 1 \
  --image-resolution 1080p \
  --image-format png \
  --image-content blank \
  --random-input-len 64 --random-output-len 128 \
  --num-prompts 100 \
  --apply-chat-template
```

5. Generated shared prefix (long system prompts + short questions):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name generated-shared-prefix \
  --gsp-num-groups 64 --gsp-prompts-per-group 16 \
  --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \
  --num-prompts 1024
```

Zipfian / power-law prefix popularity (opt-in via `--gsp-group-distribution=zipf`):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name generated-shared-prefix \
  --gsp-num-groups 64 --gsp-prompts-per-group 16 \
  --gsp-system-prompt-len 2048 --gsp-question-len 128 --gsp-output-len 256 \
  --gsp-group-distribution zipf --gsp-zipf-alpha 1.2 \
  --seed 42
```

`zipf` mode samples each request's prefix group from the rank-based distribution `p(rank) = (1/rank**alpha) / sum_k(1/k**alpha)` with rank starting at 1, so group index 0 is the hottest. The total request count stays `num_groups * prompts_per_group` — identical to `uniform` mode — and only the per-request group assignment changes. `alpha` must be a finite float strictly greater than 0; larger values concentrate requests on lower-ranked (hotter) groups.

The on-disk dataset cache at `~/.cache/sglang/benchmark/gen_shared_prefix_*.pkl` includes `group_distribution` and `zipf_alpha` in its key, so uniform-mode and zipf-mode runs (or two zipf runs with different alpha) never share a cache file. Uniform-mode filenames are unchanged from the legacy format, so existing caches remain valid.

This flag controls prefix-popularity shape only. It does not by itself reproduce any production trace or guarantee an observed cache-hit rate for a given engine.

6. Tokenized prompts (ids) for strict length control (sglang only):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --tokenize-prompt \
  --random-input-len 2048 --random-output-len 256 --random-range-ratio 0.2
```

7. Profiling and cache flush (sglang):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --profile \
  --flush-cache
```

8. TensorRT-LLM streaming endpoint:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend trt \
  --base-url http://127.0.0.1:8000 \
  --model your-trt-llm-model \
  --dataset-name random \
  --num-prompts 100 \
  --disable-ignore-eos
```

9. Evaluating large-scale KVCache sharing with mooncake trace (sglang only):

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30000 \
  --model model-name \
  --dataset-name mooncake \
  --mooncake-slowdown-factor 1.0 \
  --mooncake-num-rounds 1000 \
  --mooncake-workload conversation|mooncake|agent|synthetic
  --use-trace-timestamps true \
  --random-output-len 256
```

10. Fake decode stress testing (PD disaggregation, decode-only):

When benchmarking pure decode performance in a PD disaggregation setup, you can bypass the prefill node entirely by using `--fake-prefill`. This requires the decode server to be started with `--disaggregation-transfer-backend fake`:

```bash Command theme={null}
# Step 1: Start a decode-only server with fake transfer backend
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend fake \
  --port 30001

# Step 2: Run bench_serving with --fake-prefill
python3 -m sglang.bench_serving \
  --backend sglang \
  --host 127.0.0.1 --port 30001 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --num-prompts 500 \
  --random-input-len 1024 --random-output-len 256 \
  --fake-prefill
```

Similarly, `bench_one_batch_server` also supports `--fake-prefill`:

```bash Command theme={null}
python3 -m sglang.bench_one_batch_server \
  --base-url http://127.0.0.1:30001 \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --batch-size 32 --input-len 1024 --output-len 256 \
  --fake-prefill
```

The `--fake-prefill` flag automatically injects special sentinel values into each request, telling the decode server to skip real KV transfer and generate fake KV data locally.

### Troubleshooting

* All requests failed: verify `--backend`, server URL/port, `--model`, and authentication. Check warmup errors printed by the script.
* Throughput seems too low: adjust `--request-rate` and `--max-concurrency`; verify server batch size/scheduling; ensure streaming is enabled if appropriate.
* Token counts look odd: prefer chat/instruct models with proper chat templates; otherwise tokenization of gibberish may be inconsistent.
* Image/MMMU datasets: ensure you installed extra deps (`pillow`, `datasets`, `pybase64`).
* Authentication errors (401/403): set `OPENAI_API_KEY` or disable auth on your server.

### Notes

* The script raises the file descriptor soft limit (`RLIMIT_NOFILE`) to help with many concurrent connections.
* For sglang, `/server_info` is queried post-run to report speculative decoding accept length when available.


# Benchmark and Profiling
Source: https://docs.sglang.io/docs/developer_guide/benchmark_and_profiling



## Benchmark

SGLang provides four benchmark tools that operate at different levels of the stack. The table below summarizes their key differences:

<table>
  <thead>
    <tr>
      <th>Tool</th>
      <th>HTTP Server</th>
      <th>Scheduler</th>
      <th>Use Case</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`bench_serving`</td>
      <td>Yes (async HTTP client to a running server)</td>
      <td>Yes (indirectly, via server)</td>
      <td>Realistic online serving benchmarks with latency metrics (TTFT, TPOT, ITL)</td>
    </tr>

    <tr>
      <td>`bench_one_batch_server`</td>
      <td>Yes (sends HTTP requests to a running server)</td>
      <td>Yes (indirectly, via server)</td>
      <td>End-to-end single-batch latency including HTTP and scheduler overhead</td>
    </tr>

    <tr>
      <td>`bench_offline_throughput`</td>
      <td>No</td>
      <td>Yes (directly uses `Engine` in-process)</td>
      <td>Maximum throughput measurement without HTTP overhead</td>
    </tr>

    <tr>
      <td>`bench_one_batch`</td>
      <td>No</td>
      <td>No (directly calls `ModelRunner`)</td>
      <td>Kernel-level latency profiling of a single static batch</td>
    </tr>
  </tbody>
</table>

Use `bench_serving` by default unless there are specific needs.

**`bench_serving`** is an async HTTP load-testing client that sends requests at controlled rates with configurable concurrency to a running server. It measures realistic online serving metrics including time-to-first-token (TTFT), time-per-output-token (TPOT), inter-token latency (ITL), and throughput. Use `num-prompts >= 5 * max-concurrency` to measure steady-state performance. Launch a server with `sglang.launch_server` first.

```bash Command theme={null}
python3 -m sglang.bench_serving --backend sglang --max-concurrency 16 --num-prompts 80 --random-input-len 256 --random-output-len 32 --dataset-name random
```

**`bench_one_batch_server`** sends a single batch as one HTTP request to a running server. Due to only having a single batch, the server is never in a steady-state and metrics will be biased. Launch a server with `sglang.launch_server` first.

```bash Command theme={null}
python3 -m sglang.bench_one_batch_server --base-url http://127.0.0.1:30000 --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32
```

* Pass `--enable-multi-batch` and set `--batch-size` to a multiple of the server's `--max-running-requests` to stabilize throughput measurements. Surplus requests are queued by the scheduler and promoted batch-by-batch, amortizing per-request prefill and first-step transients into steady-state decode. Under this flag, only `overall_throughput` is authoritative; `input_throughput`, `output_throughput`, `last_ttft`, and ITL include cross-batch queueing in their denominators and should be treated as informational.
* Pass `--disable-ignore-eos` when benchmarking with real prompts, where forcing decode past EOS would shift the output distribution toward gibberish. Requests then early-stop, so the whole-run `output_throughput`/ITL include the decaying-batch tail; the report gains steady-state columns measured only over the window where every request is still decoding (from the last request's first token to the first request's finish). Replay recorded traffic with `--dataset-name sharegpt` or `--dataset-name custom --dataset-path <conversations.jsonl>` (recorded completion lens are ignored; `--output-len` is the shared `max_new_tokens` cap). For OpenAI-format traces with per-request parameters, use `bench_serving`.
* Pass `--lora-name <name>` to route every prompt through a pre-loaded LoRA adapter. Requires the server to be launched with `--enable-lora --lora-paths <name>=<path>`.

**`bench_offline_throughput`** directly instantiates the `Engine` object in-process (no HTTP server) and submits all requests at once via `engine.generate()`. The engine's scheduler handles batching and execution. This measures maximum achievable throughput without any network overhead.

```bash Command theme={null}
python3 -m sglang.benchmark.offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10
```

**`bench_one_batch`** is the lowest-level tool. It directly instantiates a `ModelRunner` and calls `extend()` / `decode()` on a fixed static batch, bypassing the scheduler entirely. The prefill and decode phases are run separately, making profiling easier but rendering the metrics unrealistic. Because there is no dynamic batching, it may run out of memory for batch sizes that a real server can handle (a real server chunks prefill into smaller batches). This is best suited for profiling individual kernel performance.

```bash Command theme={null}
python3 -m sglang.benchmark.one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32
```

## Profile with PyTorch Profiler

[Pytorch Profiler](https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html) is a convenient basic tool to inspect kernel execution time, call stack, and kernel overlap and occupancy.

### Profile a server with `sglang.bench_serving`

```bash Command theme={null}
# set trace path
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log

# start server
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct

# send profiling request from client
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile
```

For `bench_serving --profile`, the output directory is selected on the client side from `--profile-output-dir` or `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`), then sent in the `/start_profile` request.
If you call `/start_profile` directly and do not provide `output_dir`, the server uses its own `SGLANG_TORCH_PROFILER_DIR` (fallback: `/tmp`).

Setting `SGLANG_TORCH_PROFILER_DIR` on both server and client is still recommended to avoid confusion about where traces are written.

For more details, please refer to [Bench Serving Guide](./bench_serving).

### Profile In PD Disaggregation Mode

When profiling in PD disaggregation mode, prefill and decode workers **must be profiled separately** due to torch profiler limitations. The `bench_serving` command provides dedicated options for this:

#### Profile Prefill Workers

```bash Command theme={null}
# set trace path
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log

# start prefill and decode servers (see PD disaggregation docs for setup)
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode prefill
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disaggregation-mode decode --port 30001 --base-gpu-id 1

# start router
python -m sglang_router.launch_router --pd-disaggregation --prefill http://127.0.0.1:30000 --decode http://127.0.0.1:30001 --host 0.0.0.0 --port 8000

# send profiling request targeting prefill workers
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000
```

#### Profile Decode Workers

```bash Command theme={null}
# send profiling request targeting decode workers
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --sharegpt-output-len 100 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001
```

#### Important Notes

* `--profile-prefill-url` and `--profile-decode-url` are **mutually exclusive** - you cannot profile both at the same time
* Both options support multiple worker URLs for multi-instance setups:
  ```bash Command theme={null}
  # Profile multiple prefill workers
  python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-prefill-url http://127.0.0.1:30000 http://127.0.0.1:30002

  # Profile multiple decode workers
  python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 10 --profile --pd-separated --profile-decode-url http://127.0.0.1:30001 http://127.0.0.1:30003
  ```
* Make sure `SGLANG_TORCH_PROFILER_DIR` is set on all worker nodes before starting the servers
* For more details on setting up PD disaggregation, see [PD Disaggregation Guide](../advanced_features/pd_disaggregation)

### Profile a server with `sglang.benchmark.offline_throughput`

```bash Command theme={null}
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log

# profile one batch with bench_one_batch.py
# batch size can be controlled with --batch argument
python3 -m sglang.benchmark.one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile

# profile multiple batches with bench_offline_throughput.py
python -m sglang.benchmark.offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8
```

### Profile a server with `sglang.profiler`

When the server is running (e.g., processing a decoding request), you can start live profiling immediately by sending a profile request to the server.

You can do this by running `python3 -m sglang.profiler`. For example:

```text Output theme={null}
# Terminal 1: Send a generation request
python3 -m sglang.test.send_one

# Terminal 2: Before the above request finishes, quickly launch the following command in a separate terminal.
# It will generate a profile of the above request for several decoding batches.
python3 -m sglang.profiler
```

You can also combine the above operations into a single command

```text Output theme={null}
python3 -m sglang.test.send_one --profile
```

### Profile a server with HTTP API endpoints

SGLang provides HTTP API endpoints to control profiling on a running server. This allows you to start and stop profiling programmatically, which is useful for capturing specific workload patterns.

#### Using `/start_profile` endpoint

The `/start_profile` endpoint starts profiling on the server. You can control when profiling begins and how long it runs using the following parameters:

**Basic usage:**

```bash Command theme={null}
# Start profiling immediately for 10 steps
curl -X POST http://127.0.0.1:30000/start_profile \
  -H "Content-Type: application/json" \
  -d '{
    "num_steps": 10
  }'
```

**Parameters:**

* `output_dir` (optional): Directory where profile traces will be saved. If not specified, uses `SGLANG_TORCH_PROFILER_DIR` environment variable, or `/tmp` as the default
* `num_steps` (optional): Number of steps to profile. If not specified, profiling continues until manually stopped with `/stop_profile`
* `start_step` (optional): Step number at which to start profiling (inclusive). Useful for skipping warmup iterations
* `activities` (optional): List of activities to profile, e.g., `["CPU", "GPU"]`. Default is `["CPU", "GPU"]`
* `merge_profiles` (optional): Whether to merge distributed traces. Default is `false`
* `detailed_annotations` (optional): Whether to fold per-iteration request and KV-length aggregates into the trace's `step[...]` markers, for detailed analysis. Default is `false`. See [Detailed annotations](#detailed-annotations) below.

**Note on step ranges:** Profiling starts at `start_step` (inclusive) and continues for `num_steps` iterations. For example, with `start_step=3` and `num_steps=10`, profiling captures steps 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 (10 steps total, starting from step 3).

**Advanced usage with `start_step`:**

```bash Command theme={null}
# Wait 5 steps (warmup), then profile for 10 steps
curl -X POST http://127.0.0.1:30000/start_profile \
  -H "Content-Type: application/json" \
  -d '{
    "output_dir": "/tmp/profiles",
    "start_step": 5,
    "num_steps": 10,
    "activities": ["CPU", "GPU"]
  }'
```

**Continuous profiling (manual stop):**

```bash Command theme={null}
# Start profiling without num_steps - must manually stop with /stop_profile
curl -X POST http://127.0.0.1:30000/start_profile
```

#### Detailed annotations

Set `detailed_annotations` to `true` to fold per-iteration aggregates into SGLang's existing per-forward `step[...]` span. For every execution step that runs while profiling is active, SGLang augments that step's marker on the GPU stream with the request and KV-length distribution of the step, so you can reconstruct compute and memory bounds directly from the trace without per-request details.

All four per-request aggregates are appended, prefixed by phase — `c_` for context (prefill) and `g_` for generation (decode). The per-phase `sq` is always emitted so each `step[...]` label is self-contained for roofline analysis, even where it duplicates the base label's `bs` (decode) or `toks` (prefill):

* `sq`: total query tokens (`Σ N_Q`)
* `sqsq`: sum of squared query tokens per request (`Σ N_Q²`)
* `sqsk`: sum of query·KV tokens per request (`Σ N_Q·N_KV`)
* `sk`: total KV tokens (`Σ N_KV`)

A pure prefill (`EXTEND`) or decode (`DECODE`) forward emits a single group; a mixed forward emits both, with `c=`/`g=` request counts. Example labels:

```text theme={null}
step[EXTEND bs=1 toks=1025 c_sq=1025 c_sqsq=1050625 c_sqsk=1050625 c_sk=1025]
step[DECODE bs=64 g_sq=64 g_sqsq=64 g_sqsk=100032 g_sk=100032]
step[MIXED bs=66 c=2 g=64 c_sq=2048 c_sk=2048 c_sqsq=2097152 c_sqsk=2097152 g_sq=64 g_sk=65600 g_sqsq=64 g_sqsk=65600]
```

With speculative decoding (EAGLE/MTP) each request contributes multiple query tokens per step, so `sq` no longer equals `bs`. Both draft-decode and target-verify (`TARGET_VERIFY`) steps are emitted in the **generation** group. For example, a verify step with 3 draft tokens across 2 requests (`seq_lens=[10, 20]`):

```text theme={null}
step[TARGET_VERIFY bs=2 g_sq=6 g_sqsq=18 g_sqsk=90 g_sk=30]
```

```bash Command theme={null}
# Profile 10 steps with detailed annotations enabled
curl -X POST http://127.0.0.1:30000/start_profile \
  -H "Content-Type: application/json" \
  -d '{
    "output_dir": "/tmp/profiles",
    "num_steps": 10,
    "activities": ["CPU", "GPU"],
    "detailed_annotations": true
  }'
```

The annotations only appear when profiling is active with `detailed_annotations` enabled, so they add no overhead on the normal serving path. The behavior is identical in eager and CUDA graph modes. When viewing the trace (see [View traces](#view-traces)), the augmented `step[...]` markers appear on the GPU stream alongside the kernels for each step.

#### Using `/stop_profile` endpoint

The `/stop_profile` endpoint stops an ongoing profiling session and saves the trace file.

```bash Command theme={null}
# Stop profiling and save traces
curl -X POST http://127.0.0.1:30000/stop_profile
```

This is only needed when you start profiling without specifying `num_steps`. If `num_steps` is specified, profiling will automatically stop after that many steps.

#### Example workflow

```bash Command theme={null}
# Terminal 1: Start the server
export SGLANG_TORCH_PROFILER_DIR=/tmp/profiles
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct

# Terminal 2: Start continuous profiling
curl -X POST http://127.0.0.1:30000/start_profile \
  -H "Content-Type: application/json" \
  -d '{
    "start_step": 3
  }'

# Terminal 3: Send requests to generate load
python -m sglang.bench_serving --backend sglang --num-prompts 100

# Terminal 2: Stop profiling when done
curl -X POST http://127.0.0.1:30000/stop_profile
```

### Profiler Trace Merger for Distributed Traces

SGLang now supports automatic merging of profiling traces from distributed setups with multiple parallelism types (TP, DP, PP, EP). This feature is particularly useful for analyzing performance across distributed runs.

#### Multi-Node Profiling and Shared Storage Considerations

Single-node profiler output merging is completely supported. When profiling in distributed environments spanning multiple nodes, shared storage (e.g., NFS, Lustre) should be accessible by all nodes for the output directory to enable merging of trace files.

If there is no shared storage accessible across nodes, automatic merging of trace files during profiling is not supported directly as of now.

#### HTTP API Usage

```bash Command theme={null}
# Start profiling with automatic trace merging enabled
curl -X POST <BASE_URL>/start_profile \
  -H "Content-Type: application/json" \
  -d '{
    "output_dir": "/tmp/profiles", # where to store profile traces
    "num_steps": 10,
    "activities": ["CPU", "GPU"],
    "merge_profiles": true # optional argument to merge profile traces (default=False)
  }'
```

#### Command Line Usage

```bash Command theme={null}
# Start profiling with merge enabled
python -m sglang.profiler \
  --num-steps 10 \
  --cpu \
  --gpu \
  --output-dir /tmp/profiles \
  --merge-profiles # optional argument to merge profile traces (default=False)
```

#### Output Files

The profile merger generates:

* Individual rank trace files: `&#123;profile_id&#125;-TP-&#123;tp&#125;-DP-&#123;dp&#125;-PP-&#123;pp&#125;-EP-&#123;ep&#125;.trace.json.gz`
* Merged trace file: `merged-&#123;profile_id&#125;.trace.json.gz`

### Profile the CUDA graph capture phase

The tools above profile the steady-state runtime (prefill / decode). To instead profile the **CUDA graph capture phase** that runs once at server startup, launch the server with `--enable-profile-cuda-graph`. This runs a PyTorch Profiler pass over the decode CUDA-graph capture, which is useful for diagnosing slow or memory-heavy graph capture.

`--enable-profile-cuda-graph` (server arg) builds the capture profiler and always emits the per-kernel CPU/CUDA time summary tables and a CUDA memory snapshot. Persisting Chrome traces to disk is opt-in via one of two env vars (both no-ops unless `--enable-profile-cuda-graph` is also set):

* `SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE=1` — writes **one combined trace per tensor-parallel rank** for the whole capture pass, named `cuda_graph_capture-&#123;runner&#125;-TP-&#123;tp_rank&#125;.json.gz`.
* `SGLANG_GRAPH_BATCH_CAPTURE=1` — writes **one trace per captured batch size per rank**, named `&#123;runner&#125;_bs_&#123;bs&#125;_rank&#123;tp_rank&#125;.json.gz`. The profiler runs on a `wait=2, warmup=0, active=1` schedule (the two dummy runs before each capture are skipped) with `record_shapes`, `with_stack`, `with_flops`, and `profile_memory` enabled, giving per-shape kernel identities, input shapes, FLOPs, and memory.

If both env vars are set, `SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE` (the single combined trace) takes precedence.

```bash Command theme={null}
# set trace path
export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log

# opt in to per-batch-size capture traces (or set
# SGLANG_ENABLE_CUDA_GRAPH_CAPTURE_TRACE=1 for a single combined trace per rank)
export SGLANG_GRAPH_BATCH_CAPTURE=1

# launch the server with CUDA graph capture profiling enabled
python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --enable-profile-cuda-graph
```

Behavior and output:

* All traces are written to `$&#123;SGLANG_TORCH_PROFILER_DIR&#125;/graph_capture_profile/` (defaults to `/tmp/graph_capture_profile/` if the variable is unset). Files are namespaced by runner class and TP rank so concurrent capture passes (e.g. EAGLE target/draft/draft-extend) and ranks don't collide.
* A CUDA memory snapshot (`cuda_graph_runner_memory_usage.pickle`) and per-kernel CPU/CUDA time summary tables are always emitted for the capture phase (independent of the env vars above).
* Only the decode CUDA-graph runner is profiled.

The capture traces are viewed the same way as other PyTorch Profiler traces (see [View traces](#view-traces)).

### Possible PyTorch bugs

If in any cases you encounter the following error (for example, using qwen 2.5 VL):

```bash Command theme={null}
RuntimeError: !stack.empty() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/autograd/profiler_python.cpp":983, please report a bug to PyTorch. Python replay stack is empty.
```

This is likely a PyTorch Bug reported in [Bug: vLLM Profiler](https://github.com/vllm-project/vllm/issues/18240) and [Bug: torch.profiler.profile](https://github.com/pytorch/pytorch/issues/101632). As a workaround, you may disable `with_stack` with an environment variable such as follows:

```bash Command theme={null}
export SGLANG_PROFILE_WITH_STACK=False
python -m sglang.benchmark.offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8
```

### View traces

Trace files can be loaded and visualized from:

1. [https://ui.perfetto.dev/](https://ui.perfetto.dev/) (any browser)
2. chrome://tracing (Chrome browser only)

If browser cannot open trace file due to its large size,
client can generate a small trace file (\<100MB) by controlling number of prompts and lengths of prompt outputs.
For example, when profiling a server,

```bash Command theme={null}
python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B-Instruct --num-prompts 2 --sharegpt-output-len 100 --profile
```

This command sets the number of prompts to 2 with `--num-prompts` argument and limits the length of output sequences to 100 with `--sharegpt-output-len` argument, which can generate a small trace file for browser to open smoothly.

Additionally, if you want to locate the SGLang Python source code through the cuda kernel in Trace, you need to disable CUDA Graph when starting the service. This can be done by using the `--disable-cuda-graph` parameter in the command to start the service.

## Profile with Nsight

[Nsight systems](https://docs.nvidia.com/nsight-systems/) is an advanced tool that exposes more profiling details, such as register and shared memory usage, annotated code regions and low-level CUDA APIs and events.

1. Prerequisite:

   Install using apt, or run inside a [NVIDIA Docker container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags) or [SGLang Docker container](https://github.com/sgl-project/sglang/tree/main/docker).

   ```bash Command theme={null}
   # install nsys
   # https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html
   apt update
   apt install -y --no-install-recommends gnupg
   echo "deb http://developer.download.nvidia.com/devtools/repos/ubuntu$(source /etc/lsb-release; echo "$DISTRIB_RELEASE" | tr -d .)/$(dpkg --print-architecture) /" | tee /etc/apt/sources.list.d/nvidia-devtools.list
   apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
   apt update
   apt install nsight-systems-cli
   ```

2. To profile a single batch, use

   ```bash Command theme={null}
   nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node python3 -m sglang.benchmark.one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 64 --input-len 512
   ```

3. To profile a server, e.g.

   ```bash Command theme={null}
   # launch the server, set the delay and duration times according to needs
   # after the duration time has been used up, server will be killed by nsys

   nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node -o sglang.out --delay 60 --duration 70 python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --disable-radix-cache

   # client
   python3 -m sglang.bench_serving --backend sglang --num-prompts 1000 --dataset-name random --random-input 1024 --random-output 512
   ```

   In practice, we recommend users to set `--duration` argument to a large value. Whenever user wants the server to stop profiling. Firstly run:

   ```bash Command theme={null}
   nsys sessions list
   ```

   to get the session id in the form of `profile-XXXXX`, then run:

   ```bash Command theme={null}
   nsys stop --session=profile-XXXXX
   ```

   to manually kill the profiler and generate `nsys-rep` files instantly.

4. Use NVTX to annotate code regions, e.g. to see their execution time.

   ```bash Command theme={null}
   # install nvtx
   pip install nvtx
   ```

   ```python Example theme={null}
   # code snippets
   import nvtx
   with nvtx.annotate("description", color="color"):
       # some critical code
   ```

### Layer-wise NVTX Profiling with Nsight Systems

SGLang provides built-in layerwise NVTX annotations that can be combined with the CUDA Profiler for detailed per-layer profiling in Nsight Systems. This is particularly useful for identifying performance bottlenecks at the layer level.

#### Using `--enable-layerwise-nvtx-marker` with Nsight Systems and `/start_profile`

The `--enable-layerwise-nvtx-marker` flag automatically adds NVTX markers to every layer in your model. This is particularly powerful when combined with Nsight Systems profiling to see detailed per-layer performance.

**Method 1: Using `/start_profile` with CUDA\_PROFILER (for programmatic control)**

This method allows you to control exactly when profiling starts/stops via HTTP API while Nsight Systems is running.

1. Launch the server with layerwise NVTX enabled under Nsight Systems:

   ```bash Command theme={null}
   # Terminal 1: Start server with nsys and capture-range option
   nsys profile --trace-fork-before-exec=true \
     --cuda-graph-trace=node \
     --capture-range=cudaProfilerApi \
     --capture-range-end=stop \
     -o layerwise_profile \
     python -m sglang.launch_server \
       --model-path meta-llama/Llama-3.1-8B-Instruct \
       --enable-layerwise-nvtx-marker \
       --disable-cuda-graph
   ```

   Note: NVTX markers are not emitted for kernel launches captured by CUDA graphs. Use `--disable-cuda-graph` to ensure all layerwise NVTX markers are emitted in the trace.

2. In another terminal, control profiling via `/start_profile` with `CUDA_PROFILER` activity:

   ```bash Command theme={null}
   # Terminal 2: Wait for server to be ready, then start CUDA profiling
   # Wait 3 steps for warmup, then profile for 10 steps
   curl -X POST http://127.0.0.1:30000/start_profile \
     -H "Content-Type: application/json" \
     -d '{
       "start_step": 3,
       "num_steps": 10,
       "activities": ["CUDA_PROFILER"]
     }'
   ```

3. Send requests to generate load:

   ```bash Command theme={null}
   # Terminal 3: Generate workload
   python -m sglang.bench_serving --backend sglang --num-prompts 100
   ```

4. Profiling will automatically stop after 10 steps (due to `num_steps: 10`). If you hadn't specified `num_steps`, you would need to manually stop it:

   ```bash Command theme={null}
   # Terminal 2: Only needed if num_steps was not specified
   curl -X POST http://127.0.0.1:30000/stop_profile
   ```

The `--capture-range=cudaProfilerApi` option tells Nsight Systems to only capture data between `cudaProfilerStart()` and `cudaProfilerStop()` calls (triggered by `/start_profile` and `/stop_profile`), reducing overhead and file size. The `start_step` parameter skips the first 3 steps to avoid capturing warmup overhead.

**Method 2: Simpler approach without `/start_profile` API**

For simpler use cases where you don't need fine-grained control over profiling start/stop, you can profile with Nsight Systems capturing the entire workload:

```bash Command theme={null}
# Terminal 1: Start server with layerwise NVTX
# Note: --disable-cuda-graph ensures all NVTX markers are emitted
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --enable-layerwise-nvtx-marker \
  --disable-cuda-graph

# Terminal 2: Profile the benchmarking client
nsys profile --trace-fork-before-exec=true \
  --cuda-graph-trace=node \
  -o layerwise_profile \
  python -m sglang.bench_serving --backend sglang --num-prompts 10
```

This approach profiles the entire client execution, including all server interactions. The layerwise NVTX markers will be visible in the Nsight Systems timeline.

**Viewing the profiling results:**

Open the generated `.qdrep` file with Nsight Systems:

```bash Command theme={null}
nsys-ui layerwise_profile.qdrep
```

In the Nsight Systems GUI, you'll see:

* **NVTX ranges**: Each layer appears as a labeled range in the timeline with detailed information in the marker metadata
* **CUDA kernels**: All GPU kernels are shown alongside the layer annotations
* **Layer hierarchy**: The full module path (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct.model.layers.0.self_attn.qkv_proj`) helps identify specific layers. The prefix uses the full model path from `--model-path`.
* **Tensor shapes**: Input/output dimensions and parameter shapes are included in the NVTX marker data

**Benefits of layerwise NVTX profiling:**

* **Granular visibility**: See exactly which layers are taking the most time
* **Memory tracking**: Identify layers with large memory allocations
* **Bottleneck identification**: Quickly locate inefficient operations
* **Communication overhead**: In multi-GPU setups, see per-layer communication costs
* **Development debugging**: Validate that model architecture changes have the expected performance impact

## Other tips

1. You can benchmark a model using dummy weights by only providing the config.json file. This allows for quick testing of model variants without training. To do so, add `--load-format dummy` to the above commands and then you only need a correct `config.json` under the checkpoint folder.

2. You can benchmark a model with modified configs (e.g., less layers) by using `--json-model-override-args`. For example, you can benchmark a model with only 2 layers and 2 kv heads using:

   ```bash Command theme={null}
   python -m sglang.benchmark.one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32 --load-format dummy --json-model-override-args '{"num_hidden_layers": 1, "num_key_value_heads": 1}'
   ```

3. You can use `--python-backtrace=cuda` to see python call stack for all CUDA kernels, as in PyTorch Profiler. (Caveat: this can cause inaccurately long kernel runtimes for CUDA event based timing)

4. For more arguments see [Nsight Systems User Guide](https://docs.nvidia.com/nsight-systems/UserGuide/index.html).


# Contribution Guide
Source: https://docs.sglang.io/docs/developer_guide/contribution_guide



Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.

## Install SGLang from Source

### Fork and clone the repository

**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.

```bash theme={null}
git clone https://github.com/<your_user_name>/sglang.git
```

### Build from source

Refer to [Install SGLang from Source](../get-started/install#method-2-from-source).

## Format code with pre-commit

We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:

```bash theme={null}
pip3 install pre-commit
pre-commit install
pre-commit run --all-files
```

* **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
* **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
* Documentation links and anchors are checked with Mintlify in CI. To run the same version locally, install it with `npm install -g mint@4.2.559`, then run `cd docs && mint broken-links --check-anchors --check-redirects`.
* The manual `lychee` pre-commit hook checks links in the repository-level `README.md`: `pre-commit run --hook-stage manual lychee --all-files`.

## Run and add unit tests

If you add a feature or fix a bug, add focused regression coverage when the test protects a concrete behavior, invariant, or bookkeeping contract.

### Unit tests (no server required)

Unit tests live under [`test/registered/unit/`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit), organized to mirror the `python/sglang/srt/` source tree. These tests validate component logic **without** launching a server or loading real model weights.
SGLang supports both Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework and [pytest](https://docs.pytest.org/). Registered CI tests must use `CustomTestCase` instead of raw `unittest.TestCase`, register themselves with `register_*_ci(...)`, and include a standard `__main__` entry point. CI discovers registrations with `test/run_suite.py` and executes each selected test file directly with fail-fast enabled.

**When to add a unit test:** If you modify a file under `python/sglang/srt/`, check whether a corresponding test exists in `test/registered/unit/` and add coverage for your changes. For example:

```text theme={null}
srt/mem_cache/radix_cache.py   →  unit/mem_cache/test_radix_cache_unit.py
srt/sampling/sampling_params.py →  unit/sampling/test_sampling_params.py
```

**Run tests locally:**

```bash theme={null}
# Closest to how CI executes an individual registered test file
python3 test/registered/unit/mem_cache/test_radix_cache_unit.py

# Run a registered CI suite
python3 test/run_suite.py --hw cpu --suite base-a-test-cpu

# Pytest remains useful for local directory-level discovery
pytest test/registered/unit/mem_cache/ -v
```

**Run with coverage:**

```bash Command theme={null}
pytest test/registered/unit/ --cov --cov-config=.coveragerc -v
```

For conventions on CI registration, test structure, and examples, see [`test/registered/unit/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit/README.md).

### E2E tests (server required)

For tests that require launching a server, refer to [`test/registered/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/README.md) for guidance on where to place your test.

For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).

## Write documentation

Documentation is a good way for new contributors to learn the SGLang codebase. See [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md) for the current Mintlify setup and validation commands.

## Test the accuracy

If your code changes model output, run an accuracy evaluation appropriate for the affected model and feature. For example, after launching a server, run the unified GSM8K evaluator:

```bash theme={null}
# Launch a server
sglang serve --model-path Qwen/Qwen3-8B

# Evaluate
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.

GSM8K is too easy for many state-of-the-art models, so use a more challenging or feature-specific evaluation when appropriate. See [Evaluating New Models with SGLang](/docs/developer_guide/evaluating_new_models) for current MMLU, GSM8K, HellaSwag, GPQA, HumanEval, and MMMU commands. Additional examples are available in [test/manual/eval](https://github.com/sgl-project/sglang/tree/main/test/manual/eval).

## Benchmark the speed

Refer to [Benchmark and Profiling](./benchmark_and_profiling).

## Pull request titles

Use the repository-wide format checked by CI:

```text theme={null}
[Type] scope: title
[Type] [Hardware] scope: title
```

* **Type**: `Feat`, `Fix`, `Refactor`, `Perf`, `Doc`, `Test`, `CI`, or `Chore`.
* **Scope**: A required module or model name. Use `/` for submodules, such as `diffusion/scheduler`; there is no fixed scope list or letter-case requirement.
* **Hardware**: Optional, for hardware-specific changes only. Use a single uppercase tag, such as `AMD` or `NVIDIA`, except for the canonical spellings `Intel`, `Apple`, `Huawei`, and `Ascend`.
* **Title**: A short, clear description in the imperative mood. Use one space between fields and after the colon, with no leading or trailing whitespace.

For example:

```text theme={null}
[Fix] attention: Fix KV cache indexing
[Perf] diffusion/scheduler: Reduce scheduling overhead
[Perf] [AMD] diffusion/attention: Optimize decode performance
```

For reverts, keep GitHub's default `Revert "original title"` format. The PR-title check validates PR titles, not individual development commit messages.

## Requesting a review for merge

You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
Then your PR can be merged.

## How to Trigger CI Tests

We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
Users with permission are listed in the [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)

**PR authors** can use `/rerun-failed-ci` on their own PRs even if they are not listed in `CI_PERMISSIONS.json`. Selective reruns have additional rules because they execute PR code on self-hosted runners; see the permission table below.

For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:

* `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
* `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, skipped, cancelled, or timed out**.
* `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
* `/run-full-ci`: Baseline **and** extra CI. Short form of `/tag-and-rerun-ci extra` — applies both labels, then reruns everything at the current commit that didn't succeed.
* `/run-extra-ci`: Extra CI **only**. Applies both labels, then reruns just `PR Test Extra`, leaving baseline CI runs alone. Use it when baseline is already green and you only need the extra tier.
* `/rerun-test <test-spec> [<test-spec> ...]`: Reruns one or more specific tests directly. A spec may select a file, class, or method using `<file>::<TestClass>[.<test_method>]`. Multiple specs and file globs are supported. `--changed` (short form `-c`) adds every test file the PR itself adds or modifies (under `test/registered/` or the multimodal test directory), so a PR that touches several tests can rerun them all without listing them. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py`, `/rerun-test test_*backend*.py`, `/rerun-test --changed`, or `/rerun-test -c`.
* `/rerun-group <group> [<group> ...]`: Expands one or more registered test groups (for example, `/rerun-group rust-server`) and dispatches their tests through the same selective-rerun workflow. Groups are defined in [`scripts/ci/rerun_test_groups.json`](https://github.com/sgl-project/sglang/blob/main/scripts/ci/rerun_test_groups.json).
* `/run-cpu-ci`: Runs only the CPU unit suite (`base-a-test-cpu`) against the PR head, partitioned the same way as in PR CI, without the `run-ci` label or any GPU stage. For a small PR, `/run-cpu-ci` plus `/rerun-group <module>` covers the unit tests and the affected module's tests. Alias: `/rerun-cpu-ci`.

The rerun commands have the following permission rules:

| Command | PR author | Other users | Additional rule for fork PRs |
| - | - | - | - |
| `/rerun-failed-ci` | Always allowed on their own PR | Must have `can_rerun_failed_ci` in `CI_PERMISSIONS.json` | None |
| `/run-full-ci`, `/run-extra-ci` | Not auto-allowed -- these apply labels, which the PR-author allowance does not cover | Must have both `can_tag_run_ci_label` and `can_rerun_failed_ci` in `CI_PERMISSIONS.json` (every listed user has both) | None |
| `/rerun-test`, `/rerun-group`, `/run-cpu-ci` | No special allowance; judged as any other commenter | Must have `cooldown_interval_minutes: 0` in `CI_PERMISSIONS.json`, or `write`/`admin` repository permission | None -- the same rule applies wherever the PR comes from |

> [!WARNING]
>
> Selective reruns do not build or install a PR-local `sglang-kernel` wheel. If a PR changes `python/sglang/kernels/aot/` or code that depends on that AOT change, use the normal/full PR CI workflow. A `/rerun-test` or `/rerun-group` result may use the pinned released kernel and does not validate the combined change.

If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).

If a command isn't recognized, the handler reacts 😕 instead, and replies with the closest matching command when there is one. A comment with no reaction at all means the handler never saw it — check the command is spelled as listed above.

To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).

If you don’t have permission and you’re not the PR author, please ask maintainers to trigger CI for you.

### CI control labels

Four labels change how a PR's CI runs. Label before you trigger:
`max-concurrency` fixes the shard fan-out when the run is created.

| Label | Effect |
| - | - |
| `bypass-fail-fast` | A job failing no longer aborts its siblings; the run continues instead of stopping at the first root-cause failure. A failing `lint` still stops everything. |
| `parallel-stages` | Stages stop waiting on each other. `base-a`, `base-b` and `base-c` dispatch together, the way a scheduled run does. |
| `max-concurrency` | A suite fans out to all of its shards at once rather than a third of them. |
| `highest-priority` | All three of the above, and the PR is also skipped by the batch-cancel workflow, sorted first in the active-runs report, and never closed as stale. |

Each spends extra runner capacity, so they unblock a specific PR rather than
serve as a default.

### CI rate limits

Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.

Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:

```yaml Config theme={null}
cool-down-minutes:
  description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting"
  type: number
  default: 120
```

Users listed in [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.

## Code style guidance

* Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
* Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
* Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
  * A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible.
* Make functions as pure as possible. Avoid in-place modification of arguments.
* Prefer immutable data and compute configuration-derived values once during initialization when their inputs cannot change.
* Keep functions under roughly 100 lines and make orchestration functions read like high-level pseudocode by extracting details into focused helpers.
* Keep files concise. If a file exceeds 2,000 lines of code, split it into cohesive smaller modules.
* Avoid adding mixins; prefer composition or plain functions. Use `msgspec.Struct` for new data containers instead of `dataclasses.dataclass` or `attrs`.
* Prefer keyword arguments for calls with two or more arguments, and pass callees the specific values they need instead of a large state-holding object.
* In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file.
* Keep tests run fast.
  * If a single test file runs longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
  * If a single job in a GitHub Actions workflow runs longer than 30 minutes, split it into smaller jobs or steps.
  * Reuse server launches across test methods in E2E test files.
* Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python's [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead.
* When supporting new hardware or features, follow these guidelines:
  * Do not drastically change existing code.
  * Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`).
  * If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.

## How to update kernels in SGLang

Choose the implementation and release path based on the kernel's dependencies. For a lightweight kernel that does not depend on CUTLASS or another large C++ project, prefer the in-tree JIT kernel path. Use the AOT `sglang-kernel` path for heavyweight kernels, large C++ dependencies, or operations that need wheel packaging and Torch operator registration. FlashInfer-based kernels are an exception and may still use the JIT path. SGLang also consumes separately released custom builds of DeepGEMM and DeepEP; update those in their source repositories as described below.

### Update sglang-kernel

The `sglang-kernel` distribution (formerly `sgl-kernel`) is a separate Python package, but its source now lives in this repository under `python/sglang/kernels/aot/`. Normal PR CI builds and installs a PR-local `sglang-kernel` wheel, so it can test an AOT kernel and its caller together. This does not establish compatibility after merge: installed SGLang and scheduled CI use the released version pinned in `python/pyproject.toml`. If a caller unconditionally requires a new operator or a changed kernel contract, first merge and release the kernel, update the pinned `sglang-kernel` version, and then land the caller. A combined PR is acceptable only when every supported path remains correct with the pinned wheel—for example, when the caller checks operator availability and retains a semantically equivalent fallback until the released wheel includes the change. Selective `/rerun-test` and `/rerun-group` workflows do not install PR-local wheels and cannot validate the combined AOT path.

For an AOT kernel change:

1. Implement the kernel under `python/sglang/kernels/aot/csrc/` and update its declaration, Torch registration, and CMake source list.
2. Expose the Python API under `python/sglang/kernels/aot/python/sgl_kernel/`.
3. Add correctness tests under `python/sglang/kernels/aot/tests/` and a benchmark under `python/sglang/kernels/aot/benchmark/` when applicable.
4. Build and test from `python/sglang/kernels/aot/` following its [README](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/aot/README.md). Do not bump the pinned `sglang-kernel` version in `python/pyproject.toml` merely to test a PR-local kernel change.

### Update sgl-deep-gemm

Develop SGLang's customized DeepGEMM package on the [`dev` branch of `sgl-project/DeepGEMM`](https://github.com/sgl-project/DeepGEMM/tree/dev). Rebase incoming changes onto `dev`, place new or modified package tests under `sgl_deep_gemm/tests/`, and follow the [`sgl-deep-gemm` README](https://github.com/sgl-project/DeepGEMM/blob/dev/sgl_deep_gemm/README.md) to build and install a local wheel. After the implementation is merged, ask the SGLang team to run the [sgl-deep-gemm release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepgemm.yml) with the new version, CUDA target, and DeepGEMM branch. Once all required wheels are published and verified, update the `sgl-deep-gemm` pin in `python/pyproject.toml` before landing SGLang code that requires the new or changed behavior.

### Update sgl-deep-ep

Develop `sgl-deep-ep` in [`sgl-project/DeepEP`](https://github.com/sgl-project/DeepEP). Use `sgl-deepep` for both implementation and packaging changes for CUDA 13 on x86\_64 or aarch64. The [`sgl-deep-ep` README](https://github.com/sgl-project/DeepEP/blob/sgl-deepep/sgl_deep_ep/README.md) describes the platform prerequisites and release matrix.

To validate locally, check out `sgl-deepep` (or your development branch containing both the implementation and `sgl_deep_ep`) as `DeepEP-source`. Install the required build dependencies first. The following example builds a wheel for the host architecture, installs that exact wheel, and verifies that its guarded package import succeeds:

```bash theme={null}
DEEPEP_OUTPUT_DIR="$(mktemp -d)"
bash DeepEP-source/sgl_deep_ep/build_sgl_deep_ep.sh \
  "${DEEPEP_OUTPUT_DIR}" \
  13.0 \
  "$(uname -m)"
python3 -m pip install --force-reinstall --no-deps \
  "${DEEPEP_OUTPUT_DIR}"/sgl_deep_ep-*.whl
python3 -c "import deep_ep; print(deep_ep.__file__)"
```

The import check validates packaging and binary loading, but not communication correctness. On a configured multi-GPU host, also run:

```bash theme={null}
python3 DeepEP-source/tests/elastic/test_ep.py --num-processes 8
```

Adjust `--num-processes` to the available GPUs and run the internode or low-latency tests when those transports changed. After local validation, ask the SGLang team to run the [sgl-deep-ep release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepep.yml) with the new version and DeepEP ref. The workflow builds the implementation and packaging from that single ref (default: `sgl-deepep`). After verifying the published wheels for the supported Python versions and architectures, update the `sgl-deep-ep` pin in `python/pyproject.toml` before landing dependent SGLang changes.

## Tips for newcomers

If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase.

Also check out the following materials as startup guide:

* [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang.
* [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
* [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance.

If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).

Thank you for your interest in SGLang. Happy coding!


# Development Guide Using Docker
Source: https://docs.sglang.io/docs/developer_guide/development_guide_using_docker



The `lmsysorg/sglang:dev` image is a nightly build from `main` with additional development tools and dependencies for modifying and testing SGLang. For running models without a development environment, use the stable or nightly images described in the [Installation guide](/docs/get-started/install).

## Setup VSCode on a Remote Host

(Optional - you can skip this step if you plan to run sglang dev container locally)

1. In the remote host, download `code` from [VSCode](https://code.visualstudio.com/download) and run `code tunnel` in a shell.

Example

```bash Command theme={null}
wget https://vscode.download.prss.microsoft.com/dbazure/download/stable/fabdb6a30b49f79a7aba0f2ad9df9b399473380f/vscode_cli_alpine_x64_cli.tar.gz
tar xf vscode_cli_alpine_x64_cli.tar.gz

# https://code.visualstudio.com/docs/remote/tunnels
./code tunnel
```

2. In your local machine, press F1 in VSCode and choose "Remote Tunnels: Connect to Tunnel".

## Setup Docker Container

### Option 1. Use the default dev container automatically from VSCode

There is a `.devcontainer` folder in the sglang repository root folder to allow VSCode to automatically start up within dev container. You can read more about this VSCode extension in VSCode official document [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).

![VSCode Dev Container Architecture](https://github.com/user-attachments/assets/6a245da8-2d4d-4ea8-8db1-5a05b3a66f6d)

*Figure 1: Diagram from VSCode official documentation [Developing inside a Container](https://code.visualstudio.com/docs/devcontainers/containers).*

To enable this, you only need to:

1. Start Visual Studio Code and install [VSCode dev container extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
2. Press F1, type and choose "Dev Container: Open Folder in Container.
3. Input the `sglang` local repo path in your machine and press enter.

The first time you open it in dev container might take longer due to docker pull and build. Once it's successful, you should set on your status bar at the bottom left displaying that you are in a dev container:

![VSCode Dev Container Status Bar](https://github.com/user-attachments/assets/650bba0b-c023-455f-91f9-ab357340106b)

Now when you run `sglang.launch_server` in the VSCode terminal or start debugging using F5, sglang server will be started in the dev container with all your local changes applied automatically:

![SGLang Server Running in Dev Container](https://github.com/user-attachments/assets/748c85ba-7f8c-465e-8599-2bf7a8dde895)

### Option 2. Start up containers manually (advanced)

The following startup command is an example for internal development by the SGLang team. You can **modify or add directory mappings as needed**, especially for model weight downloads, to prevent repeated downloads by different Docker containers.

❗️ **Note on RDMA**

1. `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them but keeping them there does not harm. Thus, we enable these two flags by default in the commands below.
2. You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.

```bash Command theme={null}
# Change the name to yours
docker run -itd --shm-size 32g --gpus all -v <volumes-to-mount> --ipc=host --network=host --privileged --name sglang_dev lmsysorg/sglang:dev /bin/zsh
docker exec -it sglang_dev /bin/zsh
```

Some useful volumes to mount are:

1. **Huggingface model cache**: mounting model cache can avoid re-download every time docker restarts. Default location on Linux is `~/.cache/huggingface/`.
2. **SGLang repository**: code changes in the SGLang local repository will be automatically synced to the .devcontainer.

Example 1: Monting local cache folder `/opt/dlami/nvme/.cache` but not the SGLang repo. Use this when you prefer to manually transfer local code changes to the devcontainer.

```bash Command theme={null}
docker run -itd --shm-size 32g --gpus all -v /opt/dlami/nvme/.cache:/root/.cache --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
docker exec -it sglang_zhyncs /bin/zsh
```

Example 2: Mounting both HuggingFace cache and local SGLang repo. Local code changes are automatically synced to the devcontainer as the SGLang is installed in editable mode in the dev image.

```bash Command theme={null}
docker run -itd --shm-size 32g --gpus all -v $HOME/.cache/huggingface/:/root/.cache/huggingface -v $HOME/src/sglang:/sgl-workspace/sglang --ipc=host --network=host --privileged --name sglang_zhyncs lmsysorg/sglang:dev /bin/zsh
docker exec -it sglang_zhyncs /bin/zsh
```

## Debug SGLang with VSCode Debugger

1. (Create if not exist) open `launch.json` in VSCode.

2. Add the following config and save. Please note that you can edit the script as needed to apply different parameters or debug a different program (e.g. benchmark script).
   ```JSON Config theme={null}
     {
        "version": "0.2.0",
        "configurations": [
            {
                "name": "Python Debugger: launch_server",
                "type": "debugpy",
                "request": "launch",
                "module": "sglang.launch_server",
                "console": "integratedTerminal",
                "args": [
                    "--model-path", "meta-llama/Llama-3.2-1B",
                    "--host", "0.0.0.0",
                    "--port", "30000",
                    "--trust-remote-code",
                ],
                "justMyCode": false
            }
        ]
    }
   ```

3. Press "F5" to start. VSCode debugger will ensure that the program will pause at the breakpoints even if the program is running at remote SSH/Tunnel host + dev container.

## Profile

```bash Command theme={null}
# Change batch size, input, output and add `disable-cuda-graph` (for easier analysis)
# e.g. DeepSeek V3
nsys profile -o deepseek_v3 python3 -m sglang.benchmark.one_batch --batch-size 1 --input 128 --output 256 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --tp 8 --disable-cuda-graph
```

## Evaluation

```bash Command theme={null}
# GSM8K via sgl-eval: all 1319 test questions
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1319 \
  --num-threads 2000
```


# Development Guide for JIT Kernels
Source: https://docs.sglang.io/docs/developer_guide/development_jit_kernel_guide



## Environment Setup

We strongly recommend using `clangd` as the language server for JIT kernel development.
For Ubuntu/Debian, you can download clangd from [apt.llvm.org](https://apt.llvm.org/).
If you are using VS Code, we recommend installing the `clangd` extension for better IDE integration.

All JIT-related files are located in `python/sglang/kernels/jit`.
Unlike `sgl-kernel`, which compiles CUDA/C++ binaries ahead of time (AOT), just-in-time (JIT) kernels are compiled at runtime.
Consequently, a static `compile_commands.json` cannot be generated.
To enable code completion with `clangd`, run `python -m sglang.kernels.jit` to generate a `.clangd` configuration file in your current directory.
After generating the file, restart the clangd language server. It should now recognize all JIT kernel files.

## Code Structure

### C++ Implementation

C++ source code is located in `python/sglang/kernels/jit/csrc`.
Reusable functions should be placed in `python/sglang/kernels/jit/include`.

JIT C++ lives in `namespace sglang`: open it after the include block and close it at the
end of the file, with the device kernels and the host wrapper both inside.
The shared `host::` and `device::` helpers are nested in it as well, so they resolve unqualified
and need no `sglang::` prefix.

We use [tvm-ffi](https://github.com/apache/tvm-ffi) for efficient foreign language bindings.
Refer to the [documentation](https://tvm.apache.org/ffi/) for advanced usage, such as exporting C++ objects.
Typically, `tvm::ffi::TensorView` is sufficient for passing PyTorch Tensors from Python.

### Python Interface

Python interfaces are defined in `python/sglang/kernels/jit`.
The `load_jit` utility function in `python/sglang/kernels/jit/utils/compile.py` loads and returns the compiled module.
To export a C++ function (e.g., `cpp_func`), pass `cuda_wrappers=[("func", "cpp_func")]` to `load_jit`.
The function can then be called in Python as `module.func`.
`load_jit` emits the export wrapper inside `namespace sglang`, so write `cpp_func` without a
`sglang::` prefix.

For caching compiled modules, prefer `sglang.kernels.jit.utils.cache_once` over `functools.lru_cache`.
`functools.lru_cache` is not compatible with `torch.compile`.

### C++ Utilities

The following C++ utilities are available:

#### Integer Range

Similar to PyTorch, we provide an `irange` function to represent an integer range.

```C++ Example theme={null}
#include <sgl_kernel/utils.h>

void test() {
  for (auto i : host::irange(100)) { // [0, 100)
    // do something
  }
  for (auto i : host::irange(0, 100)) { // [0, 100)
    // do something
  }
}

```

#### Runtime Checking

`CHECK_HOST` is the preferred runtime check: stream-style, and zero overhead when the
check passes — the message expressions are only evaluated on failure.
`RuntimeCheck` is the function-style alternative; note its message arguments are always
evaluated, even when the check passes.
`RuntimeDeviceCheck` verifies the status of the last kernel launch, and `CHECK_CUDA`
is its stream-style equivalent for checking a `cudaError_t` with extra context.

```C++ Example theme={null}
#include <sgl_kernel/utils.h>
#include <sgl_kernel/utils.cuh>

void test() {
  CHECK_HOST(1 + 1 == 2) << 1 + 1 << " != " << 2;  // preferred
  host::RuntimeCheck(1 + 1 == 2, 1 + 1, " != ", 2);
  host::RuntimeDeviceCheck();
  // check the provided `cudaError_t`
  host::RuntimeDeviceCheck(cudaGetLastError());
  CHECK_CUDA(cudaGetLastError()) << "after my_kernel launch";
}

```

#### Tensor Checking

`TensorMatcher` provides a readable way to validate and extract tensor shape information.

```cpp Example theme={null}
#include <sgl_kernel/tensor.h>

void test(const tvm::ffi::TensorView k_cache, const tvm::ffi::TensorView v_cache) {
  using namespace host;

  auto D = SymbolicSize{"D"};  // cache dimension
  auto N = SymbolicSize{"N"};  // kvcache stride
  auto dtype = SymbolicDType{};
  auto device = SymbolicDevice{};

  TensorMatcher({-1, D})  //
      .with_strides({N, 1})
      .with_dtype<int32_t, int64_t>(dtype)
      .with_device<kDLCUDA, kDLCPU>(device)
      .verify(k_cache)
      .verify(v_cache);
}
```

Configure the `TensorMatcher` with expected stride, dtype, and device properties before verification.

* If `with_strides` is omitted, the tensor is expected to be contiguous.
* Template arguments in `with_dtype` restrict the allowed data types.
* Template arguments in `with_device` restrict the allowed devices.
* Values passed to `with_xxx` methods enforce equality checks.
* Passing `-1` for size or stride allows matching any value.

A `Symbolic` variable must resolve to the same value across all verifications.
Use `.unwrap()` to retrieve the matched value after verification.

> Note: `TensorMatcher` is a temporary expression and should not be stored in a variable.

> Tip: Add `//` at the end of the `TensorMatcher` chain to enforce proper indentation.

#### Kernel Launching

`LaunchKernel::resolve_device` retrieves the current `cudaStream` from PyTorch.
Kernels can also be launched directly using `LaunchKernel`.

```cpp Example theme={null}
#include <sgl_kernel/utils.cuh>

#include <dlpack/dlpack.h>

__global__ void kernel() {}

void test() {
  const auto num_blocks = 1;
  const auto num_threads = 32;
  const auto dynamic_smem = 0;

  DLDevice dev;  // suppose this is initialized properly
  host::LaunchKernel(num_blocks, num_threads, dev)(kernel);

  cudaStream_t stream = host::LaunchKernel::resolve_device(dev);
  host::LaunchKernel(num_blocks, num_threads, stream, dynamic_smem)(kernel);
}

```

## Add new kernels

This section walks through a complete, end-to-end example of adding a new JIT kernel to the system.
We use a simple add\_constant kernel as a running example, which adds a constant integer value to every element of an input tensor.

Conceptually, the Python interface looks like this:

```python Example theme={null}
def add_constant(src: torch.Tensor, c: int):
    return src + c
```

### STEP 1: Write the C++ kernel

Write your CUDA kernel in [kernels/jit/csrc/elementwise/add\_constant.cuh](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/jit/csrc/elementwise/add_constant.cuh). For demonstration purposes, we pass the constant value as a template parameter.

```cpp Example theme={null}
#include <sgl_kernel/tensor.h>   // For TensorMatcher, SymbolicSize, SymbolicDevice
#include <sgl_kernel/utils.cuh>  // For LaunchKernel
#include <sgl_kernel/utils.h>    // For div_ceil, CHECK_HOST

#include <dlpack/dlpack.h>
#include <tvm/ffi/container/tensor.h>

#include <cstddef>
#include <cstdint>

namespace sglang {

template <int32_t kConstant>
__global__ void add_constant_kernel(int32_t* dst, const int32_t* src, size_t length) {
  size_t idx = blockIdx.x * blockDim.x + threadIdx.x;
  if (idx < length) {
    dst[idx] = src[idx] + kConstant;
  }
}

constexpr size_t kBlockSize = 256;

// You can also use struct with static method as an alternative
template <int32_t kConstant>
void add_constant(tvm::ffi::TensorView dst, tvm::ffi::TensorView src) {
  using namespace host;

  // 1. Validate input tensors
  SymbolicSize N = {"num_elements"};
  SymbolicDevice device_;
  TensorMatcher({N})                  // 1D tensor, must be contiguous
      .with_dtype<int32_t>()          // must be int32
      .with_device<kDLCUDA>(device_)  // must be on CUDA device
      .verify(dst)                    // check tensor dst
      .verify(src);                   // check tensor src

  // 2. Extract required parameters, prepare for kernel launch
  const size_t num_elements = N.unwrap();
  const size_t grid_size = div_ceil(num_elements, kBlockSize);
  const DLDevice device = device_.unwrap();
  // some extra runtime checks using CHECK_HOST
  CHECK_HOST(num_elements > 0) << "We only support non-empty tensors, got num_elements = " << num_elements;

  // 3. Launch the kernel. Error code will be automatically checked.
  LaunchKernel(grid_size, kBlockSize, device /*, dynamic_smem*/)(
      // kernel function
      add_constant_kernel<kConstant>,
      // kernel arguments
      static_cast<int32_t*>(dst.data_ptr()),
      static_cast<int32_t*>(src.data_ptr()),
      num_elements);
}

}  // namespace sglang

```

### STEP 2: Create Python Interfaces

Next, expose the kernel through a Python wrapper.
Create a new file at [kernels/ops/elementwise/add\_constant.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/ops/elementwise/add_constant.py) and expose the needed interfaces.

```python Example theme={null}
from __future__ import annotations
from typing import TYPE_CHECKING

import torch

from sglang.kernels.jit.utils import cache_once, load_jit, make_cpp_args

if TYPE_CHECKING:
    from tvm_ffi.module import Module


@cache_once
def _jit_add_constant_module(constant: int) -> Module:
    args = make_cpp_args(constant)  # pass all the template argument
    return load_jit(
        "add_constant",
        *args,
        cuda_files=["elementwise/add_constant.cuh"],
        cuda_wrappers=[("add_constant", f"add_constant<{args}>")],
    )


def add_constant(src: torch.Tensor, constant: int) -> torch.Tensor:
    if not src.is_cuda:
        raise RuntimeError("src must be a CUDA tensor")
    if src.dtype != torch.int32:
        raise RuntimeError(f"Unsupported dtype {src.dtype}. Supported: int32")
    dst = torch.empty_like(src)
    module = _jit_add_constant_module(constant)
    module.add_constant(dst, src)
    return dst

```

Keep the Python wrapper thin, but still validate the basic invariants such as device and dtype before dispatch. In the current JIT/FFI path, invalid tensors are not always rejected safely before launch.

### STEP 3: Use your kernel

Finally, import and use the kernel like a regular Python function:

```python Example theme={null}
from sglang.kernels.ops.elementwise.add_constant import add_constant
```

For a complete, runnable example, refer to [test\_add\_constant.py](https://github.com/sgl-project/sglang/blob/main/test/registered/kernels/ops/elementwise/test_add_constant.py).

## C++ Include Library Reference

The JIT kernel framework provides a set of reusable C++ headers in
`python/sglang/kernels/jit/include/sgl_kernel/`. Each header is designed
to be lightweight and self-contained. Below is a summary of each header
and its key APIs.

### Core Utilities

<table>
  <thead>
    <tr>
      <th>Header</th>
      <th>Namespace</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`utils.h`</td>
      <td>`host`</td>
      <td>Host-side essentials: `RuntimeCheck`, `CHECK_HOST(cond) << ...`, `Panic`, `div_ceil`, `irange`</td>
    </tr>

    <tr>
      <td>`utils.cuh`</td>
      <td>`device` / `host`</td>
      <td>Type aliases (`fp16_t`, `bf16_t`, ...), `SGL_DEVICE` macro, PDL helpers, `LaunchKernel`, `RuntimeDeviceCheck`, `CHECK_CUDA(expr) << ...`</td>
    </tr>

    <tr>
      <td>`source_location.h`</td>
      <td>(global)</td>
      <td>Portable `std::source_location` wrapper for error reporting</td>
    </tr>

    <tr>
      <td>`runtime.cuh`</td>
      <td>`host::runtime`</td>
      <td>CUDA runtime queries: `get_blocks_per_sm`, `get_sm_count`, `get_cc_major`, `get_runtime_version`, `get_available_dynamic_smem_per_block`</td>
    </tr>
  </tbody>
</table>

### Tensor Validation

<table>
  <thead>
    <tr>
      <th>Header</th>
      <th>Namespace</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`tensor.h`</td>
      <td>`host`</td>
      <td>`TensorMatcher`, `SymbolicSize`, `SymbolicDType`, `SymbolicDevice`</td>
    </tr>
  </tbody>
</table>

### Math & Type System

<table>
  <thead>
    <tr>
      <th>Header</th>
      <th>Namespace</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`math.cuh`</td>
      <td>`device::math`</td>
      <td>`max`, `min`, `abs`, `sqrt`, `rsqrt`, `exp`, `sin`, `cos`, constants</td>
    </tr>

    <tr>
      <td>`type.cuh`</td>
      <td>(global) / `device`</td>
      <td>`DTypeTrait<T>`, `packed_t<T>`, `device::cast<To>(from)`</td>
    </tr>
  </tbody>
</table>

### Memory Access

<table>
  <thead>
    <tr>
      <th>Header</th>
      <th>Namespace</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`vec.cuh`</td>
      <td>`device`</td>
      <td>`AlignedVector<T, N>` - vectorized load/store (up to 128-bit; 256-bit requires Blackwell GPUs)</td>
    </tr>

    <tr>
      <td>`tile.cuh`</td>
      <td>`device::tile`</td>
      <td>`Memory<T>` - cooperative tiled memory I/O (thread/warp/CTA)</td>
    </tr>
  </tbody>
</table>

### Parallel Primitives

<table>
  <thead>
    <tr>
      <th>Header</th>
      <th>Namespace</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`warp.cuh`</td>
      <td>`device::warp`</td>
      <td>`reduce<Op, kNumThreads, kInner>` (SUM/MAX/MIN, grouped or inter-group) and `reduce_sum` / `reduce_max` / `reduce_min` wrappers via `__shfl_xor_sync`</td>
    </tr>

    <tr>
      <td>`cta.cuh`</td>
      <td>`device::cta`</td>
      <td>`reduce_max` across warps via shared memory</td>
    </tr>

    <tr>
      <td>`atomic.cuh`</td>
      <td>`device::atomic`</td>
      <td>`max` - atomic float max (CUDA + ROCm fallback)</td>
    </tr>
  </tbody>
</table>

### Reusable Kernel Templates

<table>
  <thead>
    <tr>
      <th>Header</th>
      <th>Namespace</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`impl/norm.cuh`</td>
      <td>`host::norm` / `device::norm`</td>
      <td>RMSNorm building blocks (warp & CTA paths, `StorageType`)</td>
    </tr>
  </tbody>
</table>


# Evaluating New Models with SGLang
Source: https://docs.sglang.io/docs/developer_guide/evaluating_new_models



This document provides commands for evaluating models' accuracy and performance. Before open-sourcing new models, we strongly suggest running these commands to verify whether the score matches your internal benchmark results.

**For cross verification, please submit commands for installation, server launching, and benchmark running with all the scores and hardware requirements when open-sourcing your models.**

[Reference: MiniMax M2](https://github.com/sgl-project/sglang/pull/12129)

## Accuracy

### LLMs

SGLang provides built-in scripts to evaluate common benchmarks.

**MMLU**

```bash Command theme={null}
sgl-eval run mmlu \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 1000 \
  --max-tokens 8192
```

**GSM8K**

```bash Command theme={null}
sgl-eval run gsm8k \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 200
```

**HellaSwag**

```bash Command theme={null}
python benchmark/hellaswag/bench_sglang.py \
  --host 127.0.0.1 \
  --port 30000 \
  --num-questions 200 \
  --num-shots 20
```

**GPQA**

```bash Command theme={null}
sgl-eval run gpqa \
  --base-url http://127.0.0.1:30000/v1 \
  --num-examples 198 \
  --max-tokens 120000 \
  --n-repeats 8
```

> [!TIP]
>
> For reasoning models, add `--thinking` (DeepSeek-V3 / Kimi-K2 style `thinking` key) or `--chat-template-kwarg enable_thinking=true` (Qwen3 / GLM style). You may skip it if the model has forced thinking enabled.

**HumanEval**

```bash Command theme={null}
pip install human_eval

python -m sglang.test.run_eval \
  --eval-name humaneval \
  --num-examples 10 \
  --port 30000
```

### VLMs

**MMMU**

```bash Command theme={null}
python benchmark/mmmu/bench_sglang.py \
  --port 30000 \
  --concurrency 64
```

> [!TIP]
>
> You can set max tokens by passing `--extra-request-body '{"max_tokens": 4096}'`.

For models capable of processing video, we recommend extending the evaluation to include `VideoMME`, `MVBench`, and other relevant benchmarks.

## Performance

Performance benchmarks measure **Latency** (Time To First Token - TTFT) and **Throughput** (tokens/second).

### LLMs

**Latency-Sensitive Benchmark**

This simulates a scenario with low concurrency (e.g., single user) to measure latency.

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --host 0.0.0.0 \
  --port 30000 \
  --dataset-name random \
  --num-prompts 10 \
  --max-concurrency 1
```

**Throughput-Sensitive Benchmark**

This simulates a high-traffic scenario to measure maximum system throughput.

```bash Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --host 0.0.0.0 \
  --port 30000 \
  --dataset-name random \
  --num-prompts 1000 \
  --max-concurrency 100
```

**Single Batch Performance**

You can also benchmark the performance of processing a single batch offline.

```bash Command theme={null}
python -m sglang.bench_one_batch_server \
  --model <model-path> \
  --batch-size 8 \
  --input-len 1024 \
  --output-len 1024
```

You can run more granular benchmarks:

* **Low Concurrency**: `--num-prompts 10 --max-concurrency 1`
* **Medium Concurrency**: `--num-prompts 80 --max-concurrency 16`
* **High Concurrency**: `--num-prompts 500 --max-concurrency 100`

## Reporting Results

For each evaluation, please report:

1. **Metric Score**: Accuracy % (LLMs and VLMs); Latency (ms) and Throughput (tok/s) (LLMs only).
2. **Environment settings**: GPU type/count, SGLang commit hash.
3. **Launch configuration**: Model path, TP size, and any special flags.
4. **Evaluation parameters**: Number of shots, examples, max tokens.


# Layer boundaries
Source: https://docs.sglang.io/docs/developer_guide/layer_boundary

Declare stage inputs and outputs, bind communication and residual operations, and integrate model computation with layer boundaries.

The `sglang.srt.layers.layer_boundary` package manages the transition between compute stages. It completes outstanding reductions, moves token rows, applies the producer's residual update, and prepares the consumer's normalized or quantized input. The model continues to run attention, dense FFNs, and MoE computation.

Use this guide when integrating a model or changing a boundary implementation. The first part explains the concepts, the second shows how to integrate a model, and the last describes the internals you touch when you extend the package. It documents the current internal interfaces; they are not a versioned plugin API. For model development setup, see the [contribution guide](/docs/developer_guide/contribution_guide).

## Scope and ownership

A boundary connects a **producer output** to a **consumer input**. An attention-to-FFN transition and an FFN-to-attention transition use the same contracts. A model can also have consecutive mixers, consecutive FFNs, or a branch containing three or more stages. Here a mixer is a token-mixing stage, such as attention or the Mamba block in Nemotron-H.

| Responsibility | Owner |
| - | - |
| Attention, projections, activation functions, expert routing, EP dispatch/combine | Model compute modules |
| How a producer contributes to the residual | Producer's `ResidualUpdate` |
| How a consumer reads the residual | Consumer's `ResidualReadout` and norm |
| Supported row transitions and reduction ordering | Bound boundary paths |
| Actual outstanding work and residual tensors | Per-forward `ResidualStream` |
| Hardware-specific fused completion | Fusion providers |
| Retention of auxiliary tensors | Auxiliary-output collector |

EP all-to-all is an operation inside MoE computation, not an input preparation step. The boundary does describe the MoE's input/output rows and any output reduction that compute delegates to it.

## Core concepts

### Token layouts and sums

`Layout.sharded` describes which supported token axes partition a rank's rows:

* `ATTN_DP`: different attention data-parallel token sets.
* `ATTN_CP`: context-parallel (CP) token partitions for an active CP batch.
* `ATTN_TP`: token slices distributed across attention-TP ranks.

Size-one axes are omitted. Layout is not a `DTensor` placement or a general cross-mesh redistribution engine. It does not describe arbitrary hidden-dimension sharding. The actual row ordering, padding, and per-rank counts come from batch metadata and the paired movement operations.

A sum is described separately by `SumGroup`: attention TP, full TP, or the MoE output reduction group. Separating rows from sums lets the boundary express both an unreduced contribution and the rows on which its completion must land.

### Producer contracts and output state

A declaration states how its compute cooperates with the boundary's reduction decision:

| Contract or state | Meaning |
| - | - |
| `ProducerReduction.ALWAYS_PARTIAL` | Attention only (the `declare_attn()` default): compute always returns a contribution requiring a sum and publishes it with `finish()`; the next stage's input completes the sum. Build its output projection so that it never reduces (for example `reduce_results=False`) |
| `ProducerReduction.EXIT_SCOPED` | The exit completes the sum or carries it to the next stage's input: every FFN, and `declare_attn(reduction=ProducerReduction.EXIT_SCOPED)` for a mixer |
| `OutputContract.may_*` | Construction-time permission to leave the sum to a particular later step |
| `Contribution.owed` | Work this specific output still requires |
| `OwedOutput` | Model-facing handle preventing ordinary tensor use of that unfinished output |

`UnreducedOutput` is an internal adapter form. `ResidualStream.record()` converts it into an owned contribution and an opaque model handle. `DeferredFinalize` represents producer-specific work such as a deferred MoE finalize; its `complete()` supplies an unfused completion path. A handoff is consumed by the next layer's input or, at the end of the stack, by a final norm that accepts it.

### Residual stream

`ForwardBatch.residual_stream` carries the residual and the producer's actual contribution from one boundary to the next. A stream has three states:

| State | Stored data | Next preparation |
| - | - | - |
| Initial | Neither residual nor contribution | Initialize residual through the consumer's `init_residual` |
| Written | Residual, no pending contribution | Read input without applying another update |
| Pending | Contribution, optionally an existing residual | Complete owed work, apply the producer update, then read |

The model keeps the tensor or handle that each boundary call returns, and passes exactly that value to the next call. Rebinding a layer's output to a different object, or passing another microbatch's output, raises an error instead of applying the wrong pending work. The check compares object identity, so an in-place change to a complete tensor is not detected; do not modify layer outputs in place, and add extra contributions through `add_to_output()`.

Communication completion and residual update are distinct steps. The `residual.batch` helpers expose them:

| Helper | Effect |
| - | - |
| `complete_output(hidden_states, forward_batch)` | Finishes outstanding communication/finalize work and leaves the update pending |
| `fold(hidden_states, forward_batch)` | Completes a plain contribution and folds it into the residual, leaving a written stream |
| `set_written(hidden_states, forward_batch)` | Replaces the stream with a written one whose residual is `hidden_states`. Use it after compute that produced the full residual itself. It discards any pending contribution, so call `fold()` first when one may be pending |
| `take_output(hidden_states, forward_batch)` | Releases the stream at a terminal exit whose output is already written, such as Nemotron-H MTP after its internal fold and norm. It rejects a pending contribution |

Always retain the returned tensor or handle after each of these calls. A stack-entry boundary accepts a written stream and skips its `init_residual` operation.

### Residual update and input read

The producer owns `ResidualUpdate`; the consumer owns `ResidualReadout` and its norm. The actual update travels with the contribution. Bound paths use declared update capabilities to select valid ordering:

* A plain add can participate in compatible fused add+norm kernels or add the residual on one rank before summation.
* A nonlinear update cannot generally move before reduction. The post operation of manifold-constrained hyper-connections (MHC) is one such case.
* `applied_at_exit` requests an update at the producer exit. A deferred update must guarantee that its parameters and state outlive that producer.
* `ResidualReadout.reads_before_dp_gather` preserves reads that must execute on the source rows before a DP gather.
* A read that needs every row of the attention's, such as one that adds a contribution computed for every row, declares `reads_after_attn_tp_gather`. A producer that would leave its output on this rank's attention-TP slice then returns it to the attention's rows; Qwen4-exp's per-layer embedding read and Kimi K3's attention-residual bank declare it.

A read returns `(compute_input, residual)`. `update_and_read()` receives the producer's update together with the read, so an implementation can fuse them: the plain add+norm read does, and keeps its FP32 accumulation instead of normalizing a separately rounded snapshot. Fusion is not guaranteed. An implementation such as MHC may run the update and the read as separate steps, provided it preserves its existing rounding order.

The MHC adapter implements the expanded residual and coefficient state used by its integrated models. Do not assume every MHC model uses it: DeepSeek V4 retains its model-specific MHC path.

## Integrate a model

### Declare a standard attention and FFN layer

A `StageDeclaration` is the model-facing description. It contains read/update operations, dense versus sparse compute facts, reduction cooperation, and optional source declarations. It does not own a norm or an execution plan.

The following constructor fragment assumes the model has already initialized its compute modules and norms, and the worker's runtime parallel configuration is available. It follows the dense Qwen3 integration:

```python theme={null}
from sglang.srt.layers.layer_boundary import (
    append_stages,
    declare_attn,
    declare_ffn,
)

self.attn_boundary, self.ffn_boundary = append_stages(
    (declare_attn(), self.input_layernorm),
    (declare_ffn(), self.post_attention_layernorm),
)
```

`append_stages()` extends the open layer stack: the stack records the last stage appended, and the next append takes it as its producer, so a layer needs no description of its neighbours. `make_layers()` (and `make_pp_layers()`, which calls it) opens a stack around the layers it builds. A model that builds its layers another way, such as a hand-written `nn.ModuleList` or a NextN/multi-token-prediction (MTP) draft, opens the stack itself; appending outside an open stack raises:

```python theme={null}
from sglang.srt.layers.layer_boundary import layer_stack

with layer_stack():
    self.layers = nn.ModuleList(
        DecoderLayer(config, layer_id) for layer_id in range(config.num_hidden_layers)
    )
```

Every stage binds when the stack closes, once its producer and its consumer are known: the stage appended before it and the one appended after it. The last stage ends the model's layer stack and is marked terminal, unless a later layer declares a stage. A boundary's `declaration` is usable at once; when the stack closes, it is replaced by the declaration with its place in the stack (its `previous` and `terminal`), and the plan is bound. An error raised while binding carries a note naming the `append_stages()` call it came from. A terminal stage leaves no work for a following layer, except the final-norm finalize handoff described in [Enter and leave the layer stack](#enter-and-leave-the-layer-stack). A NextN/MTP draft module is its own stack: its layer enters the stack and ends it, even when it reuses the target's decoder class.

Pipeline parallelism splits one layer stack across ranks, so the first and last local stages have neighbours that another rank builds. The model gives `make_layers()` (or `make_pp_layers()`) its shared declaration function, `stage_facts`: given a global layer index, it returns the stages that layer declares, in order (an empty tuple for a layer that declares none), from the config, the layer index, construction options known before the layer is built, and the fixed parallel topology, never from built modules or provider state. The layer declares its own stages with the same function, so both ranks derive a boundary from one source; under pipeline parallelism `make_layers()` also checks each layer it builds against the function. When the stack closes, it asks the function for the stages next to it, nearest layer first, passing over a layer that declares none, and keeps only what binding reads of them (`facts_of()`: the capabilities a read, update and output transform declare, not the objects). A model whose layers declare stages must give the function to run under pipeline parallelism; no rank builds a layer another rank holds.

Each item may include a third element containing constructor options: an attention stage accepts `qkv_latent_func` (for example `{"qkv_latent_func": self.self_attn.prepare_qkv_latent}`) and `fusions`; an FFN stage accepts only `fusions` (see [Fusion providers](#fusion-providers-and-output-decisions)). An option a stage does not take is rejected when appended. Pass declarations fresh from `declare_attn()` / `declare_ffn()`: one taken from a bound boundary already has its place in a stack, and is rejected.

### Single-stage mixers and heterogeneous stacks

For heterogeneous layer stacks, append the actual stages in order; the framework does not require an alternating attention/FFN pattern, and a layer may append a single stage.

A single-stage `ProducerReduction.EXIT_SCOPED` mixer's consumer decides whether the mixer leaves its attention-TP sum to the next stage: always before an FFN, and per batch before another mixer. The stack binds the mixer only once its consumer is known, so the two stages always agree, across a pipeline boundary too: there the consumer is the next rank's first stage, which `make_layers()` reads from the neighbouring layer. At a pipeline boundary `to_pp()` completes the sum before the handoff, and the receiving FFN takes the completed path its entry binds beside the one for the declared sum.

`models/nemotron_h_utils.py` (used by `models/nemotron_h.py`) builds each Nemotron-H stage as `append_stages((decl, norm))`, with mixers declared as `declare_attn(reduction=ProducerReduction.EXIT_SCOPED, gathers_attn_tp_input=False)`.

### More than two stages and branches

There is no two-stage limit. For example, LongCat constructs a dense FFN, attention, and another dense FFN from an input already prepared for its MoE branch:

```python theme={null}
first_ffn, second_attn, second_ffn = append_stages(
    (declare_ffn(), first_ffn_norm),
    (declare_attn(), second_attn_norm),
    (declare_ffn(), second_ffn_norm),
    prepared_from=moe_boundary.declaration,
)
```

This fragment assumes the source boundary and the three norms already exist. `prepared_from` means the source has already performed its read. A branch is a side path that the model merges back explicitly: it neither extends the stack nor waits for a consumer, so the next append still follows the source. Enter through `branch_input(source, hidden_states, forward_batch)`, not `prepare()`, to move that input and fork the residual stream without normalizing twice. The first stage's norm is therefore never applied; LongCat passes the MoE stage's norm there.

The branch adapter also provides `branch_output()` to complete the sum a branch's contribution owes and place it on the branch's exit rows and `merge_branch()` to combine it with another branch's output. These methods encode row movement and residual ownership; the model still determines its compute schedule. Branch transport currently supports only ordinary token rows: construction rejects context-parallel, input-scattered, and sequence-parallel variants. See `python/sglang/srt/models/longcat_flash.py` for a complete integration.

### Execute a layer

The corresponding forward fragment keeps computation in the model:

```python theme={null}
hidden_states = self.attn_boundary.prepare(hidden_states, forward_batch)
if hidden_states.shape[0] != 0:
    hidden_states = self.self_attn(
        positions=positions,
        hidden_states=hidden_states,
        forward_batch=forward_batch,
    )
hidden_states = self.attn_boundary.finish(hidden_states, forward_batch)

hidden_states = self.ffn_boundary.prepare(hidden_states, forward_batch)
hidden_states = self.mlp(hidden_states, forward_batch=forward_batch)
hidden_states = self.ffn_boundary.finish(hidden_states, forward_batch)
```

This example uses Qwen3's compute member name `mlp`; boundary construction does not require renaming compute classes or checkpoint parameters.

`prepare()` returns input in the consumer read's format, which can include quantized tensors. An attention with an `ALWAYS_PARTIAL` output contract uses `finish()` to register its contribution, and an FFN or single-stage mixer uses `finish()` to complete or carry on the sum its output owes. An FFN whose boundary has fusions, which may take a MoE finalize handoff, runs its compute under `exit()` instead and calls the returned object's `finish()` once (`finish()` on the boundary rejects it); the exit context restores the previous runtime flags even when compute raises, so do not call `finish()` on a failed computation.

Compute never sums a stage's output: build every output projection, MLP and MoE block of a stage with `reduce_results=False`, and leave the sum the output owes to the boundary. The boundary completes it at the exit, or carries it to the next stage's input, where a fused all-reduce + add + norm may take it:

* `FfnExit` decides before compute runs whether the exit completes the sum, a reduce-scatter on the way back completes it, or the next input does. It publishes only `defer_moe_finalize` on `get_forward()`: a MoE may then return a `DeferredFinalize`.
* `MixerExit`: before an FFN, the mixer always leaves its attention-TP sum to the FFN's input, including under attention DP. Before another mixer the exit may carry the sum to that mixer's input, but never under attention DP; otherwise it completes the sum itself.

One exception: when an FFN's output sum is fused with a reduction its computation needs anyway, so the two cannot be separated without an extra collective, the computation completes it and the stage declares `declare_ffn(output_complete=True)`; the exit then owes no sum. Kimi K3's latent MoE reduces its latent before the latent norm and completes the shared experts' sum beside it.

A component every TP rank computes in full (a shared expert replicated with `tp_size=1`, LoRA-B on a partial sum) is added on TP rank 0 only, so that the one sum counts it once. Model loading fails if a module under a stage boundary is built with `reduce_results=True`. Returning a handoff while `defer_moe_finalize` is false fails in `finish()`.

A complete tensor after `finish()` can still have a pending **residual update**. An incomplete reduction is exposed to the model as an opaque `OwedOutput`; pass it to the next boundary or a supported access method. Do not inspect it as a tensor, slice it, or unwrap its contribution in model code.

### Enter and leave the layer stack

Initialize a fresh stream once on the embedding path, and leave the stack through the final norm:

```python theme={null}
from sglang.srt.layers.layer_boundary.residual import batch as residual_batch

residual_batch.start(forward_batch)
hidden_states = self.embed_tokens(input_ids)
for layer in self.layers:
    hidden_states = layer(positions, hidden_states, forward_batch)
hidden_states = residual_batch.final_norm(
    hidden_states, forward_batch, self.norm, skip_empty=True
)
```

`skip_empty=True` still completes owed work on an empty batch, which may be a collective that other ranks join, and skips only the norm kernel. `final_norm()` and `to_pp()` release the batch's stream after they hand on its output.

When the final read is more than a norm, pass it to `make_layers(final_read=...)` (or `layer_stack(final_read=...)`): it is then the last stage's consumer. Its `attn_tp_gather` gathers the rows the last FFN leaves on this rank's attention-TP slice, and with `reads_attn_tp_slices` it reads that slice itself and the last FFN leaves its output there. Kimi K3's `AttnBankOutputRead` is such a read.

This is a non-pipeline stack fragment. Pipeline reception uses the first boundary's `from_pp(tensors, forward_batch)` to reconstruct a stream; the sender uses `residual_batch.to_pp()`. A stream the producer already wrote travels as `hidden_states` alone and arrives written: MHC's expanded streams, and the output of an FFN on its own rows (its attention-TP slice, or an unpadded batch's rows), which writes it at its exit before a pipeline handoff. The receiver knows this from the producer's declaration, not from which tensors arrive, since a captured CUDA graph's input buffers hold every key. `to_pp()` completes every sum the output owes before transport, a declared one too: the pipeline sends each tensor as one slice per attention-TP rank and gathers the slices back on the next rank, which reassembles only a tensor every attention-TP rank holds alike. The receiving stage's entry then takes the path for a sum already completed, which it binds beside the one for a sum its own producer leaves.

A terminal FFN normally completes its output before the final norm. The exception is a MoE finalize handoff. When the stage's fusion provider can defer the finalize at the terminal layer (for CuteDSL, `install_cutedsl_fusion(..., terminal_finalize=True)`), the exit may leave a `DeferredFinalize` for the final norm. Pass the handle straight to `residual_batch.final_norm(hidden_states, forward_batch, self.norm, finalize_norm=service)`, where `service` is the object `install_cutedsl_fusion()` returned; its `finalize(handoff, residual, gamma)` fuses finalize, all-reduce, residual add, and norm. The fused path reads `layernorm.gemma_weight`, so it currently expects a `GemmaRMSNorm` final norm. Without `finalize_norm`, `final_norm()` first completes the handoff unfused. With it, `finalize()` also returns the updated residual, which `capture` receives as a borrowed value. Calling `complete_output()` first runs the producer's unfused finalize, and `snapshot()` raises on a handoff. See `python/sglang/srt/models/qwen3_5.py`.

`model_forward_stages()` in `python/sglang/srt/batch_overlap/two_batch_overlap.py` gives each TBO microbatch its own stream: it splits with `ResidualStream.from_handoff()` and merges the results afterwards. Do not share one mutable stream between interleaved microbatches or hold an `exit()` scope open across a TBO yield point.

In the operation-scheduled TBO path, the FFN runs without an `exit()` scope and returns its partial output. `boundary.complete_now(hidden_states, forward_batch)` completes the sum it owes, applies any output transform, moves that output back to the layer's exit rows (a plain scatter under attention DP), and leaves nothing owed.

### Auxiliary capture and extra contributions

Choose the access method according to what the consumer needs:

| Operation | Effect on main output |
| - | - |
| `residual_batch.snapshot()` | Produces an owned plain residual snapshot; a sum can run on a copy without consuming the main contribution |
| Attention `boundary.prepare(..., capture=collector.capture)` | Completes the main output's owed communication (unfused; the add+norm kernel still runs), then captures the updated residual this read produces, moved back to the producer's rows. At stack entry it captures the input before `init_residual`; with `post_residual_addition` it captures output plus residual before the addition. FFN `prepare()` does not accept it |
| Attention `boundary.prepare(..., capture_gathered=collector)` | Captures the same updated residual on the attention's compute-input rows, after its input gather |
| `boundary.capture_output()` | Preserves its explicit capture convention: completes dynamic work on the main contribution, but snapshots a statically declared sum on a copy; returns both updated handle and snapshot |
| `residual_batch.add_to_output(..., extra)` | Completes the contribution before adding extra once, keeping the residual update pending |
| `residual_batch.final_norm(..., capture=collector.capture)` | Final norm and capture of its updated residual, including a fused finalize handoff |

Snapshots currently require a plain residual update. Producer-specific finalize handoffs need an explicit main-output capture adapter.

A separate BF16 snapshot followed by norm can change results relative to a fused add+norm kernel's FP32 accumulation. Capture from the existing update/read operation when that ordering matters.

A capture callback has the signature `capture(value, *, owned=False)`, as `AuxHiddenStateList.capture` does. With `owned=False` the value may alias the mutable residual or a reusable communication buffer, and the collector must copy it before retaining it. The boundary passes `owned=True` only for storage nothing else will write: a capture move that gathers rows (`EntryPath.capture_move_allocates`), a residual sum computed only for the capture, or a residual that the following local stage's input path certifies it leaves untouched. `append_stages()` copies that certificate into the producer's entry at construction; no plan is consulted during forward. `residual_batch.final_norm(..., capture=...)` always passes a borrowed value.

### Choose a reduction policy

The `--boundary-reduction` option controls optional FFN output transport choices for decoders built with these boundaries; other models ignore it:

| Value | Permitted optional path |
| - | - |
| `ar` | All-reduce path |
| `rs` | Fixed-size reduce-scatter where eligible |
| `rsv` | Variable-size attention-DP reduce-scatter (RSv) where eligible |
| `rs+rsv` | Both scatter choices where eligible, with RSv tried first |
| `auto` | Default. Resolves a per-architecture policy during server-argument processing; a separate draft checkpoint resolves its own `speculative_boundary_reduction`. See [FFN boundary reduction](/docs/advanced_features/server_arguments#ffn-boundary-reduction) |

Required attention and MoE collectives are unaffected. These are permissions, not commands to force unsupported collectives; an ineligible optional scatter falls back to the all-reduce path. Shape, topology, producer behavior, and required token movement still constrain the selected path. The resolved policy must be available before model construction. Model defaults are defined in `python/sglang/srt/arg_groups/boundary_reduction.py`. Reduction fusion has its own backend enablement and eligibility checks; selecting `ar` alone does not disable all-reduce fusion.

### Integration checklist and common errors

Before you open a pull request for a new model, check each item:

* Every decoder layer appends its stages inside a layer stack: `make_layers()`, or `layer_stack()` around any other construction, including a NextN/MTP draft's own layer.
* The model gives `make_layers()` its shared declaration function (`stage_facts`), and each layer declares its stages with it: another pipeline rank binds the stages next to its own from that function alone.
* The model forward calls `residual_batch.start()` before the first layer, and leaves the stack through `residual_batch.final_norm()`, `to_pp()`, or `take_output()`.
* Pipeline reception calls `from_pp(tensors, forward_batch)` on the first local boundary.
* Split prefill starts the stream only in the segment that runs the first layer and completes it only in the final segment.
* Extra contributions such as deepstack embeddings go through `residual_batch.add_to_output()`, never an in-place add on the layer output.
* Auxiliary capture uses `prepare(..., capture=...)` or `final_norm(..., capture=...)` with a callback that honours `owned`.

Common errors and their usual cause:

| Error | Usual cause |
| - | - |
| `start the layer stack before entering a stage` | The forward path, a split-prefill segment, or an MTP head skipped `residual_batch.start()` |
| `output does not belong to this residual stream; change layer outputs through boundary accessors` | The model replaced a layer's output object (for example rebinding it to a new tensor) or passed another microbatch's output. This check compares object identity, so it does not catch in-place changes; the interface still forbids modifying layer outputs in place |
| `write the residual update before taking the final output` | `take_output()` ran while a contribution was still pending; fold or norm it first |
| `no stage boundary path for the active <VARIANT> batch` | The active batch variant (CP, input-scattered, sequence parallel, unpadded) is not supported by this stage; add a server-argument check that rejects the configuration |
| `append_stages needs an open layer stack` | The layers are built outside `make_layers()`; wrap their construction in `layer_stack()` |
| `a prepared branch must enter through branch_input, not prepare` | A `prepared_from` stage was entered with `prepare()` |
| `snapshot requires a plain residual update` | A snapshot was taken on a nonlinear (for example MHC) update |

## Boundary internals

### Architecture

```mermaid theme={null}
flowchart TD
    D["Stage<br/>declarations"] --> C["Connections<br/>per batch<br/>variant"]
    C --> P["StagePlan<br/>entry and<br/>exit paths"]
    P --> B["StageBoundary"]
    F["ForwardBatch<br/>runtime facts"] --> B
    S["ResidualStream<br/>residual and<br/>pending work"] <--> B
    B --> I["prepare()<br/>complete,<br/>update, read,<br/>move rows"]
    I -->|"FFN or mixer"| X["exit()<br/>choose<br/>completion"]
    X --> M["Model<br/>computation"]
    I -->|"ALWAYS_PARTIAL"| M
    M --> O["finish()<br/>record the<br/>contribution"]
    O --> S
```

Construction and execution have different lifetimes:

1. **At model initialization**, `declare_attn()` and `declare_ffn()` describe compute behavior. `append_stages()` connects those declarations in stack order and binds each stage's own norm and hooks.
2. **During construction**, factories resolve token rows and reduction contracts for supported batch variants. `bind_entry()` binds consumer work; `bind_exit()` binds output transport. `StagePlan` retains those paths.
3. **At each forward**, batch facts select an existing path. An exit can make a batch-dependent completion decision, but it does not rebuild the declarations. The decision supplies the matching completion action; the only flag it publishes to compute is `defer_moe_finalize`, for a MoE that may hand off its finalize.
4. **Across stages**, `ForwardBatch.residual_stream` carries residual state and the producer's actual contribution. Stages do not store forward tensors on their reusable plans.

The resolved `StageContract` is a lower-level type than `StageDeclaration`: one `InputContract` and one `OutputContract` after topology and batch variant have been applied. An `EdgeContract` combines a producer output, consumer requirement, and the residual's source/destination layouts.

### Module map

Paths in this table are relative to `python/sglang/srt/layers/layer_boundary/`.

| Module | Role |
| - | - |
| `factories.py` | Model declarations, local sequence assembly, and stage factories |
| `contracts.py` | Resolved input/output contracts, edge declarations, batch variants, and bound step records |
| `layout.py` | Token axes, layouts, named sum groups, and topology/batch facts |
| `construction.py` | Build a `StagePlan` and select its batch variant |
| `boundary.py` | Select and bind producer/consumer boundary operations from contracts; for prefill CP, one `CpMoves` (`contracts.py`) pairs the FFN-input gather, the take-back of a complete output, and, where there is one, the reduce-scatter that completes a sum |
| `prepare.py` | Execute completion, update, read, and input movement in the selected order |
| `ops.py` | Collective and token-row movement primitives |
| `stage.py` | Model-facing `StageBoundary` methods |
| `exit.py` | Select output completion and publish scoped compute flags |
| `output.py` | Unreduced outputs, deferred finalizes and output transforms |
| `facts.py` | What binding keeps of a stage it does not run (`facts_of()`): a stack's neighbours on other pipeline ranks and a bound stage's producer |
| `row_transport.py` | Row transports that run a stage's compute on token rows other than its residual's (`RowTransport`, `ComputeTransport`) |
| `residual/__init__.py` | `ResidualUpdate`/`ResidualReadout` protocols and `LayerResidualOps` |
| `residual/stream.py` | Per-forward state and ownership of outstanding work |
| `residual/batch.py` | Model-stack entry, final norm, pipeline-parallel (PP) transport, snapshots, and extra contributions |
| `residual/access.py` | Tensor access and materialization helpers |
| `residual/add_norm.py`, `residual/mhc.py` | Plain add/norm and MHC read/update implementations |
| `residual/ihc.py`, `residual/gated.py` | Gated and gated-injection hyper-connection read/update implementations |
| `residual/post_norm.py` | Post-norm residual: each stage's output is normalized before it is added |
| `residual/attn_bank.py` | A residual read that aggregates a bank of residual snapshots |
| `residual/short_conv.py` | Deferred channelwise convolution followed by residual add and normalization |
| `adapters/` | Attention input, TBO, branches, and LoRA integration |
| `fusions/` | Consumer all-reduce fusion and CuteDSL integration |

Models normally use factories, `StageBoundary`, and `residual.batch`. Keep row arithmetic, process-group selection, and calls such as `token_axis_sizes()` inside the boundary implementation.

### Batch variants

`StagePlan` preconstructs ordinary, CP, input-scattered, sequence-parallel, and unpadded variants when supported by the configured topology. The unpadded variant serves batches whose rows do not divide over attention TP, which `--disable-attn-tp-gather` lets through without attention DP: an FFN that would run on this rank's attention-TP slice stays on the attention's rows instead. A forward selects a path from active runtime facts. An absent active variant raises an error rather than silently using ordinary token rows.

Configuration validation rejects unsupported model/parallel combinations. Boundary construction also rejects transitions without a correct implementation. Adding an enum value or a layout does not by itself provide a new collective path.

### Fusion providers and output decisions

Each exit has its own policy, and what it decides from fixed facts (the parallel configuration, the declarations and the bound path) is decided when the plan is built and recorded on the path as `ExitFacts`; only what depends on the batch is decided for it:

| Exit | Entry point | May decide |
| - | - | - |
| FFN | `exit()` or `finish()` | Complete the sum now, possibly in an attention-DP or CP / input-scattered reduce-scatter on the way back; leave it to the next input (an all-reduce, or under attention DP the scatter back to this rank's tokens); hand a MoE finalize to the next input |
| Operation-scheduled (TBO) | `complete_now()` | Only complete it now: under attention DP the move back is a plain scatter; never defers or hands off |
| Mixer | `finish()` | Leave its declared sum to the next stage's input (before an FFN); carry it to the next mixer's input (without attention DP); or complete it |
| Stack end | `exit()` or `finish()` | Complete it now; a MoE finalize handoff may reach a final norm that accepts it |
| Pipeline handoff | `exit()` or `finish()`, then `to_pp()` | Complete it now; an FFN on its own rows also writes it into the residual |

Fixed per path: whether an FFN may leave its sum to the next input at all, whether a reduce-scatter completes it without an attention-DP one, which sum an input-scattered batch's next input completes, whether the attention-DP return is the reduce-scatterv, whether the post-expert sum is one all-reduce, and whether a mixer carries its sum. Decided per batch: its padding, whether the attention-DP reduce-scatter is usable (an elastic EP scale-up changes the DP group), whether it has tokens, and whether a fused consumer takes the sum.

An output decision is made once. Its flags control whether compute performs reduction/finalize work; the matching completion action either executes that work or records exactly what the next boundary owes. Do not set a skip flag independently of a completion action.

A backend fusion provider is passed per stage as the `fusions` option, `(declaration, norm, {"fusions": provider})`; give a layer's attention and FFN stages the same provider. The boundary calls it as follows:

* `attn_input_fusions(plan)` and `ffn_input_fusions(plan)` return ordered consumer candidates, tried before the built-in all-reduce fusion.
* `can_defer_finalize(plan, forward_batch)` decides whether an FFN may return a `DeferredFinalize`. It is called on every FFN exit that has a provider.
* `can_defer_all_reduce(plan, forward_batch)` lets a LoRA or `SGLANG_SHARED_EXPERT_TP1` FFN leave its all-reduce to the next input. Such an FFN defers only without an expert all-to-all backend or quantized communication, when its sum is over the TP group, and when the provider, FlashInfer fusion, or AITER fusion accepts the batch. Other FFNs defer when the post-expert sum is a single all-reduce.

`install_cutedsl_fusion()` and `prepare_cutedsl_fusion()` find providers through the decoder layer's `attn_boundary` and `ffn_boundary` attributes.

Consumer fusion candidates are ordered. A candidate returning `None` must do so before modifying inputs or starting a collective, so fallback remains valid. `FfnInputFusion` declares the sum group it completes with residual add and norm. A residual read may supply its own candidates as `completing_fusions`: each `ReadoutFusion` completes a sum with the residual add, ahead of the read, and is tried before the built-in collective; one with `reads=True` also does the read. Where a stage reads its input on this rank's attention-TP slice and then gathers it, a read's `gathering_reads` may do both in one kernel. A stage may declare its own implementation of the gather over attention TP that brings rows into it (`declare_attn(attn_tp_gather=...)`), tried on every batch before the built-in gather: at its own entry, or at its producer's exit, since an exit runs its consumer's gather. Across a pipeline boundary the producer's rank takes that gather from the neighbouring stage's declaration, so the gather may use only the communication state of the rank it runs on, not the declaring layer's weights or buffers. The consumer only tries compatible candidates for the declared update/read capabilities.

An `OutputTransform` declares contribution processing before residual update. Its explicit `before_reduce_scatter` setting preserves a supported implementation's order; it is not permission to commute arbitrary operations across a reduction.

Producer GEMM plus reduce-scatter fusion is not a generic implemented API here. When adding such a backend, its output contract and actual completion must describe both the finished sum and returned token rows; do not leave the next boundary believing it still owes either operation.

### Special adapters

* **Attention/DeepSeek Sparse Attention (DSA) adapters:** retain `AttnTpContext` and its input-scattered/latent-input hooks. They are not a general asynchronous dependency scheduler.
* **LoRA adapter:** publishes the selected compute input's rows for LoRA execution.
* **TBO and branch adapters:** manage layer exits and entries that cannot be represented as a normal sequential prepare call.

### Extend and validate the module

Place changes according to their responsibility:

1. Add model compute facts to a declaration only when they describe compute behavior. Put user policy and unsupported configuration checks in server-argument handling.
2. Add new update/read semantics in `residual/`; declare only capabilities the implementation really supports.
3. Add a token transition in boundary selection and movement primitives, with its padding and row-order contract. Do not duplicate that logic in each model.
4. Put backend completion in `fusions/` and specialized exit and entry behavior in `adapters/`.
5. Keep models on stage factories and boundary methods. Avoid reading a neighbour's execution plan or reconstructing a collective decision during forward.

Focused tests live in `test/registered/unit/layer_boundary/`. Cover numerical output and residual state, collective group/row correctness, and supported fallback behavior for the changed path. Relevant examples include `test_ffn_exit.py`, `test_reduction_fusion.py`, and `test_append_stages.py`. A declaration-only assertion cannot establish that a new parallel path moves and reduces the correct values.


# MSProbe Debugging Guide
Source: https://docs.sglang.io/docs/developer_guide/msprobe_debugging_guide



MSProbe is a debugging tool for AI models that diagnoses accuracy anomalies and
numerical errors during model training and inference. It captures and monitors intermediate data (feature maps, weights,
activations, layer outputs) and contextual metadata (prompts, tensor dtypes, hardware configuration), and supports
visual analysis to systematically trace the root cause of accuracy degradation or numerical errors (e.g., NaN/Inf,
output drift, mismatched predictions).

## Basic Details

### Background Concepts: MSProbe Dumping Levels

MSProbe supports three accuracy levels for data dumping, each for different debugging needs:

* **L0**: Dumps tensors/statistics at the **module level** and generates `construct.json` (for network structure
  reconstruction in visualization). Requires passing a model/submodule handle.
* **L1**: Dumps tensors/statistics at the **torch API level**, suitable for fine-grained API-level numerical checking.
* **mix**: Combines L0 + L1, ideal for scenarios that require both **graph reconstruction** and **numerical comparison**.

### Prerequisites: Install MSProbe

Install MSProbe with pip:

```shell theme={null}
pip install mindstudio-probe --pre
```

### Key Configuration Parameters

MSProbe uses a JSON configuration file for customized data dumping. All core parameters are listed in the table below,
with the default JSON configuration provided for reference.

#### Configuration Parameter Table

| Field | Description | Required |
| :-: | :- | :-: |
| `task` | Type of dump task. Common PyTorch values include `"statistics"` and `"tensor"`. A statistics task collects tensor statistics (mean, variance, max, min, etc.) while a tensor task captures arbitrary tensors. | Yes |
| `dump_path` | Directory where dump results are stored. When omitted, `MSProbe` uses its default path. | No |
| `rank` | Ranks to sample. An empty list collects every rank. For single-card tasks you must set this field to `[]`. | No |
| `step` | Token iteration(s) to sample. An empty list means every iteration. | No |
| `level` | Dump level string (`"L0"`, `"L1"`, or `"mix"`). `L0` targets `nn.Module`, `L1` targets `torch.api`, and `mix` collects both. | Yes |
| `async_dump` | Whether to enable asynchronous dump (supported for PyTorch `statistics`/`tensor` tasks). Defaults to `false`. | No |
| `scope` | Customize the scope of dump. Provide two module or API names that follow the tool's naming convention to lock a range, only data between the two names will be dumped. An empty list dumps every module or torch API. Examples: `"scope": ["Module.conv1.Conv2d.forward.0", "Module.fc2.Linear.forward.0"]` `"scope": ["Tensor.add.0.forward", "Functional.square.2.forward"]` The `level` setting determines what can be provided—modules when `level=L0`, APIs when `level=L1`, and either modules or APIs when `level=mix`. | No |
| `list` | Customize dump list, only dumps elements from the list. An empty list dumps every module or torch API. Options include: •Supply the full names of specific APIs in PyTorch eager mode to only dump those APIs. Example: `"list": ["Tensor.permute.1.forward", "Tensor.transpose.2.forward", "Torch.relu.3.backward"]`. •When `level=mix`, you can provide module names so that the dump expands to everything produced while the module is running. Example: `"list": ["Module.module.language_model.encoder.layers.0.mlp.ParallelMlp.forward.0"]`. •Provide a substring such as `"list": ["relu"]` to dump every API whose name contains the substring. When `level=mix`, modules whose names contain the substring are also expanded. | No |

#### Default configuration

```json theme={null}
{
  "task": "statistics",
  "dump_path": "./dump_path",
  "rank": [],
  "step": [],
  "level": "L1",
  "async_dump": false,
  "statistics": {
    "scope": [],
    "list": [],
    "data_mode": [
      "all"
    ],
    "summary_mode": "statistics"
  },
  "tensor": {
    "scope": [],
    "list": [],
    "data_mode": [
      "all"
    ],
    "file_format": "npy"
  },
  "acc_check": {
    "white_list": [],
    "black_list": [],
    "error_data_path": "./"
  }
}
```

#### Outputs

Dump files are written into `dump_path` you defined. They usually contain:

* `dump.json`, which records metadata such as dtype, shape, min, max, mean, L2 norm, and `requires_grad`.
* `construct.json`, hierarchical structure description, when `level` is `L0` or `mix` (required for visualization), its
  content is not empty.
* `stack.json`, record the call stack information of API/Module.
* `dump_tensor_data`, generated when `task` is `tensor` and save the collected tensor data.

See [dump directory description](#dump-directory-description) for details.

> **Note**: When MSProbe is enabled, cuda graph is disabled (disable\_cuda\_graph=True) because MSProbe only supports dump
> in eager mode, warmup is disabled (skip\_server\_warmup=True) because there is no need to dump data for this stage.

## End-to-End Examples

MSProbe’s full debugging workflow follows **Enable → Collect Data → Visualize → Analyze Root Cause**. Below is a common
E2E example for SGLang-based model inference debugging.

### Example : Advanced Debugging with Custom Configuration

Suitable for targeted debugging (e.g., only collect statistics data for specific ranks/steps, enable mix level for graph
reconstruction + numerical comparison) and root cause analysis via **problem vs. benchmark comparison**.

#### Step 1: Enable

##### Prepare Custom Configuration JSON

Create `msprobe-config.json` (dump statistics data for rank0/1, step0/1, mix level):

```json theme={null}
{
  "task": "statistics",
  "dump_path": "./problem_dump",
  "rank": [
    0,
    1
  ],
  "step": [
    0,
    1
  ],
  "level": "mix",
  "async_dump": false,
  "statistics": {
    "scope": [],
    "list": [],
    "data_mode": [
      "all"
    ],
    "summary_mode": "statistics"
  }
}
```

##### Enable MSProbe with Custom Configuration in SGLang

Launch the SGLang server and specify the configuration file path with `--msprobe-dump-config`:

```bash theme={null}
python3 -m sglang.launch_server \
 --model-path Qwen/Qwen2.5-0.5B-Instruct \
 --host 127.0.0.1 \
 --port 1027 \
 --msprobe-dump-config /home/msprobe-config.json
```

#### Step 2: Collect Data

##### Collect Dump Data for Problem & Benchmark Sides

Send normal inference requests to trigger model running (MSProbe automatically collects data during request processing):

```bash theme={null}
curl -H "Content-type: application/json" \
 -X POST \
 -d '{
     "model": "Qwen/Qwen2.5-0.5B-Instruct",
     "messages": [
         {
             "role": "user",
             "content": "Hello, my name is"
         }
     ],
     "max_tokens": 10
 }' \
 http://127.0.0.1:1027/v1/chat/completions
```

* **Problem side**: Run the above SGLang server (with the accuracy/numerical issue) and send inference request; dump
  data is saved to `./problem_dump`.
* **Benchmark side**: Launch a normal SGLang server (without the issue, e.g., stable framework version/operator) with
  the **same custom configuration** and send the **same inference request**; rename the dump directory
  to `./bench_dump`.

> **Key Requirement**: Problem and benchmark dumps must use the same inputs and sampling points (rank/step)
> for valid comparison.

##### Check Generated Dump Files

Dump files are saved to `./problem_dump` and `./bench_dump` you defined and include core files for subsequent analysis:

* `dump.json`: Records tensor metadata of APIs and modules (dtype, shape, min/max/mean, L2 norm, `requires_grad`, etc.).
* `stack.json`: Logs call stack information of APIs and modules.
* `construct.json`: hierarchical structure description, required for visualization, its content is not empty.

#### Step 3: Visualize

##### Visualize Problem vs. Benchmark Comparison (Multi-Rank)

Generate a multi-rank comparison visualization file (mix level generates `construct.json` for graph reconstruction):

```shell theme={null}
msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output
```

* `-tp`: Path to problem-side dump data
* `-gp`: Path to benchmark-side dump data
* `-o`: Output directory for visualization files

If you want overflow check (for NaN/Inf detection), please specify the parameter `-oc`

```shell theme={null}
msprobe graph_visualize -tp ./problem_dump/step0 -gp ./bench_dump/step0 -o ./graph_output -oc
```

After the comparison or build task finishes, a `compare_{timestamp}.vis.db` file is created under `graph_output`.

The dumped data can be used to visualize and analyze differences using tables or charts generated with visualization tools such as Matplotlib and Excel.

##### Launch TensorBoard

Start TensorBoard:

```bash theme={null}
tensorboard --logdir ./graph_output --bind_all --port 6006
```

#### Step 4: Analyze Root Cause

##### Locate Root Cause

Root Cause Analysis in TensorBoard:

* Divergent nodes (with accuracy/numerical differences) are highlighted in **red** (darker red = larger difference).
* Click on divergent nodes to view detailed tensor data (inputs/outputs, parameters) and API/module call stacks.
* Use the **search/filter** function to quickly locate key layers/APIs (e.g., "relu", "conv").
* Switch between ranks/steps via the UI to check cross-rank/cross-step divergence.
* Check the **overflow check** tab for NaN/Inf values in specific nodes (the direct cause of numerical instability).

##### Verify the Root Cause

After locating the divergent node (e.g., a specific Conv layer or torch API with abnormal tensor values), verify by:

* Narrowing the dump scope to this node (via `scope`/`list` in the configuration file) for fine-grained data collection.
* Modifying the problematic layer/API (e.g., replacing the operator, adjusting the dtype) and re-running the debugging
  workflow to confirm the issue is resolved.

## Troubleshooting

### No Dump Files Generated

1. To confirm if MSProbe is installed, use `pip show mindstudio-probe` to troubleshoot. If it is installed, the MSProbe
   version information will be printed. If it is confirmed that it has not been installed, please
   use `pip install mindstudio-probe --pre` for installation;
2. Confirm the `--msprobe-dump-config` parameter points to the **correct JSON file path**.

### Dump Files Are Too Large (Excessive Data)

1. Start with `task: "statistics"` instead of `"tensor"` to collect only tensor statistics (avoids raw tensor dump);
2. Narrow the dump range with the `scope` field (specify start/end module/API);
3. Filter dump targets with the `list` field (only dump specific modules/APIs or substrings);
4. Sample specific `rank` and `step` (avoid dumping all ranks/iterations).

### TensorBoard Visualization Fails

1. Confirm `construct.json` is not empty (requires `level: L0` or `mix` – L1 does not generate graph files);
2. Check that the `-tp` (problem dump) and `-gp` (benchmark dump) paths point to **valid rank/step subdirectories** (
   e.g., `step0/rank0`);
3. Ensure the MSProbe version is up-to-date (reinstall with `pip install mindstudio-probe --pre --upgrade`);
4. Verify TensorBoard is installed and the `--logdir` parameter points to the directory containing `.vis.db` files (not
   the file itself).

### Numerical Comparison Shows No Divergence But Model Accuracy Is Low

1. Expand the dump `step` range (check more token iterations for late-stage divergence);
2. Switch to `task: "tensor"` (statistics may mask subtle numerical differences in raw tensor data);
3. Ensure the problem and benchmark dumps use **the same input data/hardware configuration** (different inputs lead to
   invalid comparisons);
4. Use the `manual mapping` feature in TensorBoard (automatic mapping may miss some nodes for custom models).

***

## Appendix

### Dump directory description

```text theme={null}
├── problem_dump or bench_dump
│   ├── step0
│   │   ├── rank0
│   │   │   ├── dump_tensor_data
│   │   │   │    ├── Tensor.permute.1.forward.pt
│   │   │   │    ├── Functional.linear.5.backward.output.pt    # Format: {api_type}.{api_name}.{call_count}.{forward/backward}.{input/output}.{arg_index}.
│   │   │   │    │                                              # arg_index is the nth input or output of the API. If an input is a list, keep numbering with decimals (e.g., 1.1 is the first element of the first argument).
│   │   │   │    ├── Module.conv1.Conv2d.forward.0.input.0.pt          # Format: {Module}.{module_name}.{class_name}.{forward/backward}.{call_count}.{input/output}.{arg_index}.
│   │   │   │    ├── Module.conv1.Conv2d.forward.0.parameters.bias.pt  # Module parameter data: {Module}.{module_name}.{class_name}.forward.{call_count}.parameters.{parameter_name}.
│   │   │   │    └── Module.conv1.Conv2d.parameters_grad.weight.pt     # Module parameter gradients: {Module}.{module_name}.{class_name}.parameters_grad.{parameter_name}. Gradients do not include call_count because the same gradient updates all invocations.
│   │   │   │                                                          # When the `model` argument passed to dump is a List[torch.nn.Module] or Tuple[torch.nn.Module], module-level data names also include the index inside the list ({Module}.{index}.*), e.g., Module.0.conv1.Conv2d.forward.0.input.0.pt.
│   │   │   ├── dump.json
│   │   │   ├── stack.json
│   │   │   ├── dump_error_info.log
│   │   │   └── construct.json
│   │   ├── rank1
│   │   │   ├── dump_tensor_data
│   │   │   │   └── ...
│   │   │   ├── dump.json
│   │   │   ├── stack.json
│   │   │   ├── dump_error_info.log
│   │   │   └── construct.json
│   │   ├── ...
│   │   │
│   │   └── rank7
│   ├── step1
│   │   ├── ...
│   ├── step2
```

* `rank`: Device ID. Each card writes its data to the corresponding `rank{ID}` directory. In non-distributed scenarios
  the directory is simply named `rank`.
* `dump_tensor_data`: Save the collected tensor data.
* `dump.json`: Statistics for the forward data of each API or module, including names, dtype, shape, max, min, mean, L2
  norm (square root of the L2 variance), and CRC-32 when `summary_mode="md5"`.
  See [dump.json file description](#dump-json-file-description) for details.
* `dump_error_info.log`: Present only when the dump tool encountered an error and records the failure log.
* `stack.json`: Call stacks for APIs/modules.
* `construct.json`: Hierarchical structure description. Empty when `level=L1`.

### dump.json file description

#### L0 level

An L0 `dump.json` contains forward/backward I/O for modules together with parameters and parameter gradients. Using
PyTorch's `Conv2d` as an example, the network code looks like:

`output = self.conv2(input)  # self.conv2 = torch.nn.Conv2d(64, 128, 5, padding=2, bias=True)`

`dump.json` contains the following entries:

* `Module.conv2.Conv2d.forward.0`: Forward data of the module. `input_args` represents positional inputs, `input_kwargs`
  represents keyword inputs, `output` stores forward outputs, and `parameters` stores weights/biases.
* `Module.conv2.Conv2d.parameters_grad`: Parameter gradients (weight and bias).
* `Module.conv2.Conv2d.backward.0`: Backward data of the module. `input` represents gradients that flow into the
  module (gradients of the forward outputs) and `output` represents gradients that flow out (gradients of the module
  inputs).

**Note**: When the `model` parameter passed to the dump API is `List[torch.nn.Module]` or `Tuple[torch.nn.Module]`,
module-level names include the index inside the list (`{Module}.{index}.*`). Example: `Module.0.conv1.Conv2d.forward.0`.

<details>
  <summary>L0 dump.json</summary>

  ```json theme={null}
  {
    "task": "tensor",
    "level": "L0",
    "framework": "pytorch",
    "dump_data_dir": "/dump/path",
    "data": {
      "Module.conv2.Conv2d.forward.0": {
        "input_args": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              8,
              16,
              14,
              14
            ],
            "Max": 1.638758659362793,
            "Min": 0.0,
            "Mean": 0.2544615864753723,
            "Norm": 70.50277709960938,
            "requires_grad": true,
            "data_name": "Module.conv2.Conv2d.forward.0.input.0.pt"
          }
        ],
        "input_kwargs": {},
        "output": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              8,
              32,
              10,
              10
            ],
            "Max": 1.6815717220306396,
            "Min": -1.5120246410369873,
            "Mean": -0.025344856083393097,
            "Norm": 149.65576171875,
            "requires_grad": true,
            "data_name": "Module.conv2.Conv2d.forward.0.output.0.pt"
          }
        ],
        "parameters": {
          "weight": {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32,
              16,
              5,
              5
            ],
            "Max": 0.05992485210299492,
            "Min": -0.05999220535159111,
            "Mean": -0.0006165213999338448,
            "Norm": 3.421217441558838,
            "requires_grad": true,
            "data_name": "Module.conv2.Conv2d.forward.0.parameters.weight.pt"
          },
          "bias": {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32
            ],
            "Max": 0.05744686722755432,
            "Min": -0.04894155263900757,
            "Mean": 0.006410328671336174,
            "Norm": 0.17263513803482056,
            "requires_grad": true,
            "data_name": "Module.conv2.Conv2d.forward.0.parameters.bias.pt"
          }
        }
      },
      "Module.conv2.Conv2d.parameters_grad": {
        "weight": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32,
              16,
              5,
              5
            ],
            "Max": 0.018550323322415352,
            "Min": -0.008627401664853096,
            "Mean": 0.0006675920449197292,
            "Norm": 0.26084786653518677,
            "requires_grad": false,
            "data_name": "Module.conv2.Conv2d.parameters_grad.weight.pt"
          }
        ],
        "bias": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32
            ],
            "Max": 0.014914230443537235,
            "Min": -0.006656786892563105,
            "Mean": 0.002657240955159068,
            "Norm": 0.029451673850417137,
            "requires_grad": false,
            "data_name": "Module.conv2.Conv2d.parameters_grad.bias.pt"
          }
        ]
      },
      "Module.conv2.Conv2d.backward.0": {
        "input": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              8,
              32,
              10,
              10
            ],
            "Max": 0.0015069986693561077,
            "Min": -0.001139344065450132,
            "Mean": 3.3215508210560074e-06,
            "Norm": 0.020567523315548897,
            "requires_grad": false,
            "data_name": "Module.conv2.Conv2d.backward.0.input.0.pt"
          }
        ],
        "output": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              8,
              16,
              14,
              14
            ],
            "Max": 0.0007466732058674097,
            "Min": -0.00044813455315306783,
            "Mean": 6.814070275140693e-06,
            "Norm": 0.01474067009985447,
            "requires_grad": false,
            "data_name": "Module.conv2.Conv2d.backward.0.output.0.pt"
          }
        ]
      }
    }
  }
  ```
</details>

#### L1 level

An L1 `dump.json` records forward/backward I/O for APIs. Using PyTorch's `relu` function as an
example (`output = torch.nn.functional.relu(input)`), the file contains:

* `Functional.relu.0.forward`: Forward data of the API. `input_args` are positional inputs, `input_kwargs` are keyword
  inputs, and `output` stores the forward outputs.
* `Functional.relu.0.backward`: Backward data of the API. `input` represents the gradients of the forward outputs,
  and `output` represents the gradients that flow back to the forward inputs.

<details>
  <summary>L1 dump.json</summary>

  ```json theme={null}
  {
    "task": "tensor",
    "level": "L1",
    "framework": "pytorch",
    "dump_data_dir": "/dump/path",
    "data": {
      "Functional.relu.0.forward": {
        "input_args": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32,
              16,
              28,
              28
            ],
            "Max": 1.3864083290100098,
            "Min": -1.3364859819412231,
            "Mean": 0.03711778670549393,
            "Norm": 236.20692443847656,
            "requires_grad": true,
            "data_name": "Functional.relu.0.forward.input.0.pt"
          }
        ],
        "input_kwargs": {},
        "output": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32,
              16,
              28,
              28
            ],
            "Max": 1.3864083290100098,
            "Min": 0.0,
            "Mean": 0.16849493980407715,
            "Norm": 175.23345947265625,
            "requires_grad": true,
            "data_name": "Functional.relu.0.forward.output.0.pt"
          }
        ]
      },
      "Functional.relu.0.backward": {
        "input": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32,
              16,
              28,
              28
            ],
            "Max": 0.0001815402356442064,
            "Min": -0.00013352684618439525,
            "Mean": 0.00011915402356442064,
            "Norm": 0.007598237134516239,
            "requires_grad": false,
            "data_name": "Functional.relu.0.backward.input.0.pt"
          }
        ],
        "output": [
          {
            "type": "torch.Tensor",
            "dtype": "torch.float32",
            "shape": [
              32,
              16,
              28,
              28
            ],
            "Max": 0.0001815402356442064,
            "Min": -0.00012117840378778055,
            "Mean": 2.0098118724831693e-08,
            "Norm": 0.006532244384288788,
            "requires_grad": false,
            "data_name": "Functional.relu.0.backward.output.0.pt"
          }
        ]
      }
    }
  }
  ```
</details>

#### mix level

A `mix` dump.json contains both L0 and L1 level data; the file format is the same as the examples above.


# Developer Guide
Source: https://docs.sglang.io/docs/developer_guide/overview

Contributing to SGLang — development setup, benchmarking, and evaluation.

* [Contribution Guide](./contribution_guide)
* [Add an out-of-tree serve backend](/docs/developer_guide/serve_backend_plugins)
* [Development Guide (Docker)](./development_guide_using_docker)
* [JIT Kernels](./development_jit_kernel_guide)
* [Quantization Contribution Guide](./quantization_contribution_guide)
* [Layer boundaries](/docs/developer_guide/layer_boundary)
* [Benchmark and Profiling](./benchmark_and_profiling)
* [Bench Serving](./bench_serving)
* [Evaluating New Models](./evaluating_new_models)
* [MSProbe Debugging Guide](./msprobe_debugging_guide)


# Quantization Contribution Guide
Source: https://docs.sglang.io/docs/developer_guide/quantization_contribution_guide



This guide describes how to add or refactor quantization support in SGLang. It focuses on the common structure used by weight-only and weight-activation quantization methods such as AWQ, GPTQ, compressed-tensors, ModelSlim, Quark, and related backend kernels.

## Design Goals

Quantization code should keep the quantization format semantics separate from hardware-specific execution. This makes it easier to add new formats, reuse kernels across formats, and review platform-specific changes independently.

Follow the architecture proposed in [Quantization Modifications](https://github.com/sgl-project/sglang/issues/15194):

* **Config**: parses model and runtime quantization parameters, validates supported options, and selects the proper scheme.
* **Scheme**: owns quantized weight creation, weight loading, post-processing, and quantized layer wiring for Linear, MoE, embedding, or other module types.
* **Backend kernel**: wraps hardware-specific execution, layout conversion, kernel selection, and kernel calls for GPU (CUDA/HIP/XPU), NPU, or other backends.

Avoid putting config parsing, weight loading, and backend kernel calls in a single monolithic file. If a method needs multiple formats or backends, add a package under `python/sglang/srt/layers/quantization/<method>/` and split schemes into `schemes/`.

## Recommended File Layout

Use this layout for a quantization method that has multiple schemes or backend-specific execution paths:

```text theme={null}
python/sglang/srt/layers/quantization/<method>/
  __init__.py
  <method>.py
  schemes/
    __init__.py
    <method>_scheme.py
    <method>_linear.py
    <method>_moe.py
    <method>_<variant>.py
```

Backend kernels should live under the hardware backend they target:

```text theme={null}
python/sglang/srt/hardware_backend/gpu/quantization/<method>_kernels.py
python/sglang/srt/hardware_backend/npu/quantization/<method>_kernels.py
```

Keep shared method selection in the quantization package and keep backend imports narrow. This prevents circular imports and keeps non-target platforms from importing unavailable kernel dependencies.

## Adding or Refactoring a Quantization Method

1. Define the config entry point and register it through `python/sglang/srt/layers/quantization/__init__.py` when needed.
2. Add explicit scheme selection helpers such as `get_linear_scheme` and `get_moe_scheme`.
3. Move layer-specific weight creation and weight loading into scheme classes.
4. Move GPU (CUDA/HIP/XPU), NPU, or other hardware kernel calls into backend kernel modules.
5. Keep Linear, MoE, embedding, and non-linear module handling explicit. Do not assign a Linear quantization method to a module type that needs different semantics.
6. Preserve compatibility for existing quantized checkpoints and runtime flags.
7. Add tests that cover both config parsing and execution paths touched by the change.

For examples, see the AWQ and GPTQ refactors:

* [PR #21126](https://github.com/sgl-project/sglang/pull/21126): splits AWQ schemes, weight initialization, and backend kernel calls.
* [PR #26402](https://github.com/sgl-project/sglang/pull/26402): applies the same scheme/kernel split to GPTQ.

## Tests and Validation

Quantization changes can affect both accuracy and performance. Include validation that matches the blast radius of the change.

For Python-only structure changes:

```bash theme={null}
ruff check <changed-python-files>
git diff --check
```

For quantized model behavior:

* Launch at least one representative model for each touched quantization method.
* Send a `/generate` request and confirm the output path succeeds.
* Run an accuracy sanity test if the change can affect numerics.
* Include warmup-aware benchmark results when the change affects kernel calls, layout conversion, or dispatch.

For backend-specific changes:

* Validate GPU changes on a supported GPU environment (NVIDIA, AMD, or Intel).
* Validate NPU changes on a supported Ascend environment.
* Include the exact model, quantization flag, backend flag, hardware, and command used in the PR description.

## PR Checklist

Before requesting review, make sure the PR description includes:

* The quantization method and backend paths changed.
* The issue, design proposal, or roadmap item the PR follows.
* Any compatibility notes for existing checkpoints or flags.
* Accuracy results when model outputs can change.
* Benchmark or profiling results when runtime performance can change.
* The exact local checks and model launch tests that were run.

Use the general [Contribution Guide](./contribution_guide) for source setup, formatting, unit tests, CI triggering, and review process details.


# Add an out-of-tree serve backend
Source: https://docs.sglang.io/docs/developer_guide/serve_backend_plugins

Connect an ecosystem runtime to sglang serve through the versioned serve backend plugin API.

A serve backend plugin connects an out-of-tree runtime to the SGLang-owned CLI:

```bash theme={null}
sglang serve MODEL_PATH --model-type BACKEND_NAME [BACKEND_OPTIONS]
```

The extension retains its own argument parser, process topology, API endpoints, hardware requirements, and release cycle. The core CLI owns backend selection and common child-process cleanup.

## Prerequisites

* Install your extension and a compatible SGLang version in the same Python 3.11+ environment.
* Declare the SGLang version range tested by your extension in its package dependencies.
* Keep the backend factory and detector independent of GPU initialization and model loading.

The plugin API itself is platform-independent. Your backend defines its supported operating systems, accelerators, parallelism options, and authentication requirements.

## Keep one owner for the executable

Only the `sglang` distribution should publish a console script named `sglang`. Your extension registers package metadata under `sglang.serve_backends`; it must not publish another `sglang` script.

This prevents installation order from replacing the command and ensures uninstalling an extension does not remove the core executable. You can retain a project-specific executable as a compatibility alias:

```bash theme={null}
my-runtime serve MODEL_PATH
sglang serve MODEL_PATH --model-type my_runtime
```

Both commands should call the same backend implementation.

## Register a backend factory

Add a zero-argument factory to your extension's `pyproject.toml`:

```toml theme={null}
[project]
name = "my-sglang-runtime"
version = "0.1.0"
dependencies = ["sglang"]

[project.entry-points."sglang.serve_backends"]
my_runtime = "my_sglang_runtime.sglang_backend:create_backend"
```

The entry point name, `my_runtime`, becomes an accepted `--model-type` value. Choose a distinctive name. `auto` and SGLang's in-tree backend names are reserved.

## Implement the backend

Create `my_sglang_runtime/sglang_backend.py`:

```python theme={null}
import argparse

from sglang.cli.serve_backends import (
    ServeBackend,
    ServeBackendDetection,
    ServeRequest,
)


def detect(request: ServeRequest) -> ServeBackendDetection:
    if request.model_path is None:
        return ServeBackendDetection.UNKNOWN
    if supports_model(request.model_path):
        return ServeBackendDetection.MATCH
    return ServeBackendDetection.NO_MATCH


def run(request: ServeRequest) -> None:
    parser = argparse.ArgumentParser(prog="sglang serve")
    parser.add_argument("--model-path", required=True)
    parser.add_argument("--pipeline-parallel", type=int, default=1)
    args, remaining = parser.parse_known_args(request.argv)
    launch_runtime(args, remaining)


def create_backend() -> ServeBackend:
    return ServeBackend(api_version=1, run=run, detect=detect)
```

Replace `supports_model()` and `launch_runtime()` with your extension's lightweight metadata check and blocking server launcher. A real `run()` call should block for the server lifetime. It must also honor `-h` and `--help` without launching a server; `argparse` does this automatically.

Among serve backend entry points, explicit selection imports only the selected provider. Automatic selection loads installed backend factories and invokes their detectors, so importing this module and calling `detect()` must not initialize accelerators, import model weights, or start workers.

## Handle forwarded arguments

SGLang removes `--model-type` and normalizes a positional Hugging Face model ID or local model directory before dispatch. For example:

```bash theme={null}
sglang serve org/model --model-type my_runtime --pipeline-parallel 2
```

Your backend receives:

```python theme={null}
("--model-path", "org/model", "--pipeline-parallel", "2")
```

The selected backend owns all remaining argument parsing and validation. Target backend-specific help with:

```bash theme={null}
sglang serve --model-type my_runtime --help
```

## Support config-only runtimes

SGLang requires a model path by default. If your runtime resolves its model and parallelism settings from a configuration file, disable that validation:

```python theme={null}
def create_backend() -> ServeBackend:
    return ServeBackend(
        api_version=1,
        run=run,
        detect=detect,
        requires_model_path=False,
    )
```

You can then accept commands such as:

```bash theme={null}
sglang serve --model-type my_runtime --config pipeline.yaml
```

Config-only requests generally require explicit `--model-type` unless your detector can identify the backend from the remaining arguments.

## Understand automatic routing

The default `--model-type auto` follows these rules:

1. Backends without a detector remain explicit-only.
2. One `MATCH` selects that backend.
3. Multiple matches fail and require an explicit `--model-type`.
4. `UNKNOWN` and detector failures do not claim the request.
5. No matches preserve the existing LLM fallback.

The registry does not resolve overlap by package installation order or a hidden priority. A backend can opt out of automatic routing by omitting its detector:

```python theme={null}
ServeBackend(api_version=1, run=run, requires_model_path=False)
```

## Maintain compatibility

Declare the API version implemented by your extension as a literal. Do not copy SGLang's current version constant at runtime; a fixed value lets a future SGLang release detect an older plugin contract. SGLang rejects incompatible, duplicate, and reserved backend registrations with an actionable error.

The public extension contract consists of:

* `ServeRequest`: normalized backend arguments and the optional model path
* `ServeBackend`: the runner, optional detector, and model-path requirement
* `ServeBackendDetection`: `MATCH`, `NO_MATCH`, or `UNKNOWN`

Test explicit selection, backend-specific help, automatic detection, ambiguous models, and installation or removal alongside the core `sglang` package.


# Examples
Source: https://docs.sglang.io/docs/get-started/examples

Run a model locally, add chat to an application, process a batch of texts, or build a tool-using assistant.

Choose an example for what you want to do:

| I want to… | Example |
| - | - |
| Run a model on my own computer | [Local inference](#run-a-model-locally) on an NVIDIA GPU or Apple Silicon Mac. |
| Serve a larger model on a GPU server | [Deploy Qwen3.8-27B](#serve-qwen3-8-27b) on an NVIDIA H200. |
| Add chat to an application | [Stream a reply](#stream-a-reply) through the OpenAI-compatible API. |
| Process a collection of documents | [Summarize multiple texts](#process-multiple-prompts) concurrently. |
| Build an agent that uses tools | [Look up an order](#build-a-customer-support-agent) and answer a customer’s question. |

## Run a model locally

Run `Qwen/Qwen3-0.6B` on your own computer and ask it a question. This small model is useful for trying the workflow; choose a larger model for more demanding tasks.

**NVIDIA GPU (Linux)**

This example uses one CUDA 13-compatible NVIDIA GPU, Linux, Docker, and NVIDIA Container Toolkit. For setup, see [Installation](/docs/get-started/install).

Run this command on your computer. Docker pulls the image if needed, and SGLang downloads the model on first use:

```bash theme={null}
docker run --rm --gpus all --ipc=host \
  -p 127.0.0.1:30000:30000 \
  lmsysorg/sglang:latest \
  sglang serve Qwen/Qwen3-0.6B \
    --host 0.0.0.0 --port 30000
```

If you already installed SGLang with uv, run this in that Python environment instead:

```bash theme={null}
sglang serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 30000
```

**Apple Silicon Mac**

On an Apple Silicon Mac with macOS 14 or newer, first follow the [Apple Silicon installation instructions](/docs/hardware-platforms/apple_metal#install-sglang) to install SGLang with its MLX dependencies. Use that Python environment to launch the model with 4-bit quantization:

```bash theme={null}
SGLANG_USE_MLX=1 python -m sglang.launch_server \
  --model-path Qwen/Qwen3-0.6B \
  --quantization mlx_q4 \
  --disable-cuda-graph \
  --host 127.0.0.1 --port 30000
```

The model runs on your Mac through MLX. It downloads the weights on first use and quantizes them in memory when loading.

Keep the server running. Once you see `The server is fired up and ready to roll!`, open another terminal on the same computer and send a request:

```bash theme={null}
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-0.6B",
    "messages": [{"role": "user", "content": "What is the capital of France? /no_think"}],
    "max_tokens": 128
  }'
```

Look for the answer in `choices[0].message.content`. Both the model and the API server are running locally. Press `Ctrl+C` in the server terminal when you are done, or leave it running for the chat and batch examples below.

## Serve Qwen3.8-27B

This configuration runs `Qwen/Qwen3.8-27B-FP8` on one NVIDIA H200. It requires Linux, a CUDA 13-compatible driver, Docker, and NVIDIA Container Toolkit configured for GPU access. For environment setup, see [Installation](/docs/get-started/install). For other hardware configurations, see the [Qwen3.8-27B Cookbook](/cookbook/autoregressive/Qwen/Qwen3.8-27B).

Set `HF_CACHE_DIR` to an absolute directory on the host for downloaded model weights, replacing the placeholder below. Run this command on the GPU host:

```bash theme={null}
HF_CACHE_DIR="/path/to/huggingface-cache"
docker run --rm --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
  --ipc=host \
  lmsysorg/sglang:latest \
  sglang serve \
    --trust-remote-code \
    --model-path Qwen/Qwen3.8-27B-FP8 \
    --kv-cache-dtype fp8_e4m3 \
    --mem-fraction-static 0.85 \
    --attention-backend flashinfer \
    --chunked-prefill-size 32768 \
    --max-prefill-tokens 32768 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 4.59 \
    --host 0.0.0.0 \
    --port 30000 \
    --mamba-radix-cache-strategy extra_buffer \
    --mamba-ssm-dtype float32
```

Docker downloads the image if needed. SGLang downloads the model on first use and starts the server. Keep this terminal open and wait for `The server is fired up and ready to roll!` before sending a request.

> **Already installed with uv?**
>
> Run this command in the activated Python environment where you installed SGLang. It starts the same server without Docker:
>
> ```bash theme={null}
> sglang serve \
>   --trust-remote-code \
>   --model-path Qwen/Qwen3.8-27B-FP8 \
>   --kv-cache-dtype fp8_e4m3 \
>   --mem-fraction-static 0.85 \
>   --attention-backend flashinfer \
>   --chunked-prefill-size 32768 \
>   --max-prefill-tokens 32768 \
>   --reasoning-parser qwen3 \
>   --tool-call-parser qwen3_coder \
>   --mamba-full-memory-ratio 4.59 \
>   --host 0.0.0.0 \
>   --port 30000 \
>   --mamba-radix-cache-strategy extra_buffer \
>   --mamba-ssm-dtype float32
> ```
>
> For Python environment setup, see [Install with uv](/docs/get-started/install#method-1-with-pip-or-uv).

Verify the running server with the [Quickstart request](/docs/get-started/quickstart#send-a-request), using `Qwen/Qwen3.8-27B-FP8` as the model name.

## Connect to a model

The following Python examples connect to the local server above. You can also use a remote SGLang server; only the server needs hardware that can run the model.

Install the Python client in your activated environment:

```bash theme={null}
uv pip install openai
```

Run this setup before each example in the same Python session:

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:30000/v1",
    api_key="EMPTY",
)
model = "Qwen/Qwen3-0.6B"
```

These settings match the local Qwen3-0.6B server above. For a remote server, replace `localhost` with its reachable address, use its model name, and supply an API key if authentication is enabled.

## Stream a reply

Ask a question and print the answer as it arrives:

```python theme={null}
stream = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Explain prefix caching in two sentences."}],
    stream=True,
)

for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
print()
```

The terminal displays the model's explanation incrementally. You can use the same stream to update a chat interface.

See the [API guide](/docs/basic_usage/openai_api_completions) for more request options.

## Process multiple prompts

Send several independent requests concurrently, for example to summarize customer feedback:

```python theme={null}
from concurrent.futures import ThreadPoolExecutor

feedback = [
    "Setup was quick, but I could not find the export button.",
    "The new search is faster and finds the documents I need.",
    "Please add a way to share saved reports with my team.",
]

def summarize(text):
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": f"Summarize in one sentence: {text}"}],
    )
    return response.choices[0].message.content

with ThreadPoolExecutor(max_workers=3) as pool:
    for summary in pool.map(summarize, feedback):
        print(summary)
```

You get one summary per input, in the original order. For batch processing inside a Python process without an HTTP server, see the [offline engine API](/docs/basic_usage/offline_engine_api).

## Build a customer support agent

Let an agent answer “Where is my order?” by looking up order data. The model chooses a tool, your application executes it, and the model uses the returned data to answer the customer.

Use the [Qwen3.8-27B server above](#serve-qwen3-8-27b), which enables a tool parser, or configure another model with the [tool-calling guide](/docs/advanced_features/tool_parser). The small local example does not configure tool calling. This script uses the OpenAI client installed above; update the endpoint and model ID to match your server.

Save this as `order_agent.py` and run `python order_agent.py`. The order data is a local fixture, so no external service or API key is needed for the lookup.

```python theme={null}
import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
model = "Qwen/Qwen3.8-27B-FP8"

orders = {
    "SGL-1042": {"status": "shipped", "estimated_delivery": "Friday"},
}

def lookup_order(order_id):
    return orders.get(order_id, {"error": "Order not found"})

tools = [{
    "type": "function",
    "function": {
        "name": "lookup_order",
        "description": "Look up an order's shipping status and estimated delivery.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string"}},
            "required": ["order_id"],
            "additionalProperties": False,
        },
    },
}]

messages = [
    {"role": "system", "content": "Use lookup_order for order status. Do not invent order details."},
    {"role": "user", "content": "Where is order SGL-1042, and when should it arrive?"},
]

for _ in range(3):
    response = client.chat.completions.create(
        model=model, messages=messages, tools=tools, tool_choice="auto",
    )
    message = response.choices[0].message
    messages.append(message.model_dump(exclude_none=True))
    if not message.tool_calls:
        print(message.content)
        break

    for call in message.tool_calls:
        try:
            if call.function.name != "lookup_order":
                raise ValueError("Unknown tool")
            arguments = json.loads(call.function.arguments)
            result = lookup_order(**arguments)
        except (ValueError, TypeError) as error:
            result = {"error": str(error)}
        print(f"{call.function.name}: {result}")
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),
        })
else:
    print("Stopped after three model turns without a final answer.")
```

You should see the lookup result followed by an answer that the order has shipped and is expected on Friday. The wording can vary. SGLang serves the model; the Python application runs the tools and manages the conversation. Replace `lookup_order` with an authorized order-system query to use real data.


# Installation
Source: https://docs.sglang.io/docs/get-started/install

Install SGLang with pip/uv, source, Docker, Kubernetes, and cloud deployment options.

Choose one installation method below. These instructions target Linux with NVIDIA GPUs; for other devices, see [Hardware platforms](/docs/hardware-platforms/overview).

## Install SGLang

**Docker**

Docker includes Python, SGLang, and its dependencies; no separate host Python installation is required.

The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
Use Linux with a supported NVIDIA GPU, a CUDA 13-compatible driver, [Docker Engine](https://docs.docker.com/engine/install/), and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Complete the Toolkit guide's Docker runtime configuration before starting a GPU container.

SGLang images ship a CUDA 13 environment. CUDA 12 images (`-cu12` / `-cu129`) are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag.

**Pull the image**

`latest` contains the newest stable release:

```bash theme={null}
docker pull lmsysorg/sglang:latest
```

**Image options**

The examples use `latest`. To use another image, replace the tag in the pull and run commands:

* Nightly: Built daily from the latest `main`, including changes not yet in a stable release. Choose a tag from the [nightly image list](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
* Runtime: A smaller image of the same stable release, with inference dependencies and fewer development utilities. Use the `latest-runtime` tag.
* AWS EFA: Any of the images above plus the AWS EFA userspace stack, for EFA-enabled EC2 instances. Add `-efa` to the tag: `latest-efa`, `latest-runtime-efa`, or the dated `nightly-dev-efa-<date>-<sha>` (moving tag `dev-efa`). See [AWS EFA](/docs/advanced_features/pd_disaggregation#aws-efa) for what the images include and how to run them.

For reproducible deployments, use a specific release or dated nightly tag from [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags).

**Start a container**

Set `HF_CACHE_DIR` to an absolute directory path on the host for downloaded model weights, replacing the placeholder below.

For gated or private models, export your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) as `HF_TOKEN` on the host and add `--env HF_TOKEN` before the image name in this command.

```bash theme={null}
HF_CACHE_DIR="/path/to/huggingface-cache"
docker run -it --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
  --ipc=host \
  lmsysorg/sglang:latest bash
```

The container shell is ready for `sglang serve`.

**pip / uv**

Use Python 3.11 or higher and a compatible CUDA 13 environment.

> **CUDA version compatibility**
>
> SGLang requires CUDA 13. CUDA 12 (`cu129`) wheels and images were retired with the upgrade to PyTorch 2.14, which publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last release with a CUDA 12 lane (PyTorch 2.13).

**Prepare the environment**

Use uv for faster installation. Install it with pip, or follow the [uv installation instructions](https://docs.astral.sh/uv/getting-started/installation/):

```bash theme={null}
pip install --upgrade pip
pip install uv
```

Create and activate a virtual environment:

```bash Command theme={null}
uv venv --python 3.12
source .venv/bin/activate
```

**Install SGLang**

```bash theme={null}
uv pip install --prerelease=allow sglang
```

> **Why --prerelease=allow?**
>
> Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op.

> **Nightly builds**
>
> To pick up the latest features and fixes before the next stable release, install a nightly build. Nightly wheels are built from the latest `main` and published to the SGLang wheel index. Add that index with `--extra-index-url`, and combine `--prerelease=allow` with `--index-strategy unsafe-best-match` so uv considers the nightly (pre-release) version alongside PyPI:
>
> ```bash Command theme={null}
> uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
> ```

For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).

**From source**

Use Python 3.11 or higher in an activated Python environment, with CUDA dependencies appropriate for the version you build.

**Get the source**

Clone a release tag (the example below pins one); choose another from [GitHub releases](https://github.com/sgl-project/sglang/releases) as needed:

```bash theme={null}
git clone -b v0.5.21 https://github.com/sgl-project/sglang.git
cd sglang
```

**Install SGLang**

Install the Python package in editable mode so local source changes are reflected in your environment:

```bash theme={null}
pip install --upgrade pip
pip install -e "python"
```

> **Development environment**
>
> For contributing to SGLang, follow the [development environment guide](/docs/developer_guide/development_guide_using_docker#setup-docker-container), which covers a container with development tools and dependencies installed.

For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).

## Start a model server

Start a model server inside your Docker container or activated Python environment. Replace `MODEL_PATH` with a Hugging Face model ID or a local model directory:

```bash theme={null}
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
```

Additional launch arguments depend on the model and your hardware.

* [Quickstart](/docs/get-started/quickstart): Launch a model and send a request to verify the server.
* [Cookbook](/cookbook/intro): Find model-specific launch arguments for your hardware.

## Deployment options

Choose how to run SGLang as a service. Expand an option for setup instructions.

> **Kubernetes**
>
> Serve models on a Kubernetes cluster, on one node or across multiple nodes.
>
> Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs).
>
> 1. Option 1: For single node serving (typically when the model size fits into GPUs on one node)
>
>    Execute command `kubectl apply -f docker/k8s-sglang-service.yaml`, to create k8s deployment and service, with llama-31-8b as example.
>
> 2. Option 2: For multi-node serving (usually when a large model requires more than one GPU node, such as `DeepSeek-R1`)
>
>    Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service.

> **Docker Compose**
>
> Run a model service on a single Docker host with a Compose configuration.
>
> > This method is recommended if you plan to serve it as a service.
> > A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
>
> 1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine
> 2. Execute the command `docker compose up -d` in your terminal.

> **SkyPilot**
>
> Deploy on Kubernetes or across cloud providers from one YAML configuration.
>
> To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).
>
> 1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
>
> 2. Deploy on your own infra with a single command and get the HTTP API endpoint:
>
> > **SkyPilot YAML: sglang.yaml**
> >
> > ```yaml Config theme={null}
> > # sglang.yaml
> > envs:
> >   HF_TOKEN: null
> >
> > resources:
> >   image_id: docker:lmsysorg/sglang:latest
> >   accelerators: A100
> >   ports: 30000
> >
> > run: |
> >   conda deactivate
> >   python3 -m sglang.launch_server \
> >     --model-path meta-llama/Llama-3.1-8B-Instruct \
> >     --host 0.0.0.0 \
> >     --port 30000
> > ```
>
> ```bash Command theme={null}
> # Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
> HF_TOKEN=<secret> sky launch -c sglang --env HF_TOKEN sglang.yaml
>
> # Get the HTTP API endpoint
> sky status --endpoint 30000 sglang
> ```
>
> 3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).

> **AWS SageMaker**
>
> Deploy on SageMaker with a pre-built SGLang image or your own container.
>
> To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy)
>
> Amazon Web Services provide supports for SGLang containers along with routine security patching. For available SGLang containers, check out [AWS SGLang DLCs](https://aws.github.io/deep-learning-containers/reference/available_images/#sglang).
>
> To deploy a pre-built SGLang Deep Learning Container without building your own image, see [Amazon SageMaker AI](/docs/basic_usage/aws_sagemaker).
>
> To host a model with your own container, follow the following steps:
>
> 1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script.
>
> 2. Push your container onto AWS ECR.
>
> > **Build script: build-and-push.sh**
> >
> > ```bash Command theme={null}
> > #!/bin/bash
> > AWS_ACCOUNT="<YOUR_AWS_ACCOUNT>"
> > AWS_REGION="<YOUR_AWS_REGION>"
> > REPOSITORY_NAME="<YOUR_REPOSITORY_NAME>"
> > IMAGE_TAG="<YOUR_IMAGE_TAG>"
> >
> > ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
> > IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}"
> >
> > echo "Starting build and push process..."
> >
> > # Login to ECR
> > echo "Logging into ECR..."
> > aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY}
> >
> > # Build the image
> > echo "Building Docker image..."
> > docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile .
> >
> > echo "Pushing ${IMAGE_URI}"
> > docker push ${IMAGE_URI}
> >
> > echo "Build and push completed successfully!"
> > ```
>
> 3. Deploy a model for serving on AWS Sagemaker, refer to [deploy\_and\_serve\_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk).
>    1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker.
>    2. To modify your model serving parameters, the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script allows for all available options within `python3 -m sglang.launch_server --help` cli by specifying environment variables with prefix `SM_SGLANG_`.
>    3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli.
>    4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`.

## Troubleshooting

### CUDA\_HOME

If installation fails with `OSError: CUDA_HOME environment variable is not set`, set `CUDA_HOME` to your CUDA Toolkit installation directory. Replace the path below with the actual path on your machine:

```bash theme={null}
export CUDA_HOME="/path/to/cuda"
```

For a versioned installation, this is typically `/usr/local/cuda-VERSION`. Alternatively, install FlashInfer first using the [FlashInfer installation instructions](https://docs.flashinfer.ai/installation.html), then retry the SGLang installation.

### FlashInfer

* [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub.
* To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`.


# Quickstart
Source: https://docs.sglang.io/docs/get-started/quickstart

Start a model server and verify it with one request.

## Install and start

These commands target Linux with an NVIDIA GPU and a CUDA 13-compatible driver. See [Installation](/docs/get-started/install) for environment requirements and other setup options.

**Docker**

With Docker and NVIDIA Container Toolkit configured, pull the image. Skip this command if you have already pulled it:

```bash theme={null}
docker pull lmsysorg/sglang:latest
```

Start the server on the GPU host. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`):

```bash theme={null}
docker run --rm --gpus all --ipc=host \
  -p 30000:30000 \
  lmsysorg/sglang:latest \
  sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
```

**uv**

With uv installed, create a Python environment and install SGLang. Skip this block if you already have an environment with SGLang installed:

```bash theme={null}
uv venv --python 3.12
source .venv/bin/activate
uv pip install --prerelease=allow sglang
```

Start the server in that environment. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`):

```bash theme={null}
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
```

For additional model-specific launch arguments, see the [Cookbook](/cookbook/intro). Wait for `The server is fired up and ready to roll!`.

## Send a request

In another terminal on the same host, send a request to verify the server. Use the same model ID for `MODEL_PATH`:

```bash theme={null}
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_PATH",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'
```

Check `choices[0].message.content` for the answer. Press `Ctrl+C` in the server terminal to stop.

See [Examples](/docs/get-started/examples) for a complete 27B deployment, streaming, and tool calls.


# AMD GPUs
Source: https://docs.sglang.io/docs/hardware-platforms/amd_gpu



This document describes how to run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).

## System Configuration

When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:

* [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
* [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
* [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
* [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
* [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)

**NOTE:** We strongly recommend reading these docs and guides entirely to fully utilize your system.

Below are a few key settings to confirm or enable for SGLang:

### Update GRUB Settings

In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:

```text GRUB Configuration theme={null}
pci=realloc=off iommu=pt
```

Afterward, run `sudo update-grub` (or your distro’s equivalent) and reboot.

### Disable NUMA Auto-Balancing

```bash Disable NUMA theme={null}
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
```

You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).

Again, please go through the entire documentation to confirm your system is using the recommended configuration.

## Install SGLang

You can install SGLang using one of the methods below.

### Install from Source

```bash Command theme={null}
# Use the last release branch
git clone -b v0.5.21 https://github.com/sgl-project/sglang.git
cd sglang

# Compile sgl-kernel
pip install --upgrade pip
cd python/sglang/kernels/aot
python setup_rocm.py install

# Install sglang python package along with diffusion support
cd ../../../..
rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
pip install -e "python[all_hip]"
```

### Install Using Docker (Recommended)

The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).

The steps below show how to build and use an image.

1. Build the docker image.
   If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.

   ```bash Command theme={null}
   docker build -t sglang_image -f rocm.Dockerfile .
   ```

2. Create a convenient alias.

   ```bash Command theme={null}
   alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
       --ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
       --security-opt seccomp=unconfined \
       -v $HOME/dockerx:/dockerx \
       -v /data:/data'
   ```

   If you are using RDMA, please note that:

   * `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
   * You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.

3. Launch the server.

   **NOTE:** Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).

   ```bash Command theme={null}
   drun -p 30000:30000 \
       -v ~/.cache/huggingface:/root/.cache/huggingface \
       --env "HF_TOKEN=<secret>" \
       sglang_image \
       python3 -m sglang.launch_server \
       --model-path NousResearch/Meta-Llama-3.1-8B \
       --host 0.0.0.0 \
       --port 30000
   ```

4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.

   ```bash Command theme={null}
   drun sglang_image \
       python3 -m sglang.bench_serving \
       --backend sglang \
       --dataset-name random \
       --num-prompts 4000 \
       --random-input 128 \
       --random-output 128
   ```

With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities.

## Quantization on AMD GPUs

The [Quantization documentation](../advanced_features/quantization#platform-compatibility) has a full compatibility matrix. The short version: FP8, AWQ, MXFP4, W8A8, GPTQ, compressed-tensors, Quark, and **petit\_nvfp4** (NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel)) all work on AMD. Methods that depend on Marlin or NVIDIA-specific kernels (`awq_marlin`, `gptq_marlin`, `gguf`, `modelopt_fp8`, `modelopt_fp4`) do not.

A few things to keep in mind:

* FP8 works via Aiter or Triton. Pre-quantized FP8 models like DeepSeek-V3/R1 work out of the box.
* AWQ uses Triton dequantization kernels on AMD. The faster Marlin path is not available.
* MXFP4 requires CDNA3/CDNA4 and `SGLANG_USE_AITER=1`.
* `petit_nvfp4` enables NVFP4 models (e.g., [Llama 3.3 70B FP4](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4)) on MI250/MI300X via [Petit](https://github.com/causalflow-ai/petit-kernel). Install with `pip install petit-kernel`; no `--quantization` flag needed when loading pre-quantized NVFP4 models.
* `quark_int4fp8_moe` is an AMD-only online quantization method for MoE models on CDNA3/CDNA4.

Several of these backends are accelerated by [Aiter](https://github.com/ROCm/aiter). Enable it with:

```bash Command theme={null}
export SGLANG_USE_AITER=1
```

Example -- serving an AWQ model:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4 \
    --trust-remote-code \
    --port 30000 --host 0.0.0.0
```

Example -- FP8 online quantization:

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --quantization fp8 \
    --port 30000 --host 0.0.0.0
```

## Examples

### Running DeepSeek-V3

The only difference when running DeepSeek-V3 is in how you start the server. Here's an example command:

```bash Command theme={null}
drun -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    --env "HF_TOKEN=<secret>" \
    sglang_image \
    python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V3 \ # <- here
    --tp 8 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000
```

[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.

### Running Llama3.1

Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server, shown by the following example command:

```bash Command theme={null}
drun -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    --env "HF_TOKEN=<secret>" \
    sglang_image \
    python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here
    --tp 8 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000
```

### Warmup Step

When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.

## Strix Halo / gfx1151 (RDNA3.5 iGPU)

AMD Ryzen AI MAX+ / Strix Halo (`gfx1151`) and Strix Point (`gfx1150`) are wave32 RDNA3.5 iGPUs. They are **not** covered by the Instinct/CDNA `rocm.Dockerfile` image. Use the dedicated gfx1151 image instead.

**Official image tag pattern** (nightly on Docker Hub [`rocm/sgl-dev`](https://hub.docker.com/r/rocm/sgl-dev/tags)):

```bash theme={null}
rocm/sgl-dev:*-gfx1151*
# example: rocm/sgl-dev:v0.5.20-rocm724-gfx1151-20260924
```

The image sets `SGLANG_USE_AITER=0` (see `docker/rocm-gfx1151.Dockerfile`). AITER kernels are CDNA/wave64-oriented and are unsafe as defaults on gfx115x.

**Host requirements**: expose `/dev/kfd` and `/dev/dri`, and add the container to the `video` and `render` groups. Strix Halo has no discrete VRAM — the GPU carves memory from system RAM (configure GTT / BIOS aperture as needed on the host).

```bash theme={null}
docker run -it --rm \
  --device=/dev/kfd --device=/dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc=host --shm-size 16g \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  rocm/sgl-dev:v0.5.20-rocm724-gfx1151-20260924 \
  bash
```

**Serving defaults on gfx1151**:

* LLM / SRT: `--attention-backend triton` (or rely on the gfx115 auto-default), `--num-gpus 1`
* Diffusion / DiT (e.g. Wan2.2): `--attention-backend torch_sdpa`, `--num-gpus 1`
* Keep `SGLANG_USE_AITER=0` (already set in the official image)


# Apple Silicon with Metal
Source: https://docs.sglang.io/docs/hardware-platforms/apple_metal



This document describes how to run the SGLang serving runtime on Apple Silicon
using [MLX](https://opensource.apple.com/projects/mlx/). SGLang Diffusion uses
PyTorch MPS instead; see its [installation guide](/docs/sglang-diffusion/installation#platform-specific-apple-mps).
If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).

## Prerequisites

The MLX runtime requires Apple Silicon with macOS 14 or newer, stable PyTorch
2.13.x, and stable MLX 0.32.0 or newer. The `srt_mps` extra installs PyTorch
2.13.0 and MLX 0.32.0 or newer; startup accepts stable PyTorch 2.13 patch
releases and newer stable MLX releases.

With `SGLANG_USE_MLX=1`, SGLang validates both framework versions and Metal
availability during argument initialization and stops before resolving or
downloading a model when the runtime is incompatible.

Building the optional native Metal kernels in `sgl-kernel` requires the Metal
shader compiler from the full Xcode application. The standalone Xcode Command
Line Tools are not sufficient. After installing Xcode, select it with:

```bash theme={null}
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
```

Verify the compiler with `xcrun -sdk macosx metal --version`.

## Install SGLang

You can install SGLang using one of the methods below.

### Install from Source

```bash theme={null}
# Use the default branch
git clone https://github.com/sgl-project/sglang.git
cd sglang

# Create and activate a virtual environment
uv venv -p 3.12 sglang-metal
source sglang-metal/bin/activate

# (Optional) Compile sgl-kernel
uv pip install --upgrade pip
uv run python/sglang/kernels/aot/setup_metal.py install

# Install sglang python package along with diffusion support
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
uv pip install -e "python[all_mps]"
```

## Launch of the Serving Engine

Launch the server with:

```bash theme={null}
SGLANG_USE_MLX=1 python -m sglang.launch_server \
  --model <MODEL_ID_OR_PATH> \
  --disable-cuda-graph \
  --host 0.0.0.0
```

**Key Parameters Explained:**

1. `SGLANG_USE_MLX=1` - Enables the use of MLX as the SGLang runtime backend (if disabled, SGLang will fall back to `torch.mps`, which has less support)
2. `--disable-cuda-graph` - Disables usage of CUDA graph, which is not relevant for Apple Metal.
3. `--disable-overlap-schedule` - Disables overlap scheduling (enabled/not present by default) achieved using MLX's `async_eval()`
4. `SGLANG_MLX_USE_CUSTOM_ROPE=1` - Enables the optional custom Metal RoPE kernel. It is disabled by default, so the MLX backend uses the standard RoPE path unless you opt in for A/B testing.
5. `SGLANG_MLX_FUSE_SWIGLU=1` - Enables the use of fused Swish-Gated Linear Unit kernel (disabled by default)
6. `SGLANG_MLX_CLEAR_CACHE_STEPS=256` - Sets the number of decode steps before clearing the MLX cache (256 by default)

## Quantization

The MLX backend supports two quantization paths on Apple Silicon:

1. **Pre-quantized HF repos.** Any `mlx-community/<model>-4bit` (or `-8bit`) repo loads directly through `mlx_lm.load(...)` — no extra flag needed.
   ```bash theme={null}
   SGLANG_USE_MLX=1 python -m sglang.launch_server \
     --model-path mlx-community/Qwen3-0.6B-4bit \
     --disable-cuda-graph
   ```
2. **On-the-fly quantization.** For any fp16 model, pass `--quantization mlx_q4` or `--quantization mlx_q8` to have sglang quantize the weights at load time via `mlx_lm.utils.quantize_model` (group size 64, the mlx-community default). The quantized weights stay in process memory; the on-disk model is untouched.
   ```bash theme={null}
   SGLANG_USE_MLX=1 python -m sglang.launch_server \
     --model-path Qwen/Qwen3-0.6B \
     --quantization mlx_q4 \
     --disable-cuda-graph
   ```
   Expected log line:
   ```
   Quantizing MLX model on-the-fly: bits=4 group_size=64 (preset=mlx_q4)
   Quantization complete in 0.13s — active mem: 1.11 GB -> 0.31 GB (71.9% reduction)
   ```
   The MLX backend silently ignores `--quantization mlx_q4` when the model is already quantized in its HF config (path 1), so the same flag is safe to pass either way.

## Benchmarking with Requests

`sglang.benchmark.one_batch` calls the synchronous prefill/decode methods directly without going through the scheduler and the overlap code path.

`sglang.benchmark.offline_throughput` can toggle overlap scheduling as it uses the scheduler and the overlap code path by using the flag `--disable-overlap-schedule`.

### Throughput Testing

Basic synchronous one batch throughput:

```bash theme={null}
SGLANG_USE_MLX=1 python -m sglang.benchmark.one_batch \
  --model-path <MODEL_ID_OR_PATH> \
  --disable-cuda-graph \
  --tp-size 1 \
  --batch-size 1 \
  --input-len 60 \
  --output-len 10
```

Synchronous offline throughput:

```bash theme={null}
SGLANG_USE_MLX=1 python -m sglang.benchmark.offline_throughput \
  --model-path <MODEL_ID_OR_PATH> \
  --disable-cuda-graph \
  --num-prompts 1 \
  --disable-overlap-schedule
```

Asynchronous offline throughput:

```bash theme={null}
SGLANG_USE_MLX=1 python -m sglang.benchmark.offline_throughput \
  --model-path <MODEL_ID_OR_PATH> \
  --disable-cuda-graph \
  --num-prompts 1
```


# Contribution Guide
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/contribution_guide



Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.

## Install SGLang from Source

### Prepare Environment

Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../getting-started/installation) to install the necessary dependencies. We recommend [using docker](../getting-started/installation#method-2-using-docker-image) to build the environment.

### Fork and clone the repository

**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.

```bash theme={null}
git clone https://github.com/<your_user_name>/sglang.git
# if you are using docker, the environment is already set up.
cd sglang
export PYTHONPATH=$PWD/python:$PYTHONPATH
```

## Format code with pre-commit

We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:

```bash theme={null}
pip3 install pre-commit
pre-commit install
pre-commit run --all-files
```

* **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
* **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
* Link checking with lychee is **enforced in CI**. By default, it is not blocking local commits.
* To run local link checks manually, use: `pre-commit run --hook-stage manual lychee --all-files`.

## Run and add tests

All NPU tests are end-to-end (E2E) and require launching a server with real model weights.
Tests live under [`test/registered/npu/`](https://github.com/sgl-project/sglang/tree/main/test/registered/npu), organized by model type and functionality:

```
ascend/
├── llm_models/          # Per-model inference accuracy
├── vlm_models/          # Vision-language models
├── embedding_models/    # Embedding model tests
├── rerank_models/       # Reranker model tests
├── reward_models/       # Reward model tests
├── interface/           # API correctness, function calling
├── basic_function/      # Cache, sampling, quantization, etc.
└── test_npu_memory_consumption.py
```

### Adding a test

See [`test_npu_sampling_backend.py`](https://github.com/sgl-project/sglang/tree/main/test/registered/npu/basic_function/backends/test_npu_sampling_backend.py) for a complete example. Key steps:

1. Place your test file in the appropriate directory under `test/registered/npu/`.
2. Extend `CustomTestCase` (from `sglang.test.test_utils`) for CI retry support.
3. Launch server with `popen_launch_server()` in `setUpClass` and clean up with `kill_process_tree()` in `tearDownClass`.
4. Register your test with `register_npu_ci()`:
   ```python theme={null}
   from sglang.test.ci.ci_register import register_npu_ci
   register_npu_ci(est_time=400, suite="stage-b-test-1-npu-a3", nightly=False)
   register_npu_ci(est_time=400, suite="nightly-1-npu-a3", nightly=True)
   ```

### Running tests locally

```bash theme={null}
pytest test/registered/npu/llm_models/test_npu_qwen3_0_6b.py -v
```

For detailed instructions, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).

### Register models for CI

If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps:

1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
2. Make sure your model is pre-cached on the CI server and is at the path "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models//".
   If this is not the case, use following command on CI server:

   ```bash theme={null}
   modelscope download \
     --model {your_model_repo}/{your_model} \
     --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
   ```

> Note: If you don’t have access to CI server, please ask maintainers ([zl19940307@163.com](mailto:zl19940307@163.com)) to download your model.

3. Add model to `python/sglang/test/ascend/test_ascend_utils.py` (use docker `"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"` path).

## Write documentation

We recommend new contributors start by writing documentation, which helps you quickly understand SGLang codebase.
For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).

## Test the accuracy

If your code changes the model output, run the accuracy tests. A quick sanity check is GSM8K through sgl-eval.

```bash theme={null}
# Launch a server
python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct

# Evaluate. --base-url must point at the server launched above.
# The default SGLang server port is 30000; change it if you launched
# the server with a different --port.
sgl-eval run gsm8k \
  --base-url http://localhost:30000/v1 \
  --num-examples 200
```

Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.

GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
You can find additional accuracy eval examples in:

* [test\_eval\_accuracy\_large.py](https://github.com/sgl-project/sglang/blob/main/test/manual/eval/test_eval_accuracy_large.py)

## Benchmark the speed

Refer to [Benchmark and Profiling](../../../developer_guide/benchmark_and_profiling).

## Requesting a review for merge

You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
Then your PR can be merged.

## How to Trigger CI Tests

We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
Users with permission are listed in the [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)

**PR authors** can always use `/rerun-failed-ci` on their own PRs, even if they are not listed in `CI_PERMISSIONS.json`.

For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:

* `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
* `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, skipped, cancelled, or timed out**.
* `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
* `/rerun-test <test-spec> [<test-spec> ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `<test-spec>` is pytest-style `<file>::<TestClass>[.<test_method>]` (the `::TestClass` and `.<test_method>` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once).

If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).

To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).

If you don’t have permission, please ask maintainers to trigger CI for you.

### CI rate limits

Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.

Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:

```yaml theme={null}
cool-down-minutes:
  description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting"
  type: number
  default: 120
```

Users listed in [CI\_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.

## Code style guidance

* Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
* Minimize device synchronization. Reduce expensive CPU-NPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
* Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
  * A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible.
* Make functions as pure as possible. Avoid in-place modification of arguments.
* Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_pp_mixin.py`)
* In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file.
* Keep tests run fast.
  * If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
  * If a single job in a github workflow runs longer than 30 min, split it into smaller jobs/steps.
  * Reuse server launches in your unit tests to make tests run faster.
* Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python’s [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead.
* When supporting new hardware or features, follow these guidelines:
  * Do not drastically change existing code.
  * Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_npu.py`).
  * If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.

## How to update sgl-kernel-npu

Sgl-kernel-npu is the separate kernel package for Ascend NPU, containing both Ascend C and Triton operators. It is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository.

For detailed guidance on developing and integrating operators (Ascend C directory structure, PyTorch op registration, build, test, and code style), see the [Ascend NPU Operator Development Guide](./operator_development).

### Multi-PR workflow

Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow:

1. **Submit sgl-kernel-npu PR**: Add or modify operators in the sgl-kernel-npu repository following the operator development guide. Ensure all tests pass.
2. **Bump sgl-kernel-npu version**: Update the version number. Merging triggers an automatic PyPI release. If not urgent, wait for a regular release (typically within one week).
3. **Reference the new version in SGLang**:
   * Update the `SGLANG_KERNEL_NPU_TAG` argument in [`docker/npu.Dockerfile`](https://github.com/sgl-project/sglang/blob/main/docker/npu.Dockerfile) to the new sgl-kernel-npu release tag.
   * Use the new operator in SGLang code.

## Tips for newcomers

If you want to contribute but don’t have a specific idea in mind, pick issues labeled ["good first issue" or "help wanted"](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase.

Also check out the following materials as startup guide:

* [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang.
* [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
* [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance.

If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).

Thank you for your interest in SGLang. Happy coding!


# Operator Development
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/operator_development

How to develop custom operators (Ascend C / Triton) for Ascend NPU and integrate them into the SGLang inference engine.

## Overview

[SGL-Kernel-NPU](https://github.com/sgl-project/sgl-kernel-npu) is the official
operator library provided by the SGLang framework for Ascend NPU. It includes
two types of operator implementations:

1. **Ascend C operators**: High-performance C++ kernels written in Ascend C,
   compiled into `libsgl_kernel_npu.so`, and registered through PyTorch's custom
   operator mechanism (`TORCH_LIBRARY_FRAGMENT`). Called in SGLang via
   `torch.ops.npu.<op_name>()`.
2. **Triton operators**: Python kernels written in Triton, adapted for Ascend
   NPU. Called directly via `from sgl_kernel_npu.xxx import ...`.

When SGLang detects an NPU device, it automatically loads `sgl_kernel_npu` and
uses its operators in place of GPU counterparts, providing optimized inference
on Ascend hardware.

## Directory Structure

> [!NOTE]
>
> The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g., `retrive_index`,
> `retrive_next_token`, `retrive_next_sibling`) in this guide match the spelling used in the
> upstream [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository and are kept verbatim for consistency.

```text theme={null}
sgl-kernel-npu/
├── csrc/                          # Ascend C operator C++ sources
│   ├── CMakeLists.txt             # Build configuration
│   ├── pytorch_extensions.cpp     # PyTorch op registration (core integration file)
│   └── <op_name>/                 # One directory per operator
│       ├── op_host/               # Host-side code (validation, tiling, launch)
│       │   ├── <op_name>.cpp
│       │   └── tiling/            # Optional: tiling data
│       └── op_kernel/             # Device-side code (Ascend C kernel on AI Core)
│           └── <op_name>_kernel.cpp
├── include/
│   └── sgl_kenel_npu_ops.h        # C++ interface declarations
├── python/
│   └── sgl_kernel_npu/
│       └── sgl_kernel_npu/
│           ├── __init__.py        # Loads libsgl_kernel_npu.so
│           ├── attention/         # Triton attention kernels
│           ├── norm/              # Triton normalization kernels
│           ├── activation/        # Triton activation kernels
│           ├── fla/               # Triton linear attention kernels
│           ├── mamba/             # Triton Mamba kernels
│           ├── moe/               # Triton MoE kernels
│           └── sample/            # Triton speculative decoding kernels
├── tests/
│   └── python/sgl_kernel_npu/     # One test file per operator
├── build.sh                       # Build script
└── CMakeLists.txt                 # Root CMake configuration
```

## Developing Ascend C Operators

A complete Ascend C operator consists of two parts:

* **Device part**: Kernel code running on the NPU AI Core, responsible for actual
  computation. Written using the Ascend C API.
* **Host part**: Code running on the CPU, responsible for parameter validation,
  data pre-processing, tiling, and kernel launch.

We recommend starting with the
[helloworld](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld)
example, a simple operator that performs element-wise addition on two tensors.

### Step 1: Create the operator directory and files

Create a new operator directory under `csrc/`, following the `op_host/` +
`op_kernel/` structure:

```text theme={null}
csrc/<op_name>/
├── op_host/
│   └── <op_name>.cpp
└── op_kernel/
    └── <op_name>_kernel.cpp
```

### Step 2: Write the Device-side Kernel (op\_kernel)

Device-side code runs on AI Core and follows the Ascend C programming model. The
core structure is a class with `Init()` and `Process()` methods, plus an
`extern "C"` entry function.

Using helloworld as an example:

```cpp theme={null}
// csrc/helloworld/op_kernel/kernel_helloworld.cpp
#include "kernel_operator.h"

constexpr int32_t BUFFER_NUM = 2;

class KernalHelloworld {
public:
    __aicore__ inline KernalHelloworld() {}

    __aicore__ inline void Init(GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength)
    {
        // Compute workload for current block
        this->blockLength = totalLength / AscendC::GetBlockNum();
        this->tileNum = 8;
        this->tileLength = this->blockLength / this->tileNum / BUFFER_NUM;

        // Set global memory buffers
        xGm.SetGlobalBuffer((__gm__ half *)x + this->blockLength * AscendC::GetBlockIdx(), this->blockLength);
        yGm.SetGlobalBuffer((__gm__ half *)y + this->blockLength * AscendC::GetBlockIdx(), this->blockLength);
        zGm.SetGlobalBuffer((__gm__ half *)z + this->blockLength * AscendC::GetBlockIdx(), this->blockLength);

        // Initialize pipeline queues
        pipe.InitBuffer(inQueueX, BUFFER_NUM, this->tileLength * sizeof(half));
        pipe.InitBuffer(inQueueY, BUFFER_NUM, this->tileLength * sizeof(half));
        pipe.InitBuffer(outQueueZ, BUFFER_NUM, this->tileLength * sizeof(half));
    }

    __aicore__ inline void Process()
    {
        int32_t loopCount = this->tileNum * BUFFER_NUM;
        for (int32_t i = 0; i < loopCount; i++) {
            CopyIn(i);    // Move data from Global Memory to Local Memory
            Compute(i);   // Compute on Local Memory
            CopyOut(i);   // Move results back to Global Memory
        }
    }

private:
    __aicore__ inline void CopyIn(int32_t progress) { /* data copy-in... */ }
    __aicore__ inline void Compute(int32_t progress) { /* core computation... */ }
    __aicore__ inline void CopyOut(int32_t progress) { /* data copy-out... */ }

private:
    AscendC::TPipe pipe;
    AscendC::TQue<AscendC::TPosition::VECIN, BUFFER_NUM> inQueueX, inQueueY;
    AscendC::TQue<AscendC::TPosition::VECOUT, BUFFER_NUM> outQueueZ;
    AscendC::GlobalTensor<half> xGm, yGm, zGm;
    uint32_t blockLength, tileNum, tileLength;
};

// Entry function: the compile tool auto-generates aclrtlaunch_<op_name>.h from this name
extern "C" __global__ __aicore__ void helloworld(
    GM_ADDR x, GM_ADDR y, GM_ADDR z, uint32_t totalLength)
{
    KernalHelloworld op;
    op.Init(x, y, z, totalLength);
    op.Process();
}
```

**Key points:**

* Class methods must be marked with `__aicore__`, indicating they run on AI Core.
* Use `AscendC::TPipe` + `AscendC::TQue` to build a pipeline that overlaps data
  movement and computation.
* The entry function must be declared `extern "C" __global__ __aicore__`. The
  compile tool generates a host-callable launch header
  `aclrtlaunch_<func_name>.h` from the function name.
* Simple operators (e.g., helloworld, cache\_assign, lora) do not need extra
  workspace memory. Complex operators (e.g., mla\_preprocess, alloc\_extend,
  build\_tree) require temporary workspace memory and are compiled separately in
  `CMakeLists.txt`.

For more in-depth Ascend C programming knowledge, refer to the
[Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html).

### Step 3: Write the Host-side Code (op\_host)

Host-side code is responsible for passing PyTorch Tensors to the kernel and
launching it. The key macro is `EXEC_KERNEL_CMD` (located in
`csrc/utils/torch_helper.h`).

```cpp theme={null}
// csrc/helloworld/op_host/helloworld.cpp
#include "defines.h"                   // Provides HOST_API macro
#include "torch_helper.h"              // Provides EXEC_KERNEL_CMD macro
#include "aclrtlaunch_helloworld.h"    // Auto-generated by compile tool

namespace sglang {
namespace npu_kernel {

HOST_API at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y)
{
    // Create output tensor
    at::Tensor z = at::empty_like(x);

    // Define block count
    uint32_t blockDim = 8;

    // Compute total element count
    uint32_t totalLength = 1;
    for (uint32_t size : x.sizes()) {
        totalLength *= size;
    }

    // Launch kernel via EXEC_KERNEL_CMD macro
    EXEC_KERNEL_CMD(helloworld, blockDim, x, y, z, totalLength);
    return z;
}

} // namespace npu_kernel
} // namespace sglang
```

**Key points:**

* The namespace must be `sglang::npu_kernel`.
* Function signatures follow the pattern
  `at::Tensor <op_name>(const at::Tensor &input, ...)`.
* For operators with multiple outputs, use
  `std::tuple<at::Tensor, at::Tensor, ...>` or non-const reference parameters.

### Step 4: Declare the C++ Interface (include/sgl\_kenel\_npu\_ops.h)

Add the operator function declaration in `include/sgl_kenel_npu_ops.h`:

```cpp theme={null}
// include/sgl_kenel_npu_ops.h
namespace sglang {
namespace npu_kernel {

at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y);

} // namespace npu_kernel
} // namespace sglang
```

### Step 5: Register the PyTorch Custom Operator (pytorch\_extensions.cpp)

Register the operator in `csrc/pytorch_extensions.cpp` in two steps: define the
schema and bind the implementation.

```cpp theme={null}
// csrc/pytorch_extensions.cpp
namespace {

// 1. Define operator schema (used by torch.compile, etc.)
TORCH_LIBRARY_FRAGMENT(npu, m)
{
    m.def("helloworld(Tensor x, Tensor y) -> Tensor");
    // ... other operator schemas ...
}

// 2. Bind implementation for the PrivateUse1 device (i.e., NPU)
TORCH_LIBRARY_IMPL(npu, PrivateUse1, m)
{
    m.impl("helloworld", TORCH_FN(sglang::npu_kernel::helloworld));
    // ... other operator implementations ...
}

} // namespace
```

**Schema conventions:**

* The namespace is fixed to `npu`. In SGLang, operators are called via
  `torch.ops.npu.<op_name>()`.
* Output tensor parameters use the `Tensor(a!)` mutating annotation.
* Optional parameters use the `Tensor?` annotation, with `c10::optional<T>`
  handling in the impl.
* For detailed schema syntax, see the
  [PyTorch Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func).

**Implementation binding rules:**

* The device name is fixed to `PrivateUse1` (PyTorch NPU backend identifier).
* Use the `TORCH_FN` macro to bind to the implementation function.
* For complex operators with optional parameters, use lambda expressions to
  unpack the arguments.

### Step 6: Update the Build Configuration (csrc/CMakeLists.txt)

Add the new operator's source files to `csrc/CMakeLists.txt`:

**For operators not requiring workspace** (simple operators), add kernel source
to `no_workspace_kernel`:

```cmake theme={null}
ascendc_library(no_workspace_kernel STATIC
    # ... existing kernel files ...
    ${PROJECT_OP_SRC_BASE}/<op_name>/op_kernel/<op_name>_kernel.cpp
)
```

**For operators requiring workspace** (complex operators), add kernel source to
`workspace_kernel` with the `-DHAVE_WORKSPACE -DHAVE_TILING` compile flags:

```cmake theme={null}
ascendc_library(workspace_kernel STATIC
    # ... existing kernel files ...
    ${PROJECT_OP_SRC_BASE}/<op_name>/op_kernel/<op_name>_kernel.cpp
)
```

**Add host source files to `OP_SRCS`:**

```cmake theme={null}
FILE(GLOB OP_SRCS
    # ... existing host files ...
    ${PROJECT_OP_SRC_BASE}/<op_name>/op_host/<op_name>.cpp
)
```

### Step 7: Build

Build following the steps in the
[python/sgl\_kernel\_npu/README.md](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md):

```bash theme={null}
cd sgl-kernel-npu

# Build all modules
bash build.sh

# Install the sgl_kernel_npu wheel
pip install output/sgl_kernel_npu*.whl
```

The compiled `libsgl_kernel_npu.so` is copied into
`python/sgl_kernel_npu/sgl_kernel_npu/lib/` and loaded by the Python package.

## Developing Triton Operators

Triton operators are located under `python/sgl_kernel_npu/sgl_kernel_npu/`,
organized by function category:

```text theme={null}
python/sgl_kernel_npu/sgl_kernel_npu/
├── attention/     # Attention (decode_attention, sinks_attention)
├── norm/          # Normalization (rmsnorm, fused_qk_norm, l1_norm)
├── activation/    # Activation (swiglu_oai, swiglu_quant)
├── fla/           # Linear attention (chunk, cumsum, wy_fast)
├── mamba/         # Mamba-related (causal_conv1d, state_update)
├── moe/           # MoE-related (mul_add, zero_experts)
└── sample/        # Speculative decoding (verify_tree_greedy)
```

**Development steps:**

1. Create a new `.py` file in the appropriate category subdirectory.
2. Write the kernel using the Triton language, using existing operators in the
   same directory as templates.
3. Export functions in the corresponding `__init__.py` if needed.
4. Write tests under `tests/python/sgl_kernel_npu/`.

**Note:** Many Triton operators are adapted from SGLang's GPU Triton kernels
(e.g., comments in `fla/utils.py` note the original source). Pay special
attention to differences between NPU and GPU when adapting.

## Integrating Operators into SGLang

### Ascend C Operator Integration

After completing the [Steps 1-7](#developing-ascend-c-operators) above (writing
the kernel, registering the torch op, building), and installing the
`sgl-kernel-npu` wheel, call the operator in SGLang as follows:

```python theme={null}
import sgl_kernel_npu  # Loading the library auto-triggers libsgl_kernel_npu.so loading

# Call the operator
result = torch.ops.npu.helloworld(x, y)
```

Real-world usage in SGLang (from `sglang/srt/speculative/eagle_utils.py`):

```python theme={null}
torch.ops.npu.build_tree_kernel_efficient(
    parent_list, selected_index, verified_seq_len, tree_mask,
    positions, retrive_index, retrive_next_token,
    retrive_next_sibling, topk, depth, draft_token_num, tree_mask_mode
)
```

### Triton Operator Integration

Import and call directly via Python:

```python theme={null}
from sgl_kernel_npu.attention.decode_attention import decode_attention_fwd
from sgl_kernel_npu.norm.rmsnorm_bias import rmsnorm_bias
from sgl_kernel_npu.mamba.causal_conv1d import causal_conv1d_fwd

# Direct function call
output = decode_attention_fwd(q, k, v, ...)
```

Real-world usage in SGLang (from `sglang/srt/models/llama.py`):

```python theme={null}
from sgl_kernel_npu.norm.split_qkv_rmsnorm_rope import split_qkv_rmsnorm_rope
```

### sgl-kernel-npu Wheel Update Process

Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates
require a multi-PR workflow:

1. **Submit sgl-kernel-npu PR**: Add/modify operators in the sgl-kernel-npu
   repository, ensuring all tests pass.
2. **Bump sgl-kernel-npu version**: Update the version number in sgl-kernel-npu.
   Merging triggers an automatic PyPI release.
3. **Reference the new version in SGLang**:
   * Update the `sgl-kernel-npu` version requirement in SGLang's
     `python/pyproject.toml`.
   * Use the new operator in SGLang code.

If not urgent, you can wait for a regular release (typically within one week).

## Writing Unit Tests

Each operator needs a corresponding unit test under
`tests/python/sgl_kernel_npu/`, using Python's `unittest` framework.

Test file naming convention: `test_<op_name>.py`

```python theme={null}
# tests/python/sgl_kernel_npu/test_helloworld.py
import unittest
import torch
import sgl_kernel_npu

class TestHelloworld(unittest.TestCase):
    def test_helloworld_basic(self):
        x = torch.randn(2048, dtype=torch.float16, device="npu")
        y = torch.randn(2048, dtype=torch.float16, device="npu")

        z = torch.ops.npu.helloworld(x, y)
        expected = x + y

        torch.testing.assert_close(z, expected)

if __name__ == "__main__":
    unittest.main()
```

Run tests:

```bash theme={null}
python tests/python/sgl_kernel_npu/test_helloworld.py
```

**Testing checklist:**

* Cover typical input shapes (power-of-2 sizes and non-standard sizes).
* Cover different data types (bf16 / fp16, etc.).
* For operators with in-place behavior, verify correctness of output tensors.
* Compare against PyTorch native computation to verify accuracy.

## Code Style

### Pre-commit Checks

sgl-kernel-npu uses pre-commit for consistent code style:

```bash theme={null}
pip3 install pre-commit
cd sgl-kernel-npu
pre-commit install
pre-commit run --all-files
```

**Note:** If `pre-commit run --all-files` fails the first time, run it again to
ensure all lint errors are auto-fixed. All code must pass checks before
submitting a PR.

### C++ Code Style

* Use the C++17 standard.
* Place all operator implementations under the `sglang::npu_kernel` namespace.
* Follow existing code style; format using `.clang-format`.
* Use `TORCH_CHECK` for error checking (not the standard GE `OP_ADD` macros).
* Do not include unnecessary GE registration code (e.g., `OP_ADD()` macros).

### Python Code Style

* Follow PEP 8.
* Use snake\_case for file and function names.
* When adapting from SGLang GPU code, note the original source in the file
  header.

### General Principles

* Avoid code duplication: extract shared functions for any repeated code blocks
  over 5 lines.
* Minimize device synchronization: reduce CPU-NPU sync operations like
  `tensor.item()` or `tensor.cpu()`.
* Keep functions pure: avoid in-place argument modification.
* Keep files concise: split files exceeding 2,000 lines.

## Submitting a PR

1. **Fork the repo**: Fork
   [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) on GitHub,
   then clone locally.

2. **Create a branch**: Create a new branch from `main`, e.g.,
   `feature/add-my-op`.

3. **Develop and test**: Develop the operator and write tests following the
   steps above. Ensure all tests pass.

4. **Run pre-commit**: Ensure code formatting compliance.

5. **Commit and push**:

   ```bash theme={null}
   git add .
   git commit -m "feat: add <op_name> operator"
   git push origin feature/add-my-op
   ```

6. **Create a PR**: Open a Pull Request on GitHub from your branch to
   `sgl-project/sgl-kernel-npu:main`.

7. **Wait for CI and review**: CI checks include linting, compilation, and
   operator tests. After passing, wait for maintainer review and merge.

## References

* [SGL-Kernel-NPU Official Repository](https://github.com/sgl-project/sgl-kernel-npu)
* [SGL-Kernel-NPU Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md)
* [Ascend C Kernel Development Guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850alpha001/opdevg/Ascendcopdevg/atlas_ascendc_10_0001.html)
* [PyTorch Custom Ops Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func)
* [helloworld Example Operator](https://github.com/sgl-project/sgl-kernel-npu/tree/main/csrc/helloworld)
* [SGLang Contribution Guide](/docs/hardware-platforms/ascend-npus/development/contribution_guide)


# Operator Performance Optimization
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/operator_performance_optimizing



## Performance\_benchmark

### Obtaining Performance Data

Before optimizing the performance, you need to obtain accurate performance data, understand the current performance status, and analyze the next optimization direction based on the performance status. MindStudio provides realistic methods for testing the performance of Triton operators.

#### Device-end

The msProf tool is used to collect and analyze key performance indicators of operators running on the Ascend AI Processor. You can use the output performance data to quickly locate the software and hardware performance bottlenecks of operators and improve the efficiency of operator performance analysis.

```bash theme={null}
msprof op python3 test_xxxxx.py
```

The following is a case of using msprof for data collection.

| Attribute | Value |
| - | - |
| Name | DequantSwigluQuant\_int32\_high\_performance\_100000000 |
| Type | DequantSwigluQuant |
| OP State | static |
| Accelerator Core | AI\_VECTOR\_CORE |
| Start Time(us) | 1774489226717521.715 |
| Duration(us) | 102.824 |
| Wait Time(us) | 0 |
| Block Dim | 36 |
| Mix Block Dim | 0 |
| HF32 Eligible | NO |
| Input Shapes | 163840,1024;128,1024;163840;;;;128 |
| Input Data Types | INT32;FLOAT;FLOAT;DT\_UNDEFINED;DT\_UNDEFINED;DT\_UNDEFINED;INT64 |
| Input Formats | ND;ND;ND;NULL;NULL;NULL;ND |
| Output Shapes | 163840,512;163840 |
| Output Data Types | INT8;FLOAT |
| Output Formats | ND;ND |
| Context ID | N/A |
| aicore\_time(us) | 0 |
| aic\_total\_cycles | 0 |
| aic\_mac\_time(us) | 0 |
| aic\_mac\_ratio | 0 |
| aic\_scalar\_time(us) | 0 |
| aic\_scalar\_ratio | 0 |
| aic\_mte1\_time(us) | 0 |
| aic\_mte1\_ratio | 0 |
| aic\_mte2\_time(us) | 0 |
| aic\_mte2\_ratio | 0 |
| aic\_fixpipe\_time(us) | 0 |
| aic\_fixpipe\_ratio | 0 |
| aic\_icache\_miss\_rate | 0 |
| aiv\_time(us) | 59.128 |
| aiv\_total\_cycles | 3512188 |
| aiv\_vec\_time(us) | 36.708 |
| aiv\_vec\_ratio | 0.621 |
| aiv\_scalar\_time(us) | 41.403 |
| aiv\_scalar\_ratio | 0.7 |
| aiv\_mte2\_time(us) | 11.975 |
| aiv\_mte2\_ratio | 0.203 |
| aiv\_mte3\_time(us) | 9.738 |
| aiv\_mte3\_ratio | 0.165 |
| aiv\_icache\_miss\_rate | 0.005 |
| cube\_utilization(%) | 0 |

Below is the field-by-field breakdown of the operator performance record, aligned with the official specification.

### 1. Basic Identification Fields

| Field | Value | Definition (per official docs) |
| - | - | - |
| Name | DequantSwigluQuant\_int32\_high\_performance\_100000000 | **Op Name**: Name of the fused operator (dequantization + SwiGLU activation + quantization), with an int32 high-performance implementation suffix. |
| Type | DequantSwigluQuant | **OP Type**: Functional category of the operator. |
| OP State | static | **OP State**: Indicates a static operator whose shape and scheduling logic are determined at compile time. |
| Accelerator Core | AI\_VECTOR\_CORE | **Task Type**: The operator runs on the AI Vector Core; other common types include AI\_CORE (matrix computation core) and AI\_CPU. |

### 2. Timing & Scheduling Fields

| Field | Value | Definition (per official docs) |
| - | - | - |
| Start Time(us) | 1774489226717521.715 | **Task Start Time**: Absolute start timestamp of the operator task on the device side, in microseconds. |
| Duration(us) | 102.824 | **Task Duration**: End-to-end total latency of the operator, including dispatch time, accelerator execution time, and completion response time, in microseconds. |
| Wait Time(us) | 0 | **Task Wait Time**: Time interval between the end of the previous task and the start of the current task. A value of 0 means no idle wait between task dispatches. |

### 3. Core Configuration & Precision Fields

| Field | Value | Definition (per official docs) |
| - | - | - |
| Block Dim | 36 | **Block Num**: Number of parallel thread blocks for the operator task, corresponding to Block Dim in the SIMT programming model. One AI Vector Core executes only one thread block at a time, so this value reflects the scale of occupied parallel compute resources. |
| Mix Block Dim | 0 | **Mix Block Num**: Number of blocks on the secondary accelerator if the operator runs on both AI Core and Vector Core. A value of 0 means the operator runs exclusively on AI\_VECTOR\_CORE with no hybrid core scheduling. |
| HF32 Eligible | NO | **HF32 Eligible**: Indicates whether the HF32 high-precision floating-point format is enabled; `NO` means it is not used. This field is reported only at the `--task-time=l1` collection level. |

### 4. Input & Output Information

| Field | Value | Definition (per official docs) |
| - | - | - |
| Input Shapes | 163840,1024;128,1024;163840;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. |
| Input Data Types | INT32;FLOAT;FLOAT;DT\_UNDEFINED;DT\_UNDEFINED;DT\_UNDEFINED;INT64 | **Input Data Types**: Data types of inputs, in the same order as input shapes. |
| Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | **Input Formats**: Memory layout of inputs; `ND` stands for N-dimensional tensor format, `NULL` corresponds to scalar/undefined inputs. |
| Output Shapes | 163840,512;163840 | **Output Shapes**: Dimensions of two output tensors, separated by semicolons: `[163840,512]` and `[163840]`. |
| Output Data Types | INT8;FLOAT | **Output Data Types**: Output 1 is INT8 (quantized result), output 2 is FLOAT. |
| Output Formats | ND;ND | **Output Formats**: Both outputs use standard ND layout. |
| Context ID | N/A | **Context ID**: Identifier for sub-tasks at Sub Task granularity; N/A means no sub-task splitting for this operator. |

### 5. AI Core Performance Metrics (aic\_\* series)

| Field | Value | Definition (per official docs) |
| - | - | - |
| aicore\_time(us) | 0 | Theoretical execution time on AI Core, in microseconds. |
| aic\_total\_cycles | 0 | Total execution cycles on AI Core. |
| aic\_mac\_time(us) / aic\_mac\_ratio | 0 / 0 | Latency and cycle ratio of cube (matrix multiplication) instructions. |
| aic\_scalar\_time(us) / aic\_scalar\_ratio | 0 / 0 | Latency and cycle ratio of scalar instructions. |
| aic\_mte1\_time(us) / aic\_mte1\_ratio | 0 / 0 | Latency and cycle ratio of L1→L0A/L0B data move instructions. |
| aic\_mte2\_time(us) / aic\_mte2\_ratio | 0 / 0 | Latency and cycle ratio of DDR→AICORE read move instructions. |
| aic\_fixpipe\_time(us) / aic\_fixpipe\_ratio | 0 / 0 | Latency and cycle ratio of L0C→OUT/L1 move instructions. |
| aic\_icache\_miss\_rate | 0 | Instruction cache miss rate of AI Core. |

### 6. AI Vector Core Performance Metrics (aiv\_\* series)

| Field | Value | Definition & Interpretation |
| - | - | - |
| aiv\_time(us) | 59.128 | **aiv\_time**: Theoretical execution time on Vector Core under ideal conditions (all blocks scheduled simultaneously with equal duration). In practice, this value is slightly smaller than real execution time due to staggered block startup. |
| aiv\_total\_cycles | 3512188 | **aiv\_total\_cycles**: Total cycles executed on Vector Core, summed across all blocks. |
| aiv\_vec\_time(us) / aiv\_vec\_ratio | 36.708 / 0.621 | Latency (us) and cycle ratio (62.1%) of vector computation instructions. Vector operations are the core compute workload of this operator. |
| aiv\_scalar\_time(us) / aiv\_scalar\_ratio | 41.403 / 0.7 | Latency (us) and cycle ratio (70%) of scalar instructions. The sum exceeds 100% because scalar and vector pipelines run in parallel with independent cycle counters. |
| aiv\_mte2\_time(us) / aiv\_mte2\_ratio | 11.975 / 0.203 | Latency and cycle ratio (20.3%) of read memory move instructions (DDR/on-chip memory → Vector Core). |
| aiv\_mte3\_time(us) / aiv\_mte3\_ratio | 9.738 / 0.165 | Latency and cycle ratio (16.5%) of write memory move instructions (Vector Core → DDR/on-chip memory). |
| aiv\_icache\_miss\_rate | 0.005 | Vector Core instruction cache miss rate of 0.5%, extremely low and indicates efficient instruction fetch. |

### 7. Utilization Metrics

| Field | Value | Definition (per official docs) |
| - | - | - |
| cube\_utilization(%) | 0 | **cube\_utilization**: Utilization rate of the matrix multiplication unit. The value is 0 because the operator is purely vector-based. |

## Optimization

### Specification

#### 1. Ascend core compute units

* AI Core: the core that actually performs matrix/vector computation
* Vector Unit: responsible for SIMD computation (similar to CUDA Core)
* Scalar Unit: responsible for control/loop
* L0/L1/L2 cache: The smaller the size, the faster the speed. L0 is only 64KB, L1 is 256KB, and L2 is shared.

#### 2. Ascend memory hierarchy (from fastest to slowest)

* Register → Fastest
* L0/L1 cache → Very fast
* On-chip cache (L2) → Fast
* DDR (host memory) → Slowest

### 3. Characteristics of Ascend instructions

Good at accessing large contiguous memory blocks

Dislikes discrete access, stride access, and random access

Must be 128-bit/256-bit aligned

Must be vectorized.

### Tips

1. A2/A3 Series products usually have only 40 or 48 vector cores. If the number of grids exceeds 40 or 48 vector cores, the grids will be delivered in a queue, resulting in a long waiting time. Therefore, the number of cores for high-performance implementation does not exceed the number of vector cores.
2. Try to use up all the UB as much as possible. Move a large block size at a time to ensure that the bound is in the MTE. No Redundant Copy.
3. If the offset is a negative number, the current triton-ascend considers it as a discrete memory access scenario. As a result, the performance severely deteriorates, and the data is read from the entire DMA block instead of being read in scalar mode.
4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation.
5. Use Double Buffer, parallelizes computation and data transfer. While computing one block of data, another block of data is being transferred to L1.
6. If hostbound behavior is severe, core binding can be used to address it.


# How to Support New Models
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/development/support_new_models



This document explains how to add support for new language models and multimodal large language models (MLLMs) in
SGLang. It also covers how to test new models and register external implementations.

## How to Support a New Language Model

To support a new model in SGLang, you only need to add a single file under
the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn
from existing model implementations and create a new file for your model. For most models, you should be able to find a
similar model to start with (e.g., starting from Llama). Also refer to how
to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang)

NPU adaptations are embedded in existing model files (e.g., `llama.py`, `qwen3_vl.py`) through `_is_npu` conditional
branches. The NPU hardware backend lives at `sglang/srt/hardware_backend/npu/`. Some ops may need to use `torch_npu`
APIs in place of CUDA equivalents.

## How to Support a New Multimodal Large Language Model

To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the
standard LLM support:

1. **Register your new model as multimodal**:
   Extend `is_multimodal_model`
   in [model\_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561)
   to return `True` for your model.

2. **Register a new chat-template**:
   Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py) and the corresponding matching function.

3. **Multimodal Data Processor**:
   Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your
   model’s dedicated processor.
   See [Multimodal Processors](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors)
   for more details.

4. **Handle Multimodal Tokens**:
   Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be
   expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data
   with `RadixAttention`.

5. **Handle Image Feature Extraction**:
   Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model.

6. **Adapt to Vision Attention**:
   Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.

You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or
other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs.

> [!TIP]
>
> On Ascend NPU, ensure vision processors and image feature extraction are compatible with the `torch_npu` backend.
> Refer to `vit_npu_graph_runner.py` under `hardware_backend/npu/graph_runner/` and `qwen_vl_processor.py` under
> `hardware_backend/npu/modules/` for NPU vision adaptation patterns.

## Testing and Debugging

Please note all your testing and benchmarking results in PR description.

### Benchmark

* **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md)
  to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than
  that from HF Transformer run. Similarly, follow the [benchmark and profiling guide](/docs/developer_guide/benchmark_and_profiling)
  to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer).
* **(Optional) Other evals**: If you ran other evals, please note the results in PR description.

> [!TIP]
>
> For NPU-adapted models: add the corresponding test under `test/registered/npu/` and verify correctness on Ascend NPU
> hardware; run benchmarks on the NPU device and report performance metrics (TTFT, throughput), comparing against SGLang
> GPU results as the primary baseline. Fall back to HF Transformer comparison when no GPU adaptation is available.

## Port a Model from vLLM to SGLang

The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable
resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models
from vLLM to SGLang.

To port a model from vLLM to SGLang:

* Compare these two files for guidance:
  * [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py)
  * [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py)
* The major differences include:
  * **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`).
  * **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.**
  * **(For multimodal models) Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.**
  * **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers.
  * **Change the `forward()` functions** to accept a `forward_batch: ForwardBatch` argument carrying the full batch
    state (positions, `input_ids`, KV cache indices, sampling info, etc.). SGLang’s top-level model `forward()` is the
    entry point and returns the final `LogitsProcessorOutput` directly — it does both the backbone forward and the
    logits computation, unlike vLLM which splits these into `forward()` + `compute_logits()`.
  * **Add `EntryClass`** at the end.
  * **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components.
  * **For Ascend NPU**: Reference existing NPU-adapted models (e.g., `llama.py`, `deepseek_v2.py`) for NPU-specific
    patterns, such as replacing CUDA kernels with `torch_npu` equivalents. The NPU backend is at
    `sglang/srt/hardware_backend/npu/`.

Note: make sure you add your new model to the supported models list in the
[supported models documentation](https://github.com/sgl-project/sglang/blob/main/docs/docs/hardware-platforms/ascend-npus/reference/support_models.mdx).

## Registering an External Model Implementation

In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server.
This allows you to integrate your model without modifying the source code.

For example:

```python Register Model theme={null}
from sglang.srt.models.registry import ModelRegistry
from sglang.srt.entrypoints.http_server import launch_server

# For a single model, add it to the registry:
ModelRegistry.models[model_name] = model_class

# For multiple models, you can imitate the import_model_classes() function:
from functools import lru_cache

@lru_cache()
def import_new_model_classes():
    model_arch_name_to_cls = {}
    # Populate model_arch_name_to_cls with your new model classes.
    ...
    return model_arch_name_to_cls

ModelRegistry.models.update(import_new_model_classes())

# Launch the server with your server arguments:
launch_server(server_args)
```

## Example: Implementing and Serving a Llama Wrapper Model

Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](/docs/basic_usage/offline_engine_api).

### Implementing Our Model

To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit.

Let's start by defining our model in a file called `llama_wrapper.py`.
The first step is to import the necessary libraries from SRT, which is SGLang's internal backend.

```python Example theme={null}
# In the file `llama_wrapper.py`

import torch
from transformers import LlamaConfig
from typing import Optional
from sglang.srt.layers.logits_processor import LogitsProcessorOutput
from sglang.srt.layers.quantization.base_config import QuantizationConfig
from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors

from sglang.srt.models.llama import LlamaForCausalLM
```

Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`.
Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219).
Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us.

```python Class Definition theme={null}
class LlamaWrapper(LlamaForCausalLM):
    def __init__(
        self,
        config: LlamaConfig,
        quant_config: Optional[QuantizationConfig] = None,
        prefix: str = "",
    ) -> None:
        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
```

Now, we want to define the `forward` method, which is what will be called at inference time.
Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references.
To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py).

```python Forward Method Signature theme={null}
    @torch.no_grad()
    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        forward_batch: ForwardBatch,
        pp_proxy_tensors: Optional[PPProxyTensors] = None,
        input_embeds: Optional[torch.Tensor] = None,
        get_embedding: bool = False,
    ) -> LogitsProcessorOutput:
```

We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method.
After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`).

```python Call Model and LogitsProcessor theme={null}
        hidden_states = self.model(
            input_ids,
            positions,
            forward_batch,
            input_embeds,
            pp_proxy_tensors=pp_proxy_tensors,
        )

        res: LogitsProcessorOutput = self.logits_processor(
            input_ids,
            hidden_states,
            self.lm_head,
            forward_batch,
        )
```

After receiving the logits for the next token, we can finally perform our biasing step.

```python Logit Biasing theme={null}
        orig_logits = res.next_token_logits
        res.next_token_logits = torch.where(
            orig_logits > 0,
            orig_logits.sqrt(),
            orig_logits
        )

        return res
```

Now, our `LlamaWrapper` model is created and ready to be served!

### Serving Our Model Via SGLang's Offline Engine

The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server.

First, create a new file called `run.py`.
Now, we must ensure that SGLang's `ModelRegistry` can find our model.
To do this, we first download the model's configuration and weights from Huggingface.

```python Example theme={null}
# In the file `run.py`

import asyncio
from functools import lru_cache
from huggingface_hub import snapshot_download
from llama_wrapper import LlamaWrapper # Make sure to import our new model!
import sglang as sgl
from sglang.srt.models.registry import ModelRegistry

# Make sure to request access to this model on Huggingface, then export your
# `HF_TOKEN` to download the model snapshot
llama_dir = snapshot_download(
    repo_id="meta-llama/Llama-3.1-8B-Instruct",
    local_dir="./llama_ckpt",
)
```

Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`.
That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model.

```json Example theme={null}
{
  "architectures": [
    "LlamaWrapper"
  ],
  ...
}
```

However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model.
Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation".

```python Register LlamaWrapper theme={null}
@lru_cache()
def import_new_model_classes():
    model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper}
    return model_arch_name_to_cls

ModelRegistry.models.update(import_new_model_classes())
```

Lastly, when we create our `Engine`, we just pass in the path to the local model directory.
Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint.

```python Example theme={null}
def main():
    llm = sgl.Engine(model_path="./llama_ckpt")
    sampling_params = {"temperature": 0.2, "top_k": 5}
    prompts = [
        "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
        "Provide a concise factual statement about France’s capital city. The capital of France is",
        "Explain possible future trends in artificial intelligence. The future of AI is",
    ]

    asyncio.run(run_llm(llm, sampling_params, prompts))

    llm.shutdown()

async def run_llm(
    llm,
    sampling_params,
    prompts,
) -> None:
    outputs = await llm.async_generate(prompts, sampling_params)

    for prompt, output in zip(prompts, outputs):
        print(f"\nPrompt: {prompt}")
        print(f"Generated text: {output['text']}")

if __name__ == "__main__":
    main()
```

Now, when we call `python run.py`, we will get the outputs of our newly created model!

## Serving External Models via the Standard CLI

The previous sections show how to register a model programmatically via `ModelRegistry` and serve it through the Offline Engine. Similar to vLLM model plugin, there is an alternative that lets you keep using the standard `python -m sglang.launch_server` CLI without modifying any SGLang source code: you can register your model using the `SGLANG_EXTERNAL_MODEL_PACKAGE` environment variable.

> [!TIP]
>
> On Ascend NPU, `--device` and `--attention-backend` are auto-detected and can be omitted from the launch command.
> SGLang sets the device to `npu` and attention backend to `ascend` automatically when `torch.npu.is_available()`.

### The `EntryClass` Variable

When SGLang scans a model package, it looks for the variable `EntryClass` at the module level of your Python file. The [model registry](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/registry.py) imports your file, checks for `EntryClass`, and registers the class assigned to it. If you are using a model based on HuggingFace, the name of this class needs to match the `"architectures"` field in your model's `config.json`.

For example, if you are implementing a Llama wrapper, add this line at the end of your model file:

```python Example theme={null}
# This is what "Add EntryClass at the end" means
EntryClass = LlamaWrapper
```

### Example: Text-Only Model

Using the same Llama wrapper from the previous section, here is how to package and serve it via the CLI.

1. Create your project

```text theme={null}
sglang_custom_project/
|----setup.py
|----custom_llm/
     |----__init__.py
     |----llama_wrapper.py
```

Write the `setup.py`:

```python Example theme={null}
# sglang_custom_project/setup.py

from setuptools import setup, find_packages
setup(
    name="sglang-custom-plugins",
    version="0.1",
    packages=find_packages(),
)
```

2. Write your model code

Inside `llama_wrapper.py`, write your model and include `EntryClass`:

```python Example theme={null}
# sglang_custom_project/custom_llm/llama_wrapper.py

import torch
from typing import Optional
from sglang.srt.layers.logits_processor import LogitsProcessorOutput
from sglang.srt.layers.quantization.base_config import QuantizationConfig
from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors
from sglang.srt.models.llama import LlamaForCausalLM

class LlamaWrapper(LlamaForCausalLM):
    def __init__(self, config, quant_config: Optional[QuantizationConfig] = None,
                 prefix: str = "") -> None:
        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
    @torch.no_grad()
    def forward(self, input_ids, positions, forward_batch,
                pp_proxy_tensors=None, input_embeds=None, get_embedding=False):
        hidden_states = self.model(
            input_ids, positions, forward_batch, input_embeds,
            pp_proxy_tensors=pp_proxy_tensors,
        )
        res: LogitsProcessorOutput = self.logits_processor(
            input_ids, hidden_states, self.lm_head, forward_batch,
        )

        orig = res.next_token_logits
        res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig)
        return res

# Don't forget to add EntryClass
EntryClass = LlamaWrapper
```

3. Install your package

Run this inside your `sglang_custom_project` directory to install your code into the active Python environment:

```bash Command theme={null}
pip install -e .
```

4. Update your `config.json`

Update the `config.json` under your HuggingFace model checkpoint directory so the `architectures` field matches your class name:

```json Config theme={null}
{
    "architectures": ["LlamaWrapper"],
    ...
}
```

5. Launch the server

Set the environment variable before running the CLI:

```bash Command theme={null}
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_llm
python -m sglang.launch_server \
    --model-path /path/to/Llama-3.1-8B-Instruct \
    --port 8000
```

The `SGLANG_EXTERNAL_MODEL_PACKAGE` should be the parent folder name containing your model-related code. In this example, it should be `custom_llm`.

### Example: Multimodal Model

If you are working with multimodal models, setting `SGLANG_EXTERNAL_MODEL_PACKAGE` alone is not enough. SGLang also needs to recognize your architecture as multimodal to enable the image/video processing pipelines, and it needs a custom processor.

You can handle this by setting two additional environment variables:

* `SGLANG_EXTERNAL_MM_MODEL_ARCH`: Adds your architecture name to SGLang's internal list of multimodal models.
* `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`: Tells SGLang where to find your custom processor class.

For example, let's build a custom model based on Qwen2-VL-Instruct that takes the square root of the logits.

Create the project:

```text theme={null}
sglang_custom_project_vl/
|----setup.py
|----custom_vlm/
     |----__init__.py
     |----qwenvl_wrapper.py
```

Write `setup.py`:

```python Example theme={null}
# sglang_custom_project_vl/setup.py

from setuptools import setup, find_packages
setup(
    name="sglang-custom-plugins-vl",
    version="0.1",
    packages=find_packages(),
)
```

Write the model in `qwenvl_wrapper.py`:

```python Example theme={null}
# sglang_custom_project_vl/custom_vlm/qwenvl_wrapper.py
import torch
from sglang.srt.models.qwen2_vl import Qwen2VLForConditionalGeneration
from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor

class CustomQwen2VL(Qwen2VLForConditionalGeneration):
    def forward(self, input_ids, positions, forward_batch,
                input_embeds=None, get_embedding=False):
        res = super().forward(
            input_ids, positions, forward_batch,
            input_embeds=input_embeds, get_embedding=get_embedding
        )
        if not get_embedding:
            orig = res.next_token_logits
            res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig)
        return res

class CustomQwen2VLProcessor(QwenVLImageProcessor):
    models = [CustomQwen2VL]

    def __init__(self, hf_config, server_args, _processor, *args, **kwargs):
        super().__init__(hf_config, server_args, _processor, *args, **kwargs)

EntryClass = CustomQwen2VL
```

**Note:** you don't need a separate `EntryClass` for the custom processor as long as you associate the processor with the specific model class.

Install the package, update `config.json`, and launch:

```bash Command theme={null}
pip install -e .
```

```json Config theme={null}
{
  "architectures": ["CustomQwen2VL"],
  ...
}
```

```bash Command theme={null}
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_vlm
export SGLANG_EXTERNAL_MM_MODEL_ARCH=CustomQwen2VL
export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=custom_vlm

python -m sglang.launch_server \
    --model-path /path/to/Qwen2-VL-2B-Instruct \
    --port 8000 \
    --enable-multimodal
```

## Documentation

Add to table of supported models in [generative\_models.mdx](/docs/supported-models/generative_models) or [multimodal\_language\_models.mdx](/docs/supported-models/multimodal_language_models)

> [!TIP]
>
> For NPU-adapted models, also add entries to the NPU support models table in
> [reference/support\_models.mdx](../reference/support_models).

***

By following these guidelines, you can add support for new language models and multimodal large language models in
SGLang and ensure they are thoroughly tested and easily integrated into the system.


# Accuracy Evaluation
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation



# Ascend NPU Accuracy Evaluation

This document describes how to perform accuracy evaluation for SGLang models running on Ascend NPU using a tool: **EvalScope**. The following scenarios are covered:

* **Online Testing**: Evaluate via API interface after starting SGLang server
* **Text Models**: Using Qwen2.5-7B-Instruct as example
* **Multimodal Models**: Using Qwen2.5-VL-7B-Instruct as example

***

## Environment Setup

> [!WARNING]
>
> Ensure sufficient disk space before proceeding. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.

First, launch the SGLang environment using the provided container image:

**Ascend A3 Series**

```shell Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily uses the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
export IMAGE=quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
    --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --volume /usr/local/sbin:/usr/local/sbin \
    --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    --volume /etc/ascend_install.info:/etc/ascend_install.info \
    --volume /var/queue_schedule:/var/queue_schedule \
    --volume ~/.cache/:/root/.cache/ \
    --entrypoint=bash \
    $IMAGE
```

***

## Using EvalScope

[EvalScope](https://github.com/modelscope/evalscope) is a comprehensive model evaluation framework from ModelScope, supporting both accuracy evaluation and performance stress testing.

### Install EvalScope

```shell Command theme={null}
# Method 1: Installing via pip
pip install evalscope

# Method 2: Installing from source
git clone https://github.com/modelscope/evalscope.git
cd evalscope/
pip install -e .
```

### Online Text Model Testing

This section covers online evaluation scenarios where the SGLang server is already running.

#### Start SGLang Server

```shell Command theme={null}
# Set HuggingFace mirror (if network access is restricted)
export HF_ENDPOINT=https://hf-mirror.com

# Start text model server
sglang serve --model-path /home/weights/Qwen2.5-7B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000 &
```

For more details of SGLang server, refer to the [Ascend NPU Quick Start](/docs/hardware-platforms/ascend-npus/getting-started/quick_start)

#### Execute Accuracy Evaluation

Use sgl-eval for GSM8K and MMLU. It owns the datasets, chat prompts and graders. EvalScope remains available for the other benchmarks below:

```shell Command theme={null}
pip install sgl-eval
sgl-eval run gsm8k \
 --model /home/weights/Qwen2.5-7B-Instruct \
 --base-url http://localhost:30000/v1 \
 --num-examples 10 --max-tokens 2048
```

The run directory contains `metrics.json` and per-example predictions. Use the model name reported by `/v1/models`.

#### Common Datasets for Online Evaluation

```shell Command theme={null}
# MMLU
sgl-eval run mmlu \
 --model /home/weights/Qwen2.5-7B-Instruct \
 --base-url http://localhost:30000/v1 --max-tokens 2048

# CEval (Chinese evaluation)
evalscope eval \
 --model /home/weights/Qwen2.5-7B-Instruct \
 --api-url http://localhost:30000/v1 \
 --api-key EMPTY \
 --eval-type openai_api \
 --datasets ceval

# MATH-500
evalscope eval \
 --model /home/weights/Qwen2.5-7B-Instruct \
 --api-url http://localhost:30000/v1 \
 --api-key EMPTY \
 --eval-type openai_api \
 --datasets math_500

# HumanEval (code generation)
evalscope eval \
 --model /home/weights/Qwen2.5-7B-Instruct \
 --api-url http://localhost:30000/v1 \
 --api-key EMPTY \
 --eval-type openai_api \
 --datasets humaneval
```

### Online Multimodal Model Testing

#### Start Multimodal Model Server

```shell Command theme={null}
# Start multimodal model server (Qwen2.5-VL-7B-Instruct)
# Multimodal models require both --attention-backend and --mm-attention-backend
sglang serve --model-path /home/weights/Qwen2.5-VL-7B-Instruct \
    --attention-backend ascend \
    --mm-attention-backend ascend_attn \
    --host 0.0.0.0 --port 30000 &
```

#### Execute Multimodal Accuracy Evaluation

```shell Command theme={null}
# MMBench (multimodal evaluation)
evalscope eval \
 --model /home/weights/Qwen2.5-VL-7B-Instruct \
 --api-url http://localhost:30000/v1 \
 --api-key EMPTY \
 --eval-type openai_api \
 --datasets mm_bench

# MMMU (multimodal comprehensive understanding)
evalscope eval \
 --model /home/weights/Qwen2.5-VL-7B-Instruct \
 --api-url http://localhost:30000/v1 \
 --api-key EMPTY \
 --eval-type openai_api \
 --datasets mmmu

# HallusionBench (hallucination evaluation)
evalscope eval \
 --model /home/weights/Qwen2.5-VL-7B-Instruct \
 --api-url http://localhost:30000/v1 \
 --api-key EMPTY \
 --eval-type openai_api \
 --datasets hallusion_bench
```

For more details, refer to the [EvalScope documentation](https://evalscope.readthedocs.io/).

***

## Troubleshooting

### SGLang Server Startup Failure

1. Verify device mapping: A3 Series uses `davinci[0-15]`
2. Confirm image tag matches device type: A3 Series uses `...-a3`
3. Check NPU status with `npu-smi info`
4. First run requires model download; set `HF_ENDPOINT=https://hf-mirror.com` if network access is restricted

### EvalScope Connection Failure to Server

1. Confirm SGLang server started successfully (look for `Application startup complete` in logs)
2. Verify `--api-url` points to the correct port (SGLang defaults to `30000`)
3. Ensure URL ends with `/v1`, e.g., `http://localhost:30000/v1`

### EvalScope SSL certificate verification failed

When using EvalScope commands without specifying a dataset or model path, it will attempt to download automatically, which may encounter an SSL certificate verification error:

```text theme={null}
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 605, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 592, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 706, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/adapters.py", line 676, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.modelscope.cn', port=443): Max retries exceeded with url: /api/v1/datasets/AI-ModelScope/ceval (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1016)')))
[ERROR] 2026-05-13-02:20:01 (PID:876, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

**Temporary workaround (test only):**
Navigate to `/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py`, find the `class Session` definition, and set `self.verify = False`.

> [!WARNING]
>
> This **disables TLS certificate validation globally** for the Python `requests` library. Use it **only as a temporary diagnostic step** in isolated test environments — never in production.

**Stable solution:**
The error is caused by a corporate TLS proxy injecting a self-signed certificate. Point `requests` to the proxy's CA bundle:

```shell theme={null}
# Obtain the CA certificate from your network administrator
# Then set the environment variable:
export REQUESTS_CA_BUNDLE=/path/to/your-proxy-ca-bundle.crt
```

> [!NOTE]
>
> This is a common workaround for corporate proxy environments. If it does not resolve your issue, consult your IT department — proxy configurations vary across organizations.

If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below.

### EvalScope Request Retry Timeout

If EvalScope keeps retrying requests with errors like:

```
2026-06-22 03:09:03 - evalscope - WARNING: Attempt 4 / 5 failed: ....... Retrying...
2026-06-22 03:09:14 - evalscope - INFO: Evaluating[ceval]   0%| 0/520 [Elapsed: 02:00 < Remaining: ?, ?it/s]
2026-06-22 03:09:19,557 - openai._base_client - INFO: Retrying request to /chat/completions in 0.447260 seconds
2026-06-22 03:09:26,088 - openai._base_client - INFO: Retrying request to /chat/completions in 0.992551 seconds
```

This is usually caused by the HTTP proxy intercepting requests to the local SGLang server. Disable the proxy with:

```shell Command theme={null}
unset http_proxy
unset https_proxy
unset HTTP_PROXY
unset HTTPS_PROXY
```

### Download Dataset Error

For this error

```text theme={null}
root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
--2026-05-12 12:08:01--  https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
Connecting to <PROXY_IP>:<PROXY_PORT>... connected.
ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘<CERT_ISSUER>’:
  Self-signed certificate encountered.
To connect to www.modelscope.cn insecurely, use `--no-check-certificate`.
```

You can add `--no-check-certificate`

```bash theme={null}
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate
```

For additional assistance, refer to [SGLang GitHub Issues](https://github.com/sgl-project/sglang/issues).


# Performance Testing
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/evaluation/performance_testing



This page walks through performance testing your SGLang deployment on Ascend NPUs. We cover three model types — text generation (`Qwen/Qwen2.5-7B-Instruct`), multimodal vision (`Qwen/Qwen2.5-VL-7B-Instruct`), and embedding (`Qwen/Qwen3-Embedding-8B`) — in both online and offline serving modes. You can use [Evalscope](https://evalscope.readthedocs.io/en/latest/), [AISBench](https://ais-bench-benchmark.readthedocs.io/en/latest/), or SGLang's built-in benchmarking tools.

> [!NOTE]
>
> The benchmark output examples in this guide are for illustration only. Actual performance depends on your hardware, model version, SGLang version, and deployment configuration. Always run benchmarks on your own hardware to obtain accurate performance data.

## 1. Prepare

### 1.1 Start SGLang server

Launch the server with the appropriate flags for each model type. Make sure SGLang is installed first — see [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) for environment setup.

**Text Generation**

```shell Command theme={null}
# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
sglang serve --model-path Qwen/Qwen2.5-7B-Instruct
```

**Multimodal**

```shell Command theme={null}
# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
sglang serve --model-path Qwen/Qwen2.5-VL-7B-Instruct --mm-attention-backend ascend_attn
```

**Embedding**

```shell Command theme={null}
# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
sglang serve --model-path Qwen/Qwen3-Embedding-8B --is-embedding
```

> [!TIP]
>
> Add `&` at the end of the command to run the server in the background, or open a new terminal to run the benchmark commands in the following sections.

> [!NOTE]
>
> The server binds to `http://127.0.0.1:30000` by default. All online benchmarks below assume the server is running at that address. The `--is-embedding` flag is required for embedding models.

### 1.2 Install benchmarking tools

`bench_serving` and `bench_offline_throughput` are built into SGLang and require no extra installation. For Evalscope and AISBench, set up each in its own virtual environment:

**Evalscope**

```shell Command theme={null}
python3 -m venv .evalscope_venv
source .evalscope_venv/bin/activate
pip install evalscope[perf] -U
```

**AISBench**

```shell Command theme={null}
python3 -m venv .aisbench_venv
source .aisbench_venv/bin/activate

git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517

pip3 install -r requirements/api.txt
pip3 install -r requirements/extra.txt
```

Run `ais_bench -h` to verify.

> [!NOTE]
>
> AISBench requires Python 3.10-3.12. After installation, all AISBench commands must be run from the `benchmark/` directory (the cloned repo root). Set `stream=True` and `ignore_eos=True` in the model config for accurate results.

## 2. Online Service: Text Generation Model

Test `Qwen/Qwen2.5-7B-Instruct` via the online serving endpoint.

> [!NOTE]
>
> Before running any benchmark in this section, make sure the SGLang text-generation server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) for the launch command.

> [!TIP]
>
> For performance testing, prefer random datasets (`--dataset random`, `--dataset-name random`) over real datasets. Random datasets let you pin `--min-prompt-length` / `--max-prompt-length` and `--min-tokens` / `--max-tokens` to fixed values, producing consistent, repeatable results. Real datasets (ShareGPT, openqa, etc.) have variable input lengths that add noise and make cross-run comparisons unreliable.

### 2.1 Using Evalscope

> [!NOTE]
>
> Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang server running at `http://127.0.0.1:30000`.

Run the following command to run a performance test against the server:

```shell Command theme={null}
evalscope perf \
  --parallel 10 \
  --number 20 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --url http://127.0.0.1:30000/v1/chat/completions \
  --api openai \
  --dataset random \
  --max-tokens 1024 \
  --min-tokens 1024 \
  --prefix-length 0 \
  --min-prompt-length 1024 \
  --max-prompt-length 1024 \
  --tokenizer-path Qwen/Qwen2.5-7B-Instruct \
  --extra-args '{"ignore_eos": true}'
```

> [!TIP]
>
> If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.

Example output (for illustration only — actual results depend on your hardware and configuration):

```text theme={null}
Benchmarking summary:
┌────────────────────────────┬─────────────┐
│ Metric                     │       Value │
├────────────────────────────┼─────────────┤
│ ── General ──              │             │
│ Test Duration (s)          │       89.34 │
│ Concurrency                │          10 │
│ Request Rate (req/s)       │       -1.00 │
│ Total / Success / Failed   │ 20 / 20 / 0 │
│ Req Throughput (req/s)     │        0.22 │
│ ── Latency ──              │             │
│ Avg Latency (s)            │       44.67 │
│ TTFT (ms)                  │      578.51 │
│ TPOT (ms)                  │       43.10 │
│ ITL (ms)                   │       43.12 │
│ ── Tokens ──               │             │
│ Avg Input Tokens           │     1024.00 │
│ Avg Output Tokens          │     1024.00 │
│ Output Throughput (tok/s)  │      229.24 │
│ Total Throughput (tok/s)   │      458.49 │
│ ── Speculative Decoding ── │             │
│ Decoded Tok/Iter           │        1.00 │
│ Spec. Accept Rate          │        0.00 │
└────────────────────────────┴─────────────┘

Percentile results:
┌────────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ Metric         │      1% │      5% │     10% │     25% │     50% │     75% │     90% │     95% │     99% │
├────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Latency (s)    │   44.47 │   44.47 │   44.47 │   44.47 │   44.86 │   44.86 │   44.86 │   44.86 │   44.86 │
│ TTFT (ms)      │  138.12 │  142.07 │  426.17 │  426.87 │  783.67 │  785.26 │  786.85 │  787.97 │  787.97 │
│ ITL (ms)       │   41.84 │   42.14 │   42.22 │   42.36 │   42.57 │   42.80 │   42.99 │   49.24 │   49.84 │
│ TPOT (ms)      │   42.71 │   42.71 │   42.71 │   43.05 │   43.08 │   43.43 │   43.43 │   43.71 │   43.71 │
│ Input tokens   │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │
│ Output tokens  │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │
│ Output (tok/s) │   22.83 │   22.83 │   22.83 │   22.83 │   23.02 │   23.03 │   23.03 │   23.03 │   23.03 │
│ Total (tok/s)  │   45.65 │   45.65 │   45.65 │   45.65 │   46.05 │   46.05 │   46.05 │   46.05 │   46.05 │
│ Decode (tok/s) │   22.88 │   23.03 │   23.03 │   23.07 │   23.21 │   23.42 │   23.42 │   23.42 │   23.42 │
└────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
...
```

> [!NOTE]
>
> See the [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) for full details.

### 2.2 Using AISBench

> [!NOTE]
>
> Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands must be run from the `benchmark/` directory. SGLang server running at `http://127.0.0.1:30000`. Set `stream=True` and `ignore_eos=True` in the model config for accurate results.

Two files need to be configured for performance testing.

First, describe the model and server settings in `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py`:

```python vllm_api_stream_chat.py theme={null}
# more details: https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/scenes_intro/performance_benchmark.html
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr="vllm-api-stream-chat",
        path="Qwen/Qwen2.5-7B-Instruct",
        model="Qwen/Qwen2.5-7B-Instruct",
        stream=True,
        request_rate=0,
        use_timestamp=False,
        retry=2,
        api_key="",
        host_ip="127.0.0.1",
        host_port=30000,
        url="",
        max_out_len=512,
        batch_size=32,
        trust_remote_code=False,
        generation_kwargs=dict(
            temperature=0.01,
            ignore_eos=True,
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content),
    )
]
```

> [!TIP]
>
> If the model has already been downloaded, point `path` to the local model path instead of the model id.

Second, configure random prompt lengths in `ais_bench/datasets/synthetic/synthetic_config.py`:

```python synthetic_config.py theme={null}
# more details: https://ais-bench-benchmark.readthedocs.io/en/latest/advanced_tutorials/synthetic_dataset.html
synthetic_config = {
    "Type":"tokenid",
    "RequestCount": 10,
    "TrustRemoteCode": False,
    "StringConfig" : {
        "Input" : {
            "Method": "uniform",
            "Params": {"MinValue": 1, "MaxValue": 200}
        },
        "Output" : {
            "Method": "gaussian",
            "Params": {"Mean": 100, "Var": 200, "MinValue": 1, "MaxValue": 100}
        }
    },
    "TokenIdConfig" : {
        "RequestSize": 10,
        "PrefixLen": 0
    }
}
```

Run with a synthetic dataset:

```shell Command theme={null}
ais_bench --models vllm_api_stream_chat --datasets synthetic_gen_string -m perf
```

Example output (for illustration only — actual results depend on your hardware and configuration):

```text theme={null}
╒══════════════════════════╤═════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════╕
│ Performance Parameters   │ Stage   │ Average         │ Min             │ Max             │ Median          │ P75             │ P90             │ P99             │  N  │
╞══════════════════════════╪═════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════╡
│ E2EL                     │ total   │ 3896.4 ms       │ 3081.6 ms       │ 4175.3 ms       │ 4013.8 ms       │ 4123.4 ms       │ 4137.1 ms       │ 4171.5 ms       │ 10  │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ TTFT                     │ total   │ 411.6 ms        │ 346.7 ms        │ 439.7 ms        │ 416.3 ms        │ 426.6 ms        │ 434.4 ms        │ 439.2 ms        │ 10  │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ TPOT                     │ total   │ 38.3 ms         │ 37.4 ms         │ 39.0 ms         │ 38.3 ms         │ 38.7 ms         │ 38.9 ms         │ 39.0 ms         │ 10  │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ ITL                      │ total   │ 38.7 ms         │ 0.0 ms          │ 156.5 ms        │ 38.9 ms         │ 39.0 ms         │ 39.2 ms         │ 117.1 ms        │ 10  │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ InputTokens              │ total   │ 123.4           │ 34.0            │ 228.0           │ 130.5           │ 170.5           │ 217.2           │ 226.92          │ 10  │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ OutputTokens             │ total   │ 92.1            │ 69.0            │ 100.0           │ 95.0            │ 99.75           │ 100.0           │ 100.0           │ 10  │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ OutputTokenThroughput    │ total   │ 23.5937 token/s │ 22.3912 token/s │ 24.2616 token/s │ 23.7399 token/s │ 23.9919 token/s │ 24.2027 token/s │ 24.2557 token/s │ 10  │
╘══════════════════════════╧═════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════╛
╒══════════════════════════╤═════════╤══════════════════╕
│ Common Metric            │ Stage   │ Value            │
╞══════════════════════════╪═════════╪══════════════════╡
│ Benchmark Duration       │ total   │ 4175.4485 ms     │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Requests           │ total   │ 10               │
├──────────────────────────┼─────────┼──────────────────┤
│ Failed Requests          │ total   │ 0                │
├──────────────────────────┼─────────┼──────────────────┤
│ Success Requests         │ total   │ 10               │
├──────────────────────────┼─────────┼──────────────────┤
│ Concurrency              │ total   │ 9.3317           │
├──────────────────────────┼─────────┼──────────────────┤
│ Max Concurrency          │ total   │ 32               │
├──────────────────────────┼─────────┼──────────────────┤
│ Request Throughput       │ total   │ 2.395 req/s      │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Input Tokens       │ total   │ 1234             │
├──────────────────────────┼─────────┼──────────────────┤
│ Prefill Token Throughput │ total   │ 299.8329 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Generated Tokens   │ total   │ 921              │
├──────────────────────────┼─────────┼──────────────────┤
│ Input Token Throughput   │ total   │ 295.5371 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Output Token Throughput  │ total   │ 220.5751 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Token Throughput   │ total   │ 516.1122 token/s │
╘══════════════════════════╧═════════╧══════════════════╛
```

> [!NOTE]
>
> See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details.

### 2.3 Using bench\_serving

SGLang's built-in `bench_serving` requires no extra installation. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark.

> [!NOTE]
>
> See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for all backends, datasets, and advanced options.

```shell Command theme={null}
python -m sglang.bench_serving \
  --backend sglang-oai \
  --base-url http://127.0.0.1:30000 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 512 \
  --random-range-ratio 1 \
  --num-prompts 100 \
  --max-concurrency 32
```

> [!NOTE]
>
> `--dataset-name random` samples token IDs from the ShareGPT dataset to generate realistic input; the first run downloads ShareGPT from Hugging Face automatically.
>
> 1. If you have network issues, set `export HF_ENDPOINT=https://hf-mirror.com` to use domestic mirror.
> 2. If downloading still fails, manually download the dataset file `ShareGPT_V3_unfiltered_cleaned_split.json` locally, upload it to your server, then specify the file directory via `--dataset-path` to run offline.

> [!TIP]
>
> Set `--random-range-ratio 1` for fixed input/output lengths (recommended for consistent comparisons) or `0` (default) for uniform distribution. Add `--request-rate` to control the request rate. For all backends, datasets, and advanced options, see the full [Bench Serving Guide](/docs/developer_guide/bench_serving).

Example output (for illustration only — actual results depend on your hardware and configuration):

```text theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-oai
Traffic request rate:                    inf
Max request concurrency:                 32
Successful requests:                     100
Benchmark duration (s):                  47.51
Total input tokens:                      102400
Total input text tokens:                 102400
Total generated tokens:                  51200
Total generated tokens (retokenized):    51195
Request throughput (req/s):              2.10
Input token throughput (tok/s):          2155.35
Output token throughput (tok/s):         1077.68
Peak output token throughput (tok/s):    1587.00
Peak concurrent requests:                64
Total token throughput (tok/s):          3233.03
Concurrency:                             26.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   12793.49
Median E2E Latency (ms):                 12940.17
P90 E2E Latency (ms):                    13049.86
P99 E2E Latency (ms):                    13051.61
---------------Time to First Token----------------
Mean TTFT (ms):                          1423.99
Median TTFT (ms):                        1489.29
P99 TTFT (ms):                           2325.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          22.25
Median TPOT (ms):                        22.22
P99 TPOT (ms):                           25.08
---------------Inter-Token Latency----------------
Mean ITL (ms):                           22.26
Median ITL (ms):                         20.74
P95 ITL (ms):                            21.40
P99 ITL (ms):                            23.62
Max ITL (ms):                            2229.30
==================================================
```

#### SGLang Serving Benchmark Result — Complete Reference

The output format is **hardcoded in `bench_serving.py`**. All formatting decisions — including column widths, alignment, and decimal precision — are statically defined in the source and cannot be changed via command-line arguments.

##### Test Configuration

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`Backend`</td>
      <td>The serving backend under test (e.g., `sglang`, `vllm`).</td>
    </tr>

    <tr>
      <td>`Traffic request rate`</td>
      <td>Request generation rate in req/s. `inf` means maximum rate (concurrency-bounded). `trace` indicates trace timestamp mode. A fixed value enforces constant inter-arrival time.</td>
    </tr>

    <tr>
      <td>`Max request concurrency`</td>
      <td>Maximum number of concurrent requests from the client side. Displays `not set` when unspecified.</td>
    </tr>
  </tbody>
</table>

##### Core Statistics & Throughput Metrics

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Format Specification</th>
    </tr>
  </thead>

  <tbody>
    <tr><td>`Successful requests`</td><td>Total number of successfully completed requests (HTTP 200, no generation errors).</td><td>Integer, no decimal places</td></tr>
    <tr><td>`Benchmark duration (s)`</td><td>Total elapsed time from first request sent to last response fully received (seconds).</td><td>2 decimal places</td></tr>
    <tr><td>`Total input tokens`</td><td>Total number of input (prompt) tokens across all requests, counted by server-side tokenizer.</td><td>Integer, no decimal places</td></tr>
    <tr><td>`Total input text tokens`</td><td>Same as `Total input tokens`. For multimodal inputs, this may differ.</td><td>Integer, no decimal places</td></tr>
    <tr><td>`Total generated tokens`</td><td>Total number of output tokens actually generated by the server (server-side tokenizer count).</td><td>Integer, no decimal places</td></tr>
    <tr><td>`Total generated tokens (retokenized)`</td><td>Output text re-tokenized by the client using its own tokenizer. A large discrepancy indicates tokenizer mismatch or special tokens in output.</td><td>Integer, no decimal places</td></tr>
    <tr><td>`Request throughput (req/s)`</td><td>Number of successful requests processed per second. Formula: `Successful requests / Benchmark duration (s)`.</td><td>2 decimal places</td></tr>
    <tr><td>`Input token throughput (tok/s)`</td><td>Number of input tokens processed per second. Formula: `Total input tokens / Benchmark duration (s)`.</td><td>2 decimal places</td></tr>
    <tr><td>`Output token throughput (tok/s)`</td><td>Number of output tokens generated per second. Formula: `Total generated tokens / Benchmark duration (s)`.</td><td>2 decimal places</td></tr>
    <tr><td>`Peak output token throughput (tok/s)`</td><td>Observed instantaneous peak output token generation rate during the test (computed over a sliding window).</td><td>2 decimal places</td></tr>
    <tr><td>`Peak concurrent requests`</td><td>Maximum number of requests being processed simultaneously on the server side. May exceed client-side `Max request concurrency` due to queueing.</td><td>Integer, no decimal places</td></tr>
    <tr><td>`Total token throughput (tok/s)`</td><td>Sum of input and output token throughputs. Formula: `Input token throughput + Output token throughput`.</td><td>2 decimal places</td></tr>
    <tr><td>`Concurrency`</td><td>Average number of concurrent requests during the test (Little's Law). Formula: `Sum of all E2E latencies / Benchmark duration`.</td><td>2 decimal places</td></tr>
  </tbody>
</table>

##### End-to-End Latency (E2E Latency)

<table>
  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>

  <tbody>
    <tr><td>`Mean E2E Latency (ms)`</td><td>Arithmetic mean</td><td>2 decimal places</td></tr>
    <tr><td>`Median E2E Latency (ms)`</td><td>50th percentile</td><td>2 decimal places</td></tr>
    <tr><td>`P90 E2E Latency (ms)`</td><td>90th percentile (90% of requests have latency ≤ this value)</td><td>2 decimal places</td></tr>
    <tr><td>`P99 E2E Latency (ms)`</td><td>99th percentile</td><td>2 decimal places</td></tr>
  </tbody>
</table>

##### Time to First Token (TTFT)

<table>
  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>

  <tbody>
    <tr><td>`Mean TTFT (ms)`</td><td>Arithmetic mean</td><td>2 decimal places</td></tr>
    <tr><td>`Median TTFT (ms)`</td><td>50th percentile</td><td>2 decimal places</td></tr>
    <tr><td>`P99 TTFT (ms)`</td><td>99th percentile</td><td>2 decimal places</td></tr>
  </tbody>
</table>

##### Time per Output Token (TPOT) – Excluding First Token

Formula: `(E2E Latency - TTFT) / (Number of output tokens - 1)`

<table>
  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>

  <tbody>
    <tr><td>`Mean TPOT (ms)`</td><td>Arithmetic mean</td><td>2 decimal places</td></tr>
    <tr><td>`Median TPOT (ms)`</td><td>50th percentile</td><td>2 decimal places</td></tr>
    <tr><td>`P99 TPOT (ms)`</td><td>99th percentile</td><td>2 decimal places</td></tr>
  </tbody>
</table>

##### Inter-Token Latency (ITL)

<table>
  <thead><tr><th>Statistic</th><th>Description</th><th>Format</th></tr></thead>

  <tbody>
    <tr><td>`Mean ITL (ms)`</td><td>Average inter-token interval</td><td>2 decimal places</td></tr>
    <tr><td>`Median ITL (ms)`</td><td>50th percentile inter-token interval</td><td>2 decimal places</td></tr>
    <tr><td>`P95 ITL (ms)`</td><td>95th percentile (used to detect stalls)</td><td>2 decimal places</td></tr>
    <tr><td>`P99 ITL (ms)`</td><td>99th percentile</td><td>2 decimal places</td></tr>
    <tr><td>`Max ITL (ms)`</td><td>Maximum observed inter-token interval; useful for identifying severe blocking events</td><td>2 decimal places</td></tr>
  </tbody>
</table>

## 3. Online Service: Multimodal Model

Test `Qwen/Qwen2.5-VL-7B-Instruct` for vision-language tasks.

> [!NOTE]
>
> Before running any benchmark in this section, make sure the SGLang multimodal server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Multimodal tab for the launch command.

> [!TIP]
>
> For consistent, repeatable results, set `--random-range-ratio 1` to fix input/output lengths, or `0` (default) for uniform distribution.

### 3.1 Using Evalscope

> [!NOTE]
>
> Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang multimodal server running at `http://127.0.0.1:30000`.

Evalscope's `perf` tool uses the OpenAI-compatible `/v1/chat/completions` endpoint. Use `--dataset random_vl` for randomized multimodal data with image generation:

```shell Command theme={null}
evalscope perf \
  --parallel 10 \
  --number 20 \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --url http://127.0.0.1:30000/v1/chat/completions \
  --api openai \
  --dataset random_vl \
  --min-tokens 1024 \
  --max-tokens 1024 \
  --prefix-length 0 \
  --min-prompt-length 1024 \
  --max-prompt-length 1024 \
  --image-width 512 \
  --image-height 512 \
  --image-format RGB \
  --image-num 1 \
  --tokenizer-path Qwen/Qwen2.5-VL-7B-Instruct \
  --extra-args '{"ignore_eos": true}'
```

> [!TIP]
>
> If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.

### 3.2 Using AISBench

> [!NOTE]
>
> Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands run from the `benchmark/` directory. SGLang multimodal server running at `http://127.0.0.1:30000`. AISBench does not include a built-in multimodal dataset — you must provide your own.

First, edit `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py` to configure the vision model:

```python vllm_api_stream_chat.py theme={null}
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr="vllm-api-stream-chat",
        path="Qwen/Qwen2.5-VL-7B-Instruct",
        model="Qwen/Qwen2.5-VL-7B-Instruct",
        stream=True,
        request_rate=0,
        use_timestamp=False,
        retry=2,
        api_key="",
        host_ip="127.0.0.1",
        host_port=30000,
        url="",
        max_out_len=256,
        batch_size=16,
        trust_remote_code=False,
        generation_kwargs=dict(
            temperature=0.01,
            ignore_eos=True,
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content),
    )
]
```

> [!TIP]
>
> If the model has already been downloaded, point `path` to the local model path instead of the model id.

Next, download a multimodal dataset such as mmstar:

```shell Command theme={null}
# Download the mmstar dataset (from within the benchmark/ directory)
cd ais_bench/datasets
mkdir mmstar
cd mmstar
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
```

Run the performance test:

```shell Command theme={null}
ais_bench --models vllm_api_stream_chat --datasets mmstar_gen -m perf
```

Example output (for illustration only — actual results depend on your hardware and configuration):

```text theme={null}
╒══════════════════════════╤═════════╤═════════════════╤════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤══════╕
│ Performance Parameters   │ Stage   │ Average         │ Min            │ Max             │ Median          │ P75             │ P90             │ P99             │  N   │
╞══════════════════════════╪═════════╪═════════════════╪════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪══════╡
│ E2EL                     │ total   │ 6190.9 ms       │ 5071.4 ms      │ 8464.8 ms       │ 6126.6 ms       │ 6475.2 ms       │ 6833.5 ms       │ 7897.9 ms       │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ TTFT                     │ total   │ 693.3 ms        │ 96.0 ms        │ 2161.5 ms       │ 747.4 ms        │ 870.9 ms        │ 1032.3 ms       │ 1620.8 ms       │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ TPOT                     │ total   │ 21.6 ms         │ 17.8 ms        │ 32.1 ms         │ 21.3 ms         │ 23.1 ms         │ 24.5 ms         │ 29.1 ms         │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ ITL                      │ total   │ 25.5 ms         │ 0.0 ms         │ 1951.1 ms       │ 18.8 ms         │ 19.7 ms         │ 37.3 ms         │ 121.8 ms        │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ InputTokens              │ total   │ 0.0             │ 0.0            │ 0.0             │ 0.0             │ 0.0             │ 0.0             │ 0.0             │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ OutputTokens             │ total   │ 256.0           │ 256.0          │ 256.0           │ 256.0           │ 256.0           │ 256.0           │ 256.0           │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ OutputTokenThroughput    │ total   │ 41.6779 token/s │ 30.243 token/s │ 50.4791 token/s │ 41.7847 token/s │ 44.6424 token/s │ 45.6484 token/s │ 46.0932 token/s │ 1500 │
╘══════════════════════════╧═════════╧═════════════════╧════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧══════╛
╒═════════════════════════╤═════════╤══════════════════╕
│ Common Metric           │ Stage   │ Value            │
╞═════════════════════════╪═════════╪══════════════════╡
│ Benchmark Duration      │ total   │ 582099.6816 ms   │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Requests          │ total   │ 1500             │
├─────────────────────────┼─────────┼──────────────────┤
│ Failed Requests         │ total   │ 0                │
├─────────────────────────┼─────────┼──────────────────┤
│ Success Requests        │ total   │ 1500             │
├─────────────────────────┼─────────┼──────────────────┤
│ Concurrency             │ total   │ 15.9532          │
├─────────────────────────┼─────────┼──────────────────┤
│ Max Concurrency         │ total   │ 16               │
├─────────────────────────┼─────────┼──────────────────┤
│ Request Throughput      │ total   │ 2.5769 req/s     │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Input Tokens      │ total   │ 0                │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Generated Tokens  │ total   │ 384000           │
├─────────────────────────┼─────────┼──────────────────┤
│ Input Token Throughput  │ total   │ 0.0 token/s      │
├─────────────────────────┼─────────┼──────────────────┤
│ Output Token Throughput │ total   │ 659.6808 token/s │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Token Throughput  │ total   │ 659.6808 token/s │
╘═════════════════════════╧═════════╧══════════════════╛
```

> [!NOTE]
>
> See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details.

### 3.3 Using bench\_serving (image dataset)

Set `--dataset-name image` for image datasets. `bench_serving` will generate random prompts with image inputs. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark.

> [!NOTE]
>
> See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for the full list of image-related flags.

```shell Command theme={null}
python -m sglang.bench_serving \
  --backend sglang \
  --base-url http://127.0.0.1:30000 \
  --model Qwen/Qwen2.5-VL-7B-Instruct \
  --dataset-name image \
  --random-input-len 1024 \
  --random-output-len 512 \
  --random-range-ratio 1 \
  --num-prompts 32 \
  --max-concurrency 16 \
  --image-count 1 \
  --image-resolution 720p
```

Example output (for illustration only — actual results depend on your hardware and configuration):

```text theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang
Traffic request rate:                    inf
Max request concurrency:                 16
Successful requests:                     32
Benchmark duration (s):                  51.74
Total input tokens:                      73464
Total input text tokens:                 35128
Total input vision tokens:               38336
Total generated tokens:                  16384
Total generated tokens (retokenized):    9300
Request throughput (req/s):              0.62
Input token throughput (tok/s):          1419.96
Output token throughput (tok/s):         316.68
Peak output token throughput (tok/s):    800.00
Peak concurrent requests:                32
Total token throughput (tok/s):          1736.64
Concurrency:                             15.98
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   25841.84
Median E2E Latency (ms):                 25842.85
P90 E2E Latency (ms):                    26296.42
P99 E2E Latency (ms):                    26303.13
---------------Time to First Token----------------
Mean TTFT (ms):                          12211.59
Median TTFT (ms):                        14405.77
P99 TTFT (ms):                           15837.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          26.67
Median TPOT (ms):                        21.75
P99 TPOT (ms):                           41.89
---------------Inter-Token Latency----------------
Mean ITL (ms):                           26.67
Median ITL (ms):                         20.34
P95 ITL (ms):                            20.85
P99 ITL (ms):                            21.70
Max ITL (ms):                            11309.91
==================================================
```

## 4. Online Service: Embedding Model

Test `Qwen/Qwen3-Embedding-8B` on the embedding API endpoint.

> [!NOTE]
>
> Before running any benchmark in this section, make sure the SGLang embedding server is running with `--is-embedding` at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Embedding tab for the launch command. AISBench does not support embedding endpoints — use `bench_serving` or Evalscope instead.

### 4.1 Using Evalscope

> [!NOTE]
>
> Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang embedding server running with `--is-embedding` at `http://127.0.0.1:30000`.

Evalscope supports embedding evaluation. For performance testing the embedding API directly:

```shell Command theme={null}
evalscope perf \
  --parallel 10 \
  --number 20 \
  --model Qwen/Qwen3-Embedding-8B \
  --url http://127.0.0.1:30000/v1/embeddings \
  --api openai_embedding \
  --dataset random_embedding \
  --min-prompt-length 1024 \
  --max-prompt-length 1024 \
  --tokenizer-path Qwen/Qwen3-Embedding-8B
```

> [!TIP]
>
> If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.

> [!NOTE]
>
> Evalscope's embedding performance testing support may vary by version. If the `perf` command does not accept the embeddings endpoint, use [`bench_serving` with `--backend sglang-embedding`](#4-2-using-bench_serving-embedding-backend) as the primary option.

### 4.2 Using bench\_serving (embedding backend)

`bench_serving` is built into SGLang. Use `--backend sglang-embedding` to target the `/v1/embeddings` endpoint. Make sure the server is running with `--is-embedding` at `http://127.0.0.1:30000`.

```shell Command theme={null}
python -m sglang.bench_serving \
  --backend sglang-embedding \
  --base-url http://127.0.0.1:30000 \
  --model Qwen/Qwen3-Embedding-8B \
  --dataset-name random \
  --random-input-len 512 \
  --random-output-len 0 \
  --num-prompts 1000 \
  --max-concurrency 64 \
  --request-rate 32
```

> [!NOTE]
>
> `--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available. Set `--random-output-len 0` for embedding benchmarks — no output tokens are generated.

Example output (for illustration only — actual results depend on your hardware and configuration):

```text theme={null}
============ Serving Benchmark Result ============
Backend:                                 sglang-embedding
Traffic request rate:                    32.0
Max request concurrency:                 64
Successful requests:                     1000
Benchmark duration (s):                  31.86
Total input tokens:                      257891
Total input text tokens:                 257891
Request throughput (req/s):              31.39
Input token throughput (tok/s):          8094.67
Peak concurrent requests:                62
Concurrency:                             6.67
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   212.34
Median E2E Latency (ms):                 160.97
P90 E2E Latency (ms):                    267.31
P99 E2E Latency (ms):                    1445.94
==================================================
```

## 5. Offline Performance Testing

SGLang's `Engine` API runs inference in-process, without an HTTP server, letting you measure maximum throughput. `bench_offline_throughput` is built into SGLang and requires no extra installation or running server.

> [!NOTE]
>
> `bench_offline_throughput` currently only supports text-generation (LLM) benchmarks. Multimodal and embedding models are not supported.

### 5.1 Using bench\_offline\_throughput

`bench_offline_throughput` uses the `Engine` API internally and measures pure inference throughput without HTTP overhead:

```shell Command theme={null}
python -m sglang.benchmark.offline_throughput \
  --model-path Qwen/Qwen2.5-7B-Instruct \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 512 \
  --num-prompts 500
```

> [!NOTE]
>
> `--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available.

> [!TIP]
>
> `--dataset-name random` with `--random-input-len` and `--random-output-len` gives you full control over input/output token counts. Fixed-length random data eliminates variance from real datasets, making throughput comparisons across runs deterministic and reliable.

## See also

* [Bench Serving Guide](/docs/developer_guide/bench_serving) — all backends, datasets, and advanced options for `bench_serving`
* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — environment setup for Ascend NPUs
* [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) — full Evalscope documentation
* [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) — full AISBench documentation


# Troubleshooting and FAQ
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/faq



Contributions are welcome. Feel free to add more.

## 1. Context corruption with GLOO op.preamble.length \<= op.nbytes in PD disaggregation

### Error message

```text highlight=2,5-7 theme={null}
[2026-04-07 13:24:13 TP0] Decode batch, #running-req: 10, #token: 485248, token usage: 0.94, pre-allocated usage: 0.51, #prealloc-req: 1, #transfer-req: 12, #retracted-req: 0, npu graph: True, gen throughput (token/s): 259.82, #queue-req: 0
[2026-04-07 13:24:13 TP0] Context corruption detected: Request 3b5dcfe1575d4e1f9b18c953de878a93 (bootstrap_room=7451500070298748792) received metadata from bootstrap_room=4125156593077881415. Metadata buffer index: 1. This indicates metadata buffer index collision.
[2026-04-07 13:24:13] INFO:      127.0.0.1:59272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-04-07 13:24:13] INFO:      127.0.0.1:34000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
terminate called after throwing an instance of 'gloo::EnforceNotMet'
 what():  [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:456] op.preamble.length <= op.nbytes. 4 vs 3
Fatal Python error: Aborted

Thread 0x0000fff873f6f120 (most recent call first):
 File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/srt/disaggregation/mooncake/conn.py", line 1499 in heartbeat_checker
```

### Cause

(Possibly, not precisely located) High-concurrency long sequences fill up the transfer buffer, causing buffer index collision and data corruption.

### Solution

1. Disable overlap on the Prefill node with `--disable-overlap-schedule`. The Prefill node in PD disaggregation must not enable overlap, otherwise it causes timing issues that lead to out-of-order reception on the Decode node.
2. Even with overlap disabled, multi-Prefill node high-concurrency long-sequence scenarios may still encounter this issue with low probability. This is a known issue pending resolution.

## 2. Graph mode aclnnInplaceFillScalar error

### Error message

```text highlight=2 theme={null}
(SGLangEngine pid=3872176) [rank0]:[E414 12:14:41.204711510 compiler_depend.ts:444] operator():build/CMakeFiles/torch_npu.dir/compiler_depend.ts:26 NPU function error:
 call aclnnInplaceFillScalar failed, error code is 507000
(SGLangEngine pid=3872176) [ERROR] 2026-04-14-12:14:41 (PID:3874122, Device:0, RankID:-1) ERR00100 PTA call acl api failed
(SGLangEngine pid=3872176) [Error]: An internal error occurs in the runtime module on the host.
(SGLangEngine pid=3872176) Rectify the fault based on the error information in the ascend log.
(SGLangEngine pid=3872176) [PID: 3874122] 2026-04-14-12:14:41.897.548 AclNN_Runtime_Error(EZ9903): aclrtLaunchKernelWithHostArgs failed, return: 507000
(SGLangEngine pid=3872176)  Solution: In this scenario, collect the plog when the fault occurs and locate the fault based on the plog.
(SGLangEngine pid=3872176)  TraceBack (most recent call last):
(SGLangEngine pid=3872176)  Check kernel task failed, stream_id=2028, task_id=48, retCode=0x7080005.[FUNC:LaunchKernel][FILE:context.cc][LINE:1585]
(SGLangEngine pid=3872176)  rtsLaunchKernelWithHostArgs execution failed, reason=kernel type error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:61]
(SGLangEngine pid=3872176)  rtsLaunchKernelWithHostArgs failed, runtime result = 507000.[FUNC:ReportCallError][FILE:Log_inner.cpp][LINE:148]
(SGLangEngine pid=3872176)  aclrtLaunchKernelWithHostArgs failed, return: 507000
(SGLangEngine pid=3872176)  Launch kernel failed.
(SGLangEngine pid=3872176)  #### KernelLaunch failed: /home/850b160/cann-8.5.8/opp/built-in/op_impl/ai_core/tbe//kernel/ascend910_93/ops_legacy/fill/Fill_41dadce325bOf810d03359af2a38990b_high_performance.o
(SGLangEngine pid=3872176)  Kernel Run failed. opType: 18, Fill
(SGLangEngine pid=3872176)  launch failed for Fill, errno:361001.
(SGLangEngine pid=3872176)
(SGLangEngine pid=3872176) Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler depend.ts:26 (most recent call first):
(SGLangEngine pid=3872176) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+ 0xb0 (0xffff806848c0 in /root/anaconda3/envs/slime_re/Lib/python3.11/site-packages/torch/lib/libc10.so)
(SGLangEngine pid=3872176) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x68(0xffff8062c140 in /root/anaconda3/envs/slime_re/Tib/puthon3.11/site-packages/torch/lib/libc10.so)
(SGLangEngine pid=3872176) frame #2: <unknown function> + 0x110e2b4 (0xffff6d44e2b4 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
(SGLangEngine pid=3872176) frame #3: <unknown function> + 0x29f0894(0xffff6ed30894 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
(SGLangEngine pid=3872176) frame #4: <unknown function> + 0x9cc708(0xffff6cd0c700 in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
(SGLangEngine pid=3872176) frame #5: <unknown function> + 0x9cd2dc (0xffff6cd0d2dc in /root/anaconda3/envs/slime_re/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
```

### Cause

Too many captured graphs cause a conflict in the graph mode update stream. Each graph is placed on a separate stream, but the number of streams is limited. If too many graphs are captured, conflicts occur.

### Solution

* CANN 8.5 + TorchNPU 2.8 should have resolved this issue.
* If your versions do not match, reduce the number of captured graphs to 10 or fewer.

## 3. alloc\_extend\_kernel error

### Error message

```text theme={null}
[root@os-node-created-6z9tp pd_7p1d_tp2_20260414_030537]# grep -nR -E "aivec error|ACL" *,log
prefill 1.log:5739:EZ9999[PID: 164243] 2026-04-14-06:40:35.033.745 (EZ9999): The error from device(chipId:0, dieId:1), serial number is 1, there is an exception of aivec error, core id is 23, error code = 0, dump info: pc start: 0x12420156a000, current: 0x12420156ad74, vec error info: 0xdb1751f50e, mte error info: 0x98f6388707, ifu error info: 0x212c93fc00000, ccu error info: 0x998e981c00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc00.[FUNC:PrintCoreInfo][FILE:device error core proc.cc][LINE:347]
prefill_1.log:5747:EZ9999[PID: 164242] 2026-04-14-06:40:35.034.456 (EZ9999):  The error from device(chipId:0, dieId:0), serial number is 1, there is an exception of aivec error, core id is 45, error code = O, dump info: pc start: 0x12400156a000, current: 0x12400156ad74, vec error info: 0x7304583407, mte error info: 0x27e2b1c765, ifu error info: 0x212c93fc00000, ccu error info: 0x6e0836f300000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc0O.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
prefill_1.Log:5784:RuntimeError: ACL stream synchronize failed, error code:507035
prefill_1.Log:5816:RuntimeError: ACL stream synchronize failed, error code:507035
prefill 2.log:2566:EZ9999[PID: 163475] 2026-04-14-05:28:43.282.016 (EZ9999):  The error from device(chipId:1, dieId:1), serial number is 1, there is an exception of aivec error, core id is 20, error code = 0, dump info: pc start: 0x124601550000, current: 0x124601550d74, vec error info: 0x9b07e496e3, mte error info: 0xc2a6806828, ifu error info: 0x212c93f200000, ccu error info: 0xaa64401100000000, cube error info: O, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core proc.cc][LINE:347]
prefill 2.log:2574:EZ9999[PID: 163474] 2026-04-14-05:28:43.282.809 (EZ9999):  The error from device(chipId:1, dieId:0), serial number is 1, there is an exception of aivec error, core id is 22, error code = 0, dump info: pc start: 0x124401550000, current: 0x124401550d74, vec error info: 0xa80496049a, mte error info: 0xa0770e2bf2, ifu error info: 0x212c93f200000, ccu error info: 0x1083000000000000, cube error info: O, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100c72400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
prefill 2.log:2611:RuntimeError: ACL stream synchronize failed, error code:507035
prefill 2.log:2644:RuntimeError: ACL stream synchronize failed, error code:507035
prefill 4.log:1323:EZ9999[PID: 163478] 2026-04-14-05:08:15.363.509 (EZ9999):  The error from device(chipId:3, dieId:0), serial number is 1, there is an exception of aivec error, core id is 5, error code = 0, dump info: pc start: 0x124c00dff000, current: 0x124c00dff860, vec error info: 0xf311fb8727, mte error info: 0x45418486a, ifu error info: 0x212c93fa00000, ccu error info: 0x17a9996f00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
prefill 4.log:1331:EZ9999[PID: 163479] 2026-04-14-05:08:15.364.016 (EZ9999):  The error from device(chipId:3, dieId:1), serial number is 1, there is an exception of aivec error, core id is 40, error code = 0, dump info: pc start: 0x124e00dff000, current: 0x124e00dff860, vec error info: 0xf11a9cf188, mte error info: 0xfb7710514a, ifu error info: 0x212c93fa00000, ccu error info: 0x1f5828a900000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c100d40c00.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
prefill 4.Log:1368:RuntimeError: ACL stream synchronize failed, error code:507035
prefill 4.log:1400:RuntimeError: ACL stream synchronize failed, error code:507035
prefill 7.log:3017:EZ9999[PID: 164628] 2026-04-14-05:34:12.369.755 (EZ9999): The error from device(chipId:6, dieId:0), serial number is 1, there is an exception of aivec error, core id is 33, error code = 0, dump info: pc start: 0x1258015ae000, current: 0x1258015aed74, vec error info: 0x740053222c, mte error info: 0x97103df5a0, ifu error info: 0x212c93f400000, ccu error info: 0x391c89ab00000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
prefill 7.log:3025:EZ9999[PID: 164629] 2026-04-14-05:34:12.369.742 (EZ9999):  The error from device(chipId:6, dieId:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = 0, dump info: pc start: 0x125a015ae000, current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: O, aic error mask: 0x6500020bd00028c, para base: 0x12c10Ocd6400.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347]
prefill 7.log:3062:RuntimeError: ACL stream synchronize failed, error code:507035
prefill 7.log:3094:RuntimeError: ACL stream synchronize failed, error code:507035
```

### Error plog

```text highlight=21 theme={null}
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.353.461 [stars engine.cc:1534]170327 ProcLogicCaReport:Task run failed, device id=13, stream id=43, task id=13145, sqe type=0(ffts), errType=0x1(task exception), sqSwStatus=0
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.720 device error core proc.cc:3211170327 AddExceptionReqInfo:add error register: core id=42, stream id=43, task id=13145
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.730 device error core proc.cc:3471170327 PrintCorelnfo:The error from device(chipld:6, dield:1), serial number is 1, there is an exception of aivec error, core id is 42, error code = O, dump info: pc start: 0x125a015ae000,current: 0x125a015aed74, vec error info: 0xb91ae07b35, mte error info: 0xfc000670ef, ifu error info: 0x212c93f400000, ccu error info: 0x57c069b000000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100cd6400.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.774 device error core proc.cc:3601170327 PrintCorelnto:The extend into: errcode:(0, 0x8000, 0) errorStr: When the D-cache reads and writes data to the UB, the response value returned bv the bus is a non-zero value. fixp_error0 info: 0x670ef, fixp error1 info: 0xfc, fsmId:0, tslot:2, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.787 device error core proc.cc:4341170327 ProcessStarsCoreErrorInfo:devId=13, streamId=43, taskId=13145, MTE errorCode=0.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.369.795 davinci task.cc:2011170327 SetStarsResultForDavinciTask:AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.644 davinci kernel task.cc:15821170327 PreCheckTaskErr:Kernel task happen error retCode=0x31, vector core exception1.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.705 davinci kernel task.cc:14241170327 GetArasInfo:[AIC INFO] aras(0 to 9) after execute:0x3fffffb9000, 0, 0, 0x12c9323ff600, 0x12c93231ee00, 0x12c93f1d8800,0x12c93f3ff600,0x12c958200000,0x100000003, 0xaaaa00000001.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.710 [davinci kernel task.cc:1427]170327 GetArgsInfo:tilingKey = 0, print 1 Times totalLen=(10*8), argsSize=80, blockDim=3
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.717 [davinci kernel task.cc:1468]170327 PrintErrorInfoForDavinciTask:[AIC INFO] after execute:arqs print end
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.751 davinci kernel task.cc:14981170327 PrintErrorInfoForDavinciTask:[DFX INFO]Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, flip num=56, fault kernel_name=alloc_extend_kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248.
[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.823 [dump manager.cpp:41][tid:170327] An exception callback message is received.
[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.971 [kernel info collector.cpp:384][tid:170327] Get error register information. coreNum=0
[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.981 kernel info collector.cpp:4771tid:1703271 It is Non-SuperKernel. functionCount=1, qlobalCount=1
[ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.987 [dump args.cpp:668][tid:170327] In arqAddr[0x12c100cd6400]|arqSize[80]dfxAddr[(nil)]|dfxSize[0] has invalid attribute.
[ERROR] IDEDD(164629,):2026-04-14-05:34:12.383.894 [dump_printf.cpp:1118][tid:170327] infoAddr is null
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.908 [stream.cc:1332]170327 GetError:Stream Synchronize failed, stream id=43, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.911 [stream.cc:1335]170327 GetError:AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.929 [stream.cc:1335]170327 GetError:[AIC_INFO] after execute:args print end
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.936 stream.cc:13351170327 GetError: DFX INFO1Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, tlip num=56,
 fault kernel_name=alloc_extend_kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.943 [stream.cc:3549]170327 EnterFailureAbort:stream id=43 enter failure abort.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.973 [stars_engine.cc:1427]170327 StarsResumeRtsa:stop scheduling in abort failure mode: stream id=43, sq id=6,sq head=801, task id=13145, taskType=66.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.972 [stream.cc:1463]164629 SynchronizeExecutedTask:context is abort, status=0x715005e.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.978 [stream.cc:1516]164629 Synchronizelmpl:failed, stream_id=43, error=0x715005e
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.982 [api error.cc:1015]164629 StreamSynchronize:Stream synchronize failed, stream_id=43, timeout=-1ms.
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.991 [apic stream.cc:154]164629 rtStreamSynchronize:ErrCode=507035, desc=[vector core exception], InnerCode=0x715005e
[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.997 [error_message_manage.cc:61]164629 FuncErrorReason:rtStreamSynchronize execution failed, reason=vector core exception
[ERROR] ASCENDCL(164629,):2026-04-14-05:34:12.384.269 [stream.cpp:140]164629 acIrtSynchronizeStreamImpl:synchronize stream failed, runtime result = 507035
```

### Cause

The `alloc_extend_kernel` operator appears to have a memory allocation issue. Pending resolution.

### Solution

Modify `sglang/srt/hardware_backend/npu/allocator_npu.py` to comment out the affected branch and use the else branch instead.

```python highlight=15-36 theme={null}
    def alloc_extend(
        self,
        prefix_lens: torch.Tensor,
        prefix_lens_cpu: torch.Tensor,
        seq_lens: torch.Tensor,
        seq_lens_cpu: torch.Tensor,
        last_loc: torch.Tensor,
        extend_num_tokens: int,
        num_new_pages: int = None,
    ):
        ...
        if num_new_pages_item > len(self.free_pages):
            return None

        # if num_new_pages_item < 200:
        #     from sgl_kernel_npu.mem_cache.allocator import alloc_extend_kernel

        #     out_indices = torch.empty(
        #         (extend_num_tokens,),
        #         dtype=torch.int64,
        #         device=self.device,
        #     )
        #     max_num_extend_tokens = next_power_of_2(extend_num_tokens)
        #     bs = prefix_lens.shape[0]
        #     alloc_extend_kernel[(bs,)](
        #         prefix_lens,
        #         seq_lens,
        #         last_loc,
        #         self.free_pages,
        #         out_indices,
        #         next_power_of_2(bs),
        #         self.page_size,
        #         max_num_extend_tokens,
        #     )

        # else:
        out_indices = torch.empty(
            (extend_num_tokens,),
            dtype=torch.int32,
            device=self.device,
        )
        ...
```

## 4. Out of NPU memory

```text highlight=7 theme={null}
  File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 175, in calculate_pool_sizes
    return MemoryPoolConfig(max_total_num_tokens=max_total_num_tokens)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 8, in __init__
  File "/home/code/sglang/python/sglang/srt/model_executor/pool_configurator.py", line 44, in __post_init__
    raise RuntimeError(msg)
RuntimeError: Not enough memory. Please try to increase --mem-fraction-static.
```

### Solution

First, use the `npu-smi info` command to check the NPU memory usage.

If the NPUs are occupied by other processes, use `--base-gpu-id` to specify the starting device index.

If the NPUs are not occupied, you can use `--tp` to deploy across multiple devices, or reduce the KV cache memory usage by increasing the `--mem-fraction-static` value. For detailed tuning guidance, see [Hyperparameter Tuning](/docs/advanced_features/hyperparameter_tuning).

## 5. How to update sgl-kernel-npu

### Solution

```bash theme={null}
git clone https://github.com/sgl-project/sgl-kernel-npu.git

source /usr/local/Ascend/ascend-toolkit/set_env.sh
cd sgl-kernel-npu
# Building Project
bash build.sh

pip install output/sgl_kernel_npu*.whl --force-reinstall

# (Optional) Confirm whether the import can be successfully
python -c "import sgl_kernel_npu; print(sgl_kernel_npu.__path__)"

rm -rf sgl-kernel-npu
```

## 6. `[Errno 101] Network is unreachable` when downloading HuggingFace datasets

### Error message

```text highlight=1-2 theme={null}
'[Errno 101] Network is unreachable' thrown while requesting HEAD https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
Retrying in 1s [Retry 1/5].
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 2353, in <module>
    run_benchmark(args)
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/bench_serving.py", line 1848, in run_benchmark
    input_requests = get_dataset(args, tokenizer, model_id)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/__init__.py", line 44, in get_dataset
    return dataset.load(tokenizer=tokenizer, model_id=model_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 45, in load
    return sample_random_requests(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/datasets/random.py", line 89, in sample_random_requests
    dataset_path = download_and_cache_hf_file(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/python3.11.14/lib/python3.11/site-packages/sglang/benchmark/utils.py", line 98, in download_and_cache_hf_file
    return hf_hub_download(repo_id=repo_id, filename=filename, repo_type=repo_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ...
RuntimeError: Cannot send a request, as the client has been closed.
[ERROR] 2026-05-18-11:58:31 (PID:215, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
```

### Cause

The machine cannot directly access the HuggingFace server due to network restrictions (firewall, proxy, or regional access limitations).

### Solution

* **Use an HF mirror site** — set the `HF_ENDPOINT` environment variable to a mirror (e.g., hf-mirror.com):

  ```bash theme={null}
  export HF_ENDPOINT=https://hf-mirror.com
  ```

* **Use a proxy** — if you have an HTTP proxy available:

  ```bash theme={null}
  export http_proxy=http://your-proxy:port
  export https_proxy=http://your-proxy:port
  ```

* **Download the dataset manually** — use a machine with network access to download the file, then transfer it to the target machine. Use `--dataset-path` to specify the local file path:

  ```bash theme={null}
  python -m sglang.bench_serving \
    --dataset-name random \
    --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
    ...
  ```

## 7. Unexpected type fp8

### Cause

FP8 model is not supported

### Solution

Replace model weights, e.g., switch Qwen/Qwen3.5-27B-FP8 to Eco-Tech/Qwen3.5-27B-w8a8-mtp.

## 8. Docker image versions: stable release vs. daily build

Docker images for Ascend NPU are available in two types:

* **Stable release** — validated version with a specific tag. Recommended for production deployments.
* **Daily build** — includes the latest development changes, e.g., `quay.io/ascend/sglang:main-cann9.1.0-a3`. Use this
  if you need the latest features or bug fixes that have not yet been included in a stable release.

If you encounter issues with a stable release, try switching to a daily build to see if the issue has been resolved in the latest development version.


# Installation
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/installation

Complete installation guide for SGLang on Ascend NPUs, including component version mapping, environment setup, and launching inference services.

You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.

## Component Version Mapping For SGLang

Each CANN version requires its own matched set of component versions — do not mix versions across sets.

**Ascend A3 Series**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Component</th>
      <th>Version</th>
      <th>Obtain Method</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>HDK</td>
      <td>25.5.2</td>
      <td>[link](https://www.hiascend.com/hardware/firmware-drivers/community?product=1\&model=30\&cann=9.1.0\&driver=Ascend+HDK+25.5.2)</td>
    </tr>

    <tr>
      <td>CANN</td>
      <td>9.1.0</td>
      <td>[Obtain Images](#obtain-cann-image)</td>
    </tr>

    <tr>
      <td>TorchNPU</td>
      <td>2.10.0.post6</td>
      <td>[link](https://gitcode.com/Ascend/pytorch/releases)</td>
    </tr>

    <tr>
      <td>MemFabric</td>
      <td>1.2.1</td>
      <td>[memfabric\_hybrid-1.2.1](https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl)\
      (x86\_64 host: use the manylinux\_2\_27\_x86\_64.manylinux\_2\_28\_x86\_64 wheel from the same directory)</td>
    </tr>

    <tr>
      <td>memcache</td>
      <td>1.2.1</td>
      <td>[memcache\_hybrid-1.2.1](https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/memcache/v1.2.1/20260923.4/memcache_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl)\
      (x86\_64 host: use the manylinux\_2\_27\_x86\_64.manylinux\_2\_28\_x86\_64 wheel from the same directory)</td>
    </tr>

    <tr>
      <td>Triton</td>
      <td>3.2.2</td>
      <td>[triton\_ascend-3.2.2](https://github.com/triton-lang/triton-ascend/releases/download/v3.2.2/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl)\
      (x86\_64 host: replace aarch64 with x86\_64)</td>
    </tr>

    <tr>
      <td>SGLang NPU Kernel</td>
      <td>2026.9.0.post9</td>
      <td>[link](https://github.com/sgl-project/sgl-kernel-npu/releases/tag/2026.9.0.post9)</td>
    </tr>

    <tr>
      <td>MemFabric-zbal</td>
      <td>1.1.3</td>
      <td>`pip install memfabric-zbal==1.1.3`</td>
    </tr>
  </tbody>
</table>

**Ascend 950PR/DT Series**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Component</th>
      <th>Version</th>
      <th>Obtain Method</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>HDK</td>
      <td>25.5.2</td>
      <td>[link](https://www.hiascend.com/hardware/firmware-drivers/community?product=1\&model=30\&cann=9.1.0\&driver=Ascend+HDK+25.5.2)</td>
    </tr>

    <tr>
      <td>CANN</td>
      <td>9.1.0</td>
      <td>[Obtain Images](#obtain-cann-image)</td>
    </tr>

    <tr>
      <td>TorchNPU</td>
      <td>2.10.0.post6</td>
      <td>[link](https://gitcode.com/Ascend/pytorch/releases)</td>
    </tr>

    <tr>
      <td>MemFabric</td>
      <td>1.2.1</td>
      <td>[memfabric\_hybrid-1.2.1](https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl)\
      (x86\_64 host: use the manylinux\_2\_27\_x86\_64.manylinux\_2\_28\_x86\_64 wheel from the same directory)</td>
    </tr>

    <tr>
      <td>memcache</td>
      <td>1.2.1</td>
      <td>[memcache\_hybrid-1.2.1](https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/memcache/v1.2.1/20260923.4/memcache_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl)\
      (x86\_64 host: use the manylinux\_2\_27\_x86\_64.manylinux\_2\_28\_x86\_64 wheel from the same directory)</td>
    </tr>

    <tr>
      <td>Triton</td>
      <td>3.2.2</td>
      <td>[triton\_ascend-3.2.2](https://github.com/triton-lang/triton-ascend/releases/download/v3.2.2/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl)\
      (x86\_64 host: replace aarch64 with x86\_64)</td>
    </tr>

    <tr>
      <td>SGLang NPU Kernel</td>
      <td>2026.9.0.post9</td>
      <td>[link](https://github.com/sgl-project/sgl-kernel-npu/releases/tag/2026.9.0.post9)</td>
    </tr>
  </tbody>
</table>

### Obtain CANN Image

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space.

You can obtain the dependency of a specified version of CANN through an image.

**Ascend A3 Series**

```bash Command theme={null}
docker pull quay.io/ascend/cann:9.1.0-a3-ubuntu22.04-py3.12
```

**Ascend 950PR/DT Series**

```bash Command theme={null}
docker pull quay.io/ascend/cann:9.1.0-950-ubuntu22.04-py3.12
```

## Preparing the Running Environment

### Method 1: Installing from source with prerequisites

The steps in this method install the component set from the [version mapping table](#component-version-mapping-for-sglang).

#### Python Version

**The CANN 9.1.0 set requires `python==3.12`.** If you don't want to break
system pre-installed python, try installing with [conda](https://github.com/conda/conda).

```bash Command theme={null}
conda create --name sglang_npu python=3.12
conda activate sglang_npu
```

**Note on Anaconda repository restrictions**

If you encounter an error like “Terms of Service have not been accepted” during the conda create step, the default Anaconda repository is blocking package downloads. To resolve this, configure a mirror (e.g., Tsinghua Open Source Mirror):

```bash Command theme={null}
# Add Tsinghua mirrors
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
conda config --set show_channel_urls yes
conda config --remove channels defaults
```

Edit the system-level conda config to remove any hardcoded defaults, e.g., `vi ~/miniconda3/.condarc`.
Then remove the failed environment and recreate it:

```bash Command theme={null}
conda clean -i
conda env remove -n sglang_npu
conda create --name sglang_npu python=3.12
conda activate sglang_npu
```

#### CANN

Prior to starting work with SGLang on Ascend, you need to install CANN Toolkit, Kernels operator package and NNAL version 9.1.0, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910/softwareinst/instg/instg_0008.html?OS=openEuler\&InstallType=local)

#### MemFabric-Hybrid

If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.

```bash Command theme={null}
# On an x86_64 host, use memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl from the same directory instead
pip install https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl

# Install the device-specific MemFabric kernels:
# --soc-version is A5 on the 950PR/DT Series, A3 on the A3 Series
mfcli kernel install --soc-version A5
```

#### MemFabric-zbal

MemFabric-zbal is a Zero Buffer Acceleration Library of high-performance operators for LLM inference and training on Ascend,
accelerating computation by eliminating intermediate memory buffers; it is required only on **aarch64** clusters and is
installed in addition to MemFabric-Hybrid.

```bash Command theme={null}
# Only needed on aarch64 (arm64) hosts
# A3 Series:
pip install memfabric-zbal==1.1.3
```

#### PyTorch and PyTorch Framework Adapter on Ascend

```bash Command theme={null}
PYTORCH_VERSION=2.10.0
TORCHVISION_VERSION=0.25.0
TORCH_NPU_VERSION=2.10.0.post6
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu==$TORCH_NPU_VERSION --extra-index-url https://ascend.devcloud.huaweicloud.com/pypi/simple/
```

If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md)

#### Triton on Ascend

We provide our own implementation of Triton for Ascend.

```bash Command theme={null}
# On an x86_64 host, replace aarch64 with x86_64 in the wheel URL
pip install https://github.com/triton-lang/triton-ascend/releases/download/v3.2.2/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl
```

For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://github.com/triton-lang/triton-ascend/blob/main/docs/en/installation_guide.md)

#### SGLang Kernels NPU

We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md).

> [!NOTE]
>
> Gemma-family models (including Qwen3.5) call `sgl_kernel_npu.norm.gemma_rmsnorm`,
> whose implementation is chosen when the wheel is built: the native `torch_npu`
> operator for Ascend 910B/910C, and ACLNN RMSNorm for Ascend 950, which does not
> register that operator. **Ascend 950 users must install the `950` release
> package** (`950` appears in the asset name), or build from source with
> `bash build.sh -a kernels Ascend950PR_9599`, the target that package is built
> with. On Ascend 950, a `910b` or `a3` package fails on the first Gemma forward,
> and so does a package from a release that predates this module; check with
> `python -c "import sgl_kernel_npu.norm.gemma_rmsnorm"`.

#### DeepEP-compatible Library

We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md).

#### Some other dependencies

```bash Command theme={null}
# libGL
apt update
apt install libgl1 libglib2.0-0

# ensure setuptools contains pkg_resources module
pip install "setuptools<80"
```

#### Installing SGLang from source

```bash Command theme={null}
# Use the last release branch
git clone https://github.com/sgl-project/sglang.git
cd sglang
mv python/pyproject_npu.toml python/pyproject.toml
pip install -e python[all_npu]
```

### Method 2: Using Docker Image

#### Obtain Image

You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space.

1. Download SGLang image

> [!TIP]
>
> We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version, or a daily build tag (e.g., `main-cann9.1.0-a3`, `main-cann9.1.0-950`) if you need the latest development changes. CANN 9.1.0 has no stable image yet, so use the daily build tag for now.
>
> Validated release builds are also published as `cann9.1.0-<device>-B<NN>` tags (e.g., `cann9.1.0-950-B080`). The exact component versions inside each release image are listed in the [image dependency manifest](https://ascend.github.io/sglang/) — see for example the `cann9.1.0-950-B100` entry for the 950PR/DT Series.

**Ascend A3 Series**

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3
```

**Ascend 950PR/DT Series**

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-950
```

2. Build an image based on Dockerfile

```bash Command theme={null}
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker

# Build the docker image
# Replace <arch_tag> with the target architecture, e.g., amd64, arm64.
# Optional build arguments:
#   --build-arg APTMIRROR=<mirror_url>    # Use a custom APT mirror to improve download speed
# If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV.
docker build --build-arg TARGETARCH=<arch_tag> -t <image_name> -f npu.Dockerfile .
```

#### Create Docker

**Notice:** `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.

**Ascend A3 Series**

```bash Command theme={null}
# Create a shortcut 'drun' to launch a privileged Docker container
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
    --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
    --device=/dev/davinci_manager --device=/dev/hisi_hdc \
    --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    --volume /etc/ascend_install.info:/etc/ascend_install.info \
    --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'

# Add HF_TOKEN env for downloading models by SGLang.
# The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C)
drun --env "HF_TOKEN=<secret>" \
    <image_name> \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```

**Ascend 950PR/DT Series**

```bash Command theme={null}
# Create a shortcut 'drun' to launch a privileged Docker container
# The 950PR/DT Series exposes 8 devices: /dev/davinci0 through /dev/davinci7
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
    --env ASCEND_USE_FIA=1 \
    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci_manager --device=/dev/hisi_hdc \
    --volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    --volume /etc/ascend_install.info:/etc/ascend_install.info \
    --volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'

# Add HF_TOKEN env for downloading models by SGLang.
# The container runs with the '--rm' flag, so it will be automatically removed after the command finishes (including Ctrl+C)
drun --env "HF_TOKEN=<secret>" \
    <image_name> \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```

> [!TIP]
>
> SGLang will serve on `http://127.0.0.1:30000` by default. You can change the host and port by `--host` and `--port` parameters.

## System Settings

### CPU performance power scheme

The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.

```bash Command theme={null}
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Make sure changes are applied successfully
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
```

### Disable NUMA balancing

```bash Command theme={null}
sudo sysctl -w kernel.numa_balancing=0
# Check
cat /proc/sys/kernel/numa_balancing # shows 0
```

### Prevent swapping out system memory

```bash Command theme={null}
sudo sysctl -w vm.swappiness=10

# Check
cat /proc/sys/vm/swappiness # shows 10
```

## Running SGLang Service

### Running Service For Large Language Models

#### PD Mixed Scene

**Ascend A3 Series**

```bash Command theme={null}
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --attention-backend ascend \
  --host 127.0.0.1 \
  --port 30000
```

**Ascend 950PR/DT Series**

```bash Command theme={null}
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --attention-backend ascend \
  --host 127.0.0.1 \
  --port 30000
```

#### PD Disaggregation Scene

1. Launch Prefill Server

**Ascend A3 Series**

```bash Command theme={null}
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1

# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode prefill \
    --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8995 \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 0 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30001
```

**Ascend 950PR/DT Series**

```bash Command theme={null}
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
# PD disaggregation transfer protocol on the 950PR/DT Series
export ASCEND_MF_TRANSFER_PROTOCOL="device_urma"

# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode prefill \
    --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8995 \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 0 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30001
```

2. Launch Decode Server

**Ascend A3 Series**

```bash Command theme={null}
# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode decode \
    --disaggregation-transfer-backend ascend \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 1 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30002
```

**Ascend 950PR/DT Series**

```bash Command theme={null}
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
# PD disaggregation transfer protocol on the 950PR/DT Series
export ASCEND_MF_TRANSFER_PROTOCOL="device_urma"

# PREFILL_IP: IP address of the first Prefill Server
# FREE_PORT: any available port
# all SGLang servers need to be configured with the same PREFILL_IP and FREE_PORT
export ASCEND_MF_STORE_URL="tcp://PREFILL_IP:FREE_PORT"
python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --disaggregation-mode decode \
    --disaggregation-transfer-backend ascend \
    --attention-backend ascend \
    --device npu \
    --base-gpu-id 1 \
    --tp-size 1 \
    --host 127.0.0.1 \
    --port 30002
```

3. Launch Router

```bash Command theme={null}
python3 -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://127.0.0.1:30001 8995 \
    --decode http://127.0.0.1:30002 \
    --host 127.0.0.1 \
    --port 30000
```

> [!NOTE]
>
> The `8995` in command script is the disaggregation bootstrap port. It must match the
> `--disaggregation-bootstrap-port` value set on the prefill server in step 1.

### Running Service For Multimodal Language Models

#### PD Mixed Scene

**Ascend A3 Series**

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
    --host 127.0.0.1 \
    --port 30000 \
    --tp 4 \
    --device npu \
    --attention-backend ascend \
    --mm-attention-backend ascend_attn \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-multimodal \
    --sampling-backend ascend
```

**Ascend 950PR/DT Series**

```bash Command theme={null}
# ASCEND_USE_FIA=1 is required on the 950PR/DT Series
export ASCEND_USE_FIA=1
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \
    --host 127.0.0.1 \
    --port 30000 \
    --tp 4 \
    --device npu \
    --attention-backend ascend \
    --mm-attention-backend ascend_attn \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-multimodal \
    --sampling-backend ascend
```

## Testing the Service

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests.

### Which port to send requests to

The port you use depends on your deployment mode:

| Scenario | Where to send requests |
| - | - |
| Non-PD (single server) | The server's `--port` (e.g., `30000` in the examples above) |
| Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes |
| PD disaggregation | The router's `--port` (e.g., `30000` in the examples above); do **not** send requests directly to prefill or decode servers |

> [!TIP]
>
> SGLang defaults to port `30000` when `--port` is not specified. The examples in this guide use explicit ports for clarity.

### Health Check

```bash Command theme={null}
curl http://127.0.0.1:30000/health
```

A successful response returns HTTP 200 with an empty body.

### Generate (Native Endpoint)

```bash Command theme={null}
curl http://127.0.0.1:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "What is the capital of France?",
    "sampling_params": {"temperature": 0, "max_new_tokens": 128}
  }'
```

The expected output should contain "Paris".

### Chat Completions (OpenAI-Compatible)

```bash Command theme={null}
curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'
```

Some models return responses accompanied by thinking process content. To disable this output, configure parameters as follows:

```bash Command theme={null}
curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Eco-Tech/Qwen3.5-27B-w8a8-mtp",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "chat_template_kwargs": {"enable_thinking": false}
  }'
```

The expected output should contain "Paris".

### Multimodal Chat Completions

> [!NOTE]
>
> The image URL in the example below references an external resource (`raw.githubusercontent.com`). Make sure the server has internet access so the image can be downloaded at inference time. Alternatively, you can use a locally accessible URL or base64-encoded image data.

```bash Command theme={null}
curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-VL-30B-A3B-Instruct",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
        {"type": "text", "text": "Describe this image."}
      ]
    }]
  }'
```


# Quickstart
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/quick_start



> [!NOTE]
>
> This page covers only the simplest deployment flow using the official container image. For the complete installation guide
> across all scenarios (source install, Docker build, system settings, version mapping, etc.),
> see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus/getting-started/installation).

## Prerequisites

### Supported Devices

* Ascend A3 Series Products

To identify your device, run `npu-smi info -l`: A3 Series reports `Chip Count: 2` per NPU.
For hardware details, see the [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

### Docker

Ensure Docker is installed and the Docker daemon is running on your host machine. Verify with:

```bash theme={null}
docker --version && docker info
```

If Docker is not installed, follow the [official Docker installation guide](https://docs.docker.com/engine/install/) for
your operating system.

## Setup environment using container

> [!WARNING]
>
> Ensure sufficient disk space before proceeding. Run `df -h` to check the available disk space. The Docker image requires
> at least **30GB** of free space. If you need to download model weights, check the model size at
> [ModelScope](https://www.modelscope.cn/models) to reserve enough space.

> [!TIP]
>
> We publish both **stable releases** and **daily builds**. Choose a stable release tag if you prefer a validated version,
> or a daily build tag (e.g., `main-cann9.1.0-a3`) if you need the latest development changes. CANN 9.1.0 has no stable
> image yet, so use the daily build tag for now.

If you have already downloaded model weights to a local path (e.g., `/path/to/model`), mount the path into the container by
adding `--volume /path/to/model:/path/to/model` to the `docker run` command below.

**Ascend A3 Series**

```shell Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so only the daily build tag is
# available for now. Once the stable image is published, restore the stable
# release option here.
export IMAGE=quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
    --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
    --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
    --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
    --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --volume /usr/local/sbin:/usr/local/sbin \
    --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    --volume /etc/ascend_install.info:/etc/ascend_install.info \
    --volume /var/queue_schedule:/var/queue_schedule \
    --volume ~/.cache/:/root/.cache/ \
    --entrypoint=bash \
    $IMAGE
```

## Usage

The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version.

### Start SGLang server

SGLang will automatically download the model from Hugging Face. If the model is already downloaded to a local path
(and has been mounted into the container), use that path directly like `--model-path /path/to/model`.

```shell Command theme={null}
# Set HF_ENDPOINT to a mirror site if network is not available
export HF_ENDPOINT=https://hf-mirror.com

# Set your own HF_TOKEN to download restricted models
export HF_TOKEN=<secret>

# Start SGLang server
# It may take several minutes to download the model on the first run
sglang serve --model-path Qwen/Qwen2.5-7B-Instruct --attention-backend ascend &
```

Server startup may take several minutes. Once you see output like the following, the server is running.

```log Output theme={null}
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
The server is fired up and ready to roll!
```

### Send a test request

You can do inference using the server:

```shell Command theme={null}
curl -X POST http://localhost:30000/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "The capital of France is",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 16
        }
    }'
```

If the "text" field in the response contains "Paris", the server is working as expected.

### Stop server and exit container

The SGLang server is running as a background process. You can send a `SIGINT` signal to stop it.

```shell Command theme={null}
SGLANG_PID=$(pgrep -f "sglang serve")
kill -SIGINT $SGLANG_PID
```

Wait a moment for the server to shut down gracefully. The output should be like the following:

```log Output theme={null}
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [<SGLANG_PID>]
```

The server has now stopped. You can verify it with `ps -ef | grep sglang` — the expected output is nothing (no matching process),
then exit the container by pressing `Ctrl+D`.


# Mindspore backend
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/mindspore_backend



## Introduction

MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang.

## Requirements

MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages.
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2.

## Supported Models

Currently, the following models are supported:

* **Qwen3**: Dense and MoE models
* **DeepSeek V3/R1**
* *More models coming soon...*

## Installation

> [!NOTE]
>
> Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](./getting-started/installation) and then install `sgl-mindspore`:

```shell Install theme={null}
git clone https://github.com/mindspore-lab/sgl-mindspore.git
cd sgl-mindspore
pip install -e .
```

## Run Model

Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.

### Offline Inference

Use the following script for offline inference:

```python Offline Inference theme={null}
import sglang as sgl

# Initialize the engine with MindSpore backend
llm = sgl.Engine(
    model_path="/path/to/your/model",  # Local model path
    device="npu",                      # Use NPU device
    model_impl="mindspore",            # MindSpore implementation
    attention_backend="ascend",        # Attention backend
    tp_size=1,                         # Tensor parallelism size
    dp_size=1                          # Data parallelism size
)

# Generate text
prompts = [
    "Hello, my name is",
    "The capital of France is",
    "The future of AI is"
]

sampling_params = {"temperature": 0, "top_p": 0.9}
outputs = llm.generate(prompts, sampling_params)

for prompt, output in zip(prompts, outputs):
    print(f"Prompt: {prompt}")
    print(f"Generated: {output['text']}")
    print("---")
```

### Start server

Launch a server with MindSpore backend:

```bash Launch Server theme={null}
# Basic server startup
python3 -m sglang.launch_server \
    --model-path /path/to/your/model \
    --host 0.0.0.0 \
    --device npu \
    --model-impl mindspore \
    --attention-backend ascend \
    --tp-size 1 \
    --dp-size 1
```

For distributed server with multiple nodes:

```bash Multi-node Distributed theme={null}
# Multi-node distributed server
python3 -m sglang.launch_server \
    --model-path /path/to/your/model \
    --host 0.0.0.0 \
    --device npu \
    --model-impl mindspore \
    --attention-backend ascend \
    --dist-init-addr 127.0.0.1:29500 \
    --nnodes 2 \
    --node-rank 0 \
    --tp-size 4 \
    --dp-size 2
```

## Troubleshooting

### Debug Mode

Enable sglang debug logging by log-level argument.

```bash Debug Mode theme={null}
python3 -m sglang.launch_server \
    --model-path /path/to/your/model \
    --host 0.0.0.0 \
    --device npu \
    --model-impl mindspore \
    --attention-backend ascend \
    --log-level DEBUG
```

Enable mindspore info and debug logging by setting environments.

```bash Set Log Level theme={null}
export GLOG_v=1  # INFO
export GLOG_v=0  # DEBUG
```

### Explicitly select devices

Use the following environment variable to explicitly select the devices to use.

```shell Select Devices theme={null}
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7  # to set device
```

### Some communication environment issues

In case of some environment with special communication environment, users need set some environment variables.

```shell Disable LCCL theme={null}
export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore
```

### Some dependencies of protobuf

In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch.

```shell Fix Protobuf theme={null}
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python  # to avoid protobuf binary version mismatch
```

## Support

For MindSpore-specific issues:

* Refer to the [MindSpore documentation](https://www.mindspore.cn/)


# DeepSeek-R1
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for DeepSeek-R1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-R1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-20ms) |
| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k-19-0ms) |
| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.9k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k9-out1k-19-0ms) |
| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 6k+1.6k | 20.5ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in6k-out1k6-20-5ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| DeepSeek-R1 | Ascend A3 Series Products | 16 | PD Disaggregation | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-1p1d-16p-in3k5-out1k5-50ms) |
| DeepSeek-R1 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-8p-in3k5-out1k5-50ms) |
| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-50ms) |

## Optimal Configuration

### DeepSeek-R1 W4A8 1P1D 16P IN3K5 OUT1K5 50ms

**Model**: DeepSeek-R1

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ENABLE_MOE_NZ=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_USE_FIA_NZ=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=4000
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --nnodes 1 \
        --tp-size 16 \
        --mem-fraction-static 0.8 \
        --quantization modelslim \
        --max-running-requests 32 \
        --context-length 8192 \
        --disable-radix-cache \
        --chunked-prefill-size 20480 \
        --max-prefill-tokens 20480 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --attn-dp-size 8 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --max-total-tokens 80000 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=800
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export TASK_QUEUE_ENABLE=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --nnodes 1 \
        --tp-size 16 \
        --attn-dp-size 16 \
        --mem-fraction-static 0.9 \
        --max-running-requests 416 \
        --quantization modelslim \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --enable-dp-lm-head \
        --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 \
        --watchdog-timeout 9000 \
        --context-length 8192 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 2 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 3 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --tokenizer-worker-num 4 \
        --load-balance-method round_robin \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --max-total-tokens 145792 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --request-rate 24 \
    --max-concurrency 416 \
    --num-prompts 1664 \
    --random-input-len 3584 \
    --random-output-len 1536 \
    --random-range-ratio 1 \
    --seed 1
```

Referenced by external docs. Verify before removing.

### DeepSeek-R1 W4A8 8P IN3K5 OUT1K5 50ms

**Model**: DeepSeek-R1

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=2000
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
export DEEPEP_NORMAL_LONG_SEQ_ROUND=10
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=72
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_USE_FIA_NZ=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --watchdog-timeout 9000 \
    --cuda-graph-bs-decode 4 8 12 14 \
    --mem-fraction-static 0.9 \
    --max-running-requests 224 \
    --context-length 8188 \
    --disable-radix-cache \
    --chunked-prefill-size 65536 \
    --max-prefill-tokens 3000 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --attn-dp-size 16 \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --dtype bfloat16 \
    --reasoning-parser deepseek-r1 \
    --tool-call-parser deepseekv3 \
    --max-total-tokens 100000
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 224 \
    --num-prompts 896 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

Referenced by external docs. Verify before removing.

### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K5 20ms

**Model**: DeepSeek-R1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_USE_FIA_NZ=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1536
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port $((8998 + $i)) \
        --node-rank 0 \
        --nnodes 1 \
        --tp-size 16 \
        --mem-fraction-static 0.81 \
        --quantization modelslim \
        --max-running-requests 4 \
        --context-length 8192 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 28680 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --attn-dp-size 2 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --enable-attn-tp-input-scattered \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
        export TASK_QUEUE_ENABLE=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp-size 32 \
        --attn-dp-size 16 \
        --mem-fraction-static 0.75 \
        --max-running-requests 32 \
        --quantization modelslim \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --cuda-graph-bs-decode 2 4 6 \
        --watchdog-timeout 9000 \
        --context-length 8192 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>: prefill node IP addresses
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

export SGLANG_DP_ROUND_ROBIN=1
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --request-rate 16 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 42
```

### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K5 50ms

**Model**: DeepSeek-R1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_USE_FIA_NZ=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=800
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=131072
        export SGLANG_NPU_FUSED_MOE_MODE=2
        export SGLANG_USE_AG_AFTER_QLORA=1
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port $((8998 + $i)) \
        --node-rank 0 \
        --nnodes 1 \
        --tp-size 16 \
        --mem-fraction-static 0.778 \
        --quantization modelslim \
        --max-running-requests 16 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 60000 \
        --moe-a2a-backend ascend_fuseep \
        --deepep-mode normal \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --attn-dp-size 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --enable-attn-tp-input-scattered \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=600
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_LM_HEAD_TP=8
        export SGLANG_NPU_FUSED_MOE_MODE=1
        export TASK_QUEUE_ENABLE=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp-size 32 \
        --attn-dp-size 32 \
        --mem-fraction-static 0.82 \
        --max-running-requests 1024 \
        --quantization modelslim \
        --moe-a2a-backend ascend_fuseep \
        --deepep-mode low_latency \
        --moe-dense-tp 1 \
        --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \
        --watchdog-timeout 9000 \
        --context-length 8192 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>: prefill node IP addresses
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

export SGLANG_DP_ROUND_ROBIN=1
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --request-rate 40 \
    --max-concurrency 1024 \
    --num-prompts 7168 \
    --random-input-len 3584 \
    --random-output-len 1536 \
    --random-range-ratio 1 \
    --seed 1
```

### DeepSeek-R1 W8A8 2P1D 32P IN3K5 OUT1K 19.0ms

**Model**: DeepSeek-R1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1k

**TPOT**: 19.0ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_USE_FIA_NZ=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1536
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port $((8998 + $i)) \
        --node-rank 0 \
        --nnodes 1 \
        --tp-size 16 \
        --mem-fraction-static 0.81 \
        --quantization modelslim \
        --max-running-requests 4 \
        --context-length 8192 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 28680 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --attn-dp-size 2 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --enable-attn-tp-input-scattered \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
        export TASK_QUEUE_ENABLE=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp-size 32 \
        --attn-dp-size 16 \
        --mem-fraction-static 0.75 \
        --max-running-requests 32 \
        --quantization modelslim \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --cuda-graph-bs-decode 2 4 6 \
        --watchdog-timeout 9000 \
        --context-length 8192 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>: prefill node IP addresses
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

export SGLANG_DP_ROUND_ROBIN=1
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --request-rate 16 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 3500 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### DeepSeek-R1 W8A8 2P1D 32P IN3K9 OUT1K 19.0ms

**Model**: DeepSeek-R1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 3.9k+1k

**TPOT**: 19.0ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_USE_FIA_NZ=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1536
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port $((8998 + $i)) \
        --node-rank 0 \
        --nnodes 1 \
        --tp-size 16 \
        --mem-fraction-static 0.81 \
        --quantization modelslim \
        --max-running-requests 4 \
        --context-length 8192 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 28680 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --attn-dp-size 2 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --enable-attn-tp-input-scattered \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
        export TASK_QUEUE_ENABLE=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp-size 32 \
        --attn-dp-size 16 \
        --mem-fraction-static 0.75 \
        --max-running-requests 32 \
        --quantization modelslim \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --cuda-graph-bs-decode 2 4 6 \
        --watchdog-timeout 9000 \
        --context-length 8192 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>: prefill node IP addresses
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

export SGLANG_DP_ROUND_ROBIN=1
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --request-rate 16 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 3900 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### DeepSeek-R1 W8A8 2P1D 32P IN6K OUT1K6 20.5ms

**Model**: DeepSeek-R1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 6k+1.6k

**TPOT**: 20.5ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_USE_FIA_NZ=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1536
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port $((8998 + $i)) \
        --node-rank 0 \
        --nnodes 1 \
        --tp-size 16 \
        --mem-fraction-static 0.81 \
        --quantization modelslim \
        --max-running-requests 4 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 28680 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --attn-dp-size 2 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --enable-attn-tp-input-scattered \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
        export TASK_QUEUE_ENABLE=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp-size 32 \
        --attn-dp-size 8 \
        --mem-fraction-static 0.75 \
        --max-running-requests 32 \
        --quantization modelslim \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --cuda-graph-bs-decode 2 4 6 \
        --watchdog-timeout 9000 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --reasoning-parser deepseek-r1 \
        --tool-call-parser deepseekv3 \
        --disaggregation-transfer-backend ascend \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>: prefill node IP addresses
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

export SGLANG_DP_ROUND_ROBIN=1
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --request-rate 16 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 6000 \
    --random-output-len 1600 \
    --random-range-ratio 1 \
    --seed 1
```


# DeepSeek-V3.2
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for DeepSeek-V3.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-V3.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) |
| DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K 26ms

**Model**: DeepSeek-V3.2

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 128k+1k

**TPOT**: 26ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --nnodes 2 \
        --tp 32 \
        --watchdog-timeout 9000 \
        --mem-fraction-static 0.73 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 68000 \
        --max-running-requests 1 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --disable-cuda-graph \
        --moe-dense-tp-size 1 \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 32 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --reasoning-parser deepseek-v3 \
        --tool-call-parser deepseekv32 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=400
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp 32 \
        --attn-dp-size 8 \
        --ep 32 \
        --moe-dense-tp-size 1 \
        --enable-dp-lm-head \
        --watchdog-timeout 9000 \
        --mem-fraction-static 0.79 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 68000 \
        --max-running-requests 32 \
        --cuda-graph-max-bs-decode 4 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --quantization modelslim \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --disaggregation-transfer-backend ascend \
        --reasoning-parser deepseek-v3 \
        --tool-call-parser deepseekv32 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 8 \
    --num-prompts 8 \
    --random-input-len 131072 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS16

**Model**: DeepSeek-V3.2

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 128k+1k

**TPOT**: 107ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --nnodes 2 \
        --tp 32 \
        --watchdog-timeout 9000 \
        --mem-fraction-static 0.73 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 68000 \
        --max-running-requests 1 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --disable-cuda-graph \
        --moe-dense-tp-size 1 \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 32 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --reasoning-parser deepseek-v3 \
        --tool-call-parser deepseekv32 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=400
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp 32 \
        --attn-dp-size 8 \
        --ep 32 \
        --moe-dense-tp-size 1 \
        --enable-dp-lm-head \
        --watchdog-timeout 9000 \
        --mem-fraction-static 0.79 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 68000 \
        --max-running-requests 32 \
        --cuda-graph-max-bs-decode 4 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --quantization modelslim \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --disaggregation-transfer-backend ascend \
        --reasoning-parser deepseek-v3 \
        --tool-call-parser deepseekv32 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 16 \
    --num-prompts 16 \
    --random-input-len 131072 \
    --random-output-len 1024 \
    --random-range-ratio 1
```

### DeepSeek-V3.2 W8A8 1P1D 32P IN128K OUT1K BS8

**Model**: DeepSeek-V3.2

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 128k+1k

**TPOT**: 26ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --nnodes 2 \
        --tp 32 \
        --watchdog-timeout 9000 \
        --mem-fraction-static 0.73 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 68000 \
        --max-running-requests 1 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --disable-cuda-graph \
        --moe-dense-tp-size 1 \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 32 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --reasoning-parser deepseek-v3 \
        --tool-call-parser deepseekv32 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=400
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --nnodes 2 \
        --tp 32 \
        --attn-dp-size 8 \
        --ep 32 \
        --moe-dense-tp-size 1 \
        --enable-dp-lm-head \
        --watchdog-timeout 9000 \
        --mem-fraction-static 0.79 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --max-prefill-tokens 68000 \
        --max-running-requests 32 \
        --cuda-graph-max-bs-decode 4 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --quantization modelslim \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --disaggregation-transfer-backend ascend \
        --reasoning-parser deepseek-v3 \
        --tool-call-parser deepseekv32 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 8 \
    --num-prompts 8 \
    --random-input-len 131072 \
    --random-output-len 1024 \
    --random-range-ratio 1
```


# DeepSeek-V4-Flash
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for DeepSeek-V4-Flash on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-V4-Flash Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| DeepSeek-V4-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-1p1d-16p-in8k-out1k-50ms) |
| DeepSeek-V4-Flash | Ascend A3 Series Products | 8 | PD Mixed | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-8p-in8k-out1k-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### DeepSeek-V4-Flash W8A8 1P1D 16P IN8K OUT1K 50ms

**Model**: DeepSeek-V4-Flash

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 8k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export FORCE_DRAFT_MODEL_NON_QUANT=1
export HCCL_OP_EXPANSION_MODE=AIV
export INF_NAN_MODE_FORCE_DISABLE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DSV4_FP4_EXPERTS=False
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False
export SGLANG_OPT_FP8_WO_A_GEMM=0
export SGLANG_OPT_FUSE_WQA_WKV=0
export SGLANG_OPT_USE_FUSED_HASH_TOPK=False
export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False
export SGLANG_OPT_USE_TILELANG_MHC_POST=False
export SGLANG_OPT_USE_TILELANG_MHC_PRE=False
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=2048
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=35
        export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
        export SGLANG_ENABLE_WAR_BARRIER=1
        export SGLANG_FORCE_COARSE_WAR_BARRIER=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --page-size 128 \
        --tp-size 16 \
        --trust-remote-code \
        --device npu \
        --attention-backend dsv4 \
        --watchdog-timeout 9000 \
        --disaggregation-transfer-backend ascend \
        --disaggregation-bootstrap-port 8998 \
        --mem-fraction-static 0.68 \
        --prefill-max-requests 6 \
        --max-prefill-tokens 80000 \
        --chunked-prefill-size 131072 \
        --max-running-requests 112 \
        --dp-size 16 \
        --enable-dp-attention \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --quantization modelslim \
        --enable-dp-lm-head \
        --kv-cache-dtype bfloat16 \
        --disable-cuda-graph \
        --disable-radix-cache \
        --load-balance-method round_robin \
        --ep-dispatch-algorithm static
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=8
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=256
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --page-size 128 \
        --tp-size 16 \
        --trust-remote-code \
        --device npu \
        --attention-backend dsv4 \
        --watchdog-timeout 9000 \
        --mem-fraction-static 0.75 \
        --prefill-max-requests 1 \
        --disable-radix-cache \
        --chunked-prefill-size -1 \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 896 \
        --dp-size 16 \
        --enable-dp-attention \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --quantization modelslim \
        --enable-dp-lm-head \
        --kv-cache-dtype bfloat16 \
        --cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 \
        --speculative-algorithm EAGLE \
        --speculative-num-steps 2 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 3
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --random-input-len 8000 \
    --random-output-len 1000 \
    --num-prompts 2400 \
    --max-concurrency 800 \
    --random-range-ratio 1 \
    --warmup-requests 16 \
    --request-rate inf \
    --seed 1
```

Referenced by external docs. Verify before removing.

### DeepSeek-V4-Flash W8A8 8P IN8K OUT1K 50ms

**Model**: DeepSeek-V4-Flash

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 8k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash

export DEEPEP_HCCL_BUFFSIZE=2500
export DEEPEP_HYBRID_DEPLOYMENT=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export FORCE_DRAFT_MODEL_NON_QUANT=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export INF_NAN_MODE_FORCE_DISABLE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96
export SGLANG_DSPARK_FAST_KERNEL=0
export SGLANG_DSV4_FP4_EXPERTS=False
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_ENABLE_WAR_BARRIER=1
export SGLANG_FORCE_COARSE_WAR_BARRIER=1
export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False
export SGLANG_OPT_FP8_WO_A_GEMM=0
export SGLANG_OPT_FUSE_WQA_WKV=0
export SGLANG_OPT_USE_FUSED_HASH_TOPK=False
export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False
export SGLANG_OPT_USE_TILELANG_MHC_POST=False
export SGLANG_OPT_USE_TILELANG_MHC_PRE=False
export SGLANG_RAGGED_VERIFY_MODE=static
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:14699
export SGLANG_ZBAL_LOCAL_MEM_SIZE=61000
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --page-size 128 \
    --tp-size 16 \
    --trust-remote-code \
    --device npu \
    --prefill-max-requests 192 \
    --max-prefill-tokens 80000 \
    --attention-backend dsv4 \
    --watchdog-timeout 9000 \
    --mem-fraction-static 0.68 \
    --chunked-prefill-size 131072 \
    --max-running-requests 192 \
    --dp-size 16 \
    --enable-dp-attention \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --quantization modelslim \
    --enable-dp-lm-head \
    --kv-cache-dtype bfloat16 \
    --load-balance-method round_robin \
    --cuda-graph-bs-decode 1 2 4 6 8 10 \
    --speculative-algorithm DSPARK \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-draft-model-quantization modelslim \
    --speculative-draft-attention-backend ascend \
    --speculative-num-draft-tokens 7 \
    --speculative-dspark-block-size 6 \
    --disable-radix-cache
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --random-input-len 8000 \
    --random-output-len 1000 \
    --num-prompts 160 \
    --max-concurrency 160 \
    --random-range-ratio 1 \
    --warmup-requests 0 \
    --request-rate inf \
    --seed 1
```


# GLM-5.1
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for GLM-5.1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [GLM-5.1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| - | - | - | - | - | - | - | - | - |
| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 65k+1.5k (90% prefix cache hit rate) | 25ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in65k-out1k5-prefix90-25ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| - | - | - | - | - | - | - | - | - |
| GLM-5.1 | Ascend A3 Series Products | 16 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-16p-in3k5-out1k5-50ms) |
| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 56.4ms | 13.1s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in128k-out1k-56-4ms) |
| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 16k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in16k-out1k-50ms) |
| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 64k+1k | 55.2ms | 7.58s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-55-2ms) |
| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 64k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### GLM-5.1 W4A8 16P IN3K5 OUT1K5 50ms

**Model**: GLM-5.1

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   NODE_IPS: IP addresses of each node in the cluster
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
NODE_IPS=('<your node1 ip>' '<your node2 ip>')

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=2500
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"

for i in "${!NODE_IPS[@]}";
do
    if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]];
    then
        echo "${NODE_IPS[$i]}"
        python3 -m sglang.launch_server \
        --model-path $MODEL_PATH \
        --host ${NODE_IPS[$i]} --port 6688 \
        --nnodes 2 \
        --dist-init-addr ${NODE_IPS[0]}:5000 \
        --node-rank $i \
        --attention-backend ascend \
        --device npu \
        --tp-size 32 \
        --dp-size 16 \
        --enable-dp-attention \
        --chunked-prefill-size 65536 \
        --max-prefill-tokens 280000 \
        --trust-remote-code \
        --mem-fraction-static 0.65 \
        --served-model-name glm-5 \
        --cuda-graph-max-bs-decode 16 \
        --max-running-requests 256 \
        --quantization modelslim \
        --speculative-draft-model-quantization unquant \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --load-balance-method round_robin \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47
        break
    fi
done
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 128 \
    --num-prompts 128 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

Referenced by external docs. Verify before removing.

### GLM-5.1 W4A8 1P1D 32P IN128K OUT1K 56.4ms

**Model**: GLM-5.1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k

**TPOT**: 56.4ms

**TTFT**: 13.1s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=8
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export TASK_QUEUE_ENABLE=2
        export ZBAL_HCCL_OP=send,recv
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 16 \
        --served-model-name glm-5 \
        --chunked-prefill-size 524288 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --dp-size 1 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --enable-prefill-cp \
        --cp-strategy interleave \
        --attn-cp-size 32 \
        --disable-radix-cache \
        --moe-dense-tp 1 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=600
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_ENABLE_SPEC_V2=1
        export SGLANG_NPU_USE_MULTI_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --trust-remote-code \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.85 \
        --max-running-requests 64 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 16 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 131072 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### GLM-5.1 W4A8 1P1D 32P IN16K OUT1K 50ms

**Model**: GLM-5.1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 16k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=16
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --mem-fraction-static 0.75 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 64 \
        --served-model-name glm-5 \
        --chunked-prefill-size 524288 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --dp-size 1 \
        --load-balance-method round_robin \
        --enable-prefill-cp \
        --cp-strategy interleave \
        --attn-cp-size 32 \
        --moe-dense-tp 1 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_ENABLE_SPEC_V2=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.87 \
        --max-running-requests 96 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 128 \
    --num-prompts 512 \
    --random-input-len 16384 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 55.2ms

**Model**: GLM-5.1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 64k+1k

**TPOT**: 55.2ms

**TTFT**: 7.58s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=8
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export TASK_QUEUE_ENABLE=2
        export ZBAL_HCCL_OP=send,recv
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --trust-remote-code \
        --tp-size 32 \
        --nnodes 2 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 32 \
        --served-model-name glm-5 \
        --chunked-prefill-size 524288 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --dp-size 1 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --enable-prefill-cp \
        --cp-strategy interleave \
        --attn-cp-size 32 \
        --disable-radix-cache \
        --moe-dense-tp 1 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=600
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_ENABLE_SPEC_V2=1
        export SGLANG_NPU_USE_MULTI_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --trust-remote-code \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --enable-dp-attention \
        --ep-size 32 \
        --mem-fraction-static 0.85 \
        --max-running-requests 64 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 16 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --random-input-len 65536 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### GLM-5.1 W4A8 1P1D 32P IN64K OUT1K 50ms

**Model**: GLM-5.1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 64k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 4 \
        --nnodes 2 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 16 \
        --served-model-name glm-5 \
        --chunked-prefill-size 32768 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --speculative-draft-model-quantization unquant \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 4 \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --pp-size 8 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --enable-dp-attention \
        --ep-size 32 \
        --mem-fraction-static 0.85 \
        --max-running-requests 32 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 16 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --trust-remote-code
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 32 \
    --num-prompts 64 \
    --random-input-len 65536 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### GLM-5.1 W4A8 1P1D 32P IN65K OUT1K5 PREFIX90 25ms

**Model**: GLM-5.1

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 65k+1.5k (90% prefix cache hit rate)

**TPOT**: 25ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip1>' '<your prefill ip2>')
D_IP=('<your decode ip1>' '<your decode ip2>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=16
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export TASK_QUEUE_ENABLE=2

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --dist-init-addr ${P_IP[0]}:5000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --mem-fraction-static 0.75 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 64 \
        --served-model-name glm-5 \
        --chunked-prefill-size 53248 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --dp-size 1 \
        --load-balance-method round_robin \
        --enable-prefill-cp \
        --cp-strategy interleave \
        --attn-cp-size 32 \
        --moe-dense-tp 1 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=650
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_ENABLE_SPEC_V2=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --dist-init-addr ${D_IP[0]}:5000 \
        --node-rank $i \
        --tp-size 32 \
        --nnodes 2 \
        --dp-size 32 \
        --ep-size 32 \
        --enable-dp-attention \
        --mem-fraction-static 0.87 \
        --max-running-requests 96 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-bs-decode 1 2 3 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 4 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --reasoning-parser glm45 \
        --tool-call-parser glm47 \
        --trust-remote-code
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip1>: first decode node IP address (decode may have distributed nodes)
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip1>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 59904` = `round(66560 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6656` = `round(66560 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 480 \
    --gsp-system-prompt-len 59904 \
    --gsp-question-len 6656 \
    --gsp-output-len 1536 \
    --max-concurrency 100 \
    --num-prompts 480 \
    --request-rate inf
```


# GLM-5.2
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for GLM-5.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [GLM-5.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| GLM-5.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#glm-5-2-w4a8-3p1d-32p-in16k-out1k-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### GLM-5.2 W4A8 3P1D 32P IN16K OUT1K 50ms

**Model**: GLM-5.2

**Hardware**: Ascend A3 Series Products

**Cards**: 32

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 16k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TRANSFORMERS_VERBOSITY=error

P_IP=('<your prefill ip1>' '<your prefill ip2>' '<your prefill ip3>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip1>:24667"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export DEEP_USE_ALLTOALL_MODE=1
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=128
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_PP_LAYER_PARTITION=18,20,24,16
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export TASK_QUEUE_ENABLE=2
        export ZBAL_HCCL_OP=send,recv
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port $((8998 + $i)) \
        --trust-remote-code \
        --tp-size 4 \
        --mem-fraction-static 0.72 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --disaggregation-transfer-backend ascend \
        --max-running-requests 16 \
        --served-model-name glm-5 \
        --chunked-prefill-size 4096 \
        --max-prefill-tokens 180000 \
        --moe-a2a-backend deepep \
        --deepep-mode normal \
        --disable-shared-experts-fusion \
        --disable-cuda-graph \
        --dtype bfloat16 \
        --speculative-draft-model-quantization unquant \
        --enable-prefill-cp \
        --cp-strategy zigzag \
        --attn-cp-size 4 \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --pp-size 4 \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 1 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=300
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_ENABLE_SPEC_V2=1
        export SGLANG_NPU_USE_MULTI_STREAM=1
        export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
        export TASK_QUEUE_ENABLE=0

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --trust-remote-code \
        --tp-size 16 \
        --dp-size 16 \
        --enable-dp-attention \
        --ep-size 16 \
        --mem-fraction-static 0.895 \
        --max-running-requests 128 \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --served-model-name glm-5 \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency \
        --cuda-graph-max-bs-decode 8 \
        --disaggregation-transfer-backend ascend \
        --watchdog-timeout 9000 \
        --context-length 180000 \
        --tokenizer-worker-num 8 \
        --disable-shared-experts-fusion \
        --dtype bfloat16 \
        --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip1>, <your prefill ip2>, <your prefill ip3>: prefill node IP addresses
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip1>:8000 8998 \
    --prefill http://<your prefill ip2>:8000 8999 \
    --prefill http://<your prefill ip3>:8000 9000 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy round_robin
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 64 \
    --num-prompts 200 \
    --random-input-len 16000 \
    --random-output-len 1000 \
    --random-range-ratio 1
```


# Kimi-K2.6
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Kimi-K2.6 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Kimi-K2.6 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| - | - | - | - | - | - | - | - | - |
| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 20ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| - | - | - | - | - | - | - | - | - |
| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Mixed | 64k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms) |
| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 128k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms) |
| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 100ms | 5s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) |
| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 64k+1.5k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms) |
| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 64k+1.5k (90% prefix cache hit rate) | 100ms | 3s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) |
| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 1024x1024 (30)+1024 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1024x1024-30-out1024-50ms) |
| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 1080p\_30+256 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1080p-30-out256-50ms) |
| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Kimi-K2.6 W4A8 16P IN64K OUT1K 100ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 64k+1k

**TPOT**: 100ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   NODE_IPS: IP addresses of each node in the cluster
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

NODE_IPS=('<your node1 ip>' '<your node2 ip>')

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=4400
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"

for i in "${!NODE_IPS[@]}";
do
    if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]];
    then
        echo "${NODE_IPS[$i]}"
        python3 -m sglang.launch_server \
        --model-path $MODEL_PATH \
        --host ${NODE_IPS[$i]} --port 6688 \
        --nnodes 2 \
        --dist-init-addr ${NODE_IPS[0]}:5000 \
        --node-rank $i \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --dtype bfloat16 \
        --tp-size 32 \
        --mem-fraction-static 0.662 \
        --max-running-requests 32 \
        --chunked-prefill-size 262144 \
        --context-length 75000 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --attn-dp-size 32 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs-decode 1 \
        --disable-radix-cache \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        break
    fi
done
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 32 \
    --num-prompts 32 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

Referenced by external docs. Verify before removing.

### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K 100ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k

**TPOT**: 100ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=8
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export ZBAL_ENABLE_GRAPH=1
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --disable-radix-cache \
        --mem-fraction-static 0.78 \
        --max-running-requests 2 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 2 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 2 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs-decode 1 2 4 6 8 16 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --request-rate inf \
    --random-input-len 128000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K PREFIX90 100ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k (90% prefix cache hit rate)

**TPOT**: 100ms

**TTFT**: 5s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=8
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export ZBAL_ENABLE_GRAPH=1
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.78 \
        --max-running-requests 2 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 2 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 2 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs-decode 1 2 4 6 8 16 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 115200` = `round(128000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 12800` = `round(128000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 4 \
    --gsp-system-prompt-len 115200 \
    --gsp-question-len 12800 \
    --gsp-output-len 1000 \
    --max-concurrency 1 \
    --num-prompts 4 \
    --request-rate inf
```

### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 100ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 64k+1.5k

**TPOT**: 100ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1800
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --disable-radix-cache \
        --disable-cuda-graph \
        --mem-fraction-static 0.78 \
        --max-running-requests 1 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 1 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 16 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs-decode 16 \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 4 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 5 \
        --speculative-draft-model-quantization unquant
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --request-rate inf \
    --random-input-len 64000 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 PREFIX90 100ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 64k+1.5k (90% prefix cache hit rate)

**TPOT**: 100ms

**TTFT**: 3s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1800
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.78 \
        --max-running-requests 2 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 2 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1200
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 2 \
        --attn-dp-size 2 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs-decode 1 2 4 6 8 \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 4 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 5 \
        --speculative-draft-model-quantization unquant
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 57600` = `round(64000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6400` = `round(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 8 \
    --gsp-system-prompt-len 57600 \
    --gsp-question-len 6400 \
    --gsp-output-len 1500 \
    --max-concurrency 2 \
    --num-prompts 8 \
    --request-rate inf
```

Referenced by external docs. Verify before removing.

### Kimi-K2.6 W4A8 8P IN1024X1024 30 OUT1024 50ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 1024x1024 (30)+1024

*Format: resolution (input tokens) + output tokens*

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=1280
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --quantization modelslim \
    --dtype bfloat16 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --trust-remote-code \
    --device npu \
    --attention-backend ascend \
    --tp-size 16 \
    --mem-fraction-static 0.865 \
    --max-running-requests 176 \
    --chunked-prefill-size 32768 \
    --context-length 8192 \
    --max-prefill-tokens 16384 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --attn-dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 4 8 9 10 11 \
    --disable-radix-cache \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 2 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 3 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 200 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1024x1024 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --image-resolution 1024x1024 \
    --image-count 1 \
    --max-concurrency 160 \
    --num-prompts 640 \
    --request-rate inf \
    --random-input-len 30 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --warmup-requests 16 \
    --seed 1
```

### Kimi-K2.6 W4A8 8P IN1080P 30 OUT256 50ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 1080p\_30+256

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=1024
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --quantization modelslim \
    --dtype bfloat16 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --trust-remote-code \
    --device npu \
    --attention-backend ascend \
    --tp-size 16 \
    --mem-fraction-static 0.852 \
    --max-running-requests 64 \
    --chunked-prefill-size 16384 \
    --context-length 8192 \
    --max-prefill-tokens 16384 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --attn-dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 3 4 \
    --disable-radix-cache \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 2 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 3 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 200 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1920x1080 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --image-resolution 1920x1080 \
    --image-count 1 \
    --max-concurrency 48 \
    --num-prompts 196 \
    --request-rate inf \
    --random-input-len 30 \
    --random-output-len 256 \
    --random-range-ratio 1 \
    --warmup-requests 16 \
    --seed 1
```

### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 20ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=1200
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --dtype bfloat16 \
    --tp-size 16 \
    --mem-fraction-static 0.865 \
    --max-running-requests 80 \
    --chunked-prefill-size 32768 \
    --context-length 6144 \
    --max-prefill-tokens 65536 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --attn-dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 3 4 5 \
    --disable-radix-cache \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 200 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-attempts 5 \
    --max-concurrency 64 \
    --num-prompts 256 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --warmup-requests 0 \
    --seed 1
```

### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 50ms

**Model**: Kimi-K2.6

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=1200
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=200
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --dtype bfloat16 \
    --tp-size 16 \
    --mem-fraction-static 0.895 \
    --max-running-requests 208 \
    --chunked-prefill-size 32768 \
    --context-length 6144 \
    --max-prefill-tokens 16384 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --attn-dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 4 8 11 12 13 \
    --disable-radix-cache \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 50 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 192 \
    --num-prompts 768 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --warmup-requests 0 \
    --seed 1
```


# MiMo-V2.5-Pro
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro



> [!NOTE]
>
> This page focuses on the deployment of MiMo-V2.5-Pro (FP4) with DFlash speculative decoding in PD disaggregation mode on the Ascend NPU.
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Model Deployment

MiMo-V2.5-Pro-FP4-DFlash is an MXFP4-quantized checkpoint with a built-in DFlash draft model (located in the `dflash/` subdirectory of the weights). The following example deploys it in 1P1D mode (1 prefill node + 1 decode node, TP8 + DP2 per node).

#### Common environment setup (both nodes)

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   HCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME: network interface name
# ============================================================

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export HCCL_BUFFSIZE=300
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600

# Use the AscendC flash attention
export ASCEND_USE_FIA=1

# PD disaggregation transfer config
export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24669"
export ASCEND_MF_TRANSFER_PROTOCOL="device_urma"

# DeepEP
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export HCCL_SOCKET_IFNAME=<network-interface>
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_HOST_SOCKET_PORT_RANGE=auto

MODEL_PATH=/path/to/MiMo-V2.5-Pro-FP4-DFlash
```

#### Prefill node

```bash Command theme={null}
export DEEPEP_HCCL_BUFFSIZE=2500
# Enable chunked dispatch for long sequences
export DEEPEP_NORMAL_LONG_SEQ_ROUND=10
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=0

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --attention-backend ascend \
    --device npu \
    --tp-size 8 --nnodes 1 --node-rank 0 \
    --chunked-prefill-size 8192 \
    --trust-remote-code --port 10001 \
    --host <your prefill ip> --max-running-requests 32 \
    --mem-fraction-static 0.90 \
    --swa-full-tokens-ratio 0.3 \
    --disaggregation-mode prefill --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8996 \
    --cuda-graph-backend-prefill disabled \
    --attn-dp-size 2 --enable-dp-lm-head \
    --moe-a2a-backend deepep --deepep-mode normal
```

#### Decode node

```bash Command theme={null}
# Use eagle_worker_v2 and overlap plan stream to hide the draft/target preparation
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
# DFlash draft model has a longer context length than the derived value
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1

export DEEPEP_HCCL_BUFFSIZE=1200

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --speculative-draft-model-path $MODEL_PATH/dflash \
    --attention-backend ascend \
    --device npu \
    --tp-size 8 --nnodes 1 --node-rank 0 \
    --trust-remote-code --port 20001 \
    --host <your decode ip> --max-running-requests 32 \
    --mem-fraction-static 0.88 \
    --swa-full-tokens-ratio 0.3 \
    --cuda-graph-bs 1 2 4 8 12 16 \
    --disaggregation-mode decode --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8996 \
    --moe-a2a-backend deepep --deepep-mode low_latency \
    --attn-dp-size 2 --enable-dp-lm-head \
    --speculative-algorithm DFLASH \
    --speculative-num-draft-tokens 8
```

#### Router

```bash Command theme={null}
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:10001 \
    --decode http://<your decode ip>:20001 \
    --host 127.0.0.1 \
    --port 6688 \
    --health-check-interval-secs 3600 --mini-lb
```

### Benchmark

We tested it based on the `RANDOM` dataset.

#### Benchmark Prefill Node (TTFT)

```bash Command theme={null}
python3 -m sglang.bench_serving \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --model /path/to/MiMo-V2.5-Pro-FP4-DFlash \
    --dataset-name random \
    --tokenize-prompt \
    --random-input-len 16000 \
    --random-output-len 1 \
    --request-rate 0.4 \
    --random-range-ratio 1 \
    --num-prompts 128 \
    --max-concurrency 32
```

#### Benchmark Decode Node (TPOT)

```bash Command theme={null}
python3 -m sglang.bench_serving \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --model /path/to/MiMo-V2.5-Pro-FP4-DFlash \
    --dataset-name random \
    --tokenize-prompt \
    --random-input-len 16000 \
    --random-output-len 1000 \
    --request-rate inf \
    --random-range-ratio 1 \
    --num-prompts 128 \
    --max-concurrency 32
```


# MiMo-V2-Flash
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for MiMo-V2-Flash on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiMo-V2-Flash Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| - | - | - | - | - | - | - | - | - |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration |
| - | - | - | - | - | - | - | - | - |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) |
| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### MiMo-V2-Flash 1P1D 12P IN16K OUT1 TTFT 5s

**Model**: MiMo-V2-Flash

**Hardware**: Ascend A3 Series Products

**Cards**: 12

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 16k+1

**TTFT**: 5s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1024
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
        export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --attention-backend ascend \
        --device npu \
        --tp-size 8 \
        --nnodes 1 \
        --chunked-prefill-size 8192 \
        --trust-remote-code \
        --max-running-requests 64 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --disaggregation-transfer-backend ascend \
        --disable-radix-cache \
        --disable-cuda-graph \
        --cuda-graph-backend-prefill=disabled \
        --dp-size 2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=800
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --nnodes 1 \
        --trust-remote-code \
        --max-running-requests 64 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
        --disaggregation-transfer-backend ascend \
        --speculative-algorithm EAGLE \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --enable-multi-layer-eagle \
        --disable-radix-cache \
        --attn-dp-size 2 \
        --enable-dp-lm-head \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --health-check-interval-secs 3600 --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 64 \
    --num-prompts 128 \
    --request-rate 0.4 \
    --random-input-len 16000 \
    --random-output-len 1 \
    --random-range-ratio 1 \
    --seed 1
```

### MiMo-V2-Flash 1P1D 12P IN16K OUT1K TPOT 20ms

**Model**: MiMo-V2-Flash

**Hardware**: Ascend A3 Series Products

**Cards**: 12

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 16k+1k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1024
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
        export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --attention-backend ascend \
        --device npu \
        --tp-size 8 \
        --nnodes 1 \
        --chunked-prefill-size 8192 \
        --trust-remote-code \
        --max-running-requests 64 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --disaggregation-transfer-backend ascend \
        --disable-radix-cache \
        --disable-cuda-graph \
        --cuda-graph-backend-prefill=disabled \
        --dp-size 2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=800
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --nnodes 1 \
        --trust-remote-code \
        --max-running-requests 32 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --cuda-graph-bs-decode 1 2 4 8 12 16 \
        --disaggregation-transfer-backend ascend \
        --speculative-algorithm EAGLE \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --enable-multi-layer-eagle \
        --disable-radix-cache \
        --attn-dp-size 2 \
        --enable-dp-lm-head \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --health-check-interval-secs 3600 --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 32 \
    --num-prompts 128 \
    --request-rate inf \
    --random-input-len 16000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

### MiMo-V2-Flash 1P1D 12P IN32K OUT1 TTFT 5s

**Model**: MiMo-V2-Flash

**Hardware**: Ascend A3 Series Products

**Cards**: 12

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 32k+1

**TTFT**: 5s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1024
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
        export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --attention-backend ascend \
        --device npu \
        --tp-size 8 \
        --nnodes 1 \
        --chunked-prefill-size 8192 \
        --trust-remote-code \
        --max-running-requests 64 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --disaggregation-transfer-backend ascend \
        --disable-radix-cache \
        --disable-cuda-graph \
        --cuda-graph-backend-prefill=disabled \
        --dp-size 2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=800
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --nnodes 1 \
        --trust-remote-code \
        --max-running-requests 64 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
        --disaggregation-transfer-backend ascend \
        --speculative-algorithm EAGLE \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --enable-multi-layer-eagle \
        --disable-radix-cache \
        --attn-dp-size 2 \
        --enable-dp-lm-head \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --health-check-interval-secs 3600 --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 64 \
    --num-prompts 128 \
    --request-rate 0.4 \
    --random-input-len 32000 \
    --random-output-len 1 \
    --random-range-ratio 1 \
    --seed 1
```

### MiMo-V2-Flash 1P1D 12P IN32K OUT1K TPOT 20ms

**Model**: MiMo-V2-Flash

**Hardware**: Ascend A3 Series Products

**Cards**: 12

**Deploy Mode**: PD Disaggregation

**Quantization**: W8A8 INT8

**Dataset**: 32k+1k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=1024
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
        export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --attention-backend ascend \
        --device npu \
        --tp-size 8 \
        --nnodes 1 \
        --chunked-prefill-size 8192 \
        --trust-remote-code \
        --max-running-requests 64 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --disaggregation-transfer-backend ascend \
        --disable-radix-cache \
        --disable-cuda-graph \
        --cuda-graph-backend-prefill=disabled \
        --dp-size 2
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export DEEPEP_HCCL_BUFFSIZE=800
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --nnodes 1 \
        --trust-remote-code \
        --max-running-requests 64 \
        --mem-fraction-static 0.8 \
        --swa-full-tokens-ratio 0.3 \
        --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \
        --disaggregation-transfer-backend ascend \
        --speculative-algorithm EAGLE \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --enable-multi-layer-eagle \
        --disable-radix-cache \
        --attn-dp-size 2 \
        --enable-dp-lm-head \
        --moe-a2a-backend deepep \
        --deepep-mode low_latency
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --health-check-interval-secs 3600 --mini-lb
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 48 \
    --num-prompts 192 \
    --request-rate inf \
    --random-input-len 32000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```


# MiniMax-M2.5
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for MiniMax-M2.5 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiniMax-M2.5 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 24.44ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in128k-out1k-prefix90-24-44ms) |
| MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-20ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| MiniMax-M2.5 | Ascend A3 Series Products | 4 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in32k-out1k-50ms) |
| MiniMax-M2.5 | Ascend A3 Series Products | 4 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in64k-out1k-prefix90-50ms) |
| MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### MiniMax-M2.5 W8A8 4P IN32K OUT1K 50ms

**Model**: MiniMax-M2.5

**Hardware**: Ascend A3 Series Products

**Cards**: 4

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 32k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
export DEEPEP_NORMAL_LONG_SEQ_ROUND=64
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=128
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_LOCAL_MEM_SIZE=60184
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1
export ZBAL_ENABLE_GRAPH=1
export ZBAL_HCCL_OP=allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 8 \
    --disable-radix-cache \
    --mem-fraction-static 0.74 \
    --max-running-requests 18 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 32768 \
    --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 24 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16 \
    --trust-remote-code \
    --tokenizer-worker-num 4 \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --device npu
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 18 \
    --num-prompts 72 \
    --random-input-len 32768 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1
```

### MiniMax-M2.5 W8A8 4P IN64K OUT1K PREFIX90 50ms

**Model**: MiniMax-M2.5

**Hardware**: Ascend A3 Series Products

**Cards**: 4

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 64k+1k (90% prefix cache hit rate)

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=1024
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=140000
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 8 \
    --mem-fraction-static 0.63 \
    --max-running-requests 26 \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --enable-prefill-delayer \
    --prefill-max-requests 10 \
    --chunked-prefill-size 67072 \
    --max-prefill-tokens 67000 \
    --cuda-graph-bs-decode 2 4 8 12 16 18 20 22 24 26 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16 \
    --trust-remote-code \
    --device npu
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 58982` = `round(65536 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6554` = `round(65536 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 104 \
    --gsp-system-prompt-len 58982 \
    --gsp-question-len 6554 \
    --gsp-output-len 1024 \
    --max-concurrency 26 \
    --num-prompts 104 \
    --request-rate inf
```

### MiniMax-M2.5 W8A8 8P IN128K OUT1K PREFIX90 24.44ms

**Model**: MiniMax-M2.5

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 128k+1k (90% prefix cache hit rate)

**TPOT**: 24.44ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=1024
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=160000
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --attn-dp-size 2 \
    --mem-fraction-static 0.65 \
    --max-running-requests 4 \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --enable-prefill-delayer \
    --prefill-max-requests 4 \
    --chunked-prefill-size 160000 \
    --max-prefill-tokens 80000 \
    --cuda-graph-bs-decode 2 4 6 8 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --tokenizer-worker-num 4 \
    --dtype bfloat16 \
    --device npu
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 117965` = `round(131072 * 0.9)` is the shared prefix portion.
`--gsp-question-len 13107` = `round(131072 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 16 \
    --gsp-system-prompt-len 117965 \
    --gsp-question-len 13107 \
    --gsp-output-len 1024 \
    --max-concurrency 4 \
    --num-prompts 16 \
    --request-rate inf
```

### MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 20ms

**Model**: MiniMax-M2.5

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=2048
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_NPU_FUSED_MOE_MODE=2
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --attn-dp-size 16 \
    --mem-fraction-static 0.53 \
    --max-running-requests 96 \
    --disable-radix-cache \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --prefill-delayer-max-delay-passes 500 \
    --enable-prefill-delayer \
    --prefill-max-requests 3 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 8192 \
    --cuda-graph-bs-decode 1 2 3 4 5 6 \
    --moe-a2a-backend ascend_fuseep \
    --deepep-mode auto \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16 \
    --device npu
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 112 \
    --num-prompts 448 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### MiniMax-M2.5 W8A8 8P IN3K5 OUT1K5 50ms

**Model**: MiniMax-M2.5

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights
export PYTHONPATH=${DRAFT_MODEL_PATH}:$PYTHONPATH

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=1024
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 16 \
    --attn-dp-size 16 \
    --mem-fraction-static 0.75 \
    --max-running-requests 320 \
    --disable-radix-cache \
    --reasoning-parser minimax-append-think \
    --tool-call-parser minimax-m2 \
    --prefill-delayer-max-delay-passes 500 \
    --enable-prefill-delayer \
    --chunked-prefill-size 196608 \
    --max-prefill-tokens 8192 \
    --cuda-graph-bs-decode 1 2 4 8 12 16 20 \
    --moe-a2a-backend ascend_fuseep \
    --fuseep-mode 2 \
    --quantization modelslim \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --dtype bfloat16 \
    --device npu
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 320 \
    --num-prompts 1280 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```


# Qwen3-235B-A22B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3-235B-A22B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-235B-A22B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

> [!WARNING]
>
> The optimal configurations on this page are validated with the image `quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3`. Use exactly this version; later versions are not covered by this page and may introduce breaking changes.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-235B-A22B | Ascend A3 Series Products | 8 | PD Mixed | 11k+1.5k | 8ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-bf16-8p-in11k-out1k5-8ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-235B-A22B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50.1ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-w8a8-8p-in3k5-out1k5-50-1ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3-235B-A22B BF16 8P IN11K OUT1K5 8ms

**Model**: Qwen3-235B-A22B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 11k+1.5k

**TPOT**: 8ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=1600
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --max-running-requests 1 \
    --dtype bfloat16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 16384 \
    --speculative-draft-model-quantization unquant \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --disable-radix-cache \
    --enable-dp-lm-head \
    --tp 16 \
    --mem-fraction-static 0.78 \
    --cuda-graph-bs-decode 1 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen25
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --random-input-len 11000 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-235B-A22B W8A8 8P IN3K5 OUT1K5 50.1ms

**Model**: Qwen3-235B-A22B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50.1ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=570
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=188416
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 432 \
    --context-length 8192 \
    --dtype bfloat16 \
    --chunked-prefill-size 94208 \
    --max-prefill-tokens 458880 \
    --sampling-backend ascend \
    --ep-dispatch-algorithm static \
    --disable-radix-cache \
    --moe-a2a-backend ascend_fuseep \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --tp 16 \
    --dp-size 16 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --mem-fraction-static 0.8 \
    --cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen25
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 432 \
    --num-prompts 1728 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```


# Qwen3-30B-A3B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3-30B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-30B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-10ms) |
| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 6k+1.5k | 10.25ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in6k-out1k5-bs16) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1k+100 | 10000ms | BF16 | [Optimal Configuration](#qwen3-30b-a3b-bf16-1p-in1k-out100) |
| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3-30B-A3B BF16 1P IN1K OUT100

**Model**: Qwen3-30B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 1k+100

**TPOT**: 10000ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_LAUNCH_BLOCKING=0
export DP_ROUND_ROBIN=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_ALGO="level0:NA;level1:ring"
export HCCL_SOCKET_IFNAME=<network-interface>
export INF_NAN_MODE_FORCE_DISABLE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_USE_MAX_DP_ATT=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --max-running-requests 168 \
    --disable-radix-cache \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 8300 \
    --speculative-draft-model-quantization unquant \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 7 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 8 \
    --tp-size 2 \
    --attn-dp-size 2 \
    --mem-fraction-static 0.85 \
    --cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 162 \
    --num-prompts 624 \
    --random-input-len 1000 \
    --random-output-len 100 \
    --random-range-ratio 1 \
    --seed 1 \
    --max-attempts 4
```

### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 10ms

**Model**: Qwen3-30B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 10ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_LAUNCH_BLOCKING=0
export DEEPEP_HCCL_BUFFSIZE=400
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 162 \
    --disable-radix-cache \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 35000 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --tp-size 2 \
    --mem-fraction-static 0.87 \
    --cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-30B-A3B W8A8 1P IN3K5 OUT1K5 50ms

**Model**: Qwen3-30B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_LAUNCH_BLOCKING=0
export DEEPEP_HCCL_BUFFSIZE=400
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 162 \
    --disable-radix-cache \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 35000 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --tp-size 2 \
    --mem-fraction-static 0.87 \
    --cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 160 \
    --num-prompts 640 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-30B-A3B W8A8 1P IN6K OUT1K5 BS16

**Model**: Qwen3-30B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 6k+1.5k

**TPOT**: 10.25ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEPEP_HCCL_BUFFSIZE=400
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export TRANSFORMERS_VERBOSITY=error

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 16 \
    --disable-radix-cache \
    --speculative-draft-model-quantization unquant \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 35000 \
    --tp-size 2 \
    --mem-fraction-static 0.6 \
    --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-attempts 5 \
    --max-concurrency 16 \
    --num-prompts 16 \
    --random-input-len 6144 \
    --random-output-len 1500 \
    --random-range-ratio 1
```


# Qwen3-32B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3-32B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-32B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-32B | Ascend A3 Series Products | 8 | PD Mixed | 18k+4k | 6ms | BF16 | [Optimal Configuration](#qwen3-32b-bf16-8p-in18k-out4k-6ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-32B | Ascend A3 Series Products | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms) |
| Qwen3-32B | Ascend A2 Series Products | 2 | PD Mixed | 3.5k+1.5k | 55ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-55ms-a2-series) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3-32B BF16 8P IN18K OUT4K 6ms

**Model**: Qwen3-32B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 18k+4k

**TPOT**: 6ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --max-running-requests 1 \
    --disable-radix-cache \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 65536 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --tp-size 16 \
    --mem-fraction-static 0.72 \
    --cuda-graph-bs-decode 1 \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --random-input-len 18000 \
    --random-output-len 4000 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-32B W8A8 2P IN3K5 OUT1K5 50ms

**Model**: Qwen3-32B

**Hardware**: Ascend A3 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_DEEPGEMM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 101 \
    --disable-radix-cache \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 35000 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --tp-size 4 \
    --mem-fraction-static 0.845 \
    --cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 100 \
    --num-prompts 400 \
    --random-input-len 3584 \
    --random-output-len 1536 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-32B W8A8 2P IN3K5 OUT1K5 55ms A2 Series

**Model**: Qwen3-32B

**Hardware**: Ascend A2 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 55ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_DEEPGEMM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 101 \
    --disable-radix-cache \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 35000 \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --tp-size 4 \
    --mem-fraction-static 0.845 \
    --cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \
    --dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 100 \
    --num-prompts 400 \
    --random-input-len 3584 \
    --random-output-len 1536 \
    --random-range-ratio 1 \
    --seed 1
```


# Qwen3.5-397B-A17B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3.5-397B-A17B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.5-397B-A17B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-20ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 16k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-20ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 22.2ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-22-2ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-20ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-50ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-prefix90-50ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-50ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-50ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-50ms) |
| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-prefix90-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3.5-397B-A17B W4A8 8P IN128K OUT1K 20ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=60672
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 131072 \
    --prefill-max-requests 1 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 16 \
    --mem-fraction-static 0.6 \
    --cuda-graph-bs-decode 2 3 4 5 6 8 10 12 14 16 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 2 \
    --max-concurrency 3 \
    --num-prompts 3 \
    --random-input-len 131072 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf
```

### Qwen3.5-397B-A17B W4A8 8P IN128K OUT1K 50ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=60672
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 131072 \
    --prefill-max-requests 1 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 16 \
    --mem-fraction-static 0.6 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 8 \
    --max-concurrency 10 \
    --num-prompts 10 \
    --random-input-len 131072 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf \
    --temperature 0.6 \
    --top-p 0.95
```

### Qwen3.5-397B-A17B W4A8 8P IN128K OUT1K PREFIX90 50ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k (90% prefix cache hit rate)

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=2200
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=60672
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 131072 \
    --max-mamba-cache-size 320 \
    --prefill-max-requests 10 \
    --mamba-radix-cache-strategy extra_buffer \
    --trust-remote-code \
    --max-running-requests 64 \
    --mem-fraction-static 0.6 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 117965` = `round(131072 * 0.9)` is the shared prefix portion.
`--gsp-question-len 13107` = `round(131072 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 40 \
    --gsp-system-prompt-len 117965 \
    --gsp-question-len 13107 \
    --gsp-output-len 1024 \
    --max-concurrency 40 \
    --num-prompts 40 \
    --request-rate inf
```

### Qwen3.5-397B-A17B W4A8 8P IN16K OUT1K 20ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 16k+1k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=20
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 50000 \
    --prefill-max-requests 4 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 48 \
    --mem-fraction-static 0.8 \
    --max-total-tokens 210000 \
    --cuda-graph-bs-decode 2 4 6 8 10 12 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --dp-size 4 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 32 \
    --max-concurrency 40 \
    --num-prompts 40 \
    --random-input-len 16384 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf
```

### Qwen3.5-397B-A17B W4A8 8P IN16K OUT1K 50ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 16k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=20
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=58624
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 65536 \
    --prefill-max-requests 4 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 144 \
    --mem-fraction-static 0.8 \
    --max-total-tokens 635000 \
    --cuda-graph-bs-decode 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --dp-size 4 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 8 \
    --max-concurrency 132 \
    --num-prompts 132 \
    --random-input-len 16384 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf
```

### Qwen3.5-397B-A17B W4A8 8P IN3K5 OUT1K5 22.2ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 22.2ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export DEEPEP_NORMAL_LONG_SEQ_ROUND=6
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=58624
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 35000 \
    --max-total-tokens 128000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 160 \
    --mem-fraction-static 0.8 \
    --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --dp-size 8 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 64 \
    --max-concurrency 160 \
    --num-prompts 160 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf
```

### Qwen3.5-397B-A17B W4A8 8P IN3K5 OUT1K5 50ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
export DEEPEP_NORMAL_LONG_SEQ_ROUND=6
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 17500 \
    --max-total-tokens 280000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 432 \
    --mem-fraction-static 0.8 \
    --cuda-graph-bs-decode 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --dp-size 8 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 16 \
    --max-concurrency 432 \
    --num-prompts 432 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf
```

### Qwen3.5-397B-A17B W4A8 8P IN64K OUT1K 20ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 64k+1k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=20
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=58672
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 65536 \
    --prefill-max-requests 1 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 16 \
    --mem-fraction-static 0.6 \
    --max-total-tokens 1065000 \
    --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --dp-size 2 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 6 \
    --max-concurrency 6 \
    --num-prompts 6 \
    --random-input-len 65536 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf \
    --temperature 0.6 \
    --top-p 0.95
```

### Qwen3.5-397B-A17B W4A8 8P IN64K OUT1K 50ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 64k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=20
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=58672
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 65536 \
    --prefill-max-requests 1 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 32 \
    --mem-fraction-static 0.6 \
    --max-total-tokens 1065000 \
    --cuda-graph-bs-decode 2 4 6 8 12 14 16 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --dp-size 2 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 8 \
    --max-concurrency 24 \
    --num-prompts 24 \
    --random-input-len 65536 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --seed 1 \
    --request-rate inf \
    --temperature 0.6 \
    --top-p 0.95
```

### Qwen3.5-397B-A17B W4A8 8P IN64K OUT1K PREFIX90 50ms

**Model**: Qwen3.5-397B-A17B

**Hardware**: Ascend A3 Series Products

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 64k+1k (90% prefix cache hit rate)

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=2200
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=20
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 65536 \
    --max-mamba-cache-size 640 \
    --mamba-radix-cache-strategy extra_buffer \
    --trust-remote-code \
    --max-running-requests 128 \
    --mem-fraction-static 0.6 \
    --max-total-tokens 1310720 \
    --quantization modelslim \
    --enable-multimodal \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --dp-size 2 \
    --enable-dp-attention \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 58982` = `round(65536 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6554` = `round(65536 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 96 \
    --gsp-system-prompt-len 58982 \
    --gsp-question-len 6554 \
    --gsp-output-len 1024 \
    --max-concurrency 96 \
    --num-prompts 96 \
    --request-rate inf
```


# Qwen3.6-27B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3.6-27B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-27B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) |
| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 1080p\_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) |
| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms) |
| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) |
| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) |
| Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) |
| Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 16k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in16k-out1k-50ms) |
| Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in64k-out1k-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3.6-27B 1P IN1024X1024 30 OUT1024 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 1024x1024 (30)+1024

*Format: resolution (input tokens) + output tokens*

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=300
export SGLANG_PREFILL_DELAYER_MAX_PREFILL_BS_WINDOW_SIZE=128
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 52000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 60 \
    --max-mamba-cache-size 60 \
    --mem-fraction-static 0.74 \
    --cuda-graph-bs-decode 2 4 8 16 24 32 40 42 45 50 60 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --mm-enable-dp-encoder \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1024x1024 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 60 \
    --num-prompts 240 \
    --random-input-len 30 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --image-resolution 1024x1024 \
    --image-count 1 \
    --seed 1
```

### Qwen3.6-27B 1P IN1080P 30 OUT256 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 1080p\_30+256

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_PREFILL_BS_WINDOW_SIZE=128
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 82688 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 38 \
    --max-mamba-cache-size 38 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \
    --enable-prefill-delayer \
    --prefill-delayer-queue-min-ratio 0.45 \
    --prefill-delayer-max-delay-ms 5500 \
    --prefill-delayer-max-delay-passes 300 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1920x1080 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --warmup-requests 38 \
    --max-concurrency 42 \
    --num-prompts 152 \
    --random-input-len 30 \
    --random-output-len 256 \
    --random-range-ratio 1 \
    --image-resolution 1920x1080 \
    --image-count 1 \
    --seed 1
```

### Qwen3.6-27B 1P IN64K OUT1K PREFIX90 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 64k+1k (90% prefix cache hit rate)

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 32768 \
    --max-prefill-tokens 32768 \
    --mamba-radix-cache-strategy extra_buffer \
    --trust-remote-code \
    --max-running-requests 20 \
    --max-mamba-cache-size 160 \
    --mem-fraction-static 0.82 \
    --cuda-graph-bs-decode 1 2 5 10 15 17 19 20 \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 57600` = `round(64000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6400` = `round(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 20 \
    --gsp-system-prompt-len 57600 \
    --gsp-question-len 6400 \
    --gsp-output-len 1000 \
    --max-concurrency 20 \
    --num-prompts 20 \
    --request-rate inf
```

### Qwen3.6-27B W8A8 1P IN3K5 OUT1K5 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=130
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 60000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 64 \
    --max-mamba-cache-size 74 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 2 8 16 32 40 45 50 54 \
    --enable-multimodal \
    --quantization modelslim \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 54 \
    --num-prompts 216 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3.6-27B W8A8 1P IN64K OUT1K 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 64k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 48000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 6 \
    --max-mamba-cache-size 16 \
    --mem-fraction-static 0.6 \
    --cuda-graph-bs-decode 1 2 4 5 6 \
    --quantization modelslim \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 6 \
    --num-prompts 12 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --random-range-ratio 1
```

### Qwen3.6-27B W8A8 2P IN128K OUT1K 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 128k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 74000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 6 \
    --max-mamba-cache-size 7 \
    --mem-fraction-static 0.63 \
    --cuda-graph-bs-decode 1 2 4 5 6 \
    --enable-multimodal \
    --quantization modelslim \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 4 \
    --num-prompts 16 \
    --random-input-len 128000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3.6-27B W8A8 2P IN16K OUT1K 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 16k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 58000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 37 \
    --max-mamba-cache-size 74 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \
    --quantization modelslim \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 37 \
    --warmup-requests 4 \
    --num-prompts 37 \
    --random-input-len 16000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3.6-27B W8A8 2P IN64K OUT1K 50ms

**Model**: Qwen3.6-27B

**Hardware**: Ascend A3 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 64k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 50000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 28 \
    --max-mamba-cache-size 50 \
    --mem-fraction-static 0.7 \
    --cuda-graph-bs-decode 2 4 6 \
    --enable-multimodal \
    --quantization modelslim \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 6 \
    --num-prompts 24 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```


# Qwen3.6-35B-A3B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3.6-35B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-35B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 254k+1k | 16.1ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k-16-1ms) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1080p\_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) |
| Qwen3.6-35B-A3B | Ascend A3 Series Products | 2 | PD Mixed | 984k+1k | 40.91ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k-40-91ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3.6-35B-A3B 1P IN1024X1024 30 OUT1024 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 1024x1024 (30)+1024

*Format: resolution (input tokens) + output tokens*

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 9999999 \
    --max-total-tokens 365000 \
    --prefill-max-requests 30 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 120 \
    --max-mamba-cache-size 120 \
    --mem-fraction-static 0.85 \
    --cuda-graph-bs-decode 4 16 32 48 64 110 165 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1024x1024 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 120 \
    --num-prompts 120 \
    --warmup-requests 120 \
    --random-input-len 30 \
    --random-output-len 1024 \
    --random-range-ratio 1 \
    --image-resolution 1024x1024 \
    --image-count 1 \
    --seed 1 \
    --request-rate inf
```

### Qwen3.6-35B-A3B 1P IN1080P 30 OUT256 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 1080p\_30+256

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_VIT_ENABLE_CUDA_GRAPH=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-prefill-tokens 150000 \
    --max-total-tokens 200000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 42 \
    --max-mamba-cache-size 42 \
    --mem-fraction-static 0.75 \
    --cuda-graph-bs-decode 4 8 16 24 48 64 80 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1920x1080 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 42 \
    --num-prompts 42 \
    --warmup-requests 42 \
    --random-input-len 30 \
    --random-output-len 256 \
    --random-range-ratio 1 \
    --image-resolution 1920x1080 \
    --image-count 1 \
    --seed 1 \
    --request-rate inf
```

### Qwen3.6-35B-A3B 1P IN128K OUT1K 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 128k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=1600
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=20
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 420000 \
    --max-prefill-tokens 128000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 3 \
    --max-mamba-cache-size 3 \
    --mem-fraction-static 0.9 \
    --cuda-graph-bs-decode 1 2 3 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 3 \
    --num-prompts 3 \
    --random-input-len 128000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3.6-35B-A3B 1P IN128K OUT1K PREFIX90 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 128k+1k (90% prefix cache hit rate)

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 16384 \
    --max-prefill-tokens 65536 \
    --trust-remote-code \
    --enable-prefill-delayer \
    --mamba-radix-cache-strategy extra_buffer \
    --max-running-requests 103 \
    --max-mamba-cache-size 85 \
    --mem-fraction-static 0.85 \
    --cuda-graph-bs-decode 2 4 8 16 32 48 64 80 96 103 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 115200` = `round(128000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 12800` = `round(128000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 412 \
    --gsp-system-prompt-len 115200 \
    --gsp-question-len 12800 \
    --gsp-output-len 1000 \
    --max-concurrency 103 \
    --num-prompts 412 \
    --request-rate inf
```

### Qwen3.6-35B-A3B 1P IN254K OUT1K 16.1ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 254k+1k

**TPOT**: 16.1ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 131072 \
    --max-prefill-tokens 254000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 1 \
    --max-mamba-cache-size 6 \
    --mem-fraction-static 0.65 \
    --cuda-graph-bs-decode 1 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --warmup-requests 1 \
    --num-prompts 1 \
    --random-input-len 254000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3.6-35B-A3B 1P IN3K5 OUT1K5 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_BUFFSIZE=100
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 659840 \
    --max-prefill-tokens 43400 \
    --disable-radix-cache \
    --trust-remote-code \
    --prefill-max-requests 12 \
    --max-running-requests 122 \
    --max-mamba-cache-size 122 \
    --mem-fraction-static 0.9 \
    --cuda-graph-bs-decode 4 16 32 64 96 116 120 122 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 122 \
    --num-prompts 122 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3.6-35B-A3B 1P IN64K OUT1K 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 64k+1k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 600000 \
    --max-prefill-tokens 65536 \
    --disable-radix-cache \
    --trust-remote-code \
    --enable-prefill-delayer \
    --max-running-requests 10 \
    --max-mamba-cache-size 20 \
    --mem-fraction-static 0.65 \
    --cuda-graph-bs-decode 2 4 8 12 14 16 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 10 \
    --num-prompts 40 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3.6-35B-A3B 1P IN64K OUT1K PREFIX90 50ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 64k+1k (90% prefix cache hit rate)

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=300
export GDN_ATTN_BACKEND_TRITON=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 2 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size -1 \
    --max-total-tokens 470784 \
    --max-prefill-tokens 65536 \
    --trust-remote-code \
    --mamba-radix-cache-strategy extra_buffer \
    --max-running-requests 40 \
    --max-mamba-cache-size 200 \
    --mem-fraction-static 0.9 \
    --cuda-graph-bs-decode 2 8 16 24 32 36 40 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 58982` = `round(65536 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6554` = `round(65536 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 40 \
    --gsp-system-prompt-len 58982 \
    --gsp-question-len 6554 \
    --gsp-output-len 1024 \
    --max-concurrency 40 \
    --num-prompts 40 \
    --request-rate inf
```

### Qwen3.6-35B-A3B 2P IN984K OUT1K 40.91ms

**Model**: Qwen3.6-35B-A3B

**Hardware**: Ascend A3 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: BF16

**Dataset**: 984k+1k

**TPOT**: 40.91ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export GLOO_SOCKET_IFNAME=lo
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --tp-size 4 \
    --nnodes 1 \
    --attention-backend ascend \
    --device npu \
    --chunked-prefill-size 131072 \
    --max-prefill-tokens 984000 \
    --disable-radix-cache \
    --trust-remote-code \
    --max-running-requests 1 \
    --max-mamba-cache-size 6 \
    --mem-fraction-static 0.68 \
    --cuda-graph-bs-decode 1 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --dtype bfloat16 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --context-length 1010000 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --warmup-requests 1 \
    --num-prompts 1 \
    --random-input-len 984000 \
    --random-output-len 1000 \
    --random-range-ratio 1 \
    --seed 1
```


# Qwen3-8B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3-8B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-8B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 5ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-5ms) |
| Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 6k+1.5k | 11.79ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in6k-out1k5-bs16) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 37ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-37ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3-8B W8A8 1P IN3K5 OUT1K5 37ms

**Model**: Qwen3-8B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 37ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50
export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 70 \
    --max-prefill-tokens 16384 \
    --disable-radix-cache \
    --chunked-prefill-size 16384 \
    --tp-size 1 \
    --mem-fraction-static 0.85 \
    --cuda-graph-bs-decode 8 12 24 36 48 51 55 60 63 64 66 68 70 \
    --dtype bfloat16 \
    --speculative-draft-model-quantization unquant \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 64 \
    --num-prompts 256 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-8B W8A8 1P IN3K5 OUT1K5 5ms

**Model**: Qwen3-8B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 5ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 1 \
    --max-prefill-tokens 16384 \
    --disable-radix-cache \
    --chunked-prefill-size -1 \
    --tp-size 2 \
    --mem-fraction-static 0.894 \
    --cuda-graph-bs-decode 1 \
    --dtype bfloat16 \
    --speculative-draft-model-quantization unquant \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-attempts 5 \
    --max-concurrency 1 \
    --num-prompts 4 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-8B W8A8 1P IN6K OUT1K5 BS16

**Model**: Qwen3-8B

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 6k+1.5k

**TPOT**: 11.79ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --nnodes 1 \
    --node-rank 0 \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --max-running-requests 16 \
    --max-prefill-tokens 16384 \
    --disable-radix-cache \
    --chunked-prefill-size -1 \
    --tp-size 2 \
    --mem-fraction-static 0.894 \
    --cuda-graph-bs-decode 1 5 15 16 \
    --dtype bfloat16 \
    --speculative-draft-model-quantization unquant \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-attempts 5 \
    --max-concurrency 16 \
    --num-prompts 16 \
    --random-input-len 6144 \
    --random-output-len 1500 \
    --random-range-ratio 1
```


# Qwen3-Next-80B-A3B-Instruct
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct



> [!NOTE]
>
> This page focuses on optimal configuration and benchmark results for Qwen3-Next-80B-A3B-Instruct on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-Next-80B-A3B-Instruct Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct).
>
> On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details.

### Low Latency

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-1p-in3k5-out1k5-20ms) |
| Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) |

### High Throughput

| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
| - | - | - | - | - | - | - | - |
| Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-50ms) |

## Optimal Configuration

Referenced by external docs. Verify before removing.

### Qwen3-Next-80B-A3B-Instruct W8A8 1P IN3K5 OUT1K5 20ms

**Model**: Qwen3-Next-80B-A3B-Instruct

**Hardware**: Ascend A3 Series Products

**Cards**: 1

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=2000
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
export DEEPEP_NORMAL_LONG_SEQ_ROUND=10
export FORCE_DRAFT_MODEL_NON_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_NPU_USE_MULTI_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_WARMUP_TIMEOUT=3600
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1
export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export ZBCCL_ENABLE_GRAPH=1
export ZBCCL_LOCAL_MEM_SIZE=60416
export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --page-size 128 \
    --tp-size 2 \
    --watchdog-timeout 9000 \
    --mem-fraction-static 0.85 \
    --disable-radix-cache \
    --max-prefill-tokens 28672 \
    --context-length 26384 \
    --max-total-tokens 122304 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-running-requests 2 \
    --cuda-graph-bs-decode 2 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --num-prompts 1 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN3K5 OUT1K5 50ms

**Model**: Qwen3-Next-80B-A3B-Instruct

**Hardware**: Ascend A3 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export FORCE_DRAFT_MODEL_NON_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=64
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=330
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_NPU_USE_MULTI_STREAM=0
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_WARMUP_TIMEOUT=3600
export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_HCCL_OP=allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --page-size 128 \
    --tp-size 4 \
    --watchdog-timeout 9000 \
    --mem-fraction-static 0.75 \
    --disable-radix-cache \
    --max-prefill-tokens 14080 \
    --context-length 26384 \
    --chunked-prefill-size -1 \
    --max-running-requests 300 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --attn-dp-size 2 \
    --enable-dp-lm-head \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 300 \
    --num-prompts 300 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --random-range-ratio 1 \
    --seed 1
```

### Qwen3-Next-80B-A3B-Instruct W8A8 2P IN6K OUT1K5 BS16

**Model**: Qwen3-Next-80B-A3B-Instruct

**Hardware**: Ascend A3 Series Products

**Cards**: 2

**Deploy Mode**: PD Mixed

**Quantization**: W8A8 INT8

**Dataset**: 6k+1.5k

**TPOT**: 15.62ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export ASCEND_USE_FIA=1
export DEEPEP_HCCL_BUFFSIZE=2000
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048
export DEEPEP_NORMAL_LONG_SEQ_ROUND=10
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export FORCE_DRAFT_MODEL_NON_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_NPU_USE_MULTI_STREAM=0
export SGLANG_WARMUP_TIMEOUT=3600
export STREAMS_PER_DEVICE=32
export TASK_QUEUE_ENABLE=1
export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669
export ZBCCL_ENABLE_GRAPH=1
export ZBCCL_LOCAL_MEM_SIZE=60416
export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --page-size 128 \
    --tp-size 4 \
    --watchdog-timeout 9000 \
    --mem-fraction-static 0.85 \
    --disable-radix-cache \
    --max-prefill-tokens 28672 \
    --context-length 81920 \
    --max-total-tokens 122304 \
    --attn-dp-size 2 \
    --enable-dp-lm-head \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --speculative-draft-model-quantization unquant \
    --chunked-prefill-size -1 \
    --max-running-requests 16 \
    --cuda-graph-bs-decode 2 4 8 \
    --mamba-ssm-dtype bfloat16 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-attempts 5 \
    --max-concurrency 16 \
    --num-prompts 16 \
    --random-input-len 6144 \
    --random-output-len 1500 \
    --random-range-ratio 1
```


# DeepSeek-R1
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1



## Introduction

DeepSeek-R1 is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 671B total parameters
with 37B active parameters. It employs Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, with built-in
multi-token prediction (MTP) for speculative decoding. The model excels at reasoning, math, and code tasks through
reinforcement learning-based training.

This document demonstrates the deployment of DeepSeek-R1 on Ascend NPUs using SGLang, including single-node PD mixed
mode, multi-node PD disaggregation mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-R1) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 20 21 22` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 2 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
| MLAPO | `export SGLANG_NPU_USE_MLAPO=1` |
| Multistream MoE | `export SGLANG_NPU_USE_MULTI_STREAM=1` |
| NZ Weight Format | `export SGLANG_USE_FIA_NZ=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [DeepSeek-R1-0528-W4A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8) (Quantized version, 376GB)
* [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version, 647GB)

> [!NOTE]
>
> The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to
> 8 dies on an A3 Series node (4 cards).
>
> The W8A8 variant (647GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full
> A3 Series node (8 cards, 16 dies).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[DeepSeek-R1 Best Practice — W4A8 8P PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#single-node-pd-mixed).

### Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[DeepSeek-R1 Best Practice — W8A8 32P PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#pd-disaggregation).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[DeepSeek-R1 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# DeepSeek-V3.2
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2



## Introduction

DeepSeek-V3.2 is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 685B total parameters
with 37B active parameters. It employs Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, with built-in
multi-token prediction (MTP) for speculative decoding. DeepSeek-V3.2 introduces the DeepSeek Sparse Attention (DSA)
mechanism, a fine-grained sparse attention mechanism powered by a lightning indexer, enabling significant efficiency
improvements in long-context scenarios.

This document demonstrates the deployment of DeepSeek-V3.2 on Ascend NPUs using SGLang, including multi-node PD
disaggregation mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-V3.2) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding, DSA context parallel), it is
recommended to use v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 32` |
| Data Parallelism | `--dp-size 8` |
| Expert Parallelism | `--ep-size 32 \` `--moe-a2a-backend deepep \` `--deepep-mode low_latency` |
| Context Parallelism | `--enable-prefill-cp \` `--cp-strategy zigzag \` `--attn-cp-size 32` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [DeepSeek-V3.2-W8A8](https://www.modelscope.cn/models/sgl-npu/DeepSeek-V3.2-W8A8) (Quantized version, 694.47GB)

> [!NOTE]
>
> The W8A8 variant (694.47GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full
> A3 Series node (8 cards, 16 dies).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[DeepSeek-V3.2 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[DeepSeek-V3.2 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# DeepSeek-V4-Flash
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash



## Introduction

DeepSeek-V4-Flash is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 284B total
parameters with 13B active parameters. It employs hybrid sparse attention (CSA + HCA) and DeepSeekMoE architecture with
a bundled multi-token prediction (MTP) draft head for speculative decoding, supporting a 1M-token context length.

This document demonstrates the deployment of DeepSeek-V4-Flash on Ascend NPUs using SGLang, including multi-node PD
disaggregation mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (DeepSeek-V4-Flash) is fully
supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is
recommended to use v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Attention Backend | `--attention-backend dsv4` |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56` |
| Speculative Decoding | `--speculative-algorithm EAGLE \` `--speculative-num-steps 2 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
| DP Attention | `--dp-size 16 \` `--enable-dp-attention` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [DeepSeek-V4-Flash-w8a8-mtp](https://www.modelscope.cn/models/Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp) (W8A8 quantized version, 300GB)

> [!NOTE]
>
> The W8A8 variant (300GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to
> 8 dies on an A3 Series node (4 cards).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node PD mixed deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. In such scenarios, DeepSeek-V4-Flash can be launched with half of an Ascend A3 Series
node (4 cards, 8 dies). The configuration below was validated with `--tp-size 8`.

```bash Command theme={null}
# ============================================================
# Before running, update the following variable:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export FORCE_DRAFT_MODEL_NON_QUANT=1
export HCCL_BUFFSIZE=8
export HCCL_OP_EXPANSION_MODE=AIV
export INF_NAN_MODE_FORCE_DISABLE=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DSV4_FP4_EXPERTS=False
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False
export SGLANG_OPT_FP8_WO_A_GEMM=0
export SGLANG_OPT_FUSE_WQA_WKV=0
export SGLANG_OPT_USE_FUSED_HASH_TOPK=False
export SGLANG_OPT_USE_OVERLAP_STORE_CACHE=False
export SGLANG_OPT_USE_TILELANG_MHC_POST=False
export SGLANG_OPT_USE_TILELANG_MHC_PRE=False
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://127.0.0.1:14699"
export SGLANG_ZBAL_LOCAL_MEM_SIZE=62084
export STREAMS_PER_DEVICE=32
export ZBAL_ENABLE_GRAPH=1
export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

python3 -m sglang.launch_server \
    --model-path "${MODEL_PATH}" \
    --page-size 128 \
    --tp-size 8 \
    --trust-remote-code \
    --device npu \
    --attention-backend dsv4 \
    --watchdog-timeout 9000 \
    --host 127.0.0.1 --port 6688 \
    --mem-fraction-static 0.8 \
    --prefill-max-requests 160 \
    --max-prefill-tokens 80000 \
    --chunked-prefill-size 131072 \
    --max-running-requests 160 \
    --dp-size 8 --enable-dp-attention \
    --moe-a2a-backend deepep --deepep-mode auto \
    --quantization modelslim --enable-dp-lm-head \
    --kv-cache-dtype bfloat16
```

For the complete benchmark data and additional single-node configurations, see
[DeepSeek-V4-Flash Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#single-node-pd-mixed).

### Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[DeepSeek-V4-Flash Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#pd-disaggregation).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "deepseek-v4-flash",
        "messages": [{"role": "user", "content": "What is the capital of France?"}]
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[DeepSeek-V4-Flash Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# GLM-5.1
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1



## Introduction

GLM-5.1 is a Mixture-of-Experts (MoE) large language model developed by Z.ai, featuring 744B total parameters with 40B
active parameters. It uses 256 routed experts (top-8) plus one shared expert, with Multi-head Latent Attention (MLA)
and DeepSeek Sparse Attention (DSA), and a built-in multi-token prediction (MTP) head for speculative decoding.
The model features built-in bilingual (Chinese-English) capabilities with a unified pre-training framework, excelling at
reasoning, math, code, and tool calling tasks. GLM-5.1 supports both Thinking mode (step-by-step reasoning) and Instruct
mode (direct response), with a native context window of approximately 200k tokens.

This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including multi-node PD mixed mode,
multi-node PD disaggregation mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (GLM-5.1) is fully supported in
this version. To use the latest features (e.g., speculative decoding, multi-node deployment), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| Context Parallelism | `--enable-prefill-cp \` `--cp-strategy zigzag \` `--attn-cp-size 4` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [GLM-5.1](https://www.modelscope.cn/models/ZhipuAI/GLM-5.1) (BF16, 1.51TB)
* [GLM-5.1-w4a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) (Quantized version, 420.17GB)
* You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.

> [!NOTE]
>
> We recommend deploying the W4A8 variant for reduced resource usage and higher throughput.
> It (420.17GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to
> 8 dies on an A3 Series node (4 cards).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Multi-node PD mixed deployment

Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping
prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single
node can provide. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[GLM-5.1 Best Practice — Multi-node PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#multi-node-pd-mixed).

### Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[GLM-5.1 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#pd-disaggregation).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[GLM-5.1 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# GLM-5.2
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2



## Introduction

GLM-5.2 is a large language model in the GLM (General Language Model) series, jointly developed by the KEG Laboratory
of Tsinghua University and Zhipu AI. GLM-5.2 adopts the DeepSeek-V3/V3.2 architecture, including DeepSeek Sparse
Attention (DSA) and multi-token prediction (MTP), and supports high-throughput inference with SGLang on Ascend NPUs.

This document demonstrates the deployment of GLM-5.2 on Ascend NPUs using SGLang, including single-node deployment,
multi-node deployment, prefill-decode disaggregation, feature configuration, and performance optimization.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [GLM-5.2](https://www.modelscope.cn/models/ZhipuAI/GLM-5.2) (BF16, 1.51TB)
* [GLM-5.2-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.2-w8a8/) (Quantized version without MTP, 774.08GB)
* You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model naively.

> [!NOTE]
>
> We recommend deploying the W8A8 variant for reduced resource usage and higher throughput.
> It (774.08GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node deployment

Quantized model `GLM-5.2-w8a8` can be deployed on one Ascend A3 Series node.

Run the following script to execute online inference.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export STREAMS_PER_DEVICE=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
# MTP OVERLAP
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MULTI_STREAM=1

export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
# DEEPEP
export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1

MODEL_PATH=/path/to/model-weights

python3 -m sglang.launch_server \
        --model-path $MODEL_PATH \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 --nnodes 1 --node-rank 0 \
        --chunked-prefill-size 16384 --max-prefill-tokens 280000 \
        --trust-remote-code \
        --host 127.0.0.1 \
        --mem-fraction-static 0.7 \
        --port 8000 \
        --served-model-name glm-5 \
        --cuda-graph-bs-decode 16 \
        --quantization modelslim \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4  \
        --moe-a2a-backend deepep --deepep-mode auto
```

### Multi-node deployment

Quantized model `GLM-5.2-w8a8` can be deployed on two Ascend A3 Series nodes.

Modify the IP addresses of the two nodes, then run the same script on both nodes.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   IPS: IP addresses of each node in the cluster
#   IP_MASTER: rank 0 node IP address with port
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export STREAMS_PER_DEVICE=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
# MTP OVERLAP
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1

export SGLANG_NPU_USE_MULTI_STREAM=1
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV

# Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here
export HCCL_SOCKET_IFNAME=<network-interface>
export GLOO_SOCKET_IFNAME=<network-interface>

# DEEPEP
export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1


IPS=('<your node1 ip>' '<your node2 ip>')
IP_MASTER="${IPS[0]}:5000"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
for i in "${!IPS[@]}";
do
    if [[ "$LOCAL_HOST1" == "${IPS[$i]}" || "$LOCAL_HOST2" == "${IPS[$i]}" ]];
    then
        echo "${IPS[$i]}"
        python3 -m sglang.launch_server \
        --model-path $MODEL_PATH \
        --attention-backend ascend \
        --device npu \
        --tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $IP_MASTER \
        --chunked-prefill-size 16384 --max-prefill-tokens 131072 \
        --trust-remote-code \
        --host 127.0.0.1 \
        --mem-fraction-static 0.8 \
        --port 8000 \
        --served-model-name glm-5 \
        --cuda-graph-max-bs-decode 32 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4  \
        --disable-radix-cache
        NODE_RANK=$i
        break
    fi
done
```

### Prefill-decode disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   ASCEND_MF_STORE_URL: prefill master IP address with port
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
# pd transfer, prefill master IP
export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24707"
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

MODEL_PATH=/path/to/model-weights

export TRANSFORMERS_VERBOSITY=error

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"

# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export TASK_QUEUE_ENABLE=2
        export HCCL_SOCKET_IFNAME=<network-interface>
        export GLOO_SOCKET_IFNAME=<network-interface>

        # prefill node
        python -m sglang.launch_server --model-path ${MODEL_PATH}  --disaggregation-mode prefill --host ${P_IP[$i]} \
        --port 8000 --disaggregation-bootstrap-port 8998 --trust-remote-code --nnodes 1 --node-rank $i \
        --tp-size 16 --mem-fraction-static 0.8 --attention-backend ascend --device npu --quantization modelslim \
        --disaggregation-transfer-backend ascend --max-running-requests 64 \
        --served-model-name glm-5 --chunked-prefill-size 524288 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \
        --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \
        --dp-size 4 --enable-dp-attention \
        --load-balance-method round_robin \
        --enable-dp-lm-head --moe-dense-tp 1 \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2

        # cp
        #--enable-prefill-cp \
        #--cp-strategy zigzag \
        #--attn-cp-size 4 \
        NODE_RANK=$i
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"

        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export HCCL_BUFFSIZE=650

        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
        export TASK_QUEUE_ENABLE=0

        export HCCL_SOCKET_IFNAME=<network-interface>
        export GLOO_SOCKET_IFNAME=<network-interface>

        export SGLANG_NPU_USE_MULTI_STREAM=1

        python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
        --port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --enable-dp-attention --ep-size 16 \
        --mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \
        --served-model-name glm-5 --moe-a2a-backend deepep --deepep-mode low_latency \
        --cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
        --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16  --load-balance-method round_robin \
        --speculative-draft-model-quantization unquant \
        --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
        NODE_RANK=$i
        break
    fi
done
```

Launch the router after the prefill and decode services are ready.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   P_MASTER_IP: prefill master IP address
#   D_MASTER_IP: decode master IP address
#   ROUTER_HOST_IP: router node IP address
# ============================================================

P_MASTER_IP="<your prefill ip>"
D_MASTER_IP="<your decode ip>"
ROUTER_HOST_IP="<your router ip>"

python3 -m sglang_router.launch_router \
--pd-disaggregation \
--policy round_robin \
--prefill http://${P_MASTER_IP}:8000 8998 \
--decode http://${D_MASTER_IP}:8003 \
--host ${ROUTER_HOST_IP} \
--port 6688
```

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 8000)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[GLM-5.2 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# GLM-5.3
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3



## Introduction

GLM-5.3 is a large-scale Mixture-of-Experts (MoE) language model in the GLM (General Language Model) series, developed
by Zhipu AI. It uses DeepSeek Sparse Attention (DSA), supports multi-token prediction (MTP), and is designed for
long-context text generation. GLM-5.3 is distinct from the separately released GLM-5.3-Flash model, which uses a
hybrid sparse/linear-attention architecture.

This document demonstrates the deployment of GLM-5.3 on Ascend NPUs using SGLang, including single-node deployment,
multi-node co-located (mixed) deployment, prefill-decode disaggregation, feature configuration, and performance optimization.
The multi-node and prefill-decode disaggregation examples use a locally prepared `GLM-5.3-w8a8` checkpoint and a CANN 9.1.0 runtime.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` or `--tp-size 32` |
| Data Parallelism | `--dp-size 16` (mixed) or `--dp-size 32` (decode) with `--enable-dp-attention` |
| Expert Parallelism | `--ep-size 32` with `--moe-a2a-backend deepep` and `--deepep-mode low_latency` |
| Pipeline Parallelism | `--pp-size 4` with `SGLANG_PP_LAYER_PARTITION=18,20,24,16` |
| Prefill Context Parallelism | `--attn-cp-size 8` with `--enable-prefill-cp` and `--cp-strategy interleave` |
| PD Disaggregation | `--disaggregation-mode prefill` / `decode` with `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | `--chunked-prefill-size 65536` / `--max-prefill-tokens 280000` (mixed deployment) |
| NPU Graph | `--cuda-graph-bs-decode 2` (mixed); `--disable-cuda-graph` (prefill); `--cuda-graph-bs-decode 1 2 3` (decode) |
| Speculative Decoding | `--speculative-algorithm NEXTN` with `--speculative-num-steps 3` and `--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column reflect the deployment scenarios described below and are not universal
> defaults. Some flags require a specific SGLang version or model checkpoint. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching a container, starting the SGLang server, and sending a test request.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers version compatibility (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile,
  and recommended system settings (CPU power scheme, NUMA, swap).

The mixed and PD disaggregation scripts below require a CANN **9.1.0** environment, the corresponding ATB and customized
operator libraries, a compatible SGLang source checkout, and working inter-node HCCL/Gloo connectivity. Ensure that
all participating nodes use compatible software versions and can access the model weights. The PD topology additionally requires the configured MemFabric components.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes or distribute an
identical checkpoint to the same local path on every node.

* [GLM-5.3-BF16](https://www.modelscope.cn/models/ZhipuAI/GLM-5.3-BF16) (BF16, approximately 1.51 TB).
* `GLM-5.3-w8a8` (a locally prepared W8A8 ModelSlim-quantized checkpoint used in the scripts below; set `MODEL_PATH`
  to the actual location). The checkpoint size and MTP-weight availability depend on the conversion output.
* You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize model weights for Ascend devices.

> [!NOTE]
>
> The deployment examples below use a W8A8 checkpoint with `--quantization modelslim`. The PD disaggregation topology
> uses **two Atlas 800I A3 nodes for prefill and two additional A3 nodes for decode**. Make sure that the checkpoint
> contains the required MTP weights before enabling NEXTN speculative decoding.

For hardware specifications (memory per device, devices per node, and differences between A2 and A3),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. A CANN Docker image requires at least **30GB** of free space.
> CANN 9.1.0 base images do **not** by themselves include a ready-to-run SGLang environment: install a compatible
> SGLang build, TorchNPU, ATB, custom operators, and other required dependencies before running the deployment scripts.

The commands below start CANN 9.1.0 base images. Follow
[SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation)
to install SGLang and the required runtime components inside the container. Use the matching image for your hardware.

**Atlas 800I A3**

```bash Command theme={null}
docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci8:/dev/davinci8 \
--device=/dev/davinci9:/dev/davinci9 \
--device=/dev/davinci10:/dev/davinci10 \
--device=/dev/davinci11:/dev/davinci11 \
--device=/dev/davinci12:/dev/davinci12 \
--device=/dev/davinci13:/dev/davinci13 \
--device=/dev/davinci14:/dev/davinci14 \
--device=/dev/davinci15:/dev/davinci15 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:cann9.1.0-a3-B080
```

**Atlas 800I A2**

```bash Command theme={null}
docker pull quay.io/ascend/sglang:cann9.1.0-a3-B080

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci8:/dev/davinci8 \
--device=/dev/davinci9:/dev/davinci9 \
--device=/dev/davinci10:/dev/davinci10 \
--device=/dev/davinci11:/dev/davinci11 \
--device=/dev/davinci12:/dev/davinci12 \
--device=/dev/davinci13:/dev/davinci13 \
--device=/dev/davinci14:/dev/davinci14 \
--device=/dev/davinci15:/dev/davinci15 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:cann9.1.0-a3-B080
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your container name or remove `--name` to use the default name.
> * The listed `/dev/davinci*` mappings follow the 16-device A3 example in the original tutorial. Remove mappings for
>   devices that do not exist on your host, particularly when using A2 nodes.
> * If you already have a validated CANN 9.1.0 SGLang development image, you can use it instead of building from the base image.

## Online service deployment

### Single-node deployment

The following single-node example is adapted from the GLM-5.2 tutorial and is provided as a
**reference configuration**.

```shell theme={null}
#!/usr/bin/env bash
# Reference only: adapted from the GLM-5.2 tutorial; not yet verified for GLM-5.3.
# Update MODEL_PATH, SGLANG_ROOT, and the installed custom-op paths as required.
MODEL_PATH='/path/to/GLM-5.3-w8a8'
SGLANG_ROOT='/path/to/sglang'

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1000
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
export SGLANG_NPU_USE_MULTI_STREAM=1

python3 -m sglang.launch_server \
    --model-path "${MODEL_PATH}" \
    --attention-backend ascend --device npu \
    --tp-size 16 --nnodes 1 --node-rank 0 \
    --chunked-prefill-size 4096 --max-prefill-tokens 8192 \
    --trust-remote-code \
    --host 127.0.0.1 --port 8000 \
    --mem-fraction-static 0.75 --max-running-requests 32 \
    --served-model-name glm-5 \
    --quantization modelslim --dtype bfloat16 \
    --moe-a2a-backend deepep --deepep-mode auto \
    --disable-shared-experts-fusion \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4
```

### Multi-node deployment

For co-located (mixed) serving across **two Atlas 800I A3 nodes**, both nodes participate in a single GLM-5.3
inference instance. The following script is adapted from the supplied GLM-5.3 W8A8 multi-node configuration, using
**TP=32**, **DP=16**, DeepEP `auto`, and NEXTN speculative decoding.

Modify the node IP addresses, model/source paths, and network interface, then run the **same script on both nodes**.
The two nodes must use the same compatible runtime and be able to communicate with rank 0 via the distributed
initialization address. In this script, the mixed-serving HTTP port is **6688** (not 8000).

```shell theme={null}
#!/usr/bin/env bash
# GLM-5.3 W8A8: two-node co-located (mixed) deployment (Atlas 800I A3).
# Run the same script on both nodes after replacing the variables below.

# ============================================================
# Before running, update the following variables:
#   IPS: IP addresses of two nodes (rank 0 first)
#   NIC: network interface name for HCCL and Gloo
#   MODEL_PATH: GLM-5.3 W8A8 weight directory
#   SGLANG_ROOT: compatible SGLang source checkout
# ============================================================
IPS=('<node-0-ip>' '<node-1-ip>')
NIC='<network-interface>'
MODEL_PATH='/path/to/GLM-5.3-w8a8'
SGLANG_ROOT='/path/to/sglang'

# High-performance CPU settings (requires sufficient permissions).
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH:-}"
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"

# Keep the allocator configuration disabled as in the supplied script.
# export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export HCCL_SOCKET_IFNAME="${NIC}"
export GLOO_SOCKET_IFNAME="${NIC}"
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1

# Optional tuning flags from the original script (disabled by default).
# export SGLANG_NPU_USE_MLAPO=1
# export HCCL_OP_EXPANSION_MODE=AIV
# export SGLANG_NPU_USE_MULTI_STREAM=1
# export SGLANG_USE_AG_AFTER_QLORA=1
# export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1

# Match the configured node IP against all local IPv4 addresses.
LOCAL_IPS=( $(hostname -I) )
has_local_ip() {
    local wanted="$1" local_ip
    for local_ip in "${LOCAL_IPS[@]}"; do
        [[ "${wanted}" == "${local_ip}" ]] && return 0
    done
    return 1
}

for i in "${!IPS[@]}"; do
    if has_local_ip "${IPS[$i]}"; then
        echo "Starting GLM-5.3 mixed deployment on ${IPS[$i]} (rank ${i})"
        export HCCL_BUFFSIZE=2000
        exec python3 -m sglang.launch_server \
            --model-path "${MODEL_PATH}" \
            --attention-backend ascend \
            --device npu \
            --dist-init-addr "${IPS[0]}:5000" \
            --tp-size 32 --nnodes 2 --node-rank "$i" \
            --dp-size 16 --enable-dp-attention \
            --chunked-prefill-size 65536 --max-prefill-tokens 280000 \
            --trust-remote-code \
            --host "${IPS[$i]}" \
            --mem-fraction-static 0.68 \
            --port 6688 \
            --served-model-name glm-5 \
            --cuda-graph-bs-decode 2 \
            --max-running-requests 32 \
            --quantization modelslim \
            --moe-a2a-backend deepep --deepep-mode auto \
            --load-balance-method round_robin \
            --speculative-algorithm NEXTN \
            --speculative-num-steps 3 \
            --speculative-eagle-topk 1 \
            --speculative-num-draft-tokens 4
    fi
done

echo "ERROR: Neither IPS address matches this host: ${LOCAL_IPS[*]}" >&2
exit 1
```

### Prefill-decode disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate node groups, reducing interference between
long-prompt processing and autoregressive generation. This configuration uses **four Atlas 800I A3 nodes**:

* **Prefill group:** 2 nodes, `TP=8`, `PP=4`, `DP=1`, and prefill context parallelism `CP=8`. It runs DeepEP in `normal`
  mode and disables NPU graph capture.
* **Decode group:** 2 nodes, `TP=32`, `DP=32`, `EP=32`. It runs DeepEP in `low_latency` mode with decode graph capture
  and NEXTN speculative decoding.

Run the following **same script on all four nodes**. The script selects the role and node rank by matching a configured
IP address against the local network addresses. Update the model/source paths, the two prefill and two decode IP
addresses, and the HCCL/Gloo interface before running it. The nodes must reach the prefill master for MemFabric
bootstrap and should be started before the router.

```shell theme={null}
#!/usr/bin/env bash
# GLM-5.3 W8A8: 2 prefill nodes + 2 decode nodes (Atlas 800I A3).
# Run this same script on all four nodes, after replacing the placeholders below.
# The IP addresses must match addresses returned by `hostname -I` on each node.

# ============================================================
# Before running, update the following variables:
#   P_IP: two prefill node IP addresses (rank 0 first)
#   D_IP: two decode node IP addresses (rank 0 first)
#   NIC: HCCL/Gloo network interface name
#   MODEL_PATH: local/shared GLM-5.3 W8A8 model directory
#   SGLANG_ROOT: SGLang source directory
# Ensure MemFabric, ATB, and CANN 9.1.0 custom ops are installed.
# ============================================================
P_IP=('<prefill-node-0-ip>' '<prefill-node-1-ip>')
D_IP=('<decode-node-0-ip>' '<decode-node-1-ip>')
NIC='<network-interface>'
MODEL_PATH='/path/to/GLM-5.3-w8a8'
SGLANG_ROOT='/path/to/sglang'

# High-performance CPU configuration (requires sufficient permissions).
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
export SGLANG_SET_CPU_AFFINITY=1

unset https_proxy http_proxy HTTPS_PROXY HTTP_PROXY ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
export LD_LIBRARY_PATH="/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib:${LD_LIBRARY_PATH:-}"
export PYTHONPATH="${SGLANG_ROOT}/python:${PYTHONPATH:-}"

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export ASCEND_MF_LOG_LEVEL=2
export ASCEND_MF_STORE_URL="tcp://${P_IP[0]}:24707"
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export HCCL_SOCKET_IFNAME="${NIC}"
export GLOO_SOCKET_IFNAME="${NIC}"

# Test every local interface rather than assuming the relevant IP is first.
LOCAL_IPS=( $(hostname -I) )
has_local_ip() {
    local wanted="$1" local_ip
    for local_ip in "${LOCAL_IPS[@]}"; do
        [[ "${wanted}" == "${local_ip}" ]] && return 0
    done
    return 1
}

# Prefill: two A3 nodes (32 NPU devices in total); TP=8, PP=4, CP=8.
for i in "${!P_IP[@]}"; do
    if has_local_ip "${P_IP[$i]}"; then
        export HCCL_BUFFSIZE=1200
        export DEEPEP_NORMAL_LONG_SEQ_ROUND=72
        export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
        export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
        export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
        export TASK_QUEUE_ENABLE=2
        export ENABLE_PROFILING=0
        export SGLANG_PP_LAYER_PARTITION=18,20,24,16

        exec python3 -m sglang.launch_server \
            --model-path "${MODEL_PATH}" \
            --disaggregation-mode prefill \
            --host "${P_IP[$i]}" --port 8000 \
            --disaggregation-bootstrap-port 8998 \
            --dist-init-addr "${P_IP[0]}:5000" \
            --trust-remote-code --nnodes 2 --node-rank "$i" \
            --mem-fraction-static 0.72 \
            --attention-backend ascend --device npu \
            --quantization modelslim --dtype bfloat16 \
            --tp-size 8 --pp-size 4 --dp-size 1 \
            --attn-cp-size 8 --enable-prefill-cp --cp-strategy interleave \
            --enable-dp-attention --enable-dp-lm-head \
            --moe-dense-tp 1 --load-balance-method round_robin \
            --disaggregation-transfer-backend ascend \
            --max-running-requests 64 \
            --served-model-name glm-5 \
            --chunked-prefill-size 4096 --max-prefill-tokens 180000 \
            --moe-a2a-backend deepep --deepep-mode normal \
            --disable-shared-experts-fusion --disable-cuda-graph \
            --speculative-algorithm NEXTN \
            --speculative-num-steps 1 \
            --speculative-eagle-topk 1 \
            --speculative-num-draft-tokens 2
    fi
done

# Decode: two A3 nodes (32 NPU devices in total); TP=32, DP=32, EP=32.
for i in "${!D_IP[@]}"; do
    if has_local_ip "${D_IP[$i]}"; then
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export DEEPEP_HCCL_BUFFSIZE=200
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
        export TASK_QUEUE_ENABLE=0
        export SGLANG_NPU_USE_MULTI_STREAM=1

        exec python3 -m sglang.launch_server \
            --model-path "${MODEL_PATH}" \
            --disaggregation-mode decode \
            --host "${D_IP[$i]}" --port 8003 \
            --trust-remote-code \
            --dist-init-addr "${D_IP[0]}:5000" \
            --nnodes 2 --node-rank "$i" \
            --tp-size 32 --dp-size 32 --enable-dp-attention --ep-size 32 \
            --mem-fraction-static 0.97 --max-running-requests 32 \
            --attention-backend ascend --device npu \
            --quantization modelslim --dtype bfloat16 \
            --served-model-name glm-5 \
            --moe-a2a-backend deepep --deepep-mode low_latency \
            --cuda-graph-bs-decode 1 2 3 \
            --disaggregation-transfer-backend ascend \
            --watchdog-timeout 9000 --context-length 180000 \
            --tokenizer-worker-num 4 \
            --disable-shared-experts-fusion \
            --load-balance-method round_robin \
            --speculative-algorithm NEXTN \
            --speculative-num-steps 3 \
            --speculative-eagle-topk 1 \
            --speculative-num-draft-tokens 4
    fi
done

echo "ERROR: None of the P_IP/D_IP addresses matches this host: ${LOCAL_IPS[*]}" >&2
exit 1
```

Launch the router after the prefill and decode services are ready. The router example below preserves the
GLM-5.2 document's PD routing syntax and uses the prefill/decode group master addresses.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   P_MASTER_IP: first prefill node IP (prefill rank 0)
#   D_MASTER_IP: first decode node IP (decode rank 0)
#   ROUTER_HOST_IP: router listening IP
# ============================================================
P_MASTER_IP="<prefill-node-0-ip>"
D_MASTER_IP="<decode-node-0-ip>"
ROUTER_HOST_IP="<router-host-ip>"

python3 -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy round_robin \
    --prefill http://${P_MASTER_IP}:8000 8998 \
    --decode http://${D_MASTER_IP}:8003 \
    --host "${ROUTER_HOST_IP}" \
    --port 6688
```

> [!NOTE]
>
> The prefill and decode groups use separate `--dist-init-addr` endpoints. The `ASCEND_MF_STORE_URL` must point to the
> prefill rank-0 node. The `--max-prefill-tokens 180000` and `--context-length 180000` settings describe this example
> configuration, **not** the model's maximum supported context. For troubleshooting, turn off optional overlap or
> multi-stream settings one at a time to isolate compatibility issues.

## Functional verification

After the service is started, invoke the model by sending a prompt. For PD disaggregation, use the router address and
port `6688`; for single-node co-located serving, use port `8000`, and for the two-node mixed deployment, use port `6688`.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: router/server host IP address
#   PORT: 6688 for the PD router or mixed deployment; 8000 for single-node mode
# ============================================================
HOST="<router-or-server-ip>"
PORT=6688

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response containing the generated answer (for example, "Paris").

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For
additional testing examples (Health Check, Generate, Chat Completions, and port usage),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

GLM-5.3 does not have a validated standalone best-practice page in this tutorial. The
[Online service deployment](#online-service-deployment) section above includes a four-node PD disaggregation
configuration and a two-node mixed deployment configuration based on supplied GLM-5.3 scripts. The single-node
co-located example remains a GLM-5.2-derived reference. The configurations have not been independently verified on hardware.

The two-node mixed example uses `TP=32`, `DP=16`, `HCCL_BUFFSIZE=2000`, and `--cuda-graph-bs-decode 2`.
Disable profiling when measuring uninstrumented service performance.

For this PD disaggregation configuration, prefill prioritizes long sequences using pipeline/context parallelism and
DeepEP `normal`, while decode prioritizes low latency using DP/EP, DeepEP `low_latency`, NPU multi-stream,
and NEXTN. Tune `--mem-fraction-static`, graph capture batch sizes, and concurrency based on observed HBM usage,
TTFT, TPOT, and request throughput. Check that the weight conversion and MTP settings are compatible.

For the full catalog of optimization features and their parameter and compatibility details,
see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## Performance tuning

For supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and parameter issues, refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# GLM-5.3-Flash
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_3_flash



## Introduction

GLM-5.3-Flash is a model in the GLM series. SGLang supports its inference on Ascend NPUs with DeepSeek Sparse
Attention (DSA), multi-token prediction (MTP), and ModelSlim quantized weights.

This document demonstrates the deployment of GLM-5.3-Flash on Ascend NPUs using SGLang, including single-node
prefill-decode mixed deployment, prefill-decode disaggregation, feature configuration, and performance tuning.
The examples use W8A8 weights on Atlas 800I A3 and W8A8\_MXFP8 weights on Ascend 950PR/DT series.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Pipeline Parallelism | `--pp-size 4` |
| Data Parallelism | `--dp-size 16` and `--enable-dp-attention` |
| DeepEP | `--moe-a2a-backend deepep` and `--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill` and `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | `--chunked-prefill-size 8192` |
| NPU Graph | `--cuda-graph-bs-decode 16` |
| Speculative Decoding | `--speculative-algorithm NEXTN` `--speculative-num-steps 3` `--speculative-eagle-topk 1` `--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflicts, see
[Feature Compatibility](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For PD disaggregation, download the weights to a shared directory accessible to both nodes.

* [GLM-5.3-Flash-w8a8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8) (W8A8, 332.97GB, for Atlas 800I A3)
* [GLM-5.3-Flash-w8a8-mxfp8](https://www.modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8-mxfp8) (W8A8\_MXFP8, 338.12GB, for Ascend 950PR/DT series)
* You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize the model. For quantization guidance, see
  [Quantization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_quantization).

The checkpoint directory must contain the model configuration, tokenizer files, quantized weights,
`quant_model_description.json`, and the MTP weights required by NEXTN.
Set `MODEL_PATH` to this directory inside the container.

> [!NOTE]
>
> A3 single-node deployment uses one full Atlas 800I A3 node: eight cards and 16 dies, with `--tp-size 16`.
> 950PR/DT series single-node deployment uses eight NPU devices, with `--tp-size 8`.
> Each PD disaggregation configuration uses two nodes of the same platform: one for prefill and one for decode.

For hardware specifications and terminology, see
[Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware).
For the PD disaggregation configuration reference, see [Best practices](#best-practices).

## Installation

> [!WARNING]
>
> Reserve enough disk space for both the container image and model weights before pulling the image.

Use a SGLang image with the runtime components required by your target platform and deployment mode.
Replace `IMAGE`, `NAME`, and `MODEL_DIR` below, then start the container.
For 950PR/DT series PD disaggregation, choose an image that supports DP attention.

**Atlas 800I A3**

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   IMAGE: SGLang image for Ascend A3
#   NAME: container name
#   MODEL_DIR: model weights directory on the host
# ============================================================

IMAGE="swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-a3-glm5.3-flash"
NAME=sglang-glm53-flash-a3
MODEL_DIR=/path/to/model-weights

docker pull "$IMAGE"

docker run -itd --shm-size=16g --name "$NAME" \
    --privileged=true --net=host \
    -v /var/queue_schedule:/var/queue_schedule \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /usr/local/sbin:/usr/local/sbin \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    -v "${MODEL_DIR}:/models/GLM-5.3-Flash" \
    --device=/dev/davinci0:/dev/davinci0 \
    --device=/dev/davinci1:/dev/davinci1 \
    --device=/dev/davinci2:/dev/davinci2 \
    --device=/dev/davinci3:/dev/davinci3 \
    --device=/dev/davinci4:/dev/davinci4 \
    --device=/dev/davinci5:/dev/davinci5 \
    --device=/dev/davinci6:/dev/davinci6 \
    --device=/dev/davinci7:/dev/davinci7 \
    --device=/dev/davinci8:/dev/davinci8 \
    --device=/dev/davinci9:/dev/davinci9 \
    --device=/dev/davinci10:/dev/davinci10 \
    --device=/dev/davinci11:/dev/davinci11 \
    --device=/dev/davinci12:/dev/davinci12 \
    --device=/dev/davinci13:/dev/davinci13 \
    --device=/dev/davinci14:/dev/davinci14 \
    --device=/dev/davinci15:/dev/davinci15 \
    --device=/dev/davinci_manager:/dev/davinci_manager \
    --device=/dev/hisi_hdc:/dev/hisi_hdc \
    --entrypoint=bash \
    "$IMAGE"

docker exec -it "$NAME" bash
```

**Ascend 950PR/DT series**

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   IMAGE: SGLang image for Ascend 950PR/DT series
#   NAME: container name
#   MODEL_DIR: model weights directory on the host
# ============================================================

IMAGE="<your 950PR/DT series image>"
NAME=sglang-glm53-flash-950pr-dt-series
MODEL_DIR=/path/to/model-weights

docker pull "$IMAGE"

docker run -itd --shm-size=16g --name "$NAME" \
    --privileged=true --net=host \
    -v /var/queue_schedule:/var/queue_schedule \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /usr/local/sbin:/usr/local/sbin \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
    -v "${MODEL_DIR}:/models/GLM-5.3-Flash" \
    --device=/dev/davinci0:/dev/davinci0 \
    --device=/dev/davinci1:/dev/davinci1 \
    --device=/dev/davinci2:/dev/davinci2 \
    --device=/dev/davinci3:/dev/davinci3 \
    --device=/dev/davinci4:/dev/davinci4 \
    --device=/dev/davinci5:/dev/davinci5 \
    --device=/dev/davinci6:/dev/davinci6 \
    --device=/dev/davinci7:/dev/davinci7 \
    --device=/dev/davinci_manager:/dev/davinci_manager \
    --device=/dev/hisi_hdc:/dev/hisi_hdc \
    --entrypoint=bash \
    "$IMAGE"

docker exec -it "$NAME" bash
```

> [!TIP]
>
> * The examples mount the host checkpoint directory at `/models/GLM-5.3-Flash`. Set
>   `MODEL_PATH=/models/GLM-5.3-Flash` in the launch scripts when using this mount.
> * Give each container its own `NAME`. For PD disaggregation, start a container on both nodes.
> * The examples use the SGLang package installed in the image and require no source-code path or `PYTHONPATH` override.

## Online service deployment

### Single-node deployment

Prefill and decode run in the same instance. Select the script for your platform and run it inside the container.

**Atlas 800I A3**

Use the ModelSlim W8A8 checkpoint on one Atlas 800I A3 node, with TP16.
Replace `MODEL_PATH` with the checkpoint directory inside the container.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights

source /usr/local/Ascend/cann-9.1.0/opp/vendors/customize/bin/set_env.bash
source /usr/local/Ascend/nnal/atb/set_env.sh

export HCCL_BUFFSIZE=1536
unset ASCEND_LAUNCH_BLOCKING

python3 -m sglang.launch_server \
    --model-path "$MODEL_PATH" \
    --quantization modelslim \
    --attention-backend ascend \
    --device npu \
    --tp-size 16 --nnodes 1 \
    --chunked-prefill-size 4096 \
    --max-prefill-tokens 8192 \
    --trust-remote-code \
    --mem-fraction-static 0.78 \
    --page-size 128 \
    --served-model-name GLM-5.3-Flash \
    --cuda-graph-bs-decode 16 \
    --max-running-requests 16 \
    --load-balance-method round_robin \
    --enable-multimodal \
    --moe-a2a-backend deepep --deepep-mode auto \
    --speculative-draft-model-path "$MODEL_PATH" \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --watchdog-timeout 9000 \
    --host 127.0.0.1 --port 8810
```

**Ascend 950PR/DT series**

Use the ModelSlim W8A8\_MXFP8 checkpoint on one 950PR/DT series node, with TP8 and no DP attention.
Replace `MODEL_PATH` with the checkpoint directory inside the container.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
# ============================================================

MODEL_PATH=/path/to/model-weights

source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/memfabric_hybrid/set_env.sh

export SGLANG_NPU_PROFILING=0
export ASCEND_LAUNCH_BLOCKING=1
export HCCL_BUFFSIZE=256

python3 -m sglang.launch_server \
    --model-path "$MODEL_PATH" \
    --quantization modelslim \
    --attention-backend ascend \
    --device npu \
    --tp-size 8 --nnodes 1 \
    --chunked-prefill-size 8192 \
    --max-prefill-tokens 8192 \
    --trust-remote-code \
    --mem-fraction-static 0.83 \
    --page-size 64 \
    --served-model-name GLM-5.3-Flash \
    --load-format auto \
    --max-running-requests 16 \
    --pre-warm-nccl \
    --speculative-draft-model-path "$MODEL_PATH" \
    --speculative-draft-kv-cache-dtype bf16 \
    --speculative-algorithm NEXTN \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --watchdog-timeout 1200 \
    --cuda-graph-bs-decode 1 8 64 \
    --host 127.0.0.1 --port 8810
```

### Prefill-decode disaggregation deployment

PD disaggregation places prefill and decode on separate nodes. Replace the model path, node IP addresses, and
network interface names, then run the same script on both nodes. The script selects its role by matching the local
IP address.

Ensure the worker HTTP ports, prefill bootstrap port, and MemFabric store port are reachable between the nodes.

**Atlas 800I A3**

Use W8A8 weights on two Atlas 800I A3 nodes. The prefill instance uses TP4 / PP4 / DP4; the decode instance uses
TP16 / PP1 / DP16 and NEXTN speculative decoding with three steps and four draft tokens.
Run the CPU tuning commands with the required host privileges.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   PREFILL_IP: prefill node IP address
#   DECODE_IP: decode node IP address
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
PREFILL_IP="<your prefill ip>"
DECODE_IP="<your decode ip>"

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
export SGLANG_SET_CPU_AFFINITY=1

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export HCCL_SOCKET_IFNAME="<network-interface>"
export GLOO_SOCKET_IFNAME="<network-interface>"
export ASCEND_MF_STORE_URL="tcp://${PREFILL_IP}:24707"
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200

unset ASCEND_LAUNCH_BLOCKING
unset http_proxy https_proxy HTTP_PROXY HTTPS_PROXY all_proxy ALL_PROXY
export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${no_proxy:-}"
export NO_PROXY="$no_proxy"
LOCAL_IPS=" $(hostname -I) "

if [[ "$LOCAL_IPS" == *" $PREFILL_IP "* ]]; then
    export SGLANG_HOST_IP="$PREFILL_IP"
    export HCCL_BUFFSIZE=1200
    export DEEPEP_HCCL_BUFFSIZE=1200
    export DEEPEP_NORMAL_LONG_SEQ_ROUND=128
    export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
    export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
    export TASK_QUEUE_ENABLE=2
    export HCCL_IF_BASE_PORT=61000

    python3 -m sglang.launch_server \
        --model-path "$MODEL_PATH" \
        --quantization modelslim \
        --device npu \
        --attention-backend ascend \
        --trust-remote-code \
        --enable-multimodal \
        --served-model-name GLM-5.3-Flash \
        --disaggregation-mode prefill \
        --disaggregation-bootstrap-port 8998 \
        --disaggregation-transfer-backend ascend \
        --dist-init-addr "${PREFILL_IP}:5000" \
        --tp-size 4 --pp-size 4 --dp-size 4 --nnodes 1 \
        --enable-dp-attention \
        --enable-dp-lm-head \
        --moe-dense-tp 1 \
        --moe-a2a-backend deepep --deepep-mode normal \
        --load-balance-method round_robin \
        --mem-fraction-static 0.78 \
        --page-size 128 \
        --max-running-requests 16 \
        --chunked-prefill-size 4096 \
        --max-prefill-tokens 8192 \
        --watchdog-timeout 9000 \
        --host "$PREFILL_IP" --port 8810
elif [[ "$LOCAL_IPS" == *" $DECODE_IP "* ]]; then
    export SGLANG_HOST_IP="$DECODE_IP"
    export HCCL_BUFFSIZE=512
    export DEEPEP_HCCL_BUFFSIZE=512
    export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
    export SGLANG_ENABLE_SPEC_V2=1
    export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1
    export SGLANG_NPU_USE_MULTI_STREAM=1
    export TASK_QUEUE_ENABLE=0
    export HCCL_IF_BASE_PORT=61010

    python3 -m sglang.launch_server \
        --model-path "$MODEL_PATH" \
        --quantization modelslim \
        --device npu \
        --attention-backend ascend \
        --trust-remote-code \
        --enable-multimodal \
        --served-model-name GLM-5.3-Flash \
        --disaggregation-mode decode \
        --disaggregation-transfer-backend ascend \
        --dist-init-addr "${DECODE_IP}:5000" \
        --tp-size 16 --pp-size 1 --dp-size 16 --nnodes 1 \
        --enable-dp-attention \
        --moe-a2a-backend deepep --deepep-mode low_latency \
        --load-balance-method round_robin \
        --mem-fraction-static 0.75 \
        --page-size 128 \
        --max-running-requests 32 \
        --cuda-graph-bs-decode 1 2 \
        --speculative-draft-model-path "$MODEL_PATH" \
        --speculative-algorithm NEXTN \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --watchdog-timeout 9000 \
        --host "$DECODE_IP" --port 8811
else
    echo "Local IP does not match PREFILL_IP or DECODE_IP: $LOCAL_IPS" >&2
    exit 1
fi
```

**Ascend 950PR/DT series**

Use W8A8\_MXFP8 weights on two 950PR/DT series nodes. Both instances use TP8 / PP1 / DP2 and NEXTN speculative decoding
with four steps and five draft tokens. The nodes must be able to communicate over the `device_urma` transport.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   PREFILL_IP: prefill node IP address
#   DECODE_IP: decode node IP address
# ============================================================

MODEL_PATH=/path/to/model-weights
PREFILL_IP="<your prefill ip>"
DECODE_IP="<your decode ip>"

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash

export SGLANG_NPU_PROFILING=0
export HCCL_BUFFSIZE=256
export ASCEND_MF_STORE_URL="tcp://${PREFILL_IP}:24670"
export ASCEND_MF_TRANSFER_PROTOCOL=device_urma
unset ASCEND_LAUNCH_BLOCKING
export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${no_proxy:-}"
export NO_PROXY="$no_proxy"
LOCAL_IPS=" $(hostname -I) "

if [[ "$LOCAL_IPS" == *" $PREFILL_IP "* ]]; then
    export SGLANG_HOST_IP="$PREFILL_IP"
    python3 -m sglang.launch_server \
        --model-path "$MODEL_PATH" \
        --quantization modelslim --load-format auto \
        --device npu --attention-backend ascend --trust-remote-code \
        --served-model-name GLM-5.3-Flash \
        --disaggregation-mode prefill \
        --disaggregation-bootstrap-port 8998 \
        --disaggregation-transfer-backend ascend \
        --tp-size 8 --pp-size 1 --dp-size 2 --enable-dp-attention --nnodes 1 \
        --chunked-prefill-size 16384 --max-prefill-tokens 16384 \
        --mem-fraction-static 0.83 --page-size 64 --max-running-requests 16 \
        --speculative-draft-model-path "$MODEL_PATH" \
        --speculative-draft-kv-cache-dtype bf16 \
        --speculative-algorithm NEXTN --speculative-num-steps 4 \
        --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
        --pre-warm-nccl --watchdog-timeout 1200 --cuda-graph-bs-decode 1 8 64 \
        --host "$PREFILL_IP" --port 8810
elif [[ "$LOCAL_IPS" == *" $DECODE_IP "* ]]; then
    export SGLANG_HOST_IP="$DECODE_IP"
    python3 -m sglang.launch_server \
        --model-path "$MODEL_PATH" \
        --quantization modelslim --load-format auto \
        --device npu --attention-backend ascend --trust-remote-code \
        --served-model-name GLM-5.3-Flash \
        --disaggregation-mode decode \
        --disaggregation-bootstrap-port 8998 \
        --disaggregation-transfer-backend ascend \
        --tp-size 8 --pp-size 1 --dp-size 2 --enable-dp-attention --nnodes 1 \
        --chunked-prefill-size 16384 --max-prefill-tokens 16384 \
        --mem-fraction-static 0.83 --page-size 64 --max-running-requests 16 \
        --speculative-draft-model-path "$MODEL_PATH" \
        --speculative-draft-kv-cache-dtype bf16 \
        --speculative-algorithm NEXTN --speculative-num-steps 4 \
        --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
        --pre-warm-nccl --watchdog-timeout 1200 --cuda-graph-bs-decode 1 8 64 \
        --host "$DECODE_IP" --port 8811
else
    echo "Local IP does not match PREFILL_IP or DECODE_IP: $LOCAL_IPS" >&2
    exit 1
fi
```

Launch the router after the prefill and decode services are ready. Use the same node IP addresses as in the worker
scripts. Client requests go to the router on port 8000; the worker ports are 8810 and 8811, and the prefill bootstrap
port is 8998.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   PREFILL_IP: prefill node IP address
#   DECODE_IP: decode node IP address
#   ROUTER_HOST_IP: router node IP address
# ============================================================

PREFILL_IP="<your prefill ip>"
DECODE_IP="<your decode ip>"
ROUTER_HOST_IP="<your router ip>"
export no_proxy="127.0.0.1,localhost,${PREFILL_IP},${DECODE_IP},${ROUTER_HOST_IP},${no_proxy:-}"
export NO_PROXY="$no_proxy"

python3 -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill "http://${PREFILL_IP}:8810" 8998 \
    --decode "http://${DECODE_IP}:8811" \
    --host "$ROUTER_HOST_IP" --port 8000 \
    --policy round_robin \
    --worker-startup-timeout-secs 3600 \
    --request-timeout-secs 7200 \
    --disable-retries
```

## Functional verification

After the service starts, send a prompt to verify inference. For mixed deployment, run the command inside the
container with `HOST=127.0.0.1` and `PORT=8810`. For PD disaggregation, use the router IP address and `PORT=8000`.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: server or router host address
#   PORT: 8810 for mixed deployment; 8000 for the router
# ============================================================

HOST=127.0.0.1
PORT=8810

curl "http://${HOST}:${PORT}/generate" \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Check for an HTTP 200 response with the generated answer. Once the server prints
`The server is fired up and ready to roll!`, it is ready to accept requests.
For Health Check, Generate, Chat Completions, and port usage examples, see
[Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service).

## Accuracy evaluation

For evaluation methods and datasets, see
[Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation).

## Performance

For benchmark methods and metric definitions, see
[Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing).

For a generated shared-prefix benchmark workload, use approximately 128K input tokens and 1K output tokens.
For mixed deployment, use the worker host and port 8810. For PD disaggregation, use the router host and port 8000.
Set both `--model` and `--tokenizer` to the checkpoint path and use `--served-model-name GLM-5.3-Flash`.

## Best practices

### Best practice configuration reference

The [Online service deployment](#online-service-deployment) section above contains complete scripts for
single-node mixed deployment and PD disaggregation.

For the optimization feature catalog and compatibility details, see
[Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization).

## Performance tuning

For the feature overview, see [Supported features](#supported-features). For tuning parameters and feature
compatibility, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization).

## FAQ

For environment, installation, and general parameter issues, see
[Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/ascend_npu_faq).


# Hy3
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3



## Introduction

Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer
parameters, developed by the Tencent Hy Team. It features 192 experts with top-8 activation per token, GQA attention
(64 heads, 8 KV heads, head dim 128), and a 256K context length. The model supports built-in reasoning via
`reasoning_effort`, tool calling, and multi-token prediction (MTP) for speculative decoding. Hy3 significantly
outperforms similar-size models and rivals flagship open-source models with 2–5× the parameters, with notable
improvements in agentic workflows, long-context tasks, and production reliability.

This document demonstrates the deployment of Hy3 on Ascend NPUs using SGLang, including single-node deployment and
speculative decoding.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode`; e.g. `--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40` |
| Speculative Decoding | `--speculative-algorithm EAGLE \` `--speculative-num-steps 2 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| Reasoning Mode | `extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}` — deep chain-of-thought; `"reasoning_effort": "low"` — short thinking chain; `"reasoning_effort": "no_think"` — direct response, no thinking |
| Tool Calling | `--tool-call-parser auto --reasoning-parser auto` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

> [!WARNING]
>
> Hy3 has 295B total parameters. If you need to download model weights, check the model size before downloading to reserve enough space.

* [Hy3](https://www.modelscope.cn/models/Tencent-Hunyuan/Hy3) (BF16, 597.60GB)

> [!TIP]
>
> The BF16 variant (597.60GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full
> Ascend A3 Series node (8 cards × 2 dies).

It is recommended to download the model weights to a shared directory accessible from within the container.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

> [!NOTE]
>
> Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
> command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
> release tag.
> For details, see
> [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=64g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.
> * `--shm-size=64g` is recommended for 16-NPU deployments with large batch sizes.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode). Hy3 with 295B
parameters requires all 16 logical NPUs on an Ascend A3 Series server (8 physical NPUs × 2 logical cores each, 64 GB HBM
per logical NPU).

Set the following environment variables before launching the server:

```bash theme={null}
# Performance tuning
export SGLANG_SET_CPU_AFFINITY=1
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0

# CANN environment (adjust paths to match your installation)
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

# Ascend NPU settings
export ASCEND_USE_FIA=1
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=3000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo

# SGLang settings
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
```

Launch the server:

```bash theme={null}
MODEL_PATH="/path/to/Hy3"

python3 -m sglang.launch_server \
    --model-path ${MODEL_PATH} \
    --attention-backend ascend \
    --reasoning-parser auto \
    --tool-call-parser auto \
    --device npu \
    --tp-size 16 \
    --host 0.0.0.0 \
    --port 9999 \
    --mem-fraction-static 0.84 \
    --dtype bfloat16 \
    --base-gpu-id 0 \
    --prefill-max-requests 40 \
    --max-running-requests 40 \
    --cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 2 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 3
```

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 9999)
# ============================================================

curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Hy3",
        "messages": [{"role": "user", "content": "What is the capital of France?"}],
        "max_tokens": 64,
        "temperature": 0.9,
        "extra_body": {"chat_template_kwargs": {"reasoning_effort": "no_think"}}
    }'
```

Expected result: an HTTP 200 response with `"Paris"` in the content field.

To enable chain-of-thought reasoning for math or coding tasks, set `reasoning_effort` to `"high"`:

```shell theme={null}
curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Hy3",
        "messages": [{"role": "user", "content": "Solve: if 3x + 7 = 22, what is x?"}],
        "max_tokens": 8192,
        "temperature": 0.9,
        "extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}}
    }'
```

The response will contain a `reasoning_content` field with the thinking process and a `content` field with the final
answer.

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

Hy3 has no standalone best practice page yet. The [Single-node online deployment](#single-node-online-deployment) section above provides
the ready-to-use script for the supported PD mixed case.

For the full catalog of optimization features and their parameter and compatibility details,
see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

### Reasoning mode selection

Hy3 supports three reasoning modes via `chat_template_kwargs`:

| `reasoning_effort` | Behavior | Recommended for |
| - | - | - |
| `no_think` | No chain-of-thought, direct reply | Simple Q\&A, low-latency tasks |
| `low` | Short thinking chain | Moderate reasoning tasks |
| `high` | Full chain-of-thought | Math, coding, complex reasoning |

Use `no_think` for interactive or latency-sensitive applications to reduce output token count significantly.

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).
This section only covers model-specific issues.


# Kimi-K2.6
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6



## Introduction

Kimi-K2.6 is an open-source, native multimodal agentic model developed by Moonshot AI, built through continual
pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It is a Mixture-of-Experts
(MoE) model featuring Multi-head Latent Attention (MLA) and MoE architecture, with 1T total parameters and 32B
active parameters. The model seamlessly integrates vision and language understanding with advanced agentic
capabilities, supporting both instant and thinking modes as well as conversational and agentic paradigms.

This document demonstrates the deployment of Kimi-K2.6 on Ascend NPUs using SGLang, including single-node PD mixed
mode, multi-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance
optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Kimi-K2.6) is fully supported in
this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 24 32 48 64 96 120` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \` `--speculative-draft-model-path /path/to/draft-model-weights \` `--speculative-num-steps 4 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 5 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
| MLAPO | `export SGLANG_NPU_USE_MLAPO=1` |
| Multistream MoE | `export SGLANG_NPU_USE_MULTI_STREAM=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Kimi-K2.6](https://www.modelscope.cn/models/moonshotai/Kimi-K2.6) (BF16, 595.21GB)
* [Kimi-K2.6-w4a8](https://www.modelscope.cn/models/Eco-Tech/Kimi-K2.6-w4a8) (W4A8 quantized version, 535.91GB)
* [kimi-k2.6-eagle3](https://www.modelscope.cn/models/lightseekorg/kimi-k2.6-eagle3) (EAGLE3 draft model for speculative decoding)
* You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Kimi-K2.6-w4a8` from `Kimi-K2.6`.

> [!NOTE]
>
> We recommend deploying the W4A8 variant for reduced resource usage and higher throughput.
> It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Kimi K2.6 Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#single-node-pd-mixed).

### Multi-node online deployment

Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping
prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single
node can provide. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Kimi-K2.6 Best Practice — Multi-node On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#multi-node-pd-mixed).

### Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Kimi-K2.6 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#pd-disaggregation).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

For multimodal requests (text + image):

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Kimi-K2.6-w4a8",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
                    {"type": "text", "text": "Describe this image."}
                ]
            }
        ]
    }'
```

Expected result: an HTTP 200 response with a description of the image.

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Kimi-K2.6 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# MiMo-V2-Flash
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash



## Introduction

MiMo-V2-Flash is a Mixture-of-Experts (MoE) large language model developed by Xiaomi. It employs advanced architecture
with speculative decoding capabilities for accelerated inference. The model is optimized for high throughput and low
latency scenarios through PD disaggregation deployment.

This document demonstrates the deployment of MiMo-V2-Flash on Ascend NPUs using SGLang, including multi-node PD
disaggregation mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (MiMo-V2-Flash) is fully supported in
this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 8` (prefill) or `--tp-size 16` (decode) |
| Data Parallelism | `--dp-size 2` |
| Expert Parallelism | `--moe-a2a-backend deepep \` `--deepep-mode low_latency` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32` |
| Speculative Decoding | `--speculative-algorithm EAGLE \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--enable-multi-layer-eagle` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` |
| DP LM Head | `--enable-dp-lm-head` |
| DP Attention | `--dp-size 2 \` `--enable-dp-attention` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [MiMo-V2-Flash-W8A8](https://www.modelscope.cn/models/iridiumine/MiMo-V2-Flash-W8A8) (Quantized version, 311.5GB)

> [!NOTE]
>
> The W8A8 variant (311.5GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to
> 8 dies on an A3 Series node (4 cards).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving
throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[MiMo-V2-Flash Best Practice — W8A8 24P PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash#pd-disaggregation).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 9903)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[MiMo-V2-Flash Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# MiniMax-M2.5
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5



## Introduction

MiniMax-M2.5 is a Mixture-of-Experts (MoE) large language model developed by MiniMax, featuring a sparse MoE
architecture with approximately 230B total parameters and 10B active parameters. It supports native long-context
processing up to 200k tokens. The model supports EAGLE3 speculative decoding with a
custom eagle model for accelerated inference, and excels at general language understanding, reasoning, and
long-context tasks.

This document demonstrates the deployment of MiniMax-M2.5 on Ascend NPUs using SGLang, including single-node PD mixed
mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (MiniMax-M2.5) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend ascend_fuseep \` `--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 8 16 24 32 48 64 80` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \` `--speculative-draft-model-path /path/to/draft-model-weights \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.

* [MiniMax-M2.5-w8a8-QuaRot](https://www.modelscope.cn/models/Eco-Tech/MiniMax-M2.5-w8a8-QuaRot) (W8A8 quantized version, 230.82GB)
* [MiniMax-M2.5-eagle-model](https://www.modelscope.cn/models/sgl-npu/MiniMax-M2.5-eagel-model-0318) (EAGLE3 draft model for speculative decoding)

> [!NOTE]
>
> The W8A8 variant (230.82GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to
> 8 dies on an A3 Series node (4 cards).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[MiniMax-M2.5 Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[MiniMax-M2.5 Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3-235B-A22B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b



## Introduction

Qwen3-235B-A22B is a Mixture-of-Experts (MoE) large language model developed by Alibaba, featuring 235B total
parameters with 22B active parameters. It employs Grouped-Query Attention (GQA) and Qwen3MoE architecture, with
support for EAGLE3 speculative decoding for accelerated inference. The model excels at instruction following,
logical reasoning, text comprehension, mathematics, science, coding, and tool usage, available in both standard
and thinking/reasoning-enhanced editions.

This document demonstrates the deployment of Qwen3-235B-A22B on Ascend NPUs using SGLang, including single-node PD mixed
mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel, feature configuration,
and performance optimization.

This document is validated and written based on **SGLang v0.5.18**. The current model (Qwen3-235B-A22B) is fully supported in
this version. It is recommended to use exactly v0.5.18 (the `v0.5.18-cann9.0.0-a3` image); later versions are not covered
by this document and may introduce breaking changes.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 16` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend ascend_fuseep` |
| PD Disaggregation | `--disaggregation-mode prefill \` `--disaggregation-transfer-backend ascend` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \` `--speculative-draft-model-path /path/to/draft-model-weights \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
| Context Parallelism | `--enable-prefill-cp \` `--attn-cp-size 2 \` `--moe-dp-size 2` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Qwen3-235B-A22B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-235B-A22B-Instruct-2507) (BF16, 470.21GB)
* [Qwen3-235B-A22B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-235B-A22B-W8A8) (W8A8 quantized version, 236.80GB)
* [Qwen3-235B-A22B-Eagle3](https://www.modelscope.cn/models/nv-community/Qwen3-235B-A22B-Eagle3) (EAGLE3 draft model for speculative decoding)

> [!NOTE]
>
> The BF16 variant (470.21GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full
> A3 Series node (8 cards, 16 dies).
>
> The W8A8 variant (236.80GB) can be deployed on 8 × 64GB (`--tp-size 8`), which corresponds to 8 dies on an A3 Series node (4 cards).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

The following command is based on the stable release tag `v0.5.18-cann9.0.0-a3`.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
docker pull quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.18-cann9.0.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3-235B-A22B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b#single-node-pd-mixed).

### Multi-node PD disaggregation deployment

#### 256K long-sequence PD disaggregation on 2 x Ascend A3 Series nodes (without CP)

This configuration uses PD disaggregation for **256K long-sequence inference** on 2 x Ascend A3 Series nodes with context parallel disabled.
The following command is based on the **W8A8** quantized model.

1. Set the shared environment variables on both prefill and decode nodes:

```bash Shared environment theme={null}
#============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   PREFILL_HOST_IP: prefill node IP address
#   HCCL_SOCKET_IFNAME: network interface name for HCCL (use ifconfig to find)
#   GLOO_SOCKET_IFNAME: network interface name for Gloo (use ifconfig to find)
#============================================================

export ASCEND_USE_FIA=1
export SGLANG_SET_CPU_AFFINITY=1
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:12345"
export HCCL_SOCKET_IFNAME=<network-interface>
export GLOO_SOCKET_IFNAME=<network-interface>

MODEL_PATH=/path/to/model-weights
```

2. Run on the **prefill node**:

```bash Prefill node theme={null}
#============================================================
# Before running, update the following variable:
#   PREFILL_HOST_IP: prefill node IP address
#============================================================

export ASCEND_LAUNCH_BLOCKING=1
export HCCL_BUFFSIZE=1500
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
export DEEPEP_NORMAL_LONG_SEQ_ROUND=128
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1

python3 -m sglang.launch_server \
    --model-path ${MODEL_PATH} \
    --disaggregation-mode prefill \
    --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8995 \
    --attention-backend ascend \
    --disable-radix-cache \
    --chunked-prefill-size -1 \
    --skip-server-warmup \
    --device npu \
    --quantization modelslim \
    --tp-size 16 \
    --mem-fraction-static 0.45 \
    --max-running-requests 1 \
    --host <PREFILL_HOST_IP> \
    --port 8000 \
    --dist-init-addr <PREFILL_HOST_IP>:5000 \
    --nnodes 1 \
    --node-rank 0 \
    --moe-a2a-backend deepep \
    --deepep-mode normal
```

3. Run on the **decode node**:

```bash Decode node theme={null}
#============================================================
# Before running, update the following variable:
#   DECODE_HOST_IP: decode node IP address
#============================================================

export HCCL_BUFFSIZE=4000
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=16

python3 -m sglang.launch_server \
    --model-path ${MODEL_PATH} \
    --disaggregation-mode decode \
    --disaggregation-transfer-backend ascend \
    --attention-backend ascend \
    --mem-fraction-static 0.8 \
    --disable-cuda-graph \
    --device npu \
    --quantization modelslim \
    --disable-radix-cache \
    --chunked-prefill-size 8192 \
    --skip-server-warmup \
    --tp-size 16 \
    --max-running-requests 1 \
    --host <DECODE_HOST_IP> \
    --port 8232 \
    --moe-a2a-backend deepep \
    --deepep-mode low_latency \
    --disable-overlap-schedule
```

4. Launch the SGLang Router (on any reachable node):

```bash Router theme={null}
#============================================================
# Before running, update the following variables:
#   PREFILL_HOST_IP: prefill node IP address
#   DECODE_HOST_IP: decode node IP address
#   ROUTER_HOST_IP: router node IP address
#============================================================

python3 -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://<PREFILL_HOST_IP>:8000 8995 \
    --decode http://<DECODE_HOST_IP>:8232 \
    --host <ROUTER_HOST_IP> \
    --port 6689
```

#### Prefill Context Parallel (PCP) on 2 x Ascend A3 Series nodes

This configuration enables **Prefill Context Parallel** (`--enable-prefill-cp`) to split the context
across CP ranks during prefill, reducing per-device memory pressure and improving TTFT for long sequences.
PD disaggregation is required. The following command is based on the **W8A8** quantized model.

> [!WARNING]
>
> **Constraints:**
>
> * Prefill side must set `--max-running-requests 1` (PCP only supports batch\_size=1)
> * `--attn-cp-size` must evenly divide `--tp-size`; each CP rank occupies `tp_size / cp_size` NPUs

1. Run on the **prefill node**:

```bash Prefill node theme={null}
#============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   PREFILL_HOST_IP: prefill node IP address
#============================================================

export SGLANG_SET_CPU_AFFINITY=1
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:23456"
export ASCEND_USE_FIA=True

python3 -m sglang.launch_server \
    --model-path ${MODEL_PATH} \
    --trust-remote-code \
    --disaggregation-mode prefill \
    --disaggregation-transfer-backend ascend \
    --disaggregation-bootstrap-port 8995 \
    --quantization modelslim \
    --attention-backend ascend \
    --skip-server-warmup \
    --mem-fraction-static 0.7 \
    --chunked-prefill-size 32768 \
    --device npu \
    --base-gpu-id 0 \
    --tp-size 16 \
    --enable-prefill-cp \
    --attn-cp-size 2 \
    --moe-dp-size 2 \
    --max-running-requests 1 \
    --host <PREFILL_HOST_IP> \
    --port 8000 \
    --nnodes 1 \
    --node-rank 0 \
    --dist-init-addr <PREFILL_HOST_IP>:6688
```

Key parameters for PCP:

| Parameter | Value | Description |
| - | - | - |
| `--enable-prefill-cp` | flag | Enable PCP feature |
| `--attn-cp-size` | 2 | Split context across 2 CP ranks (each rank handles half the sequence) |
| `--moe-dp-size` | 2 | MoE DP size, should match `--attn-cp-size` |
| `--max-running-requests` | 1 | Required by PCP (batch\_size=1 constraint) |

2. Run on the **decode node**:

```bash Decode node theme={null}
#============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DECODE_HOST_IP: decode node IP address
#   PREFILL_HOST_IP: prefill node IP address (for ASCEND_MF_STORE_URL)
#============================================================

export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:23456"
export ASCEND_USE_FIA=True

python3 -m sglang.launch_server \
    --model-path ${MODEL_PATH} \
    --trust-remote-code \
    --disaggregation-mode decode \
    --disaggregation-transfer-backend ascend \
    --quantization modelslim \
    --attention-backend ascend \
    --disable-radix-cache \
    --disable-cuda-graph \
    --mem-fraction-static 0.7 \
    --chunked-prefill-size 32768 \
    --skip-server-warmup \
    --device npu \
    --base-gpu-id 0 \
    --tp-size 8 \
    --max-running-requests 32 \
    --host <DECODE_HOST_IP> \
    --port 8001 \
    --nnodes 1 \
    --node-rank 0 \
    --dist-init-addr <DECODE_HOST_IP>:6688
```

> [!TIP]
>
> `ASCEND_MF_STORE_URL` on both nodes must point to the same KV store (typically the prefill node IP).
> `ASCEND_USE_FIA=True` enables fast interconnect aggregation for KV transfer.
> PCP is a prefill-only feature; the decode side needs no CP-related flags.

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3-235B-A22B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3-30B-A3B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b



## Introduction

Qwen3-30B-A3B is a Mixture-of-Experts (MoE) large language model developed by Alibaba, featuring 30B total
parameters with 3B active parameters. It employs Grouped-Query Attention (GQA) and Qwen3MoE architecture, with
support for EAGLE3 speculative decoding for accelerated inference. The model excels at instruction following,
logical reasoning, text comprehension, mathematics, science, coding, and tool usage, available in both standard
and thinking/reasoning-enhanced editions.

This document demonstrates the deployment of Qwen3-30B-A3B on Ascend NPUs using SGLang, including single-node PD mixed
mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-30B-A3B) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 2` |
| Data Parallelism | `--dp-size 2` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \` `--speculative-draft-model-path /path/to/draft-model-weights \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Qwen3-30B-A3B-Instruct-2507](https://www.modelscope.cn/models/Qwen/Qwen3-30B-A3B-Instruct-2507) (BF16, 61.08GB, recommended)
* [Qwen3-30B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8) (W8A8 quantized version, 31.29GB)
* [Qwen3-a3B\_eagle3](https://www.modelscope.cn/models/vllm-ascend/Qwen3-a3B_eagle3) (EAGLE3 draft model for speculative decoding)

> [!NOTE]
>
> Both variants fit within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on A3 Series.

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3-30B-A3B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3-30B-A3B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3-32B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b



## Introduction

Qwen3-32B is a dense model in the Qwen3 series developed by Alibaba, featuring 32B parameters with Grouped-Query
Attention (GQA) and up to 128k (131k with YaRN) context length. It delivers significant improvements in instruction following, logical
reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports EAGLE3 speculative
decoding for accelerated inference and supports both standard and thinking/reasoning modes.

This document demonstrates the deployment of Qwen3-32B on Ascend NPUs using SGLang, including single-node PD mixed mode,
feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-32B) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later
version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 4` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 4 8 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \` `--speculative-draft-model-path /path/to/draft-model-weights \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Qwen3-32B](https://www.modelscope.cn/models/Qwen/Qwen3-32B) (BF16, 65.54GB)
* [Qwen3-32B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-32B-W8A8) (W8A8 quantized version, 42.77GB)
* [Eagle3-Qwen3-32B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-32B-zh) (EAGLE3 draft model for speculative decoding)

> [!NOTE]
>
> The BF16 variant (65.54GB) can be deployed on 2 × 64GB of device memory (`--tp-size 2`), which corresponds to 1 card (2 dies) on A3 Series.
>
> The W8A8 variant (42.77GB) fits within a single die.

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3-32B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3-32B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3.5-397B-A17B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b



## Introduction

Qwen3.5-397B-A17B is the latest flagship model in the Qwen series developed by Alibaba, featuring a Gated Delta
Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated). It employs hybrid
attention with Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer, and MoE
routing with Top-10 active out of 512 routed experts plus a dedicated shared expert. The model supports multimodal
inputs (text, image, video) with native context lengths of up to 262,144 tokens, and includes built-in multi-token
prediction (MTP) for speculative decoding.

This document demonstrates the deployment of Qwen3.5-397B-A17B on Ascend NPUs using SGLang, including single-node PD
mixed mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.5-397B-A17B) is fully
supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to
use v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 16` |
| Data Parallelism | `--dp-size 8` |
| Expert Parallelism | `--ep-size 16 \` `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP, 235.88GB)

> [!NOTE]
>
> The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to 8 dies on an A3 Series node (4 cards).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3.5-397B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

For multimodal requests (text + image):

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen3.5-397B-A17B-w4a8-mtp",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
                    {"type": "text", "text": "Describe this image."}
                ]
            }
        ]
    }'
```

Expected result: an HTTP 200 response with a description of the image.

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3.5-397B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3.6-27B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b



## Introduction

Qwen3.6-27B is a dense model in the Qwen3.6 series developed by Alibaba, featuring 27B parameters with a hybrid
architecture combining Gated Delta Networks (linear, O(n) complexity) with full attention every 4th layer. It supports
multimodal inputs (text, image, video) with native context lengths of up to 262,144 tokens, and includes built-in
multi-token prediction (MTP) for speculative decoding. The model delivers strong performance in instruction following,
reasoning, text comprehension, and tool usage.

This document demonstrates the deployment of Qwen3.6-27B on Ascend NPUs using SGLang, including single-node PD mixed
mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.6-27B) is fully supported
in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 8 16 32 48` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version, 36.45GB)

> [!NOTE]
>
> The W8A8 variant (36.45GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on A3 Series.

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3.6-27B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

For multimodal requests (text + image):

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen3.6-27B-w8a8",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
                    {"type": "text", "text": "Describe this image."}
                ]
            }
        ]
    }'
```

Expected result: an HTTP 200 response with a description of the image.

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3.6-27B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3.6-35B-A3B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b



## Introduction

Qwen3.6-35B-A3B is a Mixture-of-Experts model in the Qwen3.6 series developed by Alibaba, featuring 35B total
parameters with 3B active parameters per token. It uses a hybrid architecture that combines Gated DeltaNet (linear,
O(n) complexity) with full attention applied every Nth layer, enabling efficient long-context modeling. The model
supports multimodal inputs (text, image, video) with long context lengths, and includes built-in multi-token prediction
(NEXTN) for speculative decoding. It delivers strong performance in instruction following, reasoning, text
comprehension, and tool usage.

This document demonstrates the deployment of Qwen3.6-35B-A3B on Ascend NPUs using SGLang, including single-node PD mixed
mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3.6-35B-A3B) is fully supported
in this version. To use the latest features (e.g., speculative decoding / NEXTN, multimodal), use **v0.5.16 or a later
version**.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 2` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 16 24 32 48 64 80 96 112 120` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version, 39.81GB)

> [!NOTE]
>
> The W8A8 variant (39.81GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on A3 Series.

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3.6-35B-A3B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

For multimodal requests (text + image):

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Qwen3.6-35B-A3B-w8a8",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}},
                    {"type": "text", "text": "Describe this image."}
                ]
            }
        ]
    }'
```

Expected result: an HTTP 200 response with a description of the image.

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3.6-35B-A3B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3.8-Max
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max



## Introduction

Qwen3.8-Max (published as Qwen3.8-2.4T-A95B) is a Mixture-of-Experts (MoE) model with 2.4 trillion total
parameters and 95B activated per token. It uses a 92-layer hybrid architecture that combines Gated Delta
Network (GDN) linear-attention with full-attention layers. The model has a native context length of 262,144
tokens (extensible to over 1 million) and built-in multi-token prediction (MTP) weights for speculative
decoding.

This document demonstrates the deployment of Qwen3.8-Max on Ascend NPUs using SGLang, including multi-node PD
mixed mode, feature configuration, and performance optimization.

Qwen3.8-Max is newly released. This document is validated and written based on the **SGLang main branch (daily
build)**. It is recommended to use the latest daily build Docker image, or build SGLang from the main branch
source.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 64` |
| DP Attention | `--dp-size 4 \` `--enable-dp-attention` |
| Expert Parallelism | `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Qwen/Qwen3.8-2.4T-A95B](https://www.modelscope.cn/models/Qwen/Qwen3.8-2.4T-A95B) (BF16, approximately 4.8TB)
* [Eco-Tech/Qwen3.8-2.4T-A95B-w4a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.8-2.4T-A95B-w4a8) (W4A8 quantized version, approximately 1.33TB)
* You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantize `Qwen3.8-2.4T-A95B-w4a8` from `Qwen3.8-2.4T-A95B`.

> [!NOTE]
>
> We recommend deploying the W4A8 variant for reduced resource usage and higher throughput.
> The validated configuration in this tutorial deploys the W4A8 variant (approximately 1.33TB) on 4 Ascend A3 Series
> nodes (`--tp-size 64`, 64 dies in total). The BF16 checkpoint alone weighs approximately 4.8TB and requires additional nodes.

For the hardware specifications (memory per die, dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

The following command is based on the daily build tag, which contains the latest SGLang main branch changes.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Multi-node online deployment

Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping
prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single
node can provide. The validated configuration deploys the W4A8 quantized checkpoint on 4 nodes with `--tp-size 64`
(16 dies per node), DP attention (`--dp-size 4` + `--enable-dp-attention`), and DeepEP in `auto` mode.

Modify the IP addresses of the four nodes, then run the same script on all four nodes. Each node determines its own
rank by matching the local IP address against `IPS`.

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   IPS: IP addresses of each node in the cluster
#   MODEL_PATH: path to the model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=10
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_ONE_VISIBLE_DEVICE_PER_PROCESS=1

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32

# deepep
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
export HCCL_BUFFSIZE=2300
export DEEPEP_NORMAL_LONG_SEQ_ROUND=64
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512
export HCCL_OP_EXPANSION_MODE=AIV

# Run command ifconfig on each node, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here
export HCCL_SOCKET_IFNAME=<network-interface>
export GLOO_SOCKET_IFNAME=<network-interface>

IPS=('<your node1 ip>' '<your node2 ip>' '<your node3 ip>' '<your node4 ip>')
IP_MASTER="${IPS[0]}:5000"

MODEL_PATH=/path/to/model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
for i in "${!IPS[@]}";
do
    if [[ "$LOCAL_HOST1" == "${IPS[$i]}" || "$LOCAL_HOST2" == "${IPS[$i]}" ]];
    then
        echo "${IPS[$i]}"

        # overlap schedule
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_RAGGED_VERIFY_MODE=static

        sglang serve \
            --model-loader-extra-config '{"enable_multithread_load": true}' \
            --dist-init-addr $IP_MASTER --nnodes 4 --node-rank $i \
            --model-path $MODEL_PATH \
            --tokenizer-path $MODEL_PATH \
            --trust-remote-code \
            --attention-backend ascend \
            --device npu \
            --quantization modelslim \
            --dtype bfloat16 \
            --tp-size 64 \
            --dp-size 4 --enable-dp-attention --enable-dp-lm-head \
            --mem-fraction-static 0.8 \
            --chunked-prefill-size 8192 \
            --cuda-graph-bs-decode 16 \
            --disable-radix-cache \
            --max-running-requests 64 \
            --host 0.0.0.0 \
            --port 30000 \
            --moe-a2a-backend deepep \
            --deepep-mode auto \
            --watchdog-timeout 9000
        break
    fi
done
```

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 30000)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

Qwen3.8-Max has no standalone best practice page yet, as tuning parameters are not finalized. Instead, the
[Online service deployment](#online-service-deployment) section above provides a ready-to-use script for multi-node
deployment, which embeds the recommended feature combinations and tuning parameters (e.g., DP attention, DeepEP,
overlap schedule).

For the full catalog of optimization features and their parameter and compatibility details,
see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3-8B
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b



## Introduction

Qwen3-8B is a compact dense model in the Qwen3 series developed by Alibaba, featuring 8B parameters with
Grouped-Query Attention (GQA) and up to 128k context length. It delivers significant improvements in instruction
following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports
EAGLE3 speculative decoding for accelerated inference and is available in both standard and thinking/reasoning-enhanced
editions.

This document demonstrates the deployment of Qwen3-8B on Ascend NPUs using SGLang, including single-node PD mixed mode,
feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-8B) is fully supported in
this version. To use the latest features (e.g., speculative decoding), it is recommended to use v0.5.16 or a later
version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 2` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value; disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 6 9 10 15 16` |
| Speculative Decoding | `--speculative-algorithm EAGLE3 \` `--speculative-draft-model-path /path/to/draft-model-weights \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Qwen3-8B](https://www.modelscope.cn/models/Qwen/Qwen3-8B) (BF16, 16.40GB)
* [Qwen3-8B-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-8B-w8a8) (W8A8 quantized version, 11.27GB)
* [Eagle3-Qwen3-8B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-8B-zh) (EAGLE3 draft model for speculative decoding)

> [!NOTE]
>
> We recommend deploying the W8A8 variant for reduced resource usage and higher throughput.
> It (11.27GB) fits within a single 64GB die, so `--tp-size 1` is sufficient on A3 Series.

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3-8B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3-8B Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Qwen3-Next-80B-A3B-Instruct
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct



## Introduction

Qwen3-Next-80B-A3B-Instruct is a hybrid Gated DeltaNet-Transformer Mixture-of-Experts (MoE) model developed by Alibaba,
featuring 80B total parameters with 3B active parameters. It combines Gated DeltaNet SSM layers with attention layers in a
sparse MoE architecture, enabling efficient long-sequence modeling with linear complexity in the SSM layers. The model
supports multi-token prediction (NEXTN) for speculative decoding, delivering strong performance in instruction
following, reasoning, and text generation tasks.

This document demonstrates the deployment of Qwen3-Next-80B-A3B-Instruct on Ascend NPUs using SGLang, including
single-node PD mixed mode, feature configuration, and performance optimization.

This document is validated and written based on **SGLang v0.5.16**. The current model (Qwen3-Next-80B-A3B-Instruct) is
fully supported in this version. To use the latest features (e.g., speculative decoding), it is recommended to use
v0.5.16 or a later version.

## Supported features

| Feature | Example usage |
| - | - |
| Tensor Parallelism | `--tp-size 4` |
| Data Parallelism | `--dp-size 2` |
| Expert Parallelism | `--ep-size 4 \` `--moe-a2a-backend deepep \` `--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| NPU Graph | enabled by default; disable with `--disable-cuda-graph`; control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 8` |
| Speculative Decoding | `--speculative-algorithm NEXTN \` `--speculative-num-steps 3 \` `--speculative-eagle-topk 1 \` `--speculative-num-draft-tokens 4 \` `--speculative-draft-model-quantization unquant \` `--speculative-draft-model-path /path/to/draft-model-weights` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |

> [!NOTE]
>
> The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
> mode, and workload. For parameter details, see
> [Feature descriptions](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-descriptions); for
> recommended configurations for each deployment scenario, see [Best practices](#best-practices).

For feature compatibility and conflict information between features,
see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning#feature-compatibility).

## Prerequisites

### Environment

Before following this tutorial, complete the environment setup in the documents below:

* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — the fastest way to get started.
  It walks you through launching the official container image, starting the SGLang server, and sending a test request.
  Recommended if you are new to SGLang on Ascend.
* [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/getting-started/installation) — the full installation guide.
  It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a
  Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize
  the environment instead of using the official image.

### Model weights

Before downloading model weights, check the model size to reserve enough disk space.
For multi-node deployment, download the weights to a shared directory accessible to all nodes.

* [Qwen3-Next-80B-A3B-Instruct](https://www.modelscope.cn/models/Qwen/Qwen3-Next-80B-A3B-Instruct) (BF16, 162.68GB) — used as the draft model for NEXTN speculative decoding
* [Qwen3-Next-80B-A3B-Instruct-W8A8](https://www.modelscope.cn/models/vllm-ascend/Qwen3-Next-80B-A3B-Instruct-W8A8) (W8A8 quantized version, 84.90GB)

> [!NOTE]
>
> We recommend deploying the W8A8 variant as the main model for reduced resource usage and higher throughput.
> It (84.90GB) can be deployed on 2 × 64GB (`--tp-size 2`), which corresponds to 1 card (2 dies) on A3 Series.
> The BF16 weights serve as the NEXTN draft model (set `--speculative-draft-model-path` to the BF16 weight path).

This is the minimum recommended configuration. For optimized configurations,
see [Best practices](#best-practices), which may require additional cards or nodes.

For the hardware specifications (memory per die and dies per card),
see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware).

## Installation

> [!WARNING]
>
> Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the
online platform. You can directly pull it.

Both **stable releases** and **daily builds** are available. CANN 9.1.0 has no stable image yet, so the following
command uses the daily build tag `main-cann9.1.0-a3`. Once the stable image is published, switch back to the stable
release tag.
For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build).

```bash Command theme={null}
# TODO: CANN 9.1.0 has no stable image yet, so this temporarily pulls the
# daily build tag. Once the stable image is published, switch back to the
# stable release tag.
docker pull quay.io/ascend/sglang:main-cann9.1.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:main-cann9.1.0-a3
```

> [!TIP]
>
> * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the
>   container, for example: `-v /path/to/models:/models`.
> * Replace `${NAME}` with your own container name or remove `--name` to use default name.

## Online service deployment

### Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios
with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized
deployment commands and benchmark data, see
[Qwen3-Next-80B-A3B-Instruct Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct#single-node-pd-mixed).

## Functional verification

After the service is started, you can invoke the model by sending a prompt:

```shell theme={null}
# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6688)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'
```

Expected result: an HTTP 200 response with the generated text containing "Paris".

Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more
testing examples (Health Check, Generate, Chat Completions, and port usage guidance),
see [Testing the Service](/docs/hardware-platforms/ascend-npus/getting-started/installation#testing-the-service).

## Accuracy evaluation

For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation).

## Performance

For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing).

## Best practices

### Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the
[Qwen3-Next-80B-A3B-Instruct Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct) page.

## Performance tuning

For the full list of supported features, see [Supported features](#supported-features). For detailed optimization
guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning).

## FAQ

For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/faq).


# Parameter Tuning
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning



This guide explains the role of each parameter used in SGLang deployments on Ascend NPU. It uses
the [DeepSeek-V3.2 best practice configuration](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation)
as the reference example. For a complete list of tested deployment configurations, see the
[Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) page.

> [!NOTE]
>
> Parameters in this guide fall into two categories:
>
> * **Required configurations** (marked with `[Required]`): These must be set correctly for the target deployment scenario (e.g., multi-node communication, PD disaggregation). Incorrect values will cause deployment failures or incorrect behavior.
> * **Performance optimizations**: These improve throughput, latency, or memory efficiency. The optimal values depend on your specific model, hardware, and workload and may require tuning. Where the optimal value is not obvious, tuning guidance is provided.

## System-Level Optimizations

The following system-level tuning steps reduce OS interference and improve CPU scheduling determinism:

<table>
  <thead>
    <tr>
      <th>Command / Variable</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`</td>
      <td>Locks all CPU cores to the maximum frequency, eliminating DVFS-induced latency jitter during inference-critical paths.</td>
    </tr>

    <tr>
      <td>`sysctl -w vm.swappiness=0`</td>
      <td>Minimizes kernel swapping of anonymous pages. Reduces the risk of page faults on NPU memory buffers pinned to host RAM.</td>
    </tr>

    <tr>
      <td>`sysctl -w kernel.numa_balancing=0`</td>
      <td>Disables automatic NUMA page migration. Prevents the kernel from moving memory pages between NUMA nodes while inference is running, which would cause latency spikes.</td>
    </tr>

    <tr>
      <td>`sysctl -w kernel.sched_migration_cost_ns=50000`</td>
      <td>Sets a minimum task migration cost, discouraging the scheduler from moving inference threads between CPU cores unnecessarily.</td>
    </tr>

    <tr>
      <td>`SGLANG_SET_CPU_AFFINITY=1`</td>
      <td>Binds SGLang worker processes to specific CPU cores, avoiding cross-core migration overhead for high-frequency scheduling loops.</td>
    </tr>
  </tbody>
</table>

## Memory & Device Configuration

<table>
  <thead>
    <tr>
      <th>Variable / Argument</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`PYTORCH_NPU_ALLOC_CONF=expandable_segments:True`</td>
      <td>Enables the expandable NPU memory allocator, allowing the memory pool to grow dynamically. This avoids out-of-memory errors when workloads have variable memory requirements and is essential for large models such as MoE architectures.</td>
      <td>`expandable_segments:True`</td>
    </tr>

    <tr>
      <td>`PYTORCH_NPU_ALLOC_CONF=expandable_segments:True,pin_memory_expandable_segments:True`</td>
      <td>Additionally extends the expandable allocator to pinned (page-locked) host memory, reducing fragmentation and mitigating intermittent pinned host memory allocation failures. Consider appending `pin_memory_expandable_segments:True` when the workload allocates large pinned host buffers — e.g., large hierarchical cache (HiCache) host pools or KV cache offload — and host memory allocation failures are observed. This option changes memory usage and may introduce performance fluctuations; validate on your own workload before adopting.</td>
      <td>`expandable_segments:True,pin_memory_expandable_segments:True`</td>
    </tr>

    <tr>
      <td>`STREAMS_PER_DEVICE=32`</td>
      <td>Sets the maximum number of parallel streams per NPU device. More streams allow better overlap between compute and communication operations. The default of `32` is sufficient for most deployments; increase only if profiling reveals stream contention in complex pipeline parallelism setups.</td>
      <td>`32`</td>
    </tr>

    <tr>
      <td>`--mem-fraction-static`</td>
      <td>Controls the fraction of NPU memory allocated to model weights and the KV cache pool. Lower values leave headroom for intermediate activations; higher values maximize KV cache capacity for serving more concurrent requests. The optimal value depends on your model size, sequence length, and available NPU memory. Start conservatively and increase gradually while monitoring for out-of-memory errors.</td>
      <td>Prefill: `0.73`, Decode: `0.79`</td>
    </tr>
  </tbody>
</table>

## Communication Configuration

<table>
  <thead>
    <tr>
      <th>Variable</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`DEEPEP_HCCL_BUFFSIZE`</td>
      <td>Sets the HCCL communication buffer size in MB for DeepEP MoE dispatch/combine groups. Larger buffers increase throughput for bulk transfers but consume more host memory. The optimal value depends on your communication pattern — larger values benefit prefill (bulk token transfers), while smaller values are sufficient for decode (small batches). Falls back to `HCCL_BUFFSIZE` (CANN's global HCCL buffer variable) when unset.</td>
      <td>Prefill: `1200`, Decode: `400`</td>
    </tr>

    <tr>
      <td>`HCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME`</td>
      <td>**\[Required for multi-node]** Specifies the network interface used for HCCL and GLOO distributed communication. Must be set to the high-bandwidth inter-node network interface (e.g., RDMA-capable NIC) for multi-node deployments. Without this, the framework may default to a low-bandwidth interface, severely degrading distributed communication performance. Not needed for single-node deployments.</td>
      <td>Set per-cluster</td>
    </tr>
  </tbody>
</table>

## MoE & Expert Parallelism

<table>
  <thead>
    <tr>
      <th>Variable / Argument</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--moe-a2a-backend`</td>
      <td>Selects the all-to-all communication backend for MoE expert dispatch and combine. On Ascend NPU, the primary options are `deepep` (DeepEP) and `ascend_fuseep` (Ascend Fused EP). DeepEP is optimized for large-scale models with flexible prefill/decode dispatch paths; `ascend_fuseep` provides a general fused MoE dispatch path.</td>
      <td>`deepep`</td>
    </tr>

    <tr>
      <td>`--deepep-mode`</td>
      <td>Selects the DeepEP operating mode. Available options: `normal` (optimized for high throughput, long sequences, and large token counts — suitable for prefill), `low_latency` (optimized for low latency, CUDA Graph compatible, small batches — suitable for decode), and `auto` (switches automatically based on the operation type). Use `auto` if unsure; use explicit modes when managing prefill/decode independently in PD disaggregation.</td>
      <td>Prefill: `normal`, Decode: `low_latency`</td>
    </tr>

    <tr>
      <td>`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
      <td>Sets the per-rank dispatch buffer capacity (in tokens) for DeepEP low-latency mode, bounding how many tokens one rank can dispatch per decode iteration. Hard upper limit: `1024`. Larger values allocate larger buffers. Size it to the tokens actually dispatched per iteration: `max-running-requests * --speculative-num-draft-tokens` with speculative decoding (just `max-running-requests` without it) — exceeding the buffer fails at runtime inside DeepEP. Not needed for prefill: normal-mode dispatch is dynamic and does not use this value.</td>
      <td>Decode: `8`</td>
    </tr>

    <tr>
      <td>`DEEP_NORMAL_MODE_USE_INT8_QUANT`</td>
      <td>When set to `1`, quantizes intermediate activations to INT8 in the DeepEP dispatch operator, reducing communication volume during MoE dispatch. This is most beneficial for large-scale multi-node prefill with many tokens. The trade-off is a small accuracy impact from quantization and additional compute for the quantize/dequantize operations.</td>
      <td>Prefill: `1`</td>
    </tr>

    <tr>
      <td>`TASK_QUEUE_ENABLE`</td>
      <td>Controls the ASCEND Runtime task queue optimization level: `0` = disabled, `1` = default optimization, `2` = aggressive optimization with greater task fusion and overlap. Higher levels improve throughput but may interfere with NPU Graph-launched tasks. Start with `1` for general use. Use `2` for throughput-critical prefill workloads; use `0` for decode where NPU Graph compatibility is needed.</td>
      <td>Prefill: `2`, Decode: `0`</td>
    </tr>

    <tr>
      <td>`--moe-dense-tp-size`</td>
      <td>Sets the tensor parallelism size for MoE dense (shared) MLP layers. When using DP attention, setting this to `1` avoids an unnecessary all-reduce across the DP group for the dense MLP layers, since each DP shard already has the full weight. In deployments without DP attention, set this to match your TP size.</td>
      <td>`1`</td>
    </tr>
  </tbody>
</table>

## Prefill Optimizations

These arguments and environment variables are critical for tuning prefill performance:

<table>
  <thead>
    <tr>
      <th>Argument / Variable</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--chunked-prefill-size`</td>
      <td>Sets the maximum number of tokens per prefill chunk. A positive value enables chunked prefill, which interleaves prefill and decode for better concurrency in mixed workloads. Set to `-1` to disable chunking and process each request in a single forward pass, which is preferred for dedicated prefill servers with long-context sequences.</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`--max-prefill-tokens`</td>
      <td>Limits the total number of tokens the prefill server can process in one batch. The effective bound is `max(this value, model_max_context_length)`. Set this based on your target sequence length and available NPU memory to bound memory usage while maximizing throughput. Tune by increasing until you encounter out-of-memory errors.</td>
      <td>`68000`</td>
    </tr>

    <tr>
      <td>`--max-running-requests`</td>
      <td>Limits the number of concurrent requests being processed. For prefill, a low value (e.g., `1`) dedicates more compute and memory to each request, achieving higher per-request throughput — ideal for dedicated prefill nodes processing long sequences. For general-purpose serving, use a higher value to support multi-request concurrency.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`--disable-radix-cache`</td>
      <td>Disables prefix caching via RadixAttention. Set this flag when processing non-overlapping long sequences where prefix caching provides no benefit and only consumes memory. Leave unset (radix cache enabled) for chat/conversation workloads with shared system prompts.</td>
      <td>true</td>
    </tr>

    <tr>
      <td>`--disable-cuda-graph`</td>
      <td>Disables CUDA Graph capture. CUDA Graphs reduce kernel launch overhead for small, predictable batch sizes, making them ideal for decode. For prefill with large and variable batch sizes, CUDA Graphs provide minimal benefit and can cause issues with dynamic shapes. Set this flag on prefill nodes; leave unset on decode nodes.</td>
      <td>true</td>
    </tr>

    <tr>
      <td>`--attn-cp-size`</td>
      <td>Specifies the context parallelism group size for attention computation. Larger values distribute the sequence across more ranks, reducing per-rank memory and compute at the cost of increased communication. For models with DSA, this controls the CP size for sparse attention prefill. Set to the number of available devices for maximum parallelization.</td>
      <td>`32`</td>
    </tr>
  </tbody>
</table>

## Decode Optimizations

These arguments and environment variables are critical for tuning decode performance:

<table>
  <thead>
    <tr>
      <th>Argument / Variable</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--attn-dp-size` / `--attention-data-parallel-size`</td>
      <td>Sets the attention DP size for the decode server: the number of data-parallel attention groups inside the TP group. This is a key optimization for decode throughput — attention is DP-sharded to reduce KV cache duplication, while FFN/MoE layers remain TP-sharded to leverage expert parallelism. Best suited for MoE models where attention is not the compute bottleneck. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides the TP size evenly.</td>
      <td>`8`</td>
    </tr>

    <tr>
      <td>`--ep` / `--ep-size` / `--expert-parallel-size`</td>
      <td>Sets the expert parallelism degree. For MoE models, this distributes experts across cards, reducing per-card expert loading overhead and enabling all-to-all dispatch. The code default is `1`; set explicitly for MoE models. The optimal value depends on your model's expert count and architecture. DeepSeek V3.2 with 256 routed experts uses `ep=32`. For models with fewer experts, use a proportionally smaller value.</td>
      <td>`32`</td>
    </tr>

    <tr>
      <td>`--enable-dp-lm-head`</td>
      <td>Enables vocabulary parallelism across the DP attention group, sharding the LM head weight across ranks. Each rank only computes logits for its vocabulary shard, avoiding a costly all-gather of logits across the DP group. This is essential when DP attention is enabled to maintain throughput.</td>
      <td>Enabled</td>
    </tr>

    <tr>
      <td>`--cuda-graph-max-bs-decode`</td>
      <td>Caps the maximum batch size for which CUDA Graphs are captured. Larger values cover more batch sizes but increase graph capture time and memory overhead. If your `max-running-requests` is high but typical batch sizes are lower, use a smaller value to reduce capture overhead. Tune based on your observed batch size distribution during serving.</td>
      <td>`4`</td>
    </tr>

    <tr>
      <td>`SGLANG_SCHEDULER_SKIP_ALL_GATHER=1`</td>
      <td>When DP attention is enabled, the scheduler normally performs an all-gather across DP ranks to determine the full set of ready requests. Setting this to `1` skips that operation, reducing decode scheduling latency. Only safe when load is balanced across DP ranks (e.g., via a round-robin load balancing policy). Disable if you observe uneven load distribution across DP ranks.</td>
      <td>`1`</td>
    </tr>
  </tbody>
</table>

## Speculative Decoding

Speculative decoding reduces per-token latency by generating draft tokens that are then verified by the target model:

<table>
  <thead>
    <tr>
      <th>Argument / Variable</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-algorithm`</td>
      <td>Selects the speculative decoding algorithm. `NEXTN` (aliased to `EAGLE`) uses the model's built-in MTP (Multi-Token Prediction) heads, requiring no separate draft model. `EAGLE3` uses an external draft model, which can achieve higher acceptance rates at the cost of additional memory. Other built-in options include `STANDALONE`, `NGRAM`, and `DFLASH`, plus any plugin-registered name via `SpeculativeAlgorithm.register`. Choose `NEXTN` for models with native MTP support (e.g., DeepSeek V3.2/R1); choose `EAGLE3` for models without MTP (e.g., Qwen MoE).</td>
      <td>`NEXTN`</td>
    </tr>

    <tr>
      <td>`--speculative-num-steps`</td>
      <td>Number of speculative forward passes per iteration. More steps can increase the acceptance length and throughput but add latency. For prefill, use a small value (`1`) to minimize prefill latency impact. For decode, use a larger value (`2`–`4`) to maximize throughput. Tune based on your latency vs throughput requirements.</td>
      <td>Prefill: `1`, Decode: `3`</td>
    </tr>

    <tr>
      <td>`--speculative-eagle-topk`</td>
      <td>Limits the number of draft tokens considered per position. Lower values reduce compute on unlikely tokens and are required for the overlap scheduler (enabled by default). Higher values may increase acceptance rates but add overhead. Start with `1` for the overlap scheduler; otherwise, `4`–`8` is typical.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>Number of draft tokens generated per speculative step. Higher values increase potential acceptance length and throughput but add per-step computation. Balance against your latency budget — prefill typically uses fewer draft tokens (`2`) to minimize overhead; decode can use more (`4`) to maximize throughput.</td>
      <td>Prefill: `2`, Decode: `4`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1`</td>
      <td>Enables a dedicated plan stream for spec-v2 speculative decoding workers (EAGLE/NEXTN, EAGLE3, STANDALONE, DFLASH, etc.). Batch-preparation kernels such as draft-extend metadata and KV-location planning run on this separate stream, overlapping scheduling prep overhead with other enqueued work instead of serializing it. Incompatible with LoRA.</td>
      <td>`1`</td>
    </tr>
  </tbody>
</table>

## Quantization

<table>
  <thead>
    <tr>
      <th>Argument</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--quantization modelslim`</td>
      <td>Uses the Ascend ModelSlim quantization tool to load W4A4/W8A8/W4A8 pre-quantized model weights (plus MXFP8/MXFP4 variants). This reduces model weight footprint by approximately 50% (for w8) or 75% (for w4) compared to BF16, allowing larger models to fit in NPU memory with minimal accuracy degradation. The quantization method is auto-detected from the model's `quant_model_description.json` file.</td>
      <td>`modelslim`</td>
    </tr>
  </tbody>
</table>

## Throughput Configuration

<table>
  <thead>
    <tr>
      <th>Argument / Variable</th>
      <th>Purpose</th>
      <th>Reference Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--tokenizer-worker-num`</td>
      <td>Sets the number of parallel tokenizer worker processes. Increasing this allows concurrent tokenization of multiple input/output streams, preventing the tokenizer from becoming a bottleneck under high request concurrency. Set based on your CPU core count and expected request rate.</td>
      <td>`4`</td>
    </tr>

    <tr>
      <td>`--load-balance-method`</td>
      <td>Selects the DP load balancing strategy. Available options: `auto` (default, automatically selects the best strategy), `round_robin` (assigns requests to DP ranks in rotation for even distribution), `total_tokens` (balances by token load), `total_requests` (balances by request count), `follow_bootstrap_room` (follows the bootstrap room assignment). Start with `round_robin` for simple even distribution; use `total_tokens` if your requests have highly variable lengths.</td>
      <td>`round_robin`</td>
    </tr>

    <tr>
      <td>`ASCEND_MF_STORE_URL`</td>
      <td>**\[Required for PD disaggregation]** Sets the MemFabric config store address for PD disaggregation. This is the prefill primary node's IP with an arbitrary port, used by the decode nodes to discover and connect to the MemFabric-based KV cache transfer service. Omit this for non-disaggregated deployments.</td>
      <td>Prefill IP with port</td>
    </tr>
  </tbody>
</table>

## Additional Ascend NPU-Specific Parameters

The following environment variables are used in other best practice configurations and may be applicable depending on your model and deployment:

<table>
  <thead>
    <tr>
      <th>Variable</th>
      <th>Purpose</th>
      <th>Typical Usage</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`HCCL_OP_EXPANSION_MODE=AIV`</td>
      <td>Sets where HCCL communication algorithms are orchestrated and executed: `AIV` places them on the device's AI Vector cores, offloading communication from the AI Core compute units so it can overlap with compute (the alternative is `AI_CPU`). Recommended for PD-hybrid deployments in CANN's tuning guide; verify the benefit for your workload, as gains are workload-dependent.</td>
      <td>Used in Qwen MoE, DeepSeek R1, Kimi K2.6, and MiMo configurations</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_USE_MLAPO=1`</td>
      <td>**(DeepSeek MLA-specific)** Adopts the `MLAPO` fusion operator in the MLA (Multi-Head Latent Attention) preprocessing stage for DeepSeek models with MLA architecture.</td>
      <td>Used with DeepSeek R1</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_FIA_NZ=1`</td>
      <td>**(DeepSeek MLA-specific)** Reshapes the KV Cache into FIA NZ format for improved memory access efficiency. Must be used together with `SGLANG_NPU_USE_MLAPO=1`.</td>
      <td>Used with DeepSeek R1</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_USE_MULTI_STREAM=1`</td>
      <td>**(DeepSeek MoE-specific)** Enables dual-stream computation for shared experts and routing experts in DeepSeek MoE models, allowing the two expert types to execute concurrently on separate streams.</td>
      <td>Used with DeepSeek R1</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_AG_AFTER_QLORA=1`</td>
      <td>Delays the all-gather operation until after Q-LoRA processing. This reduces communication overhead by performing Q-LoRA projection before the all-gather, requiring fewer bytes to be transferred.</td>
      <td>Used with DeepSeek V3.2/R1 prefill</td>
    </tr>
  </tbody>
</table>

## Feature Compatibility

The table shows feature-by-feature compatibility of SGLang capabilities on Ascend NPU. Features that have performance impact on Ascend are included. Each cell at the intersection of two features indicates whether they can be used together.

The symbols used are defined as follows:

* 🟢 = Full compatibility
* 🟠 = Partial compatibility
* ❌ = No compatibility
* ❔ = Unknown or TBD

<table>
  <thead>
    <tr>
      <th>Feature</th>
      <th>Tensor Parallelism</th>
      <th>Data Parallelism</th>
      <th>Expert Parallelism</th>
      <th>Context Parallelism</th>
      <th>PD Disaggregation</th>
      <th>Quantization</th>
      <th>Chunked Prefill</th>
      <th>NPU Graph</th>
      <th>Speculative Decoding</th>
      <th>PrefixCache</th>
      <th>Overlap Schedule</th>
      <th>DP LM Head</th>
      <th>MLAPO</th>
      <th>Multistream MoE</th>
      <th>EPLB</th>
      <th>NZ Weight Format</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Tensor Parallelism</td>

      <td />

      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>Data Parallelism</td>
      <td>🟢</td>

      <td />

      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>Expert Parallelism</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>Context Parallelism</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>

      <td />

      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>❔</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>PD Disaggregation</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>Quantization</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>Chunked Prefill</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>

      <td />

      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>NPU Graph</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>

      <td />

      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
    </tr>

    <tr>
      <td>Speculative Decoding</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>

      <td />

      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>PrefixCache</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>Overlap Schedule</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>DP LM Head</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>❔</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>MLAPO</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>🟢</td>
      <td>❔</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>Multistream MoE</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>

      <td />

      <td>❔</td>
      <td>🟢</td>
    </tr>

    <tr>
      <td>EPLB</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>❔</td>
      <td>❔</td>

      <td />

      <td>🟠</td>
    </tr>

    <tr>
      <td>NZ Weight Format</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟢</td>
      <td>🟠</td>

      <td />
    </tr>
  </tbody>
</table>

## Feature descriptions

### Tensor Parallelism (`--tp-size`)

Splits model weights across multiple NPU devices so that large models can be loaded and run cooperatively. Each device holds a shard of every weight tensor and communicates via HCCL all-reduce. This is the primary mechanism for deploying models that exceed single-device memory.

### Data Parallelism (`--dp-size`, `--attn-dp-size`)

`--dp-size` replicates the model across independent device groups to increase throughput by processing multiple requests in parallel. `--attn-dp-size` instead runs attention data parallel inside the TP group: the attention layers are replicated across attention DP ranks while the FFN/MoE layers remain tensor-parallel/expert-parallel. This reduces the communication overhead of attention layers in TP (which is significant during small-batch decode) while keeping the heavy compute and memory demands of MoE layers distributed. Model replicas (`--dp-size` > 1) cannot be combined with attention DP.

### Expert Parallelism (`--ep-size`)

Each device holds a subset of experts and routes tokens via all-to-all communication based on the model's gating network. Supports load balancing methods including default round-robin expert placement and dynamic Expert Parallelism Load Balancing (EPLB).

### Context Parallelism (`--attn-cp-size`)

Splits long input sequences across devices so that the KV cache and attention computation for a single request are distributed. This enables serving very long context lengths (e.g., 128k tokens) that exceed single-device memory. On Ascend, `--attn-cp-size` must equal `--tp-size`.

### PD Disaggregation (`--disaggregation-mode`)

Separates the prefill (P) and decode (D) phases onto different device groups, allowing each phase to be independently optimized for its compute and memory characteristics. Prefill nodes handle long input processing at high throughput, while decode nodes focus on low-latency token generation. Uses the Ascend MemFabric transfer backend (`--disaggregation-transfer-backend ascend`).

### Quantization (`--quantization`)

Reduces model weight and activation precision (e.g., W8A8, W4A8, W4A16) to decrease memory usage and increase throughput.
On Ascend, the natively supported and highly optimized quantization method is ModelSlim (supporting W4A4, W8A8, W4A8 dynamic/static, plus MXFP8/MXFP4 variants).
Support for other community formats (such as AWQ, GPTQ, Auto-round, and Compressed-tensors — the source of W4A16 support) depends on the availability
of specific Ascend custom kernels in your environment;
see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/optimization/quantization) for details and compatibility matrices.

### Chunked Prefill (`--chunked-prefill-size`)

Breaks large prefill computations into smaller, fixed-size chunks for better scheduling interleaving with decode batches. This prevents long prefill requests from blocking decode latency. Use `-1` to disable chunked prefill on dedicated prefill nodes. Has partial compatibility with most features because chunk boundaries introduce scheduling complexity.

### NPU Graph (`--cuda-graph-bs-decode`, `--cuda-graph-max-bs-decode`)

Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes: capture an explicit batch-size list via `--cuda-graph-bs-decode`, or let sizes be bucketed automatically up to `--cuda-graph-max-bs-decode`. Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes.

### Speculative Decoding (`--speculative-algorithm`)

Reduces per-token latency by predicting multiple future tokens in a single forward pass, then verifying them against the model. Ascend supports `NEXTN` (for DeepSeek models, using the model's own hidden states) and `EAGLE3` (for Qwen MoE models, using a separate draft model). Controlled by `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. On Ascend, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` moves speculative-decoding batch-preparation kernels to a dedicated stream (see the Speculative Decoding table above); it is incompatible with LoRA.

For non-greedy requests, Ascend target verification applies request-level `temperature`, `top_k`, and `top_p`. The default target-only mode supports linear and tree-shaped candidates. Classic rejection sampling is available with `--speculative-use-rejection-sampling` for linear chains configured with `--speculative-eagle-topk 1`.

### PrefixCache (`--disable-radix-cache`)

Reuses KV cache across requests that share common prompt prefixes (Radix Cache), reducing repeated prefill computation and lowering time-to-first-token. Enabled by default; disable with `--disable-radix-cache` when prefix reuse is not expected (e.g., dedicated prefill nodes in PD disaggregation, or random-input benchmarks).

### Overlap Schedule

Overlaps CPU scheduling work with NPU compute so the next batch is prepared while the current one runs. Enabled by default; disable with `--disable-overlap-schedule`. Note that `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` is a separate, complementary optimization: it moves speculative-decoding batch-preparation kernels to a dedicated device stream (see the Speculative Decoding table above).

### DP LM Head (`--enable-dp-lm-head`)

Shards the language model head (the final vocabulary projection layer) across DP ranks to reduce memory consumption and improve decode throughput. Without this, each DP rank holds a full copy of the LM head, which is wasteful for large vocabularies. Typically used together with `--attn-dp-size`.

### MLAPO (`SGLANG_NPU_USE_MLAPO=1`)

Enables the MLAPO (MLA Pre-Processing Optimization) fusion operator for DeepSeek-style Multi-head Latent Attention models, replacing multiple separate kernels with a single fused kernel for Q/K/V projection and absorption. Reduces prefill latency and memory bandwidth usage. Used in most DeepSeek best-practice configurations.

### Multistream MoE (`SGLANG_NPU_USE_MULTI_STREAM=1`)

Enables dual-stream parallel execution for MoE layers, where shared experts and routing experts run concurrently on separate NPU streams. This overlaps shared-expert computation with routed-expert dispatch, improving MoE layer throughput.

### EPLB (`--enable-eplb`)

Enables Expert Parallelism Load Balancing to dynamically redistribute experts across devices based on workload, ensuring even expert utilization and preventing hotspots. Uses the `deepseek` balancing algorithm. Compatible with the `deepep` MoE A2A backend, but not with `ascend_fuseep`.

### NZ Weight Format (`SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT`)

Casts model weight tensors to the Ascend NPU ACL FRACTAL\_NZ format (format 29) for improved memory access efficiency on Da Vinci AI cores. Enabled by default globally; disable with `SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT=1` when the format conflicts with other features (e.g., EPLB). Applied to linear weights, MoE weights, quantized weights, MLA QKV projections, and KV Cache (via `PA_NZ` / `nzcache` modes for FIA). Related env var: `SGLANG_USE_FIA_NZ=1` (requires `SGLANG_NPU_USE_MLAPO=1`).

## See Also

* [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) — Complete deployment configurations and benchmark results for all supported models
* [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/reference/environment_variables) — Reference for all Ascend NPU-related environment variables
* [DeepSeek V3.2 Guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) — Detailed usage guide for DeepSeek V3.2 deployment
* [Expert Parallelism](/docs/advanced_features/expert_parallelism) — DeepEP configuration and tuning guide


# Performance Profiling
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/profiling



During inference serving, it is sometimes necessary to monitor the internal
execution flow of the serving framework to identify performance issues. By
collecting start/end timestamps of key flows, identifying critical functions or
iterations, recording key events, and gathering relevant information, you can
quickly locate performance bottlenecks.

This guide walks you through the complete workflow of collecting performance
data in an SGLang Ascend NPU inference service — from preparation, collection,
and analysis to visualization — helping you get started with performance
profiling quickly.

For more profiling scenarios (e.g., Nsight Systems, PD disaggregation, etc.),
see [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling).

## Ascend PyTorch Profiler

SGLang has built-in PyTorch Profiler support. Through the Ascend `torch_npu`
backend, you can directly collect NPU operator-level performance data. No
additional packages are required — profiling start/stop is controlled via API
requests.

### 1. Environment Setup

Launch an SGLang online service and set the `SGLANG_TORCH_PROFILER_DIR`
environment variable to control where performance files are saved. Once the
service starts, profiling is ready on standby.

```shell Command theme={null}
# Set the performance data output directory
export SGLANG_TORCH_PROFILER_DIR=./sglang_profile

# Start SGLang server (use local model path or HuggingFace model id)
sglang serve \
  --model-path /path/to/your/model \
  --attention-backend ascend \
  --host 0.0.0.0 --port 30000 \
  --tp-size 1 \
  --max-running-requests 128
```

**Profiling-related environment variables:**

<table>
  <thead>
    <tr>
      <th>Variable</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_TORCH_PROFILER_DIR`</td>
      <td>Trace file output directory</td>
      <td>`/tmp`</td>
    </tr>

    <tr>
      <td>`SGLANG_PROFILE_WITH_STACK`</td>
      <td>Record Python call stack (True / False)</td>
      <td>`True`</td>
    </tr>

    <tr>
      <td>`SGLANG_PROFILE_RECORD_SHAPES`</td>
      <td>Record operator input shapes (True / False)</td>
      <td>`True`</td>
    </tr>
  </tbody>
</table>

> [!NOTE]
>
> Priority order: **API parameter > Environment variable > Default value**.

### 2. Collection Methods

SGLang provides four collection methods. The core differences are **whether you
need to manually send `/start_profile` and `/stop_profile`**. All four methods
produce identical results — choose the most convenient one.

**Method comparison:**

<table>
  <thead>
    <tr>
      <th>Method</th>
      <th>Manual start\_profile</th>
      <th>Manual stop\_profile</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>A: API manual start/stop</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Maximum flexibility for precise control</td>
    </tr>

    <tr>
      <td>B: API auto-stop</td>
      <td>Yes</td>
      <td>No</td>
      <td>Set `num_steps`, auto-stops and generates output</td>
    </tr>

    <tr>
      <td>C: bench\_serving --profile</td>
      <td>No</td>
      <td>No</td>
      <td>Benchmark + profiling in one command</td>
    </tr>

    <tr>
      <td>D: sglang.profiler CLI</td>
      <td>No</td>
      <td>No</td>
      <td>Standalone profiling CLI tool</td>
    </tr>
  </tbody>
</table>

#### Method A: API Manual Start/Stop

Send `/start_profile` to start → send workload requests → send `/stop_profile`
to stop. After stopping, the server automatically parses the data — **no need to
manually call `analyse()`**.

```bash Command theme={null}
# Step 1: Start profiling (no num_steps, requires manual stop)
curl -X POST http://127.0.0.1:30000/start_profile \
  -H "Content-Type: application/json" \
  -d '{
    "output_dir": "./sglang_profile",
    "start_step": 1,
    "activities": ["CPU", "GPU"]
  }'

# Step 2: Send workload requests (using curl as example)
curl http://127.0.0.1:30000/generate \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 10}}'

# Step 3: Stop profiling
curl -X POST http://127.0.0.1:30000/stop_profile
```

> [!NOTE]
>
> `/stop_profile` returns `"Stop profiling. This will take some time."` — the
> server needs time to flush trace data to disk and parse it. Wait for the
> response to complete.
> This method takes a significant amount of time to parse
> profiling data; consider using **Method B** instead to avoid lengthy waits.

#### Method B: API Auto-Stop

Specify `num_steps` in the `/start_profile` request. Profiling stops
automatically after N steps and generates output — **no need to manually send
`/stop_profile`**.

```bash Command theme={null}
# num_steps=10, wait 3 warmup steps, auto-stop after 10 steps
curl -X POST http://127.0.0.1:30000/start_profile \
  -H "Content-Type: application/json" \
  -d '{
    "output_dir": "./sglang_profile",
    "start_step": 3,
    "num_steps": 10,
    "activities": ["CPU", "GPU"]
  }'

# Just send workload — no /stop_profile needed
curl http://127.0.0.1:30000/generate \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}'
```

#### Method C: bench\_serving --profile

Use SGLang's built-in `bench_serving` with the `--profile` flag.
**Automatically handles `/start_profile` and `/stop_profile`** — no manual API
calls needed.

```bash Command theme={null}
# With --profile-steps: auto-stops after N steps and generates output
python -m sglang.bench_serving \
  --backend sglang \
  --base-url http://127.0.0.1:30000 \
  --model /path/to/your/model \
  --tokenizer /path/to/your/model \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 100 \
  --num-prompts 10 \
  --profile \
  --profile-steps 10 \
  --profile-output-dir ./sglang_profile

# Without --profile-steps: /stop_profile sent automatically after benchmark
python -m sglang.bench_serving \
  --backend sglang \
  --base-url http://127.0.0.1:30000 \
  --model /path/to/your/model \
  --tokenizer /path/to/your/model \
  --dataset-name random \
  --random-input-len 1024 \
  --random-output-len 100 \
  --num-prompts 10 \
  --profile \
  --profile-output-dir ./sglang_profile
```

> [!NOTE]
>
> `--profile-steps N` sends `"num_steps": N` to the server's `/start_profile`, so
> the server auto-stops and parses data after N steps — bench\_serving skips
> sending `/stop_profile`.

> [!NOTE]
>
> `bench_serving --profile` creates a timestamp subdirectory inside
> `--profile-output-dir` (e.g., `<output_dir>/<timestamp>/`). The output path is
> shown in the server log as `Profiling done. Traces are saved to: <path>`.

> [!NOTE]
>
> On Ascend NPU, SGLang uses `torch_npu._apply_patches()` to automatically
> redirect PyTorch Profiler's CUDA activity to NPU, so
> `activities: ["CPU", "GPU"]` actually captures NPU operator events.

**`bench_serving --profile` parameters:**

<table>
  <thead>
    <tr><th>Parameter</th><th>Description</th></tr>
  </thead>

  <tbody>
    <tr><td>`--profile`</td><td>Enable auto profiling start/stop</td></tr>
    <tr><td>`--profile-steps N`</td><td>Auto-stop after N steps (skips /stop\_profile)</td></tr>
    <tr><td>`--profile-output-dir`</td><td>Trace output directory</td></tr>
  </tbody>
</table>

#### Method D: sglang.profiler CLI

Use the `sglang.profiler` CLI module, which automatically sends
`/start_profile` and waits for completion. **Start `sglang.profiler` first,
then send inference requests** (otherwise there are no steps to capture and the
profiler will wait indefinitely).

```bash Command theme={null}
# Terminal 1: Start sglang.profiler first (sends /start_profile, then waits for completion)
python3 -m sglang.profiler \
  --url http://127.0.0.1:30000 \
  --output-dir ./my_profiles \
  --num-steps 3 \
  --cpu --gpu &
```

```bash Command theme={null}
# Terminal 2: Wait for "Waiting for N steps" output from Terminal 1, then send requests.
# The profiler starts recording once /start_profile is received by the server.
# Requests sent before the server receives /start_profile will not be captured.
curl http://127.0.0.1:30000/generate \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 32}}'
```

A simpler and more reliable approach is to use `bench_serving --profile`, which
handles both steps automatically:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --base-url http://127.0.0.1:30000 \
  --model /path/to/your/model \
  --tokenizer /path/to/your/model \
  --dataset-name random \
  --random-input-len 128 \
  --random-output-len 32 \
  --num-prompts 10 \
  --profile \
  --profile-steps 3 \
  --profile-output-dir ./my_profiles
```

> [!NOTE]
>
> `sglang.profiler` is essentially a CLI wrapper around the `/start_profile` API.
> Advanced options like `--profile-by-stage` are also supported. On Ascend NPU,
> trace flushing is asynchronous and may take a while — the CLI may occasionally
> block waiting for flush. If it times out, use Method B (API auto-stop) or
> Method C (bench\_serving --profile) instead.

**`sglang.profiler` CLI parameters:**

<table>
  <thead>
    <tr><th>Parameter</th><th>Description</th></tr>
  </thead>

  <tbody>
    <tr><td>`--url`</td><td>SGLang server address</td></tr>

    <tr>
      <td>`--output-dir`</td>
      <td>Output directory (defaults to `SGLANG_TORCH_PROFILER_DIR`)</td>
    </tr>

    <tr><td>`--num-steps`</td><td>Number of steps to profile</td></tr>

    <tr>
      <td>`--profile-by-stage`</td>
      <td>Profile prefill / decode stages separately</td>
    </tr>

    <tr><td>`--profile-prefix`</td><td>Trace filename prefix</td></tr>

    <tr>
      <td>`--cpu` / `--gpu` / `--mem` / `--rpd`</td>
      <td>Activity types to collect</td>
    </tr>
  </tbody>
</table>

### 3. Full Parameter Reference

All methods ultimately send a `/start_profile` request to the server. The full
set of supported parameters:

<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`output_dir`</td>

      <td>
        Output directory. Falls back to
        `SGLANG_TORCH_PROFILER_DIR` or `/tmp`
      </td>

      <td>`/tmp`</td>
    </tr>

    <tr>
      <td>`num_steps`</td>

      <td>
        Number of steps. If set, profiling auto-stops — no /stop\_profile needed
      </td>

      <td>None</td>
    </tr>

    <tr>
      <td>`start_step`</td>

      <td>
        Step index to start profiling (inclusive), for skipping warmup
      </td>

      <td>0</td>
    </tr>

    <tr>
      <td>`activities`</td>

      <td>
        Activity types: CPU, GPU, MEM, RPD. On Ascend NPU, only CPU and GPU are supported.
        MEM depends on CUDA memory APIs and will be silently ignored. RPD requires ROCm and will cause an error.
      </td>

      <td>`["CPU", "GPU"]`</td>
    </tr>

    <tr>
      <td>`profile_by_stage`</td>
      <td>Profile prefill and decode stages separately</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`with_stack`</td>

      <td>
        Record Python call stack. Also controllable via
        `SGLANG_PROFILE_WITH_STACK`
      </td>

      <td>`true`</td>
    </tr>

    <tr>
      <td>`record_shapes`</td>

      <td>
        Record operator input shapes. Also controllable via
        `SGLANG_PROFILE_RECORD_SHAPES`
      </td>

      <td>`true`</td>
    </tr>

    <tr>
      <td>`profile_prefix`</td>
      <td>Prefix for trace filenames</td>
      <td>None</td>
    </tr>

    <tr>
      <td>`profile_stages`</td>

      <td>
        Stages to profile, e.g., `["prefill", "decode"]`.
        Requires `profile_by_stage`
      </td>

      <td>None</td>
    </tr>
  </tbody>
</table>

### 4. Finding Output Files

**The server log explicitly indicates where traces are saved.** You can find
them via:

* **When profiling starts**: server log outputs
  `Profiling starts. Traces will be saved to: <path> (with profile id: <id>)`

```text theme={null}
[2026-05-19 13:23:15] Profiling starts. Traces will be saved to: /tmp/1779196995.6948605 (with profile id: 1779196995.6979997)
[2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text.
[2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text.
[2026-05-19 13:23:15] INFO:     127.0.0.1:40714 - "POST /start_profile HTTP/1.1" 200 OK
```

* **When profiling stops**: server log outputs
  `Profiling done. Traces are saved to: <path>`

```text theme={null}
[2026-05-19 13:23:17] Stop profiling...
[2026-05-19 13:23:17] [WARNING] [350443] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data.
[rank0]:[W519 13:23:17.084812760 compiler_depend.ts:3136] Warning: The indexFromRank 0 is not equal indexFromCurDevice 4 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator())
[2026-05-19 13:23:17] [INFO] [352725] profiler.py: Start parsing profiling data: /tmp/1779196995.6948605/localhost.localdomain_350443_20260519132315700_ascend_pt
[2026-05-19 13:23:22] [INFO] [352734] profiler.py: CANN profiling data parsed in a total time of 0:00:04.022310
[2026-05-19 13:23:32] [INFO] [352725] profiler.py: All profiling data parsed in a total time of 0:00:14.305669
[2026-05-19 13:23:32] Profiling done. Traces are saved to: /tmp/1779196995.6948605
```

* **CLI output**: `sglang.profiler` outputs `Dump profiling traces to <path>`

```text theme={null}
Dump profiling traces to /tmp/1779243331.3219
Waiting for 10 steps and the trace to be flushed.... (profile_by_stage=False)
```

The directory structure is
`<output_dir>/<hostname>_<pid>_<timestamp>_ascend_pt/`. When using Method C
(`bench_serving --profile`), a timestamp subdirectory is added:
`<output_dir>/<timestamp>/<hostname>_<pid>_<timestamp>_ascend_pt/`. Always check the server log for the exact path:
`Profiling done. Traces are saved to: <path>`.

### 5. Viewing Results

After profiling stops (either `/stop_profile` returns or `num_steps`
auto-triggers), the server **automatically parses the raw data**. The
`ASCEND_PROFILER_OUTPUT` directory directly contains the following visualization
files — **no need to manually call `analyse()`**:

<table>
  <thead>
    <tr><th>File</th><th>Description</th></tr>
  </thead>

  <tbody>
    <tr>
      <td>`trace_view.json`</td>

      <td>
        Chrome Tracing format. Open in
        [MindStudio Insight](https://www.hiascend.com/document/detail/zh/mindstudio/81RC1/GUI_baseddevelopmenttool/msascendinsightug/Insight_userguide_0002.html)
      </td>
    </tr>

    <tr><td>`analysis.db`</td><td>Database-format performance data</td></tr>

    <tr>
      <td>`ascend_pytorch_profiler_0.db`</td>
      <td>Database-format performance data</td>
    </tr>

    <tr><td>`kernel_details.csv`</td><td>Kernel-level data</td></tr>
    <tr><td>`operator_details.csv`</td><td>Operator-level data</td></tr>
    <tr><td>`step_trace_time.csv`</td><td>Step trace timing data</td></tr>
  </tbody>
</table>

> [!NOTE]
>
> `trace_view.json` can also be opened using Chrome's built-in
> `chrome://tracing` or [Perfetto UI](https://ui.perfetto.dev/).

> [!NOTE]
>
> In a multi-node deployment, each node generates its own trace files.
> To merge them into a single unified trace, set `"merge_profiles": true` in the `/start_profile` request.
> However, on Ascend NPU, the merge feature does **not** fully support the `*_ascend_pt` format — merged results may be incomplete or incorrect.
> It is recommended to view `trace_view.json` on each node individually instead.
> For more details, see [Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling#profiler-trace-merger-for-distributed-traces).

### 6. Re-parsing Raw Data (Optional)

If you need to **re-parse existing data** with different parameters, or if
profiling was interrupted and `ASCEND_PROFILER_OUTPUT` was not auto-generated,
use `torch_npu`'s `analyse()` tool:

```python theme={null}
from torch_npu.profiler.profiler import analyse
analyse("./sglang_profile/<hostname>_*_ascend_pt/")
```

> [!NOTE]
>
> Normally **no need** to manually run `analyse()` — the server already parses
> data automatically. Only use this for re-parsing or handling interrupted data.
> Running `analyse()` when ASCEND\_PROFILER\_OUTPUT already exists will overwrite the original directory.
> If the original data is still needed, back it up before running `analyse()`.

## Best Practices

### Common Notes

* **Finding output**: Check the server log for
  `Profiling starts. Traces will be saved to: <path>` and
  `Profiling done. Traces are saved to: <path>`, or `sglang.profiler` output for
  `Dump profiling traces to <path>`.
* **Control trace file size**: Reduce the number of requests and output length
  using `--num-prompts` and `--random-output-len` to avoid trace files too large
  for browsers.
* **Warmup iterations**: Set `start_step` to skip the first few warmup steps and
  capture performance data under steady state.
* **Profile step count**: Large values for `num_steps` or `--profile-steps` can
  lead to lengthy profiling data parsing times. Reduce these values
  appropriately when you only need a quick overview.
* **CUDA Graph impact**: To see the full Python call stack → operator mapping in
  traces, add `--disable-cuda-graph` when starting the server. Note that this
  reduces decode performance — only use during profiling. To analyze CUDA Graph
  capture specifically, use `--enable-profile-cuda-graph` — traces are saved to
  `SGLANG_TORCH_PROFILER_DIR/graph_capture_profile/`.
* **Multi-node deployment**: In multi-node environments, performance data is
  distributed across nodes. On Ascend NPU, the `merge_profiles` feature has
  limited support — check `*_ascend_pt/ASCEND_PROFILER_OUTPUT/trace_view.json`
  on each node individually. In PD disaggregation mode, prefill and decode
  workers must be profiled separately — see
  [Profile In PD Disaggregation Mode](/docs/developer_guide/benchmark_and_profiling#profile-in-pd-disaggregation-mode).

## See Also

* [SGLang Benchmark and Profiling](/docs/developer_guide/benchmark_and_profiling)
  — General SGLang profiling guide
* [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start)
  — Ascend NPU environment setup
* [Ascend NPU Optimization](/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning)
  — Ascend NPU optimization parameters
* [Ascend NPU Performance Testing](/docs/hardware-platforms/ascend-npus/evaluation/performance_testing)
  — Ascend NPU performance benchmarking
* [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/reference/environment_variables)
  — Environment variable reference


# Quantization on Ascend
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/quantization



To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.

SGLang supports **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.

[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)

<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Series Supported</th>
      <th>A3 Series Supported</th>
      <th>950PR/DT Series Supported</th>
      <th>Diffusion models</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>W4A4 dynamic</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>**√**</td>
    </tr>

    <tr>
      <td>W8A8 static</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>**√**</td>
    </tr>

    <tr>
      <td>W8A8 dynamic</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>**√**</td>
    </tr>

    <tr>
      <td>MXFP8 ([Diffusion](https://github.com/sgl-project/sglang/pull/20922), [LLM dense](https://github.com/sgl-project/sglang/pull/22352))</td>
      <td>Linear</td>
      <td>**x**</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>**√**</td>
    </tr>

    <tr>
      <td>[MXFP4](https://github.com/sgl-project/sglang/pull/22338)</td>
      <td>Linear</td>
      <td>**x**</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>**√**</td>
    </tr>

    <tr>
      <td>[MXFP4 W4A8](https://github.com/sgl-project/sglang/pull/23650)</td>
      <td>Linear</td>
      <td>**x**</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>**x**</td>
    </tr>

    <tr>
      <td>[MXFP4 W4A8 (ModelSlim)](https://github.com/sgl-project/sglang/pull/30318)</td>
      <td>MoE</td>
      <td>**x**</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>**x**</td>
    </tr>

    <tr>
      <td>[MXFP4 W4A4](https://github.com/sgl-project/sglang/pull/23795)</td>
      <td>Linear</td>
      <td>**x**</td>
      <td>**x**</td>
      <td>**WIP**</td>
      <td>**x**</td>
    </tr>

    <tr>
      <td>[MXFP4 W4A4 (ModelSlim)](https://github.com/sgl-project/sglang/pull/30319)</td>
      <td>MoE</td>
      <td>**x**</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>**x**</td>
    </tr>

    <tr>
      <td>W4A4 dynamic</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>**x**</td>
    </tr>

    <tr>
      <td>W4A8 dynamic</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>**x**</td>
    </tr>

    <tr>
      <td>W8A8 dynamic</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>**x**</td>
    </tr>

    <tr>
      <td>[MXFP8](https://github.com/sgl-project/sglang/pull/20922) (LLM MoE)</td>
      <td>MoE</td>
      <td>**x**</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>**x**</td>
    </tr>
  </tbody>
</table>

[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):

<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Series Supported</th>
      <th>A3 Series Supported</th>
      <th>950PR/DT Series Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>W4A16</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>W8A16</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>W4A16</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>
  </tbody>
</table>

GPTQ on Ascend support

<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Series Supported</th>
      <th>A3 Series Supported</th>
      <th>950PR/DT Series Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[W4A16](https://github.com/sgl-project/sglang/pull/15203)</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>[W8A16](https://github.com/sgl-project/sglang/pull/15203)</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>[W4A16 MOE](https://github.com/sgl-project/sglang/pull/16364)</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>[W8A16 MOE](https://github.com/sgl-project/sglang/pull/16364)</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>
  </tbody>
</table>

[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)

<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Series Supported</th>
      <th>A3 Series Supported</th>
      <th>950PR/DT Series Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>W4A16</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>W8A16</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>W4A16</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>W8A16</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>
  </tbody>
</table>

Compressed-tensors (LLM Compressor) on Ascend support:

<table>
  <thead>
    <tr>
      <th>Quantization scheme</th>
      <th>Layer type</th>
      <th>A2 Series Supported</th>
      <th>A3 Series Supported</th>
      <th>950PR/DT Series Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504)</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>[W4A8 dynamic with/without activation clip](https://github.com/sgl-project/sglang/pull/14736)</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>[W4A16 MOE](https://github.com/sgl-project/sglang/pull/12759)</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>[W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504)</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>
  </tbody>
</table>

[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)

<table>
  <thead>
    <tr>
      <th>Quantization type</th>
      <th>Layer type</th>
      <th>A2 Series Supported</th>
      <th>A3 Series Supported</th>
      <th>950PR/DT Series Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>All GGUF types (standard, K-quant)</td>
      <td>Linear</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>

    <tr>
      <td>All GGUF types (standard, K-quant)</td>
      <td>MoE</td>
      <td>**√**</td>
      <td>**√**</td>
      <td>**TBD**</td>
    </tr>
  </tbody>
</table>

**Usage Examples:**

* Dense model (e.g., Qwen3-14B-Q4\_K\_M.gguf):

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen3-14B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.7 --tp-size 2
```

* MoE model (e.g., Qwen3-30B-A3B-Q4\_K\_M.gguf):

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen3-30B-A3B-Q4_K_M.gguf \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 2
```

> **Implementation Notes:**
>
> * GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead).
> * MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation.
> * TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.

**MXFP8 for LLM dense models (e.g., Qwen3 / Qwen3.5):**

LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend 950PR/DT Series or newer (`npu_dynamic_mx_quant` is not available on A2/A3 Series).

* Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time):

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

* Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`):

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path /path/to/Qwen3-8B-W8A8-MXFP8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

> **Implementation Notes:**
>
> * Online path: `Fp8Config.get_quant_method()` dispatches to `NPUMXFP8LinearMethod`. Weights are quantized once at load via `npu_dynamic_mx_quant(weight, dst_type=torch_npu.float8_e4m3fn)` and pre-transposed to `[in, out]`; activations are per-token quantized at inference and matmul runs via `npu_quant_matmul(..., group_sizes=[1, 1, 32])` (block\_size = 32).
> * Offline path: `ModelSlimMXFP8Scheme` loads `float8_e4m3fn` weights + `float8_e8m0fnu` block scales pre-exported by msmodelslim. Transpose is kept as a non-contiguous view (`.data` assignment) — calling `.contiguous()` would physically reorder the pre-quantized layout and break the block-scale mapping.
> * MoE MXFP8 (FusedMoE) for LLMs is documented in **MXFP8 for LLM MoE models** below.

**MXFP8 for LLM MoE models (e.g. Qwen3-30B-A3B / Qwen3.5 MoE):**

LLM MoE W8A8 MXFP8 (FusedMoE) support builds on the dense MXFP8 path. Requires Ascend 950PR/DT Series or newer — the fused MoE MX kernels (`npu_grouped_matmul_swiglu_quant_v2`, `npu_dynamic_mx_quant`) are only available on the 950PR/DT Series.

* Online MXFP8 quantization (BF16/FP16 expert weights → MXFP8 at load time):

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-30B-A3B \
    --quantization mxfp8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

* Offline MXFP8 quantization (msmodelslim pre-quantized weights, `W8A8_MXFP8` scheme). No `--quantization` flag is needed: the `quant_model_description.json` shipped with the checkpoint selects both the ModelSlim path and the scheme automatically.

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path /path/to/Qwen3-30B-A3B-W8A8-MXFP8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

> **Implementation Notes:**
>
> * Both paths share the per-gmm kernel `NPUMXFP8MoEMethod` (`hardware_backend/npu/quantization/moe_methods.py`), which tells online from offline by weight dtype. Expert weights and their e8m0 block scales are kept as non-contiguous transpose views — calling `.contiguous()` would tank HBM bandwidth.
> * Online path: `Fp8Config.get_quant_method()` dispatches FusedMoE layers to `NPUMXFP8OnlineMoEMethod`, which subclasses `UnquantizedFusedMoEMethod` and overrides only `create_moe_runner` to swap in the MXFP8 kernels — weight creation, weight post-processing and the forward pass are the unquantized Ascend ones. BF16 expert weights `w13`/`w2` are quantized once at load via `npu_dynamic_mx_quant(dst_type=torch.float8_e4m3fn)` (a 3D `[E, N, K]` input is accepted directly).
> * Offline path: `ModelSlimMXFP8MoEScheme` (one instance per weight group) loads `float8_e4m3fn` expert weights + uint8 (e8m0, exponent + 127) block scales. The scale is reshaped `[E, N, K/32] → [E, N, K/64, 2]` (contiguous pairing, matching `npu_dynamic_mx_quant`) then transposed.
> * Forward: `AscendTPDispatcher` runs `npu_moe_init_routing_v2(quant_mode=3)`, which fuses the per-token MX activation quant into routing (e4m3 payload + e8m0 block scale, reshaped to the pair-split layout). `AscendRunnerCore` then runs gmm1 `npu_grouped_matmul_swiglu_quant_v2` (cumulative `group_list`; fuses gate/up + swiglu + requant, so no separate activation step) → gmm2 `npu_grouped_matmul` (count `group_list`). The UE8M0 (`float8_e8m0fnu`) scale dtypes are passed explicitly; the e4m3 `x`/`weight` dtypes are left implicit.
> * **Router gate**: msmodelslim may also quantize `mlp.gate` (`W8A8_MXFP8`). The gate is a `ReplicatedLinear`, so its quantization must be **description-driven**: for the offline `modelslim` path the gate is passed the quant config and dequantized correctly; the online path keeps it in BF16. Loading a quantized gate as BF16 without its block scale scrambles routing and produces garbage output.
> * Where the activation quant happens depends on the dispatcher. On `ascend_tp` it is fused into routing as described above. DeepEP has no MXFP8 dispatch dtype, so it keeps dispatching BF16 and gmm1 quantizes the hidden states itself via `npu_dynamic_mx_quant` before the fused kernel — the two paths reach the same gmm1 input. Only the `ascend_tp` path has been validated end-to-end on the Ascend 950PR/DT Series.

**MXFP4 W4A8 for LLM dense models (e.g., Qwen3 / Qwen3.5):**

LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend 950PR/DT Series or newer.

* Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time):

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp_w4a8 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

* Offline W4A8 quantization (msmodelslim pre-quantized weights, `W4A8_MXFP` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`).

> **Implementation Notes:**
>
> * Weights are packed FP4 (`float4_e2m1fn_x2`, two nibbles per byte) with a UE8M0 per-block shared exponent (block\_size = 32); activations are per-token MXFP8. Matmul runs via `npu_quant_matmul(..., x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[0, 0, 32])`.
> * The packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype` / `input_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject.
> * Online and offline share the same kernel path and layout; they differ only in the weight source (RTN at load vs msmodelslim calibration).

**ModelSlim W4A8 MXFP4 for LLM MoE models:**

SGLang auto-detects offline ModelSlim `W4A8_MXFP` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950PR/DT Series or newer.

```bash Command theme={null}
MODEL_PATH=/path/to/w4a8-mxfp4-moe-model
python3 -m sglang.launch_server \
    --model-path "$MODEL_PATH" \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --tp-size 1
```

> **Implementation Notes:**
>
> * ModelSlim supplies packed MXFP4 `w13` and `w2` expert weights with UE8M0 block scales (block size 32).
> * Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP8 dispatch. SGLang dynamically quantizes each expert input to MXFP8 immediately before grouped matmul.

**MXFP4 W4A4 for LLM dense models (e.g. Qwen3 / Qwen3.5):**

LLM dense W4A4 (MXFP4 4-bit weights + 4-bit activations) Linear support was added in [PR #23795](https://github.com/sgl-project/sglang/pull/23795). Requires Ascend 950PR/DT Series or newer — the dual-level online path uses the `DualLevelQuantBatchMatmul` op, which A2/A3 Series lack. On the Ascend NPU backend `--quantization mxfp4` selects this W4A4 path (on GPU the same flag selects the upstream OCP MXFP4 MoE config instead).

* Online W4A4 quantization (BF16/FP16 weights → dual-level MXFP4 at load time):

```bash Command theme={null}
python3 -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --quantization mxfp4 \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --mem-fraction-static 0.8 --tp-size 1
```

* Offline W4A4 quantization (msmodelslim pre-quantized weights, `W4A4_MXFP4` scheme; no `--quantization` flag needed — auto-detected from `quant_model_description.json`).

> **Implementation Notes:**
>
> * **Online** (`NPUDualLevelMXFP4LinearMethod`) uses **dual-level** MXFP4: both weights and activations are quantized with a fine FP8 (E4M3) L0 block scale plus a coarser L1 scale via `npu_dynamic_dual_level_mx_quant`, and the matmul runs via `npu_dual_level_quant_matmul` (weight in FRACTAL\_NZ). Dual-level captures per-block dynamic range far better than a single UE8M0 (power-of-2) scale, which is what made an earlier single-level RTN online path degenerate (greedy decoding could loop without emitting EOS).
> * **Offline** (`ModelSlimMXFP4Scheme` → `NPUSingleLevelMXFP4OfflineLinearMethod`) is **single-level**: msmodelslim's `W4A4_MXFP4` checkpoint ships single-level UE8M0 block scales (block\_size = 32), so the matmul runs via `npu_quant_matmul(..., x1_dtype=x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[1, 1, 32])`. The online and offline paths therefore use different matmul kernels — they no longer share the matmul path.
> * As with W4A8, the packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject.
> * Validated end-to-end on Ascend 950PR/DT Series hardware.

**ModelSlim W4A4 MXFP4 for LLM MoE models:**

SGLang auto-detects offline ModelSlim `W4A4_MXFP4` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950PR/DT Series or newer.

```bash Command theme={null}
MODEL_PATH=/path/to/w4a4-mxfp4-moe-model
python3 -m sglang.launch_server \
    --model-path "$MODEL_PATH" \
    --device npu --attention-backend ascend \
    --host 0.0.0.0 --port 30000 \
    --tp-size 1
```

> **Implementation Notes:**
>
> * ModelSlim supplies packed MXFP4 `w13` and `w2` expert weights with UE8M0 block scales (block size 32).
> * Ascend TP and DeepEP dispatch activations as BF16; this path does not request MXFP4 dispatch. SGLang dynamically quantizes each expert input to MXFP4 immediately before grouped matmul.

## Diffusion Model Quantization on Ascend NPU

SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires Ascend 950PR/DT Series; the ModelSlim W8A8/W4A4 schemes work on A2/A3 Series.

**Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend 950PR/DT Series

<table>
  <thead>
    <tr>
      <th>Quantization method</th>
      <th>`quant_type` in JSON</th>
      <th>Scheme class</th>
      <th>Mode</th>
      <th>A2/A3 Series Supported</th>
      <th>950PR/DT Series Supported</th>
      <th>Trigger</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>MXFP8 (W8A8)</td>
      <td>—</td>
      <td>`MXFP8Config`</td>
      <td>Online</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>`--quantization mxfp8`</td>
    </tr>

    <tr>
      <td>MXFP8 (W8A8)</td>
      <td>`W8A8_MXFP8`</td>
      <td>`ModelSlimMXFP8Scheme`</td>
      <td>Offline</td>
      <td>**x**</td>
      <td>**√**</td>
      <td>auto-detected from `quant_model_description.json`</td>
    </tr>

    <tr>
      <td>W8A8 static</td>
      <td>`W8A8`</td>
      <td>`ModelSlimW8A8Int8`</td>
      <td>Offline</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>auto-detected from `quant_model_description.json`</td>
    </tr>

    <tr>
      <td>W8A8 dynamic</td>
      <td>`W8A8_DYNAMIC`</td>
      <td>`ModelSlimW8A8Int8`</td>
      <td>Offline</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>auto-detected from `quant_model_description.json`</td>
    </tr>

    <tr>
      <td>W4A4 dynamic</td>
      <td>`W4A4_DYNAMIC`</td>
      <td>`ModelSlimW4A4Int4`</td>
      <td>Offline</td>
      <td>**√**</td>
      <td>**TBD**</td>
      <td>auto-detected from `quant_model_description.json`</td>
    </tr>
  </tbody>
</table>

### Online MXFP8 Quantization

Online quantization dynamically quantizes FP16/BF16 weights to MXFP8 at load time using `npu_dynamic_mx_quant` + `npu_quant_matmul` CANN kernels. Pass `--quantization mxfp8` to override auto-detection.

```bash Command theme={null}
# Start the diffusion server with online MXFP8 quantization
sglang serve \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --num-gpus 4
```

```bash Command theme={null}
# One-shot generation
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --prompt "a beautiful sunset over the mountains" \
  --save-output
```

### Offline MXFP8 Quantization (ModelSlim)

For offline quantization, pre-quantize the model with msModelSlim and load the resulting checkpoint. The quantization scheme is auto-detected from `quant_model_description.json`, so no extra `--quantization` flag is needed.

**Step 1: Quantize with msModelSlim**

```bash Command theme={null}
msmodelslim quant \
  --model_path /path/to/wan2_2_float_weights \
  --save_path /path/to/wan2_2_mxfp8_weights \
  --device npu \
  --model_type Wan2_2 \
  --quant_type mxfp8 \
  --trust_remote_code True
```

> Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim.

**Step 2: Convert to Diffusers format**

msModelSlim saves quantized Wan2.2 weights in the original Wan format. Convert to Diffusers format using the provided repack script:

```bash Command theme={null}
python python/sglang/multimodal_gen/tools/wan_repack.py \
  --input-path /path/to/wan2_2_mxfp8_weights \
  --output-path /path/to/wan2_2_mxfp8_diffusers
```

Then copy all files from the original Diffusers checkpoint (except the `transformer`/`transformer_2` folders) into the output directory.

**Step 3: Run inference**

```bash Command theme={null}
sglang generate \
  --model-path /path/to/wan2_2_mxfp8_diffusers \
  --prompt "a beautiful sunset over the mountains" \
  --save-output
```

For pre-quantized checkpoints available on ModelScope, see [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).


# Ring-SP Performance (Wan2.1-T2V-1.3B)
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/optimization/ring_sp_performance



This page reports Ring-SP performance on Ascend NPU with `torch_npu==2.10.0`.

* Baseline config: `ulysses=1, ring=1` (short: `u1r1`)
* Ring-SP config: `ulysses=1, ring=2` (short: `u1r2`)

## Benchmark Setup

* Model: `Wan2.1-T2V-1.3B-Diffusers`
* Prompt: `"a cat is playing piano"`
* Framework command: `sglang generate`
* Runtime: `torch_npu==2.10.0`

## Generate Commands

### Baseline (`u1r1`)

```bash theme={null}
sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
    --prompt "a cat is playing piano" --num-gpus 1 --ring-degree 1 \
    --save-output
```

### Ring-SP (`u1r2`)

```bash theme={null}
sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
    --prompt "a cat is playing piano" --num-gpus 2 --ring-degree 2 \
    --save-output
```

## Benchmarks

Benchmark Disclaimer

These numbers are from one fixed setup and one prompt case. Actual performance may vary by model settings, environment, and workload.

### Stage Time Breakdown

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Stage / Metric</th>
      <th>`u1r2` (s)</th>
      <th>`u1r1` baseline (s)</th>
      <th>Speedup</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>InputValidation</td>
      <td>0.0003</td>
      <td>0.0002</td>
      <td>0.67x</td>
    </tr>

    <tr>
      <td>TextEncoding</td>
      <td>3.5936</td>
      <td>3.5820</td>
      <td>1.00x</td>
    </tr>

    <tr>
      <td>LatentPreparation</td>
      <td>0.0007</td>
      <td>0.0055</td>
      <td>7.86x</td>
    </tr>

    <tr>
      <td>TimestepPreparation</td>
      <td>0.0008</td>
      <td>0.0007</td>
      <td>0.88x</td>
    </tr>

    <tr>
      <td>Denoising</td>
      <td>121.2788</td>
      <td>239.2580</td>
      <td>1.97x</td>
    </tr>

    <tr>
      <td>Decoding</td>
      <td>13.8685</td>
      <td>16.4969</td>
      <td>1.19x</td>
    </tr>

    <tr>
      <td>**Total (Pixel data generated)**</td>
      <td>**141.86**</td>
      <td>**266.50**</td>
      <td>**1.88x**</td>
    </tr>
  </tbody>
</table>

## Summary

* With `torch_npu==2.10.0`, Ring-SP (`u1r2`) runs successfully on NPU for this case.
* End-to-end generation time improves from `266.50s` to `141.86s` (`1.88x`).
* The main gain comes from `DenoisingStage` (`1.97x`), while decoding also improves (`1.19x`).


# Environment Variables
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/environment_variables



SGLang supports various environment variables related to Ascend NPU that can be used to configure its runtime behavior.
This document provides a list of commonly used environment variables and aims to stay updated over time.

## Directly Used in SGLang

<table>
  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_NPU_USE_MLAPO`</td>
      <td>Adopts the `MLAPO` fusion operator in the attention \
      &#x20;preprocessing stage of the MLA model.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_FIA_NZ`</td>
      <td>Reshapes KV Cache for FIA NZ format.\
      &#x20;`SGLANG_USE_FIA_NZ` must be enabled with `SGLANG_NPU_USE_MLAPO`</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_USE_MULTI_STREAM`</td>
      <td>Enable dual-stream computation of shared experts \
      &#x20;and routing experts in DeepSeek models.\
      &#x20;Enable dual-stream computation in DeepSeek DSA Indexer.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT`</td>
      <td>Disable casting model weight tensor to a specific NPU \
      &#x20;ACL format.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
      <td>The maximum number of dispatched tokens on each rank.</td>
      <td>`128`</td>
    </tr>
  </tbody>
</table>

## Used in DeepEP Ascend

<table>
  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`</td>
      <td>Enable long-sequence token pipelining in dispatch stage. Indicates \
      &#x20;the number of tokens transmitted per round on each rank.</td>
      <td>`8192`</td>
    </tr>

    <tr>
      <td>`DEEPEP_NORMAL_LONG_SEQ_ROUND`</td>
      <td>Enable long-sequence token pipelining in dispatch stage. Indicates \
      &#x20;the number of rounds transmitted on each rank.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`</td>
      <td>Enable long-sequence token pipelining in combine stage. \
      &#x20;The value `0` means disabled.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`MOE_ENABLE_TOPK_NEG_ONE`</td>
      <td>Needs to be enabled when the expert ID to be processed by \
      &#x20;DEEPEP contains -1.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`DEEP_NORMAL_MODE_USE_INT8_QUANT`</td>

      <td>
        **Deprecated — will be removed in a future release.**

        When set to `1`, quantizes intermediate activations to INT8 in
        \
        the DeepEP dispatch operator during normal mode, reducing communication
        \
        volume for W8A8-quantized MoE models. This variable will become a no-op;
        \
        the quantization behavior will be inferred automatically.
      </td>

      <td>`0`</td>
    </tr>

    <tr>
      <td>`DEEPEP_HCCL_BUFFSIZE`</td>
      <td>Configures the HCCL buffer size (in MB) for the process groups used by \
      &#x20;DeepEP MoE All-to-All communication (the default group and `moe`-named \
      &#x20;groups), allowing the MoE communication buffer to be tuned independently from \
      &#x20;non-MoE groups. If unset, falls back to `HCCL_BUFFSIZE`, then to `200`. \
      &#x20;A larger buffer reduces All-to-All latency at the cost of more HBM usage.</td>
      <td>`200`</td>
    </tr>
  </tbody>
</table>

## Others

<table>
  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`TASK_QUEUE_ENABLE`</td>
      <td>Used to control the optimization level of the dispatch queue\
      &#x20;for the task\_queue operator. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/730/comref/Envvariables/docs/zh/environment_variable_reference/TASK_QUEUE_ENABLE.md)</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`INF_NAN_MODE_ENABLE`</td>
      <td>Controls whether the chip uses saturation mode or INF\_NAN mode. [Detail](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0056.html)</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`STREAMS_PER_DEVICE`</td>
      <td>Configures the maximum number of streams for the stream pool. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/720/comref/Envvariables/Envir_041.html)</td>
      <td>`32`</td>
    </tr>

    <tr>
      <td>`PYTORCH_NPU_ALLOC_CONF`</td>
      <td>Controls the behavior of the cache allocator. \
      This variable changes memory usage and may cause performance fluctuations. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/700/comref/Envvariables/Envir_012.html)</td>

      <td />
    </tr>

    <tr>
      <td>`ASCEND_MF_STORE_URL`</td>
      <td>The address of config store in MemFabric during PD separation, \
      which is generally set to the IP address of the P primary node\
      &#x20;with an arbitrary port number.</td>

      <td />
    </tr>

    <tr>
      <td>`ASCEND_LAUNCH_BLOCKING`</td>
      <td>Controls whether synchronous mode is enabled during operator execution. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/710/comref/Envvariables/Envir_006.html)</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`HCCL_OP_EXPANSION_MODE`</td>
      <td>Configures the expansion position for communication algorithm scheduling. [Detail](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0094.html)</td>

      <td />
    </tr>

    <tr>
      <td>`HCCL_BUFFSIZE`</td>
      <td>Controls the size of the buffer area for shared data between two NPUs. \
      The unit is MB, and the value must be greater than or equal to 1. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/60RC3/ptmoddevg/trainingmigrguide/performance_tuning_0047.html)</td>
      <td>`200`</td>
    </tr>

    <tr>
      <td>`HCCL_SOCKET_IFNAME`</td>
      <td>Configures the name of the network card used by the Host \
      during HCCL initialization. [Detail](https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/apiref/envvar/envref_07_0075.html)</td>

      <td />
    </tr>

    <tr>
      <td>`GLOO_SOCKET_IFNAME`</td>
      <td>Configures the network interface name for GLOO communication.</td>

      <td />
    </tr>
  </tbody>
</table>


# Glossary
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/glossary



This page covers the hardware concepts, communication libraries, deployment terminology, and common abbreviations
you will encounter throughout the Ascend NPU documentation. Refer back here when you run into unfamiliar terms.

## Hardware

### Supported devices

SGLang supports the following Ascend inference hardware:

| Hardware | Devices | Dies per card | Memory configuration |
| - | - | - | - |
| Ascend A3 Series Products | 16 | 2 | 8(cards) × 2(dies/card) × 64(GB/die) |
| Ascend 950PR/DT Series | 8 | 1 | 8(cards) × 1(die/card) — HBM depends on the chip |

> [!NOTE]
>
> Throughout these docs, **A3 Series** and **950PR/DT Series** are used as shorthand for the hardware above.
> Docker image tags use `a3` for the A3 Series. For example, `main-cann9.1.0-a3`.
>
> Tags for the 950PR/DT Series use `950`. For example, `main-cann9.1.0-950`.
>
> On the A3 Series, each card has 2 dies, giving 16 devices per node.
> Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on the A3 Series means `--tp-size 8`.
> On the 950PR/DT Series, each card is a single die, so card count, device count, and `--tp-size` match one-to-one.

### NPU

**NPU** stands for Neural Processing Unit. Each NPU device is a single `davinci` core.
The terms "NPU" and "davinci" are used interchangeably in commands and error logs.

On the A3 Series, devices are numbered `/dev/davinci0` through `/dev/davinci15` (16 devices).
On the 950PR/DT Series, each card exposes a single device, numbered `/dev/davinci0` through `/dev/davinci7` (8 devices).

On an A3 Series server, run `npu-smi info` to view NPU information such as device
health, memory usage, and chip status. If the command is not found or reports no devices,
the driver is likely not installed. Follow the
[Ascend driver installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/910/softwareinst/instg/instg_0005.html?OS=openEuler\&InstallType=local)
to install it.

## Communication libraries

| Library | Description |
| - | - |
| **HCCL** (Huawei Collective Communication Library) | The primary communication backend for multi-card data transfer on Ascend NPUs. Equivalent to NVIDIA NCCL. Used in `--nnodes`, `--tp-size`, and all distributed scenarios. |
| **GLOO** | Meta's collective communications library. Used alongside HCCL for distributed initialization and coordination. |
| **DeepEP** (Deep Expert Parallelism) | A communication library optimized for Mixture-of-Experts (MoE) all-to-all dispatch and combine operations. Used with `--moe-a2a-backend deepep`. |
| **RDMA** (Remote Direct Memory Access) | Enables direct memory access between nodes over InfiniBand or RoCE networks. Required for multi-node PD disaggregation. |

## Quantization and precision

| Notation | Meaning |
| - | - |
| **W8A8** | 8-bit weights, 8-bit activations |
| **W4A8** | 4-bit weights, 8-bit activations |
| **W4A16** | 4-bit weights, 16-bit activations |
| **BF16** | Brain Floating Point 16 — 16-bit format optimized for ML workloads |
| **FP8** | 8-bit Floating Point — not supported on the A3 Series (on Ascend, FP8-class quantization goes through MXFP8 below) |
| **MXFP8** | Microscaling FP8 (OCP MX block-scaled format) — requires the 950PR/DT Series |
| **MXFP4** | Microscaling FP4 (OCP MX block-scaled format) — requires the 950PR/DT Series |
| **INT8** | 8-bit Integer quantization |

To apply quantization, use `--quantization modelslim` for W8A8 INT8, or load a pre-quantized
checkpoint directly from a model hub.

## Deployment terminology

### Prefill-Decode (PD) disaggregation

PD disaggregation separates inference into two stages running on different nodes:

* **Prefill** (P): Processes the entire input prompt at once. Compute-bound.
* **Decode** (D): Generates tokens one at a time. Memory-bandwidth-bound.

**PD Mixed**: Both stages run on the same set of nodes.

| Shorthand | Meaning |
| - | - |
| **1P1D** | 1 prefill node + 1 decode node |
| **2P1D** | 2 prefill nodes + 1 decode node |
| **1P2D** | 1 prefill node + 2 decode nodes |

You will see these in [Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1)
section headings, e.g., `W8A8 2P1D 32P` means "W8A8 quantization, 2 prefill nodes +
1 decode node, 32 cards total."

### Parallelism strategies

| Strategy | Flag | Description |
| - | - | - |
| **Tensor Parallelism (TP)** | `--tp-size` | Splits model weights across NPUs within a node |
| **Data Parallelism (DP)** | `--dp-size` | Replicates the model across nodes for higher throughput |
| **Expert Parallelism (EP)** | `--ep-size` | Distributes MoE experts across devices; requires `--moe-a2a-backend` |
| **Context Parallelism (CP)** | `--attn-cp-size` | Splits long context windows across devices for extended sequence length |
| **Pipeline Parallelism (PP)** | `--pp-size` | Splits model layers across devices sequentially |

### Speculative decoding

| Algorithm | Description |
| - | - |
| **EAGLE3** | Uses an external draft model (specified via `--speculative-draft-model-path`) to propose candidate tokens. Supports top-k sampling. |
| **NEXTN** | Uses the model's built-in Multi-Token Prediction (MTP) heads — no separate draft model needed. Available for models with native MTP support. |
| **MTP** (Multi-Token Prediction) | A model architecture feature where the model predicts multiple future tokens per step. The foundation for NEXTN speculative decoding. |

### Performance metrics

| Metric | Description |
| - | - |
| **TPOT** (Time Per Output Token) | Average time to generate each output token. Lower is better. |
| **TTFT** (Time To First Token) | Latency from request arrival to first generated token. Critical for interactive use. |

## Model architecture terms

| Term | Description |
| - | - |
| **MoE** (Mixture of Experts) | Model architecture where only a subset of parameters (experts) is activated per token, reducing compute. Common in DeepSeek, Qwen3-30B-A3B, and MiMo models. |
| **MLA** (Multi-head Latent Attention) | Attention variant that compresses key-value representations into a lower-dimensional latent space. Used by DeepSeek models. |
| **GQA** (Grouped-Query Attention) | Attention variant where multiple query heads share a single key-value head. Used by Qwen3 dense models. |
| **DSA** (DeepSeek Sparse Attention) | DeepSeek's sparse attention mechanism; reduces KV cache overhead for long contexts. |
| **FFN** (Feed-Forward Network) | The non-attention component of each transformer layer. In MoE models, the FFN is replaced by multiple expert FFN layers selected by a router. |

## Other common abbreviations

| Abbreviation | Expansion |
| - | - |
| **ACL** (Ascend Computing Language) | Low-level Ascend compute API; seen in error logs |
| **DVFS** (Dynamic Voltage and Frequency Scaling) | Hardware frequency scaling to maintain performance stability |
| **NUMA** (Non-Uniform Memory Access) | Memory architecture affecting multi-socket performance |
| **KV Cache** (Key-Value Cache) | Cached attention key-value tensors to avoid recomputation during decode |
| **LoRA** (Low-Rank Adaptation) | Parameter-efficient fine-tuning method |
| **HF** (Hugging Face) | Model hub; `HF_TOKEN` / `HF_ENDPOINT` control model download access |
| **UB** (Unified Buffer) | On-chip NPU memory; referenced in operator optimization |
| **DMA** (Direct Memory Access) | Data transfer mechanism between host and device memory |

## Where to go next

* [Quickstart](/docs/hardware-platforms/ascend-npus/getting-started/quick_start) — launch your first server
* [Installation Guide](/docs/hardware-platforms/ascend-npus/getting-started/installation) — full installation with component version mapping
* [Supported Features](/docs/hardware-platforms/ascend-npus/reference/support_features) — per-parameter Ascend support status
* [Supported Models](/docs/hardware-platforms/ascend-npus/reference/support_models) — models verified on Ascend NPU
* [Model Tutorials](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1) — step-by-step deployment guides
* [Best Practice](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1) — benchmark configurations and results


# Supported Features
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/support_features



This section describes the basic functions and features supported by the Ascend NPU. If you encounter issues or have any
questions, please [open an issue](https://github.com/sgl-project/sglang/issues).

If you want to know the meaning and usage of each parameter,
click [Server Arguments](../../../advanced_features/server_arguments).

## Model and tokenizer

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--model-path`\
      `--model`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tokenizer-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tokenizer-mode`</td>
      <td>`auto`</td>
      <td>`auto`, `slow`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tokenizer-backend`</td>
      <td>`huggingface`</td>
      <td>`huggingface`, `fastokens`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tokenizer-worker-num`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--detokenizer-worker-num`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--skip-tokenizer-init`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--load-format`</td>
      <td>`auto`</td>
      <td>`auto`, `safetensors`, `gguf`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--model-loader-extra-config`</td>
      <td>`{}`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--context-length`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--is-embedding`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-multimodal`</td>
      <td>`None`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--revision`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--model-impl`</td>
      <td>`auto`</td>
      <td>`auto`, `sglang`,\
      &#x20;`transformers`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--model-config-parser`</td>
      <td>`auto`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## HTTP server

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--host`</td>
      <td>`127.0.0.1`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--port`</td>
      <td>`30000`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--skip-server-warmup`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--warmups`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--nccl-port`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--fastapi-root-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--grpc-mode`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>Planned</td>
    </tr>
  </tbody>
</table>

## SSL/TLS

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--ssl-keyfile`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--ssl-certfile`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--ssl-keyfile-password`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-ssl-refresh`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-http2`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Quantization and data type

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dtype`</td>
      <td>`auto`</td>
      <td>`auto`,\
      &#x20;`float16`,\
      &#x20;`bfloat16`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--quantization`</td>
      <td>`None`</td>
      <td>`modelslim`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--quantization-param-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--kv-cache-dtype`</td>
      <td>`auto`</td>
      <td>`auto`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-fp32-lm-head`</td>
      <td>`False`</td>
      <td>bool flag \
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--modelopt-quant`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--modelopt-checkpoint-restore-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--modelopt-checkpoint-save-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--modelopt-export-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--quantize-and-serve`</td>
      <td>`False`</td>
      <td>bool flag \
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--rl-quant-profile`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>
  </tbody>
</table>

## Memory and scheduling

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--mem-fraction-static`</td>
      <td>`None`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--max-running-requests`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-max-requests`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--max-queued-requests`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--max-total-tokens`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--chunked-prefill-size`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--max-prefill-tokens`</td>
      <td>`16384`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--schedule-policy`</td>
      <td>`fcfs`</td>
      <td>`lpm`,\
      `fcfs`,\
      `random`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-priority-scheduling`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-priority-preemption`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--default-priority-value`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--schedule-low-priority-values-first`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--priority-scheduling-preemption-threshold`</td>
      <td>`10`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--retraction-policy`</td>
      <td>`length`</td>
      <td>`length`,\
      &#x20;`priority`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--schedule-conservativeness`</td>
      <td>`1.0`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--page-size`</td>
      <td>`128`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--swa-full-tokens-ratio`</td>
      <td>`0.8`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--radix-eviction-policy`</td>
      <td>`lru`</td>
      <td>`lru`,\
      `lfu`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-prefill-delayer`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-max-delay-passes`</td>
      <td>`30`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-token-usage-low-watermark`</td>
      <td>`None`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-forward-passes-buckets`</td>
      <td>`None`</td>
      <td>List\[float]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-wait-seconds-buckets`</td>
      <td>`None`</td>
      <td>List\[float]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-queue-min-ratio`</td>
      <td>`None`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-delayer-max-delay-ms`</td>
      <td>`None`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--abort-on-priority-when-disabled`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-dynamic-chunking`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Runtime options

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--device`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tensor-parallel-size`\
      `--tp-size`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--pipeline-parallel-size`\
      `--pp-size`</td>
      <td>`1`</td>
      <td>Type: int; Currently `2` not supported; Cannot be used together with TP</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--attention-context-parallel-size`\
      `--attn-cp-size`</td>
      <td>`1`</td>
      <td>Type: int; must be equal to --tp-size</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--moe-data-parallel-size`\
      `--moe-dp-size`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--pp-max-micro-batch-size`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--pp-async-batch-depth`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--stream-interval`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--batch-notify-size`</td>
      <td>`16`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--incremental-streaming-output`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--stream-response-default-include-usage`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-streaming-session`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-session-radix-cache`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--random-seed`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--constrained-json-whitespace-pattern`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--constrained-json-disable-any-whitespace`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--watchdog-timeout`</td>
      <td>`300`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--soft-watchdog-timeout`</td>
      <td>`300`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--dist-timeout`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--download-dir`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--model-checksum`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--base-gpu-id`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--gpu-id-step`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--sleep-on-idle`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--load-snapshot-publish-interval`</td>
      <td>`15`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--use-ray`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--custom-sigquit-handler`</td>
      <td>`None`</td>
      <td>Only for engine</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Logging

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--log-level`</td>
      <td>`info`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--log-level-http`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--log-requests`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--log-requests-level`</td>
      <td>`2`</td>
      <td>`0`, `1`, `2`, `3`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--log-requests-format`</td>
      <td>text</td>
      <td>`text`, `json`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--crash-dump-folder`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-metrics`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-mfu-metrics`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-metrics-for-all-schedulers`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tokenizer-metrics-custom-labels-header`</td>
      <td>`x-custom-labels`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tokenizer-metrics-allowed-custom-labels`</td>
      <td>`None`</td>
      <td>List\[str]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--extra-metric-labels`</td>
      <td>`None`</td>
      <td>Type: JSON/Dict</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--bucket-time-to-first-token`</td>
      <td>`None`</td>
      <td>List\[float]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--bucket-inter-token-latency`</td>
      <td>`None`</td>
      <td>List\[float]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--bucket-e2e-request-latency`</td>
      <td>`None`</td>
      <td>List\[float]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prompt-tokens-buckets`</td>
      <td>`None`</td>
      <td>List\[str]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--generation-tokens-buckets`</td>
      <td>`None`</td>
      <td>List\[str]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--gc-warning-threshold-secs`</td>
      <td>`0.0`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--decode-log-interval`</td>
      <td>`40`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-request-time-stats-logging`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--kv-events-config`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--enable-forward-pass-metrics`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--forward-pass-metrics-worker-id`</td>
      <td>\`\`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--forward-pass-metrics-ipc-name`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-trace`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--trace-modules`</td>
      <td>`request`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--otlp-traces-endpoint`</td>
      <td>`localhost:4317`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--log-requests-target`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--uvicorn-access-log-exclude-prefixes`</td>
      <td>`[]`</td>
      <td>List\[str]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## RequestMetricsExporter configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--export-metrics-to-file`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--export-metrics-to-file-dir`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## API related

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--api-key`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--admin-api-key`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--served-model-name`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--weight-version`</td>
      <td>`default`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--chat-template`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--hf-chat-template-name`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--completion-template`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--file-storage-path`</td>
      <td>`sglang_storage`</td>
      <td>Type: str</td>
      <td>Unused reserved parameter</td>
    </tr>

    <tr>
      <td>`--enable-cache-report`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--reasoning-parser`</td>
      <td>`None`</td>
      <td>`deepseek-r1`\
      `deepseek-v3`\
      `glm45`\
      `gpt-oss`\
      `kimi`\
      `qwen3`\
      `qwen3-thinking`\
      `step3`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--default-chat-template-kwargs`</td>
      <td>`None`</td>
      <td>Type: JSON / Dict</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--strip-thinking-cache`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-strict-thinking`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--tool-call-parser`</td>
      <td>`None`</td>
      <td>`llama3`\
      &#x20;`pythonic`\
      &#x20;`qwen`\
      &#x20;`qwen3_coder`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--sampling-defaults`</td>
      <td>`model`</td>
      <td>`openai`, `model`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--asr-max-buffer-seconds`</td>
      <td>`60`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--asr-max-concurrent-sessions`</td>
      <td>`32`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Data parallelism

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--data-parallel-size`\
      `--dp-size`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--load-balance-method`</td>
      <td>`auto`</td>
      <td>`auto`,\
      &#x20;`round_robin`,\
      &#x20;`follow_bootstrap_room`,\
      &#x20;`total_requests`,\
      &#x20;`total_tokens`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Multi-node distributed serving

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dist-init-addr`\
      `--nccl-init-addr`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--nnodes`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--node-rank`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Model override args

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--json-model-override-args`</td>
      <td>`{}`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--preferred-sampling-params`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## LoRA (restricted to Qwen series models)

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-lora`</td>
      <td>`False`</td>
      <td>bool flag \
      (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-lora-overlap-loading`</td>
      <td>`False`</td>
      <td>bool flag \
      (set to enable)</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--max-lora-rank`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--lora-target-modules`</td>
      <td>`None`</td>
      <td>`all`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--lora-paths`</td>
      <td>`None`</td>
      <td>Type: List\[str] /\
      &#x20;JSON objects</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--max-loras-per-batch`</td>
      <td>`8`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--max-loaded-loras`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--lora-eviction-policy`</td>
      <td>`lru`</td>
      <td>`lru`,\
      &#x20;`fifo`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--lora-backend`</td>
      <td>`csgmv`</td>
      <td>`triton`,\
      &#x20;`csgmv`,\
      &#x20;`ascend`,\
      &#x20;`torch_native`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--experts-shared-outer-loras`</td>
      <td>`None`</td>
      <td>Type: bool</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--lora-use-virtual-experts`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--lora-strict-loading`</td>
      <td>`False`</td>
      <td>Type: bool</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--lora-drain-wait-threshold`</td>
      <td>`0.0`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--max-lora-chunk-size`</td>
      <td>`16`</td>
      <td>`16`, `32`,\
      &#x20;`64`, `128`</td>
      <td>Special for GPU</td>
    </tr>
  </tbody>
</table>

## Kernel backends (attention, sampling, grammar, GEMM)

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--attention-backend`</td>
      <td>`None`</td>
      <td>`ascend`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--prefill-attention-backend`</td>
      <td>`None`</td>
      <td>`ascend`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--decode-attention-backend`</td>
      <td>`None`</td>
      <td>`ascend`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--sampling-backend`</td>
      <td>`None`</td>
      <td>`pytorch`,\
      `ascend`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--grammar-backend`</td>
      <td>`None`</td>
      <td>`xgrammar`,\
      `outlines`,\
      `llguidance`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--radix-cache-backend`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mm-attention-backend`</td>
      <td>`None`</td>
      <td>`ascend_attn`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--dsa-prefill-backend`</td>
      <td>`flashmla_sparse`</td>
      <td>`flashmla_sparse`,\
      &#x20;`flashmla_decode`,\
      `fa3`,\
      &#x20;`tilelang`,\
      &#x20;`aiter`</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--dsa-decode-backend`</td>
      <td>`fa3`</td>
      <td>`flashmla_prefill`,\
      &#x20;`flashmla_kv`,\
      &#x20;`fa3`,\
      `tilelang`,\
      &#x20;`aiter`</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--fp8-gemm-backend`</td>
      <td>`auto`</td>
      <td>`auto`,\
      &#x20;`deep_gemm`,\
      &#x20;`flashinfer_trtllm`,\
      `flashinfer_cutlass`,\
      `flashinfer_deepgemm`,\
      `cutlass`,\
      &#x20;`triton`,\
      &#x20;`aiter`</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--disable-flashinfer-autotune`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>
  </tbody>
</table>

## Speculative decoding

Ascend NPU supports non-greedy target verification for the supported EAGLE-based speculative algorithms. Request-level
`temperature`, `top_k`, and `top_p` are applied when constructing the target probability distribution. Request-level
`top_k` controls token sampling and is separate from `--speculative-eagle-topk`, which controls the draft tree branching
factor.

The default target-only mode supports both linear and tree-shaped draft candidates. Classic rejection sampling is
available for EAGLE3 and NEXTN with `--speculative-use-rejection-sampling` and requires
`--speculative-eagle-topk 1`. Rejection sampling uses the draft proposal distribution and cannot be combined with
non-default speculative acceptance thresholds or deterministic inference.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-algorithm`</td>
      <td>`None`</td>
      <td>`EAGLE`,\
      &#x20;`EAGLE3`,\
      &#x20;`NEXTN`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-path`\
      `--speculative-draft-model`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-revision`</td>
      <td>`None`</td>
      <td>Type: str,\
      &#x20;`branch name`,\
      &#x20;`tag name`,\
      &#x20;`commit id`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-draft-load-format`</td>
      <td>`auto`</td>
      <td>`auto`,\
      &#x20;`dummy`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-num-steps`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-eagle-topk`</td>
      <td>`None`</td>
      <td>`1` (the only supported value on Ascend NPU)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-num-draft-tokens`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-dflash-block-size`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-single`</td>
      <td>`1.0`</td>
      <td>Type: float</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--speculative-accept-threshold-acc`</td>
      <td>`1.0`</td>
      <td>Type: float</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--speculative-use-rejection-sampling`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(requires `--speculative-eagle-topk 1`)</td>
      <td>Planned</td>
    </tr>

    <tr>
      <td>`--speculative-token-map`</td>
      <td>`None`</td>
      <td>Type: str; requires `--tp-size=1`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-attention-mode`</td>
      <td>`prefill`</td>
      <td>`prefill`,\
      &#x20;`decode`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-moe-runner-backend`</td>
      <td>`None`</td>
      <td>`auto`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-moe-a2a-backend`</td>
      <td>`None`</td>
      <td>`ascend_fuseep` (the only supported value on Ascend NPU)</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--speculative-draft-attention-backend`</td>
      <td>`None`</td>
      <td>`ascend`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-draft-window-size`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>Planned</td>
    </tr>

    <tr>
      <td>`--speculative-draft-model-quantization`</td>
      <td>`None`</td>
      <td>`unquant` (the only supported value for speculative decoding on Ascend NPU)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Ngram speculative decoding

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--speculative-ngram-min-match-window-size`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-match-window-size`</td>
      <td>`12`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-min-bfs-breadth`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-bfs-breadth`</td>
      <td>`10`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-match-type`</td>
      <td>`BFS`</td>
      <td>`BFS`,\
      &#x20;`PROB`</td>
      <td>Experimental. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion.</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-max-trie-depth`</td>
      <td>`18`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-capacity`</td>
      <td>`10000000`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-external-corpus-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-external-sam-budget`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-ngram-external-corpus-max-tokens`</td>
      <td>`10000000`</td>
      <td>Type: int</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--speculative-adaptive`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--speculative-adaptive-config`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Expert parallelism

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--expert-parallel-size`\
      `--ep-size`\
      `--ep`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--moe-a2a-backend`</td>
      <td>`none`</td>
      <td>`none`,\
      &#x20;`deepep`,\
      &#x20;`ascend_fuseep`(It is incompatible with eplb)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--moe-runner-backend`</td>
      <td>`auto`</td>
      <td>`auto`, `triton`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--flashinfer-mxfp4-moe-precision`</td>
      <td>`default`</td>
      <td>`default`,\
      &#x20;`bf16`,\
      &#x20;`fp8`</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--enable-flashinfer-allreduce-fusion`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--deepep-mode`</td>
      <td>`auto`</td>
      <td>`normal`, \
      `low_latency`,\
      &#x20;`auto`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--fuseep-mode`</td>
      <td>`2`</td>
      <td>`1`,\
      &#x20;`2`\
      (When the value is `1`, the batch size must be ≤ 256)</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--deepep-dispatcher-output-dtype`</td>
      <td>`auto`</td>
      <td>`auto`,\
      &#x20;`bf16`,\
      &#x20;`int8`\
      (When enabling DeepEP for a quantized model, set the dispatcher output dtype according to your model’s quantization.
      If the value is int8, you must also set the environment variable:DEEP\_NORMAL\_MODE\_USE\_INT8\_QUANT=1)</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--deepep-config`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--ep-num-redundant-experts`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--ep-dispatch-algorithm`</td>
      <td>`None`</td>
      <td>`static`,\
      &#x20;`dynamic`,\
      &#x20;`fake`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--init-expert-location`</td>
      <td>`trivial`</td>
      <td>`trivial`,\
      &#x20;`<path.pt>`,\
      &#x20;`<path.json>`,\
      &#x20;`<json_string>`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-eplb`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--eplb-algorithm`</td>
      <td>`deepseek`</td>
      <td>`auto`,\
      &#x20;`deepseek`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--eplb-rebalance-num-iterations`</td>
      <td>`1000`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--eplb-rebalance-layers-per-chunk`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--eplb-min-rebalancing-utilization-threshold`</td>
      <td>`1.0`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--expert-distribution-recorder-mode`</td>
      <td>`None`</td>
      <td>`stat`,\
      &#x20;`stat_approx`,\
      &#x20;`per_pass`,\
      &#x20;`per_token`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--expert-distribution-recorder-buffer-size`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--moe-dense-tp-size`</td>
      <td>`None`</td>
      <td>`1`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--elastic-ep-backend`</td>
      <td>`None`</td>
      <td>`none`, `mooncake`</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--mooncake-ib-device`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--enable-waterfill`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Mamba cache

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--max-mamba-cache-size`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mamba-ssm-dtype`</td>
      <td>`float32`</td>
      <td>`float32`,\
      `bfloat16`,\
      `float16`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mamba-full-memory-ratio`</td>
      <td>`0.9`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mamba-radix-cache-strategy`</td>
      <td>`auto`</td>
      <td>`auto`,\
      `no_buffer`,\
      `extra_buffer`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mamba-track-interval`</td>
      <td>`256`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Hierarchical cache

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-hierarchical-cache`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable).\
      &#x20;Currently, mamba cache is not supported.</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--hicache-ratio`</td>
      <td>`2.0`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--hicache-size`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--hicache-write-policy`</td>
      <td>`write_through`</td>
      <td>`write_back`,\
      &#x20;`write_through`,\
      &#x20;`write_through_selective`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--hicache-io-backend`</td>
      <td>`kernel`</td>
      <td>`kernel_ascend`,\
      &#x20;                    `direct`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--hicache-mem-layout`</td>
      <td>`layer_first`</td>
      <td>`page_first_direct`,\
      &#x20;                 `page_first_kv_split`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--hicache-storage-backend`</td>
      <td>`None`</td>
      <td>`file`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--hicache-storage-prefetch-policy`</td>
      <td>`timeout`</td>
      <td>`best_effort`,\
      &#x20;`wait_complete`,\
      &#x20; `timeout`</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--hicache-storage-backend-extra-config`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>
  </tbody>
</table>

## LMCache

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-lmcache`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--lmcache-config-file`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>
  </tbody>
</table>

## Diffusion LLM

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--dllm-algorithm`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--dllm-algorithm-config`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Offloading (must be used with `--disable-cuda-graph`)

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--cpu-offload-gb`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--offload-group-size`</td>
      <td>`-1`</td>
      <td>Type: int (DeepSeek only)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--offload-num-in-group`</td>
      <td>`1`</td>
      <td>Type: int (DeepSeek only)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--offload-prefetch-step`</td>
      <td>`1`</td>
      <td>Type: int (DeepSeek only)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--offload-mode`</td>
      <td>`cpu`</td>
      <td>`cpu` (DeepSeek only) \
      `meta` (DeepSeek only) \
      `sharded_gpu` (DeepSeek only, only support tp=1 dp>1)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Optimization/debug options

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--disable-radix-cache`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--cuda-graph-config`</td>
      <td>`None`</td>
      <td>Type: JSON</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--cuda-graph-backend-decode`</td>
      <td>`None`</td>
      <td>`full`,\
      &#x20;`disabled`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--cuda-graph-backend-prefill`</td>
      <td>`None`</td>
      <td>`disabled`\
      &#x20;(the former `tc_piecewise` prefill backend has been removed)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--cuda-graph-max-bs-decode`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--cuda-graph-max-bs-prefill`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--cuda-graph-bs-decode`</td>
      <td>`None`</td>
      <td>Type: List\[int]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--cuda-graph-bs-prefill`</td>
      <td>`None`</td>
      <td>Type: List\[int]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-prefill-cuda-graph`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-decode-cuda-graph`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-cuda-graph`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-cuda-graph-padding`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-profile-cuda-graph`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-cudagraph-gc`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-nccl-nvls`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--enable-symm-mem`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--disable-flashinfer-cutlass-moe-fp4-allgather`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--enable-tokenizer-batch-encode`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-tokenizer-batch-decode`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-custom-all-reduce`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-mscclpp`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--pre-warm-nccl`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-torch-symm-mem`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--disable-overlap-schedule`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-mixed-chunk`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-dp-attention`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-dp-attention-local-control-broadcast`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-dp-lm-head`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-two-batch-overlap`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Planned</td>
    </tr>

    <tr>
      <td>`--enable-single-batch-overlap`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--tbo-token-distribution-threshold`</td>
      <td>`0.48`</td>
      <td>Type: float</td>
      <td>Planned</td>
    </tr>

    <tr>
      <td>`--enable-p2p-check`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--triton-attention-reduce-in-fp32`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--triton-attention-num-kv-splits`</td>
      <td>`8`</td>
      <td>Type: int</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--triton-attention-split-tile-size`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--delete-ckpt-after-loading`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-memory-saver`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-weights-cpu-backup`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-draft-weights-cpu-backup`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--allow-auto-truncate`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-custom-logit-processor`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--flashinfer-mla-disable-ragged`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--disable-shared-experts-fusion`</td>
      <td>`True`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-chunked-prefix-cache`</td>
      <td>`True`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--image-processor-backend`</td>
      <td>`auto`</td>
      <td>`auto`, `torchvision`, `pil`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-fast-image-processor`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-return-hidden-states`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-return-routed-experts`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--scheduler-recv-interval`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--numa-node`</td>
      <td>`None`</td>
      <td>List\[int]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-deterministic-inference`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--rl-on-policy-target`</td>
      <td>`None`</td>
      <td>`fsdp`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disable-attn-tp-gather`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-layerwise-nvtx-marker`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--enable-attn-tp-input-scattered`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Experimental</td>
    </tr>

    <tr>
      <td>`--enable-prefill-cp`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)\
      (Currently only DeepSeek V4 is supported)</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--cp-strategy`</td>
      <td>`None`</td>
      <td>`zigzag`\
      (Currently only DeepSeek V4 is supported)</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--enable-fused-qk-norm-rope`</td>
      <td>`None`</td>
      <td>Type: bool</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--enable-precise-embedding-interpolation`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--gc-threshold`</td>
      <td>`None`</td>
      <td>List\[int]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Dynamic batch tokenizer

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-dynamic-batch-tokenizer`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--dynamic-batch-tokenizer-batch-size`</td>
      <td>`32`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--dynamic-batch-tokenizer-batch-timeout`</td>
      <td>`0.002`</td>
      <td>Type: float</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Debug tensor dumps

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--debug-tensor-dump-output-folder`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--debug-tensor-dump-layers`</td>
      <td>`None`</td>
      <td>List\[int]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--debug-tensor-dump-input-file`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## PD disaggregation

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--disaggregation-mode`</td>
      <td>`null`</td>
      <td>`null`,\
      &#x20;`prefill`,\
      &#x20;`decode`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disaggregation-transfer-backend`</td>
      <td>`mooncake`</td>
      <td>`ascend` (default `mooncake` not supported on NPU, must be specified manually)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disaggregation-bootstrap-port`</td>
      <td>`8998`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disaggregation-ib-device`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--disaggregation-decode-enable-offload-kvcache`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--num-reserved-decode-tokens`</td>
      <td>`512`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--disaggregation-decode-polling-interval`</td>
      <td>`1`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--optimistic-prefill-attempts`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Encode prefill disaggregation

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-adaptive-dispatch-to-encoder`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable adaptive dispatch)</td>
      <td>A2, A3 Series</td>
    </tr>

    <tr>
      <td>`--encoder-only`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to launch an encoder-only server)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--language-only`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to load weights for the language model only)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--encoder-transfer-backend`</td>
      <td>`zmq_to_scheduler`</td>
      <td>`zmq_to_scheduler`, \
      &#x20;`zmq_to_tokenizer`,\
      &#x20; `mooncake`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--encoder-urls`</td>
      <td>`[]`</td>
      <td>List\[str]\
      &#x20;(List of encoder server urls)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--encoder-bootstrap-port`</td>
      <td>`8997`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--encoder-register-urls`</td>
      <td>`[]`</td>
      <td>List\[str]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Custom weight loader

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--custom-weight-loader`</td>
      <td>`None`</td>
      <td>List\[str]</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--weight-loader-disable-mmap`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--weight-loader-prefetch-checkpoints`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--weight-loader-prefetch-num-threads`</td>
      <td>`4`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-seed-instance-ip`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-seed-instance-service-port`</td>
      <td>`None`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-send-weights-group-ports`</td>
      <td>`None`</td>
      <td>Type: JSON\
      &#x20;list</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-backend`</td>
      <td>`nccl`</td>
      <td>`transfer_engine`, \
      &#x20;`nccl`, \
      &#x20;`modelexpress`</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--weight-loader-drop-cache-after-load`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--remote-instance-weight-loader-start-seed-via-transfer-engine`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>
  </tbody>
</table>

## For PD-multiplexing

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-pdmux`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--pdmux-config-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>Special for GPU</td>
    </tr>

    <tr>
      <td>`--sm-group-num`</td>
      <td>`8`</td>
      <td>Type: int</td>
      <td>Special for GPU</td>
    </tr>
  </tbody>
</table>

## For multi-modal

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-broadcast-mm-inputs-process`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mm-process-config`</td>
      <td>`None`</td>
      <td>Type: JSON / Dict</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mm-enable-dp-encoder`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mm-processor-worker-num`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--mm-io-worker-num`</td>
      <td>`0`</td>
      <td>Type: int</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--limit-mm-data-per-request`</td>
      <td>`None`</td>
      <td>Type: JSON / Dict</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## For checkpoint decryption

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--decrypted-config-file`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--decrypted-draft-config-file`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-prefix-mm-cache`</td>
      <td>`False`</td>
      <td>bool flag\
      &#x20;(set to enable)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Forward hooks

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--forward-hooks`</td>
      <td>`None`</td>
      <td>Type: JSON list</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>

    <tr>
      <td>`--enable-quant-communications`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)\
      (When using DeepEP with a quantized model, you must set --deepep-dispatcher-output-dtype)</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## msProbe

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--msprobe-dump-config`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Configuration file support

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
      <th>Server supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--config`</td>
      <td>`None`</td>
      <td>Type: str</td>
      <td>A2, A3, 950PR/DT Series</td>
    </tr>
  </tbody>
</table>

## Other params

The following parameters are not supported because the required third-party components
(e.g., Ktransformer, checkpoint-engine) are not compatible with the NPU.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--checkpoint-engine-wait-weights-before-ready`</td>
      <td>`False`</td>
      <td>bool flag (set to enable)</td>
    </tr>

    <tr>
      <td>`--kt-weight-path`</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--kt-method`</td>
      <td>`AMXINT4`</td>
      <td>Type: str</td>
    </tr>

    <tr>
      <td>`--kt-cpuinfer`</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--kt-threadpool-count`</td>
      <td>2</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--kt-num-gpu-experts`</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>

    <tr>
      <td>`--kt-max-deferred-experts-per-token`</td>
      <td>`None`</td>
      <td>Type: int</td>
    </tr>
  </tbody>
</table>

The following parameters have known functional deficiencies in the upstream community codebase

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Defaults</th>
      <th>Options</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--tool-server`</td>
      <td>`None`</td>
      <td>Type: str</td>
    </tr>
  </tbody>
</table>


# Supported Models
Source: https://docs.sglang.io/docs/hardware-platforms/ascend-npus/reference/support_models



This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
Models, Diffusion Language Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included.
You are welcome to enable various models based on your business requirements.

## Large Language Models

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Models</th>
      <th>Model Family</th>
      <th>Ascend A2 Series Products Supported</th>
      <th>Ascend A3 Series Products Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Eco-Tech/DeepSeek-V4-Pro-w4a8-mtp</td>
      <td>DeepSeek</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp</td>
      <td>DeepSeek</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>sgl-npu/DeepSeek-V3.1-w8a8</td>
      <td>DeepSeek</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>sgl-npu/DeepSeek-V3.2-W8A8</td>
      <td>DeepSeek</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>sgl-npu/DeepSeek-R1-0528-W8A8</td>
      <td>DeepSeek</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>sgl-npu/DeepSeek-V2-Lite-W8A8</td>
      <td>DeepSeek</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3.8-2.4T-A95B</td>
      <td>Qwen3.8</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3.6-35B-A3B</td>
      <td>Qwen3.6</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Qwen3.6-27B-w8a8</td>
      <td>Qwen3.6</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp</td>
      <td>Qwen3.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Qwen3.5-122B-A10B-w8a8-mtp</td>
      <td>Qwen3.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Qwen3.5-35B-A3B-w8a8-mtp</td>
      <td>Qwen3.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Qwen3.5-27B-w8a8-mtp</td>
      <td>Qwen3.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3.5-9B</td>
      <td>Qwen3.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3.5-4B</td>
      <td>Qwen3.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3.5-0.8B</td>
      <td>Qwen3.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-30B-A3B-Instruct-2507</td>
      <td>Qwen3</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-32B</td>
      <td>Qwen3</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-0.6B</td>
      <td>Qwen3</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>sgl-npu/Qwen3-235B-A22B-W8A8</td>
      <td>Qwen3</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-Next-80B-A3B-Instruct</td>
      <td>Qwen3</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot</td>
      <td>Qwen3</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen2.5-7B-Instruct</td>
      <td>Qwen2.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>sgl-npu/QwQ-32B-W8A8</td>
      <td>QWQ</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>LLM-Research/Llama-4-Scout-17B-16E-Instruct</td>
      <td>Llama</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>AI-ModelScope/Llama-3.1-8B-Instruct</td>
      <td>Llama</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>LLM-Research/llama-2-7b</td>
      <td>Llama</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>LLM-Research/Llama-3.2-1B-Instruct</td>
      <td>Llama</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>mistralai/Mistral-7B-Instruct-v0.2</td>
      <td>Mistral</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>google/gemma-3-4b-it</td>
      <td>Gemma</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>microsoft/Phi-4-multimodal-instruct</td>
      <td>Phi</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>allenai/OLMoE-1B-7B-0924</td>
      <td>OLMoE</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>stabilityai/stablelm-2-1\_6b</td>
      <td>StableLM</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>CohereForAI/c4ai-command-r-v01</td>
      <td>Command-R</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>ZhipuAI/chatglm2-6b</td>
      <td>ChatGLM</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct</td>
      <td>ExaONE 3</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>xverse/XVERSE-MoE-A36B</td>
      <td>XVERSE</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>HuggingFaceTB/SmolLM-1.7B</td>
      <td>SmolLM</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/GLM-5.2-w8a8</td>
      <td>GLM-5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/GLM-5.1-w4a8</td>
      <td>GLM-5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/GLM-5-w4a8</td>
      <td>GLM-5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>ZhipuAI/glm-4-9b-chat</td>
      <td>GLM-4</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>iridiumine/MiMo-V2-Flash-W8A8</td>
      <td>MiMo</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>XiaomiMiMo/MiMo-7B-RL</td>
      <td>MiMo</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>arcee-ai/AFM-4.5B-Base</td>
      <td>Arcee AFM-4.5B</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Howeee/persimmon-8b-chat</td>
      <td>Persimmon</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>inclusionAI/Ling-lite</td>
      <td>Ling</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>ibm-granite/granite-3.1-8b-instruct</td>
      <td>Granite</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>AI-ModelScope/dbrx-instruct</td>
      <td>DBRX (Databricks)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>baichuan-inc/Baichuan2-13B-Chat</td>
      <td>Baichuan 2 (7B, 13B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>PaddlePaddle/ERNIE-4.5-21B-A3B-PT</td>
      <td>ERNIE-4.5 (4.5, 4.5MoE series)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>OpenBMB/MiniCPM3-4B</td>
      <td>MiniCPM (v3, 4B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>sgl-npu/Kimi-K3-W4A8</td>
      <td>Kimi</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Kimi-K2.6-w4a8</td>
      <td>Kimi</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/Kimi-K2.5-w4a8</td>
      <td>Kimi</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>moonshotai/Kimi-K2-Thinking</td>
      <td>Kimi</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>eigen-ai-labs/gpt-oss-120b-bf16</td>
      <td>GPTOSS</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>allenai/OLMo-2-1124-7B-Instruct</td>
      <td>OLMo</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Eco-Tech/MiniMax-M2.5-w8a8-QuaRot</td>
      <td>MiniMax-M2.5</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>cyankiwi/MiniMax-M2-BF16</td>
      <td>MiniMax-M2</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>upstage/SOLAR-10.7B-Instruct-v1.0</td>
      <td>Solar</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>bigcode/starcoder2-7b</td>
      <td>StarCoder2</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>arcee-ai/Trinity-Mini</td>
      <td>Trinity (Nano, Mini)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>OrionStarAI/Orion-14B-Base</td>
      <td>Orion (14B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>EleutherAI/gpt-j-6b</td>
      <td>GPT-J (6B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>ibm-granite/granite-4.0-micro</td>
      <td>IBM Granite 4.0 (Hybrid, Dense)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>poolside/Laguna-XS.2</td>
      <td>Laguna XS.2 (poolside)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Tencent-Hunyuan/Hy3</td>
      <td>Hunyuan</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
  </tbody>
</table>

## Multimodal Language Models

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Models</th>
      <th>Model Family (Variants)</th>
      <th>Ascend A2 Series Products Supported</th>
      <th>Ascend A3 Series Products Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Qwen/Qwen2.5-VL-3B-Instruct</td>
      <td>Qwen2.5-VL</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen2.5-VL-72B-Instruct</td>
      <td>Qwen2.5-VL</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-VL-30B-A3B-Instruct</td>
      <td>Qwen3-VL</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-VL-8B-Instruct</td>
      <td>Qwen3-VL</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-VL-4B-Instruct</td>
      <td>Qwen3-VL</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-VL-235B-A22B-Instruct</td>
      <td>Qwen3-VL</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>deepseek-ai/deepseek-vl2</td>
      <td>DeepSeek-VL2</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>deepseek-ai/Janus-Pro-1B</td>
      <td>Janus-Pro (1B, 7B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>deepseek-ai/Janus-Pro-7B</td>
      <td>Janus-Pro (1B, 7B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>OpenBMB/MiniCPM-V-2\_6</td>
      <td>MiniCPM-V / MiniCPM-o</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>OpenBMB/MiniCPM-o-2\_6</td>
      <td>MiniCPM-V / MiniCPM-o</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>google/gemma-3-4b-it</td>
      <td>Gemma 3 (Multimodal)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>mistralai/Mistral-Small-3.1-24B-Instruct-2503</td>
      <td>Mistral-Small-3.1-24B</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>microsoft/Phi-4-multimodal-instruct</td>
      <td>Phi-4-multimodal-instruct</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>XiaomiMiMo/MiMo-VL-7B-RL</td>
      <td>MiMo-VL (7B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>AI-ModelScope/llava-v1.6-34b</td>
      <td>LLaVA (v1.5 & v1.6)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>lmms-lab/llava-next-72b</td>
      <td>LLaVA-NeXT (8B, 72B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>moonshotai/Kimi-VL-A3B-Instruct</td>
      <td>Kimi-VL (A3B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>ZhipuAI/GLM-4.5V</td>
      <td>GLM-4.5V (106B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>LLM-Research/Llama-3.2-11B-Vision-Instruct</td>
      <td>Llama 3.2 Vision (11B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>rednote-hilab/dots.ocr</td>
      <td>DotsVLM-OCR</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-Omni-30B-A3B-Instruct</td>
      <td>Qwen3-Omni</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>stepfun-ai/Step3-VL-10B</td>
      <td>Step3-VL (10B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>deepseek-ai/DeepSeek-OCR-2</td>
      <td>DeepSeek-OCR / OCR-2</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-ASR-1.7B</td>
      <td>Qwen3-ASR (0.6B, 1.7B)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
  </tbody>
</table>

## Embedding Models

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Models</th>
      <th>Model Family</th>
      <th>Ascend A2 Series Products Supported</th>
      <th>Ascend A3 Series Products Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>intfloat/e5-mistral-7b-instruct</td>
      <td>E5 (Llama/Mistral based)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>iic/gte-Qwen2-1.5B-instruct</td>
      <td>GTE-Qwen2</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-Embedding-8B</td>
      <td>Qwen3-Embedding</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>iic/gme-Qwen2-VL-2B-Instruct</td>
      <td>GME (Multimodal)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>BAAI/bge-large-en-v1.5</td>
      <td>BGE</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
  </tbody>
</table>

## Reward Models

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Models</th>
      <th>Model Family</th>
      <th>Ascend A2 Series Products Supported</th>
      <th>Ascend A3 Series Products Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Skywork/Skywork-Reward-Llama-3.1-8B-v0.2</td>
      <td>Llama3.1 Reward</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Shanghai\_AI\_Laboratory/internlm2-7b-reward</td>
      <td>InternLM 2 Reward</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen2.5-Math-RM-72B</td>
      <td>Qwen2.5 Reward - Math</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Howeee/Qwen2.5-1.5B-apeach</td>
      <td>Qwen2.5 Reward - Sequence</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
  </tbody>
</table>

## Rerank Models

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Models</th>
      <th>Model Family</th>
      <th>Ascend A2 Series Products Supported</th>
      <th>Ascend A3 Series Products Supported</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>BAAI/bge-reranker-v2-m3</td>
      <td>BGE-Reranker</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-Reranker-8B</td>
      <td>Qwen3-Reranker (decoder-only yes/no)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Qwen/Qwen3-VL-Reranker-2B</td>
      <td>Qwen3-VL-Reranker (multimodal yes/no)</td>
      <td>✅</td>
      <td>✅</td>
    </tr>
  </tbody>
</table>


# CPU Servers
Source: https://docs.sglang.io/docs/hardware-platforms/cpu_server



The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers.
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
which are 4th generation or newer Intel® Xeon® Scalable Processors.

A number of popular LLMs are optimized and run efficiently on CPU,
including the most notable open-source models like Llama series, Qwen series,
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro)
in which the support status and example commands can be found.

## Installation

### Install Using Docker

It is recommended to use Docker for setting up the SGLang environment.

#### Pull from Docker Hub

Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository.
The [CPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xeon) end with `xeon` suffix.
The image pulling command is like:

```bash Command theme={null}
docker pull lmsysorg/sglang:v0.5.13-xeon
```

#### Build from Dockerfile

A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation from latest source code.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).

```bash Command theme={null}
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker

# Build the docker image
docker build -t sglang-cpu:latest -f xeon.Dockerfile .

# Initiate a docker container
docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=<secret>" \
    sglang-cpu:latest /bin/bash
```

### Install From Source

If you prefer to install SGLang in a bare metal environment,
the setup process is as follows:

Please install the required packages and libraries beforehand if
they are not already present on your system.
You can refer to the Ubuntu-based installation commands in
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L7)
for guidance.

1. Install `uv` package manager, then create and activate a virtual environment:

```bash Command theme={null}
# Taking '/opt' as the example uv env folder, feel free to change it as needed
cd /opt
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv --python 3.12
source .venv/bin/activate
```

2. Create a config file to direct the installation channel
   (a.k.a. index-url) of `torch` related packages:

```bash Command theme={null}
vim .venv/uv.toml
```

Press 'a' to enter insert mode of `vim`, paste the following content into the created file

```file theme={null}
[[index]]
name = "torch"
url = "https://download.pytorch.org/whl/cpu"

[[index]]
name = "torchvision"
url = "https://download.pytorch.org/whl/cpu"

[[index]]
name = "torchaudio"
url = "https://download.pytorch.org/whl/cpu"

[[index]]
name = "triton"
url = "https://download.pytorch.org/whl/cpu"

```

Save the file (in `vim`, press 'esc' to exit insert mode, then ':x+Enter'),
and set it as the default `uv` config.

```bash Command theme={null}
export UV_CONFIG_FILE=/opt/.venv/uv.toml
```

3. Clone the `sglang` source code and build the packages

```bash Command theme={null}
# Clone the SGLang code
git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout <YOUR-DESIRED-VERSION>

# Use dedicated toml file
cd python
cp pyproject_cpu.toml pyproject.toml
# Install SGLang dependent libs, and build SGLang main package
uv pip install --upgrade pip setuptools
uv pip install .  # If installing sgl-diffusion, please use: uv pip install ".[diffusion]"

# Build the CPU backend kernels
cd sglang/kernels/aot
cp pyproject_cpu.toml pyproject.toml
uv pip install .
```

4. Set the required environment variables

```bash Command theme={null}
export SGLANG_USE_CPU_ENGINE=1

# Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2
```

Notes:

* Note that the environment variable `SGLANG_USE_CPU_ENGINE=1`
  is required to enable the SGLang service with the CPU engine.

* If you encounter code compilation issues during the `sgl-kernel` building process,
  please check your `gcc` and `g++` versions and upgrade them if they are outdated.
  It is recommended to use `gcc-15` and `g++-15` as they have been verified
  in the official Docker container.

* The system library path is typically located in one of the following directories:
  `~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/`
  and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu`
  is used. Please adjust the path according to your server configuration.

* It is recommended to add the following to your `~/.bashrc` file to
  avoid setting these variables every time you open a new terminal:

  ```bash Command theme={null}
  source .venv/bin/activate
  export SGLANG_USE_CPU_ENGINE=1
  export LD_LIBRARY_PATH=<YOUR-SYSTEM-LIBRARY-FOLDER>
  export LD_PRELOAD=<YOUR-LIBS-PATHS>
  ```

## Launch of the Serving Engine

Example command to launch SGLang serving:

```bash Launch Server theme={null}
sglang serve                          \
    --model-path <MODEL_ID_OR_PATH>   \
    --trust-remote-code               \
    --disable-overlap-schedule        \
    --device cpu                      \
    --host 0.0.0.0                    \
    --tp 6
```

Notes:

1. For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`.

2. The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6).
   The number of TP specified is how many TP ranks will be used during the execution.
   On a CPU platform, a TP rank means a sub-NUMA cluster (SNC).
   Usually we can get the SNC information (How many available) from the Operating System with e.g. `lscpu` command.

   If the specified TP rank number differs from the total SNC count,
   the system will automatically utilize the first `n` SNCs.
   Note that `n` cannot exceed the total SNC number, doing so will result in an error.

   `SGLANG_CPU_OMP_THREADS_BIND` allows explicit control of CPU cores for each tensor parallel (TP) rank.

   **example 1**: Run SGLang service with TP=6, using the first 40 cores of each SNC on a Xeon® 6980P server,
   which has 43-43-42 cores on the 3 SNCs of a socket, we should set:

   ```bash Command theme={null}
   export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253"
   ```

   This configuration is equivalent to:

   * rank 0: `numactl -C 0-39 -m 0`
   * rank 1: `numactl -C 43-82 -m 1`
   * rank 2: `numactl -C 86-125 -m 2`
   * rank 3: `numactl -C 128-167 -m 3`
   * rank 4: `numactl -C 171-210 -m 4`
   * rank 5: `numactl -C 214-253 -m 5`

   **example 2**: Run SGLang service with TP=2, using 96 cores cross 3 SNCs on a Xeon® 6972P server,
   which has 32-32-32 cores on the 3 SNCs in a socket, we should set:

   ```bash Command theme={null}
   export SGLANG_CPU_OMP_THREADS_BIND="0-95|96-191"
   ```

   This configuration is equivalent to:

   * rank 0: `numactl -C 0-95 -m 0-2`
   * rank 1: `numactl -C 96-191 -m 3-5`

   Please beware that with SGLANG\_CPU\_OMP\_THREADS\_BIND set,
   the available memory amounts of the ranks may not be determined in prior.
   You may need to set proper `--max-total-tokens` to avoid the out-of-memory error.

3. For optimizing decoding with torch.compile, please add the flag `--enable-torch-compile`.
   To specify the maximum batch size when using `torch.compile`, set the flag `--torch-compile-max-bs`.
   For example, `--enable-torch-compile --torch-compile-max-bs 4` means using `torch.compile`
   and setting the maximum batch size to 4.

4. A warmup step is automatically triggered when the service is started.
   The server is ready when you see the log `The server is fired up and ready to roll!`.

## Benchmarking with Requests

You can benchmark the performance via the `bench_serving` script.
Run the command in another terminal. An example command would be:

```bash Run Benchmark theme={null}
python -m sglang.bench_serving   \
    --dataset-name random        \
    --random-input-len 1024      \
    --random-output-len 1024     \
    --num-prompts 1              \
    --request-rate inf           \
    --random-range-ratio 1.0
```

Detailed parameter descriptions are available via the command:

```bash Benchmark Help theme={null}
python -m sglang.bench_serving -h
```

Additionally, requests can be formatted using
[the OpenAI Completions API](../basic_usage/openai_api_completions)
and sent via the command line (e.g., using `curl`) or through your own scripts.

## Example Usage Commands

Large Language Models can range from fewer than 1 billion to several hundred billion parameters.
Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors
with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer,
or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common
4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors.

### Example: Running DeepSeek-V3.1-Terminus

An example command to launch service of W8A8\_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server:

```bash W8A8_INT8 theme={null}
sglang serve                                                        \
    --model-path IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
    --trust-remote-code                                             \
    --disable-overlap-schedule                                      \
    --device cpu                                                    \
    --quantization w8a8_int8                                        \
    --enable-torch-compile                                          \
    --torch-compile-max-bs 4                                        \
    --host 0.0.0.0                                                  \
    --tp 6
```

Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be:

```bash FP8 theme={null}
sglang serve                                         \
    --model-path deepseek-ai/DeepSeek-V3.1-Terminus  \
    --trust-remote-code                              \
    --disable-overlap-schedule                       \
    --device cpu                                     \
    --enable-torch-compile                           \
    --torch-compile-max-bs 4                         \
    --host 0.0.0.0                                   \
    --tp 6
```

Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment.
The value `4` in the examples is illustrative.

### Example: Running Llama-3.2-3B

An example command to launch service of Llama-3.2-3B with BF16 precision:

```bash BF16 theme={null}
sglang serve                                         \
    --model-path meta-llama/Llama-3.2-3B-Instruct    \
    --trust-remote-code                              \
    --disable-overlap-schedule                       \
    --device cpu                                     \
    --enable-torch-compile                           \
    --torch-compile-max-bs 16                        \
    --host 0.0.0.0                                   \
    --tp 3
```

The example command to launch service of W8A8\_INT8 version of Llama-3.2-3B:

```bash W8A8_INT8 theme={null}
sglang serve                                            \
    --model-path RedHatAI/Llama-3.2-3B-quantized.w8a8   \
    --trust-remote-code                                 \
    --disable-overlap-schedule                          \
    --device cpu                                        \
    --quantization w8a8_int8                            \
    --enable-torch-compile                              \
    --torch-compile-max-bs 16                           \
    --host 0.0.0.0                                      \
    --tp 3
```

Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup.
For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server.

Once the server have been launched, you can test it using the `bench_serving` command or create
your own commands or scripts following [the benchmarking example](#benchmarking-with-requests).


# Moore Threads GPUs
Source: https://docs.sglang.io/docs/hardware-platforms/mthreads_gpu



This document describes how to run SGLang on Moore Threads GPUs. If you
encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).

SGLang uses the MUSA runtime through
[`torch_musa`](https://github.com/MooreThreads/torch_musa) and
[`torchada`](https://github.com/MooreThreads/torchada). The
`python[all_musa]` extra installs the MUSA torch, Triton, TileLang, MATE, and
runtime Python stack from the Moore Threads package index, with public PyPI as
the fallback for common Python dependencies.

## Prerequisites

Install the Moore Threads driver and MUSA toolkit before installing SGLang. For
MTT S5000, follow the [official driver installation guide](https://docs.mthreads.com/driver-linux-server/driver-linux-server-doc-online/MTT_S5000/install_guide).

## Install SGLang

You can install SGLang using one of the methods below.

### Install from Source

Start from an environment with the prerequisites above installed.

```bash Command theme={null}
git clone https://github.com/sgl-project/sglang.git
cd sglang

python -m pip install --upgrade pip "setuptools<82" wheel
cp python/pyproject_other.toml python/pyproject.toml
python -m pip install -e "python[all_musa]" \
  --index-url https://dl.mthreads.com/repo/api/pypi/pypi/simple \
  --extra-index-url https://pypi.org/simple \
  --trusted-host dl.mthreads.com \
  --no-build-isolation

# Ensure the MUSA Triton wheel (with triton.backends.mtgpu) is installed.
python -m pip install --no-cache-dir --force-reinstall --no-deps \
  --index-url https://dl.mthreads.com/repo/api/pypi/pypi/simple \
  --trusted-host dl.mthreads.com \
  triton==3.2.0
python -c 'import triton.backends.mtgpu'

cd python/sglang/kernels/aot
cp pyproject_musa.toml pyproject.toml
MTGPU_TARGET=mp_31 python setup_musa.py install
```

### Install Using Docker

The Dockerfile installs the MUSA runtime and Python stack declared by
`python/pyproject_other.toml`.

```bash Command theme={null}
git clone https://github.com/sgl-project/sglang.git
cd sglang

docker build -f docker/musa.Dockerfile -t sglang:main-musa .
```

Run the image with MUSA devices exposed by the host container toolkit.

```bash Command theme={null}
alias drun='docker run -it --rm --network=host \
  --env MTHREADS_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  --env MTHREADS_DRIVER_CAPABILITIES=all \
  --shm-size=32g \
  -v ~/.cache/huggingface:/root/.cache/huggingface'

drun sglang:main-musa \
  python3 -c 'import triton.backends.mtgpu; print("MUSA Triton backend OK")'
```

Launch the server:

```bash Command theme={null}
drun sglang:main-musa \
  python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --host 0.0.0.0 \
  --port 30000
```

When the server displays `The server is fired up and ready to roll!`, startup is
successful.

## Verify

You can send an OpenAI-compatible request to the engine:

```bash Command theme={null}
curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 32
  }'
```

Or run a benchmark:

```bash Command theme={null}
drun sglang:main-musa \
  python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 4000 \
  --random-input 128 \
  --random-output 128
```


# NVIDIA GPUs
Source: https://docs.sglang.io/docs/hardware-platforms/nvidia-gpus



Please refer to the [Installation Guide](../get-started/install) to get started with SGLang on NVIDIA GPUs.


# NVIDIA Jetson Orin
Source: https://docs.sglang.io/docs/hardware-platforms/nvidia_jetson

Guide for installing and running SGLang on NVIDIA Jetson Orin devices.

## Prerequisites

Before starting, ensure the following:

* [**NVIDIA Jetson AGX Orin Devkit**](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with **JetPack 6.1** or later.
* **CUDA Toolkit** and **cuDNN** are installed.
* Verify that the Jetson AGX Orin is in **high-performance mode**:

```bash theme={null}
sudo nvpmodel -m 0
```

***

## Installing and running SGLang with Jetson Containers

Clone the jetson-containers github repository:

```bash theme={null}
git clone https://github.com/dusty-nv/jetson-containers.git
```

Run the installation script:

```bash theme={null}
bash jetson-containers/install.sh
```

Build the container image:

```bash theme={null}
jetson-containers build sglang
```

Run the container:

```
jetson-containers run $(autotag sglang)
```

Or you can also manually run a container with this command:

```
docker run --runtime nvidia -it --rm --network=host IMAGE_NAME
```

***

## Running Inference

Launch the server:

```bash theme={null}
python -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
  --device cuda \
  --dtype half \
  --attention-backend flashinfer \
  --mem-fraction-static 0.8 \
  --context-length 8192
```

The quantization and limited context length (`--dtype half --context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments).

After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#usage) to test the usability.

***

## Structured output with XGrammar

Please refer to [SGLang doc structured output](../advanced_features/structured_outputs).

***

Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez).

## References

* [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin)


# Hardware Platforms
Source: https://docs.sglang.io/docs/hardware-platforms/overview

Platform-specific guides for running SGLang on GPUs, TPUs, NPUs, CPUs, and more.

* [NVIDIA GPUs](./nvidia-gpus)
* [AMD GPUs](./amd_gpu)
* [Ascend NPUs](./ascend-npus/getting-started/installation)
* [CPU Server](./cpu_server)
* [NVIDIA Jetson Orin](./nvidia_jetson)
* [TPU](./tpu)
* [XPU](./xpu)
* [Moore Threads GPUs](./mthreads_gpu)


# SGLang Plugin System
Source: https://docs.sglang.io/docs/hardware-platforms/plugin



## Overview

Allows hardware vendors and developers to extend SGLang **without modifying the main repository code**.

The framework provides two plugin types, both discovered via Python's standard `setuptools` entry\_points:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Plugin Type</th>
      <th>Entry Point Groups</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Hardware Platform Plugin**</td>
      <td>`sglang.srt.platforms`\
      `sglang.multimodal_gen.platforms`</td>
      <td>Register a custom hardware platform (device operations, KV cache pools, attention backends, graph capture, compilation backends, etc.)</td>
    </tr>

    <tr>
      <td>**General Plugin**</td>
      <td>`sglang.srt.plugins`\
      `sglang.multimodal_gen.plugins`</td>
      <td>Inject hooks (before/after/around/replace) into any function/method, or replace entire classes</td>
    </tr>
  </tbody>
</table>

### Principles

* **Non-intrusive**: Built-in platforms remain the fallback when no OOT platform activates.
* **Install-time discovery**: Plugins are discovered from Python entry points after installation.
* **Environment variable control**: `SGLANG_PLATFORM` selects an SRT platform, `SGLANG_DIFFUSION_PLATFORM_OVERRIDE` selects a diffusion platform, and `SGLANG_PLUGINS` filters general plugins in either hook group.

### Current scope

The platform plugin system targets **out-of-tree (OOT) hardware platforms**. Diffusion support is experimental and covers the seams documented below, not every device-specific branch.

## Architecture

### Runtime-specific platform interfaces

SRT and diffusion have separate platform base classes and platform identity types in `sglang.srt.platforms` and `sglang.multimodal_gen.runtime.platforms`. A package supporting both runtimes should define separate platform classes and use each runtime's own `PlatformEnum.OOT` value.

Each platform entry point resolves to a zero-argument activation callback:

```python theme={null}
def activate() -> str | None:
    """Return the platform class qualname when this hardware is available."""
```

Return a fully qualified class name when the provider can run, or `None` otherwise. Keep activation import-safe: do not access `current_platform` or initialize runtime or device state. Put required backend setup in the platform's `init_backend()` method and reserve general plugins for hooks that the platform interface cannot express.

Explicit selection enumerates entry-point metadata and imports only the selected callback. Automatic selection invokes installed platform callbacks to determine which provider is active.

Diffusion reserves `cpu`, `cuda`, `rocm`, `xpu`, `mps`, `npu`, and `musa`. Automatic discovery also rejects duplicate entry-point names; explicit selection validates only the selected name and does not import unrelated providers.

#### SRT selection

`current_platform` is a **lazy singleton** in `sglang.srt.platforms`. On first access it resolves the active platform through the following priority chain:

```text theme={null}
entry_points("sglang.srt.platforms")  → Enumerate ALL plugins by name (metadata only)
  │
  ├─ SGLANG_PLATFORM set (front-loading filter):
  │   ├─ Name not found in discovered → RuntimeError
  │   ├─ activate() returns non-None  → load that platform
  │   └─ activate() returns None      → RuntimeError (hardware unavailable)
  │
  └─ SGLANG_PLATFORM unset (auto-discover, activate all):
      ├─ 0 activated + SGLANG_USE_CPU_ENGINE=1 → fallback CpuSRTPlatform
      ├─ 0 activated + CUDA available          → fallback CudaSRTPlatform
      ├─ 0 activated + ROCm available          → fallback RocmSRTPlatform
      ├─ 0 activated + XPU available           → fallback XpuSRTPlatform
      ├─ 0 activated + none of the above       → fallback base SRTPlatform
      ├─ 1 activated → use it
      └─ N activated → RuntimeError (must set SGLANG_PLATFORM)
```

#### Diffusion selection

SGLang Diffusion resolves its platform in this order:

1. When `SGLANG_DIFFUSION_PLATFORM_OVERRIDE` names `cpu`, `cuda`, `rocm`, `mps`, `npu`, or `musa`, select that built-in platform without hardware probing. XPU remains automatic-only, preserving the existing selector behavior.
2. When it contains another name, load only the matching `sglang.multimodal_gen.platforms` entry point. An unknown name or a callback that returns `None` is an error.
3. When it is unset, activate installed diffusion platform plugins. No active plugin continues to built-in detection, one selects that plugin, and multiple active plugins raise an error that asks you to set the selector. An activation callback that raises aborts startup rather than falling back, so a broken vendor runtime cannot silently run the job on a built-in platform.
4. Try built-in platforms in order: MPS, XPU, ROCm, CUDA, NPU, MUSA, then CPU.

The existing override variable is therefore the single explicit selector for supported built-in aliases and OOT entry-point names.

Selection resolves lazily, the first time anything in a process touches `current_platform`, so it needs no call site and happens in every process automatically. SGLang Diffusion also records which distribution supplied the selected platform, and skips the hooks of every other installed platform package.

#### Required platform initialization vs. optional hooks

The two mechanisms have different failure semantics, and it matters which one you use:

| | Platform contract | General plugin |
| - | - | - |
| Entry-point group | `sglang.multimodal_gen.platforms` | `sglang.multimodal_gen.plugins` |
| Delivery | methods on your `Platform` subclass | hooks that monkey-patch a target |
| Activation | lazy `current_platform`, plus guarded `init_backend()` in each worker | registration with `load_plugins()`, then explicit `apply_plugin_hooks()` per process |
| A failure | aborts startup | aborts startup if the plugin ships in the selected platform's distribution; otherwise logged |

Anything your hardware needs in order to be correct belongs on the `Platform` subclass, so a broken platform cannot silently serve. Reach for a general plugin only when the `Platform` interface has no seam for what you need — and please report that gap.

A plugin shipped in the selected platform's own distribution is treated as part of that platform's contract: a failure to load it, to run its callback, or to apply any hook it registered aborts startup rather than leaving the platform half-initialized. Plugins from any other installed package stay best-effort, so a broken third party cannot take the server down. An explicit `SGLANG_PLUGINS` allowlist can disable any general plugin, including one from the selected platform package; required hardware setup therefore belongs in `init_backend()`.

### Plugin Loading Flow

Each runtime has a process-local hook registry. SRT retains its single
`load_plugins()` activation step. Diffusion separates registration
(`load_plugins()`) from target resolution (`apply_plugin_hooks()`), because
resolving a dotted hook target can import that target's entire module graph.
Both runtimes honor `SGLANG_PLUGINS`.

The loader is called at these SRT entry points:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Call Site</th>
      <th>Process</th>
      <th>Timing</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`cli/serve.py` `serve()`</td>
      <td>Main</td>
      <td>Before `prepare_server_args()`</td>
    </tr>

    <tr>
      <td>`launch_server.py` `main`</td>
      <td>Main</td>
      <td>Before `prepare_server_args()`</td>
    </tr>

    <tr>
      <td>`engine.py` `_launch_subprocesses()`</td>
      <td>Main</td>
      <td>Before `server_args.check_server_args()`</td>
    </tr>

    <tr>
      <td>`scheduler.py` `run_scheduler_process()`</td>
      <td>Subprocess</td>
      <td>Before `Scheduler()` construction</td>
    </tr>
  </tbody>
</table>

> **Note**: Diffusion plugin registration and hook application each run once per process. Spawned subprocesses start from a blank interpreter and establish their own hooks — nothing the parent patched survives the spawn boundary. SRT's `load_plugins()` performs both phases in one call.

```text theme={null}
load_plugins()
  ├── _get_excluded_dists()                       → compute dists to skip (via SGLANG_PLATFORM)
  ├── load_plugins_by_group("sglang.srt.plugins",     → discover entry_points, filter by SGLANG_PLUGINS
  │     excluded_dists=...)                          skip plugins from unselected platform packages
  ├── for each plugin:                            → set _current_plugin_source context var
  │     func()                                      side effects (register hooks with source tracking)
  └── HookRegistry.apply_hooks()                  → monkey-patch targets
```

Diffusion launchers call `apply_plugin_hooks()`, which first performs registration
if necessary and then resolves and patches targets. Scheduler children use a
stricter lifecycle in `runtime/managers/worker_bootstrap.py`:

```text theme={null}
spawn unpickles SchedulerProcessSpec (stdlib types + opaque ServerArgs bytes)
  -> initialize_current_platform()
  -> load_plugins()                 # discover callbacks and register hooks only
  -> apply_plugin_hooks()           # target imports are allowed from here onward
  -> materialize ServerArgs
  -> resolve and invoke the patched run_scheduler_process
```

HTTP children likewise finish plugin registration and hook application before
importing `runtime.launch_server` or materializing `ServerArgs`. This matters for
class replacements and other hooks whose effects cannot be retroactively applied
to classes or registrations created while importing the server module graph.

The bootstrap module is the `mp.Process` target and imports no diffusion runtime
module at module scope. `ServerArgs` is serialized inside `ServerArgsPayload`, so
the multiprocessing unpickler cannot import pipeline configuration modules
before the target starts. Workers always use a local `spawn` context, independent
of an embedding application's global multiprocessing setting.

This ordering makes backend initialization precede plugin callback imports,
hook-target resolution, `ServerArgs` materialization, and worker imports. A
platform activation module necessarily loads before its own `init_backend()`;
activation modules must therefore remain import-safe. Prefer platform methods
and registries for required behavior.

Each activation phase runs once per process behind a lock: a re-entrant call
from a plugin callback returns and lets the outer call finish, another thread
waits for it, and a failure is terminal. A callback must not hand activation to
a second thread and join it — that deadlocks.

#### Offline scripts and the spawn boundary

`spawn` re-executes the launching script's module scope in every child *before*
it unpickles the target's arguments, so an offline script's own imports run
ahead of that child's platform initialization and `ServerArgsPayload` cannot
help. The supported script layout is:

```python theme={null}
from sglang.multimodal_gen import DiffGenerator  # a proxy: imports nothing yet

if __name__ == "__main__":
    generator = DiffGenerator.from_pretrained(model_path="...")
```

`DiffGenerator` on the `sglang.multimodal_gen` facade is a lazy proxy, so
binding it at module scope costs no diffusion import and the child still reaches
`initialize_current_platform()` with a clean module table. Module scope also
stays open to `envs` and `runtime.platforms`, including `runtime.platforms.plugins`.
These modules remain import-safe so a plugin can subclass `Platform` and register
hooks before backend initialization. Everything else, including `SamplingParams` and `PipelineConfig`
from the same facade, belongs inside the `if __name__ == "__main__":` guard or
inside the function that uses it. A child
that finds runtime modules already imported names them in a warning: hook
application can still patch those modules, but classes and registrations
created while importing them are already past reach.

***

## Plugin Type 1: Hardware Platform Plugin

### Description

A hardware platform plugin registers an SRT `SRTPlatform` subclass, a diffusion `Platform` subclass, or both. The selected class tells that runtime how to interact with a specific hardware backend.

### SRT quick start

**1. Create a minimal package:**

```
my_platform_plugin/
├── pyproject.toml
└── my_platform_plugin/
    ├── __init__.py    # activate() function
    ├── device.py      # MyDeviceMixin
    └── platform.py    # MySRTPlatform
```

**2. `pyproject.toml`:**

```toml theme={null}
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "my-platform-plugin"
version = "0.1.0"

[project.entry-points."sglang.srt.platforms"]
my_device = "my_platform_plugin:activate"
```

**3. `__init__.py`** — activation function:

```python theme={null}
def activate():
    """Return fully-qualified class name to activate, or None to skip."""
    if _my_device_is_available():
        return "my_platform_plugin.platform.MySRTPlatform"
    return None
```

**4. `device.py`** — device mixin:

```python theme={null}
from sglang.srt.platforms.device_mixin import DeviceMixin, PlatformEnum

class MyDeviceMixin(DeviceMixin):
    _enum = PlatformEnum.OOT
    device_name = "my_device"
    device_type = "my_device"   # torch device type

    def set_device(self, device) -> None: ...
    def get_device_name(self, device_id=0) -> str: ...
    def get_device_total_memory(self, device_id=0) -> int: ...
    def get_current_memory_usage(self, device=None) -> float: ...
    def get_device_capability(self, device_id=0): ...
    def get_torch_distributed_backend_str(self) -> str: ...
```

**5. `platform.py`** — SRT platform:

```python theme={null}
from sglang.srt.platforms.interface import PlatformCapabilities, SRTPlatform
from my_platform_plugin.device import MyDeviceMixin

class MySRTPlatform(SRTPlatform, MyDeviceMixin):
    capabilities = PlatformCapabilities(supports_triton=False, graph_capture=True)

    def get_default_attention_backend(self) -> str: ...
    # ... override other methods as needed; vendor-private helpers start with `_`
```

Every public name on the subclass (and on any mixin it composes) must exist on
`SRTPlatform` with a compatible signature. The check runs when the class is
defined, so a hook that core renamed or never had raises `TypeError` naming it
instead of silently never being called.

**6. Install and verify:**

```bash theme={null}
pip install -e my_platform_plugin/
python -c "from sglang.srt.platforms import current_platform; print(current_platform)"
```

### SRT platform interface reference

#### Identity Queries (from DeviceMixin)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`is_cuda()`</td>
      <td>Based on `_enum`</td>
      <td>Whether this is an NVIDIA CUDA platform</td>
    </tr>

    <tr>
      <td>`is_rocm()`</td>
      <td>Based on `_enum`</td>
      <td>Whether this is an AMD ROCm platform</td>
    </tr>

    <tr>
      <td>`is_npu()`</td>
      <td>Based on `_enum`</td>
      <td>Whether this is a Huawei NPU platform</td>
    </tr>

    <tr>
      <td>`is_cpu()`</td>
      <td>Based on `_enum`</td>
      <td>Whether this is a CPU-only platform</td>
    </tr>

    <tr>
      <td>`is_xpu()`</td>
      <td>Based on `_enum`</td>
      <td>Whether this is an Intel XPU platform</td>
    </tr>

    <tr>
      <td>`is_musa()`</td>
      <td>Based on `_enum`</td>
      <td>Whether this is a Moore Threads MUSA platform</td>
    </tr>

    <tr>
      <td>`is_cuda_alike()`</td>
      <td>CUDA+ROCM+MUSA</td>
      <td>True if the hardware supports CUDA-like APIs</td>
    </tr>

    <tr>
      <td>`is_out_of_tree()`</td>
      <td>`True` for OOT</td>
      <td>Automatically detected based on `_enum = PlatformEnum.OOT`</td>
    </tr>
  </tbody>
</table>

#### Device Operations (from DeviceMixin)

> Methods annotated **\[Active]** are called by SGLang core through `current_platform` — OOT implementations take effect immediately.
> Methods annotated **\[Planned]** are reserved interfaces — SGLang core still uses hardcoded calls (e.g. `torch.cuda.empty_cache()`). OOT implementations will NOT take effect until the core is migrated in a future PR.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Default</th>
      <th>Status</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`get_device(local_rank)`</td>
      <td>`raise NotImplementedError`</td>
      <td>Planned</td>
      <td>Return `torch.device` for a given local rank</td>
    </tr>

    <tr>
      <td>`set_device(device)`</td>
      <td>`raise NotImplementedError`</td>
      <td>Planned</td>
      <td>Set the current device</td>
    </tr>

    <tr>
      <td>`get_device_name(device_id)`</td>
      <td>`raise NotImplementedError`</td>
      <td>Planned</td>
      <td>Get human-readable device name</td>
    </tr>

    <tr>
      <td>`get_device_uuid(device_id)`</td>
      <td>`raise NotImplementedError`</td>
      <td>Planned</td>
      <td>Get unique device identifier</td>
    </tr>

    <tr>
      <td>`get_device_capability(device_id)`</td>
      <td>`raise NotImplementedError`</td>
      <td>Planned</td>
      <td>Get `DeviceCapability(major, minor)`. None if N/A</td>
    </tr>

    <tr>
      <td>`empty_cache()`</td>
      <td>`pass`</td>
      <td>Planned</td>
      <td>Release cached device memory</td>
    </tr>

    <tr>
      <td>`synchronize()`</td>
      <td>`pass`</td>
      <td>Planned</td>
      <td>Synchronize device operations</td>
    </tr>

    <tr>
      <td>`get_device_total_memory(device_id)`</td>
      <td>`raise NotImplementedError`</td>
      <td>**Active**</td>
      <td>Get total device memory in bytes</td>
    </tr>

    <tr>
      <td>`get_available_memory(device_id)`</td>
      <td>`raise NotImplementedError`</td>
      <td>Planned</td>
      <td>Return `(free_bytes, total_bytes)`</td>
    </tr>

    <tr>
      <td>`get_current_memory_usage(device)`</td>
      <td>`raise NotImplementedError`</td>
      <td>**Active**</td>
      <td>Get current peak memory usage in bytes</td>
    </tr>

    <tr>
      <td>`is_pin_memory_available(device=None)`</td>
      <td>`False`</td>
      <td>**Active**</td>
      <td>Whether pinned host memory is available for a target device</td>
    </tr>

    <tr>
      <td>`get_torch_distributed_backend_str()`</td>
      <td>`raise NotImplementedError`</td>
      <td>Planned</td>
      <td>Distributed backend string (e.g. "nccl", "hccl")</td>
    </tr>

    <tr>
      <td>`get_communicator_class()`</td>
      <td>`None`</td>
      <td>Planned</td>
      <td>Platform-specific communicator class</td>
    </tr>

    <tr>
      <td>`inference_mode()`</td>
      <td>`torch.inference_mode(True)`</td>
      <td>Planned</td>
      <td>Return inference mode context manager</td>
    </tr>

    <tr>
      <td>`seed_everything(seed)`</td>
      <td>Set random/np/torch seeds</td>
      <td>Planned</td>
      <td>Set random seeds for reproducibility</td>
    </tr>

    <tr>
      <td>`verify_quantization(quant)`</td>
      <td>`pass`</td>
      <td>Planned</td>
      <td>Validate quantization method support</td>
    </tr>

    <tr>
      <td>`get_cpu_architecture()`</td>
      <td>Auto-detect x86/arm</td>
      <td>Planned</td>
      <td>Detect CPU architecture (`CpuArchEnum`)</td>
    </tr>
  </tbody>
</table>

#### Types (from DeviceMixin)

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Type</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`PlatformEnum`</td>
      <td>Enumeration of platform types: CUDA, ROCM, CPU, XPU, MUSA, NPU, TPU, MPS, OOT, UNSPECIFIED</td>
    </tr>

    <tr>
      <td>`CpuArchEnum`</td>
      <td>CPU architecture: X86, ARM, UNSPECIFIED</td>
    </tr>

    <tr>
      <td>`DeviceCapability`</td>
      <td>`NamedTuple(major, minor)` with comparison support. Methods: `as_version_str()`, `to_int()`</td>
    </tr>
  </tbody>
</table>

#### Capabilities (from SRTPlatform)

`capabilities` is a frozen `PlatformCapabilities` class attribute. Core reads it
before constructing anything and picks a code path the platform can serve; every
field defaults to the conservative answer.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Field</th>
      <th>Default</th>
      <th>What core does with it</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`supports_triton`</td>
      <td>`False`</td>
      <td>When False, the paged allocator and the req-to-token writers use their torch-native paths instead of Triton kernels.</td>
    </tr>

    <tr>
      <td>`graph_capture`</td>
      <td>`False`</td>
      <td>Runs the decode graph runner (`get_graph_runner_cls()`).</td>
    </tr>

    <tr>
      <td>`hicache_device_kernels`</td>
      <td>`False`</td>
      <td>The sgl\_kernel HiCache transfer / write-back kernels are available; otherwise host pools skip their staging buffers.</td>
    </tr>

    <tr>
      <td>`supports_pynccl`</td>
      <td>`False`</td>
      <td>Model-parallel groups create a PyNccl communicator when the caller does not choose. Set it to True only if the device runs NCCL (or an NCCL-compatible library PyNccl loads).</td>
    </tr>
  </tbody>
</table>

#### Subsystem Factory Methods (from SRTPlatform)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`get_default_attention_backend()`</td>
      <td>`raise NotImplementedError`</td>
      <td>Default attention backend name</td>
    </tr>

    <tr>
      <td>`get_graph_runner_cls()`</td>
      <td>`return None`</td>
      <td>Graph Runner class. Adaptive speculative decoding also builds one target verify runner per step count from it, passing `attn_backend`, `speculative_num_steps` and `speculative_num_draft_tokens` as keyword arguments.</td>
    </tr>

    <tr>
      <td>`get_speculative_graph_runner_cls(algorithm, phase)`</td>
      <td>`return None`</td>
      <td>EAGLE draft graph runner for `"draft_decode"` or `"draft_extend"` on the built-in worker, constructed like the built-in runner for that phase. `None` runs the phase eagerly; not consulted when `capabilities.graph_capture` is False.</td>
    </tr>

    <tr>
      <td>`get_speculative_worker_cls(algorithm)`</td>
      <td>`return None`</td>
      <td>Worker class that replaces the built-in speculative worker, for any algorithm. It is constructed with the built-in worker's keyword arguments, must expose the same scheduler-facing interface, and must reject flags it cannot serve (such as multi-layer EAGLE). `--speculative-adaptive` falls back to static parameters with a platform worker, since only the built-in worker implements it. EAGLE, EAGLE3, NEXTN (asked as `"EAGLE"`) and STANDALONE also require `supports_speculative_algorithm(algorithm)` to return True.</td>
    </tr>

    <tr>
      <td>`get_kv_pool_cls(kind=...)`</td>
      <td>`return None`</td>
      <td>Return this platform's KV pool class for `kind` in `"mha"` / `"mla"` / `"dsa"`, or `None` for the in-tree class. The class must subclass `MHATokenToKVPool` / `MLATokenToKVPool` / `DSATokenToKVPool`; core constructs it with exactly the keyword arguments the in-tree class receives, at every site the in-tree class is built (standalone, inside the SWA composite, as the hybrid-linear full-attention leaf). Override `_create_buffers` to change how the backing storage is carved -- addressing, index translation and PD registration stay core's. If you override `_create_buffers` entirely you also own `_kv_buffer_descs` (PD transfer and post-capture backing read it); override `_create_buffers_normal` or `_kv_buffer_shapes` instead to keep the base descriptors, which require the leading buffer axis to be tokens or pages. The in-tree pool allocates on `device` and is correct on any torch device, so most platforms return `None`.</td>
    </tr>

    <tr>
      <td>`get_paged_allocator_cls()`</td>
      <td>`return None`</td>
      <td>Paged allocator class, honored at every paged-allocator construction site. With `capabilities.supports_triton=False` the in-tree allocator already uses torch-native kernels, so most platforms return `None`. At `page_size == 1` core uses the flat allocator on every platform.</td>
    </tr>

    <tr>
      <td>`get_quantization_config(quantization)`</td>
      <td>`return None`</td>
      <td>Return hardware-specific quantization config for the specific quantization scheme, raise an error if not supported or return None to use the default config.</td>
    </tr>

    <tr>
      <td>`get_compile_backend(mode)`</td>
      <td>`"inductor"`</td>
      <td>Compilation backend string</td>
    </tr>

    <tr>
      <td>`get_dispatch_key_name()`</td>
      <td>`"native"`</td>
      <td>BaseFusedOp (fused-op) dispatch key name</td>
    </tr>

    <tr>
      <td>`get_group_coordinator_device(local_rank)`</td>
      <td>`torch.device("cpu")`</td>
      <td>Device each `GroupCoordinator` binds to on this rank (e.g. `torch.device(f"mydev:")`). Without it, device collectives on an out-of-tree platform run against a CPU device.</td>
    </tr>
  </tbody>
</table>

SGLang custom ops (e.g. the `inplace_all_reduce` collective) register their implementation under the `PrivateUse1` dispatch key on an out-of-tree platform, the key torch reserves for out-of-tree backends.

#### Lifecycle Hooks (from SRTPlatform)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Method</th>
      <th>Invocation Timing</th>
      <th>Purpose</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`apply_server_args_defaults(server_args)`</td>
      <td>After ServerArgs parsing, in `post_init`</td>
      <td>Set platform-specific defaults</td>
    </tr>

    <tr>
      <td>`init_backend()`</td>
      <td>In each worker, before model construction</td>
      <td>One-time backend initialization</td>
    </tr>

    <tr>
      <td>`post_load_model(model)`</td>
      <td>In each worker, once the model's weights are loaded</td>
      <td>Relocate parameters, swap quant methods or install forward hooks on an in-tree model class; mutate `model` in place</td>
    </tr>
  </tbody>
</table>

### Platform and plugin environment variables

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Variable</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_PLATFORM`</td>
      <td>Select the platform plugin by entry\_point name (e.g. `kunlun`, `demo_cuda`). When set, **only** the named plugin's `activate()` is called (front-loading filter) — other plugins are not touched. Additionally, general plugins (`sglang.srt.plugins`) from unselected platform packages are automatically skipped to avoid importing their dependencies. Required when multiple plugins would activate. Errors if the name is not found or if the plugin's hardware is unavailable.</td>
    </tr>

    <tr>
      <td>`SGLANG_PLUGINS`</td>
      <td>Comma-separated whitelist of general plugin names to load from either hook group. It also filters automatic SRT platform discovery when `SGLANG_PLATFORM` is unset; explicit platform selection ignores it.</td>
    </tr>
  </tbody>
</table>

### Add diffusion support

A package may support SRT, SGLang Diffusion, or both. Diffusion uses a zero-argument `Platform` subclass and separate platform and hook entry points:

```toml theme={null}
[project.entry-points."sglang.multimodal_gen.platforms"]
my_device = "my_platform_plugin:activate_diffusion"

[project.entry-points."sglang.multimodal_gen.plugins"]
my_device = "my_platform_plugin:register_hooks"
```

Keep activation import-safe and return the fully qualified class name only when the backend is available:

```python theme={null}
def activate_diffusion() -> str | None:
    try:
        import my_device_runtime
    except ImportError:
        return None
    if not my_device_runtime.is_available():
        return None
    return "my_platform_plugin.diffusion_platform.MyDiffusionPlatform"
```

The referenced class must be zero-argument constructible:

```python theme={null}
from sglang.multimodal_gen.runtime.platforms import Platform, PlatformEnum

class MyDiffusionPlatform(Platform):
    _enum = PlatformEnum.OOT
    device_name = "my_device"
    device_type = "my_device"
    dispatch_key = "PrivateUse1"

    def get_dispatch_key_name(self) -> str:
        return "my_device"
```

Set `dispatch_key` to the PyTorch dispatcher key used by direct `torch.library` registrations. Keep it separate from `get_dispatch_key_name()`, which selects `CustomOp` implementations. Bring up the remaining contract in dependency order:

1. Implement `get_device_name()`, `get_device_total_memory()`, and `get_available_gpu_memory()` before constructing `ServerArgs`.
2. Implement `get_device()` and `get_local_torch_device()` before worker binding.
3. Configure distributed initialization, attention selection, and custom-op implementations for supported workloads.
4. Run an end-to-end workload and audit model-specific kernels, compilation, and remaining device-family branches.

Register custom-op implementations from `init_backend()`, which runs once in each worker before any pipeline module is constructed:

```python theme={null}
class MyDiffusionPlatform(Platform):
    ...

    def init_backend(self) -> None:
        from my_platform_plugin.ops import rms_norm_forward
        from sglang.multimodal_gen.runtime.layers.custom_op import CustomOp
        from sglang.multimodal_gen.runtime.layers.layernorm import RMSNorm

        CustomOp.register_oot_forward(
            RMSNorm, fn=rms_norm_forward, platform_key="my_device"
        )
```

`init_backend()` runs at most once per worker process, before the worker implementation is imported. Raising aborts startup and the failed initialization is not retried in that process, because registrations and other backend side effects may be only partially reversible. Custom-op dispatch is resolved when each operation is constructed, after backend initialization has completed. The selected callable therefore remains stable when the operation is compiled instead of changing on the first compiled call.

The function receives the operation instance before its normal arguments. Registration matches the exact operation class and the value returned by `get_dispatch_key_name()`. Without a registration, SGLang looks for a matching `forward_<key>()` implementation on the operation, then falls back to `forward_oot()`, whose base implementation calls `forward_native()`. Returning `"cuda"`, for example, lets a CUDA-compatible OOT platform reuse operation-specific `forward_cuda()` implementations without reporting CUDA platform identity.

The `sglang.multimodal_gen.plugins` entry point remains for seams the `Platform` interface does not cover. Its callbacks run in launchers and workers before scheduler or model construction, so use a `BEFORE` or `AROUND` hook on `GPUWorker.init_device_and_model()` for rank-aware initialization. Hooks from platform distributions other than the selected one are skipped. Because these ship in your platform's distribution, a failure at any stage — load, callback, or hook application — aborts startup, the same as `init_backend()`.

#### Diffusion platform contract

**Active** interfaces are called through `current_platform`. **Compatibility** interfaces are retained adapters; unlisted `Platform` methods are not a stable OOT contract.

| Status | Area | Interface |
| - | - | - |
| Active | Initialization | `init_backend()` |
| Active | Configuration | `apply_server_args_defaults()` |
| Active | Device and memory | `get_device()`, `set_device()`, `get_local_torch_device()`, `get_device_name()`, `get_device_uuid()`, `get_device_total_memory()`, `get_available_gpu_memory()`, `get_device_capability()` |
| Active | Distributed | `get_torch_distributed_backend_str()`, `supports_distributed_device_id()`, `get_all_to_all_communicator_cls()`, `get_cpu_architecture()` |
| Active | Dispatch and model | `dispatch_key` (read by `get_torch_library_dispatch_key()`), `get_dispatch_key_name()`, `get_attn_backend_cls_str()`, `verify_model_arch()`, `optimize_vae()` |
| Active | Execution | `get_compile_backend()`, `get_compile_options()`, `inference_mode()`, `seed_everything()`, `enable_dit_layerwise_offload_by_default()` |
| Compatibility | Communicator | `get_device_communicator_cls()` remains the fallback used by `get_all_to_all_communicator_cls()` for existing platform subclasses. New OOT platforms should override the latter. |

Install the package, restart Python to refresh entry-point metadata, and verify explicit selection:

```bash theme={null}
pip install -e my_platform_plugin/
SGLANG_DIFFUSION_PLATFORM_OVERRIDE=my_device python -c \
  "from sglang.multimodal_gen.runtime.platforms import current_platform; print(current_platform)"
```

### Diffusion platform limitations

* SRT and diffusion require separate platform classes.
* Only one external platform can be active per process; set the runtime selector if multiple callbacks activate.
* `get_all_to_all_communicator_cls()` controls only `all_to_all_4D()`, not every collective, graph-capture, or synchronization path; `get_device_communicator_cls()` remains for compatibility.
* Compile settings affect only `build_torch_compile_kwargs()` callers; audit static `@torch.compile` decorators and other direct compile paths.
* Explicit attention selectors accept only `AttentionBackendEnum` names. Returning a custom backend works when none is selected; selecting it by name requires a hook or downstream patch.
* Device-family branches remain outside the interface. Each supported workload needs a native fallback or an actionable unsupported-feature error.
* The diffusion `PlatformEnum.OOT` identifies an external provider. Override built-in identity predicates only after auditing every enabled branch.

***

## Plugin Type 2: General Plugin

### Description

General function plugins inject behavior into SRT or diffusion **without requiring a custom platform**. The two runtimes have separate entry-point groups and hook registries:

```python theme={null}
# SRT plugin
from sglang.srt.plugins.hook_registry import HookType, plugin_hook

# Diffusion plugin
from sglang.multimodal_gen.runtime.platforms.plugins import HookType, plugin_hook
```

Register an SRT plugin under `sglang.srt.plugins` and a diffusion plugin under
`sglang.multimodal_gen.plugins`, and import the hook API from the matching
module above. Each runtime applies only its own registry, so a hook registered
through the other runtime's API never runs. The examples below use SRT.

The diffusion hook API lives alongside platform discovery in
`sglang.multimodal_gen.runtime.platforms.plugins`;
`sglang.multimodal_gen.plugins` is its entry-point group, not a Python module path.

Use cases include:

* **Observability**: Add logging, metrics, and tracing to any function
* **Behavior modification**: Modify function arguments or return values
* **Performance profiling**: Add timing to critical functions
* **A/B testing**: Replace implementations at runtime

### Quick Start

**1. Create a minimal package:**

```
my_general_plugin/
├── pyproject.toml
└── my_general_plugin/
    └── __init__.py    # register() function
```

**2. `pyproject.toml`:**

```toml theme={null}
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "my-general-plugin"
version = "0.1.0"

[project.entry-points."sglang.srt.plugins"]
my_plugin = "my_general_plugin:register"
```

**3. `__init__.py`** — register hooks:

```python theme={null}
from sglang.srt.plugins.hook_registry import HookRegistry, HookType

def register():
    """Entry point called by load_plugins()."""
    HookRegistry.register(
        "sglang.srt.managers.scheduler.Scheduler.__init__",
        my_hook,
        HookType.AROUND,
    )

def my_hook(original_fn, self, *args, **kwargs):
    result = original_fn(self, *args, **kwargs)
    print(f"Scheduler initialized! gpu_id={self.gpu_id}")
    return result
```

**4. Install and run:**

```bash theme={null}
pip install -e my_general_plugin/
sglang serve --model-path <model> [options]
# Look for "Scheduler initialized!" in logs
```

### Hook Types

`HookRegistry` supports four hook types:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Hook Type</th>
      <th>Signature</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BEFORE**</td>
      <td>`fn(*args, **kwargs) -> (args, kwargs) | None`</td>
      <td>Runs before the original. Return `None` to keep args unchanged, or `(args, kwargs)` to modify.</td>
    </tr>

    <tr>
      <td>**AFTER**</td>
      <td>`fn(result, *args, **kwargs) -> new_result | None`</td>
      <td>Runs after the original. Return `None` to keep result, or a new value to replace.</td>
    </tr>

    <tr>
      <td>**AROUND**</td>
      <td>`fn(original_fn, *args, **kwargs) -> result`</td>
      <td>Wraps the original. You must call `original_fn` yourself. Full control over execution.</td>
    </tr>

    <tr>
      <td>**REPLACE**</td>
      <td>`fn(*args, **kwargs) -> result` or `class`</td>
      <td>Replace the original function or class entirely. For class targets, pass a replacement class directly — it is substituted via `setattr` preserving `isinstance()`/`issubclass()` semantics.</td>
    </tr>
  </tbody>
</table>

> **Note**: Only `REPLACE` accepts a class as the hook. Passing a class to `BEFORE`/`AFTER`/`AROUND` raises `TypeError` at registration time.

### Registration API

Hooks can be registered using the **imperative API** or the **decorator API**:

```python theme={null}
# --- Imperative API ---
from sglang.srt.plugins.hook_registry import HookRegistry, HookType

def my_timer(original_fn, *args, **kwargs):
    start = time.perf_counter()
    result = original_fn(*args, **kwargs)
    print(f"Elapsed: {time.perf_counter() - start:.3f}s")
    return result

HookRegistry.register(
    "sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run",
    my_timer,
    HookType.AROUND,
)

# --- Decorator API ---
from sglang.srt.plugins.hook_registry import plugin_hook, HookType

@plugin_hook(
    "sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run",
    type=HookType.AROUND,
)
def my_timer(original_fn, *args, **kwargs):
    start = time.perf_counter()
    result = original_fn(*args, **kwargs)
    print(f"Elapsed: {time.perf_counter() - start:.3f}s")
    return result

# --- Class replacement (REPLACE) ---
from sglang.srt.plugins.hook_registry import plugin_hook, HookType
from sglang.srt.managers.scheduler import Scheduler

@plugin_hook(
    "sglang.srt.managers.scheduler.Scheduler",
    type=HookType.REPLACE,
)
class MyScheduler(Scheduler):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        print("Enhanced scheduler initialized!")
```

### Hook Target Resolution

Target paths use fully-qualified dotted notation. Both formats are supported:

* **Dotted**: `sglang.srt.managers.scheduler.Scheduler.__init__`
* **Entry-points style**: `sglang.srt.managers.scheduler:Scheduler.__init__` (colon treated as dot)

### Common SRT Hook Targets

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Target</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`sglang.srt.server_args.ServerArgs.add_cli_args`</td>
      <td>Add custom CLI arguments</td>
    </tr>

    <tr>
      <td>`sglang.srt.server_args.ServerArgs.post_init`</td>
      <td>Modify ServerArgs after parsing</td>
    </tr>

    <tr>
      <td>`sglang.srt.server_args.ServerArgs.check_server_args`</td>
      <td>Add/relax validation</td>
    </tr>

    <tr>
      <td>`sglang.srt.managers.scheduler.Scheduler.init`</td>
      <td>Custom scheduler state</td>
    </tr>

    <tr>
      <td>`sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run`</td>
      <td>Custom scheduling policy</td>
    </tr>

    <tr>
      <td>`sglang.srt.managers.scheduler.Scheduler.run_batch`</td>
      <td>Profiling / inspection</td>
    </tr>

    <tr>
      <td>`sglang.srt.managers.scheduler.Scheduler.process_batch_result`</td>
      <td>Custom metrics</td>
    </tr>

    <tr>
      <td>`sglang.srt.managers.tp_worker.TpModelWorker.init`</td>
      <td>Custom worker state</td>
    </tr>

    <tr>
      <td>`sglang.srt.managers.tp_worker.TpModelWorker.forward_batch_generation`</td>
      <td>Forward pass wrapping</td>
    </tr>
  </tbody>
</table>

***

## File Reference

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>File</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`sglang/srt/platforms/device_mixin.py`</td>
      <td>`DeviceMixin` base class and SRT platform identity types</td>
    </tr>

    <tr>
      <td>`sglang/srt/platforms/interface.py`</td>
      <td>`SRTPlatform` base class (extends DeviceMixin)</td>
    </tr>

    <tr>
      <td>`sglang/srt/platforms/init.py`</td>
      <td>`current_platform` lazy singleton + discovery logic</td>
    </tr>

    <tr>
      <td>`sglang/multimodal_gen/runtime/platforms/interface.py`</td>
      <td>Diffusion `Platform` base class</td>
    </tr>

    <tr>
      <td>`sglang/multimodal_gen/runtime/platforms/init.py`</td>
      <td>Diffusion `current_platform` lazy singleton and built-in fallback order</td>
    </tr>

    <tr>
      <td>`sglang/multimodal_gen/runtime/platforms/plugins.py`</td>
      <td>Diffusion plugin registration, explicit hook-application phase, and hook registry</td>
    </tr>

    <tr>
      <td>`sglang/multimodal_gen/runtime/managers/worker_bootstrap.py`</td>
      <td>Import-neutral process specifications and spawn targets that initialize the backend before resolving runtime hooks</td>
    </tr>

    <tr>
      <td>`sglang/srt/plugins/init.py`</td>
      <td>`load_plugins()` + `load_plugins_by_group()`</td>
    </tr>

    <tr>
      <td>`sglang/srt/plugins/hook_registry.py`</td>
      <td>`HookRegistry`, `HookType`, `plugin_hook` decorator</td>
    </tr>
  </tbody>
</table>


# T-HEAD SAILs
Source: https://docs.sglang.io/docs/hardware-platforms/sail



This document describes how to run SGLang on T-HEAD SAILs. If you
encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).

## Prerequisites

Install the T-HEAD SAIL driver and the PPU SDK before installing SGLang, and export
the `PPU_SDK` environment variable pointing at the SDK installation (for
example, `/usr/local/PPU_SDK`). The Triton SAIL backend requires this variable
when compiling kernels at runtime.

Verify that the device is visible:

```bash Command theme={null}
python3 -c 'import torch; print(torch.cuda.get_device_name())'
# Expected: a ZW-series device, e.g. "ZW810E"
```

## Install SGLang

### Install from Source

Start from an environment with the prerequisites above installed.

```bash Command theme={null}
git clone https://github.com/sgl-project/sglang.git
cd sglang

python -m pip install --upgrade pip wheel

cp python/pyproject_other.toml python/pyproject.toml
python -m pip install -e "python[all_sail]" \
  --no-build-isolation

# Install sailify
python -m pip install sailify==1.0.0

# Build sgl-kernel for SAIL from source
cd python/sglang/kernels/aot
python setup_sail.py install
```

Launch the server:

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --host 0.0.0.0 \
  --port 30000
```

When the server displays `The server is fired up and ready to roll!`, startup is
successful.

## Verify

You can send an OpenAI-compatible request to the engine:

```bash Command theme={null}
curl http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 32
  }'
```

Or run a benchmark:

```bash Command theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 4000 \
  --random-input 128 \
  --random-output 128
```


# TPU
Source: https://docs.sglang.io/docs/hardware-platforms/tpu

Serve models on Google TPUs with SGLang-TorchTPU for PyTorch or SGL-JAX for JAX.

SGLang provides two TPU serving backends. Choose the backend that matches your framework and model configuration:

| Backend | Framework | Hardware scope |
| - | - | - |
| [SGLang-TorchTPU](https://github.com/sgl-project/sglang-torchtpu) | PyTorch through torch\_tpu, with Pallas kernels | TPU7x (Ironwood) |
| [SGL-JAX](https://github.com/sgl-project/sglang-jax) | JAX | TPU v6e and v7 |

Each backend maintains its own runtime, model coverage, and deployment configurations. Select its tab below for details.

**SGLang-TorchTPU**

## PyTorch serving on TPU

[SGLang-TorchTPU](https://github.com/sgl-project/sglang-torchtpu) runs the SGLang serving engine on Google TPUs through torch\_tpu and Pallas kernels. You can use PyTorch models with SGLang's OpenAI-compatible serving API, eager execution, and TPU compilation.

Development and testing focus on TPU7x (Ironwood). A single host has four chips and eight TensorCores. Multi-host configurations use tensor and expert parallelism; disaggregated configurations separate prefill and decode across hosts.

## Model coverage

Model integrations include Muse Spark Open, GLM-5.3-Flash, Kimi-K3, DeepSeek-V4-Flash, Qwen3.8-27B, GPT-OSS, and Gemma 4. Coverage ranges from serving recipes to registered accuracy tests; Kimi-K3 currently has a small-model compile test with dummy weights.

The backend includes prefix caching, host-memory KV offload, EAGLE3 and DFlash speculative decoding, and prefill/decode disaggregation. Available combinations depend on the model and topology. Check the repository's [support guide](https://github.com/sgl-project/sglang-torchtpu/blob/main/SUPPORT.md) for feature restrictions and experimental capabilities.

## Deployment resources

Follow the [SGLang-TorchTPU getting-started guide](https://github.com/sgl-project/sglang-torchtpu#get-started) for runtime prerequisites, installation, and a first request.

Find model-specific configurations in the repository's [TPU serving recipes](https://github.com/sgl-project/sglang-torchtpu/tree/main/benchmark/tpu/recipe). Run these recipes from the TorchTPU checkout with its matching runtime.

Report PyTorch TPU issues in [sglang-torchtpu](https://github.com/sgl-project/sglang-torchtpu/issues), including the TPU generation, topology, model, and runtime versions.

**SGL-JAX**

This tab covers the JAX backend. Report JAX-specific issues in [sglang-jax](https://github.com/sgl-project/sglang-jax/issues).

## System Requirements

### Supported TPU Hardware

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>TPU Type</th>
      <th>HBM Memory</th>
      <th>Availability</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>TPU v6e</td>
      <td>32 GB</td>
      <td>Google Cloud</td>
    </tr>

    <tr>
      <td>TPU v7</td>
      <td>96 GB per core</td>
      <td>Google Cloud</td>
    </tr>
  </tbody>
</table>

### Software Requirements

* **Python:** 3.12 or higher
* **JAX:** Latest version with TPU support
* **Environment:** Google Cloud TPU VM or compatible TPU runtime
* **Optional:** SkyPilot for simplified cloud deployment

## Feature Support Matrix

SGLang-JAX provides comprehensive TPU-optimized features for production LLM serving:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Feature</th>
      <th>Support Status</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>High-Throughput Continuous Batching</td>
      <td>✅</td>
      <td>Dynamic request batching for maximum TPU utilization</td>
    </tr>

    <tr>
      <td>Radix Tree KV Cache</td>
      <td>✅</td>
      <td>Memory-efficient prefix sharing between requests</td>
    </tr>

    <tr>
      <td>FlashAttention Backend</td>
      <td>✅</td>
      <td>TPU-optimized attention kernel for long sequences</td>
    </tr>

    <tr>
      <td>Tensor Parallelism</td>
      <td>✅</td>
      <td>Distribute models across multiple TPU cores</td>
    </tr>

    <tr>
      <td>Paged Attention</td>
      <td>✅</td>
      <td>Flexible KV cache management with paging</td>
    </tr>

    <tr>
      <td>Speculative Decoding (EAGLE/EAGLE3)</td>
      <td>✅</td>
      <td>20-40% throughput improvement for compatible models</td>
    </tr>

    <tr>
      <td>Chunked Prefill</td>
      <td>✅</td>
      <td>Mixed prefill-decode batching</td>
    </tr>

    <tr>
      <td>OpenAI-Compatible API</td>
      <td>✅</td>
      <td>Drop-in replacement for OpenAI API</td>
    </tr>

    <tr>
      <td>Data Parallel Attention</td>
      <td>🚧</td>
      <td>In development - Attention computation with data parallelism</td>
    </tr>

    <tr>
      <td>Quantization</td>
      <td>🚧</td>
      <td>In development - Model quantization for reduced memory usage</td>
    </tr>

    <tr>
      <td>Multi-LoRA</td>
      <td>🚧</td>
      <td>In development - Serve multiple LoRA adapters simultaneously</td>
    </tr>
  </tbody>
</table>

### Attention Backend Comparison

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>**Backend**</th>
      <th>**Paged Attention**</th>
      <th>**Spec Decoding**</th>
      <th>**MLA**</th>
      <th>**Sliding Window**</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>FlashAttention (fa)</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
    </tr>

    <tr>
      <td>Native</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
    </tr>
  </tbody>
</table>

**NOTE:** FlashAttention backend is recommended for production workloads due to superior memory efficiency and performance.

## Optimized Model List

SGLang-JAX supports the following model families. Use the linked TPU cookbooks for deployment instructions.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>Cookbook</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>[Qwen 3](https://huggingface.co/Qwen)</td>
      <td>[Qwen3](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Qwen/Qwen3.md)</td>
    </tr>

    <tr>
      <td>[Qwen 3 MoE](https://huggingface.co/Qwen)</td>
      <td>[Qwen3 MoE](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Qwen/Qwen3-MoE.md)</td>
    </tr>

    <tr>
      <td>[Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-32B-Instruct)</td>
      <td>[Qwen2.5-VL](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Qwen/Qwen2.5-VL.md)</td>
    </tr>

    <tr>
      <td>[Llama/LLaMA](https://huggingface.co/meta-llama)</td>
      <td>[Llama 3.1](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Llama/Llama3.1.md) / [Llama 3.3 70B](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Llama/Llama3.3-70B.md)</td>
    </tr>

    <tr>
      <td>[DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3) / [R1](https://huggingface.co/deepseek-ai/DeepSeek-R1)</td>
      <td>[DeepSeek V3](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.md) / [DeepSeek R1](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/DeepSeek/DeepSeek-R1.md)</td>
    </tr>

    <tr>
      <td>[Kimi-Linear](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct)</td>
      <td>[Kimi-Linear](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Moonshotai/Kimi-Linear.md)</td>
    </tr>

    <tr>
      <td>[Grok-2](https://huggingface.co/xai-org)</td>
      <td>[Grok-2](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Grok/Grok2.md)</td>
    </tr>

    <tr>
      <td>[Gemma 2](https://huggingface.co/google)</td>
      <td>[Gemma 2](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Google/Gemma2.md)</td>
    </tr>

    <tr>
      <td>[MiMo-V2-Flash](https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash)</td>
      <td>[MiMo-V2-Flash](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Xiaomi/MiMo-V2-Flash.md)</td>
    </tr>

    <tr>
      <td>[MiMo-V2.5-Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro)</td>
      <td>[MiMo-V2.5-Pro](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.5-Pro.md)</td>
    </tr>

    <tr>
      <td>[Ling-2.6](https://huggingface.co/inclusionAI/Ling-2.6-1T)</td>
      <td>[Ling-2.6](https://github.com/sgl-project/sglang-jax/blob/main/docs/cookbook/autoregressive/InclusionAI/Ling-2.6.md)</td>
    </tr>
  </tbody>
</table>

## Installation

### Method 1: Using PyPI (Recommended)

```bash Command theme={null}
pip install sglang-jax
```

### Method 2: From Source

```bash Command theme={null}
git clone https://github.com/sgl-project/sglang-jax
cd sglang-jax
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -e "python[all]"
```

### Method 3: Using Docker

**NOTE:** Docker support for TPU is currently under development. Please use PyPI or source installation methods.

### Method 4: Cloud TPU with SkyPilot

[SkyPilot](https://github.com/skypilot-org/skypilot) provides simplified deployment on Google Cloud TPU:

1. Install SkyPilot and configure GCP access (see [SkyPilot documentation](https://skypilot.readthedocs.io/))

2. Create a SkyPilot configuration file:

> ```yaml Config theme={null}
> # sglang-jax.sky.yaml
> resources:
>    accelerators: tpu-v6e-4
>    accelerator_args:
>       tpu_vm: True
>       runtime_version: v2-alpha-tpuv6e
>
> run: |
>   git clone https://github.com/sgl-project/sglang-jax.git
>   cd sglang-jax
>   uv venv --python 3.12
>   source .venv/bin/activate
>   uv pip install -e "python[all]"
> ```

3. Launch your TPU cluster:

```bash Command theme={null}
# Standard deployment
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp

# With spot instances for cost savings
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp --use-spot
```

## Launch of the Serving Engine

### Basic Example: Qwen-7B

```bash Command theme={null}
JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache python3 -u -m sgl_jax.launch_server \
    --model-path Qwen/Qwen-7B-Chat \
    --trust-remote-code \
    --dist-init-addr=0.0.0.0:10011 \
    --nnodes=1 \
    --tp-size=4 \
    --device=tpu \
    --random-seed=3 \
    --node-rank=0 \
    --mem-fraction-static=0.8 \
    --max-prefill-tokens=8192 \
    --download-dir=/tmp \
    --dtype=bfloat16 \
    --skip-server-warmup \
    --host 0.0.0.0 \
    --port 30000
```

**Key Parameters Explained:**

1. `JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache` - Enables JIT compilation caching to accelerate server startup on subsequent runs
2. `--tp-size=4` - Tensor parallelism size; match this to your TPU core count (typically 1, 4, or 8)
3. `--device=tpu` - Specifies TPU device (this is the default for sglang-jax)
4. `--dtype=bfloat16` - Uses bfloat16 precision, which TPUs are optimized for
5. `--mem-fraction-static=0.8` - Allocates 80% of TPU HBM for static memory (adjustable from 0.2 to 0.9)
6. `--max-prefill-tokens=8192` - Maximum number of tokens processed in the prefill phase

### High-Performance Configuration: Qwen3-8B

For production workloads with optimal throughput:

```bash Command theme={null}
python3 -u -m sgl_jax.launch_server \
    --model-path Qwen/Qwen3-8B \
    --trust-remote-code \
    --tp-size=4 \
    --device=tpu \
    --mem-fraction-static=0.8 \
    --chunked-prefill-size=2048 \
    --dtype=bfloat16 \
    --max-running-requests=256 \
    --page-size=128 \
    --attention-backend=fa
```

### Advanced: Speculative Decoding (EAGLE3)

Speculative decoding can improve throughput by 20-40% for compatible models:

```bash Command theme={null}
python3 -u -m sgl_jax.launch_server \
    --model-path Qwen/Qwen3-32B \
    --trust-remote-code \
    --device=tpu \
    --tp-size=4 \
    --mem-fraction-static=0.8 \
    --max-prefill-tokens=4096 \
    --attention-backend=fa \
    --dtype=bfloat16 \
    --port=30000 \
    --host=0.0.0.0 \
    --disable-overlap-schedule \
    --speculative-algorithm=EAGLE3 \
    --speculative-draft-model-path=AngelSlim/Qwen3-32B_eagle3 \
    --page-size=64 \
    --speculative-eagle-topk=1 \
    --speculative-num-steps=3 \
    --speculative-num-draft-tokens=4
```

**NOTE:** Speculative decoding is currently supported for Qwen3 and LLaMA model families. See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration guidance.

### Multi-Node Distributed Serving

For large models requiring multiple TPU VMs:

```bash Command theme={null}
# Node 0 (coordinator)
python3 -m sgl_jax.launch_server \
    --model-path MODEL_PATH \
    --dist-init-addr=NODE0_IP:10011 \
    --nnodes=2 \
    --node-rank=0 \
    --tp-size=8 \
    [other parameters...]

# Node 1 (worker)
python3 -m sgl_jax.launch_server \
    --model-path MODEL_PATH \
    --dist-init-addr=NODE0_IP:10011 \
    --nnodes=2 \
    --node-rank=1 \
    --tp-size=8 \
    [other parameters...]
```

## Benchmarking with Requests

### Throughput Testing

Basic throughput benchmark:

```bash Command theme={null}
python3 -m sgl_jax.bench_serving \
    --backend sgl-jax \
    --dataset-name random \
    --num-prompts=100 \
    --random-input=512 \
    --random-output=128 \
    --max-concurrency=8 \
    --random-range-ratio=1 \
    --warmup-requests=0
```

### Latency Testing

Measure single-batch latency:

```bash Command theme={null}
python3 -m sgl_jax.bench_one_batch_server \
    --base-url http://127.0.0.1:30000 \
    --model-path Qwen/Qwen-7B-Chat \
    --batch-size=32 \
    --input-len=256 \
    --output-len=32
```

### Comprehensive Benchmark Script

For systematic performance evaluation across different configurations:

```bash Command theme={null}
#!/bin/bash
set -e

backend=${1:-sgl-jax}
num_prompts_per_concurrency=3
input_seq_lens=(1024 4096 8192)
output_seq_lens=(1 1024)
max_concurrencies=(8 16 32 64 128 256)

for input_seq_len in "${input_seq_lens[@]}"; do
    for output_seq_len in "${output_seq_lens[@]}"; do
        echo "======================================="
        echo "Testing ISL/OSL: $input_seq_len/$output_seq_len"
        echo "======================================="
        for max_concurrency in "${max_concurrencies[@]}"; do
            num_prompts=$((num_prompts_per_concurrency * max_concurrency))
            python3 -m sgl_jax.bench_serving \
                --backend ${backend} \
                --dataset-name random \
                --num-prompts ${num_prompts} \
                --random-input ${input_seq_len} \
                --random-output ${output_seq_len} \
                --max-concurrency ${max_concurrency} \
                --random-range-ratio 1 \
                --disable-ignore-eos \
                --warmup-requests 0
        done
    done
done
```

For detailed help on all benchmark parameters:

```bash Command theme={null}
python3 -m sgl_jax.bench_serving --help
```

See the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for advanced benchmarking techniques and profiling with JAX Profiler.

## Performance Optimization

### Memory Optimization

**Reduce memory usage:**

* Lower `--mem-fraction-static` (from 0.8 → 0.5 → 0.3)
* Decrease `--max-prefill-tokens` (from 16384 → 8192 → 4096)
* Reduce `--max-running-requests`

**Handle OOM errors:**

* Start with conservative memory settings (`--mem-fraction-static=0.5`)
* Gradually increase until you find the optimal balance
* Increase `--page-size` for better memory locality (1 → 16 → 64 → 128)

### Throughput Optimization

To maximize tokens per second:

* Use FlashAttention backend: `--attention-backend=fa`
* Enable speculative decoding (EAGLE3) for Qwen3 models (20-40% improvement)
* Increase `--max-running-requests` to 256+
* Set `--mem-fraction-static` to 0.8+ (if memory allows)
* Use larger page sizes (64-128)
* Enable chunked prefill: `--chunked-prefill-size=2048`

### Latency Optimization

To minimize time-to-first-token (TTFT) and inter-token latency:

* Reduce `--page-size` to 1-4
* Lower `--max-running-requests` (16-32) for smaller batches
* Reduce `--chunked-prefill-size`
* Use conservative memory settings to avoid GC pauses

### TPU-Specific Optimizations

1. **JIT Compilation Cache:**
   ```bash Command theme={null}
   export JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache
   ```
   Always set this environment variable to cache compiled kernels and accelerate server startup.

2. **Data Type Optimization:**
   Use `--dtype=bfloat16` for TPU native optimization. TPUs are specifically designed for bfloat16 computations.

3. **Tensor Parallelism:**
   Match `--tp-size` to your TPU core configuration (1, 4, or 8) for optimal model distribution.

4. **Attention Backend:**
   Always use `--attention-backend=fa` (FlashAttention) for production workloads.

## Troubleshooting

### OOM (Out of Memory) Errors

If you encounter out-of-memory errors:

1. Reduce `--mem-fraction-static` from 0.8 to 0.5 or lower
2. Decrease `--max-prefill-tokens` from 8192 to 4096 or 2048
3. Lower `--max-running-requests` to reduce concurrent batch size
4. Increase `--page-size` for better memory layout efficiency

### Compilation Long-Time

If the server takes too long to start:

1. Ensure `JAX_COMPILATION_CACHE_DIR` is properly set
2. Understand that the first run requires JIT compilation (this is normal)
3. Subsequent runs will be significantly faster with cached compilations
4. Consider using `--skip-server-warmup` to defer compilation until first request

### Low Throughput

If you're not achieving expected throughput:

1. Verify `--tp-size` matches your TPU core configuration
2. Check that `--attention-backend=fa` is enabled
3. Increase `--max-running-requests` to enable larger batch formation
4. Consider enabling speculative decoding for compatible models
5. Ensure memory settings allow for sufficient batch sizes

### Connection Issues

If clients cannot connect to the server:

1. Ensure `--host=0.0.0.0` for external access (not just `127.0.0.1`)
2. Verify firewall rules allow traffic on the specified port (default: 30000)
3. Check that the server process is running: `curl http://localhost:30000/health`

## Advanced Features

### Speculative Decoding

SGLang-JAX supports EAGLE and EAGLE3 speculative decoding algorithms for Qwen3 and LLaMA model families. Speculative decoding can improve throughput by 20-40% without affecting output quality.

See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration and supported model combinations.

### Chunked Prefill

Enable mixed prefill-decode batching for better TPU utilization:

```bash Command theme={null}
--chunked-prefill-size=2048 --enable-mixed-chunk
```

This allows the scheduler to mix prefill operations with decode operations in the same batch, improving overall throughput.

### Custom Attention Backends

SGLang-JAX supports a plugin-based attention backend system. You can implement custom attention kernels optimized for specific use cases.

See the [Attention Backend documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/attention_backend.md) for implementation details.

### Environment Verification

Verify your TPU setup before deploying:

```bash Command theme={null}
python -c "from sgl_jax import check_env; check_env.check_env()"
```

This command checks:

* Installed package versions
* TPU device availability and specifications
* System resources and configuration
* Compatibility of settings

## Contributing

We welcome contributions to improve TPU support in SGLang-JAX!

### Areas for Contribution

**Check the [Development Roadmap](https://github.com/sgl-project/sglang-jax/issues/190)** to see planned features and find opportunities to contribute new functionality.

Current contribution areas include:

* Performance optimizations for specific TPU generations
* Support for additional model architectures
* Documentation improvements and examples
* Bug reports and fixes
* Benchmark results and performance analysis

### How to Contribute

1. Visit the [sglang-jax repository](https://github.com/sgl-project/sglang-jax)
2. Read the [Contribution Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/contribution_guide.md)
3. Join the [SGL-JAX Slack community](https://sgl-fru7574.slack.com/archives/C09EBE5HT5X) for discussions
4. Report issues at [sglang-jax/issues](https://github.com/sgl-project/sglang-jax/issues)

### Testing on TPU

For contributors who need TPU access for testing:

* Refer to the [TPU Resources Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/tpu_resources_guide.md) for information on accessing TPU hardware
* Use SkyPilot with spot instances for cost-effective testing
* Follow the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for performance validation

## References

### Documentation

* [SGLang-JAX Repository](https://github.com/sgl-project/sglang-jax)
* [SGLang-JAX Installation Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/get_started/install.md)
* [Qwen Models Quick Start](https://github.com/sgl-project/sglang-jax/blob/main/docs/basic_usage/qwen.md)
* [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md)
* [Speculative Decoding](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md)

### External Resources

* [JAX Documentation](https://jax.readthedocs.io/)
* [Google Cloud TPU Documentation](https://cloud.google.com/tpu/docs)
* [SkyPilot Documentation](https://skypilot.readthedocs.io/)


# XPU
Source: https://docs.sglang.io/docs/hardware-platforms/xpu



The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html).

Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [
Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).

A number of popular LLMs are optimized and run efficiently on XPU, including the most notable open-source models like Llama series, Qwen series, and Diffusion model series like FLUX and Z-Image.
Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro) in which the support status and example commands can be found.

Quantized MoE models are covered separately in
[MXFP4 MoE Quantization](#mxfp4-moe-quantization) below.

## Installation

### Install From Source

Currently SGLang XPU only supports installation from source. Please refer to ["Getting Started on Intel GPU"](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency.

```bash Command theme={null}
# Below commands shows Ubuntu/Debian OS as example, other OS requires similar command change
# add sudo if required to install below packages
# Install curl and protobuf compiler

apt-get update && apt-get install -y curl protobuf-compiler

# Install Rust (non-interactive)
curl --proto '=https' --tlsv1.2 --retry 3 --retry-delay 2 -sSf https://sh.rustup.rs | sh -s -- -y
# Make cargo available in the current shell
source "$HOME/.cargo/env"

# Create and activate a conda environment
conda create -n sgl-xpu python=3.12 -y
conda activate sgl-xpu

# Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues.
pip3 install torch==2.14.0+xpu torchvision==0.29.0+xpu torchaudio==2.11.0+xpu --index-url https://download.pytorch.org/whl/xpu
pip3 install xgrammar==0.1.33 --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU

# Clone the SGLang code
git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout <YOUR-DESIRED-VERSION>

# Use dedicated toml file
cd python
cp pyproject_xpu.toml pyproject.toml
# Install SGLang dependent libs, and build SGLang main package
pip install --upgrade pip setuptools
pip install -v ".[dev,diffusion]" --extra-index-url https://download.pytorch.org/whl/xpu
```

### Install Using Docker

It is recommended to use Docker for setting up the SGLang environment.

#### Pull from Docker Hub

Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository.
The [XPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu) end with `xpu` suffix.
The image pulling command is like:

```bash Command theme={null}
docker pull lmsysorg/sglang:v0.5.22-xpu
```

#### Build from Dockerfile

[The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).

```bash Command theme={null}
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker

# Build the docker image
docker build -t sglang-xpu:latest -f xpu.Dockerfile .

# Initiate a docker container
docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=<secret>" \
    sglang-xpu:latest /bin/bash
```

## Launch of the Serving Engine

Example command to launch SGLang serving:

```bash theme={null}
sglang serve                         \
    --model-path <MODEL_ID_OR_PATH>  \
    --trust-remote-code              \
    --device xpu                     \
    --host 0.0.0.0                   \
    --tp 2                           \   # using multi GPUs
    --attention-backend intel_xpu    \   # using intel optimized XPU attention backend
    --page-size                      \   # intel_xpu attention backend supports [32, 64, 128]
```

## MXFP4 MoE Quantization

Native MXFP4 MoE checkpoints (OCP microscaling FP4: packed `e2m1` weights plus
per-32-element `ue8m0` block scales) run on Intel GPUs through the
`sgl-kernel-xpu` W4A16 grouped GEMM. The expert weights stay in the checkpoint's
packed layout end to end — there is no dequantization to BF16 — so GPT-OSS-20B
loads in roughly 13 GB rather than the \~42 GB a BF16 upcast would need, which is
what lets it fit on a single 24 GB card.

The `mxfp4` method is registered automatically on `--device xpu`, which already
requires `sgl-kernel-xpu` — no extra flag is needed:

```bash theme={null}
sglang serve                        \
    --model-path openai/gpt-oss-20b \
    --device xpu                    \
    --attention-backend intel_xpu
```

**Tested models:**

| Model | Notes |
| :- | :- |
| [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | 32 experts, top-k 4, hidden 2880, intermediate 2880; clamped-swiglu activation (`gemm1_alpha` 1.702, limit 7.0) with expert biases |

**Requirements and limitations:**

| Item | Status |
| - | - |
| GPU architecture | Xe2 / BMG only (Intel® Arc™ B-Series and Arc™ Pro B-Series) |
| Quantized layers | Fused MoE experts only; attention, router, embeddings and LM head stay BF16 (per the checkpoint's `modules_to_not_convert`) |
| Activations | BF16 (W4A16). Clamped swiglu (GPT-OSS) and plain SiLU are supported |
| Expert parallelism | Not yet validated on this path |

## Benchmarking with Requests

You can benchmark the performance via the `bench_serving` script.
Run the command in another terminal.

```bash theme={null}
python -m sglang.bench_serving   \
    --dataset-name random        \
    --random-input-len 1024      \
    --random-output-len 1024     \
    --num-prompts 1              \
    --request-rate inf           \
    --random-range-ratio 1.0
```

The detail explanations of the parameters can be looked up by the command:

```bash theme={null}
python -m sglang.bench_serving -h
```

Additionally, the requests can be formed with
[OpenAI Completions API](../basic_usage/openai_api_completions)
and sent via the command line (e.g. using `curl`) or via your own script.

## XPU Graph \[Experimental]

SGLang enables XPU graph capture to reduce per-step kernel-launch overhead.

| Phase | Backend | Mechanism | Default |
| - | - | - | - |
| Decode | `full` | One `torch.xpu.XPUGraph` per batch size, captured on startup | **Off** (opt-in) |
| Prefill | `breakable` | Segmented `torch.xpu.XPUGraph` capture/replay (no `torch.compile`); eager break points at attention / MoE boundaries | **Off** (opt-in) |

### Enable Decode Graph

Decode graph capture is **opt-in** on XPU. Enable it explicitly:

```bash theme={null}
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-decode full
```

### Enable Prefill Graph

Prefill graph capture is **opt-in** on XPU and must be enabled explicitly.
Use the `breakable` backend. The former `tc_piecewise` backend has been removed.

#### breakable

Captures the transformer stack as segmented `XPUGraph`s with eager break points
at attention / MoE boundaries, without `torch.compile`:

```bash theme={null}
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill breakable
```

You can also configure both phases together with a single `--cuda-graph-config` JSON argument:

```bash theme={null}
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-config '{"decode":{"backend":"full"},"prefill":{"backend":"breakable"}}'
```

### Enable torch.compile for Decode

`--enable-torch-compile` adds a `torch.compile` pass on top of the decode
XPU graph: the model forward is compiled first, and the compiled forward is
then captured as an `XPUGraph`. This can reduce per-kernel overhead further
but increases startup time.

```bash theme={null}
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --enable-torch-compile
```

### Disable XPU Graph

Both phases are disabled by default. To explicitly disable them anyway:

```bash theme={null}
# Disable decode graph (already off by default; explicit form)
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-decode=disabled

# Disable prefill graph (already off by default; explicit form)
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill=disabled

# Disable both phases
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-decode=disabled \
    --cuda-graph-backend-prefill=disabled
```

### Customize Capture Buckets

By default, prefill capture sizes are derived from `--chunked-prefill-size`.
To specify explicit token-length buckets:

```bash theme={null}
python -m sglang.launch_server \
    --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill breakable \
    --cuda-graph-bs-prefill 64 128 256 512
```

To specify explicit decode graph batch sizes:

```bash theme={null}
python -m sglang.launch_server \
    --model-path <MODEL> --device xpu \
    --cuda-graph-bs-decode 1 2 4 8
```

### Server Args

| Argument | XPU allowed values | Default | Description |
| - | - | - | - |
| `--cuda-graph-backend-decode` | `full`, `disabled` | `disabled` | Backend for the decode phase. Only `full` is supported on XPU. Set to `full` to enable. |
| `--cuda-graph-backend-prefill` | `breakable`, `disabled` | `disabled`\* | Backend for the prefill phase. Set to `breakable` explicitly to enable. |
| `--cuda-graph-bs-prefill` | list of ints | auto | Explicit token-length buckets to capture for prefill. |
| `--cuda-graph-bs-decode` | list of ints | auto | Explicit batch sizes to capture for decode. |
| `--cuda-graph-config` | JSON string | — | One-shot JSON config for both phases, e.g. `'{"decode":{"backend":"full"},"prefill":{"backend":"breakable"}}'`. Overrides all per-phase flags. |
| `--disable-decode-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-decode=disabled`. |
| `--disable-prefill-cuda-graph` | — | `False` | Shorthand for `--cuda-graph-backend-prefill=disabled`. |
| `--enable-torch-compile` | — | `False` | Apply `torch.compile` on top of the decode XPU graph for further kernel optimization. |
| `--torch-compile-max-bs` | int | `32` | Maximum batch size compiled by `torch.compile` when `--enable-torch-compile` is set. |

\* Prefill graph is auto-disabled on XPU unless you lock the backend explicitly
via `--cuda-graph-backend-prefill` or `--cuda-graph-config`.

### Limitations

| Feature | Status |
| - | - |
| Two-batch overlap (`--enable-two-batch-overlap`) | Not yet supported |
| Speculative decoding | Supported |

## Decode Context Parallel (DCP) on Intel XPU \[Experimental]

DCP shards the KV cache across the ranks of a TP group by token position. It grows
the servable context only when `tp_size` exceeds the model's KV heads, which TP would
otherwise replicate, and decode gets slower, so use it only when the context you need
does not fit.

```bash theme={null}
python -m sglang.launch_server --model-path Qwen/Qwen3.5-4B --device xpu \
    --attention-backend intel_xpu --tp 2 --dcp-size 2 \
    --disable-radix-cache --mem-fraction-static 0.60
```

It needs the `intel_xpu` attention backend and ranks that share their DCP group's
KV heads: Qwen3.5 models, or models with at most `tp_size / dcp_size` KV heads.
Unsupported features are refused at startup. If startup fails with
`UR_RESULT_ERROR_OUT_OF_RESOURCES`, lower `--mem-fraction-static`.

With P/D disaggregation, DCP on XPU skips the DCP pack fast path: the pack
kernel runs only on CUDA, so KV is transferred per token instead.

## Memory Saver (release/resume memory occupation) on Intel XPU \[Experimental]

SGLang can temporarily release most of the GPU memory it holds — model weights
and/or KV cache — and reclaim it later without restarting the process. This is
the same `release_memory_occupation` / `resume_memory_occupation` feature
available on CUDA, used for RL rollout/training hand-off and for freeing the
device between inference bursts.

This is backed by the [`torch_memory_saver`](https://github.com/fzyzcjy/torch_memory_saver)
package — the same package used on CUDA — which gained an Intel XPU backend
built natively on Level Zero (keeping virtual addresses fixed while
releasing/re-committing physical pages via `zeVirtualMemUnmap` /
`zeVirtualMemMap`).

**Install `torch_memory_saver`.** Unlike CUDA (prebuilt wheel), the XPU backend
is built from source against your local oneAPI + `torch+xpu` runtime (the `.so`
links `libsycl.so.<N>`, which must match the installed `intel-sycl-rt`).
`TMS_PLATFORM=xpu` forces the XPU backend, and `--no-build-isolation` lets the
build import your installed `torch` so it can match the `libsycl` major to it:

The published wheels are CUDA-only, so install from git and let it build. The ref
below is the `v0.0.10b2` release, pinned so the build is reproducible.

```bash theme={null}
source /opt/intel/oneapi/setvars.sh
TMS_PLATFORM=xpu pip install --no-build-isolation \
    git+https://github.com/fzyzcjy/torch_memory_saver.git@a5c99f11b18ebb8e9fda71a68812e476ae49e417
```

**Use it** by launching with `--enable-memory-saver` (the XPU backend is
selected automatically); optionally add `--enable-weights-cpu-backup` to keep
weights in host RAM across a release:

```bash theme={null}
python -m sglang.launch_server --model-path Qwen/Qwen3-0.6B \
    --trust-remote-code --device xpu --enable-memory-saver
```

```bash theme={null}
# Release GPU memory while idle, then reclaim it (server must be idle).
curl -X POST http://127.0.0.1:30000/release_memory_occupation
curl -X POST http://127.0.0.1:30000/resume_memory_occupation
```

The Python engine API (`engine.release_memory_occupation(tags=...)` /
`engine.resume_memory_occupation(tags=...)`) and the `weights` / `kv_cache` tags
behave the same as on CUDA. Pauseable CUDA-graph capture is not used on XPU, so
the `cuda_graph` tag is a no-op there.

> **Verifying memory was freed:** neither `torch.xpu.memory_allocated()` nor
> `torch.xpu.mem_get_info()` drops when physical pages are released — the first is
> allocator accounting, and the second stays flat because torch keeps the freed block
> cached. Query actual device memory via sysman (`ZES_ENABLE_SYSMAN=1`) instead.

## Prefill-Decode (P/D) Disaggregation on Intel XPU \[Experimental]

SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend.

**Tested models:**

| Model | Notes |
| :-: | :-: |
| [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | Used in integration tests; verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) |
| [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | Verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) |

**Prerequisites:** `pip install nixl sglang-router`

**Start the prefill server (GPU 0):**

```bash theme={null}
ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
    --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
    --disaggregation-mode prefill --disaggregation-transfer-backend nixl \
    --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000
```

**Start the decode server (GPU 1):**

```bash theme={null}
ZE_AFFINITY_MASK=1 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
    --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
    --disaggregation-mode decode --disaggregation-transfer-backend nixl \
    --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30001
```

**Start the router:**

```bash theme={null}
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://127.0.0.1:30000 \
    --decode  http://127.0.0.1:30001 \
    --host 0.0.0.0 --port 8000
```

**Send a request:**

```bash theme={null}
curl http://127.0.0.1:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 32}'
```

> **Note:** `UCX_POSIX_USE_PROC_LINK=n` is required on Intel XPU to avoid UCX shared-memory transport issues.


# Custom Chat Template
Source: https://docs.sglang.io/docs/references/custom_chat_template



**NOTE**: There are two chat template systems in SGLang project. This document is about setting a custom chat template for the OpenAI-compatible API server (defined at [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py)). It is NOT related to the chat template used in the SGLang language frontend (defined at [chat\_template.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/lang/chat_template.py)).

By default, the server uses the chat template specified in the model tokenizer from Hugging Face.
It should just work for most official models such as Llama-2/Llama-3.

If needed, you can also override the chat template when launching the server:

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-2-7b-chat-hf \
  --port 30000 \
  --chat-template llama-2
```

If the chat template you are looking for is missing, you are welcome to contribute it or load it from a file.

## JSON Format

You can load the JSON format, which is defined by `conversation.py`.

```json Config theme={null}
{
  "name": "my_model",
  "system": "<|im_start|>system",
  "user": "<|im_start|>user",
  "assistant": "<|im_start|>assistant",
  "sep_style": "CHATML",
  "sep": "<|im_end|>",
  "stop_str": ["<|im_end|>", "<|im_start|>"]
}
```

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-2-7b-chat-hf \
  --port 30000 \
  --chat-template ./my_model_template.json
```

## Jinja Format

You can also use the [Jinja template format](https://huggingface.co/docs/transformers/main/en/chat_templating) as defined by Hugging Face Transformers.

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path meta-llama/Llama-2-7b-chat-hf \
  --port 30000 \
  --chat-template ./my_model_template.jinja
```


# Environment Variables
Source: https://docs.sglang.io/docs/references/environment_variables



SGLang supports various environment variables that can be used to configure its runtime behavior. This document provides a comprehensive list and aims to stay updated over time.

*Note: The canonical prefix for all SGLang environment variables is `SGLANG_`. The legacy `SGL_` prefix is deprecated: any `SGL_*` variable set in the environment is automatically rewritten to its `SGLANG_*` equivalent at import time with a deprecation warning, and the alias will be removed in a future release. A few variables keep an upstream/vendor prefix (e.g. `MOONCAKE_*`, `ASCEND_*`) because that is their canonical name.*

## General Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_USE_MODELSCOPE`</td>
      <td>Enable using models from ModelScope</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_HOST_IP`</td>
      <td>Host IP address for the server</td>
      <td>`0.0.0.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_PORT`</td>
      <td>Port for the server</td>
      <td>auto-detected</td>
    </tr>

    <tr>
      <td>`SGLANG_LOGGING_CONFIG_PATH`</td>
      <td>Custom logging configuration path</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_LOG_REQUEST_HEADERS`</td>
      <td>Comma-separated list of additional HTTP headers to log when `--log-requests` is enabled. Appends to the default `x-smg-routing-key`.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_HEALTH_CHECK_TIMEOUT`</td>
      <td>Timeout for health check in seconds</td>
      <td>`20`</td>
    </tr>

    <tr>
      <td>`SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL`</td>
      <td>The interval of passes to collect the metric of selected count of physical experts on each layer and GPU rank. 0 means disabled.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_EPLB_P2P_BATCH_CHUNK_SIZE`</td>
      <td>Number of expert IDs per batch when submitting P2P ops during EPLB rebalance (CUDA and ROCm). Smaller values prevent NCCL/RCCL GPU-side accumulation hangs but increase overhead; set `>= num_physical_experts` to submit a single batch. Deprecated alias: `SGLANG_EPLB_ROCM_P2P_BATCH_CHUNK_SIZE`.</td>
      <td>`32`</td>
    </tr>

    <tr>
      <td>`SGLANG_FORWARD_UNKNOWN_TOOLS`</td>
      <td>Forward unknown tool calls to clients instead of dropping them</td>
      <td>`false` (drop unknown tools)</td>
    </tr>

    <tr>
      <td>`SGLANG_REQ_WAITING_TIMEOUT`</td>
      <td>Timeout (in seconds) for requests waiting in the queue before being scheduled</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_REQ_RUNNING_TIMEOUT`</td>
      <td>Timeout (in seconds) for requests running in the decode batch</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIR`</td>
      <td>Cache directory for model weights and other data. Also the default root for compiled-kernel caches: Triton, Inductor, FlashInfer, the CUDA driver and DeepGEMM are pointed under it unless their own env vars (`TRITON_CACHE_DIR`, `TORCHINDUCTOR_CACHE_DIR`, `FLASHINFER_WORKSPACE_BASE`, `CUDA_CACHE_PATH`, `SGLANG_DG_CACHE_DIR`) are set explicitly</td>
      <td>`~/.cache/sglang`</td>
    </tr>

    <tr>
      <td>`SGLANG_QWEN4_PLE_FILE_DIR`</td>
      <td>Default directory for the file-backed Qwen4-Exp PLE table (`--ple-offload-backend file`); one subdirectory per model path is created under it</td>
      <td>`/ple`</td>
    </tr>

    <tr>
      <td>`SGLANG_QWEN4_PLE_FILE_PREFETCH`</td>
      <td>Hint the page cache (`posix_fadvise(WILLNEED)`) with the rows a prefill-sized PLE gather is about to read from the file-backed table. Set to `0` to disable</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_QWEN4_PLE_FILE_SKIP_DEVICE_CHECK`</td>
      <td>Skip the load-time check that the device reads pageable host memory through the host page tables (`cudaDevAttrPageableMemoryAccessUsesHostPageTables`) before using the file-backed PLE table; only for devices that cannot be queried</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB`</td>
      <td>Cap (GiB) on the resident set of the file-backed PLE mapping. A row fault maps a whole page-cache folio, so the mapping otherwise creeps towards the full table and consumes the free memory that sizes the KV pool; over the cap the pages are dropped with `MADV_DONTNEED` and stay in the page cache. Set to `0` to disable</td>
      <td>`8.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_QWEN4_PLE_FILE_RSS_INTERVAL_S`</td>
      <td>How often the file-backed PLE mapping's resident set is compared against `SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB`</td>
      <td>`30.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_CUTE_AOT_CACHE_DIR`</td>
      <td>Trusted directory for persistent CuTe DSL AOT objects shared across process restarts. Artifacts are namespaced by source, runtime ABI, host platform, and target GPU architecture. Set to an empty string to keep compilation process-local. Only use a directory writable by trusted users because SGLang loads cached object files into the process.</td>
      <td>`{SGLANG_CACHE_DIR}/cute_aot`</td>
    </tr>

    <tr>
      <td>`SGLANG_PREFETCH_BLOCK_SIZE_MB`</td>
      <td>Block size (in MB) for sequential checkpoint prefetch reads that warm the OS page cache before workers load weights via mmap</td>
      <td>`16`</td>
    </tr>
  </tbody>
</table>

## Performance Tuning

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_ENABLE_TORCH_INFERENCE_MODE`</td>
      <td>Control whether to use torch.inference\_mode</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_TORCH_COMPILE`</td>
      <td>Enable torch.compile</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_SET_CPU_AFFINITY`</td>
      <td>Enable CPU affinity setting (often set to `1` in Docker builds)</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN`</td>
      <td>Allows the scheduler to overwrite longer context length requests (often set to `1` in Docker builds)</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_IS_FLASHINFER_AVAILABLE`</td>
      <td>Control FlashInfer availability check</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_AUTOTUNE_CACHE`</td>
      <td>Reuse persisted FlashInfer autotune results from `SGLANG_CACHE_DIR` across runs, including MegaMoE capacity profiles. Set to `0` to disable disk-cache reuse; the fresh result is written to a `runs/<rank>.<timestamp>.<uuid>.json` sibling file (the canonical cache is left untouched). Target MegaMoE rank files include the physical GPU UUID. Use separate `SGLANG_CACHE_DIR` roots for engines sharing physical GPUs; a running process retains tactics for already-pooled workspaces.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_SKIP_P2P_CHECK`</td>
      <td>Skip P2P (peer-to-peer) access check</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD`</td>
      <td>Sets the threshold for enabling chunked prefix caching</td>
      <td>`8192`</td>
    </tr>

    <tr>
      <td>`SGLANG_MAX_KV_CHUNK_CAPACITY`</td>
      <td>Maximum number of tokens in each KV chunk for DeepSeek MHA chunked prefix cache</td>
      <td>`131072`</td>
    </tr>

    <tr>
      <td>`SGLANG_FUSED_MLA_ENABLE_ROPE_FUSION`</td>
      <td>Enable RoPE fusion in Fused Multi-Layer Attention</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISABLE_CONSECUTIVE_PREFILL_OVERLAP`</td>
      <td>Disable overlap schedule for consecutive prefill batches</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_SCHEDULER_MAX_RECV_PER_POLL`</td>
      <td>Set the maximum number of requests per poll, with a negative value indicating no limit</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_MAX_NEW_TOKENS_LIMIT`</td>
      <td>Hard server-side limit for each generation request's `max_new_tokens`; requests asking for more are clipped. Disabled when unset or non-positive.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_DATA_PARALLEL_BUDGET_INTERVAL`</td>
      <td>Interval for DPBudget updates</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DEFAULT`</td>
      <td>Default weight value for scheduler recv skipper counter (used when forward mode doesn't match specific modes). Only active when `--scheduler-recv-interval > 1`. The counter accumulates weights and triggers request polling when reaching the interval threshold.</td>
      <td>`1000`</td>
    </tr>

    <tr>
      <td>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_DECODE`</td>
      <td>Weight increment for decode forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during decode phase.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_TARGET_VERIFY`</td>
      <td>Weight increment for target verify forward mode in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency during verification phase.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_SCHEDULER_RECV_SKIPPER_WEIGHT_NONE`</td>
      <td>Weight increment when forward mode is None in scheduler recv skipper. Works with `--scheduler-recv-interval` to control polling frequency when no specific forward mode is active.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_MM_BUFFER_SIZE_MB`</td>
      <td>Size of preallocated GPU buffer (in MB) for multi-modal feature hashing optimization. When set to a positive value, temporarily moves features to GPU for faster hash computation, then moves them back to CPU to save GPU memory. Larger features benefit more from GPU hashing. Set to `0` to disable.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_MM_PRECOMPUTE_HASH`</td>
      <td>Enable precomputing of hash values for MultimodalDataItem</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH`</td>
      <td>Enable NCCL for gathering when preparing mlp sync batch under overlap scheduler (without this flag gloo is used for gathering)</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_SYMM_MEM_PREALLOC_GB_SIZE`</td>
      <td>Size of preallocated GPU buffer (in GB) for NCCL symmetric memory pool to limit memory fragmentation. Only have an effect when server arg `--enable-symm-mem` is set.</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR`</td>
      <td>Skip-softmax threshold scale factor for TRT-LLM prefill attention in flashinfer. `None` means standard attention. See [https://arxiv.org/abs/2512.12087](https://arxiv.org/abs/2512.12087)</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`SGLANG_SKIP_SOFTMAX_DECODE_THRESHOLD_SCALE_FACTOR`</td>
      <td>Skip-softmax threshold scale factor for TRT-LLM decode attention in flashinfer. `None` means standard attention. See [https://arxiv.org/abs/2512.12087](https://arxiv.org/abs/2512.12087)</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_SGL_FA3_KERNEL`</td>
      <td>Use sgl-kernel implementation for FlashAttention v3</td>
      <td>`true`</td>
    </tr>
  </tbody>
</table>

## DeepGEMM Configuration (Advanced Optimization)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_ENABLE_JIT_DEEPGEMM`</td>
      <td>Enable Just-In-Time compilation of DeepGEMM kernels (enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) GPUs when the DeepGEMM package is installed; set to `"0"` to disable)</td>
      <td>`"true"`</td>
    </tr>

    <tr>
      <td>`SGLANG_JIT_DEEPGEMM_PRECOMPILE`</td>
      <td>Enable precompilation of DeepGEMM kernels</td>
      <td>`"true"`</td>
    </tr>

    <tr>
      <td>`SGLANG_IN_DEEPGEMM_PRECOMPILE_STAGE`</td>
      <td>Indicator flag used during the DeepGEMM precompile script</td>
      <td>`"false"`</td>
    </tr>

    <tr>
      <td>`SGLANG_DG_CACHE_DIR`</td>
      <td>Directory for caching compiled DeepGEMM kernels</td>
      <td>`{SGLANG_CACHE_DIR}/deep_gemm`</td>
    </tr>

    <tr>
      <td>`SGLANG_DG_USE_NVRTC`</td>
      <td>Use NVRTC instead of NVCC for DeepGEMM JIT compilation (Experimental)</td>
      <td>`"false"`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_DEEPGEMM_BMM`</td>
      <td>Use DeepGEMM for Batched Matrix Multiplication (BMM) operations</td>
      <td>`"false"`</td>
    </tr>

    <tr>
      <td>`SGLANG_JIT_DEEPGEMM_FAST_WARMUP`</td>
      <td>Precompile less kernels during warmup, which reduces the warmup time from 30min to less than 3min. Might cause performance degradation during runtime.</td>
      <td>`"false"`</td>
    </tr>
  </tbody>
</table>

## DeepEP Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
      <td>The maximum number of dispatched tokens on each GPU</td>
      <td>`"128"`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
      <td>The maximum number of dispatched tokens on each GPU for --moe-a2a-backend=flashinfer</td>
      <td>`"1024"`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_MEGAMOE_COMBINE_DTYPE`</td>
      <td>FlashInfer NVFP4 MegaMOE cross-rank combine wire format. Supported values are `bf16`, `mxfp8`, and `nvfp4`. Quantized formats are incompatible with `SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE=1`.</td>
      <td>`"bf16"`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_MEGAMOE_MAX_TOKENS_PER_RANK`</td>
      <td>Per-rank FlashInfer MegaMOE symmetric-workspace token capacity. A value of `0` derives the capacity from the runtime token limits.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE`</td>
      <td>Permit CuTe DSL MegaMOE in-kernel FC2 reduction for NVFP4 W4A4, NVFP4 W4A16, and MXFP8. FlashInfer's tuning profile may still select external reduction. BF16 atomic accumulation is nondeterministic; leave disabled for reproducibility. Requires BF16 combine.</td>
      <td>`"false"`</td>
    </tr>

    <tr>
      <td>`SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS`</td>
      <td>Number of SMs used for DeepEP combine when single batch overlap is enabled</td>
      <td>`"32"`</td>
    </tr>

    <tr>
      <td>`SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO`</td>
      <td>Run shared experts on an alternate stream when single batch overlap is enabled on GB200. When not setting this flag, shared experts and down gemm will be overlapped with DeepEP combine together.</td>
      <td>`"false"`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_QWEN_DEEPEP_SHARED_OVERLAP`</td>
      <td>Run the Qwen shared expert on an alternate CUDA stream while DeepEP executes the routed experts.</td>
      <td>`"true"`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISABLED_MODEL_ARCHS`</td>
      <td>Comma-separated list of model architectures to disable from auto-registration.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_SORT_WEIGHT_FILES`</td>
      <td>Controls weight-file ordering for load-time I/O optimization. `-1` disables sorting/staggering (original order); `0` sorts files only; a value `k` greater than 0 sorts and staggers per-rank order with factor `k` for better multi-rank I/O concurrency.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_RETURN_ORIGINAL_LOGPROB`</td>
      <td>Return the original (pre-temperature) logprobs instead of the post-sampling values.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_COLOCATED_BATCH_GEN`</td>
      <td>Enable colocated batch generation.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_MOE_DEFERRED_FINALIZE`</td>
      <td>Defer the MoE finalize step to overlap it with other work.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_PATCH_TOKENIZER`</td>
      <td>Patch the tokenizer to cache `all_special_tokens`/`all_special_ids` (notably for Kimi tiktoken, where ITL can otherwise regress under high batch size).</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_LOGPROB_CHUNK`</td>
      <td>Process logits in chunks to reduce peak memory.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_LOGPROB_CHUNK_SIZE`</td>
      <td>Chunk size (in tokens) used when logits-processor chunking is enabled.</td>
      <td>`2048`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_USE_PAGED`</td>
      <td>Use the paged FlashInfer attention path.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_WORKSPACE_SIZE`</td>
      <td>FlashInfer workspace size in bytes (default ≈ 384 MiB).</td>
      <td>`402653184`</td>
    </tr>

    <tr>
      <td>`SGLANG_EAGER_INPUT_NO_COPY`</td>
      <td>In eager forward, wrap the ForwardBatch's own tensors instead of copying them into the CUDA graph buffer registry (skips a per-iter device-to-device copy).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DEEPGEMM_SANITY_CHECK`</td>
      <td>Run extra sanity checks on DeepGEMM kernels.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DEEPGEMM_PDL`</td>
      <td>Enable Programmatic Dependent Launch (PDL) for DeepGEMM kernels.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_PP_PARALLEL_DEEPGEMM_WARMUP`</td>
      <td>Run DeepGEMM warmup in parallel across pipeline-parallel ranks.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISABLE_STATIC_WATERFILL`</td>
      <td>Force dynamic Waterfill with runtime EP all-reduce instead of the default static local-batch path.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NIXL_EP_BF16_DISPATCH`</td>
      <td>Use BF16 for NIXL-EP dispatch.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NIXL_EP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
      <td>Maximum number of dispatched tokens per GPU for NIXL-EP.</td>
      <td>`128`</td>
    </tr>
  </tbody>
</table>

## MORI Configuration

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_MORI_DISPATCH_DTYPE`</td>
      <td>Override MoRI-EP dispatch quantization type. `auto` uses auto-detection from weight dtype; `bf16`/`fp8`/`fp4` forces the specified type for all layers</td>
      <td>`"auto"`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_COMBINE_DTYPE`</td>
      <td>Combine dtype: `auto`, `bf16`, `fp8`, `fp4`, or `fp8_direct_cast`</td>
      <td>`"auto"`</td>
    </tr>

    <tr>
      <td>`MORI_DISABLE_AUTO_XGMI`</td>
      <td>Set to `0` to allow Mori to automatically use XGMI for same-node PD disaggregation when no active RDMA device is available.</td>
      <td>`unset`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
      <td>Maximum number of dispatch tokens per rank for MORI-EP buffer allocation</td>
      <td>`4096`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_DISPATCH_INTER_KERNEL_SWITCH_THRESHOLD`</td>
      <td>Threshold for switching between `InterNodeV1` and `InterNodeV1LL` kernel types. `InterNodeV1LL` is used if `SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK` is less than or equal to this threshold; otherwise, `InterNodeV1` is used.</td>
      <td>`256`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_PREALLOC_MAX_RECV_TOKENS`</td>
      <td>This argument devives `SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK` which indicates customized amount of tokens preallocated for a rank, valid range from 1 to world\_size\*SGLANG\_MORI\_NUM\_MAX\_DISPATCH\_TOKENS\_PER\_RANK, by default `0` means maximum. Setting a smaller value will reduce memory footprint but too small value could cause buffer overflow.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_MOE_MAX_INPUT_TOKENS`</td>
      <td>Manual MoE input row limit for MORI EPv1 and EPv2. A positive value overrides `SGLANG_MORI_RECV_BOUND` and is used unrounded, capped at the receive capacity; it only narrows the MoE input view, not the communication buffer. It must be >= the actual received tokens, or results are incorrect. `0` leaves it to `SGLANG_MORI_RECV_BOUND`.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_EP_V2`</td>
      <td>Select MORI EPv2 for `--moe-a2a-backend mori`. When unset, EPv2 is used only for DeepSeek-V4 models on AITER with `ep_size` \<= 8 (on gfx1250, any `ep_size` that fits one UALink vPOD), a bfloat16 model dtype and `MORI_ENABLE_SDMA` off, and MORI EPv1 otherwise. Set to `true` to use EPv2 even when these checks fail (a warning names the failed check), or `false` to always use EPv1.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_RECV_BOUND`</td>
      <td>Hand the MoE only the receive rows the per-rank DP token counts can fill (rounded up) instead of the full MORI receive buffer. Unverified layouts, missing metadata and two-batch overlap keep the full view.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_PPLX_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
      <td>Maximum number of dispatched tokens per rank for the PPLX (`--moe-a2a-backend pplx`) NVSHMEM buffer allocation. Must be ≥ the worst-case per-rank forward tokens (the per-rank prefill chunk `chunked_prefill_size / dp_size`, or the decode CUDA-graph batch size); the server refuses to start otherwise.</td>
      <td>`128`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_QP_PER_TRANSFER`</td>
      <td>Number of RDMA Queue Pairs (QPs) used per transfer operation</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_POST_BATCH_SIZE`</td>
      <td>Number of RDMA work requests posted in a single batch to each QP</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_NUM_WORKERS`</td>
      <td>Number of worker threads in the RDMA executor thread pool</td>
      <td>`1`</td>
    </tr>
  </tbody>
</table>

## DSA Backend Configuration (For DeepSeek V3.2)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_DSA_FUSE_TOPK`</td>
      <td>Fuse the operation of picking topk logits and picking topk indices from page table.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSA_TOPK_FLASHINFER_DETERMINISTIC`</td>
      <td>Use deterministic FlashInfer topk kernels when either `--dsa-topk-backend=flashinfer` or `--speculative-dsa-topk-backend=flashinfer`.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSA_TOPK_FLASHINFER_TIE_BREAK`</td>
      <td>Tie-break mode for FlashInfer DSA topk when either `--dsa-topk-backend=flashinfer` or `--speculative-dsa-topk-backend=flashinfer`: unset disables explicit tie-breaking, `small` prefers the smaller candidate index for equal scores, and `large` prefers the larger candidate index for equal scores. Setting this variable makes FlashInfer use deterministic topk.</td>
      <td>`unset`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD`</td>
      <td>When the maximum kv len in current prefill batch exceeds this value, the sparse mla kernel will be applied, else it falls back to dense MHA implementation. Default to the index topk of model (2048 for DeepSeek V3.2).</td>
      <td>`2048`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSA_TOPK_BROADCAST`</td>
      <td>Experimental. When enabled, broadcast the finalized NSA/DSA indexer top-k result from attention TP rank 0 to the other attention TP ranks. This can mitigate top-k mismatches in TP attention runs at the cost of some speed.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_SEND_AUX_RDMA`</td>
      <td>Send CPU-resident AUX data via RDMA instead of ZMQ TCP.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_TRANSFER_SHARDS`</td>
      <td>Number of sharded synchronous worker threads draining KV transfers; also bounds outstanding transfers (primary RDMA send-queue throttle).</td>
      <td>`8`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_WAIT_POLL_MS`</td>
      <td>Poll cadence (ms) at which a transfer worker wakes to check the SLA while waiting for completion.</td>
      <td>`1000`</td>
    </tr>

    <tr>
      <td>`SGLANG_MORI_TRANSFER_TIMEOUT_MS`</td>
      <td>Per-transfer SLA (ms) before a KV transfer is failed; `0` disables the SLA.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSA_HIP_DISABLE_PRESHUFFLE`</td>
      <td>Disable weight pre-shuffle on the HIP DSA path.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSA_MQA_LOGITS_FREE_MEM_FRACTION`</td>
      <td>Fraction of free memory the MQA-logits step may use on the DSA path.</td>
      <td>`0.2`</td>
    </tr>
  </tbody>
</table>

## Memory Management

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_DEBUG_MEMORY_POOL`</td>
      <td>Enable memory pool debugging</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_CLIP_MAX_NEW_TOKENS_ESTIMATION`</td>
      <td>Clip max new tokens estimation for memory planning</td>
      <td>`4096`</td>
    </tr>

    <tr>
      <td>`SGLANG_DETOKENIZER_MAX_STATES`</td>
      <td>Maximum states for detokenizer</td>
      <td>Default value based on system</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK`</td>
      <td>Enable checks for memory imbalance across Tensor Parallel ranks</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_MOONCAKE_CUSTOM_MEM_POOL`</td>
      <td>Configure the custom memory pool type for Mooncake. Supports `NVLINK`, `BAREX`, `INTRA_NODE_NVLINK`. If set to `true`, it defaults to `NVLINK`.</td>
      <td>`None`</td>
    </tr>
  </tbody>
</table>

## Model-Specific Options

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_USE_AITER`</td>
      <td>Use AITER optimize implementation</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ROCM_USE_MULTI_STREAM`</td>
      <td>Allocate alt CUDA/HIP stream on ROCm/AITER to overlap shared and routed experts in DeepseekV2 MoE. Requires the HIP env `GPU_MAX_HW_QUEUES>=5` (default `4`, the cap on HSA/ROCr HW queues HIP creates) so the alt stream gets its own queue instead of serializing with the main stream. Best paired with `--deepep-mode low_latency` so Mori's AsyncLL kernel offloads dispatch/combine to copy engines and frees CUs.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MOE_PADDING`</td>
      <td>Enable MoE padding (sets padding size to 128 if value is `1`, often set to `1` in Docker builds)</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_FUSED_PARALLEL_QKNORM`</td>
      <td>Use the fused parallel QK RMSNorm kernel for MiniMax-M2.x on CUDA when attention TP size > 1</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_BUSY`</td>
      <td>Enable strict memory checks while the scheduler is busy.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE`</td>
      <td>Enable strict memory checks while the scheduler is idle.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_NATIVE_MOVE_KV_CACHE`</td>
      <td>Use the native implementation to move KV cache entries.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_BREAKABLE_CUDA_GRAPH`</td>
      <td>Use a breakable CUDA graph so it can be interrupted/rebuilt at runtime.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MEMORY_SAVER_CUDA_GRAPH`</td>
      <td>Allow CUDA graphs under the release/resume memory saver.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_GEMMA_OUT_OF_PLACE_POSITION_MUTATION`</td>
      <td>Use out-of-place position mutation for Gemma models.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MAMBA_CONV_DTYPE`</td>
      <td>dtype for the Mamba convolution state.</td>
      <td>`bfloat16`</td>
    </tr>

    <tr>
      <td>`SGLANG_MAMBA_SSM_DTYPE`</td>
      <td>dtype for the Mamba SSM state (defaults to the model dtype when unset).</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_EMBEDDINGS_SPARSE_HEAD`</td>
      <td>Name of the sparse-embeddings head to expose for embedding models.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_DSV4_FP4_EXPERTS`</td>
      <td>Whether DeepSeek V4 experts use FP4. Set to `false` when using an FP4-to-FP8 converted DeepSeek V4 checkpoint.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSV4_REASONING_EFFORT`</td>
      <td>Default `reasoning_effort` for DeepSeek-V4 when a request does not set it: a deepseek-recipe tier name. Unset means deepseek-recipe's default.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSV41_REASONING_EFFORT`</td>
      <td>Default `reasoning_effort` for DeepSeek-V4.1 when a request does not set it: a deepseek-recipe tier name or an integer budget in \[1, 100]. Unset means deepseek-recipe's default.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_DSV4_USE_BF16_KV_QUANT_SOURCE`</td>
      <td>For DeepSeek V4, quantize the SWA FP8 KV cache from BF16-rounded values instead of FP32 registers. This matches trainer-side QAT and the DSA prefill-CP path, at the cost of an extra BF16 KV materialization and separate cache-store kernels.</td>
      <td>`false`</td>
    </tr>
  </tbody>
</table>

## Quantization

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_INT4_WEIGHT`</td>
      <td>Enable INT4 weight quantization</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_FORCE_FP8_MARLIN`</td>
      <td>Force using FP8 MARLIN kernels even if other FP8 kernels are available</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NVFP4_CKPT_FP8_GEMM_IN_ATTN`</td>
      <td>Quantize q\_b\_proj from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MOE_NVFP4_DISPATCH`</td>
      <td>Use nvfp4 for moe dispatch (on flashinfer\_cutlass or flashinfer\_cutedsl moe runner backend)</td>
      <td>`"false"`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION`</td>
      <td>Enable FlashInfer TRT-LLM or CuTe DSL v2 (no A2A or FlashInfer A2A) per-token FP32 activation scaling for serialized `modelopt_fp4` checkpoints; checkpoint activation scales are treated as `1`</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16`</td>
      <td>Use BF16 activations and outputs with FlashInfer CuTe DSL NVFP4 weights on NVIDIA SM100-family GPUs (SM100/SM103). Serialized dense linear layers use this mode with `--fp4-gemm-backend flashinfer_cutedsl`. With `--moe-runner-backend flashinfer_cutedsl`, MoE layers support online weight quantization and serialized ModelOpt NVFP4 weights with either no A2A or FlashInfer A2A, and honor `SGLANG_FLASHINFER_MOE_FUSED_FINALIZE`. With `--moe-runner-backend flashinfer_megamoe --moe-a2a-backend flashinfer_megamoe`, MoE layers support serialized ModelOpt NVFP4 weights and `--quantization nvfp4_online` for gated SiLU/SwiGLU or SiTU. MegaMOE uses BF16 activations and combine, with equal TP, attention-DP, and EP sizes. For more on the W4A16 MoE design and performance, see [Faster-than-light NVFP4 decode—with precision](https://humansand.ai/blog/faster-than-light-nvfp4-decode) by Ziang Li at humans& ai.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_MOE_FUSED_FINALIZE`</td>
      <td>Use FlashInfer's fused atomic CUTLASS and CuTe DSL MoE finalize. Disabled by default for better numerical accuracy. Set to `1` to opt in for more aggressive performance. Deterministic inference overrides this to `false`.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`FLASHINFER_NVFP4_4OVER6`</td>
      <td>Enable FlashInfer NVFP4 4over6 scaling for NVFP4 per-token activation and online NVFP4 MoE weight quantization paths</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`FLASHINFER_NVFP4_4OVER6_E4M3_USE_256`</td>
      <td>Use `256` as the E4M3 scale maximum for FlashInfer NVFP4 4over6 scaling; otherwise uses `448`</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE`</td>
      <td>Quantize moe of nextn layer from BF16 to FP8 when launching DeepSeek NVFP4 checkpoint</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_QUANT_ALLOW_DOWNCASTING`</td>
      <td>Allow weight dtype downcasting during loading (e.g., fp32 → fp16). By default, SGLang rejects this kind of downcasting when using quantization.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_FP8_IGNORED_LAYERS`</td>
      <td>A comma-separated list of layer names to ignore during FP8 quantization. For example: `model.layers.0,model.layers.1.,qkv_proj`.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_FP4_IGNORED_LAYERS`</td>
      <td>A comma-separated list of layer names to keep out of online FP4 conversion for `modelopt_fp4` or `nvfp4_online`. For example: `model.layers.40,model.layers.41`.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_HUMMING_ONLINE_QUANT_CONFIG`</td>
      <td>JSON object or JSON file path for Humming online weight quantization. When a layer has no checkpoint quantization config, this config tells Humming how to quantize the loaded fp16/bf16 weight. When the checkpoint already has a Humming config, add `"force_requant": true` to requantize it to this schema during loading. Common keys include `dtype`/`weight_dtype`, `scale_dtype`, `group_size`, `scale_type`, `ignored_layers`, `ignore`, and `modules_to_not_convert`.</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`SGLANG_HUMMING_INPUT_QUANT_CONFIG`</td>
      <td>JSON object or JSON file path for Humming input activation quantization. This controls the activation dtype and scale grouping passed into Humming kernels, independently of the weight schema. For example, `{"dtype": "float8e4m3"}` quantizes Humming inputs to FP8 E4M3.</td>
      <td>`None`</td>
    </tr>

    <tr>
      <td>`SGLANG_HUMMING_USE_F16_ACCUM`</td>
      <td>Use FP16 accumulation in Humming compute/tuning config. This is only meaningful for Humming dtype combinations that support FP16 accumulation, such as fp16 or FP8 E4M3 activations with float16 output. Leave it `false` for the default accumulator behavior.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_HUMMING_MOE_GEMM_TYPE`</td>
      <td>Select the Humming MoE GEMM path for standard dispatch and DeepEP normal dispatch. `indexed` uses top-k expert ids directly and is the fallback for unset or unknown values. `grouped` maps to Humming grouped-contiguous GEMM. DeepEP low-latency dispatch uses grouped-masked GEMM internally and does not use this selector.</td>
      <td>`""` (`indexed`)</td>
    </tr>
  </tbody>
</table>

## Distributed Computing

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_BLOCK_NONZERO_RANK_CHILDREN`</td>
      <td>Control blocking of non-zero rank children processes</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_IS_FIRST_RANK_ON_NODE`</td>
      <td>Indicates if the current process is the first rank on its node</td>
      <td>`"true"`</td>
    </tr>

    <tr>
      <td>`SGLANG_PP_LAYER_PARTITION`</td>
      <td>Pipeline parallel layer partition specification</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_ONE_VISIBLE_DEVICE_PER_PROCESS`</td>
      <td>Set one visible device per process for distributed computing</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_RAY_BUNDLE_INDICES`</td>
      <td>Comma-separated bundle indices for Ray actor placement (e.g., `"0,1,2,3"`). Must match world\_size. Enables fine-grained GPU assignment in custom placement groups.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_CPU_QUANTIZATION`</td>
      <td>Enable CPU-side quantization.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`USE_TRITON_W8A8_FP8_KERNEL`</td>
      <td>Use the Triton W8A8 FP8 kernel.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_MESSAGE_QUEUE_BROADCASTER`</td>
      <td>Use the shared-memory message-queue broadcaster for inter-process tensor broadcast.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISTRIBUTED_INIT_METHOD_OVERRIDE`</td>
      <td>Override the init method used by `torch.distributed.init_process_group`. Set to `env://` to use an externally-created TCPStore via `MASTER_ADDR`/`MASTER_PORT`.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_TCP_STORE_PORT`</td>
      <td>Port for the torch.distributed TCPStore.</td>
      <td>`29600`</td>
    </tr>

    <tr>
      <td>`SGLANG_SYNC_TOKEN_IDS_ACROSS_TP`</td>
      <td>Synchronize sampled token ids across tensor-parallel ranks.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_KV_SHARD_ZERO_SM_GATHER`</td>
      <td>With `--enable-kv-cache-sharding`, register the prefix gather scratch of single-node shard groups as an NCCL symmetric window so NCCL 2.31.2+ runs the all-gather on copy engines (zero SMs). Older NCCL versions keep the SM-kernel all-gather. On single-node deployments this also makes the server set `NCCL_CUMEM_ENABLE=1` for every NCCL communicator unless it is already set, except under `--enable-memory-saver` or `SGLANG_MOONCAKE_CUSTOM_MEM_POOL`; see `--enable-kv-cache-sharding` for the full conditions. Set to `false` to keep the SM-kernel all-gather and the default `NCCL_CUMEM_ENABLE=0`.</td>
      <td>`true`</td>
    </tr>
  </tbody>
</table>

## PD Disaggregation — Staging Buffer (Heterogeneous TP)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_DISAGG_STAGING_BUFFER`</td>
      <td>Enable GPU staging buffer for heterogeneous TP KV transfer. Required when prefill and decode use different TP/attention-TP sizes. Only for non-MLA models (e.g. GQA, MHA).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGG_STAGING_POOL_SIZE_MB`</td>
      <td>Decode-side ring buffer pool total size in MB. Shared buffer receiving RDMA data from all prefill ranks. Larger values support higher concurrency.</td>
      <td>`4096`</td>
    </tr>

    <tr>
      <td>`SGLANG_STAGING_USE_TORCH`</td>
      <td>Force using PyTorch gather/scatter fallback instead of Triton fused kernels for staging operations. Useful for debugging.</td>
      <td>`false`</td>
    </tr>
  </tbody>
</table>

## Testing & Debugging (Internal/CI)

*These variables are primarily used for internal testing, continuous integration, or debugging.*

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_IS_IN_CI`</td>
      <td>Indicates if running in CI environment</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_IS_IN_CI_AMD`</td>
      <td>Indicates running in AMD CI environment</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_TEST_RETRACT`</td>
      <td>Enable retract decode testing</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_TEST_RETRACT_NO_PREFILL_BS`</td>
      <td>When SGLANG\_TEST\_RETRACT is enabled, no prefill is performed if the batch size exceeds SGLANG\_TEST\_RETRACT\_NO\_PREFILL\_BS.</td>
      <td>`2 ** 31`</td>
    </tr>

    <tr>
      <td>`SGLANG_RECORD_STEP_TIME`</td>
      <td>Record step time for profiling</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_TEST_REQUEST_TIME_STATS`</td>
      <td>Test request time statistics</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DEBUG_SYMM_MEM`</td>
      <td>Enable debug checks that verify tensors passed to NCCL communication ops are allocated in the symmetric memory pool. Logs warnings (rank 0 only) with stack traces for any tensor not in the pool.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_LOGLEVEL`</td>
      <td>Controls crash-debug kernel API logging. `0` disables logging, `1` logs API names, `3` logs tensor metadata, `5` adds tensor statistics, and `10` also writes pre-call dump snapshots.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_LOGDEST`</td>
      <td>Destination for crash-debug kernel API logs. Use `stdout`, `stderr`, or a file path. `%i` is replaced with the process PID.</td>
      <td>`stdout`</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_DUMP_DIR`</td>
      <td>Output directory for level-10 kernel API input/output dumps. `%i` is replaced with the process PID.</td>
      <td>`sglang_kernel_api_dumps`</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_DUMP_INCLUDE`</td>
      <td>Comma-separated wildcard patterns for kernel API names to include in level-10 dumps.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_DUMP_EXCLUDE`</td>
      <td>Comma-separated wildcard patterns for kernel API names to exclude from level-10 dumps.</td>
      <td>Not set</td>
    </tr>
  </tbody>
</table>

## Profiling & Benchmarking

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_TORCH_PROFILER_DIR`</td>
      <td>Directory for PyTorch profiler output</td>
      <td>`/tmp`</td>
    </tr>

    <tr>
      <td>`SGLANG_PROFILE_WITH_STACK`</td>
      <td>Set `with_stack` option (bool) for PyTorch profiler (capture stack trace)</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_PROFILE_RECORD_SHAPES`</td>
      <td>Set `record_shapes` option (bool) for PyTorch profiler (record shapes)</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS`</td>
      <td>Config BatchSpanProcessor.schedule\_delay\_millis if tracing is enabled</td>
      <td>`500`</td>
    </tr>

    <tr>
      <td>`SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE`</td>
      <td>Config BatchSpanProcessor.max\_export\_batch\_size if tracing is enabled</td>
      <td>`64`</td>
    </tr>

    <tr>
      <td>`SGLANG_TRACE_ASYNC`</td>
      <td>Enable async tracing: span creation is offloaded to a dedicated exporter process via ZMQ, reducing OTel overhead on the inference hot path</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_TRACE_ASYNC_FLUSH_THRESHOLD`</td>
      <td>Number of buffered trace operations before an automatic flush to the exporter process</td>
      <td>`100`</td>
    </tr>

    <tr>
      <td>`SGLANG_PROFILE_V2`</td>
      <td>Use the v2 profiler implementation.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DETECT_SLOW_RANK`</td>
      <td>Detect and report ranks that fall behind during collective ops.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_RANK_CONSENSUS_CHECKER`</td>
      <td>Check for PP/TP-rank divergence.  Kill the server when divergence occurs.  Helpful for trouble-shooting server hangs issues.</td>
      <td>`False`</td>
    </tr>

    <tr>
      <td>`SGLANG_FORCE_SHUTDOWN`</td>
      <td>Force an immediate process-group shutdown on exit.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_PYSPY_DUMP_BEFORE_CRASH`</td>
      <td>Capture a py-spy stack dump of all processes before crashing.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_CUDA_COREDUMP`</td>
      <td>Enable CUDA coredump generation (auto-injects the required `CUDA_*` env vars).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_CUDA_COREDUMP_DIR`</td>
      <td>Directory for CUDA coredumps. If unset, resolves to `RUNNER_TEMP` in CI, else `/tmp`.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_CUDA_COREDUMP_BEFORE_CRASH`</td>
      <td>Trigger a CUDA coredump before crashing.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_CUDA_COREDUMP_BEFORE_CRASH_WAIT_SECS`</td>
      <td>Seconds to wait for the CUDA coredump to finish before exiting.</td>
      <td>`60.0`</td>
    </tr>
  </tbody>
</table>

## Storage & Caching

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_WAIT_WEIGHTS_READY_TIMEOUT`</td>
      <td>Timeout period for waiting on weights</td>
      <td>`120`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISABLE_OUTLINES_DISK_CACHE`</td>
      <td>Disable Outlines disk cache</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_CUSTOM_TRITON_KERNEL_CACHE`</td>
      <td>Use SGLang's custom Triton kernel cache implementation for lower overheads (automatically enabled on CUDA)</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_DECODE_OFFLOAD_STRIDE`</td>
      <td>Decode-side incremental KV cache offload stride. Rounded down to a multiple of `--page-size` (min is `--page-size`). If unset/invalid/\<=0, it falls back to `--page-size`.</td>
      <td>Not set (uses `--page-size`)</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_NIXL_USE_DIRECT_IO`</td>
      <td>Enable `O_DIRECT` for any file-based NIXL backend (POSIX, GDS, GDS\_MT, 3FS) when opening cache files (bypasses the OS page cache, reducing memory pressure and improving throughput on NVMe). Can also be disabled via `{"use_direct_io": false}` in `--hicache-storage-backend-extra-config`. Falls back to buffered I/O with a warning when `O_DIRECT` is unavailable on the current OS.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_HUGEPAGE_SIZE`</td>
      <td>Use huge pages for host KV cache allocations (HiCache / disaggregation offload). Valid values: `2MB` (2 MiB pages via `MAP_HUGE_2MB`) or `1GB` (1 GiB pages via `MAP_HUGE_1GB`). Requires huge pages to be pre-allocated on the host OS (`/proc/sys/vm/nr_hugepages` or `/sys/kernel/mm/hugepages`). If the allocation fails, the allocator logs a warning and falls back to regular page-size mmap automatically.</td>
      <td>Not set (uses OS default page size)</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_HF3FS_CONFIG_PATH`</td>
      <td>Path to the HiCache HF3FS backend config file.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR`</td>
      <td>Storage directory for the HiCache file backend.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_FILE_BACKEND_MAX_SIZE`</td>
      <td>Max size for HiCache file-backend LRU eviction (accepts SI/IEC suffixes; `0` disables eviction).</td>
      <td>Not set (eviction off)</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_FILE_BACKEND_EVICTION_RATIO`</td>
      <td>Target fraction to evict down to when the file-backend max size is reached.</td>
      <td>`0.9`</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_FILE_BACKEND_MIN_FREE_SPACE`</td>
      <td>Minimum free space to keep on the file-backend volume (accepts SI/IEC suffixes).</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR`</td>
      <td>Storage directory for the HiCache NIXL backend.</td>
      <td>Not set</td>
    </tr>
  </tbody>
</table>

## Function Calling / Tool Use

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_TOOL_STRICT_LEVEL`</td>
      <td>Controls the strictness level of tool call parsing and validation. \<br>**Level 0**: Off - No strict validation \<br>**Level 1**: Function strict - Enables structural tag constraints for all tools (even if none have `strict=True` set) \<br>**Level 2**: Parameter strict - Enforces strict parameter validation for all tools, treating them as if they all have `strict=True` set</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_DEFAULT_THINKING`</td>
      <td>Enable model thinking/reasoning output by default.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MAX_THINK_TOKENS`</td>
      <td>Cap on thinking tokens. Negative means unlimited; `0` or greater caps the count.</td>
      <td>`-1`</td>
    </tr>
  </tbody>
</table>

## Logging & Observability

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_LOG_GC`</td>
      <td>Log Python garbage-collection pauses.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_LOG_FORWARD_ITERS`</td>
      <td>Log each forward iteration.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_LOG_MS`</td>
      <td>Log per-step timing in milliseconds.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_LOG_REQUEST_EXCEEDED_MS`</td>
      <td>Log requests whose processing time exceeds this many milliseconds. `-1` disables.</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_LOG_SCHEDULER_STATUS_TARGET`</td>
      <td>Target (e.g. a file path) for periodic scheduler-status logging.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_LOG_SCHEDULER_STATUS_INTERVAL`</td>
      <td>Interval (seconds) between scheduler-status log lines.</td>
      <td>`60.0`</td>
    </tr>
  </tbody>
</table>

## Constrained Decoding (Grammar)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_GRAMMAR_POLL_INTERVAL`</td>
      <td>Poll interval (seconds) for asynchronous grammar compilation.</td>
      <td>`0.005`</td>
    </tr>

    <tr>
      <td>`SGLANG_GRAMMAR_MAX_POLL_ITERATIONS`</td>
      <td>Maximum poll iterations before grammar compilation is treated as stuck.</td>
      <td>`10000`</td>
    </tr>
  </tbody>
</table>

## Scheduler & Batching

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_INIT_NEW_TOKEN_RATIO`</td>
      <td>Initial new-token ratio used for memory planning.</td>
      <td>`0.7`</td>
    </tr>

    <tr>
      <td>`SGLANG_MIN_NEW_TOKEN_RATIO_FACTOR`</td>
      <td>Floor factor for the new-token ratio after decay.</td>
      <td>`0.14`</td>
    </tr>

    <tr>
      <td>`SGLANG_NEW_TOKEN_RATIO_DECAY_STEPS`</td>
      <td>Number of steps over which the new-token ratio decays.</td>
      <td>`600`</td>
    </tr>

    <tr>
      <td>`SGLANG_RETRACT_DECODE_STEPS`</td>
      <td>Number of decode steps to look ahead when deciding to retract.</td>
      <td>`20`</td>
    </tr>

    <tr>
      <td>`SGLANG_EMPTY_CACHE_INTERVAL`</td>
      <td>Interval (seconds) at which to empty the device cache; set this if memory accumulates over a long serving period. `-1` disables.</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_FORCE_STREAM_INTERVAL`</td>
      <td>For non-streaming requests, flush intermediate output batches to the tokenizer manager every N decoded tokens (lower to `1` for accurate TTFT benchmarking).</td>
      <td>`50`</td>
    </tr>

    <tr>
      <td>`SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR`</td>
      <td>Smoothing factor for dynamic prefill chunking.</td>
      <td>`0.75`</td>
    </tr>

    <tr>
      <td>`SGLANG_SWA_EVICTION_INTERVAL_MULTIPLIER`</td>
      <td>Multiplier applied to the sliding-window-attention eviction interval.</td>
      <td>`1.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_RADIX_FORCE_MISS`</td>
      <td>Force radix-cache misses (debugging/benchmarking).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_SCHEDULER_SKIP_ALL_GATHER`</td>
      <td>Skip the scheduler all-gather step.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_WAR_BARRIER`</td>
      <td>Force-enable the write-after-read barrier for the overlap scheduler even when CUDA is not detected (e.g. AMD/ROCm). On CUDA the barrier is always enabled.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_PP_SKIP_PURE_CHUNKED_OUTPUT_COMM`</td>
      <td>In pipeline parallel, skip output send/recv when a batch is entirely non-final chunked-prefill requests.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_KILLPG_ON_SCHEDULER_EXCEPTION`</td>
      <td>Kill the whole process group when the scheduler raises an exception.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_REQUEST_STATE_WAIT_TIMEOUT`</td>
      <td>Tokenizer-manager request-state wait timeout (seconds).</td>
      <td>`4`</td>
    </tr>
  </tbody>
</table>

## PD Disaggregation (Runtime)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`</td>
      <td>Thread-pool size for KV transfers. Defaults to a value computed from the CPU count at runtime.</td>
      <td>Not set (computed at runtime)</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_QUEUE_SIZE`</td>
      <td>Disaggregation transfer queue size.</td>
      <td>`4`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT`</td>
      <td>Timeout (seconds) for the disaggregation bootstrap handshake.</td>
      <td>`300`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_WAITING_TIMEOUT`</td>
      <td>Timeout (seconds) for a request waiting on KV transfer.</td>
      <td>`300`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL`</td>
      <td>Interval (seconds) between disaggregation heartbeats.</td>
      <td>`5.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILURE`</td>
      <td>Consecutive heartbeat failures tolerated before a peer is considered dead.</td>
      <td>`2`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_ZMQ_SEND_TIMEOUT`</td>
      <td>Send timeout (seconds) for decode's ZMQ sockets to prefill peers. Raise if healthy sends exceed the default.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_BOOTSTRAP_ENTRY_CLEANUP_INTERVAL`</td>
      <td>Interval (seconds) for cleaning up stale bootstrap entries.</td>
      <td>`120`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_NIXL_BACKEND`</td>
      <td>NIXL transport backend for disaggregation.</td>
      <td>`UCX`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_NIXL_BACKEND_PARAMS`</td>
      <td>JSON parameters passed to the NIXL backend.</td>

      <td>
        `{}`
      </td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER`</td>
      <td>Have all context-parallel ranks participate in KV transfer.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK`</td>
      <td>Force querying the prefill DP rank for routing.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS`</td>
      <td>Extra slots in `req_to_token_pool` for decode workers (effective when `max_num_reqs` greater than 32), letting more KV transfers overlap decode.</td>
      <td>`0`</td>
    </tr>
  </tbody>
</table>

## Mooncake KV Store & Transfer

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_HICACHE_MOONCAKE_CONFIG_PATH`</td>
      <td>Path to the HiCache Mooncake store config file.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_HICACHE_MOONCAKE_REUSE_TE`</td>
      <td>Reuse the Mooncake transfer engine across HiCache operations.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_MOONCAKE_SEND_AUX_TCP`</td>
      <td>Send Mooncake AUX data over TCP.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_FAILED_SESSION_PROBE`</td>
      <td>Periodically probe blacklisted Mooncake sessions and un-blacklist the ones that answer. Probes run on their own thread, off the KV transfer path.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_FAILED_SESSION_PROBE_INTERVAL_S`</td>
      <td>How often (seconds) the probe loop runs, and the first retry interval of a failing session.</td>
      <td>`30.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_FAILED_SESSION_PROBE_MAX_INTERVAL_S`</td>
      <td>Cap (seconds) on a session's probe interval, which doubles after each failed probe.</td>
      <td>`300.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_FAILED_SESSION_IDLE_TIMEOUT_S`</td>
      <td>Drop a blacklisted Mooncake session that no request has targeted for this many seconds, e.g. the old session id of a restarted decode.</td>
      <td>`600.0`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_MASTER`</td>
      <td>Address of the Mooncake master.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`MOONCAKE_CLIENT`</td>
      <td>Mooncake client identifier.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`MOONCAKE_LOCAL_HOSTNAME`</td>
      <td>Local hostname advertised to Mooncake.</td>
      <td>`localhost`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_TE_META_DATA_SERVER`</td>
      <td>Mooncake transfer-engine metadata server.</td>
      <td>`P2PHANDSHAKE`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_GLOBAL_SEGMENT_SIZE`</td>
      <td>Mooncake global segment size.</td>
      <td>`4gb`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_PROTOCOL`</td>
      <td>Mooncake transport protocol.</td>
      <td>`rdma`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_DEVICE`</td>
      <td>Mooncake RDMA device(s).</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_MASTER_METRICS_PORT`</td>
      <td>Port for Mooncake master metrics.</td>
      <td>`9003`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_CHECK_SERVER`</td>
      <td>Check connectivity to the Mooncake server on startup.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_STANDALONE_STORAGE`</td>
      <td>Run Mooncake in standalone storage mode.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_ENABLE_SSD_OFFLOAD`</td>
      <td>Enable SSD offload in Mooncake.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`MOONCAKE_OFFLOAD_FILE_STORAGE_PATH`</td>
      <td>File storage path for Mooncake SSD offload.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`ENABLE_ASCEND_TRANSFER_WITH_MOONCAKE`</td>
      <td>Enable Ascend NPU transfers via Mooncake.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`ASCEND_NPU_PHY_ID`</td>
      <td>Physical Ascend NPU id used for Mooncake transfers. `-1` auto-detects.</td>
      <td>`-1`</td>
    </tr>
  </tbody>
</table>

## Attention & Kernels

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_TRITON_DECODE_ATTN_STATIC_KV_SPLITS`</td>
      <td>Use static KV splits in the Triton decode-attention kernel.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MUSA_FA3_FORCE_UPDATE_METADATA`</td>
      <td>Force FA3 metadata updates on MThreads MUSA.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK`</td>
      <td>Skip the sgl-kernel version compatibility check.</td>
      <td>`false`</td>
    </tr>
  </tbody>
</table>

## Deterministic Inference

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_ENABLE_DETERMINISTIC_INFERENCE`</td>
      <td>Enable deterministic inference (fixed reduction/accumulation order).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_1STAGE_ALLREDUCE`</td>
      <td>Use the 1-stage all-reduce kernel on AMD (deterministic, fixed accumulation order). If unset, it is auto-enabled when deterministic inference is on.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_PCIE_IPC_ALLREDUCE`</td>
      <td>Use FlashInfer's PCIe-IPC all-reduce on switch-free intra-node hosts (no NVLink, no multicast), where the other custom all-reduce backends do not apply. Serves bf16 decode reductions that fit the workspace bound; everything else stays on NCCL. Left off under deterministic inference.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_PCIE_IPC_MAX_NUMEL`</td>
      <td>Elements the PCIe-IPC workspace is sized for; it cannot grow after construction. `0` derives it from the widest decode batch the runner captures for. Raising it hands larger reductions to the kernels, at roughly `2 * world_size * max_numel * itemsize` bytes per rank.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_PREFILL_SPLIT_TILE_SIZE`</td>
      <td>FlashInfer prefill split-tile size for deterministic attention.</td>
      <td>`4096`</td>
    </tr>

    <tr>
      <td>`SGLANG_FLASHINFER_DECODE_SPLIT_TILE_SIZE`</td>
      <td>FlashInfer decode split-tile size for deterministic attention.</td>
      <td>`2048`</td>
    </tr>

    <tr>
      <td>`SGLANG_TRITON_DECODE_SPLIT_TILE_SIZE`</td>
      <td>Triton decode split-tile size for deterministic attention.</td>
      <td>`256`</td>
    </tr>
  </tbody>
</table>

## Speculative Decoding & Overlap

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_ENABLE_OVERLAP_PLAN_STREAM`</td>
      <td>Plan the next step on a separate stream to overlap with the current step (Overlap Spec V2).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_SPEC_SKIP_ZERO_STEP_DRAFT_EXTEND`</td>
      <td>Skip draft\_extend while adaptive spec is at steps=0; saves a draft forward but the draft KV goes stale.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NGRAM_FORCE_GREEDY_VERIFY`</td>
      <td>Force greedy verification for the n-gram speculative path.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_SANITIZE_NAN_LOGITS`</td>
      <td>Sanitize NaN logits before sampling kernels and emit a throttled warning.</td>
      <td>`true`</td>
    </tr>
  </tbody>
</table>

## EPLB (Expert Parallel Load Balancing)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR`</td>
      <td>Output directory for the expert-distribution recorder.</td>
      <td>`/tmp`</td>
    </tr>

    <tr>
      <td>`SGLANG_LOG_EXPERT_LOCATION_METADATA`</td>
      <td>Log expert-location metadata.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_EXPERT_LOCATION_UPDATER_LOG_INPUT`</td>
      <td>Log inputs to the expert-location updater.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_EXPERT_LOCATION_UPDATER_LOG_METRICS`</td>
      <td>Log metrics from the expert-location updater.</td>
      <td>`false`</td>
    </tr>
  </tbody>
</table>

## AMD & ROCm

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_USE_AITER_AG`</td>
      <td>Use the AITER all-gather implementation.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_AITER_UNIFIED_ATTN`</td>
      <td>Use the AITER unified attention kernel.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_AITER_FP8_PER_TOKEN`</td>
      <td>Use AITER FP8 per-token quantization.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION`</td>
      <td>Use an explicitly passed `--mem-fraction-static` as-is on AITER with context length above 8192, instead of scaling it by 0.85 to reserve non-static attention workspace. Skipping the reserve can OOM long-context serving; set it only when the scaled fraction is too small to hold the model weights.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_AITER_GFX942_BLOCKSCALE_USE_CK`</td>
      <td>On gfx942, run dense block-FP8 linears with the AITER CK blockscale GEMM instead of the AITER Triton one. Faster, especially at decode batch sizes, but check accuracy: it lowers GSM8K on Qwen3-32B-FP8 by about 2 points.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_AITER_MOE_GU_ITLV`</td>
      <td>Select the AITER MoE gate/up tile layout: `true` interleaves, `false` uses the separated layout.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_AITER_FUSE_RMSNORM_PAD`</td>
      <td>Fuse the residual-add + RMSNorm + zero-pad triplet before the MoE block via the AITER Triton kernel (TP=1, post-attention layernorm path only).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_AITER_KV_CACHE_LAYOUT`</td>
      <td>Physical layout for the MHA KV cache on AITER: `nhd` or `vectorized_5d` (SHUFFLE layout enabling pa\_decode\_gluon).</td>
      <td>`nhd`</td>
    </tr>

    <tr>
      <td>`SGLANG_AITER_PAGED_PREFILL_ASM`</td>
      <td>Let gfx950 fp8 hd256 chunked prefill read the KV pool in place through AITER's paged-varlen ASM kernel, instead of gathering the prefix into a contiguous buffer first. The ASM kernel is compiled for a 4D LINEAR `[N, 64, H, D]` KV cache, so it serves `--page-size 64` on the NHD pool and nothing else; at any other page size this variable has no effect whichever way it is set. Where it does apply, the gain grows with context and is flat to marginally negative at moderate ISL, so set it to `false` for workloads that are not long-context.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_ROCM_FUSED_DECODE_MLA`</td>
      <td>Use the fused decode MLA kernel on ROCm.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ROCM_DISABLE_LINEARQUANT`</td>
      <td>Disable linear-layer quantization on ROCm.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_HACK_FLASHMLA_BACKEND`</td>
      <td>DeepSeek-V4 HIP decode attention kernel: `auto` takes aiter's gluon sparse kernel on gfx950 and the tilelang partial + combine elsewhere; `aiter_sparse`, `tilelang`, `triton`, `torch`, `comparison` or `unified_kv_triton` force one.</td>
      <td>`auto`</td>
    </tr>

    <tr>
      <td>`SGLANG_ROCM_K3_FUSE_KDA_INPROJ`</td>
      <td>Kimi-K3 on ROCm: fold the KDA `[f_a|b]` tail into the wide `[q,k,v,g]` projection so the whole input projection is one GEMM. Applies to unquantized weights only; falls back to the split projection otherwise.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_ROCM_K3_FUSE_KDA_INPROJ_MAX_TOKENS`</td>
      <td>Token count above which `SGLANG_ROCM_K3_FUSE_KDA_INPROJ` stops applying and the split projection runs instead. The merged shape is only faster while the projection is bandwidth bound.</td>
      <td>`256`</td>
    </tr>
  </tbody>
</table>

## NPU (Ascend)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT`</td>
      <td>Disable ACL-format weight conversion on NPU.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_USE_MULTI_STREAM`</td>
      <td>Use multiple streams on NPU.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_USE_MLAPO`</td>
      <td>Use the MLAPO path on NPU.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_FORWARD_NATIVE_GELUTANH`</td>
      <td>Use the native gelu-tanh activation forward (for Skywork-Reward-Gemma-2-27B-v0.2).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_NPU_FORWARD_NATIVE_GEMMA_RMS_NORM`</td>
      <td>Use the native Gemma RMSNorm forward (for Skywork-Reward-Gemma-2-27B-v0.2).</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_AG_AFTER_QLORA`</td>
      <td>Delay all-gather until after QLoRA for better DeepSeek V3.2 performance.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_EXPERIMENTAL_LORA_OPTI`</td>
      <td>Master switch for the experimental TRT-LLM LoRA fast path. When off, all fine-grained opt switches read false.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ZBAL_LOCAL_MEM_SIZE`</td>
      <td>Local memory size for the ZBAL (zero-buffer accelerate library) path (NPU only).</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`SGLANG_ZBAL_BOOTSTRAP_URL`</td>
      <td>Bootstrap URL for the ZBAL path (NPU only).</td>
      <td>`""`</td>
    </tr>
  </tbody>
</table>

## Apple Silicon (MLX / MPS)

These variables configure the SRT MLX backend. SGLang Diffusion uses PyTorch
MPS and does not read them.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_USE_MLX`</td>
      <td>Use the MLX backend on Apple Silicon.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MLX_USE_CUSTOM_ROPE`</td>
      <td>Use the custom RoPE kernel on MLX.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MLX_FUSE_SWIGLU`</td>
      <td>Fuse the SwiGLU activation on MLX.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MLX_CLEAR_CACHE_STEPS`</td>
      <td>Number of decode steps between `mx.clear_cache()` calls. `0` disables cache clearing.</td>
      <td>`256`</td>
    </tr>
  </tbody>
</table>

## Multimodal (VLM)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_VLM_CACHE_SIZE_MB`</td>
      <td>Size (MB) of the VLM feature cache.</td>
      <td>`100`</td>
    </tr>

    <tr>
      <td>`SGLANG_IMAGE_MAX_PIXELS`</td>
      <td>Maximum number of pixels per image before resizing.</td>
      <td>`12845056`</td>
    </tr>

    <tr>
      <td>`SGLANG_RESIZE_RESAMPLE`</td>
      <td>Resampling filter used when resizing images (e.g. `bilinear`, `bicubic`).</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_MM_SKIP_COMPUTE_HASH`</td>
      <td>Skip computing multimodal-item hashes.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_MM_AVOID_RETOKENIZE`</td>
      <td>For pre-tokenized (list\[int]) multimodal prompts, preserve the user's original tokens to avoid retokenization drift.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_VIT_ENABLE_CUDA_GRAPH`</td>
      <td>Capture the vision encoder (ViT) in a CUDA graph.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_CUDA_IPC_TRANSPORT`</td>
      <td>Use CUDA IPC transport for multimodal-item tensors.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_IPC_POOL_HANDLE_CACHE`</td>
      <td>When CUDA IPC multimodal feature transport is selected, reuse mappings to its existing bounded pool. This does not enable CUDA IPC transport or reserve another pool.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_MM_FEATURE_CACHE_MB`</td>
      <td>Size (MB) of the multimodal feature cache.</td>
      <td>`1024`</td>
    </tr>

    <tr>
      <td>`SGLANG_MM_ITEM_MEM_POOL_RECYCLE_INTERVAL_SEC`</td>
      <td>Interval (seconds) for recycling the multimodal-item memory pool.</td>
      <td>`0.05`</td>
    </tr>
  </tbody>
</table>

## Encoder / EPD (Multimodal Disaggregation)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_ENCODER_MM_RECEIVER_MODE`</td>
      <td>Encoder receiver selection used by EPD paths: `http` or `grpc`.</td>
      <td>`http`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_GRPC_TIMEOUT_SECS`</td>
      <td>gRPC timeout (seconds) for encoder communication.</td>
      <td>`60`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_RECV_TIMEOUT`</td>
      <td>Encoder receive timeout (seconds).</td>
      <td>`180.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_SEND_TIMEOUT`</td>
      <td>Encoder send timeout (seconds).</td>
      <td>`180.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_HTTP_TIMEOUT`</td>
      <td>Encoder HTTP timeout (seconds).</td>
      <td>`1800.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_REQ_TIMEOUT`</td>
      <td>Encoder per-request timeout (seconds).</td>
      <td>`180.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_DISPATCH_MIN_ITEMS`</td>
      <td>Minimum items before the encoder dispatches a batch.</td>
      <td>`2`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_MAX_BATCH_SIZE`</td>
      <td>Maximum encoder batch size.</td>
      <td>`8`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_PREPROC_WORKERS`</td>
      <td>Number of encoder preprocessing workers.</td>
      <td>`8`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_IMAGE_PROCESSOR_USE_GPU`</td>
      <td>Run the image processor on the GPU.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_BOOTSTRAP_HEALTH_CHECK_INTERVAL`</td>
      <td>EncoderBootstrapServer health-check interval (seconds). `0` disables it.</td>
      <td>`10.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_BOOTSTRAP_HEALTH_CHECK_TIMEOUT`</td>
      <td>EncoderBootstrapServer health-check timeout (seconds).</td>
      <td>`2.0`</td>
    </tr>

    <tr>
      <td>`SGLANG_EMBEDDING_POOL_SIZE_MB`</td>
      <td>Persistent receiver-side GPU embedding pool size (MB) for Mooncake EPD transport. `0` disables (per-request register/deregister).</td>
      <td>`4096`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENCODER_DP_WORKER_MAX_INFLIGHT`</td>
      <td>Maximum in-flight requests per encoder DP worker.</td>
      <td>`64`</td>
    </tr>

    <tr>
      <td>`SGLANG_BACKUP_PORT_BASE`</td>
      <td>Base port for elastic-EP backup ports.</td>
      <td>`10000`</td>
    </tr>
  </tbody>
</table>

## HTTP & gRPC Server

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_TIMEOUT_KEEP_ALIVE`</td>
      <td>HTTP keep-alive timeout (seconds).</td>
      <td>`5`</td>
    </tr>

    <tr>
      <td>`SGLANG_UVICORN_WORKER_HEALTHCHECK_TIMEOUT`</td>
      <td>Uvicorn multiprocess supervisor per-worker health-check interval (seconds).</td>
      <td>`10`</td>
    </tr>

    <tr>
      <td>`SGLANG_HTTP_SERVER_APP_PATH`</td>
      <td>ASGI import string (`module:attr`) that Uvicorn and Granian worker processes load when `--tokenizer-worker-num` is greater than 1. Point it at an app that extends the SGLang app with extra routes.</td>
      <td>`sglang.srt.entrypoints.http_server:app`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION`</td>
      <td>Have the `/health` endpoint run a generation as part of the check.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_WARMUP_TIMEOUT`</td>
      <td>If a warmup forward batch takes longer than this many seconds, the server crashes to avoid hanging. `-1` disables; increase (e.g. to 1800) to accommodate kernel JIT precompile.</td>
      <td>`-1`</td>
    </tr>

    <tr>
      <td>`SGLANG_GRPC_PORT`</td>
      <td>Port for the native gRPC server.</td>
      <td>Not set</td>
    </tr>

    <tr>
      <td>`SGLANG_GRANIAN_PARENT_PID`</td>
      <td>Parent PID for the Granian HTTP/2 worker supervisor.</td>
      <td>Not set</td>
    </tr>
  </tbody>
</table>

## NUMA & CPU

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_NUMA_BIND_V2`</td>
      <td>Select the NUMA binding implementation. `true` uses a pre-launch `numactl` wrapper; `false` binds in the worker process. This variable does not enable or disable NUMA binding.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_AUTO_NUMA_BIND`</td>
      <td>Automatically detect each NVIDIA GPU's local NUMA node and bind its worker when `--numa-node` is not specified. Without `CAP_SYS_NICE`, only CPU affinity is set. Set to `false` to leave workers unbound; an explicit `--numa-node` takes precedence.</td>
      <td>`true`</td>
    </tr>

    <tr>
      <td>`SGLANG_CRASH_ON_NUMA_BIND_FAILURE`</td>
      <td>Crash if NUMA binding fails instead of warning.</td>
      <td>`false`</td>
    </tr>
  </tbody>
</table>

## Metrics

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_ENABLE_METRICS_DEVICE_TIMER`</td>
      <td>Enable device-timer-based metrics.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`SGLANG_ENABLE_METRICS_DP_ATTENTION`</td>
      <td>Enable data-parallel attention metrics.</td>
      <td>`false`</td>
    </tr>
  </tbody>
</table>

## External Models

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_EXTERNAL_MODEL_PACKAGE`</td>
      <td>Python package providing external model implementations.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_EXTERNAL_MM_MODEL_ARCH`</td>
      <td>External multimodal model architecture name.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`</td>
      <td>Python package providing the external multimodal processor.</td>
      <td>`""`</td>
    </tr>
  </tbody>
</table>

## Plugin System

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Description</th>
      <th>Default Value</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_PLATFORM`</td>
      <td>Platform plugin name to load.</td>
      <td>`""`</td>
    </tr>

    <tr>
      <td>`SGLANG_PLUGINS`</td>
      <td>Comma-separated list of plugins to load.</td>
      <td>`""`</td>
    </tr>
  </tbody>
</table>


# Troubleshooting and Frequently Asked Questions
Source: https://docs.sglang.io/docs/references/faq



## Troubleshooting

This page lists common errors and tips for resolving them.

### CUDA Out of Memory

If you encounter out-of-memory (OOM) errors, you can adjust the following parameters:

* If OOM occurs during prefill, try reducing `--chunked-prefill-size` to `4096` or `2048`. This saves memory but slows down the prefill speed for long prompts.
* If OOM occurs during decoding, try lowering `--max-running-requests`.
* You can also decrease `--mem-fraction-static` to a smaller value, such as 0.8 or 0.7. This decreases the memory usage of the KV cache memory pool and helps prevent OOM errors during both prefill and decoding. However, it limits maximum concurrency and reduces peak throughput.
* Another common case for OOM is requesting input logprobs for a long prompt as it requires significant memory. To address this, set `logprob_start_len` in your sampling parameters to include only the necessary parts. If you do need input logprobs for a long prompt, try reducing `--mem-fraction-static`.

### CUDA Error: Illegal Memory Access Encountered

This error may result from kernel errors or out-of-memory issues:

* If it is a kernel error, resolving it may be challenging. Please file an issue on GitHub.
* If it is an out-of-memory issue, it may sometimes be reported as this error instead of "Out of Memory." Refer to the section above for guidance on avoiding OOM issues.

### The server hangs

* If the server hangs during initialization or running, it can be memory issues (out of memory), network issues (nccl errors), or other bugs in sglang.
  * If it is out of memory, you might see that `avail mem` is very low during the initialization or right after initialization. In this case,
    you can try to decrease `--mem-fraction-static`, decrease `--cuda-graph-max-bs-decode`, or decrease `--chunked-prefill-size`.
* Other bugs, please file an issue on GitHub.

## Frequently Asked Questions

### The results are not deterministic, even with a temperature of 0

You may notice that when you send the same request twice, the results from the engine will be slightly different, even when the temperature is set to 0.

From our initial investigation, this indeterminism arises from two factors: dynamic batching and prefix caching. Roughly speaking, dynamic batching accounts for about 95% of the indeterminism, while prefix caching accounts for the remaining portion. The server runs dynamic batching under the hood. Different batch sizes can cause PyTorch/CuBLAS to dispatch to different CUDA kernels, which can lead to slight numerical differences. This difference accumulates across many layers, resulting in nondeterministic output when the batch size changes. Similarly, when prefix caching is enabled, it can also dispatch to different kernels. Even when the computations are mathematically equivalent, small numerical differences from different kernel implementations lead to the final nondeterministic outputs.

To achieve more deterministic outputs in the current code, you can add `--disable-radix-cache` and send only one request at a time. The results will be mostly deterministic under this setting.

**Update**:
Recently, we also introduced a deterministic mode, you can enable it with `--enable-deterministic-inference`.
Please find more details in this blog post: [https://lmsys.org/blog/2025-09-22-sglang-deterministic/](https://lmsys.org/blog/2025-09-22-sglang-deterministic/)


# Choices Methods in SGLang
Source: https://docs.sglang.io/docs/references/frontend/choices_methods



This doc describes the choices methods supported by SGLang.

The optional `choices_method` arg determines how options supplied to SGLang's `choices` primitive are selected. Only the `RuntimeEndpoint` backend supports the `choices_method` arg. Other backends, such as `OpenAI`, have bespoke selection implementations due to API limitations.

## Methods

### Token Length Normalized

Token length normalized is the default SGLang choices method. It selects the option with the highest average logprob across all of its tokens.

Usage example (alternatively, simply omit the `choices_method` arg):

```python Example theme={null}
@sgl.function
def example(s):
    s += sgl.user("What is the capital of France?")
    s += sgl.assistant(
        sgl.gen(
            "answer",
            choices=["London", "Paris", "Berlin"],
            choices_method=sgl.token_length_normalized,
        )
    )
```

This can perform poorly if an option contains many tokens, where its later tokens are predicted with high confidence based on its earlier tokens. For instance, even strong models will fail the above example if the specified options are `["Paris", "Antidisestablishmentarianism"]`.

### Greedy Token Selection

Greedy token selection simply selects the option with the highest logprob for its initial token. For overlapping options where one option is a subset of a longer option, the logprobs of the shorter option are extended using its average logprob for comparison against the longer option.

Usage example:

```python Example theme={null}
@sgl.function
def example(s):
    s += sgl.user("What is the capital of France?")
    s += sgl.assistant(
        sgl.gen(
            "answer",
            choices=["London", "Paris", "Berlin"],
            choices_method=sgl.greedy_token_selection,
        )
    )
```

This can perform poorly if an option misleads the model down a bad path based on an attractive initial token. For instance, greedy selection will result in an incorrect response for this example:

```python Example theme={null}
@sgl.function
def us_president_example(s):
    s += sgl.user("Name a US president.")
    s += sgl.assistant(
        sgl.gen(
            "answer",
            choices=["Donald Duck", "Millard Fillmore"],
            choices_method=sgl.greedy_token_selection,
        )
    )
```

### Unconditional Likelihood Normalized

Unconditional likelihood normalized selects the option with the highest average token logprob once normalized by the unconditional token logprobs, as described in [this EleutherAI blogpost](https://blog.eleuther.ai/multiple-choice-normalization/). This method incurs an additional LLM call to obtain the unconditional likelihoods.

Usage example:

```python Example theme={null}
@sgl.function
def example(s):
    s += sgl.user("What is the capital of France?")
    s += sgl.assistant(
        sgl.gen(
            "answer",
            choices=["London", "Paris", "Berlin"],
            choices_method=sgl.unconditional_likelihood_normalized,
        )
    )
```


# Frontend Language
Source: https://docs.sglang.io/docs/references/frontend/frontend_index



* [Frontend Tutorial](./frontend_tutorial)
* [Choices Methods](./choices_methods)


# SGLang Frontend Language
Source: https://docs.sglang.io/docs/references/frontend/frontend_tutorial



SGLang frontend language can be used to define simple and easy prompts in a convenient, structured way.

## Launch A Server

Launch the server in your terminal and wait for it to initialize.

```python Example theme={null}
from sglang import assistant_begin, assistant_end
from sglang import assistant, function, gen, system, user
from sglang import image
from sglang import RuntimeEndpoint
from sglang.lang.api import set_default_backend
from sglang.srt.utils import load_image
from sglang.test.doc_patch import launch_server_cmd
from sglang.utils import print_highlight, terminate_process, wait_for_server

server_process, port = launch_server_cmd(
    "python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=server_process)
print(f"Server started on http://localhost:{port}")
```

Set the default backend. Note: Besides the local server, you may use also `OpenAI` or other API endpoints.

```python Example theme={null}
set_default_backend(RuntimeEndpoint(f"http://localhost:{port}"))
```

## Basic Usage

The most simple way of using SGLang frontend language is a simple question answer dialog between a user and an assistant.

```python Example theme={null}
@function
def basic_qa(s, question):
    s += system(f"You are a helpful assistant than can answer questions.")
    s += user(question)
    s += assistant(gen("answer", max_tokens=512))
```

```python Example theme={null}
state = basic_qa("List 3 countries and their capitals.")
print_highlight(state["answer"])
```

## Multi-turn Dialog

SGLang frontend language can also be used to define multi-turn dialogs.

```python Example theme={null}
@function
def multi_turn_qa(s):
    s += system(f"You are a helpful assistant than can answer questions.")
    s += user("Please give me a list of 3 countries and their capitals.")
    s += assistant(gen("first_answer", max_tokens=512))
    s += user("Please give me another list of 3 countries and their capitals.")
    s += assistant(gen("second_answer", max_tokens=512))
    return s


state = multi_turn_qa()
print_highlight(state["first_answer"])
print_highlight(state["second_answer"])
```

## Control flow

You may use any Python code within the function to define more complex control flows.

```python Example theme={null}
@function
def tool_use(s, question):
    s += assistant(
        "To answer this question: "
        + question
        + ". I need to use a "
        + gen("tool", choices=["calculator", "search engine"])
        + ". "
    )

    if s["tool"] == "calculator":
        s += assistant("The math expression is: " + gen("expression"))
    elif s["tool"] == "search engine":
        s += assistant("The key word to search is: " + gen("word"))


state = tool_use("What is 2 * 2?")
print_highlight(state["tool"])
print_highlight(state["expression"])
```

## Parallelism

Use `fork` to launch parallel prompts. Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel.

```python Example theme={null}
@function
def tip_suggestion(s):
    s += assistant(
        "Here are two tips for staying healthy: "
        "1. Balanced Diet. 2. Regular Exercise.\n\n"
    )

    forks = s.fork(2)
    for i, f in enumerate(forks):
        f += assistant(
            f"Now, expand tip {i+1} into a paragraph:\n"
            + gen("detailed_tip", max_tokens=256, stop="\n\n")
        )

    s += assistant("Tip 1:" + forks[0]["detailed_tip"] + "\n")
    s += assistant("Tip 2:" + forks[1]["detailed_tip"] + "\n")
    s += assistant(
        "To summarize the above two tips, I can say:\n" + gen("summary", max_tokens=512)
    )


state = tip_suggestion()
print_highlight(state["summary"])
```

## Constrained Decoding

Use `regex` to specify a regular expression as a decoding constraint. This is only supported for local models.

```python Example theme={null}
@function
def regular_expression_gen(s):
    s += user("What is the IP address of the Google DNS servers?")
    s += assistant(
        gen(
            "answer",
            temperature=0,
            regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)",
        )
    )


state = regular_expression_gen()
print_highlight(state["answer"])
```

Use `regex` to define a `JSON` decoding schema.

```python Example theme={null}
character_regex = (
    r"""\{\n"""
    + r"""    "name": "[\w\d\s]{1,16}",\n"""
    + r"""    "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n"""
    + r"""    "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n"""
    + r"""    "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n"""
    + r"""    "wand": \{\n"""
    + r"""        "wood": "[\w\d\s]{1,16}",\n"""
    + r"""        "core": "[\w\d\s]{1,16}",\n"""
    + r"""        "length": [0-9]{1,2}\.[0-9]{0,2}\n"""
    + r"""    \},\n"""
    + r"""    "alive": "(Alive|Deceased)",\n"""
    + r"""    "patronus": "[\w\d\s]{1,16}",\n"""
    + r"""    "bogart": "[\w\d\s]{1,16}"\n"""
    + r"""\}"""
)


@function
def character_gen(s, name):
    s += user(
        f"{name} is a character in Harry Potter. Please fill in the following information about this character."
    )
    s += assistant(gen("json_output", max_tokens=256, regex=character_regex))


state = character_gen("Harry Potter")
print_highlight(state["json_output"])
```

## Batching

Use `run_batch` to run a batch of prompts.

```python Example theme={null}
@function
def text_qa(s, question):
    s += user(question)
    s += assistant(gen("answer", stop="\n"))


states = text_qa.run_batch(
    [
        {"question": "What is the capital of the United Kingdom?"},
        {"question": "What is the capital of France?"},
        {"question": "What is the capital of Japan?"},
    ],
    progress_bar=True,
)

for i, state in enumerate(states):
    print_highlight(f"Answer {i+1}: {states[i]['answer']}")
```

## Streaming

Use `stream` to stream the output to the user.

```python Example theme={null}
@function
def text_qa(s, question):
    s += user(question)
    s += assistant(gen("answer", stop="\n"))


state = text_qa.run(
    question="What is the capital of France?", temperature=0.1, stream=True
)

for out in state.text_iter():
    print(out, end="", flush=True)
```

## Complex Prompts

You may use `&#123;system|user|assistant&#125;_&#123;begin|end&#125;` to define complex prompts.

```python Example theme={null}
@function
def chat_example(s):
    s += system("You are a helpful assistant.")
    # Same as: s += s.system("You are a helpful assistant.")

    with s.user():
        s += "Question: What is the capital of France?"

    s += assistant_begin()
    s += "Answer: " + gen("answer", max_tokens=100, stop="\n")
    s += assistant_end()


state = chat_example()
print_highlight(state["answer"])
```

```python Example theme={null}
terminate_process(server_process)
```

## Multi-modal Generation

You may use SGLang frontend language to define multi-modal prompts.
See [here](../../supported-models/multimodal_language_models) for supported models.

```python Example theme={null}
server_process, port = launch_server_cmd(
    "python -m sglang.launch_server --model-path Qwen/Qwen2.5-VL-7B-Instruct --host 0.0.0.0 --log-level warning"
)

wait_for_server(f"http://localhost:{port}", process=server_process)
print(f"Server started on http://localhost:{port}")
```

```python Example theme={null}
set_default_backend(RuntimeEndpoint(f"http://localhost:{port}"))
```

Ask a question about an image.

```python Example theme={null}
@function
def image_qa(s, image_file, question):
    s += user(image(image_file) + question)
    s += assistant(gen("answer", max_tokens=256))


image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
image_bytes, _ = load_image(image_url)
state = image_qa(image_bytes, "What is in the image?")
print_highlight(state["answer"])
```

```python Example theme={null}
terminate_process(server_process)
```


# Deploy On Kubernetes
Source: https://docs.sglang.io/docs/references/multi_node_deployment/deploy_on_k8s



This document is for deploying a RoCE network-based SGLang two-node inference service on a Kubernetes (K8S) cluster.

[LeaderWorkerSet (LWS)](https://github.com/kubernetes-sigs/lws) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads. A major use case is for multi-host/multi-node distributed inference.

SGLang can also be deployed with LWS on Kubernetes for distributed model serving.

Please see this guide for more details on deploying SGLang on Kubernetes using LWS.

Here we take the deployment of DeepSeek-R1 as an example.

## Prerequisites

1. At least two Kubernetes nodes, each with two H20 systems and eight GPUs, are required.

2. Make sure your K8S cluster has LWS correctly installed. If it hasn't been set up yet, please follow the [installation instructions](https://lws.sigs.k8s.io/docs/installation/). **Note:** For LWS versions ≤0.5.x, you must use the Downward API to obtain `LWS_WORKER_INDEX`, as native support for this feature was introduced in v0.6.0.

## Basic example

For the basic example documentation, refer to [Deploy Distributed Inference Service with SGLang and LWS on GPUs](https://github.com/kubernetes-sigs/lws/tree/main/docs/examples/sglang).

However, that document only covers the basic NCCL socket mode.

In this section, we’ll make some simple modifications to adapt the setup to the RDMA scenario.

## RDMA RoCE case

* Check your env:

```bash Command theme={null}
[root@node1 ~]# ibstatus
Infiniband device 'mlx5_bond_0' port 1 status:
        default gid:     fe80:0000:0000:0000:0225:9dff:fe64:c79a
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (2X NDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_1' port 1 status:
        default gid:     fe80:0000:0000:0000:0225:9dff:fe6e:c3ec
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (2X NDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_2' port 1 status:
        default gid:     fe80:0000:0000:0000:0225:9dff:fe73:0dd7
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (2X NDR)
        link_layer:      Ethernet

Infiniband device 'mlx5_bond_3' port 1 status:
        default gid:     fe80:0000:0000:0000:0225:9dff:fe36:f7ff
        base lid:        0x0
        sm lid:          0x0
        state:           4: ACTIVE
        phys state:      5: LinkUp
        rate:            200 Gb/sec (2X NDR)
        link_layer:      Ethernet
```

* Prepare the `lws.yaml` file for deploying on k8s.

```yaml Config theme={null}
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: sglang
spec:
  replicas: 1
  leaderWorkerTemplate:
    size: 2
    restartPolicy: RecreateGroupOnPodRestart
    leaderTemplate:
      metadata:
        labels:
          role: leader
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        hostIPC: true
        containers:
          - name: sglang-leader
            image: sglang:latest
            securityContext:
              privileged: true
            env:
              - name: NCCL_IB_GID_INDEX
                value: "3"
            command:
              - python3
              - -m
              - sglang.launch_server
              - --model-path
              - /work/models
              - --mem-fraction-static
              -  "0.93"
              - --torch-compile-max-bs
              - "8"
              - --max-running-requests
              - "20"
              - --tp
              - "16" # Size of Tensor Parallelism
              - --dist-init-addr
              - $(LWS_LEADER_ADDRESS):20000
              - --nnodes
              - $(LWS_GROUP_SIZE)
              - --node-rank
              - $(LWS_WORKER_INDEX)
              - --trust-remote-code
              - --host
              - "0.0.0.0"
              - --port
              - "40000"
            resources:
              limits:
                nvidia.com/gpu: "8"
            ports:
              - containerPort: 40000
            readinessProbe:
              tcpSocket:
                port: 40000
              initialDelaySeconds: 15
              periodSeconds: 10
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
              - name: model
                mountPath: /work/models
              - name: ib
                mountPath: /dev/infiniband
        volumes:
          - name: dshm
            emptyDir:
              medium: Memory
          - name: model
            hostPath:
              path: '< your models dir >' # modify it according your models dir
          - name: ib
            hostPath:
              path: /dev/infiniband
    workerTemplate:
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        hostIPC: true
        containers:
          - name: sglang-worker
            image: sglang:latest
            securityContext:
              privileged: true
            env:
            - name: NCCL_IB_GID_INDEX
              value: "3"
            command:
              - python3
              - -m
              - sglang.launch_server
              - --model-path
              - /work/models
              - --mem-fraction-static
              - "0.93"
              - --torch-compile-max-bs
              - "8"
              - --max-running-requests
              - "20"
              - --tp
              - "16" # Size of Tensor Parallelism
              - --dist-init-addr
              - $(LWS_LEADER_ADDRESS):20000
              - --nnodes
              - $(LWS_GROUP_SIZE)
              - --node-rank
              - $(LWS_WORKER_INDEX)
              - --trust-remote-code
            resources:
              limits:
                nvidia.com/gpu: "8"
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
              - name: model
                mountPath: /work/models
              - name: ib
                mountPath: /dev/infiniband
        volumes:
          - name: dshm
            emptyDir:
              medium: Memory
          - name: ib
            hostPath:
              path: /dev/infiniband
          - name: model
            hostPath:
              path: /data1/models/deepseek_v3_moe
***
apiVersion: v1
kind: Service
metadata:
  name: sglang-leader
spec:
  selector:
    leaderworkerset.sigs.k8s.io/name: sglang
    role: leader
  ports:
    - protocol: TCP
      port: 40000
      targetPort: 40000

```

* Then use  `kubectl apply -f lws.yaml` you will get this output.

```text Output theme={null}
NAME           READY   STATUS    RESTARTS       AGE
sglang-0       0/1     Running   0              9s
sglang-0-1     1/1     Running   0              9s
```

Wait for the sglang leader (`sglang-0`) status to change to 1/1, which indicates it is `Ready`.

You can use the command `kubectl logs -f sglang-0` to view the logs of the leader node.

Once successful, you should see output like this:

```text Output theme={null}
[2025-02-17 05:27:24 TP1] Capture cuda graph end. Time elapsed: 84.89 s
[2025-02-17 05:27:24 TP6] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24 TP0] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24 TP7] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24 TP3] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24 TP2] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24 TP4] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24 TP1] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24 TP5] max_total_num_tokens=712400, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=50, context_len=163840
[2025-02-17 05:27:24] INFO:     Started server process [1]
[2025-02-17 05:27:24] INFO:     Waiting for application startup.
[2025-02-17 05:27:24] INFO:     Application startup complete.
[2025-02-17 05:27:24] INFO:     Uvicorn running on http://0.0.0.0:40000 (Press CTRL+C to quit)
[2025-02-17 05:27:25] INFO:     127.0.0.1:48908 - "GET /get_model_info HTTP/1.1" 200 OK
[2025-02-17 05:27:25 TP0] Prefill batch. #new-seq: 1, #new-token: 7, #cached-token: 0, cache hit rate: 0.00%, token usage: 0.00, #running-req: 0, #queue-req: 0
[2025-02-17 05:27:32] INFO:     127.0.0.1:48924 - "POST /generate HTTP/1.1" 200 OK
[2025-02-17 05:27:32] The server is fired up and ready to roll!
```

If it doesn’t start up successfully, please follow these steps to check for any remaining issues. Thanks!

### Debug

* Set `NCCL_DEBUG=TRACE` to check if it is a NCCL communication problem.

This should resolve most NCCL-related issues.

***Notice: If you find that NCCL\_DEBUG=TRACE is not effective in the container environment, but the process is stuck or you encounter hard-to-diagnose issues, try switching to a different container image. Some images may not handle standard error output properly.***

#### RoCE scenario

* Please make sure that RDMA devices are available in the cluster environment.

* Please make sure that the nodes in the cluster have Mellanox NICs with RoCE. In this example, we use Mellanox ConnectX 5 model NICs, and the proper OFED driver has been installed. If not, please refer to the document [Install OFED Driver](https://docs.nvidia.com/networking/display/mlnxofedv24102180lts/installing+the+driver) to install the driver.

* Check your env:

  ```shell Command theme={null}
  $ lspci -nn | grep Eth | grep Mellanox
  0000:7f:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  0000:7f:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  0000:c7:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  0000:c7:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  0001:08:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  0001:08:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  0001:a2:00.0 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  0001:a2:00.1 Ethernet controller [0200]: Mellanox Technologies MT43244 BlueField-3 integrated ConnectX-7 network controller [15b3:a2dc] (rev 01)
  ```

* Check the OFED driver:

  ```shell Command theme={null}
  ofed_info -s
  OFED-internal-23.07-0.5.0:
  ```

* Show RDMA link status and check IB devices:

  ```shell Command theme={null}
  $ rdma link show
  8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0
  9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2
  10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4
  11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6

  $ ibdev2netdev
  8/1: mlx5_bond_0/1: state ACTIVE physical_state LINK_UP netdev reth0
  9/1: mlx5_bond_1/1: state ACTIVE physical_state LINK_UP netdev reth2
  10/1: mlx5_bond_2/1: state ACTIVE physical_state LINK_UP netdev reth4
  11/1: mlx5_bond_3/1: state ACTIVE physical_state LINK_UP netdev reth6
  ```

* Test RoCE network speed on the host:

  ```shell Command theme={null}
  yum install qperf
  # for server：
  execute qperf
  # for client
  qperf -t 60 -cm1 <server_ip>   rc_rdma_write_bw
  ```

* Check RDMA accessible in your container:

  ```shell Command theme={null}
  # ibv_devices
  # ibv_devinfo
  ```

## Keys to success

* In the YAML configuration above, pay attention to the NCCL environment variable. For older versions of NCCL, you should check the NCCL\_IB\_GID\_INDEX environment setting.
* NCCL\_SOCKET\_IFNAME is also crucial, but in a containerized environment, this typically isn’t an issue.
* In some cases, it’s necessary to configure GLOO\_SOCKET\_IFNAME correctly.
* NCCL\_DEBUG is essential for troubleshooting, but I've found that sometimes it doesn't show error logs within containers. This could be related to the Docker image you're using. You may want to try switching images if needed.
* Avoid using Docker images based on Ubuntu 18.04, as they tend to have compatibility issues.

## Remaining issues

* In Kubernetes, Docker, or Containerd environments, we use hostNetwork to prevent performance degradation.
* We utilize privileged mode, which  isn’t secure. Additionally, in containerized environments, full GPU isolation cannot be achieved.

## TODO

* Integrated with [k8s-rdma-shared-dev-plugin](https://github.com/Mellanox/k8s-rdma-shared-dev-plugin).


# LWS Based PD Deploy
Source: https://docs.sglang.io/docs/references/multi_node_deployment/lws_pd/lws_pd_deploy



## 0. Prerequisites

1. k8s >=1.26
2. lws installed on k8s.

## 1. Image Preparation

`lmsysorg/sglang:deepep`

## 2. Deployment Manifest Files

***Notice: We will package all deployment files into Helm Chart format in the near future. Interested community members can contact us to contribute***

### Prefill

Prefill manifest file [prefill.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p.yaml)

*Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment*

```yaml Config theme={null}
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: deepseekr10528-prefill-main
spec:
  leaderWorkerTemplate:
    leaderTemplate:
      metadata:
        labels:
          role: leader
      spec:
        containers:
        - command:
          - python3
          - -m
          - sglang.launch_server
          - --port
          - "30000"
          - --host
          - "0.0.0.0"
          - --model-path
          - /work/models
          - --disaggregation-ib-device
          # should modify according your rdma env
          - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3
          - --chunked-prefill-size
          - "524288"
          - --max-prefill-tokens
          - "32768"
          - --page-size
          - "64"
          #          - --init-expert-location
          #          - /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json
          - --ep-dispatch-algorithm
          - dynamic
          - --eplb-algorithm
          - deepseek
          #          - --deepep-config
          #          -  /home/aiges/tuned/tuned_8sms.json
          - --enable-dp-lm-head
          - --attn-dp-size
          - "16"
          - --disable-radix-cache
          - --moe-a2a-backend
          - deepep
          - --disaggregation-mode
          - prefill
          - --mem-fraction-static
          - "0.7"
          - --context-length
          - "32768"
          - --tp
          - "16"
          - --dist-init-addr
          - $(LWS_LEADER_ADDRESS):20102
          - --nnodes
          - $(LWS_GROUP_SIZE)
          - --node-rank
          - $(LWS_WORKER_INDEX)
          - --trust-remote-code
          - --ep-num-redundant-experts
          - "32"
          - --moe-dense-tp-size
          - "1"
          - --max-running-requests
          - "1024"
          env:
#          - name: NVSHMEM_HCA_PE_MAPPING
#            value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2"
#          - name: NVSHMEM_HCA_LIST
#            value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1"
          - name: NVSHMEM_IB_GID_INDEX
            value: "3"
          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
            value: "1"
          - name: SGLANG_SET_CPU_AFFINITY
            value: "true"
          - name: SGLANG_ENABLE_JIT_DEEPGEMM
            value: "1"
          - name: NCCL_IB_QPS_PER_CONNECTION
            value: "8"
          - name: NCCL_IB_SPLIT_DATA_ON_QPS
            value: "1"
          - name: NCCL_NET_PLUGIN
            value: none
          - name: NCCL_IB_TC
            value: "136"
          - name: NCCL_MIN_NCHANNELS
            value: "4"
          - name: MC_TE_METRIC
            value: "false"
          - name: NCCL_IB_SL
            value: "5"
          - name: NCCL_IB_HCA
            value: ^=mlx5_0,mlx5_5,mlx5_6
          - name: LWS_WORKER_INDEX
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
          image: lmsysorg/sglang:deepep
          name: sglang-leader
          ports:
          - containerPort: 30000
            protocol: TCP
          readinessProbe:
            periodSeconds: 30
            tcpSocket:
              port: 30000
          resources:
            limits:
              nvidia.com/gpu: "8"
          securityContext:
            capabilities:
              add:
              - IPC_LOCK
            privileged: true
          volumeMounts:
          - mountPath: /dev/shm
            name: dshm
          - mountPath: /work/models
            name: model
          - mountPath: /dev/infiniband
            name: ib
          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
            name: cf
          - mountPath: /root/.cache
            name: sgl-cache
        dnsPolicy: ClusterFirstWithHostNet
        hostIPC: true
        hostNetwork: true
        nodeSelector:
          pd: "yes"
        tolerations:
        - key: pd
          operator: Exists
        - key: node-role
          operator: Exists
        volumes:
        - emptyDir:
            medium: Memory
          name: dshm
        - hostPath:
            # modify according to you deployment env
            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
          name: model
        - hostPath:
            path: /dev/infiniband
          name: ib
        - hostPath:
            # modify according to you deployment env
            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
          name: cf
        - hostPath:
            # modify according to you deployment env
            path: /data1/sgl_cache
            type: DirectoryOrCreate
          name: sgl-cache
    restartPolicy: RecreateGroupOnPodRestart
    size: 2
    workerTemplate:
      metadata: {}
      spec:
        containers:
        - command:
          - python3
          - -m
          - sglang.launch_server
          - --model-path
          - /work/models
          - --disaggregation-ib-device
          - mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3
          - --chunked-prefill-size
          - "524288"
          - --max-prefill-tokens
          - "32768"
          - --page-size
          - "64"
          #- --init-expert-location
          #- /home/aiges/tuned/attachment_ep_statistics/prefill_in1024.json
          - --ep-dispatch-algorithm
          - dynamic
          - --eplb-algorithm
          - deepseek
#          - --deepep-config
#          -  /home/aiges/tuned/tuned_8sms.json
          - --enable-dp-lm-head
          - --attn-dp-size
          - "16"
          - --disable-radix-cache
          - --moe-a2a-backend
          - deepep
          - --disaggregation-mode
          - prefill
          - --mem-fraction-static
          - "0.7"
          - --context-length
          - "32768"
          - --tp
          - "16"
          - --dist-init-addr
          - $(LWS_LEADER_ADDRESS):20102
          - --nnodes
          - $(LWS_GROUP_SIZE)
          - --node-rank
          - $(LWS_WORKER_INDEX)
          - --trust-remote-code
          - --ep-num-redundant-experts
          - "32"
          - --moe-dense-tp-size
          - "1"
          - --max-running-requests
          - "1024"
          env:
          - name: SGLANG_SET_CPU_AFFINITY
            value: "true"
          - name: SGLANG_HACK_DEEPEP_NUM_SMS
            value: "8"
          - name: SGLANG_HACK_DEEPEP_NEW_MODE
            value: "0"
#          - name: NVSHMEM_HCA_PE_MAPPING
#            value: "mlx5_bond_0:1:2,mlx5_bond_1:1:2,mlx5_bond_2:1:2,mlx5_bond_3:1:2"
#          - name: NVSHMEM_HCA_LIST
#            value: "mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1"
          - name: NCCL_IB_HCA
            value: ^=mlx5_0,mlx5_5,mlx5_6
          - name: NVSHMEM_IB_TRAFFIC_CLASS
            value: "16"
          - name: NVSHMEM_IB_GID_INDEX
            value: "3"
          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
            value: "1"
          - name: CUDA_LAUNCH_BLOCKING
            value: "0"
          - name: SGLANG_MOONCAKE_TRANS_THREAD
            value: "8"
          - name: SGLANG_ENABLE_JIT_DEEPGEMM
            value: "1"
          - name: SGLANG_CHUNKED_PREFIX_CACHE_THRESHOLD
            value: "0"
          - name: NCCL_IB_QPS_PER_CONNECTION
            value: "8"
          - name: NCCL_IB_SPLIT_DATA_ON_QPS
            value: "1"
          - name: NCCL_NET_PLUGIN
            value: none
          - name: NCCL_IB_TC
            value: "136"
          - name: NCCL_MIN_NCHANNELS
            value: "4"
          - name: MC_TE_METRIC
            value: "true"
          - name: NCCL_IB_SL
            value: "5"
          - name: LWS_WORKER_INDEX
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
          image: lmsysorg/sglang:deepep
          name: sglang-worker
          ports:
          - containerPort: 30001
            protocol: TCP
          resources:
            limits:
              nvidia.com/gpu: "8"
          securityContext:
            capabilities:
              add:
              - IPC_LOCK
            privileged: true
          volumeMounts:

          - mountPath: /root/.cache
            name: sgl-cache
          - mountPath: /dev/shm
            name: dshm
          - mountPath: /work/models
            name: model
          - mountPath: /dev/infiniband
            name: ib
          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
            name: cf
        dnsPolicy: ClusterFirstWithHostNet
        hostIPC: true
        hostNetwork: true
        nodeSelector:
          pd: "yes"
        tolerations:
        - key: pd
          operator: Exists
        - key: node-role
          operator: Exists
        volumes:
        - emptyDir:
            medium: Memory
          name: dshm
        - hostPath:
            path: /dev/infiniband
          name: ib
        - hostPath:
            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
          name: model
        - hostPath:
            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
          name: cf
        - hostPath:
            path: /data1/sgl_cache
            type: DirectoryOrCreate
          name: sgl-cache

```

### Decode

Decode node deployment manifest file [decode.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d.yaml)

*Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment*

```yaml Config theme={null}
apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: deepseekr10528-decode-main
spec:
  leaderWorkerTemplate:
    leaderTemplate:
      metadata:
        labels:
          role: leader
      spec:
        containers:
        - command:
          - python3
          - -m
          - sglang.launch_server
          - --port
          - "30000"
          - --host
          - "0.0.0.0"
          - --model-path
          - /work/models
          - --chunked-prefill-size
          - "262144"
          - --page-size
          - "64"
          - --enable-dp-lm-head
          - --attn-dp-size
          - "16"
          - --moe-a2a-backend
          - deepep
          - --disaggregation-mode
          - decode
          - --mem-fraction-static
          -  "0.849"
          - --context-length
          - "32768"
          - --disaggregation-ib-device
          - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3"
          - --cuda-graph-max-bs-decode
          - "64"
          - --max-running-requests
          - "2048"
          - --tp-size
          - "16" # Size of Tensor Parallelism
          - --dist-init-addr
          - $(LWS_LEADER_ADDRESS):20102
          - --nnodes
          - $(LWS_GROUP_SIZE)
          - --node-rank
          - $(LWS_WORKER_INDEX)
          - --trust-remote-code
          - --ep-num-redundant-experts
          - "32"
          - --moe-dense-tp-size
          - "1"
          env:
          - name: CUDA_LAUNCH_BLOCKING
            value: "0"
          - name: NVSHMEM_IB_GID_INDEX
            value: "3"
          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
            value: "1"
          - name:  NCCL_IB_QPS_PER_CONNECTION
            value: "8"
          - name: NCCL_IB_SPLIT_DATA_ON_QPS
            value: "1"
          - name: NCCL_NET_PLUGIN
            value: "none"
          - name: NCCL_IB_TC
            value: "136"
          - name: NCCL_MIN_NCHANNELS
            value: "4"
          - name: NCCL_IB_SL
            value: "5"
          - name: MC_TE_METRIC
            value: "true"
          - name: SGLANG_MOONCAKE_TRANS_THREAD
            value: "16"
          - name: SGLANG_ENABLE_JIT_DEEPGEMM
            value: "1"
          - name: NCCL_IB_HCA
            value: ^=mlx5_0,mlx5_5,mlx5_6
          - name: LWS_WORKER_INDEX
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
          image: lmsysorg/sglang:deepep
          name: sglang-leader
          ports:
          - containerPort: 30000
            protocol: TCP
          readinessProbe:
            periodSeconds: 30
            tcpSocket:
              port: 30000
          resources:
            limits:
              nvidia.com/gpu: "8"
          securityContext:
            capabilities:
              add:
              - IPC_LOCK
            privileged: true
          volumeMounts:
          - mountPath: /root/.cache
            name: sgl-cache
          - mountPath: /dev/shm
            name: dshm
          - mountPath: /work/models
            name: model
          - mountPath: /dev/infiniband
            name: ib
          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
            name: cf
        dnsPolicy: ClusterFirstWithHostNet
        hostIPC: true
        hostNetwork: true
        nodeSelector:
          pd: "yes"
        tolerations:
        - key: pd
          operator: Exists
        - key: node-role
          operator: Exists
        volumes:
        - hostPath:
            path: /data1/sgl_cache1
            type: DirectoryOrCreate
          name: sgl-cache
        - emptyDir:
            medium: Memory
          name: dshm
        - hostPath:
            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
          name: model
        - hostPath:
            path: /dev/infiniband
          name: ib
        - hostPath:
            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
          name: cf
    restartPolicy: RecreateGroupOnPodRestart
    size:  2
    workerTemplate:
      metadata: {}
      spec:
        containers:
        - command:
          - python3
          - -m
          - sglang.launch_server
          - --model-path
          - /work/models
          - --chunked-prefill-size
          - "262144"
          - --page-size
          - "64"
          - --enable-dp-lm-head
            #- --enable-two-batch-overlap
          - --attn-dp-size
          - "16"
          - --moe-a2a-backend
          - deepep
          - --disaggregation-mode
          - decode
          - --mem-fraction-static
          -  "0.849"
          - --context-length
          - "32768"
          - --disaggregation-ib-device
          # should modify according your rdma env
          - "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3"
          - --cuda-graph-max-bs-decode
          - "64"
          - --max-running-requests
          - "2048"
          - --tp-size
          - "16" # Size of Tensor Parallelism
          - --dist-init-addr
          - $(LWS_LEADER_ADDRESS):20102
          - --nnodes
          - $(LWS_GROUP_SIZE)
          - --node-rank
          - $(LWS_WORKER_INDEX)
          - --trust-remote-code
          - --ep-num-redundant-experts
          - "32"
          - --moe-dense-tp-size
          - "1"
          env:
          - name: SGLANG_HACK_DEEPEP_NUM_SMS
            value: "24"
          - name: SGLANG_HACK_DEEPEP_NEW_MODE
            value: "0"
          - name: NVSHMEM_IB_TRAFFIC_CLASS
            value: "16"
          - name: NVSHMEM_IB_GID_INDEX
            value: "3"
          - name: NVSHMEM_ENABLE_NIC_PE_MAPPING
            value: "1"
          - name:  NCCL_IB_QPS_PER_CONNECTION
            value: "8"
          - name: NCCL_IB_SPLIT_DATA_ON_QPS
            value: "1"
          - name: NCCL_NET_PLUGIN
            value: "none"
          - name: NCCL_IB_TC
            value: "136"
          - name: NCCL_MIN_NCHANNELS
            value: "4"
          - name: MC_TE_METRIC
            value: "true"
          - name: NCCL_IB_SL
            value: "5"
          - name: SGLANG_MOONCAKE_TRANS_THREAD
            value: "16"
          - name: SGLANG_ENABLE_JIT_DEEPGEMM
            value: "1"
          - name: NCCL_IB_HCA
            value: ^=mlx5_0,mlx5_5,mlx5_6
          - name: LWS_WORKER_INDEX
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
          image: lmsysorg/sglang:deepep
          name: sglang-worker
          ports:
          - containerPort: 30001
          resources:
            limits:
              nvidia.com/gpu: "8"
          securityContext:
            capabilities:
              add:
              - IPC_LOCK
            privileged: true
          volumeMounts:
          - mountPath: /root/.cache
            name: sgl-cache
          - mountPath: /dev/shm
            name: dshm
          - mountPath: /work/models
            name: model
          - mountPath: /dev/infiniband
            name: ib
          - mountPath: /sgl-workspace/sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/configs
            name: cf
        dnsPolicy: ClusterFirstWithHostNet
        hostIPC: true
        hostNetwork: true
        nodeSelector:
          pd: "yes"
        tolerations:
        - key: pd
          operator: Exists
        - key: node-role
          operator: Exists
        volumes:
        - hostPath:
            path: /data1/sgl_cache1
            type: DirectoryOrCreate
          name: sgl-cache
        - emptyDir:
            medium: Memory
          name: dshm
        - hostPath:
            path: /dev/infiniband
          name: ib
        - hostPath:
            # modify according to you deployment env
            path: /data1/maas_hosted_models/models/DeepSeek-R1-0528/deepseek_r1_0528
          name: model
        - hostPath:
            # modify according to you deployment env
            path: /data1/maas_hosted_models/models/fused_moe_triton/configs
          name: cf
  networkConfig:
    subdomainPolicy: Shared
  replicas: 1
  rolloutStrategy:
    rollingUpdateConfiguration:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  startupPolicy: LeaderCreated
```

Execute separately:

```bash Command theme={null}
kubectl apply -f p.yaml
kubectl apply -f d.yaml
```

At this point, we have completed the deployment of the 1P1D SGLang engine part.

To allow our users to directly experience the model API, we still need a load balancer to handle sequential calls between prefill and decode. Different companies implement LBs differently, and the community will also officially release a new LB component written in Rust in the near future.

Currently, we use a static K8S service + minilb approach to implement model API calls.

### Creating Service for Prefill and Decode

#### Create prefill k8s service

[p-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/p-svc.yaml)

```yaml Config theme={null}
apiVersion: v1
kind: Service
metadata:
  name: deepseekr10528-prefill-main
spec:
  selector:
    leaderworkerset.sigs.k8s.io/name: deepseekr10528-prefill-main
    role: leader
  ports:
    - protocol: TCP
      port: 30000
      targetPort: 30000
```

Execute `kubectl apply -f p-svc.yaml`

#### Create decode k8s service

[d-svc.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/d-svc.yaml)

```yaml Config theme={null}
apiVersion: v1
kind: Service
metadata:
  name: deepseekr10528-decode-main
spec:
  selector:
    leaderworkerset.sigs.k8s.io/name: deepseekr10528-decode-main
    role: leader
  ports:
    - protocol: TCP
      port: 30000
      targetPort: 30000
```

Execute `kubectl apply -f d-svc.yaml`

#### Deploy minilb and lb service

[lb.yaml](https://github.com/sgl-project/sglang/blob/main/docs/references/multi_node_deployment/lws_pd/lws-examples/lb.yaml)

```yaml Config theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: deepseekr10528-lb-main
  labels:
    app: deepseekr10528-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: deepseekr10528-lb
  template:
    metadata:
      labels:
        app: deepseekr10528-lb
    spec:
      nodeSelector:
          pd: "yes"
      tolerations:
        - key: pd
          operator: Exists
        - key: node-role
          operator: Exists
      containers:
        - name: sgl-minilb
          image: lmsysorg/sglang:deepep
          command:
          - python
          - -m
          - sglang_router.launch_router
          - --pd-disaggregation
          - --prefill
          - http://deepseekr10528-prefill-main:30000
          - --decode
          - http://deepseekr10528-decode-main:30000
          - --host
          - 0.0.0.0
          - --port
          -  "8000"
          ports:
            - containerPort: 8000
***
apiVersion: v1
kind: Service
metadata:
  name: deepseekr10528-lb-service
spec:
  type: NodePort
  selector:
    app: deepseekr10528-lb
  ports:
    - protocol: TCP
      port: 8000         # Service Port（In-Cluster）
      targetPort: 8000   # Exposed Container
      nodePort: 30800
```

Execute `kubectl apply -f lb.yaml`

After waiting for all model deployments to succeed, you will get the following output:

```bash Command theme={null}
[root@ecs-001]# kubectl get po
deepseekr10528-decode-main-0             1/1     Running   0          74m
deepseekr10528-decode-main-0-1           1/1     Running   0          74m
deepseekr10528-lb-main-9c5dbfc57-6lcbd   1/1     Running   0          22m
deepseekr10528-prefill-main-0            1/1     Running   0          74m
deepseekr10528-prefill-main-0-1          1/1     Running   0          74m
[root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl  get svc |grep dee
deepseekr10528-decode-main    ClusterIP   None             <none>        <none>           97m
deepseekr10528-lb-service     NodePort    172.16.242.169   <none>        8000:30800/TCP   22m
deepseekr10528-prefill-main   ClusterIP   None             <none>        <none>           97m
```

At this point, select a nodePort:30800 to access:

```bash Command theme={null}
[root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \
>     -H "Content-Type: application/json" \
>     -H "Authorization: Bearer None" \
>     -d '{
>        "rid":"ccccdd",
>         "model": "r1",
>         "messages": [
>             {"role": "system", "content": "0: You are a helpful AI assistant"},
>             {"role": "user", "content": "你是谁？."}
>         ],
>         "max_tokens":221
>     }'
{"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"<think>\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n</think>\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-R1**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}}

```

## FAQ

1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments.

2. Some preset, optimized configurations for EPLB are not used here. You can adjust them according to [6017](https://github.com/sgl-project/sglang/issues/6017) as needed.


# Multi-Node Deployment
Source: https://docs.sglang.io/docs/references/multi_node_deployment/multi_node



## Llama 3.1 405B

**Run 405B (fp16) on Two Nodes**

```bash Command theme={null}
# replace 172.16.4.52:20000 with your own node ip address and port of the first node

python3 -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \
  --tp 16 \
  --dist-init-addr 172.16.4.52:20000 \
  --nnodes 2 \
  --node-rank 0

python3 -m sglang.launch_server \
  --model-path meta-llama/Meta-Llama-3.1-405B-Instruct \
  --tp 16 \
  --dist-init-addr 172.16.4.52:20000 \
  --nnodes 2 \
  --node-rank 1
```

Note that LLama 405B (fp8) can also be launched on a single node.

```bash Command theme={null}
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8
```

## DeepSeek V3/R1

Please refer to [DeepSeek documents for reference](/cookbook/autoregressive/DeepSeek/DeepSeek-V3#4-2-5-multi-node-deployment).

## Multi-Node Inference on SLURM

This example showcases how to serve SGLang server across multiple nodes by SLURM. Submit the following job to the SLURM cluster.

```bash Command theme={null}
#!/bin/bash -l

#SBATCH -o SLURM_Logs/%x_%j_master.out
#SBATCH -e SLURM_Logs/%x_%j_master.err
#SBATCH -D ./
#SBATCH -J Llama-405B-Online-Inference-TP16-SGL

#SBATCH --nodes=2
#SBATCH --ntasks=2
#SBATCH --ntasks-per-node=1  # Ensure 1 task per node
#SBATCH --cpus-per-task=18
#SBATCH --mem=224GB
#SBATCH --partition="lmsys.org"
#SBATCH --gres=gpu:8
#SBATCH --time=12:00:00

echo "[INFO] Activating environment on node $SLURM_PROCID"
if ! source ENV_FOLDER/bin/activate; then
    echo "[ERROR] Failed to activate environment" >&2
    exit 1
fi

# Define parameters
model=MODEL_PATH
tp_size=16

echo "[INFO] Running inference"
echo "[INFO] Model: $model"
echo "[INFO] TP Size: $tp_size"

# Set NCCL initialization address using the hostname of the head node
HEAD_NODE=$(scontrol show hostname "$SLURM_NODELIST" | head -n 1)
NCCL_INIT_ADDR="${HEAD_NODE}:8000"
echo "[INFO] NCCL_INIT_ADDR: $NCCL_INIT_ADDR"

# Launch the model server on each node using SLURM
srun --ntasks=2 --nodes=2 --output="SLURM_Logs/%x_%j_node$SLURM_NODEID.out" \
    --error="SLURM_Logs/%x_%j_node$SLURM_NODEID.err" \
    python3 -m sglang.launch_server \
    --model-path "$model" \
    --grammar-backend "xgrammar" \
    --tp "$tp_size" \
    --dist-init-addr "$NCCL_INIT_ADDR" \
    --nnodes 2 \
    --node-rank "$SLURM_NODEID" &

# Wait for the NCCL server to be ready on port 30000
while ! nc -z "$HEAD_NODE" 30000; do
    sleep 1
    echo "[INFO] Waiting for $HEAD_NODE:30000 to accept connections"
done

echo "[INFO] $HEAD_NODE:30000 is ready to accept connections"

# Keep the script running until the SLURM job times out
wait
```

Then, you can test the server by sending requests following other [documents](../../basic_usage/openai_api_completions).

Thanks for [aflah02](https://github.com/aflah02) for providing the example, based on his [blog post](https://aflah02.substack.com/p/multi-node-llm-inference-with-sglang).


# Multi-Node Deployment
Source: https://docs.sglang.io/docs/references/multi_node_deployment/multi_node_index



* [Multi Node](./multi_node)
* [Deploy On K8S](./deploy_on_k8s)
* [Lws Pd Deploy](./lws_pd/lws_pd_deploy)
* [Deepseekv32 Pd](./rbg_pd/deepseekv32_pd)
* [Deploying DeepSeek with PD Disaggregation on 96 H100 GPUs](https://lmsys.org/blog/2025-05-05-large-scale-ep/)
* [Deploying Kimi K2 with PD Disaggregation on 128 H200 GPUs](https://lmsys.org/blog/2025-07-20-k2-large-scale-ep/)


# DeepSeekV32-Exp RBG Based PD Deploy
Source: https://docs.sglang.io/docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd



## 0. Prerequisites

1. k8s >=1.26
2. lws installed on k8s.
3. rbg installed on k8s.

For RBG installation, please refer to: [https://github.com/sgl-project/rbg](https://github.com/sgl-project/rbg)

## 1. Image Preparation

`lmsysorg/sglang:latest`

### 2. All In One manifest file

*Note: The NodeSelector section, model location section, and taint toleration section can be adjusted according to your actual deployment environment*

rbg-dsv32.yml

```yaml Config theme={null}
apiVersion: workloads.x-k8s.io/v1alpha1
kind: RoleBasedGroup
metadata:
  name: deepseek-rbg-32exp
  namespace: default
spec:
  roles:
    - name: prefill
      replicas: 1
      workload:
        apiVersion: leaderworkerset.x-k8s.io/v1
        kind: LeaderWorkerSet
      restartPolicy: None
      leaderWorkerSet:
        size: 1
        patchLeaderTemplate:
          metadata:
            labels:
              role: leader
              pd_role: prefill
          spec:
            containers:
            - command:
              - python3
              - -m
              - sglang.launch_server
              - --model-path
              - /work/models
              - --port
              - "30000"
              - --trust-remote
              - --host
              -  0.0.0.0
              - --disaggregation-ib-device
              -  mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
              - --disable-radix-cache
              - --chunked-prefill-size
              - "131072"
              - --page-size
              - "64"
    #          - --enable-eplb
              - --ep-dispatch-algorithm
              - dynamic
              - --eplb-algorithm
              - deepseek
              - --enable-dp-lm-head
              - --attn-dp-size
              - "8"
              - --moe-a2a-backend
              - deepep
              - --deepep-mode
              - normal
              - --disaggregation-mode
              - prefill
              - --mem-fraction-static
              - "0.8"
              - --max-prefill-tokens
              - "32768"
              - --context-length
              - "32768"
              - --tp
              - "8"
              - --dist-init-addr
              - $(LWS_LEADER_ADDRESS):20102
              - --nnodes
              - $(LWS_GROUP_SIZE)
              - --node-rank
              - $(LWS_WORKER_INDEX)
              - --trust-remote-code
              - --ep-num-redundant-experts
              - "32"
              - --moe-dense-tp-size
              - "1"
              - --max-running-requests
              - "1024"
              env:
              - name: LWS_WORKER_INDEX
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
              livenessProbe:
                failureThreshold: 3000
                httpGet:
                  path: /health
                  port: 30000
                initialDelaySeconds: 300
                periodSeconds: 60
                successThreshold: 1
                timeoutSeconds: 10
              readinessProbe:
                failureThreshold: 20
                httpGet:
                  path: /health
                  port: 30000
                periodSeconds: 30
                successThreshold: 1
                timeoutSeconds: 10
              name: sglang
              ports:
              - containerPort: 30000
                name: sglang-http
                protocol: TCP

        patchWorkerTemplate: {}
      template:
        metadata:
          labels:
            inference-framework: sglang
            inference-stack.io/monitoring: "enabled"
        spec:
            containers:
            - name: sglang
              image: lmsysorg/sglang:latest
              env:
                - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK
                  value: "1"
                - name: CUDA_LAUNCH_BLOCKING
                  value: "0"
                - name:  SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT
                  value: "1000000000"
                - name: NVSHMEM_IB_TRAFFIC_CLASS
                  value: "16"
                - name: NVSHMEM_DISABLE_P2P
                  value: "0"
                - name: ENABLE_METRICS
                  value: "true"
                - name: NVSHMEM_IB_GID_INDEX
                  value: "3"
                - name: NVSHMEM_IB_SL
                  value: "5"
                - name: SGLANG_SET_CPU_AFFINITY
                  value: "true"
                - name: SGL_ENABLE_JIT_DEEPGEMM
                  value: "1"
                - name:  NCCL_IB_QPS_PER_CONNECTION
                  value: "8"
                - name: NCCL_IB_SPLIT_DATA_ON_QPS
                  value: "1"
                - name: NCCL_NET_PLUGIN
                  value: "none"
                - name: NCCL_IB_TC
                  value: "136"
                - name: NCCL_IB_SL
                  value: "5"
                - name: NCCL_IB_TIMEOUT
                  value: "22"
                - name: NCCL_IB_GID_INDEX
                  value: "3"
                - name: NCCL_MIN_NCHANNELS
                  value: "4"
                - name: NCCL_SOCKET_IFNAME
                  value: bond0
                - name: GLOO_SOCKET_IFNAME
                  value: bond0
                - name: NCCL_IB_HCA
                  value: ^=mlx5_0,mlx5_5,mlx5_6
                - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
                  value: "bond0"
                - name: MC_TE_METRIC
                  value: "false"
              resources:
                limits:
                  nvidia.com/gpu: "8"
              securityContext:
                capabilities:
                  add:
                  - IPC_LOCK
                privileged: true
              volumeMounts:
                - mountPath: /root/.cache
                  name: sgl-cache
                - mountPath: /dev/shm
                  name: dshm
                - mountPath: /work/models
                  name: model
                - mountPath: /dev/infiniband
                  name: ib
                - mountPath: /sgl-workspace/sglang
                  name: src

            dnsPolicy: ClusterFirstWithHostNet
            hostIPC: true
            hostNetwork: true
            nodeSelector:
              pd: "yes"
            tolerations:
              - key: pd
                operator: Exists
            volumes:
            - hostPath:
                path: /var/run/sys-topology
              name: topo
            - hostPath:
                path: /data1/sgl_cache4
                type: DirectoryOrCreate
              name: sgl-cache
            - emptyDir:
                medium: Memory
              name: dshm
            - hostPath:
                path: /data/DeepSeek-V3.2-Exp
              name: model
            - hostPath:
                path: /dev/infiniband
              name: ib
            - hostPath:
                path: /data/src/sglang
                type: DirectoryOrCreate
              name: src

    - name: decode
      replicas: 1
      workload:
        apiVersion: leaderworkerset.x-k8s.io/v1
        kind: LeaderWorkerSet
      leaderWorkerSet:
        size: 1
        patchLeaderTemplate:
          metadata:
            labels:
              role: leader
              pd_role: decode
          spec:
            containers:
            - command:
                  - python3
                  - -m
                  - sglang.launch_server
                  - --model-path
                  - /work/models
                  - --port
                  - "30000"
                  - --trust-remote
                  - --host
                  -  0.0.0.0
                  - --disaggregation-ib-device
                  -  mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
                  - --chunked-prefill-size
                  - "131072"
                  - --eplb-rebalance-layers-per-chunk
                  - "29"
                  - --page-size
                  - "64"
                  - --enable-dp-lm-head
                  - --attn-dp-size
                  - "8"
                  - --moe-a2a-backend
                  - deepep
                  - --deepep-mode
                  - low_latency
                  - --disaggregation-mode
                  - decode
                  - --mem-fraction-static
                  -  "0.8"
                  - --context-length
                  - "32768"
                  - --max-running-requests
                  - "2048"
                  - --tp-size
                  - "8" # Size of Tensor Parallelism
                  - --cuda-graph-max-bs-decode
                  - "16"
                  - --dist-init-addr
                  - $(LWS_LEADER_ADDRESS):20102
                  - --nnodes
                  - $(LWS_GROUP_SIZE)
                  - --node-rank
                  - $(LWS_WORKER_INDEX)
                  - --trust-remote-code
                  - --ep-num-redundant-experts
                  - "32"
                  - --moe-dense-tp-size
                  - "1"
              env:
              - name: LWS_WORKER_INDEX
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
              livenessProbe:
                failureThreshold: 30000
                httpGet:
                  path: /health
                  port: 30000
                initialDelaySeconds: 300
                periodSeconds: 60
                successThreshold: 1
                timeoutSeconds: 10
              name: sglang
              readinessProbe:
                failureThreshold: 20
                httpGet:
                  path: /health
                  port: 30000
                periodSeconds: 30
                successThreshold: 1
                timeoutSeconds: 10
        patchWorkerTemplate:
          spec:
            containers:
            - command:
                - python3
                - -m
                - sglang.launch_server
                - --model-path
                - /work/models
                - --crash-dump-folder
                -  /log
                - --chunked-prefill-size
                - "262144"
                - --eplb-rebalance-layers-per-chunk
                - "29"
                - --page-size
                - "64"
                - --enable-dp-lm-head
                - --attn-dp-size
                - "32"
                - --moe-a2a-backend
                - "deepep"
                - --deepep-mode
                - low_latency
                - --disaggregation-mode
                - decode
                - --mem-fraction-static
                -  "0.849"
                - --context-length
                - "32768"
                - --disaggregation-ib-device
                -  mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
                - --max-running-requests
                - "4096"
                - --cuda-graph-max-bs-decode
                - "16"
                - --tp-size
                - "8" # Size of Tensor Parallelism
                - --dist-init-addr
                - $(LWS_LEADER_ADDRESS):20102
                - --nnodes
                - $(LWS_GROUP_SIZE)
                - --node-rank
                - $(LWS_WORKER_INDEX)
                - --trust-remote-code
                - --ep-num-redundant-experts
                - "32"
                - --moe-dense-tp-size
                - "1"
              env:
              - name: LWS_WORKER_INDEX
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.labels['leaderworkerset.sigs.k8s.io/worker-index']
              name: sglang
      template:
        metadata:
          labels:
            inference-framework: sglang-unuse
            inference-stack.io/monitoring: "enabled"
        spec:
            containers:
            - image: lmsysorg/sglang:latest
              name: sglang
              resources:
                limits:
                  nvidia.com/gpu: "8"
              securityContext:
                capabilities:
                  add:
                  - IPC_LOCK
                privileged: true
              volumeMounts:
                - mountPath: /root/.cache
                  name: sgl-cache
                - mountPath: /dev/shm
                  name: dshm
                - mountPath: /work/models
                  name: model
                - mountPath: /dev/infiniband
                  name: ib
                - mountPath: /sgl-workspace/sglang
                  name: src
              env:
                - name: SGLANG_SKIP_SGL_KERNEL_VERSION_CHECK
                  value: "1"
                - name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
                  value: "100000000"
                - name: NVSHMEM_DISABLE_P2P
                  value: "0"
                - name: NVSHMEM_IB_TRAFFIC_CLASS
                  value: "16"
                - name: NVSHMEM_IB_SL
                  value: "5"
                - name: ENABLE_METRICS
                  value: "true"
                - name: CUDA_LAUNCH_BLOCKING
                  value: "0"
                - name: NVSHMEM_IB_GID_INDEX
                  value: "3"
                - name:  NCCL_IB_QPS_PER_CONNECTION
                  value: "8"
                - name: NCCL_IB_SPLIT_DATA_ON_QPS
                  value: "1"
                - name: NCCL_NET_PLUGIN
                  value: "none"
                - name: NCCL_IB_TC
                  value: "136"
                - name: NCCL_IB_SL
                  value: "5"
                - name: NCCL_IB_TIMEOUT
                  value: "22"
                - name: NCCL_IB_GID_INDEX
                  value: "3"
                - name: NCCL_MIN_NCHANNELS
                  value: "4"
                - name: NCCL_SOCKET_IFNAME
                  value: bond0
                - name: GLOO_SOCKET_IFNAME
                  value: bond0
                - name: NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME
                  value: "bond0"
                - name: NCCL_IB_HCA
                  value: ^=mlx5_0,mlx5_5,mlx5_6
                - name: MC_TE_METRIC
                  value: "false"
                - name: SGL_ENABLE_JIT_DEEPGEMM
                  value: "1"
            dnsPolicy: ClusterFirstWithHostNet
            hostIPC: true
            hostNetwork: true
            nodeSelector:
              pd: "yes"
            tolerations:
            - key: pd
              operator: Exists
            volumes:
            - hostPath:
                path: /var/run/sys-topology
              name: topo
            - hostPath:
                path: /data1/sgl_cache4
                type: DirectoryOrCreate
              name: sgl-cache
            - hostPath:
                path: /data/src/sglang
                type: DirectoryOrCreate
              name: src
            - emptyDir:
                medium: Memory
              name: dshm
            - hostPath:
                path: /data/DeepSeek-V3.2-Exp
              name: model
            - hostPath:
                path: /dev/infiniband
              name: ib
    - name: router
      replicas: 1
      dependencies: [ "decode", "prefill" ]
      template:
        spec:
          containers:
            - name: scheduler
              image: lmsysorg/sglang:latest
              command:
              - sh
              - -c
              - >
                python3 -m sglang_router.launch_router
                --host 0.0.0.0
                --port 8080
                --pd-disaggregation
                --policy random
                --service-discovery
                --service-discovery-namespace ${NAMESPACE}
                --service-discovery-port 30000
                --prefill-selector pd_role=prefill
                --decode-selector pd_role=decode
                --max-payload-size 2147483648
                --worker-startup-timeout-secs 1200
              env:
              - name: NAMESPACE
                valueFrom:
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
***
apiVersion: v1
kind: Service
metadata:
  labels:
    app: deepseek-rbg-32exp
  name: deepseek-rbg-32exp
  namespace: default
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 30080

  selector:
    rolebasedgroup.workloads.x-k8s.io/name: deepseek-rbg-32exp
    rolebasedgroup.workloads.x-k8s.io/role: router
  type: NodePort

```

```bash Command theme={null}
[root@ecs-001]# kubectl get po -n default
deepseek-rbg-32exp-decode-main-0             1/1     Running   0          74m
deepseek-rbg-32exp-decode-0-1                1/1     Running   0          74m
deepseek-rbg-32exp-router-9c5dbfc57          1/1     Running   0          22m
deepseek-rbg-32exp-prefill-0                 1/1     Running   0          74m

[root@ecs-cbm-x1-pd-cpu-001 main_doc]# kubectl  get svc |grep dee
deepseek-rbg-32exp-decode             ClusterIP   None             <none>        <none>           97m
deepseek-rbg-32exp-router-service     NodePort    172.16.242.169   <none>        8000:30800/TCP   22m
deepseek-rbg-32exp-prefill            ClusterIP   None             <none>        <none>           97m
```

At this point, select a nodePort:30800 to access:

```bash Command theme={null}
[root@ecs-001]# curl -X POST "http://{nodePort}:30800/v1/chat/completions" \
>     -H "Content-Type: application/json" \
>     -H "Authorization: Bearer None" \
>     -d '{
>        "rid":"ccccdd",
>         "model": "dsv32",
>         "messages": [
>             {"role": "system", "content": "0: You are a helpful AI assistant"},
>             {"role": "user", "content": "你是谁？."}
>         ],
>         "max_tokens":221
>     }'
{"id":"ccccdd","object":"chat.completion","created":1750252498,"model":"qwen2","choices":[{"index":0,"message":{"role":"assistant","content":"&lt;think&gt;\n嗯，用户问了一个很基础的自我介绍问题"你是谁？"。这可能是第一次互动时的常规开场白，也可能是想确认我的身份和功能范围。\n\n用户没有提供任何背景信息，语气简洁中性。这种场景下新用户的可能性较高，需要给出清晰友好的自我介绍，同时突出实用价值来降低陌生感。\n\n考虑到中文用户，应该用简体中文回复。重点要说明三点：身份归属（深度求索）、功能定位（AI助手）、服务范围（学习/工作/生活）。结尾用开放性问题引导对话很关键——既能了解需求，又能避免让用户面对空白输入框时不知所措。\n\n用波浪线结尾可以软化语气，那个笑脸表情😊刚好能中和AI的机械感。不过要控制表情符号数量，避免显得轻浮。\n&lt;/think&gt;\n你好呀！我是你的AI助手，由深度求索公司（DeepSeek）开发的语言模型，名字叫 **DeepSeek-V32**。你可以把我当成一个知识丰富、随叫随到的小帮手～😊\n\n我的任务就是陪你聊天、解答问题、","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"length","matched_stop":null}],"usage":{"prompt_tokens":14,"total_tokens":235,"completion_tokens":221,"prompt_tokens_details":null}}

```

## FAQ

1. The current deployment startup parameters may not be fully compatible with all RDMA scenarios. Different RDMA NCCL-related environment configurations may be needed in different network environments.

2. Please ensure that the sglang code in the image has incorporated the changes from [PR #10912](https://github.com/sgl-project/sglang/pull/10912).


# Nightly precision regression
Source: https://docs.sglang.io/docs/references/nightly_precision_regression



# Nightly Precision Regression Testing

## Overview

The nightly precision regression framework detects silent numerical regressions in the SGLang serving engine by comparing **per-layer hidden states** between consecutive runs. It runs as a nightly CI job on 8×H200 GPUs and can also be invoked locally for development and debugging.

The framework operates on a **rolling-baseline** model:

1. **Baseline creation or comparison:** Launch the server, send a fixed prompt, dump per-layer hidden states to disk. If a previous baseline exists, compare the new tensors against it using the SGLang tensor comparator. If the comparison passes, the new tensors become the updated baseline.
2. On the first run (or when the capture shape changes), the dumped tensors are saved as a new baseline with no comparison.

Baselines are stored locally on disk and synced to a **HuggingFace dataset** so they survive across CI runners and can be shared across machines. The HF dataset store is **required** — the test errors if `SGLANG_PRECISION_HF_REPO` is unset.

***

## How It Works

### Step-by-step flow

```
┌──────────────────────────────────────────────────────────────┐
│  1. Resolve model config (layer count, capture layers)        │
│     ↓                                                         │
│  2. Compute capture_signature (schema, layers, TP, filter)    │
│     ↓                                                         │
│  3. Fetch baseline from HF dataset (signature-matched)        │
│     ↓                                                         │
│  4. Launch SGLang server with DUMPER enabled                   │
│     ↓                                                         │
│  5. POST /dumper/configure  (set layer filter + cleanup)      │
│     ↓                                                         │
│  6. POST /v1/chat/completions  (fixed prompt, 2 tokens,       │
│     ignore_eos=true to force decode path)                      │
│     ↓                                                         │
│  7. Kill server; assert decode tensors were captured           │
│     ↓                                                         │
│  8. Baseline exists (with matching signature)?                 │
│     ├── YES → Run comparator → pass/fail                      │
│     │         ├── PASS  → update baseline, push to HF         │
│     │         └── FAIL  → push diagnostics to HF              │
│     └── NO  → copy today's tensors as initial baseline        │
│                → push to HF as "baseline_established"          │
│     ↓                                                         │
│  9. Report summary (stdout + GitHub Step Summary)             │
└──────────────────────────────────────────────────────────────┘
```

### Key components

| Component | File | Purpose |
| - | - | - |
| Test entry point | `test/registered/debug_utils/test_nightly_precision_regression.py` | Orchestrates server launch, dump, compare, and reporting |
| HF baseline store | `python/sglang/test/precision_baseline_store.py` | Push / fetch / prune baselines on a HuggingFace dataset |
| Tensor comparator | `python/sglang/srt/debug_utils/comparator/` | Compares two directories of `.pt` tensors, emits JSONL report |
| Dumper infrastructure | `python/sglang/srt/debug_utils/dumper.py` | Captures per-layer hidden states at runtime |
| CI workflow | `.github/workflows/nightly-test-nvidia.yml` | Schedules the nightly job on 8×H200 |

***

## What Gets Dumped and Compared

### Strided layer capture

Not every layer is dumped — the framework uses a **strided capture** to reduce I/O and storage overhead. By default, it captures:

* Layer 0 (always)
* The last layer (always)
* Every 8th layer in between (configurable via `LAYER_CAPTURE_STRIDE`)

The layer count is resolved automatically from the model's HuggingFace `config.json` (`num_hidden_layers` or `num_layers`). If resolution fails, all layers are captured as a safe fallback.

The dumper filter is built dynamically as a regex matching only the selected layer indices, e.g.:

```
match(r'^non_intrusive__model\.layers\.(0|7|15|23)\.(inputs\.1|self_attn\.inputs\.hidden_states)$', name)
```

### Decode-path verification

The test generates **2 tokens** with `ignore_eos=True` to ensure the model's decode path is exercised. After the dump, `_assert_decode_captured()` verifies that tensors from the decode step were actually captured (not just prefill). If only prefill tensors are found, the test fails immediately — this catches misconfigurations where `--max-total-tokens` is too low for the decode loop to run.

### Comparator

The comparator computes **relative differences** (`rel_diff`) for each tensor and checks them against a configurable threshold (default `1e-3`). The comparison is restricted to the post-fusion attention input, which `--override-dims` declares replicated across TP ranks:

```
--filter self_attn\.inputs\.hidden_states
--override-dims ^non_intrusive__model\.layers\.\d+\.self_attn\.inputs\.hidden_states$:bs h # tp:replicated
```

Declaring the axis replicated makes the comparator check that every rank holds the same value, rather than summing partial contributions across ranks. Layer-entry `inputs.1` tensors are still captured, but they are excluded from the comparison by `--filter` (see below).

If the comparator returns exit code 0 but compared **zero layers** (baseline/target name mismatch), the test fails with a diagnostic message rather than silently passing.

### Capture signature

A `capture_signature` (SHA-1 hash of schema version, max\_tokens, ignore\_eos, TP size, dumper filter, comparator filter, and fusion backend) is computed per run. The HF store uses this signature during fetch to ensure only baselines with an identical capture and comparison contract are considered. If the signature changes (e.g. you add layers to the capture set or change TP), the framework establishes a fresh baseline instead of erroring on incompatible tensors.

### Fusion and what gets compared

The harness pins FlashInfer all-reduce fusion **on** (`--flashinfer-allreduce-fusion-backend trtllm`; the SM90 auto-enable was dropped in #23402). Two names are captured per layer:

* `inputs.1` — hidden states entering the layer. The FFN boundary defers the cross-layer all-reduce, so with fusion active these are rank-local TP-partial sums. CPU-side reduction of them is not the kernel's semantic output and drifts across self-hosted runners, so they are **not** compared. `_assert_fused_tp_layout()` keeps them only as a guard: if they come back replicated, fusion silently fell back and the test fails.
* `self_attn.inputs.hidden_states` — the value attention consumes after `attn_boundary.prepare()`, i.e. post fused all-reduce and residual RMSNorm. This is replicated across ranks and is what the comparator actually diffs.

This is why a stale baseline captured with fusion *not* initialized is incompatible: it holds replicated layer-entry tensors where the target holds TP-partial ones, which reads as an approximately `tp_size`x mismatch. Fusion initialization failures are tracked in [FlashInfer #3676](https://github.com/flashinfer-ai/flashinfer/issues/3676) and [SGLang #30875](https://github.com/sgl-project/sglang/issues/30875).

***

## Environment Variables

| Variable | Default | Description |
| - | - | - |
| `SGLANG_PRECISION_MODELS` | `zai-org/GLM-5.1-FP8` | Comma-separated HuggingFace model IDs to test |
| `SGLANG_PRECISION_BASELINE_DIR` | `/tmp/sglang_precision_baselines` | Local directory for baseline tensors |
| `SGLANG_PRECISION_DIFF_THRESHOLD` | `1e-3` | Per-tensor relative diff threshold |
| `SGLANG_PRECISION_FORCE_UPDATE` | `0` | Set to `1` to skip comparison and unconditionally refresh baseline |
| `SGLANG_PRECISION_COMMIT` | *(auto-detected from git)* | Override the sglang commit SHA tagged on push |
| `SGLANG_PRECISION_HF_REPO` | *(required)* | HuggingFace dataset repo for cross-runner baseline storage |
| `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset |
| `SGLANG_PRECISION_HF_TOKEN` | *(required in CI)* | HuggingFace token with write access to the dataset. Kept off `HF_TOKEN`, which already carries the runner's gated-model read token |
| `SGLANG_PRECISION_HF_READ_ONLY` | `0` | Set to `1` to fetch and compare without updating the shared baseline store |

***

## CI Integration

### Workflow job

The test is registered on the `nightly-8-gpu-h200` stage in `.github/workflows/nightly-test-nvidia.yml`, which runs through `_pr-test-stage.yml` like every other CUDA stage. The baseline env below is exported on every scheduled stage; only this test reads it.

Key CI configuration:

```yaml theme={null}
- name: Export precision baseline env
  if: inputs.scheduled
  env:
    BASELINE_HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
  run: |
    {
      echo "SGLANG_PRECISION_BASELINE_DIR=/tmp/sglang_precision_baselines"
      echo "SGLANG_PRECISION_HF_REPO=${{ vars.SGLANG_PRECISION_HF_REPO }}"
      echo "SGLANG_PRECISION_HF_REVISION=${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}"
      echo "SGLANG_PRECISION_COMMIT=${{ github.sha }}"
      echo "SGLANG_PRECISION_HF_TOKEN=${BASELINE_HF_TOKEN}"
    } >> "$GITHUB_ENV"
```

`SGLANG_PRECISION_HF_TOKEN` rather than `HF_TOKEN`: the latter already carries the runner's gated-model read token, and overwriting it would turn every gated model on the job into a 401.

Standalone `/rerun-test` runs fetch the nightly baseline without changing it. Maintainers can refresh it from an in-repo PR with `/rerun-test --refresh-precision-baseline test/registered/debug_utils/test_nightly_precision_regression.py`.

### Required GitHub secrets/variables

| Name | Type | Purpose |
| - | - | - |
| `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset |
| `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) |
| `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset; exported to the job as `SGLANG_PRECISION_HF_TOKEN` |

### GitHub Step Summary

When running in CI, the test writes a Markdown table to the GitHub Actions job summary showing each model's status (`PASSED`, `FAILED`, `BASELINE_ESTABLISHED`, or `ERROR`).

***

## HF Dataset Storage Layout

Baselines are organized in the HF dataset as:

```
<model_sanitized>/<YYYY>/<MM>/<DD>/run-<sha7>/
├── meta.json                    # Run metadata (model, commit, hardware, thresholds, stats)
├── comparator_report.jsonl      # Per-tensor comparison results
└── tensors/
    ├── layer_0_inputs_1.pt
    ├── layer_7_inputs_1.pt
    └── ...
```

A top-level `manifest.jsonl` tracks all runs with one JSON object per line. Each manifest row carries a `capture_signature` field so that fetch selects only baselines with a matching capture shape.

The `prune_old_runs()` function (callable manually) retains daily runs for 30 days and keeps one run per week beyond that window.

***

## How to Add a New Model

### Option A: Add to the default model list (CI)

Edit the default in `test/registered/debug_utils/test_nightly_precision_regression.py`:

```python theme={null}
DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.1-FP8,your-org/your-model"
```

Or set the `SGLANG_PRECISION_MODELS` environment variable in the CI workflow to override the default.

### Option B: Run locally for a specific model

```bash theme={null}
export SGLANG_PRECISION_MODELS="your-org/your-model"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines"
export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export SGLANG_PRECISION_HF_TOKEN="hf_..."

cd test
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
```

### Step-by-step: adding a model to the nightly CI

1. **Verify the model works with the dumper.** Run locally first to ensure hidden states are captured correctly:

   ```bash theme={null}
   export SGLANG_PRECISION_MODELS="your-org/your-model"
   export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines"
   export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
   export SGLANG_PRECISION_HF_TOKEN="hf_..."
   export SGLANG_PRECISION_FORCE_UPDATE="1"  # first run: establish baseline

   cd test
   python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision
   ```

2. **Run a comparison pass** (remove `FORCE_UPDATE`):

   ```bash theme={null}
   unset SGLANG_PRECISION_FORCE_UPDATE
   python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision
   ```

   This should report `PASSED` if the engine is numerically stable for the model.

3. **Set the tensor-parallelism size.** If the model requires TP > 1, the test harness defaults to `tp_size=8` for all models. To customize, modify the `ModelLaunchSettings` construction in the test or pass extra server arguments:

   ```python theme={null}
   # In setUpClass or via env-driven logic
   cls.models = [ModelLaunchSettings("your-org/your-model", tp_size=4)]
   ```

4. **Adjust the diff threshold if needed.** FP8 or quantized models may exhibit larger numerical differences. Set `SGLANG_PRECISION_DIFF_THRESHOLD` to an appropriate value (e.g., `1e-2` for FP8).

5. **Add to the default model list** or configure `SGLANG_PRECISION_MODELS` in the CI workflow.

### Considerations for model-specific adjustments

| Concern | How to handle |
| - | - |
| TP size != 8 | Override `tp_size` in `ModelLaunchSettings` or add model-specific logic |
| Quantized models (FP8, GPTQ) | Loosen `SGLANG_PRECISION_DIFF_THRESHOLD` (e.g., `1e-2`) |
| Model needs extra server args | Pass them via `ModelLaunchSettings(model, extra_args=["--quantization", "fp8"])` |
| Model needs different prompt | Modify `PROMPT` constant or make it model-configurable |
| MoE models with TP partial sums | Already handled by `--override-dims` (`bs h[tp:partial]`) |
| Fewer/more capture layers | Adjust `LAYER_CAPTURE_STRIDE` (default 8); set lower for smaller models |
| Decode not captured | Ensure `--max-total-tokens` is well above the scheduler's decode reservation (default 512); the test uses 4096 |

***

## Running Locally

### Prerequisites

* SGLang installed in development mode
* GPUs matching the model's requirements
* `huggingface_hub` installed
* A **HuggingFace dataset** for baseline storage and a write-capable `SGLANG_PRECISION_HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback.

### Quick local test

```bash theme={null}
# All three are required — the test errors if SGLANG_PRECISION_HF_REPO is unset.
export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export SGLANG_PRECISION_HF_TOKEN="hf_..."

# First run: establish baseline
cd test
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v

# Second run: compare against baseline
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
```

### Force-refresh a baseline

```bash theme={null}
export SGLANG_PRECISION_FORCE_UPDATE="1"
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
```

***

## Interpreting Results

### Status codes

| Status | Meaning |
| - | - |
| `BASELINE_ESTABLISHED` | No prior baseline with a matching signature existed; today's tensors saved as the new baseline |
| `PASSED` | All per-layer hidden states are within the diff threshold; baseline updated |
| `FAILED` | One or more layers exceeded the diff threshold, or 0 layers were compared (baseline/target mismatch); diagnostic data pushed to HF |
| `ERROR` | Server launch, inference, or comparison encountered an unexpected error |

### Output example

```
============================================================
Nightly Precision Regression Summary
============================================================
Model                                          Status                   Details
------------------------------------------------------------
zai-org/GLM-5.1-FP8                           PASSED                   comparison ok, baseline updated
Qwen/Qwen2.5-0.5B-Instruct                    FAILED                   tensor=layer_23.inputs_1 rel_diff=0.0152
============================================================
```

### When a failure is detected

1. The comparator output is saved to `/tmp/nightly_precision_<model>_*.log`
2. The failing tensors and comparator report are pushed to the HF dataset with `pass_label="failed"` for offline diagnosis
3. The GitHub Step Summary includes the failure details
4. The CI job exits with a non-zero status

***

## Baseline Management

### Local baselines

Baselines are stored at:

```
$SGLANG_PRECISION_BASELINE_DIR/<model_sanitized>/nightly_precision/*.pt
```

A `baseline_meta.json` next to the tensors records the timestamp and commit that produced the baseline.

### HF dataset baselines

* **Fetch:** At test start, if no local baseline exists, the latest signature-matched baseline is downloaded from the HF dataset.
* **Push:** After each run, tensors and metadata are uploaded to the dataset.
* **Prune:** Use `prune_old_runs()` to garbage-collect old baselines (keeps 30 days of daily runs, one per week after that).

### Refreshing a stale baseline

If an intentional numerical change (e.g., kernel optimization, model refactor) causes a comparison failure:

1. Verify the change is intentional
2. Set `SGLANG_PRECISION_FORCE_UPDATE=1` and run the test once to establish a new baseline
3. Commit any necessary threshold adjustments

If you change the capture configuration (stride, TP size, etc.), the `capture_signature` will differ and the framework automatically establishes a fresh baseline — no manual intervention needed.

***

## Known Limitations

### Baseline drift

The framework uses a **rolling baseline**: every successful comparison updates the baseline to the current run's tensors. This means the reference shifts forward each day. While individual day-to-day diffs stay within the configured threshold, tiny numerical differences can **accumulate over time**, causing the baseline to silently drift away from the original golden values.

**Implications:**

* The framework detects **regressions** (a sudden, large numerical change between consecutive runs), not **absolute accuracy** relative to a fixed reference.
* Over weeks or months, the cumulative drift may become significant enough to mask a real regression that happened gradually, or to cause a false-positive failure when the drift eventually crosses the threshold.

**Mitigation strategies (not yet implemented):**

* Periodically re-establish a fresh anchor baseline from a known-good reference commit.
* Track the cumulative drift in the manifest metadata and alert when it exceeds a long-term budget.
* Compare against a fixed "epoch" baseline in addition to the rolling one.

### No local-only mode

The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `SGLANG_PRECISION_HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token.

***

## File Reference

| File | Role |
| - | - |
| `test/registered/debug_utils/test_nightly_precision_regression.py` | Main test — server lifecycle, dump, compare, report |
| `python/sglang/test/precision_baseline_store.py` | HF dataset store — push, fetch, prune baselines |
| `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine |
| `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture |
| `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition |
| `test/run_suite.py` | Test suite registration (includes `nightly-test-8-gpu-h200`) |


# References
Source: https://docs.sglang.io/docs/references/overview

FAQ, environment variables, production metrics, deployment guides, and more.

* [FAQ](./faq)
* [Environment Variables](./environment_variables)
* [Production Metrics](./production_metrics)
* [Production Request Trace](./production_request_trace)
* [Multi-Node Deployment](./multi_node_deployment/multi_node)
* [Custom Chat Template](./custom_chat_template)
* [Frontend Language](./frontend/frontend_tutorial)
* [Post-Training Integration](./post_training_integration)


# Post-Training Integration
Source: https://docs.sglang.io/docs/references/post_training_integration



SGLang has become the de facto inference backend for modern LLM training frameworks, powering state-of-the-art models across the industry. From GLM-4.6 to Qwen3, leading models leverage SGLang's high-performance inference during reinforcement learning and post-training workflows.

What makes SGLang essential for post-training?

* Open-To-Use Refit Functionality: diverse method for colocate or disaggregate
* Easy To Postpone Generation: enable partial rollout and dedicated rollout control
* Fine-Grained Engine Sleep And Wake Up: facilitate maxium-powered rollout and training
* Training Serving Alignment: ensure the performance consistency in training and serving
* Load Balancing Router: cache-aware load-balancing for high-throughput rollout
* Deterministic Inference: ensure zero kl divergence between rollout and training

These capabilities, combined with native integration support across major frameworks, have established SGLang as the infrastructure backbone for modern LLM/VLMs post-training. We also share our latest work in this slide, [Optimizing Large-Scale RL with SGLang](https://gamma.app/docs/Optimizing-RL-with-SGLang-y0kqgj877k34779).

## Adoption

* [**Miles**](https://github.com/radixark/miles): Enterprise-scale RL framework for large MoE models with SGLang-native rollout, speculative training, and production-grade stability
* [**slime**](https://github.com/THUDM/slime): Post-training framework combining Megatron and SGLang, used to train GLM-4.6
* [**AReaL**](https://github.com/inclusionAI/AReaL): Fully asynchronous RL system achieving 2.77x speedup with SGLang backend for continuous rollout generation
* [**ROLL**](https://github.com/alibaba/ROLL): ROLL is an efficient and user-friendly RL library designed for Large Language Models utilizing Large Scale GPU resources
* [**verl**](https://github.com/volcengine/verl): Full-stack RLHF framework supporting PPO, GRPO, and ReMax with modular SGLang integration
* [**Unsloth**](https://docs.unsloth.ai/basics/inference-and-deployment/sglang-guide): 2x faster fine-tuning with optimized kernels, deploys seamlessly with SGLang inference
* [**LLaMA Factory**](https://github.com/hiyouga/LLaMA-Factory): Unified framework for training 100+ LLMs with LoRA, QLoRA, and full fine-tuning methods
* [**Tunix**](https://github.com/google/tunix): Google's JAX-native library for LLM post-training with SFT, DPO, PPO, and GRPO support
* [**RL2**](https://github.com/ChenmienTan/RL2): Ray Less Reinforcement Learning, a concise library of post-training for large language models

## Collaboration

Due to the privacy of the design parternes, we cannot list the companies that adopt SGLang for post-training. However, we are happy to share the details with you if you are interested and trust the choice among 10+ top companies and frontier labs across US and China. If you are interested in integrating SGLang with your training framework or need technical support, we're here to help! Reach out to us at **[rl\_team@lmsys.org](mailto:rl_team@lmsys.org)** for partnerships, integration guidance, and custom feature development.


# Production Metrics
Source: https://docs.sglang.io/docs/references/production_metrics



SGLang exposes the following metrics via Prometheus. You can enable it by adding `--enable-metrics` when you launch the server.

An example of the monitoring dashboard is available in [examples/monitoring/grafana.json](https://github.com/sgl-project/sglang/blob/main/examples/monitoring/grafana/dashboards/json/sglang-dashboard.json).

## Language model metrics

`sglang:time_to_first_token_seconds` measures the time from request creation in the API server to the first token. For streaming requests it ends when the first output arrives after detokenization, which may contain no printable text. Non-streaming clients only see the final response, so for them it ends when the scheduler produced the first token.

Here is an example of the metrics:

```text Output theme={null}
$ curl http://localhost:30000/metrics
# HELP sglang:prompt_tokens_total Number of prefill tokens processed.
# TYPE sglang:prompt_tokens_total counter
sglang:prompt_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 8.128902e+06
# HELP sglang:generation_tokens_total Number of generation tokens processed.
# TYPE sglang:generation_tokens_total counter
sglang:generation_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.557572e+06
# HELP sglang:token_usage The token usage
# TYPE sglang:token_usage gauge
sglang:token_usage{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.28
# HELP sglang:cache_hit_rate The cache hit rate
# TYPE sglang:cache_hit_rate gauge
sglang:cache_hit_rate{model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.007507552643049313
# HELP sglang:time_to_first_token_seconds Histogram of time to first token in seconds.
# TYPE sglang:time_to_first_token_seconds histogram
sglang:time_to_first_token_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2.3518979474117756e+06
sglang:time_to_first_token_seconds_bucket{le="0.001",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
sglang:time_to_first_token_seconds_bucket{le="0.005",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
sglang:time_to_first_token_seconds_bucket{le="0.01",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
sglang:time_to_first_token_seconds_bucket{le="0.02",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
sglang:time_to_first_token_seconds_bucket{le="0.04",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1.0
sglang:time_to_first_token_seconds_bucket{le="0.06",model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0
sglang:time_to_first_token_seconds_bucket{le="0.08",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:time_to_first_token_seconds_bucket{le="0.1",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:time_to_first_token_seconds_bucket{le="0.25",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:time_to_first_token_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:time_to_first_token_seconds_bucket{le="0.75",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:time_to_first_token_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 27.0
sglang:time_to_first_token_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 140.0
sglang:time_to_first_token_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 314.0
sglang:time_to_first_token_seconds_bucket{le="7.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 941.0
sglang:time_to_first_token_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1330.0
sglang:time_to_first_token_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1970.0
sglang:time_to_first_token_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2326.0
sglang:time_to_first_token_seconds_bucket{le="25.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2417.0
sglang:time_to_first_token_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 2513.0
sglang:time_to_first_token_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0
sglang:time_to_first_token_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11008.0
# HELP sglang:e2e_request_latency_seconds Histogram of End-to-end request latency in seconds
# TYPE sglang:e2e_request_latency_seconds histogram
sglang:e2e_request_latency_seconds_sum{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.116093850019932e+06
sglang:e2e_request_latency_seconds_bucket{le="0.3",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0
sglang:e2e_request_latency_seconds_bucket{le="0.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:e2e_request_latency_seconds_bucket{le="0.8",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:e2e_request_latency_seconds_bucket{le="1.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:e2e_request_latency_seconds_bucket{le="1.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:e2e_request_latency_seconds_bucket{le="2.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:e2e_request_latency_seconds_bucket{le="2.5",model_name="meta-llama/Llama-3.1-8B-Instruct"} 6.0
sglang:e2e_request_latency_seconds_bucket{le="5.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.0
sglang:e2e_request_latency_seconds_bucket{le="10.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 10.0
sglang:e2e_request_latency_seconds_bucket{le="15.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11.0
sglang:e2e_request_latency_seconds_bucket{le="20.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 14.0
sglang:e2e_request_latency_seconds_bucket{le="30.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 247.0
sglang:e2e_request_latency_seconds_bucket{le="40.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 486.0
sglang:e2e_request_latency_seconds_bucket{le="50.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 845.0
sglang:e2e_request_latency_seconds_bucket{le="60.0",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1513.0
sglang:e2e_request_latency_seconds_bucket{le="+Inf",model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0
sglang:e2e_request_latency_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 11228.0
# HELP sglang:func_latency_seconds Function latency in seconds
# TYPE sglang:func_latency_seconds histogram
sglang:func_latency_seconds_sum{name="generate_request"} 4.514771912145079
sglang:func_latency_seconds_bucket{le="0.05",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="0.07500000000000001",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="0.1125",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="0.16875",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="0.253125",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="0.3796875",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="0.56953125",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="0.8542968750000001",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="1.2814453125",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="1.9221679687500002",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="2.8832519531250003",name="generate_request"} 14006.0
sglang:func_latency_seconds_bucket{le="4.3248779296875",name="generate_request"} 14007.0
sglang:func_latency_seconds_bucket{le="6.487316894531251",name="generate_request"} 14007.0
sglang:func_latency_seconds_bucket{le="9.730975341796876",name="generate_request"} 14007.0
sglang:func_latency_seconds_bucket{le="14.596463012695313",name="generate_request"} 14007.0
sglang:func_latency_seconds_bucket{le="21.89469451904297",name="generate_request"} 14007.0
sglang:func_latency_seconds_bucket{le="32.84204177856446",name="generate_request"} 14007.0
sglang:func_latency_seconds_bucket{le="49.26306266784668",name="generate_request"} 14007.0
sglang:func_latency_seconds_bucket{le="+Inf",name="generate_request"} 14007.0
sglang:func_latency_seconds_count{name="generate_request"} 14007.0
# HELP sglang:num_running_reqs The number of running requests
# TYPE sglang:num_running_reqs gauge
sglang:num_running_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 162.0
# HELP sglang:num_used_tokens The number of used tokens
# TYPE sglang:num_used_tokens gauge
sglang:num_used_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 123859.0
# HELP sglang:gen_throughput The generate throughput (token/s)
# TYPE sglang:gen_throughput gauge
sglang:gen_throughput{model_name="meta-llama/Llama-3.1-8B-Instruct"} 86.50814177726902
# HELP sglang:num_queue_reqs The number of requests in the waiting queue
# TYPE sglang:num_queue_reqs gauge
sglang:num_queue_reqs{model_name="meta-llama/Llama-3.1-8B-Instruct"} 2826.0
# HELP sglang:spec_num_steps Currently active speculative_num_steps.
# TYPE sglang:spec_num_steps gauge
sglang:spec_num_steps{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0
# HELP sglang:spec_num_draft_tokens Currently active speculative_num_draft_tokens (decouples from steps under topk>1).
# TYPE sglang:spec_num_draft_tokens gauge
sglang:spec_num_draft_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.0
```

## Request-level time per output token (TPOT)

`sglang:request_time_per_output_token_seconds` records TPOT once per successfully finished request:

```text theme={null}
TPOT = (finished_time - first_token_time) / (completion_tokens - 1)
     = 1 / meta_info["decode_throughput"]
```

The first token is excluded because its latency belongs to TTFT. Requests with fewer than two output tokens, streaming requests that finish in their first output, aborted requests, and PD prefill workers do not contribute. The histogram is split by `is_streaming` and reuses the `--bucket-inter-token-latency` buckets.

Unlike the token-weighted `sglang:inter_token_latency_seconds`, this metric gives every qualifying request equal weight.

p95 TPOT per model:

```promql theme={null}
histogram_quantile(
  0.95,
  sum by (model_name, is_streaming, le) (
    rate(sglang:request_time_per_output_token_seconds_bucket[5m])
  )
)
```

Mean TPOT is `rate(..._sum[5m]) / rate(..._count[5m])`. Per-request decode tokens/s is its reciprocal, so p5 TPS is `1 / p95 TPOT`; the reciprocal of mean TPOT is not mean TPS.

## Diffusion metrics

SGLang Diffusion exposes request, queue, stage and LoRA metrics with
`--enable-metrics`. See [Diffusion production metrics](/docs/sglang-diffusion/production_metrics)
for the metric reference, counting semantics and disaggregated scraping setup.

## Setup Guide

This section describes how to set up the monitoring stack (Prometheus + Grafana) provided in the `examples/monitoring` directory.

### Prerequisites

* Docker and Docker Compose installed
* SGLang server running with metrics enabled

### Usage

1. **Start your SGLang server with metrics enabled:**

   ```bash Command theme={null}
   python -m sglang.launch_server \
     --model-path <your_model_path> \
     --port 30000 \
     --enable-metrics \
     --enable-mfu-metrics
   ```

   Replace `<your_model_path>` with the actual path to your model (e.g., `meta-llama/Meta-Llama-3.1-8B-Instruct`). Ensure the server is accessible from the monitoring stack (you might need `--host 0.0.0.0` if running in Docker). By default, the metrics endpoint will be available at `http://<sglang_server_host>:30000/metrics`.

2. **Navigate to the monitoring example directory:**
   ```bash Command theme={null}
   cd examples/monitoring
   ```

3. **Start the monitoring stack:**
   ```bash Command theme={null}
   docker compose up -d
   ```
   This command will start Prometheus and Grafana in the background.

4. **Access the monitoring interfaces:**
   * **Grafana:** Open your web browser and go to [http://localhost:3000](http://localhost:3000).
   * **Prometheus:** Open your web browser and go to [http://localhost:9090](http://localhost:9090).

5. **Log in to Grafana:**
   * Default Username: `admin`
   * Default Password: `admin`
     You will be prompted to change the password upon your first login.

6. **View the Dashboard:**
   The SGLang dashboard is pre-configured and should be available automatically. Navigate to `Dashboards` -> `Browse` -> `SGLang Monitoring` folder -> `SGLang Dashboard`.

### Troubleshooting

* **Port Conflicts:** If you encounter errors like "port is already allocated," check if other services (including previous instances of Prometheus/Grafana) are using ports `9090` or `3000`. Use `docker ps` to find running containers and `docker stop <container_id>` to stop them, or use `lsof -i :<port>` to find other processes using the ports. You might need to adjust the ports in the `docker-compose.yaml` file if they permanently conflict with other essential services on your system.

To modify Grafana's port to the other one(like 3090) in your Docker Compose file, you need to explicitly specify the port mapping under the grafana service.

Option 1: Add GF\_SERVER\_HTTP\_PORT to the environment section:

```
  environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_SERVER_HTTP_PORT=3090  # <-- Add this line
```

Option 2: Use port mapping:

```
grafana:
  image: grafana/grafana:latest
  container_name: grafana
  ports:
  - "3090:3000"  # <-- Host:Container port mapping
```

* **Connection Issues:**
  * Ensure both Prometheus and Grafana containers are running (`docker ps`).
  * Verify the Prometheus data source configuration in Grafana (usually auto-configured via `grafana/datasources/datasource.yaml`). Go to `Connections` -> `Data sources` -> `Prometheus`. The URL should point to the Prometheus service (e.g., `http://prometheus:9090`).
  * Confirm that your SGLang server is running and the metrics endpoint (`http://<sglang_server_host>:30000/metrics`) is accessible *from the Prometheus container*. If SGLang is running on your host machine and Prometheus is in Docker, use `host.docker.internal` (on Docker Desktop) or your machine's network IP instead of `localhost` in the `prometheus.yaml` scrape configuration.
* **No Data on Dashboard:**
  * Generate some traffic to your SGLang server to produce metrics. For example, run a benchmark:
    ```bash Command theme={null}
    python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompts 100 --random-input 128 --random-output 128
    ```
  * Check the Prometheus UI (`http://localhost:9090`) under `Status` -> `Targets` to see if the SGLang endpoint is being scraped successfully.
  * Verify the `model_name` and `instance` labels in your Prometheus metrics match the variables used in the Grafana dashboard. You might need to adjust the Grafana dashboard variables or the labels in your Prometheus configuration.

### Configuration Files

The monitoring setup is defined by the following files within the `examples/monitoring` directory:

* `docker-compose.yaml`: Defines the Prometheus and Grafana services.
* `prometheus.yaml`: Prometheus configuration, including scrape targets.
* `grafana/datasources/datasource.yaml`: Configures the Prometheus data source for Grafana.
* `grafana/dashboards/config/dashboard.yaml`: Tells Grafana to load dashboards from the specified path.
* `grafana/dashboards/json/sglang-dashboard.json`: The actual Grafana dashboard definition in JSON format.

You can customize the setup by modifying these files. For instance, you might need to update the `static_configs` target in `prometheus.yaml` if your SGLang server runs on a different host or port.

#### Check if the metrics are being collected

Run:

```text Output theme={null}
python3 -m sglang.bench_serving \
  --backend sglang \
  --dataset-name random \
  --num-prompts 3000 \
  --random-input 1024 \
  --random-output 1024 \
  --random-range-ratio 0.5
```

to generate some requests.

Then you should be able to see the metrics in the Grafana dashboard.

## Estimated Performance Metrics (MFU-related)

SGLang exports the following estimated per-GPU counters that can be used to derive
Model FLOPs Utilization (MFU)-related signals:

* `sglang:estimated_flops_per_gpu_total`: Estimated floating-point operations.
* `sglang:estimated_read_bytes_per_gpu_total`: Estimated bytes read from memory.
* `sglang:estimated_write_bytes_per_gpu_total`: Estimated bytes written to memory.

These metrics are available when both `--enable-metrics` and
`--enable-mfu-metrics` are enabled.

These are cumulative counters. Use Prometheus `rate(...)` to get per-second values.

### PromQL examples

Average TFLOPS per GPU:

```promql theme={null}
rate(sglang:estimated_flops_per_gpu_total[1m]) / 1e12
```

Average estimated memory bandwidth in GB/s:

```promql theme={null}
(rate(sglang:estimated_read_bytes_per_gpu_total[1m]) +
 rate(sglang:estimated_write_bytes_per_gpu_total[1m])) / 1e9
```

### Notes

* These metrics are estimates intended for observability and trend analysis.
* Estimated memory bytes reflect modeled traffic and are not a direct hardware
  counter from GPU profilers.


# Production Request Tracing
Source: https://docs.sglang.io/docs/references/production_request_trace



SGLang exports request trace data based on the OpenTelemetry Collector. You can enable tracing by adding the `--enable-trace` and configure the OpenTelemetry Collector endpoint using `--otlp-traces-endpoint` when launching the server.

You can find example screenshots of the visualization in [https://github.com/sgl-project/sglang/issues/8965](https://github.com/sgl-project/sglang/issues/8965).

## Setup Guide

This section explains how to configure the request tracing and export the trace data.

1. Install the required packages and tools
   * install Docker and Docker Compose
   * install the dependencies
   ```bash Command theme={null}
   # enter the SGLang root directory
   pip install -e "python[tracing]"

   # or manually install the dependencies using pip
   pip install opentelemetry-sdk opentelemetry-api opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-grpc
   ```

2. Launch OpenTelemetry collector and Jaeger
   ```bash Command theme={null}
   docker compose -f examples/monitoring/tracing_compose.yaml up -d
   ```

3. Start your SGLang server with tracing enabled

   ```bash Command theme={null}
   # set env variables
   export SGLANG_OTLP_EXPORTER_SCHEDULE_DELAY_MILLIS=500
   export SGLANG_OTLP_EXPORTER_MAX_EXPORT_BATCH_SIZE=64
   # start the prefill and decode server
   python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other option>
   # start the model-gate-way
   python -m sglang_router.launch_router --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other option>
   ```

   Replace `0.0.0.0:4317` with the actual endpoint of the OpenTelemetry collector. If you launched the openTelemetry collector with tracing\_compose.yaml, the default receiving port is 4317.

   To use the HTTP/protobuf span exporter, set the following environment variable and point to an HTTP endpoint, for example, `http://0.0.0.0:4318/v1/traces`.

   ```bash Command theme={null}
   export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
   ```

4. Raise some requests

5. Observe whether trace data is being exported
   * Access port 16686 of Jaeger using a web browser to visualize the request traces.
   * The OpenTelemetry Collector also exports trace data in JSON format to /tmp/otel\_trace.json. In a follow-up patch, we will provide a tool to convert this data into a Perfetto-compatible format, enabling visualization of requests in the Perfetto UI.

6. Dynamically adjust trace level
   The trace level accepts configurable values from `0` to `3`. The meanings of different trace level values are as follows:

   ```
   0: disable tracing
   1: Trace important slices
   2: Trace all slices except nested ones
   3: Trace all slices (default)
   ```

   **At startup** — set `SGLANG_TRACE_LEVEL` before launching the server:

   ```bash Command theme={null}
   SGLANG_TRACE_LEVEL=2 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other options>
   ```

   **At runtime** — dynamically adjust via HTTP API without restarting:

   ```bash Command theme={null}
   curl http://0.0.0.0:30000/set_trace_level?level=2
   ```

   Replace `0.0.0.0:30000` with your actual server address, and replace `level=2` with the level you want to set.

   **Note**: You must set the parameter `--enable-trace`; otherwise, the trace capability will not be enabled regardless of any dynamic adjustments to the trace level.

## Async Tracing (Reducing Performance Overhead)

When batch sizes are large, synchronous OTel span creation can degrade inference throughput due to thread-safe locking and background export threads. Async tracing moves all span creation to a dedicated exporter process via ZMQ, keeping the scheduler and tokenizer hot paths free of OTel overhead.

**Enable async tracing:**

```bash theme={null}
SGLANG_TRACE_ASYNC=1 python -m sglang.launch_server --enable-trace --otlp-traces-endpoint 0.0.0.0:4317 <other options>
```

**How it works:**

* A daemon exporter process is started per worker process (scheduler, tokenizer, etc.).
* The root span is still created in the caller process (one per request, negligible cost), preserving cross-process span linking via `traceparent`.
* Thread spans and slice spans are buffered as lightweight operation dicts and flushed to the exporter via ZMQ PUSH/PULL.
* Span IDs are pre-generated in the caller process using `TraceCustomIdGenerator.preset_next_span_id()`, so the exported span tree is identical to synchronous mode.
* Thread info (scheduler label, TP/DP/PP ranks) is registered once via a callback on `trace_set_thread_info()`.

**Tuning:**

| Environment Variable | Description | Default |
| - | - | - |
| `SGLANG_TRACE_ASYNC` | Enable async tracing | `false` |
| `SGLANG_TRACE_ASYNC_FLUSH_THRESHOLD` | Max buffered ops before auto-flush | `100` |

## How to add Tracing for slices you're interested in?(API introduction)

We have already inserted instrumentation points in the tokenizer and scheduler main threads. If you wish to trace additional request execution segments or perform finer-grained tracing, please use the APIs from the tracing package as described below.

**All of the following implementations are done in python/sglang/srt/observability/req\_time\_stats.py. If you want to add another slice, please do it here.**

1. Initialization

   Every process involved in tracing during the initialization phase should execute:

   ```python Example theme={null}
   process_tracing_init(otlp_traces_endpoint, server_name)
   ```

   The otlp\_traces\_endpoint is obtained from the arguments, and you can set server\_name freely, but it should remain consistent across all processes.

   Every thread involved in tracing during the initialization phase should execute:

   ```python Example theme={null}
   trace_set_thread_info("thread label", tp_rank, dp_rank)
   ```

   The "thread label" can be regarded as the name of the thread, used to distinguish different threads in the visualization view.

2. Create a trace context for a request
   Each request needs to call `TraceReqContext()` to initialize a request context, which is used to generate slice spans and record request stage info. You can either store it within the request object or maintain it as a global variable.

3. Mark the beginning and end of a request
   ```
   trace_ctx.trace_req_start().
   trace_ctx.trace_req_finish()
   ```
   trace\_req\_start() and trace\_req\_finish() must be called within the same process, for example, in the tokenizer.

4. Add tracing for a slice

   * Add slice tracing normally:
     ```python Example theme={null}
     trace_ctx.trace_slice_start(RequestStage.TOKENIZER.stage_name)
     trace_ctx.trace_slice_end(RequestStage.TOKENIZER.stage_name)

     or
     trace_ctx.trace_slice(slice: TraceSliceContext)
     ```

   - The end of the last slice in a thread must be marked with thread\_finish\_flag=True, or explicitly call trace\_ctx.abort(); otherwise, the thread's span will not be properly generated.
     ```python Example theme={null}
     trace_ctx.slice_end(RequestStage.D.stage_name, thread_finish_flag = True)
     trace_ctx.abort()
     ```

5. When the request execution flow transfers to another thread, the thread context needs to be explicitly rebuilt.
   * receiver: Execute the following code after receiving the request via ZMQ
     ```python Example theme={null}
     trace_ctx.rebuild_thread_context()
     ```

## How to Extend the Tracing Framework to Support Complex Tracing Scenarios

The currently provided tracing package still has potential for further development. If you wish to build more advanced features upon it, you must first understand its existing design principles.

The core of the tracing framework's implementation lies in the design of the span structure and the trace context. To aggregate scattered slices and enable concurrent tracking of multiple requests, we have designed a three-level trace context structure or span structure: `TraceReqContext`, `TraceThreadContext` and `TraceSliceContext`. Their relationship is as follows:

```
TraceReqContext (req_id="req-123")
├── TraceThreadContext(thread_label="scheduler", tp_rank=0)
|     └── TraceSliceContext(slice_name="prefill")
|
└── TraceThreadContext(thread_label="scheduler", tp_rank=1)
      └── TraceSliceContext(slice_name="prefill")
```

Each traced request maintains a global `TraceReqContext` and creates a corresponding request span. For every thread that processes the request, a `TraceThreadContext` is recorded and a thread span is created. The `TraceThreadContext` is nested within the `TraceReqContext`, and each currently traced code slice—potentially nested—is stored in its associated `TraceThreadContext`.

In addition to the above hierarchy, each slice also records its previous slice via Span.add\_link(), which can be used to trace the execution flow.


# CLI reference
Source: https://docs.sglang.io/docs/sglang-diffusion/api/cli

Run one-off generation tasks and launch the HTTP server from the command line.

Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`.

### Overlay repos for non-diffusers models

If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it
through a self-hosted overlay repo.

SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface.

Override example:

```bash Command theme={null}
export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{
  "Wan-AI/Wan2.2-S2V-14B": {
    "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay",
    "overlay_revision": "main"
  }
}'

sglang generate \
  --model-path Wan-AI/Wan2.2-S2V-14B \
  --config configs/wan_s2v.yaml
```

The overlay repo should be a complete diffusers-style/componentized repo

You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`.

Notes:

1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for
   development and debugging. It accepts either a JSON object or a path to a JSON
   file, and can extend or replace built-in entries for the current process.
2. On the first load, SGLang will:
   * download overlay metadata from the overlay repo
   * download the required files from the original source repo
   * materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/`
3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory.

## Quick Start

### Generate

```bash Command theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --prompt "A beautiful sunset over the mountains" \
  --save-output
```

### Serve

```bash Command theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010
```

For request and response examples, see [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api).

> [!TIP]
>
> Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags.

## Common Options

### Model and runtime

* `--model-path {MODEL}`: model path or Hugging Face model ID
* `--served-model-name {NAME}`: stable model name exposed by serving APIs. Defaults to `--model-id` when set, otherwise `--model-path`.
* `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`, or `hybrid` with explicit merged transformer weights to serve all three tasks (see the [H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#singularity-hybrid-weights)). This is a server/load-time choice, unlike a request's `task`.
* `--minimax-h3-adaln-cache-path {FILE}`: advanced MiniMax-H3-only inference cache. It replaces the checkpoint's AdaLN projection weights with precomputed outputs and only accepts requests whose exact FP32 timestep plan is included in the cache. It requires unquantized weights and the matching model variant.
* `--minimax-h3-adaln-online {true,false}`: rebuild MiniMax-H3 AdaLN outputs from the checkpoint on demand instead of keeping the 24.2 GiB of `adaln_proj` weights resident. Works with any step count or schedule; requires the unquantized native-layout checkpoint. Built plans live in a GPU slab with per-plan LRU eviction and, by default, a pinned-host cache so previously seen schedules swap back in over PCIe instead of re-reading the checkpoint.
* `--minimax-h3-adaln-plan-width {N}`: widest timestep plan the online slab is sized for (default 4 covers every task; t2va needs 2, fl2va 3).
* `--minimax-h3-adaln-host-cache-gb {GB}`: pinned host memory per rank caching built AdaLN plans (default 8; 0 disables). A 50-step schedule needs about 0.9 (t2va) / 1.33 (fl2va) / 1.77 (ref2va) GB; over-capacity plan sets simply recompute. Expert escape hatches (GPU slot count, experimental fp32 rebuild) are the `SGLANG_DIFFUSION_MINIMAX_H3_ADALN_*` environment variables.
* `--model-subfolder {PATH}`: advanced direct override for a component subfolder inside the model repository. Prefer `--model-variant` when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition.
* `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter from a local path, Hugging Face repo/subfolder, or exact Hub file URL
* `--lora-weight-name {FILE}`: select one adapter file from a repository that contains multiple LoRA revisions. The Hub download is filtered to that file plus JSON metadata, so unused weights are not downloaded.
* `--lora-alpha {N}`: supply the training alpha when a single-file adapter omits both per-layer alpha tensors and `adapter_config.json`. Do not set it when the adapter already records alpha metadata.
* PEFT `adapter_config.json` semantics are applied automatically for named adapter slots, RSLoRA, and per-layer `alpha_pattern`. Adapters that require unsupported auxiliary parameters or runtime behavior, such as DoRA, fail before weight injection instead of silently using ordinary LoRA math.
* `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` normally merges regular weights, but keeps FSDP-sharded weights dynamic to avoid full-gather peaks. For a single adapter, it also stays dynamic when an estimate shows that merging would round away more than 50% of the update's norm. `merge` forces merging; `dynamic` applies the adapter at forward time.
* `--num-gpus {N}`: number of GPUs to use
* `--numa-node {N} [N ...]`: NUMA node for each GPU worker, the i-th value for the worker on local GPU i. When unset, each worker binds its CPUs and host memory to its GPU's NUMA node on multi-socket hosts, as srt does (CPUs only without `CAP_SYS_NICE`); set `SGLANG_AUTO_NUMA_BIND=0` to turn that off. Unbound workers on a two-socket host (e.g. GB200/GB300) can drift to the other socket, which slows host-bound steps and makes latency vary run to run.
* `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and dispatches residency from selected-GPU headroom and workload type: image DiTs stay resident above the 45 GiB threshold, while video DiT placement remains model-specific. It uses FSDP only for validated DiT-offload replacement paths. `speed` keeps `torch.compile` disabled unless a model-specific deployment config opts in after validation; pass `--enable-torch-compile true` to enable it explicitly. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes.
* `--direct-gpu-weight-loading {true|false}`: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Compatible tensors become parameter storage without another GPU copy, while tensors requiring transforms can still need temporary allocations. It is disabled by default; startup time and peak GPU memory remain model-dependent, so benchmark the target model before deployment. It is incompatible with DiT CPU/layerwise offload and FSDP.
* `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both.
* `--sp-degree {N}`: sequence parallelism size
* `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only.
* `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
* `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading.
* `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism. Requests that disable CFG remain valid on a CFG-parallel server, but extra CFG ranks redundantly recompute the single active branch.
* `--encoder-parallel {auto|fold|dp|replicate}`: how native encoders use the GPUs in each DiT replica. `auto` TP-folds a native text/image encoder wide enough to benefit, selects batch DP for an explicitly supported native text encoder when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards native text/image encoders across the full replica whenever dimensions allow; `dp` splits a batched encode across supported native text encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).
* `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic
* `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency.
* `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP.
* `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Each served shape is captured separately at warmup: pass every served resolution in `--warmup-resolutions` (and the frame count in `--warmup-num-frames` for video), otherwise only the model default shape is captured and other shapes run eager. Unset, it is on for `LTX2TwoStagePipeline` on LTX-2/LTX-2.3 when CUDA serving on GPUs with at least 130 GiB keeps the DiTs resident (server warmup; no torch.compile, Cache-DiT, batching, or `--lora-path`; LTX-2.3 in `resident` mode, its default on those GPUs). On a model outside the validated list it stays enabled with a warning; compare outputs against eager. If the startup warmup captures no graph (the model's denoising stage does not use BCG), the server warns that it runs eager. An `lossless` or `high` request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only request-gated paths remain compatible.
* `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse.
* `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines
* `--enable-attention-backend-autotune {true|false}`: for SGLang-native pipelines, benchmark compatible attention backends on each layer's first sufficiently large input and keep a backend only when it is both numerically compatible and measurably faster. Disabled by default and currently validated on SM90 and SM12x. Explicit `--attention-backend`, component overrides, and model-required backends are never replaced.
* `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa`. A key may carry a `.self` or `.cross` role suffix to override one kind of attention only, for example `transformer.cross=fa`
* `--attention-backend-config {CONFIG}`: attention backend configuration
* `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](/docs/sglang-diffusion/models_with_ar).
* `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server
* `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server
* `--scheduler-rpc-timeout {SECONDS}`: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it.
* `--sleep-on-idle`: reduce idle scheduler CPU usage by waiting for socket events. Disabled by default, matching SRT, to minimize idle-to-active wake-up latency. Applies only to monolithic serving with one rank per DP replica; TP/SP/CFG multi-rank replicas keep polling. Each wait ends when a message arrives, a pending allocator-cache release is due, or one second elapses. Active generation is unchanged; benchmark low-QPS tail latency before enabling it in latency-sensitive deployments.
* `--enable-trace`: export OpenTelemetry traces for the diffusion scheduler and worker stages.
* `--otlp-traces-endpoint {HOST:PORT}`: OTLP collector endpoint used with `--enable-trace` (default: `localhost:4317`).
* `--otlp-service-name {NAME}`: `service.name` attached to exported traces. Resolution order is this flag, `OTEL_SERVICE_NAME`, then `sglang-diffusion`.
* `--enable-metrics`: expose Prometheus metrics at `/metrics` (default: disabled). Includes request counts, queue time, host-side stage timing and LoRA state, separated by role and DP replica. See [Production metrics](/docs/sglang-diffusion/production_metrics) for metric semantics and disaggregated scraping.
* `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](/docs/sglang-diffusion/models_with_pe).
* `--prompt-enhancer-config {JSON_FILE}`: configure an independent SRT LLM/VLM for opt-in HTTP prompt rewriting. Requests use `enhance_prompt=true`; no enhancer runs by default. See [Prompt enhancement](/docs/sglang-diffusion/models_with_pe).

Keep internal scheduler ports on a trusted network: they use Python pickle,
not an authenticated public API. On a monolithic scheduler, malformed payloads
in valid request envelopes receive an error without discarding other requests
received in the same poll.
This isolation does not make untrusted pickle safe or recover from GPU failures.

### Disaggregated serving

* `--disagg-role {monolithic|encoder|prompt_encoder|image_encoder|denoiser|decoder|server}`: select the stages run by this process. `monolithic` is the default; the split `prompt_encoder` and `image_encoder` roles currently support FLUX.3 Image.
* `--disagg-server-addr {TCP_ENDPOINT}`: head scheduler endpoint used by workers for registration and result routing.
* `--encoder-urls {ENDPOINTS}`: combined encoder pool on the head. For split encoding, replace it with both `--prompt-encoder-urls` and `--image-encoder-urls`; do not mix the two topologies.
* `--denoiser-urls {ENDPOINTS}` and `--decoder-urls {ENDPOINTS}`: downstream worker pools on the head. Each pool URL flag accepts a quoted, semicolon-separated list of TCP endpoints.
* `--disagg-p2p-hostname {HOST}` and `--disagg-ib-device {DEVICE}`: worker address and optional RDMA device for tensor transfers. The advertised worker address and scheduler port must match its entry in the head's URL list.
* `--disagg-transfer-pool-size {BYTES}`: transfer buffer per worker (default: 268435456 bytes). Include space for incoming tensors, outgoing tensors, and preallocated receive slots.

See [Disaggregated diffusion](/docs/sglang-diffusion/disaggregation#flux3-split-encoding)
for deployment commands, ports, and the validation scope. Generation and editing
still use the existing image APIs; encoding is not exposed as a standalone API.

### Sampling and output

* `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
* `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation
* `--num-inference-steps {STEPS}` and `--seed {SEED}`
* `--num-outputs-per-prompt {N}` / `--num-outputs {N}`: generate multiple outputs for each prompt. A scalar seed expands as `seed + output_index`.
* `--quality {exact,lossless,high}`: cumulative request-level optimization tier. `lossless` (default) runs every fast path that keeps the reference math and the precision of every operand, moving only where the rounding happens; it does not enable quantization, sparsity, caching, or another approximate path. `exact` restricts it to the fast paths that reproduce the reference's rounding exactly, so output is bit-identical in the same environment -- ask for it when a run has to be reproducible. `high` includes the complete `lossless` set and may also enable model-owned approximate optimizations. `extra-high` is accepted as the former name of `lossless`. Separately configured quantization, attention, or caching options still apply. Support and validation constraints are model-specific.
* `skip_softmax_params` (online request only): explicitly enables lossy BLASST/Skip-Softmax attention for compatible FA self-attention layers. See [Attention Backends](../attention_backends#request-scoped-skip-softmax).
* `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
* `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`

For frame interpolation and upscaling, see [Post-Processing](/docs/sglang-diffusion/api/post_processing).

### Quantization

Component checkpoint paths are selected separately, so changing DiT precision
never silently changes prompt embeddings. The component-scoped forms are the
canonical interface for any component key from `model_index.json` or a native
pipeline's registered module name:

| Intent | Canonical option | Convenience alias | Behavior |
| - | - | - | - |
| Replace a component | `--component-paths.<component> {MODEL}` | `--<component>-path {MODEL}` | Load the replacement component's configuration and weights |
| Replace only its weights | `--component-weights-paths.<component> {WEIGHTS}` | `--<component>-weights-path {WEIGHTS}` | Retain the base component configuration and replace its weights |
| Select component precision | `--component-precisions.<component> {DTYPE}` | — | Use a loader-supported exact parameter and execution dtype |
| Direct-load an eligible component | `--component-direct-gpu-weight-loading.<component>` | None | Use that component's audited direct-GPU loader; it must stay resident |
| Quantize an unquantized component online | `--component-quantizations.<component> {METHOD}` | `--<component>-quantization {METHOD}` | Apply a method supported by that component's native loader |
| Keep selected component layers unquantized | `--component-quantization-ignored-layers.<component> {PATTERN...}` | None | Pass component-local ignored-layer patterns to its online quantizer |

For example, pair any replacement text-encoder configuration with a separate
single-file checkpoint as follows:

```bash theme={null}
--component-paths.text_encoder COMPONENT_REPO_OR_DIRECTORY \
--component-weights-paths.text_encoder WEIGHTS_FILE_OR_REPO_FILE
```

The transformer-specific `--transformer-weights-path` spelling remains
supported for the primary DiT. Do not mechanically replace it with
`--component-weights-paths.transformer`: the component-scoped form requires an
actual component name, which is pipeline-specific. Use `--quantization` only
to override the method inferred by the transformer loader, and
`--quantization-ignored-layers` to keep matching transformer layers
unquantized during online quantization.

For a native text encoder:

* `--component-paths.text_encoder {MODEL}` replaces the text-encoder checkpoint; `--text-encoder-path {MODEL}` is its shorter alias
* `--component-quantizations.text_encoder {METHOD}` applies supported online quantization to an unquantized native encoder; pair it with `--component-quantization-ignored-layers.text_encoder {PATTERN...}` when selected layers must remain unquantized
* Quantization metadata is auto-detected from that checkpoint. The native loader accepts compatible serialized formats without a model-name allowlist and rejects implementations that do not construct the required quantized layers.

The same contract applies to every weighted component: path routing is generic,
while quantized materialization is capability-based. Native auxiliary loaders
whose current materializer expects plain state dicts reject unsupported
quantization metadata before model construction. See
[Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories)
for the current component matrix. A model cookbook is the source of truth for
published, model-specific checkpoint examples; for example, all H3 sources and
their exact overlays are kept in one
[MiniMax-H3 compatibility table](/cookbook/diffusion/MiniMax/MiniMax-H3#checkpoint-and-adapter-formats).

Exact precision overrides are capability-based. Native text and image encoders,
standard VAE components, and native plain-state components support them;
other component loaders reject the option instead of accepting a dtype that
their execution stage would not honor.

Direct-GPU loading is also capability-based. The existing
`--direct-gpu-weight-loading` remains the primary DiT path. The component form
currently supports standard native `vae` and `video_vae` state dicts on CUDA:
it streams each safetensors tensor directly to the resident
module, rather than materializing a complete CPU state dict. It rejects custom
Diffusers `auto_map` classes, quantized checkpoints, tied state entries, and
any component-offload or layerwise-offload placement.

Component overrides accept a local component directory, a standalone Hub
repository, or a Hub component subfolder written as `owner/repo/subfolder`.
For transformer and native encoder loaders, an explicit weight filename keeps
the base component configuration and replaces only its weights; the selected
file format must still be supported by that loader.

For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}`
compresses completed KV-cache chunks independently of transformer weight
quantization. It is lossy and disabled by default.

See [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models) for the runtime and
model scope, and [Quantization](/docs/sglang-diffusion/quantization) for supported quantization
families and examples.

### Request logging

* `--log-requests`: Log user-facing fields of all requests (default: `False`). The verbosity is decided by `--log-requests-level`.
* `--log-requests-level &#123;0|1|2|3&#125;`: Verbosity level for request logging (default: `2`). 0: Log metadata (request id). 1: Log metadata and sampling config (seed, steps, guidance, resolution, frames, fps, ...). 2: Log metadata, sampling config and prompt (truncated to 2 KiB). 3: Log metadata, sampling config and full prompt.
* `--log-requests-format &#123;text|json&#125;`: Format for request logging (default: `text`). `text` is human-readable; `json` outputs structured JSON lines.
* `--log-requests-target &#123;TARGET...&#125;`: Target(s) for request logging. Use `stdout` for console output and/or directory path(s) for file output. Can specify multiple targets, e.g., `--log-requests-target stdout /my/log/dir`.

## Configuration Files

Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file.

```bash Command theme={null}
sglang generate --config config.yaml
```

Example:

```yaml Config theme={null}
model_path: FastVideo/FastHunyuan-diffusers
prompt: A beautiful woman in a red dress walking down a street
output_path: outputs/
num_gpus: 2
sp_degree: 2
tp_size: 1
num_frames: 45
height: 720
width: 1280
num_inference_steps: 6
seed: 1024
fps: 24
precision: bf16
vae_precision: fp16
vae_tiling: true
vae_sp: true
enable_torch_compile: false
```

HunyuanVideo and FastHunyuan use tiled VAE decode by default so multi-GPU runs
distribute VAE tiles instead of selecting spatial-shard decode. At
HunyuanVideo's supported 960×544×77 shape, spatial-shard decode can consume
99.8 GiB per rank before requesting another 49.61 GiB causal mask. At
FastHunyuan's default 1280×720×125 shape, the mask alone would require
197.75 GiB. You can still override the policy with
`--vae-config.parallel-decode-mode`, but `spatial` and `spatial_shard` should
only be used for smaller validated shapes.

## Generate

`sglang generate` runs a single generation job and exits when the job finishes.

```bash Command theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --prompt "A curious raccoon" \
  --save-output \
  --output-path outputs \
  --output-file-name "a-curious-raccoon.mp4"
```

> [!NOTE]
>
> HTTP server-only arguments are ignored by `sglang generate`.

For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. It can run with DiT layerwise offload; it cannot run with FSDP. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](/docs/sglang-diffusion/cache_dit).

Native conditioning reuse is enabled by default, independently of Cache-DiT:

* `--disable-conditioning-cache`: disable cross-request text/image conditioning reuse and VAE posterior caching. Reuse within a grouped stage remains enabled.
* `--conditioning-cache-max-size-mb {MiB}`: shared CPU and device LRU capacity per executor/rank, default `512`; `0` disables cross-request caching. Consumed negative text conditioning stays on its original device; other entries live on the CPU. The cache holds at most 128 entries and bypasses oversized outputs.

See [conditioning cache](/docs/sglang-diffusion/caching-acceleration#conditioning-cache) for model coverage, invalidation, and the distinction between cold and repeated-input latency.

For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures.

LongCat-Image is supported by this path. Its DiT always consumes a fixed
512-token prompt body, so different prompt lengths reuse the same graph without
model-specific padding. For a 1024x1024 deployment, use:

```bash Command theme={null}
sglang serve --model-path meituan-longcat/LongCat-Image \
  --enable-breakable-cuda-graph \
  --warmup-resolutions 1024x1024 \
  --enable-torch-compile false \
  --port 30010
```

SANA-Video supports the same path for fixed-shape serving. Its default text
stage emits a fixed 300-token prompt shape, so the runtime reuses one graph
across prompt lengths without padding it to a generic text bucket. BCG also
requires the serving frame count to match warmup; the example captures the
model's default 81-frame signature:

```bash Command theme={null}
sglang serve \
  --model-path Efficient-Large-Model/SANA-Video_2B_480p_diffusers \
  --enable-breakable-cuda-graph \
  --warmup-resolutions 832x480 \
  --enable-torch-compile false \
  --port 30010
```

### Component Residency

Use `--component-residency COMPONENT=MODE` to assign one runtime residency mode to each native pipeline component:

```bash Command theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --component-residency all=resident text_encoder=layerwise-offload vae=component-offload \
  --prompt "A quiet city street after rain"
```

The available modes are:

* `resident`: keep the complete component on the accelerator.
* `component-offload`: keep the complete component on CPU between uses, moving it to the accelerator before each declared use and back to CPU afterward.
* `snapshot-offload`: retain CPU parameter storage while a complete component runs on GPU. At the end of a use, restore the CPU parameters instead of copying GPU weights back. Mutable buffers still move back to CPU.
* `layerwise-offload`: keep component weights on CPU and stream its declared layers during execution.

Selectors match exact loaded component keys from `model_index.json`, including names such as `transformer_2`, `audio_vae`, and `connectors`. The group selectors `dit`, `text_encoder`, `image_encoder`, and `vae` are also available, together with `all`. An exact key overrides a matching group, and a group overrides `all`. Components without a matching canonical selector retain their explicit legacy setting or automatic/model default.

The existing `--dit-cpu-offload`, `--text-encoder-cpu-offload`, `--image-encoder-cpu-offload`, `--vae-cpu-offload`, and `--cpu-offload-components` options remain supported. New and legacy options may be mixed: `--component-residency` wins only for components it matches, while unmatched legacy settings remain effective. Legacy layerwise selectors take precedence over legacy component-offload selectors for the same component. Explicit `--dit-layerwise-offload false` makes the DiT resident unless another explicit DiT selector, such as `--dit-cpu-offload true` or `--component-residency dit=component-offload`, selects a different mode.

Explicit component placement also applies during warmup and request cleanup. Pipeline
preferences cannot keep or reload an explicitly offloaded component merely to
prepare for the next request. An explicitly `resident` component remains resident.
This applies to matching exact, group, and `all` selectors, as well as legacy
offload options. Components without an explicit placement may still use the
pipeline's warmup retention and request-end preload preferences. Selecting one component does
not disable those preferences for other components. On-demand loading and
prefetch for upcoming uses within a request are unchanged; explicit offload may
therefore add weight-transfer latency to the first real request after warmup.

Layerwise selection is strict. A native weighted component selected for `layerwise-offload` must declare its layer structure; otherwise startup fails with the unsupported component name instead of silently changing modes. Explicit non-resident placement also requires a request-time component-use declaration, so it cannot silently select a module that the pipeline does not manage. FSDP applies only to resident components. The Diffusers backend supports only pipeline-wide `all=resident` and `all=component-offload`.

#### Snapshot offload

On NVIDIA CUDA GPUs with separate host and device memory, explicitly select
`--component-residency vae=snapshot-offload` to avoid repeated weight D2H transfers.
Existing defaults and `component-offload` behavior are unchanged.

Snapshot offload retains a full host copy during GPU execution. With
`--pin-cpu-memory` enabled (the default), it shares a per-worker host pin budget
with layerwise offload. Layerwise initialization claims its allowance first;
snapshot weights use the remainder. Shared parameter storage is pinned once and
reused across requests; its allowance is returned when the storage is released.
Pinning also checks current host headroom, including the cgroup limit.

Weights that do not fit the budget, or have pinning disabled, retain their
existing CPU storage. Checkpoint mmap stays file-backed unless pinning, dtype
conversion, or weight mutation materializes a copy. Pageable H2D can be slower,
and pinning temporarily needs space for both the original and pinned storage.
This budget limits new pinned weight allocations, not total process RAM or the
CUDA host allocator's cache. Benchmark repeated requests with the actual host
storage and workload; avoiding D2H does not guarantee an end-to-end speedup.

The whole component must still fit on GPU. Unlike layerwise offload with all layers
retained across denoising steps, snapshot offload does not require a layer-structure
declaration and also covers parameters outside transformer blocks. Weight updates,
LoRA merge/unmerge, and sleep restore the CPU weights before mutation or release.
FSDP-managed components, shared-memory GPUs, and snapshot-offloaded DiTs with
breakable CUDA graphs are not supported.

### Layerwise Offload Tuning

Use layerwise offload when a component does not fit comfortably in GPU memory. The compatibility options `--dit-layerwise-offload` and `--layerwise-offload-components` remain available (`--layerwise-offload-modules` is an alias), while new deployments can select the mode directly:

```bash Command theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --component-residency transformer=layerwise-offload text_encoder=layerwise-offload \
  --dit-offload-prefetch-size 0 \
  --prompt "A quiet city street after rain"
```

Values passed to the compatibility option `--layerwise-offload-components` must match loaded component keys, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Its `default` group selects text encoders, image encoders, and VAEs. Use `all` to select every layerwise-offloadable component.

Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, `--dit-layerwise-residency-policy`, and `--dit-layerwise-residency-lifetime` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency. DiT layerwise offload can run with Cache-DiT: skipped blocks are not streamed, and the first layer after a skip may sync-load. Cache-DiT remains incompatible with FSDP.

Those four set the default for every streamed component. To give one component its own values, use the `component=value` forms, which also accept JSON:

```bash theme={null}
sglang serve --model-path <MODEL> \
  --layerwise-offload-components dit,text_encoder \
  --layerwise-prefetch-size text_encoder=2 \
  --layerwise-resident-layers transformer=4 \
  --layerwise-residency-policy transformer=strided \
  --layerwise-residency-lifetime transformer=permanent
```

The `component=value` maps take the loaded component's name — `transformer`, `text_encoder`, `video_vae` — not a group alias: `--layerwise-offload-components` understands `dit`, but `--layerwise-resident-layers dit=4` matches no component and is silently ignored.

* `--layerwise-prefetch-size`: how many layers to fetch ahead. Fractional values are a share of the stack, `>= 1` an absolute count. Deeper prefetch overlaps more of the transfer with compute, at the cost of staging buffers.
* `--layerwise-resident-layers`: how many layers stay on the GPU instead of being streamed. Fractional values are a share of the stack. Resident layers pay for a component that runs its layers many times per request — a DiT across the denoise steps, a video VAE across the latent chunks — and do nothing for one that runs them once per request, such as a text encoder.
* `--layerwise-residency-policy`: `leading` keeps the first layers, `strided` spreads them across the stack so the transfers do not arrive as one burst.
* `--layerwise-residency-lifetime`: how long the resident layers stay. `forward` (default) places them when the component starts running its layers for a request and releases them when it finishes, so they are transferred once per request and keep a copy in host memory. `permanent` places them at load time, keeps no host copy, and never releases them: one transfer for the life of the server.

A component without an entry keeps the group default, so adding these options changes nothing until one is set. All of these trade VRAM for transfer. The lifetime decides *whose* VRAM: with `forward`, only the component that is running holds its resident layers, so the peak is the largest single component's set; with `permanent`, every component's resident layers are on the GPU at once, so the peak is their sum. Pick `permanent` when that sum fits alongside the activations — it also frees the pinned host memory the copies would have held — and `forward` when it does not. Measure before raising any of them.

## Serve

`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.

```bash Command theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --text-encoder-cpu-offload \
  --pin-cpu-memory \
  --num-gpus 4 \
  --ulysses-degree 2 \
  --ring-degree 2 \
  --port 30010
```

### Asynchronous output saving

Use `--async-output-save` to overlap output saving with the next request during
native NVIDIA CUDA serving. It is disabled by default. See
[Asynchronous output saving](/docs/sglang-diffusion/performance-optimization#asynchronous-output-saving)
for an example, eligible requests, and the response latency and memory tradeoffs.

### Graceful shutdown

Set `--http-graceful-shutdown-timeout 30` to allow up to 30 seconds for open HTTP
connections and request tasks to finish after shutdown begins. At the deadline,
unfinished HTTP tasks are cancelled, including uploads that have not sent their
complete request body. The default is unset (wait indefinitely); `0` cancels
outstanding HTTP tasks without waiting.

This controls HTTP draining only, not the full process shutdown time or an
inference deadline. Application cleanup and GPU worker teardown can take longer;
allow for them in your orchestrator's termination grace period. It does not make
asynchronous video jobs durable or guarantee that queued jobs finish before
shutdown. Use `--scheduler-rpc-timeout` separately to bound scheduler reply waits.

### Health endpoints

SGLang Diffusion separates process liveness from inference readiness:

| Endpoint | Success condition | Recommended use |
| - | - | - |
| `GET /liveness` | The HTTP server is accepting requests. It remains `200` during server warmup. | Kubernetes liveness probe |
| `GET /health` | Server-based synthetic warmup has completed, tracked local workers are alive, and the local broker and any disaggregation routing loop are running. Returns `503` otherwise. | Startup and readiness probes |
| `GET /health_generate` | Compatibility alias for `/health`. It does not currently issue a generation request in SGLang Diffusion. | Existing integrations only |

With `--warmup-mode off` or `--warmup-mode request`, `/health` returns `200`
once the HTTP server starts, provided its local backend is running; those modes
do not promise that compilation or other first-request work has completed. If
server-based warmup fails, the server terminates instead of reporting ready.

For standard serving (including `--webui`) and locally launched disaggregated
pools, losing any tracked worker makes `/health` return `503`; `/liveness`
continues to report HTTP availability. The same applies when the local offline
broker or disaggregation routing loop exits, even if its process is still alive.
The broker replies to malformed jobs and inference errors without stopping;
an unrecoverable socket receive or send error stops it and makes the server unready.
A local monitor checks these components once per
second, fails outstanding scheduler requests, and rejects new POST requests with
`503`, even without an RPC timeout. Completed job status and content remain
readable until the server stops. Remove the instance from routing and restart
the whole server; workers and in-flight jobs are not automatically recovered.

An allocator OOM during a multi-rank forward terminates the affected worker:
other ranks may still be inside collectives, so continuing the next request is
unsafe. Single-GPU replicas, including independent DP replicas, retain normal
request-level OOM handling. The monitor does not detect arbitrary hangs, unusable
CUDA contexts, or workers on other machines. A standalone disaggregation router
does not own remote worker processes and cannot validate their health through this
check; it only validates its own routing loop and broker.

Do not use `/health` as a liveness probe: a long server warmup can legitimately
keep it at `503` for several minutes.

### Cloud Storage

SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation.

```bash Command theme={null}
export SGLANG_CLOUD_STORAGE_TYPE=s3
export SGLANG_S3_BUCKET_NAME=my-bucket
export SGLANG_S3_ACCESS_KEY_ID=your-access-key
export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
```

See [Environment Variables](/docs/sglang-diffusion/environment_variables) for the full set of storage options.

Cancellation waits for active uploads to finish before request cleanup can remove
their source files. Allow for the storage SDK's timeouts and retries in your
shutdown grace period. Cancellation does not roll back objects already uploaded;
use your bucket's lifecycle rules for object retention.

## Component Path Overrides

Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `--<component>-path`.

```bash Command theme={null}
sglang serve \
  --model-path black-forest-labs/FLUX.2-dev \
  --vae-path fal/FLUX.2-Tiny-AutoEncoder
```

The component key must match a key in the model's `model_index.json` or a native
pipeline's registered module name. The path must be either a Hugging Face repo
ID or a complete component directory.

Path selection and quantized checkpoint support are separate capabilities. A
pre-quantized component repo must carry its quantization metadata, and the
selected loader must support that serialized format. Native plain-state loaders
fail closed; library-managed components inherit the corresponding Transformers
or Diffusers support. The transformer-specific `--quantization` flag does not
select the format of component checkpoints; their own metadata does. See
[Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories).

## Component Attention Backend Overrides

Use `--component-attention-backends` when one pipeline component needs a different native attention backend from the global `--attention-backend`.

```bash Command theme={null}
sglang generate \
  --model-path Lightricks/LTX-2.3 \
  --attention-backend fa \
  --component-attention-backends text_encoder=torch_sdpa
```

The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` while that component is being constructed and fail if the component cannot satisfy them. A native component may explicitly defer backend selection until first use; components with fixed attention reject the override. Sparse self-attention backends use a compatible dense backend for cross-attention layers. The global backend remains strict for DiT components, while auxiliary components may fall back to a compatible backend. The Diffusers backend supports only the global backend passthrough.

You can also pass dotted CLI entries:

```bash Command theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --component-attention-backends.text_encoder torch_sdpa \
  --component-attention-backends.transformer fa
```

A component key may end in `.self` or `.cross` to override one kind of attention
only. This helps when a backend's per-call overhead pays off for self-attention
but not for the lighter cross-attention:

```bash Command theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --component-attention-backends.transformer.self sage_attn \
  --component-attention-backends.transformer.cross fa
```

A role override outranks the component-wide backend for layers of that role and
is equally strict: it fails the load if it never reaches a layer of that role.
See [Attention Backends](/docs/sglang-diffusion/attention_backends#per-role-overrides).

## Diffusers Backend

Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class.

### Key Options

<table>
  <thead>
    <tr>
      <th>Argument</th>
      <th>Values</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--backend`</td>
      <td>`auto`, `sglang`, `diffusers`</td>
      <td>Choose native SGLang, force native, or force diffusers</td>
    </tr>

    <tr>
      <td>`--attention-backend`</td>
      <td>`flash`, `_flash_3_hub`, `sage`, `xformers`, `native`</td>
      <td>Attention backend for diffusers pipelines</td>
    </tr>

    <tr>
      <td>`--trust-remote-code`</td>
      <td>flag</td>
      <td>Required for models with custom pipeline classes</td>
    </tr>

    <tr>
      <td>`--vae-tiling` and `--vae-slicing`</td>
      <td>flag</td>
      <td>Lower memory usage for VAE decode</td>
    </tr>

    <tr>
      <td>`--dit-precision` and `--vae-precision`</td>
      <td>`fp16`, `bf16`, `fp32`</td>
      <td>Precision controls</td>
    </tr>

    <tr>
      <td>`--enable-torch-compile`</td>
      <td>flag</td>
      <td>Enable `torch.compile`</td>
    </tr>

    <tr>
      <td>`--cache-dit-config`</td>

      <td />

      <td>Cache-DiT config for diffusers pipelines</td>
    </tr>
  </tbody>
</table>

### Example

```bash theme={null}
sglang generate \
  --model-path AIDC-AI/Ovis-Image-7B \
  --backend diffusers \
  --trust-remote-code \
  --attention-backend flash \
  --prompt "A serene Japanese garden with cherry blossoms" \
  --height 1024 \
  --width 1024 \
  --num-inference-steps 30 \
  --save-output \
  --output-path outputs \
  --output-file-name ovis_garden.png
```

For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file.


# OpenAI API
Source: https://docs.sglang.io/docs/sglang-diffusion/api/openai_api

Image and video generation endpoints with LoRA adapter management.

The SGLang diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as LoRA adapter management.

## Prerequisites

* Python 3.11+ if you plan to use the OpenAI Python SDK.

## Serve

Launch the server using the `sglang serve` command.

### Start the server

```bash theme={null}
SERVER_ARGS=(
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
  --served-model-name wan-t2v
  --text-encoder-cpu-offload
  --pin-cpu-memory
  --num-gpus 4
  --ulysses-degree=2
  --ring-degree=2
  --port 30010
)

sglang serve "${SERVER_ARGS[@]}"
```

* **--model-path**: Path to the model or model ID.
* **--served-model-name**: Stable model name exposed by the serving APIs. It defaults to `--model-id` when set, otherwise `--model-path`.
* **--port**: HTTP port to listen on (default: `30000`).

### Served model name

`--served-model-name` separates the public API identity from the checkpoint location. This is useful when replicas use different local mount paths or when a gateway needs one stable model name:

```bash theme={null}
sglang serve \
  --model-path /models/Wan2.1-T2V-1.3B-Diffusers \
  --served-model-name wan-t2v \
  --port 30010
```

`--model-id` is not a free-form deployment alias: it selects the registered model configuration for checkpoints whose local path cannot be identified. `--served-model-name` only controls the name exposed by serving APIs. When both are set, the served name takes precedence for API responses.

### Discover the served model

**Endpoint:** `GET /v1/models`

Returns the public model name together with diffusion-specific runtime information.

**Curl Example:**

```bash curl theme={null}
curl -sS "http://localhost:30010/v1/models"
```

**Response Example:**

```json theme={null}
{
  "object": "list",
  "data": [
    {
      "id": "wan-t2v",
      "object": "model",
      "created": 1786348800,
      "owned_by": "sglang",
      "root": "wan-t2v",
      "parent": null,
      "max_model_len": null,
      "num_gpus": 4,
      "task_type": "T2V",
      "supported_task_types": ["T2V"],
      "dit_precision": "bf16",
      "vae_precision": "fp16",
      "pipeline_name": "WanPipeline",
      "pipeline_class": "WanPipeline"
    }
  ]
}
```

Retrieve the same model by its served name:

```bash curl theme={null}
curl -sS "http://localhost:30010/v1/models/wan-t2v"
```

`GET /server_info` also reports `served_model_name` for gateway discovery. Video and action responses use this name when the request does not provide a model explicitly.

***

## Endpoints

### Select a request task

`GET /v1/models` reports `task_type` (the configured default) and
`supported_task_types` (the tasks implemented by that configured pipeline).
`GET /model_info` also reports the supported `output_types`. A pipeline may
support image and video output on one server when its implementation and
loaded components provide both paths.

Send the optional SGLang extension `task_type` as a case-insensitive enum name:

| Interface | Task field | Output constraint |
| - | - | - |
| `POST /v1/images/generations` JSON | `"task_type": "t2i"` | Image |
| `POST /v1/images/edits` multipart | `-F task_type=i2i` | Image |
| `POST /v1/videos` JSON or multipart | `"task_type": "i2v"` or `-F task_type=i2v` | Video |
| OpenAI Python client | `extra_body={"task_type": "t2i"}` | Selected endpoint |
| Native Python `generate` sampling parameters | `"task_type": "t2i"` | Selected task |
| `sglang generate` | `--task-type t2i` | Selected task |

Choose only a task in the server's advertised set. An unsupported task, a task
with the wrong output type for the endpoint, or missing required conditioning
returns HTTP 400. For example, `i2v` needs image input, while `v2v` and `f2v`
need video input. Model-specific size, frame count, and reference constraints
still apply.

If you omit `task_type`, the server uses the endpoint and supplied inputs to
choose a compatible task, preferring its default when compatible. It asks for
an explicit task if several alternatives remain (for example `v2v` versus
`f2v`). Pipelines with one task keep that task as their default. Sending a
request never changes the server default or subsequent requests.

### Prompt enhancement

Image generation, image edits and video creation accept `enhance_prompt=true`
when the server is configured with `--prompt-enhancer-config`. An independent
SRT service rewrites the prompt before diffusion runs. The default is `false`;
image responses return `data[].revised_prompt`, and video responses return
`revised_prompt`. See [deployment and model compatibility](/docs/sglang-diffusion/models_with_pe).

### Model-specific request fields

The image and video request schemas contain only OpenAI fields and stable
SGLang extensions shared across model families. Put model-specific controls in
`extra_body` when using the OpenAI Python client:

```python Python theme={null}
from openai import OpenAI

client = OpenAI(api_key="EMPTY", base_url="http://localhost:30010/v1")
response = client.images.generate(
    model="ideogram-ai/ideogram-4-nf4",
    prompt="A typeset poster",
    extra_body={"preset": "V4_QUALITY_48"},
)
```

Raw JSON requests may send these fields at the top level. Multipart video
requests may send individual declared form fields or one JSON object in
`extra_params`. SGLang forwards model-specific extensions only when the active
model declares them, so a field for one model family does not silently change
another model.

### Image Generation

The server implements an OpenAI-compatible Images API under the `/v1/images` namespace.

**Create an image**

**Endpoint:** `POST /v1/images/generations`

#### Request quality

`quality` selects a cumulative request-level optimization tier: `exact` keeps the selected deployment's reference path and all unconditional bit-exact replacements, so output is bit-identical to it in the same environment; `lossless` additionally enables the request-gated DiT/VAE fusions that keep the reference math and every operand's precision and only move the rounding; `high` includes the full `lossless` set and may also enable model-owned sparse, caching, lower-precision, or other approximate paths. `extra-high` is accepted as the former name of `lossless`. The tier does not override separately configured quantization, attention, or caching options. Omit it (or send OpenAI's default `auto`) to keep the `lossless` runtime default. It is distinct from `output_quality`, which controls only output-file compression. The same extension is accepted by image edits and video requests.

**Python Example (b64\_json response):**

```python Python theme={null}
import base64
from openai import OpenAI

client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")

img = client.images.generate(
    prompt="A calico cat playing a piano on stage",
    size="1024x1024",
    n=1,
    response_format="b64_json",
)

image_bytes = base64.b64decode(img.data[0].b64_json)
with open("output.png", "wb") as f:
    f.write(image_bytes)
```

**Curl Example:**

```bash curl theme={null}
curl -sS -X POST "http://localhost:30010/v1/images/generations" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -d '{
        "prompt": "A calico cat playing a piano on stage",
        "size": "1024x1024",
        "n": 1,
        "response_format": "b64_json"
      }'
```

> **Note**
> If `response_format=url` is used and cloud storage is not configured, the API returns
> a relative URL like `/v1/images/<IMAGE_ID>/content`.

**Edit an image**

**Endpoint:** `POST /v1/images/edits`

This endpoint accepts a multipart form upload with input images and a text prompt. The server can return either a base64-encoded image or a URL to download the image.

Uploaded filenames are sanitized, and saved uploads are constrained to the
server's upload directory. This also applies to image uploads for video and mesh
generation.

**Curl Example (b64\_json response):**

```bash Command theme={null}
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -F "image=@local_input_image.png" \
  -F "url=image_url.jpg" \
  -F "prompt=A calico cat playing a piano on stage" \
  -F "size=1024x1024" \
  -F "response_format=b64_json"
```

**Curl Example (URL response):**

```bash Command theme={null}
curl -sS -X POST "http://localhost:30010/v1/images/edits" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -F "image=@local_input_image.png" \
  -F "url=image_url.jpg" \
  -F "prompt=A calico cat playing a piano on stage" \
  -F "size=1024x1024" \
  -F "response_format=url"
```

**Download image content**

When `response_format=url` is used with `POST /v1/images/generations` or `POST /v1/images/edits`,
the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.

**Endpoint:** `GET /v1/images/&#123;image_id&#125;/content`

**Curl Example:**

```bash theme={null}
curl -sS -L "http://localhost:30010/v1/images/<IMAGE_ID>/content" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -o output.png
```

### Mesh generation (SGLang extension)

For a mesh-generation model, `POST /v1/meshes` accepts an input image and returns
an asynchronous job. This endpoint is a SGLang extension, not an OpenAI API.

```bash theme={null}
curl -sS http://localhost:30010/v1/meshes \
  -F 'image=@input.png' \
  -F 'output_format=glb'
```

Poll `GET /v1/meshes/{id}` until `status` is `completed` or `failed`. The content
endpoint, `GET /v1/meshes/{id}/content`, returns 404 until the job completes
successfully; an unfinished or failed export is not downloadable even if a
partial file exists. When cloud storage is enabled, use the completed job's
`url` instead.

Job metadata is process-local and does not survive server restart.
`DELETE /v1/meshes/{id}` removes the record, not the running generation or its
output files. Manage output retention separately.

### Video Generation

The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace.

**Create a video (text-to-video)**

**Endpoint:** `POST /v1/videos`

**Python Example:**

```python Python theme={null}
from openai import OpenAI

client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")

video = client.videos.create(
    prompt="A calico cat playing a piano on stage",
    size="1280x720"
)
print(f"Video ID: {video.id}, Status: {video.status}")
```

**Curl Example:**

```bash curl theme={null}
curl -sS -X POST "http://localhost:30010/v1/videos" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -d '{
        "prompt": "A calico cat playing a piano on stage",
        "size": "1280x720"
      }'
```

**Create a video (image-to-video)**

For I2V or TI2V models (e.g., Wan2.1 I2V, LTX-2.3 two-stage), pass an input image via multipart form upload or a reference URL.

**Curl Example (multipart form upload):**

```bash Command theme={null}
curl -sS -X POST "http://localhost:30010/v1/videos" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -F "prompt=A cat playing a piano" \
  -F "input_reference=@input_image.png" \
  -F "size=1280x720"
```

**Curl Example (reference URL):**

```bash Command theme={null}
curl -sS -X POST "http://localhost:30010/v1/videos" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -d '{
        "prompt": "A cat playing a piano",
        "reference_url": "https://example.com/input_image.png",
        "size": "1280x720"
      }'
```

> [!NOTE]
>
> `size` is the literal output resolution for most pipelines. Wan-Animate-2
> treats it as a pixel-area budget instead: the output keeps the reference image's
> aspect ratio and uses the largest frame with at most `width × height` pixels whose
> sides are multiples of 16, so `size=640x800` with a 9:16 reference image yields a
> 528×944 video. Request a budget with the reference image's aspect ratio to get a
> specific size. Wan-Animate-2 also ignores `seconds` and `num_frames`: the output
> has as many frames as the (resampled) reference video, generated in chunks of
> `clip_len` frames; see the
> [Wan-Animate-2 cookbook](/cookbook/diffusion/Wan/Wan-Animate-2#1-2-defaults).
> Wan-Animate-2 keeps the reference video's audio track on the output; the
> model-specific `enable_audio` field (form field or `extra_body`, default `true`)
> set to `false` gives a silent output.

#### MP4 encoding

`x264_preset` picks the libx264 preset for the returned MP4: `ultrafast`,
`superfast`, `veryfast`, `faster`, `fast` (the default), `medium`, `slow`,
`slower`, `veryslow` or `placebo`. Faster presets finish encoding sooner and
trade file size and fidelity at the same `output_compression`/`output_quality`.
Send it at the top level of a JSON body or as a multipart form field;
`sglang generate` takes `--x264-preset`.

**List videos**

**Endpoint:** `GET /v1/videos`

Returns up to 20 jobs by default. Set `limit` from 1 to 100 and `order` to
`asc` or `desc` (default). Responses include `first_id`, `last_id`, and
`has_more`; pass `after=last_id` to fetch the next page. The same pagination
contract applies to `GET /v1/meshes`. Pagination bounds each response, not
the number of stored jobs or queued generation requests.

**Python Example:**

```python Python theme={null}
for item in client.videos.list(limit=20):
    print(item.id, item.status)
```

**Curl Example:**

```bash curl theme={null}
curl -sS -X GET "http://localhost:30010/v1/videos" \
  -H "Authorization: Bearer sk-proj-1234567890"
```

**Download video content**

**Endpoint:** `GET /v1/videos/&#123;video_id&#125;/content`

**Python Example:**

```python Python theme={null}
import time

# Poll for completion
while True:
    item = client.videos.retrieve(video_id)
    if item.status == "completed":
        break
    if item.status == "failed":
        raise RuntimeError(item.error)
    time.sleep(5)

# Download content
resp = client.videos.download_content(video_id=video_id)
with open("output.mp4", "wb") as f:
    f.write(resp.read())
```

**Curl Example:**

```bash curl theme={null}
curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
  -H "Authorization: Bearer sk-proj-1234567890" \
  -o output.mp4
```

***

### LoRA Management

The server supports dynamic loading, merging, and unmerging of LoRA adapters.

**Important Notes:**

* Mutual Exclusion: Only one LoRA configuration can be active per target at a time
* Switching: To switch LoRAs, deactivate the current LoRA with `unmerge_lora_weights`, then `set` the new one
* Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost

**Set LoRA Adapter**

Loads one or more LoRA adapters and applies them to the model. By default, regular weights are statically merged, while FSDP-sharded weights use dynamic LoRA to avoid full-gather memory peaks.

**Endpoint:** `POST /v1/set_lora`

**Parameters:**

* `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs
* `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname`
* `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values:
  * `"all"` (default): Apply to all transformers
  * `"transformer"`: Apply only to the primary transformer (high noise for Wan2.2)
  * `"transformer_2"`: Apply only to transformer\_2 (low noise for Wan2.2)
  * `"critic"`: Apply only to the critic model
* `strength` (float or list of floats, optional): LoRA strength for merge, default 1.0. If a list, must match the length of `lora_nickname`. Values \< 1.0 reduce the effect, values > 1.0 amplify the effect
* `merge_mode` (string, optional): `"auto"` (default server policy), `"merge"` (force static merge), or `"dynamic"` (apply LoRA at forward time)

**Single LoRA Example:**

```bash Command theme={null}
curl -X POST http://localhost:30010/v1/set_lora \
  -H "Content-Type: application/json" \
  -d '{
        "lora_nickname": "lora_name",
        "lora_path": "/path/to/lora.safetensors",
        "target": "all",
        "strength": 0.8
      }'
```

**Multiple LoRA Example:**

```bash Command theme={null}
curl -X POST http://localhost:30010/v1/set_lora \
  -H "Content-Type: application/json" \
  -d '{
        "lora_nickname": ["lora_1", "lora_2"],
        "lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"],
        "target": ["transformer", "transformer_2"],
        "strength": [0.8, 1.0]
      }'
```

**Multiple LoRA with Same Target:**

```bash Command theme={null}
curl -X POST http://localhost:30010/v1/set_lora \
  -H "Content-Type: application/json" \
  -d '{
        "lora_nickname": ["style_lora", "character_lora"],
        "lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"],
        "target": "all",
        "strength": [0.7, 0.9]
      }'
```

> [!NOTE]
>
> When using multiple LoRAs:
>
> * All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length
> * If `target` or `strength` is a single value, it will be applied to all LoRAs
> * Multiple LoRAs applied to the same target are applied in order

**Merge LoRA Weights**

Manually merges the currently set LoRA weights into the base model.

> [!NOTE]
>
> With FSDP-sharded weights, manual merge may require a full-gather and can OOM. Use `set_lora` with `merge_mode="auto"` or `"dynamic"` for the lower-peak path.

**Endpoint:** `POST /v1/merge_lora_weights`

**Parameters:**

* `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer\_2", "critic"
* `strength` (float, optional): LoRA strength for merge, default 1.0. Values \< 1.0 reduce the effect, values > 1.0 amplify the effect

**Curl Example:**

```bash theme={null}
curl -X POST http://localhost:30010/v1/merge_lora_weights \
  -H "Content-Type: application/json" \
  -d '{"strength": 0.8}'
```

**Unmerge LoRA Weights**

Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA.

**Endpoint:** `POST /v1/unmerge_lora_weights`

**Curl Example:**

```bash theme={null}
curl -X POST http://localhost:30010/v1/unmerge_lora_weights \
  -H "Content-Type: application/json"
```

**List LoRA Adapters**

Returns loaded LoRA adapters and current application status per module.

**Endpoint:** `GET /v1/list_loras`

**Curl Example:**

```bash theme={null}
curl -sS -X GET "http://localhost:30010/v1/list_loras"
```

**Response Example:**

```json theme={null}
{
  "loaded_adapters": [
    { "nickname": "lora_a", "path": "/weights/lora_a.safetensors" },
    { "nickname": "lora_b", "path": "/weights/lora_b.safetensors" }
  ],
  "active": {
    "transformer": [
      {
        "nickname": "lora2",
        "path": "tarn59/pixel_art_style_lora_z_image_turbo",
        "merged": true,
        "mode": "merged",
        "strength": 1.0
      }
    ]
  }
}
```

Notes:

* If LoRA is not enabled for the current pipeline, the server will return an error.
* `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter.

### Example: Switching LoRAs

1. Set LoRA A:
   ```bash Command theme={null}
   curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}'
   ```
2. Generate with LoRA A...
3. Unmerge LoRA A:
   ```bash Command theme={null}
   curl -X POST http://localhost:30010/v1/unmerge_lora_weights
   ```
4. Set LoRA B:
   ```bash Command theme={null}
   curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}'
   ```
5. Generate with LoRA B...

### Adjust Output Quality

The server supports adjusting output quality and compression levels for both image and video generation through the `output-quality` and `output-compression` parameters.

#### Parameters

* **`output-quality`** (string, optional): Preset quality level that automatically sets compression. **Default is `"default"`**. Valid values:
  * `"maximum"`: Highest quality (100)
  * `"high"`: High quality (90)
  * `"medium"`: Medium quality (55)
  * `"low"`: Lower quality (35)
  * `"default"`: Auto-adjust based on media type (50 for video, 75 for image)

* **`output-compression`** (integer, optional): Direct compression level override (0-100). **Default is `None`**. When provided (not `None`), takes precedence over `output-quality`.
  * `0`: Lowest quality, smallest file size
  * `100`: Highest quality, largest file size

#### Notes

* **Precedence**: When both `output-quality` and `output-compression` are provided, `output-compression` takes precedence
* **Format Support**: Quality settings apply to JPEG, and video formats. PNG is lossless: by default it is stored without zlib compression, which encodes fastest but keeps files close to the raw pixel size; an `output-compression` other than 75 selects zlib level `round(output-compression / 100 * 9)` instead
* **File Size vs Quality**: Lower compression values (or "low" quality preset) produce smaller files but may show visible artifacts


# Post-Processing
Source: https://docs.sglang.io/docs/sglang-diffusion/api/post_processing



SGLang diffusion supports optional post-processing steps that run after
generation to improve temporal smoothness (frame interpolation) or spatial
resolution (upscaling). These steps are independent of the diffusion model and
can be combined in a single run.

When both are enabled, **frame interpolation runs first** (increasing the frame
count), then **upscaling runs on every frame** (increasing the spatial
resolution).

***

## Frame Interpolation (video only)

Frame interpolation synthesizes new frames between each pair of consecutive
generated frames, producing smoother motion without re-running the diffusion
model.

The `--frame-interpolation-exp` flag controls how many rounds of interpolation
to apply: each round inserts one new frame into every gap between adjacent
frames, so the output frame count follows the formula:

> **(N − 1) × 2^exp + 1**
>
> e.g. 5 original frames with `exp=1` → 4 gaps × 1 new frame + 5 originals = **9** frames;
> with `exp=2` → **17** frames.

### CLI Arguments

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-frame-interpolation`</td>
      <td>Enable frame interpolation. Model weights are downloaded automatically on first use.</td>
    </tr>

    <tr>
      <td>`--frame-interpolation-exp {EXP}`</td>
      <td>Interpolation exponent — `1` = 2× temporal resolution, `2` = 4×, etc. (default: `1`)</td>
    </tr>

    <tr>
      <td>`--frame-interpolation-scale {SCALE}`</td>
      <td>RIFE inference scale; use `0.5` for high-resolution inputs to save memory (default: `1.0`)</td>
    </tr>

    <tr>
      <td>`--frame-interpolation-model-path {PATH}`</td>
      <td>Local directory or HuggingFace repo ID containing RIFE `flownet.pkl` weights (default: `elfgum/RIFE-4.22.lite`, downloaded automatically)</td>
    </tr>
  </tbody>
</table>

### Supported Models

Frame interpolation uses the [RIFE](https://github.com/hzwer/Practical-RIFE)
(Real-Time Intermediate Flow Estimation) architecture. Only **RIFE 4.22.lite**
(`IFNet` with 4-scale `IFBlock` backbone) is supported. The network topology is
hard-coded, so custom weights provided via `--frame-interpolation-model-path`
must be a `flownet.pkl` checkpoint that is compatible with this architecture.

Other RIFE versions (e.g., older `v4.x` variants with different block counts)
or entirely different frame interpolation methods (FILM, AMT, etc.) are **not
supported**.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Weight</th>
      <th>HuggingFace Repo</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>RIFE 4.22.lite *(default)*</td>
      <td>[`elfgum/RIFE-4.22.lite`](https://huggingface.co/elfgum/RIFE-4.22.lite)</td>
      <td>Lightweight model, downloaded automatically on first use</td>
    </tr>
  </tbody>
</table>

### Example

Generate a 5-frame video and interpolate to 9 frames ((5 − 1) × 2¹ + 1 = 9):

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --prompt "A dog running through a park" \
  --num-frames 5 \
  --enable-frame-interpolation \
  --frame-interpolation-exp 1 \
  --save-output
```

***

## Upscaling (image and video)

Upscaling increases the spatial resolution of generated images or video frames
using [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN). The model weights
are downloaded automatically on first use and cached for subsequent runs.

### CLI Arguments

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Argument</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--enable-upscaling`</td>
      <td>Enable post-generation upscaling using Real-ESRGAN.</td>
    </tr>

    <tr>
      <td>`--upscaling-scale {SCALE}`</td>
      <td>Desired upscaling factor (default: `4`). The 4× model is used internally; if a different scale is requested, a bicubic resize is applied after the network output.</td>
    </tr>

    <tr>
      <td>`--upscaling-model-path {PATH}`</td>
      <td>Local `.pth` file, HuggingFace repo ID, or `repo_id:filename` for Real-ESRGAN weights (default: `ai-forever/Real-ESRGAN` with `RealESRGAN_x4.pth`, downloaded automatically). Use the `repo_id:filename` format to specify a custom weight file from a HuggingFace repo (e.g. `my-org/my-esrgan:weights.pth`).</td>
    </tr>
  </tbody>
</table>

### Supported Models

Upscaling supports two Real-ESRGAN network architectures. The correct
architecture is **auto-detected** from the checkpoint keys, so you only need to
point `--upscaling-model-path` at a valid `.pth` file:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Architecture</th>
      <th>Example Weights</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**RRDBNet**</td>
      <td>`RealESRGAN_x4plus.pth`</td>
      <td>Heavier model with higher quality; best for photos</td>
    </tr>

    <tr>
      <td>**SRVGGNetCompact**</td>
      <td>`RealESRGAN_x4.pth` *(default)*, `realesr-animevideov3.pth`, `realesr-general-x4v3.pth`</td>
      <td>Lightweight model; faster inference, good for video</td>
    </tr>
  </tbody>
</table>

The default weight file is
[`ai-forever/Real-ESRGAN`](https://huggingface.co/ai-forever/Real-ESRGAN) with
`RealESRGAN_x4.pth` (SRVGGNetCompact, 4× native scale).

Other super-resolution models (e.g., SwinIR, HAT, BSRGAN) are **not supported**
— only Real-ESRGAN checkpoints using the two architectures above are
compatible.

### Examples

Generate a 1024×1024 image and upscale to 4096×4096:

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --prompt "A cat sitting on a windowsill" \
  --output-size 1024x1024 \
  --enable-upscaling \
  --save-output
```

Generate a video and upscale each frame by 4×:

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --prompt "A curious raccoon" \
  --enable-upscaling \
  --upscaling-scale 4 \
  --save-output
```

***

## Combining Frame Interpolation and Upscaling

Frame interpolation and upscaling can be combined in a single run.
Interpolation is applied first (increasing the frame count), then upscaling is
applied to every frame (increasing the spatial resolution).

Example — generate 5 frames, interpolate to 9 frames, and upscale each frame
by 4×:

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --prompt "A curious raccoon" \
  --num-frames 5 \
  --enable-frame-interpolation \
  --frame-interpolation-exp 1 \
  --enable-upscaling \
  --upscaling-scale 4 \
  --save-output
```


# Attention Backends
Source: https://docs.sglang.io/docs/sglang-diffusion/attention_backends

Select and configure attention backends for SGLang diffusion pipelines.

This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them.

## Overview

Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`.

Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`). `--attention-backend` is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use `--component-attention-backends` when an auxiliary component must use a specific backend; incompatible overrides, including an override for a component that constructs no SGLang attention layer, fail unless a sparse backend is being replaced for cross-attention. A component key may carry a `.self` or `.cross` role suffix to override just one kind of attention; see [Per-role overrides](#per-role-overrides).

When using the diffusers backend, `--attention-backend` is passed through to diffusers'
`set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`).

* **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel.
* **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
* **Intel XPU**: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA.
* **MUSA**: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA.
* **MPS**: always uses PyTorch SDPA.
* **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA.

## Backend options

For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>CLI value</th>
      <th>Enum value</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`fa` / `fa3` / `fa4`</td>
      <td>`FA`</td>
      <td>FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.post_init`).</td>
    </tr>

    <tr>
      <td>`torch_sdpa`</td>
      <td>`TORCH_SDPA`</td>
      <td>PyTorch `scaled_dot_product_attention`.</td>
    </tr>

    <tr>
      <td>`sliding_tile_attn`</td>
      <td>`SLIDING_TILE_ATTN`</td>
      <td>Sliding Tile Attention (STA). Requires `st_attn`. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`sage_attn`</td>
      <td>`SAGE_ATTN`</td>
      <td>Requires `sageattention`. On Hopper (SM90), PyPI `sageattention==2.2.0` is unsupported because it lacks the upstream SM90 binding fix. Install `pip install --force-reinstall git+https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4 --no-build-isolation`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120; see upstream `setup.py`.</td>
    </tr>

    <tr>
      <td>`sage_attn_3`</td>
      <td>`SAGE_ATTN_3`</td>
      <td>Requires SageAttention3 installed per upstream instructions.</td>
    </tr>

    <tr>
      <td>`sparge_attn`</td>
      <td>`SPARGE_ATTN`</td>
      <td>Training-free sparse SageAttention2. CUDA SM80/86/87/89/90, FP16/BF16, head dim 64/128, and square self-attention with sequence length at least 128. Other attention shapes use dense SDPA. Install `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation`; configure retained block ratio with `topk`.</td>
    </tr>

    <tr>
      <td>`sol_attn`</td>
      <td>`SOL_ATTN`</td>
      <td>Requires the upstream `sol-attn` package. Install with `pip install git+https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends`. BF16, head dim 128. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`video_sparse_attn`</td>
      <td>`VIDEO_SPARSE_ATTN`</td>
      <td>Requires `vsa`. Configure `sparsity` via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`subblock_sparse_attn`</td>
      <td>`SUBBLOCK_SPARSE_ATTN`</td>
      <td>Training-free SubBlock sparsity for MiniMax-H3 on CUDA SM90, SM100, and SM120. The default compute mode is BF16; `compute_mode=sage_fp8` is an approximate SM90-only path that requires `SpargeAttn`. See the [MiniMax-H3 recipe](/cookbook/diffusion/MiniMax/MiniMax-H3#subblock-sparse-attention).</td>
    </tr>

    <tr>
      <td>`video_sparse_attn_h3`</td>
      <td>`VIDEO_SPARSE_ATTN_H3`</td>
      <td>Video Sparse Attention for MiniMax-H3 / FastH3 (VSA-H3). In-tree kernels, no external package: FastVideo's native tcgen05 block-sparse forward on SM100 / SM103, the Triton tile-64 kernel on SM90. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`hybrid_window_attn_h3`</td>
      <td>`HYBRID_WINDOW_ATTN_H3`</td>
      <td>VDN-H3 hybrid attention: chunk-aligned window softmax (exact, gated) plus the Video Delta linear branch over the window complement. The window runs as a union of FlashAttention varlen calls (FA4 on SM100 / SM103 / SM120, FA3 on SM90; on SM80 / SM86 / SM89 the same FA3 build runs its Sm80 mainloop at FA2-class speed). Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`vmoba_attn`</td>
      <td>`VMOBA_ATTN`</td>
      <td>Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`aiter`</td>
      <td>`AITER`</td>
      <td>Requires `aiter`.</td>
    </tr>

    <tr>
      <td>`aiter_sage`</td>
      <td>`AITER_SAGE`</td>
      <td>Requires `aiter`.</td>
    </tr>

    <tr>
      <td>`sla_attn`</td>
      <td>`SLA_ATTN`</td>
      <td>Sparse Linear Attention. Requires `SpargeAttn`. Install with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation`.</td>
    </tr>

    <tr>
      <td>`sage_sla_attn`</td>
      <td>`SAGE_SLA_ATTN`</td>
      <td>SageAttention + Sparse Linear Attention. Requires `SpargeAttn` (same install as SLA).</td>
    </tr>

    <tr>
      <td>`sparse_video_gen_2_attn`</td>
      <td>`SPARSE_VIDEO_GEN_2_ATTN`</td>
      <td>Requires `svg`. See installation instructions at [https://github.com/svg-project/Sparse-VideoGen](https://github.com/svg-project/Sparse-VideoGen).</td>
    </tr>

    <tr>
      <td>`laser_attn`</td>
      <td>`LASER_ATTN`</td>
      <td>Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU.</td>
    </tr>

    <tr>
      <td>`block_sparse_attn`</td>
      <td>`BLOCK_SPARSE_ATTN`</td>
      <td>Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU.</td>
    </tr>

    <tr>
      <td>`rain_fusion_attn`</td>
      <td>`RAIN_FUSION_ATTN`</td>
      <td>Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU.</td>
    </tr>

    <tr>
      <td>`fp8_fa_sm120`</td>
      <td>`FP8_FA_SM120`</td>
      <td>FP8 (E4M3) dense attention for SM120 GPUs (GeForce RTX 50, RTX PRO Blackwell), written in CuTe-DSL. Q/K/V are quantized per head on the fly; the output stays BF16. Non-causal, batch 1, head dim 128, BF16 inputs; other calls use `torch_cudnn_sdpa`. Opt-in only: outputs differ from BF16 attention by about 5% relative RMS per call. Each new sequence length compiles once (about 10 s).</td>
    </tr>
  </tbody>
</table>

## Selection priority

The selection order in `runtime/layers/attention/selector.py` is:

1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)`
2. Component *and role* override from `--component-attention-backends.<component>.<role>` while that component is being constructed
3. Component override from `--component-attention-backends` while that component is being constructed
4. CLI `--attention-backend` (`ServerArgs.attention_backend`)
5. Layer or component default, when declared
6. Auto selection (platform capability, dtype, and installed packages)

An explicit global backend mismatch fails for DiT self-attention. Auxiliary
components may fall back to their declared default or another compatible backend.
Sparse backends selected for self-attention similarly fall back to a compatible
dense backend for cross-attention. Explicit component overrides are otherwise
strict.

### Per-role overrides

A role is the kind of attention a layer performs: `self` or `cross`. Appending
a role to a component key selects a backend for just that role, which is useful
when a quantized or otherwise expensive backend pays off for self-attention but
not for the much lighter cross-attention:

```bash Command theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --component-attention-backends.transformer.self sage_attn \
  --component-attention-backends.transformer.cross fa
```

A role override outranks the component-wide backend for layers of that role and
leaves the other role alone. Layers of a role with no override fall through to
the component-wide backend as usual, and `<component>_2` inherits
`<component>`'s role overrides unless it declares its own.

Role overrides are as strict as component-wide ones: a role override that never
reaches a layer fails the load, including when the component builds no layers of
that role at all. The one tolerated divergence is the shared sparse rule above —
a sparse backend named for `cross` is replaced by a compatible dense backend and
logged, rather than failing.

Model and layer backend sets guide automatic selection; they are not positive
allowlists for explicit requests. A backend selected with
`--attention-backend` or `--component-attention-backends` can therefore be used
without first adding it to every model's set. It still fails closed when the
platform cannot resolve it or when it lacks a required operation such as dense
cross-attention, packed variable-length attention, or ring-attention LSE.
Model paths that require one backend for correctness declare it as required;
those layers keep that backend even when the surrounding component is
overridden.

For an automatic, measurement-based choice, pass
`--enable-attention-backend-autotune true`. On each layer's first sufficiently
large input, the native runtime times compatible candidates and switches only
when the output remains within its numerical guard and the measured gain exceeds
the noise margin. The tuner is disabled by default and currently validated on
SM90 and SM12x. It does not replace a backend selected explicitly through
`--attention-backend` or `--component-attention-backends`, or a backend required
by the model for correctness.

## Configuration

Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts:

* A path to a JSON or YAML configuration file.
* A JSON string (e.g., `'&#123;"sparsity": 0.5&#125;'`).
* Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`).

### Supported Configuration Parameters

**Sliding Tile Attention (`sliding_tile_attn`)**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`mask_strategy_file_path`</td>
      <td>`str`</td>
      <td>**Required.** Path to the mask strategy JSON file.</td>
      <td>-</td>
    </tr>

    <tr>
      <td>`sta_mode`</td>
      <td>`str`</td>
      <td>Mode of STA.</td>
      <td>`STA_inference`</td>
    </tr>

    <tr>
      <td>`skip_time_steps`</td>
      <td>`int`</td>
      <td>Number of steps to use full attention before switching to sparse attention.</td>
      <td>`15`</td>
    </tr>
  </tbody>
</table>

**Video Sparse Attention (`video_sparse_attn`)**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`sparsity`</td>
      <td>`float`</td>
      <td>Validation sparsity (0.0 - 1.0).</td>
      <td>`0.0`</td>
    </tr>
  </tbody>
</table>

**SpargeAttention (`sparge_attn`)**

| Parameter | Type | Description | Default |
| - | - | - | - |
| `topk` | `float` | Fraction of predicted attention blocks retained per head. Higher values preserve more attention work and generally improve quality. Must be in `(0, 1]`. | `0.5` |

SpargeAttention is approximate even when `topk=1`: the recommended upstream
kernel quantizes attention through SageAttention2. Validate output quality and
end-to-end latency on the target model and resolution before deployment.

**Video Sparse Attention for H3 (`video_sparse_attn_h3`)**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`VSA_sparsity`</td>
      <td>`float`</td>
      <td>Fraction of video tiles excluded from the top-k selection (0.0 - 1.0). FastH3 defaults to its trained `0.8`.</td>
      <td>`0.8` (FastH3), `0.9` (base H3)</td>
    </tr>

    <tr>
      <td>`vsa_mode`</td>
      <td>`str`</td>
      <td>`exempt`: non-video keys (text/audio prefix tiles) are always selected. `compete`: they compete with video tiles in the top-k.</td>
      <td>`exempt`</td>
    </tr>

    <tr>
      <td>`vsa_dense_first_n_steps`</td>
      <td>`int`</td>
      <td>Use dense attention for the first N denoising steps.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`vsa_dense_layers`</td>
      <td>`list[int]`</td>
      <td>Layer indices kept dense, e.g. `[0, 1]`.</td>
      <td>`[]`</td>
    </tr>

    <tr>
      <td>`vsa_tile_size`</td>
      <td>`int`</td>
      <td>Kernel tile size. Only `64` (the trained (4, 4, 4) geometry) is accepted.</td>
      <td>`64`</td>
    </tr>
  </tbody>
</table>

VSA-H3 constraints:

* Only the DiT runs sparse; the token refiner, text encoder, and VAEs keep
  their dense defaults. An explicit `--component-attention-backends
  text_encoder=fa` is rejected because the H3 text encoder has SDPA-only layers.
* Uses the checkpoint's trained `to_gate_compress` compression branch. Base
  MiniMax-H3 weights load zero gates and run pure sparse.
* Ulysses sequence parallelism is supported; `--ring-degree` greater than 1,
  `torch.compile`, and breakable CUDA graph execution are rejected.

**Hybrid window attention for VDN-H3 (`hybrid_window_attn_h3`)**

[VDN-H3](/cookbook/diffusion/MiniMax/MiniMax-H3#7-vdn-h3-hybrid-attention-8-step-distill)
replaces every DiT block's dense self-attention with two branches. The
softmax branch is an exact softmax over a chunk-aligned frame window (frame
`t` belongs to chunk `t // 5` and attends to chunks `c - 1 .. c + 1`; frames
0 and F-1 are dense as rows and columns; every text / audio pair stays dense)
scaled by a per-head sigmoid gate. The linear branch (a frame-wise Video Delta
rule recurrence, forward and reverse over frames) covers exactly the window's
complement and is driven by the attention module. The mask is request-static,
so the metadata is built once per request. The checkpoint's
`transformer/config.json` carries the window geometry; the backend reads it.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`vdn_h3_dense_smoke`</td>
      <td>`bool`</td>
      <td>Allows a dense transformer backend (`fa`) on the VDN-H3 weights for the base-H3 + LoRA equivalence smoke. The gates and the linear branch are skipped: a study, not a sample.</td>
      <td>`false`</td>
    </tr>

    <tr>
      <td>`vdn_max_gather_rows`</td>
      <td>`int`</td>
      <td>Upper bound on the gathered K/V rows per window FlashAttention call; consecutive chunk groups fill one call up to it. Splitting changes no query's kept key set, only the pass count and the gather's peak memory.</td>
      <td>`200000`</td>
    </tr>
  </tbody>
</table>

Hybrid window attention constraints:

* VDN-H3 requires this backend for the transformer; a dense backend on these
  weights would silently skip the linear branch and the gates, so it is
  rejected unless `vdn_h3_dense_smoke` is set. The token refiner, text
  encoder, and VAEs keep dense attention.
* Base MiniMax-H3 and FastH3 checkpoints have no linear branch and are
  rejected by this backend.
* Ulysses sequence parallelism is supported (QK-norm + RoPE run after the
  all-to-all on the head shard); `--ring-degree` greater than 1,
  `torch.compile`, and breakable CUDA graph execution are rejected.

**V-MoBA (`vmoba_attn`)**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`temporal_chunk_size`</td>
      <td>`int`</td>
      <td>Chunk size for temporal dimension.</td>
      <td>-</td>
    </tr>

    <tr>
      <td>`temporal_topk`</td>
      <td>`int`</td>
      <td>Top-K tokens to select in temporal dimension.</td>
      <td>-</td>
    </tr>

    <tr>
      <td>`spatial_chunk_size`</td>
      <td>`list[int]`</td>
      <td>Chunk size for spatial dimension (H, W).</td>
      <td>-</td>
    </tr>

    <tr>
      <td>`spatial_topk`</td>
      <td>`int`</td>
      <td>Top-K tokens to select in spatial dimension.</td>
      <td>-</td>
    </tr>

    <tr>
      <td>`st_chunk_size`</td>
      <td>`list[int]`</td>
      <td>Chunk size for spatiotemporal dimension (T, H, W).</td>
      <td>-</td>
    </tr>

    <tr>
      <td>`st_topk`</td>
      <td>`int`</td>
      <td>Top-K tokens to select in spatiotemporal dimension.</td>
      <td>-</td>
    </tr>

    <tr>
      <td>`moba_select_mode`</td>
      <td>`str`</td>
      <td>Selection mode (e.g., `threshold`).</td>
      <td>`threshold`</td>
    </tr>

    <tr>
      <td>`moba_threshold`</td>
      <td>`float`</td>
      <td>Threshold value for selection.</td>
      <td>`0.25`</td>
    </tr>

    <tr>
      <td>`moba_threshold_type`</td>
      <td>`str`</td>
      <td>Type of thresholding (e.g., `query_head`).</td>
      <td>`query_head`</td>
    </tr>

    <tr>
      <td>`first_full_step`</td>
      <td>`int`</td>
      <td>Number of initial steps to use full attention.</td>
      <td>`12`</td>
    </tr>

    <tr>
      <td>`first_full_layer`</td>
      <td>`int`</td>
      <td>Number of initial layers to use full attention.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`temporal_layer`</td>
      <td>`int`</td>
      <td>Number of temporal layers.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`spatial_layer`</td>
      <td>`int`</td>
      <td>Number of spatial layers.</td>
      <td>`1`</td>
    </tr>

    <tr>
      <td>`st_layer`</td>
      <td>`int`</td>
      <td>Number of spatiotemporal layers.</td>
      <td>`1`</td>
    </tr>
  </tbody>
</table>

**Block Sparse Attention (`block_sparse_attn`)**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`skip_first_steps`</td>
      <td>`int`</td>
      <td>Number of steps to use laser attention before switching to sparse attention.</td>
      <td>`10`</td>
    </tr>

    <tr>
      <td>`sparsity`</td>
      <td>`float`</td>
      <td> The sparsity coefficient must be in the range (0, 1).</td>
      <td>`0.2`</td>
    </tr>
  </tbody>
</table>

**Sol-Attn (`sol_attn`)**

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
      <th>Default</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`tau`</td>
      <td>`float`</td>
      <td>Routing threshold scale. Higher values select fewer exact KV blocks.</td>
      <td>`1.0`</td>
    </tr>

    <tr>
      <td>`thresh_type`</td>
      <td>`str`</td>
      <td>Threshold mode: `diag` or `exact`.</td>
      <td>`diag`</td>
    </tr>

    <tr>
      <td>`sink_tokens`</td>
      <td>`int`</td>
      <td>Exact KV sink length for prefix tokens such as text/audio rows.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`sink_start`</td>
      <td>`int`</td>
      <td>Start index of the exact KV sink range.</td>
      <td>`0`</td>
    </tr>

    <tr>
      <td>`dense_steps`</td>
      <td>`int`</td>
      <td>Use dense attention for the first N denoising steps.</td>
      <td>`10`</td>
    </tr>

    <tr>
      <td>`dense_layers`</td>
      <td>`str`</td>
      <td>Layer indices kept dense, e.g. `0,1` or `0-2`.</td>
      <td>`0,1`</td>
    </tr>

    <tr>
      <td>`dense_backend`</td>
      <td>`str`</td>
      <td>Backend used for the dense prefix: `fa` (default) or `sage_attn`. `sage_attn` is approximate.</td>
      <td>`fa`</td>
    </tr>

    <tr>
      <td>`kv_splits`</td>
      <td>`int | str`</td>
      <td>KV split factor passed to the Sol-Attn kernel. Use `auto` on long sequences.</td>
      <td>`auto`</td>
    </tr>
  </tbody>
</table>

## Platform support matrix

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Backend</th>
      <th>CUDA</th>
      <th>ROCm</th>
      <th>XPU</th>
      <th>MUSA</th>
      <th>MPS</th>
      <th>NPU</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`fa`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>✅</td>
      <td>✅</td>
      <td>❌</td>
      <td>✅</td>
      <td>CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. No extra installations are required for NPU</td>
    </tr>

    <tr>
      <td>`torch_sdpa`</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>Yes</td>
      <td>✅</td>
      <td>✅</td>
      <td>Most compatible option across platforms.</td>
    </tr>

    <tr>
      <td>`sliding_tile_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`sage_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>Yes</td>
      <td>❌</td>
      <td>❌</td>
      <td>Optional dependency on CUDA and MUSA. On Hopper, also falls back to FlashAttention when the installed package lacks the SM90 binding fix.</td>
    </tr>

    <tr>
      <td>`sage_attn_3`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only (optional dependency).</td>
    </tr>

    <tr>
      <td>`sparge_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA SM80/86/87/89/90 only. Requires `SpargeAttn`; head dim 64/128 and square self-attention with sequence length at least 128.</td>
    </tr>

    <tr>
      <td>`sol_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only. Requires `sol-attn`. Install with `pip install git+https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends`. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`video_sparse_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`video_sparse_attn_h3`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only (SM90 / SM100 / SM103). In-tree kernels (native tcgen05 forward on Blackwell, Triton on Hopper), no external dependency. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`sla_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only. Requires `SpargeAttn`.</td>
    </tr>

    <tr>
      <td>`sage_sla_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only. Requires `SpargeAttn`.</td>
    </tr>

    <tr>
      <td>`vmoba_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`aiter`</td>
      <td>❌</td>
      <td>✅</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>Requires `aiter`.</td>
    </tr>

    <tr>
      <td>`aiter_sage`</td>
      <td>❌</td>
      <td>✅</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>Requires `aiter`.</td>
    </tr>

    <tr>
      <td>`sparse_video_gen_2_attn`</td>
      <td>Yes</td>
      <td>No</td>
      <td>No</td>
      <td>No</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA-only. Requires `svg`.</td>
    </tr>

    <tr>
      <td>`laser_attn`</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>NPU-only. Requires `attentions` from `sgl_kernel_npu`. Uses SDPA if seqlen less than 2048.</td>
    </tr>

    <tr>
      <td>`block_sparse_attn`</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>NPU-only. Requires `attentions` from `sgl_kernel_npu`. Configuration via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`rain_fusion_attn`</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>✅</td>
      <td>NPU-only. Requires `attentions` from `sgl_kernel_npu` Configuration via `--attention-backend-config`.</td>
    </tr>

    <tr>
      <td>`fp8_fa_sm120`</td>
      <td>Yes</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>❌</td>
      <td>CUDA SM120 only; falls back to `torch_cudnn_sdpa` on other devices and for unsupported calls.</td>
    </tr>
  </tbody>
</table>

## Usage

### Select a backend via CLI

```bash theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend fa
```

```bash theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend torch_sdpa
```

### Override one component

Use component overrides when a specific module needs different attention semantics from the main transformer:

```bash theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend fa \
  --component-attention-backends text_encoder=torch_sdpa
```

Component keys match pipeline module names from `model_index.json`, such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`.
Use this override when the fallback must be pinned: unlike the global backend,
an incompatible component override raises an error instead of selecting another
backend. The one role-based exception is a sparse self-attention backend, which
uses a compatible dense backend for cross-attention layers in the same component.
The component must construct SGLang-selectable attention or explicitly defer
selection until first use; components with fixed attention reject the override.
Per-component overrides apply only to native pipelines. The Diffusers backend
accepts the global `--attention-backend` passthrough instead.

### Per-request override (denoise loop)

A single server can serve exact and approximate attention side by side: requests
may switch the DiT denoise attention backend via the `attention_backend_override`
sampling param. Valid values are the exact/drop-in dense kernels — `fa`,
`torch_sdpa`, `sage_attn`, `sage_attn_3`. The field participates in the
dynamic-batch signature, so requests with different backends never share a batch.

```bash theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend-override sage_attn
```

```python theme={null}
client.images.generate(
    model="<MODEL_PATH_OR_ID>",
    prompt="...",
    extra_body={"attention_backend_override": "sage_attn"},
)
```

Incompatible server settings **reject the request** (with a server log) instead
of silently falling back: breakable CUDA graphs and `torch.compile` bake the
attention kernel into a captured/traced graph; sparse server backends
(`sliding_tile_attn`, `video_sparse_attn`, ...) cannot be mixed with per-request
dense switching; under ring parallelism the target must be ring-capable. Note
`sage_attn` / `sage_attn_3` are lossy (quantized attention) — validate quality
on your workload.

### Request-scoped Skip Softmax

Skip Softmax (BLASST) keeps the QK matmul, but skips the exponential,
softmax-state update, V load, and PV matmul for attention tiles whose estimated
softmax mass is below a threshold. It is an explicit lossy optimization backed
by the FlashInfer kernels shipped with SGLang's pinned dependencies. See the
[BLASST paper](https://arxiv.org/abs/2512.12087) and NVIDIA's
[video-generation study](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog28_Accelerating_Video_Generation_with_GEMM_Quantization_Attention_Quantization_and_Skip_Softmax_Attention_in_TensorRT-LLM.md)
for the algorithm and measured quality/performance trade-offs.

Pass `skip_softmax_params` in one image or video request. The runtime routes
compatible self-attention layers through the FA/FlashInfer path for that batch;
cross-attention keeps its normal backend, and the next request restores the
server default.

```python theme={null}
client.videos.create(
    model="<MODEL_PATH_OR_ID>",
    prompt="...",
    extra_body={
        "skip_softmax_params": {
            "threshold_scale_factor": 500.0,
            "start_step": 14,
        }
    },
)
```

* `threshold_scale_factor` is required and must be positive. The kernel uses
  `threshold_scale_factor / context_length`; larger values skip more work and
  usually increase quality loss.
* `start_step` is the zero-based denoising step at which sparse execution
  starts. It defaults to `0`; keeping early high-noise steps dense is generally
  safer.

There is intentionally no default threshold. Calibrate the threshold and start
step against output-quality metrics for each model, resolution, step count, and
deployment backend; values tuned for one workload are not portable.

Current support is FP16/BF16, head dimension 128 or 256, and unmasked
self-attention on Hopper SM90 and Blackwell SM100/SM103/SM107. Ulysses sequence
parallelism is supported because the kernel runs after its all-to-all. Ring
Attention, attention masks, `torch.compile`, and breakable CUDA graphs reject
the request instead of silently running dense attention. Models that merely
offer an FA backend do not automatically qualify: the runtime still checks the
GPU, dtype, head dimension, attention role, and execution mode.

### Using SpargeAttention

Install the optional CUDA extension, then select the backend explicitly:

```bash theme={null}
pip install ninja
pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend sparge_attn \
  --attention-backend-config topk=0.5
```

`sparge_attn` applies to compatible self-attention layers. Cross-attention uses
the existing dense fallback; short self-attention and asymmetric Q/KV also use
dense SDPA. LTX-2, LTX-2.3, and LTX-2.5 have compatible video (128) and audio
(64) head dimensions, so their sufficiently long, unmasked self-attention uses
SpargeAttention while prompt and audio-video cross-attention remain dense.
Ulysses sequence parallelism is supported, but ring attention is not because the
upstream kernel does not expose the softmax LSE needed for ring merging. K/V-gather
SP therefore follows the normal sparse-backend rule and uses Ulysses instead.

### Sage then Sol hybrid

`sol_attn` keeps the first `dense_steps` steps dense. Set
`dense_backend=sage_attn` to run that prefix on SageAttention and the tail on
Sol sparse attention. Keep the text encoder on `torch_sdpa`:

```bash theme={null}
sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --attention-backend sol_attn \
  --attention-backend-config dense_backend=sage_attn,dense_steps=10 \
  --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output
```

Both `sage_attn` and `sol_attn` are approximate. The default DiT backend remains `fa`.

### Using Sliding Tile Attention (STA)

```bash theme={null}
# Pass the mask strategy file path via config
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend sliding_tile_attn \
  --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"
```

### Notes for ROCm / MPS

* ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment. On gfx1151 / Strix Halo (wave32), do **not** use `aiter` — prefer `torch_sdpa` (image sets `SGLANG_USE_AITER=0`).
* MPS: the platform implementation always uses `torch_sdpa`.


# Cache-DiT Acceleration
Source: https://docs.sglang.io/docs/sglang-diffusion/cache_dit

Configure Cache-DiT acceleration for diffusion inference.

SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss.

## Overview

**Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop:

* **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences
* **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions
* **DMD Calibrator**: An **exponential-basis** forecasting calibrator (Dynamic Mode Decomposition, not Distribution Matching Distillation) that serves as a drop-in alternative to TaylorSeer's polynomial basis; strongest on flow-matching models
* **SCM (Step Computation Masking)**: Step-level caching control for additional speedup

Cache-DiT also ships **SVDQuant** W4A4 (int4 / NVFP4) dynamic quantization, which can be combined
with DBCache caching (see [Quantization](#quantization)).

## Basic Usage

Cache-DiT is a **per-request** switch: each request decides whether to run
cached or lossless, and requests with different Cache-DiT settings never share
a batch. The `SGLANG_CACHE_DIT_*` environment variables remain available as
server-wide defaults for requests that leave the switch unset.

Enable it for a single generation:

```bash theme={null}
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A beautiful sunset over the mountains" \
    --enable-cache-dit true
```

Or per request against a running server, via the OpenAI-compatible API:

```python theme={null}
client.images.generate(
    model="Qwen/Qwen-Image",
    prompt="A beautiful sunset over the mountains",
    extra_body={
        "enable_cache_dit": True,
        # optional knob overrides for this request only
        "cache_dit_params": {"residual_diff_threshold": 0.12, "scm_preset": "fast"},
    },
)
```

`enable_cache_dit` accepts three states: `true` (on for this request), `false`
(off for this request, overriding the server default), and unset (follow the
`SGLANG_CACHE_DIT_ENABLED` server default). `cache_dit_params` accepts the
DBCache knobs (`Fn_compute_blocks`, `Bn_compute_blocks`, `max_warmup_steps`,
`residual_diff_threshold`, `max_continuous_cached_steps`, `enable_taylorseer`,
`taylorseer_order`), the DMD knobs (`enable_dmd`, `dmd_history`, `dmd_rank`,
`dmd_ridge`, `dmd_svd_precision`; DMD and TaylorSeer are mutually exclusive
calibrators and cannot be enabled together), the SCM knobs (`scm_preset`,
`scm_compute_bins`, `scm_cache_bins`, `scm_policy`), and a nested `secondary`
dict with the DBCache
knobs for the second transformer of dual-DiT models (unset secondary keys
inherit the request's primary values, then the
`SGLANG_CACHE_DIT_SECONDARY_*` defaults).

To make Cache-DiT the default for every request instead, export the
environment variable when launching:

```bash theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
sglang serve --model-path Qwen/Qwen-Image
```

## Diffusers Backend

Cache-DiT supports loading acceleration configs from a custom YAML file. For
diffusers pipelines (`diffusers` backend), pass the YAML/JSON path via `--cache-dit-config`. This
flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`).

### Single GPU inference

Define a `cache.yaml` file that contains:

* DBCache + TaylorSeer

```yaml theme={null}
cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
```

Then apply the config with:

```bash theme={null}
sglang generate \
  --backend diffusers \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config cache.yaml \
  --prompt "A beautiful sunset over the mountains"
```

* DBCache + TaylorSeer + SCM (Step Computation Mask)

```yaml Config theme={null}
cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
  # Must set the num_inference_steps for SCM. The SCM will automatically
  # generate the steps computation mask based on the num_inference_steps.
  # Reference: https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#scm-steps-computation-masking
  num_inference_steps: 28
  steps_computation_mask: fast
```

* DBCache + TaylorSeer + SCM (Step Computation Mask) + Cache CFG

```yaml Config theme={null}
cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
  num_inference_steps: 28
  steps_computation_mask: fast
  enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc.
```

* DBCache + DMD Calibrator

Instead of TaylorSeer, you can use the DMD calibrator: an **exponential-basis** forecasting
calibrator that serves as a drop-in alternative to TaylorSeer's polynomial basis. DMD models
the cached feature stream as a linear dynamical system (`Y_{t+1} ~= A @ Y_t`), forecasts
cached features from the fitted eigen-modes, and stays accurate over longer cache skips where
polynomial extrapolation diverges. DMD here refers to Dynamic Mode Decomposition (Schmid
2010\), **not** Distribution Matching Distillation. DMD works best on flow-matching models
(e.g., FLUX), while TaylorSeer is often better on DDPM-style models — try both. DMD and
TaylorSeer are mutually exclusive — enable only one of `enable_dmd` / `enable_taylorseer`:

```yaml Config theme={null}
cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0 # Bn=0 since the DMD calibrator replaces the Bn calibrator
  residual_diff_threshold: 0.12
  enable_dmd: true
  dmd_history: 6 # snapshot window length, 5-6 typical
  dmd_svd_precision: "medium" # "low", "medium" or "high"
```

A `dmd_history` window of 5–6 snapshots is typically the sweet spot — longer histories do not
always help, because the feature dynamics drift across timesteps. With fewer than 4 uniformly
spaced snapshots available, DMD transparently falls back to the Taylor expansion it maintains
internally. See the
[Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition)
for the mathematical principle and quantitative comparisons. A ready-made config is available
at
[examples/configs/cache\_dmd.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd.yaml)
in the Cache-DiT repository. Apply it with the same `--cache-dit-config` flag:

```bash theme={null}
sglang generate \
  --backend diffusers \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config cache_dmd.yaml \
  --prompt "A beautiful sunset over the mountains"
```

### Distributed inference

* 1D Parallelism

Define a parallelism only config yaml `parallel.yaml` file that contains:

```yaml Config theme={null}
parallelism_config:
  ulysses_size: auto
  attention_backend: native
```

Then, apply the distributed inference acceleration config from yaml. `ulysses_size: auto` means that cache-dit will auto detect the `world_size` as the ulysses\_size. Otherwise, you should manually set it as specific int number, e.g, 4.

Then apply the distributed config with: (Note: please add `--num-gpus N` to specify the number of gpus for distributed inference)

```bash theme={null}
sglang generate \
  --backend diffusers \
  --num-gpus 4 \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config parallel.yaml \
  --prompt "A futuristic cityscape at sunset"
```

* 2D Parallelism

You can also define a 2D parallelism config yaml `parallel_2d.yaml` file that contains:

```yaml Config theme={null}
parallelism_config:
  ulysses_size: auto
  tp_size: 2
  attention_backend: native
```

Then, apply the 2D parallelism config from yaml. Here `tp_size: 2` means using tensor parallelism with size 2. The `ulysses_size: auto` means that cache-dit will auto detect the `world_size // tp_size` as the ulysses\_size.

* 3D Parallelism

You can also define a 3D parallelism config yaml `parallel_3d.yaml` file that contains:

```yaml Config theme={null}
parallelism_config:
  ulysses_size: 2
  ring_size: 2
  tp_size: 2
  attention_backend: native
```

Then, apply the 3D parallelism config from yaml. Here `ulysses_size: 2`, `ring_size: 2`, `tp_size: 2` means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2.

* Ulysses Anything Attention

To enable Ulysses Anything Attention, you can define a parallelism config yaml `parallel_uaa.yaml` file that contains:

```yaml Config theme={null}
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  ulysses_anything: true
```

* Ulysses FP8 Communication

For device that don't have NVLink support, you can enable Ulysses FP8 Communication to further reduce the communication overhead. You can define a parallelism config yaml `parallel_fp8.yaml` file that contains:

```yaml Config theme={null}
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  ulysses_float8: true
```

* Async Ulysses CP

You can also enable async ulysses CP to overlap the communication and computation. Define a parallelism config yaml `parallel_async.yaml` file that contains:

```yaml Config theme={null}
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  ulysses_async: true # Now, only support for FLUX.1, Qwen-Image, Ovis-Image and Z-Image.
```

Then, apply the config from yaml. Here `ulysses_async: true` means enabling async ulysses CP.

* TE-P and VAE-P

You can also specify the extra parallel modules in the yaml config. For example, define a parallelism config yaml `parallel_extra.yaml` file that contains:

```yaml Config theme={null}
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  extra_parallel_modules: ["text_encoder", "vae"]
```

### Hybrid Cache and Parallelism

Define a hybrid cache and parallel acceleration config yaml `hybrid.yaml` file that contains:

```yaml Config theme={null}
cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  extra_parallel_modules: ["text_encoder", "vae"]
```

Then, apply the hybrid cache and parallel acceleration config from yaml.

```bash theme={null}
sglang generate \
  --backend diffusers \
  --num-gpus 4 \
  --model-path Qwen/Qwen-Image \
  --cache-dit-config hybrid.yaml \
  --prompt "A beautiful sunset over the mountains"
```

### Attention Backend

In some cases, users may want to only specify the attention backend without any other optimization configs. In this case, you can define a yaml file `attention.yaml` that only contains:

```yaml Config theme={null}
attention_backend: "flash" # '_flash_3' for Hopper
```

### Quantization

You can also specify the quantization config in the yaml file, required `torchao>=0.16.0`. For example, define a yaml file `quantize.yaml` that contains:

```yaml Config theme={null}
quantize_config: # quantization configuration for transformer modules
  # float8 (DQ), float8_weight_only, float8_blockwise, int8 (DQ), int8_weight_only, etc.
  quant_type: "float8"
  # layers to exclude from quantization (transformer). layers that contains any of the
  # keywords in the exclude_layers list will be excluded from quantization. This is useful
  # for some sensitive layers that are not robust to quantization, e.g., embedding layers.
  exclude_layers:
    - "embedder"
    - "embed"
  verbose: false # whether to print verbose logs during quantization
```

Then, apply the quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. For example:

```bash Command theme={null}
sglang generate \
  --backend diffusers \
  --model-path Qwen/Qwen-Image \
  --warmup-mode request \
  --cache-dit-config quantize.yaml \
  --enable-torch-compile \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false \
  --prompt "A beautiful sunset over the mountains"
```

#### SVDQuant (W4A4 int4 / NVFP4)

SVDQuant is Cache-DiT's built-in W4A4 PTQ quantization (weights and activations in int4 or
NVFP4, with smoothed low-rank branches). It can be freely combined with DBCache caching and
the DMD calibrator for the largest speedups.

::::note
SVDQuant requires a cache-dit build **with CUDA extension support** — a plain
`pip install cache-dit` does NOT include it. Install one of:

```bash Command theme={null}
# Option 1: prebuilt CUDA 13 wheel
pip install cache-dit-cu13==<version> --no-deps

# Option 2: build from source with SVDQuant enabled
git clone https://github.com/vipshop/cache-dit
cd cache-dit
export CUDA_HOME=/usr/local/cuda
CACHE_DIT_BUILD_SVDQUANT=1 pip install ".[quantization]" --no-build-isolation
```

::::

Valid `quant_type` values are `svdq_int4_r{32,64,128,256}_dq` (int4 W4A4) and
`svdq_nvfp4_r{32,64,128,256}_dq` (NVFP4 W4A4; requires a Blackwell GPU). Example config
combining SVDQuant NVFP4 with DBCache + DMD (see
[examples/configs/blackwell/cache\_dmd\_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/blackwell/cache_dmd_svdq.yaml)):

```yaml Config theme={null}
cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_dmd: true
  dmd_history: 6
  dmd_svd_precision: "medium"
quantize_config:
  quant_type: "svdq_nvfp4_r128_dq" # nvfp4 for Blackwell; use svdq_int4_r128_dq for int4
  svdq_kwargs:
    quantize_device: "cuda"
    fused_mlp: true
  exclude_layers:
    - "embedder"
    - "embed"
  verbose: false
```

For int4 W4A4 (pre-Blackwell GPUs), the same config with
`quant_type: "svdq_int4_r128_dq"` is available at
[examples/configs/cache\_dmd\_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd_svdq.yaml)
(add `runtime_kernel: "v2"` to `svdq_kwargs`).

Enable `torch.compile` for the best SVDQuant performance, and make sure `--warmup-steps`
covers the compile warmup (use the same value as `--num-inference-steps`):

```bash Command theme={null}
sglang generate \
  --backend diffusers \
  --model-path black-forest-labs/FLUX.1-dev \
  --num-inference-steps=28 \
  --warmup-mode request \
  --warmup-steps 28 \
  --cache-dit-config cache_dmd_svdq.yaml \
  --enable-torch-compile \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false \
  --prompt "A beautiful sunset over the mountains"
```

You can verify from the log that the quantization is active:
`[Cache-DiT] SVDQuant Type: svdq_nvfp4_r128_dq, Rank: 128`.

### Combined Configs: Cache + Parallelism + Quantization

You can also combine all the above configs together in a single yaml file `combined.yaml` that contains:

```yaml Config theme={null}
cache_config:
  max_warmup_steps: 8
  warmup_interval: 2
  max_cached_steps: -1
  max_continuous_cached_steps: 2
  Fn_compute_blocks: 1
  Bn_compute_blocks: 0
  residual_diff_threshold: 0.12
  enable_taylorseer: true
  taylorseer_order: 1
parallelism_config:
  ulysses_size: auto
  attention_backend: native
  extra_parallel_modules: ["text_encoder", "vae"]
quantize_config:
  quant_type: "float8"
  exclude_layers:
    - "embedder"
    - "embed"
  verbose: false
```

Then, apply the combined cache, parallelism and quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization.

## Advanced Configuration

### DBCache Parameters

DBCache controls block-level caching behavior:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fn</td>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>Number of first blocks to always compute</td>
    </tr>

    <tr>
      <td>Bn</td>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Number of last blocks to always compute</td>
    </tr>

    <tr>
      <td>W</td>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching starts</td>
    </tr>

    <tr>
      <td>R</td>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>MC</td>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Maximum continuous cached steps</td>
    </tr>
  </tbody>
</table>

### TaylorSeer Configuration

TaylorSeer improves caching accuracy using Taylor expansion:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>Order</td>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>Taylor expansion order (1 or 2)</td>
    </tr>
  </tbody>
</table>

### DMD Calibrator Configuration

DMD (Dynamic Mode Decomposition, Schmid 2010 — **not** Distribution Matching Distillation) is
an **exponential-basis** forecasting calibrator and a drop-in alternative to TaylorSeer's
polynomial basis. At each full-compute step it records a snapshot of the computed features; at
a cached step it identifies a linear propagator from the recent snapshot window (one economy
SVD with rank truncation, then eigendecomposition) and forecasts the current features via
eigenvalue powers — cheap to advance, and stable over longer cache skips where polynomial
extrapolation diverges. It typically improves both speed and quality over pure DBCache.
**DMD and TaylorSeer are mutually exclusive** (enabling both raises a `ValueError`); DMD is
best for flow-matching models, TaylorSeer for DDPM-style ones. See the
[Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition)
for details:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Env Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Enable</td>
      <td>`SGLANG_CACHE_DIT_DMD`</td>
      <td>false</td>
      <td>Enable the DMD calibrator</td>
    </tr>

    <tr>
      <td>History</td>
      <td>`SGLANG_CACHE_DIT_DMD_HISTORY`</td>
      <td>6</td>
      <td>Snapshot window length; 5-6 typical. Needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer</td>
    </tr>

    <tr>
      <td>Rank</td>
      <td>`SGLANG_CACHE_DIT_DMD_RANK`</td>
      <td>0</td>
      <td>SVD truncation rank; 0 = automatic (drop modes below 1e-4 of the leading singular value)</td>
    </tr>

    <tr>
      <td>Ridge</td>
      <td>`SGLANG_CACHE_DIT_DMD_RIDGE`</td>
      <td>1e-8</td>
      <td>Tikhonov regularization added to the inverted singular values</td>
    </tr>

    <tr>
      <td>SVD Precision</td>
      <td>`SGLANG_CACHE_DIT_DMD_SVD_PRECISION`</td>
      <td>medium</td>
      <td>SVD precision: "low", "medium" or "high"</td>
    </tr>
  </tbody>
</table>

Usage (SGLD backend, env-driven):

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_DMD=true \
sglang generate --model-path black-forest-labs/FLUX.1-dev \
    --prompt "A curious raccoon in a forest"
```

On the diffusers backend, enable DMD from the yaml config instead
(`enable_dmd: true` in `cache_config`, see
[Diffusers Backend](#diffusers-backend)); DMD can also be set per request via
`cache_dit_params: {"enable_dmd": true}`.

### Combined Configuration Example

DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters
simultaneously:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang generate --model-path black-forest-labs/FLUX.1-dev \
    --prompt "A curious raccoon in a forest"
```

### SCM (Step Computation Masking)

SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and
which to use cached results.

**SCM Presets**

SCM is configured with presets:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Preset</th>
      <th>Compute Ratio</th>
      <th>Speed</th>
      <th>Quality</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`none`</td>
      <td>100%</td>
      <td>Baseline</td>
      <td>Best</td>
    </tr>

    <tr>
      <td>`slow`</td>
      <td>\~75%</td>
      <td>\~1.3x</td>
      <td>High</td>
    </tr>

    <tr>
      <td>`medium`</td>
      <td>\~50%</td>
      <td>\~2x</td>
      <td>Good</td>
    </tr>

    <tr>
      <td>`fast`</td>
      <td>\~35%</td>
      <td>\~3x</td>
      <td>Acceptable</td>
    </tr>

    <tr>
      <td>`ultra`</td>
      <td>\~25%</td>
      <td>\~4x</td>
      <td>Lower</td>
    </tr>
  </tbody>
</table>

**Usage**

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_PRESET=medium \
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A futuristic cityscape at sunset"
```

**Custom SCM Bins**

For fine-grained control over which steps to compute vs cache:

```bash Command theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A futuristic cityscape at sunset"
```

**SCM Policy**

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Policy</th>
      <th>Env Variable</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`dynamic`</td>
      <td>`SGLANG_CACHE_DIT_SCM_POLICY=dynamic`</td>
      <td>Adaptive caching based on content (default)</td>
    </tr>

    <tr>
      <td>`static`</td>
      <td>`SGLANG_CACHE_DIT_SCM_POLICY=static`</td>
      <td>Fixed caching pattern</td>
    </tr>
  </tbody>
</table>

## Environment Variables

All Cache-DiT parameters can also be configured via environment variables,
which act as the server-wide defaults for requests that don't set
`enable_cache_dit` / `cache_dit_params`.
See [Environment Variables](./environment_variables) for the complete list.

## Supported Models

SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>Example Models</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Wan</td>
      <td>Wan2.1, Wan2.2</td>
    </tr>

    <tr>
      <td>Flux</td>
      <td>FLUX.1-dev, FLUX.2-dev, FLUX.3 Image Klein</td>
    </tr>

    <tr>
      <td>Z-Image</td>
      <td>Z-Image-Turbo</td>
    </tr>

    <tr>
      <td>Qwen</td>
      <td>Qwen-Image, Qwen-Image-Edit, Qwen-Image 2.1</td>
    </tr>

    <tr>
      <td>Hunyuan</td>
      <td>HunyuanVideo</td>
    </tr>

    <tr>
      <td>MiniMax</td>
      <td>MiniMax-H3 (T2VA, FL2VA, and Ref2VA)</td>
    </tr>
  </tbody>
</table>

## Performance Tips

1. **Start with defaults**: The default parameters work well for most models
2. **Use TaylorSeer**: It typically improves both speed and quality
3. **Tune R threshold**: Lower values = better quality, higher values = faster
4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance
5. **Warmup matters**: Higher warmup = more stable caching decisions

## Limitations

* **SGLang-native pipelines**: Distributed Cache-DiT paths exist for supported pipelines. Hybrid SP+TP configurations add communication and cache coordination overhead, so validate them on the target model and hardware before using them as production defaults.
* **DiT layerwise offload**: Compatible. Skipped blocks are not streamed, and the first layer after a skip may sync-load. Still incompatible with `--use-fsdp-inference`.
* **SCM minimum steps**: SCM requires >= 8 inference steps to be effective. Some pipelines report `steps - 1` NFEs (for example MiniMax-H3 `num_inference_steps=8` is 7 NFEs), which trips the upstream `steps_mask` assertion; use at least 9 requested steps or custom bins.
* **Model support**: The model must be registered in Cache-DiT's
  `BlockAdapterRegister` or have an SGLang custom block adapter.

## Troubleshooting

### SCM disabled for low step count

For models with \< 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache
acceleration still works.

### SVDQuant unavailable or load failure

SVDQuant cases raise `svdq_is_available() = False` or
`undefined symbol: ... materialize_cow_storage ...` when the installed cache-dit has no CUDA
extension, or the prebuilt wheel was compiled against an incompatible torch. Fix: reinstall
from the `cache-dit-cu13` wheel matching your torch version, or build cache-dit from source
with `CACHE_DIT_BUILD_SVDQUANT=1` (see [Quantization](#quantization)). Quick self-check:

```bash Command theme={null}
python -c "from cache_dit.quantization.svdquant import svdq_is_available, svdq_get_load_error as e; print(svdq_is_available(), e())"
```

## References

* [Cache-DiT](https://github.com/vipshop/cache-dit)
* [SGLang Diffusion](./performance-optimization)


# Caching Acceleration
Source: https://docs.sglang.io/docs/sglang-diffusion/caching-acceleration

Compare caching acceleration strategies for diffusion models.

SGLang can reuse exact conditioning across requests and cache intermediate denoising results. Conditioning reuse preserves the encoded values; denoising caches such as Cache-DiT, TeaCache, and Spectrum use approximations.

## Conditioning cache

Native pipelines enable a conditioning cache by default. Repeated inputs can reuse text embeddings (including negative prompts), image or vision-language encoder outputs, and VAE posteriors. The shared LRU holds at most **512 MiB per executor/rank** across CPU and device entries, with at most 128 entries. The common text stage keeps consumed negative conditioning on its original device to avoid repeated CPU-to-GPU transfers and postprocessing. Other entries live on the CPU. Oversized results are not admitted.

Within a grouped or batched stage, the same conditioning boundary can reuse a device result before consulting the cross-request cache. The common text stage and the Qwen-Image 2.1, Ming-Image, and LingBot-Video text stages share read-only consumed tensors and give each request its own containers. Other cached boundaries return private copies, with device snapshots limited to 512 MiB and 128 entries per stage. These temporary references are released when the stage finishes or fails. Group reuse remains available when cross-request caching is disabled or its budget is too small.

The standard text encoding stage stores the final embeddings, pooled outputs, masks, and sequence lengths. A hit skips encoder weight preparation, token uploads, and mask construction; CPU tokenization still runs to match the inputs. Qwen-Image 2.1 also caches its consumed prompt embeddings and image-token slots before processor execution, while retaining separate vision-feature reuse for changed edit instructions. Other custom stages and batch-DP encoding retain caching at the native encoder boundary.

CPU hits restore outputs to their original devices; device hits return private device copies. Negative conditioning can therefore retain GPU memory between requests, within the shared cache budget. Measure memory as well as latency when using a tight VRAM budget.

This is most useful when generating several variations from the same prompt or reference image, especially with a layerwise-offloaded encoder. A changed edit instruction recomputes the joint vision-language embedding, while the unchanged image can still reuse its vision features and VAE posterior. Separate CLI invocations start separate processes and cannot share this cache. Denoising and output decoding still run for every request, so end-to-end gains depend on how much time the original request spends encoding conditioning.

To disable cross-request conditioning reuse:

```bash theme={null}
sglang serve --model-path Qwen/Qwen-Image-2.1 --disable-conditioning-cache
```

To increase the shared cache budget:

```bash theme={null}
sglang serve --model-path Qwen/Qwen-Image-2.1 --conditioning-cache-max-size-mb 1024
```

The same settings are accepted by `DiffGenerator.from_pretrained()` as `disable_conditioning_cache` and `conditioning_cache_max_size_mb`. A capacity of `0` also disables both cross-request tiers. Neither setting disables conditioning reuse within a grouped stage or changes per-output seeds.

### Matching and correctness

* Keys include the loaded encoder instance, actual input contents, shapes, dtypes, masks, and autocast precision. Image tensors and PIL images are matched by content, so replacing a file at the same path cannot reuse stale pixels. Alpha is part of the VAE input.
* Positive and negative inputs use the same bounded cache. Different seeds can reuse conditioning; initial noise is still generated for each request.
* VAE caching stores the posterior before sampling and normalization. Every request still calls `sample()` or `mode()` as its model requires, preserving the random-number sequence.
* Cross-request hits return independent copies. Grouped consumed outputs may share read-only tensors; other boundaries isolate mutable tensors and posteriors. In-place normalization and downstream mutations cannot change a persistent entry.
* Encoder TP and folding groups agree on hits before skipping collective work. Weight updates and LoRA changes invalidate the cache, including failed updates that may have modified weights.
* Warmup refreshes cached results and executes each distinct encoding within a group once. This primes kernels while preserving multi-output reuse; subsequent requests can reuse conditioning such as the model-default negative prompt. Compare disabled, cold, and repeated-input latency separately when benchmarking.

### Coverage and limits

The common encoder boundary covers native CLIP, T5/UMT5, Llama, Mistral 3, Gemma 2/3, Qwen 2.5-VL, Qwen 3/3-VL, Ideogram, and MiniMax-H3 encoders, including custom pipeline stages. Qwen-VL and Gemma 3 also cache vision features independently of text. Hunyuan3D image encoders and the LTX-2.3 video condition encoder participate. Native image, video, and audio VAE encode entry points cache deterministic tensors or supported posterior containers.

Cosmos3 has no separate text encoder: its deterministic understanding pathway caches immutable per-layer K/V results, which are copied into fresh request state. SenseNova-U1 caches reference-image vision features; its noisy-image vision path still runs at each denoising step. VLA keeps its model-owned observation/prefix cache.

FSDP disables cross-request caching and caching inside sharded modules. Consumed text-conditioning outputs still support group-local reuse, with all ranks agreeing before skipping the encoder. The common text stage also reuses consumed outputs from library fallback encoders within a group and keeps their negative conditioning device-hot across requests. Batch-DP can skip encoding and its output gather only when all participating ranks have the consumed result; this includes device-hot negatives. Otherwise each encoder copy retains its normal cache and gather behavior.

Stateful autoregressive calls (`use_cache=True` or existing `past_key_values`), training/autograd, CUDA Graph capture, unsupported tensor or output types, and distributed VAE encoding bypass the shared caches. External SRT encoders and the Diffusers pipeline backend keep their own cache behavior. The global disable flag also disables existing realtime text and VLA prefix reuse; those model-specific caches otherwise retain their own memory settings.

Joint vision-language embeddings depend on both the prompt and image. The cache does not reuse a previous turn's combined embedding when either changes, and does not carry mutable denoising KV state between requests.

Dynamic batching still decides which requests can execute together. Conditioning reuse does not change encoder batch shapes, padding, or output order: a batched encoder call is matched as a whole, rather than split into individually cached rows. Model-specific grouped execution that also prepares materials, broadcasts results, or manages state retains its own execution rules.

The common text stage also keeps group-local stage deduplication: equivalent requests execute the stage once and receive their own output containers. This avoids repeating tokenization and stage bookkeeping. When stage deduplication leaves only one computation, it also skips the temporary group cache and its hit synchronization; cross-request caching remains available. The conditioning cache shares encodings between otherwise different requests, including a common negative prompt. Custom stages that prepare request-specific metadata or realtime session state use encoder-result reuse while still preparing each request.

## Overview

The following approaches reduce denoising computation:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Strategy</th>
      <th>Scope</th>
      <th>Mechanism</th>
      <th>Best For</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Cache-DiT</td>
      <td>Block-level</td>
      <td>Skip individual transformer blocks dynamically</td>
      <td>Advanced, higher speedup</td>
    </tr>

    <tr>
      <td>TeaCache</td>
      <td>Timestep-level</td>
      <td>Skip entire denoising steps based on L1 similarity</td>
      <td>Simple, built-in</td>
    </tr>

    <tr>
      <td>Spectrum</td>
      <td>Timestep-level</td>
      <td>Forecast DiT features to skip selected denoising steps</td>
      <td>Experimental, model-validated tuning</td>
    </tr>
  </tbody>
</table>

## Cache-DiT

[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with
advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**.

See [Cache-DiT](./cache_dit) for detailed configuration.

> [!NOTE]
>
> Cache-DiT currently cannot be combined with `--use-fsdp-inference`. Keep FSDP disabled when enabling Cache-DiT. DiT layerwise offload is compatible: skipped blocks are not streamed, and the first layer after a skip may sync-load.

### Quick Start

```bash theme={null}
SGLANG_CACHE_DIT_ENABLED=true \
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A beautiful sunset over the mountains"
```

### Key Features

* **DBCache**: Dynamic block-level caching based on residual differences
* **TaylorSeer**: Taylor expansion-based calibration for optimized caching
* **SCM**: Step-level computation masking for additional speedup

## TeaCache

TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.

See [TeaCache](./teacache) for detailed documentation.

### Quick Overview

* Tracks L1 distance between modulated inputs across timesteps
* When accumulated distance is below threshold, reuses cached residual
* Uses separate positive/negative caches for supported CFG model families

### Supported Models

* Wan2.1
* Z-Image
* Wan2.2: coefficients are not calibrated yet; enabling TeaCache is accepted but currently no-ops
* HunyuanVideo: not supported yet

For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.

## Spectrum

Spectrum forecasts DiT features and skips selected denoising steps. It is
approximate and currently applies only to selected native implementation paths.

See [Spectrum Acceleration](./spectrum) for supported model families,
constraints, and request controls.

## References

* [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
* [TeaCache Paper](https://arxiv.org/abs/2411.14324)


# CI Performance Baselines
Source: https://docs.sglang.io/docs/sglang-diffusion/ci_perf

Generate and update diffusion performance baselines used in CI.

## Sequential Request Checks

Server cases can exercise request-to-request state without restarting the server:

```python theme={null}
DiffusionTestCase(
    id="image_t2i",
    server_args=server_args,
    sampling_params=sampling_params,
    perf_repeat_requests=2,
)
```

Every request is checked, including the first one. Performance checks include
load/runtime peak memory; consistency checks include audio when
`expect_audio_output=True`. A failure in either request fails the case, even if
the other succeeds. Reports retain the request index instead of overwriting the
first result with the second.

Keep workloads small, but regenerate baselines and GT whenever sampling
parameters change. MiniMax-H3 T2VA and ref2va use different checkpoint
partitions, so they remain separate cases, each issuing two requests to its
own server.
The T2VA case includes VAE layerwise offload to exercise weight restoration
between requests; ref2va retains the resident VAE path.

Use `SGLANG_GEN_BASELINE=1 pytest -s ... -k <case_id>` to collect baselines in
the same sequential order, retaining the worst observed value of each metric.
`SGLANG_GEN_GT=1` with `SGLANG_GT_OUTPUT_DIR` saves
GT; repeated requests share one GT.

## Perf Baseline Generation Script

`python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`.

### Usage

Update a single case:

```bash theme={null}
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --case qwen_image_t2i
```

Select by regex:

```bash theme={null}
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --match 'qwen_image_.*'
```

Run all keys from the baseline file `scenarios`:

```bash theme={null}
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --all-from-baseline
```

Specify input/output paths and timeout:

```bash theme={null}
python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --baseline python/sglang/multimodal_gen/test/server/perf_baselines.json --out /tmp/perf_baselines.json --timeout 600
```

## Nightly Comparison Telemetry

The nightly comparison reports the median of three client-side measurements.
SGLang requests also write server perf dumps for stage-level timing. An
unreadable dump does not invalidate a completed client measurement, but the
dashboard reports server sample coverage (for example, `2/3`) and computes the
server total and stage medians only from readable dumps. Treat incomplete
coverage as a telemetry warning rather than a performance regression by itself.


# ComfyUI plugin
Source: https://docs.sglang.io/docs/sglang-diffusion/comfyui

Use SGLang Diffusion from ComfyUI in server mode or as a per-step DiT backend.

The [ComfyUI SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion) has two modes.

**Server mode** talks to a standalone `sglang serve` process over HTTP. ComfyUI sends prompts and receives images or video. This is the same path as the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api).

**Integrated mode** keeps ComfyUI's CLIP, VAE, and sampler loop. SGLang loads only the DiT and runs one forward per sampler step. The worker starts the native Flux / Qwen-Image / Z-Image / MiniMax-H3 pipeline under `--comfyui-mode`. A single-file ComfyUI `.safetensors` (or a GGUF overlay) is loaded through a checkpoint spec. Each `apply_model` call is translated by a per-model adapter.

Integrated mode does not ship a separate `comfyui_*` pipeline class per model.

## Install the plugin

1. Install `sglang[diffusion]`. See [Installation](/docs/sglang-diffusion/installation).
2. Copy `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion` into ComfyUI's `custom_nodes/` directory.
3. Restart ComfyUI.

Example workflows live next to the plugin under `workflows/`.

## Integrated mode

1. Load the DiT with `SGLDiffusion UNET Loader`.
2. Set `num_gpus`, `tp_size`, `model_type`, or compile flags with `SGLDiffusion Options`.
3. Connect the loaded model to a standard ComfyUI sampler.

Supported integrated-mode families: Flux, Qwen-Image, Z-Image, MiniMax-H3. Qwen-Image edit is experimental.

## How a sampler step reaches the worker

The plugin README has the [architecture diagram](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion#architecture). The hop is:

1. A model adapter packs ComfyUI tensors into an SGLang `Req`.
2. Local ZMQ replaces CUDA tensors with IPC handles so latents stay on GPU.
3. Rank 0 materializes the handles. Multi-rank `--comfyui-mode` then detaches CUDA tensors and broadcasts them over NCCL. The general SP / CFG / TP path is still the original `broadcast_pyobj`.
4. The worker pipeline keeps `transformer` plus a pass-through scheduler. After the first step, conditioning stays in a worker session; later steps send latents and the timestep.

Integrated mode currently supports:

| Family | ComfyUI `model_type` | Native pipeline | Example checkpoints |
| - | - | - | - |
| Flux | `flux` | `FluxPipeline` | `FLUX.1-dev` |
| Z-Image | `lumina2` | `ZImagePipeline` | `Z-Image-Turbo` |
| Qwen-Image | `qwen_image` | `QwenImagePipeline` | `Qwen-Image`, `Qwen-Image-2512` |
| Qwen-Image edit | `qwen_image_edit` | `QwenImageEditPlusPipeline` | `Qwen-Image-Edit-2511` (experimental) |
| MiniMax-H3 | `minimax_h3` | `MiniMaxH3Pipeline` | `minimax_h3_fl2va_*.safetensors`, `minimax_h3_ref2va_*.safetensors` |

MiniMax-H3 example graphs: `minimax_h3_t2v_sgld.json` (T2V / I2VA / FL2VA on the FL2VA DiT), `minimax_h3_r2v_sgld.json` (reference-to-video; use a `ref2va` file), `minimax_h3_t2v_sgld_upscaler.json` (two-pass 3D ×2). Set `SGLDOptions.model_type` to `minimax_h3`. On 24 GB, keep the 32B CLIP on CPU and prefer pruned BF16, GGUF, or `convrot_int8` with DiT layerwise offload. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#comfyui-integrated-mode).

To add a model, register a checkpoint spec under `runtime/loader/comfyui_checkpoints/` and a `ComfyUIModelAdapter`. Do not add another ComfyUI pipeline class.


# Supported Models
Source: https://docs.sglang.io/docs/sglang-diffusion/compatibility_matrix

Browse model families and public checkpoints supported by SGLang Diffusion.

## Supported model inventory

Use a listed checkpoint as `--model-path` with `sglang generate` or
`sglang serve`. This registry-backed list contains known public entry points;
family detection may also support compatible local directories. Open the linked
Cookbook recipe for launch commands, optimizations, adapters, and model-specific
notes.

**Image and 3D**

**Video and audio**

**World and action**


# Contributing to SGLang Diffusion
Source: https://docs.sglang.io/docs/sglang-diffusion/contributing



This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`).

## Contributor Guides

* [Support New Models](/docs/sglang-diffusion/support_new_models): implementation guide for adding new diffusion pipelines
* [Helper ownership](/docs/sglang-diffusion/support_new_models#place-helpers-with-their-owners): where to put shared and model-specific utilities
* [CI Performance](./ci_perf): update and regenerate perf baselines
* [SGLang Plugin System](/docs/hardware-platforms/plugin): package a hardware platform outside the SGLang repository

## On AI-Assisted ("Vibe Coding") PRs

Vibe-coded PRs are welcome — we judge code quality, not how it was produced. The bar is the same for all PRs:

* **No over-commenting.** If the name says it all, skip the docstring.
* **No over-catching.** Don't guard against errors that virtually never happen in practice.
* **Test before submitting.** AI-generated code can be subtly wrong — verify correctness end-to-end.

## Pull request titles

Follow the repository-wide [PR title convention](/docs/developer_guide/contribution_guide#pull-request-titles). Use `diffusion` as the scope, or `diffusion/<submodule-or-model>` when a more specific scope helps.

```text theme={null}
[Perf] diffusion/scheduler: Reduce scheduling overhead
[Fix] diffusion/cli: Fix argument parsing
[Feat] diffusion/wan: Add model support
[Doc] diffusion: Update the contribution guide
[Perf] [NVIDIA] diffusion/attention: Optimize attention kernels
```

Put the change type in the first bracketed tag. `diffusion` belongs in the scope, not in a hardware tag or a separate `[Diffusion]` prefix. Omit the hardware tag unless the change is hardware-specific.

## Performance Reporting

For PRs that impact **latency**, **throughput**, or **memory usage**, you **should** provide a performance comparison report.

### How to Generate a Report

1. **Baseline**: run the benchmark (for a single generation task)
   ```bash theme={null}
   $ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path baseline.json
   ```

2. **New**: run the same benchmark, without modifying any server\_args or sampling\_params
   ```bash theme={null}
   $ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path new.json
   ```

3. **Compare**: run the compare script, which will print a Markdown table to the console
   ```bash theme={null}
   $ python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json [new2.json ...]
   ### Performance Comparison Report
   ...
   ```

4. **Paste**: paste the table into the PR description

## CI-Based Change Protection

Consider adding tests to the `pr-test` or `nightly-test` suites to safeguard your changes, especially for PRs that:

* support a new model
  * add a testcase for this new model to `testcase_configs.py`
* support or fix important features
* significantly improve performance

Please run the according testcase, then update/add the baseline to `perf_baselines.json` by following the instruction in console if applicable.

See [test](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/test) for examples

### Host-sync regression tests

A denoising step or a DiT forward should not block the host on the device. One `item()`, `float(t)`, `if t:`, `.cpu()`, `nonzero` or pageable `torch.tensor(..., device="cuda")` per step stops the host from queueing the next step while the GPU runs this one. `test/unit/host_sync_guard.py` has three guards for unit tests:

* `no_cuda_sync(allow=...)` and `assert_no_cuda_sync(fn, ...)` run the code under `torch.cuda.set_sync_debug_mode` and fail with the call site of every sync whose issuing function is not listed in `allow` (`"<file suffix>:<function>"`). They skip without CUDA. Call the code once outside the guard first, so first-use setup is not counted.
* `SyncOnRead` is a CPU stand-in for device tensors (timesteps, sigmas, masks): reading its values on the host raises. Run the step once with plain tensors and once with stand-ins, and require bitwise-equal outputs.
* `forbid_host_syncs()` is a dispatch mode for code that builds its own tensors: it fails on the ops that sync on CUDA whatever tensor they read (scalar reads, `nonzero`, boolean-mask indexing, data-dependent output shapes).

`test_host_sync.py` shows both on the shared denoising loop. Unit tests run in the `multimodal-gen-unit-test` CI job on an H100, so the CUDA guard runs there.

## Out-of-tree platform changes

Use the `sglang.multimodal_gen.platforms` entry-point group when a hardware integration can live in a separately installed package. The [plugin guide](/docs/hardware-platforms/plugin#add-diffusion-support) describes the activation callback, diffusion platform contract, and selection behavior.

When you change the platform interface or discovery implementation in SGLang:

* Add focused CPU-only tests with a fake OOT platform.
* Run the relevant SRT plugin tests when changing shared hook-registry behavior.
* Document workload-specific kernels or optimized paths that still require vendor integration.


# Deployment and Performance Modes
Source: https://docs.sglang.io/docs/sglang-diffusion/deployment_cookbook

Choose component residency, FSDP, CFG parallelism, SP, TP, and performance-mode presets in SGLang Diffusion.

This page gives practical defaults for choosing `--performance-mode`, component residency, FSDP, CFG parallelism, SP, and TP.

## Quick Rule

Use the simplest setting that fits your memory target:

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Goal</th>
      <th>Recommended setting</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Fastest single-GPU run when the model fits</td>
      <td>Use resident components and do not use FSDP.</td>
    </tr>

    <tr>
      <td>Lower single-GPU memory usage</td>
      <td>Use component offload, then layerwise offload when a complete component still does not fit comfortably.</td>
    </tr>

    <tr>
      <td>Faster multi-GPU Qwen/Wan CFG generation</td>
      <td>Use FSDP with CFG parallelism and keep the sharded component resident.</td>
    </tr>

    <tr>
      <td>Sequence length or video-shape scaling</td>
      <td>Use SP/Ulysses/Ring when the model benefits from sequence parallelism.</td>
    </tr>

    <tr>
      <td>TP compatibility or encoder-heavy paths</td>
      <td>Set TP explicitly; do not treat TP as the default latency optimization.</td>
    </tr>
  </tbody>
</table>

Base the decision on available memory on the selected GPU(s).

* For multi-GPU deployment: the least-free selected GPU is the bottleneck. A busy 80GiB GPU can behave like a much smaller GPU.
* For single-GPU deployment: FSDP shards weights across multiple GPUs. It is not useful for keeping a single-GPU deployment on one GPU; use component or layerwise offload instead.

## Health Probes

Use `/liveness` to check that the HTTP process is alive and `/health` to check
that the server is ready for inference. During server-based warmup, `/liveness`
returns `200` while `/health` returns `503`. Configure the startup probe with a
failure budget large enough for model loading and compilation:

```yaml theme={null}
startupProbe:
  httpGet:
    path: /health
    port: 30010
  periodSeconds: 10
  failureThreshold: 180
readinessProbe:
  httpGet:
    path: /health
    port: 30010
livenessProbe:
  httpGet:
    path: /liveness
    port: 30010
```

See [Health endpoints](/docs/sglang-diffusion/api/cli#health-endpoints) for the
status-code contract and warmup-mode behavior.

## Checkpoint read failures

Safetensors header validation fails without deleting checkpoint files, symlinks,
or shared cache blobs. A read error can indicate permissions or storage I/O
problems, not just an incomplete download. Check the underlying error before
repairing or re-downloading weights. Do not remove shared cache files while other
replicas may be using them.

## Stable Model Identity

Use `--served-model-name` when the public model name must remain stable across replicas, hosts, or checkpoint mount paths:

```bash theme={null}
sglang serve \
  --model-path /mnt/checkpoints/Qwen-Image \
  --model-id Qwen-Image \
  --served-model-name image-production \
  --port 30010
```

The resolved public name follows `--served-model-name`, then `--model-id`, then `--model-path`. `--model-id` remains an internal model registry and configuration-resolution hint; it is not a replacement for a deployment alias. The resolved name is exposed through `/server_info` and `/v1/models` and is used by video and action responses when a request does not supply its own model.

See [OpenAI API: Served model name](/docs/sglang-diffusion/api/openai_api#served-model-name) for discovery and retrieval examples.

## Performance Modes

`--performance-mode` applies safe presets without overriding explicit offload, FSDP, or parallelism flags. `auto` is the default. Use `manual` when you need to keep performance-related server args under explicit user control. `--mode` is a short alias.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Mode</th>
      <th>Meaning</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`manual`</td>
      <td>Keeps performance-related server args under explicit user control.</td>
    </tr>

    <tr>
      <td>`auto`</td>
      <td>Default. Keeps every component resident when the least-free GPU has at least twice the model's per-GPU weights free (sized from the cached checkpoint; not with batching or on shared-memory devices). Otherwise keeps legacy safe offload defaults and uses FSDP/CFG only on validated multi-GPU deployments where FSDP can replace DiT offload.</td>
    </tr>

    <tr>
      <td>`speed`</td>
      <td>Favors GPU-resident execution for lower latency and higher throughput. Disables CPU offload when unset. `torch.compile` stays off unless the model has a validated default or it is enabled explicitly; may OOM.</td>
    </tr>

    <tr>
      <td>`memory`</td>
      <td>Favors lower GPU memory. Uses component offload, or Wan/MOVA layerwise DiT offload when supported.</td>
    </tr>
  </tbody>
</table>

`auto` checks selected GPU memory before applying FSDP. In multi-GPU runs it uses the least available memory across selected GPUs, and only turns on FSDP automatically when doing so can replace DiT offload. For image workloads with at least 45 GiB available per selected GPU, it keeps the repeatedly reused DiT resident and uses layerwise offload for large auxiliary encoders; below that threshold it keeps the DiT offloaded. Model-specific components such as VAEs become resident only when their configured memory threshold is met. `memory` instead keeps the VAE in its default layerwise set to maximize memory headroom; use `--component-residency vae=resident` for a measured recipe with sufficient capacity. Video DiT residency remains model- and workload-specific because frame count and resolution change its peak memory substantially. When the model default uses CFG and the user did not set a parallelism policy, `auto` may also enable CFG parallelism. `speed` intentionally does not check memory; it is the mode for users who prefer latency/throughput and accept OOM risk. It keeps `torch.compile` disabled by default because its effect varies by model and workload. A model-specific deployment config may enable a validated compile path, and `--enable-torch-compile true` always opts in explicitly.

The modes tune native pipeline components declared to the component residency manager. DiTs, text/image encoders, VAEs, vocoders, adapters, and upsamplers can use layerwise offload when their native module declares its executable layer structure. Explicitly selecting an unsupported component fails at startup instead of falling back to another residency mode.

For direct control, assign one of `resident`, `component-offload`, `snapshot-offload`, or `layerwise-offload` with `--component-residency COMPONENT=MODE`:

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --component-residency dit=layerwise-offload text_encoder=component-offload vae=resident
```

Existing per-component CPU-offload and layerwise flags remain supported. Canonical selectors override matching legacy settings only; unmatched legacy settings and automatic defaults remain effective. See [Component Residency](/docs/sglang-diffusion/api/cli#component-residency) for the complete precedence rules.

When `torch.compile` is enabled, `--offload-during-compile` stays on by default. During compile warmup it temporarily offloads the DiT and evicts resident non-DiT components so `max-autotune` fits on tighter-memory GPUs, then restores the configured serving residency before real traffic.

Breakable CUDA graph is a separate manual opt-in for supported image pipelines. If you enable `--enable-breakable-cuda-graph`, declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures.

> [!NOTE]
>
> The preset is intentionally coarse. A future continuous value such as `0.0` to `1.0` could express the speed-memory tradeoff more precisely, but it would need model-specific memory models and clearer user expectations. Until then, use the preset plus explicit flags for overrides.

Examples:

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --num-gpus 2 \
  --performance-mode auto
```

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --performance-mode memory
```

Explicit flags win over the mode:

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --num-gpus 2 \
  --performance-mode auto \
  --use-fsdp-inference false
```

In this example, `auto` will not re-enable FSDP. The same applies to parallelism; for example, `--enable-cfg-parallel false` keeps CFG parallelism disabled.

## Interpreting The Levers

**Resident** keeps the complete component on the accelerator. It is usually fastest when memory is sufficient.

**Component offload** keeps a complete component on CPU between declared uses. It is simple and robust, but each use pays a whole-component transfer.

**Layerwise offload** streams the declared layers of any supported native weighted component. It lowers peak accelerator memory further, but may increase latency and lower throughput.

### Which components to stream

`--layerwise-offload-components` takes a list, and more is not better. Streaming pays one host-to-device transfer of a component's weights **every time that component runs**, overlapped with that run's compute. What matters is how often it runs per request, not how large it is.

* The **DiT** runs once per denoising step. Its transfer amortizes over every step and hides behind attention, so it is the component layerwise offload is for.
* A **video VAE** runs once per temporal chunk, not once: H3's decoder re-runs whole for each chunk, so streaming its blocks pays their transfer per chunk.
* A **text encoder** runs once. Keep it resident if it fits; otherwise stream it with its blocks resident.

There are three placements, not two. A component can be in the list and still hold its blocks — `--layerwise-offload-components dit,text_encoder,vae` together with `--layerwise-resident-layers video_vae=36` transfers them once instead of per pass, and gives the VRAM back when the component finishes rather than holding it through denoise. On MiniMax-H3 at 864x480 / 124 frames that is 13 s of decode against 150 s streamed.

That hand-back is the default lifetime, `forward`, and it is what lets the DiT's resident layers and the VAE's share one budget: each set is on the GPU only while its component runs. When both sets fit at once, `--layerwise-residency-lifetime video_vae=permanent` (or `--dit-layerwise-residency-lifetime permanent` for the DiT) places them at load instead — no transfer per request, and no host copy pinned for a reload that never comes. The trade is peak memory: `permanent` adds every component's resident set together, `forward` only the largest. On a card where the sum does not fit, `permanent` fails at startup rather than slowly; keep `forward` there.

On a consumer card the trade is usually bad, so measure before reaching for it. Qwen-Image-2.1 on one RTX 5090 (1024px, 40 steps, three interleaved rounds, five requests each), making the text encoder's resident set permanent:

| `text_encoder` resident | p50 | vs `forward` | peak at load | of a 31.4 GiB card |
| - | - | - | - | - |
| none (default) | 13.53 s | — | 25168 MiB | 78% |
| 0.3 permanent | 13.45 s | −0.6% | 29570 MiB | 92% |
| 0.5 permanent | 13.41 s | −0.8% | 31480 MiB | 98% |
| 0.8 permanent | 13.37 s | −1.2% | 32092 MiB | 100% |

Every round reproduces to 0.1% and the ordering is consistent, so the win is real — and it is 0.6% to 1.2% in exchange for nearly all of the card's remaining headroom, on a machine that also has a desktop to run. The faster the card, the smaller the share the saved transfer represents: the same flag measured −2.8% on a slower box whose baseline was 14.14 s. Output stays within the usual floating-point reordering (0.5 is bit-identical to the default; 0.8 deviates by at most 1/255 with no pixel beyond one LSB).

`permanent` earns its memory where one use makes many passes and the sum still fits — a datacenter card, or a component whose set is small next to the budget.

So do not read "drop it from the list" as the fix for a one-shot component: that keeps it resident for the whole process. With no resident blocks on either side, moving H3's 10.4 GB video VAE out of the list took decode from 39.9 s to 5.3 s on one RTX 4090 — but it also added 5.8 GB of peak, which is the budget a 12 GB card does not have.

### How deep to prefetch

`--dit-offload-prefetch-size` is not monotonic. Deeper prefetch hides more of the copy, but its staging buffers crowd out activations, so latency turns back up and peak memory keeps climbing. On the H3 configuration above, 1/2/3/4 layers measured 17.3 / 15.3 / 15.8 / 16.7 s of denoise, with the deepest setting reaching 96% of a 24 GB card. Sweep two or three values rather than trusting the default.

### When not to bother

Before tuning residency or prefetch, measure whether the transfer is exposed — and measure it rather than inferring it from the checkpoint size. Bytes over bandwidth bounds what *could* be exposed, not what is, because prefetch exists to hide exactly that.

A 1.3B video DiT moves about 2.6 GB a step, on the order of a tenth of a one-second step if none of it overlapped, and its residency settings measure flat: it overlaps, so there is nothing to recover. A 50-layer model at 1.4 GB a layer moves 66 GB a step, and there the same flags decide whether the model runs at all. Sweep two or three values on the target configuration and keep the measured winner.

**FSDP** shards DiT weights across multiple GPUs and all-gathers weights during forward. It can reduce DiT CPU offload cost on multi-GPU deployments, especially for validated Wan I2V workloads.

FSDP sharding granularity matters. SGLang Diffusion prefers sharding direct repeated transformer block entries such as `transformer_blocks.0` or `blocks.0`. Coarser sharding lowers wrapper count but can increase all-gather peak memory; finer sharding can reduce transient memory but adds communication and scheduling overhead. If a model does not define an explicit sharding rule, the loader falls back to repeated block class names and common direct numbered block paths.

**CFG parallelism** splits positive and negative CFG branches across GPUs. For Qwen/Wan workloads with normal step counts, this is the most reliable multi-GPU speedup observed so far.

**SP/Ulysses/Ring** splits sequence work. It can help video workloads, but validated Qwen/Wan runs showed CFG parallelism outperforming SP for latency.

**TP** is supported for compatibility and some model structures, but current measurements do not make it the default latency path for Qwen/Wan.

## Current Benchmark Takeaways

Observed regular-scale trends:

* Z-Image: single-GPU no-offload was faster than FSDP/SP in the tested setting; keep FSDP off unless memory or parallelism requires it.
* Qwen-Image: keep the default non-FSDP path unless a specific FSDP/SP/Ring setting has been benchmarked on the target hardware.
* Wan: FSDP can replace DiT offload on validated multi-GPU workloads, while text/image encoders may still need component offload. Keep model-specific precision checks before making FSDP automatic for a path.
* Component offload mainly reduced memory; it did not improve latency in the tested no-offload-vs-offload runs.

Always benchmark with your actual resolution, frame count, step count, and GPU type before locking production defaults.


# Disaggregated Diffusion Pipeline
Source: https://docs.sglang.io/docs/sglang-diffusion/disaggregation



Split a monolithic text-to-video/image pipeline into independent **Encoder**, **Denoiser**, and **Decoder** roles, each running on its own GPU(s). A central **DiffusionServer** routes requests through the pipeline.

## Quick Start

Disaggregation is controlled by a single flag: `--disagg-role`. Each component is launched independently, just like LLM PD disaggregation.

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>`--disagg-role`</th>
      <th>What it runs</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`monolithic`</td>
      <td>(Default) Standard single-server mode</td>
    </tr>

    <tr>
      <td>`encoder`</td>
      <td>All stages with the default `RoleType.ENCODER` affinity: `InputValidationStage`, `TextEncodingStage` (plus `ImageEncodingStage` / `ImageVAEEncodingStage` for image-conditioned pipelines), `LatentPreparationStage`, `TimestepPreparationStage`, and any model-specific "before denoising" stage (e.g. `QwenImageLayeredBeforeDenoisingStage`, `GlmImageBeforeDenoisingStage`).</td>
    </tr>

    <tr>
      <td>`prompt_encoder` / `image_encoder`</td>
      <td>Separate text and reference-image encoding stages (currently FLUX.3 Image).</td>
    </tr>

    <tr>
      <td>`denoiser`</td>
      <td>`DenoisingStage` (and its subclasses: `CausalDMDDenoisingStage`, `DmdDenoisingStage`, `LTX2AVDenoisingStage`, `LTX2RefinementStage`, `Hunyuan3DShapeDenoisingStage`, ...) — the DiT forward loop plus the scheduler stepping it drives.</td>
    </tr>

    <tr>
      <td>`decoder`</td>
      <td>`DecodingStage` (VAE decode) and its subclasses (`LTX2AVDecodingStage`, `HeliosDecodingStage`, ...).</td>
    </tr>

    <tr>
      <td>`server`</td>
      <td>DiffusionServer head node + HTTP server (no GPU)</td>
    </tr>
  </tbody>
</table>

> Each stage declares its role via the `role_affinity` property on `PipelineStage` (default `ENCODER`). Non-monolithic pipelines only instantiate matching stages. The combined `encoder` role also includes stages with `PROMPT_ENCODER` or `IMAGE_ENCODER` affinity.

### Single-Machine Example (Verified)

The following commands have been tested end-to-end on an 8×H200 machine with
`Wan-AI/Wan2.1-T2V-1.3B-Diffusers`. Each role runs on a separate GPU via
`--base-gpu-id`; the `server` head node requires no GPU.

```bash theme={null}
# Terminal 1: Encoder (GPU 0)
sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
    --disagg-role encoder \
    --disagg-server-addr tcp://127.0.0.1:19655 \
    --scheduler-port 19000 \
    --num-gpus 1 --base-gpu-id 0

# Terminal 2: Denoiser (GPU 1)
sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
    --disagg-role denoiser \
    --disagg-server-addr tcp://127.0.0.1:19655 \
    --scheduler-port 19001 \
    --num-gpus 1 --base-gpu-id 1

# Terminal 3: Decoder (GPU 2)
sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
    --disagg-role decoder \
    --disagg-server-addr tcp://127.0.0.1:19655 \
    --scheduler-port 19002 \
    --num-gpus 1 --base-gpu-id 2

# Terminal 4: DiffusionServer head (no GPU, receives HTTP requests)
sglang serve --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
    --disagg-role server \
    --encoder-urls  "tcp://127.0.0.1:19000" \
    --denoiser-urls "tcp://127.0.0.1:19001" \
    --decoder-urls  "tcp://127.0.0.1:19002" \
    --host 0.0.0.0 --port 22000 \
    --scheduler-port 19655

# Send request (video generation)
curl http://127.0.0.1:22000/v1/videos \
    -H "Content-Type: application/json" \
    -d '{"model": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers", "prompt": "A curious raccoon exploring a garden, cinematic", "size": "832x480"}'
```

> **Tested result (8×H200):**
> Encoder 2.3 s (TextEncoding) → Denoiser 312.8 s (50 steps, layerwise offload) → Decoder 7.1 s (VAE decode).
> Total \~322 s for 81-frame 1024×1024 video.

> **Tip:** `--base-gpu-id` controls which physical GPU the role uses.
> Encoder and Decoder can share a GPU (e.g. both `--base-gpu-id 0`) to save resources,
> but make sure the combined GPU memory is sufficient.

### Multi-Machine Example

The exact same CLI pattern — just replace `127.0.0.1` with actual IPs and add
RDMA flags for direct transfer:

```bash theme={null}
# Machine A (10.0.0.1): Encoder
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \
    --disagg-role encoder \
    --disagg-server-addr tcp://10.0.0.4:19655 \
    --scheduler-port 19000 \
    --num-gpus 1 \
    --disagg-p2p-hostname 10.0.0.1 --disagg-ib-device mlx5_0

# Machine B (10.0.0.2): Denoiser (4 GPUs with SP)
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \
    --disagg-role denoiser \
    --disagg-server-addr tcp://10.0.0.4:19655 \
    --scheduler-port 19001 \
    --num-gpus 4 --denoiser-sp 4 --denoiser-ulysses 2 --denoiser-ring 2 \
    --disagg-p2p-hostname 10.0.0.2 --disagg-ib-device mlx5_0

# Machine C (10.0.0.3): Decoder
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \
    --disagg-role decoder \
    --disagg-server-addr tcp://10.0.0.4:19655 \
    --scheduler-port 19002 \
    --num-gpus 1 \
    --disagg-p2p-hostname 10.0.0.3 --disagg-ib-device mlx5_0

# Machine D (10.0.0.4): DiffusionServer head
sglang serve --model-path Wan-AI/Wan2.1-T2V-14B-Diffusers \
    --disagg-role server \
    --encoder-urls  "tcp://10.0.0.1:19000" \
    --denoiser-urls "tcp://10.0.0.2:19001" \
    --decoder-urls  "tcp://10.0.0.3:19002" \
    --host 0.0.0.0 --port 30000 \
    --scheduler-port 19655 \
    --disagg-dispatch-policy max_free_slots
```

> ZMQ handles startup order gracefully — instances and head can start in any order.

## FLUX.3 split encoding

FLUX.3 Image supports both the three-role topology above and four independently
deployable worker pools:

```text theme={null}
T2I: HTTP head -> prompt_encoder -> denoiser -> decoder -> HTTP head
i2i: HTTP head -> prompt_encoder -> image_encoder -> denoiser -> decoder -> HTTP head
```

The prompt encoder loads only the text encoder. The image encoder loads only
the VAE for reference-image encoding; the decoder has its own VAE instance.
The denoiser loads only the DiT and creates noise from the request seed locally.
T2I bypasses the image encoder and goes directly from prompt encoder to denoiser.
Stages within one request run sequentially; different requests can occupy
different worker pools concurrently. This is not a latency optimization for a
single request: separation adds transfers and allows independent placement and
scaling. The head still serves complete image-generation requests; this is not
a standalone API for returning training embeddings or VAE latents.

Use the same FLUX.3 package, variant, and numerical settings on every process.
Each host currently needs a complete package on disk, even though a worker
loads only its own components into memory. Follow the
[FLUX.3 cookbook](/cookbook/diffusion/FLUX/FLUX.3) for model access and licensing.
For gated checkpoints, authenticate on each host with an authorized account
using `hf auth login`, or provide an accessible local package path.
Install the transfer dependencies described below on every worker.
For long prompts, increase `--disagg-transfer-pool-size` on the encoding and
denoising workers to cover their largest incoming/outgoing tensors, as well as
reserved receive slots; for example, `1073741824` reserves a 1 GiB pool.

For a Linux NVIDIA deployment with four GPUs, start the workers in one shell:

```bash theme={null}
MODEL_PATH=black-forest-labs/flux-3-image-klein
for spec in 'prompt_encoder 0 19000' 'image_encoder 1 19010' \
            'denoiser 2 19020' 'decoder 3 19030'; do
  read -r ROLE GPU WORK_PORT <<< "$spec"
  CUDA_VISIBLE_DEVICES="$GPU" sglang serve --model-path "$MODEL_PATH" \
    --num-gpus 1 --disagg-role "$ROLE" \
    --disagg-server-addr tcp://127.0.0.1:18000 \
    --disagg-p2p-hostname 127.0.0.1 \
    --scheduler-port "$WORK_PORT" --master-port "$((WORK_PORT + 50))" &
done
```

Then start the GPU-less head in another shell:

```bash theme={null}
sglang serve --model-path black-forest-labs/flux-3-image-klein \
  --disagg-role server --host 127.0.0.1 --port 30000 --scheduler-port 18000 \
  --prompt-encoder-urls tcp://127.0.0.1:19000 \
  --image-encoder-urls tcp://127.0.0.1:19010 \
  --denoiser-urls tcp://127.0.0.1:19020 \
  --decoder-urls tcp://127.0.0.1:19030
```

The existing image API is unchanged: use `/v1/images/generations` for T2I and
multipart `/v1/images/edits` for references:

```bash theme={null}
curl --fail-with-body -sS http://127.0.0.1:30000/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"A sunset over mountains","size":"1024x1024","seed":42,"response_format":"b64_json"}' \
  -o flux3-t2i.json

curl --fail-with-body -sS http://127.0.0.1:30000/v1/images/edits \
  -F 'image=@reference.png' \
  -F 'prompt=Turn the scene into a watercolor painting' \
  -F 'seed=42' -F 'response_format=b64_json' \
  -o flux3-i2i.json
```

Replace `reference.png` with your input image. Omit `size` on edits to keep the
reference dimensions, including the model's padding/crop behavior. The head
inlines locally uploaded reference files before dispatch, so GPU workers do
not need access to the head's upload directory. URL inputs must remain
reachable from the image encoder.

Specify both `--prompt-encoder-urls` and `--image-encoder-urls`, instead of
`--encoder-urls`. Each accepts semicolon-separated worker endpoints, with the
same dispatch policy as the existing pools. Alternatively, use the existing
`encoder` role and `--encoder-urls` to colocate both encoders in one worker.
Image encoder and decoder processes can also share a GPU if their combined
weights, transfer pools, and concurrent activations fit.

Across machines, replace loopback addresses with routable IPs. Advertise each
worker's IP with `--disagg-p2p-hostname`, and use that exact work endpoint in
the head's URL list. The head also listens on scheduler port +4 for prompt
encoder results and +5 for image encoder results; +1/+2/+3 retain their
existing meanings. Keep worker work/control ports (+1), internal scheduler
ports, and distributed master ports distinct. Run the internal transport on a
trusted private network, not a public interface.

The BF16 flow-map package has been validated on H200 with same-node,
multi-process Mooncake RDMA: concurrent T2I, reference-size-preserving i2i,
and both three-role and four-role topologies matched the monolithic API's
output pixels with `quality=exact`. Cross-node execution has not been validated.

## Multiple Instances per Role

Use semicolons in `--*-urls` to register multiple instances:

```bash theme={null}
# 2 encoders + 2 denoisers (4-GPU SP each) + 1 decoder
sglang serve --model-path ... --disagg-role server \
    --encoder-urls  "tcp://10.0.0.1:35000;tcp://10.0.0.2:35000" \
    --denoiser-urls "tcp://10.0.0.3:35000;tcp://10.0.0.4:35000" \
    --decoder-urls  "tcp://10.0.0.5:35000"
```

### GLM-Image distributed mode

GLM-Image can batch AR generation in the head and dispatch the resulting prior
tokens to distributed denoiser workers. Each worker runs prompt/glyph preparation,
DiT, and VAE decoding locally; no latent or embedding tensors are transferred.

The following 16-device deployment uses devices 0-1 for the external AR server
and devices 2-15 for 14 independent batch-1 Cache-DiT denoisers.

```bash theme={null}
# Run 14 distributed denoisers
DISAGG_SERVER="tcp://127.0.0.1:19655"
MODEL_PATH="zai-org/GLM-Image"
BASE_MASTER_PORT=29005

export SGLANG_CACHE_DIT_FN=2
export SGLANG_CACHE_DIT_BN=1
export SGLANG_CACHE_DIT_WARMUP=4
export SGLANG_CACHE_DIT_RDT=0.4
export SGLANG_CACHE_DIT_MC=4
export SGLANG_CACHE_DIT_TAYLORSEER=true
export SGLANG_CACHE_DIT_TS_ORDER=2
export SGLANG_CACHE_DIT_ENABLED=true

worker_pids=()

cleanup() {
    trap - EXIT
    if ((${#worker_pids[@]})); then
        kill "${worker_pids[@]}" 2>/dev/null || true
        wait "${worker_pids[@]}" 2>/dev/null || true
    fi
}

trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

for i in $(seq 2 15); do
    scheduler_port=$((19001 + i))
    master_port=$((BASE_MASTER_PORT + i))

    sglang serve \
        --model-path "$MODEL_PATH" \
        --disagg-role denoiser \
        --disagg-server-addr "$DISAGG_SERVER" \
        --srt-encoder-url http://127.0.0.1:30020 \
        --scheduler-port "$scheduler_port" \
        --master-port "$master_port" \
        --num-gpus 1 \
        --base-gpu-id "$i" \
        --denoiser-sp 1 \
        --cfg-parallel-size 1 \
        --batching-max-size 1 \
        --dit-cpu-offload false \
        --attention-backend fa &
    worker_pids+=("$!")
done

# Stop all denoisers if any worker exits or fails during startup.
wait -n "${worker_pids[@]}"
```

Run the external AR server:

```bash theme={null}
sglang serve \
  --model-path ./zai-org/GLM-Image/vision_language_encoder/ \
  --tokenizer-path ./zai-org/GLM-Image/processor/ \
  --enable-multimodal \
  --cuda-graph-max-bs-decode 28 \
  --device npu \
  --attention-backend ascend \
  --disable-fast-image-processor \
  --tp-size 2 \
  --host 0.0.0.0 \
  --port 30020 \
  --mem-fraction-static 0.8
```

Run the public head. `--encoder-urls` and `--decoder-urls` are intentionally
omitted for this topology.

```bash theme={null}
sglang serve \
  --model-path zai-org/GLM-Image \
  --disagg-role server \
  --srt-encoder-url http://127.0.0.1:30020 \
  --srt-encoder-timeout 300 \
  --denoiser-urls "tcp://127.0.0.1:19003;tcp://127.0.0.1:19004;tcp://127.0.0.1:19005;tcp://127.0.0.1:19006;tcp://127.0.0.1:19007;tcp://127.0.0.1:19008;tcp://127.0.0.1:19009;tcp://127.0.0.1:19010;tcp://127.0.0.1:19011;tcp://127.0.0.1:19012;tcp://127.0.0.1:19013;tcp://127.0.0.1:19014;tcp://127.0.0.1:19015;tcp://127.0.0.1:19016" \
  --batching-mode dynamic \
  --batching-max-size 28 \
  --batching-delay-ms 30 \
  --enable-batching-metrics \
  --host 0.0.0.0 \
  --port 30052 \
  --scheduler-port 19655 \
  --output-path ./outputs
```

Workers return decoded pixels to the head, which saves and serves the final
files from `--output-path`. The PR benchmark used
[longtext-bench.zip](https://github.com/user-attachments/files/29779516/longtext-bench.zip):

```bash theme={null}
python fetch_images.py \
  --base-url http://localhost:30052/v1 \
  --model GLM-Image-distributed-test \
  --output-dir generated_images \
  --max-concurrency 56
```

## Port Convention

Result endpoints are derived deterministically from the head node's `--scheduler-port` (default: 5555):

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Socket</th>
      <th>Port</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>DS frontend (ROUTER)</td>
      <td>`scheduler_port`</td>
    </tr>

    <tr>
      <td>Encoder result (PULL)</td>
      <td>`scheduler_port + 1`</td>
    </tr>

    <tr>
      <td>Denoiser result (PULL)</td>
      <td>`scheduler_port + 2`</td>
    </tr>

    <tr>
      <td>Decoder result (PULL)</td>
      <td>`scheduler_port + 3`</td>
    </tr>

    <tr>
      <td>Prompt encoder result (PULL, split encoding)</td>
      <td>`scheduler_port + 4`</td>
    </tr>

    <tr>
      <td>Image encoder result (PULL, split encoding)</td>
      <td>`scheduler_port + 5`</td>
    </tr>
  </tbody>
</table>

Role instances derive their result endpoint automatically from `--disagg-server-addr`. No manual endpoint configuration needed.

## Transfer Mechanism

Tensor data between adjacent roles is transferred via a P2P transfer engine. The DiffusionServer routes control messages and request metadata; tensors flow directly between workers. Uploaded reference files are forwarded inline as request metadata.

**mooncake-transfer-engine** is required for disaggregated diffusion. It provides RDMA for direct GPU-to-GPU data movement.

The GLM-Image distributed mode is an exception: it relays only prior token IDs
and request metadata over ZMQ and does not require Mooncake.

```bash theme={null}
pip install mooncake-transfer-engine
```

### Transfer Flow

1. **Sender** stages output tensors in its transfer buffer, then notifies DiffusionServer.
2. **DiffusionServer** selects a downstream worker and reserves one of its preallocated receive slots. If none fits, it sends `transfer_alloc` and waits for `transfer_allocated`.
3. **DiffusionServer** sends `transfer_push` to the sender with the receiver's session and destination address.
4. **Sender** pushes the tensors through Mooncake and reports `transfer_pushed` after completion.
5. **DiffusionServer** sends `transfer_ready` to the receiver and releases the sender's scheduling slot.
6. **Receiver** loads the tensors on its transfer stream and executes its stages. Receive prefetch can overlap the previous request's compute.

Decoder results (final output) flow back through DiffusionServer as raw ZMQ frames to the HTTP client.

### Failure Isolation

The head rejects malformed scheduler requests and duplicate active request IDs
without replacing the original request. Each scheduler message carries one
request; use concurrent API calls for independent requests, or `n` for multiple
outputs from one request.

Tensor decoding errors fail only the associated request when its ID can be
recovered. Results without an identifiable request are logged and discarded;
affected requests still rely on `--disagg-timeout`. Reported worker errors release
their scheduling capacity and reserved receive slots so later requests can run.

Keep scheduler and worker ports on a trusted private network. This error handling
does not make the pickle-based scheduler protocol safe for untrusted clients, or
provide durable jobs and transparent recovery after a head restart.

### RDMA Flags

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Flag</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--disagg-p2p-hostname`</td>
      <td>`127.0.0.1`</td>
      <td>RDMA-reachable hostname/IP of this instance</td>
    </tr>

    <tr>
      <td>`--disagg-ib-device`</td>
      <td>`None`</td>
      <td>InfiniBand device (e.g., `mlx5_0`, `mlx5_roce0`)</td>
    </tr>

    <tr>
      <td>`--disagg-transfer-pool-size`</td>
      <td>256 MiB</td>
      <td>Pinned memory pool per instance</td>
    </tr>
  </tbody>
</table>

Set `--disagg-p2p-hostname` to the actual IP on each machine. For multi-machine, `--disagg-ib-device` specifies the RDMA NIC.

## Per-Role Parallelism

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Flag</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--encoder-tp`</td>
      <td>Encoder tensor parallelism</td>
    </tr>

    <tr>
      <td>`--denoiser-tp` / `--denoiser-sp` / `--denoiser-ulysses` / `--denoiser-ring`</td>
      <td>Denoiser parallelism</td>
    </tr>

    <tr>
      <td>`--decoder-sp`</td>
      <td>Decoder sequence parallelism</td>
    </tr>
  </tbody>
</table>

If not specified, parallelism is auto-derived from `--num-gpus`.

## Other Options

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Flag</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--disagg-timeout`</td>
      <td>`600`</td>
      <td>Timeout (seconds) for pending requests</td>
    </tr>

    <tr>
      <td>`--disagg-dispatch-policy`</td>
      <td>`round_robin`</td>
      <td>`round_robin` or `max_free_slots`</td>
    </tr>
  </tbody>
</table>

## Python API

For programmatic single-machine deployment, `launch_pool_disagg_server()` is available. It spawns its workers, and each child re-executes this script's module scope before initializing its platform, so keep the diffusion imports inside the guard:

```python theme={null}
if __name__ == "__main__":
    from sglang.multimodal_gen.runtime.launch_server import launch_pool_disagg_server
    from sglang.multimodal_gen.runtime.server_args import ServerArgs

    server_args = ServerArgs.from_kwargs(
        model_path="Wan-AI/Wan2.1-T2V-14B-Diffusers",
        denoiser_sp=4, denoiser_ulysses=2, denoiser_ring=2,
        disagg_ib_device="mlx5_0",
    )

    launch_pool_disagg_server(
        server_args,
        encoder_gpus=[[0]],
        denoiser_gpus=[[1, 2, 3, 4], [5, 6, 7, 8]],
        decoder_gpus=[[0]],
    )
```

## Architecture

```
Client ─── HTTP (port 30000) ──► FastAPI Server
                                      │
                                      ▼
                              DiffusionServer (ROUTER, scheduler_port)
                              ┌───────┼───────┐
                   PUSH work  │       │       │  PUSH work
                              ▼       │       ▼
                    Encoder[0..N]     │    Decoder[0..K]
                              │       │       ▲
                   P2P tensor │       │       │ P2P tensor
                   transfer   ▼       │       │ transfer
                          Denoiser[0..M] ─────┘
                                      │
                    PULL results ◄────┘  (decoder → DS → client)
```

### Request State Machine

```
PENDING → ENCODER_WAITING → ENCODER_RUNNING → ENCODER_DONE
                                                    │
                        DENOISING_WAITING → DENOISING_RUNNING → DENOISING_DONE
                                                                       │
                                    DECODER_WAITING → DECODER_RUNNING → DONE
```

Any state can transition to `FAILED` or `TIMED_OUT`.


# Inference Batching
Source: https://docs.sglang.io/docs/sglang-diffusion/dynamic_batching

Batch compatible native SGLang-Diffusion requests during serving.

Dynamic batching is an opt-in SGLang-Diffusion serving mode that merges compatible queued requests into one native pipeline batch. It is separate from LLM continuous batching and tokenizer batching.

Use it for concurrent T2I or T2V traffic with the same model and sampling shape. Keep singleton serving for latency-sensitive or highly mixed traffic.

## Enable

Dynamic batching is disabled by default with `--batching-max-size 1`.

For GLM-Image T2I with an external AR server, compatible requests share one
batched AR `/generate` call. DiT denoising and VAE decoding remain independent
per-request executions; GLM-Image does not batch DiT work.

```bash Command theme={null}
sglang serve \
  --model-path black-forest-labs/FLUX.1-dev \
  --port 30010 \
  --batching-mode dynamic \
  --batching-max-size 8 \
  --batching-delay-ms 5 \
  --enable-batching-metrics
```

For request formats, see the [OpenAI-Compatible API](./api/openai_api).

Use `--batching-config /path/to/batching_config.json` to load JSON rules when a model or resolution needs a lower cap than `--batching-max-size`:

```json Config theme={null}
{
  "schema_version": 1,
  "rules": [
    {
      "model_contains": "Qwen-Image",
      "resolution": "1024x1024",
      "max_batch_size": 1
    }
  ]
}
```

## Compatibility

An initial implementation of dynamic batching for T2I and T2V models can be found in [#18764](https://github.com/sgl-project/sglang/pull/18764). The current compatibility grid is below and will be updated as more coverage is added. See [Supported Models](./compatibility_matrix) for common model IDs.

`✅` means supported, `❌` means not currently supported, `?` means untested, and `-` means not applicable.

### Image

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>T2I</th>
      <th>I2I</th>
    </tr>
  </thead>

  <tbody>
    <tr><td>FLUX.1-dev</td><td>✅</td><td>-</td></tr>
    <tr><td>FLUX.2-dev</td><td>✅</td><td>❌</td></tr>
    <tr><td>FLUX.2-dev-NVFP4</td><td>?</td><td>?</td></tr>
    <tr><td>FLUX.2-Klein-4B</td><td>✅</td><td>❌</td></tr>
    <tr><td>FLUX.2-Klein-9B</td><td>?</td><td>?</td></tr>
    <tr><td>FLUX.2-Klein-Base-4B</td><td>?</td><td>?</td></tr>
    <tr><td>FLUX.2-Klein-Base-9B</td><td>?</td><td>?</td></tr>
    <tr><td>Z-Image</td><td>?</td><td>-</td></tr>
    <tr><td>Z-Image-Turbo</td><td>✅</td><td>-</td></tr>
    <tr><td>GLM-Image (external AR)</td><td>✅</td><td>-</td></tr>
    <tr><td>Qwen Image</td><td>✅</td><td>-</td></tr>
    <tr><td>Qwen Image 2512</td><td>✅</td><td>-</td></tr>
    <tr><td>Qwen Image 2.1</td><td>✅</td><td>❌</td></tr>
    <tr><td>Qwen Image Edit</td><td>-</td><td>❌</td></tr>
    <tr><td>Qwen Image Edit 2509</td><td>-</td><td>?</td></tr>
    <tr><td>Qwen Image Edit 2511</td><td>-</td><td>?</td></tr>
    <tr><td>Qwen Image Layered</td><td>?</td><td>?</td></tr>
    <tr><td>SD3 Medium</td><td>?</td><td>-</td></tr>
    <tr><td>SD3.5 Medium</td><td>?</td><td>-</td></tr>
    <tr><td>SD3.5 Large</td><td>?</td><td>-</td></tr>
    <tr><td>Hunyuan3D-2</td><td>?</td><td>-</td></tr>
    <tr><td>SANA 1.5 1.6B</td><td>✅</td><td>-</td></tr>
    <tr><td>SANA 1.5 4.8B</td><td>✅</td><td>-</td></tr>
    <tr><td>SANA 1600M 1024px</td><td>?</td><td>-</td></tr>
    <tr><td>SANA 600M 1024px</td><td>?</td><td>-</td></tr>
    <tr><td>SANA 1600M 512px</td><td>?</td><td>-</td></tr>
    <tr><td>SANA 600M 512px</td><td>?</td><td>-</td></tr>
    <tr><td>FireRed-Image-Edit 1.0</td><td>-</td><td>?</td></tr>
    <tr><td>FireRed-Image-Edit 1.1</td><td>-</td><td>?</td></tr>
    <tr><td>ERNIE-Image</td><td>?</td><td>-</td></tr>
    <tr><td>ERNIE-Image-Turbo</td><td>?</td><td>-</td></tr>
  </tbody>
</table>

Qwen Image 2.1 supports merging compatible text-to-image requests. Image edits
are not merged across requests; `n > 1` still produces multiple outputs within
one edit request. See its [cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#batching)
for deployment settings and batching tradeoffs.

### Video

<table>
  <colgroup>
    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Support</th>
    </tr>
  </thead>

  <tbody>
    <tr><td>FastWan2.1 T2V 1.3B</td><td>✅</td></tr>
    <tr><td>FastWan2.2 TI2V 5B Full Attn</td><td>❌</td></tr>
    <tr><td>Wan2.2 TI2V 5B</td><td>❌</td></tr>
    <tr><td>Wan2.2 T2V A14B</td><td>✅</td></tr>
    <tr><td>Wan2.2 I2V A14B</td><td>❌</td></tr>
    <tr><td>HunyuanVideo</td><td>❌</td></tr>
    <tr><td>FastHunyuan</td><td>❌</td></tr>
    <tr><td>Wan2.1 T2V 1.3B</td><td>✅</td></tr>
    <tr><td>Wan2.1 T2V 14B</td><td>✅</td></tr>
    <tr><td>Wan2.1 I2V 480P</td><td>?</td></tr>
    <tr><td>Wan2.1 I2V 720P</td><td>?</td></tr>
    <tr><td>TurboWan2.1 T2V 1.3B</td><td>✅</td></tr>
    <tr><td>TurboWan2.1 T2V 14B</td><td>✅</td></tr>
    <tr><td>TurboWan2.1 T2V 14B 720P</td><td>✅</td></tr>
    <tr><td>TurboWan2.2 I2V A14B</td><td>?</td></tr>
    <tr><td>Wan2.1 Fun 1.3B InP</td><td>?</td></tr>
    <tr><td>Helios Base</td><td>?</td></tr>
    <tr><td>Helios Mid</td><td>?</td></tr>
    <tr><td>Helios Distilled</td><td>?</td></tr>
    <tr><td>LTX-2</td><td>?</td></tr>
    <tr><td>LTX-2.3</td><td>?</td></tr>
  </tbody>
</table>

## Notes

* Requests batch only when model inputs, sampling parameters, output handling, and any configured rules are compatible.
* There is no startup probing, runtime learning, OOM retry, or automatic fallback to singletons. If a merged batch fails or cannot be split, every request in that batch receives an error.
* Batch shape can change kernels, so singleton and dynamic outputs are not expected to be bit-exact.
* Use `--enable-batching-metrics` to inspect realized batches:

```text theme={null}
Dynamic batch dispatch: size=2/8, user_max=8, queue_wait=5.12ms, stop_reason=delay
Dynamic batch dispatch: size=1/8, user_max=8, queue_wait=0.04ms, stop_reason=config_cap:1
Dynamic batch stats (last 5 dispatches): avg_size=2.80, merged_rate=60.0%, full_rate=20.0%, utilization=35.0%, wait_avg=3.21ms, wait_p95=5.12ms, top_rejects=none
```


# Encoder Parallelism
Source: https://docs.sglang.io/docs/sglang-diffusion/encoder_parallel



While the DiT denoises, the text and image encoders are idle — and while they
encode, the whole DiT replica is idle. `--encoder-parallel` decides how native
encoders use those otherwise-unused GPUs for the encoding stage. Folding applies
to native text and image encoders; within-replica batch DP currently requires an
explicitly supported native text encoder.

```bash theme={null}
--encoder-parallel {auto,fold,dp,replicate}
```

| Mode | What it does | Use when |
| - | - | - |
| `auto` | Picks folding or the existing layout for native text/image encoders, and batch DP for supported native text encoders | Default; you want the decision made per encoder |
| `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode |
| `dp` | Supported native text encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` |
| `replicate` | Keeps the encoder on its DiT TP group and encodes redundantly across the other replica ranks | You want to disable folding and batch DP |

The two accelerated modes are mutually exclusive per encoder: folding shards the
weights for the lifetime of the loaded model, so a folded encoder cannot also be
data-parallel.

## Which Mode Wins

Measured on H100 across T5 (hidden 4096), Qwen3 (2560), and CLIP-L (768) at
batch 1–8 and replica sizes 2 and 4:

* **Folding** pays when the encoder is wide enough that sharding its GEMMs beats
  the per-layer all-reduce it adds. T5 gains; Qwen3 (+35%) and CLIP-L (+50%) get
  slower, so folding is gated at hidden ≥ 4096. Its benefit also saturates as
  the replica grows, since each rank's slice keeps shrinking.
* **Data-parallel** pays only when the encode is compute-bound, which needs a
  wide encoder (hidden ≥ 1024 — CLIP-L is slower at every batch and replica
  measured) and more than one prompt in a single encode call.
* **Replication** is the right answer whenever neither condition holds, which is
  most single-request latency work.

`auto` encodes exactly these rules, so prefer it unless you are pinning a
configuration you measured yourself.

## Numerics

`replicate` matches single-GPU encoding bit-for-bit only when the encoder TP
degree is one. Both the existing DiT TP layout and `fold` can reorder parallel
reductions; they are mathematically equivalent but are not generally bitwise
identical to a single-GPU kernel.

`dp` is also not bitwise-identical: each encoder copy sees a smaller batch, so
GEMM tiling can differ from the unsplit reference. It may compose with encoder
TP: all ranks in one TP group receive the same batch slice, and corresponding TP
ranks gather outputs across the orthogonal encoder-DP group. The result remains
mathematically equivalent and deterministic for a fixed topology and batch
shape, but long video sampling can amplify small floating-point differences.

## Recommended Commands

Throughput serving. A single encode call must carry more than one prompt for DP
to engage, so raise the batching ceiling too; an encoder flag deliberately does
not change DiT batching for you:

```bash theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --model-type diffusion \
  --num-gpus 2 \
  --tp-size 1 \
  --ulysses-degree 2 \
  --encoder-parallel dp \
  --batching-max-size 2
```

Single-request latency with one wide text encoder:

```bash theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --model-type diffusion \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --encoder-parallel fold
```

Bit-exact reproducibility against a single-GPU reference:

```bash theme={null}
sglang serve \
  --model-path Qwen/Qwen-Image-2512 \
  --model-type diffusion \
  --num-gpus 2 \
  --encoder-parallel replicate
```

## Interaction With Other Flags

* **Tensor parallel**: encoder DP composes with TP. A TP group jointly encodes
  one batch slice; the orthogonal ranks inside the same pipeline replica split
  and gather the batch. A pure-TP replica has one encoder copy, so there is no
  additional batch-DP degree.
* **Data parallel**: encoder collectives never cross pipeline replicas. With
  `--dp-size > 1`, each replica independently uses its own TP/SP/CFG ranks.
* **Dynamic batching**: `dp` only pays with a wide batch. Selecting it does not
  change `--batching-max-size`; configure that separately. See
  [Inference Batching](./dynamic_batching).
* **Sequence parallelism**: SP splits the DiT latent sequence. During encoding,
  those ranks either hold encoder copies for batch DP or join a folded encoder.
  See
  [Sequence Parallelism](./ring_sp_performance).


# Environment Variables
Source: https://docs.sglang.io/docs/sglang-diffusion/environment_variables

Configure SGLang diffusion behavior with environment variables.

## Runtime

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_EXTERNAL_MODEL_PACKAGE`</td>
      <td>not set</td>
      <td>Installed package that registers out-of-tree diffusion pipelines and component models. The package is imported once in every process.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_PLATFORM_OVERRIDE`</td>
      <td>not set</td>
      <td>Select `cpu`, `cuda`, `rocm`, `mps`, `npu`, `musa`, or an installed `sglang.multimodal_gen.platforms` entry-point name. XPU remains automatic-only. When unset, SGLang Diffusion detects installed platform plugins before trying built-in platforms.</td>
    </tr>

    <tr>
      <td>`SGLANG_PLUGINS`</td>
      <td>not set</td>
      <td>Comma-separated allowlist for installed `sglang.multimodal_gen.plugins` hooks. When unset, all discovered diffusion hooks load.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_ATTENTION_CONFIG`</td>
      <td>not set</td>
      <td>Path to attention backend configuration file (JSON/YAML)</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_STAGE_LOGGING`</td>
      <td>false</td>
      <td>Enable per-stage timing logs</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_TORCH_PROFILER_DIR`</td>
      <td>not set</td>
      <td>Directory for torch profiler traces (absolute path). Enables profiling when set</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_CACHE_ROOT`</td>
      <td>`~/.cache/sgl_diffusion`</td>
      <td>Root directory for cache files</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_IPC_A2A`</td>
      <td>`true`</td>
      <td>Enable CUDA-IPC all-to-all for eligible same-host, peer-accessible TP1 + two-rank Ulysses groups. Unsupported topology falls back to NCCL; set `0` to force NCCL.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_IPC_A2A_TIMEOUT_MS`</td>
      <td>`10000`</td>
      <td>Peer-wait timeout in milliseconds. This is a hang backstop, not a per-step budget; raise it only for known long stalls such as layerwise offload.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_IPC_A2A_MAX_BUFFERS`</td>
      <td>`16`</td>
      <td>Maximum cached IPC staging-buffer shape pairs. Cap this for multi-resolution serving to bound permanent staging memory.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS`</td>
      <td>`-1`</td>
      <td>Head groups for pipelining the Ulysses exchange against dense attention over the copy engine. `-1` picks the largest count that divides the heads per rank while each group still fills the GPU; `0` keeps the sequential exchange; `N` forces `N` groups. Results are bit-identical either way.</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_RUNAI_MODEL_STREAMER`</td>
      <td>true</td>
      <td>Use Run:AI model streamer for model loading</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_CONVROT_INT8_BACKEND`</td>
      <td>`auto`</td>
      <td>Kernel backend for `convrot_int8`: `auto` takes SGLang's JIT-compiled ConvRot ops where they build and run (CC 9.0, 10.0, 12.0, 12.1 with nvcc) and `comfy_kitchen` otherwise; `jit` or `comfy_kitchen` forces one.</td>
    </tr>

    <tr>
      <td>`SGLANG_KITCHEN_INT8_MAX_ROWS`</td>
      <td>`0`</td>
      <td>Max activation rows per `convrot_int8` fused GEMM on the `comfy_kitchen` backend. `0` (default) issues one call; a positive value splits large-M GEMMs by rows.</td>
    </tr>

    <tr>
      <td>`SGLANG_KITCHEN_INT8_MIN_SPLIT_N`</td>
      <td>`8192`</td>
      <td>Minimum output features before `convrot_int8` (`comfy_kitchen` backend) splits large-M GEMMs when `SGLANG_KITCHEN_INT8_MAX_ROWS` is positive. Narrow outputs stay as a single call.</td>
    </tr>
  </tbody>
</table>

## Platform-Specific

### Apple MPS

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_USE_MLX`</td>
      <td>not set</td>
      <td>SRT only: enables the MLX serving backend. It has no effect on SGLang Diffusion, which uses PyTorch MPS.</td>
    </tr>
  </tbody>
</table>

### ROCm (AMD GPUs)

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_USE_ROCM_VAE`</td>
      <td>false</td>
      <td>Use AITer GroupNorm in VAE for improved performance on ROCm</td>
    </tr>

    <tr>
      <td>`SGLANG_USE_ROCM_CUDNN_BENCHMARK`</td>
      <td>false</td>
      <td>Enable MIOpen auto-tuning for VAE conv layers on ROCm</td>
    </tr>
  </tbody>
</table>

### Quantization

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND`</td>
      <td>not set</td>
      <td>Optional FlashInfer FP4 GEMM backend override for diffusion NVFP4. When unset, SGLang defaults to `flashinfer_trtllm`.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_ENABLE_W8A8_FP8_GEMM`</td>
      <td>false</td>
      <td>Experimental opt-in for fused W8A8 FP8 GEMM in diffusion weight-only FP8 linears. When disabled, FP8 weights are dequantized to the compute dtype before matmul. Enabling this dynamically quantizes activations to FP8 and may change output quality.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_ENABLE_MXFP8_ATTENTION`</td>
      <td>false</td>
      <td>Enable Ascend MXFP8 FA for supported non-causal self-attention layers. Applies to online `MXFP8Config` and offline ModelSlim `W8A8_MXFP8` checkpoints. Unsupported calls continue to use the regular attention path.</td>
    </tr>

    <tr>
      <td>`SGLANG_DIFFUSION_MXFP8_FA_HEAD_CHUNK_SIZE`</td>
      <td>4</td>
      <td>Maximum number of attention heads processed by each Ascend MXFP8 FA call. Smaller chunks may improve performance for large workloads but add kernel launches; the optimal value depends on the model and input shape. Set to 0 to disable head splitting.</td>
    </tr>
  </tbody>
</table>

## Caching Acceleration

These variables configure caching acceleration for Diffusion Transformer (DiT) models.
SGLang supports multiple caching strategies - see [caching documentation](./caching-acceleration) for an overview.

### Cache-DiT Configuration

See [cache-dit documentation](./cache_dit) for detailed configuration.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_CACHE_DIT_ENABLED`</td>
      <td>false</td>
      <td>Enable Cache-DiT acceleration</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_FN`</td>
      <td>1</td>
      <td>First N blocks to always compute</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_BN`</td>
      <td>0</td>
      <td>Last N blocks to always compute</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_WARMUP`</td>
      <td>4</td>
      <td>Warmup steps before caching</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_RDT`</td>
      <td>0.24</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_MC`</td>
      <td>3</td>
      <td>Max continuous cached steps</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_TAYLORSEER`</td>
      <td>false</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_TS_ORDER`</td>
      <td>1</td>
      <td>TaylorSeer order (1 or 2)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_DMD`</td>
      <td>false</td>
      <td>Enable the DMD (Dynamic Mode Decomposition) calibrator (mutually exclusive with TaylorSeer)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_DMD_HISTORY`</td>
      <td>6</td>
      <td>DMD snapshot window length (5-6 typical; needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_DMD_RANK`</td>
      <td>0</td>
      <td>DMD SVD truncation rank (0 = automatic)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_DMD_RIDGE`</td>
      <td>1e-8</td>
      <td>DMD Tikhonov regularization added to the inverted singular values</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_DMD_SVD_PRECISION`</td>
      <td>medium</td>
      <td>DMD SVD precision (low/medium/high)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SCM_PRESET`</td>
      <td>none</td>
      <td>SCM preset (none/slow/medium/fast/ultra)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SCM_POLICY`</td>
      <td>dynamic</td>
      <td>SCM caching policy</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SCM_COMPUTE_BINS`</td>
      <td>not set</td>
      <td>Custom SCM compute bins</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SCM_CACHE_BINS`</td>
      <td>not set</td>
      <td>Custom SCM cache bins</td>
    </tr>
  </tbody>
</table>

### Cache-DiT Secondary Transformer

For dual-transformer models (e.g., Wan2.2 with high/low-noise experts), these variables configure caching for the secondary transformer. Each falls back to its primary counterpart if not set.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_FN`</td>
      <td>(from primary)</td>
      <td>First N blocks to always compute</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_BN`</td>
      <td>(from primary)</td>
      <td>Last N blocks to always compute</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_WARMUP`</td>
      <td>(from primary)</td>
      <td>Warmup steps before caching</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_RDT`</td>
      <td>(from primary)</td>
      <td>Residual difference threshold</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_MC`</td>
      <td>(from primary)</td>
      <td>Max continuous cached steps</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_TAYLORSEER`</td>
      <td>(from primary)</td>
      <td>Enable TaylorSeer calibrator</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_TS_ORDER`</td>
      <td>(from primary)</td>
      <td>TaylorSeer order (1 or 2)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_DMD`</td>
      <td>(from primary)</td>
      <td>Enable the DMD calibrator (mutually exclusive with TaylorSeer)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_DMD_HISTORY`</td>
      <td>(from primary)</td>
      <td>DMD snapshot window length</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_DMD_RANK`</td>
      <td>(from primary)</td>
      <td>DMD SVD truncation rank (0 = automatic)</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_DMD_RIDGE`</td>
      <td>(from primary)</td>
      <td>DMD Tikhonov regularization term</td>
    </tr>

    <tr>
      <td>`SGLANG_CACHE_DIT_SECONDARY_DMD_SVD_PRECISION`</td>
      <td>(from primary)</td>
      <td>DMD SVD precision (low/medium/high)</td>
    </tr>
  </tbody>
</table>

## Cloud Storage

These variables configure S3-compatible cloud storage for automatically uploading generated images and videos.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_CLOUD_STORAGE_TYPE`</td>
      <td>not set</td>
      <td>Set to `s3` to enable cloud storage</td>
    </tr>

    <tr>
      <td>`SGLANG_S3_BUCKET_NAME`</td>
      <td>not set</td>
      <td>The name of the S3 bucket</td>
    </tr>

    <tr>
      <td>`SGLANG_S3_ENDPOINT_URL`</td>
      <td>not set</td>
      <td>Custom endpoint URL (for MinIO, OSS, etc.)</td>
    </tr>

    <tr>
      <td>`SGLANG_S3_REGION_NAME`</td>
      <td>us-east-1</td>
      <td>AWS region name</td>
    </tr>

    <tr>
      <td>`SGLANG_S3_ACCESS_KEY_ID`</td>
      <td>not set</td>
      <td>AWS Access Key ID</td>
    </tr>

    <tr>
      <td>`SGLANG_S3_SECRET_ACCESS_KEY`</td>
      <td>not set</td>
      <td>AWS Secret Access Key</td>
    </tr>
  </tbody>
</table>

## CUDA Crash Debugging

These variables enable kernel API logging and optional input/output dumps around diffusion CUDA kernel call boundaries. They are useful when tracking down CUDA crashes such as illegal memory access, device-side assert, or shape mismatches in custom kernels.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Environment Variable</th>
      <th>Default</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`SGLANG_KERNEL_API_LOGLEVEL`</td>
      <td>`0`</td>
      <td>Controls crash-debug kernel API logging. `1` logs API names, `3` logs tensor metadata, `5` adds tensor statistics, and `10` also writes dump snapshots.</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_LOGDEST`</td>
      <td>`stdout`</td>
      <td>Destination for crash-debug kernel API logs. Use `stdout`, `stderr`, or a file path. `%i` is replaced with the process PID.</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_DUMP_DIR`</td>
      <td>`sglang_kernel_api_dumps`</td>
      <td>Output directory for level-10 kernel API dumps. `%i` is replaced with the process PID.</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_DUMP_INCLUDE`</td>
      <td>not set</td>
      <td>Comma-separated wildcard patterns for kernel API names to include in level-10 dumps.</td>
    </tr>

    <tr>
      <td>`SGLANG_KERNEL_API_DUMP_EXCLUDE`</td>
      <td>not set</td>
      <td>Comma-separated wildcard patterns for kernel API names to exclude from level-10 dumps.</td>
    </tr>
  </tbody>
</table>


# Fused Kernels
Source: https://docs.sglang.io/docs/sglang-diffusion/fused_kernels

The fused CUDA/Triton kernels SGLang Diffusion ships, what each one replaces, and which are on by default.

Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise and selection chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues, and MoE routing — each of which can require several kernel launches and HBM round trips in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion).

This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is `--quality`, described below.

## Numerical contracts and quality tiers

Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. The `quality` switch distinguishes unconditional bit-exact replacements from non-bit-exact eager-chain fusions:

**Bit-exact — mounted unconditionally.** The kernel reproduces every rounding boundary of the eager chain, so `torch.equal` holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch's `vectorized_layer_norm_kernel` down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer's CuTe-DSL `RMSNormKernel` fragment order and `shfl.bfly` fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch.

**Same math, different rounding — request-gated at `lossless`.** These keep the reference function and the precision of every operand and accumulator; only the place or the order of the rounding moves, so the result is within rounding error of the reference rather than equal to it. They are mounted for `quality="lossless"` (the default) and `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. `quality="exact"` runs the unmodified reference chain instead.

**Approximate — request-gated at `high`.** A fusion that quantizes, lowers an operand or accumulator precision, or sits upstream of a quantizer or threshold is no longer a rounding difference, so it needs model-level quality evaluation and is mounted only for `quality="high"`. Today that is the FLUX.2 NVFP4 FC1+SwiGLU+quant path (it quantizes FC2's input before the reference BF16 intermediate exists), Ideogram's BF16-native gate RMSNorm (the reference keeps the norm statistics in fp32), and SANA-Video's BF16-input linear attention (the reference promotes those inputs to FP32).

**Model/checkpoint-native.** Generic close-contract kernels, sparse operators, and FP8/NVFP4 producers can be part of a model implementation or a separately selected deployment path. They are documented in the inventory, but `quality` does not select or undo those choices.

**Selection-equivalent routing — enabled unconditionally.** LingBot Video's fused group-limited top-k returns the same selected expert-id set as its guarded CUDA `torch.topk(..., sorted=False)` reference chain. The order of those ids is not part of either path's contract. Because the selected experts are unchanged, this path does not depend on the request `quality` tier.

> [!NOTE]
>
> A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR, which is what motivated the bit-exact rewrite of that path.

The quality levels are cumulative:

| `quality` | Included optimization set |
| - | - |
| `exact` | The selected deployment's reference path plus every unconditional bit-exact replacement |
| `lossless` | Everything in `exact`, plus request-gated DiT and VAE fusions that keep the reference math and every operand's precision; this level does not itself enable quantization, sparsity, caching, or another approximate path |
| `high` | Everything in `lossless`, plus any model-owned approximate optimization, such as an audited Cache-DiT policy, lower-precision VAE decode, or the approximate fusions listed above |

A fusion claiming `lossless` has to pass the admission checks in
`sglang/multimodal_gen/test/quality_tier_admission.py`: scored against an fp64
evaluation of the same math, its RMS and max error may not exceed 1.5x the
path the model runs today, its rounding may not be biased, and NaN, Inf and
non-contiguous inputs have to behave as before. Lowering an accumulation below
fp32, or quantizing, makes it `high` by construction — no measurement admits it.

If a model has no eligible request-gated fusion, `lossless` can execute the same path as `exact`. Likewise, `high` adds only the model-specific high-only paths that the active pipeline implements.

> [!NOTE]
>
> `quality` is not a master precision switch. A quantized checkpoint, an explicitly selected approximate attention backend, or an independently enabled cache remains active at every quality tier.

## Enabling the request-gated set

```bash theme={null}
sglang generate --model-path MODEL_PATH --prompt "..." --quality lossless
```

The server default is `lossless`, so this is the set a request gets unless it asks otherwise; the OpenAI-compatible endpoints carry the level per request. Images:

```bash theme={null}
curl -X POST http://${HOST}:${PORT}/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "lossless"}'
```

Video, same field:

```bash theme={null}
curl -X POST http://${HOST}:${PORT}/v1/videos \
  -H 'Content-Type: application/json' \
  -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "lossless"}'
```

> [!WARNING]
>
> The `quality` field in a **video response** body is unrelated. It is Sora-compatible response metadata and is always reported as `"standard"`; it does not reflect the sampling quality the request ran with.

`quality` participates in the dynamic-batch signature, so mixed-quality traffic is batched separately and the transition happens safely at a batch boundary. Mounting is all-or-nothing: if any marked site on a transformer fails its static guards, no site on that transformer is fused.

> [!WARNING]
>
> Do not combine request-gated DiT fusions with `--enable-breakable-cuda-graph`.
> BCG warmup captures the lossless module branches before an `lossless` or
> `high` request mounts its DiT fusions, so replay would bypass the requested
> kernels. SGLang rejects this combination for models with eligible DiT quality
> sites. Models whose request-gated path changes only VAE decode remain allowed
> because BCG captures the DiT only.

These fusion families mount under both `quality="lossless"` and
`quality="high"`:

| Fusion | What it folds |
| - | - |
| Linear + tanh-GELU | Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip |
| Wan NVFP4 linear + GELU | Bias-add and GELU fused into Wan's NVFP4 FFN projection output |
| Qwen-Image added-QKV | Added Q/K/V projections fused into joint-buffer production |
| LayerNorm + modulate | `layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate |
| LTX-2 RMSNorm + modulate | `rms_norm(x) * (1 + scale) + shift` in one launch |
| Gate RMSNorm (BF16-native) | `RMSNorm + tanh + mul + add` in one pass |
| HunyuanVideo strided QK RMSNorm | Per-head QK RMSNorm over the packed QKV layout |
| LingBot Video fused RMSNorm | Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels |
| LingBot Video per-token gated residual + RMSNorm modulate | Folds `residual + gate * update` (per-token `[B, S, 1]` gate) and the `rmsnorm(x) * (1 + scale) + shift` adaLN chain (strided `[B, S, 6D]` chunk views) into single kernels |
| SANA-Video BF16-input linear attention | Keeps the first linear-attention GEMM's inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32 |
| Dynamic LoRA delta | An unmerged adapter's scaled update accumulated onto the base output by its second GEMM (`addmm`, one rounding), and a row-parallel layer's bias added in the base GEMM's epilogue, instead of materializing the `[tokens, out]` delta and scaling and adding it in separate passes |
| FLUX-family VAE fast paths | Channels-last decode, GroupNorm(+SiLU), upsample, and attention replacements for FLUX.2 and AutoencoderKL-based FLUX.1, Z-Image, and SD3 pipelines |
| Wan VAE RMSNorm + SiLU | Replaces the channel-first RMSNorm/SiLU chain while keeping the decode in `channels_last_3d` |
| MiniMax-H3 VAE decoder fast path | Per-head fp32 RMSNorm, rounded to the activation dtype, and NeoX RoPE in one launch over the strided Q/K views of the ViT decoder's QKV buffer; cuDNN SDPA for the block's attention; up to `vae_config.decoder_tile_batch` spatial tiles and `vae_config.window_batch` temporal windows per decoder forward. On SM100+ the temporal blend write and output de-normalization run as bit-exact kernels at every quality level |

## Kernel inventory

46 operators are registered in the kernel registry across 52 implementations (some operators carry several backends). Backends are named by provenance, not device: `KDA` identifies Kernel Design Agents implementations, `JIT` compiles under nvcc *and* hipcc, `TRITON` identifies Triton sources, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, and `AOT` comes from the `sgl_kernel` wheel. Per-operator capability metadata determines which devices can load each implementation.

### Normalization

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `rmsnorm_scale_shift` | Triton | bit-exact | RMSNorm + `* (1 + scale) + shift` (4 kernels) |
| `scale_residual_norm_scale_shift` | KDA / Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) or backend-specific close contract | the above plus the preceding `residual + gate * update` |
| `scale_residual_norm_scale_shift_nvfp4` | JIT CUDA | matches the selected NVFP4 producer contract | Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization |
| `layernorm_modulate` | Triton | bit-exact | affine-free LayerNorm + adaLN modulate |
| `try_fused_fp32_layernorm_bf16` | KDA / Triton | bit-exact against the verified native dispatch | LingBot World causal FP32 LayerNorm on Hopper with FP32 affine or adaLN post-ops, including BF16 input/output conversions; other devices retain native LayerNorm |
| `qk_head_layernorm` | Triton | bit-exact | per-head LayerNorm on q/k |
| `qk_rmsnorm_native` | Triton | bit-exact | Z-Image per-head QK RMSNorm |
| `norm_scale_shift` | KDA / CuTe-DSL / FlyDSL | backend-specific close contract | LN-or-RMS + scale/shift, many broadcast modes |
| `rmsnorm_scale`, `rmsnorm_tanh_residual` | Triton | bf16-native statistics | `RMSNorm(x) * scale`, `x + tanh(gate) * RMSNorm(y)` |
| `apply_group_norm_silu` | Triton | close | `GroupNorm + SiLU`, NCHW-contiguous |
| `group_norm_silu_4d`, `group_norm_silu_rows` | Triton | close | channels-last GroupNorm(+SiLU); what lets a VAE decoder run channels\_last end to end with no `nchwToNhwc` transposes |
| `wan_rmsnorm_silu` | Triton | close | Wan VAE `channels_last_3d` RMSNorm + SiLU |
| `wan_norm_silu_post` | JIT CUDA | bit-exact, first-sight verified | Wan VAE FP32 divide, scale, affine, and SiLU after the unchanged native L2 reduction; dense NCDHW and `channels_last_3d` |

### adaLN modulation and gating

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `modulate_scale_shift` | JIT CUDA | bit-exact | `x * (1 + scale) + shift` |
| `residual_gate_add` | KDA (JIT CUDA) | bit-exact | `residual + gate * update` |
| `try_fused_scaled_residual_bf16` | Triton | bit-exact against the verified native dispatch | LingBot World causal BF16 residual with an FP32 per-frame gate; skips the unused normalized output before camera conditioning |
| `timestep_embedding` | JIT CUDA | close | sinusoidal timestep embedding |
| `temb_table_slices` | Triton | bit-exact | see note below |
| `ltx2_ada_values` | Triton | bit-exact | LTX-2 nine-way adaLN value split, slices come out contiguous |

LingBot World camera conditioning also uses `modulate_scale_shift` with one
affine row per token. The causal normalization and residual paths verify each
new signature outside CUDA graph capture and fall back to their native chains
on mismatch or unsupported layouts. These paths preserve `lossless` arithmetic;
kernel replay support does not enable BCG for a model configuration that disables it.

> [!TIP]
>
> `temb_table_slices` is worth knowing about. The eager `(scale_shift_table + temb.float()).chunk(6, dim=2)` materializes roughly 8 GB of fp32 at 704p/121f *and* hands six **strided** slices downstream, whose `.contiguous()` calls then copy each one again. The fused kernel produces the six slices in one pass, each naturally contiguous, so the downstream copies become no-ops.

### RoPE and QK-norm

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True` | separate QK-norm kernel + RoPE |
| `flux2_qkv_epilogue` | KDA (JIT CUDA) | bit-exact against its selected BF16 reference chain | FLUX.2 QK RMSNorm + RoPE + joint text/image QKV packing |
| `flux2_strided_qknorm_rope` | Triton | bit-exact against native tiled RMSNorm followed by FlashInfer interleaved RoPE; verified per signature | FLUX.2 Klein single-stream packed Q/K copies + QK RMSNorm + RoPE on Hopper, BF16, 24 heads of dimension 128 |
| `qwen_qkv_epilogue` | JIT CUDA | bit-exact against its selected BF16 reference chain | Qwen-Image QK RMSNorm + RoPE + joint QKV writes on SM100+ |
| `rope_rotate_half` | Triton | bit-exact | `chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection |
| `interleaved_rope_fp64` | JIT CUDA | bit-exact | paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels |
| `helios_qk_rope` | JIT CUDA | bit-exact | paired in-place Helios Q/K RoPE with transposed frequency layout |
| `ltx2_qknorm_split_rope` | KDA (JIT CUDA) | close (validated on B200) | LTX-2 QK-norm + split RoPE |
| `ltx25_decoder_rope` | JIT CUDA | bit-exact | paired LTX-2.5 decoder 3D RoPE from cached compact axis tables |
| `hunyuan_qkv_rope_pack` | Triton | bit-exact | QKV pack and RoPE in one pass |

The Klein strided path preserves the BF16 rounding between RMSNorm and RoPE,
including signed zeros. It reads packed Q/K views without changing the adjacent
V or MLP values. Unsupported layouts, devices, or RMSNorm dispatches use the
original helper. A new signature is verified outside CUDA Graph capture; an
unverified capture falls back. `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0` also disables
this path. This kernel does not enable model-level BCG support for FLUX.2.

### Activation

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `silu_mul` | Triton | bit-exact | `F.silu(a) * b` for split-projection SwiGLU, where the concatenated `silu_and_mul` kernels do not apply without an extra full-width `cat` |
| `bias_silu`, `bias_glu` | Triton | bit-exact | Sana GLUMB conv bias + SiLU / GLU post-processing |
| `linear_gelu_tanh` | AOT (cublasLt) | not bit-exact, request-gated | bias-add and tanh-GELU folded into the GEMM epilogue |

### Attention

| Operator | Backend | Notes |
| - | - | - |
| `sparse_linear_attn_fwd` | Triton | block-map, compression, and forward for sparse linear attention |
| `bigdn` | Triton | Sana-WM bidirectional gated delta-net; the chunkwise form splits phase A along the KV and Z streams so two blocks stay resident per SM, and stores `(I - P)` so phase B's MMA folds the identity-add in |
| `vsa_block_sparse_sm100` | JIT CUDA | FastVideo's warp-specialized tcgen05 block-sparse VSA forward for 64-token tiles (sm\_100a / sm\_103a); a CTA owns two adjacent query tiles with their own key lists. VSA-H3 uses it on Blackwell and keeps the Triton tile-64 kernel elsewhere |

### MoE routing

| Operator | Backend | Contract | Replaces |
| - | - | - | - |
| `group_limited_topk` | Triton | selected expert-id set matches the guarded CUDA reference; output order is unspecified | LingBot Video's per-group top-2 reduction, group top-k, mask construction, masked expert scores, and final expert top-k |

### Data movement

Every kernel here only moves values (plus zero fill, plus at most one same-order add), so each is bitwise identical to the aten chain it replaces.

| Operator | Backend | Replaces |
| - | - | - |
| `usp_merge_heads` | JIT CUDA | USP all-to-all output head merge (`permute` + `contiguous`) |
| `pack_qkv_destination_major` | Triton | Ulysses destination-major QKV pack |
| `varlen_pack_qkv`, `varlen_scatter_to_padded` | Triton | varlen gather/scatter around the masked attention path |
| `varlen_pack_segmented_qkv` | Triton | varlen gather from a virtual prefix/main Q/K/V sequence |
| `joint_qkv_cat` | Triton | Joy Image Edit's three image/text Q/K/V concatenations, including strided packed-V inputs |
| `causal_conv3d_cat_pad` | KDA (JIT CUDA) / Triton | causal Conv3d `cat` + `pad` |
| `cat_pad_channels_last_3d` | Triton | Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass |
| `dup_up3d_add` | Triton | `repeat_interleave + permute().contiguous() + add` |

Joy Image Edit uses the joint-copy path for eligible CUDA FP16/BF16 image
tensors of at least 32 MiB. It preserves image-first token order, copies
values without arithmetic, and verifies each new shape/stride signature
against the native concatenations before enabling it. Small inputs,
unsupported layouts, gradient-bearing inputs, and unverified signatures
during graph capture use the native path. This does not enable model-level
BCG support.

For large Hopper BF16 image Q/K with 32 heads of width 128, Joy also uses
the existing out-of-place QK-Norm + RoPE kernel to read the packed projection
directly. This removes two input copies while retaining the original CUDA
arithmetic and contiguous outputs. Each new signature is checked bitwise;
inputs remain intact if the operation fails. Other shapes and platforms,
compilation, unverified capture, and `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0`
retain the original helper.

### Quantized layout producers

These kernels preserve the quantized checkpoint path's selected reference operation. They are not a claim that FP8 or NVFP4 is equivalent to an unquantized BF16 checkpoint.

| Operator | Backend | Replaces |
| - | - | - |
| `flux2_layernorm_modulate_fp8_quant` | KDA (Triton) | FLUX.2 LayerNorm plus adaLN modulation directly into static FP8 output |
| `flux2_token_cat_fp8` | KDA (Triton) | FLUX.2 single-block attention/MLP concatenation plus static FP8 quantization |
| `flux2_token_cat_nvfp4` | JIT CUDA | FLUX.2 single-block attention/MLP concatenation plus NVFP4 quantization |

## Coverage by model

Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal.

| Model | Fused paths |
| - | - |
| FLUX.1 | LN+modulate, modulate, residual-gate add, linear+GELU |
| FLUX.2 | LN+modulate, fused LN+modulate-to-FP8, packed SwiGLU, gated residual/norm, residual-gate add, QK RMSNorm+RoPE+joint QKV packing, FP8/NVFP4 token-cat producers |
| Qwen-Image | linear+GELU, select-0/1 LN modulation, added-QKV fusion, QK RMSNorm+RoPE+joint QKV writes, residual norm/modulate+NVFP4 producer |
| GLM-Image | LN+modulate, per-head qk LN, residual-gate add, linear+GELU |
| ERNIE-Image | RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add |
| Joy Image Edit | Bitwise image/text QKV concatenation; strided image QK-Norm + RoPE on Hopper |
| Z-Image | BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm |
| Ideogram 4 | gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add |
| LTX-2 | QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU |
| LTX-2.5 decoder | paired 3D RoPE with shared axis-table cache |
| HunyuanVideo / Helios | QKV+RoPE pack, strided QK RMSNorm, linear+GELU; Helios also has paired in-place Q/K RoPE |
| LingBot Video MoE | Default-on group-limited top-k expert selection; fused RMSNorm, per-token gated residual, and fused RMSNorm+modulate at `quality=lossless` or `quality=high` |
| Sana | LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add |
| SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; GLUMB bias+SiLU / bias+GLU in eager and BCG; LN+modulate and residual-gate add during BCG; BF16-input linear attention at `quality=lossless` or `quality=high` |
| Sana-WM | bidirectional gated delta-net, fused QK inverse-RMS |
| Wan / LongLive 2 VAE | temb table slices (Wan); VAE cat+pad and DupUp3D add; lossless FP32 normalization post-ops; full `channels_last_3d` RMSNorm+SiLU at `quality=lossless` or `quality=high` |
| Cosmos3 | QK-norm + RoPE + KV packing; rounded T2I fusion also supports Hopper TP2/SP1 Super-Text2Image with the 5120-wide SwiGLU architecture |
| Krea2 / MiniMax-H3 | QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation) |
| FLUX.2 VAE / HunyuanVAE / latent upsampler | GroupNorm + SiLU (channels-last two-pass for FLUX.2) |

## Inspecting what is registered

Every kernel is described by a `KernelSpec` in the process-wide registry, so the inventory is queryable without importing any backend:

```python theme={null}
from sglang.kernels.registry import registry

diffusion_ops = [op for op in registry.ops() if op.startswith("diffusion.")]
for spec in registry.get("diffusion.scale_residual_norm_scale_shift"):
    print(spec.backend, spec.target, spec.capabilities)
```

Registration is metadata only — it imports neither torch nor a backend and triggers no JIT build. To pick a specific implementation of an operator that has several:

```python theme={null}
from sglang.kernels import select_kernel, KernelBackend

fn = select_kernel(
    "diffusion.scale_residual_norm_scale_shift", backend=KernelBackend.CUTE_DSL
).load()
```

## Importing the kernels

Runtime code imports from the package, never from a submodule:

```python theme={null}
from sglang.kernels.ops.diffusion import fused_rmsnorm_scale_shift_bitexact
```

Resolution is lazy: the backends have disjoint, heavy dependencies (Triton, CUTLASS/CuTe-DSL, and FlyDSL on ROCm), so an eager re-export would make every one of them an import-time requirement on every platform. Each public kernel is a predicate-plus-kernel pair — call `can_use_<op>(...)` first and fall back to the reference chain when it returns `False`; the kernel raises on an unsupported input rather than silently returning `None`.

The package `README.md` carries a selection matrix for the cases where several kernels look interchangeable and are not. The normalization domain alone holds more than a dozen implementations that differ by numerical contract, activation layout, and backend rather than by speed.

## References

* [Performance Optimization](/docs/sglang-diffusion/performance-optimization)
* [Attention Backends](/docs/sglang-diffusion/attention_backends)
* [Quantization](/docs/sglang-diffusion/quantization)
* [Profiling](/docs/sglang-diffusion/profiling)
* [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion) — source and selection matrix
* [RFC #29630](https://github.com/sgl-project/sglang/issues/29630) — the unified `sglang.kernels` namespace


# SGLang Diffusion
Source: https://docs.sglang.io/docs/sglang-diffusion/index

Accelerated image and video generation with diffusion models.

SGLang Diffusion is SGLang's built-in inference engine for image and video generation. It ships in the same [repository](https://github.com/sgl-project/sglang) and `sglang` Python package, using the shared `sglang generate` and `sglang serve` commands.

It provides native SGLang pipelines, diffusers backend support, an OpenAI-compatible server, and an optimized kernel stack built on both precompiled `sgl-kernel` operators and JIT kernels for key inference paths.

## Key Features

* Broad model support across Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more
* Fast inference with `sgl-kernel`, JIT kernels, scheduler improvements, and caching acceleration
* Multiple interfaces: `sglang generate`, `sglang serve`, and an OpenAI-compatible API
* Multi-platform support for NVIDIA, AMD, Intel XPU, Ascend, Apple Silicon, and Moore Threads

## Quick Start

Start with the [recommended Docker setup](/docs/sglang-diffusion/installation) for Linux GPU deployments. The installation guide also covers pip/uv, source installation, and other platforms.

Run the following commands inside the container or your activated Python environment. Generate an image:

```bash theme={null}
sglang generate --model-path Qwen/Qwen-Image \
  --prompt "A beautiful sunset over the mountains" \
  --save-output
```

Or start an HTTP server:

```bash theme={null}
sglang serve --model-path Qwen/Qwen-Image --host 0.0.0.0 --port 30000
```

## Start Here

* [Installation](/docs/sglang-diffusion/installation): install SGLang Diffusion and platform dependencies
* [Supported Models](/docs/sglang-diffusion/compatibility_matrix): browse supported model families, tasks, and public checkpoints
* [CLI](/docs/sglang-diffusion/api/cli): run one-off generation jobs or launch a persistent server
* [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server
* [ComfyUI plugin](/docs/sglang-diffusion/comfyui): use SGLang from ComfyUI in server mode or as a per-step DiT backend
* [Performance Overview](/docs/sglang-diffusion/performance-optimization): choose speed, memory, parallelism, caching, and quality-tradeoff levers
* [Caching Acceleration](/docs/sglang-diffusion/caching-acceleration): use Cache-DiT, TeaCache, or Spectrum to reduce denoising cost
* [Quantization](/docs/sglang-diffusion/quantization): configure component checkpoint and causal KV-cache quantization
* [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models): understand session state, causal caches, and realtime-only controls
* [Contributing](/docs/sglang-diffusion/contributing): contribution workflow, adding new models, and CI perf baselines

## Additional Documentation

* [Post-Processing](/docs/sglang-diffusion/api/post_processing): frame interpolation and upscaling
* [Models with AR Stage](/docs/sglang-diffusion/models_with_ar): run hybrid diffusion pipelines like GLM-Image with a separate AR encoder server
* [Models with Prompt Enhancement](/docs/sglang-diffusion/models_with_pe): run ERNIE-Image with either built-in PE or a separate PE server
* [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook): choose `--performance-mode`, offload, FSDP, CFG parallelism, SP, and TP
* [Attention Backends](/docs/sglang-diffusion/attention_backends): choose the best backend for your model and hardware
* [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance): configure SP, Ulysses, and ring-based splitting for long sequences
* [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel): fold, data-parallel, or replicate the text/image encoders across idle GPUs
* [Inference Batching](/docs/sglang-diffusion/dynamic_batching): batch compatible native diffusion requests during serving
* [Progressive Resolution Generation](/docs/sglang-diffusion/progressive_resolution): run early denoising steps at lower latent resolution for selected pipelines
* [Environment Variables](/docs/sglang-diffusion/environment_variables): platform, caching, storage, and debugging configuration

## Developer Documentation

* [Support New Models](/docs/sglang-diffusion/support_new_models): implementation guide for new diffusion pipelines
* [CI Performance Baselines](/docs/sglang-diffusion/ci_perf): generate and update performance baselines used in CI

## References

* [SGLang GitHub](https://github.com/sgl-project/sglang)
* [Cache-DiT](https://github.com/vipshop/cache-dit)
* [FastVideo](https://github.com/hao-ai-lab/FastVideo)
* [xDiT](https://github.com/xdit-project/xDiT)
* [Diffusers](https://github.com/huggingface/diffusers)


# Install SGLang Diffusion
Source: https://docs.sglang.io/docs/sglang-diffusion/installation

Run SGLang's built-in diffusion engine with Docker, or install the diffusion extra with pip, uv, or from source.

SGLang Diffusion is SGLang's built-in image and video generation engine. It ships in the same [repository](https://github.com/sgl-project/sglang) and **`sglang` Python package**, using `sglang generate` and `sglang serve`.

Docker is the recommended setup for Linux GPU deployments. The official NVIDIA images include SGLang, its diffusion dependencies, and the optimized kernel stack; no separate Python installation or `pip install` inside the container is needed for the standard diffusion runtime. Optional backends and model-specific dependencies are documented in the [cookbook](/cookbook/diffusion/intro).

For Python installation, use `sglang[diffusion]`: `[diffusion]` adds optional dependencies to the same package, not a separate engine.

## Install on NVIDIA GPUs

**Docker (recommended)**

Use Linux with a supported NVIDIA GPU, a driver compatible with the image's CUDA version, [Docker Engine](https://docs.docker.com/engine/install/), and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Configure Docker's GPU runtime before continuing. See the [SGLang installation guide](/docs/get-started/install) for current CUDA requirements.

The example uses the latest stable release. For reproducible deployments, replace `latest` with a release tag or image digest from [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags). For models or features not yet released, choose a [nightly image](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly) containing the required changes, or install from source.

```bash theme={null}
docker pull lmsysorg/sglang:latest
```

Start a container with persistent model and output directories. The working directory is `/outputs`, so generated files remain in the host's `outputs` directory after the container exits. The server port is published only on the host's loopback interface.

```bash theme={null}
mkdir -p "$HOME/.cache/huggingface" "$PWD/outputs"
docker run --rm -it --gpus all \
  --ipc=host \
  -p 127.0.0.1:30000:30000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -v "$PWD/outputs:/outputs" \
  -w /outputs \
  lmsysorg/sglang:latest bash
```

For gated or private models, export your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) as `HF_TOKEN` on the host and add `--env HF_TOKEN` before the image name. Mount any local input media or checkpoints into the container and use their container paths in requests.

Run the commands in [Generate or serve](#generate-or-serve) inside this container. For development tools and source changes, see the [development environment guide](/docs/developer_guide/development_guide_using_docker#setup-docker-container).

**pip / uv**

Use this option for an existing Python environment. Follow the [Python and CUDA prerequisites](/docs/get-started/install), then install the diffusion extra in a virtual environment:

```bash theme={null}
pip install uv
uv venv --python 3.12
source .venv/bin/activate
uv pip install "sglang[diffusion]" --prerelease=allow
```

With pip, use `pip install "sglang[diffusion]" --pre` in your activated environment.

**From source**

Use source installation to develop SGLang or try changes that are not in a published image. Follow the [Python and CUDA prerequisites](/docs/get-started/install). The commands below install the current `main`; check out a release tag or commit before installation when you need a fixed revision.

```bash theme={null}
git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install uv
uv venv --python 3.12
source .venv/bin/activate
uv pip install -e "python[diffusion]" --prerelease=allow
```

The diffusion implementation lives in [`python/sglang/multimodal_gen`](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen).

## Generate or serve

Run these commands inside the container or your activated Python environment. For a one-off image:

```bash theme={null}
sglang generate --model-path Qwen/Qwen-Image \
  --prompt "A beautiful sunset over the mountains" \
  --save-output
```

Alternatively, start an HTTP server. Binding to `0.0.0.0` makes the published container port reachable from the host:

```bash theme={null}
sglang serve --model-path Qwen/Qwen-Image --host 0.0.0.0 --port 30000
```

Choose a model and GPU configuration from the [cookbook](/cookbook/diffusion/intro). See the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api) to send requests.

## AMD GPUs (ROCm)

Docker is also recommended for AMD GPUs. Use a ROCm image matched to your GPU and host driver, not the NVIDIA `latest` image. Follow [AMD GPUs](/docs/hardware-platforms/amd_gpu) for the current image tags, device passthrough flags, and source installation options.

For **Strix Halo / gfx1151**, use `rocm/sgl-dev:*-gfx1151*` (image sets `SGLANG_USE_AITER=0`) and pass `--attention-backend torch_sdpa --num-gpus 1`. Do not use AITER on wave32 gfx115x. See the Strix Halo section in [AMD GPUs](../hardware-platforms/amd_gpu).

## Platform-Specific: MUSA (Moore Threads GPUs)

## Moore Threads GPUs (MUSA)

For Moore Threads GPUs (MTGPU) with the MUSA software stack, follow the platform guide first. If the source tree still requires the alternate platform `pyproject` fallback, keep a backup of the default file before switching:

```bash Command theme={null}
# Clone the repository
git clone https://github.com/sgl-project/sglang.git
cd sglang

# Install the Python packages
pip install --upgrade pip
mv python/pyproject.toml python/pyproject.toml.bak
cp python/pyproject_other.toml python/pyproject.toml
pip install -e "python[all_musa]"
```

## Intel XPU

For Intel Data Center GPU Max or Arc GPUs, follow the Docker instructions in the [XPU installation guide](/docs/hardware-platforms/xpu). The Dockerfile already includes diffusion dependencies.

## Ascend NPU

For Ascend NPU, please follow the [NPU installation guide](/docs/hardware-platforms/ascend-npus/getting-started/installation).

Quick test:

```bash Command theme={null}
sglang generate --model-path black-forest-labs/FLUX.1-dev \
    --prompt "A logo With Bold Large text: SGL Diffusion" \
    --save-output
```

## Platform-Specific: Apple MPS

For Apple MPS, install natively on macOS using the source instructions below; the Linux Docker image is not an MPS runtime. If the source tree still requires the alternate platform `pyproject` fallback, keep a backup of the default file before switching:

```bash Command theme={null}
# Install ffmpeg
brew install ffmpeg

# Install uv
brew install uv

# Clone the repository
git clone https://github.com/sgl-project/sglang.git
cd sglang

# Create and activate a virtual environment
uv venv -p 3.12 sglang-diffusion
source sglang-diffusion/bin/activate

# Install the Python packages
uv pip install --upgrade pip
mv python/pyproject.toml python/pyproject.toml.bak
cp python/pyproject_other.toml python/pyproject.toml
uv pip install -e "python[all_mps]"
```

SGLang Diffusion uses PyTorch MPS. The `all_mps` extra also installs the SRT
MLX backend dependencies; `SGLANG_USE_MLX` applies only to SRT serving.


# Diffusion models with autoregressive stages
Source: https://docs.sglang.io/docs/sglang-diffusion/models_with_ar

Run diffusion pipelines with in-process or separately deployed autoregressive encoders.

SGLang Diffusion supports two AR execution paths. Qwen Image Layered and
LongCat-Image use the native Qwen2.5-VL component in process. GLM-Image can use
its bundled Transformers implementation or delegate AR inference to a separate
SGLang server.

## GLM-Image Quick Start

Run GLM-Image with its bundled Transformers implementation (default):

```bash theme={null}
# Terminal 1 : launch server
sglang serve --model-path zai-org/GLM-Image --port ${PORT}
```

```bash theme={null}
# Terminal 2 : launch client
curl http://${HOST}:${PORT}/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "prompt",
    "n": 1,
    "size": "widthxheight"
  }'
```

Run GLM-Image with a separate SGLang server for its AR stage:

```bash theme={null}
# Terminal 1 : launch server with AR model
sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \
--tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal --port ${AR_PORT}
```

```bash theme={null}
# Terminal 2 : launch server with Diffusion model
sglang serve --model-path /path/to/zai-org/GLM-Image/ \
  --srt-encoder-url "http://${HOST}:${AR_PORT}" \
  --port ${PORT}
```

```bash theme={null}
# Terminal 3 : launch client
curl http://${HOST}:${PORT}/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "prompt",
    "n": 1,
    "size": "widthxheight"
  }'
```

## Support matrix

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model</th>
      <th>Transformers backend</th>
      <th>SGLang backend</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>GLM-Image</td>
      <td>T2I, I2I, V2I</td>
      <td>T2I</td>
    </tr>

    <tr>
      <td>Qwen Image Layered</td>
      <td>Not used</td>
      <td>I2I, in-process native Qwen2.5-VL</td>
    </tr>

    <tr>
      <td>LongCat-Image</td>
      <td>Not used</td>
      <td>T2I, in-process native Qwen2.5-VL</td>
    </tr>
  </tbody>
</table>

## Deployment Assumptions & Limitations

:::warning
**Network Latency & Timeouts:** In SGLang backend mode, the Diffusion server sends an HTTP request to `--srt-encoder-url` for **every auto-regressive (AR) step**.

* To prevent requests from breaking during long model generations, increase `--srt-encoder-timeout` (e.g., set to 100 seconds).

* To protect the system against temporary network delays or brief drops in connection, use `--srt-encoder-connection-timeout`.
  :::

* **Recommended Setup:** Run both servers on the same machine or inside the same fast local network.

* **Cross-Region Warning:** Running the Diffusion server and the AR server in different geographic regions will slow down token generation and heavily reduce performance.

* **Startup Connection Check:** SGLang automatically checks the connection to `--srt-encoder-url` when starting up. The server will stop immediately if the remote AR host is offline.

## Ascend NPU ENV

To run 2 servers on same group of NPU you need to specify env variables
[https://www.hiascend.com/document/detail/zh/canncommercial/850/maintenref/envvar/envref\_07\_0144.html](https://www.hiascend.com/document/detail/zh/canncommercial/850/maintenref/envvar/envref_07_0144.html)

Example:

```bash theme={null}
# Terminal 1 : server with AR model
export HCCL_IF_BASE_PORT=23000
export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199"
export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399"
```

```bash theme={null}
# Terminal 2 : server with diffusion model
export HCCL_IF_BASE_PORT=24000
export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199"
export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399"
```

## Best practices

GLM-Image example for Ascend A3 Series, 2 cards (4 devices)

```bash theme={null}
# Terminal 1 : server with AR model
export HCCL_IF_BASE_PORT=23000
export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199"
export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399"
sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \
--tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal \
--cuda-graph-bs-decode 1 --device npu --attention-backend ascend --image-processor-backend pil \
--tp-size 4 --port ${PORT} --mem-fraction-static 0.4
```

Second terminal with diffusion server:

```bash theme={null}
# Terminal 2 : run SGL-Diffusion generate command
export HCCL_IF_BASE_PORT=24000
export HCCL_HOST_SOCKET_PORT_RANGE="24000-24199"
export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399"
SGLANG_CACHE_DIT_FN=2 SGLANG_CACHE_DIT_BN=1 SGLANG_CACHE_DIT_WARMUP=4 SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 SGLANG_CACHE_DIT_TAYLORSEER=true SGLANG_CACHE_DIT_TS_ORDER=2 \
SGLANG_CACHE_DIT_ENABLED=true sglang generate --model-path /path/to/zai-org/GLM-Image/ \
--prompt "A curious raccoon" --height 1920 --width 1088 --num-inference-steps 50 --num-gpus 4 \
--sp-degree 4 --srt-encoder-url "http://${HOST}:${PORT}" --warmup-mode request
```

Result:

```bash theme={null}
Warmed-up request processed in 33.82 seconds (with warmup excluded)
```


# Prompt enhancement
Source: https://docs.sglang.io/docs/sglang-diffusion/models_with_pe

Connect an independently deployed SGLang LLM or VLM to diffusion image and video APIs.

Run an SRT server for prompt expansion and a diffusion server for generation.
The diffusion HTTP frontend calls SRT's `/v1/chat/completions` once before
constructing the generation request. The enhancer does not replace the
diffusion model's text encoder or run inside its GPU workers.

```text theme={null}
Application -> diffusion HTTP server -> SRT enhancer server
                                    <- rewritten prompt
            <- image/video          <- diffusion pipeline
```

Prompt enhancement is opt-in per request. It changes conditioning and can
change intent or quality; it is not a lossless acceleration feature. Evaluate
the chosen enhancer and template on your workload before enabling it.

## Deploy two services

This example uses Linux with NVIDIA CUDA and two GPUs, one for each server.
Install [SGLang](/docs/get-started/install) for SRT and
[SGLang Diffusion](/docs/sglang-diffusion/installation) for diffusion. Separate
environments and hosts are supported; the diffusion host needs HTTP access to SRT.

Start a text-only enhancer:

```bash theme={null}
CUDA_VISIBLE_DEVICES=0 sglang serve \
  --model-path Qwen/Qwen3-4B-Instruct-2507 \
  --port 31000
```

Create `enhancer.json` on the diffusion host:

```json theme={null}
{
  "base_url": "http://127.0.0.1:31000/v1",
  "model": "Qwen/Qwen3-4B-Instruct-2507"
}
```

Start diffusion in another terminal:

```bash theme={null}
CUDA_VISIBLE_DEVICES=1 sglang serve \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --prompt-enhancer-config enhancer.json \
  --port 30000
```

Keep the SRT port separate from diffusion's HTTP and internal ports; diffusion
reserves `--port + 1` for its broker. For separate hosts, bind SRT to its service
interface with `--host` and update
`base_url`. Restrict network access and configure SRT authentication. Do not
expose an unauthenticated enhancer publicly. A shared enhancer can serve
multiple diffusion servers; size its concurrency and memory independently.

Send a generation request:

```bash theme={null}
curl --fail-with-body http://127.0.0.1:30000/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "A red teapot on a windowsill, morning light",
    "enhance_prompt": true,
    "size": "512x512",
    "seed": 42,
    "response_format": "b64_json"
  }'
```

Omit `enhance_prompt` or set it to `false` to use the original prompt without
calling SRT. With the OpenAI Python SDK, pass `extra_body={"enhance_prompt": True}`
to `images.generate()`.

The same switch works with `/v1/images/edits` and `/v1/videos`, including
multipart requests (`-F 'enhance_prompt=true'`). Image responses expose the
rewritten text as `data[].revised_prompt`; video responses use `revised_prompt`.
Video creation waits for enhancement before returning the queued job. All `n`
outputs share one rewritten prompt.

Include enhancement in client-observed end-to-end latency. The response's
`inference_time_s` measures diffusion execution, not the preceding SRT call.

## Configure the enhancer

The configuration file is read at server startup. Its supported fields are:

| Field | Meaning |
| - | - |
| `base_url` | SRT OpenAI API base URL, including `/v1` |
| `model` | SRT model ID or its configured served model name |
| `system_prompt` | Optional replacement for the built-in, task-aware rewriting instruction |
| `include_images` | Forward reference images to a VLM; defaults to `false` |
| `timeout` | HTTP operation timeout in seconds; defaults to `60` |
| `api_key_env` | Name of an environment variable on the diffusion server containing the SRT API key |
| `generation_kwargs` | SRT chat-completion options, such as `max_tokens`, `temperature`, `chat_template_kwargs`, or `response_format` |
| `tasks` | Per-task settings for `image`, `image_edit`, or `video`; see below |
| `protocol` | Optional built-in enhancer protocol; currently `flux3-klein-pup` |
| `protocol_source` | Local directory or Hugging Face repo containing the protocol's files; omit to use the protocol's default repository |

Without a built-in protocol, default generation options are `max_tokens=512`
and `temperature=0`.
`generation_kwargs` cannot replace the model, messages, single-output or
non-streaming settings, or enable tools. Choose a non-thinking model or configure
its supported chat template to suppress reasoning; only message `content` is
used as the diffusion prompt.

Unless a task sets `user_template`, the enhancer receives a JSON text message
containing `task` (`image`, `image_edit`, or `video`) and `prompt`.
It returns the replacement prompt as
text. Seed, negative prompt, references and diffusion sampling options are not
rewritten. The diffusion frontend reuses HTTP connections and makes no enhancer
calls during synthetic warmup.

Timeouts return HTTP 504. Upstream failures, empty responses and truncated
completions return HTTP 502 before diffusion is queued. There is no automatic
retry or fallback to the original prompt. An unconfigured server rejects
`enhance_prompt=true` with HTTP 400.

### Task-specific templates and sampling

Each entry in `tasks` accepts:

| Field | Meaning |
| - | - |
| `system_prompt` | Override the top-level system instruction for this task |
| `user_template` | User message text containing `{prompt}`, which is replaced by the request prompt |
| `images_first` | Put image content before text; defaults to `false` and requires top-level `include_images=true` to forward images |
| `generation_kwargs` | Override individual top-level chat-completion options for this task |

For example, keep image rewriting on the default JSON protocol while giving
video rewriting its own instruction and token limit:

```json theme={null}
{
  "base_url": "http://127.0.0.1:31000/v1",
  "model": "Qwen/Qwen3-4B-Instruct-2507",
  "tasks": {
    "video": {
      "system_prompt": "Expand the video prompt while preserving its subjects, motion and timing. Return only the rewritten prompt.",
      "user_template": "Video prompt: {prompt}",
      "generation_kwargs": {"max_tokens": 256}
    }
  }
}
```

Use `protocol="flux3-klein-pup"` for the checkpoint-specific
[FLUX.3 prompt upsampler](/cookbook/diffusion/FLUX/FLUX.3#7-prompt-upsampler).
It loads `system_prompts.json` and supplies image/edit templates, sampling
defaults and image forwarding. Top-level `generation_kwargs` override protocol
sampling defaults; task-level options take precedence over both. An explicit
`tasks` entry replaces that protocol task's entire configuration, not just its
specified fields. Include the required template and system instruction when
overriding a protocol task.

## Choose a model combination

There is no enhancer or diffusion model allowlist in this integration. Protocol
compatibility does not establish quality: use a template appropriate for the
target checkpoint and verify prompt adherence.

| Diffusion input | Enhancer requirements |
| - | - |
| Natural-language image/video prompts, such as Z-Image, FLUX, Qwen-Image, Wan, LTX or MiniMax-H3 | An SRT-served instruction model with chat completions; the default template is a starting point |
| Image editing or image-to-video | Text-only instruction rewriting, or an SRT VLM with `include_images=true` for image-aware rewriting |
| Structured captions, such as Ideogram 4 or LingBot-Video-MoE | A model-specific `system_prompt` and preferably schema-constrained `response_format`; the default prose template is unsuitable |
| ERNIE-Image or LongCat with built-in rewriting | Disable the built-in rewriter for the request when using this generic enhancer, to avoid rewriting twice |

For a VLM, deploy an image-capable SRT model and set `include_images=true` in
its configuration. Uploaded images are sent as data URLs, so the SRT server
does not need access to the diffusion host's filesystem. HTTP image URLs must
be reachable from SRT. Text-only models should leave this option disabled.
Audio and video references are not forwarded to the enhancer; it cannot inspect
their content. Image edits forward the supplied images; video creation forwards
the primary image reference, not model-specific reference bundles.

For [Ideogram 4](/cookbook/diffusion/Ideogram/Ideogram4), instruct the enhancer
to produce the checkpoint's JSON caption schema, including
`high_level_description` and `compositional_deconstruction`. Set the matching
JSON schema through `generation_kwargs.response_format`. JSON text is passed
through unchanged, not converted to prose. Valid JSON alone does not guarantee
the correct caption schema or equivalent output to a hosted service's expander.

This integration applies to the three HTTP image/video endpoints above,
including their normal parallelism, caching and disaggregated diffusion paths.
It does not add enhancement to offline `sglang generate`, realtime streaming,
mesh or action endpoints. For those workflows, call SRT in your application
first and pass its returned text as the diffusion prompt. For reproducibility,
store the rewritten prompt and replay it with `enhance_prompt=false`; a fixed
diffusion seed does not fix the enhancer's output.

## ERNIE-Image's checkpoint-provided enhancer

ERNIE-Image also has a separate, existing PE integration. By default, diffusion
loads its native Ministral3 enhancer in-process:

```bash theme={null}
sglang serve --model-path baidu/ERNIE-Image --port 30000
```

Use `use_pe=false` to skip it for a request. For a memory-constrained deployment,
`--layerwise-offload-components pe` streams the native PE decoder from CPU.

To serve the checkpoint-provided PE through SRT, use a local checkpoint's PE
directory with its tokenizer and retain the ERNIE-specific protocol:

```bash theme={null}
CUDA_VISIBLE_DEVICES=0 sglang serve \
  --model-path /models/ERNIE-Image/pe --port 31000

CUDA_VISIBLE_DEVICES=1 sglang serve \
  --model-path /models/ERNIE-Image \
  --pe-server-url http://127.0.0.1:31000 --port 30000
```

`--pe-server-url` uses the checkpoint's PE tokenizer/template and SRT's
`/generate`; it is not the generic chat-completion integration. Keep using it
when you need ERNIE's trained PE behavior. If you instead use
`--prompt-enhancer-config`, send both `enhance_prompt=true` and `use_pe=false`.
For LongCat, use `enable_prompt_rewrite=false` with the generic enhancer.
These request flags skip native rewriting, not loading its model components.

For Ascend setup, see [models with AR stages](/docs/sglang-diffusion/models_with_ar#ascend-npu-env).


# Parallelism Overview
Source: https://docs.sglang.io/docs/sglang-diffusion/parallelism



SGLang Diffusion ships several parallelism strategies. Each one splits a
different dimension of the DiT forward pass, which is exactly why they can be
combined: the total GPU count is the product of the degrees,

```text theme={null}
num_gpus = cfg_parallel_degree × tp_size × sp_degree
sp_degree = ulysses_degree × ring_degree
```

This page is the map — what each axis does, which combinations are legal, and
how to pick one. Per-axis depth lives in
[Sequence Parallelism](./ring_sp_performance),
[Encoder Parallelism](./encoder_parallel) (text/image encoders are a separate
axis with their own knob), and the [CLI reference](./api/cli).

## The axes

| Strategy | Splits | Communication | Flag |
| - | - | - | - |
| CFG parallel | guidance branches | one combine per denoise step | `--cfg-parallel-size` |
| Tensor parallel (TP) | weights and attention heads | all-reduce per transformer block | `--tp-size` |
| Ulysses SP | sequence outside attention ↔ heads inside it | two all-to-alls per attention | `--ulysses-degree` |
| Ring SP | sequence rows inside attention | neighbor-only K/V rotation, overlapped with compute | `--ring-degree` |
| K/V-gather CP | sequence rows inside attention | one K/V all-gather per attention | `--kv-gather-degree` |
| Data parallel | requests | none between replicas | `--dp-size` |

Two strategies compose when they split different dimensions. TP and Ulysses
both touch heads but compose serially — TP splits the projection weights, then
Ulysses splits the activations of the TP-local heads. Ring and Ulysses compose
because ring splits rows while Ulysses splits heads. Ring has no composition
with a K/V-all-gather style of attention parallelism: both answer the same
question (how a rank's query rows see remote K/V), so they are alternatives for
one slot, not complements.

## What happens to the shapes

For `tp_size = T`, `ulysses_degree = U`, `ring_degree = R`, one attention runs:

```text theme={null}
[B, S/(U·R), H/T, D]                     sequence-sharded activations
   │  Ulysses input all-to-all (inside each Ulysses group)
   ▼
[B, S/R, H/(T·U), D]                     full sequence of this ring block, few heads
   │  ring attention (R−1 neighbor hops; Q never moves, K/V rotate)
   ▼
[B, S/R, H/(T·U), D]
   │  Ulysses output all-to-all (inverse)
   ▼
[B, S/(U·R), H/T, D]
```

The ring merge (online softmax) requires every rank in a ring group to hold the
*same heads* over *different rows*; the group construction guarantees this. A
K/V-gather (CP-style) variant fills the same slot differently: instead of R−1
overlapped hops it all-gathers K/V once and computes the local Q rows against
the full sequence in one shot — fewer, larger transfers, paid for by holding the
whole K/V per rank. Like ring it splits rows, so it adds no head constraint. When no SP degree is
set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` — its
measured-win zone — and higher degrees default to Ulysses.
Ulysses groups are laid out on contiguous ranks and ring groups on strided
ranks, so with a node-major rank mapping, Ulysses traffic stays on intra-node
NVLink (all-to-all needs full-bisection bandwidth) while ring hops cross the
slower interconnect where neighbor-only transfers overlap with compute. A
mis-mapped layout stays numerically correct and silently loses the performance
— worth checking when a sharded run is unexpectedly slow.

## Constraints

* `num_attention_heads % tp_size == 0` — TP splits heads at the projections.
* `(num_attention_heads / tp_size) % ulysses_degree == 0` — Ulysses splits the
  **TP-local** heads. `H % U == 0` alone is not sufficient: 56 heads pass with
  `tp=2, ulysses=4` (28 % 4) and fail with `tp=4, ulysses=4` (14 % 4).
* Ring adds no head constraint (it splits rows), but the sequence — including
  any model-specific packing alignment — must divide by `ulysses × ring`, since
  ring adds an outer row split on top of Ulysses's inner one.
* Ring requires an attention backend that declares
  `supports_ring_rotation()` — the per-hop merge needs the kernel's softmax
  LSE. `fa` and `sage_attn` declare it; the launcher auto-selects `fa` when
  unset.
* `USPAttention`'s masked/tail-padded text path and its replicated-prefix,
  -suffix, and -kv-prefix paths all support ring: the sharded K/V rotates
  through the ring while the tail-pad or replicated portion is attended
  locally once and combined into the ring result with the same online-softmax
  merge. This covers the joint text+image attention most models use (flux,
  flux\_2, qwen\_image, zimage, glm\_image, ernie\_image, and others).
* What still raises `NotImplementedError` under ring: `USPAttention`'s generic
  varlen path (multiple packed segments per row, as used by HunyuanVideo —
  no ring-aware rotation for it yet), and the legacy stacked-QKV
  `UlyssesAttention` layer, now down to one user (Wan's VSA sparse-attention
  variant) that has no softmax LSE to merge and can't gain ring support
  without a different kernel.
* The launcher validates `num_gpus` against the product of the degrees and
  fails fast on any mismatch.

## The Ulysses transport

The all-to-alls normally run over NCCL. On exactly 2 GPUs with peer-to-peer
access, a CUDA-IPC transport replaces them by default: each rank writes its
half directly into the peer's mapped staging buffer, with GPU-side sequence
counters instead of a NCCL rendezvous. An all-to-all is a permutation, never a
reduction, so the transport cannot change results — outputs are bitwise
identical to the NCCL path.

| Environment variable | Default | Meaning |
| - | - | - |
| `SGLANG_DIFFUSION_IPC_A2A` | `1` | set `0` to force NCCL |
| `SGLANG_DIFFUSION_IPC_A2A_TIMEOUT_MS` | `10000` | deadlock backstop for the peer wait; on expiry the transport retires on every rank and the request fails rather than returning incomplete data |
| `SGLANG_DIFFUSION_IPC_A2A_MAX_BUFFERS` | `16` | staging pairs kept alive; raise for many-resolution serving |
| `SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS` | `-1` | head groups for the pipelined exchange below; `0` keeps the sequential exchange, `N >= 2` forces `N` groups |

On a single host with peer-to-peer access, Ulysses attention also pipelines
the exchange against attention, one head group at a time: while group g
attends, group g + 1's q/k/v moves on the copy engine, and finished groups send
their output back while later groups still attend. Every Ulysses path of
`USPAttention` goes through it under FlashAttention or cuDNN SDPA, at any batch
size and with GQA: dense attention, key masks and tail padding, and text tokens
replicated on every rank (a prefix, a suffix, or a K/V-only prefix), which stay
out of the exchange. MiniMax-H3's attention uses the same transport. Ring
attention, uneven sequence shards (`seq_lens`) and the K/V-gather mode keep the
sequential exchange. The automatic count is the largest of 7, 5, 4, 3 and 2
that divides the heads per rank while each group's call still fills four waves
of 128-row tiles, so short sequences such as a 1024² image keep the sequential
exchange. Its buffers are one pool per dtype that every shape shares, grown to
the largest (about 4 × rows per rank × heads × head\_dim); a shape that would
not fit on every rank keeps the sequential exchange. The first call of every
shape runs both exchanges and keeps the pipeline only if they give the same
bytes on every rank, so results never change; that check runs the sequential
exchange on per-call buffers, so no staging for it stays allocated. A custom
attention module reuses it through
`ulysses_pipelined_attention(q, k, v, attend, groups, replicated=..., sequential=...)`.

Independently of the transport, the default path already packs the three
Q/K/V input exchanges into one destination-major collective. A handful of
models (e.g. LTX-2) instead opt into `enable_packed_qkv_input_a2a`, which
pipelines three separate exchanges over a dedicated stream rather than
merging them into one payload — a different trade-off, not a strict
upgrade over the default.

## Which axes tolerate crossing nodes

Each axis has a fixed communication pattern, and the pattern — volume per step,
how often it fires, and whether it can hide behind compute — decides whether the
axis survives the drop from NVLink to the inter-node fabric. Ordered from most
to least cross-node friendly:

| Axis | Pattern | Traffic per denoise step | Cross-node verdict |
| - | - | - | - |
| Data parallel | none between replicas | zero on the request path | **Best.** Replicas only share startup init and control-op fan-out; no fast interconnect needed at all. |
| CFG parallel | one branch-combine | one latent-sized exchange, once | **Good candidate.** Once per step, small payload, naturally deadline-tolerant. Unmeasured across nodes so far. |
| K/V-gather CP | one K/V all-gather per attention | (R−1)/R of K/V, one unoverlapped burst | **Between Ulysses and ring:** only K/V moves (queries never do), but the burst cannot hide behind compute — prefer ring across nodes, gather at small degrees within one. |
| Ring SP | neighbor-only K/V rotation | K/V ÷ ring\_degree, (R−1) hops, overlapped with attention tiles | **Designed for it.** The mechanism now covers most models' joint attention (see Constraints). Actually crossing nodes end-to-end is validated for MiniMax H3 (Ulysses intra-node × ring across, net positive and growing with sequence length); other models' ring support is same-node-validated so far. |
| Ulysses SP | all-to-all | full q/k/v activations, twice per attention, every layer | **Keep intra-node.** All-to-all needs full-bisection bandwidth; across nodes it becomes R² flows with receiver incast. |
| Tensor parallel | all-reduce | hidden-sized reduction per transformer block (×60 blocks for qwen-class DiTs) | **Worst.** Highest frequency, no overlap, already \~70% of sharded kernel time on NVLink. |

Two caveats keep this a map rather than a promise. Cross-node launch
(`--nnodes`/`--node-rank`/`--dist-init-addr`) is merged, and per-model ring
support is broad now (see the constraints above) — but those two facts
together still don't add up to "any model, any node count." The only
configuration actually run end-to-end across nodes is the H3 recipe; other
models' ring support is same-node-validated so far, which means crossing
nodes with them is untested, not disallowed. And the data-parallel row
describes the design: the current
implementation binds each replica's ingress on the local host, so replicas
spanning hosts additionally need per-replica host addressing before `--dp-size`
can place one replica per node.

## Choosing a configuration

Measured guidance rather than rules — the right combination depends on the
model's communication profile and the hardware topology, and legal does not
mean profitable:

* **Multi-branch (true-CFG) models**: CFG parallelism first. Branches run the
  whole DiT independently and combine once per step, avoiding per-layer
  communication entirely.
* **Single-branch image models on 2 GPUs**: Ulysses and TP trade places by
  model. Communication-heavy DiTs measured faster with Ulysses; smaller DiTs
  with TP. Measure both; do not copy a winner across models.
* **Long video / packed sequences**: Ulysses up to the head-divisibility limit,
  then ring for the remaining factor — sequence length scales past the head
  count where Ulysses alone cannot.
* **When Ulysses's head divisibility blocks the degree you need** (`H/T` not
  divisible by the target `U`): the row-splitting slot sidesteps it — ring
  today, or `--kv-gather-degree` (it splits rows, so it adds no head
  constraint either).
* **TP beyond 2 ranks** rarely improves image-DiT latency: the per-block
  all-reduce grows with rank count faster than the GEMM savings.

## Data parallelism

`--dp-size N` runs N full engine replicas on `num_gpus / N` GPUs each, every
replica with its own ingress. Generation requests round-robin across replicas,
realtime sessions stick to the replica holding their state, and control
operations (weights, LoRA, memory occupation, shutdown) apply to every replica;
replicas exchange nothing on the request path. Monolithic serving only, and the
ingress currently binds on the local host — one replica per node needs
per-replica host addressing first.


# Performance Optimization
Source: https://docs.sglang.io/docs/sglang-diffusion/performance-optimization

Choose performance levers for SGLang Diffusion by latency, throughput, memory, and quality tradeoffs.

Use this page as the starting point for SGLang Diffusion performance work. It separates performance levers into two decision classes:

* **Output-preserving / lossless-style:** system settings that should preserve model behavior while changing residency, parallelism, kernels, or scheduling.
* **Quality-tradeoff / lossy or approximate:** techniques that can change the denoising path, numerical representation, or generated output.

The docs use "output-preserving" instead of promising bit-exact "lossless" because different kernels, GPU types, or precision paths can still introduce small numerical differences. The decision boundary is whether the optimization intentionally trades quality or output equivalence for speed.

## Start Here

1. Pick a serving or generation mode from [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook). `--performance-mode auto` is the default; use `speed` when the model fits in GPU memory and latency matters most, `memory` when GPU memory is the bottleneck, and `manual` when every performance flag should be explicit.
2. Choose the right attention backend from [Attention Backends](/docs/sglang-diffusion/attention_backends).
3. Use [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance) only when the model and video shape benefit from sequence splitting.
4. Use [Inference Batching](/docs/sglang-diffusion/dynamic_batching) for concurrent compatible requests during serving.
5. Use [Profiling](/docs/sglang-diffusion/profiling) before changing several levers at once.

## Image and Video Input Processing

Eligible RGB inputs on NVIDIA CUDA use GPU Lanczos resize, crop, and normalization before
encoding. This path retains Pillow's uint8 rounding, model-specific crop
geometry, and the encoder input layout.
It covers Cosmos3 image/video conditioning, Wan TI2V, FLUX.2/Klein reference
images, Qwen-Image-Edit 2509/2511 VAE inputs, and RGB conversion in the shared
image-VAE stage. Small images and unsupported image modes keep the CPU path.
GPU preprocessing uses temporary device buffers and compiles its kernels on first
use. Measure warmed-up end-to-end requests as well as peak memory; faster input
processing does not imply the same speedup for the full generation pipeline.

Set `SGLANG_FORCE_CPU_IMAGE_PREPROCESSING=1` before starting the process to
compare against CPU preprocessing or troubleshoot a numerical issue. This does
not move the learned encoder to CPU. File/image decoding, tokenization, and
model-specific preprocessing outside these paths are unchanged; video readers
stop early where only a prefix is required. GPU hardware codecs are not enabled
by this setting.

## Choose a request quality tier

`--quality` is cumulative: a broader tier never drops an optimization from a
stricter tier.

| Tier | Optimization boundary |
| - | - |
| `exact` | The selected deployment's reference execution plus all unconditional bit-exact replacements. Output is bit-identical to the reference path in the same environment |
| `lossless` (default) | Everything in `exact`, plus request-gated DiT/VAE fusions that keep the reference math and the precision of every operand and accumulator, changing only where or in which order the rounding happens. Output matches the reference to rounding error, not bit for bit |
| `high` | Everything in `lossless`, plus model-owned paths that change what is computed -- lower precision, sparsity, or reuse across steps, such as an audited Cache-DiT policy or lower-precision VAE decode |

Bit-identical describes reproducibility, not quality: a `lossless` fast path is
the same mathematical function, and its rounding is no further from the exact
value than the reference's own. On FLUX.1-dev the two tiers differ by 34.5 dB
PSNR, while the same tier differs by 32.5 dB between one GPU and two -- the
tier moves the image less than the parallel layout does, which is why it is
the default. Ask for `exact` when a run has to be byte-for-byte reproducible,
or when bisecting a numerical bug.

`extra-high` is accepted as the former name of `lossless`. Note that
`lossless` itself was the former name of `exact`, and that it is now the
default: a request that passed `quality="lossless"` for bit-identical output
has to pass `quality="exact"`, and a deployment that never set `quality` moves
from the bit-exact path to this one.

A tier may be a no-op when the active model has no eligible path.
Separately configured quantization, attention, or caching options still apply.
See [Fused Kernels](/docs/sglang-diffusion/fused_kernels) for the current
request-gated families and their numerical contracts.

### Library numerics below the tiers

PyTorch enables two approximations by default, underneath the whole tier
system: `torch.backends.cudnn.allow_tf32` truncates fp32 convolution inputs to
10 mantissa bits, and
`torch.backends.cuda.matmul.allow_bf16_reduced_precision_reduction` lets a bf16
GEMM accumulate split-K partials below fp32. Both change what is computed, so
by the definitions above they belong to `high`.

They are process-global and a server mixes tiers across concurrent batches, so
they cannot follow a request. The runtime sets both explicitly per worker and
logs the result, instead of inheriting whatever the installed PyTorch defaults
to:

| Flag | Default | Effect of turning it off |
| - | - | - |
| `--allow-cudnn-tf32` | `true` (PyTorch's own default) | fp32 convolutions run on fp32 CUDA cores instead of TF32 tensor cores. Affects the VAE decoders that default to `vae_precision="fp32"` |
| `--allow-bf16-reduced-precision-reduction` | `true` (PyTorch's own default) | bf16 GEMMs accumulate split-K partials at fp32 |

The defaults keep PyTorch's values so that `exact` is no slower than it was
before these flags existed. A deployment that needs the strict contract — fp32
and bf16 math at reference precision throughout — passes both as `false` and
accepts the slower convolutions.

## Output-Preserving / Lossless-Style Levers

These settings should preserve model behavior while changing residency, parallelism, kernels, or scheduling. They are the first choices for production tuning.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Lever</th>
      <th>Use when</th>
      <th>Docs</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--performance-mode`</td>
      <td>You want a safe preset for speed or memory without overriding explicit flags.</td>
      <td>[Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook)</td>
    </tr>

    <tr>
      <td>Breakable CUDA graph</td>
      <td>A supported pipeline serves a fixed set of shapes and eager execution is launch-bound.</td>
      <td>[CLI reference](/docs/sglang-diffusion/api/cli)</td>
    </tr>

    <tr>
      <td>Offload, FSDP, CFG parallelism</td>
      <td>GPU memory, multi-GPU residency, or CFG branch splitting is the main bottleneck.</td>
      <td>[Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook)</td>
    </tr>

    <tr>
      <td>Sequence parallelism</td>
      <td>Long image/video sequences need sequence-level parallelism.</td>
      <td>[Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance)</td>
    </tr>

    <tr>
      <td>`--encoder-parallel`</td>
      <td>Text/image encoding is a visible share of the request and the DiT replica sits idle during it.</td>
      <td>[Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel)</td>
    </tr>

    <tr>
      <td>Attention backend</td>
      <td>Kernel choice dominates DiT latency or memory.</td>
      <td>[Attention Backends](/docs/sglang-diffusion/attention_backends)</td>
    </tr>

    <tr>
      <td>Fused kernels</td>
      <td>You want to know which elementwise chains are already fused, or to opt into the request-gated set.</td>
      <td>[Fused Kernels](/docs/sglang-diffusion/fused_kernels)</td>
    </tr>

    <tr>
      <td>Dynamic batching</td>
      <td>Serving many compatible requests concurrently.</td>
      <td>[Inference Batching](/docs/sglang-diffusion/dynamic_batching)</td>
    </tr>

    <tr>
      <td>`--async-output-save`</td>
      <td>Image/video encoding and disk writes leave the GPU idle between queued requests.</td>
      <td>[Asynchronous output saving](/docs/sglang-diffusion/performance-optimization#asynchronous-output-saving)</td>
    </tr>
  </tbody>
</table>

## Asynchronous output saving

Enable `--async-output-save` to overlap output saving with the next request's GPU
work during serving. Frame materialization, image/video encoding, and disk writes
run on a background thread. The flag is disabled by default and does not change
the model's generation settings or the requested output format.

Use it when you have concurrent requests and saving takes a noticeable share of
each request, such as PNG images or large videos from a fast model. A client that
waits for each response before sending the next request has no work to overlap.

### Enable

The feature requires NVIDIA CUDA and the native SGLang backend in monolithic
serving mode. After following [Installation](/docs/sglang-diffusion/installation),
you can start this example on one GPU with the public Hugging Face model
`Tongyi-MAI/Z-Image-Turbo`:

```bash Command theme={null}
sglang serve \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --backend sglang \
  --num-gpus 1 \
  --batching-max-size 1 \
  --async-output-save
```

Send concurrent requests using the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api).
`--batching-max-size 1` keeps each forward separate while allowing requests to
queue. Grouped and dynamically batched forwards currently keep synchronous output
saving.

### Scope and tradeoffs

* Only the output rank performs asynchronous saving, and only for requests whose
  outputs it encodes: `return_file_paths_only=true` together with
  `save_output=true` or `return_image_bytes=true` (set by `/v1/images`) in the
  sampling parameters.
* Warmup requests, raw-frame responses, other requests with `save_output=false`, and
  requests with `perf_dump_path` set or `SGLANG_DIFFUSION_STAGE_LOGGING=1` keep the
  synchronous path. This includes the frame-return path used by `/rollout/generate`.
* Each reply is sent from the saving thread as soon as its output is saved,
  even while the next request's forward is running. Measure both throughput and
  response latency against the flag disabled.
* Up to two pending finalizations retain request outputs in GPU memory. The
  scheduler waits for capacity before starting another forward; leave memory
  headroom for the pending outputs as well as inference.
* Per-request peak memory metrics can include allocations from the next request.
  Asynchronous finalization skips the usual per-request allocator cache release.

Compare the flag on and off with the same model, output shape, format, and client
concurrency. Keep it disabled when the reply latency or extra memory outweighs
the throughput gain.

## Encoding during VAE decode

Native Wan, H3, and LTX video VAEs can hand completed frames to the shared MP4
encoder within the current request. Unlike `--async-output-save`, this does not
require a second request to obtain a benefit. Eligible single-video, CUDA,
file-path-only requests use this path automatically; raw frames, compilation,
interpolation, upscaling, and custom post-decoding hooks keep the normal path.
LTX's diffusion decoder is not included.

Ordinary VAEs without frame callbacks keep the existing GPU-direct saving path.
Handing over a complete clip cannot overlap encoding with decode, and is not
automatically enabled: it can add overhead to short requests. The shared
`DecodingStage.decode(..., on_frames=...)` supports an explicit complete-clip
hand-off for custom integrations. Custom decoding stages retain their original
path unless they explicitly integrate the encoder, preserving output transforms
and audio muxing. Image, audio, and mesh outputs are not MP4 inputs.

LTX without temporal tiling hands over the complete clip after decode. This
reduces CPU float-frame materialization but does not overlap VAE decoding with
encoding. Existing temporal tiling can emit finalized, blended prefixes while
later tiles decode. This optimization does **not** enable temporal tiling:
changing the tile schedule can change the video.

Keep the same decode, output compression, and x264 preset when measuring.
Include time until the MP4 is fully written, not just the decoding-stage timer.
The encoder bounds queued frames and frame-conversion intermediates, and a
failed encode falls back to normal saving. Model authors can follow the
[frame hand-off contract](/docs/sglang-diffusion/support_new_models#video-encode-hand-off).

## Quality-Tradeoff / Lossy Or Approximate Levers

These techniques can change the denoising path, numerical representation, or generated output. They are useful after you have a baseline and an acceptance criterion for quality.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Lever</th>
      <th>Tradeoff</th>
      <th>Docs</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Cache-DiT</td>
      <td>Skips selected DiT block or step computation based on cache decisions.</td>
      <td>[Cache-DiT](/docs/sglang-diffusion/cache_dit)</td>
    </tr>

    <tr>
      <td>TeaCache</td>
      <td>Reuses residuals when consecutive denoising steps are similar enough.</td>
      <td>[TeaCache](/docs/sglang-diffusion/teacache)</td>
    </tr>

    <tr>
      <td>Progressive resolution</td>
      <td>Runs early denoising at lower latent resolution for supported pipelines.</td>
      <td>[Progressive Resolution Generation](/docs/sglang-diffusion/progressive_resolution)</td>
    </tr>

    <tr>
      <td>Quantization</td>
      <td>Uses lower-precision transformer weights or activations.</td>
      <td>[Quantization](/docs/sglang-diffusion/quantization)</td>
    </tr>
  </tbody>
</table>

## Practical Order

1. Establish a baseline with the target model, resolution, frame count, step count, and GPU type.
2. Select `--performance-mode` and explicit residency or parallelism flags.
3. Compare `quality=lossless` with `quality=lossless` to isolate the request-gated fusion set.
4. Compare breakable CUDA graph against eager execution for supported fixed-shape pipelines. Pass every served resolution to `--warmup-resolutions` and confirm capture in the server log. Models with request-gated DiT fusions cannot combine those fusions with a graph captured from the lossless branches.
5. Tune attention backend and batching for the deployment pattern.
6. Profile if the bottleneck is unclear.
7. Add `quality=high`, caching, progressive resolution, or quantization only after comparing output quality against your acceptance target.

## Per-model tuning starting points

Warmup and breakable CUDA graph (BCG) solve different problems. `--warmup-mode request` runs a warmup copy derived from the first request to prime one-time compilation and caches; it does not remove recurring Python launches from each denoising step. BCG captures supported DiT segments and can reduce that recurring launch overhead for captured shapes. Use the table below as a first experiment, then keep a lever only when profiling confirms that it addresses the active bottleneck.

| Observed bottleneck or constraint | Example models | First experiment | What to verify |
| - | - | - | - |
| The first execution of a shape pays substantial compilation or cache setup | ERNIE-Image-Turbo, GLM-Image, FastWan / TurboWan family, FLUX.2-klein-4B, LingBot-World, LongLive, Cosmos3, SANA, LongCat-Image-Edit-Turbo | Compare `--warmup-mode request` with `off` | Separate warmup time, first-real-request latency, and warmed steady-state latency. Request-based warmup is primarily a benchmark aid and still consumes time before the first real request completes. |
| A supported fixed-shape pipeline shows recurring host-launch gaps between GPU kernels | GLM-Image, Z-Image-Turbo, Qwen-Image-2512, Ideogram-4, LongCat-Image, LTX-2 / LTX-2.3, SANA-1.5 1.6B, JoyEcho | Compare eager execution with `--enable-breakable-cuda-graph` | Confirm graph capture and replay in the server log, then compare steady-state latency and GPU memory. Pass additional production shapes through `--warmup-resolutions`; without it, BCG captures only the model's default warmup resolution. |
| The model does not fit while both DiT stages are resident | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, LingBot-Video-MoE | Start with `--dit-layerwise-offload` | Verify peak GPU memory first, then measure the transfer overhead. Add warmup separately if first-run compilation is also material. |
| Dynamic masks, metadata, or image conditions make graph replay ineffective | Qwen-Image-Edit family, FireRed-Image-Edit | Keep eager execution as the baseline | Profile before attempting BCG. Dynamic per-step host work can outweigh graph replay savings, and unsupported pipelines fall back to eager execution. |
| Kernels or collectives already dominate a many-step workload | Qwen-Image, FLUX.1-dev, FLUX.2-dev, LTX-2, Wan2.1 14B, HunyuanVideo, MOVA | Warm the baseline, then profile before enabling another lever | A small host gap limits the benefit available from BCG. Prioritize kernels, attention, parallelism, or residency according to the trace. |
| A one-shot or very few-step workload cannot amortize warmup | FastVideo-FastH3 (8-step) | Start with `--warmup-mode off` | Compare end-to-end latency including warmup. Use request warmup only when intentionally separating cold-start setup from the measured request. |

The example assignments come from single-GPU NVIDIA B300/GB300 profiles and are directional rather than an exhaustive compatibility list. A model can match more than one row as resolution, frame count, step count, parallelism, or GPU type changes; the measured bottleneck takes precedence over the model name.

BCG is enabled only for model and pipeline configurations accepted by the runtime support check. It captures the default warmup shape automatically. Use `--warmup-resolutions` for additional served resolutions; for video and variable prompt lengths, set `--warmup-num-frames` and `--bcg-text-buckets` to cover the intended workload. Requests that do not match a captured signature fall back to eager execution.

Warmup and BCG do not intentionally trade output quality for speed, but that is not a guarantee of byte-identical output. Different execution paths can introduce numerical differences, and some pipelines customize the scheduler or schedule used by a synthetic warmup request. Before production rollout, compare output hashes when bitwise stability is required, otherwise run the project's quality acceptance check.

Performance results are configuration-dependent. Record the exact checkpoint revision, GPU, precision, resolution, frame count, step count, parallelism, command line, and whether the measurement includes warmup. Re-profile with [Profiling](/docs/sglang-diffusion/profiling) on the target workload rather than transferring a percentage from another model or shape.

## Diagnostics

[Profiling](/docs/sglang-diffusion/profiling) is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers.

## References

* [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook)
* [Attention Backends](/docs/sglang-diffusion/attention_backends)
* [Fused Kernels](/docs/sglang-diffusion/fused_kernels)
* [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance)
* [Caching Strategies](/docs/sglang-diffusion/caching-acceleration)
* [Exact conditioning reuse across requests](/docs/sglang-diffusion/caching-acceleration#conditioning-cache)
* [Profiling](/docs/sglang-diffusion/profiling)


# Production Metrics
Source: https://docs.sglang.io/docs/sglang-diffusion/production_metrics

Monitor SGLang Diffusion requests, queues, stages and LoRA state with Prometheus.

## Enable metrics

Enable metrics on an NVIDIA CUDA deployment, for example:

```bash theme={null}
sglang serve --model-path black-forest-labs/FLUX.2-klein-4B \
  --num-gpus 1 --enable-metrics --port 30000
curl http://localhost:30000/metrics
```

Metrics are opt-in. Disabled metrics do not scan queues or collect LoRA status.
Enabled metrics add host-side bookkeeping, not GPU synchronization or collectives.

## Metric reference

All diffusion metrics carry `role` and `replica` labels. `replica` is the
scheduler endpoint; only each DP replica's leader publishes, so TP/SP ranks do
not multiply request counts. The table lists additional labels.

| Metric | Type | Labels | Description |
| - | - | - | - |
| `sglang:diffusion_num_queue_reqs` | Gauge | none | Original generation requests waiting for their first dispatch. |
| `sglang:diffusion_num_running_reqs` | Gauge | none | Number of diffusion generation requests dispatched by the scheduler and not yet finished. |
| `sglang:diffusion_requests_total` | Counter | `status`, `is_warmup` | Completed diffusion generation requests. Dynamic batches are counted per original scheduler request after the merged output is split. |
| `sglang:diffusion_request_latency_seconds` | Histogram | `status`, `is_warmup` | Scheduler acceptance to completion, excluding HTTP preprocessing, media encoding and response delivery. |
| `sglang:diffusion_queue_time_seconds` | Histogram | `is_warmup` | Time spent waiting in the diffusion scheduler queue. |
| `sglang:diffusion_generation_batch_size` | Histogram | `stop_reason` | Generation batch size selected by the diffusion scheduler at dispatch time. |
| `sglang:diffusion_stage_host_latency_seconds` | Histogram | `stage` | Host wall time around a stage, not GPU kernel execution time. Step labels are normalized to `DenoisingStep`. |
| `sglang:diffusion_lora_loaded_adapters` | Gauge | none | Number of loaded diffusion LoRA adapters. |
| `sglang:diffusion_lora_active_modules` | Gauge | none | Number of diffusion modules with active LoRA adapters. |
| `sglang:diffusion_lora_active_adapters` | Gauge | none | Number of unique active diffusion LoRA adapters. |
| `sglang:diffusion_lora_module_active` | Gauge | `module` | Whether a diffusion module currently has an active LoRA adapter. |

Request counts refer to original scheduler requests, not generated images,
denoising steps or distributed shards. `status` is `success` or `error`;
`is_warmup` is `true` or `false`. Queue and running gauges include warmup.
Stage observations include warmup and count stage invocations, not requests;
asynchronous GPU work can complete in a later stage. For synchronized diagnostic
timings, use `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1` separately, accepting its
synchronization overhead. LoRA gauges update at startup and after LoRA control operations.

## Disaggregated serving

In disaggregated serving, the head (`role="server"`) records the original request
lifecycle, including role handoffs and errors. Queue time ends at the first
encoder dispatch; intermediate waits remain part of request latency. Workers
report their own stage and LoRA metrics, not duplicate completed requests.
The generation-batch histogram currently describes monolithic scheduling only.

Single-host pool mode exposes all child metrics through the head's `/metrics`.
A scrape aggregates only processes sharing that host's metrics directory.
For standalone remote roles, pass `--enable-metrics` to each process and scrape
its `--host`/`--port` as well as the head. Each role serves a metrics-only HTTP
endpoint. Use a separate, empty `PROMETHEUS_MULTIPROC_DIR` per server launch if
you set it yourself; otherwise SGLang creates and owns a temporary directory.
Do not share this directory between independent servers or reuse stale files.

## Query throughput

Successful, non-warmup request throughput:

```promql theme={null}
sum(rate(sglang:diffusion_requests_total{status="success",is_warmup="false"}[5m]))
```


# Profiling
Source: https://docs.sglang.io/docs/sglang-diffusion/profiling

Profile SGLang diffusion workloads with PyTorch Profiler and Nsight Systems.

This guide covers profiling techniques for multimodal generation pipelines in SGLang.

## PyTorch Profiler

PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics.

### Denoising Stage Profiling

Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step):

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --seed 0 \
  --profile
```

**Parameters:**

* `--profile`: Enable profiling for the denoising stage
* `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
  * Smaller values reduce trace file size
  * Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step

### Full Pipeline Profiling

Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.):

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --seed 0 \
  --profile \
  --profile-all-stages
```

**Parameters:**

* `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising

### Image API requests

For an already running image server, enable profiling on an individual request.
The JSON generations endpoint accepts the same controls with underscores:

```bash theme={null}
curl -sS http://127.0.0.1:30000/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "A mountain lake at sunrise",
    "seed": 0,
    "profile": true,
    "num_profiled_timesteps": 5,
    "profile_all_stages": false
  }'
```

For `/v1/images/edits`, pass these controls as multipart form fields alongside
the image and prompt: `-F profile=true -F num_profiled_timesteps=5`.
Add `-F profile_all_stages=true` to profile the full pipeline instead.

Traces are written on the server, not returned as image-response attachments.
Use `perf_dump_path` for a lightweight timing report instead of a profiler trace.
Run latency benchmarks separately with profiling disabled, since collecting a
trace adds overhead.

### Output Location

By default, trace files are saved in the ./logs/ directory.

The exact output file path will be shown in the console output, for example:

```bash theme={null}
[mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz
```

### View Traces

Load and visualize trace files at:

* [https://ui.perfetto.dev/](https://ui.perfetto.dev/) (recommended)
* chrome://tracing (Chrome only)

For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.

### `--perf-dump-path` (Stage/Step Timing Dump)

Besides profiler traces, you can also dump a lightweight JSON report that contains:

* stage-level timing breakdown for the full pipeline
* step-level timing breakdown for the denoising stage (per diffusion step)

This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike).

The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key.

Example:

```bash theme={null}
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "<PROMPT>" \
  --perf-dump-path perf.json
```

## Nsight Systems

Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns.

### Installation

See the [SGLang profiling guide](../developer_guide/benchmark_and_profiling#profile-with-nsight) for installation instructions.

### Basic Profiling

Profile the entire pipeline execution:

```bash theme={null}
nsys profile \
  --trace-fork-before-exec=true \
  --cuda-graph-trace=node \
  --force-overwrite=true \
  -o QwenImage \
  sglang generate \
    --model-path Qwen/Qwen-Image \
    --prompt "A Logo With Bold Large Text: SGL Diffusion" \
    --seed 0
```

### Targeted Stage Profiling

Use `--delay` and `--duration` to capture specific stages and reduce file size:

```bash theme={null}
nsys profile \
  --trace-fork-before-exec=true \
  --cuda-graph-trace=node \
  --force-overwrite=true \
  --delay 10 \
  --duration 30 \
  -o QwenImage_denoising \
  sglang generate \
    --model-path Qwen/Qwen-Image \
    --prompt "A Logo With Bold Large Text: SGL Diffusion" \
    --seed 0
```

**Parameters:**

* `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
* `--duration N`: Capture for N seconds (focus on specific stages)
* `--force-overwrite`: Overwrite existing output files

## Notes

* **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems
* **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline
* **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads

## FAQ

* If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time.


# Progressive Resolution Generation
Source: https://docs.sglang.io/docs/sglang-diffusion/progressive_resolution

Experimental spectral progressive resolution growing for selected SGLang Diffusion pipelines.

Progressive resolution growing is an experimental feature for selected SGLang Diffusion pipelines. It runs early denoising steps at a coarser latent resolution and spectrally upsamples the latent before the full-resolution steps. On the benchmark setup below, this reduces the quadratic attention cost of the DiT transformer and yields up to **1.63× speedup on FLUX.1**, **1.93× speedup on FLUX.2**, **2.33× speedup on Z-Image**, **2.78× speedup on Wan 2.1 T2V**, **1.69× speedup on Qwen-Image**, and **1.56× speedup on Ideogram 4**.

Based on [Spectral Progressive Diffusion (arXiv 2605.18736)](https://arxiv.org/abs/2605.18736).

## Overview

DiT attention is O(n²) in sequence length. Running the first N denoising steps at half the spatial resolution cuts the attention cost to \~6% for those steps.

The transition point — how many steps to run at each resolution — is computed from the **Bayes-optimal frequency-activation criterion**: frequencies that cannot be resolved at the coarse scale are not denoised there. The method is designed to preserve quality under this criterion, but generated outputs can still differ from the full-resolution baseline.

| Model | Full-res tokens | Half-res tokens | Token-step ratio |
| - | - | - | - |
| FLUX.1 1024×1024 | 4,096 | 1,024 | 4.0× |
| FLUX.2 1024×1024 | 4,096 | 1,024 | 4.0× |
| Z-Image 1024×1024 | 4,096 | 1,024 | 4.0× |
| Wan 2.1 T2V 480×832 (81 frames) | 6,240 | 1,560 | 4.0× |
| Ideogram 4 1024×1024 | 4,096 | 1,024 | 4.0× |

## Parameters

| Parameter | CLI flag | Default | Description |
| - | - | - | - |
| `progressive_mode` | `--progressive-mode` | `"fullres"` | `"fullres"` disables (identical to standard generation). `"dct_rewind"` enables spectral upsample with scheduler rewind (recommended). `"dct"` enables upsample without rewind. |
| `progressive_levels` | `--progressive-levels` | `1` | Number of resolution halvings. `1` = one coarse stage (64×64 latent → 128×128). `2` = two coarse stages (32×32 → 64×64 → 128×128). |
| `progressive_delta` | `--progressive-delta` | `0.01` | Noise-dominated tolerance δ. Controls how many steps run at coarse resolution. Higher δ = more coarse steps = more speedup. |

> **Tip:** Add `--dit-cpu-offload false` to keep the transformer GPU-resident. With CPU offload each step pays a fixed PCIe transfer cost regardless of sequence length, which dilutes the speedup.

***

## FLUX.1

### Usage

```bash theme={null}
sglang generate \
    --model-path black-forest-labs/FLUX.1-dev \
    --prompt "A serene mountain lake at golden hour, photorealistic" \
    --num-inference-steps 50 \
    --dit-cpu-offload false \
    --progressive-mode dct_rewind \
    --progressive-levels 1 \
    --progressive-delta 0.05
```

### Choosing delta

| δ | Coarse steps (50 total) | Denoising speedup |
| - | - | - |
| `0.01` | 18 @ 64² + 32 @ 128² | **1.32×** |
| `0.05` | 28 @ 64² + 22 @ 128² | **1.63×** |

For most prompts `0.05` is recommended — it gives the largest speedup with no visible degradation.

### Benchmark

Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Timing = denoising loop only.

| Config | Stage split | Denoise | Speedup |
| - | - | - | - |
| Fullres (baseline) | 50 @ 128² latent | 36.65 s | 1.00× |
| dct\_rewind L1 δ=0.01 | 18\@64² + 32\@128² | 27.67 s | **1.32×** |
| dct\_rewind L1 δ=0.05 | 28\@64² + 22\@128² | 22.58 s | **1.62×** |
| dct\_rewind L2 δ=0.01 | 10\@32² + 8\@64² + 32\@128² | 26.48 s | **1.38×** |

### Python API

```python theme={null}
from sglang.multimodal_gen import DiffGenerator

gen = DiffGenerator.from_pretrained(
    model_path="black-forest-labs/FLUX.1-dev",
    dit_cpu_offload=False,
)
result = gen.generate(sampling_params_kwargs={
    "prompt": "A serene mountain lake at golden hour, photorealistic",
    "num_inference_steps": 50,
    "height": 1024,
    "width": 1024,
    "progressive_mode": "dct_rewind",
    "progressive_levels": 1,
    "progressive_delta": 0.05,
})
```

***

## FLUX.2

Supports `FLUX.2-dev`, `FLUX.2-klein-4B`, and `FLUX.2-klein-9B`.

### Usage

```bash theme={null}
sglang generate \
    --model-path black-forest-labs/FLUX.2-klein-4B \
    --prompt "A serene mountain lake at golden hour, photorealistic" \
    --num-inference-steps 30 \
    --dit-cpu-offload false \
    --progressive-mode dct_rewind \
    --progressive-levels 1 \
    --progressive-delta 0.10
```

### Benchmark

Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Model: FLUX.2-klein-4B, 30 steps, 1024×1024.
Timing = denoising loop only, averaged across 10 diverse prompts.

| Config | Stage split | Denoise | Speedup |
| - | - | - | - |
| Fullres (baseline) | 30 @ 64² latent | 9.72 s | 1.00× |
| dct\_rewind L1 δ=0.05 | 18\@32² + 12\@64² | 5.50 s | **1.77×** |
| dct\_rewind L1 δ=0.10 | 20\@32² + 10\@64² | 5.03 s | **1.93×** |

### Python API

```python theme={null}
from sglang.multimodal_gen import DiffGenerator

gen = DiffGenerator.from_pretrained(
    model_path="black-forest-labs/FLUX.2-klein-4B",
    dit_cpu_offload=False,
)
result = gen.generate(sampling_params_kwargs={
    "prompt": "A serene mountain lake at golden hour, photorealistic",
    "num_inference_steps": 30,
    "progressive_mode": "dct_rewind",
    "progressive_levels": 1,
    "progressive_delta": 0.10,
})
```

***

## Wan 2.1 T2V

Supports `Wan-AI/Wan2.1-T2V-1.3B-Diffusers` and `Wan-AI/Wan2.1-T2V-14B-Diffusers`.

> **Note:** Progressive generation grows only the **spatial** H×W dimensions. The temporal dimension T (number of latent frames) is kept fixed across all stages.

### Usage

```bash theme={null}
sglang generate \
    --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
    --prompt "A cheetah sprinting across the Serengeti at sunset, slow motion, photorealistic" \
    --num-inference-steps 50 \
    --num-frames 81 \
    --height 480 \
    --width 832 \
    --guidance-scale 5.0 \
    --flow-shift 5.0 \
    --dit-cpu-offload false \
    --progressive-mode dct_rewind \
    --progressive-levels 1 \
    --progressive-delta 0.05
```

### Choosing delta

| δ | Coarse steps (50 total) | Denoising speedup |
| - | - | - |
| `0.01` | 23 @ 30×52 + 27 @ 60×104 | **1.65×** |
| `0.02` | 27 @ 30×52 + 23 @ 60×104 | **1.86×** |
| `0.05` | 33 @ 30×52 + 17 @ 60×104 | **2.32×** |
| `0.10` | 37 @ 30×52 + 13 @ 60×104 | **2.78×** |

For most prompts `0.05` is recommended. `0.10` provides maximum speedup but should be validated on motion-heavy scenes.

### Python API

```python theme={null}
from sglang.multimodal_gen import DiffGenerator

gen = DiffGenerator.from_pretrained(
    model_path="Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
    dit_cpu_offload=False,
    flow_shift=5.0,
)
result = gen.generate(sampling_params_kwargs={
    "prompt": "A cheetah sprinting across the Serengeti at sunset, slow motion, photorealistic",
    "num_inference_steps": 50,
    "num_frames": 81,
    "height": 480,
    "width": 832,
    "guidance_scale": 5.0,
    "progressive_mode": "dct_rewind",
    "progressive_levels": 1,
    "progressive_delta": 0.05,
})
```

***

## Z-Image

Supports `Tongyi-MAI/Z-Image`. Z-Image uses the same VAE as FLUX.1 (`FluxVAEConfig`), so the power-law spectrum constants are identical. The progressive stage handles Z-Image's 5-D latent format `[B, C, 1, H, W]` with squeeze/unsqueeze hooks and recomputes caption+image RoPE positional embeddings on each stage transition.

> **Note:** Always specify `--height 1024 --width 1024` (or another resolution where H\_lat and W\_lat are both divisible by 2). Z-Image's default resolution (360×640) produces a 45×80 latent where H=45 is not divisible by the patch size.

### Usage

```bash theme={null}
# Standard fullres — unchanged behavior
sglang generate --model-path Tongyi-MAI/Z-Image \
    --prompt "A serene mountain lake at golden hour, photorealistic" \
    --height 1024 --width 1024

# Progressive dct_rewind L1 δ=0.10 → 2.33× denoising speedup
sglang generate --model-path Tongyi-MAI/Z-Image \
    --prompt "A serene mountain lake at golden hour, photorealistic" \
    --height 1024 --width 1024 \
    --num-inference-steps 50 \
    --dit-cpu-offload false \
    --progressive-mode dct_rewind \
    --progressive-levels 1 \
    --progressive-delta 0.10
```

### Choosing delta

| δ | Coarse steps (50 total) | Denoising speedup |
| - | - | - |
| `0.01` | 26 @ 64² + 24 @ 128² | **1.53×** |
| `0.05` | 35 @ 64² + 15 @ 128² | **2.03×** |
| `0.10` | 42 @ 64² + 8 @ 128² | **2.33×** |

Z-Image achieves higher progressive speedups than FLUX.1 at the same δ because it uses dual CFG (two forward passes per step), doubling the absolute attention savings at coarse resolution. `0.10` is the recommended tradeoff.

### Python API

```python theme={null}
from sglang.multimodal_gen import DiffGenerator

gen = DiffGenerator.from_pretrained(
    model_path="Tongyi-MAI/Z-Image",
    dit_cpu_offload=False,
)
result = gen.generate(sampling_params_kwargs={
    "prompt": "A serene mountain lake at golden hour, photorealistic",
    "num_inference_steps": 50,
    "height": 1024,
    "width": 1024,
    "progressive_mode": "dct_rewind",
    "progressive_levels": 1,
    "progressive_delta": 0.10,
})
```

***

## Qwen-Image

Qwen-Image uses the same 2×2 patchify convention as FLUX.1 (in\_channels=64, C=16), so the same progressive stage wires in with model-specific hooks for RoPE (`freqs_cis`) and spatial metadata (`img_shapes`).

```bash theme={null}
# Standard fullres — unchanged behavior
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A serene mountain lake at golden hour"

# Progressive dct_rewind L1 δ=0.20 → 1.69× denoising speedup
sglang generate --model-path Qwen/Qwen-Image \
    --prompt "A serene mountain lake at golden hour" \
    --progressive-mode dct_rewind --progressive-levels 1 --progressive-delta 0.20 \
    --num-inference-steps 30 --dit-cpu-offload false
```

Hardware: RTX A6000 48 GB, `--dit-cpu-offload false`. Timing = denoising loop only.

| Config | Stage split | Denoise | Speedup |
| - | - | - | - |
| Fullres (baseline) | 30 @ 128² | 43.00 s | 1.00× |
| dct\_rewind L1 δ=0.05 | 13\@64² + 17\@128² | 33.25 s | **1.29×** |
| dct\_rewind L1 δ=0.10 | 16\@64² + 14\@128² | 33.86 s | **1.27×** |
| dct\_rewind L1 δ=0.20 | 19\@64² + 11\@128² | 25.40 s | **1.69×** |

## Ideogram 4

Supports `ideogram-ai/ideogram-4`. Ideogram 4 uses a **dual-transformer architecture**: a conditional transformer (text + image tokens) and a separately-weighted unconditional transformer (image tokens only, zero LLM features). Both transformers shrink at coarse resolution, providing the same token-ratio benefit as single-transformer models.

> **Note:** Ideogram 4's logit-normal noise schedule (`std=1.75`, `mu=0`) concentrates steps near the mid-sigma range. Fewer steps fall in the high-sigma coarse-eligible region compared to FLUX, which limits the achievable speedup at a given δ.

### Usage

**20-step (V4\_DEFAULT\_20 preset)**

```bash theme={null}
sglang generate \
    --model-path ideogram-ai/ideogram-4 \
    --prompt "A serene mountain lake at golden hour, photorealistic" \
    --height 1024 --width 1024 \
    --num-inference-steps 20 \
    --dit-cpu-offload false \
    --progressive-mode dct_rewind \
    --progressive-levels 1 \
    --progressive-delta 0.05
```

**48-step (V4\_QUALITY\_48 preset)**

```bash theme={null}
sglang generate \
    --model-path ideogram-ai/ideogram-4 \
    --prompt "A serene mountain lake at golden hour, photorealistic" \
    --height 1024 --width 1024 \
    --num-inference-steps 48 \
    --dit-cpu-offload false \
    --progressive-mode dct_rewind \
    --progressive-levels 1 \
    --progressive-delta 0.05
```

### Benchmark

Hardware: RTX A6000 48 GB, `torch_sdpa`, `--dit-cpu-offload false`. Timing = denoising loop only.

**20-step (V4\_DEFAULT\_20)**

| Config | Stage split | Denoise | Speedup |
| - | - | - | - |
| Fullres (baseline) | 20 @ 64² | 53.99 s | 1.00× |
| dct\_rewind L1 δ=0.01 | 6 @ 32² + 14 @ 64² | 43.47 s | **1.24×** |
| dct\_rewind L1 δ=0.05 | 9 @ 32² + 11 @ 64² | 38.14 s | **1.42×** |
| dct\_rewind L1 δ=0.10 | 11 @ 32² + 9 @ 64² | 34.60 s | **1.56×** |

**48-step (V4\_QUALITY\_48)**

| Config | Stage split | Denoise | Speedup |
| - | - | - | - |
| Fullres (baseline) | 48 @ 64² | 130.92 s | 1.00× |
| dct\_rewind L1 δ=0.01 | 12 @ 32² + 36 @ 64² | 109.79 s | **1.19×** |
| dct\_rewind L1 δ=0.05 | 21 @ 32² + 27 @ 64² | 93.83 s | **1.40×** |
| dct\_rewind L1 δ=0.10 | 26 @ 32² + 22 @ 64² | 84.94 s | **1.54×** |

### Python API

```python theme={null}
from sglang.multimodal_gen import DiffGenerator

gen = DiffGenerator.from_pretrained(
    model_path="ideogram-ai/ideogram-4",
    dit_cpu_offload=False,
)
result = gen.generate(sampling_params_kwargs={
    "prompt": "A serene mountain lake at golden hour, photorealistic",
    "num_inference_steps": 48,
    "height": 1024,
    "width": 1024,
    "progressive_mode": "dct_rewind",
    "progressive_levels": 1,
    "progressive_delta": 0.05,
})
```

***

## Limitations

* **Sequence parallelism incompatible.** Cannot be combined with `--ulysses-degree` or `--ring-degree`. The stage raises a `RuntimeError` if SP is enabled.
* **torch.compile incompatible.** Compiled kernels have a fixed sequence length; the resolution transition causes a recompile or error. Use progressive without `--enable-torch-compile`.
* **Cache-DiT interaction is experimental.** The stage refreshes Cache-DiT context at resolution transitions, but quality and speedup should be benchmarked before relying on this combination.

## References

* [Spectral Progressive Diffusion (arXiv 2605.18736)](https://arxiv.org/abs/2605.18736)
* [SGLang Diffusion Performance Optimization](./performance-optimization)


# Quantization
Source: https://docs.sglang.io/docs/sglang-diffusion/quantization



SGLang-Diffusion treats component path selection and quantized checkpoint
materialization as separate capabilities. Every loaded component can use an
independent checkpoint path, but that checkpoint is quantized only when its
selected loader supports the serialized format. Transformer, encoder, and VAE
precision are resolved independently.

## Quick Reference

Choose the option from the source you have, not from the precision name:

| Source or operation | CLI | Result |
| - | - | - |
| Complete component repo/directory | `--component-paths.<component> <source>` | Replaces that component's config and weights. |
| Weight file/directory for an existing component | `--component-weights-paths.<component> <source>` | Keeps the base component config and replaces only its weights. |
| Online quantization of unquantized weights | `--component-quantizations.<component> <method>` | Constructs a supported quantized implementation during load; use `--component-quantization-ignored-layers.<component>` for exclusions. |
| Primary DiT convenience spellings | `--transformer-path`, `--transformer-weights-path`, `--quantization` | Aliases for pipelines with one primary DiT; they remain supported but are not generic component names. |
| Causal KV-cache compression | `--kv-cache-quant` | Quantizes completed runtime cache chunks, not checkpoint weights. |

`--model-path` always selects the base model. The component key must be a real
key from its `model_index.json` or native pipeline registry; examples include
`transformer`, `transformer_2`, `text_encoder`, `video_vae`, and `audio_vae`,
but no one name is universal.

Recommended example for pre-quantized checkpoints:

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --component-weights-paths.transformer black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "a curious pikachu"
```

For quantized transformers-style transformer component folders:

```bash theme={null}
sglang generate \
  --model-path /path/to/base-model \
  --component-paths.transformer /path/to/quantized-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion"
```

NOTE: Some model-specific integrations also accept a quantized repo or local
directory directly as `--model-path`, but that is a compatibility path. If a
repo contains multiple candidate checkpoints, select an exact component or
weight source explicitly.

Pre-quantized files are self-describing. Pass one selected file by local path,
`owner/repo/path/file.safetensors`, or direct Hugging Face file URL and do not
add an online quantization option. The
[MiniMax-H3 compatibility table](/cookbook/diffusion/MiniMax/MiniMax-H3#checkpoint-and-adapter-formats)
is the canonical list of H3 component formats and published example sources;
this page defines the backend families and their shared constraints.

## Quantized Component Repositories

Every loaded component can point to an independent repository, but path routing
does not imply that every loader can materialize every quantization format.
SGLang resolves quantized component checkpoints through one of three explicit
paths:

* load through an SGLang quantization implementation;
* delegate a standard component to Transformers or Diffusers;
* fail closed in a native plain-state loader when the format cannot be restored.

| Component path | Quantized checkpoint behavior |
| - | - |
| `transformer`, `transformer_2`, `unconditional_transformer`, `audio_dit`, `video_dit` | Uses the SGLang transformer quantization adapters documented below. |
| `text_encoder*`, `image_encoder*` | Prefers a compatible registered native encoder. Otherwise, a standard top-level `quantization_config` recognized by the installed Transformers version is delegated to its `from_pretrained` path; unsupported formats and metadata locations fail closed. |
| `vae`, `video_vae`, `audio_vae` | A standard top-level Diffusers `quantization_config` is delegated to `AutoModel.from_pretrained`. Native-only VAEs and nested/compression metadata fail closed. |
| Library-managed Transformers or Diffusers components | Delegates to the upstream `from_pretrained` path and inherits its format support and validation behavior. The local PE model uses this path; compatible formats remain model-specific. |
| Native auxiliary components that load raw state dicts | Quantized checkpoints are rejected before model construction until that component has a quantized materialization implementation. This includes connectors, duration heads, bridges, diffusion decoders, sound tokenizers, spatial upsamplers, and vocoders. |

`--quantization` is the explicit override for the primary transformer loader;
`--component-quantizations.<component>` expresses the same intent for one
supported component. Pair the latter with
`--component-quantization-ignored-layers.<component>` to keep matching layers
unquantized. Pre-quantized component repositories instead select their format
through their own metadata and the capability of the selected loader. A generic
string override without a matching materialization backend would advertise
support that the component does not have, while a quantized weight file without
matching config metadata cannot be identified or restored generically.

## Quant Families

Here, `quant_family` means a checkpoint and loading family with shared CLI
usage and loader behavior. It is not just the numeric precision or a kernel
backend.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>quant\_family</th>
      <th>checkpoint form</th>
      <th>checkpoint selector</th>
      <th>supported models</th>
      <th>extra dependency</th>
      <th>platform / notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`fp8` / `mxfp4` (online quantization)</td>
      <td>Unquantized checkpoint (offline via AMD Quark coming soon)</td>
      <td>`--component-quantizations.<component> {fp8,mxfp4}`</td>
      <td>Z-Image-Turbo (validated), others likely work. More support coming soon.</td>
      <td>MXFP4: `aiter` on ROCm</td>
      <td>MXFP4 requires ROCm and MI350+ (gfx95x). Weights quantized at load time, activations quantized to `fp8` / `mxfp4` dynamically.</td>
    </tr>

    <tr>
      <td>`convrot_int8` (online quantization)</td>
      <td>Unquantized BF16/FP16 checkpoint</td>
      <td>`--component-quantizations.<component> convrot_int8`</td>
      <td>MiniMax-H3, Qwen-Image, Qwen-Image-Edit (quality-validated); other BF16 DiTs load but are not quality-validated</td>
      <td>SGLang JIT ops (bundled, need nvcc) or `comfy-kitchen`</td>
      <td>Data-free INT8 ConvRot W8A8 at load time, per layer on the fused JIT-compiled ops (CC 9.0, 10.0, 12.0, 12.1) or on `comfy_kitchen.int8_linear` (Turing+); `SGLANG_DIFFUSION_CONVROT_INT8_BACKEND` forces one. Layers whose input dim is not divisible by the group size stay in BF16. TP, LoRA, FSDP and memory notes under ConvRot INT8 below.</td>
    </tr>

    <tr>
      <td>`fp8` (offline quantization)</td>
      <td>Quantized transformer component folder, or safetensors with `quantization_config` metadata</td>
      <td>`--component-paths.<component>` or `--component-weights-paths.<component>`</td>
      <td>Native DiTs whose linear layers support the selected FP8 method; validate quality per model</td>
      <td>None</td>
      <td>Component-folder and single-file flows are both supported</td>
    </tr>

    <tr>
      <td>`modelopt-fp8`</td>
      <td>Converted ModelOpt FP8 transformer directory or repo with `config.json`</td>
      <td>`--component-paths.<component>`</td>
      <td>FLUX.1, FLUX.2, Wan2.2, HunyuanVideo, Qwen Image, Qwen Image Edit</td>
      <td>None</td>
      <td>Serialized config stays `quant_method=modelopt` with `quant_algo=FP8`; `dit_layerwise_offload` is supported and `dit_cpu_offload` stays disabled</td>
    </tr>

    <tr>
      <td>`auto-round` W4A16</td>
      <td>Transformer component repo with a self-describing `quantization_config` and `auto_round:auto_gptq` packing</td>
      <td>`--component-paths.<component>`</td>
      <td>Native dense DiTs with compatible component parameter mappings; MiniMax-H3 Diffusers components are supported</td>
      <td>None</td>
      <td>Auto-detected; reuses the SRT GPTQ/Marlin backend. No `--quantization` flag is needed; use TP/sequence parallelism rather than FSDP.</td>
    </tr>

    <tr>
      <td>`modelopt-nvfp4`</td>
      <td>Mixed transformer directory/repo with `config.json`, raw or Comfy layer-marked NVFP4 safetensors, or full ModelOpt Diffusers repo</td>
      <td>`--component-paths.<component>` for component repos; `--component-weights-paths.<component>` for raw files; `--model-path` for full repos</td>
      <td>FLUX.1, FLUX.2, Wan2.2, Qwen Image, Qwen Image 2512, Qwen Image Edit, Qwen Image Edit 2511, MiniMax-H3</td>
      <td>None</td>
      <td>Mixed override repos keep the base model separate; full Qwen Image exports can be loaded directly as `--model-path`; raw exports such as `black-forest-labs/FLUX.2-dev-NVFP4` use the weights-path flow. Comfy markers select NVFP4 plus INT8 or FP8 companion linears automatically; omit `--quantization`.</td>
    </tr>

    <tr>
      <td>`gguf`</td>
      <td>One selected GGUF DiT or native encoder file</td>
      <td>`--component-weights-paths.<component>`</td>
      <td>MiniMax-H3 original or pruned FL2VA / Ref2VA DiTs and its Qwen3-VL text encoder</td>
      <td>None</td>
      <td>CUDA only and auto-detected; supports standard and K-quant GGML types. TP requires aligned GGML blocks; component/layerwise offload is supported and FSDP is not.</td>
    </tr>

    <tr>
      <td>`comfy-fp8`</td>
      <td>One selected Comfy safetensors file with per-layer `comfy_quant` metadata</td>
      <td>`--component-weights-paths.<component>`</td>
      <td>MiniMax-H3 pruned FL2VA / Ref2VA DiTs</td>
      <td>None</td>
      <td>CUDA; auto-detected; TP, sequence parallelism, and component/layerwise offload are supported, while FSDP is not. Checkpoint-marked `fc2` layers retain FP8 storage and use compute-dtype matmul.</td>
    </tr>

    <tr>
      <td>`comfy-int8-convrot`</td>
      <td>One selected safetensors file with per-layer `int8_tensorwise` and ConvRot metadata</td>
      <td>`--component-weights-paths.<component>`</td>
      <td>Native DiTs and encoders whose parameter mappings preserve each marked linear; MiniMax-H3 DiT and Qwen3-VL encoder checkpoints have validated tensor contracts</td>
      <td>SGLang JIT ops (bundled, need nvcc) or `comfy-kitchen`</td>
      <td>CUDA; auto-detected; each layer runs on the JIT-compiled ConvRot ops where they apply (CC 9.0/10.0/12.0/12.1, group size 64 or 256) and on `comfy_kitchen.int8_linear` otherwise; weight/scale layout is validated before model construction. TP requires every row-parallel input shard to preserve the checkpoint's ConvRot group boundary. The H3 256-group DiT supports TP1/2/4, not TP8; its Qwen3-VL encoder keeps TP8 by replicating only incompatible row projections. Offload is supported; FSDP is not.</td>
    </tr>

    <tr>
      <td>`mxfp8`</td>
      <td>Self-describing serialized weights, or BF16/FP16 weights for online quantization</td>
      <td>`--component-weights-paths.<component>`, or `--component-quantizations.<component> mxfp8` online</td>
      <td>Native diffusion linears; mixed MiniMax-H3 checkpoints are selected per layer</td>
      <td>SRT's platform MXFP8 backend</td>
      <td>Serialized metadata is auto-detected. NVIDIA and ROCm reuse SRT's dense MXFP8 kernels; Ascend keeps its native online path. Mixed per-layer checkpoints do not support FSDP.</td>
    </tr>

    <tr>
      <td>`comfy-w4a8-convrot`</td>
      <td>Safetensors with serialized `asym_w4a8_int8` layer metadata and packed weights</td>
      <td>`--component-weights-paths.<component>`</td>
      <td>MiniMax-H3 FL2VA / Ref2VA DiTs and its native Qwen3-VL encoder</td>
      <td>`comfy-kitchen>=0.2.27`</td>
      <td>Auto-detected; omit `--quantization`. Requires SM80+ and validates packed weights, group/channel scales, and optional codebooks before model construction. Mixed encoder files may keep their embedding tensorwise INT8. TP must preserve ConvRot group boundaries; offload is supported and FSDP is not.</td>
    </tr>

    <tr>
      <td>`comfy-w4a4-convrot`</td>
      <td>Safetensors with serialized `convrot_w4a4` metadata, optionally mixed with `int8_tensorwise` layers</td>
      <td>`--component-weights-paths.<component>`</td>
      <td>Native DiTs and encoders with matching parameter mappings; MiniMax-H3 FL2VA / Ref2VA DiTs and Qwen3-VL encoder layouts are recognized</td>
      <td>`comfy-kitchen`</td>
      <td>Auto-detected; omit `--quantization`. Each layer dispatches to its serialized W4A4 or INT8 ConvRot kernel. CUDA requires SM75+; TP must preserve each format's quantization and ConvRot group boundaries. Offload is supported and FSDP is not.</td>
    </tr>

    <tr>
      <td>`comfy-nvfp4`</td>
      <td>Safetensors with serialized `nvfp4` and optional scalar/row-wise `int8_tensorwise` layer metadata</td>
      <td>`--component-weights-paths.text_encoder`</td>
      <td>MiniMax-H3 native Qwen3-VL encoder</td>
      <td>`comfy-kitchen` for NVFP4 matmul</td>
      <td>Auto-detected; omit explicit quantization. Preserves high-nibble-first weights, swizzled block scales, and optional AWQ input pre-scales. Layers declaring `full_precision_matrix_mult` use BF16/FP16 compute with packed storage. Other NVFP4 layers use dynamic activation quantization and NVFP4 matmul on NVIDIA SM100+. Companion INT8 embeddings support scalar or per-row scales without expanding the full table.</td>
    </tr>

    <tr>
      <td>`quanto-int8`</td>
      <td>One native encoder safetensors file with an embedded Quanto quantization map</td>
      <td>`--component-weights-paths.<component>`</td>
      <td>Native encoders whose mapped linear layers consume every declared qint8 entry; MiniMax-H3's Qwen3-VL encoder is supported</td>
      <td>None</td>
      <td>Auto-detected weight-only qint8 storage. Each active matrix is dequantized to the compute dtype for the ordinary linear operation, so this reduces stored/resident weight memory rather than promising INT8 GEMM speed. TP and offload are supported; FSDP is not.</td>
    </tr>

    <tr>
      <td>`qvg-kv`</td>
      <td>Unquantized model with runtime causal KV-cache compression</td>
      <td>`--kv-cache-quant {int4,int2}`</td>
      <td>LingBot World realtime causal path</td>
      <td>`quant-videogen`</td>
      <td>CUDA only; compresses completed cache chunks rather than model weights; lossy and disabled by default</td>
    </tr>

    <tr>
      <td>`nunchaku-svdq`</td>
      <td>Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4|fp4}_r{rank}-...`</td>
      <td>`--transformer-weights-path`</td>
      <td>Model-specific support such as Qwen-Image, FLUX, and Z-Image</td>
      <td>`nunchaku`</td>
      <td>SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4`</td>
    </tr>

    <tr>
      <td>`msmodelslim`</td>
      <td>Pre-quantized msmodelslim transformer weights</td>
      <td>`--model-path`</td>
      <td>Wan2.2 family</td>
      <td>None</td>
      <td>Currently only compatible with the Ascend NPU family and supports `mxfp8`, `mxfp4`, `w8a8`, and `w4a4`</td>
    </tr>
  </tbody>
</table>

## Causal KV-Cache Quantization

Quant-VideoGen KV-cache quantization targets long-running autoregressive video
sessions, where the causal self-attention cache can become comparable to the
model weights. It does not change or quantize the checkpoint weights.

See [Realtime and Causal Video Models](./realtime_models) for the session
lifecycle, supported pipelines, and the distinction between realtime and
request-based causal generation.

Install the optional dependency without allowing its stale Torch requirement to
replace SGLang's pinned Torch version, then enable int4 compression when serving
a supported LingBot World realtime pipeline:

```bash theme={null}
pip install "sglang[diffusion,diffusion-qvg]"
pip install --no-deps quant-videogen==0.1.0

sglang serve \
  --model-path robbyant/lingbot-world-fast-diffusers \
  --pipeline-class-name LingBotWorldCausalDMDPipeline \
  --num-gpus 4 \
  --ulysses-degree 4 \
  --kv-cache-quant int4 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false
```

### Storage Policy

The current chunk is rewritten at every denoising step, so it remains in BF16.
The newest `--kv-cache-quant-keep-recent` completed chunks also remain in BF16.
Older completed chunks are stable and are packed once with Progressive Residual
Quantization (PRQ); their dense BF16 tensors are then released.

When a transformer layer runs attention, its packed visible chunks are
dequantized and concatenated with the recent BF16 chunks. This creates one
layer's dense attention view at a time instead of keeping dense windows
resident for every transformer layer.

### How PRQ Works

For each K or V vector, PRQ uses k-means to select a centroid, then quantizes
the remaining error:

```text theme={null}
x = centroid_1 + residual_1
residual_1 = centroid_2 + residual_2
...
x_hat = centroid_1 + centroid_2 + ... + dequantize(low_bit_residual)
```

Each additional stage applies another centroid lookup to the previous stage's
residual. SGLang's default uses one stage, 128 centroids, and an int4 or int2
block-quantized residual. More stages or centroids can reduce reconstruction
error but add codebook storage and packing work.

PRQ is the compression algorithm; selecting older completed chunks is the
runtime storage policy that makes it practical. Stable chunks are compressed
once, while mutable and recent chunks avoid repeated packing and retain higher
precision.

### Quality And Performance

> [!WARNING]
>
> KV-cache quantization is lossy. Disabling it uses the original dense BF16 cache
> and is bit-exact with the unmodified path. Enabling int4 or int2 reconstructs an
> approximation of K and V, so fixed-seed generated frames are not expected to be
> pixel-identical to BF16.

In the initial LingBot measurements, int4 used about 47% of the dense resident
KV-cache memory for a 24-frame window and added about 18% per-chunk latency.
Int2 used about 37% of the dense resident KV-cache memory but introduces more
quantization error. These measurements are configuration-specific; benchmark
memory, latency, temporal consistency, identity stability, and motion quality
on the intended session length. Start with int4 unless capacity requires int2.

The current implementation is limited to the LingBot realtime
sliding-window-and-sink path, including Ulysses sequence sharding. It does not
support LongLive2 pinned sinks, global sinks, or dynamically growing caches.

### Tuning Options

| Option | Default | Effect |
| - | -: | - |
| `--kv-cache-quant {off,int4,int2}` | `off` | Enables QVG KV-cache compression and selects residual precision. |
| `--kv-cache-quant-stages` | `1` | Number of progressive centroid-residual stages. |
| `--kv-cache-quant-centroids` | `128` | Number of k-means centroids per stage. |
| `--kv-cache-quant-block-size` | `64` | Block size used to quantize the final residual. |
| `--kv-cache-quant-iters` | `2` | K-means iterations used while packing a chunk. |
| `--kv-cache-quant-asymmetric` | disabled | Uses asymmetric residual quantization. |
| `--kv-cache-quant-keep-recent` | `1` | Number of newest completed chunks retained in BF16. |
| `--kv-cache-quant-sink {0,1}` | `1` | Whether to quantize completed sink chunks. |
| `--kv-cache-quant-sink-keep` | `0` | Number of leading sink chunks retained in BF16. |

## Online Quantization

This section describes the online methods currently implemented by the
transformer loader. They are useful when a pre-quantized transformer checkpoint
is not available. Encoders, VAEs, and auxiliary components are independent:
their repositories may still carry serialized quantized weights, which are
restored when the selected component loader supports that format.

### FP8 Online Quantization

Apply FP8 quantization to a supported unquantized DiT checkpoint:

```bash theme={null}
sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --component-quantizations.transformer fp8 \
  --prompt "a beautiful sunset" \
  --save-output
```

For a pipeline whose primary DiT is named `transformer`, the shorter
`--quantization fp8` spelling is equivalent.

MiniMax-H3 supports this path while preserving its required FP32 patch,
timestep, and output projections. See the
[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes)
for its distributed serving recipe.

### MXFP4 Online Quantization

MXFP4 provides aggressive 4-bit compression with online quantization. **Note: Requires ROCm and MI350+ (gfx95x) GPU.**

```bash theme={null}
sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --component-quantizations.transformer mxfp4 \
  --prompt "a beautiful sunset" \
  --save-output
```

**Note:** Requires `aiter` package with MXFP4 kernel support

### ConvRot INT8

`convrot_int8`
is online W8A8 INT8 for DiTs and compatible native encoders:
every linear whose input dim is divisible by the Hadamard group size (256) is
rotated and quantized per output channel after loading a stock BF16 checkpoint
(ConvRot, arXiv:2512.03673; no calibration data). Each forward call runs one
fused op per linear: group-wise Hadamard rotation, dynamic per-row INT8
activation quantization, INT8 GEMM, dequantization and bias add.

Two kernel backends implement the same quantization and consume the same INT8
weights and per-row scales, so the choice is made per layer at load time:

* `jit`: the `convrot_int8` ops in `sglang.kernels.ops.gemm.convrot_int8`,
  compiled by SGLang's JIT for the local GPU on first use (about a minute with
  `nvcc` on the machine; cached under `SGLANG_JIT_CACHE_DIR`, default
  `~/.cache/sglang/jit`, for later runs). CC 9.0 (Hopper), 10.0 (B200, GB200)
  and 12.0 / 12.1 (RTX PRO 6000 Blackwell, RTX 50 series, DGX Spark); group
  size 64, 128, 256 or 512; output width a multiple of 8. CC 10.3 (B300,
  GB300) is refused on purpose: Blackwell Ultra cuts INT8 tensor-core
  throughput to a fraction of its BF16 rate, so use FP8 or NVFP4 there. Linears sharing one input (a block's Q/K/V projections)
  rotate and quantize it once, and the Qwen-Image FFN down-projection folds its
  GELU(tanh) into the rotate kernel; both are bitwise identical to the
  per-linear ops.
* `comfy_kitchen`: `comfy_kitchen.int8_linear` from the optional
  `comfy-kitchen` package (`pip install comfy-kitchen`). Turing and later
  (SM75); group size 16, 64 or 256.

The default `auto` takes `jit` wherever its ops build and run (no CUDA
toolchain, or a failed build, counts as unavailable and is logged) and
`comfy_kitchen` otherwise; loading stops with an install hint if
`comfy-kitchen` is then missing. A layer whose output width is not a multiple
of 8 goes to `comfy_kitchen` when it is installed and stays BF16 otherwise.
`SGLANG_DIFFUSION_CONVROT_INT8_BACKEND=jit` or `=comfy_kitchen` forces one
backend. LoRA adapters run in dynamic mode (they cannot be merged into
rotated INT8 weights); under FSDP `auto` resolves to `comfy_kitchen` and an
explicit `jit` is refused. Serialized Comfy ConvRot INT8 DiTs and native
encoders use `--component-weights-paths.<component>`; they are auto-detected
from per-layer markers, load INT8 weights and row scales directly (omit online
quantization), and follow the same per-layer backend choice.
`--transformer-weights-path` remains the primary-DiT convenience spelling.

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --quantization convrot_int8 \
  --prompt "a beautiful sunset" \
  --save-output
```

```bash theme={null}
sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --component-quantizations.transformer convrot_int8 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output
```

Quantization runs after the model's weight loaders, so MiniMax-H3's grouped
`qkv` reorder is already applied. Layers whose input dim is not divisible by
the group size stay in BF16 instead of failing the load (H3's AdaLN
projections take that path); under TP every row-parallel input shard must also
divide it, or the load stops naming the layer, which
`--quantization-ignored-layers` keeps in BF16. Without offload, loading stages
the BF16 weights on the GPU before quantizing them, so it needs the BF16
checkpoint's worth of free VRAM (38 GiB for Qwen-Image); under layerwise
offload the weights stay on the host and are quantized one layer at a time.
`--quantization-ignored-layers img_mod txt_mod txt_mlp.net.2` keeps the
few-row Qwen-Image GEMMs (AdaLN modulation,
text-stream FFN down-projection) in BF16 at the cost of memory (45.5 GB peak)
for no consistent speed gain, so the default quantizes everything.

Measured against BF16 on GB200 in eager execution with every eligible layer
quantized:

| Model | Backend | Speedup vs BF16 | Peak GPU memory |
| - | - | -: | - |
| Qwen-Image 1024x1024, 50 steps, CFG 4, 1 GPU | `jit` | 1.19x (1.13x on 2 GPUs, CFG parallel) | 56 -> 37 GB |
| Qwen-Image-Edit 1024x1024, 1 GPU | `jit` | 1.32x | |
| MiniMax-H3 1344x768, 5 s, 50 steps, 1 GPU | `jit` | 1.07x (1.15x on 4 GPUs, Ulysses 4) | 125 -> 107 GiB |
| MiniMax-H3, same | `comfy_kitchen` | 0.86x | 125 -> 107 GiB |

On MiniMax-H3 both backends quantize the same 209 linears and sit in the same
quality class against BF16 over six prompt/seed pairs (video PSNR 23.4 vs 23.8
dB, SSIM 0.80 vs 0.81, audio spectral similarity 0.96 vs 0.97; the same-seed
distance between the two backends equals each one's distance to BF16). Loading
an online-quantized H3 without offload peaks at 115 GiB resident; the
serialized Comfy INT8 checkpoint loads at 96 GiB resident and runs on `jit` at
the same speed.

> [!WARNING]
>
> `convrot_int8` is approximate: same-seed outputs are not bit-exact with BF16 or
> between the two backends (both stay within INT8 quantization noise of BF16), so
> it is not a consistency ground-truth mode. Quality was validated against BF16 on
> MiniMax-H3, Qwen-Image and Qwen-Image-Edit; other DiTs load but are not
> quality-validated. Without `convrot_int8` neither backend is loaded and the BF16
> path is untouched. See the
> [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes)
> for the 24 GB offload recipe, including why `vae` must stay out of
> `--layerwise-offload-components`.

With the `comfy_kitchen` backend, each GEMM is one fused call by default.
`SGLANG_KITCHEN_INT8_MAX_ROWS=N` splits large-M GEMMs (`rows > N` and
`out_features >= SGLANG_KITCHEN_INT8_MIN_SPLIT_N`, default 8192) into N-row
calls, which keeps them on comfy\_kitchen's data-parallel CUTLASS config; on
MiniMax-H3 with comfy-kitchen 0.2.37 the extra calls and output copies cost more
than they save.

### Skipping Layers

By default, transformer online quantization quantizes every supported linear
layer in that component. However, `--quantization-ignored-layers` can keep
specific transformer layers in their original precision:

```bash theme={null}
sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization fp8 \
  --quantization-ignored-layers attention.to_ \
  --prompt "a beautiful sunset" \
  --save-output

sglang generate \
  --model-path Tongyi-MAI/Z-Image-Turbo \
  --quantization mxfp4 \
  --quantization-ignored-layers attention.to_ \
  --prompt "a beautiful sunset" \
  --save-output
```

Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantized if its prefix contains any of the given patterns.

## Transformers-managed Quantized Components

Model components that already have a native Transformers loading path delegate
self-describing checkpoints whose standard top-level `quantization_config` is
recognized by the installed Transformers version. Successful loading still
depends on that backend's optional dependencies, platform, and model/checkpoint
compatibility. An available native SGLang implementation remains preferred when
it can restore the same format. For example, replace FLUX's T5 component with
the official BitsAndBytes checkpoint:

```bash Command theme={null}
sglang serve \
  --model-path black-forest-labs/FLUX.1-dev \
  --component-paths.text_encoder_2 \
    diffusers/FLUX.1-dev-bnb-4bit/text_encoder_2
```

Transformers owns format validation and places the quantized component directly
on its resident device. SGLang rejects component/layerwise offload, FSDP,
nonstandard metadata locations, unsupported upstream formats, and native-only
fallbacks instead of silently changing the checkpoint contract. Diffusion DiT
components declared under the Diffusers library use the separate quantization
backends documented above.

## Validated ModelOpt Checkpoints

This section is the canonical support matrix for the thirteen published
diffusion ModelOpt checkpoints currently wired up in SGLang docs and validation
coverage.

Published checkpoints keep the serialized quantization config as
`quant_method=modelopt`; the FP8 vs NVFP4 split below is a documentation label
derived from `quant_algo`.

Twelve of the thirteen repos live under `lmsys/*`. The FLUX.2 NVFP4 entry keeps
the official `black-forest-labs/FLUX.2-dev-NVFP4` repo.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Quant Algo</th>
      <th>Base Model</th>
      <th>Preferred CLI</th>
      <th>HF Repo</th>
      <th>Current Scope</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`FP8`</td>
      <td>`black-forest-labs/FLUX.1-dev`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/flux1-dev-modelopt-fp8-sglang-transformer`</td>
      <td>single-transformer override, deterministic latent/image comparison, H100 benchmark, torch-profiler trace</td>
      <td>SGLang converter keeps a validated BF16 fallback set for modulation and FF projection layers; use `--model-id FLUX.1-dev` for local mirrors</td>
    </tr>

    <tr>
      <td>`FP8`</td>
      <td>`black-forest-labs/FLUX.2-dev`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/flux2-dev-modelopt-fp8-sglang-transformer`</td>
      <td>single-transformer override load and generation path</td>
      <td>published SGLang-ready transformer override</td>
    </tr>

    <tr>
      <td>`FP8`</td>
      <td>`Wan-AI/Wan2.2-T2V-A14B-Diffusers`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer`</td>
      <td>primary `transformer` quantized, `transformer_2` kept BF16</td>
      <td>primary-transformer-only path; keep `transformer_2` on the base checkpoint, and do not describe this as dual-transformer full-model FP8 unless that path is validated separately</td>
    </tr>

    <tr>
      <td>`FP8`</td>
      <td>`hunyuanvideo-community/HunyuanVideo`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer`</td>
      <td>single-transformer override, BF16-vs-FP8 video comparison, H100 benchmark, torch-profiler trace</td>
      <td>HunyuanVideo uses different ModelOpt/diffusers and SGLang runtime module names; the converter maps those names before writing FP8 scale tensors and BF16 fallback ignores</td>
    </tr>

    <tr>
      <td>`FP8`</td>
      <td>`Qwen/Qwen-Image`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/qwen-image-modelopt-fp8-sglang-transformer`</td>
      <td>single-transformer override, BF16-vs-FP8 image comparison, H100 benchmark, torch-profiler trace</td>
      <td>shares the Qwen Image FP8 fallback preset; keep `img_in`, `txt_in`, timestep embedder, `norm_out.linear`, `proj_out`, `img_mod`/`txt_mod`, and `img_mlp.net.2` in BF16</td>
    </tr>

    <tr>
      <td>`FP8`</td>
      <td>`Qwen/Qwen-Image-Edit-2511`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer`</td>
      <td>TI2I edit path, BF16-vs-FP8 image comparison, H100 benchmark</td>
      <td>shares `QwenImageTransformer2DModel` with Qwen Image and uses the same Qwen Image FP8 fallback preset</td>
    </tr>

    <tr>
      <td>`NVFP4`</td>
      <td>`black-forest-labs/FLUX.1-dev`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer`</td>
      <td>mixed BF16+NVFP4 transformer override, correctness validation, 4x RTX 5090 benchmark, torch-profiler trace</td>
      <td>use `build_modelopt_nvfp4_transformer.py`; validated builder keeps selected FLUX.1 modules in BF16 and sets `swap_weight_nibbles=false`</td>
    </tr>

    <tr>
      <td>`NVFP4`</td>
      <td>`black-forest-labs/FLUX.2-dev`</td>
      <td>`--transformer-weights-path`</td>
      <td>`black-forest-labs/FLUX.2-dev-NVFP4`</td>
      <td>packed-QKV load path</td>
      <td>official raw export repo; validated packed export detection and runtime layout handling</td>
    </tr>

    <tr>
      <td>`NVFP4`</td>
      <td>`Wan-AI/Wan2.2-T2V-A14B-Diffusers`</td>
      <td>`--transformer-path`</td>
      <td>`lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer`</td>
      <td>primary `transformer` quantized with ModelOpt NVFP4, `transformer_2` kept BF16</td>
      <td>primary-transformer-only path; keep `transformer_2` on the base checkpoint; the default FP4 GEMM backend is `flashinfer_trtllm`</td>
    </tr>

    <tr>
      <td>`NVFP4`</td>
      <td>`Qwen/Qwen-Image`</td>
      <td>`--model-path`</td>
      <td>`lmsys/qwen-image-modelopt-nvfp4-sglang`</td>
      <td>full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison</td>
      <td>full repo loaded directly; exported with ModelOpt PR #1706 SVDQuant NVFP4 (`--format fp4`, max calibration, block size 16) and BF16 fallbacks for attention-sensitive modules plus first/last transformer blocks</td>
    </tr>

    <tr>
      <td>`NVFP4`</td>
      <td>`Qwen/Qwen-Image-2512`</td>
      <td>`--model-path`</td>
      <td>`lmsys/qwen-image-2512-modelopt-nvfp4-sglang`</td>
      <td>full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison, B200 CI case</td>
      <td>same full-repo loader path as Qwen Image; this is the Qwen Image NVFP4 representative in `multimodal-gen-test-1-b200`</td>
    </tr>

    <tr>
      <td>`NVFP4`</td>
      <td>`Qwen/Qwen-Image-Edit`</td>
      <td>`--model-path`</td>
      <td>`lmsys/qwen-image-edit-modelopt-nvfp4-sglang`</td>
      <td>TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison</td>
      <td>full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe</td>
    </tr>

    <tr>
      <td>`NVFP4`</td>
      <td>`Qwen/Qwen-Image-Edit-2511`</td>
      <td>`--model-path`</td>
      <td>`lmsys/qwen-image-edit-2511-modelopt-nvfp4-sglang`</td>
      <td>TI2I edit full ModelOpt NVFP4 Diffusers repo, BF16-vs-NVFP4 B200 image comparison</td>
      <td>full repo loaded directly with normal image-edit inputs; exported with the same ModelOpt PR #1706 NVFP4 recipe</td>
    </tr>
  </tbody>
</table>

These thirteen checkpoints are the intended ModelOpt documentation support
set. The B200 diffusion CI job (`multimodal-gen-test-1-b200`) uses a
representative NVFP4 subset and includes
`lmsys/qwen-image-2512-modelopt-nvfp4-sglang` for Qwen Image coverage.

## ModelOpt FP8

### Usage Examples

Converted ModelOpt FP8 transformer repos should be loaded as transformer
component overrides. If the repo or local directory already contains
`config.json`, use `--transformer-path`. Full Diffusers repos such as the
NVIDIA Wan2.2 FP8 checkpoint can be passed directly with `--model-path`.

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-path lmsys/flux2-dev-modelopt-fp8-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path lmsys/wan22-t2v-a14b-modelopt-fp8-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output
```

```bash theme={null}
sglang generate \
  --model-path hunyuanvideo-community/HunyuanVideo \
  --transformer-path lmsys/hunyuanvideo-modelopt-fp8-sglang-transformer \
  --height 544 --width 960 --num-frames 17 \
  --prompt "A cinematic shot of a red sports car driving through rain at night" \
  --save-output
```

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-path lmsys/qwen-image-modelopt-fp8-sglang-transformer \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output
```

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image-Edit-2511 \
  --transformer-path lmsys/qwen-image-edit-modelopt-fp8-sglang-transformer \
  --image-path /path/to/input.png \
  --prompt "Turn the scene into a warm watercolor illustration" \
  --save-output
```

### Notes

* `--transformer-path` is the primary-DiT convenience spelling for converted
  ModelOpt FP8 component repos or directories that already carry `config.json`.
* If the override repo or local directory contains its own `config.json`,
  SGLang reads the quantization config from that override instead of relying on
  the base model config.
* `--transformer-weights-path` still works when you intentionally point at raw
  weight files or a directory that should be metadata-probed as weights first.
* `dit_layerwise_offload` is supported for ModelOpt FP8 checkpoints.
* `dit_cpu_offload` still stays disabled for ModelOpt FP8 checkpoints.
* The layerwise offload path now preserves the non-contiguous FP8 weight stride
  expected by the runtime FP8 GEMM path.
* On disk, the quantization config stays `quant_method=modelopt` with
  `quant_algo=FP8`; the `modelopt-fp8` label in this document is a support
  family name, not a serialized config key.
* To build the converted checkpoint yourself from a ModelOpt diffusers export,
  use `python -m sglang.multimodal_gen.tools.build_modelopt_fp8_transformer`.

## ModelOpt NVFP4

### Usage Examples

For mixed ModelOpt NVFP4 transformer overrides that already contain
`config.json`, keep the base model and quantized transformer separate and use
`--transformer-path`:

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/FLUX.1-dev \
  --transformer-path lmsys/flux1-dev-modelopt-nvfp4-sglang-transformer \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

For raw NVFP4 exports such as the official FLUX.2 release, use
`--transformer-weights-path`:

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev \
  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

SGLang also supports passing the NVFP4 repo or local directory directly as
`--model-path`:

```bash theme={null}
sglang generate \
  --model-path black-forest-labs/FLUX.2-dev-NVFP4 \
  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
  --save-output
```

For a dual-transformer Wan2.2 export where only the primary `transformer`
was quantized:

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --transformer-path lmsys/wan22-t2v-a14b-modelopt-nvfp4-sglang-transformer \
  --prompt "a fox walking through neon rain" \
  --save-output
```

For full Qwen Image NVFP4 exports, load the published repo directly:

```bash theme={null}
sglang generate \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output
```

For high-resolution Qwen-Image-family generations on B200, the FlashInfer
CUTLASS FP4 GEMM backend can be faster than the default TensorRT-LLM backend:

```bash theme={null}
SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass \
sglang generate \
  --model-path lmsys/qwen-image-2512-modelopt-nvfp4-sglang \
  --width 2048 --height 2048 \
  --prompt "A tiny astronaut reading a book under a glass greenhouse" \
  --save-output
```

Qwen-Image 2.1 also supports independent NVFP4 DiT and native language-encoder
component directories through `--component-paths.transformer` and
`--component-paths.text_encoder`. Private max-calibrated exports passed B200
generation, editing, transparent RGBA, offload, and TP2 checks. These checks
do not cover arbitrary exports or RTX 5090. See its
[cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#nvfp4-components) for
component-loading instructions and hardware requirements.

### Notes

* Use `--transformer-path` for mixed ModelOpt NVFP4 transformer repos or local
  directories that already include `config.json`.
* Use `--transformer-weights-path` for raw NVFP4 exports, individual
  safetensors files, or repo layouts that should be treated as weights first.
* For dual-transformer pipelines such as `Wan2.2-T2V-A14B-Diffusers`, the
  primary `--transformer-path` override targets only `transformer`. Use a
  per-component override such as `--transformer-2-path` only when you
  intentionally want a non-default `transformer_2`.
* On B200, the diffusion ModelOpt NVFP4 path defaults to FlashInfer
  TensorRT-LLM FP4 GEMM (`flashinfer_trtllm`). On SM120, including RTX 5090,
  the default is `auto`; TensorRT-LLM FP4 GEMM does not support that architecture.
* The published Qwen Image NVFP4 exports keep the `img_mod`/`txt_mod`
  modulation projections and first/last transformer blocks in BF16.
* Qwen-Image NVFP4 does not always improve latency at 1024x1024. On B200, the
  validated ModelOpt exports were faster than BF16 at 2048x2048 with
  `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND=cutlass`, while 1024x1024
  remained BF16-faster.
* Direct `--model-path` loading is the canonical path for full Qwen Image
  ModelOpt NVFP4 repos and a compatibility path for FLUX.2 NVFP4-style repos
  or local directories.
* If `--transformer-weights-path` is provided explicitly, it takes precedence
  over the compatibility `--model-path` flow.
* For local directories, SGLang first looks for `*-mixed.safetensors`, then
  falls back to loading from the directory.
* To force the diffusion ModelOpt FP4 path onto a different FlashInfer
  backend, set `SGLANG_DIFFUSION_FLASHINFER_FP4_GEMM_BACKEND`. Supported values
  include `flashinfer_cudnn`, `flashinfer_cutlass`, and `flashinfer_trtllm`.
* On disk, the quantization config stays `quant_method=modelopt` with
  `quant_algo=NVFP4`; the `modelopt-nvfp4` label here is again a documentation
  family name rather than a serialized config key.

## GGUF

GGUF loads a community-quantized transformer from a single `.gguf` file while
the rest of the pipeline — VAE, text encoder, scheduler, tokenizer — keeps
loading from the base model.

GGUF primarily reduces checkpoint, host-memory, and resident-weight size. For
example, MiniMax-H3's transformer is 17.5 GiB as Q4\_K\_M versus 61.7 GiB as
BF16. With full layerwise offload, VAE decode and offload buffers can still
dominate peak GPU memory, but each streamed DiT layer also transfers fewer
bytes.

Packed linears reuse SRT's GGUF type definitions and CUDA dequantization, then
run the native GEMM. SRT's fused MMVQ/MMQ kernels target the low-token LLM
regime and are slower at diffusion sequence lengths. GGUF remains a
capacity-oriented option; latency depends on the quantization type, activation
shape, and placement policy.

Layers the checkpoint stores unquantized (F32/F16/BF16) take the ordinary linear
path rather than the packed one. Their precision is then whatever the model
declares for that layer, exactly as on the safetensors path — a checkpoint
cannot raise a layer above the model's own dtype by storing it wider. For
MiniMax-H3 the two agree: the layers it pins to FP32 are the ones the validated
checkpoint leaves unquantized, and `post_load_weights` fails the load if any of
them ends up narrower.

### Usage

No extra install: `gguf` is already a core SGLang dependency.

`--model-path` stays the base model;
`--component-weights-paths.<component>` takes the GGUF. A local path,
`owner/repo/file.gguf`, or `owner/repo:QUANT_TYPE` all work. The quant-type
shorthand is accepted only when exactly one repository file matches it;
otherwise SGLang lists the candidates and asks for a full path.

```bash theme={null}
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --component-weights-paths.transformer \
    leejet/MiniMax-H3-GGUF/minimax_h3_fl2va-Q4_K_M.gguf \
  --num-gpus 1 \
  --attention-backend fa \
  --performance-mode memory \
  --layerwise-offload-components dit,text_encoder \
  --dit-offload-prefetch-size 1 \
  --dit-layerwise-resident-layers 0 \
  --enable-torch-compile false \
  --port 30010
```

Here `transformer` is H3's registered DiT component name. The
`--transformer-weights-path` alias produces the same result. Note that
`--quantization gguf` is not the selector: the quantization is read from the
file itself, so passing the file is what enables the path.

MiniMax-H3 GGUF checkpoints may use either the original timestep MLP or the
pruned AdaLN curve architecture. Repositories often contain both FL2VA and
Ref2VA files, so use the full Hub file reference instead of an ambiguous
`owner/repo:QUANT_TYPE` selector. The
[H3 compatibility table](/cookbook/diffusion/MiniMax/MiniMax-H3#checkpoint-and-adapter-formats)
links both published layouts without duplicating their commands here.

The pruned architecture keeps its sampled curve and reduced AdaLN projections
in FP32, matching the published checkpoint implementation. This precision
island deliberately bypasses the BF16-only fused modulation kernels.

#### Whether to offload the VAE

Adding `vae` to `--layerwise-offload-components` trades a lot of latency for
some peak VRAM, because the video VAE decoder is re-streamed per decode tile.
Measured on 1× RTX 5090 with this checkpoint, 1344×768 × 107 frames:

| `--layerwise-offload-components` | Peak VRAM | Denoise | VAE decode |
| - | -: | -: | -: |
| `dit,text_encoder` | 26.3 GiB | 39.0 s | **9.5 s** |
| `dit,text_encoder,vae` | **19.6 GiB** | 39.0 s | 57.2 s |

Denoise is unaffected, and the output is bit-identical either way. Leave the VAE
resident unless the 6.7 GiB matters — on a 24 GB card by this measurement it
does, and the 6× slower decode is the price of fitting.

### Constraints

| Constraint | Reason |
| - | - |
| TP shard boundaries must align to GGML blocks | Column-parallel rows shard directly; row-parallel packed columns require each local input partition to contain whole quantization blocks |
| No `--use-fsdp-inference` | FSDP does not preserve the GGUF packed-block layout |
| CUDA only | The reused SRT GGML dequantization kernel currently ships for CUDA |
| Native byte order only | A quantized block embeds its scales, so a non-native file cannot be byte-swapped as a whole |
| No LoRA, and none of the H3 AdaLN cache flags | An adapter cannot be merged into packed blocks, and the AdaLN paths read the transformer's safetensors |
| No `--quantization` | The checkpoint fixes the quantization; the flag would be a second, conflicting selector |

Every constraint above fails at startup with an explanatory error rather than
silently producing wrong output.

Sequence parallelism (`--ulysses-degree` / `--ring-degree`) remains available
because it shards activations rather than packed weights. TP is also available;
startup rejects a degree that cuts a row-parallel matrix inside a GGML block.

### Validated scope

| Model | Checkpoint | Hardware | Result |
| - | - | - | - |
| MiniMax-H3 `fl2va` | [`leejet/MiniMax-H3-GGUF`](https://huggingface.co/leejet/MiniMax-H3-GGUF) `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x RTX 5090 (32 GiB) | t2va 1344x768, 107 frames, video + audio; 19.6-26.3 GiB peak depending on VAE offload |
| MiniMax-H3 `fl2va` | `minimax_h3_fl2va-Q4_K_M.gguf` (17.5 GiB) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 10.13 s and 17.17 GiB peak |
| MiniMax-H3 `fl2va`, pruned AdaLN curve | [`unsloth/MiniMax-H3-GGUF`](https://huggingface.co/unsloth/MiniMax-H3-GGUF) `minimax_h3_fl2va_pruned-Q4_K.gguf` (10.7 GiB loaded DiT) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC |
| MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 1x GB300 (CUDA 13, PyTorch 2.13) | 50-step t2va 1344x768, 107 frames, H.264 + AAC; 105.38 s and 80.88 GB peak |
| MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 2x GB300, TP2 (CUDA 13, PyTorch 2.13) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 7.55 s and 51.90 GB peak per rank |
| Qwen-Image 2.1 | Private native-name Q4\_0 exports; DiT 3.91 GiB, encoder 7.03 GiB | 1x B200; separate TP2 check | 1024px/40-step generation, editing, and transparent RGBA; each component and both together; combined offload matched resident pixels |
| Qwen-Image 2.1 | Community ComfyUI-GGUF DiT [`abenzerps/Qwen-Image-2.1-Uncensored-GGUF`](https://huggingface.co/abenzerps/Qwen-Image-2.1-Uncensored-GGUF) `qwen-image-2.1-UC-Q4_K_M.gguf` (4.6 GB) | 1x RTX 5090 (32 GiB) | 1024px/40-step text-to-image; 8.25 GB peak vs 17.0 GB for the BF16 DiT, 0.47 vs 0.43-0.48 s/step |

The unpruned MiniMax-H3 DiT loads at 17.5 GiB against 61.7 GiB for the BF16 checkpoint. Weight
fidelity was checked tensor-by-tensor against the BF16 reference: cosine
1.00000 for the F32/BF16 tensors and 0.9973 for Q4\_K/Q4\_0.

The MiniMax-H3 measurements do not validate other quantization types, the
`ref2va` partition, or a BF16-vs-GGUF output comparison.

Qwen-Image 2.1's Q4\_0 exports use the base component configs and native tensor
names. Its [cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#gguf-components)
describes how to load compatible component files and choose precision settings.
These private exports establish loading
and execution compatibility, not general output quality or compatibility with
other GGUF exports.

## Nunchaku (SVDQuant)

### Install

Install the runtime dependency first:

```bash theme={null}
pip install nunchaku
```

For platform-specific installation methods and troubleshooting, see the
[Nunchaku installation guide](https://nunchaku.tech/docs/nunchaku/installation/installation.html).

### File Naming and Auto-Detection

For Nunchaku checkpoints, `--model-path` should still point to the original
base model, while `--transformer-weights-path` points to the quantized
transformer weights.

If the basename of `--transformer-weights-path` contains the pattern
`svdq-(int4|fp4)_r{rank}`, SGLang will automatically:

* enable SVDQuant
* infer `--quantization-precision`
* infer `--quantization-rank`

Examples:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>checkpoint name fragment</th>
      <th>inferred precision</th>
      <th>inferred rank</th>
      <th>notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`svdq-int4_r32`</td>
      <td>`int4`</td>
      <td>`32`</td>
      <td>Standard INT4 checkpoint</td>
    </tr>

    <tr>
      <td>`svdq-int4_r128`</td>
      <td>`int4`</td>
      <td>`128`</td>
      <td>Higher-quality INT4 checkpoint</td>
    </tr>

    <tr>
      <td>`svdq-fp4_r32`</td>
      <td>`nvfp4`</td>
      <td>`32`</td>
      <td>`fp4` in the filename maps to CLI value `nvfp4`</td>
    </tr>

    <tr>
      <td>`svdq-fp4_r128`</td>
      <td>`nvfp4`</td>
      <td>`128`</td>
      <td>Higher-quality NVFP4 checkpoint</td>
    </tr>
  </tbody>
</table>

Common filenames:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>filename</th>
      <th>precision</th>
      <th>rank</th>
      <th>typical use</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`svdq-int4_r32-qwen-image.safetensors`</td>
      <td>`int4`</td>
      <td>`32`</td>
      <td>Balanced default</td>
    </tr>

    <tr>
      <td>`svdq-int4_r128-qwen-image.safetensors`</td>
      <td>`int4`</td>
      <td>`128`</td>
      <td>Quality-focused</td>
    </tr>

    <tr>
      <td>`svdq-fp4_r32-qwen-image.safetensors`</td>
      <td>`nvfp4`</td>
      <td>`32`</td>
      <td>RTX 50-series / NVFP4 path</td>
    </tr>

    <tr>
      <td>`svdq-fp4_r128-qwen-image.safetensors`</td>
      <td>`nvfp4`</td>
      <td>`128`</td>
      <td>Quality-focused NVFP4</td>
    </tr>

    <tr>
      <td>`svdq-int4_r32-qwen-image-lightningv1.0-4steps.safetensors`</td>
      <td>`int4`</td>
      <td>`32`</td>
      <td>Lightning 4-step</td>
    </tr>

    <tr>
      <td>`svdq-int4_r128-qwen-image-lightningv1.1-8steps.safetensors`</td>
      <td>`int4`</td>
      <td>`128`</td>
      <td>Lightning 8-step</td>
    </tr>
  </tbody>
</table>

If your checkpoint name does not follow this convention, pass
`--enable-svdquant`, `--quantization-precision`, and `--quantization-rank`
explicitly.

### Usage Examples

Recommended auto-detected flow:

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \
  --prompt "a beautiful sunset" \
  --save-output
```

Manual override when the filename does not encode the quant settings:

```bash theme={null}
sglang generate \
  --model-path Qwen/Qwen-Image \
  --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \
  --enable-svdquant \
  --quantization-precision int4 \
  --quantization-rank 128 \
  --prompt "a beautiful sunset" \
  --save-output
```

### Notes

* `--transformer-weights-path` is the primary-DiT convenience spelling used by
  Nunchaku examples. Older config names such as `quantized_model_path` are
  treated as compatibility aliases.
* Auto-detection only happens when the checkpoint basename matches
  `svdq-(int4|fp4)_r{rank}`.
* The CLI values are `int4` and `nvfp4`. In filenames, the NVFP4 variant is
  written as `fp4`.
* Lightning checkpoints usually expect matching `--num-inference-steps`, such
  as `4` or `8`.
* Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x)
  or SM12x GPUs. Hopper (SM90) is currently rejected.

## [ModelSlim](https://gitcode.com/Ascend/msmodelslim)

MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.

* **Installation**

  ```bash theme={null}
  # Clone repo and install msmodelslim:
  git clone https://gitcode.com/Ascend/msmodelslim.git
  cd msmodelslim
  bash install.sh
  ```

* **Multimodal\_sd quantization**

  Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to [Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card).

  > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).

  Run quantization using one-click quantization (recommended):

  ```bash theme={null}
  msmodelslim quant \
    --model_path /path/to/wan2_2_float_weights \
    --save_path /path/to/wan2_2_quantized_weights \
    --device npu \
    --model_type Wan2_2 \
    --quant_type w8a8 \
    --trust_remote_code True
  ```

  For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/README.md) section in ModelSLim repo.

  > Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim.

* **Auto-Detection and different formats**

  For msmodelslim checkpoints, it's enough to specify only `--model-path`, the detection of quantization occurs automatically for each layer using parsing of      `quant_model_description.json` config.

  In the case of `Wan2.2` only `Diffusers` weights storage format are supported, whereas modelslim saves the quantized model in the original `Wan2.2` format.
  For conversion, use the one-step `wan_repack.py` script:

  ```bash theme={null}
  python wan_repack.py \
    --model-type Wan2.2-TI2V-5B \
    --original-model-path {path_to_original_diffusers_model} \
    --quant-path {path_to_quantized_model} \
    --output-path {path_to_converted_model}
  ```

  Supported `--model-type` values: `Wan2.2-TI2V-5B` (single-transformer), `Wan2.2-T2V-A14B` and `Wan2.2-I2V-A14B` (Cascade dual-transformer).
  The script automatically handles: copying the base model, converting quantized weights to Diffusers format, and restoring `config.json`.

* **Usage Example**

  With auto-detected flow:

  ```bash theme={null}
  sglang generate \
    --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \
    --prompt "a beautiful sunset" \
    --save-output
  ```

* **Available Quantization Methods**:
  * [x] &#x20;`W4A4_DYNAMIC` linear with online quantization of activations
  * [x] &#x20;`W8A8` linear with offline quantization of activations
  * [x] &#x20;`W8A8_DYNAMIC` linear with online quantization of activations
  * [x] &#x20;`W8A8_MXFP8` linear with offline quantization (msmodelslim pre-quantized weights)
  * [x] &#x20;`mxfp8` linear with online quantization (`--quantization mxfp8`)
  * [x] &#x20;`W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` linear with offline quantization (msmodelslim pre-quantized weights)
  * [x] &#x20;`mxfp4_npu` linear with online quantization (`--quantization mxfp4_npu`)

## MXFP8 Quantization

Self-describing serialized MXFP8 transformers are inferred from their
checkpoint metadata. For online MXFP8 quantization, load the original
FP16/BF16 model and add `--quantization mxfp8`. NVIDIA and ROCm reuse SRT's
dense MXFP8 implementation; Ascend uses `npu_dynamic_mx_quant` and
`npu_quant_matmul` with a block size of 32.

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp8 \
  --prompt "a fox walking through neon rain" \
  --save-output
```

The selected SRT backend must provide an MXFP8 kernel on NVIDIA or ROCm. On
Ascend, the hardware requirement remains 950PR/DT Series or newer;
`npu_dynamic_mx_quant` is not available on A2/A3 Series.

## MXFP8 Offline Quantization (msmodelslim)

Pre-quantized MXFP8 weights exported by msmodelslim are auto-detected via `quant_model_description.json`
(`W8A8_MXFP8` scheme). Use `wan_repack.py` to convert the quantized weights to Diffusers format,
then load the converted model with `--model-path`:

```bash theme={null}
sglang generate \
  --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-mxfp8 \
  --prompt "a beautiful sunset" \
  --save-output
```

## MXFP4 Online Quantization

For online MXFP4 quantization on Ascend NPU, load the original FP16/BF16 model and add
`--quantization mxfp4_npu`. The `mxfp4_npu` key is used for Ascend because `mxfp4`
is reserved for the ROCm/aiter backend.

Weights are quantized at load time via `npu_dynamic_dual_level_mx_quant`, and activations
are quantized per-token during inference before `npu_dual_level_quant_matmul`. MXFP4 uses
dual-level block scales with an L1 block size of 32 and an L0 block size of 512.

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
  --quantization mxfp4_npu \
  --prompt "a fox walking through neon rain" \
  --save-output
```

> **Hardware requirement:** Ascend 950PR/DT Series or newer. `npu_dynamic_dual_level_mx_quant`
> and `npu_dual_level_quant_matmul` are not available on A2/A3 Series.
>
> **Note:** Online MXFP4 weight quantization is experimental. The offline msmodelslim
> flow uses pre-quantized weights and may produce different numerical results.

## MXFP4 Offline Quantization (msmodelslim)

Pre-quantized MXFP4 weights exported by msmodelslim are auto-detected via
`quant_model_description.json` (`W4A4_MXFP4` / `W4A4_MXFP4_DUALSCALE` scheme).
Use `wan_repack.py` to convert the quantized weights to Diffusers format, then load
the converted model with `--model-path`:

```bash theme={null}
sglang generate \
  --model-path {path_to_converted_mxfp4_model} \
  --prompt "a beautiful sunset" \
  --save-output
```

The offline MXFP4 checkpoint stores weights in an FP8 container and includes dual-level
scales (`weight_scale`, `weight_dual_scale`). If exported with smooth quantization,
`mul_scale` is loaded and applied before activation quantization to keep activations
aligned with the calibrated weights.


# Realtime and Causal Video Models
Source: https://docs.sglang.io/docs/sglang-diffusion/realtime_models



Realtime and causal video pipelines generate video incrementally and reuse state
across chunks. This differs from offline diffusion pipelines, which denoise one
bounded latent sequence and release all request state when generation finishes.

## Execution Modes

SGLang Diffusion exposes two related but distinct modes:

| Mode | Lifetime | Interface | Examples |
| - | - | - | - |
| Realtime session | State persists until the client disconnects or the session ends | `/v1/realtime_video/generate` WebSocket | LingBot World, SANA-WM realtime |
| Request-based causal generation | State is reused across chunks within one request, then released | Standard video generation API | LongLive 2.0, batch-streaming SANA-WM |

The realtime server retains model-specific state such as the causal self-attention
KV cache, cross-attention cache, decoder history, and pending control events.
State is isolated per session and is not reused by unrelated requests.

> [!NOTE]
>
> A causal DiT is not automatically a realtime session model. The pipeline must
> also register a realtime adapter and implement the WebSocket session lifecycle.

## Supported Realtime Pipelines

| Model family | Pipeline | Live controls | QVG KV-cache quantization |
| - | - | - | - |
| LingBot World | `LingBotWorldCausalDMDPipeline` | Camera actions and prompt updates | Supported |
| SANA-WM | `SanaWMRealtimePipeline` | Camera actions | Not supported |

Use the model cookbooks for launch commands, request schemas, and control-token
details:

* [LingBot World](/cookbook/diffusion/LingBot-World/LingBot-World)
* [LingBot World 2.0](/cookbook/diffusion/LingBot-World/LingBot-World-2.0)
* [SANA-WM](/cookbook/diffusion/SANA-WM/SANA-WM)

For the complete model list, see
[Supported Models](./compatibility_matrix).

## Causal Cache Controls

Realtime requests can override two model defaults:

* `realtime_causal_sink_size`: amount of stable prefix history retained as an attention sink
* `realtime_causal_kv_cache_num_frames`: recent causal history retained in the rolling KV-cache window

Larger windows preserve more history but increase resident memory and attention
work. These fields are request/session controls; supported ranges and defaults
remain model-specific.

For supported LingBot World deployments, the server-level
`--kv-cache-quant {off,int4,int2}` option compresses completed cache chunks.
It is disabled by default and is lossy when enabled. Start with `int4`; use
`int2` only when the additional memory reduction is worth the larger quality
risk.

See [Causal KV-Cache Quantization](./quantization#causal-kv-cache-quantization)
for installation, storage policy, tuning options, memory/latency tradeoffs, and
current limitations.

> [!WARNING]
>
> QVG KV-cache quantization currently supports only the LingBot realtime
> sliding-window-and-sink path. It does not apply to SANA-WM realtime, LongLive 2.0
> pinned sinks, global sinks, or dynamically growing caches.

## Deployment Considerations

* Keep `--kv-cache-quant off` when bit-exact BF16 cache behavior is required.
* Benchmark a representative session length. Short clips may not exercise cold-cache packing and can hide both its memory benefit and packing overhead.
* Treat sequence parallelism as model-specific. Follow the model cookbook and the [Sequence Parallelism](./ring_sp_performance) guide instead of assuming one mesh is best for every realtime pipeline.
* Realtime WebSocket clients must send an initialization message before control events. The exact MessagePack schema and output encoding are documented in each model cookbook.


# Sequence Parallelism
Source: https://docs.sglang.io/docs/sglang-diffusion/ring_sp_performance



Sequence parallelism splits long image or video latent sequences across GPUs. In SGLang Diffusion, the public controls are:

* `--sp-degree`: total sequence parallel degree
* `--ulysses-degree`: Ulysses parallel degree
* `--ring-degree`: ring parallel degree
* `--sp-attention-mode`: attention exchange used inside each SP group

The degrees must satisfy:

```text theme={null}
sp_degree = ulysses_degree * ring_degree
```

The default `--sp-attention-mode ulysses` uses all-to-all to redistribute
sequence shards over attention heads. `--sp-attention-mode kv_gather` keeps
queries sequence-sharded and all-gathers keys and values, then computes each
rank's local output directly. The K/V-gather mode currently supports
non-causal attention with `--ring-degree 1`. Varlen calls through the legacy
`UlyssesAttention` adapter and video sparse attention are not supported.

Use SP when sequence length or video shape makes the DiT forward pass the bottleneck and the model supports sequence sharding. For latency-oriented multi-GPU Qwen/Wan deployments, also compare against CFG parallelism and FSDP; SP is not automatically the best multi-GPU setting for every model.

## Choosing The Attention Exchange

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Mode</th>
      <th>Communication</th>
      <th>Memory</th>
      <th>Constraints</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`ulysses`</td>
      <td>All-to-all before and after attention</td>
      <td>Full sequence with a shard of the attention heads during attention</td>
      <td>Attention head divisibility must match the Ulysses degree</td>
    </tr>

    <tr>
      <td>`kv_gather`</td>
      <td>All-gather K and V; Q and output remain sequence-sharded</td>
      <td>Replicates full K and V within the SP group</td>
      <td>Non-causal attention and `ring_degree=1`; no legacy varlen or video sparse attention</td>
    </tr>
  </tbody>
</table>

Neither exchange is universally faster. K/V gather avoids the reverse
all-to-all and can help when its local attention shape or collective is more
efficient, while Ulysses can use less attention activation memory. Benchmark
both on the target model, resolution, accelerator, and interconnect.

For SP degree `P`, the approximate per-rank network payload of K/V gather
relative to Ulysses is `P / 2`, excluding each rank's local shard. The payloads
are therefore similar at SP2, while K/V gather moves about 2x as much data at
SP4 and 4x at SP8. K/V gather may still be faster when all-gather and its local
attention layout are more efficient, especially at low SP degrees, but this
scaling makes the interconnect and input shape part of the selection policy.

## Recommended Commands

### Ulysses Sequence Parallelism

The default mode needs only the total SP degree when ring parallelism is not
used:

```bash theme={null}
sglang serve \
  --model-path Qwen/Qwen-Image \
  --num-gpus 4 \
  --sp-degree 4 \
  --port 8898
```

### K/V-Gather Sequence Parallelism

Use the same SP process-group layout and select the alternative attention
exchange explicitly:

```bash theme={null}
sglang serve \
  --model-path Qwen/Qwen-Image \
  --num-gpus 4 \
  --sp-degree 4 \
  --sp-attention-mode kv_gather \
  --port 8898
```

### Tensor Plus Sequence Parallelism

TP and SP use independent dimensions. With DP and CFG parallelism disabled,
the required GPU count is `tp_size * sp_degree`. This example creates two TP
groups across a two-rank SP dimension:

```bash theme={null}
sglang serve \
  --model-path Qwen/Qwen-Image \
  --num-gpus 4 \
  --tp-size 2 \
  --sp-degree 2 \
  --sp-attention-mode kv_gather \
  --port 8898
```

Omit `--sp-attention-mode kv_gather` to use TP plus Ulysses with the same
`tp=2, sp=2` topology.

#### How TP Plus SP Works

TP and SP form orthogonal dimensions of the DiT process mesh. For `tp=2,
sp=2`, ranks `[0, 1]` and `[2, 3]` are TP groups, while ranks `[0, 2]` and
`[1, 3]` are SP groups. Each rank therefore belongs to one group of each type:

* TP shards supported attention and MLP projection weights and computation,
  then communicates partial projection results inside the TP group.
* SP shards the latent sequence and attention activations, then uses Ulysses
  or K/V gather inside the SP group.

Pure SP replicates the DiT weights on every SP rank. Adding TP reduces the
per-rank memory used by TP-sharded weights and keeps the sequence activation
sharding from SP, at the cost of adding TP communication to every applicable
DiT block. The exact memory reduction is model-dependent because not every
parameter or runtime buffer is TP-sharded.

TP plus SP should therefore be treated as a capacity and memory-latency Pareto
option, not as the default latency winner. On a single NVSwitch node, pure SP
often wins when the complete DiT weights fit on every GPU because it avoids
the repeated TP collectives. Try TP plus SP when pure SP does not fit, when
more memory headroom is required, or when its measured memory reduction is
worth a small latency increase.

The following representative eager results used eight H200 GPUs in one
NVSwitch node. Times are median scheduler-side end-to-end latency. They
illustrate the tradeoff rather than define a universal policy:

| Model and workload | Fastest tested topology | TP plus SP Pareto point | Tradeoff |
| - | - | - | - |
| Qwen-Image, 1536x1536 | CFG2xSP4 Ulysses: 972.6 ms, 62.8 GiB/GPU | CFG2xTP2xSP2 K/V: 1017.4 ms, 48.4 GiB/GPU | 4.6% slower, 22.9% less peak memory |
| Wan2.2-A14B, 832x480x81 | CFG2xSP4 K/V: 6573.5 ms, 61.9 GiB/GPU | CFG2xTP2xSP2 K/V: 7243.1 ms, 34.2 GiB/GPU | 10.2% slower, 44.7% less peak memory |
| LTX2.3, 768x512x241 | SP8 K/V: 7258.2 ms, 55.4 GiB/GPU | TP2xSP4 K/V: 10372.3 ms, 37.8 GiB/GPU | 42.9% slower, 31.8% less peak memory |

K/V gather can still improve TP plus SP at the same topology even when that
topology is not the global latency winner. In the same experiment it improved
TP2xSP4 by 4.2% for FLUX and 8.4% for LTX2.3, and improved
CFG2xTP2xSP2 by 6.0% for Qwen-Image and 2.7% for Wan2.2-A14B, relative to
Ulysses. Always compare the full candidate set, including pure SP, TP, CFG,
and their feasible combinations, rather than selecting the SP attention
backend first.

### FSDP Plus Sequence Parallelism

FSDP can shard DiT weights across the same workers that participate in SP.
Unlike TP times SP, the FSDP and SP degrees do not multiply the required GPU
count. This is useful when pure SP is fast enough but replicated DiT weights
or long-sequence activations leave too little memory headroom:

```bash theme={null}
sglang serve \
  --model-path Lightricks/LTX-2.3 \
  --num-gpus 2 \
  --use-fsdp-inference true \
  --sp-degree 2 \
  --sp-attention-mode kv_gather \
  --port 8898
```

FSDP adds weight all-gather communication, so compare it with pure SP when
both fit. K/V gather has the same non-causal and `ring_degree=1` constraints
under FSDP.

### Ring Sequence Parallelism

This example uses two GPUs with `sp=2`, `ulysses=1`, and `ring=2`.

```bash theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --num-gpus 2 \
  --sp-degree 2 \
  --ulysses-degree 1 \
  --ring-degree 2 \
  --port 8898
```

### Single-GPU Baseline

Use an explicit single-GPU baseline before attributing a gain to sequence parallelism.

```bash theme={null}
sglang serve \
  --model-path Wan-AI/Wan2.2-TI2V-5B-Diffusers \
  --num-gpus 1 \
  --sp-degree 1 \
  --ulysses-degree 1 \
  --ring-degree 1 \
  --port 8898
```

## Choosing The Degrees

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Setting</th>
      <th>Typical use</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`--sp-degree 1`</td>
      <td>Single-GPU or no sequence splitting</td>
      <td>Use this as the baseline.</td>
    </tr>

    <tr>
      <td>`--ulysses-degree N`</td>
      <td>Ulysses-only sequence parallelism</td>
      <td>When ring parallelism is not needed, keep `--ring-degree 1`.</td>
    </tr>

    <tr>
      <td>`--ring-degree N`</td>
      <td>Ring-based sequence splitting over long sequences</td>
      <td>Keep `--sp-degree` equal to `ulysses_degree * ring_degree`.</td>
    </tr>
  </tbody>
</table>

## Cross-Node Sequence Parallelism

Ulysses alone cannot scale sequence parallelism past the GPU count of one
node: going wider either violates head-count divisibility or exposes an
all-to-all across the slower inter-node link. Ring's point-to-point KV
rotation is designed to overlap with attention compute, which tolerates a
slower cross-node link far better than an all-to-all does — so the pattern
for scaling SP across nodes is **node-local Ulysses × cross-node Ring**, not
Ulysses alone.

Cross-node launches add three flags on top of the usual SP degrees:

* `--nnodes`: number of nodes. `--num-gpus` stays the *total* GPU count
  across every node; each node runs `num_gpus // nnodes` local workers.
* `--node-rank`: this node's rank, `0` on the head node (which keeps the
  HTTP/TokenizerManager surface) and `1..nnodes-1` on the others (worker-only).
* `--dist-init-addr`: a `host:port` rendezvous address reachable from every
  node — typically the head node's address.

Run the same command on every node, changing only `--node-rank`:

```bash theme={null}
# node 0 (head)
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 16 \
  --nnodes 2 \
  --node-rank 0 \
  --dist-init-addr <node0-ip>:23456 \
  --sp-degree 16 \
  --ulysses-degree 8 \
  --ring-degree 2 \
  --encoder-parallel replicate \
  --port 30010

# node 1 (worker)
sglang serve \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant ref2va \
  --num-gpus 16 \
  --nnodes 2 \
  --node-rank 1 \
  --dist-init-addr <node0-ip>:23456 \
  --sp-degree 16 \
  --ulysses-degree 8 \
  --ring-degree 2 \
  --encoder-parallel replicate \
  --port 30010
```

`--encoder-parallel replicate` is required for cross-node deployments today:
the `auto` fold decision is not yet node-boundary aware and will try to fold
the text encoder across nodes, which crashes reference-conditioned encoders.
See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel).

> [!WARNING]
>
> Cross-node ring support is model-specific, not a property of the launch
> flags alone. Confirm support in the model cookbook and see
> [Parallelism](/docs/sglang-diffusion/parallelism) for topology requirements.
> Passing `--ring-degree > 1` for a model that only
> has single-node Ulysses may either raise or, in some cases, silently
> compute incorrect output; check the model's cookbook page before assuming
> cross-node scaling is supported.

### Numerics across node boundaries

Ring's online-softmax merge across P2P hops accumulates floating-point
operations in a different order than single-node attention, so a cross-node
run is **not** expected to bit-match a single-node run of the same prompt and
seed — this is the same class of difference as choosing a different
attention backend, not a correctness regression. What *is* expected: the same
request run twice against the same cross-node deployment must produce
byte-identical output. Use that repeat-request check, not a cross-topology
comparison, to validate a cross-node deployment's determinism.

## Benchmarking Guidance

When benchmarking SP, compare the same model, precision, resolution, frame count, step count, scheduler settings, prompt type, and output path. Report both stage latency and peak GPU memory; SP can reduce per-GPU memory while adding communication overhead.

Useful metrics:

* End-to-end latency
* Denoising stage latency
* Decoding stage latency
* Peak GPU memory and peak allocated memory
* Communication or runtime overhead when available

## Reference Benchmark

The following numbers are a reference measurement for one setup. They are not a general promise for all Wan2.2 deployments.

* Model: `Wan-AI/Wan2.2-TI2V-5B-Diffusers`
* Hardware: two 48 GB RTX 40-series GPUs for sequence parallelism, one 48 GB RTX 40-series GPU for baseline
* Sequence parallel config: `sp=2, ulysses=1, ring=2` (`u1r2`)
* Baseline config: `sp=1, ulysses=1, ring=1` (`u1r1`)

### Stage Time Breakdown

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Stage / Metric</th>
      <th>`u1r2` (s)</th>
      <th>`u1r1` baseline (s)</th>
      <th>Speedup</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>InputValidation</td>
      <td>0.1060</td>
      <td>0.1029</td>
      <td>0.97x</td>
    </tr>

    <tr>
      <td>TextEncoding</td>
      <td>1.3965</td>
      <td>2.2261</td>
      <td>1.59x</td>
    </tr>

    <tr>
      <td>LatentPreparation</td>
      <td>0.0002</td>
      <td>0.0002</td>
      <td>1.00x</td>
    </tr>

    <tr>
      <td>TimestepPreparation</td>
      <td>0.0003</td>
      <td>0.0004</td>
      <td>1.33x</td>
    </tr>

    <tr>
      <td>Denoising</td>
      <td>52.6358</td>
      <td>71.6785</td>
      <td>1.36x</td>
    </tr>

    <tr>
      <td>Decoding</td>
      <td>7.6708</td>
      <td>13.4314</td>
      <td>1.75x</td>
    </tr>

    <tr>
      <td>**Total**</td>
      <td>**63.74**</td>
      <td>**90.63**</td>
      <td>**1.42x**</td>
    </tr>
  </tbody>
</table>

### Memory Usage

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Memory Metric</th>
      <th>`u1r2` (GB)</th>
      <th>`u1r1` baseline (GB)</th>
      <th>Delta</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Peak GPU Memory</td>
      <td>20.07</td>
      <td>27.40</td>
      <td>-7.33</td>
    </tr>

    <tr>
      <td>Peak Allocated</td>
      <td>13.35</td>
      <td>20.40</td>
      <td>-7.05</td>
    </tr>

    <tr>
      <td>Memory Overhead</td>
      <td>6.72</td>
      <td>7.00</td>
      <td>-0.28</td>
    </tr>

    <tr>
      <td>Overhead Ratio</td>
      <td>33.5%</td>
      <td>25.6%</td>
      <td>+7.9pp</td>
    </tr>
  </tbody>
</table>

In this setup, end-to-end latency improved from `90.63s` to `63.74s` (`1.42x`) and peak GPU memory dropped by `7.33GB`. The overhead ratio increased, so future tuning should still check communication and runtime overhead on the target hardware.

## Cross-Node Reference Benchmark

The following numbers are a reference measurement for MiniMax-H3's cross-node
Ulysses × Ring deployment. They are not a general promise for every
model or topology — see each model's cookbook page for its own verified
cross-node status.

* Model: `MiniMaxAI/MiniMax-H3`
* Hardware: 2 nodes × 8× NVIDIA H200 SXM, same cluster, InfiniBand between nodes
* Cross-node config: `--num-gpus 16 --sp-degree 16 --ulysses-degree 8 --ring-degree 2`
* Single-node baseline: `--num-gpus 8 --sp-degree 8 --ulysses-degree 8 --ring-degree 1`

Denoise-stage-only comparison, holding prompt, seed, and step count fixed:

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Task</th>
      <th>Single-node (s/step)</th>
      <th>Cross-node (s/step)</th>
      <th>Change</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>T2VA denoise</td>
      <td>0.749</td>
      <td>0.477</td>
      <td>-36.3%</td>
    </tr>

    <tr>
      <td>Ref2VA / V2V denoise</td>
      <td>2.572</td>
      <td>1.494</td>
      <td>-41.9%</td>
    </tr>
  </tbody>
</table>

The gain grows with sequence length: ring's per-hop communication cost stays
roughly constant while attention compute grows quadratically with sequence
length, so V2V's longer packed sequence benefits more than T2VA's shorter
one. With the point-to-point KV rotation pipelined against attention compute
(rather than a blocking `all_gather`), one V2V request's full denoise stage
completed in 68.1-68.3s versus 128.6s on the single-node 8-GPU baseline
(-47.0%), with byte-identical output to the unpipelined cross-node path.


# Spectrum Acceleration
Source: https://docs.sglang.io/docs/sglang-diffusion/spectrum

Approximate request-scoped denoising-step acceleration.

Spectrum forecasts DiT features and skips selected denoising steps. It is an
approximation: validate visual or video quality and latency on the exact model,
shape, hardware, and sampling settings you plan to deploy.

## Quick start

```bash theme={null}
sglang generate \
  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
  --prompt "A paper boat floating through a misty mountain lake" \
  --enable-spectrum \
  --save-output
```

## Scope and constraints

* Available on native FLUX.1, Wan, HunyuanVideo, SD3, and MiniMax-H3
  implementation paths. It is not a `--backend diffusers` feature and does
  not currently cover FLUX.2.
* The request control is available through `sglang generate` and Python
  sampling parameters. It is not a `sglang serve` or OpenAI-server request
  option yet.
* Spectrum and `--enable-teacache` are mutually exclusive. On MiniMax-H3
  it also cannot run with Cache-DiT / `quality="high"`, or with SP > 1.
* MiniMax-H3 50-step recipe: `--spectrum-warmup-steps 11 --spectrum-window-size 5 --spectrum-flex-window 1.0`. Defaults are the
  20-step schedule. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3).
* Start with the defaults. `--debug` adds shadow-prediction validation work,
  so its latency is not representative of normal Spectrum execution.

## Advanced controls

Use `--enable-spectrum` explicitly. Providing any Spectrum override also
enables it, but that implicit behavior is intended for scripts rather than new
commands.

| Flag | Default | Purpose |
| - | - | - |
| `--spectrum-window-size` | `2.0` | Initial step-skipping window |
| `--spectrum-flex-window` | `0.75` | Window growth after a real forward |
| `--spectrum-warmup-steps` | `5` | Initial exact DiT forwards |
| `--spectrum-m` | `4` | Chebyshev basis count |
| `--spectrum-lam` | `0.1` | Ridge regularization |
| `--spectrum-tau-num-steps` | `50` | Chebyshev time horizon |
| `--history-size` | `100` | Recent feature-history capacity |
| `--taylor-order` | `1` | Local predictor order (`1`, `2`, or `3`) |
| `--w` | `1.0` | Chebyshev/Taylor blend weight |

> [!WARNING]
>
> These controls trade speed against output fidelity. Change one control at a
> time and retain a lossless baseline for comparison.


# Support New Diffusion Models
Source: https://docs.sglang.io/docs/sglang-diffusion/support_new_models

A concise implementation guide for adding diffusion model families to SGLang-Diffusion.

Use this guide as a triage flow for finding the smallest change that can
support a model. Most new model work should touch a small set of files, even
though the runtime is split into separate folders.

## Read the Code in This Order

The files are split by runtime responsibility. For a new model, read the
request path first:

1. `configs/pipeline_configs/{model}.py` defines model-specific denoising and
   decoding behavior.
2. `runtime/pipelines/{model}.py` wires modules into stages.
3. `runtime/pipelines_core/stages/` runs the shared stage logic.
4. `runtime/models/` contains native model components only when the architecture
   cannot be reused.

That is the dependency direction. Avoid making a model PR that requires readers
to jump between folders in a different order.

`runtime/models/` owns modeling code: checkpoint-defined neural modules,
architecture wrappers, and weight-loading or forward-path details that are
intrinsic to one model family. Reusable serving infrastructure belongs in
SGLang-Diffusion runtime folders such as `runtime/cache/`,
`runtime/distributed/`, `runtime/utils/`, or shared pipeline stages. This
includes cache managers, graph runners, process-group transport, request
utilities, and common action-policy helpers. Model packages may call these
helpers. Keep ownership in shared runtime folders unless the code is truly
architecture-specific.

## Place helpers with their owners

Use the narrowest existing owner before adding a utility module:

| Scope | Location |
| - | - |
| Used by one file, or specific to one operation | A private helper in the consuming file; inline short one-off expressions |
| Shared within a domain | A purpose-named module in that domain, such as `runtime/realtime/video.py` or `runtime/layers/attention/mask_strategy.py` |
| Shared across domains, without model or pipeline semantics | An existing focused module under `runtime/utils/`, such as `argparse.py`, `process.py`, or `precision.py` |

Do not create a top-level `utils/` package or grow a catch-all `utils.py` or
`common.py`. Split large mixed-responsibility files along ownership boundaries,
not arbitrary line counts. A helper folder is warranted only when several
cohesive modules need it, not for a single function or hypothetical reuse.

Model code must not import pipeline stages. Put contracts shared by models and
stages in a lower-level domain module; for example, realtime cache keys belong
under `runtime/realtime/`. Keep GPU initialization, monkey patches, and model
loading out of generic utility imports.

When moving internal helpers, update all callers, tests, and cookbook examples
together. Preserve documented registration and serving entry points; do not
add re-export chains just to retain obsolete internal utility paths.

## Out-of-Tree Models and Pipelines

An installed package can register native component models and a pipeline
without modifying SGLang-Diffusion. Register them in the package's
`__init__.py`:

```python theme={null}
from sglang.multimodal_gen.registry import register_pipeline
from sglang.multimodal_gen.runtime.models.registry import ModelRegistry

from .configs import CustomPipelineConfig, CustomSamplingParams
from .pipeline import CustomPipeline


ModelRegistry.register_model(
    "CustomTransformer2DModel",
    "custom_diffusion.models:CustomTransformer2DModel",
)
register_pipeline(
    CustomPipeline,
    sampling_param_cls=CustomSamplingParams,
    pipeline_config_cls=CustomPipelineConfig,
    hf_model_paths=["my-org/custom-diffusion-model"],
    model_detectors=[lambda value: "custom-diffusion" in value.lower()],
)
```

Install the package in the server environment and set the same environment
variable used by SRT plugins:

```bash theme={null}
pip install -e /path/to/custom-diffusion
SGLANG_EXTERNAL_MODEL_PACKAGE=custom_diffusion \
  sglang serve --model-path my-org/custom-diffusion-model
```

Notes:

* The string form of `register_model` keeps component imports lazy.
* `hf_model_paths` also supports checkpoints without `model_index.json`. Other
  Diffusers checkpoints can select the pipeline through `_class_name`.
* For a standalone safetensors file, pass `--pipeline CustomPipeline`.
* Set the environment variable before startup. Each process imports the package
  once. Use `overwrite=True` only to intentionally replace a built-in pipeline.

## Start With the Smallest Change

Before adding files, decide which path fits the model.

| Situation | What to do |
| - | - |
| A new checkpoint uses an existing native family | Add the Hugging Face path and, if needed, a small `SamplingParams` or `PipelineConfig` variant. Reuse the existing pipeline and modules. |
| The model has a new native DiT/UNet architecture | Add a native SGLang pipeline and the missing model components. Keep denoising and decoding on the shared stages unless measured behavior requires model-specific logic. |
| The model is long-tail or you only need compatibility first | Prefer the Diffusers backend for compatibility-first support. Add native support later if performance or deployment needs justify it. |

Do not add a folder just to mirror the Diffusers repository layout. Add a new
file only when an existing pipeline, stage, module, config, or sampler cannot
express the behavior clearly.

## Minimal File Map

The source tree is split by runtime responsibility. That split is useful for
optimization. Keep new model PRs focused on the files required by model
behavior.

| Area | Add or edit when | Typical file |
| - | - | - |
| Runtime parameters | The request schema differs from existing models | `configs/sample/{model}.py` |
| Pipeline config | Denoising, decoding, precision, position encoding, or CFG hooks differ | `configs/pipeline_configs/{model}.py` |
| Pipeline wiring | The model needs a new stage layout or module list | `runtime/pipelines/{model}.py` |
| DiT/UNet module | The denoising network is new | `runtime/models/dits/{model}.py` |
| dVLA or policy module | The checkpoint defines a new action-policy architecture | `runtime/models/{family}/modeling_*.py` or a task-specific model subfolder |
| Shared runtime infrastructure | Cache, CUDA graph, distributed transfer, request utilities, or action-policy helpers can be reused by future models | `runtime/cache/`, `runtime/distributed/`, `runtime/utils/`, `runtime/pipelines_core/stages/` |
| Model component config | A model component has static architecture config | `configs/models/dits/{model}.py`, `configs/models/vaes/{model}.py` |
| Model-specific stage | A single stage's runtime semantics cannot be expressed by a native stage or a narrow subclass of one | `runtime/pipelines_core/stages/model_specific_stages/{model}.py` |
| Encoder, VAE, scheduler | No existing implementation can be reused | `runtime/models/encoders/`, `runtime/models/vaes/`, `runtime/models/schedulers/` |

For a new native architecture, the common minimum is:

1. `configs/sample/{model}.py`
2. `configs/pipeline_configs/{model}.py`
3. `runtime/pipelines/{model}.py`
4. `runtime/models/dits/{model}.py`

Every extra file should map to model behavior that existing code cannot express
clearly.

For dVLA or other non-image diffusion policies, keep the same ownership rule.
The policy network, VLM/action expert modules, checkpoint mapping, and
model-specific forward code belong under `runtime/models/`. Prefix caches,
request-local contexts, denoising graph runners, OpenPI-compatible transport,
and prefix/action process-group utilities should be shared SGLang-Diffusion
runtime infrastructure when they are useful beyond the first model.

## Read the Reference First

Use the model's Diffusers pipeline, official implementation, or
`model_index.json` as the source of truth. Write down:

* Which modules must be loaded: tokenizer, text encoder, image encoder,
  transformer, scheduler, VAE, processor, and any extra adapters.
* The prompt and image encoding flow.
* Latent shape, packing, scale, shift, dtype, and device rules.
* Timestep and sigma schedule.
* The exact `forward()` kwargs expected by the denoising network.
* VAE decode rules and output post-processing.

If the new model is close to Flux, Qwen-Image, GLM-Image, Wan, HunyuanVideo, or
LTX, extend that implementation before starting from an empty file.

## Video encode hand-off

Reuse `EncodeWhileDecoding` and `streamed_video_output_path` instead of adding a
model-specific ffmpeg writer. Explicit `DecodingStage.decode(..., on_frames=...)`
calls can hand over the completed RGB video for VAEs without callbacks; no decoder
modification is needed. This whole-clip path is not automatically enabled:
ordinary decoders already use GPU-direct saving and may not benefit.
To overlap encoding with existing incremental decoding, a native VAE can declare
`supports_decode_on_frames = True` and accept `decode(..., on_frames=callback)`.
Emit read-only `[1, 3, T, H, W]` runs in presentation order, only after overlap
blending and cropping are final. Concatenating these runs must exactly reproduce
the normal decode. Batched decoders can emit the complete batch once; the shared
MP4 path currently only accepts a single output.

Match the model's existing normalization, dtype, and uint8 conversion order.
Audio must be available before opening the shared encoder. Keep component-use
ordering identical across ranks, including ranks that do not write output.
Encoder failures must retain the regular decode result for saving afterward.

An untiled decoder may emit once at completion, reducing CPU frame materialization
but not overlapping encoding with decode. Do not enable temporal tiling just to
gain overlap: it can change the decoded video. Compare callback-on and callback-off
with identical tiling, audio, encoding settings, and full request timing.

## Conditioning reuse

Use the native `TextEncoder` / `ImageEncoder` base classes (or `EncoderTensorParallelMixin`) and call encoders through `encoder(...)`. This preserves the shared conditioning cache, module hooks, and encoder TP/folding group. A replicated custom encoder can use `ConditioningEncoderMixin`; do not apply it to a module with hidden mutable state or random sampling in `forward()`.

The standard `TextEncodingStage` caches postprocessed embeddings, conditioning masks, sequence lengths, and pooled outputs instead of unused intermediate hidden states. Keep its postprocessing deterministic. When adding a stage-level `cached_encoder_call`, place deterministic preprocessing and component preparation inside the compute callback and give the stage a `namespace`. The namespace avoids a duplicate full encoder entry while preserving nested vision-method caches. Declare conditional component uses with `start_at_stage_entry=False`, so the executor cannot load weights before checking the cache. Keep calling `encoder(...)` inside the callback to preserve hooks and parallel-group handling. After retrieving the conditioning, call `finish_unused_declared_component()` to release weights retained by warmup or the previous request when no encoder use began; an active use remains owned by the residency manager. Batch-DP retains per-copy caches. A gathered group result may skip encoding and the output gather only after hit consensus across the whole replica; a miss keeps every rank in the normal gather path.

For a custom VAE, apply `cached_vae_encode` only to the deterministic encode method returning a tensor or posterior. Keep posterior sampling, normalization, initial noise, scheduler state, and request-specific KV dictionaries outside that boundary. Unsupported outputs bypass caching; explicitly register a tensor-only posterior container with `register_conditioning_container` when needed. Never register an autoregressive cache or a session object as a posterior container.

For integrated models without a separate encoder, use `cached_conditioning` on a pure conditioning method, as in Cosmos3's understanding pathway or SenseNova-U1's reference-image features. Every rank in the current encoder TP group must enter that method together. Do not decorate the noisy-image or denoising path.

Prefer these function boundaries to whole-stage deduplication. A stage may also write per-request metadata, advance RNG state, release materials, or execute collectives; these actions must still run for each request. The executor gives each grouped or batched stage a temporary device-result scope, using the same keys and invalidation as the cross-request cache. Use `share_in_group=True` only for consumed tensors that remain read-only across requests; containers are copied. Other outputs get private tensor snapshots with a bounded temporary budget. The scope ends at stage exit, including exceptions, and remains available with `--disable-conditioning-cache`.

Wrap negative encoding in `prefer_conditioning_cache()` to prioritize reusable conditioning. At a consumed `cached_encoder_call` boundary with a stage namespace and `share_in_group=True`, this keeps a private device snapshot across requests, including library encoders and gathered batch-DP results. Hits skip encoder preparation and postprocessing, with the same rank consensus and invalidation as CPU entries. Device and CPU entries share one byte and entry budget; raw intermediate encoder states remain on the CPU. Avoid adding a separate model-owned negative cache.

Do not put `Req`, scheduler instances, sampled latents, or session state into a cross-request cache. Keep model-specific grouped execution for parallel work distribution and material ownership. Preserve batch shapes and padding when matching batched encoder calls; per-row regrouping needs separate numerical validation.

The common `TextEncodingStage` retains stage deduplication to avoid repeated tokenization and stage bookkeeping. If you override its `forward`, declare a complete stage-output contract or set `deduplicated_output_fields = ()` and use conditioning-result reuse. Do not inherit the common stage's output list when your stage also writes model-specific metadata. Keep request-local containers independent even when their tensors are shared.

Test cache-on versus cache-off with identical seeds, changed prompts/masks/reference pixels (including alpha), downstream in-place mutations, changed weights, and rank-local eviction under supported parallel groups. Also exercise grouped requests through the executor with a zero or undersized cache budget: check encoder call counts, per-request metadata, output ownership, and group cleanup on failure. Include the new model in the [conditioning cache coverage](/docs/sglang-diffusion/caching-acceleration#coverage-and-limits).

Preserve existing grouped-request encoding reuse when cross-request caching is unavailable. Use `cross_request=False` for boundaries that support only group-local reuse, such as library fallback encoders and gathered batch-DP outputs. FSDP scopes allow consumed-output group reuse while bypassing nested encoder caches. A boundary containing collectives needs hit consensus across every participating rank, including FSDP shards and encoder DP copies; an encoder TP group alone may be insufficient. Test these bypass paths and single-rank invalidation before removing stage deduplication.

## Choose a Pipeline Shape

SGLang-Diffusion uses `ComposedPipelineBase` to wire stages together. Most
native pipelines should choose the least invasive stage shape that preserves the
runtime semantics.

| Shape | Use when | Layout |
| - | - | - |
| Native stages | Text/image encoding, latent prep, timestep prep, denoising, and decoding match existing helpers | `add_standard_t2i_stages()`, `add_standard_ti2i_stages()`, or a similar helper |
| Native-stage subclass | One stage has model-specific details, but the stage boundary and batch contract still match an existing native stage | `{Model}TextEncodingStage(TextEncodingStage) -> LatentPreparationStage -> TimestepPreparationStage -> DenoisingStage -> DecodingStage` |
| Custom single-purpose stage | One step has a different state owner or batch-field lifecycle and cannot cleanly inherit from a native stage | Native stages with one `{Model}{Purpose}Stage` inserted or substituted |
| Aggregated custom stage | Several preparation steps are inseparable in the reference pipeline and cannot be split without fragile duplicate state | `{Model}BeforeDenoisingStage -> DenoisingStage -> DecodingStage` |

Prefer this order:

1. **Use native stages directly.** This keeps the model on shared code paths for
   offload, component readiness, profiling, disaggregation, batching, and future
   stage-level optimizations.
2. **Subclass the narrowest native stage.** If only prompt processing differs,
   inherit from `TextEncodingStage`. If only latent setup, timestep setup,
   denoising, or decode differs, inherit from that specific native stage. Preserve
   the existing input/output fields whenever possible.
3. **Add a custom single-purpose stage only when no native stage contract fits.**
   Keep the stage owner narrow: one stage should own one coherent transformation,
   such as a custom condition assembly step or a model-specific policy/action
   bridge.
4. **Use an aggregated `BeforeDenoisingStage` only as a last resort.** This is the
   least preferred shape because it hides multiple runtime responsibilities in
   one stage, increases code size and review cost, and bypasses shared hooks for
   offload, profiling, disaggregation, batching, and future stage-level
   optimizations.

## Implement the Pieces

### 1. Sampling Params

Create request parameters only for values users can set at runtime.

```python theme={null}
# python/sglang/multimodal_gen/configs/sample/my_model.py
from dataclasses import dataclass

from sglang.multimodal_gen.configs.sample.sampling_params import ImageSamplingParams


@dataclass
class MyModelSamplingParams(ImageSamplingParams):
    guidance_scale: float = 4.0
    num_inference_steps: int = 28
```

### 2. Pipeline Config

`PipelineConfig` is where shared denoising and decoding stages get model-specific
callbacks.

```python theme={null}
# python/sglang/multimodal_gen/configs/pipeline_configs/my_model.py
from dataclasses import dataclass, field


@dataclass
class MyModelPipelineConfig(ImagePipelineConfig):
    task_type: ModelTaskType = ModelTaskType.T2I
    should_use_guidance: bool = True
    dit_config: DiTConfig = field(default_factory=MyModelDiTConfig)
    vae_config: VAEConfig = field(default_factory=MyModelVAEConfig)

    def prepare_pos_cond_kwargs(self, batch, latent_model_input, t, **kwargs):
        return {
            "hidden_states": latent_model_input,
            "encoder_hidden_states": batch.prompt_embeds[0],
            "timestep": t,
        }

    def prepare_neg_cond_kwargs(self, batch, latent_model_input, t, **kwargs):
        return {
            "hidden_states": latent_model_input,
            "encoder_hidden_states": batch.negative_prompt_embeds[0],
            "timestep": t,
        }
```

Make these kwargs match the denoising module's `forward()` signature exactly.

### 3. Pipeline Wiring

Use the standard helper when the model fits it.

```python theme={null}
# python/sglang/multimodal_gen/runtime/pipelines/my_model.py
class MyModelPipeline(LoRAPipeline, ComposedPipelineBase):
    pipeline_name = "MyModelPipeline"

    _required_config_modules = [
        "text_encoder",
        "tokenizer",
        "transformer",
        "scheduler",
        "vae",
    ]

    def create_pipeline_stages(self, server_args: ServerArgs):
        self.add_standard_t2i_stages()


EntryClass = [MyModelPipeline]
```

When a standard helper is not enough, first check whether only one native stage
needs model-specific behavior. In that case, subclass that stage and keep the
rest of the pipeline standard. For example, custom tokenization or prompt-window
logic should usually inherit from `TextEncodingStage` directly.

```python theme={null}
class MyModelPipeline(LoRAPipeline, ComposedPipelineBase):
    pipeline_name = "MyModelPipeline"

    _required_config_modules = [
        "text_encoder",
        "tokenizer",
        "transformer",
        "scheduler",
        "vae",
    ]

    def create_pipeline_stages(self, server_args: ServerArgs):
        self.add_stage(InputValidationStage())
        self.add_stage_factory(
            RoleType.ENCODER,
            lambda: MyModelTextEncodingStage(
                text_encoder=self.get_module("text_encoder"),
                tokenizer=self.get_module("tokenizer"),
            ),
            "my_model_text_encoding_stage",
        )
        self.add_standard_latent_preparation_stage()
        self.add_standard_timestep_preparation_stage()
        self.add_standard_denoising_stage()
        self.add_standard_decoding_stage()


EntryClass = [MyModelPipeline]
```

Use a custom single-purpose stage only when the reference pipeline has one step
that cannot be represented cleanly by a hook or native-stage subclass. Keep the
custom stage narrow and reuse native stages before and after it.

```python theme={null}
class MyModelPipeline(LoRAPipeline, ComposedPipelineBase):
    pipeline_name = "MyModelPipeline"

    _required_config_modules = [
        "text_encoder",
        "tokenizer",
        "transformer",
        "scheduler",
        "vae",
    ]

    def create_pipeline_stages(self, server_args: ServerArgs):
        self.add_stage(InputValidationStage())
        self.add_standard_text_encoding_stage()
        self.add_stage_factory(
            RoleType.ENCODER,
            lambda: MyModelConditioningStage(
                scheduler=self.get_module("scheduler"),
                vae=self.get_module("vae"),
            ),
            "my_model_conditioning_stage",
        )
        self.add_standard_latent_preparation_stage()
        self.add_standard_timestep_preparation_stage()
        self.add_standard_denoising_stage()
        self.add_standard_decoding_stage()


EntryClass = [MyModelPipeline]
```

Use an aggregated `BeforeDenoisingStage` only when the reference pipeline couples
several preparation steps so tightly that splitting them would require fragile
duplicate state or extra synchronization. Do not start with this shape.

```python theme={null}
class MyModelPipeline(LoRAPipeline, ComposedPipelineBase):
    pipeline_name = "MyModelPipeline"

    _required_config_modules = [
        "text_encoder",
        "tokenizer",
        "transformer",
        "scheduler",
        "vae",
    ]

    def create_pipeline_stages(self, server_args: ServerArgs):
        self.add_stage(InputValidationStage())
        self.add_stage(
            MyModelBeforeDenoisingStage(
                text_encoder=self.get_module("text_encoder"),
                tokenizer=self.get_module("tokenizer"),
                scheduler=self.get_module("scheduler"),
                vae=self.get_module("vae"),
            )
        )
        self.add_standard_denoising_stage()
        self.add_standard_decoding_stage()


EntryClass = [MyModelPipeline]
```

#### Reuse component loaders

Most components should keep the default loader for their role (transformer,
text encoder, VAE, scheduler, or tokenizer). For an auxiliary module that accepts
`model_cls(**config)` and loads an unchanged state dict, select
`PlainStateDictComponentLoader` instead of adding a model-specific loader class:

```python theme={null}
from sglang.multimodal_gen.runtime.loader.component_loaders.component_loader import (
    PlainStateDictComponentLoader,
)


class MyModelPipeline(ComposedPipelineBase):
    _required_config_modules = ["transformer", "queryformer", "text_projection"]
    component_loaders = {
        "queryformer": PlainStateDictComponentLoader,
        "text_projection": PlainStateDictComponentLoader,
    }

    # Wire the stages as in the examples above.
```

The mapping uses exact pipeline component names, including aliases declared in
`_extra_config_module_map`. It is local to this pipeline; omitted components keep
their existing dispatch. An explicitly selected loader fails on errors instead
of falling back to a different implementation.

Register each module through `EntryClass` in its model file, or
`ModelRegistry.register_model` for an out-of-tree package. The shared loader:

* Resolves the registered class from `config.json`'s `_class_name`, falling back
  to the architecture in `model_index.json`, and passes non-metadata config
  fields to its constructor.
* Loads a single safetensors file or an indexed sharded checkpoint using the
  shared checkpoint selector, with strict key and shape checks.
* Honors exact overrides such as `--component-weights-paths.queryformer PATH`
  and `--component-precisions.queryformer bf16`. Precision defaults to the
  pipeline's `dit_precision`.
* Leaves eval mode and final CPU/GPU placement to the shared loading lifecycle.
  It does not implement TP/FSDP weight sharding, quantization, or direct GPU
  loading. Explicit quantization and direct-GPU-loading overrides are rejected.

Keep a specialized loader when the component needs a config-object constructor,
weight remapping or fusion, sharding, or a non-standard checkpoint format. Such
a loader can also be selected through `component_loaders`; registering a model
alone does not opt it into the plain state-dict protocol.

### 4. Last-Resort Before-Denoising Stage

A `BeforeDenoisingStage` is not a catch-all replacement for the native stages.
Use it when the model has custom latent packing, conditioning assembly, timestep
preparation, or request-local state that does not fit `LatentPreparationStage` or
`TimestepPreparationStage`, and only after checking whether the work can be a
native-stage subclass or a custom single-purpose stage. If the difference is
prompt handling, subclass `TextEncodingStage` instead.

A proper `BeforeDenoisingStage` should populate the batch fields consumed by
`DenoisingStage`.

```python theme={null}
class MyModelBeforeDenoisingStage(PipelineStage):
    @torch.no_grad()
    def forward(self, batch: Req, server_args: ServerArgs) -> Req:
        prompt_embeds, negative_prompt_embeds = self._encode_prompt(batch)
        latents = self._prepare_latents(batch)
        timesteps, sigmas = self._prepare_timesteps(batch)

        batch.prompt_embeds = [prompt_embeds]
        batch.negative_prompt_embeds = [negative_prompt_embeds]
        batch.latents = latents
        batch.timesteps = timesteps
        batch.num_inference_steps = len(timesteps)
        batch.sigmas = sigmas.tolist()
        batch.raw_latent_shape = latents.shape
        return batch
```

Required fields for `DenoisingStage`:

| Field | Notes |
| - | - |
| `batch.latents` | Initial latent tensor, including any packing required by the model. |
| `batch.timesteps` | Timestep tensor in the exact order used by the reference pipeline. |
| `batch.sigmas` | Python list when the scheduler expects sigma values. |
| `batch.prompt_embeds` | Positive embeddings, wrapped in a list. |
| `batch.negative_prompt_embeds` | Negative embeddings, wrapped in a list when CFG is used. |
| `batch.num_inference_steps` | Number of denoising iterations. |
| `batch.raw_latent_shape` | Original latent shape before packing, if decode needs it. |

### 5. Distributed and memory integration

Single-GPU parity is only the first milestone. Complete native support also
requires:

* **Encoder and DiT TP/SP:** use native parallel projections and sharded weight
  loading for TP, and `USPAttention` for SP. Handle masks, RoPE, padding, and
  output gathering without falling back to a replicated full model. TP and SP
  must work together.
* **VAE parallel decode:** subclass `ParallelTiledVAE`, or reuse an existing
  native base with the same contract. Support tiled and `spatial_shard` decode
  through `DecodingStage` and the shared decode group. Reuse
  `runtime/layers/parallel_conv.py` and
  `runtime/models/vaes/parallel/diffusers_spatial.py` where applicable.
* **Layerwise offload:** every loaded neural module must inherit
  `LayerwiseOffloadableModuleMixin` and list all repeated block paths in
  `layer_names`. Set `layerwise_offload_dit_group_enabled = False` for non-DiT
  modules. Component CPU offload is not a substitute.

See `wanvideo.py` and `qwen_image.py` for DiT TP/SP, `gemma_3.py` for encoder TP
and offload, and `autoencoder_kl_qwenimage.py` or `ltx_2_vae.py` for VAE decode.

For a projection over independently sharded inputs, such as concatenated
attention and MLP outputs, use
`RowParallelLinear(attn_dim + mlp_dim, out_dim, input_sizes=[attn_dim, mlp_dim])`.
Pass each segment's **full, pre-TP width**. Each width must be divisible by TP,
and the input must already contain the rank-local segments. The shared loader
selects the matching checkpoint columns; do not replace parameter loaders in
the model. Keep `input_sizes=None` for an ordinary contiguous input shard.

The Diffusers backend is compatibility-first and does not need to meet this
native integration contract.

### 6. Registry

Define a `register()` function in `configs/pipeline_configs/{model}.py`. The
runtime auto-discovers it on startup and calls it to register the sampling
params and pipeline config.

```python theme={null}
# python/sglang/multimodal_gen/configs/pipeline_configs/my_model.py
def register():
    from sglang.multimodal_gen.registry import register_configs

    register_configs(
        sampling_param_cls=MyModelSamplingParams,
        pipeline_config_cls=MyModelPipelineConfig,
        hf_model_paths=["org/my-model"],
        model_detectors=[lambda hf_id: "my-model" in hf_id.lower()],
    )
```

`model_detectors` matches a model path or `model_index.json` `_class_name` when
the Hugging Face path varies; see `wan.py` or `qwen_image21.py` for real
examples. The pipeline file is discovered through its `EntryClass`; do not add
a second pipeline registry unless the existing registry requires it.

## Verify the Port

Use one deterministic prompt and seed while comparing with the reference
implementation.

1. Run a single-GPU smoke test and check that the output contains coherent
   content.
2. Compare latent scale and shift, timestep order, sigma values, and conditioning
   kwargs against Diffusers or the official implementation.
3. Compare VAE decode separately, including tiled and multi-GPU `spatial_shard`.
4. Run encoder and DiT TP, SP, combined TP x SP, and
   `--layerwise-offload-components all`; compare with the single-GPU resident
   baseline.
5. If the model supports LoRA, CFG parallelism, or disaggregation, test each
   feature explicitly.
6. Add or update the cookbook, examples, and
   [Supported Models](./compatibility_matrix) catalog when users need a new
   launch command.

Common failure points:

* Wrong latent scale or shift.
* Reversed or dtype-mismatched timesteps.
* Missing negative embeddings when CFG is enabled.
* Conditioning kwarg names mismatched with the DiT `forward()`.
* Rotary embedding shape or style mismatch.
* Decoding packed latents without restoring `raw_latent_shape`.

## PR Checklist

* [ ] Reused an existing family, stage, module, scheduler, or VAE wherever
  possible.
* [ ] Kept the new-model touch surface small and justified any extra files.
* [ ] Added `SamplingParams`, `PipelineConfig`, pipeline wiring, DiT module, and
  registry entry when native support is needed.
* [ ] Confirmed `pipeline_name` matches the Diffusers `model_index.json`
  `_class_name` when applicable.
* [ ] Confirmed `_required_config_modules` matches the model repo.
* [ ] Verified image or video quality against a reference output.
* [ ] Completed the distributed and memory integration checks above.
* [ ] Tested CFG parallelism and distributed serving paths when they apply.

## Declare multiple task types

Keep `PipelineConfig.task_type` as the default task. To let one configured
pipeline serve several tasks, declare its capabilities as a class variable:

```python theme={null}
from dataclasses import dataclass
from typing import ClassVar

from sglang.multimodal_gen.configs.pipeline_configs.base import PipelineConfig
from sglang.multimodal_gen.configs.task_type import ModelTaskType


@dataclass
class MultiTaskPipelineConfig(PipelineConfig):
    task_type: ModelTaskType = ModelTaskType.T2V
    supported_task_types: ClassVar[tuple[ModelTaskType, ...]] = (
        ModelTaskType.T2I,
        ModelTaskType.I2I,
        ModelTaskType.T2V,
        ModelTaskType.I2V,
    )
```

The declaration must be nonempty, contain no duplicates, and include the
configured default. Existing pipelines that omit it retain the singleton
`(task_type,)`. Override `get_supported_task_types()` if capabilities depend
on the loaded checkpoint or configuration, preserving these invariants.
Flat CLI/Python `task_type` arguments select the request. To change the
configured default, use `pipeline_config={"task_type": ModelTaskType.I2V}`
(or its serialized configuration), choosing a supported task.

Capabilities describe implemented execution paths; declaring a task does not
load additional components or implement conditioning and decoding for it.

Requests carry `SamplingParams.task_type`, accepting an enum or its name
(case-insensitive). Admission resolves it to an enum and derives the output
`data_type`. Explicit tasks must belong to the supported set and match the
HTTP endpoint's output type. Without an explicit task, admission filters by
output type and conditioning inputs, prefers a compatible default, then
selects a unique compatible alternative. Python/CLI requests also prefer the
default's output type among alternatives. Ambiguous requests must specify
`task_type`. No request changes the shared configuration.

Use the admitted `batch.task_type` in model stages when behavior depends on
the request. Shared stages use `get_request_task_type(batch, pipeline_config)`
to preserve compatibility with direct callers that have no resolved task.
Audit model-specific frame adjustment, reference loading, latent construction,
and decoding for assumptions about the server default. Declare the union of
components used by all supported tasks in stage residency metadata. Synthetic
warmup uses the default task; provide the pipeline's warmup hook when that task
needs model-specific reference inputs. Dynamic batching compares the admitted
task along with the other sampling fields.

`I2I` and `I2V` require image input; `T2I` and `T2V` reject it. Existing `TI2I`
and `TI2V` retain optional image conditioning. `V2V` and `F2V` require video
input; `F2V` denotes video-prefix conditioning. Models own the details of
prefix length and temporal alignment. Pipelines with explicit capabilities
reject video input for tasks that do not accept it; legacy singleton pipelines
retain their model-owned video checks.

See [request task selection](/docs/sglang-diffusion/api/openai_api#select-a-request-task)
for the HTTP interface. Test task selection, invalid inputs, discovery,
serialization, batching, and alternating tasks on one running server.


# TeaCache Acceleration
Source: https://docs.sglang.io/docs/sglang-diffusion/teacache

Configure TeaCache for temporal similarity-based diffusion acceleration.

> **Note**: This is one of two caching strategies available in SGLang.
> For an overview of all caching options, see [caching](./caching-acceleration).

TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.

## Overview

TeaCache works by:

1. Tracking the L1 distance between modulated inputs across consecutive timesteps
2. Accumulating the rescaled L1 distance over steps
3. When accumulated distance is below a threshold, reusing the cached residual
4. Using separate positive/negative caches for supported CFG model families

## How It Works

### L1 Distance Tracking

At each denoising step, TeaCache computes the relative L1 distance between the current and previous modulated inputs:

```text theme={null}
rel_l1 = |current - previous|.mean() / |previous|.mean()
```

This distance is then rescaled using polynomial coefficients and accumulated:

```text theme={null}
accumulated += poly(coefficients)(rel_l1)
```

### Cache Decision

* If `accumulated >= threshold`: Force computation, reset accumulator
* If `accumulated < threshold`: Skip computation, use cached residual

### CFG Support

For models that support CFG cache separation, TeaCache maintains separate caches for positive and negative branches:

* `previous_modulated_input` / `previous_residual` for positive branch
* `previous_modulated_input_negative` / `previous_residual_negative` for negative branch

For models that do not support CFG separation, TeaCache is automatically disabled when CFG is enabled.

## Configuration

TeaCache is configured via `TeaCacheParams` in the sampling parameters:

```python theme={null}
from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams

params = TeaCacheParams(
    teacache_thresh=0.1,           # Threshold for accumulated L1 distance
    coefficients=[1.0, 0.0, 0.0],  # Polynomial coefficients for L1 rescaling
)
```

### Parameters

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Parameter</th>
      <th>Type</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>`teacache_thresh`</td>
      <td>float</td>
      <td>Threshold for accumulated L1 distance. Higher = more caching, faster but potentially lower quality</td>
    </tr>

    <tr>
      <td>`coefficients`</td>
      <td>list\[float]</td>
      <td>Polynomial coefficients for L1 rescaling. Model-specific tuning</td>
    </tr>
  </tbody>
</table>

### Model-Specific Configurations

Different models may have different optimal configurations. The coefficients are typically tuned per-model to balance speed and quality.

## Supported Models

TeaCache support status by model family:

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>CFG Cache Separation</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>Wan2.1</td>
      <td>Yes</td>
      <td>Full support</td>
    </tr>

    <tr>
      <td>Wan2.2</td>
      <td>Yes</td>
      <td>Coefficients are not calibrated yet; enabling TeaCache is accepted but currently no-ops</td>
    </tr>

    <tr>
      <td>Z-Image</td>
      <td>Yes</td>
      <td>Full support</td>
    </tr>

    <tr>
      <td>HunyuanVideo</td>
      <td>No</td>
      <td>Not supported yet</td>
    </tr>

    <tr>
      <td>Flux</td>
      <td>No</td>
      <td>To be supported</td>
    </tr>

    <tr>
      <td>Qwen</td>
      <td>No</td>
      <td>To be supported</td>
    </tr>
  </tbody>
</table>

## References

* [TeaCache: Accelerating Diffusion Models with Temporal Similarity](https://arxiv.org/abs/2411.14324)


# Supported models
Source: https://docs.sglang.io/docs/supported-models

See which families of SGLang-compatible models are actively maintained.

SGLang supports model families across text generation, retrieval, and reward workflows. Browse the sections below for the primary product paths and jump to the detail pages when you are ready to explore a specific class.

### Text generation

> **[Large language models](./supported-models/generative_models)**
>
> Production-tuned Llama and Qwen families validated for high-throughput
> serving.

> **[Vision language models](./supported-models/multimodal_language_models)**
>
> Vision-text hybrids that stay responsive on multi-GPU setups.

> **[Diffusion language models](./sglang-diffusion/index)**
>
> Score-based and diffusion backbones for structured text generation
> workflows.

### Retrieval and ranking

> **[Embedding models](./supported-models/embedding_models)**
>
> Dense and sparse embeddings optimized with FlashInfer kernels.

> **[Rerank models](./supported-models/rerank_models)**
>
> Low-latency rerankers for multi-stage retrieval pipelines.

> **[Classification models](./supported-models/classify_models)**
>
> Lightweight classifiers covering safety, intent, and context filters.

### Specialized models

> **[Reward models](./supported-models/reward_models)**
>
> RLHF and reward scoring pipelines optimized for production latency.

> **[Decision models](./supported-models/decision_models)**
>
> Typed choice, score, and yes or no answers with a probability for every option.


# Classification Models
Source: https://docs.sglang.io/docs/supported-models/classify_models



This document describes the `/v1/classify` API endpoint implementation in SGLang, which is compatible with vLLM's classification API format.

## Overview

The classification API allows you to classify text inputs using classification models. This implementation follows the same format as vLLM's 0.7.0 classification API.

## API Endpoint

```text Output theme={null}
POST /v1/classify
```

## Request Format

```json Config theme={null}
{
  "model": "model_name",
  "input": "text to classify"
}
```

### Parameters

* `model` (string, required): The name of the classification model to use
* `input` (string, required): The text to classify
* `user` (string, optional): User identifier for tracking
* `rid` (string, optional): Request ID for tracking
* `priority` (integer, optional): Request priority

## Response Format

```json Config theme={null}
{
  "id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
  "object": "list",
  "created": 1745383213,
  "model": "jason9693/Qwen2.5-1.5B-apeach",
  "data": [
    {
      "index": 0,
      "label": "Default",
      "probs": [0.565970778465271, 0.4340292513370514],
      "num_classes": 2
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "total_tokens": 10,
    "completion_tokens": 0,
    "prompt_tokens_details": null
  }
}
```

### Response Fields

* `id`: Unique identifier for the classification request
* `object`: Always "list"
* `created`: Unix timestamp when the request was created
* `model`: The model used for classification
* `data`: Array of classification results
  * `index`: Index of the result
  * `label`: Predicted class label
  * `probs`: Array of probabilities for each class
  * `num_classes`: Total number of classes
* `usage`: Token usage information
  * `prompt_tokens`: Number of input tokens
  * `total_tokens`: Total number of tokens
  * `completion_tokens`: Number of completion tokens (always 0 for classification)
  * `prompt_tokens_details`: Additional token details (optional)

## Example Usage

### Using curl

```bash Command theme={null}
curl -v "http://127.0.0.1:8000/v1/classify" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jason9693/Qwen2.5-1.5B-apeach",
    "input": "Loved the new café—coffee was great."
  }'
```

### Using Python

```python Example theme={null}
import requests
import json

# Make classification request
response = requests.post(
    "http://127.0.0.1:8000/v1/classify",
    headers={"Content-Type": "application/json"},
    json={
        "model": "jason9693/Qwen2.5-1.5B-apeach",
        "input": "Loved the new café—coffee was great."
    }
)

# Parse response
result = response.json()
print(json.dumps(result, indent=2))
```

## Supported Models

The classification API works with any classification model supported by SGLang, including:

### Classification Models (Multi-class)

* `LlamaForSequenceClassification` - Multi-class classification
* `Qwen2ForSequenceClassification` - Multi-class classification
* `Qwen3ForSequenceClassification` - Multi-class classification
* `BertForSequenceClassification` - Multi-class classification
* `Gemma2ForSequenceClassification` - Multi-class classification

**Label Mapping**: The API automatically uses the `id2label` mapping from the model's `config.json` file to provide meaningful label names instead of generic class names. If `id2label` is not available, it falls back to `LABEL_0`, `LABEL_1`, etc., or `Class_0`, `Class_1` as a last resort.

### Reward Models (Single score)

* `InternLM2ForRewardModel` - Single reward score
* `Qwen2ForRewardModel` - Single reward score
* `LlamaForSequenceClassificationWithNormal_Weights` - Special reward model

**Note**: The `/classify` endpoint in SGLang was originally designed for reward models but now supports all non-generative models. Our `/v1/classify` endpoint provides a standardized vLLM-compatible interface for classification tasks.

## Error Handling

The API returns appropriate HTTP status codes and error messages:

* `400 Bad Request`: Invalid request format or missing required fields
* `500 Internal Server Error`: Server-side processing error

Error response format:

```json Config theme={null}
{
  "error": "Error message",
  "type": "error_type",
  "code": 400
}
```

## Implementation Details

The classification API is implemented using:

1. **Rust Model Gateway**: Handles routing and request/response models in `sgl-model-gateway/src/protocols/spec.rs`
2. **Python HTTP Server**: Implements the actual endpoint in `python/sglang/srt/entrypoints/http_server.py`
3. **Classification Service**: Handles the classification logic in `python/sglang/srt/entrypoints/openai/serving_classify.py`

## Testing

Use the provided test script to verify the implementation:

```bash Command theme={null}
python test_classify_api.py
```

## Compatibility

This implementation is compatible with vLLM's classification API format, allowing seamless migration from vLLM to SGLang for classification tasks.


# Decision models
Source: https://docs.sglang.io/docs/supported-models/decision_models

Answer typed choice, score, and yes or no questions with a probability for every option from a chat model, without generating text.

`/v1/decisions` turns a chat model into a decision model. You send an input and a list of typed questions, and each answer comes back with the probability of every option, read from the model's next-token scores at the answer position. No text is generated and no output is parsed. It needs no special checkpoint: any generation model served with a Jinja chat template can answer, as long as the checks below pass for it.

> [!NOTE]
>
> `/v1/decisions` is an SGLang extension under `/v1`, like `/v1/score` and `/v1/rerank`. It is not part of the OpenAI API, so call it over HTTP rather than through an OpenAI SDK method. Until a release contains it, install a [nightly build](/docs/get-started/install#nightly-builds), which is built from the main branch.

## Supported models

| Model | Example HuggingFace identifier | Notes |
| - | - | - |
| **Qwen3.8-27B** | `Qwen/Qwen3.8-27B` | Validated on one H200 in BF16, with and without NEXTN speculative decoding. See the [model page](/cookbook/autoregressive/Qwen/Qwen3.8-27B). |
| **Qwen3.5-35B-A3B** | `Qwen/Qwen3.5-35B-A3B` | Validated on one H200 in BF16. See the [model page](/cookbook/autoregressive/Qwen/Qwen3.5). |

Other chat models are served when the answer labels are single tokens at the answer position and the answer does not start inside a reasoning block. The server checks these per request and returns a 400 that names the reason when one fails, see [Errors](#errors). It does not check how the chat template renders the question or where the template itself would put an answer, so inspect `prompt_token_ids` from `return_prompt_token_ids` before relying on a new model.

## Launch command

```bash Command theme={null}
python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --host 127.0.0.1 --port 30000
```

No extra flag is needed. The same server keeps serving `/v1/chat/completions` and `/generate`, and decisions can run alongside that traffic. A server launched with `--enable-mis` or `--dllm-algorithm`, or one that uses a built-in conversation template instead of the tokenizer's Jinja template, refuses decisions.

## Example request

```python Example theme={null}
import requests

URL = "http://127.0.0.1:30000"

response = requests.post(
    f"{URL}/v1/decisions",
    json={
        "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing. I'm losing sales.",
        "questions": [
            {
                "id": "team",
                "type": "choice",
                "question": "Which team should handle this ticket?",
                "options": [
                    {"name": "billing", "description": "Payment or subscription issues"},
                    {"name": "technical", "description": "Bugs or integration problems"},
                    {"name": "sales", "description": "Pricing or account questions"},
                ],
            },
            {
                "id": "frustration",
                "type": "score",
                "question": "How frustrated is the customer?",
                "levels": ["Calm", "Frustrated but civil", "Very angry"],
            },
            {
                "id": "urgent",
                "type": "yes_no",
                "question": "The customer needs an answer today.",
            },
        ],
    },
    timeout=60,
)
response.raise_for_status()
for question_id, answer in response.json()["answers"].items():
    print(question_id, answer)
```

Each answer contains:

* `type`: the question type, `choice`, `score`, or `yes_no`.
* `probabilities`: one value per option name, level index, or `yes` and `no`, summing to 1. For a yes or no question, `probabilities["yes"]` is the answer.
* `choice` for a choice question (the most probable option) or `score` for a score question (the probability-weighted mean level index).
* `label_mass`: the full-vocabulary probability of the answer labels at the answer position. A low value means the model puts most of its probability outside the offered answers.

Questions are answered independently and scored together in one batch. To route the ticket in this example, send it to the chosen team, and hand it to a person when the top probability falls below a threshold you validate on your own labeled tickets.

## Image inputs

Both `/v1/decisions` and `/v1/systemone` accept an optional top-level `images` list. Each item uses the same image URL object as `image_url` in [vision chat requests](/docs/basic_usage/openai_api_vision), with `url` and optional `detail`, or is a string holding the `url` alone, such as a data URL or base64 image bytes. Use a vision-capable generation model with its vision encoder loaded and a Jinja template that accepts image parts. Some vision models serve text only unless the server is launched with `--enable-multimodal`. The native processor must expose image marker metadata, and a custom chat template is refused if it fails to preserve one complete marker per image. The server uses SGLang's existing image processor and vision encoder, placing the images in list order before the existing text in every question's user message.

Save an image as `screenshot.png` and send it to the server launched above:

```python Example theme={null}
import base64
from pathlib import Path

import requests

response = requests.post(
    "http://127.0.0.1:30000/v1/decisions",
    json={
        "input": "Use the screenshot to answer the question.",
        "images": [{"url": base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")}],
        "questions": [{"id": "menu", "type": "yes_no", "question": "A menu is visible."}],
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["menu"])
```

You can also supply an image URL. On `/v1/systemone`, send the same `images` field over HTTP alongside `state` and its question map. Images are shared by all questions. Objects and arrays in `input` or `state` remain JSON text, and text-only prompt wording stays at version 1. Image requests are refused when `--allow-auto-truncate` is enabled because truncation could remove the answer position. `return_prompt_token_ids` is unsupported with images because token IDs alone cannot replay vision conditioning through `/v1/score`.

Image requests are also refused when `input`, `state` or a question contains the model's own image, video or audio marker text, because the processor would read that text as media. Each question of an image request is prepared and scheduled on its own, so a probability near a tie can differ slightly between identical requests. Launch the server with `--enable-deterministic-inference` when image answers must repeat exactly, as described in [deterministic inference](/docs/advanced_features/deterministic_inference).

## How answers are computed

The server renders each question as one user message with the model's chat template and thinking turned off, and labels the answers `A` to `Z` for options, `0` to `9` for levels, and `yes` and `no`. It checks that each label is one distinct token at the answer position, then runs one prefill pass per question through the scoring path of `/v1/score` and reads the next-token log-probabilities of the labels over the full vocabulary. For a yes or no question with log-probabilities `lp_yes` and `lp_no` and request temperature `T`:

* `probabilities["yes"] = exp(lp_yes / T) / (exp(lp_yes / T) + exp(lp_no / T))`, which equals a softmax of the two label logits divided by `T`, because the vocabulary normalizer cancels.
* `label_mass = exp(lp_yes) + exp(lp_no)`, which does not depend on `T`.

Choice and score questions work the same way over their labels. None of these values is a calibrated probability that the decision is correct. Validate any threshold on labeled data from your workload.

## Request and response reference

Request fields:

| Field | Type | Meaning |
| - | - | - |
| `input` | string, object, or array | What the questions are about. Objects and arrays are rendered as compact JSON. Must not be blank. |
| `images` | list of image URL objects, default empty | Images shared by all questions, in list order. See [Image inputs](#image-inputs). |
| `questions` | list of questions | At least one. Each question has a unique `id`, which keys its answer. |
| `temperature` | number above 0, default 1 | Divides the label logits before the softmax over the labels. It does not change `label_mass`. |
| `chat_template_kwargs` | object, default empty | Extra chat template arguments, applied after `--default-chat-template-kwargs`. The thinking toggle that the chat template or its reasoning parser names stays off in either case. |
| `prompt_format_version` | integer, optional | Pins the prompt wording. A server that serves another version returns 400. |
| `return_prompt_token_ids` | boolean, default false | Adds `prompt_token_ids` and `label_token_ids` to each text-only answer. Unsupported with images. |
| `model` | string, optional | Echoed in the response, as `default` when omitted. It does not select a LoRA adapter, and the `base:adapter` form returns 400. |

Question fields:

| Type | Fields | Labels |
| - | - | - |
| `choice` | `id`, `question`, and `options`: 2 to 26 objects with a `name` and an optional `description` | `A` to `Z` in list order |
| `score` | `id`, `question`, and `levels`: 2 to 10 level descriptions, lowest first | `0` to `9` |
| `yes_no` | `id`, `question`, and optional `yes` and `no` descriptions | `yes` and `no` |

`question`, descriptions, and levels can also be JSON objects or arrays. Each question becomes one user message: the input, a blank line, the question line, one line per option, level, or described yes or no answer, and a closing instruction to answer with one label only. An option renders as `A: billing` or `A: billing - <description>`, a level as `0: <description>`, and a described yes or no answer as `yes: <description>`.

The response has `object` set to `decisions`, `model`, `prompt_format_version`, `answers` keyed by question id, and `usage` with `prompt_tokens` for all questions and `completion_tokens` 0.

## Errors

The request fails with HTTP 400 for:

* an unknown question type or an unknown field anywhere in the request
* option or level counts outside the ranges above
* a blank input, question, question id, or level
* a repeated question id, or option names that are blank, contain control or line break characters, or repeat another name after trimming and case folding
* a label that is not one distinct token at the answer position for the served tokenizer
* a prompt that does not fit the model's context length
* a chat template that always thinks before answering or starts every answer with a reasoning block, `chat_template_kwargs` that set the thinking toggle to anything but `false`, a rendered prompt that leaves a reasoning block open, or a model whose reasoning parser expects answers to start inside a reasoning block when the rendered prompt does not close one
* a `prompt_format_version` other than the served one, or a `model` that names a LoRA adapter
* a server launched with `--enable-mis` or `--dllm-algorithm`, or one that uses a built-in conversation template, whether named with `--chat-template`, loaded from a JSON template file, or inferred from the model path
* a model whose chat route uses a built-in encoder instead of a chat template, a tokenizer that does not encode the rendered chat text back to the same ids, or a model that is not a generation model
* a chat template that raises an error for the question message, or a server launched with `--skip-tokenizer-init`
* a decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B), or a joint schema or pointer head checkpoint such as [Clef](/cookbook/autoregressive/Cloudflare/clef) or [Kev](/cookbook/autoregressive/Kev/kev), which answer through `/v1/systemone`

Errors about one question name its id or its position in `questions`.

## Pin and replay the prompt

The server owns the prompt wording of `/v1/decisions` and versions it. Every response carries `prompt_format_version`, and a change to the wording ships as a new version. Send `prompt_format_version` to fail loudly instead of receiving answers from different wording after a server upgrade.

To keep a text-only answer reproducible independently of the server's wording, ask for the scored ids and replay them through `/v1/score`, which only scores the ids you send:

```python Example theme={null}
import requests

URL = "http://127.0.0.1:30000"

body = {
    "input": "The integration keeps failing.",
    "questions": [{"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."}],
    "prompt_format_version": 1,
    "return_prompt_token_ids": True,
}
answer = requests.post(f"{URL}/v1/decisions", json=body, timeout=60).json()["answers"]["urgent"]
replay = requests.post(
    f"{URL}/v1/score",
    json={
        "query": [],
        "items": [answer["prompt_token_ids"]],
        "label_token_ids": [answer["label_token_ids"]],
        "apply_softmax": True,
        "return_token_logprobs": True,
    },
    timeout=60,
).json()
print(answer["probabilities"], replay["scores"][0])
```

The replayed scores equal the answer's probabilities when both requests use the same `temperature` and run in the same cache state. If the decision set `temperature`, send the same value in the `/v1/score` body. A replay right after the decision reuses its cached prefix, so it can differ slightly, as described in [Reproducibility](#reproducibility).

## Thinking models

Qwen3.8-27B and Qwen3.5-35B-A3B think by default. `/v1/decisions` turns thinking off for every question, refuses a request that turns it back on, and refuses a rendered prompt that leaves a reasoning block open, so the answer is never read inside the reasoning. Chat requests to the same server keep the model's default, and launching with `--reasoning-parser qwen3` affects chat only.

## Reproducibility

These models mix full and linear attention. Answer probabilities can move by up to several hundredths (0.07 in our checks), and `label_mass` by up to about 0.14, between cold and prefix-cached requests and across batch compositions, while the chosen option stayed the same in our checks. On Qwen3.5-35B-A3B, `--disable-radix-cache` gave identical values across sequential repeats, at the cost of prefix reuse.

## System One compatible API

`POST /v1/systemone` serves the same decisions in the request and response shape of the System One API, with a `state`, a map of `noul`, `choice`, and `score` questions keyed by your ids, and one answer per id. Clients written for that API, including the official TypeSafe SDKs, get decisions by pointing their base URL at the server, with the exceptions listed below. For ordinary generation models, the route uses the same rendering, label checks, and scoring as `/v1/decisions`, and refuses the same servers, chat templates, and prompts, listed in [Errors](#errors). Its request fields are checked as described below. A decision checkpoint such as [PPLX-Decider-v1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1-27B) instead gets the prompt, answer codes, and calibrated temperature it was trained with. When its `decision_config.json` sets `attention_mode` to `noncausal_full_attention`, as [PPLX-Decider-v1.1-27B](/cookbook/autoregressive/Perplexity/PPLX-Decider-v1.1-27B) does, its full-attention layers read the whole prompt, and the server turns off the radix cache and chunked prefill, which would show those layers only part of it. A joint schema checkpoint such as [Clef](/cookbook/autoregressive/Cloudflare/clef) answers all questions from one prompt in one prefill, scored by its joint schema head, and does not use the chat template. A pointer head checkpoint such as [Kev](/cookbook/autoregressive/Kev/kev) answers each question from its own prompt after the shared state, scored by its pointer head, and does not use the chat template either.

```bash Command theme={null}
pip install typesafe-sdk
```

```python Example theme={null}
from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

# A server started without --api-key accepts any key, so a placeholder works.
client = TypeSafeClient(base_url="http://127.0.0.1:30000", api_key="unused")
result = client.system_one(
    "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.",
    {
        "team": Choice(
            instructions="Which team should handle this ticket?",
            criteria={"billing": None, "technical": "Bugs or integration problems", "sales": None},
        ),
        "urgent": Noul(instructions="The customer needs an answer today."),
        "frustration": Score(
            instructions="How frustrated is the customer?",
            criteria=["Calm", "Frustrated but civil", "Very angry"],
        ),
    },
)
for name, answer in result.answers.items():
    print(name, answer)
```

What to expect from this route:

* Any `model` name is accepted, including the SDK default `jev-latest`, unless it names a LoRA adapter after a colon, as in `base:adapter`. The response `model` is always the served model.
* A `noul` answer is the probability of yes. A `choice` answer has the most probable option and a `score` answer the probability-weighted mean level, each with `probabilities` and `confidence`, and a score answer echoes its levels in `legend`.
* `confidence` follows the formulas TypeSafe publishes for its adapter. It is a statistic of the returned probabilities, which are renormalized over the answer labels and ignore probability outside them. It is not calibrated, it can read 0 for a score split between distant levels, and thresholds tuned on another provider may not transfer.
* For a model scored with answer-label tokens, each answer also carries `x_label_mass`, the full-vocabulary probability of its labels, which `/v1/decisions` returns as `label_mass`. The Python SDK drops fields it does not define, so read it from `result.raw_http_response.json()`. Decision checkpoints and joint schema checkpoints omit it, because their readout or head scores only the answers.
* A choice takes 1 to 255 options and a score 1 to 10 levels. Option names that are blank, contain control or line break characters, or repeat another name after trimming and case folding are refused, and so are blank or null levels, as on `/v1/decisions`.
* For general generation models that score answer labels, options beyond 26 get two-letter labels (`AA`, `AB`, and so on), each checked to be one token for the served tokenizer. These labels include common words and have unequal priors, so answers can depend on option order. Supporting more than 26 options also requires a Hugging Face tokenizer that reports its added tokens and a compatible chat-template boundary, as the Qwen models have with `</think>`. Without those properties, these models are limited to 26 options. Clef's joint schema head scores the options directly and does not use letter labels.
* `instructions` are optional. For general generation models, a choice or score question without them drops its question line; Clef uses the question id as its instructions. A `noul` question is answered from its `true` and `false` descriptions, and one with neither descriptions nor instructions, such as the SDK's bare `Noul()`, is refused.
* Unknown top-level fields are ignored. Unknown keys inside a question or inside `noul` criteria are refused, and so are the `/v1/decisions` fields `temperature`, `prompt_format_version`, and `return_prompt_token_ids` with any value other than null. `images` is accepted as an SGLang extension. Clef rejects nonempty `chat_template_kwargs` because it uses the prompt its head was trained with; other models check this field against their own template rules.
* Unlike `/v1/decisions`, an empty `state` and empty or blank question ids are accepted, as the System One schema allows.
* Invalid requests return 422 with the location of the error, other refusals return 400, and neither is retried by the SDKs.
* `usage.input_tokens` counts the prompt of every question, so the state is counted once per question, or once for a joint schema checkpoint, and `usage.output_tokens` is 0.
* The SDKs' `models.list()` and response `request_id` are not supported, because `GET /v1/models` keeps its OpenAI format and no request id header is sent.
* The SDKs time out after 10 seconds and retry, so raise their timeout for requests with many questions over a long state. A request whose client gives up still finishes its encoding and part of its prefill, so each retry repeats at least the encoding.

## Build decisions without `/v1/decisions`

When you need your own prompt or labels, build the prompt on the client and score it through `/v1/score`, which takes token ids and one label list per item. This needs `transformers` and `jinja2` on the client. Every label must be exactly one token at the answer position:

```python Example theme={null}
import requests
from transformers import AutoTokenizer

MODEL = "Qwen/Qwen3.8-27B"
URL = "http://127.0.0.1:30000"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

options = ["billing", "technical", "sales"]
letters = ["A", "B", "C"]
prompt = (
    "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.\n\n"
    "Question: Which team should handle this ticket?\n"
    + "\n".join(f"{letter}: {name}" for letter, name in zip(letters, options))
    + "\nAnswer with the letter of one option only."
)
text = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}], add_generation_prompt=True, enable_thinking=False, tokenize=False
)
input_ids = tokenizer.encode(text, add_special_tokens=False)
label_ids = []
for letter in letters:
    ids = tokenizer.encode(text + letter, add_special_tokens=False)
    if ids[: len(input_ids)] != input_ids or len(ids) != len(input_ids) + 1:
        raise ValueError(f"Label {letter} is not one token at the answer position")
    label_ids.append(ids[-1])

scores = requests.post(
    f"{URL}/v1/score",
    json={"query": [], "items": [input_ids], "label_token_ids": [label_ids], "apply_softmax": True},
    timeout=60,
).json()["scores"][0]
print(dict(zip(options, scores)))
```

`enable_thinking=False` renders a closed think block, so the answer token follows it directly. Do not score a prompt that leaves the think block open, because the scored position then falls inside the reasoning.

Structured output gives the same decision through generation. A one-token regex such as `(A|B|C)` with `temperature` 0 and logprobs returns probabilities renormalized over the labels, equal to the `/v1/score` values for one-character labels in the same cache state. A JSON schema with `enum` fields fills several fields in one response, but gives no clean per-option probability, because an enum value can span several tokens. See [Structured outputs](/docs/advanced_features/structured_outputs) for the request formats.

## Limitations

* Labels must be one token at the answer position. `/v1/decisions` assigns one-token labels for you, and a client-built prompt must do the same.
* For `yes_no`, `label_mass` counts only the lowercase `yes` and `no` tokens. The model also puts probability on `Yes` and `No`, so this value reads lower than for choice and score questions even on clear cases, while `probabilities["yes"]` is unaffected.
* The server tells from the chat template and its reasoning parser whether answers could start inside a reasoning block, using the parser's reasoning tags in the generation prompt and in the template's own rendering of a finished answer. Reasoning that shows neither is not detected, so check `prompt_token_ids` for such a model. Some templates of models that do not reason still contain reasoning tags without ever closing a block in the generation prompt, and those models are refused.
* A default temperature in `--preferred-sampling-params` can reach scoring requests and scale both the probabilities and `label_mass` when a decision shares a batch with generation or runs under speculative decoding. Leave it unset for decisions.
* The server tokenizes each question's full prompt once on the HTTP event loop, as `/v1/score` does for its items, and lets other requests run between questions. Tokenizers whose label check falls back to the full prompt encode it once more per label.
* A decision takes a single `input`, not a chat history. To decide about a conversation, pass the history as the input text or as a JSON array.


# Diffusion language models
Source: https://docs.sglang.io/docs/supported-models/diffusion_language_models



Diffusion language models have shown promise for non-autoregressive text generation with parallel decoding capabilities. Unlike auto-regressive language models, different diffusion language models require different decoding strategies.

## Example Launch Command

SGLang supports different DLLM algorithms such as `LowConfidence` and `JointThreshold`.

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path inclusionAI/LLaDA2.0-mini \ # example HF/local path
  --dllm-algorithm LowConfidence \
  --dllm-algorithm-config ./config.yaml \ # Optional. Uses the algorithm's default if not set.
  --host 0.0.0.0 \
  --port 30000
```

## First-Done-First-Out (FDFO) Scheduling

FDFO scheduling is **enabled by default**: each request leaves the batch as soon as its block is resolved, instead of advancing in lockstep where fast-converging requests must wait for slow long-tail requests before leaving the batch (head-of-line blocking). This improves throughput and is orthogonal to `--dllm-algorithm`, so it works with any dLLM algorithm. Pass `--no-dllm-fdfo` to fall back to synchronous lockstep scheduling:

```bash Command theme={null}
python3 -m sglang.launch_server \
  --model-path inclusionAI/LLaDA2.0-mini \
  --dllm-algorithm LowConfidence \
  --no-dllm-fdfo \
  --host 0.0.0.0 \
  --port 30000
```

## Example Configuration File

Depending on the algorithm selected, the configuration parameters vary.

LowConfidence Config:

```yaml Config theme={null}
# Confidence threshold for accepting predicted tokens
# - Higher values: More conservative, better quality but slower
# - Lower values: More aggressive, faster but potentially lower quality
# Range: 0.0 - 1.0
threshold: 0.95

# Default: 32, for LLaDA2MoeModelLM
block_size: 32
```

JointThreshold Config:

```yaml Config theme={null}
# Decoding threshold for Mask-to-Token (M2T) phase
# - Higher values: More conservative, better quality but slower
# - Lower values: More aggressive, faster but potentially lower quality
# Range: 0.0 - 1.0
threshold: 0.5
# Decoding threshold for Token-to-Token (T2T) phase
# Range: 0.0 - 1.0
# Setting to 0.0 allows full editing (recommended for most cases).
edit_threshold: 0.0
# Max extra T2T steps after all masks are removed. Prevents infinite loops.
max_post_edit_steps: 16
# 2-gram repetition penalty (default 0).
# An empirical value of 3 is often sufficient to mitigate most repetitions.
penalty_lambda: 0
```

## Example Client Code Snippet

Just like other supported models, diffusion language models can be used via the REST API or Python client.

Python client example for making a generation request to the launched server:

```python Example theme={null}
import sglang as sgl

def main():
    llm = sgl.Engine(model_path="inclusionAI/LLaDA2.0-mini",
                     dllm_algorithm="LowConfidence",
                     max_running_requests=1,
                     trust_remote_code=True)

    prompts = [
        "<role>SYSTEM</role>detailed thinking off<|role_end|><role>HUMAN</role> Write a brief introduction of the great wall <|role_end|><role>ASSISTANT</role>"
    ]

    sampling_params = {
        "temperature": 0,
        "max_new_tokens": 1024,
    }

    outputs = llm.generate(prompts, sampling_params)
    print(outputs)

if __name__ == '__main__':
    main()
```

Curl example for making a generation request to the launched server:

```bash Command theme={null}
curl -X POST "http://127.0.0.1:30000/generate" \
     -H "Content-Type: application/json" \
     -d '{
        "text": [
            "<role>SYSTEM</role>detailed thinking off<|role_end|><role>HUMAN</role> Write the number from 1 to 128 <|role_end|><role>ASSISTANT</role>",
            "<role>SYSTEM</role>detailed thinking off<|role_end|><role>HUMAN</role> Write a brief introduction of the great wall <|role_end|><role>ASSISTANT</role>"
        ],
        "stream": true,
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 1024
        }
    }'
```

## Supported Models

Below the supported models are summarized in a table.

<table>
  <thead>
    <tr>
      <th>Model Family</th>
      <th>Example Model</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**LLaDA2.0 (mini, flash)**</td>
      <td>`inclusionAI/LLaDA2.0-flash`</td>
      <td>LLaDA2.0-flash is a diffusion language model featuring a 100B Mixture-of-Experts (MoE) architecture.</td>
    </tr>

    <tr>
      <td>**SDAR (JetLM)**</td>
      <td>`JetLM/SDAR-8B-Chat`</td>
      <td>SDAR series diffusion language model (Chat), dense architecture.</td>
    </tr>

    <tr>
      <td>**SDAR (JetLM)**</td>
      <td>`JetLM/SDAR-30B-A3B-Chat`</td>
      <td>SDAR series diffusion language model (Chat), MoE architecture.</td>
    </tr>

    <tr>
      <td>**DiffusionGemma**</td>
      <td>`google/diffusiongemma-26B-A4B-it`</td>
      <td>Uniform-state (renoising) block-diffusion multimodal (text + image) MoE, 25.2B total / 3.8B active, served with the `Gemma4Renoise` sampler.</td>
    </tr>
  </tbody>
</table>


# Embedding models
Source: https://docs.sglang.io/docs/supported-models/embedding_models

Dense and sparse embedding models with FlashInfer acceleration and SGLang's batching infrastructure.

SGLang provides robust support for embedding models by integrating efficient serving mechanisms with its flexible programming interface. This integration allows for streamlined handling of embedding tasks, facilitating faster and more accurate retrieval and semantic search operations. SGLang's architecture enables better resource utilization and reduced latency in embedding model deployment.

> [!WARNING]
>
> Native encoder embedding architectures and `google/embeddinggemma-300m` are detected automatically. Decoder-style embedding models require `--is-embedding`; add `--trust-remote-code` when the model requires it.

## Quick Start

### Launch Server

```bash theme={null}
sglang serve \
  --model-path Qwen/Qwen3-Embedding-4B \
  --is-embedding
```

### EmbeddingGemma

EmbeddingGemma uses bidirectional attention and is auto-detected, so its best default command is simply:

```bash theme={null}
sglang serve --model-path google/embeddinggemma-300m
```

On CUDA, SGLang automatically uses breakable CUDA graph (BCG) for its full encoder prefill and disables incompatible radix-cache and chunked-prefill behavior. Do not add the deprecated piecewise CUDA graph knobs.

### Client Request

```python theme={null}
import requests

url = "http://127.0.0.1:30000"

payload = {
    "model": "Qwen/Qwen3-Embedding-4B",
    "input": "What is the capital of France?",
    "encoding_format": "float"  # or "base64" for compact FP32 responses
}

response = requests.post(url + "/v1/embeddings", json=payload).json()
print("Embedding:", response["data"][0]["embedding"])
```

## Multimodal Embedding Example

For multimodal models like GME that support both text and images:

```bash theme={null}
sglang serve \
  --model-path Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
  --is-embedding \
  --chat-template gme-qwen2-vl
```

```python Example theme={null}
import requests

url = "http://127.0.0.1:30000"

text_input = "Represent this image in embedding space."
image_path = "https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild/resolve/main/images/023.jpg"

payload = {
    "model": "gme-qwen2-vl",
    "input": [
        {
            "text": text_input
        },
        {
            "image": image_path
        }
    ],
}

response = requests.post(url + "/v1/embeddings", json=payload).json()

print("Embeddings:", [x.get("embedding") for x in response.get("data", [])])
```

## Matryoshka Embedding Example

[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows user to trade off between performance and cost.

### 1. Launch a Matryoshka‑capable model

If the model config already includes `matryoshka_dimensions` or `is_matryoshka` then no override is needed. Otherwise, you can use `--json-model-override-args` as below:

```bash Command theme={null}
sglang serve \
    --model-path Qwen/Qwen3-Embedding-0.6B \
    --is-embedding \
    --json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 1024, 1536]}'
```

1. Setting `"is_matryoshka": true` allows truncating to any dimension. Otherwise, the server will validate that the specified dimension in the request is one of `matryoshka_dimensions`.
2. Omitting `dimensions` in a request returns the full vector.

### 2. Make requests with different output dimensions

```python theme={null}
import requests

url = "http://127.0.0.1:30000"

# Request a truncated (Matryoshka) embedding by specifying a supported dimension.
payload = {
    "model": "Qwen/Qwen3-Embedding-0.6B",
    "input": "Explain diffusion models simply.",
    "dimensions": 512  # change to 128 / 1024 / omit for full size
}

response = requests.post(url + "/v1/embeddings", json=payload).json()
print("Embedding:", response["data"][0]["embedding"])
```

## Supported Models

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>Example Model</th>
      <th>Chat template</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>EmbeddingGemma</td>
      <td>`google/embeddinggemma-300m`</td>
      <td>N/A</td>
      <td>Bidirectional Gemma3 text encoder; auto-detected and served with BCG by default</td>
    </tr>

    <tr>
      <td>E5 (Llama/Mistral based)</td>
      <td>`intfloat/e5-mistral-7b-instruct`</td>
      <td>N/A</td>
      <td>High-quality text embeddings based on Mistral/Llama architectures</td>
    </tr>

    <tr>
      <td>GTE-Qwen2</td>
      <td>`Alibaba-NLP/gte-Qwen2-7B-instruct`</td>
      <td>N/A</td>
      <td>Alibaba's text embedding model with multilingual support</td>
    </tr>

    <tr>
      <td>Qwen3-Embedding</td>
      <td>`Qwen/Qwen3-Embedding-4B`</td>
      <td>N/A</td>
      <td>Latest Qwen3-based text embedding model for semantic representation</td>
    </tr>

    <tr>
      <td>Qwen3 (bare backbone)</td>
      <td>`microsoft/harrier-oss-v1-0.6b`</td>
      <td>N/A</td>
      <td>Bare `Qwen3Model` backbone (no LM head); served natively on SGLang's fused Qwen3 kernels and auto-classified as an embedding model</td>
    </tr>

    <tr>
      <td>BGE</td>
      <td>`BAAI/bge-large-en-v1.5`</td>
      <td>N/A</td>
      <td>BAAI's text embeddings (requires `attention-backend` triton/torch\_native)</td>
    </tr>

    <tr>
      <td>GME (Multimodal)</td>
      <td>`Alibaba-NLP/gme-Qwen2-VL-2B-Instruct`</td>
      <td>`gme-qwen2-vl`</td>
      <td>Multimodal embedding for text and image cross-modal tasks</td>
    </tr>

    <tr>
      <td>CLIP</td>
      <td>`openai/clip-vit-large-patch14-336`</td>
      <td>N/A</td>
      <td>OpenAI's CLIP for image and text embeddings</td>
    </tr>
  </tbody>
</table>


# Large Language Models
Source: https://docs.sglang.io/docs/supported-models/generative_models



These models accept text input and produce text output (e.g., chat completions). They are primarily large language models (LLMs), some with mixture-of-experts (MoE) architectures for scaling.

## Example launch Command

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Instruct \  # example HF/local path
  --host 0.0.0.0 \
  --port 30000 \
```

## Supported models

Below the supported models are summarized in a table.

If you are unsure if a specific architecture is implemented, you can search for it via GitHub. For example, to search for `Qwen3ForCausalLM`, use the expression:

```text Output theme={null}
repo:sgl-project/sglang path:/^python\/sglang\/srt\/models\// Qwen3ForCausalLM
```

in the GitHub search bar.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family (Variants)</th>
      <th>Example HuggingFace Identifier</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**DeepSeek** (v1, v2, v3/R1)</td>
      <td>`deepseek-ai/DeepSeek-R1`</td>
      <td>Series of advanced reasoning-optimized models (including a 671B MoE) trained with reinforcement learning; top performance on complex reasoning, math, and code tasks. [SGLang provides Deepseek v3/R1 model-specific optimizations](/cookbook/autoregressive/DeepSeek/DeepSeek-V3) and [Reasoning Parser](../advanced_features/separate_reasoning)</td>
    </tr>

    <tr>
      <td>**Kimi K2** (Thinking, Instruct)</td>
      <td>`moonshotai/Kimi-K2-Instruct`</td>
      <td>Moonshot AI's 1 trillion parameter MoE model (32B active) with 128K–256K context; state-of-the-art agentic intelligence with stable long-horizon agency across 200–300 sequential tool calls. Features MLA attention and native INT4 quantization. [See Reasoning Parser docs](../advanced_features/separate_reasoning)</td>
    </tr>

    <tr>
      <td>**Kimi Linear** (48B-A3B)</td>
      <td>`moonshotai/Kimi-Linear-48B-A3B-Instruct`</td>
      <td>Moonshot AI's hybrid linear attention model (48B total, 3B active) with 1M token context; features Kimi Delta Attention (KDA) for up to 6× faster decoding and 75% KV cache reduction vs full attention.</td>
    </tr>

    <tr>
      <td>**GPT-OSS**</td>
      <td>`openai/gpt-oss-20b`, `openai/gpt-oss-120b`</td>
      <td>OpenAI’s latest GPT-OSS series for complex reasoning, agentic tasks, and versatile developer use cases.</td>
    </tr>

    <tr>
      <td>**Qwen** (3.5, 3, 3MoE, 3Next, 2.5, 2 series)</td>
      <td>`Qwen/Qwen3.5-397B-A17B`, `Qwen/Qwen3-0.6B`, `Qwen/Qwen3-30B-A3B`, `Qwen/Qwen3-Next-80B-A3B-Instruct`</td>
      <td>Alibaba’s latest Qwen3 series for complex reasoning, language understanding, and generation tasks; Support for MoE variants along with previous generation 2.5, 2, etc. [SGLang provides Qwen3 specific reasoning parser](../advanced_features/separate_reasoning)</td>
    </tr>

    <tr>
      <td>**Llama** (2, 3.x, 4 series)</td>
      <td>`meta-llama/Llama-4-Scout-17B-16E-Instruct`</td>
      <td>Meta's open LLM series, spanning 7B to 400B parameters (Llama 2, 3, and new Llama 4) with well-recognized performance. [SGLang provides Llama-4 model-specific optimizations](/cookbook/autoregressive/Meta/Llama4)</td>
    </tr>

    <tr>
      <td>**Mistral** (Mixtral, NeMo, Small3)</td>
      <td>`mistralai/Mistral-7B-Instruct-v0.2`</td>
      <td>Open 7B LLM by Mistral AI with strong performance; extended into MoE (“Mixtral”) and NeMo Megatron variants for larger scale.</td>
    </tr>

    <tr>
      <td>**Gemma** (v1, v2, v3)</td>
      <td>`google/gemma-3-1b-it`</td>
      <td>Google’s family of efficient multilingual models (1B–27B); Gemma 3 offers a 128K context window, and its larger (4B+) variants support vision input.</td>
    </tr>

    <tr>
      <td>**Phi** (Phi-1.5, Phi-2, Phi-3, Phi-4, Phi-MoE series)</td>
      <td>`microsoft/Phi-4-multimodal-instruct`, `microsoft/Phi-3.5-MoE-instruct`</td>
      <td>Microsoft’s Phi family of small models (1.3B–5.6B); Phi-4-multimodal (5.6B) processes text, images, and speech, Phi-4-mini is a high-accuracy text model and Phi-3.5-MoE is a mixture-of-experts model.</td>
    </tr>

    <tr>
      <td>**MiniCPM** (v3, 4B)</td>
      <td>`openbmb/MiniCPM3-4B`</td>
      <td>OpenBMB’s series of compact LLMs for edge devices; MiniCPM 3 (4B) achieves GPT-3.5-level results in text tasks.</td>
    </tr>

    <tr>
      <td>**OLMo** (2, 3)</td>
      <td>`allenai/OLMo-3-1125-32B`, `allenai/OLMo-3-32B-Think`, `allenai/OLMo-2-1124-7B-Instruct`</td>
      <td>Allen AI’s series of Open Language Models designed to enable the science of language models.</td>
    </tr>

    <tr>
      <td>**OLMoE** (Open MoE)</td>
      <td>`allenai/OLMoE-1B-7B-0924`</td>
      <td>Allen AI’s open Mixture-of-Experts model (7B total, 1B active parameters) delivering state-of-the-art results with sparse expert activation.</td>
    </tr>

    <tr>
      <td>**MiniMax-M2** (M2, M2.1, M2.5)</td>
      <td>`MiniMaxAI/MiniMax-M2.5`, `MiniMaxAI/MiniMax-M2.1`, `MiniMaxAI/MiniMax-M2`</td>
      <td>MiniMax's SOTA LLM for coding & agentic workflows.</td>
    </tr>

    <tr>
      <td>**StableLM** (3B, 7B)</td>
      <td>`stabilityai/stablelm-tuned-alpha-7b`</td>
      <td>StabilityAI’s early open-source LLM (3B & 7B) for general text generation; a demonstration model with basic instruction-following ability.</td>
    </tr>

    <tr>
      <td>**Command-(R,A)** (Cohere)</td>
      <td>`CohereLabs/c4ai-command-r-v01`, `CohereLabs/c4ai-command-r7b-12-2024`, `CohereLabs/c4ai-command-a-03-2025`</td>
      <td>Cohere’s open conversational LLM (Command series) optimized for long context, retrieval-augmented generation, and tool use.</td>
    </tr>

    <tr>
      <td>**DBRX** (Databricks)</td>
      <td>`databricks/dbrx-instruct`</td>
      <td>Databricks’ 132B-parameter MoE model (36B active) trained on 12T tokens; competes with GPT-3.5 quality as a fully open foundation model.</td>
    </tr>

    <tr>
      <td>**Grok** (xAI)</td>
      <td>`xai-org/grok-1`</td>
      <td>xAI’s grok-1 model known for vast size(314B parameters) and high quality; integrated in SGLang for high-performance inference.</td>
    </tr>

    <tr>
      <td>**ChatGLM** (GLM-130B family)</td>
      <td>`THUDM/chatglm2-6b`</td>
      <td>Zhipu AI’s bilingual chat model (6B) excelling at Chinese-English dialogue; fine-tuned for conversational quality and alignment.</td>
    </tr>

    <tr>
      <td>**InternLM 2** (7B, 20B)</td>
      <td>`internlm/internlm2-7b`</td>
      <td>Next-gen InternLM (7B and 20B) from SenseTime, offering strong reasoning and ultra-long context support (up to 200K tokens).</td>
    </tr>

    <tr>
      <td>**ExaONE 3** (Korean-English)</td>
      <td>`LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct`</td>
      <td>LG AI Research’s Korean-English model (7.8B) trained on 8T tokens; provides high-quality bilingual understanding and generation.</td>
    </tr>

    <tr>
      <td>**Baichuan 2** (7B, 13B)</td>
      <td>`baichuan-inc/Baichuan2-13B-Chat`</td>
      <td>BaichuanAI’s second-generation Chinese-English LLM (7B/13B) with improved performance and an open commercial license.</td>
    </tr>

    <tr>
      <td>**XVERSE** (MoE)</td>
      <td>`xverse/XVERSE-MoE-A36B`</td>
      <td>Yuanxiang’s open MoE LLM (XVERSE-MoE-A36B: 255B total, 36B active) supporting \~40 languages; delivers 100B+ dense-level performance via expert routing.</td>
    </tr>

    <tr>
      <td>**SmolLM** (135M–1.7B)</td>
      <td>`HuggingFaceTB/SmolLM-1.7B`</td>
      <td>Hugging Face’s ultra-small LLM series (135M–1.7B params) offering surprisingly strong results, enabling advanced AI on mobile/edge devices.</td>
    </tr>

    <tr>
      <td>**GLM-4** (Multilingual 9B)</td>
      <td>`ZhipuAI/glm-4-9b-chat`</td>
      <td>Zhipu’s GLM-4 series (up to 9B parameters) – open multilingual models with support for 1M-token context and even a 5.6B multimodal variant (Phi-4V).</td>
    </tr>

    <tr>
      <td>**MiMo** (7B series)</td>
      <td>`XiaomiMiMo/MiMo-7B-RL`</td>
      <td>Xiaomi's reasoning-optimized model series, leverages Multiple-Token Prediction for faster inference.</td>
    </tr>

    <tr>
      <td>**ERNIE-4.5** (4.5, 4.5MoE series)</td>
      <td>`baidu/ERNIE-4.5-21B-A3B-PT`</td>
      <td>Baidu's ERNIE-4.5 series which consists of MoE with 47B and 3B active parameters, with the largest model having 424B total parameters, as well as a 0.3B dense model.</td>
    </tr>

    <tr>
      <td>**Arcee AFM-4.5B**</td>
      <td>`arcee-ai/AFM-4.5B-Base`</td>
      <td>Arcee's foundational model series for real world reliability and edge deployments.</td>
    </tr>

    <tr>
      <td>**Persimmon** (8B)</td>
      <td>`adept/persimmon-8b-chat`</td>
      <td>Adept’s open 8B model with a 16K context window and fast inference; trained for broad usability and licensed under Apache 2.0.</td>
    </tr>

    <tr>
      <td>**Solar** (10.7B)</td>
      <td>`upstage/SOLAR-10.7B-Instruct-v1.0`</td>
      <td>Upstage's 10.7B parameter model, optimized for instruction-following tasks. This architecture incorporates a depth-up scaling methodology, enhancing model performance.</td>
    </tr>

    <tr>
      <td>**Tele FLM** (52B-1T)</td>
      <td>`CofeAI/Tele-FLM`</td>
      <td>BAAI & TeleAI's multilingual model, available in 52-billion and 1-trillion parameter variants. It is a decoder-only transformer trained on \~2T tokens</td>
    </tr>

    <tr>
      <td>**Ling** (16.8B–290B)</td>
      <td>`inclusionAI/Ling-lite`, `inclusionAI/Ling-plus`</td>
      <td>InclusionAI’s open MoE models. Ling-Lite has 16.8B total / 2.75B active parameters, and Ling-Plus has 290B total / 28.8B active parameters. They are designed for high performance on NLP and complex reasoning tasks.</td>
    </tr>

    <tr>
      <td>**Granite 3.0, 3.1** (IBM)</td>
      <td>`ibm-granite/granite-3.1-8b-instruct`</td>
      <td>IBM's open dense foundation models optimized for reasoning, code, and business AI use cases. Integrated with Red Hat and watsonx systems.</td>
    </tr>

    <tr>
      <td>**Granite 3.0, 3.1 MoE** (IBM)</td>
      <td>`ibm-granite/granite-3.0-3b-a800m-instruct`, `ibm-granite/granite-3.1-3b-a800m-instruct`</td>
      <td>IBM’s Mixture-of-Experts models offering strong performance with cost-efficiency. MoE expert routing designed for enterprise deployment at scale. 3.1 adds shared experts.</td>
    </tr>

    <tr>
      <td>**GPT-J** (6B)</td>
      <td>`EleutherAI/gpt-j-6b`</td>
      <td>EleutherAI's GPT-2-like causal language model (6B) trained on the [Pile](https://pile.eleuther.ai/) dataset.</td>
    </tr>

    <tr>
      <td>**Orion** (14B)</td>
      <td>`OrionStarAI/Orion-14B-Base`</td>
      <td>A series of open-source multilingual large language models by OrionStarAI, pretrained on a 2.5T token multilingual corpus including Chinese, English, Japanese, Korean, etc, and it exhibits superior performance in these languages.</td>
    </tr>

    <tr>
      <td>**Llama Nemotron Super** (v1, v1.5, NVIDIA)</td>
      <td>`nvidia/Llama-3_3-Nemotron-Super-49B-v1`, `nvidia/Llama-3_3-Nemotron-Super-49B-v1_5`</td>
      <td>The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents.</td>
    </tr>

    <tr>
      <td>**Llama Nemotron Ultra** (v1, NVIDIA)</td>
      <td>`nvidia/Llama-3_1-Nemotron-Ultra-253B-v1`</td>
      <td>The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents.</td>
    </tr>

    <tr>
      <td>**NVIDIA Nemotron Nano 2.0**</td>
      <td>`nvidia/NVIDIA-Nemotron-Nano-9B-v2`</td>
      <td>The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) family of multimodal models provides state-of-the-art reasoning models specifically designed for enterprise-ready AI agents. `Nemotron-Nano-9B-v2` is a hybrid Mamba-Transformer language model designed to increase throughput for reasoning workloads while achieving state-of-the-art accuracy compared to similarly-sized models.</td>
    </tr>

    <tr>
      <td>**NVIDIA Nemotron 3 Super** (NVIDIA)</td>
      <td>`nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4`</td>
      <td>The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) 3 Super is a 120B-parameter MoE model (12B active) delivering high-quality reasoning and generation for enterprise AI agents.</td>
    </tr>

    <tr>
      <td>**NVIDIA Nemotron 3 Nano** (NVIDIA)</td>
      <td>`nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`</td>
      <td>The [NVIDIA Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/) 3 Nano is a compact model designed for efficient edge and enterprise deployment with strong reasoning capabilities.</td>
    </tr>

    <tr>
      <td>**StarCoder2** (3B-15B)</td>
      <td>`bigcode/starcoder2-7b`</td>
      <td>StarCoder2 is a family of open large language models (LLMs) specialized for code generation and understanding. It is the successor to StarCoder, jointly developed by the BigCode project (a collaboration between Hugging Face, ServiceNow Research, and other contributors).</td>
    </tr>

    <tr>
      <td>**Jet-Nemotron**</td>
      <td>`jet-ai/Jet-Nemotron-2B`</td>
      <td>Jet-Nemotron is a new family of hybrid-architecture language models that surpass state-of-the-art open-source full-attention language models, while achieving significant efficiency gains.</td>
    </tr>

    <tr>
      <td>**Trinity** (Nano, Mini)</td>
      <td>`arcee-ai/Trinity-Mini`</td>
      <td>Arcee's foundational MoE Trinity family of models, open weights under Apache 2.0.</td>
    </tr>

    <tr>
      <td>**LFM2** (230M, 350M, 1.2B)</td>
      <td>`LiquidAI/LFM2.5-1.2B-Instruct`</td>
      <td>Liquid AI's hybrid language model combining gated short convolutions with a small number of grouped query attention (GQA) blocks.</td>
    </tr>

    <tr>
      <td>**LFM2-MoE** (8B-A1B, 24B-A2B)</td>
      <td>`LiquidAI/LFM2-8B-A1B`</td>
      <td>Liquid AI's sparse Mixture-of-Experts variant of the LFM2 hybrid backbone, with SwiGLU experts, normalized sigmoid routing, and top-k expert selection.</td>
    </tr>

    <tr>
      <td>**Falcon-H1** (0.5B–34B)</td>
      <td>`tiiuae/Falcon-H1-34B-Instruct`</td>
      <td>TII's hybrid Mamba-Transformer architecture combining attention and state-space models for efficient long-context inference.</td>
    </tr>

    <tr>
      <td>**Mamba-Codestral** (7B, Mamba2)</td>
      <td>`mistralai/Mamba-Codestral-7B-v0.1`</td>
      <td>Mistral AI's pure Mamba2 state-space model for code generation; uses selective state spaces instead of attention, so it holds no KV cache.</td>
    </tr>

    <tr>
      <td>**Hunyuan-Large** (389B, MoE)</td>
      <td>`tencent/Tencent-Hunyuan-Large`</td>
      <td>Tencent's open-source MoE model with 389B total / 52B active parameters, featuring Cross-Layer Attention (CLA) for improved efficiency.</td>
    </tr>

    <tr>
      <td>**IBM Granite 4.0 (Hybrid, Dense)**</td>
      <td>`ibm-granite/granite-4.0-h-micro`, `ibm-granite/granite-4.0-micro`</td>
      <td>IBM Granite 4.0 micro models: hybrid Mamba–MoE (`h-micro`) and dense (`micro`) variants. Enterprise-focused reasoning models</td>
    </tr>

    <tr>
      <td>**Granite SWA** (IBM)</td>
      <td>`ibm-granite/granite-swash-2b`</td>
      <td>IBM Granite dense model with per-layer sliding window attention and learnable per-head attention sinks. Served by the Granite implementation; needs transformers ≥ 5.15.</td>
    </tr>

    <tr>
      <td>**Granite MoE SWA** (IBM)</td>
      <td>`ibm-granite/granite-swash-3b-a600m`</td>
      <td>IBM Granite Mixture-of-Experts model with per-layer sliding window attention, learnable attention sinks, and optional shared experts. Served by the GraniteMoE implementation; needs transformers ≥ 5.15.</td>
    </tr>

    <tr>
      <td>**Sarvam 2** (30B-A2B, 105B-A10B)</td>
      <td>`sarvamai/sarvam-2`</td>
      <td>Sarvam's Mixture-of-Experts models. The 105B variant uses MLA (Multi-head Latent Attention) and the 30B variant uses GQA, both with 128 routed experts.</td>
    </tr>

    <tr>
      <td>**Laguna XS.2** (poolside)</td>
      <td>`poolside/Laguna-XS.2`</td>
      <td>Poolside's hybrid sliding-window-attention MoE model (256 routed experts, 1 shared expert, sigmoid router) with per-layer-type RoPE (YARN on full-attention layers, default on sliding-attention layers) and a softplus per-head attention gate.</td>
    </tr>

    <tr>
      <td>**Mellum 2** (JetBrains)</td>
      <td>`JetBrains/Mellum2-12B-A2.5B-Base`, `JetBrains/Mellum2-12B-A2.5B-Thinking`</td>
      <td>JetBrains' Qwen3-MoE-based code generation model with interleaved sliding-window/full attention, per-layer-type RoPE, and per-layer dense/sparse MLP routing.</td>
    </tr>

    <tr>
      <td>**GigaChat 3.5**</td>
      <td>`ai-sage/GigaChat3.5-432B-A28B`</td>
      <td>GigaChat's hybrid MoE model: per-layer mix of DeepSeek-style MLA full attention and Qwen3 Gated-Delta-Net (GDN) linear attention, DeepSeek-style routed + shared experts, and multiple multi-token-prediction (MTP) heads for speculative decoding. Emits tool calls in GCML format.</td>
    </tr>
  </tbody>
</table>


# MindSpore Models
Source: https://docs.sglang.io/docs/supported-models/mindspore_models



## Introduction

MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang.

## Requirements

MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN 8.5.
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com).

## Supported Models

Currently, the following models are supported:

* **Qwen3**: Dense and MoE models
* **DeepSeek V3/R1**
* *More models coming soon...*

## Installation

> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](../hardware-platforms/ascend-npus/getting-started/installation) and then install `sgl-mindspore`:

```bash Install theme={null}
git clone https://github.com/mindspore-lab/sgl-mindspore.git
cd sgl-mindspore
pip install -e .
```

## Run Model

Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.

### Offline inference

Use the following script for offline inference:

```python Offline Inference theme={null}
import sglang as sgl

# Initialize the engine with MindSpore backend
llm = sgl.Engine(
    model_path="/path/to/your/model",  # Local model path
    device="npu",                      # Use NPU device
    model_impl="mindspore",            # MindSpore implementation
    attention_backend="ascend",        # Attention backend
    tp_size=1,                         # Tensor parallelism size
    dp_size=1                          # Data parallelism size
)

# Generate text
prompts = [
    "Hello, my name is",
    "The capital of France is",
    "The future of AI is"
]

sampling_params = {"temperature": 0, "top_p": 0.9}
outputs = llm.generate(prompts, sampling_params)

for prompt, output in zip(prompts, outputs):
    print(f"Prompt: {prompt}")
    print(f"Generated: {output['text']}")
    print("---")
```

### Start server

Launch a server with MindSpore backend:

```bash Command theme={null}
# Basic server startup
python3 -m sglang.launch_server \
    --model-path /path/to/your/model \
    --host 0.0.0.0 \
    --device npu \
    --model-impl mindspore \
    --attention-backend ascend \
    --tp-size 1 \
    --dp-size 1
```

For distributed server with multiple nodes:

```bash Command theme={null}
# Multi-node distributed server
python3 -m sglang.launch_server \
    --model-path /path/to/your/model \
    --host 0.0.0.0 \
    --device npu \
    --model-impl mindspore \
    --attention-backend ascend \
    --dist-init-addr 127.0.0.1:29500 \
    --nnodes 2 \
    --node-rank 0 \
    --tp-size 4 \
    --dp-size 2
```

## Troubleshooting

#### Debug Mode

Enable sglang debug logging by log-level argument.

```bash Debug Mode theme={null}
python3 -m sglang.launch_server \
    --model-path /path/to/your/model \
    --host 0.0.0.0 \
    --device npu \
    --model-impl mindspore \
    --attention-backend ascend \
    --log-level DEBUG
```

Enable mindspore info and debug logging by setting environments.

```bash Command theme={null}
export GLOG_v=1  # INFO
export GLOG_v=0  # DEBUG
```

#### Explicitly select devices

Use the following environment variable to explicitly select the devices to use.

```bash Command theme={null}
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7  # to set device
```

#### Some communication environment issues

In case of some environment with special communication environment, users need set some environment variables.

```bash Disable LCCL theme={null}
export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore
```

#### Some dependencies of protobuf

In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch.

```bash Command theme={null}
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python  # to avoid protobuf binary version mismatch
```

## Support

For MindSpore-specific issues:

* Refer to the [MindSpore documentation](https://www.mindspore.cn/)


# Use Models From ModelScope
Source: https://docs.sglang.io/docs/supported-models/modelscope



To use a model from [ModelScope](https://www.modelscope.cn), set the environment variable `SGLANG_USE_MODELSCOPE`.

```bash Set Environment Variable theme={null}
export SGLANG_USE_MODELSCOPE=true
```

We take [Qwen2-7B-Instruct](https://www.modelscope.cn/models/qwen/qwen2-7b-instruct) as an example.

Launch the Server:

```bash Python theme={null}
python -m sglang.launch_server --model-path qwen/Qwen2-7B-Instruct --port 30000
```

Or start it by docker:

```bash Docker theme={null}
docker run --gpus all \
    -p 30000:30000 \
    -v ~/.cache/modelscope:/root/.cache/modelscope \
    --env "SGLANG_USE_MODELSCOPE=true" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --host 0.0.0.0 --port 30000
```

Note that modelscope uses a different cache directory than huggingface. You may need to set it manually to avoid running out of disk space.


# Multimodal Language Models
Source: https://docs.sglang.io/docs/supported-models/multimodal_language_models



These models accept multi-modal inputs (e.g., images and text) and generate text output. They augment language models with multimodal encoders.

## Example launch Command

```bash Launch Server theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-11B-Vision-Instruct \  # example HF/local path
  --host 0.0.0.0 \
  --port 30000 \
```

> See the [OpenAI APIs section](../basic_usage/openai_api_vision) for how to send multimodal requests.

## Supported models

Below the supported models are summarized in a table.

If you are unsure if a specific architecture is implemented, you can search for it via GitHub. For example, to search for `Qwen2_5_VLForConditionalGeneration`, use the expression:

```text GitHub Search theme={null}
repo:sgl-project/sglang path:/^python\/sglang\/srt\/models\// Qwen2_5_VLForConditionalGeneration
```

in the GitHub search bar.

<table>
  <colgroup>
    <col />

    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family (Variants)</th>
      <th>Example HuggingFace Identifier</th>
      <th>Description</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Qwen-VL**</td>
      <td>`Qwen/Qwen3-VL-235B-A22B-Instruct`</td>
      <td>Alibaba's vision-language extension of Qwen; for example, Qwen2.5-VL (7B and larger variants) can analyze and converse about image content.</td>

      <td />
    </tr>

    <tr>
      <td>**DeepSeek-VL2**</td>
      <td>`deepseek-ai/deepseek-vl2`</td>
      <td>Vision-language variant of DeepSeek (with a dedicated image processor), enabling advanced multimodal reasoning on image and text inputs.</td>

      <td />
    </tr>

    <tr>
      <td>**DeepSeek-OCR / OCR-2**</td>
      <td>`deepseek-ai/DeepSeek-OCR-2`</td>
      <td>OCR-focused DeepSeek models for document understanding and text extraction.</td>
      <td>Use `--trust-remote-code`.</td>
    </tr>

    <tr>
      <td>**Janus-Pro** (1B, 7B)</td>
      <td>`deepseek-ai/Janus-Pro-7B`</td>
      <td>DeepSeek's open-source multimodal model capable of both image understanding and generation. Janus-Pro employs a decoupled architecture for separate visual encoding paths, enhancing performance in both tasks.</td>

      <td />
    </tr>

    <tr>
      <td>**MiniCPM-V / MiniCPM-o**</td>
      <td>`openbmb/MiniCPM-V-2_6`</td>
      <td>MiniCPM-V (2.6, \~8B) supports image inputs, and MiniCPM-o adds audio/video; these multimodal LLMs are optimized for end-side deployment on mobile/edge devices.</td>

      <td />
    </tr>

    <tr>
      <td>**Llama 3.2 Vision** (11B)</td>
      <td>`meta-llama/Llama-3.2-11B-Vision-Instruct`</td>
      <td>Vision-enabled variant of Llama 3 (11B) that accepts image inputs for visual question answering and other multimodal tasks.</td>

      <td />
    </tr>

    <tr>
      <td>**LLaVA** (v1.5 & v1.6)</td>
      <td>*e.g.* `liuhaotian/llava-v1.5-13b`</td>
      <td>Open vision-chat models that add an image encoder to LLaMA/Vicuna (e.g. LLaMA2 13B) for following multimodal instruction prompts.</td>

      <td />
    </tr>

    <tr>
      <td>**LLaVA-NeXT** (8B, 72B)</td>
      <td>`lmms-lab/llava-next-72b`</td>
      <td>Improved LLaVA models (with an 8B Llama3 version and a 72B version) offering enhanced visual instruction-following and accuracy on multimodal benchmarks.</td>

      <td />
    </tr>

    <tr>
      <td>**LLaVA-OneVision**</td>
      <td>`lmms-lab/llava-onevision-qwen2-7b-ov`</td>
      <td>Enhanced LLaVA variant integrating Qwen as the backbone; supports multiple images (and even video frames) as inputs via an OpenAI Vision API-compatible format.</td>

      <td />
    </tr>

    <tr>
      <td>**Gemma 3 (Multimodal)**</td>
      <td>`google/gemma-3-4b-it`</td>
      <td>Gemma 3's larger models (4B, 12B, 27B) accept images (each image encoded as 256 tokens) alongside text in a combined 128K-token context.</td>

      <td />
    </tr>

    <tr>
      <td>**Kimi-VL** (A3B)</td>
      <td>`moonshotai/Kimi-VL-A3B-Instruct`</td>
      <td>Kimi-VL is a multimodal model that can understand and generate text from images.</td>

      <td />
    </tr>

    <tr>
      <td>**Mistral-Small-3.1-24B**</td>
      <td>`mistralai/Mistral-Small-3.1-24B-Instruct-2503`</td>
      <td>Mistral 3.1 is a multimodal model that can generate text from text or images input. It also supports tool calling and structured output.</td>

      <td />
    </tr>

    <tr>
      <td>**Phi-4-multimodal-instruct**</td>
      <td>`microsoft/Phi-4-multimodal-instruct`</td>
      <td>Phi-4-multimodal-instruct is the multimodal variant of the Phi-4-mini model, enhanced with LoRA for improved multimodal capabilities. It supports text, vision and audio modalities in SGLang.</td>

      <td />
    </tr>

    <tr>
      <td>**MiMo-VL** (7B)</td>
      <td>`XiaomiMiMo/MiMo-VL-7B-RL`</td>
      <td>Xiaomi's compact yet powerful vision-language model featuring a native resolution ViT encoder for fine-grained visual details, an MLP projector for cross-modal alignment, and the MiMo-7B language model optimized for complex reasoning tasks.</td>

      <td />
    </tr>

    <tr>
      <td>**GLM-4.5V** (106B) / **GLM-4.1V**(9B)</td>
      <td>`zai-org/GLM-4.5V`</td>
      <td>GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning</td>
      <td>Use `--chat-template glm-4v`</td>
    </tr>

    <tr>
      <td>**PaddleOCR-VL** (0.9B, 1.5, 1.6)</td>
      <td>`PaddlePaddle/PaddleOCR-VL-1.6`</td>
      <td>Baidu's 0.9B document-parsing VLM: a NaViT-style dynamic-resolution SigLIP encoder on an ERNIE-4.5-0.3B backbone, covering text, tables, formulas, charts and seals in 109 languages. See the [cookbook page](/cookbook/autoregressive/Baidu/PaddleOCR-VL).</td>
      <td>Task is selected by the prompt (`OCR:`, `Table Recognition:`, ...). Leave `--trust-remote-code` off — `transformers` supports this architecture natively and its image processor is the faster one.</td>
    </tr>

    <tr>
      <td>**GLM-OCR**</td>
      <td>`zai-org/GLM-OCR`</td>
      <td>GLM-OCR: A fast and accurate general OCR model</td>

      <td />
    </tr>

    <tr>
      <td>**DotsVLM** (General/OCR)</td>
      <td>`rednote-hilab/dots.vlm1.inst`</td>
      <td>RedNote's vision-language model built on a 1.2B vision encoder and DeepSeek V3 LLM, featuring NaViT vision encoder trained from scratch with dynamic resolution support and enhanced OCR capabilities through structured image data training.</td>

      <td />
    </tr>

    <tr>
      <td>**DotsVLM-OCR**</td>
      <td>`rednote-hilab/dots.ocr`</td>
      <td>Specialized OCR variant of DotsVLM optimized for optical character recognition tasks with enhanced text extraction and document understanding capabilities.</td>
      <td>Don't use `--trust-remote-code`</td>
    </tr>

    <tr>
      <td>**NVILA** (8B, 15B, Lite-2B, Lite-8B, Lite-15B)</td>
      <td>`Efficient-Large-Model/NVILA-8B`</td>
      <td>`chatml`</td>
      <td>NVILA explores the full stack efficiency of multi-modal design, achieving cheaper training, faster deployment and better performance.</td>
    </tr>

    <tr>
      <td>**NVIDIA Nemotron Nano 2.0 VL**</td>
      <td>`nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16`</td>
      <td>NVIDIA Nemotron Nano v2 VL enables multi-image reasoning and video understanding, along with strong document intelligence, visual Q\&A and summarization capabilities. It builds on Nemotron Nano V2, a hybrid Mamba-Transformer LLM, in order to achieve higher inference throughput in long document and video scenarios.</td>
      <td>Use `--trust-remote-code`. You may need to adjust `--max-mamba-cache-size` \[default is 512] to fit memory constraints.</td>
    </tr>

    <tr>
      <td>**NVIDIA Cosmos3 Reasoner** (Nano)</td>
      <td>`nvidia/Cosmos3-Nano`</td>
      <td>The understanding tower (Qwen3-VL-based reasoner) of NVIDIA's Cosmos3 world model, served as a standalone VLM for image and video understanding from the unified diffusers-layout checkpoint. The generation (diffusion) tower is dropped at load time.</td>
      <td>Launch with `--model-type llm`; see [Cosmos3 LLM serving](#serving-the-llm-tower-of-cosmos3-checkpoints). Pass `--language-model-only` for text-only serving.</td>
    </tr>

    <tr>
      <td>**NVIDIA Cosmos3-Edge** (4B)</td>
      <td>`nvidia/Cosmos3-Edge`</td>
      <td>The 4B dense understanding tower of Cosmos3-Edge: an Arcee-structure text model with a SigLIP2 vision tower and an Edge-specific spatial-merge projector, supporting image and video understanding.</td>
      <td>Launch with `--model-type llm`; see [Cosmos3 LLM serving](#serving-the-llm-tower-of-cosmos3-checkpoints). Pass `--language-model-only` for text-only serving.</td>
    </tr>

    <tr>
      <td>**Ernie4.5-VL**</td>
      <td>`baidu/ERNIE-4.5-VL-28B-A3B-PT`</td>
      <td>Baidu's vision-language models(28B,424B). Support image and video comprehension, and also support thinking.</td>

      <td />
    </tr>

    <tr>
      <td>**JetVLM**</td>

      <td />

      <td>JetVLM is an vision-language model designed for high-performance multimodal understanding and generation tasks built upon Jet-Nemotron.</td>
      <td>Coming soon</td>
    </tr>

    <tr>
      <td>**Step3-VL** (10B)</td>
      <td>`stepfun-ai/Step3-VL-10B`</td>
      <td>StepFun's lightweight open-source 10B parameter VLM for multimodal intelligence, excelling in visual perception, complex reasoning, and human alignment.</td>

      <td />
    </tr>

    <tr>
      <td>**Qwen3-ASR** (0.6B, 1.7B)</td>
      <td>`Qwen/Qwen3-ASR-1.7B`</td>
      <td>Alibaba's automatic speech recognition models supporting 52 languages. Served via the `/v1/audio/transcriptions` endpoint.</td>

      <td />
    </tr>

    <tr>
      <td>**Qwen3-Omni**</td>
      <td>`Qwen/Qwen3-Omni-30B-A3B-Instruct`</td>
      <td>Alibaba's omni-modal MoE model. Currently supports the **Thinker** component (multimodal understanding for text, images, audio, and video), while the **Talker** component (audio generation) is not yet supported.</td>

      <td />
    </tr>

    <tr>
      <td>**LFM2-VL**</td>
      <td>`LiquidAI/LFM2.5-VL-1.6B`</td>
      <td>Liquid AI's vision-language model combining a SigLIP2 NaFlex vision encoder (variable resolution, native aspect ratio) with the LFM2 hybrid gated short conv + GQA language model. Supports multi-image inputs.</td>

      <td />
    </tr>

    <tr>
      <td>**LocateAnything** (3B)</td>
      <td>`nvidia/LocateAnything-3B`</td>
      <td>NVIDIA's visual grounding/detection model (MoonViT vision encoder + Qwen2 backbone) that emits \<ref>label\</ref>\<box>...\</box> outputs with coordinates normalized to \[0, 1000]. Covers object detection, phrase grounding, scene-text detection, GUI grounding, and pointing.</td>
      <td>Use `--trust-remote-code`. Set `skip_special_tokens=false` so the \<ref>/\<box> grounding tokens survive in the output. Constrained \<box> decoding is opt-in and client-side: start the server with `--enable-custom-logit-processor`, then pass `custom_logit_processor` (a top-level request field) and `custom_params` (inside `sampling_params`) together — use `LocateAnythingBoxGrammarLogitProcessor.build_sampling_params(config)` to build both from the config token ids.</td>
    </tr>
  </tbody>
</table>

## Audio Transcription

SGLang supports audio-only ASR models via the OpenAI-compatible `/v1/audio/transcriptions` endpoint. Upload an audio file and receive a transcription.

### Launch Command

```bash Command theme={null}
sglang serve \
  --model-path Qwen/Qwen3-ASR-1.7B \
  --served-model-name qwen3-asr \
  --trust-remote-code \
  --host 0.0.0.0 --port 30000
```

### Example Request

```bash Command theme={null}
curl http://localhost:30000/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=qwen3-asr \
  -F response_format=verbose_json
```

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>Example Identifier</th>
      <th>Notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Whisper**</td>
      <td>`openai/whisper-large-v3`</td>
      <td>OpenAI's speech recognition model.</td>
    </tr>

    <tr>
      <td>**Qwen3-ASR** (0.6B, 1.7B)</td>
      <td>`Qwen/Qwen3-ASR-1.7B`</td>
      <td>Use `--trust-remote-code`. Supports 52 languages.</td>
    </tr>
  </tbody>
</table>

## Video Input Support

SGLang supports video input for Vision-Language Models (VLMs), enabling temporal reasoning tasks such as video question answering, captioning, and holistic scene understanding. Video clips are decoded, key frames are sampled, and the resulting tensors are batched together with the text prompt, allowing multimodal inference to integrate visual and linguistic context.

<table>
  <colgroup>
    <col />

    <col />

    <col />
  </colgroup>

  <thead>
    <tr>
      <th>Model Family</th>
      <th>Example Identifier</th>
      <th>Video notes</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**Qwen-VL** (Qwen2-VL, Qwen2.5-VL, Qwen3-VL, Qwen3-Omni)</td>
      <td>`Qwen/Qwen3-VL-235B-A22B-Instruct`</td>
      <td>The processor gathers `video_data`, runs Qwen's frame sampler, and merges the resulting features with text tokens before inference.</td>
    </tr>

    <tr>
      <td>**Ling-VL** (Ling-3.0-flash-VL)</td>
      <td>`inclusionAI/Ling-3.0-flash-VL`</td>
      <td>InclusionAI's KDA/MLA hybrid MoE VLM (125B total / \~5.1B active) built on the Ling-3.0-flash backbone with a Qwen3-ViT-family vision tower. Image and video inputs are expanded by the Bailing processor with centered mRoPE positions. Requires `--trust-remote-code`. Audio is not supported.</td>
    </tr>

    <tr>
      <td>**GLM-4v** (4.5V, 4.1V, MOE)</td>
      <td>`zai-org/GLM-4.5V`</td>
      <td>Video clips are read with Decord, converted to tensors, and passed to the model alongside metadata for rotary-position handling.</td>
    </tr>

    <tr>
      <td>**NVILA** (Full & Lite)</td>
      <td>`Efficient-Large-Model/NVILA-8B`</td>
      <td>The runtime samples eight frames per clip and attaches them to the multimodal request when `video_data` is present.</td>
    </tr>

    <tr>
      <td>**LLaVA video variants** (LLaVA-NeXT-Video, LLaVA-OneVision)</td>
      <td>`lmms-lab/LLaVA-NeXT-Video-7B`</td>
      <td>The processor routes video prompts to the LlavaVid video-enabled architecture, and the provided example shows how to query it with `sgl.video(...)` clips.</td>
    </tr>

    <tr>
      <td>**NVIDIA Nemotron Nano 2.0 VL**</td>
      <td>`nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16`</td>
      <td>The processor samples at 2 FPS, at a max of 128 frames, as per model training. The model uses [EVS](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/multimodal/evs/README.md), a pruning method that removes redundant tokens from video embeddings. By default `video_pruning_rate=0.7`. Change this by providing: `--json-model-override-args '{"video_pruning_rate": 0.0}'` to disable EVS, for example.</td>
    </tr>

    <tr>
      <td>**JetVLM**</td>

      <td />

      <td>The runtime samples eight frames per clip and attaches them to the multimodal request when `video_data` is present.</td>
    </tr>
  </tbody>
</table>

Use `sgl.video(path, num_frames)` when building prompts to attach clips from your SGLang programs.

Example OpenAI-compatible request that sends a video clip:

```python Complete Example theme={null}
import requests

url = "http://localhost:30000/v1/chat/completions"

data = {
    "model": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s happening in this video?"},
                {
                    "type": "video_url",
                    "video_url": {
                        "url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"
                    },
                },
            ],
        }
    ],
    "max_tokens": 300,
}

response = requests.post(url, json=data)
print(response.text)
```

## Serving the LLM tower of Cosmos3 checkpoints

`nvidia/Cosmos3-Nano` and `nvidia/Cosmos3-Edge` are unified world-model checkpoints in a diffusers layout, so `sglang serve` routes them to the diffusion runtime by default (see the [Cosmos3 cookbook page](/cookbook/diffusion/Cosmos/Cosmos3) for media generation). To serve the understanding tower as a vision-language model instead, force the LLM backend with `--model-type llm`:

```bash Launch Server theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --model-type llm \
  --host 0.0.0.0 --port 30000
```

The same command serves `nvidia/Cosmos3-Edge` by swapping the model path. Both accept image and video inputs through the OpenAI-compatible API:

```bash Example Request theme={null}
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Cosmos3-Nano",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image."},
        {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}
      ]
    }],
    "max_tokens": 128
  }'
```

To serve only the text model (skipping the vision tower entirely), add `--language-model-only`. In this mode `/model_info` reports `has_image_understanding: false`, and requests containing image or video inputs are rejected:

```bash Text-Only Launch theme={null}
sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --model-type llm \
  --language-model-only \
  --host 0.0.0.0 --port 30000
```

## Usage Notes

### Performance Optimization

For multimodal models, you can use `--mm-feature-transport=cuda_ipc` to optimize for latency at the cost of increased GPU memory usage:

* **Default behavior**: Multimodal feature tensors are moved to CPU after processing to save GPU memory
* **With `--mm-feature-transport=cuda_ipc`**: Feature tensors stay in a bounded GPU pool shared over CUDA IPC, reducing device-to-host copy overhead and improving latency, but consuming more GPU memory

Use this option when you have sufficient GPU memory and want to minimize latency for multimodal inference.

### Multimodal Inputs Limitation

* **Use `--mm-process-config '&#123;"image":&#123;"max_pixels":1048576&#125;,"video":&#123;"fps":3,"max_pixels":602112,"max_frames":60&#125;&#125;'`**: To set `image`, `video`, and `audio` input limits.

This can reduce GPU memory usage, improve inference speed, and help to avoid OOM, but may impact model performance, thus set a proper value based on your specific use case. The config entries are passed as `images_kwargs`, `videos_kwargs`, and `audio_kwargs` to the HuggingFace processor, so each modality's settings are kept separate and do not collide. Refer to the HuggingFace documentation for your model's processor to understand the available parameters.

For GLM-V models, the per-request `video_url` fields `fps`, `max_frames`, `max_tokens_per_frame`, and `max_image_tokens` can only tighten the deployment's video budget. Larger `fps`, `max_frames`, and `max_image_tokens` values are clamped to the deployment setting, and a `max_tokens_per_frame` that would exceed the video token budget is ignored. Raise the limits with `--mm-process-config` instead.

### Bidirectional Attention in Multimodal Model Serving

**Note for serving the Gemma-3 multimodal model**:

As mentioned in [Welcome Gemma 3: Google's all new multimodal, multilingual, long context open LLM
](https://huggingface.co/blog/gemma3#multimodality), Gemma-3 employs bidirectional attention between image tokens during the prefill phase. Currently, SGLang only supports bidirectional attention when using the Triton Attention Backend. Note, however, that SGLang's current bidirectional attention implementation is incompatible with both CUDA Graph and Chunked Prefill.

To enable bidirectional attention, you can use the `TritonAttnBackend` while disabling CUDA Graph and Chunked Prefill. Example launch command:

```bash Bidirectional Attention theme={null}
python -m sglang.launch_server \
  --model-path google/gemma-3-4b-it \
  --host 0.0.0.0 --port 30000 \
  --enable-multimodal \
  --dtype bfloat16 --triton-attention-reduce-in-fp32 \
  --attention-backend triton \ # Use Triton attention backend
  --disable-cuda-graph \ # Disable Cuda Graph
  --chunked-prefill-size -1 # Disable Chunked Prefill
```

If higher serving performance is required and a certain degree of accuracy loss is acceptable, you may choose to use other attention backends, and you can also enable features like CUDA Graph and Chunked Prefill for better performance, but note that the model will fall back to using causal attention instead of bidirectional attention.


# Rerank models
Source: https://docs.sglang.io/docs/supported-models/rerank_models



SGLang offers comprehensive support for rerank models by incorporating optimized serving frameworks with a flexible programming interface. This setup enables efficient processing of cross-encoder reranking tasks, improving the accuracy and relevance of search result ordering. SGLang’s design ensures high throughput and low latency during reranker model deployment, making it ideal for semantic-based result refinement in large-scale retrieval systems.

> [!WARNING]
>
> Rerank models in SGLang fall into two categories:
>
> * **Cross-encoder rerank models**: run with `--is-embedding` (embedding runner).
> * **Decoder-only rerank models**: run **without** `--is-embedding` and use next-token logprob scoring (yes/no).
>   * Text-only (e.g. Qwen3-Reranker)
>   * Multimodal (e.g. Qwen3-VL-Reranker): also supports image/video content
>
> Some models may require `--trust-remote-code`.

## Supported rerank models

<table>
  <thead>
    <tr>
      <th>Model Family (Rerank)</th>
      <th>Example HuggingFace Identifier</th>
      <th>Chat Template</th>
      <th>Description</th>
    </tr>
  </thead>

  <tbody>
    <tr>
      <td>**BGE-Reranker (BgeRerankModel)**</td>
      <td>`BAAI/bge-reranker-v2-m3`</td>
      <td>N/A</td>
      <td>Currently only support `attention-backend` `triton` and `torch_native`. High-performance cross-encoder reranker model from BAAI. Suitable for reranking search results based on semantic relevance.</td>
    </tr>

    <tr>
      <td>**Qwen3-Reranker (decoder-only yes/no)**</td>
      <td>`Qwen/Qwen3-Reranker-8B`</td>
      <td>`examples/chat_template/qwen3_reranker.jinja`</td>
      <td>Decoder-only reranker using next-token logprob scoring for labels (yes/no). Launch **without** `--is-embedding`.</td>
    </tr>

    <tr>
      <td>**Qwen3-VL-Reranker (multimodal yes/no)**</td>
      <td>`Qwen/Qwen3-VL-Reranker-2B`</td>
      <td>`examples/chat_template/qwen3_vl_reranker.jinja`</td>
      <td>Multimodal decoder-only reranker supporting text, images, and videos. Uses yes/no logprob scoring. Launch **without** `--is-embedding`.</td>
    </tr>
  </tbody>
</table>

## Cross-Encoder Rerank (embedding runner)

### Launch Command

```shell theme={null}
python3 -m sglang.launch_server \
  --model-path BAAI/bge-reranker-v2-m3 \
  --host 0.0.0.0 \
  --disable-radix-cache \
  --chunked-prefill-size -1 \
  --attention-backend triton \
  --is-embedding \
  --port 30000
```

### Example Client Request

```python theme={null}
import requests

url = "http://127.0.0.1:30000/v1/rerank"

payload = {
    "model": "BAAI/bge-reranker-v2-m3",
    "query": "what is panda?",
    "documents": [
        "hi",
        "The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China."
    ],
    "top_n": 1,
    "return_documents": True
}

response = requests.post(url, json=payload)
response_json = response.json()

for item in response_json:
    if item.get("document"):
        print(f"Score: {item['score']:.2f} - Document: '{item['document']}'")
    else:
        print(f"Score: {item['score']:.2f} - Index: {item['index']}")
```

**Request Parameters:**

* `query` (required): The query text to rank documents against
* `documents` (required): List of documents to be ranked
* `model` (required): Model to use for reranking
* `top_n` (optional): Maximum number of documents to return. Defaults to returning all documents. If specified value is greater than the total number of documents, all documents will be returned.
* `return_documents` (optional): Whether to return documents in the response. Defaults to `True`.

## Qwen3-Reranker (decoder-only yes/no rerank)

### Launch Command

```shell theme={null}
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-Reranker-0.6B \
  --trust-remote-code \
  --disable-radix-cache \
  --host 0.0.0.0 \
  --port 8001 \
  --chat-template examples/chat_template/qwen3_reranker.jinja
```

> [!NOTE]
>
> Qwen3-Reranker uses decoder-only logprob scoring (yes/no). Do NOT launch it with `--is-embedding`.

### Example Client Request (supports optional instruct, top\_n, and return\_documents)

```shell theme={null}
curl -X POST http://127.0.0.1:8001/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-Reranker-0.6B",
    "query": "法国首都是哪里？",
    "documents": [
      "法国的首都是巴黎。",
      "德国的首都是柏林。",
      "香蕉是黄色的水果。"
    ],
    "instruct": "Given a web search query, retrieve relevant passages that answer the query.",
    "top_n": 2,
    "return_documents": true
  }'
```

**Request Parameters:**

* `query` (required): The query text to rank documents against
* `documents` (required): List of documents to be ranked
* `model` (required): Model to use for reranking
* `instruct` (optional): Instruction text for the reranker
* `top_n` (optional): Maximum number of documents to return. Defaults to returning all documents. If specified value is greater than the total number of documents, all documents will be returned.
* `return_documents` (optional): Whether to return documents in the response. Defaults to `True`.

### Response Format

`/v1/rerank` returns a list of objects (sorted by descending score):

* `score`: float, higher means more relevant
* `document`: the original document string (only included when `return_documents` is `true`)
* `index`: the original index in the input `documents`
* `meta_info`: optional debug/usage info (may be present for some models)

The number of returned results is controlled by the `top_n` parameter. If `top_n` is not specified or is greater than the total number of documents, all documents are returned.

Example (with `return_documents: true`):

```json theme={null}
[
  {"score": 0.99, "document": "法国的首都是巴黎。", "index": 0},
  {"score": 0.01, "document": "德国的首都是柏林。", "index": 1},
  {"score": 0.00, "document": "香蕉是黄色的水果。", "index": 2}
]
```

Example (with `return_documents: false`):

```json theme={null}
[
  {"score": 0.99, "index": 0},
  {"score": 0.01, "index": 1},
  {"score": 0.00, "index": 2}
]
```

Example (with `top_n: 2`):

```json theme={null}
[
  {"score": 0.99, "document": "法国的首都是巴黎。", "index": 0},
  {"score": 0.01, "document": "德国的首都是柏林。", "index": 1}
]
```

### Common Pitfalls

* **`--chat-template` is required.** Without `--chat-template examples/chat_template/qwen3_reranker.jinja`, the server does not recognize the model as a decoder-only reranker and returns a 400 error: `"This model does not appear to be an embedding model by default. Please add `--is-embedding`..."`. The fix is to add the chat template flag, NOT `--is-embedding`.
* If you launch Qwen3-Reranker with `--is-embedding`, `/v1/rerank` cannot compute yes/no logprob scores. Relaunch **without** `--is-embedding`.
* If you see a validation error like "score should be a valid number" and the backend returned a list, upgrade to a version that coerces `embedding[0]` into `score` for rerank responses.

## Qwen3-VL-Reranker (multimodal decoder-only rerank)

Qwen3-VL-Reranker extends the Qwen3-Reranker to support multimodal content, allowing reranking of documents containing text, images, and videos.

### Launch Command

```shell theme={null}
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen3-VL-Reranker-2B \
  --trust-remote-code \
  --disable-radix-cache \
  --host 0.0.0.0 \
  --port 30000 \
  --chat-template examples/chat_template/qwen3_vl_reranker.jinja
```

> [!NOTE]
>
> Qwen3-VL-Reranker uses decoder-only logprob scoring (yes/no) like Qwen3-Reranker. Do NOT launch it with `--is-embedding`.

### Text-Only Reranking (backward compatible)

```python theme={null}
import requests

url = "http://127.0.0.1:30000/v1/rerank"

payload = {
    "model": "Qwen3-VL-Reranker-2B",
    "query": "What is machine learning?",
    "documents": [
        "Machine learning is a branch of artificial intelligence that enables computers to learn from data.",
        "The weather in Paris is usually mild with occasional rain.",
        "Deep learning is a subset of machine learning using neural networks with many layers.",
    ],
    "instruct": "Retrieve passages that answer the question.",
    "return_documents": True
}

response = requests.post(url, json=payload)
results = response.json()

for item in results:
    print(f"Score: {item['score']:.4f} - {item['document'][:60]}...")
```

### Image Reranking (text query, image/mixed documents)

```python theme={null}
import requests

url = "http://127.0.0.1:30000/v1/rerank"

payload = {
    "query": "A woman playing with her dog on a beach at sunset.",
    "documents": [
        # Document 1: Text description
        "A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset.",
        # Document 2: Image URL
        [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://example.com/beach_dog.jpeg"
                }
            }
        ],
        # Document 3: Text + Image (mixed)
        [
            {"type": "text", "text": "A joyful scene at the beach:"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://example.com/beach_dog.jpeg"
                }
            }
        ]
    ],
    "instruct": "Retrieve images or text relevant to the user's query.",
    "return_documents": False
}

response = requests.post(url, json=payload)
results = response.json()

for item in results:
    print(f"Index: {item['index']}, Score: {item['score']:.4f}")
```

### Multimodal Query Reranking (query with image)

```python theme={null}
import requests

url = "http://127.0.0.1:30000/v1/rerank"

payload = {
    # Query with text and image
    "query": [
        {"type": "text", "text": "Find similar images to this:"},
        {
            "type": "image_url",
            "image_url": {
                "url": "https://example.com/reference_image.jpeg"
            }
        }
    ],
    "documents": [
        "A cat sleeping on a couch.",
        "A woman and her dog enjoying the sunset at the beach.",
        "A busy city street with cars and pedestrians.",
        [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://example.com/similar_image.jpeg"
                }
            }
        ]
    ],
    "instruct": "Find images or descriptions similar to the query image."
}

response = requests.post(url, json=payload)
results = response.json()

for item in results:
    print(f"Index: {item['index']}, Score: {item['score']:.4f}")
```

### Request Parameters (Multimodal)

* `query` (required): Can be a string (text-only) or a list of content parts:
  * `&#123;"type": "text", "text": "..."&#125;` for text
  * `&#123;"type": "image_url", "image_url": &#123;"url": "..."&#125;&#125;` for images
  * `&#123;"type": "video_url", "video_url": &#123;"url": "..."&#125;&#125;` for videos
* `documents` (required): List where each document can be a string or list of content parts (same format as query)
* `instruct` (optional): Instruction text for the reranker
* `top_n` (optional): Maximum number of documents to return
* `return_documents` (optional): Whether to return documents in the response (default: `false`)

### Common Pitfalls

* Always use `--chat-template examples/chat_template/qwen3_vl_reranker.jinja` for Qwen3-VL-Reranker.
* Do NOT launch with `--is-embedding`.
* For best results, use `--disable-radix-cache` to avoid caching issues with multimodal content.
* **Note**: Currently only `Qwen3-VL-Reranker-2B` is tested and supported. The 8B model may have different behavior and is not guaranteed to work with this template.


# Reward models
Source: https://docs.sglang.io/docs/supported-models/reward_models



These models output a scalar reward score or classification result, often used in reinforcement learning or content moderation tasks.

They are executed with `--is-embedding` and some may require `--trust-remote-code`.

## Example launch Command

```shell Command theme={null}
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-Math-RM-72B \  # example HF/local path
  --is-embedding \
  --host 0.0.0.0 \
  --tp-size=4 \                          # set for tensor parallelism
  --port 30000 \
```

## Supported models

| Model Family (Reward) | Example HuggingFace Identifier | Description |
| - | - | - |
| **Llama (3.1 Reward / `LlamaForSequenceClassification`)** | `Skywork/Skywork-Reward-Llama-3.1-8B-v0.2` | Reward model (preference classifier) based on Llama 3.1 (8B) for scoring and ranking responses for RLHF. |
| **Gemma 2 (27B Reward / `Gemma2ForSequenceClassification`)** | `Skywork/Skywork-Reward-Gemma-2-27B-v0.2` | Derived from Gemma‑2 (27B), this model provides human preference scoring for RLHF and multilingual tasks. |
| **InternLM 2 (Reward / `InternLM2ForRewardMode`)** | `internlm/internlm2-7b-reward` | InternLM 2 (7B)–based reward model used in alignment pipelines to guide outputs toward preferred behavior. |
| **Qwen2.5 (Reward - Math / `Qwen2ForRewardModel`)** | `Qwen/Qwen2.5-Math-RM-72B` | A 72B math-specialized RLHF reward model from the Qwen2.5 series, tuned for evaluating and refining responses. |
| **Qwen2.5 (Reward - Sequence / `Qwen2ForSequenceClassification`)** | `jason9693/Qwen2.5-1.5B-apeach` | A smaller Qwen2.5 variant used for sequence classification, offering an alternative RLHF scoring mechanism. |


# How to Support New Models
Source: https://docs.sglang.io/docs/supported-models/support_new_models

This document explains how to add support for new language models and multimodal large language models (MLLMs) in SGLang. It also covers how to test new models and register external implementations.

This document explains how to add support for new language models and multimodal large language models (MLLMs) in
SGLang. It also covers how to test new models and register external implementations.

## How to Support a New Language Model

To support a new model in SGLang, you only need to add a single file under
the [SGLang Models Directory](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/models). You can learn
from existing model implementations and create a new file for your model. For most models, you should be able to find a
similar model to start with (e.g., starting from Llama). Also refer how
to [port a Model from vLLM to SGLang](#port-a-model-from-vllm-to-sglang)

## How to Support a New Multimodal Large Language Model

To support a new multimodal large language model (MLLM) in SGLang, there are several key components in addition to the
standard LLM support:

1. **Register your new model as multimodal**:
   Extend `is_multimodal_model`
   in [model\_config.py](https://github.com/sgl-project/sglang/blob/0ab3f437aba729b348a683ab32b35b214456efc7/python/sglang/srt/configs/model_config.py#L561)
   to return `True` for your model.

2. **Register a new chat-template**:
   Only when your default chat-template is unable to accept images as input: Register a new chat template in [conversation.py](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/parser/conversation.py) and the corresponding matching function.

3. **Multimodal Data Processor**:
   Define a new `Processor` class that inherits from `BaseMultimodalProcessor` and register this processor as your
   model’s dedicated processor.
   See [multimodal\_processor.py](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/multimodal/processors)
   for more details.

4. **Handle Multimodal Tokens**:
   Implement a `pad_input_ids` function for your new model. In this function, multimodal tokens in the prompt should be
   expanded (if necessary) and padded with multimodal-data-hashes so that SGLang can recognize different multimodal data
   with `RadixAttention`.

5. **Handle Image Feature Extraction**:
   Implement a `get_image_feature` function for your new model, which extracts image features from raw image data and converts them into the embeddings used by the language model.

6. **Adapt to Vision Attention**:
   Adapt the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.

You can refer to [Qwen2VL](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen2_vl.py) or
other mllm implementations. These models demonstrate how to correctly handle both multimodal and textual inputs.

## Testing and Debugging

Please note all your testing and benchmarking results in PR description.

### Interactive Debugging

For interactive debugging, compare the outputs of Hugging Face/Transformers and SGLang. The following two commands
should give the same text output and very similar prefill logits:

* Get the reference output:
  ```bash Command theme={null}
  python3 scripts/playground/reference_hf.py --model-path [new model] --model-type {text,vlm}
  ```
* Get the SGLang output:
  ```bash Command theme={null}
  python3 -m sglang.benchmark.one_batch --correct --model [new model]
  ```

### Add the Model to the Test Suite

To ensure the new model is well maintained, add it to the test suite by including it in the `ALL_OTHER_MODELS` list in
the [test\_generation\_models.py](https://github.com/sgl-project/sglang/blob/main/test/registered/models/test_generation_models.py)
file, test the new model on your local machine and report the results on demonstrative benchmarks (GSM8K, MMLU, MMMU,
MMMU-Pro, etc.) in your PR. \\
For VLMs, also include a test in `test_vision_openai_server_&#123;x&#125;.py` (e.g. [test\_vision\_openai\_server\_a.py](https://github.com/sgl-project/sglang/blob/main/test/registered/vlm/test_vision_openai_server_a.py)).

This is an example command to run to test a new model on your local machine:

```bash Run Test theme={null}
ONLY_RUN=Qwen/Qwen2-1.5B python3 -m unittest test_generation_models.TestGenerationModels.test_others
```

### Benchmark

* **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow [https://docs.sglang.io/developer\_guide/benchmark\_and\_profiling.html](https://docs.sglang.io/developer_guide/benchmark_and_profiling.html) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer).
* **(Optional) Other evals**: If you ran other evals, please note the results in PR description.

## Port a Model from vLLM to SGLang

The [vLLM Models Directory](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) is a valuable
resource, as vLLM covers many models. SGLang reuses vLLM’s interface and some layers, making it easier to port models
from vLLM to SGLang.

To port a model from vLLM to SGLang:

* Compare these two files for guidance:
  * [SGLang Llama Implementation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama.py)
  * [vLLM Llama Implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py)
* The major differences include:
  * **Replace vLLM’s `Attention` with `RadixAttention`** (ensure you pass `layer_id` to `RadixAttention`).
  * **Replace vLLM’s `LogitsProcessor` with SGLang’s `LogitsProcessor`.**
  * **Replace the multi-headed `Attention` of ViT with SGLang’s `VisionAttention`.**
  * **Replace other vLLM layers** (such as `RMSNorm`, `SiluAndMul`) with SGLang layers.
  * **Remove `Sample`.**
  * **Change the `forward()` functions** and add a `forward_batch()` method.
  * **Add `EntryClass`** at the end.
  * **Ensure that the new implementation uses only SGLang components** and does not rely on any vLLM components.

Note: make sure you add your new model to the supported models list in the supported models documentation.

## Registering an External Model Implementation

In addition to the methods above, you can register your new model with the `ModelRegistry` before launching the server.
This allows you to integrate your model without modifying the source code.

For example:

```python Register Model theme={null}
from sglang.srt.models.registry import ModelRegistry
from sglang.srt.entrypoints.http_server import launch_server

# For a single model, add it to the registry:
ModelRegistry.models[model_name] = model_class

# For multiple models, you can imitate the import_model_classes() function:
from functools import lru_cache

@lru_cache()
def import_new_model_classes():
    model_arch_name_to_cls = {}
    # Populate model_arch_name_to_cls with your new model classes.
    ...
    return model_arch_name_to_cls

ModelRegistry.models.update(import_new_model_classes())

# Launch the server with your server arguments:
launch_server(server_args)
```

## Example: Implementing and Serving a Llama Wrapper Model

Below is an introductory, step-by-step walkthrough on how to implement a new model end-to-end in SGLang and then run it via the [Offline Engine](../basic_usage/offline_engine_api).

### Implementing Our Model

To keep things simple, this new model will be a simple wrapper around [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), and our goal will be just to bias the output logits for each `forward` call by taking the square root of each individual logit.

Let's start by defining our model in a file called `llama_wrapper.py`.
The first step is to import the necessary libraries from SRT, which is SGLang's internal backend.

```python Example theme={null}
# In the file `llama_wrapper.py`

import torch
from transformers import LlamaConfig
from typing import Optional
from sglang.srt.layers.logits_processor import LogitsProcessorOutput
from sglang.srt.layers.quantization.base_config import QuantizationConfig
from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors

from sglang.srt.models.llama import LlamaForCausalLM
```

Next, we declare a new `class` for our model and have it inherit from `LlamaForCausalLM`, which allows our model to access `LlamaForCausalLM`'s predefined modules and layers, such as `LlamaAttention` and `LlamaMLP`.
Note that almost all model implementations take in `config` and `quant_config` as arguments for their `__init__` method; `config` and `quant_config` are passed in via [`model_loader/loader.py`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_loader/loader.py#L219).
Because we have inherited from `LlamaForCausalLM`, we can pass our parameters directly to its constructor, which will set the member variables for us.

```python Class Definition theme={null}
class LlamaWrapper(LlamaForCausalLM):
    def __init__(
        self,
        config: LlamaConfig,
        quant_config: Optional[QuantizationConfig] = None,
        prefix: str = "",
    ) -> None:
        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
```

Now, we want to define the `forward` method, which is what will be called at inference time.
Note that the signature for `forward` is essentially the same for any model; you can take a look at the other models defined in the [`models` directory](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/) for references.
To see where exactly `forward` is called in the SGLang runtime's internals, take a look at [`forward_decode`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1705) and [`forward_extend`](https://github.com/sgl-project/sglang/blob/bf72b80122fd888bf619d17b96fa3e323ab809fc/python/sglang/srt/model_executor/model_runner.py#L1724) in the [`ModelRunner` class](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/model_executor/model_runner.py).

```python Forward Method Signature theme={null}
    @torch.no_grad()
    def forward(
        self,
        input_ids: torch.Tensor,
        positions: torch.Tensor,
        forward_batch: ForwardBatch,
        pp_proxy_tensors: Optional[PPProxyTensors] = None,
        input_embeds: Optional[torch.Tensor] = None,
        get_embedding: bool = False,
    ) -> LogitsProcessorOutput:
```

We now call the `__call__` method for `self.model` (which is a member variable that `LlamaForCausalLM` defines in its `__init__` method), which eventually calls `LlamaForCausalLM`'s `forward` method.
After that, we feed the `hidden_states` into our model's `LogitsProcessor` (again defined in `LlamaForCausalLM`).

```python Call Model and LogitsProcessor theme={null}
        hidden_states = self.model(
            input_ids,
            positions,
            forward_batch,
            input_embeds,
            pp_proxy_tensors=pp_proxy_tensors,
        )

        res: LogitsProcessorOutput = self.logits_processor(
            input_ids,
            hidden_states,
            self.lm_head,
            forward_batch,
        )
```

After receiving the logits for the next token, we can finally perform our biasing step.

```python Logit Biasing theme={null}
        orig_logits = res.next_token_logits
        res.next_token_logits = torch.where(
            orig_logits > 0,
            orig_logits.sqrt(),
            orig_logits
        )

        return res
```

Now, our `LlamaWrapper` model is created and ready to be served!

### Serving Our Model Via SGLang's Offline Engine

The next step of this walkthrough involves hosting our new model offline, so that it can be served locally and without an HTTP server.

First, create a new file called `run.py`.
Now, we must ensure that SGLang's `ModelRegistry` can find our model.
To do this, we first download the model's configuration and weights from Huggingface.

```python Example theme={null}
# In the file `run.py`

import asyncio
from functools import lru_cache
from huggingface_hub import snapshot_download
from llama_wrapper import LlamaWrapper # Make sure to import our new model!
import sglang as sgl
from sglang.srt.models.registry import ModelRegistry

# Make sure to request access to this model on Huggingface, then export your
# `HF_TOKEN` to download the model snapshot
llama_dir = snapshot_download(
    repo_id="meta-llama/Llama-3.1-8B-Instruct",
    local_dir="./llama_ckpt",
)
```

Now that we have our model on disk, we want to point it to `LlamaWrapper` by changing the `architectures` field in `./llama_ckpt/config.json` to be `LlamaWrapper`.
That way, when we pass in the path of our model checkpoint to SGLang, it will know that we want to use "LlamaWrapper" instead of "LlamaForCausalLM" as our model.

```python Example theme={null}
{
  "architectures": [
   #  "LlamaForCausalLM"
    "LlamaWrapper"
  ],
  ...
}
```

However, if we don't link our `LlamaWrapper` class to the "LlamaWrapper" registry keyword, then SGLang won't be able to find our model.
Thus, to register our `LlamaWrapper`, we want to follow the steps in the above section titled "Registering an External Model Implementation".

```python Register LlamaWrapper theme={null}
@lru_cache()
def import_new_model_classes():
    model_arch_name_to_cls = {"LlamaWrapper": LlamaWrapper}
    return model_arch_name_to_cls

ModelRegistry.models.update(import_new_model_classes())
```

Lastly, when we create our `Engine`, we just pass in the path to the local model directory.
Then, our `LlamaWrapper` is ready to be served; for this walkthrough, we will use SGLang `Engine`'s non-streaming asynchronous generation endpoint.

```python Example theme={null}
def main():
    llm = sgl.Engine(model_path="./llama_ckpt")
    sampling_params = {"temperature": 0.2, "top_k": 5}
    prompts = [
        "Write a short, neutral self-introduction for a fictional character. Hello, my name is",
        "Provide a concise factual statement about France’s capital city. The capital of France is",
        "Explain possible future trends in artificial intelligence. The future of AI is",
    ]

    asyncio.run(run_llm(llm, sampling_params, prompts))

    llm.shutdown()

async def run_llm(
    llm,
    sampling_params,
    prompts,
) -> None:
    outputs = await llm.async_generate(prompts, sampling_params)

    for prompt, output in zip(prompts, outputs):
        print(f"\nPrompt: {prompt}")
        print(f"Generated text: {output['text']}")

if __name__ == "__main__":
    main()
```

Now, when we call `python run.py`, we will get the outputs of our newly created model!

## Serving External Models via the Standard CLI

The previous sections show how to register a model programmatically via `ModelRegistry` and serve it through the Offline Engine. Similar to vLLM model plugin, there is an alternative that lets you keep using the standard `python -m sglang.launch_server` CLI without modifying any SGLang source code: you can register your model using the `SGLANG_EXTERNAL_MODEL_PACKAGE` environment variable.

### The `EntryClass` Variable

When SGLang scans a model package, it looks for the variable `EntryClass` at the module level of your Python file. The [model registry](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/registry.py) imports your file, checks for `EntryClass`, and registers the class assigned to it. If you are using a model based on HuggingFace, the name of this class needs to match the `"architectures"` field in your model's `config.json`.

For example, if you are implementing a Llama wrapper, add this line at the end of your model file:

```python Example theme={null}
# This is what "Add EntryClass at the end" means
EntryClass = LlamaWrapper
```

### Example: Text-Only Model

Using the same Llama wrapper from the previous section, here is how to package and serve it via the CLI.

1. Create your project

```
sglang_custom_project/
|----setup.py
|----custom_llm/
     |----__init__.py
     |----llama_wrapper.py
```

Write the `setup.py`:

```python Example theme={null}
# sglang_custom_project/setup.py

from setuptools import setup, find_packages
setup(
    name="sglang-custom-plugins",
    version="0.1",
    packages=find_packages(),
)
```

2. Write your model code

Inside `llama_wrapper.py`, write your model and include `EntryClass`:

```python Example theme={null}
# sglang_custom_project/custom_llm/llama_wrapper.py

import torch
from typing import Optional
from sglang.srt.layers.logits_processor import LogitsProcessorOutput
from sglang.srt.layers.quantization.base_config import QuantizationConfig
from sglang.srt.model_executor.forward_batch_info import ForwardBatch, PPProxyTensors
from sglang.srt.models.llama import LlamaForCausalLM

class LlamaWrapper(LlamaForCausalLM):
    def __init__(self, config, quant_config: Optional[QuantizationConfig] = None,
                 prefix: str = "") -> None:
        super().__init__(config=config, quant_config=quant_config, prefix=prefix)
    @torch.no_grad()
    def forward(self, input_ids, positions, forward_batch,
                pp_proxy_tensors=None, input_embeds=None, get_embedding=False):
        hidden_states = self.model(
            input_ids, positions, forward_batch, input_embeds,
            pp_proxy_tensors=pp_proxy_tensors,
        )
        res: LogitsProcessorOutput = self.logits_processor(
            input_ids, hidden_states, self.lm_head, forward_batch,
        )

        orig = res.next_token_logits
        res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig)
        return res

# Don't forget to add EntryClass
EntryClass = LlamaWrapper
```

3. Install your package

Run this inside your `sglang_custom_project` directory to install your code into the active Python environment:

```bash Command theme={null}
pip install -e .
```

4. Update your `config.json`

Update the `config.json` under your HuggingFace model checkpoint directory so the `architectures` field matches your class name:

```json Config theme={null}
{
  "architectures": ["LlamaWrapper"],
  ...
}
```

5. Launch the server

Set the environment variable before running the CLI:

```bash Command theme={null}
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_llm
python -m sglang.launch_server \
    --model-path /path/to/Llama-3.1-8B-Instruct \
    --port 8000
```

The `SGLANG_EXTERNAL_MODEL_PACKAGE` should be the parent folder name containing your model-related code. In this example, it should be `custom_llm`.

### Example: Multimodal Model

If you are working with multimodal models, setting `SGLANG_EXTERNAL_MODEL_PACKAGE` alone is not enough. SGLang also needs to recognize your architecture as multimodal to enable the image/video processing pipelines, and it needs a custom processor.

You can handle this by setting two additional environment variables:

* `SGLANG_EXTERNAL_MM_MODEL_ARCH`: Adds your architecture name to SGLang's internal list of multimodal models.
* `SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE`: Tells SGLang where to find your custom processor class.

For example, let's build a custom model based on Qwen2-VL-Instruct that takes the square root of the logits.

Create the project:

```
sglang_custom_project_vl/
|----setup.py
|----custom_vlm/
     |----__init__.py
     |----qwenvl_wrapper.py
```

Write `setup.py`:

```python Example theme={null}
# sglang_custom_project_vl/setup.py

from setuptools import setup, find_packages
setup(
    name="sglang-custom-plugins-vl",
    version="0.1",
    packages=find_packages(),
)
```

Write the model in `qwenvl_wrapper.py`:

```python Example theme={null}
# sglang_custom_project_vl/custom_vlm/qwenvl_wrapper.py
import torch
from sglang.srt.models.qwen2_vl import Qwen2VLForConditionalGeneration
from sglang.srt.multimodal.processors.qwen_vl import QwenVLImageProcessor

class CustomQwen2VL(Qwen2VLForConditionalGeneration):
    def forward(self, input_ids, positions, forward_batch,
                input_embeds=None, get_embedding=False):
        res = super().forward(
            input_ids, positions, forward_batch,
            input_embeds=input_embeds, get_embedding=get_embedding
        )
        if not get_embedding:
            orig = res.next_token_logits
            res.next_token_logits = torch.where(orig > 0, orig.sqrt(), orig)
        return res

class CustomQwen2VLProcessor(QwenVLImageProcessor):
    models = [CustomQwen2VL]

    def __init__(self, hf_config, server_args, _processor, *args, **kwargs):
        super().__init__(hf_config, server_args, _processor, *args, **kwargs)

EntryClass = CustomQwen2VL
```

**Note:** you don't need a separate `EntryClass` for the custom processor as long as you associate the processor with the specific model class.

Install the package, update `config.json`, and launch:

```bash Command theme={null}
pip install -e .
```

```json Config theme={null}
{
  "architectures": ["CustomQwen2VL"],
  ...
}
```

```bash Command theme={null}
export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_vlm
export SGLANG_EXTERNAL_MM_MODEL_ARCH=CustomQwen2VL
export SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGE=custom_vlm

python -m sglang.launch_server \
    --model-path /path/to/Qwen2-VL-2B-Instruct \
    --port 8000 \
    --enable-multimodal
```

## Documentation

Add to table of supported models in [generative\_models.md](./generative_models) or [multimodal\_language\_models.md](./multimodal_language_models)

***

By following these guidelines, you can add support for new language models and multimodal large language models in
SGLang and ensure they are thoroughly tested and easily integrated into the system.


# Transformers Fallback in SGLang
Source: https://docs.sglang.io/docs/supported-models/transformers_fallback



`sglang` can fall back to using models that are available in `transformers`. This works for most decoder-style language models and support for vision-language models is coming soon!

## Example launch Command

By default, we will use sglang implementation if it is available. Otherwise, we will fall back to transformers one. However, you can switch the implementation by setting `--model-impl` to `transformers`.

```shell Launch Server theme={null}
python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.2-1B-Instruct \
  --host 0.0.0.0 \
  --port 30000 \
  --model-impl transformers
```

## Supported features

### Quantization

Transformers fall back has supported most of available quantization in SGLang (except GGUF). See [Quantization page](../advanced_features/quantization) for more information about supported quantization in SGLang.

### Remote code

This fallback also means that any model on the hub that can be used in `transformers` with `trust_remote_code=True` that correctly implements attention can be used in production!

A model just needs the following two things:

```python Example theme={null}
from transformers import PreTrainedModel
from torch import nn

class MyAttention(nn.Module):

  def forward(self, hidden_states, **kwargs): # <- kwargs are required

    ...
    attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
    attn_output, attn_weights = attention_interface(
      self,
      query_states,
      key_states,
      value_states,
      **kwargs,
    )
    ...

class MyModel(PreTrainedModel):
  _supports_attention_backend = True
```

Here is what happens in the background:

1. The config is loaded
2. `MyModel` python class is loaded from the `auto_map`, and we check that the model `_supports_attention_backend`.
3. The `TransformersModel` backend is used. See `/srt/models/transformers`, which leverages `self.config._attn_implementation = "sglang"`, thus the need to use `ALL_ATTENTION_FUNCTIONS`.

That's it!


# Welcome to SGLang
Source: https://docs.sglang.io/index

High-performance serving framework for large language and multimodal models.

[Star](https://github.com/sgl-project/sglang)

[Fork](https://github.com/sgl-project/sglang/fork)

> **Performance & Runtime**
>
> Designed for low-latency, high-throughput inference with RadixAttention, prefix caching, and multi-GPU parallelism.

> **Models & Ecosystem**
>
> Broad support for Llama, Qwen, DeepSeek, and more. Compatible with Hugging
> Face and OpenAI APIs.

> **Extensive Hardware Support**
>
> Native support across [Hardware Platforms](./docs/hardware-platforms/overview)
> including NVIDIA, AMD, Intel Xeon, Google TPU, Ascend NPU, and Moore Threads MUSA accelerators.

> **Community & Training**
>
> Open-source with widespread adoption, powering 400k+ GPUs and integrated with major RL frameworks.

SGLang powers large-scale production deployments, generating trillions of tokens each day across more than 400,000 GPUs worldwide. It is hosted under the non-profit open-source organization [LMSYS](https://lmsys.org/about/).

***

## Get Started

SGLang is an inference framework meant for production level serving.
It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters.

> **[Quickstart](./docs/get-started/quickstart)**
>
> Start a model server and send a request to verify it is running.

> **[Examples](/docs/get-started/examples)**
>
> Stream replies, request tool calls, and process multiple prompts from Python.

For environment setup and other installation methods, see the [Installation guide](/docs/get-started/install).

## News and latest blogs

![SGLang Adds Day-0 Support for NVIDIA Nemotron 3.5 Super VL](https://lmsys.org/images/blog/nemotron-3-5-super-vl/image1.png)

SGLang Adds Day-0 Support for NVIDIA Nemotron 3.5 Super VL

October 15, 2026

[Open](https://lmsys.org/blog/2026-10-15-nvidia-nemotron-3-5-super-vl/)

![SGLang and Miles Add Day-0 Support for Muse Spark Open](https://lmsys.org/images/blog/meta-muse-spark-open/muse-spark-architecture.svg)

SGLang and Miles Add Day-0 Support for Muse Spark Open

October 15, 2026

[Open](https://lmsys.org/blog/2026-10-15-meta-muse-spark-open/)

![SGLang and Miles on NVIDIA Vera Rubin](https://lmsys.org/images/blog/vera-rubin/mla-pipeline.png)

SGLang and Miles on NVIDIA Vera Rubin

October 9, 2026

[Open](https://lmsys.org/blog/2026-10-09-vera-rubin/)

![Scaling JEV-like Decision Models with SGLang](https://lmsys.org/images/blog/sglang-decision-models/title-card.png)

Scaling JEV-like Decision Models with SGLang

September 25, 2026

[Open](https://lmsys.org/blog/2026-09-25-sglang-decision-models/)

![Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache](https://lmsys.org/images/blog/nvfp4-kv-cache/kv-cache-layout.svg)

Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache

September 16, 2026

[Open](https://lmsys.org/blog/2026-09-16-nvfp4-kv-cache/)

![SGLang and Miles Add Day-0 Support for DeepSeek-V4.1](https://lmsys.org/images/blog/deepseek-v41/preview.png)

SGLang and Miles Add Day-0 Support for DeepSeek-V4.1

September 10, 2026

[Open](https://lmsys.org/blog/2026-09-10-deepseek-v41/)

***

## Community and resources

* [Slack](https://slack.sglang.io/): Technical questions and development discussions.
* Events: [Meetups, workshops, and office hours](https://www.sglang.io/events) | [Developer meetings](https://meet.sglang.io/).
* Updates: [X](https://x.com/lmsysorg) | [LinkedIn](https://www.linkedin.com/company/sgl-project/) | [LMSYS Blog](https://lmsys.org/blog/).
* Contribute: [Contributor guide](/docs/developer_guide/contribution_guide) | [Roadmap](https://roadmap.sglang.io/).
* [Release notes](https://github.com/sgl-project/sglang/releases): Changes in each release.