Skip to main content

​
Deployment

Command
The deployment panel can render a complete docker run command for the selected hardware and options. See Install SGLang with Docker for host setup. The CUDA tag does not run on MI355X; use the ROCm image there; Use Xeon image for Xeon CPU and for more SGLang CPU installation approaches, see the CPU version installation guide.
Choose your Hardware Platform, Strategy, and Quantization below. The default is B300 / Low Latency / NVFP4. Low Latency keeps the existing deployment settings; High Throughput removes all speculative decoding parameters. Use the Playground to tune parallelism, caching, speculative decoding, multimodal transport, and prefill/decode disaggregation.

​
Playground

The Playground is where you experiment with SGLang features beyond the verified matrix. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing.

​
Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in SemiAnalysis InferenceX and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.
  • Router. Each configuration was submitted behind one front door: Dynamo (dynamo.frontend with dynamo.sglang workers) or SGLang (sglang.launch_server, behind sglang_router when there are several workers or DP-attention ranks).
  • KV offload. Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes --enable-unified-cache-external-linker.
  • Container image. Run the commands inside the image listed for each configuration; flag names follow that image.

​
GB300 deployments measured with Artificial Analysis AgentPerf

Five aggregated Dynamo + SGLang deployments on GB300, measured with the Artificial Analysis AgentPerf coding-agent workload. Together they span the trade-off from the lowest per-user latency (concurrency 1) to the highest throughput per GPU (concurrency 160), ordered from low latency to high throughput. Each one starts from the Artificial Analysis SGLang submission for this model and changes only flags, environment variables and parallelism.
  • Container image. lmsysorg/sglang:dev-muse-spark-open@sha256:8cfc5a72cb296771f5f8a4a4688cb6d05685610a830cbb616fa4ab0eb7bb2b0a with the Dynamo wheels added: pip install --no-deps ai-dynamo==1.6.0.dev20261019 ai-dynamo-runtime==1.6.0.dev20261019.
  • Router. One Dynamo frontend with the KV-aware router. Session affinity keeps requests that carry the same X-Dynamo-Session-ID header on one worker; the AgentPerf client sends one ID per agent session.
  • Checkpoints. meta-models/Muse-Spark-Open-816B-A42B-NVFP4 with the DFlash2 draft RadixArk/Muse-Spark-Open-DFlash2 (8 draft tokens, FP8 KV cache).
  • HiCache. --hicache-size is the host tier in GB per rank; scale it to your host memory.
  • Two-node deployments. The TP8·EP8 worker spans two GB300 trays (4 GPUs each) over multi-node NVLink: run the worker command on both nodes, with NODE_RANK 0 on the first.

​
MI355X deployment measured with SemiAnalysis InferenceX

TP8·EP8 with DFlash V2 on 8× MI355X, measured in SemiAnalysis InferenceX at concurrency 1–64.
  • Container image. lmsysorg/sglang-rocm:muse-spark-open-1020.
  • HiCache. At concurrency 16 and above, add the HiCache flags below. --hicache-size is the host tier in GB per rank, so 253 GB needs about 2 TB of host memory across the eight ranks.
Command
HiCache (concurrency 16 and above)

​
1. Model introduction

This cookbook provides deployment recipes for the Muse Spark Open 816B-A42B checkpoints:
  • NVFP4: meta-models/Muse-Spark-Open-816B-A42B-NVFP4 (NVIDIA default).
  • MXFP4: amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 (MI355X default; Quark-quantized).
  • NVFP4 on MI355X (direct load): the same meta-models/Muse-Spark-Open-816B-A42B-NVFP4 checkpoint runs on MI355X with --quantization quark_mxfp4; the weights are requantized to MXFP4 while loading, so no separate export is needed. Select MI355X / NVFP4 in the panel for this command.
  • INT4 intel-ai/Muse-Spark-Open-816B-A42B-INT4 (Xeon default; AutoRound-quantized).
  • BF16: meta-models/Muse-Spark-Open-816B-A42B.

​
2. Configuration tips

​
H200 configuration

NVFP4 runs on one 8x H200 node with Marlin W4A16 experts (--moe-runner-backend marlin), since Hopper has no TRT-LLM FP4 MoE. The BF16 checkpoint does not fit on one node, so it runs TP16/EP16 across two nodes; replace <node-rank> and <node0-ip> for each node. Prefill sequence parallel and MegaMoE need SM100 kernels and are unavailable on H200.

​
Xeon CPU service configuration

  • Xeon serves the INT4 checkpoint (quantized by Intel AutoRound, HF id: intel-ai/Muse-Spark-Open-816B-A42B-INT4) on the SGLang CPU engine. A typical Xeon 6 P-core platform (e.g., Xeon 6980P) is recommended.
  • Set --page-size 1 for the intel_amx attention backend (the CPU default attention backend).
  • We recommend setting the TP size equal to the NUMA count. See the Notes under Launch of the Serving Engine in the SGLang CPU server document for how to configure the arguments, especially TP and NUMA binding.

​
3. Advanced Usage

​
3.1 Reasoning

On the server, enable --reasoning-parser muse in your sglang serve command. The Deployment recipes above already include this flag; in the Playground, keep Reasoning Parser enabled under Parsers. The parser separates the model’s reasoning into message.reasoning_content and its final answer into message.content. On the client, set reasoning_effort for each chat completion request. Muse Spark Open supports four levels: low, medium, high, and xhigh. If you omit the parameter, the checkpoint’s chat template defaults to xhigh. Use low or medium for faster, shorter responses, and high or xhigh for complex problem solving, coding, and agentic tasks. Other values, including minimal, are not supported. See the model card for the supported settings. The example below connects to the server on port 30000 and discovers its model ID, so it works with whichever checkpoint you selected in Deployment. Change base_url if the server runs on another host or port. The current checkpoint’s chat template does not require chat_template_kwargs={"thinking": True}; set the reasoning level with reasoning_effort.
Example
Output

​
3.2 Tool calling

On the server, enable --tool-call-parser muse alongside --reasoning-parser muse in your sglang serve command. The Deployment recipes above already include both flags; in the Playground, keep Tool Call Parser and Reasoning Parser enabled under Parsers. SGLang converts Muse Spark Open’s native tool-call format into OpenAI-compatible message.tool_calls, while keeping reasoning in message.reasoning_content. On the client, pass function definitions in tools, with each function’s arguments described by a JSON Schema. The checkpoint’s chat template formats these definitions for the model. With tool_choice="auto", the model decides whether to call a tool. The example below requests a weather lookup and prints each returned function name and its JSON arguments.
Example
Output
This example inspects the requested tool calls; it does not fetch weather data. Your application executes the corresponding function. To continue the conversation, append the assistant message containing tool_calls, then a role="tool" message for each result with the matching tool_call_id, and send the updated messages with the same tool definitions.

​
3.3 Encode Prefill Decode Disaggregation

Encoder–Prefill–Decode (EPD) disaggregation separates media encoding, prompt processing, and token generation, so encoding-heavy image/video requests can be scheduled separately from language inference. The encoder sends features through zmq_to_tokenizer; prefill transfers KV state to decode through Mooncake, with a router exposing one client endpoint. The examples below use one 8× B300 NVLink host: encoder TP2 shares GPUs 0–1 with prefill TP4/EP4 on GPUs 0–3; decode TP4/EP4 uses GPUs 4–7. Run each command in a separate shell using the NVIDIA image from Deployment, with a shared host network. Start encoder, prefill, decode, then router, waiting for each to become ready. Prefill (30001), decode (30002), and router (8000) match the Playground ports; the encoder uses 30010.
These are simplified examples derived from PR #265. The source run checked transport with simulated outputs; this ordinary-serving configuration still needs runtime and accuracy validation. Leave GPU memory headroom for the encoder sharing the prefill GPUs.
Command
Command
Command
Command
The language workers use distinct encoder-bootstrap ports (8997 / 8999) on this shared host; the router’s 8998 matches the prefill KV bootstrap port. Send client requests to http://localhost:8000/v1. See the EPD guide for the architecture and additional deployment options.

​
3.4 Sequence Parallel for Prefill

Sequence parallelism (SP) splits activations along the token dimension across TP ranks. Each rank processes its own token shard for normalization, pooling, and unpooling, reducing duplicated computation and activation storage during long-prompt prefill. NVFP4 MegaMoE connects these shards to expert parallelism through integrated all-to-all dispatch and combine: tokens reach their selected experts, and weighted expert outputs return to the original token ranks. For a Blackwell NVFP4 configuration (B300, B200, or GB300), use the Playground to enable Prefill Sequence Parallel and Prefill MegaMoE together. This adds --enable-layernorm-sp --enable-megamoe-for-layernorm-sp to the launch command. Keep Attention TP and MoE EP equal; MegaMoE requires SP to remain on. These prefill options are unavailable on H200 and are omitted for a PD decode worker.

​
3.5 Speculative Decoding with DFlash v2 and Block Verification

DFlash V2 uses the target model’s hidden states to propose a block of future tokens in one draft-model forward pass; Muse Spark Open then verifies the block in a single pass, advancing by multiple tokens when proposals are accepted. V2 mixes information across neighboring draft positions and selects candidates with their dependencies in mind. Optional block verification evaluates draft prefixes jointly instead of token by token, potentially accepting longer prefixes while preserving the target sampling distribution. In the Playground, select DFlash V2 under Speculative Decoding to add --speculative-algorithm DFLASH and --speculative-draft-model-path RadixArk/Muse-Spark-Open-DFlash2. Then adjust:
  • Draft attention backend: fa4 for NVIDIA or triton for MI355X, inherited from Deployment and set with --speculative-draft-attention-backend.
  • Num draft tokens: 1–8, default 8, set with --speculative-num-draft-tokens.
  • Block Verification: turn it on to add --speculative-use-block-verification; it is off by default.
The Deployment Low Latency strategy enables DFlash V2 on supported GPUs; High Throughput leaves speculative decoding off. The Playground inherits that choice and lets you override it.

​
3.6 MegaKernel with Helion (Experimental)

We support persistent Helion megakernels for DFlash V2 target verification. They fuse attention, projection and normalization boundaries, routing, MoE, and the layer tail, with a separate fused LM head, reducing kernel launches and communication overhead. TP4/EP1 shards each expert across four GPUs, avoiding the expert-load imbalance of EP4 at batch size 1. With BS=1 + DFlash V2, the reported decode performance is roughly 10% faster than the TP4/EP4 baseline without megakernels on 4×GB300; gains depend on the workload. Use this experimental recipe only on a PD-disaggregated decode worker, with four GB300 (SM 10.3) GPUs, NVFP4 weights, FP8 E4M3 KV cache, and helion==1.5.0. Keep prefill on the regular path: EP1 prefill is slower than the TP4/EP4 baseline. This specialized TP4/EP1 command is separate from the Playground’s equal-TP/EP configuration.
Install the dependency in the serving environment, then launch the decode worker:
Command
Pair this worker with a compatible prefill worker using the same target checkpoint, TP4, --kv-cache-dtype fp8_e4m3, --page-size 128, and --served-model-name external, with bootstrap port 8998. Use the Playground PD router command with the decode URL pointing to this worker on port 30002; send client requests to the router on port 8000.