XPU

The document addresses how to set up the SGLang environment and run LLM inference on Intel GPU, see more context about Intel GPU support within PyTorch ecosystem.

Specifically, SGLang is optimized for Intel® Arc™ Pro B-Series Graphics and Intel® Arc™ B-Series Graphics.

Optimized Model List

A list of LLMs have been optimized on Intel GPU, and more are on the way:

Model Name	BF16
Llama-3.2-3B	meta-llama/Llama-3.2-3B-Instruct
Llama-3.1-8B	meta-llama/Llama-3.1-8B-Instruct
Qwen2.5-1.5B	Qwen/Qwen2.5-1.5B

Note: The model identifiers listed in the table above have been verified on Intel® Arc™ B580 Graphics.

Quantized MoE models are covered separately in MXFP4 MoE Quantization below.

Installation

Install From Source

Currently SGLang XPU only supports installation from source. Please refer to “Getting Started on Intel GPU” to install XPU dependency.

Command

# Below commands shows Ubuntu/Debian OS as example, other OS requires similar command change
# add sudo if required to install below packages
# Install curl and protobuf compiler

apt-get update && apt-get install -y curl protobuf-compiler

# Install Rust (non-interactive)
curl --proto '=https' --tlsv1.2 --retry 3 --retry-delay 2 -sSf https://sh.rustup.rs | sh -s -- -y
# Make cargo available in the current shell
source "$HOME/.cargo/env"

# Create and activate a conda environment
conda create -n sgl-xpu python=3.12 -y
conda activate sgl-xpu

# Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues.
pip3 install torch==2.13.0+xpu torchvision==0.28.0+xpu torchaudio==2.11.0+xpu --index-url https://download.pytorch.org/whl/xpu
pip3 install xgrammar==0.1.33 --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU

# Clone the SGLang code
git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout <YOUR-DESIRED-VERSION>

# Use dedicated toml file
cd python
cp pyproject_xpu.toml pyproject.toml
# Install SGLang dependent libs, and build SGLang main package
pip install --upgrade pip setuptools
pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu

Install Using Docker

The SGLang XPU Dockerfile is provided to facilitate the installation. Replace <secret> below with your HuggingFace access token.

Command

# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker

# Build the docker image
docker build -t sglang-xpu:latest -f xpu.Dockerfile .

# Initiate a docker container
docker run \
    -it \
    --privileged \
    --ipc=host \
    --network=host \
    --user root \
    --group-add $(getent group video | cut -d: -f3) \
    --device /dev/dri \
    -v /dev/dri/by-path:/dev/dri/by-path \
    -v /dev/shm:/dev/shm \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 30000:30000 \
    -e "HF_TOKEN=<secret>" \
    sglang-xpu:latest /bin/bash

Launch of the Serving Engine

Example command to launch SGLang serving:

sglang serve                         \
    --model-path <MODEL_ID_OR_PATH>  \
    --trust-remote-code              \
    --disable-overlap-schedule       \
    --device xpu                     \
    --host 0.0.0.0                   \
    --tp 2                           \   # using multi GPUs
    --attention-backend intel_xpu    \   # using intel optimized XPU attention backend
    --page-size                      \   # intel_xpu attention backend supports [32, 64, 128]

MXFP4 MoE Quantization

Native MXFP4 MoE checkpoints (OCP microscaling FP4: packed e2m1 weights plus per-32-element ue8m0 block scales) run on Intel GPUs through the sgl-kernel-xpu W4A16 grouped GEMM. The expert weights stay in the checkpoint’s packed layout end to end — there is no dequantization to BF16 — so GPT-OSS-20B loads in roughly 13 GB rather than the ~42 GB a BF16 upcast would need, which is what lets it fit on a single 24 GB card.

The mxfp4 method is registered automatically on --device xpu, which already requires sgl-kernel-xpu — no extra flag is needed:

sglang serve                        \
    --model-path openai/gpt-oss-20b \
    --device xpu                    \
    --attention-backend intel_xpu

Tested models:

Model	Notes
openai/gpt-oss-20b	32 experts, top-k 4, hidden 2880, intermediate 2880; clamped-swiglu activation (gemm1_alpha 1.702, limit 7.0) with expert biases

Requirements and limitations:

Item	Status
GPU architecture	Xe2 / BMG only (Intel® Arc™ B-Series and Arc™ Pro B-Series)
Quantized layers	Fused MoE experts only; attention, router, embeddings and LM head stay BF16 (per the checkpoint’s modules_to_not_convert)
Activations	BF16 (W4A16). Clamped swiglu (GPT-OSS) and plain SiLU are supported
Expert parallelism	Not yet validated on this path

Benchmarking with Requests

You can benchmark the performance via the bench_serving script. Run the command in another terminal.

python -m sglang.bench_serving   \
    --dataset-name random        \
    --random-input-len 1024      \
    --random-output-len 1024     \
    --num-prompts 1              \
    --request-rate inf           \
    --random-range-ratio 1.0

The detail explanations of the parameters can be looked up by the command:

python -m sglang.bench_serving -h

Additionally, the requests can be formed with OpenAI Completions API and sent via the command line (e.g. using curl) or via your own script.

XPU Graph [Experimental]

SGLang enables XPU graph capture to reduce per-step kernel-launch overhead.

Phase	Backend	Mechanism	Default
Decode	full	One torch.xpu.XPUGraph per batch size, captured on startup	Off (opt-in)
Prefill	tc_piecewise	torch.compile + XPU graph, one graph segment per token-length bucket	Off (opt-in)
Prefill	breakable	Segmented torch.xpu.XPUGraph capture/replay (no torch.compile); eager break points at attention / MoE boundaries	Off (opt-in)

Enable Decode Graph

Decode graph capture is opt-in on XPU. Enable it explicitly:

python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-decode full

Enable Prefill Graph

Prefill graph capture is opt-in on XPU and must be enabled explicitly. Two backends are available: tc_piecewise and breakable.

tc_piecewise

Uses torch.compile plus an XPU graph, one graph segment per token-length bucket:

python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill tc_piecewise

By default the prefill subgraphs are compiled with eager mode. Switch to inductor for higher-quality generated code at the cost of longer startup:

python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill tc_piecewise \
    --cuda-graph-tc-compiler inductor

breakable

Captures the transformer stack as segmented XPUGraphs with eager break points at attention / MoE boundaries, without torch.compile:

python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill breakable

You can also configure both phases together with a single --cuda-graph-config JSON argument:

python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-config '{"decode":{"backend":"full"},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}'

Enable torch.compile for Decode

--enable-torch-compile adds a torch.compile pass on top of the decode XPU graph: the model forward is compiled first, and the compiled forward is then captured as an XPUGraph. This can reduce per-kernel overhead further but increases startup time.

python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --enable-torch-compile

Note: --enable-torch-compile is mutually exclusive with the prefill tc_piecewise graph (the compatibility rules auto-disable it). Use them separately or lock the prefill backend explicitly via --cuda-graph-config if you need both.

Disable XPU Graph

Both phases are disabled by default. To explicitly disable them anyway:

# Disable decode graph (already off by default; explicit form)
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-decode=disabled

# Disable prefill graph (already off by default; explicit form)
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill=disabled

# Disable both phases
python -m sglang.launch_server --model-path <MODEL> --device xpu \
    --cuda-graph-backend-decode=disabled \
    --cuda-graph-backend-prefill=disabled

Customize Capture Buckets

By default, prefill capture sizes are derived from --chunked-prefill-size. To specify explicit token-length buckets:

python -m sglang.launch_server \
    --model-path <MODEL> --device xpu \
    --cuda-graph-backend-prefill tc_piecewise \
    --cuda-graph-bs-prefill 64 128 256 512

To specify explicit decode graph batch sizes:

python -m sglang.launch_server \
    --model-path <MODEL> --device xpu \
    --cuda-graph-bs-decode 1 2 4 8

Server Args

Argument	XPU allowed values	Default	Description
--cuda-graph-backend-decode	full, disabled	disabled	Backend for the decode phase. Only full is supported on XPU. Set to full to enable.
--cuda-graph-backend-prefill	tc_piecewise, breakable, disabled	disabled*	Backend for the prefill phase. Set to tc_piecewise or breakable explicitly to enable.
--cuda-graph-tc-compiler	eager, inductor	eager	Compiler for tc_piecewise prefill subgraphs. inductor produces more optimized code but has longer startup.
--cuda-graph-bs-prefill	list of ints	auto	Explicit token-length buckets to capture for prefill.
--cuda-graph-bs-decode	list of ints	auto	Explicit batch sizes to capture for decode.
--cuda-graph-config	JSON string	—	One-shot JSON config for both phases, e.g. '{"decode":{"backend":"full"},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}'. Overrides all per-phase flags.
--disable-decode-cuda-graph	—	False	Shorthand for --cuda-graph-backend-decode=disabled.
--disable-prefill-cuda-graph	—	False	Shorthand for --cuda-graph-backend-prefill=disabled.
--enable-torch-compile	—	False	Apply torch.compile on top of the decode XPU graph for further kernel optimization.
--torch-compile-max-bs	int	32	Maximum batch size compiled by torch.compile when --enable-torch-compile is set.

* Prefill graph is auto-disabled on XPU unless you lock the backend explicitly via --cuda-graph-backend-prefill or --cuda-graph-config.

Limitations

Feature	Status
Two-batch overlap (--enable-two-batch-overlap)	Not yet supported
Speculative decoding	Not yet implemented

Memory Saver (release/resume memory occupation) on Intel XPU [Experimental]

SGLang can temporarily release most of the GPU memory it holds — model weights and/or KV cache — and reclaim it later without restarting the process. This is the same release_memory_occupation / resume_memory_occupation feature available on CUDA, used for RL rollout/training hand-off and for freeing the device between inference bursts.

This is backed by the torch_memory_saver package — the same package used on CUDA — which gained an Intel XPU backend built natively on Level Zero (keeping virtual addresses fixed while releasing/re-committing physical pages via zeVirtualMemUnmap / zeVirtualMemMap).

Install torch_memory_saver. Unlike CUDA (prebuilt wheel), the XPU backend is built from source against your local oneAPI + torch+xpu runtime (the .so links libsycl.so.<N>, which must match the installed intel-sycl-rt). TMS_PLATFORM=xpu forces the XPU backend, and --no-build-isolation lets the build import your installed torch so it can match the libsycl major to it:

The published wheels are CUDA-only, so install from git and let it build. The ref below is the v0.0.10b2 release, pinned so the build is reproducible.

source /opt/intel/oneapi/setvars.sh
TMS_PLATFORM=xpu pip install --no-build-isolation \
    git+https://github.com/fzyzcjy/torch_memory_saver.git@a5c99f11b18ebb8e9fda71a68812e476ae49e417

Use it by launching with --enable-memory-saver (the XPU backend is selected automatically); optionally add --enable-weights-cpu-backup to keep weights in host RAM across a release:

python -m sglang.launch_server --model-path Qwen/Qwen3-0.6B \
    --trust-remote-code --device xpu --enable-memory-saver

# Release GPU memory while idle, then reclaim it (server must be idle).
curl -X POST http://127.0.0.1:30000/release_memory_occupation
curl -X POST http://127.0.0.1:30000/resume_memory_occupation

The Python engine API (engine.release_memory_occupation(tags=...) / engine.resume_memory_occupation(tags=...)) and the weights / kv_cache tags behave the same as on CUDA. Pauseable CUDA-graph capture is not used on XPU, so the cuda_graph tag is a no-op there.

Verifying memory was freed: neither torch.xpu.memory_allocated() nor torch.xpu.mem_get_info() drops when physical pages are released — the first is allocator accounting, and the second stays flat because torch keeps the freed block cached. Query actual device memory via sysman (ZES_ENABLE_SYSMAN=1) instead.

Prefill-Decode (P/D) Disaggregation on Intel XPU [Experimental]

SGLang supports prefill-decode disaggregation on Intel XPU using the NIXL KV-transfer backend.

Tested models:

Model	Notes
Qwen/Qwen3-0.6B	Used in integration tests; verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode)
Qwen/Qwen2.5-7B-Instruct	Verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode)

Prerequisites: pip install nixl sglang-router

Start the prefill server (GPU 0):

ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
    --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
    --disaggregation-mode prefill --disaggregation-transfer-backend nixl \
    --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000

Start the decode server (GPU 1):

ZE_AFFINITY_MASK=1 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
    --model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
    --disaggregation-mode decode --disaggregation-transfer-backend nixl \
    --disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30001

Start the router:

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://127.0.0.1:30000 \
    --decode  http://127.0.0.1:30001 \
    --host 0.0.0.0 --port 8000

Send a request:

curl http://127.0.0.1:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 32}'

Note: UCX_POSIX_USE_PROC_LINK=n is required on Intel XPU to avoid UCX shared-memory transport issues.