Attention Backends - SGLang Documentation

This document describes the attention backends available in sglang diffusion (sglang.multimodal_gen) and how to select them.

Overview

Attention backends are defined by AttentionBackendEnum (sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum) and selected via the CLI flag --attention-backend. Backend selection is performed by the shared attention layers (e.g. LocalAttention / USPAttention / UlyssesAttention in sglang.multimodal_gen.runtime.layers.attention.layer). --attention-backend is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use --component-attention-backends when an auxiliary component must use a specific backend; incompatible overrides, including an override for a component that constructs no SGLang attention layer, fail unless a sparse backend is being replaced for cross-attention. When using the diffusers backend, --attention-backend is passed through to diffusers’ set_attention_backend (e.g., flash, _flash_3_hub, sage, xformers, native).

CUDA: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA. On SM100/B200, dense non-causal fp16/bf16 native attention prefers cuDNN SDPA and falls back to FA4 if cuDNN has no compatible kernel.
ROCm: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
Intel XPU: uses XPU Flash Attention backend (fp16/bf16, head sizes 64/96/128/192/256); otherwise falls back to PyTorch SDPA.
MUSA: uses FlashAttention when available; also supports Sage Attention when installed; otherwise falls back to PyTorch SDPA.
MPS: always uses PyTorch SDPA.
NPU: for ring attention uses FA otherwise uses PyTorch SDPA.

Backend options

For SGLang-native pipelines, the CLI accepts the lowercase names of AttentionBackendEnum. The table below lists the backends implemented by the built-in platforms. fa3/fa4 are accepted as aliases for fa.

CLI value	Enum value	Notes
`fa` / `fa3` / `fa4`	`FA`	FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.post_init`).
`torch_sdpa`	`TORCH_SDPA`	PyTorch `scaled_dot_product_attention`.
`sliding_tile_attn`	`SLIDING_TILE_ATTN`	Sliding Tile Attention (STA). Requires `st_attn`. Configure via `—attention-backend-config`.
`sage_attn`	`SAGE_ATTN`	Requires `sageattention`. On Hopper (SM90), PyPI `sageattention==2.2.0` is unsupported because it lacks the upstream SM90 binding fix. Install `pip install —force-reinstall git+https://github.com/thu-ml/SageAttention.git@d9704247a5139ab4c03bf7fc6b35cc0e2cbb5ea4 —no-build-isolation`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120; see upstream `setup.py`.
`sage_attn_3`	`SAGE_ATTN_3`	Requires SageAttention3 installed per upstream instructions.
`sparge_attn`	`SPARGE_ATTN`	Training-free sparse SageAttention2. CUDA SM80/86/87/89/90, FP16/BF16, head dim 64/128, and square self-attention with sequence length at least 128. Other attention shapes use dense SDPA. Install `pip install git+https://github.com/thu-ml/SpargeAttn.git —no-build-isolation`; configure retained block ratio with `topk`.
`sol_attn`	`SOL_ATTN`	Requires the upstream `sol-attn` package. Install with `pip install git+https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends`. BF16, head dim 128. Configure via `—attention-backend-config`.
`video_sparse_attn`	`VIDEO_SPARSE_ATTN`	Requires `vsa`. Configure `sparsity` via `—attention-backend-config`.
`subblock_sparse_attn`	`SUBBLOCK_SPARSE_ATTN`	Training-free SubBlock sparsity for MiniMax-H3 on CUDA SM90, SM100, and SM120. The default compute mode is BF16; `compute_mode=sage_fp8` is an approximate SM90-only path that requires `SpargeAttn`. See the MiniMax-H3 recipe.
`video_sparse_attn_h3`	`VIDEO_SPARSE_ATTN_H3`	Video Sparse Attention for MiniMax-H3 / FastH3 (VSA-H3). In-tree Triton block-sparse kernel (SM90 / SM100 / SM103); no external package. Configure via `—attention-backend-config`.
`vmoba_attn`	`VMOBA_ATTN`	Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `—attention-backend-config`.
`aiter`	`AITER`	Requires `aiter`.
`aiter_sage`	`AITER_SAGE`	Requires `aiter`.
`sla_attn`	`SLA_ATTN`	Sparse Linear Attention. Requires `SpargeAttn`. Install with `pip install git+https://github.com/thu-ml/SpargeAttn.git —no-build-isolation`.
`sage_sla_attn`	`SAGE_SLA_ATTN`	SageAttention + Sparse Linear Attention. Requires `SpargeAttn` (same install as SLA).
`sparse_video_gen_2_attn`	`SPARSE_VIDEO_GEN_2_ATTN`	Requires `svg`. See installation instructions at https://github.com/svg-project/Sparse-VideoGen.
`laser_attn`	`LASER_ATTN`	Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU.
`block_sparse_attn`	`BLOCK_SPARSE_ATTN`	Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU.
`rain_fusion_attn`	`RAIN_FUSION_ATTN`	Requires `attentions` which can be installed with `sgl_kernel_npu`; available only for NPU.

Selection priority

The selection order in runtime/layers/attention/selector.py is:

global_force_attn_backend(...) / global_force_attn_backend_context_manager(...)
Component override from --component-attention-backends while that component is being constructed
CLI --attention-backend (ServerArgs.attention_backend)
Layer or component default, when declared
Auto selection (platform capability, dtype, and installed packages)

An explicit global backend mismatch fails for DiT self-attention. Auxiliary components may fall back to their declared default or another compatible backend. Sparse backends selected for self-attention similarly fall back to a compatible dense backend for cross-attention. Explicit component overrides are otherwise strict. Model and layer backend sets guide automatic selection; they are not positive allowlists for explicit requests. A backend selected with --attention-backend or --component-attention-backends can therefore be used without first adding it to every model’s set. It still fails closed when the platform cannot resolve it or when it lacks a required operation such as dense cross-attention, packed variable-length attention, or ring-attention LSE. Model paths that require one backend for correctness declare it as required; those layers keep that backend even when the surrounding component is overridden.

Configuration

Some backends require additional configuration. You can pass these parameters via --attention-backend-config. This argument accepts:

A path to a JSON or YAML configuration file.
A JSON string (e.g., '{"sparsity": 0.5}').
Key-value pairs (e.g., "sparsity=0.5,enable_x=true").

Supported Configuration Parameters

Sliding Tile Attention (sliding_tile_attn)

Parameter	Type	Description	Default
`mask_strategy_file_path`	`str`	Required. Path to the mask strategy JSON file.	-
`sta_mode`	`str`	Mode of STA.	`STA_inference`
`skip_time_steps`	`int`	Number of steps to use full attention before switching to sparse attention.	`15`

Video Sparse Attention (video_sparse_attn)

Parameter	Type	Description	Default
`sparsity`	`float`	Validation sparsity (0.0 - 1.0).	`0.0`

SpargeAttention (sparge_attn)

Parameter	Type	Description	Default
`topk`	`float`	Fraction of predicted attention blocks retained per head. Higher values preserve more attention work and generally improve quality. Must be in `(0, 1]`.	`0.5`

SpargeAttention is approximate even when topk=1: the recommended upstream kernel quantizes attention through SageAttention2. Validate output quality and end-to-end latency on the target model and resolution before deployment. Video Sparse Attention for H3 (video_sparse_attn_h3)

Parameter	Type	Description	Default
`VSA_sparsity`	`float`	Fraction of video tiles excluded from the top-k selection (0.0 - 1.0). `0.9` is the FastH3 trained policy.	`0.9`
`vsa_mode`	`str`	`exempt`: non-video keys (text/audio prefix tiles) are always selected. `compete`: they compete with video tiles in the top-k.	`exempt`
`vsa_dense_first_n_steps`	`int`	Use dense attention for the first N denoising steps.	`0`
`vsa_dense_layers`	`list[int]`	Layer indices kept dense, e.g. `[0, 1]`.	`[]`
`vsa_tile_size`	`int`	Kernel tile size. Only `64` (the trained (4, 4, 4) geometry) is accepted.	`64`

VSA-H3 constraints:

Only the DiT runs sparse; the token refiner, text encoder, and VAEs keep their dense defaults. An explicit --component-attention-backends text_encoder=fa is rejected because the H3 text encoder has SDPA-only layers.
Uses the checkpoint’s trained to_gate_compress compression branch. Base MiniMax-H3 weights load zero gates and run pure sparse.
Ulysses sequence parallelism is supported; --ring-degree greater than 1, torch.compile, and breakable CUDA graph execution are rejected.

V-MoBA (vmoba_attn)

Parameter	Type	Description	Default
`temporal_chunk_size`	`int`	Chunk size for temporal dimension.	-
`temporal_topk`	`int`	Top-K tokens to select in temporal dimension.	-
`spatial_chunk_size`	`list[int]`	Chunk size for spatial dimension (H, W).	-
`spatial_topk`	`int`	Top-K tokens to select in spatial dimension.	-
`st_chunk_size`	`list[int]`	Chunk size for spatiotemporal dimension (T, H, W).	-
`st_topk`	`int`	Top-K tokens to select in spatiotemporal dimension.	-
`moba_select_mode`	`str`	Selection mode (e.g., `threshold`).	`threshold`
`moba_threshold`	`float`	Threshold value for selection.	`0.25`
`moba_threshold_type`	`str`	Type of thresholding (e.g., `query_head`).	`query_head`
`first_full_step`	`int`	Number of initial steps to use full attention.	`12`
`first_full_layer`	`int`	Number of initial layers to use full attention.	`0`
`temporal_layer`	`int`	Number of temporal layers.	`1`
`spatial_layer`	`int`	Number of spatial layers.	`1`
`st_layer`	`int`	Number of spatiotemporal layers.	`1`

Block Sparse Attention (block_sparse_attn)

Parameter	Type	Description	Default
`skip_first_steps`	`int`	Number of steps to use laser attention before switching to sparse attention.	`10`
`sparsity`	`float`	The sparsity coefficient must be in the range (0, 1).	`0.2`

Sol-Attn (sol_attn)

Parameter	Type	Description	Default
`tau`	`float`	Routing threshold scale. Higher values select fewer exact KV blocks.	`1.0`
`thresh_type`	`str`	Threshold mode: `diag` or `exact`.	`diag`
`sink_tokens`	`int`	Exact KV sink length for prefix tokens such as text/audio rows.	`0`
`sink_start`	`int`	Start index of the exact KV sink range.	`0`
`dense_steps`	`int`	Use dense attention for the first N denoising steps.	`10`
`dense_layers`	`str`	Layer indices kept dense, e.g. `0,1` or `0-2`.	`0,1`
`dense_backend`	`str`	Backend used for the dense prefix: `fa` (default) or `sage_attn`. `sage_attn` is approximate.	`fa`
`kv_splits`	`int \| str`	KV split factor passed to the Sol-Attn kernel. Use `auto` on long sequences.	`auto`

Platform support matrix

Backend	CUDA	ROCm	XPU	MUSA	MPS	NPU	Notes
`fa`	Yes	Yes	✅	✅	❌	✅	CUDA requires SM80+ and fp16/bf16. XPU uses its own flash attention backend. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. No extra installations are required for NPU
`torch_sdpa`	Yes	Yes	Yes	Yes	✅	✅	Most compatible option across platforms.
`sliding_tile_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `st_attn`. Configure via `—attention-backend-config`.
`sage_attn`	Yes	No	No	Yes	❌	❌	Optional dependency on CUDA and MUSA. On Hopper, also falls back to FlashAttention when the installed package lacks the SM90 binding fix.
`sage_attn_3`	Yes	No	No	No	❌	❌	CUDA-only (optional dependency).
`sparge_attn`	Yes	No	No	No	❌	❌	CUDA SM80/86/87/89/90 only. Requires `SpargeAttn`; head dim 64/128 and square self-attention with sequence length at least 128.
`sol_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `sol-attn`. Install with `pip install git+https://github.com/NVlabs/Sana.git@sol-engine#subdirectory=techniques/sparse_backends`. Configure via `—attention-backend-config`.
`video_sparse_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `vsa`. Configure `sparsity` via `—attention-backend-config`.
`video_sparse_attn_h3`	Yes	No	No	No	❌	❌	CUDA-only (SM90 / SM100 / SM103). In-tree Triton kernel, no external dependency. Configure via `—attention-backend-config`.
`sla_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `SpargeAttn`.
`sage_sla_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `SpargeAttn`.
`vmoba_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `—attention-backend-config`.
`aiter`	❌	✅	No	No	❌	❌	Requires `aiter`.
`aiter_sage`	❌	✅	No	No	❌	❌	Requires `aiter`.
`sparse_video_gen_2_attn`	Yes	No	No	No	❌	❌	CUDA-only. Requires `svg`.
`laser_attn`	❌	❌	❌	❌	❌	✅	NPU-only. Requires `attentions` from `sgl_kernel_npu`. Uses SDPA if seqlen less than 2048.
`block_sparse_attn`	❌	❌	❌	❌	❌	✅	NPU-only. Requires `attentions` from `sgl_kernel_npu`. Configuration via `—attention-backend-config`.
`rain_fusion_attn`	❌	❌	❌	❌	❌	✅	NPU-only. Requires `attentions` from `sgl_kernel_npu` Configuration via `—attention-backend-config`.

Usage

Select a backend via CLI

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend fa

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend torch_sdpa

Override one component

Use component overrides when a specific module needs different attention semantics from the main transformer:

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend fa \
  --component-attention-backends text_encoder=torch_sdpa

Component keys match pipeline module names from model_index.json, such as text_encoder, text_encoder_2, transformer, transformer_2, or connectors. Use this override when the fallback must be pinned: unlike the global backend, an incompatible component override raises an error instead of selecting another backend. The one role-based exception is a sparse self-attention backend, which uses a compatible dense backend for cross-attention layers in the same component. The component must construct SGLang-selectable attention or explicitly defer selection until first use; components with fixed attention reject the override. Per-component overrides apply only to native pipelines. The Diffusers backend accepts the global --attention-backend passthrough instead.

Per-request override (denoise loop)

A single server can serve exact and approximate attention side by side: requests may switch the DiT denoise attention backend via the attention_backend_override sampling param. Valid values are the exact/drop-in dense kernels — fa, torch_sdpa, sage_attn, sage_attn_3. The field participates in the dynamic-batch signature, so requests with different backends never share a batch.

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend-override sage_attn

client.images.generate(
    model="<MODEL_PATH_OR_ID>",
    prompt="...",
    extra_body={"attention_backend_override": "sage_attn"},
)

Incompatible server settings reject the request (with a server log) instead of silently falling back: breakable CUDA graphs and torch.compile bake the attention kernel into a captured/traced graph; sparse server backends (sliding_tile_attn, video_sparse_attn, …) cannot be mixed with per-request dense switching; under ring parallelism the target must be ring-capable. Note sage_attn / sage_attn_3 are lossy (quantized attention) — validate quality on your workload.

Request-scoped Skip Softmax

Skip Softmax (BLASST) keeps the QK matmul, but skips the exponential, softmax-state update, V load, and PV matmul for attention tiles whose estimated softmax mass is below a threshold. It is an explicit lossy optimization backed by the FlashInfer kernels shipped with SGLang’s pinned dependencies. See the BLASST paper and NVIDIA’s video-generation study for the algorithm and measured quality/performance trade-offs. Pass skip_softmax_params in one image or video request. The runtime routes compatible self-attention layers through the FA/FlashInfer path for that batch; cross-attention keeps its normal backend, and the next request restores the server default.

client.videos.create(
    model="<MODEL_PATH_OR_ID>",
    prompt="...",
    extra_body={
        "skip_softmax_params": {
            "threshold_scale_factor": 500.0,
            "start_step": 14,
        }
    },
)

threshold_scale_factor is required and must be positive. The kernel uses threshold_scale_factor / context_length; larger values skip more work and usually increase quality loss.
start_step is the zero-based denoising step at which sparse execution starts. It defaults to 0; keeping early high-noise steps dense is generally safer.

There is intentionally no default threshold. Calibrate the threshold and start step against output-quality metrics for each model, resolution, step count, and deployment backend; values tuned for one workload are not portable. Current support is FP16/BF16, head dimension 128 or 256, and unmasked self-attention on Hopper SM90 and Blackwell SM100/SM103/SM107. Ulysses sequence parallelism is supported because the kernel runs after its all-to-all. Ring Attention, attention masks, torch.compile, and breakable CUDA graphs reject the request instead of silently running dense attention. Models that merely offer an FA backend do not automatically qualify: the runtime still checks the GPU, dtype, head dimension, attention role, and execution mode.

Using SpargeAttention

Install the optional CUDA extension, then select the backend explicitly:

pip install ninja
pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation

sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend sparge_attn \
  --attention-backend-config topk=0.5

sparge_attn applies to compatible self-attention layers. Cross-attention uses the existing dense fallback; short self-attention and asymmetric Q/KV also use dense SDPA. LTX-2, LTX-2.3, and LTX-2.5 have compatible video (128) and audio (64) head dimensions, so their sufficiently long, unmasked self-attention uses SpargeAttention while prompt and audio-video cross-attention remain dense. Ulysses sequence parallelism is supported, but ring attention is not because the upstream kernel does not expose the softmax LSE needed for ring merging. K/V-gather SP therefore follows the normal sparse-backend rule and uses Ulysses instead.

Sage then Sol hybrid

sol_attn keeps the first dense_steps steps dense. Set dense_backend=sage_attn to run that prefix on SageAttention and the tail on Sol sparse attention. Keep the text encoder on torch_sdpa:

sglang generate \
  --model-path MiniMaxAI/MiniMax-H3 \
  --model-variant fl2va \
  --attention-backend sol_attn \
  --attention-backend-config dense_backend=sage_attn,dense_steps=10 \
  --component-attention-backends text_encoder=torch_sdpa,transformer=sol_attn \
  --prompt "A cat walking on a sunny beach, gentle waves." \
  --save-output

Both sage_attn and sol_attn are approximate. The default DiT backend remains fa.

Using Sliding Tile Attention (STA)

# Pass the mask strategy file path via config
sglang generate \
  --model-path <MODEL_PATH_OR_ID> \
  --prompt "..." \
  --attention-backend sliding_tile_attn \
  --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"

Notes for ROCm / MPS

ROCm: use --attention-backend torch_sdpa or fa depending on what is available in your environment.
MPS: the platform implementation always uses torch_sdpa.

​Overview

​Backend options

​Selection priority

​Configuration

​Supported Configuration Parameters

​Platform support matrix

​Usage

​Select a backend via CLI

​Override one component

​Per-request override (denoise loop)

​Request-scoped Skip Softmax

​Using SpargeAttention

​Sage then Sol hybrid

​Using Sliding Tile Attention (STA)

​Notes for ROCm / MPS