Parallelism Overview

SGLang Diffusion ships several parallelism strategies. Each one splits a different dimension of the DiT forward pass, which is exactly why they can be combined: the total GPU count is the product of the degrees,

num_gpus = cfg_parallel_degree × tp_size × sp_degree
sp_degree = ulysses_degree × ring_degree

This page is the map — what each axis does, which combinations are legal, and how to pick one. Per-axis depth lives in Sequence Parallelism, Encoder Parallelism (text/image encoders are a separate axis with their own knob), and the CLI reference.

The axes

Strategy	Splits	Communication	Flag
CFG parallel	guidance branches	one combine per denoise step	--cfg-parallel-size
Tensor parallel (TP)	weights and attention heads	all-reduce per transformer block	--tp-size
Ulysses SP	sequence outside attention ↔ heads inside it	two all-to-alls per attention	--ulysses-degree
Ring SP	sequence rows inside attention	neighbor-only K/V rotation, overlapped with compute	--ring-degree
K/V-gather CP	sequence rows inside attention	one K/V all-gather per attention	--kv-gather-degree
Data parallel	requests	none between replicas	--dp-size

Two strategies compose when they split different dimensions. TP and Ulysses both touch heads but compose serially — TP splits the projection weights, then Ulysses splits the activations of the TP-local heads. Ring and Ulysses compose because ring splits rows while Ulysses splits heads. Ring has no composition with a K/V-all-gather style of attention parallelism: both answer the same question (how a rank’s query rows see remote K/V), so they are alternatives for one slot, not complements.

What happens to the shapes

For tp_size = T, ulysses_degree = U, ring_degree = R, one attention runs:

[B, S/(U·R), H/T, D]                     sequence-sharded activations
   │  Ulysses input all-to-all (inside each Ulysses group)
   ▼
[B, S/R, H/(T·U), D]                     full sequence of this ring block, few heads
   │  ring attention (R−1 neighbor hops; Q never moves, K/V rotate)
   ▼
[B, S/R, H/(T·U), D]
   │  Ulysses output all-to-all (inverse)
   ▼
[B, S/(U·R), H/T, D]

The ring merge (online softmax) requires every rank in a ring group to hold the same heads over different rows; the group construction guarantees this. A K/V-gather (CP-style) variant fills the same slot differently: instead of R−1 overlapped hops it all-gathers K/V once and computes the local Q rows against the full sequence in one shot — fewer, larger transfers, paid for by holding the whole K/V per rank. Like ring it splits rows, so it adds no head constraint. When no SP degree is set explicitly, sp_degree=2 defaults to kv_gather_degree=2 — its measured-win zone — and higher degrees default to Ulysses. Ulysses groups are laid out on contiguous ranks and ring groups on strided ranks, so with a node-major rank mapping, Ulysses traffic stays on intra-node NVLink (all-to-all needs full-bisection bandwidth) while ring hops cross the slower interconnect where neighbor-only transfers overlap with compute. A mis-mapped layout stays numerically correct and silently loses the performance — worth checking when a sharded run is unexpectedly slow.

Constraints

• num_attention_heads % tp_size == 0 — TP splits heads at the projections.
• (num_attention_heads / tp_size) % ulysses_degree == 0 — Ulysses splits the TP-local heads. H % U == 0 alone is not sufficient: 56 heads pass with tp=2, ulysses=4 (28 % 4) and fail with tp=4, ulysses=4 (14 % 4).
• Ring adds no head constraint (it splits rows), but the sequence — including any model-specific packing alignment — must divide by ulysses × ring, since ring adds an outer row split on top of Ulysses’s inner one.
• Ring requires an attention backend that declares supports_ring_rotation() — the per-hop merge needs the kernel’s softmax LSE. fa and sage_attn declare it; the launcher auto-selects fa when unset.
• USPAttention’s masked/tail-padded text path and its replicated-prefix, -suffix, and -kv-prefix paths all support ring: the sharded K/V rotates through the ring while the tail-pad or replicated portion is attended locally once and combined into the ring result with the same online-softmax merge. This covers the joint text+image attention most models use (flux, flux_2, qwen_image, zimage, glm_image, ernie_image, and others).
• What still raises NotImplementedError under ring: USPAttention’s generic varlen path (multiple packed segments per row, as used by HunyuanVideo — no ring-aware rotation for it yet), and the legacy stacked-QKV UlyssesAttention layer, now down to one user (Wan’s VSA sparse-attention variant) that has no softmax LSE to merge and can’t gain ring support without a different kernel.
• The launcher validates num_gpus against the product of the degrees and fails fast on any mismatch.

The Ulysses transport

The all-to-alls normally run over NCCL. On exactly 2 GPUs with peer-to-peer access, a CUDA-IPC transport replaces them by default: each rank writes its half directly into the peer’s mapped staging buffer, with GPU-side sequence counters instead of a NCCL rendezvous. An all-to-all is a permutation, never a reduction, so the transport cannot change results — outputs are bitwise identical to the NCCL path.

Environment variable	Default	Meaning
SGLANG_DIFFUSION_IPC_A2A	1	set 0 to force NCCL
SGLANG_DIFFUSION_IPC_A2A_TIMEOUT_MS	10000	deadlock backstop for the peer wait; on expiry the transport retires on every rank and the request fails rather than returning incomplete data
SGLANG_DIFFUSION_IPC_A2A_MAX_BUFFERS	16	staging pairs kept alive; raise for many-resolution serving
SGLANG_DIFFUSION_ULYSSES_PIPELINE_GROUPS	-1	head groups for the pipelined exchange below; 0 keeps the sequential exchange, N >= 2 forces N groups

On a single host with peer-to-peer access, Ulysses attention also pipelines the exchange against attention, one head group at a time: while group g attends, group g + 1’s q/k/v moves on the copy engine, and finished groups send their output back while later groups still attend. Every Ulysses path of USPAttention goes through it under FlashAttention or cuDNN SDPA, at any batch size and with GQA: dense attention, key masks and tail padding, and text tokens replicated on every rank (a prefix, a suffix, or a K/V-only prefix), which stay out of the exchange. MiniMax-H3’s attention uses the same transport. Ring attention, uneven sequence shards (seq_lens) and the K/V-gather mode keep the sequential exchange. The automatic count is the largest of 7, 5, 4, 3 and 2 that divides the heads per rank while each group’s call still fills four waves of 128-row tiles, so short sequences such as a 1024² image keep the sequential exchange. Its buffers are one pool per dtype that every shape shares, grown to the largest (about 4 × rows per rank × heads × head_dim); a shape that would not fit on every rank keeps the sequential exchange. The first call of every shape runs both exchanges and keeps the pipeline only if they give the same bytes on every rank, so results never change; that check runs the sequential exchange on per-call buffers, so no staging for it stays allocated. A custom attention module reuses it through ulysses_pipelined_attention(q, k, v, attend, groups, replicated=..., sequential=...).

Independently of the transport, the default path already packs the three Q/K/V input exchanges into one destination-major collective. A handful of models (e.g. LTX-2) instead opt into enable_packed_qkv_input_a2a, which pipelines three separate exchanges over a dedicated stream rather than merging them into one payload — a different trade-off, not a strict upgrade over the default.

Which axes tolerate crossing nodes

Each axis has a fixed communication pattern, and the pattern — volume per step, how often it fires, and whether it can hide behind compute — decides whether the axis survives the drop from NVLink to the inter-node fabric. Ordered from most to least cross-node friendly:

Axis	Pattern	Traffic per denoise step	Cross-node verdict
Data parallel	none between replicas	zero on the request path	Best. Replicas only share startup init and control-op fan-out; no fast interconnect needed at all.
CFG parallel	one branch-combine	one latent-sized exchange, once	Good candidate. Once per step, small payload, naturally deadline-tolerant. Unmeasured across nodes so far.
K/V-gather CP	one K/V all-gather per attention	(R−1)/R of K/V, one unoverlapped burst	Between Ulysses and ring: only K/V moves (queries never do), but the burst cannot hide behind compute — prefer ring across nodes, gather at small degrees within one.
Ring SP	neighbor-only K/V rotation	K/V ÷ ring_degree, (R−1) hops, overlapped with attention tiles	Designed for it. The mechanism now covers most models’ joint attention (see Constraints). Actually crossing nodes end-to-end is validated for MiniMax H3 (Ulysses intra-node × ring across, net positive and growing with sequence length); other models’ ring support is same-node-validated so far.
Ulysses SP	all-to-all	full q/k/v activations, twice per attention, every layer	Keep intra-node. All-to-all needs full-bisection bandwidth; across nodes it becomes R² flows with receiver incast.
Tensor parallel	all-reduce	hidden-sized reduction per transformer block (×60 blocks for qwen-class DiTs)	Worst. Highest frequency, no overlap, already ~70% of sharded kernel time on NVLink.

Two caveats keep this a map rather than a promise. Cross-node launch (--nnodes/--node-rank/--dist-init-addr) is merged, and per-model ring support is broad now (see the constraints above) — but those two facts together still don’t add up to “any model, any node count.” The only configuration actually run end-to-end across nodes is the H3 recipe; other models’ ring support is same-node-validated so far, which means crossing nodes with them is untested, not disallowed. And the data-parallel row describes the design: the current implementation binds each replica’s ingress on the local host, so replicas spanning hosts additionally need per-replica host addressing before --dp-size can place one replica per node.

Choosing a configuration

Measured guidance rather than rules — the right combination depends on the model’s communication profile and the hardware topology, and legal does not mean profitable:

• Multi-branch (true-CFG) models: CFG parallelism first. Branches run the whole DiT independently and combine once per step, avoiding per-layer communication entirely.
• Single-branch image models on 2 GPUs: Ulysses and TP trade places by model. Communication-heavy DiTs measured faster with Ulysses; smaller DiTs with TP. Measure both; do not copy a winner across models.
• Long video / packed sequences: Ulysses up to the head-divisibility limit, then ring for the remaining factor — sequence length scales past the head count where Ulysses alone cannot.
• When Ulysses’s head divisibility blocks the degree you need (H/T not divisible by the target U): the row-splitting slot sidesteps it — ring today, or --kv-gather-degree (it splits rows, so it adds no head constraint either).
• TP beyond 2 ranks rarely improves image-DiT latency: the per-block all-reduce grows with rank count faster than the GEMM savings.

Data parallelism

--dp-size N runs N full engine replicas on num_gpus / N GPUs each, every replica with its own ingress. Generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica; replicas exchange nothing on the request path. Monolithic serving only, and the ingress currently binds on the local host — one replica per node needs per-replica host addressing first.