1. Quick start
Use a SGLang Diffusion Docker environment. Until a released image contains this integration, install SGLang frommain
inside it:
The picker selects
black-forest-labs/flux-3-image-klein-base or
black-forest-labs/flux-3-image-klein, each a self-contained repository. Access
to the selected checkpoint is required. Keep --model-path at its repository
root; --model-variant selects only the quantized DiT. Both release BF16
checkpoints have single-GPU generation checks; see the validation boundaries
below. Unverified hardware or resource shapes remain selectable but are not
measured recommendations.2. Model capabilities
FLUX.3 Image Klein is Black Forest Labs’ 5.6B open-weights model for text-to-image generation and image editing. It takes a prompt and up to 10 reference images and returns one image; text-to-image, single-reference edits and multi-reference compositions run through the same DiT, with Qwen3-VL-4B-Instruct as the text encoder. The 5.6B count covers the image transformer only. Prompts can place elements with bounding boxes on a 0 to 1000 grid, then recolor, move, replace or remove them; boxes guide placement and scale rather than masking pixels. The model also renders text inside images. One SGLang server serves text-to-image and editing, chosen per request. This image pipeline does not serve the separate FLUX.3 Action robot policy.
Each repository ships its BF16 DiT with FP8 (
fp8r) and NVFP4 variants,
together with the text encoder and image autoencoder. Quantization and caching
are separate fidelity/latency tradeoffs, not lossless substitutes for the
selected checkpoint.
Start the server:
num_inference_steps, guidance_scale, size,
seed, n); unset fields take the checkpoint’s defaults from its
config.json. SGLang expands n into independently seeded requests and can
pack compatible requests into one DiT forward.
References are prepared as in the
BFL reference pipeline.
A single reference is
fitted under 4 MP, reflect-padded to the 16-pixel grid and, when you set no
output size, sets the output size; the padding is cropped from the result.
With several references each is fitted under 1 MP and centre-cropped, and the
output takes the first one’s size.
Prompting
The caption controls composition; reference images anchor content but do not lock structure.- Write full scene captions covering framing, subject, background and lighting. A short prompt gives generic output, and a one-line edit instruction (“make it blue”) reframes the shot. The prompt upsampler expands a short request into a full caption.
- For edits, describe the target image as a complete caption of the reference with the edit folded in.
- For layout, append a JSON list of elements to the caption. Each box is
[top, left, bottom, right]on a 0 to 1000 grid, and the caption tags each element as<id>. To edit by box, refer to the n-th reference as<ref_image_n>and give each elementfrom,src_bboxandtgt_bbox; anulltarget removes it. See BFL’s bounding box guide for the full format.
3. Checkpoints and precision
Both repositories use
flux-3-image-klein in configuration and safetensors
metadata. This shared identifier does not determine the sampling mode. The
release repositories do not need the pre-release sd-* variant names.
The distilled model is a step-distilled flow-map DiT: each step jumps straight
to the next time of the schedule, so it samples without CFG. SGLang tells a
flow map from its weights. It rejects --enable-cfg-parallel, a request
guidance_scale other than 1, and a non-empty negative_prompt. Tensor
parallelism, Ulysses SP and breakable CUDA graphs work as for the base model.
The Base model defaults to CFG 3.5 against an empty negative prompt. Both
variants reuse the package’s text encoder and VAE. Keep the checkpoint revision,
precision, attention backend, guidance and sampling parameters fixed when
comparing outputs.
Explicit guidance_scale=1 uses scale 1, not the package default;
true_cfg_scale, when provided, takes precedence over guidance_scale.
Quantized checkpoints
Each repo ships quantized DiTs undervariants/: select one with
--model-variant fp8r or --model-variant nvfp4 and keep --model-path
pointing at the repo root. Text encoder and VAE precision follow their own
package settings; quantizing the DiT does not quantize them. The quantization
comes from the variant’s hf_quant_config.json, so do not also pass
--quantization.
SGLANG_ENABLE_FLUX3_PRIMS_TS_AUTOTUNE=1 to
autotune them during warmup. Other GPUs use the generic rowwise-FP8 GEMM.
NVFP4 stores FP4 (E2M1) weights with FP8 block scales and needs a Blackwell
GPU and a compatible GEMM implementation. SM10x and consumer SM120 use
different kernel paths; a B200 recipe does not establish RTX 5090 performance.
Both quantization modes can change output quality and need workload-specific
comparison against BF16.
These flags remain separately configurable; see the validation boundaries below
before combining quantization with other optimizations.
Breakable CUDA graphs
BCG reserves additional device memory and captures specific request shapes. The picker generates--warmup-resolutions 1024x1024 and matching
--warmup-sampling-params for the selected quality. For editing it also sets
flux3_mode="i2i" and a server-local reference path: mount a 1024x1024 image
there before starting the server. This path can differ from the client’s upload
path. Longer prompt token buckets, other reference dimensions/counts, batches
or quality settings may run eager instead; check for a serving-signature miss
in the server log.
Changing the picker workload also changes its graph warmup command. Restart
with that command to capture the new workload. Keep eager for variable-shape
traffic, or explicitly validate every shape you intend to capture.
LoRA adapters
Use the standard LoRA API with native DiT keys, optionally prefixed bydit. or
transformer.. BF16 dynamic and merged adapters have a synthetic rank-4 generation
check, including repeatability and exact restoration after unloading; this does
not validate every community adapter. For quantized deployments requiring LoRA,
pass --lora-path at startup to select the generic quantized layout. Attaching
LoRA after PrimsTS weight preparation is rejected before replacing any layers.
FP8R and NVFP4 startup adapters also have synthetic rank-4 generation checks
on GB300, including repeatability and exact restoration after unloading. These
do not establish compatibility with arbitrary quantized community adapters.
4. Multi-GPU and memory
FLUX.3 shares its DiT with FLUX.3 Action, so the image pipeline supports tensor parallelism, Ulysses sequence parallelism, CFG parallelism, layerwise offload and breakable CUDA graphs. The picker covers RTX 5090, RTX PRO 6000, H100/H200, and B200/B300/GB200/GB300 with one to eight GPUs on one node. TP times Ulysses degree must equal the GPU count and evenly partition the 24 attention heads. Ring is not admitted by this image recipe. For example, two-GPU TP uses--num-gpus 2 --tp-size 2; two-GPU Ulysses
uses --num-gpus 2 --tp-size 1 --ulysses-degree 2. Base also permits CFG
parallelism; distilled checkpoints have no CFG branch to parallelize.
More GPUs do not guarantee lower latency, particularly over PCIe.
Ulysses inherits all-to-all streaming through the shared attention layer.
Leave its pipeline-group policy automatic; short sequences need not benefit.
TP-only deployments do not use this path. Encoding-while-decoding overlaps
video-frame decoding with video-file encoding, so it does not apply to this
still-image pipeline.
Under sequence parallelism every rank holds the full DiT, so combining it with
layerwise offload streams the whole model to each GPU every step. TP also
partitions weights, so it is the lower-memory starting point when offloading
on several GPUs. Preserve headroom for the longest prompt, references and output
size you intend to serve; a successful small request is not a capacity guarantee.
quality="lossless" can use dense-layout,
full-context memory-efficient SDPA; quality="exact" retains the original path.
It does not prune attention or reduce precision, but reduction order can change
pixels. In a Base BF16 test with a 2048x2048 reference and 1024x1024 output,
GB300 peak allocated memory fell from 54.3 to 29.5 GiB; latency was effectively
unchanged. GB300/H200 image SSIM was 0.9951-0.9992 against the exact path across
four samples. This is allocated memory, not total device usage: a 32 GB card
still needs offload or quantization headroom for that reference size.
Disaggregated serving
You can deploy the prompt encoder, reference-image VAE encoder, DiT, and VAE decoder as separate worker pools behind the same image API. Both T2I and i2i are supported. Use the split encoding deployment guide for the four-role commands, or keep the existing three-role topology with a combined encoder. This separates resources and scaling; it adds transfer hops and does not imply lower single-request latency. T2I skips the image encoder; editing keeps reference padding/crop semantics. Each pool can have a different number of workers. Keep the package variant and numerical settings consistent across workers; the guide includes the required transfer dependency, network ports, and same-node validation scope.5. Cache-DiT
Cache-DiT skips the 28 single-stream blocks on steps where their input barely changes and reuses the cached residual. Turn it on for every request withSGLANG_CACHE_DIT_ENABLED=true on sglang serve,
or per request with "enable_cache_dit": true. A request can also tune it, for
example "cache_dit_params": {"residual_diff_threshold": 0.12, "enable_taylorseer": true}.
For image editing, send -F "enable_cache_dit=true" and a JSON-encoded
cache_dit_params form field; generation uses JSON request fields.
Cache-DiT is approximate and may change composition; measure quality and latency
on representative prompts rather than treating its threshold as a fidelity guarantee.
It combines with FP8R and NVFP4, tensor parallelism, CFG parallelism and
layerwise offload. It is skipped, with a warning, under Ulysses sequence
parallelism and breakable CUDA graphs, and for a request whose CFG pair has to
run as two forwards to fit in device memory.
6. Request examples
- Text to image
- Image to image
- Offline generation
7. Prompt upsampler
black-forest-labs/flux-3-image-klein-pup is a Qwen3.5-9B vision-language fine-tune that expands a short prompt into the detailed caption FLUX.3 works best with. The BFL reference code runs it in front of the image model by default. It is a standardQwen3_5ForConditionalGeneration checkpoint, so it is served
as a regular LLM, separately from the image pipeline. The system prompt (system_prompts.json
in the repository) selects the mode; t2i is the text-to-image upsampler.
Download the package and start the upsampler on its own port:
t2i system
prompt is read from the downloaded package; thinking is disabled and decoding follows the
model card (greedy, presence penalty 1.5):
--prompt of a t2i request to the Flux3 pipeline.
To have the image server call the upsampler itself, point it at this endpoint with the
built-in flux3-klein-pup protocol. It reads system_prompts.json from the upsampler
package (protocol_source, a local directory or the Hugging Face repo, which is the
default) and follows BFL’s reference code: the t2i prompt for generations, the i2i
prompt with the references first (resized as the reference code does) for edits, greedy
decoding with presence penalty 1.5 for t2i and 0 for i2i, and thinking disabled.
enhance_prompt; it is off by default, so a full caption is sent
as typed:
--enable-deterministic-inference --disable-radix-cache when the same request must give
the same caption.
The layout prompts (t2i_bbox, i2i_bbox) are not wired in yet; write the box table
into the prompt yourself as described in Model capabilities.
8. Dynamic batching
Enable a small batching window to collect concurrent requests:Serve concurrent requests
Pre-release batching measurements
These measurements used pre-release weights, not the release repositories above. They do not establish release-checkpoint performance. The following are warmed HTTP measurements on one GB300 (about 277 GiB device memory), Base BF16, Torch SDPA, eager execution and all components resident. Each row compares two different prompt/seed requests served serially with the same two requests served together. Times include local HTTP transfer and JPEG output; medians cover three rounds after warming both shapes and reference encoding. Software: PyTorch 2.14.1+cu130, Transformers 5.19.0. Checkpoint revision:5de4fe32f17aee3dbc34375a94935f09206dba2c.
Eight steps exercise batching, not the recommended Base image-quality setting.
The standard 50-step workload shows little batching benefit;
FlashAttention gives a similarly small gain in this
setup. Batch 1 remains the default. Do not extrapolate the small-request speedup
to full-resolution generation or other GPUs.
For these 50-step requests, batch-1 versus batch-2 JPEG outputs had SSIM
0.9920-0.9977. A separate four-image PNG comparison at 512x512/8 steps had SSIM
0.9885-0.9955; JPEG can amplify small rounding differences. These are observed
samples, not a universal quality guarantee. Identical single requests before
and after batching matched exactly. Preserve additional memory headroom for
batching rather than assuming a batch-1 recipe fits on a 32 GB card at batch 2.
9. Validation boundaries
The release BF16 and FP8R checkpoints have single-GB300/H200 generation checks; NVFP4 has GB300 and RTX 5090 checks. RTX 5090 also has Base BF16 and both checkpoints’ FP8R and Cache-DiT checks. Single-GPU measurements use eager execution, all components resident, batch 1, 1024x1024 T2I and I2I, two prompts/seeds and repeated requests. Base uses 50 steps and distilled uses 16. Repeated requests matched pixel-for-pixel within each tested configuration. This does not establish official-reference parity or validate every combination of hardware, parallelism, batching and optional features in the picker.Multi-GPU measurements
Base BF16, 50 steps, CFG 3.5, batch 1 andquality="exact". These are warmed
localhost HTTP medians through PNG decoding, excluding loading and warmup;
two prompts/seeds and two timed rounds per prompt. Software: PyTorch
2.14.1+cu130, Transformers 5.19.0, FlashInfer 0.7.1. Both runs use checkpoint
d8acde427a698bdb98b5c1c941c60fa4c97e9f28. The H200 run uses FA3 and
NVLink; the RTX 4090 run uses Torch SDPA over PCIe without CUDA peer access.
The picker keeps TP as its multi-GPU starting point, not a guarantee that TP
always wins. A hybrid topology can help on PCIe; validate it on your own
interconnect. H100, B200 and other unmeasured resource shapes are extrapolated
starting points and are not labelled verified.
On the same two H200s, Ulysses streaming reduced 2048x2048 T2I from 24.00 to
22.87 s and editing a 2048x2048 reference into a 1024x1024 output from 31.01
to 29.54 s. Streamed and non-streamed outputs matched pixel-for-pixel in these
samples, with no quantization or caching. At 1024x1024, automatic dispatch
selected no streaming groups and gave no speedup. TP and SP themselves are
not bit-identical: sample SSIM was 0.9890-0.9985. None of these comparisons
establishes which path is closer to an official reference.
Single-GPU measurements
Warmed local HTTP medians, including PNG encoding and response transfer, from two rounds per prompt. PyTorch 2.14.1+cu130, Transformers 5.19.0, FP32 VAE,quality="exact"; FlashAttention on GB300/H200 and Torch SDPA on RTX 5090.
The Base checkpoint revision is d8acde427a698bdb98b5c1c941c60fa4c97e9f28.
SSIM compares each quantized result with BF16 on the same GPU, prompt, seed and
reference, and reports the lowest value across two samples, not a quality bound.
For the distilled checkpoint, quantization’s T2I benefit was small on GB300;
do not assume Base’s speedup transfers to a 16-step workload. Cache-DiT at
threshold 0.12 reduced Base I2I from 6.31 to 3.08 s on GB300, with sample SSIM
0.9786-0.9975. It remains opt-in and approximate.
Eager versus CUDA graphs
Matched ABBA comparisons usequality="lossless", resident weights, batch 1,
1024x1024 output/reference, and the same software versions above with stable
FlashInfer 0.7.1. Base uses 50 steps/CFG 3.5; distilled uses 16 steps/no CFG.
Times are warmed localhost HTTP medians through JPEG decoding: two prompts/seeds
and four timed samples per arm, excluding loading and graph capture. These
JPEG timings are not directly comparable with the earlier PNG table.
GB300 distilled NVFP4 T2I improves by 22.5% with FA4 + BCG. Peak allocated
memory increases from 16.63 to 17.46 GiB; peak reserved rises from 18.75 to
19.96 GiB. These are per-process allocator maxima across warmup/runtime,
not loading or total-device peaks. Other workloads also need headroom; memory
capacity alone is not a reason to enable graphs when they do not improve latency.
All 15 fixed-backend eager/BCG workload pairs passed repeated-request checks
and pixel-identical PNG comparisons across the two prompts. This is not an
official-reference accuracy guarantee or evidence that quantization preserves
BF16 output. GB300 measurements use Base revision
17333191c36bae1b61b0a247e33ba284f1df07fa
and distilled 875076bf83d4364b68dccdbf8307f4f01475f68d; H200/5090 use
d8acde427a698bdb98b5c1c941c60fa4c97e9f28 and
4d02a6a4dc4d5c7a0f3fa767187483b2b70f2384. TP2 uses tensor-identical Base
DiT weights with a different safetensors header.
GB300 distilled NVFP4 T2I also ran at 0.424 s with SDPA + BCG (versus 0.589 s
SDPA eager). It is selectable but not the default backend: SDPA versus FA4
outputs had sample SSIM 0.978-0.987 and MAD 1.74-1.75/255. Neither path is
established as closer to the official output. Keep the backend fixed when
evaluating the graph speedup.
The native DiT’s packed CFG forward has focused comparison tests against the
shared FLUX 3 transformer. Such component tests do not replace full-checkpoint
E2E or establish parity with every upstream release. Keep unverified hardware,
topologies and optional optimizations separate from measured deployment recipes.
The opt-in full-checkpoint HTTP regression covers concurrent prompts, image
editing, multiple outputs and a repeated single request:
SGLANG_TEST_FLUX3_IMAGE_MODEL to a local package path when using an
offline checkpoint. This test needs checkpoint access; it does not substitute
for an official-reference golden-image comparison.