Cosmos3

Serve NVIDIA Cosmos3 image, video, sound, and action generation with SGLang Diffusion.

​

1. Quick start

Follow the SGLang Diffusion installation guide for your GPU platform.

Choose a generation or policy request matching the checkpoint; the Request tab queries model metadata.

Installation and prerequisites

Follow the SGLang Diffusion installation guide. Docker is recommended for Linux GPU deployments; run the commands below inside the container or your installed Python environment.

Cosmos3 guardrails are enabled by default when the package is available:

Command

pip install "cosmos-guardrail==0.3.1"

cosmos-guardrail downloads gated NVIDIA guardrail weights, so pass a Hugging Face token if your environment needs one. If the package is not installed, SGLang skips Cosmos3 guardrails and logs a warning. To disable Cosmos3 guardrails for local experiments, set SGLANG_DISABLE_COSMOS3_GUARDRAILS=1 before starting the server.

There may be problems loading the Cosmos-1.0-Guardrail weights on Ascend NPU. If the _pickle.UnpicklingError error occurs during startup, you should change weight_only=True to weights_only=False parameter in cosmos_guardrail/cosmos_utils.py:

#!/usr/bin/env bash
COSMOS_GUARDRAIL_DIR="$(dirname "$(python -c 'import cosmos_guardrail; print(cosmos_guardrail.__file__)')")"
sed -i 's/weights_only=True/weights_only=False/g' "$COSMOS_GUARDRAIL_DIR/cosmos_utils.py"

​

2. Model capabilities

NVIDIA Cosmos3 is an omnimodal world-model family spanning text/image/video generation, optional synchronized sound, and robot action prediction. Its main advantage is breadth: the same native SGLang pipeline can serve media-generation checkpoints and the DROID policy checkpoint without routing through an LLM sampler.

Choose Nano for the broadest modality coverage and lower deployment cost, Super for the larger 64B image/video model, and a specialized checkpoint when only T2I or I2V is needed. Sound and action are checkpoint-specific heads, so they are not available from every Cosmos3 repository.

Model	Status	Notes
nvidia/Cosmos3-Nano	Supported	T2I, T2V, I2V, V2V, joint sound, and action
nvidia/Cosmos3-Super	Supported	T2I, T2V, I2V, and V2V; use multi-GPU for the 64B checkpoint
nvidia/Cosmos3-Super-Text2Image	Supported	T2I-specialized checkpoint
nvidia/Cosmos3-Super-Image2Video	Supported	I2V-specialized checkpoint
nvidia/Cosmos3-Nano-Policy-DROID	Supported	DROID policy action generation
nvidia/Cosmos3-Edge	Supported	4B dense model for T2I, T2V, I2V, V2V, and action generation
nvidia/Cosmos3-Edge-Policy-DROID	Supported	4B DROID policy action generation
nvidia/Cosmos3-Super-Text2Image-4Step	Supported	64B T2I checkpoint distilled to a fixed 4-step schedule
nvidia/Cosmos3-Super-Image2Video-4Step	Supported	64B I2V checkpoint distilled to a fixed 4-step schedule

Sound and action generation require the corresponding checkpoint heads. The pipeline reads the transformer and scheduler configs at startup, so Edge and distilled checkpoints do not require architecture-specific server flags. Non-distilled checkpoints use the flow-native FlowUniPCMultistepScheduler; distilled checkpoints use the fixed sigma schedule stored in the checkpoint.

The default flow_shift is 3.0 for T2I, 10.0 for non-Edge video and all action modes, and 3.0 for Edge video modes. Distilled checkpoints bake the schedule into their sigmas and do not use a request-level flow_shift.

​

3. Serve Cosmos3

For Intel Xeon CPU deployment, use SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu with the selected Cosmos3 checkpoint.

Command

SGLANG_DIFFUSION_PLATFORM_OVERRIDE=cpu sglang serve --model-path nvidia/Cosmos3-Nano --port 30010

Serve Cosmos3-Nano directly from the Hugging Face model ID:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Nano \
  --num-gpus 1

With --performance-mode auto, Cosmos3 Nano keeps its DiT and VAE resident when every selected GPU has at least 90 GiB available at startup. Other Cosmos3 checkpoints use a 120 GiB threshold. Below the applicable threshold, auto mode retains the conservative DiT component-offload policy. Cosmos3 runs one DiT per pipeline, so component offload above the threshold only pays to copy the weights out to host memory and back on every request. Serve Cosmos3-Super across multiple GPUs as shown below so each rank holds a shard of the weights.

For Cosmos3-Super, split the model across multiple GPUs:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Super \
  --num-gpus 4

The server also accepts the specialized nvidia/Cosmos3-Super-Text2Image and nvidia/Cosmos3-Super-Image2Video checkpoint IDs.

​

Edge checkpoints

Cosmos3-Edge is a 4B dense model and can be served on one GPU:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Edge \
  --num-gpus 1

Edge is trained for 256p and 480p generation. Its default video configuration is 832x480 with guidance_scale=5.0; its default image configuration is 640x640 with guidance_scale=7.0. Supported sizes are 832x480, 480x832, 640x480, 480x640, 480x480, 640x640, 448x256, 256x448, and 256x256.

Serve the Edge DROID policy checkpoint with the same single-GPU configuration, replacing the model path with nvidia/Cosmos3-Edge-Policy-DROID.

​

Distilled checkpoints

The distilled Super checkpoints are 64B models. Use multiple GPUs unless the complete model and request workload fit on one GPU:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Super-Text2Image-4Step \
  --num-gpus 4

For distilled I2V, replace the model path with nvidia/Cosmos3-Super-Image2Video-4Step. SGLang detects both checkpoints from scheduler/scheduler_config.json, uses the checkpoint’s fixed four-step sigma schedule, and forces guidance_scale=1.0. Do not tune num_inference_steps or flow_shift for these checkpoints.

​

Simulation and multiview checkpoints

These checkpoints use native pipelines with checkpoint-specific conditioning:

Checkpoint	Inputs	Rollout
nvidia/Cosmos3-Nano-Sim-Bimanual	Image, prompt, camera or robot actions	Causal chunks with committed KV history
nvidia/Cosmos3-Nano-Sim-Transfer	Control video and prompt	Causal control-video transfer
nvidia/Cosmos3-Nano-Transfer-Auto	Per-camera WSM clips and captions; optional LiDAR	Bidirectional multiview generation
nvidia/Cosmos3-Nano-Sim-Auto	Per-camera WSM clips and captions; optional LiDAR	Causal multiview generation

For example, start the Bimanual server:

sglang serve --model-path nvidia/Cosmos3-Nano-Sim-Bimanual

Submit image-conditioned video requests through /v1/videos. Supply action rows according to the checkpoint’s embodiment contract, selected by domain_name or domain_id. Native contracts accept raw rows with action_normalization="contract"; unified contracts require normalized 59-dimensional rows and default to action_normalization="none". Without actions, Bimanual uses null-action conditioning. These pipelines predict video from actions, not actions from video; use the policy checkpoints above for action prediction.

For realtime Bimanual sessions, add --pipeline-class-name CosmosDreamsRealtimePipeline and use /v1/realtime_video. Each tick consumes one action block and preserves causal KV history.

Transfer-Auto and Sim-Auto take a multiview object with views, each containing camera_key, control_path, optional vision_path, and optional prompt. Joint checkpoints also accept lidar; set lidar.decode=true to return decoded range maps and previews. Outputs pack all frames of each camera consecutively, in request camera order.

LiDAR uses built-in compiled FlexAttention by default. The optional NATTEN path requires a wheel matching your PyTorch/CUDA versions and SGLANG_DIFFUSION_COSMOS3_LIDAR_USE_NATTEN=1; it is not installed by default.

Bimanual and Sim-Transfer support TP, but not SP or CFG parallelism. Transfer-Auto supports Ulysses without TP or Ring; Sim-Auto supports TP and Ulysses subject to the checkpoint’s query/KV head divisibility, but not Ring or CFG parallelism. Causal pipelines use the exported fixed-step schedule and require guidance 1.0.

Use --component-residency for memory-constrained deployments. Joint checkpoints register lidar_encoder and lidar_decoder separately from the camera vae. Bimanual overlaps causal VAE decode only when the VAE is resident; offloaded VAEs use the final serial decode stage. The causal pipelines default to history_mode: auto, which follows the export’s training K/V window: a sliding cache over the artifact’s window_frames and sink_frames when the export records kv_cache_inference_size, the whole clip otherwise; frustum retrieval needs history_mode: full. History-window reduction and frustum retrieval change the attention context and are not lossless acceleration settings.

​

4. OpenAI-Compatible Requests

​

Text to image

Cosmos3 text-to-image uses /v1/images/generations. The default Cosmos3 image response is b64_json, matching vLLM-Omni’s examples.

Command

curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "1280x720",
    "n": 1,
    "num_inference_steps": 35,
    "guidance_scale": 6.0,
    "flow_shift": 3.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'

With a server running nvidia/Cosmos3-Super-Text2Image-4Step, omit the scheduler controls and use guidance_scale=1.0:

Command

curl -sS -X POST http://127.0.0.1:30010/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "A warehouse robot folds a blue cloth on a clean workbench.",
    "size": "640x640",
    "n": 1,
    "guidance_scale": 1.0,
    "seed": 0,
    "extra_body": {
      "use_resolution_template": false,
      "guardrails": true
    }
  }'

​

Text to video with sound

Use /v1/videos to create an asynchronous job, then poll the job and download the completed MP4. Set generate_sound=true to generate and mux a stereo 48 kHz audio track; omit it for a silent video.

Command

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A small warehouse robot moves a blue box across a clean floor." \
  --form-string "negative_prompt=blurry, distorted, low quality" \
  --form-string "size=1280x720" \
  --form-string "num_frames=81" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=4.0" \
  --form-string "flow_shift=10.0" \
  --form-string "generate_sound=true" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

while true; do
  status=$(curl -sS "http://127.0.0.1:30010/v1/videos/${job_id}" \
    | python -c 'import json, sys; print(json.load(sys.stdin)["status"])')
  [ "$status" = "completed" ] && break
  [ "$status" = "failed" ] && exit 1
  sleep 1
done

curl -sS -L "http://127.0.0.1:30010/v1/videos/${job_id}/content" \
  -o cosmos3_t2v.mp4

​

Image to video

This mirrors the official nvidia/Cosmos3-Nano Hugging Face image-to-video example:

Python

import json
import time
from pathlib import Path

import requests
from huggingface_hub import snapshot_download

base_url = "http://127.0.0.1:30010"
model_dir = Path(snapshot_download("nvidia/Cosmos3-Nano"))
asset_dir = model_dir / "assets"

prompt = json.dumps(json.loads((asset_dir / "example_i2v_prompt.json").read_text()))
negative_prompt = json.dumps(
    json.loads((asset_dir / "negative_prompt.json").read_text())
)

data = {
    "prompt": prompt,
    "negative_prompt": negative_prompt,
    "size": "1280x720",
    "num_frames": "189",
    "fps": "24",
    "num_inference_steps": "35",
    "guidance_scale": "6.0",
    "max_sequence_length": "4096",
    "flow_shift": "10.0",
    "seed": "1111",
    "extra_params": json.dumps(
        {
            "use_resolution_template": False,
            "use_duration_template": False,
            "guardrails": True,
        }
    ),
}

with (asset_dir / "example_i2v_input.jpg").open("rb") as image:
    response = requests.post(
        f"{base_url}/v1/videos",
        data=data,
        files={"input_reference": ("example_i2v_input.jpg", image, "image/jpeg")},
        timeout=60,
    )
response.raise_for_status()
video_id = response.json()["id"]

while True:
    job = requests.get(f"{base_url}/v1/videos/{video_id}", timeout=30).json()
    if job["status"] == "completed":
        break
    if job["status"] == "failed":
        raise RuntimeError(job.get("error") or "Video generation failed")
    time.sleep(1)

response = requests.get(f"{base_url}/v1/videos/{video_id}/content", timeout=300)
response.raise_for_status()
Path("cosmos3_i2v.mp4").write_bytes(response.content)

For the distilled I2V checkpoint, use the same API with a server running nvidia/Cosmos3-Super-Image2Video-4Step. The recommended request is 480p and does not specify scheduler controls:

Command

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A warehouse robot carefully places a blue box on a shelf." \
  --form "input_reference=@first_frame.png;type=image/png" \
  --form-string "size=832x480" \
  --form-string "num_frames=189" \
  --form-string "fps=24" \
  --form-string "guidance_scale=1.0" \
  --form-string "seed=42" \
  --form-string 'extra_params={"guardrails":true,"use_resolution_template":false,"use_duration_template":false}' \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

Poll and download this job with the same status and content endpoints used by the T2V example.

​

Video to video

Upload a source video with video_reference. Cosmos3 keeps latent frames [0, 1] by default and generates the remaining frames. Use condition_frame_indexes to select different latent frames, and condition_video_keep to take conditioning frames from the start or end of the source.

Command

job_id=$(curl -sS -X POST http://127.0.0.1:30010/v1/videos \
  --form-string "prompt=A robotic arm pours liquid into a glass on a white tabletop." \
  --form "video_reference=@robot_pouring.mp4;type=video/mp4" \
  --form-string "size=1280x704" \
  --form-string "num_frames=45" \
  --form-string "fps=24" \
  --form-string "num_inference_steps=35" \
  --form-string "guidance_scale=6.0" \
  --form-string 'condition_frame_indexes=[0,1]' \
  --form-string "condition_video_keep=first" \
  | python -c 'import json, sys; print(json.load(sys.stdin)["id"])')

Poll and download this job with the same status and content endpoints used by the T2V example.

​

Action generation

For DROID policy generation, start a server with either the Nano or Edge policy checkpoint. One GPU is enough; to split a request across GPUs, use Ulysses sequence parallelism (--num-gpus 2 --ulysses-degree 2). Cosmos3 action generation does not support CFG parallel; policy requests run with guidance_scale=1.0, so this only matters if you raise the guidance scale on a multi-GPU server.

Command

sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1

Use nvidia/Cosmos3-Edge-Policy-DROID in the same command to serve the smaller 4B policy checkpoint.

policy and inverse_dynamics return actions, so their canonical API is the synchronous /v1/actions/generations endpoint. The following request predicts a 16-step action chunk from one observation image. action_horizon=16 maps to the model’s num_frames=17 convention.

Python

import base64
from pathlib import Path

import requests

image_b64 = base64.b64encode(Path("observation.png").read_bytes()).decode()
response = requests.post(
    "http://127.0.0.1:30010/v1/actions/generations",
    json={
        "input": {
            "task": "Put the pot to the left of the purple item.",
            "observation": {
                "image": {"b64_json": image_b64},
            },
        },
        "parameters": {
            "action_mode": "policy",
            "action_horizon": 16,
            "domain_name": "droid_lerobot",
            "height": 480,
            "width": 832,
            "fps": 5,
            "num_inference_steps": 30,
            "guidance_scale": 1.0,
            "seed": 42,
        },
    },
    timeout=300,
)
response.raise_for_status()
action = response.json()["data"][0]["action"]
print(action["shape"], action["values"])

Use GET /v1/actions/metadata to inspect the action modes, default horizon, padded action dimension, and accepted observation modalities. Msgpack requests and the /v1/actions/realtime websocket use the same action envelope.

To batch policy observations inside one request, opt in with a bounded batch size:

Command

sglang serve \
  --model-path nvidia/Cosmos3-Nano-Policy-DROID \
  --num-gpus 1 \
  --batching-max-size 4

Send one image per observation as a list or [B, H, W, C] uint8 array in input.input_reference, and either one prompt per image or one scalar prompt to broadcast across the batch. Batched prompts must currently tokenize to the same length because Cosmos3 GEN cross-attention does not mask padded text K/V. All items in one request share the domain, resolution, action horizon, and denoise settings. The standard action envelope returns one data[i] item per input, each with action shape [H, D]. For a compact msgpack response containing one [B, H, D] array, set runtime.response_format="raw" and read the top-level actions field.

For JSON, input_reference can be a list of base64 image payloads. For msgpack, it can be a packed uint8 numpy array directly:

JSON

{
  "input": {
    "prompt": ["pick up the block", "close the drawer"],
    "input_reference": [
      {"b64_json": "<first-image-base64>"},
      {"b64_json": "<second-image-base64>"}
    ]
  },
  "parameters": {
    "action_mode": "policy",
    "domain_name": "droid_lerobot"
  }
}

The batch size cannot exceed --batching-max-size; this keeps one request from bypassing the server’s configured memory limit. Batching applies to action_mode="policy" only. A request seed controls the random stream for the whole batch, so a batched result is deterministic for that request but is not expected to be bit-exact with separately seeded B=1 requests.

inverse_dynamics also uses /v1/actions/generations; set action_mode="inverse_dynamics" and pass an observation video URL or server-local path as input.observation.video. Select the embodiment head with domain_name or domain_id; set raw_action_dim explicitly when it cannot be inferred from the domain name.

forward_dynamics is intentionally different: it consumes an action array and predicts video, so it remains on /v1/videos. Action-producing modes submitted to /v1/videos return HTTP 400 with the canonical action endpoint in the error message.

​

5. Cosmos3 Parameters

Cosmos3 supports the standard SGLang video and image fields such as size, num_frames, fps, num_inference_steps, guidance_scale, negative_prompt, and seed. For distilled checkpoints, SGLang replaces num_inference_steps with the checkpoint’s fixed four-step schedule and forces guidance_scale=1.0; negative-prompt CFG and request-level flow_shift do not apply.

Top-level Cosmos3 request fields:

• max_sequence_length: maximum text token length used by the Cosmos3 tokenizer.
• flow_shift: per-request scheduler shift for non-distilled checkpoints. If omitted, SGLang uses --flow-shift, then the mode default (3.0 for T2I, 10.0 for non-Edge video and all action modes, or 3.0 for Edge video).
• guidance_interval: optional [start, end] noise interval for CFG. Non-distilled T2I defaults to [400, 1000]; video modes guide at every step.

Cosmos3 omnimodal fields are accepted as extra JSON fields or multipart form fields:

• generate_sound: generate a sound track whose duration follows num_frames / fps.
• sound_duration: explicit sound duration in seconds; takes precedence over the derived duration.
• condition_frame_indexes: V2V latent-frame indexes to keep from the source video; defaults to [0, 1].
• condition_video_keep: use the first or last source frames for V2V conditioning.
• action_mode: policy, forward_dynamics, or inverse_dynamics.
• domain_name / domain_id: select the action embodiment head.
• raw_action_dim: number of active action dimensions; inferred for known domain names.
• action: action array with shape [T, D], required by forward_dynamics.
• action_fps: action-token frame rate for temporal mRoPE; defaults to the video FPS.
• action_view_point: viewpoint used in the structured action caption.
• action_normalization: dataset normalization mode, such as quantile, meanstd, or minmax.

Pass model-specific controls through extra_body with the OpenAI Python SDK. Raw JSON may keep them at the top level; multipart video requests should put them in the extra_params JSON object. The legacy image extra_args container remains accepted for compatibility, but new clients should use extra_body:

• use_duration_template: whether to append SGLang’s generated duration suffix to video prompts.
• use_resolution_template: accepted for vLLM-Omni request compatibility.
• use_system_prompt: whether to add the Cosmos3 system prompt to the chat template.
• guardrails or use_guardrails: per-request guardrail toggle when the server started with guardrails enabled.

​

6. Run in ComfyUI