LingBot World 2.0

1. Model Introduction

LingBot World 2.0 is the 14B realtime camera-controlled world model in the LingBot family. It generates continuously from a live causal session, accepts control updates between chunks, and reuses DiT and VAE state for incremental frame delivery.

Choose 2.0 when the larger model and interactive KV-window path justify an 8-GPU deployment. Like the earlier LingBot release, it prioritizes responsive control and bounded causal state over the full-clip global context of offline video diffusion.

2. Deployment

Command

export SGLANG_LINGBOT_LAZY_VAE_ENCODE_BLACK_FRAMES=60
export SGLANG_LINGBOT_ENABLE_INTERACTIVE_KV_WINDOW=true
sglang serve \
  --model-path robbyant/lingbot-world-v2-14b-causal-fast-diffusers \
  --pipeline-class-name LingBotWorldCausalDMDPipeline \
  --num-gpus 8 \
  --ulysses-degree 8 \
  --dit-cpu-offload false \
  --text-encoder-cpu-offload false \
  --vae-config.use-parallel-decode true \
  --vae-config.parallel-decode-mode spatial \
  --enable-torch-compile false

Optional KV-Cache Compression

Long-running sessions can enable lossy int4 PRQ compression for completed causal KV-cache chunks by installing quant-videogen as described in the quantization guide and adding --kv-cache-quant int4 to the server command. The current and most recent completed chunks remain in BF16. See Causal KV-Cache Quantization for the algorithm, tuning options, measured memory-latency tradeoff, and support limits.

3. Realtime WebUI

The lightweight local WebUI is useful for validating latency, frame transport, and camera control behavior.

Command

python -m http.server 18080 -d python/sglang/multimodal_gen/apps/realtime_webui

Open http://127.0.0.1:18080 and use:

Example

ws://127.0.0.1:30000/v1/realtime_video/generate

4. HTTP and WebSocket API

LingBot World 2.0 uses the realtime video WebSocket endpoint. The server keeps one live session, generates one chunk at a time, and accepts runtime control events while generation is running.

Endpoints

API	Method	Purpose	Notes
/v1/models	GET	Query the served model id before opening a session.	The WebUI uses this to fill the model field when the server exposes model metadata.
/v1/realtime_video/generate	WebSocket	Create one realtime LingBot session and stream generated video chunks.	The first client message must be an init message encoded with MessagePack.

init message

Send this MessagePack map immediately after the WebSocket opens.

Parameter	Type	Required	Meaning
type	string	Yes	Must be "init".
model	string	No	Model id. Leave empty to use the served model.
prompt	string	Yes	Text prompt for the initial scene and motion style.
first_frame	bytes or string	Yes	Initial reference image. Send bytes from the WebUI/client, or a server-readable image path/string.
size	string	Yes	Generation size as WIDTHxHEIGHT, for example 832x480.
fps	number	Yes	Target playback FPS for the generated stream.
num_frames	integer	Yes	Frames per generated chunk. LingBot uses chunked causal generation, so this controls per-chunk latency and queue size.
seed	integer	No	Random seed for deterministic sampling.
num_inference_steps	integer	No	Denoising steps per chunk. LingBot defaults to 4 when omitted.
guidance_scale	number	No	Classifier-free guidance scale. Realtime LingBot commonly uses 1.
negative_prompt	string	No	Negative prompt passed to the diffusion pipeline.
quality	"lossless", "high"	No	lossless keeps FP32 VAE decode. high uses the validated BF16 decode path for lower per-chunk latency.
max_chunks	integer	No	Stop after this many chunks. Omit for a continuous session.
realtime_causal_sink_size	integer	No	Number of sink frames/tokens retained in the causal attention window.
realtime_causal_kv_cache_num_frames	integer	No	Number of recent frames retained in the causal KV cache window.
realtime_output_format	"webp", "jpeg", "raw"	No	Preview/output transport. webp and jpeg send encoded preview frames; raw sends raw RGB; omit for lossless delta-gzip RGB.
output_compression	integer	No	Preview quality for webp or jpeg, from 1 to 100.
enable_upscaling	boolean	No	Enable server-side super resolution after frame decode.
upscaling_scale	integer	No	Super-resolution scale. Current default is 4 when upscaling is enabled.
upscaling_model_path	string	No	Optional Real-ESRGAN model path.
enable_frame_interpolation	boolean	No	Enable frame interpolation. Keep this disabled when measuring true generated FPS.
frame_interpolation_exp	integer	No	Interpolation multiplier exponent. 1 means 2x frames.
frame_interpolation_scale	number	No	RIFE internal scale for interpolation.
frame_interpolation_model_path	string	No	Optional RIFE model path.
condition_inputs.camera_actions	list[list[string]]	No	Initial scripted camera actions, one action list per frame.

Runtime event messages

After init, send MessagePack event maps to update the live session.

Parameter	Type	Required	Meaning
type	string	Yes	Must be "event".
kind	"prompt", "camera_actions", or "composite_input"	Yes	Runtime event kind.
payload	string, object, or list	Yes	For prompt, a non-empty string. For camera_actions, either scripted list[list[string]] or state-mode payload. For composite_input, a map containing input_types plus each named input.
event_id	integer	No	Client sequence id. The server echoes it in chunk/frame metadata after the event is sampled.

camera_actions supports two payload modes:

Mode	Payload shape	Meaning
Script	list[list[string]]	A fixed sequence of per-frame actions consumed by upcoming chunks.
State	{ "mode": "state", "transitions": [{"actions": [...], "client_ts_ms": ...}] }	Live control state transitions from keyboard or UI controls.

Supported LingBot action tokens include w, a, s, d for camera movement and i, j, k, l for look controls.

Use composite_input when multiple runtime inputs should be sampled together, such as updating the prompt and camera controls in one event.

Server messages

Message	Payload	Meaning
frame_batch	MessagePack map with payload bytes	One batch of frames. The map includes chunk_index, num_frames, content_type, encoding, width, height, and frame-batch metadata.
chunk_stats	MessagePack map	Per-chunk timing and transport metrics, including scheduler_forward_ms, raw_payload_build_ms, chunk_total_ms, num_frames, and ws_payload_bytes.
error	MessagePack map	Server-side validation or generation error.

Minimal client sketch

Python

import msgspec.msgpack
import websocket

ws = websocket.create_connection("ws://127.0.0.1:30000/v1/realtime_video/generate")
ws.send_binary(msgspec.msgpack.encode({
    "type": "init",
    "prompt": "A quiet rainy London alley, stable camera motion.",
    "first_frame": open("reference.jpg", "rb").read(),
    "size": "832x480",
    "fps": 25,
    "num_frames": 9,
    "num_inference_steps": 4,
    "guidance_scale": 1,
    "realtime_output_format": "webp",
    "output_compression": 95,
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "camera_actions",
    "event_id": 1,
    "payload": {"mode": "state", "transitions": [{"actions": ["w"], "client_ts_ms": 0}]},
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "prompt",
    "event_id": 2,
    "payload": "A quiet snowy Tokyo alley, stable camera motion.",
}))

ws.send_binary(msgspec.msgpack.encode({
    "type": "event",
    "kind": "composite_input",
    "event_id": 3,
    "payload": {
        "input_types": ["prompt", "camera_actions"],
        "prompt": "A quiet neon Shanghai alley, stable forward camera motion.",
        "camera_actions": [["w"], ["w"], []],
    },
}))

5. Consistency

LingBot World 2.0 uses raw-frame websocket GT plus per-chunk latency guards for consistency checks.

6. Notes

• Use the realtime endpoint for interactive sessions: /v1/realtime_video/generate.
• Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks.
• Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior.