Hy3

Deploy Tencent Hy3 with SGLang — verified launch commands and tuning for the BF16 Mixture-of-Experts model with hybrid thinking, native tool calling, 256K context, and MTP speculative decoding.

Deployment

Install SGLang

For all methods and hardware platforms, see the official SGLang installation guide. The two paths below match the Python / Docker toggle in the command panel.

Python (pip / uv)

pip install -U uv
uv venv --python 3.12 && source .venv/bin/activate

# Install from source (main carries the suffix-aware `hunyuan` parser + the
# HYV3 model code). Once a tagged release picks it up, `uv pip install sglang`
# is enough.
git clone https://github.com/sgl-project/sglang.git
cd sglang
uv pip install -e python

Run the Python output of the command panel below in that environment.

Docker

Command

# The image bundles the HYV3 model code and the suffix-aware `hunyuan` parser.
docker pull lmsysorg/sglang:dev

For how to launch the image, see Install → Method 3: Using Docker, substituting the inner sglang serve ... with what the command generator below produces.

The dev image bundles the HYV3 model code, the suffix-aware hunyuan reasoning/tool-call parsers, and the MTP draft-module runtime. The same parsers serve both the preview (suffix-less) and the shipping (suffixed) Hy3 tokenizer — no per-model hard-coding.

Pick your hardware + recipe to generate the launch command.

• Low-Latency — fastest reply for a single user. Pick for chat.
• Balanced — good speed with several users at once. Use for typical multi-user serving.

Panel controls (top of the command box):

• Python / Docker — bare sglang serve … for an existing SGLang env, or a docker run … sglang serve … wrap against the per-hardware image from the Install SGLang panel above.
• ⧉ Copy — copies the current command (with whichever framing is active) to your clipboard.
• $ cURL — a sample request against localhost:30000 to confirm the server is up.
• ⚙ Env — edits the placeholders (HOST_IP, PORT, HF_TOKEN, NODE_RANK, NODE0_IP) the command and cURL share. Persists in localStorage across cookbooks.
• Verified / Not Verified badge — green when the (hw, variant, quant, strategy, nodes) combo has been run end-to-end on real hardware; yellow when auto-derived from a neighbor and not yet re-checked.

Playground

The Playground lets you turn on additional knobs on top of whichever Deploy cell is currently selected. The base is read live from your Deploy selection — only your overrides change.

The knobs come in two flavors:

• Built-in SGLang features — parallelism overrides (TP / CP / DP-Attention), MoE backend + EP, reasoning / tool-call parsers, speculative-decoding presets, prefill/decode disaggregation, and HiCache tiers.
• Hy3 specific — --tool-call-parser auto / --reasoning-parser auto (auto-detect Hy3’s suffix-aware hunyuan parsers from the chat template; resolve the real special tokens from the tokenizer vocab at runtime).

Lines highlighted green are added by your overrides; lines with red strikethrough were in the verified base but stripped by an override. When no override differs from the base cell, the playground inherits the base’s Verified badge; any actual change flips it to Not Verified until the new configuration is run end-to-end and submitted back.

Panel controls reuse Python / Docker · ⧉ Copy · $ cURL · ⚙ Env from the Deploy panel, plus one extra:

• Submit ↗ — opens a pre-filled GitHub issue so you can land your override combo as a new verified cookbook cell. Shown only while the badge says Not Verified; click it once you’ve actually run the command on your hardware and confirmed it works.

1. Model Introduction

Hy3 is Tencent’s third-generation flagship Mixture-of-Experts language model, featuring hybrid thinking, native tool calling, long-context reasoning, and Multi-Token Prediction (MTP) for low-latency serving.

Key Features:

• MoE Architecture: 192 routed experts + 1 shared expert, top-8 activated per token. 295B total parameters with 21B active (+3.8B MTP layer), delivering dense-model quality at MoE inference cost.
• Hybrid Thinking: Reasoning modes (high, low, no_think) controllable via OpenAI-standard reasoning_effort, allowing the same weights to trade off latency and depth of reasoning.
• Native Tool Calling: Trained on a structured grammar. Pairs with SGLang’s hunyuan tool-call parser for streaming OpenAI-compatible function-calling output.
• Long Context: 256K token context window (262,144 positions) for repository-scale code and document reasoning.
• Multi-Token Prediction (MTP): Ships with a built-in MTP draft module enabling speculative decoding out of the box.

Available Model:

• tencent/Hy3 — BF16 instruct
• tencent/Hy3-FP8 — FP8

Recommended Generation Parameters:

Parameter	Value
temperature	0.9
top_p	1.0
reasoning_effort	high / low (thinking) or no_think (instant)

Special tokens. The shipping Hy3 tokenizer appends a shared suffix to every special token (e.g. <tool_calls:TAG> instead of the bare <tool_calls>). SGLang’s hunyuan parsers resolve the real token strings from the tokenizer vocab at runtime (PR #29920), so the same recipe serves both the preview (suffix-less) and the shipping (suffixed) tokenizer — no per-model hard-coding. This is why --reasoning-parser hunyuan / --tool-call-parser hunyuan work out of the box on the shipping model.

2. Configuration Tips

Hardware requirements (BF16, ~590GB weights):

GPU	VRAM	TP	Notes
H200	141GB	8	minimum single-node for BF16
B200	192GB	4	BF16 590GB → 148GB/GPU
B300 / GB300	288GB	4	BF16 590GB → 148GB/GPU; ample KV headroom
GB200	192GB	4	single-node 4×192GB = 768GB fits BF16 590GB

Blackwell attention backend. On SM100/SM103 (B200 / B300 / GB200 / GB300), SGLang auto-selects the trtllm_mha attention backend for HYV3’s MHA architecture (no flag needed) — the launch commands above omit it for that reason. Override only if you have a specific kernel reason.

MTP (Multi-Token Prediction, EAGLE).

• low-latency: steps=3, draft-tokens=4 → largest win at bs=1.
• balanced: MTP disabled — keep the prefill batch moderate so chunked-prefill stays efficient.

reasoning_effort vs thinking. The Hy3 chat template is driven by reasoning_effort (high / low / no_think), NOT by the thinking flag that some other families use. The default is no_think (instant). To opt into thinking, pass reasoning_effort="high" on the request (the OpenAI-standard field; sglang forwards it to the template). reasoning_effort: max is rejected by sglang — use high. For eval, sgl-eval’s --thinking flag translates to reasoning_effort="high" for Hy3, so the benchmark commands below use it as-is.

3. Advanced Usage

3.1 Reasoning (Hybrid Thinking)

Hy3 is a hybrid-thinking model. Control the thinking budget via reasoning_effort:

• high / low — increasing amounts of chain-of-thought in reasoning_content
• no_think — skip thinking entirely (instant responses, content-only)

Enable the reasoning parser during deployment so the thinking section is separated into reasoning_content:

Deploy with reasoning parser

Command

sglang serve \
  --model-path tencent/Hy3 \
  --tp 8 \
  --reasoning-parser auto \
  --tool-call-parser auto

Example: thinking (reasoning_effort=high)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "Solve step by step: What is 15% of 240?"}],
    reasoning_effort="high",
    max_tokens=2048,
)

msg = response.choices[0].message
print("=============== Thinking =================")
print(msg.reasoning_content)
print("=============== Content =================")
print(msg.content)

Output

=============== Thinking =================
We need to solve: "What is 15% of 240?" Step by step. 15% means 15/100 = 0.15. Multiply 0.15 by 240.
10% of 240 = 24, 5% is half of 10% = 12, so sum = 36. So answer is 36.
=============== Content =================
To find 15% of 240, follow these steps:

1. 15% = 15/100 or 0.15.
2. Multiply 240 by 0.15: 0.15 × 240 = 36.
3. Check: 10% of 240 = 24, 5% = 12, 15% = 36.

Thus, 15% of 240 is 36.

Example: instant mode (reasoning_effort=no_think)

Example

response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "Give me a one-line summary of relativity."}],
    reasoning_effort="no_think",
    max_tokens=256,
)

print("Content:", response.choices[0].message.content)

Output

Content: Relativity is Einstein's theory that space, time, mass, and gravity are interconnected and relative, not fixed, fundamentally changing our understanding of the universe.

3.2 Tool Calling

Hy3 supports streaming OpenAI-compatible tool calls. Enable both parsers together — the reasoning parser strips any thinking tokens before the tool-call parser runs:

Deploy with tool-call parser

Command

sglang serve \
  --model-path tencent/Hy3 \
  --tp 8 \
  --reasoning-parser auto \
  --tool-call-parser auto

Example: non-streaming tool call

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["city"],
            },
        },
    }
]

response = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
)

msg = response.choices[0].message
print("Reasoning:", msg.reasoning_content)
print("Content:  ", msg.content)
for tc in msg.tool_calls or []:
    print(f"Tool Call: {tc.function.name}")
    print(f"  Arguments: {tc.function.arguments}")

Output

Reasoning: None
Content:   I'll get the current weather for Beijing in Fahrenheit for you.
Tool Call: get_weather
  Arguments: {"city": "Beijing", "unit": "fahrenheit"}

Example: streaming tool call (incremental argument deltas)

Example

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

stream = client.chat.completions.create(
    model="tencent/Hy3",
    messages=[{"role": "user", "content": "What's the weather in Beijing? Use fahrenheit."}],
    tools=tools,
    stream=True,
)

tool_buffer = {}
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)
    for tc in delta.tool_calls or []:
        buf = tool_buffer.setdefault(tc.index, {"name": "", "args": ""})
        if tc.function and tc.function.name:
            buf["name"] += tc.function.name
        if tc.function and tc.function.arguments:
            buf["args"] += tc.function.arguments

for idx, buf in tool_buffer.items():
    print(f"\nTool[{idx}] {buf['name']}({buf['args']})")

Output

I'll check the current weather in Beijing for you using Fahrenheit.
Tool[0] get_weather({"city": "Beijing", "unit": "fahrenheit"})