> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sglang.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Kimi-K2.6

<Note>
  This page focuses on optimal configuration and benchmark results for Kimi-K2.6 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Kimi-K2.6 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6).

  On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware) for details.
</Note>

### Low Latency

| Model     | Hardware      | Cards | Deploy Mode | Dataset   | TPOT | TTFT | Quantization | Configuration                                                 |
| --------- | ------------- | ----- | ----------- | --------- | ---- | ---- | ------------ | ------------------------------------------------------------- |
| Kimi-K2.6 | Atlas 800I A3 | 8     | PD Mixed    | 3.5k+1.5k | 20ms | -    | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) |

### High Throughput

| Model     | Hardware      | Cards | Deploy Mode       | Dataset                              | TPOT  | TTFT | Quantization | Configuration                                                                 |
| --------- | ------------- | ----- | ----------------- | ------------------------------------ | ----- | ---- | ------------ | ----------------------------------------------------------------------------- |
| Kimi-K2.6 | Atlas 800I A3 | 16    | PD Mixed          | 64k+1k                               | 100ms | -    | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms)                |
| Kimi-K2.6 | Atlas 800I A3 | 16    | PD Disaggregation | 128k+1k                              | 100ms | -    | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms)          |
| Kimi-K2.6 | Atlas 800I A3 | 16    | PD Disaggregation | 128k+1k (90% prefix cache hit rate)  | 100ms | 5s   | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) |
| Kimi-K2.6 | Atlas 800I A3 | 16    | PD Disaggregation | 64k+1.5k                             | 100ms | -    | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms)          |
| Kimi-K2.6 | Atlas 800I A3 | 16    | PD Disaggregation | 64k+1.5k (90% prefix cache hit rate) | 100ms | 3s   | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) |
| Kimi-K2.6 | Atlas 800I A3 | 8     | PD Mixed          | 1024x1024 (30)+1024                  | 50ms  | -    | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1024x1024-30-out1024-50ms)       |
| Kimi-K2.6 | Atlas 800I A3 | 8     | PD Mixed          | 1080p\_30+256                        | 50ms  | -    | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1080p-30-out256-50ms)            |
| Kimi-K2.6 | Atlas 800I A3 | 8     | PD Mixed          | 3.5k+1.5k                            | 50ms  | -    | W4A8 INT8    | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms)                 |

## Optimal Configuration

<a id="multi-node-pd-mixed" title="Referenced by external docs. Verify before removing." />

### Kimi-K2.6 W4A8 16P IN64K OUT1K 100ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 16

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 64k+1k

**TPOT**: 100ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   NODE_IPS: IP addresses of each node in the cluster
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

NODE_IPS=('<your node1 ip>' '<your node2 ip>')

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=4400
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"

for i in "${!NODE_IPS[@]}";
do
    if [[ "$LOCAL_HOST1" == "${NODE_IPS[$i]}" || "$LOCAL_HOST2" == "${NODE_IPS[$i]}" ]];
    then
        echo "${NODE_IPS[$i]}"
        python3 -m sglang.launch_server \
        --model-path $MODEL_PATH \
        --host ${NODE_IPS[$i]} --port 6688 \
        --nnodes 2 \
        --dist-init-addr ${NODE_IPS[0]}:5000 \
        --node-rank $i \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --quantization modelslim \
        --dtype bfloat16 \
        --tp-size 32 \
        --mem-fraction-static 0.662 \
        --max-running-requests 32 \
        --chunked-prefill-size 262144 \
        --context-length 75000 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --enable-dp-attention \
        --dp-size 32 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs 1 \
        --disable-radix-cache \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        break
    fi
done
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 32 \
    --random-input-len 64000 \
    --random-output-len 1000 \
    --num-prompts 32 \
    --random-range-ratio 1
```

<a id="pd-disaggregation" title="Referenced by external docs. Verify before removing." />

### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K 100ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k

**TPOT**: 100ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=8
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export ZBAL_ENABLE_GRAPH=1
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --disable-radix-cache \
        --mem-fraction-static 0.78 \
        --max-running-requests 2 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 2 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend
        NODE_RANK=$i
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 2 \
        --enable-dp-attention \
        --dp-size 1 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs 1 2 4 6 8 16 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        NODE_RANK=$i
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --random-input-len 128000 \
    --random-output-len 1000 \
    --num-prompts 1 \
    --random-range-ratio 1 \
    --request-rate inf
```

### Kimi-K2.6 W4A8 1P1D 16P IN128K OUT1K PREFIX90 100ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 128k+1k (90% prefix cache hit rate)

**TPOT**: 100ms

**TTFT**: 5s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=8
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0
        export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24699
        export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184
        export ZBAL_ENABLE_GRAPH=1
        export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.78 \
        --max-running-requests 2 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 2 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        NODE_RANK=$i
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 2 \
        --enable-dp-attention \
        --dp-size 1 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs 1 2 4 6 8 16 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 3 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 4 \
        --speculative-draft-model-quantization unquant \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        NODE_RANK=$i
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 115200` = `int(128000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 12800` = `int(128000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 4 \
    --gsp-system-prompt-len 115200 \
    --gsp-question-len 12800 \
    --gsp-output-len 1000 \
    --max-concurrency 1 \
    --num-prompts 4 \
    --request-rate inf
```

### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 100ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 64k+1.5k

**TPOT**: 100ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1800
        export HCCL_SOCKET_IFNAME=<network-interface>

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --disable-radix-cache \
        --disable-cuda-graph \
        --mem-fraction-static 0.78 \
        --max-running-requests 1 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 1 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        NODE_RANK=$i
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 16 \
        --enable-dp-attention \
        --dp-size 1 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs 16 \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 4 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 5 \
        --speculative-draft-model-quantization unquant
        NODE_RANK=$i
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 1 \
    --random-input-len 64000 \
    --random-output-len 1500 \
    --num-prompts 1 \
    --random-range-ratio 1 \
    --request-rate inf
```

### Kimi-K2.6 W4A8 1P1D 16P IN64K OUT1K5 PREFIX90 100ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 16

**Deploy Mode**: PD Disaggregation

**Quantization**: W4A8 INT8

**Dataset**: 64k+1.5k (90% prefix cache hit rate)

**TPOT**: 100ms

**TTFT**: 3s

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   P_IP: prefill node IP address
#   D_IP: decode node IP address
#   ASCEND_MF_STORE_URL: prefill node IP with port
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================


echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=60
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

P_IP=('<your prefill ip>')
D_IP=('<your decode ip>')

export ASCEND_MF_STORE_URL="tcp://<your prefill ip>:24670"

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
    then
        echo "${P_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1800
        export HCCL_SOCKET_IFNAME=<network-interface>

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode prefill \
        --host ${P_IP[$i]} \
        --port 8000 \
        --disaggregation-bootstrap-port 8998 \
        --node-rank 0 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.78 \
        --max-running-requests 2 \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --chunked-prefill-size 16384 \
        --prefill-max-requests 2 \
        --max-prefill-tokens 65536 \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2
        NODE_RANK=$i
        break
    fi
done

# decode
for i in "${!D_IP[@]}";
do
    if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
    then
        echo "${D_IP[$i]}"
        export GLOO_SOCKET_IFNAME=<network-interface>
        export HCCL_BUFFSIZE=1200
        export HCCL_SOCKET_IFNAME=<network-interface>
        export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
        export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
        export SGLANG_NPU_USE_MLAPO=1
        export SGLANG_NPU_USE_MULTI_STREAM=1

        python3 -m sglang.launch_server \
        --model-path ${MODEL_PATH} \
        --disaggregation-mode decode \
        --host ${D_IP[$i]} \
        --port 8001 \
        --quantization modelslim \
        --dtype bfloat16 \
        --disaggregation-transfer-backend ascend \
        --nnodes 1 \
        --trust-remote-code \
        --attention-backend ascend \
        --device npu \
        --tp-size 16 \
        --mem-fraction-static 0.82 \
        --max-running-requests 2 \
        --enable-dp-attention \
        --dp-size 2 \
        --enable-dp-lm-head \
        --disable-radix-cache \
        --enable-multimodal \
        --mm-attention-backend ascend_attn \
        --sampling-backend ascend \
        --moe-a2a-backend deepep \
        --deepep-mode auto \
        --cuda-graph-bs 1 2 4 6 8 \
        --reasoning-parser kimi_k2 \
        --tool-call-parser kimi_k2 \
        --speculative-algorithm EAGLE3 \
        --speculative-draft-model-path $DRAFT_MODEL_PATH \
        --speculative-num-steps 4 \
        --speculative-eagle-topk 1 \
        --speculative-num-draft-tokens 5 \
        --speculative-draft-model-quantization unquant
        NODE_RANK=$i
        break
    fi
done
```

```bash Command theme={null}
# ============================================================
# Before running, replace the following placeholders:
#   <your prefill ip>: prefill node IP address
#   <your decode ip>: decode node IP address
# ============================================================

python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill http://<your prefill ip>:8000 8998 \
    --decode http://<your decode ip>:8001 \
    --host 127.0.0.1 \
    --port 6688 \
    --policy cache_aware
```

#### Benchmark

We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`):
`--gsp-system-prompt-len 57600` = `int(64000 * 0.9)` is the shared prefix portion.
`--gsp-question-len 6400` = `int(64000 * (1 - 0.9))` is the unique per-request suffix.
`--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name generated-shared-prefix \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --gsp-num-groups 1 \
    --gsp-prompts-per-group 8 \
    --gsp-system-prompt-len 57600 \
    --gsp-question-len 6400 \
    --gsp-output-len 1500 \
    --max-concurrency 2 \
    --num-prompts 8 \
    --request-rate inf
```

<a id="single-node-pd-mixed" title="Referenced by external docs. Verify before removing." />

### Kimi-K2.6 W4A8 8P IN1024X1024 30 OUT1024 50ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 1024x1024 (30)+1024

*Format: resolution (input tokens) + output tokens*

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=1500
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --quantization modelslim \
    --dtype bfloat16 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --trust-remote-code \
    --device npu \
    --attention-backend ascend \
    --tp-size 16 \
    --mem-fraction-static 0.872 \
    --max-running-requests 176 \
    --chunked-prefill-size 32768 \
    --context-length 8192 \
    --max-prefill-tokens 16384 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --enable-dp-attention \
    --dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 4 8 9 10 11 \
    --disable-radix-cache \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 2 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 3 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 200 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1024x1024 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 160 \
    --random-input-len 30 \
    --random-output-len 1024 \
    --num-prompts 640 \
    --random-range-ratio 1 \
    --request-rate inf \
    --warmup-requests 16 \
    --image-count 1 \
    --image-resolution 1024x1024
```

### Kimi-K2.6 W4A8 8P IN1080P 30 OUT256 50ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 1080p\_30+256

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=2400
export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --quantization modelslim \
    --dtype bfloat16 \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --trust-remote-code \
    --device npu \
    --attention-backend ascend \
    --tp-size 16 \
    --base-gpu-id 0 \
    --mem-fraction-static 0.852 \
    --max-running-requests 64 \
    --chunked-prefill-size 16384 \
    --context-length 8192 \
    --max-prefill-tokens 16384 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --enable-dp-attention \
    --dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 3 4 \
    --disable-radix-cache \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 2 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 3 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 200 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `IMAGE` dataset with 1920x1080 resolution.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name image \
    --backend sglang-oai-chat \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 48 \
    --random-input-len 30 \
    --random-output-len 256 \
    --num-prompts 196 \
    --random-range-ratio 1 \
    --request-rate inf \
    --warmup-requests 16 \
    --image-count 1 \
    --image-resolution 1920x1080
```

### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 20ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 20ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1200
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --dtype bfloat16 \
    --tp-size 16 \
    --mem-fraction-static 0.865 \
    --max-running-requests 80 \
    --chunked-prefill-size 32768 \
    --context-length 6144 \
    --max-prefill-tokens 65536 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --enable-dp-attention \
    --dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 3 4 5 \
    --disable-radix-cache \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 200 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 64 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --num-prompts 256 \
    --random-range-ratio 1 \
    --warmup-requests 0
```

### Kimi-K2.6 W4A8 8P IN3K5 OUT1K5 50ms

**Model**: Kimi-K2.6

**Hardware**: Atlas 800I A3

**Cards**: 8

**Deploy Mode**: PD Mixed

**Quantization**: W4A8 INT8

**Dataset**: 3.5k+1.5k

**TPOT**: 50ms

#### Model Deployment

```bash Command theme={null}
# ============================================================
# Before running, update the following variables:
#   MODEL_PATH: path to the model weights directory
#   DRAFT_MODEL_PATH: path to the draft model weights directory
#   HCCL_SOCKET_IFNAME: network interface name for HCCL
#   GLOO_SOCKET_IFNAME: network interface name for Gloo
# ============================================================

MODEL_PATH=/path/to/model-weights
DRAFT_MODEL_PATH=/path/to/draft-model-weights

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000

unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh

export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export GLOO_SOCKET_IFNAME=<network-interface>
export HCCL_BUFFSIZE=1200
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=<network-interface>
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_SET_CPU_AFFINITY=1
export STREAMS_PER_DEVICE=32

python3 -m sglang.launch_server \
    --model-path $MODEL_PATH \
    --host 127.0.0.1 --port 6688 \
    --trust-remote-code \
    --attention-backend ascend \
    --device npu \
    --quantization modelslim \
    --dtype bfloat16 \
    --tp-size 16 \
    --mem-fraction-static 0.895 \
    --max-running-requests 208 \
    --chunked-prefill-size 32768 \
    --context-length 6144 \
    --max-prefill-tokens 16384 \
    --enable-multimodal \
    --mm-attention-backend ascend_attn \
    --sampling-backend ascend \
    --enable-dp-attention \
    --dp-size 16 \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-bs-decode 1 2 4 8 12 13 \
    --disable-radix-cache \
    --model-loader-extra-config '{"enable_multithread_load": true}' \
    --speculative-algorithm EAGLE3 \
    --speculative-draft-model-path $DRAFT_MODEL_PATH \
    --speculative-num-steps 4 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 5 \
    --speculative-draft-model-quantization unquant \
    --prefill-delayer-max-delay-passes 200 \
    --enable-prefill-delayer \
    --reasoning-parser kimi_k2 \
    --tool-call-parser kimi_k2
```

#### Benchmark

We tested it based on the `RANDOM` dataset.

```bash Command theme={null}
python -m sglang.bench_serving \
    --dataset-name random \
    --backend sglang \
    --host 127.0.0.1 \
    --port 6688 \
    --max-concurrency 192 \
    --random-input-len 3500 \
    --random-output-len 1500 \
    --num-prompts 768 \
    --random-range-ratio 1 \
    --warmup-requests 0
```