DeepSeek-R1

Introduction

DeepSeek-R1 is a Mixture-of-Experts (MoE) large language model developed by DeepSeek, featuring 671B total parameters with 37B active parameters. It employs Multi-head Latent Attention (MLA) and DeepSeekMoE architecture, with built-in multi-token prediction (MTP) for speculative decoding. The model excels at reasoning, math, and code tasks through reinforcement learning-based training.

This document demonstrates the deployment of DeepSeek-R1 on Ascend NPUs using SGLang, including single-node PD mixed mode, multi-node PD disaggregation mode, feature configuration, and performance optimization.

This document is validated and written based on SGLang v0.5.13. The current model (DeepSeek-R1) is fully supported in this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use v0.5.13 or a later version.

Supported features

Feature	Example usage
Tensor Parallelism	--tp-size 16
Data Parallelism	--dp-size 16
Expert Parallelism	--ep-size 16 \ --moe-a2a-backend deepep \ --deepep-mode auto
PD Disaggregation	--disaggregation-mode prefill \ --disaggregation-transfer-backend ascend
Quantization	--quantization modelslim
NPU Graph	enabled by default; disable with --disable-cuda-graph; control range via --cuda-graph-bs or --cuda-graph-max-bs-decode; e.g., --cuda-graph-bs 4 8 20 21 22
Speculative Decoding	--speculative-algorithm NEXTN \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3
Overlap Schedule	export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
DP LM Head	--enable-dp-lm-head
MLAPO	export SGLANG_NPU_USE_MLAPO=1
Multistream MoE	export SGLANG_NPU_USE_MULTI_STREAM=1
NZ Weight Format	export SGLANG_USE_FIA_NZ=1

The values in the Example usage column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see Feature descriptions; for recommended configurations for each deployment scenario, see Best practices.

For feature compatibility and conflict information between features, see Feature Compatibility.

Prerequisites

Environment

Before following this tutorial, complete the environment setup in the documents below:

• Ascend NPU Quickstart — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend.
• SGLang Installation with NPU Support — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image.

Model weights

Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes.

• DeepSeek-R1-0528-W4A8 (Quantized version, 376GB)
• DeepSeek-R1-0528-W8A8 (Quantized version, 647GB)

The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (--tp-size 8), which corresponds to one full A2 node or 8 dies on A3 (4 cards).

The W8A8 variant (647GB) can be deployed on 16 × 64GB of device memory (--tp-size 16), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes.

This is the minimum recommended configuration. For optimized configurations, see Best practices, which may require additional cards or nodes.

For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see Ascend NPU Reference — Hardware.

Installation

The Docker image requires at least 30GB of free space. Ensure sufficient disk space before pulling images.

The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.

Both stable releases and daily builds are available. The following command is based on the stable release tag. For details, see Docker image versions.

Atlas 800I A3

Command

docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci8:/dev/davinci8  \
--device=/dev/davinci9:/dev/davinci9  \
--device=/dev/davinci10:/dev/davinci10  \
--device=/dev/davinci11:/dev/davinci11  \
--device=/dev/davinci12:/dev/davinci12  \
--device=/dev/davinci13:/dev/davinci13  \
--device=/dev/davinci14:/dev/davinci14  \
--device=/dev/davinci15:/dev/davinci15  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3

Atlas 800I A2

Command

docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b

docker run -itd --shm-size=16g --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0  \
--device=/dev/davinci1:/dev/davinci1  \
--device=/dev/davinci2:/dev/davinci2  \
--device=/dev/davinci3:/dev/davinci3  \
--device=/dev/davinci4:/dev/davinci4  \
--device=/dev/davinci5:/dev/davinci5  \
--device=/dev/davinci6:/dev/davinci6  \
--device=/dev/davinci7:/dev/davinci7  \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b

• If the model weights have already been downloaded to a shared directory, use -v to mount the model path into the container, for example: -v /path/to/models:/models.
• Replace ${NAME} with your own container name or remove --name to use default name.

Online service deployment

Single-node online deployment

Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see DeepSeek-R1 Best Practice — W4A8 8P PD Mixed On A3.

Multi-node PD disaggregation deployment

PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see DeepSeek-R1 Best Practice — W8A8 32P PD Disaggregation On A3.

Functional verification

After the service is started, you can invoke the model by sending a prompt:

# ============================================================
# Before running, update the following variables:
#   HOST: the server host address (e.g., localhost)
#   PORT: the server port number (e.g., 6689)
# ============================================================

curl http://${HOST}:${PORT}/generate \
    -H "Content-Type: application/json" \
    -d '{
        "text": "What is the capital of France?",
        "sampling_params": {
            "max_new_tokens": 64,
            "temperature": 0
        }
    }'

Expected result: an HTTP 200 response with the generated text containing “Paris”.

Once the server prints The server is fired up and ready to roll! in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see Testing the Service.

Accuracy evaluation

For accuracy evaluation methods and datasets, see Accuracy Evaluation on Ascend NPU.

Performance

For performance data and benchmark commands, see Performance Testing on Ascend NPU.

Best practices

Best practice configuration reference

For complete optimal configurations with deployment scripts and benchmark commands, see the DeepSeek-R1 Best Practice page.

Performance tuning

For the full list of supported features, see Supported features. For detailed optimization guidance, see Optimization on Ascend NPU.

FAQ

For common environment, installation, and general parameter issues, please refer to the Ascend NPU FAQ.