> ## Documentation Index > Fetch the complete documentation index at: https://docs.sglang.io/llms.txt > Use this file to discover all available pages before exploring further. # Qwen3.5-397B-A17B ## Introduction Qwen3.5-397B-A17B is the latest flagship model in the Qwen series developed by Alibaba, featuring a Gated Delta Networks combined with sparse Mixture-of-Experts architecture (397B total parameters, 17B activated). It employs hybrid attention with Gated Delta Networks (linear, O(n) complexity) combined with full attention every 4th layer, and MoE routing with Top-10 active out of 512 routed experts plus a dedicated shared expert. The model supports multimodal inputs (text, image, video) with native context lengths of up to 262,144 tokens, and includes built-in multi-token prediction (MTP) for speculative decoding. This document demonstrates the deployment of Qwen3.5-397B-A17B on Ascend NPUs using SGLang, including single-node PD mixed mode, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.13**. The current model (Qwen3.5-397B-A17B) is fully supported in this version. To use the latest features (e.g., speculative decoding, multimodal), it is recommended to use v0.5.13 or a later version. ## Supported features | Feature | Example usage | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Tensor Parallelism | `--tp-size 16` | | Data Parallelism | `--dp-size 8` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment mode, and workload. For parameter details, see [Feature descriptions](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-descriptions); for recommended configurations for each deployment scenario, see [Best practices](#best-practices). For feature compatibility and conflict information between features, see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-compatibility). ## Prerequisites ### Environment Before following this tutorial, complete the environment setup in the documents below: * [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — the fastest way to get started. It walks you through launching the official container image, starting the SGLang server, and sending a test request. Recommended if you are new to SGLang on Ascend. * [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide. It covers the component version mapping (CANN, TorchNPU, Triton, kernels, etc.), building from source or from a Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize the environment instead of using the official image. ### Model weights Before downloading model weights, check the model size to reserve enough disk space. For multi-node deployment, download the weights to a shared directory accessible to all nodes. * [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP, 235.88GB) The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/ascend_npu_reference#hardware). ## Installation Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it. Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/ascend_npu_faq#8-docker-image-versions-stable-release-vs-daily-build). ```bash Command theme={null} docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3 docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci8:/dev/davinci8 \ --device=/dev/davinci9:/dev/davinci9 \ --device=/dev/davinci10:/dev/davinci10 \ --device=/dev/davinci11:/dev/davinci11 \ --device=/dev/davinci12:/dev/davinci12 \ --device=/dev/davinci13:/dev/davinci13 \ --device=/dev/davinci14:/dev/davinci14 \ --device=/dev/davinci15:/dev/davinci15 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3 ``` ```bash Command theme={null} docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b docker run -itd --shm-size=16g --name ${NAME} \ --privileged=true --net=host \ -v /var/queue_schedule:/var/queue_schedule \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v /usr/local/sbin:/usr/local/sbin \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --device=/dev/davinci0:/dev/davinci0 \ --device=/dev/davinci1:/dev/davinci1 \ --device=/dev/davinci2:/dev/davinci2 \ --device=/dev/davinci3:/dev/davinci3 \ --device=/dev/davinci4:/dev/davinci4 \ --device=/dev/davinci5:/dev/davinci5 \ --device=/dev/davinci6:/dev/davinci6 \ --device=/dev/davinci7:/dev/davinci7 \ --device=/dev/davinci_manager:/dev/davinci_manager \ --device=/dev/hisi_hdc:/dev/hisi_hdc \ --entrypoint=bash \ quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b ``` * If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the container, for example: `-v /path/to/models:/models`. * Replace `${NAME}` with your own container name or remove `--name` to use default name. ## Online service deployment ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see [Qwen3.5-397B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b#single-node-pd-mixed). ## Functional verification After the service is started, you can invoke the model by sending a prompt: ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", "sampling_params": { "max_new_tokens": 64, "temperature": 0 } }' ``` Expected result: an HTTP 200 response with the generated text containing "Paris". For multimodal requests (text + image): ```shell theme={null} # ============================================================ # Before running, update the following variables: # HOST: the server host address (e.g., localhost) # PORT: the server port number (e.g., 6688) # ============================================================ curl http://${HOST}:${PORT}/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3.5-397B-A17B-w4a8-mtp", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, {"type": "text", "text": "Describe this image."} ] } ] }' ``` Expected result: an HTTP 200 response with a description of the image. Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more testing examples (Health Check, Generate, Chat Completions, and port usage guidance), see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). ## Accuracy evaluation For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation). ## Performance For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing). ## Best practices ### Best practice configuration reference For complete optimal configurations with deployment scripts and benchmark commands, see the [Qwen3.5-397B Best Practice](/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b) page. ## Performance tuning For the full list of supported features, see [Supported features](#supported-features). For detailed optimization guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization). ## FAQ For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/ascend_npu_faq).