AMD GPUs

This document describes how run SGLang on AMD GPUs. If you encounter issues or have questions, please open an issue.

System Configuration

When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:

• AMD MI300X Tuning Guides
• LLM inference performance validation on AMD Instinct MI300X
• AMD Instinct MI300X System Optimization
• AMD Instinct MI300X Workload Optimization
• Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X

We strongly recommend reading these docs and guides entirely to fully utilize your system.

Below are a few key settings to confirm or enable for SGLang:

Update GRUB Settings

In /etc/default/grub, append the following to GRUB_CMDLINE_LINUX:

GRUB Configuration

pci=realloc=off iommu=pt

Afterward, run sudo update-grub (or your distro’s equivalent) and reboot.

Disable NUMA Auto-Balancing

Disable NUMA

sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'

You can automate or verify this change using this helpful script.

Again, please go through the entire documentation to confirm your system is using the recommended configuration.

Install SGLang

Docker (Recommended)

The docker images are available on Docker Hub at lmsysorg/sglang, built from rocm.Dockerfile.

1. Build the docker image If you use pre-built images, you can skip this step and replace sglang_image with the pre-built image names in the steps below.

   Build Image

   docker build -t sglang_image -f rocm.Dockerfile .
   

2. Create a convenient alias

   Create Alias

   alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
       --ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
       --security-opt seccomp=unconfined \
       -v $HOME/dockerx:/dockerx \
       -v /data:/data'
   

   If you are using RDMA, please note that:

   • --network host and --privileged are required by RDMA. If you don’t need RDMA, you can remove them.
   • You may need to set NCCL_IB_GID_INDEX if you are using RoCE, for example: export NCCL_IB_GID_INDEX=3.

3. Launch the server

   Replace <secret> below with your huggingface hub token.

   Launch Server

   drun -p 30000:30000 \
       -v ~/.cache/huggingface:/root/.cache/huggingface \
       --env "HF_TOKEN=<secret>" \
       sglang_image \
       python3 -m sglang.launch_server \
       --model-path NousResearch/Meta-Llama-3.1-8B \
       --host 0.0.0.0 \
       --port 30000
   

4. Verify the installation You can run a benchmark in another terminal or refer to other docs to send requests to the engine.

   Run Benchmark

   drun sglang_image \
   python3 -m sglang.bench_serving \
   --backend sglang \
   --dataset-name random \
   --num-prompts 4000 \
   --random-input 128 \
   --random-output 128
   

With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities.

From Source

1. Clone the repository Clone the SGLang repository.

   git clone https://github.com/sgl-project/sglang.git
   cd sglang
   

2. Compile sgl-kernel Upgrade pip and compile the sgl-kernel for ROCm support.

   pip install --upgrade pip
   cd sgl-kernel
   python setup_rocm.py install
   

3. Install sglang package Install the SGLang Python package with HIP and diffusion support.

   cd ..
   rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
   pip install -e "python[all_hip]"
   

Examples

Running DeepSeek-V3

The only difference when running DeepSeek-V3 is in how you start the server.

DeepSeek-V3

drun -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    --env "HF_TOKEN=<secret>" \
    sglang_image \
    python3 -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V3 \
    --tp 8 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000

Running DeepSeek-R1 on a single NDv5 MI300X VM could also be a good reference.

Running Llama3.1

Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server.

Llama3.1

drun -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --ipc=host \
    --env "HF_TOKEN=<secret>" \
    sglang_image \
    python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --tp 8 \
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 30000

Warmup Step

When the server displays The server is fired up and ready to roll!, it means the startup is successful.