Use Models From ModelScope

To use a model from ModelScope, set the environment variable SGLANG_USE_MODELSCOPE.

Set Environment Variable

export SGLANG_USE_MODELSCOPE=true

We take Qwen2-7B-Instruct as an example.

Launch the Server

Python

python -m sglang.launch_server --model-path qwen/Qwen2-7B-Instruct --port 30000

ModelScope uses a different cache directory than Hugging Face. You may need to set it manually to avoid running out of disk space.