Decision models

Answer typed choice, score, and yes or no questions with a probability for every option from a chat model, without generating text.

/v1/decisions turns a chat model into a decision model. You send an input and a list of typed questions, and each answer comes back with the probability of every option, read from the model’s next-token scores at the answer position. No text is generated and no output is parsed. It needs no special checkpoint: any generation model served with a Jinja chat template can answer, as long as the checks below pass for it.

/v1/decisions is an SGLang extension under /v1, like /v1/score and /v1/rerank. It is not part of the OpenAI API, so call it over HTTP rather than through an OpenAI SDK method. Until a release contains it, install a nightly build, which is built from the main branch.

Supported models

Model	Example HuggingFace identifier	Notes
Qwen3.8-27B	Qwen/Qwen3.8-27B	Validated on one H200 in BF16, with and without NEXTN speculative decoding. See the model page.
Qwen3.5-35B-A3B	Qwen/Qwen3.5-35B-A3B	Validated on one H200 in BF16. See the model page.

Other chat models are served when the answer labels are single tokens at the answer position and the answer does not start inside a reasoning block. The server checks these per request and returns a 400 that names the reason when one fails, see Errors. It does not check how the chat template renders the question or where the template itself would put an answer, so inspect prompt_token_ids from return_prompt_token_ids before relying on a new model.

Launch command

Command

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --host 127.0.0.1 --port 30000

No extra flag is needed. The same server keeps serving /v1/chat/completions and /generate, and decisions can run alongside that traffic. A server launched with --enable-mis or --dllm-algorithm, or one that uses a built-in conversation template instead of the tokenizer’s Jinja template, refuses decisions.

Example request

Example

import requests

URL = "http://127.0.0.1:30000"

response = requests.post(
    f"{URL}/v1/decisions",
    json={
        "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing. I'm losing sales.",
        "questions": [
            {
                "id": "team",
                "type": "choice",
                "question": "Which team should handle this ticket?",
                "options": [
                    {"name": "billing", "description": "Payment or subscription issues"},
                    {"name": "technical", "description": "Bugs or integration problems"},
                    {"name": "sales", "description": "Pricing or account questions"},
                ],
            },
            {
                "id": "frustration",
                "type": "score",
                "question": "How frustrated is the customer?",
                "levels": ["Calm", "Frustrated but civil", "Very angry"],
            },
            {
                "id": "urgent",
                "type": "yes_no",
                "question": "The customer needs an answer today.",
            },
        ],
    },
    timeout=60,
)
response.raise_for_status()
for question_id, answer in response.json()["answers"].items():
    print(question_id, answer)

Each answer contains:

• type: the question type, choice, score, or yes_no.
• probabilities: one value per option name, level index, or yes and no, summing to 1. For a yes or no question, probabilities["yes"] is the answer.
• choice for a choice question (the most probable option) or score for a score question (the probability-weighted mean level index).
• label_mass: the full-vocabulary probability of the answer labels at the answer position. A low value means the model puts most of its probability outside the offered answers.

Questions are answered independently and scored together in one batch. To route the ticket in this example, send it to the chosen team, and hand it to a person when the top probability falls below a threshold you validate on your own labeled tickets.

Image inputs

Both /v1/decisions and /v1/systemone accept an optional top-level images list. Each item uses the same image URL object as image_url in vision chat requests, with url and optional detail, or is a string holding the url alone, such as a data URL or base64 image bytes. Use a vision-capable generation model with its vision encoder loaded and a Jinja template that accepts image parts. Some vision models serve text only unless the server is launched with --enable-multimodal. The native processor must expose image marker metadata, and a custom chat template is refused if it fails to preserve one complete marker per image. The server uses SGLang’s existing image processor and vision encoder, placing the images in list order before the existing text in every question’s user message.

Save an image as screenshot.png and send it to the server launched above:

Example

import base64
from pathlib import Path

import requests

response = requests.post(
    "http://127.0.0.1:30000/v1/decisions",
    json={
        "input": "Use the screenshot to answer the question.",
        "images": [{"url": base64.b64encode(Path("screenshot.png").read_bytes()).decode("ascii")}],
        "questions": [{"id": "menu", "type": "yes_no", "question": "A menu is visible."}],
    },
    timeout=60,
)
response.raise_for_status()
print(response.json()["answers"]["menu"])

You can also supply an image URL. On /v1/systemone, send the same images field over HTTP alongside state and its question map. Images are shared by all questions. Objects and arrays in input or state remain JSON text, and text-only prompt wording stays at version 1. Image requests are refused when --allow-auto-truncate is enabled because truncation could remove the answer position. return_prompt_token_ids is unsupported with images because token IDs alone cannot replay vision conditioning through /v1/score.

Image requests are also refused when input, state or a question contains the model’s own image, video or audio marker text, because the processor would read that text as media. Each question of an image request is prepared and scheduled on its own, so a probability near a tie can differ slightly between identical requests. Launch the server with --enable-deterministic-inference when image answers must repeat exactly, as described in deterministic inference.

How answers are computed

The server renders each question as one user message with the model’s chat template and thinking turned off, and labels the answers A to Z for options, 0 to 9 for levels, and yes and no. It checks that each label is one distinct token at the answer position, then runs one prefill pass per question through the scoring path of /v1/score and reads the next-token log-probabilities of the labels over the full vocabulary. For a yes or no question with log-probabilities lp_yes and lp_no and request temperature T:

• probabilities["yes"] = exp(lp_yes / T) / (exp(lp_yes / T) + exp(lp_no / T)), which equals a softmax of the two label logits divided by T, because the vocabulary normalizer cancels.
• label_mass = exp(lp_yes) + exp(lp_no), which does not depend on T.

Choice and score questions work the same way over their labels. None of these values is a calibrated probability that the decision is correct. Validate any threshold on labeled data from your workload.

Request and response reference

Request fields:

Field	Type	Meaning
input	string, object, or array	What the questions are about. Objects and arrays are rendered as compact JSON. Must not be blank.
images	list of image URL objects, default empty	Images shared by all questions, in list order. See Image inputs.
questions	list of questions	At least one. Each question has a unique id, which keys its answer.
temperature	number above 0, default 1	Divides the label logits before the softmax over the labels. It does not change label_mass.
chat_template_kwargs	object, default empty	Extra chat template arguments, applied after --default-chat-template-kwargs. The thinking toggle that the chat template or its reasoning parser names stays off in either case.
prompt_format_version	integer, optional	Pins the prompt wording. A server that serves another version returns 400.
return_prompt_token_ids	boolean, default false	Adds prompt_token_ids and label_token_ids to each text-only answer. Unsupported with images.
model	string, optional	Echoed in the response, as default when omitted. It does not select a LoRA adapter, and the base:adapter form returns 400.

Question fields:

Type	Fields	Labels
choice	id, question, and options: 2 to 26 objects with a name and an optional description	A to Z in list order
score	id, question, and levels: 2 to 10 level descriptions, lowest first	0 to 9
yes_no	id, question, and optional yes and no descriptions	yes and no

question, descriptions, and levels can also be JSON objects or arrays. Each question becomes one user message: the input, a blank line, the question line, one line per option, level, or described yes or no answer, and a closing instruction to answer with one label only. An option renders as A: billing or A: billing - <description>, a level as 0: <description>, and a described yes or no answer as yes: <description>.

The response has object set to decisions, model, prompt_format_version, answers keyed by question id, and usage with prompt_tokens for all questions and completion_tokens 0.

Errors

The request fails with HTTP 400 for:

• an unknown question type or an unknown field anywhere in the request
• option or level counts outside the ranges above
• a blank input, question, question id, or level
• a repeated question id, or option names that are blank, contain control or line break characters, or repeat another name after trimming and case folding
• a label that is not one distinct token at the answer position for the served tokenizer
• a prompt that does not fit the model’s context length
• a chat template that always thinks before answering or starts every answer with a reasoning block, chat_template_kwargs that set the thinking toggle to anything but false, a rendered prompt that leaves a reasoning block open, or a model whose reasoning parser expects answers to start inside a reasoning block when the rendered prompt does not close one
• a prompt_format_version other than the served one, or a model that names a LoRA adapter
• a server launched with --enable-mis or --dllm-algorithm, or one that uses a built-in conversation template, whether named with --chat-template, loaded from a JSON template file, or inferred from the model path
• a model whose chat route uses a built-in encoder instead of a chat template, a tokenizer that does not encode the rendered chat text back to the same ids, or a model that is not a generation model
• a chat template that raises an error for the question message, or a server launched with --skip-tokenizer-init
• a decision checkpoint such as PPLX-Decider-v1-27B, or a joint schema or pointer head checkpoint such as Clef or Kev, which answer through /v1/systemone

Errors about one question name its id or its position in questions.

Pin and replay the prompt

The server owns the prompt wording of /v1/decisions and versions it. Every response carries prompt_format_version, and a change to the wording ships as a new version. Send prompt_format_version to fail loudly instead of receiving answers from different wording after a server upgrade.

To keep a text-only answer reproducible independently of the server’s wording, ask for the scored ids and replay them through /v1/score, which only scores the ids you send:

Example

import requests

URL = "http://127.0.0.1:30000"

body = {
    "input": "The integration keeps failing.",
    "questions": [{"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."}],
    "prompt_format_version": 1,
    "return_prompt_token_ids": True,
}
answer = requests.post(f"{URL}/v1/decisions", json=body, timeout=60).json()["answers"]["urgent"]
replay = requests.post(
    f"{URL}/v1/score",
    json={
        "query": [],
        "items": [answer["prompt_token_ids"]],
        "label_token_ids": [answer["label_token_ids"]],
        "apply_softmax": True,
        "return_token_logprobs": True,
    },
    timeout=60,
).json()
print(answer["probabilities"], replay["scores"][0])

The replayed scores equal the answer’s probabilities when both requests use the same temperature and run in the same cache state. If the decision set temperature, send the same value in the /v1/score body. A replay right after the decision reuses its cached prefix, so it can differ slightly, as described in Reproducibility.

Thinking models

Qwen3.8-27B and Qwen3.5-35B-A3B think by default. /v1/decisions turns thinking off for every question, refuses a request that turns it back on, and refuses a rendered prompt that leaves a reasoning block open, so the answer is never read inside the reasoning. Chat requests to the same server keep the model’s default, and launching with --reasoning-parser qwen3 affects chat only.

Reproducibility

These models mix full and linear attention. Answer probabilities can move by up to several hundredths (0.07 in our checks), and label_mass by up to about 0.14, between cold and prefix-cached requests and across batch compositions, while the chosen option stayed the same in our checks. On Qwen3.5-35B-A3B, --disable-radix-cache gave identical values across sequential repeats, at the cost of prefix reuse.

System One compatible API

POST /v1/systemone serves the same decisions in the request and response shape of the System One API, with a state, a map of noul, choice, and score questions keyed by your ids, and one answer per id. Clients written for that API, including the official TypeSafe SDKs, get decisions by pointing their base URL at the server, with the exceptions listed below. For ordinary generation models, the route uses the same rendering, label checks, and scoring as /v1/decisions, and refuses the same servers, chat templates, and prompts, listed in Errors. Its request fields are checked as described below. A decision checkpoint such as PPLX-Decider-v1-27B instead gets the prompt, answer codes, and calibrated temperature it was trained with. When its decision_config.json sets attention_mode to noncausal_full_attention, as PPLX-Decider-v1.1-27B does, its full-attention layers read the whole prompt, and the server turns off the radix cache and chunked prefill, which would show those layers only part of it. A joint schema checkpoint such as Clef answers all questions from one prompt in one prefill, scored by its joint schema head, and does not use the chat template. A pointer head checkpoint such as Kev answers each question from its own prompt after the shared state, scored by its pointer head, and does not use the chat template either.

Command

pip install typesafe-sdk

Example

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

# A server started without --api-key accepts any key, so a placeholder works.
client = TypeSafeClient(base_url="http://127.0.0.1:30000", api_key="unused")
result = client.system_one(
    "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.",
    {
        "team": Choice(
            instructions="Which team should handle this ticket?",
            criteria={"billing": None, "technical": "Bugs or integration problems", "sales": None},
        ),
        "urgent": Noul(instructions="The customer needs an answer today."),
        "frustration": Score(
            instructions="How frustrated is the customer?",
            criteria=["Calm", "Frustrated but civil", "Very angry"],
        ),
    },
)
for name, answer in result.answers.items():
    print(name, answer)

What to expect from this route:

• Any model name is accepted, including the SDK default jev-latest, unless it names a LoRA adapter after a colon, as in base:adapter. The response model is always the served model.
• A noul answer is the probability of yes. A choice answer has the most probable option and a score answer the probability-weighted mean level, each with probabilities and confidence, and a score answer echoes its levels in legend.
• confidence follows the formulas TypeSafe publishes for its adapter. It is a statistic of the returned probabilities, which are renormalized over the answer labels and ignore probability outside them. It is not calibrated, it can read 0 for a score split between distant levels, and thresholds tuned on another provider may not transfer.
• For a model scored with answer-label tokens, each answer also carries x_label_mass, the full-vocabulary probability of its labels, which /v1/decisions returns as label_mass. The Python SDK drops fields it does not define, so read it from result.raw_http_response.json(). Decision checkpoints and joint schema checkpoints omit it, because their readout or head scores only the answers.
• A choice takes 1 to 255 options and a score 1 to 10 levels. Option names that are blank, contain control or line break characters, or repeat another name after trimming and case folding are refused, and so are blank or null levels, as on /v1/decisions.
• For general generation models that score answer labels, options beyond 26 get two-letter labels (AA, AB, and so on), each checked to be one token for the served tokenizer. These labels include common words and have unequal priors, so answers can depend on option order. Supporting more than 26 options also requires a Hugging Face tokenizer that reports its added tokens and a compatible chat-template boundary, as the Qwen models have with </think>. Without those properties, these models are limited to 26 options. Clef’s joint schema head scores the options directly and does not use letter labels.
• instructions are optional. For general generation models, a choice or score question without them drops its question line; Clef uses the question id as its instructions. A noul question is answered from its true and false descriptions, and one with neither descriptions nor instructions, such as the SDK’s bare Noul(), is refused.
• Unknown top-level fields are ignored. Unknown keys inside a question or inside noul criteria are refused, and so are the /v1/decisions fields temperature, prompt_format_version, and return_prompt_token_ids with any value other than null. images is accepted as an SGLang extension. Clef rejects nonempty chat_template_kwargs because it uses the prompt its head was trained with; other models check this field against their own template rules.
• Unlike /v1/decisions, an empty state and empty or blank question ids are accepted, as the System One schema allows.
• Invalid requests return 422 with the location of the error, other refusals return 400, and neither is retried by the SDKs.
• usage.input_tokens counts the prompt of every question, so the state is counted once per question, or once for a joint schema checkpoint, and usage.output_tokens is 0.
• The SDKs’ models.list() and response request_id are not supported, because GET /v1/models keeps its OpenAI format and no request id header is sent.
• The SDKs time out after 10 seconds and retry, so raise their timeout for requests with many questions over a long state. A request whose client gives up still finishes its encoding and part of its prefill, so each retry repeats at least the encoding.

Build decisions without /v1/decisions

When you need your own prompt or labels, build the prompt on the client and score it through /v1/score, which takes token ids and one label list per item. This needs transformers and jinja2 on the client. Every label must be exactly one token at the answer position:

Example

import requests
from transformers import AutoTokenizer

MODEL = "Qwen/Qwen3.8-27B"
URL = "http://127.0.0.1:30000"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

options = ["billing", "technical", "sales"]
letters = ["A", "B", "C"]
prompt = (
    "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.\n\n"
    "Question: Which team should handle this ticket?\n"
    + "\n".join(f"{letter}: {name}" for letter, name in zip(letters, options))
    + "\nAnswer with the letter of one option only."
)
text = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}], add_generation_prompt=True, enable_thinking=False, tokenize=False
)
input_ids = tokenizer.encode(text, add_special_tokens=False)
label_ids = []
for letter in letters:
    ids = tokenizer.encode(text + letter, add_special_tokens=False)
    if ids[: len(input_ids)] != input_ids or len(ids) != len(input_ids) + 1:
        raise ValueError(f"Label {letter} is not one token at the answer position")
    label_ids.append(ids[-1])

scores = requests.post(
    f"{URL}/v1/score",
    json={"query": [], "items": [input_ids], "label_token_ids": [label_ids], "apply_softmax": True},
    timeout=60,
).json()["scores"][0]
print(dict(zip(options, scores)))

enable_thinking=False renders a closed think block, so the answer token follows it directly. Do not score a prompt that leaves the think block open, because the scored position then falls inside the reasoning.

Structured output gives the same decision through generation. A one-token regex such as (A|B|C) with temperature 0 and logprobs returns probabilities renormalized over the labels, equal to the /v1/score values for one-character labels in the same cache state. A JSON schema with enum fields fills several fields in one response, but gives no clean per-option probability, because an enum value can span several tokens. See Structured outputs for the request formats.

Limitations

• Labels must be one token at the answer position. /v1/decisions assigns one-token labels for you, and a client-built prompt must do the same.
• For yes_no, label_mass counts only the lowercase yes and no tokens. The model also puts probability on Yes and No, so this value reads lower than for choice and score questions even on clear cases, while probabilities["yes"] is unaffected.
• The server tells from the chat template and its reasoning parser whether answers could start inside a reasoning block, using the parser’s reasoning tags in the generation prompt and in the template’s own rendering of a finished answer. Reasoning that shows neither is not detected, so check prompt_token_ids for such a model. Some templates of models that do not reason still contain reasoning tags without ever closing a block in the generation prompt, and those models are refused.
• A default temperature in --preferred-sampling-params can reach scoring requests and scale both the probabilities and label_mass when a decision shares a batch with generation or runs under speculative decoding. Leave it unset for decisions.
• The server tokenizes each question’s full prompt once on the HTTP event loop, as /v1/score does for its items, and lets other requests run between questions. Tokenizers whose label check falls back to the full prompt encode it once more per label.
• A decision takes a single input, not a chat history. To decide about a conversation, pass the history as the input text or as a JSON array.