Skip to main content

Chat Completions

Level: Intermediate · Category: API reference

Tags: Streaming · Reasoning effort · Function calling · Structured output · Vision

POST https://openai.rc.asu.edu/v1/chat/completions follows the OpenAI Chat Completions API, so the official openai SDKs, LangChain, LlamaIndex and most other OpenAI-compatible tools work with it once you set the base URL and key. LLM API Access covers keys and a first request. This page covers what a real application needs on top of that.

The Python examples use the openai package and read the key and base URL from the environment:

mamba install -c conda-forge openai
read -rs OPENAI_API_KEY && export OPENAI_API_KEY      # paste your key, then press Enter
export OPENAI_BASE_URL="https://openai.rc.asu.edu/v1"
from openai import OpenAI
client = OpenAI()  # reads OPENAI_API_KEY and OPENAI_BASE_URL

Pick a model​

Each entry in GET /v1/models says what the model can do. The same facts appear as badges under AI LLM › LLMs in the Voyager portal.

FieldMeaning
model_typechat for chat models. Other types (embedding, image, audio_speech, audio_transcription, video) use their own endpoints.
context_windowMaximum prompt plus reply length, in tokens
supports_function_callingAccepts tools
supports_response_schemaEnforces response_format JSON schemas itself
supports_visionReads image_url content parts itself
reasoningHow the model takes reasoning_effort (see Reasoning effort)
familyThe name picks one of several models by reasoning effort
boons, boon_detailsAbilities the gateway adds to the model (see Gateway add-ons)
status, replacement, retire_at, notePresent only on models being retired (see Model retirements)

For example, to list the chat models that enforce JSON schemas:

curl -s https://openai.rc.asu.edu/v1/models -H "Authorization: Bearer $OPENAI_API_KEY" \
  | jq -r '.data[] | select(.model_type=="chat" and .supports_response_schema) | .id'

Streaming​

Set stream: true to receive the reply as server-sent events while it is generated. Add stream_options: {"include_usage": true} to get a final chunk with token counts.

stream = client.chat.completions.create(
    model="gemma4-31b-it",
    messages=[{"role": "user", "content": "Explain what a Slurm job array is in two sentences."}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        print("\n", chunk.usage)

A stream ends with a chunk carrying finish_reason, then the usage chunk if you asked for it, then data: [DONE]. If a stream is cut off (the model server restarts, or the request reaches the 60-minute limit), the last event is an error object instead of [DONE], and the connection closes:

data: {"error":{"message":"upstream stream ended before the response was complete (after 61 s)","type":"obleth_gateway_error","code":502}}

Treat a stream that ends without [DONE] as failed.

Reasoning effort​

Many models can think before they answer. You control that with one scale for every model: reasoning_effort set to none, low, medium or high. The gateway translates the level into whatever the model's server expects, so you don't need model-specific settings such as chat_template_kwargs.

resp = client.chat.completions.with_raw_response.create(
    model="olmo3-32b",
    reasoning_effort="high",
    max_tokens=4000,
    messages=[{"role": "user", "content": "Is 391 prime? Answer yes or no."}],
)
print(resp.headers["x-obleth-reasoning-effort"])   # the level that was served: high
completion = resp.parse()
message = completion.choices[0].message
print(message.content)                            # the answer
print(getattr(message, "reasoning", None))        # the thinking, separate from the answer

What you need to know:

  • Thinking costs output tokens, and it comes first. With a small max_tokens, a thinking model can spend the whole budget thinking and return an empty content. In testing, olmo3-32b at high with max_tokens: 400 returned content: "". Give thinking models room (thousands of tokens), or use none when you want a short, direct answer.
  • The thinking is returned separately, in message.reasoning (and delta.reasoning when streaming), not inside content.
  • The response tells you what was served. The x-obleth-reasoning-effort header carries the level served. Some models don't have all four levels (some can't turn thinking off, for example); they serve another level in its place and add x-obleth-reasoning-effort-requested with the level you asked for. Models with no reasoning control return x-obleth-reasoning-effort: unsupported and ignore the setting.
  • Unknown values are refused with a 400: reasoning effort `turbo` is not one of none, low, medium or high. The synonyms off (none), minimal (low) and xhigh/max (high) are accepted.
  • Some models think by default. Leave reasoning_effort out and each model uses its own default. The reasoning object in /v1/models lists the levels a model has (mode: "mapped"), or says it has no control (mode: "unsupported").

Model families​

Some names are families: one name for an instruct model and its thinking sibling, with the level choosing between them. For example, olmo3-32b serves olmo3-32b-instruct at none and olmo3-32b-think at low, medium and high. Family entries in /v1/models have a family field:

curl -s https://openai.rc.asu.edu/v1/models/olmo3-32b -H "Authorization: Bearer $OPENAI_API_KEY" | jq .family
{
  "default_level": "none",
  "members": {"none": "olmo3-32b-instruct", "low": "olmo3-32b-think", "medium": "olmo3-32b-think", "high": "olmo3-32b-think"}
}

Responses from a family carry x-obleth-model-variant with the family name. You can also call a member by its own name.

Function calling​

Models with supports_function_calling accept OpenAI tools and return tool_calls for your code to run:

tools = [{
    "type": "function",
    "function": {
        "name": "get_quota",
        "description": "Storage usage for a filesystem on the cluster",
        "parameters": {
            "type": "object",
            "properties": {"filesystem": {"type": "string", "enum": ["home", "scratch"]}},
            "required": ["filesystem"],
        },
    },
}]
completion = client.chat.completions.create(
    model="qwen36-27b",
    reasoning_effort="none",
    messages=[{"role": "user", "content": "How much scratch space am I using on Sol?"}],
    tools=tools,
)
call = completion.choices[0].message.tool_calls[0]
print(call.function.name, call.function.arguments)   # get_quota {"filesystem": "scratch"}

Send the tool's result back as a {"role": "tool", "tool_call_id": call.id, "content": ...} message, after the assistant message that made the call, and ask again. tool_choice (auto, required, none, or a named function) works on models with supports_tool_choice.

Structured output​

On models with supports_response_schema, response_format with a JSON schema makes the reply valid JSON in that shape:

schema = {
    "type": "object",
    "properties": {
        "partition": {"type": "string"},
        "nodes": {"type": "integer"},
        "gpus_per_node": {"type": "integer"},
        "gpu_type": {"type": "string"},
        "walltime": {"type": "string"},
    },
    "required": ["partition", "nodes", "gpus_per_node", "gpu_type", "walltime"],
    "additionalProperties": False,
}
completion = client.chat.completions.create(
    model="gemma4-31b-it",
    temperature=0,
    messages=[{"role": "user", "content": "Extract the job: sbatch -p general -N 2 --gres=gpu:a100:4 -t 04:00:00 train.sh"}],
    response_format={"type": "json_schema", "json_schema": {"name": "job", "strict": True, "schema": schema}},
)
print(completion.choices[0].message.content)
# {"gpu_type": "a100", "gpus_per_node": 4, "nodes": 2, "partition": "general", "walltime": "04:00:00"}

Use a model with supports_response_schema; other models may not follow the schema. The schema fixes the structure, not the content of each value, so use temperature=0 for extraction: in testing, the same request at the default temperature sometimes put stray text inside a string value. For yes/no answers, categories or scores with probabilities, the Verdicts API is a better fit than a schema.

Images in the prompt​

Send images as image_url content parts, either a base64 data URL or a public https:// URL:

import base64

with open("figure.png", "rb") as f:
    image = "data:image/png;base64," + base64.b64encode(f.read()).decode()

completion = client.chat.completions.create(
    model="gemma4-31b-it",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What does this figure show? One sentence."},
        {"type": "image_url", "image_url": {"url": image}},
    ]}],
)
print(completion.choices[0].message.content)

Models with supports_vision read images themselves. A few text-only models (for example kimi-k2-7-code) have the vision add-on instead: another model describes each image in text first, and the model answers from the description (see Gateway add-ons). A text-only model without it rejects the image. The model server downloads https:// URLs itself, and some sites refuse automated downloads, so data URLs are the safer choice.

Gateway add-ons​

The gateway can give a model abilities it doesn't have natively. Each model's boon_details in /v1/models lists them, with active and, where another model does the work, helper_model:

Add-onWhat it does
visionA helper model describes images in text for a model that can't read them
web_searchThe model gets a web_search tool that the gateway runs (see Web search)
image_generationThe model gets a generate_image tool; generated images are appended to the reply as Markdown data URLs
speculationA small draft model answers first and the large model checks it (only on model names that opt in, such as glm-5-3-spec)

The x-obleth-boons response header lists the add-ons that acted on a request. Add-ons change what the model sees (search results, tool schemas and image descriptions add prompt tokens), so for reproducible experiments either record that header, or turn every add-on off for a request with the x-obleth-boons: off request header:

completion = client.chat.completions.create(
    model="glm-5-3",
    messages=[{"role": "user", "content": "Summarize the abstract below."}],
    extra_headers={"x-obleth-boons": "off"},
)

Let the gateway choose: auto​

"model": "auto" picks a chat model for each request from the request's content and the current load, and skips models that can't handle it (for example, it only picks vision models when you send images). The response's model field is the name of the model server that answered, which can differ from the catalog name; GET /v1/models/<that name> maps it back. auto is convenient for interactive use. For research results you want to reproduce, name a model.

Prompt caching​

Most chat models keep recently processed prompt prefixes and reuse them, so a request that starts with the same text as an earlier one (a long system prompt, a document you ask several questions about, an agent's growing conversation) is processed faster. To benefit:

  • put the stable part first (system prompt, instructions, documents) and the changing part last;
  • don't put timestamps or random ids at the start of the prompt;
  • ask several questions about one document in a row rather than interleaved with other work.

usage.prompt_tokens_details.cached_tokens reports how many prompt tokens were reused.

Model retirements​

A model scheduled for retirement stays in /v1/models with status: "deprecated", a replacement and a retire_at date, and every response from it carries headers that say so:

x-obleth-model-status: deprecated
deprecation: @1791179089
sunset: Mon, 19 Oct 2026 00:00:00 GMT
x-obleth-model-replacement: gemma4-31b-it
link: </v1/models/gemma4-31b-it>; rel="successor-version"

After the date, requests return 410 with "code": "model_retired" and the replacement's name, or are answered by the replacement. Check your logs for x-obleth-model-status to find code that still uses a retiring model.

Other parameters​

temperature, top_p, max_tokens (or max_completion_tokens), stop, logprobs/top_logprobs and n are passed to the model server as you send them. Don't rely on presence_penalty, frequency_penalty or repetition_penalty for now: on most models they currently have no effect. POST /v1/completions (prompt in, text out, no chat template) is also available for models that support it, without reasoning-effort translation or add-ons.