Skip to main content

Verdicts API: Typed Answers About Your Data

Level: Intermediate · Time: 15 min · Category: API reference

Tags: LLM API · Classification · Labeling · Vision

POST https://openai.rc.asu.edu/v1/verdicts asks a model typed questions about a piece of content and returns typed answers. You send a state (text, a JSON record, or images) and a set of questions. Each question comes back as a boolean, one option from a list, or a level on a scale, together with a probability distribution and a confidence score.

The answer is read from the model's first-token probabilities over a fixed set of labels, so there is no prose to parse and the answer cannot come back malformed. Use it wherever you would otherwise prompt a chat model and parse its reply:

  • Screening and labeling: abstracts, survey responses, transcripts, or a dataset you need labeled in bulk.
  • Rubric scoring: grade answers, essays, or generated text against levels you define.
  • Triage and routing: decide whether something is urgent and which queue it belongs in.
  • Record matching: decide whether two records describe the same person, sample, or paper.
  • Image checks: check that a figure has labeled axes, or that a micrograph or photo shows what you expect.
  • Gating an agent: approve or block a tool call by thresholding a probability.
Before you start
  • An API key from the Voyager portal (see LLM API Access). Keys marked Legacy in Voyager go through the older gateway, which does not have this endpoint and returns {"detail":"Not Found"}. Rotate a legacy key on the AI LLM tab to move it to the current gateway.

You do not have to pick a model. A request without model is answered by gemma4-31b-it, which handles both text and images. See Choose a model to use another one.

Quick example​

The examples on this page read your key from OPENAI_API_KEY, as set up in Use Python.

curl -s https://openai.rc.asu.edu/v1/verdicts \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": {"ticket": "My job 4821337 on Sol has been pending for two days and my thesis defense is Friday. squeue says the reason is QOSMaxGRESPerUser."},
    "questions": {
      "is_urgent": {
        "type": "boolean",
        "instructions": "Does this need a reply today?",
        "criteria": {"true": "A deadline within a week, data loss, or a security issue", "false": "Can wait for the normal queue"}
      },
      "team": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {"scheduler": "Slurm jobs, queues, QOS, partitions", "storage": "Quotas, scratch, file recovery", "accounts": "Access, sponsors, groups", "software": "Modules, builds, containers"}
      },
      "frustration": {
        "type": "score",
        "instructions": "How frustrated does the user sound?",
        "criteria": ["Calm and factual", "Mildly annoyed", "Frustrated", "Very angry"]
      }
    }
  }'

The curl request returns (numbers rounded):

{
  "model": "gemma4-31b-it",
  "verdicts": {
    "frustration": {
      "type": "score",
      "value": 3,
      "probabilities": [0.00002, 0.4073, 0.5927, 0.000001],
      "confidence": 0.51,
      "expected_value": 2.59
    },
    "is_urgent": {
      "type": "boolean",
      "value": true,
      "probabilities": {"false": 0.0, "true": 1.0},
      "confidence": 1.0
    },
    "team": {
      "type": "choice",
      "value": "scheduler",
      "probabilities": {"accounts": 0.0, "scheduler": 1.0, "software": 0.0, "storage": 0.0},
      "confidence": 1.0
    }
  },
  "usage": {"prompt_tokens": 477, "completion_tokens": 3, "total_tokens": 480}
}

Notice that frustration is split between "Mildly annoyed" and "Frustrated". Its confidence of 0.51 tells you the model was unsure, which a single text answer would hide.

Request​

FieldTypeRequiredDescription
modelstringnoA chat model ID from /v1/models, a model family, or "auto". Defaults to gemma4-31b-it. See Choose a model.
statestring, object, or arrayyes, unless images is givenThe content to judge. Strings are used as written; objects and arrays are rendered as indented JSON.
imagesarray of stringsnoUp to 8 images. See Images.
questionsobjectyes1–32 questions, keyed by an ID you choose (up to 64 bytes). The response uses the same IDs.

Each question has a type, instructions (the question itself, as a string), and criteria:

typecriteriavalue in the responseprobabilities in the response
booleanOptional. {"true": "...", "false": "..."} says what each answer means.true or false{"false": p, "true": p}
choiceRequired. An object mapping each option name to a description (or null). 2–26 options.The option name{"option": p, ...}
scoreRequired. An array of level descriptions, lowest first. 2–10 levels.The level, counting from 1[p1, p2, ...], plus expected_value

Each question is answered independently and cannot see the other answers. Unknown fields are ignored. The gateway always uses greedy decoding and generates a single token, so temperature, max_tokens, and stream have no effect. Responses are never streamed.

Response​

FieldDescription
modelThe model that answered. For a family or "auto", this tells you which model was chosen.
verdicts.<id>.typeThe question's type.
verdicts.<id>.valueThe most probable answer.
verdicts.<id>.probabilitiesThe distribution over every answer, summing to 1. An answer outside the model's top 20 first tokens gets exactly 0.0.
verdicts.<id>.confidence0 to 1. It is 0 for a uniform distribution and 1 when the model is decisive and all of its probability landed on valid answers.
verdicts.<id>.expected_valueScore questions only: the probability-weighted level, such as 2.59. Useful for ranking.
usageTokens summed over every question.

The request ID is returned in the x-obleth-request-id response header. Include it when you ask for help.

Base decisions on both numbers: threshold on the probability you care about, and send answers with low confidence to a person instead of acting on them.

Examples​

These use the verdicts() helper from the Python tab above.

Screen abstracts​

Ask every question about one item in a single request. The model processes the state once and answers all of the questions from it. Send several items in parallel, a few at a time:

from concurrent.futures import ThreadPoolExecutor

from verdicts_helper import verdicts

SCREEN = {
    "relevant": {
        "type": "boolean",
        "instructions": "Does the study apply machine learning to genomic or transcriptomic data?",
        "criteria": {
            "true": "ML methods are applied to DNA, RNA, or gene-expression data",
            "false": "No ML, or the data is not genomic",
        },
    },
    "study_type": {
        "type": "choice",
        "instructions": "What kind of study is this?",
        "criteria": {
            "method": "Introduces a new method or model",
            "application": "Applies existing methods to answer a biological question",
            "review": "Surveys or reviews prior work",
            "other": "Anything else",
        },
    },
    "rigor": {
        "type": "score",
        "instructions": "How well does the abstract describe the evaluation?",
        "criteria": [
            "No evaluation described",
            "Evaluation mentioned without detail",
            "Datasets or metrics named",
            "Datasets, metrics, and baselines or held-out validation named",
        ],
    },
}

abstracts = [
    "We present a transformer model that predicts enhancer activity from DNA sequence. Trained on "
    "STARR-seq data from five cell lines, it outperforms CNN baselines (auROC 0.91 vs 0.85) on "
    "held-out chromosomes.",
    "We surveyed 412 graduate students about their sleep habits and found that students who slept "
    "fewer than six hours reported lower satisfaction with their programs.",
    "This review summarizes deep learning approaches for single-cell RNA-seq analysis, including "
    "clustering, batch correction, and trajectory inference.",
]

with ThreadPoolExecutor(max_workers=4) as pool:
    results = list(pool.map(lambda text: verdicts({"abstract": text}, SCREEN), abstracts))

for text, v in zip(abstracts, results):
    p = v["relevant"]["probabilities"]["true"]
    print(f"{p:.2f}  {v['study_type']['value']:<12} rigor {v['rigor']['value']}  {text[:50]}")
1.00  method       rigor 4  We present a transformer model that predicts enhan
0.00  other        rigor 3  We surveyed 412 graduate students about their slee
1.00  review       rigor 1  This review summarizes deep learning approaches fo

Check figures before you submit​

Send images as base64 data URLs. When you send images, state is optional; here it gives context:

import base64
from pathlib import Path

from verdicts_helper import verdicts

def data_url(path):
    return "data:image/png;base64," + base64.b64encode(Path(path).read_bytes()).decode()

FIGURE_CHECK = {
    "axis_labels": {
        "type": "boolean",
        "instructions": "Do both axes have text labels that say what is plotted? Tick numbers alone do not count.",
    },
    "legend": {
        "type": "boolean",
        "instructions": "If the plot has more than one series, is there a legend that names each one?",
        "criteria": {"true": "One series, or a legend names every series", "false": "Several series and no legend"},
    },
    "plot_type": {
        "type": "choice",
        "instructions": "What kind of plot is this?",
        "criteria": {"line": None, "scatter": None, "bar": None, "histogram": None, "other": "Anything else"},
    },
}

for path in ["figure-good.png", "figure-bare.png"]:
    v = verdicts("A figure from a manuscript draft.", FIGURE_CHECK, images=[data_url(path)])
    print(path,
          "labels", round(v["axis_labels"]["probabilities"]["true"], 2),
          "legend", round(v["legend"]["probabilities"]["true"], 2),
          "type", v["plot_type"]["value"])

With one line plot that has axis labels and a legend, and the same plot with neither:

figure-good.png labels 1.0 legend 1.0 type line
figure-bare.png labels 0.0 legend 0.0 type line

Set thresholds from labeled examples​

Before you automate a decision, run the question over a few hundred items whose answers you already know, and pick the threshold that gives the precision and recall you need:

from verdicts_helper import verdicts

labeled = [
    ("My scratch directory vanished and my results for tomorrow's deadline were in it.", True),
    ("Could you install the latest version of R on Phoenix?", False),
    # ... a few hundred more
]
URGENT = {"is_urgent": {"type": "boolean", "instructions": "Does this need a reply today?"}}

results = [
    (verdicts(text, URGENT)["is_urgent"]["probabilities"]["true"], label)
    for text, label in labeled
]
positives = sum(label for _, label in results)

for t in (0.5, 0.7, 0.8, 0.9, 0.95, 0.99):
    flagged = [label for p, label in results if p >= t]
    precision = sum(flagged) / len(flagged) if flagged else 1.0
    recall = sum(flagged) / positives
    print(f"{t:.2f}  precision {precision:.2f}  recall {recall:.2f}  flagged {len(flagged)}")

Calibrate again whenever you change the model or the wording of a question.

Images​

  • Format: each entry in images is a base64 data URL (data:image/png;base64,...) or a public https:// URL. The model server downloads https:// URLs itself, and some sites refuse automated downloads (the error contains Failed to fetch media from URL). Data URLs always work.

  • Limits: up to 8 images and 20 MiB of image data per request. Images are sent with every question, so image size × number of questions must stay under 64 MiB. Ask fewer questions per request, or send smaller images, if you hit that limit.

  • Models: the model must accept images. List the vision models with:

    curl -s https://openai.rc.asu.edu/v1/models -H "Authorization: Bearer $OPENAI_API_KEY" \
      | jq -r '.data[] | select(.supports_vision) | .id'

    The default, gemma4-31b-it, accepts images. A model without vision returns 400 with does not accept images; use a model with vision. With "model": "auto", only vision models are considered.

Choose a model​

Leave model out and gemma4-31b-it answers. It is an instruction-tuned model that answers decisively and accepts images, and it is the model behind every example on this page. To use another model, set model in the request.

Do not use reasoning models

Reasoning (thinking) models are poor at verdicts. A verdict is read from the model's first token, and a reasoning model spends its first token starting to think. The gateway retries such a question once with thinking skipped, which doubles its token use, and the answers usually still come back with confidence near 0. In testing, qwen38-27b picked the same labels as gemma4-31b-it on the quick example, with confidence of 0.01 or less on all three questions.

  • Other instruct models that work well include granite41-30b, olmo3-32b-instruct, and qwen3-235b-a22b-instruct-2507.

  • Model families such as olmo3-32b and qwen3-235b-a22b (entries in /v1/models with a family field) pick a member by reasoning effort in chat. Verdicts always answers a family name with its instruct member, the one listed under "none", and the response's model field names it. To see a family's members:

    curl -s https://openai.rc.asu.edu/v1/models/olmo3-32b -H "Authorization: Bearer $OPENAI_API_KEY" | jq .family

  • "auto" picks a model for you, filtered to vision models when you send images. It can pick a reasoning model, and its choice can change over time. If you use fixed thresholds, stay on the default or name a model.

  • Calibrate per model. Probabilities differ between models, so recalibrate your thresholds whenever you change model.

Write good questions​

  • Ask about what the state shows, not a judgment the model has to invent.
  • Ask one thing per question. Split "is it relevant and well-designed?" into two questions.
  • Give choices distinct meanings, and add an other option so the model is not forced into a bad fit.
  • List score levels lowest first, with each level differing from its neighbors in one clear way.
  • Put definitions in criteria and keep instructions to the question itself.
  • Name options with the values your code uses, so you can switch on value directly.
  • Ask related questions about one item in one request. If a question depends on another's answer, send two requests: the second only when the first says it applies.

Usage, limits, and errors​

Each question is one model call that generates one token. The calls run concurrently and share the state (and images) as a common prefix, which the model server caches. A request with many questions costs little more time than a request with one. usage.prompt_tokens counts the state once per question. Requests count against your key's fair share like chat requests.

LimitValue
Questions per request32
Options per choice question2–26, each name up to 200 bytes
Levels per score question2–10
Rendered state2 MiB
Images8 per request, 20 MiB in total; image size × questions ≤ 64 MiB
Request body64 MiB

If any question fails, the whole request fails with 502. There are no partial results. Errors use the form {"error": {"message": "...", "type": "obleth_gateway_error"}}.

StatusExample messageFix
401invalid api keyCheck the key, or create a new one in Voyager.
404{"detail":"Not Found"}You are using a Legacy key. Rotate it on the AI LLM tab in Voyager.
404model 'x' is not registeredCheck the model ID against /v1/models, or leave model out to use the default.
400invalid request: unknown variant `number`, expected one of `boolean`, `choice`, `score` Use one of the three question types.
400`state` must be a string, object, or array, and may be left out only when `images` are givenAdd a state, or send images.
400too many questions: 40 (maximum 32)Split the questions across requests.
400model 'x' does not accept images; use a model with visionLeave model out to use the default, or pick a vision model.
413rendered state is too largeShorten the state; the limit is 2 MiB.
429 / 503token budget exceeded / timed out waiting for model capacityWait and retry with backoff, or send fewer requests at once.
502question 'q' on model 'm': backend returned no logprobs (does it support `logprobs`?)The model's server cannot return token probabilities. Use another model.
502question 'q' on model 'm': model produced no recognizable answer label (is this a reasoning/thinking model?)Use an instruction-tuned model.
502question 'q' on model 'm': upstream returned 422 ... Failed to fetch media from URLThe model server could not download an image URL. Send the image as a data URL.