Batch Requests
Level: Intermediate · Category: API reference
Tags: Batch API · Offline inference · Background priority
For large offline runs (classifying a corpus, summarizing thousands of documents, embedding a
collection, scoring a test set) send one file of requests instead of thousands of separate calls.
The gateway works through it in the background and gives you an output file when it finishes. It
follows the OpenAI Batch API, so the openai SDKs
work unchanged. Keys and setup are on LLM API Access.
| Method and path | What it does |
|---|---|
POST /v1/files | Upload a JSONL file of requests (purpose=batch) |
POST /v1/batches | Create a batch from the file |
GET /v1/batches/{id} | Status and request counts |
GET /v1/files/{id}/content | The output or error file, once the batch has finished |
GET /v1/batches/{id}/results | Results so far, while it runs (?errors=true for the error lines) |
POST /v1/batches/{id}/cancel | Stop the requests that haven't started |
DELETE /v1/batches/{id} | Delete a batch that has stopped, with its files |
GET /v1/batches | Your batches, newest first |
Background priority
Batch requests run only on capacity that interactive users aren't using. A batch request starts when nobody is waiting for that model, and batches together hold at most 75% of a model's capacity, so people using the model interactively never queue behind a batch. The trade-off:
- A busy model runs batches slowly, or not at all until it quiets down. Nights and weekends are faster.
- Throughput is the model's spare capacity, not your client's concurrency. There is no benefit to splitting one job into many batches.
- Batches from different users share the background capacity by the same fair-share rules as interactive requests (how capacity is shared).
Use batches when nobody is waiting on the answer. For interactive work, call the endpoint directly.
Run a batch
1. Write the requests
One JSON object per line. custom_id is yours and must be unique in the file; url is the
endpoint; body is the request as you would send it directly. Every line must name the same model.
{"custom_id": "doc-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen38-27b", "messages": [{"role": "user", "content": "One-line summary: the cooling loop failed at 02:10 and was restored at 02:40."}], "max_tokens": 100}}
{"custom_id": "doc-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen38-27b", "messages": [{"role": "user", "content": "One-line summary: storage quota raised from 10 TB to 25 TB for the imaging group."}], "max_tokens": 100}}
A batch can target /v1/chat/completions, /v1/completions or /v1/embeddings. Everything a
direct request supports works in body: system prompts, reasoning_effort, structured output,
tools.
2. Upload the file and create the batch
from openai import OpenAI
client = OpenAI(base_url="https://openai.rc.asu.edu/v1", api_key="<YOUR_API_KEY>")
batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=batch_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"name": "incident summaries"},
)
print(batch.id, batch.status)
Every line is checked first. If any line is invalid (bad JSON, a missing or repeated custom_id,
another endpoint or model, a model your key can't use), the batch's status is failed at once and
batch.errors lists each problem with its line number. Nothing runs.
completion_window is the deadline: from 1h to 7d, written as <n>h or <n>d. Requests still
queued when it passes don't run; give a long job a long window.
3. Wait for it
import time
while batch.status in ("validating", "in_progress", "finalizing", "cancelling"):
time.sleep(30)
batch = client.batches.retrieve(batch.id)
counts = batch.request_counts
print(batch.status, f"{counts.completed + counts.failed}/{counts.total}")
batch.obleth["waiting"] is "capacity" when requests are left and none has finished for two
minutes: the model is busy with interactive users, and the batch continues when they are done.
batch.obleth["last_progress_at"] is when a request last finished.
4. Read the results
import json
answers = {}
if batch.output_file_id:
for line in client.files.content(batch.output_file_id).text.splitlines():
row = json.loads(line)
answers[row["custom_id"]] = row["response"]["body"]["choices"][0]["message"]["content"]
if batch.error_file_id:
for line in client.files.content(batch.error_file_id).text.splitlines():
row = json.loads(line)
if row["error"]:
print(row["custom_id"], row["error"]["code"], row["error"]["message"])
else:
print(row["custom_id"], row["response"]["status_code"], row["response"]["body"])
Match lines by custom_id; neither file is in input order.
- The output file has every request the model answered successfully.
- The error file has the rest: a request the model rejected (its
responseholds the status and error body), one that still failed after 5 tries, or one that never ran, witherror.codebatch_expired(the deadline passed) orcancelled.
While it runs
-
Results so far:
GET /v1/batches/{id}/resultsreturns the output lines finished so far, in the same format as the output file, so you can start on the answers before the batch ends.curl -s https://openai.rc.asu.edu/v1/batches/<BATCH_ID>/results \-H "Authorization: Bearer <YOUR_API_KEY>" > partial.jsonl -
Cancel:
client.batches.cancel(batch.id)stops the requests that haven't started. Running ones finish, and everything that finished is in the output file. -
Delete: once a batch has stopped (
completed,failed,expiredorcancelled),DELETE /v1/batches/{id}removes it with its answers and files. A running batch returns409: cancel it first.curl -s -X DELETE https://openai.rc.asu.edu/v1/batches/<BATCH_ID> \-H "Authorization: Bearer <YOUR_API_KEY>"
Limits
| Limit | Value |
|---|---|
| Requests per file | 50,000 |
| File size | 200 MB (413 above it) |
| Batches running at once, per key | 10 (429 for another) |
completion_window | 1h to 7d |
| Tries per request | 5, on 429, 5xx and timeouts, with backoff |
| Kept for | 30 days after creation, then deleted |
| Who can see a batch | Only the API key that created it |
Each request in a batch counts like a direct request: it shows in your usage under AI LLM › Usage in the Voyager portal, marked as a batch request. The data handling of direct requests applies; the batch itself (your input file and the answers) is stored on the gateway until you delete it or it expires, so delete a batch with sensitive content once you have the results.
Related guides
- LLM API Access: keys, the endpoint, and every other endpoint.
- Limits and errors: how capacity is shared, and what each error means.
- Embeddings: embedding a collection works as a batch too, with
"url": "/v1/embeddings".