Skip to main content

Batch Requests

Level: Intermediate · Category: API reference

Tags: Batch API · Offline inference · Background priority

For large offline runs (classifying a corpus, summarizing thousands of documents, embedding a collection, scoring a test set) send one file of requests instead of thousands of separate calls. The gateway works through it in the background and gives you an output file when it finishes. It follows the OpenAI Batch API, so the openai SDKs work unchanged. Keys and setup are on LLM API Access.

Method and pathWhat it does
POST /v1/filesUpload a JSONL file of requests (purpose=batch)
POST /v1/batchesCreate a batch from the file
GET /v1/batches/{id}Status and request counts
GET /v1/files/{id}/contentThe output or error file, once the batch has finished
GET /v1/batches/{id}/resultsResults so far, while it runs (?errors=true for the error lines)
POST /v1/batches/{id}/cancelStop the requests that haven't started
DELETE /v1/batches/{id}Delete a batch that has stopped, with its files
GET /v1/batchesYour batches, newest first

Background priority​

Batch requests run only on capacity that interactive users aren't using. A batch request starts when nobody is waiting for that model, and batches together hold at most 75% of a model's capacity, so people using the model interactively never queue behind a batch. The trade-off:

  • A busy model runs batches slowly, or not at all until it quiets down. Nights and weekends are faster.
  • Throughput is the model's spare capacity, not your client's concurrency. There is no benefit to splitting one job into many batches.
  • Batches from different users share the background capacity by the same fair-share rules as interactive requests (how capacity is shared).

Use batches when nobody is waiting on the answer. For interactive work, call the endpoint directly.

Run a batch​

1. Write the requests​

One JSON object per line. custom_id is yours and must be unique in the file; url is the endpoint; body is the request as you would send it directly. Every line must name the same model.

requests.jsonl
{"custom_id": "doc-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen38-27b", "messages": [{"role": "user", "content": "One-line summary: the cooling loop failed at 02:10 and was restored at 02:40."}], "max_tokens": 100}}
{"custom_id": "doc-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen38-27b", "messages": [{"role": "user", "content": "One-line summary: storage quota raised from 10 TB to 25 TB for the imaging group."}], "max_tokens": 100}}

A batch can target /v1/chat/completions, /v1/completions or /v1/embeddings. Everything a direct request supports works in body: system prompts, reasoning_effort, structured output, tools.

2. Upload the file and create the batch​

from openai import OpenAI

client = OpenAI(base_url="https://openai.rc.asu.edu/v1", api_key="<YOUR_API_KEY>")

batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=batch_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"name": "incident summaries"},
)
print(batch.id, batch.status)

Every line is checked first. If any line is invalid (bad JSON, a missing or repeated custom_id, another endpoint or model, a model your key can't use), the batch's status is failed at once and batch.errors lists each problem with its line number. Nothing runs.

completion_window is the deadline: from 1h to 7d, written as <n>h or <n>d. Requests still queued when it passes don't run; give a long job a long window.

3. Wait for it​

import time

while batch.status in ("validating", "in_progress", "finalizing", "cancelling"):
time.sleep(30)
batch = client.batches.retrieve(batch.id)
counts = batch.request_counts
print(batch.status, f"{counts.completed + counts.failed}/{counts.total}")

batch.obleth["waiting"] is "capacity" when requests are left and none has finished for two minutes: the model is busy with interactive users, and the batch continues when they are done. batch.obleth["last_progress_at"] is when a request last finished.

4. Read the results​

import json

answers = {}
if batch.output_file_id:
for line in client.files.content(batch.output_file_id).text.splitlines():
row = json.loads(line)
answers[row["custom_id"]] = row["response"]["body"]["choices"][0]["message"]["content"]
if batch.error_file_id:
for line in client.files.content(batch.error_file_id).text.splitlines():
row = json.loads(line)
if row["error"]:
print(row["custom_id"], row["error"]["code"], row["error"]["message"])
else:
print(row["custom_id"], row["response"]["status_code"], row["response"]["body"])

Match lines by custom_id; neither file is in input order.

  • The output file has every request the model answered successfully.
  • The error file has the rest: a request the model rejected (its response holds the status and error body), one that still failed after 5 tries, or one that never ran, with error.code batch_expired (the deadline passed) or cancelled.

While it runs​

  • Results so far: GET /v1/batches/{id}/results returns the output lines finished so far, in the same format as the output file, so you can start on the answers before the batch ends.

    curl -s https://openai.rc.asu.edu/v1/batches/<BATCH_ID>/results \
    -H "Authorization: Bearer <YOUR_API_KEY>" > partial.jsonl
  • Cancel: client.batches.cancel(batch.id) stops the requests that haven't started. Running ones finish, and everything that finished is in the output file.

  • Delete: once a batch has stopped (completed, failed, expired or cancelled), DELETE /v1/batches/{id} removes it with its answers and files. A running batch returns 409: cancel it first.

    curl -s -X DELETE https://openai.rc.asu.edu/v1/batches/<BATCH_ID> \
    -H "Authorization: Bearer <YOUR_API_KEY>"

Limits​

LimitValue
Requests per file50,000
File size200 MB (413 above it)
Batches running at once, per key10 (429 for another)
completion_window1h to 7d
Tries per request5, on 429, 5xx and timeouts, with backoff
Kept for30 days after creation, then deleted
Who can see a batchOnly the API key that created it

Each request in a batch counts like a direct request: it shows in your usage under AI LLM › Usage in the Voyager portal, marked as a batch request. The data handling of direct requests applies; the batch itself (your input file and the answers) is stored on the gateway until you delete it or it expires, so delete a batch with sensitive content once you have the results.

  • LLM API Access: keys, the endpoint, and every other endpoint.
  • Limits and errors: how capacity is shared, and what each error means.
  • Embeddings: embedding a collection works as a batch too, with "url": "/v1/embeddings".