Skip to main content

LLM API Limits and Errors

Level: Intermediate · Category: API reference

Tags: Fair share · Time limits · Retries · Error codes

How https://openai.rc.asu.edu/v1 shares capacity, how long a request may run, and what each error means. Every response carries an x-obleth-request-id header; include it when you ask for help.

How capacity is shared​

Each model runs on a fixed number of Gaudi accelerators, and each model server handles a fixed number of requests at once. When a model is busy, new requests wait in a queue instead of failing, and the queue is fair: capacity is split between groups, and within a group between API keys, so one script sending hundreds of requests at once can't lock everyone else out. Popular models also scale up when they stay busy, which takes a few minutes.

What this means for your code:

  • A slow response is usually a queue, not an error. Waiting time counts toward your client's timeout, so set client timeouts in minutes, not seconds.
  • Parallelism has diminishing returns. Sending 200 requests at once to one model doesn't make them finish sooner than sending 8 at a time; it only lengthens the queue. A thread pool of 4 to 8 workers per model is a good starting point for batch jobs.
  • A request that waits 5 minutes without starting returns 503 timed out waiting for model capacity with a Retry-After: 5 header. Retry with backoff, or try a less busy model.

Time limits​

LimitValue
Wait in the queue before a model starts your request5 minutes, then 503
Total time for one request, streamed or not60 minutes
Request body64 MiB (32 MiB on /v1/responses and /v1/messages)

A request that reaches the 60-minute limit is stopped. A non-streamed request returns 504. A streamed one ends with an error event and the connection closes, without data: [DONE]:

data: {"error":{"message":"upstream stream ended before the response was complete (after 3600 s)","type":"obleth_gateway_error","code":502}}

Treat any stream that ends without [DONE] as failed. Streaming is the better choice for long generations: you see progress, and a cut-off reply still gives you the text produced so far.

Retrying​

Retry 503 and 504 responses, and connection errors, with exponential backoff (for example 5 s, 10 s, 20 s, 40 s). The openai Python SDK retries some of these on its own; raise max_retries and timeout for batch work:

from openai import OpenAI

client = OpenAI(timeout=1800, max_retries=5)  # 30-minute timeout, up to 5 retries

Don't retry 400, 401, 403, 404 or 410 without changing the request: the same request will fail the same way.

Error format​

Errors from the gateway use the OpenAI shape:

{"error": {"message": "invalid api key", "type": "obleth_gateway_error"}}

Errors from the model server itself (for example, a prompt longer than the model's context window) pass through with the model server's status and message.

Error reference​

StatusMessageWhat to do
400model is requiredAdd model, and check the body is valid JSON.
400reasoning effort `X` is not one of none, low, medium or highUse one of the four levels.
400Model server messages such as This model's maximum context length is …Shorten the prompt or lower max_tokens.
401missing bearer token / invalid api keySend Authorization: Bearer <key>, check the key, or create a new one on the Keys tab in Voyager.
403api key disabledThe key was turned off on the Keys tab. Turn it on or create a new one.
403model is disabled / model not permitted for tenantThe model is offline, or your key's group can't use it. Pick another model.
403api key term budget exhausted / tenant term budget exhaustedThe key or group has a spending budget and it is used up. Contact us.
404model 'x' is not registeredCheck the name against GET /v1/models; names are exact (qwen38-27b, not qwen3-8-27b).
404{"detail":"Not Found"}You are using a Legacy key on an endpoint the older gateway doesn't have. Rotate the key on the AI LLM tab.
404unknown endpointCheck the path. Off campus, the base URL must end in /v1.
410The model 'x' was retired on … ("code": "model_retired")Switch to the replacement named in the error. See Model retirements.
413request body too largeSend less data, or smaller images.
429token budget exceededA per-minute token budget is used up. Wait a minute and retry.
502upstream request failed, or a stream that ends with an error eventThe model server failed or restarted. Retry with backoff.
503timed out waiting for model capacity (with Retry-After: 5)The model was busy for 5 minutes. Retry with backoff or use another model.
503no model is available to satisfy the auto requestNo model can take this auto request right now (for example, none that reads images is up). Name a model.
504upstream request timed outThe request ran past the time limit. Stream it, or ask for less output.

Usage and data handling​

  • Your usage (requests, tokens, and an indicative market-rate value; there is no charge) is under AI LLM › Usage in the Voyager portal, by key and by model, with a per-request log and CSV export.
  • What the gateway records: for each request, the time, key, model, token counts, status and timing, kept for 180 days. These records do not include the text of prompts or replies.
  • Where data goes: every model runs on Research Computing hardware at ASU; prompts are not sent to commercial AI providers. Two features do reach outside: the web search add-on and endpoint send search queries to public search engines, and model servers download any https:// image URL you send.
  • Follow the ASU Data Classification Tool before sending sensitive or regulated data.