Skip to main content

Audio: Streaming Text-to-Speech and Transcription

Level: Intermediate · Category: API reference

Tags: Text-to-speech · Streaming audio · Transcription · Translation

The LLM API turns text into speech (POST /v1/audio/speech) and speech into text (POST /v1/audio/transcriptions and /v1/audio/translations), in the OpenAI API shapes. Keys and setup are on LLM API Access; the Python examples read OPENAI_API_KEY and OPENAI_BASE_URL from the environment as described there.

ModelEndpointUse it for
orpheus-3bspeechEnglish speech that starts in about a quarter of a second and streams while it is generated. 8 voices, expression tags.
qwen3-tts-customvoice-1p7bspeechOther languages, and a spoken style you describe in words. Not streamed, and much slower than real time.
whisper-large-v3transcriptions, translationsTranscription in about 100 languages, translation into English, timestamps
qwen3-asr-1p7btranscriptionsTranscription

Text-to-speech with orpheus-3b​

Voices and expression​

Voices: tara, leah, jess, leo, dan, mia, zac, zoe. List them with:

curl -s "https://openai.rc.asu.edu/v1/audio/voices?model=orpheus-3b" -H "Authorization: Bearer $OPENAI_API_KEY"
{"voices":["tara","leah","jess","leo","dan","mia","zac","zoe"],"uploaded_voices":[]}

OpenAI voice names (alloy, nova, …) are mapped onto these. Put expression tags inline in the text: <laugh>, <chuckle>, <sigh>, <cough>, <sniffle>, <groan>, <yawn>, <gasp>.

A whole file​

from openai import OpenAI

client = OpenAI()
speech = client.audio.speech.create(
    model="orpheus-3b",
    voice="tara",
    input="Your job on Sol finished. The results are in your scratch directory. <sigh> That took a while.",
    response_format="mp3",
)
speech.write_to_file("notice.mp3")

response_format can be mp3, opus, flac, wav or pcm. Audio is 24 kHz mono.

Streaming​

With response_format pcm or wav, audio is sent while it is being generated. The first audio arrives in about 0.3 seconds, and a single stream is generated at about 1.8 times real time, so a player can start right away and won't run dry. pcm is raw 16-bit little-endian samples at 24 kHz, mono.

import wave
from openai import OpenAI

client = OpenAI()
with client.audio.speech.with_streaming_response.create(
    model="orpheus-3b",
    voice="leo",
    input="Good morning. Your GPU job has been running for three hours, and utilization is low.",
    response_format="pcm",
) as response, wave.open("stream.wav", "wb") as out:
    out.setnchannels(1)
    out.setsampwidth(2)      # 16-bit
    out.setframerate(24000)
    for chunk in response.iter_bytes():
        out.writeframes(chunk)  # or hand each chunk to an audio player as it arrives

Streaming as server-sent events​

For browsers and web apps, add "stream_format": "sse" (or "stream": true) with response_format pcm or wav. Each event carries a base64 chunk of audio, and the last one carries usage:

data: {"type": "speech.audio.delta", "audio": "AAAAAAAA..."}
data: {"type": "speech.audio.delta", "audio": "..."}
data: {"type": "speech.audio.done", "usage": {"input_tokens": 12, "output_tokens": 126, "total_tokens": 138}}

Asking for SSE with mp3, opus or flac returns 400 Streaming needs response_format pcm or wav: those formats are encoded once the clip is complete.

Limits​

  • One orpheus-3b server streams 8 requests at once at faster than real time. More requests wait for a free slot, and more servers start when it stays busy (see Limits and errors).
  • Up to 20,000 characters (about 20 minutes of audio) per request.
  • speed other than 1.0 is refused, and instructions is ignored: use the voice and tags.
  • English only.

Text-to-speech in other languages: qwen3-tts-customvoice-1p7b​

Set language to chinese, english, french, german, italian, japanese, korean, portuguese, russian or spanish (the default, auto, detects it), and describe the delivery in instructions. Voices: vivian, serena, ryan, aiden, dylan, eric, ono_anna, sohee, uncle_fu.

from openai import OpenAI

client = OpenAI()
speech = client.audio.speech.create(
    model="qwen3-tts-customvoice-1p7b",
    voice="vivian",
    input="Bienvenidos al supercomputador Sol.",
    response_format="mp3",
    extra_body={"language": "spanish"},
    instructions="Speak slowly and warmly.",
)
speech.write_to_file("bienvenidos.mp3")

This model returns the whole file when it is done and runs slower than real time (in testing, about 15 seconds for one short sentence, and over a minute when the model had been idle), so use it for recorded narration, not live playback.

Transcription​

from openai import OpenAI

client = OpenAI()
with open("notice.mp3", "rb") as f:
    result = client.audio.transcriptions.create(model="whisper-large-v3", file=f)
print(result.text)
# Your job on Sol finished. The results are in your Scratch directory. That took a while.
  • response_format="verbose_json" adds the detected language, duration, and segments with start and end times. text, json, srt and vtt work too.
  • language (an ISO code such as en or es) skips language detection; prompt gives Whisper context such as spellings of names and terms.
  • POST /v1/audio/translations with whisper-large-v3 transcribes speech in another language straight into English text.
  • Upload mp3, wav, flac or ogg/opus files, up to 64 MiB per request. Long recordings are fine: a 1.5-minute WAV (4.7 MB) transcribed in about 20 seconds in testing.

To transcribe large collections on Sol without the API, see OpenAI Whisper.