Appearance
Realtime transcription
WS /v1/realtime?intent=transcription
Open a WebSocket, send audio as it is captured, and receive the transcript while the speaker is still talking. Use it for phone calls, meetings, voice agents and microphones. Billed per second of audio sent.
For a recording you already have, use file transcription instead. It is simpler and costs less.
Before you start
Four things work differently from what you may expect:
- You decide when an utterance ends. There is no server-side voice activity detection. Nothing is finalized until you send
input_audio_buffer.commit, so your client needs its own end-of-speech signal: a local VAD, a push-to-talk button, or a timer.turn_detectionmust benull. - The audio format is fixed: 24 kHz, mono, 16-bit little-endian PCM, base64 in JSON messages. There is no container or header, and binary WebSocket frames are rejected.
- The model must support realtime. Check for
realtimein the model'scapabilities.cadence-sttdoes. - Sessions last up to 55 minutes.
session.createdtells you exactly when yours ends, so you can open the next session before it does.
Connect
The OpenAI Python SDK's realtime client connects directly:
bash
pip install "openai[realtime]>=2.43"python
import asyncio
import base64
import os
from openai import AsyncOpenAI
client = AsyncOpenAI(
api_key=os.environ["XVOICE_API_KEY"],
base_url="https://api.xvoice.dev/v1",
)
async def main():
async with client.realtime.connect(
extra_query={"intent": "transcription"}
) as connection:
await connection.session.update(session={
"type": "transcription",
"audio": {"input": {
"format": {"type": "audio/pcm", "rate": 24000},
"transcription": {"model": "cadence-stt", "language": "en"},
"turn_detection": None,
}},
})
# Send 24 kHz mono PCM16 in 20 ms frames, then commit to end the utterance.
with open("speech.pcm", "rb") as audio:
while frame := audio.read(960):
await connection.input_audio_buffer.append(
audio=base64.b64encode(frame).decode()
)
await connection.input_audio_buffer.commit()
async for event in connection:
if event.type == "conversation.item.input_audio_transcription.delta":
print(event.delta, end="", flush=True)
elif event.type == "conversation.item.input_audio_transcription.completed":
break
asyncio.run(main())The realtime example is a complete client that sends and receives at the same time.
Any WebSocket client works. To try the protocol by hand with websocat:
bash
websocat -H "Authorization: Bearer $XVOICE_API_KEY" \
"wss://api.xvoice.dev/v1/realtime?intent=transcription"Connection rules
intent=transcriptionis required in the query string.- Authenticate with the
Authorization: Bearerheader. Keys in the URL are not accepted. For a browser client, relay the connection through your own server so the key never reaches the browser. - Name the model in
session.update, not in the URL. Amodelquery parameter is accepted as a fallback for clients that cannot send one in the session.
If the connection is refused, for example for a bad key (401), an empty balance (402) or too many open sessions (429), the refusal is an ordinary HTTP response with the usual error body, sent instead of the WebSocket upgrade. Read the body for the reason.
Session flow
text
server → session.created session id and expires_at
client → session.update model, language, prompt
server → session.updated the configuration in effect
client → input_audio_buffer.append audio, repeatedly
server → …input_audio_transcription.delta text, while the speaker is still talking
client → input_audio_buffer.commit the utterance is over
server → input_audio_buffer.committed
server → conversation.item.added / .done
server → …input_audio_transcription.delta any remaining text
server → …input_audio_transcription.completed the final transcriptAudio sent before session.update fails with session_not_configured.
Client events
There are four client events. Unknown events, and unknown fields in known events, fail with an error instead of being ignored.
session.update
Configures the session. The first one sets the model, and a session uses one model for its whole life: naming a different model later fails with model_already_set. You can change language and prompt at any time, and the change applies from the next utterance.
json
{
"type": "session.update",
"session": {
"type": "transcription",
"audio": {
"input": {
"format": {"type": "audio/pcm", "rate": 24000},
"transcription": {"model": "cadence-stt", "language": "en", "prompt": ""},
"turn_detection": null
}
}
}
}language works as it does for file transcription: set it to the language spoken, or leave it out.
To also receive interim text, add "include": ["item.input_audio_transcription.interim"] to session. Send "include": [] to turn it off again.
input_audio_buffer.append
json
{"type": "input_audio_buffer.append", "audio": "<base64 PCM16>"}Adds audio to the current utterance. Each chunk must be an even number of bytes, and at most 15 MB. Chunks of 20 ms (960 bytes before encoding) work well.
input_audio_buffer.commit
json
{"type": "input_audio_buffer.commit"}Ends the current utterance. You can start sending the next one straight away: the previous utterance finishes in the background. Committing an empty buffer fails with input_audio_buffer_commit_empty.
input_audio_buffer.clear
json
{"type": "input_audio_buffer.clear"}Discards the audio not yet committed. It does not cancel an utterance already committed. Cleared audio is still billed, because it has already been processed.
Any client event can include an event_id of up to 512 characters. If the event causes an error, the error repeats it as client_event_id.
Server events
| Event | When |
|---|---|
session.created | Right after you connect |
session.updated | After each session.update |
input_audio_buffer.committed | After a commit |
input_audio_buffer.cleared | After a clear |
conversation.item.added, conversation.item.done | An utterance has been created, or has finished |
conversation.item.input_audio_transcription.delta | New text for an utterance |
conversation.item.input_audio_transcription.interim | Only if you asked for it: the current best guess at an utterance's text, which may still change. See Interim text |
conversation.item.input_audio_transcription.completed | The final transcript of an utterance |
conversation.item.input_audio_transcription.failed | One utterance could not be transcribed. The session continues |
error | Something went wrong. See Errors |
Handling transcripts
- Text arrives before you commit. Deltas stream while the utterance is still open, and they carry the utterance's
item_idbeforeinput_audio_buffer.committedannounces it. Take the id from the first delta you see. - Deltas are never revised. You can display them as they arrive. Joined together, the deltas for an item equal its final transcript.
- Utterances can finish out of order. A long utterance can complete after a short one that followed it. Key everything by
item_id, and do not assume there is only one utterance in progress.
completed carries the full transcript and the audio duration billed for it:
json
{
"type": "conversation.item.input_audio_transcription.completed",
"item_id": "item_01J8...",
"transcript": "Thanks for calling. How can I help you today?",
"usage": {"type": "duration", "seconds": 3.0}
}item_id values are stable for the session and match the entries in your request logs.
Interim text
Deltas only carry text that will not change, so they start a few seconds into an utterance. If you want to show words sooner, ask for interim text in session.update. You then also receive:
json
{
"type": "conversation.item.input_audio_transcription.interim",
"item_id": "item_01J8...",
"content_index": 0,
"text": "Thanks for calling. How can I"
}- Each interim replaces the one before it.
textis the whole utterance so far, not an increment. Words at its end can change in the next one. - Interim text is a preview. Deltas and
completedwork exactly as without it and remain the transcript. Do not join interim text with deltas. - It is not billed separately. Billing is by audio duration either way.
Interim text is our addition to the OpenAI event set. The OpenAI SDKs do not reject event types they do not know, so event.type and event.text can be read with them.
Session limits
| Limit | Value |
|---|---|
| Session length | 55 minutes |
| Idle timeout | 5 minutes with no audio |
| One utterance | 10 minutes. Commit more often |
One append | 15 MB |
session.created includes expires_at, a Unix timestamp:
json
{
"type": "session.created",
"session": {
"id": "sess_01J8...",
"object": "realtime.transcription_session",
"type": "transcription",
"expires_at": 1758675300,
"audio": {
"input": {
"format": {"type": "audio/pcm", "rate": 24000},
"transcription": {"model": "cadence-stt", "language": "en", "prompt": ""},
"turn_detection": null,
"noise_reduction": null
}
},
"include": []
}
}At expires_at, you receive a session_expired error and the socket closes normally. For a call longer than 55 minutes, open a new session a little before the deadline and switch to it at the next utterance boundary, so no audio is lost.
Billing
You are billed for every second of audio you send, counted from the bytes received (bytes / 48000), not from transcription results. That means these are billed too:
- audio you
clear, - audio in progress when you disconnect,
- audio whose transcription failed or never arrived.
Your balance is checked every 60 seconds of audio. If it runs out, you receive an insufficient_credits error and the session closes.
Each open session holds one of your organization's concurrent request slots for its whole life, and opening it counts as one request. See Rate limits and Billing.
Errors
An error after the connection is open arrives as an error event, with the same body as everywhere else:
json
{
"type": "error",
"event_id": "event_01J8...",
"error": {
"type": "invalid_request",
"code": "unsupported_parameter",
"message": "Server-side turn detection is not available. Send 'input_audio_buffer.commit' when an utterance ends.",
"param": "session.audio.input.turn_detection",
"request_id": "req_01J8...",
"client_event_id": "event_from_you"
}
}Most errors leave the session open, so you can fix the event and continue. Billing errors, rate limits, the session deadline, the idle timeout and failures on our side close it. The close code does not say why the session closed. Read the error event sent just before it.
| Close code | Meaning |
|---|---|
1000 | Normal close: the session reached its deadline or idle timeout |
1008 | A problem you can fix: billing, limits or a protocol error |
1011 | A problem on our side, or the model is at capacity. Reconnect after a short wait |
| Code | Cause |
|---|---|
session_not_configured | Audio was sent before session.update |
missing_model | No model named in session.update or the URL |
invalid_model | No such model, or not an STT model |
realtime_not_supported | The model does not support realtime |
model_already_set | A later session.update named a different model |
unsupported_event | An unknown event type, or a binary frame |
unsupported_parameter | turn_detection or noise_reduction was set, or include named something other than interim text |
unsupported_audio_format | The format is not audio/pcm at 24 kHz |
unsupported_language | The model does not support that language |
frame_too_large | An append was larger than 15 MB |
audio_too_long | One utterance passed 10 minutes. Commit more often |
input_audio_buffer_commit_empty | commit with nothing buffered |
transcription_failed | One utterance could not be transcribed. The session continues |
invalid_input | The model could not process the audio |
session_expired | The session reached 55 minutes |
session_idle | No audio for 5 minutes |
insufficient_credits | The balance ran out during the session |
model_overloaded | The model is at capacity. Reconnect after a short wait |
service_unavailable | The session could not be opened. Retry shortly |
stream_interrupted | The session was interrupted. Open a new one |
Running in production
- Raise your proxy's timeouts. If the connection passes through your own reverse proxy or load balancer, its idle and read timeouts must be longer than a session. Many default to 60 seconds; nginx's
proxy_read_timeoutandproxy_send_timeoutare examples. Set them above 3,300 seconds and allow the WebSocket upgrade. - Plan for reconnects. Treat
1011and dropped connections as retryable, and roll over to a new session beforeexpires_at. - Size your concurrency. Each open session uses one concurrent slot. See Rate limits.
Not supported
Server-side VAD and turn_detection, noise reduction, logprobs, ephemeral client keys, WebRTC, and speech output over the same socket. This endpoint transcribes; it is not a speech-to-speech API.