Building Production Audio Agents with Real-Time Speech-to-Speech Models

openai · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real-time voice and audio agents

OpenAI's solution architecture team needed to show how teams can build production audio agents now that speech-to-speech models are becoming viable. The core problem is designing voice agents that balance latency, cost, accuracy, user experience, tools, evaluation, guardrails, and handoffs without assuming one model architecture fits every use case.

Constraints:

• Chained speech-to-text, LLM, and text-to-speech architectures add latency and lose semantic audio context.
• Consumer voice experiences need low latency and expressive speech, while customer-service workflows need accuracy, integrations, and determinism.
• Voice prompts must control not only task behavior but also demeanor, tone, pacing, enthusiasm, and expressiveness.
• Tool use and handoffs must preserve conversation context so the user does not repeat themselves.
• Safety guardrails cannot add obvious latency to a real-time conversation.
• Audio-agent evaluation must cover traces, human listening, transcription-based checks, and audio-specific tone/pacing issues.

Architecture

The recommended architecture uses OpenAI's real-time API as a speech-to-speech frontline agent when low-latency conversation matters. That frontline agent handles normal interaction, easy questions, voice expressiveness, and user-facing flow. For harder reasoning or business-policy decisions, it delegates through tool calls to specialized backend agents or smarter models such as O3/O4 mini. Conversation-state summaries are passed across handoffs. Observability starts with traces of audio and agent behavior, evaluation proceeds through human labeling, transcription-based LLM-as-judge checks, audio-specific tone/pacing review, and synthetic conversations, while guardrails run asynchronously in the latency gap between generated text and spoken audio.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-audio-input["User audio input"] --> real-time-api-frontline-agent["Real-time API frontline agent"]
  voice-prompt-contract["Voice prompt contract"] --> real-time-api-frontline-agent
  real-time-api-frontline-agent --> limited-tool-set["Limited tool set"]
  real-time-api-frontline-agent --> backend-specialist-agents["Backend specialist agents"]
  conversation-state-summary["Conversation-state summary"] --> backend-specialist-agents
  backend-specialist-agents --> real-time-api-frontline-agent
  real-time-api-frontline-agent --> async-guardrails["Async guardrails"]
  async-guardrails --> spoken-audio-output["Spoken audio output"]
  real-time-api-frontline-agent --> spoken-audio-output
  real-time-api-frontline-agent --> observability-traces["Observability traces"]
  observability-traces --> human-audio-labeling["Human audio labeling"]
  observability-traces --> transcription-based-evals["Transcription-based evals"]
  observability-traces --> audio-specific-evals["Audio-specific evals"]
  observability-traces --> synthetic-conversation-tests["Synthetic conversation tests"]

Components

Data flow

A user speaks to the voice agent. In a speech-to-speech design, the real-time API handles transcription, intelligence, and speech generation inside one model rather than chaining three models. The frontline agent follows voice-specific prompts for tone and flow, uses a limited tool set, and delegates hard tasks to specialized backend agents when needed. Handoff summaries preserve conversation state. Generated text becomes audio, while asynchronous guardrails inspect generated text before problematic speech reaches the user. Traces, human labels, transcription-based evals, audio evals, and synthetic conversations feed iteration.

Why this architecture

Speech-to-speech is preferred when low latency and natural conversation dominate because it removes the chained architecture's three-model latency and semantic loss. But the talk explicitly frames architecture as use-case dependent: consumer voice apps may prioritize real-time expressiveness, while customer-service workflows may accept chained designs for accuracy, determinism, and integrations. Backend delegation keeps the frontline audio model responsive while routing specialized reasoning elsewhere, and async guardrails preserve safety without blocking spoken interaction.

Models used: OpenAI real-time API · speech-to-speech model · O3 mini · O4 mini · GPT-4 Audio · LLM-as-judge models

Scale

• The real-time speech-to-speech capability is described as newly usable for high-quality production applications at scale.
  “The models are getting, I would say, good enough and are kind of at this tipping point where you can start to really meaningfully build high quality production applications at scale.”transcript, stated
• The talk compares current models to models from not even six months earlier, emphasizing rapid improvement.
  “this is actually not just not even six months ago, right? The models are incredibly slow.”transcript, stated
• A customer-service audio agent must handle business-impacting details such as order numbers and update/delete actions.
  “you can't get an order number wrong. If it uh goes to delete an order instead of updating an order, that's a big deal.”transcript, stated
• The example guardrail debounce runs every 100 generated characters, with the threshold configurable.
  “right now it runs every 100 characters to run the guardrails but you can control that if you want to.”transcript, stated
• OpenAI positions the new real-time API snapshot as good enough for scalable production applications.
  “the technology is getting to that point where it is good enough to build scalable production applications.”transcript, stated

Alternatives considered (roads not taken)

Chained STT -> LLM -> TTS architecture

Rejected because: For low-latency conversational experiences it compounds latency and loses semantic audio context, though it may still fit accuracy/determinism-heavy customer-service cases.

Give one voice agent 10 or 20 tools

Rejected because: The recommended approach is to keep each agent's tools limited and delegate different task types to other agents/models.

Evaluate only from transcripts

Rejected because: Transcripts cover many checks but cannot fully capture tone, pacing, intonation, and other voice-experience qualities.

Block the real-time response path for guardrails

Rejected because: Blocking would harm latency; generated text arrives before spoken audio, so guardrails can run asynchronously before playback.

Patterns

latency-optimization (voice-architecture)

The speech-to-speech real-time API collapses transcription, intelligence, and speech generation into one model to reduce response latency.

multi-agent-orchestration (agent-architecture)

A low-latency frontline audio agent delegates harder tasks to smarter backend agents or models through tool calls.

tool-use-orchestration (agent-tools)

Voice agents use a limited set of tools directly and route specialized tasks to other agents instead of overloading one agent with many tools.

context-compaction (handoff)

Before handing a voice interaction to another agent, the system summarizes conversation state so context is preserved without passing the full raw history.

guardrails-io-filtering (safety)

Guardrails run asynchronously on generated text while audio synthesis/playback is still pending, with configurable debounce frequency.

eval-driven-development (evaluation)

The voice-agent evaluation stack starts with traces and human labels, then adds transcript-based LLM judges, audio-specific checks, and synthetic conversations.

Tradeoffs

Use speech-to-speech for low-latency experience

Chose Real-time API single-model architecture over A three-model chained STT/LLM/TTS architecture.

Rationale: Speech-to-speech simplifies the architecture, reduces time-to-output, and preserves more semantic conversation understanding.

Route harder tasks to backend models

Chose Frontline voice agent with backend specialist delegation over One real-time voice model handling all reasoning and policy decisions.

Rationale: The frontline agent can stay responsive while more capable models handle harder tasks such as refund-policy decisions.

Run guardrails asynchronously

Chose Parallel generated-text checks before audio reaches users over Blocking the real-time response path.

Rationale: Generated text appears before the spoken audio, providing room to run safety checks without adding obvious user-perceived latency.

Use human labeling early

Chose Humans listening and labeling audio-agent behavior over Only automated evals from the beginning.

Rationale: The presenters say human labeling is the most effective evaluation method they have seen with customers and accelerates prompt iteration.

Gotchas (what bit them)

Chained audio architectures can feel too slow and lose conversational nuance.

Root cause: Separate transcription, LLM, and TTS calls compound latency and transform information between modalities.

Resolution: Use speech-to-speech real-time models when the use case prioritizes low latency and natural conversation.

Architecture choice changes by use case; real-time is not automatically best.

Root cause: Consumer apps optimize for latency and expressiveness, while customer-service systems may prioritize accuracy, determinism, and integrations.

Resolution: Choose between speech-to-speech and chained designs based on latency, cost, accuracy, user experience, and tooling constraints.

Voice agents need prompt controls that text agents do not.

Root cause: Voice quality includes sound, tone, pacing, enthusiasm, and expressiveness, not just content correctness.

Resolution: Experiment with voice-specific prompts, staged conversation instructions, and sample voices to tune brand behavior.

Context can become lossy during agent handoffs.

Root cause: A different backend model or specialist agent may not share the full conversational state from the real-time agent.

Resolution: Summarize conversation state and pass it during handoff.

Audio quality cannot be fully evaluated from text transcripts.

Root cause: Tone, pacing, intonation, and emotional expression matter to the user experience but may disappear in transcription.

Resolution: Combine transcription-based evals with audio-capable model checks and human listening.

Evaluation

The talk recommends a layered evaluation strategy: start with observability and traces of audio/agent behavior, use human labeling to iterate quickly, add transcription-based LLM-as-judge and function-call evals, add audio-specific tone/pacing checks with audio-capable models, and use synthetic real-time conversations for scale.

Metrics: Trace availability for audio, intermediate states, and tool calls. · Human labels on conversation quality and prompt issues. · Transcription-based rubric/judge scores. · Function-calling accuracy against business criteria. · Audio tone, pacing, and intonation checks. · Synthetic conversation coverage across customer personas.

Outcomes

• OpenAI positions speech-to-speech audio models as now good enough for scalable production applications.
• The talk provides architecture guidance for choosing speech-to-speech or chained designs by use case.
• The demonstrated pattern delegates hard customer-service work from a real-time frontline voice agent to backend specialist models.
• Customer examples highlight Lemonade's early eval/guardrail/feedback focus and Tinder's brand-realistic voice customization.

The source is a solution-architecture guidance talk rather than a single deployment report; scale is expressed as production readiness, customer examples, and the real-time API's new snapshot.