Voice AI Agent Development and Production Challenges

various_(canonical,_prosus,_deepmind) · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Voice AI agents and conversational production systems

Voice AI teams need to move beyond impressive demos to production systems that handle low-latency interaction, tool calls, ambiguous spoken input, emotional nuance, multilingual variation, and clear product scope.

Constraints:

Architecture

The panel's recommended architecture for production voice agents combines real-time voice APIs, structured prompting, streaming transports, asynchronous tool execution, alternate input channels for hard data, broad synthetic test sets, and production observability. For healthcare-style support, RAG over curated documents plus escalation to human appointments keeps the agent within a bounded domain.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  voice-user["Voice user"] --> real-time-voice-agent["Real-time voice agent"]
  real-time-voice-agent --> streaming-transport["Streaming transport"]
  real-time-voice-agent --> tool-execution-layer["Tool execution layer"]
  tool-execution-layer --> external-task-systems["External task systems"]
  real-time-voice-agent --> structured-input-collector["Structured input collector"]
  real-time-voice-agent --> rag-knowledge-base["RAG knowledge base"]
  synthetic-voice-test-suite["Synthetic voice test suite"] --> real-time-voice-agent
  real-time-voice-agent --> observability-platform["Observability platform"]
  synthetic-voice-test-suite --> observability-platform
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Voice user Speaks requests, corrections, emotional signals, and interruptions. Real-time voice agent
Real-time voice agent Handles speech interaction, structured prompting, interruption handling, and user feedback. Voice user, Streaming transport, Tool execution layer, Structured input collector, RAG knowledge base
Streaming transport Uses WebSocket or WebRTC-style updates to avoid silent waits during agent work. Real-time voice agent
Tool execution layer Runs short tool calls synchronously and slow operations asynchronously with follow-up. Real-time voice agent, External task systems
External task systems Web search, purchasing, booking, or appointment workflows used by voice agents. Tool execution layer
Structured input collector Collects difficult fields through structured prompts or alternate app inputs. Real-time voice agent
Synthetic voice test suite Simulates languages, emotions, background noise, dialects, and speaking styles. Real-time voice agent, Observability platform
RAG knowledge base Grounds bounded support agents in curated domain documents. Real-time voice agent
Observability platform Shows production behavior when real humans interact with agents. Real-time voice agent, Synthetic voice test suite

Data flow

User speech enters a real-time voice agent; structured prompts guide task state and field collection; short or streaming tool calls provide immediate feedback; long operations can detach and follow up asynchronously; observability captures real-user production interactions; and bounded RAG systems escalate out-of-scope questions to humans.

3 supporting quotes
“Using WebSocket or WebRTC connections to provide real-time updates to users.”summary, stated
“For particularly slow processes like completing a purchase, the agent informs the user upfront, allows them to disconnect, and follows up via SMS when complete.”summary, stated
“Appropriate escalation: Questions outside the documentation scope triggered handoff to schedule human appointments.”summary, stated

Why this architecture

Voice agents require a different architecture from text agents because users expect immediate conversational feedback and spoken input is ambiguous. Streaming, asynchronous tool handling, structured collection, and observability compensate for those voice-specific failure modes.

“Voice adds multiple additional layers of complexity on top of this foundation.”summary, stated
“When an agent needs to perform an action or retrieve information, the traditional approach of waiting for the tool to complete before responding simply doesn't work in voice contexts.”summary, stated

Models used: OpenAI Real-time API · Voice AI models · Speech synthesis models · RAG over curated documents · Synthetic data generation with OpenAI real-time API

Scale

Alternatives considered (roads not taken)

Reuse text-agent prompts unchanged for voice

Rejected because: Prosus tried this with the real-time API and it failed badly because voice lacks text's visible structure and control.

“When they attempted to use the same text prompts that worked for their text-based agent with OpenAI's real-time API for voice, it "failed miserably."”summary, stated

Wait silently for tools to finish

Rejected because: Voice users become frustrated without immediate feedback, so slow work needs streaming or asynchronous follow-up.

“Users become frustrated if the agent doesn't respond immediately, which has major implications for tool calling in LLM-powered agents.”summary, stated

Ask users to spell difficult identifiers by voice

Rejected because: Letters are especially unreliable for current voice models.

“Letters are particularly problematic for current models to both speak and understand.”summary, stated

Patterns

tool-use-orchestration (agent architecture)

Voice agents must orchestrate tools with streaming feedback and asynchronous completion instead of blocking the conversation.

“This represents a significant departure from typical LLM application architectures where tool calls can happen synchronously without immediate user feedback requirements.”summary, stated

async-message-orchestration (voice workflow)

Slow actions such as purchases can be detached from the live call and completed with SMS follow-up.

“For particularly slow processes like completing a purchase, the agent informs the user upfront, allows them to disconnect, and follows up via SMS when complete.”summary, stated

synthetic-data-generation (testing)

Prosus uses synthetic real-time voice interactions to test languages, emotions, and speaking styles.

“Synthetic test data generation: Using OpenAI's real-time API to simulate different interaction scenarios with varied languages, emotions, and speaking styles.”summary, stated

observability-tracing (production monitoring)

Canonical's tooling focuses on seeing production behavior when real users interact with voice agents.

“Tom from Canonical emphasized the importance of observability in production, noting that his company helps builders "see what's happening in production when real humans are interacting with their agents."”summary, stated

rag-retrieval (healthcare support)

The prostate-cancer recovery example grounds answers in curated medical recovery documents and escalates out-of-scope questions.

“• Retrieval-Augmented Generation (RAG): Built over curated medical documents about the recovery process.”summary, stated

Tradeoffs

Asynchronous follow-up for slow operations

Chose Inform users, let them disconnect, and follow up via SMS over Keeping users waiting on a live voice call.

Rationale: Voice agents need to preserve conversational momentum and avoid silent waits.

“Asynchronous processing for slow operations: For particularly slow processes like completing a purchase, the agent informs the user upfront, allows them to disconnect, and follows up via SMS when complete.”summary, stated

Use alternate modalities for hard input

Chose Application interfaces and phone-number recognition over Voice spelling of difficult identifiers.

Rationale: Spoken letters and complex identifiers are a known weak point for current models.

“User authentication via phone number: Recognizing users through their phone number rather than having them spell out identifying information.”summary, stated

Scope voice products narrowly

Chose Strong MVPs matched to appropriate voice characteristics over Comprehensive, vague voice-agent products.

Rationale: Voice complexity makes broad product definitions more likely to fail.

“Monica emphasized being "eternally frustrated with very vague product definitions" because voice is so complex that trying to build a comprehensive solution often leads to failure.”summary, stated

Gotchas (what bit them)

Text prompts did not transfer to voice interactions.

Root cause: Voice lacks the structured, visible context that text interfaces provide and adds latency, interruption, and UX dimensions.

Resolution: Design voice-specific prompts, feedback loops, and interaction patterns rather than porting text-agent prompts.

“The key insight is that with text, everything is nicely structured and visible, allowing for significant control.”summary, stated
“Voice interactions lack this structure and introduce entirely new dimensions of user experience that must be carefully managed.”summary, stated

Silent tool latency breaks voice UX.

Root cause: Users expect conversational immediacy and become frustrated when the agent does not respond.

Resolution: Shorten tool calls, stream progress, and move slow workflows to asynchronous follow-up.

“• Minimizing tool call duration: Tool calls need to be as short as possible because users require constant feedback during the interaction.”summary, stated
“• Leveraging streaming events: Using WebSocket or WebRTC connections to provide real-time updates to users. For example, when starting a web search, the agent can inform the user that it's beginning and allow them to interrupt or interact during the process.”summary, stated

Spoken identifiers and ambiguous input are unreliable.

Root cause: Voice models can mishear names, addresses, spelling, dialects, and background-noise-distorted speech.

Resolution: Avoid spelling flows, authenticate through phone number, collect fields structurally, or move sensitive fields to app interfaces.

“Voice input can be vague and difficult to interpret, especially when dealing with postal addresses, names, and spelling.”summary, stated
“• Leveraging other input modalities: Having users enter sensitive or complex data through an application interface rather than voice.”summary, stated

Demo behavior does not predict production voice behavior.

Root cause: Real users bring noise, dialects, emotions, interruptions, and unpredictable behavior absent from controlled tests.

Resolution: Build synthetic tests across those variations and add production observability for real interactions.

“Testing emerged as a major topic, with panelists acknowledging that voice agents behave completely differently in production compared to controlled development environments. The gap between demo quality and production quality is substantial.”summary, stated
“Testing with different types of background noise, reflecting real-world conditions like a food delivery driver in traffic in São Paulo.”summary, stated

Evaluation

The panel recommends synthetic test generation across languages, emotions, dialects, and environments, plus production observability to see real user behavior.

Metrics: Latency responsiveness · Language and dialect coverage · Background-noise robustness · Emotional-state handling · Production interaction traces

3 supporting quotes
“Dialect and language coverage: Ensuring testing covers regional variations, as models are typically trained primarily on English and may perform poorly with other languages or dialects.”summary, stated
“Emotional variation: Testing how agents handle users in different emotional states, including anger or frustration.”summary, stated
“controlled testing cannot fully replicate the unpredictability of real user interactions”summary, stated

Outcomes

The panel draws on Prosus's one-billion-customer operating context, Google's speech synthesis experience, and production tooling experience from Canonical.

“The discussion highlights that while voice AI models are impressive, significant craftsmanship is required to move from MVP to production-ready systems.”summary, stated
“Success requires attention to latency at every step of the pipeline, careful management of tool calling patterns, robust testing across languages and emotional states, and clear product scoping.”summary, stated