Building a Production-Ready AI Phone Call Assistant with Multi-Modal Processing

realchar · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real-time consumer phone-call AI assistant

RealChar wanted Revia to complete customer-service phone calls for users, which required an AI agent that could handle real-time audio, button presses, text messages, latency spikes, unreliable generative models, and human takeover without breaking the phone conversation.

Constraints:

• Phone-call UX exposes delays of even about half a second
• The agent must process audio, button presses, and incoming text messages together
• LLM first-token latency is unstable and can spike for seconds
• Open-ended customer-service calls contain many edge cases
• Reliability must be validated before agents interact with real humans at scale
• The system needs real-time observability and fallback paths when models make wrong decisions

Architecture

RealChar applied self-driving-car and robotics patterns to a phone-call AI assistant. Audio, text, DTMF/button signals, and other streams enter a low-level streaming layer, are published to an event bus, and are processed in parallel on clock cycles by perception models, traditional ML/NLP signal generators, and generative AI models. A routing/fallback layer chooses whether to wait, respond, press buttons, or fall back to faster systems. A virtual-world simulator and millisecond-level observability support validation, debugging, synthetic data generation, and human takeover.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> revia-assistant[Revia Assistant]
  revia-assistant --> phone-call-interface[Phone Call Interface]
  phone-call-interface --> audio-streaming-layer[Audio Streaming Layer]
  audio-streaming-layer --> perception-models[Perception Models]
  perception-models --> event-bus[Event Bus]
  event-bus --> traditional-ml-signals[Traditional ML Signals]
  traditional-ml-signals --> generative-ai-models[Generative AI Models]
  event-bus --> generative-ai-models
  generative-ai-models --> fallback-router[Fallback Router]
  fallback-router --> revia-assistant
  virtual-world-simulator[Virtual World Simulator] --> real-time-observability[Real Time Observability]
  real-time-observability --> event-bus
  human-supervisor[Human Supervisor] --> revia-assistant
  real-time-observability --> human-supervisor

Components

Data flow

Phone-call streams arrive over WebSocket/WebRTC-like channels. The system processes recent audio or multimodal frames at regular intervals, emits signals onto an event bus, lets downstream components subscribe and publish responses, expands task context from real-time signals, calls generative models only when there is enough context, and uses fallback mechanisms when latency or model behavior crosses thresholds.

Why this architecture

RealChar treated phone-call agents like robotics systems because voice conversations are real-time, multimodal, and safety-sensitive. Parallel event processing keeps audio critical paths from blocking on slower model calls. Traditional ML provides fast signals, generative models handle open-ended language, and simulation plus observability turns hidden production failures into debuggable millisecond events.

Models used: GPT-4 Turbo · GPT-4o · traditional NLP models · generative AI models · speech-to-text models · text-to-speech models

Scale

• The product was in small beta testing and expected to appear on the App Store.
  “um uh we are like soon to have a uh version on App Store so I think by the time like you you you listen to the podcast you should be able to download that um but we're still doing like a small beta testing right now and uh”transcript, stated

  “that um but we're still doing like a small beta testing right now and uh hopefully I can give this like uh a assistant to power your daily life yeah and it's is it only in the US uh it's uh it's the App Store I think we”transcript, stated
• RealChar processes phone-call data on roughly 100-millisecond clock cycles.
  “right so once like 100 milliseconds is starting to process all the frames for the last 100 milliseconds you use that data to generate signals for the downstream uh systems damn that is wild so basically you're doing it at consistent intervals”transcript, stated
• GPT-4 Turbo first-token latency was typically 800 ms to 1 second in their benchmark.
  “they run and you were surprised GPT for Tero the first token they were running is about uh 800 millisecond to 1 second the first token right so that's way slow for way too slow for for the human conversation like the real time conversation and they were not really”transcript, stated
• GPT-4 Turbo latency could spike to 3-4 seconds about 20% of the time.
  “stable so they can go up to like 3 second to 4 second and we encounter this probably at 20% of time right that that's the worst for any AA agent right if you're trying to handle the audio bites um then they announced this GPD 40 right like the the the even fast model”transcript, stated
• GPT-4o initially showed about 300-500 ms first-token latency at launch.
  “like for the for the the launch date we were see uh seeing like 300 milliseconds like 500 millisecond for the first token but then it's starting to like getting slow again I think with the traffic like they starting receiving right so so that's why like I'm I will be super surp”transcript, stated

Alternatives considered (roads not taken)

Rely on generative models alone

Rejected because: Real-time phone calls need faster intent signals, validation, and context gathering before invoking slower generative models.

Use a simple model gateway

Rejected because: The router must consider task type, prompt, identified intent, request state, and whether to wait or press buttons.

Validate primarily against live human calls

Rejected because: The team wanted a safe virtual world where failures could be repeated, inspected, and used for training/validation.

Let slow model calls block the audio path

Rejected because: Users notice audio lag immediately, so the critical path needs parallel processing and fallbacks.

Patterns

async-message-orchestration (runtime-architecture)

RealChar uses an event bus so parallel systems can subscribe to relevant events and publish responses without blocking each other.

latency-optimization (voice-agent-runtime)

The architecture processes streams on clock cycles, keeps audio on a critical path, and uses fast fallbacks for latency spikes.

model-routing (model-operations)

Requests are routed by priority, prompt, intent, and latency state rather than through a single fixed model path.

fallback-degradation (reliability)

Slow or failing generative model calls trigger faster fallback systems so the call does not stall.

eval-driven-development (simulation)

RealChar uses a virtual world to validate agent behavior, inspect failures, and generate synthetic training data before live exposure.

human-in-the-loop-gate (safety)

Humans can supervise and take over when the AI makes wrong decisions during a call.

Tradeoffs

Use robotics-style parallel architecture

Chose event bus and clock-cycle processing over sequential conversational pipeline.

Rationale: Parallel processing keeps multimodal inputs and audio-critical paths moving while slower models work asynchronously.

Combine generative AI with traditional ML

Chose fast traditional NLP signals plus generative responses over end-to-end generative-only action selection today.

Rationale: The team expects eventual end-to-end models, but current production needs signals, validation, and context expansion.

Use simulation before real-world exposure

Chose virtual-world calls with recorded or synthetic counterparts over random live calls against real humans.

Rationale: Simulation lets the team generate failures safely, inspect exact failure moments, and collect data for future models.

Fallback from preferred models

Chose priority lists and faster fallback tiers over waiting indefinitely for the top model.

Rationale: A real-time phone call needs both accuracy and speed, and model availability/latency fluctuates.

Gotchas (what bit them)

Voice agents expose latency much more harshly than text agents.

Root cause: A half-second audio delay is immediately noticeable, while text delays of one or two seconds feel acceptable.

Resolution: RealChar uses parallel processing, fast speech systems, latency benchmarks, and fallback tiers.

Open-ended phone calls break narrow intent systems.

Root cause: The Google Duplex-era approach required many labeled intents and still failed on conversational edge cases.

Resolution: Revia combines generative AI, traditional ML signals, multimodal context, and fallback mechanisms.

Model-latency averages hide damaging spikes.

Root cause: GPT-4 Turbo often had acceptable first-token latency but could spike to 3-4 seconds around 20% of the time.

Resolution: The system tracks real-time latency and falls back beyond thresholds.

Demo reliability does not imply production reliability.

Root cause: Agents can produce appealing recordings while working only a small fraction of the time.

Resolution: RealChar spends most effort on repeatable simulation, validation, and observability.

Multimodal call state is broader than audio transcription.

Root cause: Real calls include DTMF/button navigation and text-message verification codes alongside speech.

Resolution: The perception system handles any input stream on the fly and gathers enough context before generation.

Evaluation

RealChar evaluates the system through latency benchmarks, simulation/virtual-world tests, millisecond-level observability, synthetic data generation, and human-supervised debugging of model or traditional-ML decision failures.

Metrics: first-token latency · latency spike frequency · clock-cycle processing interval · failure moment timestamp · model vs traditional ML failure attribution · call task completion · human takeover events

Outcomes

• RealChar built Revia as a beta consumer phone-call assistant for customer-service tasks.
• The architecture applies self-driving-car patterns: simulation, multimodal perception, parallel runtime, fallback recovery, and observability.
• The system handles audio streams, button presses, and incoming text context rather than plain transcription alone.
• Latency benchmarks showed why fallback and routing are required for real-time voice.
• The team built homegrown low-level audio streaming and observability systems for supervision and takeover.

The system was in small beta and optimized around real-time constraints such as 100 ms processing intervals, 800 ms to 1 second GPT-4 Turbo first-token latency, 3-4 second spikes about 20% of the time, and 300-500 ms GPT-4o launch first-token latency.