Building Low-Latency Voice AI Agents for Home Services

elyos_ai · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Voice agents for home services

Alias/Elyos AI needed production voice agents that could answer stressed home-services customers in real time, triage issues, book appointments, invoice, take payments, and trigger on-call engineers while keeping latency, state transitions, and handoffs reliable enough for phone conversations.

Constraints:

• Voice turn-taking has a human expectation around 100 milliseconds, while cascade voice agents add ASR, LLM, and TTS latency.
• Emergency home-services calls may require someone within one to four hours, and callers can be stressed.
• Calls can last 15-20 minutes, so deployments and worker changes must drain connections gracefully.
• Sensitive workflows such as invoicing and payments need deterministic state control and cannot rely on unconstrained model updates.
• Real conversations can change intent mid-call and require recovery paths, uncertainty handling, and human escalation.
• The UK/US voice environment needs accents, English-first operation, and reliable phone-quality speech handling.

Architecture

Alias AI built a vertical voice-agent platform for home-services businesses. A phone/email/message entry point reaches an orchestration layer connected to CRM/ERP systems, telephony, ASR, LLMs, TTS, backend workflow state machines, tools/functions, monitoring, and human escalation. The system uses cascade voice architecture, regional clustering, warm workers, dynamic LLM provider routing, just-in-time context, backend-managed state machines, post-runtime reconciliation, and a parallel fast LLM judge. The agents handle triage, emergency determination, appointment booking, invoicing, payment, on-call engineer notification, and resolution tracking, while humans can take over or approve sensitive steps.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  home-services-caller[Home-services caller] --> voice-agent-entry-point[Voice agent entry point]
  voice-agent-entry-point --> telephony-provider[Telephony provider]
  voice-agent-entry-point --> orchestration-layer[Orchestration layer]
  telephony-provider --> orchestration-layer
  orchestration-layer --> speech-to-text-service[Speech-to-text service]
  orchestration-layer --> llm-reasoning-layer[LLM reasoning layer]
  orchestration-layer --> text-to-speech-service[Text-to-speech service]
  orchestration-layer --> just-in-time-context-manager[Just-in-time context manager]
  orchestration-layer --> backend-state-machine[Backend state machine]
  orchestration-layer --> tool-function-layer[Tool/function layer]
  orchestration-layer --> parallel-llm-judge[Parallel LLM judge]
  llm-reasoning-layer --> tool-function-layer
  just-in-time-context-manager --> backend-state-machine
  backend-state-machine --> crm-and-erp-systems[CRM and ERP systems]
  backend-state-machine --> human-escalation-path[Human escalation path]
  backend-state-machine --> tool-function-layer
  human-escalation-path --> home-services-caller
  text-to-speech-service --> tts-quality-monitor[TTS quality monitor]
  observability-and-reconciliation[Observability and reconciliation] --> orchestration-layer
  observability-and-reconciliation --> tts-quality-monitor
  observability-and-reconciliation --> backend-state-machine

Components

Data flow

A customer calls, emails, or messages a home-services provider. The voice worker is already warm and regionally close to the telephony provider. Audio goes through speech-to-text, the orchestration layer injects only current workflow context, the LLM reasons or selects tools, backend state machines execute deterministic steps, and text-to-speech speaks the response. For an emergency boiler call, the agent identifies the customer, decides whether the callout is an emergency, checks contract/location/coverage/cost, books the appointment, invoices, takes payment, and starts an on-call process. A fast parallel judge, post-call quality assessment, and human escalation paths monitor whether the conversation stays on track.

Why this architecture

The team stayed with a cascade architecture because it let them choose best-in-class ASR, LLM, and TTS components for latency, accents, language, and quality, even though end-to-end models promise fewer round trips. Production reliability came from orchestration rather than any one model: regional clustering, provider routing, warm starts, state machines, just-in-time context, human escalation, and parallel judging were all designed to keep conversations fast, deterministic, recoverable, and auditable.

Models used: speech-to-text models · Deepgram · Speechmatics · Gladia · AssemblyAI · OpenAI models · GPT-5 · GPT-5.1 · GPT-4.1 · Gemini real-time models · GPT-4o-mini real-time model · text-to-speech models · Cartesia

Scale

• The agents answer calls, emails, and messages 24/7.
  “we are able to answer all calls, emails, and messages 24/7.”transcript, stated
• Emergency calls may need service within one, two, or four hours rather than 36 hours.
  “The first available appointment is probably on Monday and that is like 36 hours away”transcript, stated

  “Sometimes might be like within one or two hours, sometimes maybe uh within four hours”transcript, stated
• Human conversation turn-taking was described as about 100 milliseconds.
  “humans are really used to converse in a very specific way. uh and each turn probably takes about 100 milliseconds.”transcript, stated
• Speech-to-text latency was typically 100-300 milliseconds.
  “speech to text is going to be anywhere between 100 and 300 millconds”transcript, stated
• Text-to-speech latency ranged from about 200 to 600 milliseconds.
  “starting from 200 millconds going all the way up to four five or 600 millconds”transcript, stated
• Best-case P90 total latency was estimated at 500-1200 milliseconds.
  “in a best case scenario the the P90 range here would be somewhere between 500 to 1200 milliseconds”transcript, stated
• Calls can last 15 or 20 minutes, which affects deployment and fallback strategy.
  “you might have calls lasting for 15 or 20 minutes, right?”transcript, stated
• Emergency classification may inject the top five, ten, or twenty job types as step-specific context.
  “These are the five top jobs or 10 or 20 jobs that you're going to do that.”transcript, stated
• Some customers have 0% transfers and can let agents handle 100% of traffic.
  “we have some customers that we have 0% transfers”transcript, stated

  “So the agents can handle like 100% of the traffic.”transcript, stated
• On average about 15% of calls require a human touch point.
  “I would say about 15% of all calls require a human um kind of touch point.”transcript, stated
• High-value solar interactions can involve $20,000-30,000 projects.
  “because we're talking maybe for 20 or 30,000 projects right”transcript, stated
• Custom TTS voices typically need two or three hours of high-quality audio.
  “on average like most of the um models can do pretty good with two or three hours of audio.”transcript, stated

Alternatives considered (roads not taken)

Use end-to-end real-time voice models.

Rejected because: They promised latency gains but were not mature enough for multi-turn function calling, languages, accents, and custom ASR needs at the time.

Use complicated RAG for workflow context by default.

Rejected because: Wrong retrieval can introduce cascading errors, and many vertical workflows can inject deterministic just-in-time context based on intent instead.

Let the agent automatically update sensitive workflow state.

Rejected because: Invoicing, payments, and similar operations need backend state-machine control and deterministic success/failure criteria.

Use general orchestration frameworks in the voice path.

Rejected because: Most frameworks added too much complexity and latency for real-time voice, though they are acceptable for email or text communication.

Transfer every uncertain call directly to a human.

Rejected because: The platform can also use human-in-the-loop approvals, callback decisions, post-call review, or instructions without handing over the entire call.

Patterns

latency-optimization (operations)

The system reduces end-to-end voice latency through warm workers, regional clusters near telephony providers and customers, local tool placement, LLM endpoint routing, and TTS quality monitoring.

model-routing (model-operations)

Alias monitors provider and regional endpoint performance and routes among OpenAI-hosted endpoints, Azure OpenAI endpoints, and other providers based on current latency.

context-compaction (context-management)

The voice workflow uses just-in-time context: inject only the current step's facts, turn conclusions into deterministic state, summarize tool results, and remove stale context.

tool-use-orchestration (agent-architecture)

A custom workflow engine combines deterministic functions and stochastic model steps so agents can move through customer identification, job creation, cost/coverage reasoning, booking, invoicing, and payments.

human-in-the-loop-gate (safety)

The system includes direct human transfer, human approval/instructions inside workflows, post-call review, and callbacks for complaints, high-value projects, and unsupported systems.

llm-as-judge (evaluation)

A fast LLM runs in parallel with the live conversation to monitor whether the interaction is staying on the expected path and pull the primary model back if needed.

observability-tracing (observability)

Alias monitors LLM latency, TTS quality, all calls after conversation, groundedness, conversation quality, sentiment, outcomes, and common failures.

fallback-degradation (reliability)

Recovery paths include uncertainty handling, post-runtime reconciliation, human transfer or approval, callbacks, and state-machine error states rather than repeated clarification loops.

Tradeoffs

Use cascade voice architecture for production.

Chose separate ASR, LLM, and TTS components over end-to-end real-time voice models.

Rationale: Cascade adds round trips but lets the team pick custom ASR, model, and TTS components for accents, languages, provider reliability, and latency while end-to-end models were not yet reliable enough.

Keep sensitive state updates in the backend.

Chose backend state machines with deterministic and stochastic steps over LLM-controlled state mutation for payments and invoices.

Rationale: The LLM can reason about intent and provide feedback, but payment, invoice, identification, and booking workflows need reliable sequencing and success criteria.

Use just-in-time context instead of broad prompt stuffing.

Chose minimal prompt plus intent-scoped context in/out over keeping full workflow history, all job types, and all tool results in context.

Rationale: The agent needs current facts, not every previous explanation; pruning obsolete context reduces confusion and latency.

Specialize vertically in home services.

Chose deep home-services workflows over horizontal general-purpose voice agents.

Rationale: Vertical specialization let the team model emergency callouts, booking, invoices, payments, on-call engineer dispatch, contracts, and home-service-specific edge cases.

Preserve natural interruption perception rather than trying to eliminate interruptions.

Chose observer-style TTS cancellation at word or sentence boundaries over abrupt immediate cancellation of generated speech.

Rationale: The team frames interruption handling as managing perception: the generated voice should pause naturally instead of cutting off mid-sound.

Gotchas (what bit them)

A simple cascade pipeline is too slow and fragile for natural voice conversations.

Root cause: ASR, LLM, and TTS each add latency, while real-world calls include noise, unclear speech, changing intent, and stressed callers.

Resolution: Treat orchestration as the core system: warm starts, regional clusters, provider routing, local tools, TTS monitoring, and workflow-aware recovery.

Text-to-speech latency and quality can make conversations feel unnatural even when the LLM is fast.

Root cause: Generated voice can spike or become inconsistent, creating clanky call behavior.

Resolution: Monitor TTS quality after calls, test providers, and treat TTS as a first-class latency/quality surface rather than focusing only on LLM time-to-first-token.

Leaving old tool results in context confuses subsequent tool calls.

Root cause: The model sees obsolete results and multiple call IDs when the same tool is called later with different parameters.

Resolution: Summarize or remove tool-call results once they are no longer needed and keep only deterministic current facts.

Sub-agent architectures can lose context in complex voice workflows.

Root cause: The main agent may not know what happens inside sub-agents until they finish, which risks dropped context and unclear handoffs.

Resolution: Keep general context controlled in the backend or another model, use smart summarization, and define clean handoff outcomes.

Repeatedly asking callers to clarify hard-to-transcribe fields damages the call experience.

Root cause: Speech-to-text still struggles with emails and UK postcodes, especially over phone audio.

Resolution: Use an uncertainty path instead of asking indefinitely, then escalate, callback, or reconcile after the call when appropriate.

Generic frameworks can silently break a latency-critical voice product.

Root cause: Framework abstractions add complexity and latency that may be acceptable for email or WhatsApp but not real-time speech.

Resolution: Build or choose orchestration carefully for the voice path, and keep slower frameworks to non-voice communication where latency tolerance is higher.

Evaluation

The system is evaluated through operational voice metrics, post-call monitoring, human-agent benchmarking, parallel judging, and outcome tracking. The team monitors time-to-first-token, groundedness, hallucination/script adherence, interruption quality, word repetition, task completion, sentiment, common failures, TTS quality, transfers/human touch points, and post-runtime reconciliation.

Metrics: speech-to-text latency · LLM time-to-first-token · text-to-speech latency · P90 end-to-end latency · groundedness · hallucination and script adherence · conversation interruptions · word repetition · task outcome success · caller sentiment · common failure modes · human transfer or touch-point rate · TTS quality

Outcomes

• Alias built voice agents for home-services verticals such as plumbers, electricians, HVAC installers, and fire/security companies.
• The agents handle 24/7 calls, emails, and messages; triage issues; book appointments; send invoices; take payments; and trigger on-call engineer processes.
• The architecture combines cascade ASR/LLM/TTS components with regional clustering, dynamic provider routing, just-in-time context, backend state machines, human escalation, and parallel judging.
• The team reports some customers at 0% transfers and agents handling 100% of traffic, with an average of about 15% of calls needing human touch points.
• The platform monitors calls after conversations and uses operational metrics such as groundedness, interruptions, sentiment, outcomes, and common failures to improve quality.
• The team found cascade production reliability better than end-to-end models for its requirements, especially around accents, languages, and function-calling maturity.

The source reports 24/7 coverage, emergency windows of one to four hours versus 36-hour delays, 100ms human turn-taking, 100-300ms ASR latency, 200-600ms TTS latency, 500-1200ms best-case P90 total latency, 15-20 minute calls, some customers with 0% transfer and 100% automation, about 15% human touch points on average, $20k-30k high-value projects, and two to three hours of audio for custom TTS voices.