Building and Evaluating Production Voice Agents: From Custom Infrastructure to Platform Solutions

nomore_engineering · Healthcare · 2023 · source Agentic: yes · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Healthcare appointment voice agents

The team wanted a production phone agent for Polish primary-care appointment scheduling, where patients could not reliably reach receptionists and a real-time voice workflow had to handle medical scheduling safely and naturally.

Constraints:

• The source fetch failed, so this extraction relies on the ZenML summary.
• Voice agents need low response latency and natural conversation flow.
• Healthcare appointment scheduling has safety, availability, and language-quality requirements.
• Custom voice infrastructure is operationally complex to maintain.

Architecture

The team first built a custom voice-agent stack with speech-to-text, an LLM, text-to-speech, conversation orchestration, and testing. After comparing custom infrastructure with platforms such as Vapi.ai, Bland.ai, and Retell.ai, they chose a platform approach because maintaining the full voice stack was too complex.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  patient-caller[Patient Caller] --> voice-agent[Voice Agent]
  voice-agent --> speech-to-text[Speech To Text]
  speech-to-text --> conversation-orchestrator[Conversation Orchestrator]
  conversation-orchestrator --> llm[LLM]
  conversation-orchestrator --> text-to-speech[Text To Speech]
  text-to-speech --> voice-agent[Voice Agent]
  conversation-orchestrator --> testing-harness[Testing Harness]
  voice-platform[Voice Platform] --> voice-agent[Voice Agent]

Components

Data flow

A patient phone call is transcribed by speech-to-text, passed through LLM-driven conversation orchestration, converted back to speech through text-to-speech, and tested for latency, naturalness, script adherence, availability, and language support.

Why this architecture

The team built the full custom stack to understand the architecture and test requirements, then moved to a platform because operating STT, LLM, TTS, orchestration, latency, and availability together was too costly for the use case.

Models used: LLMs · Vapi.ai · Bland.ai · Retell.ai

Scale

• The platform comparison targeted sub-2-second voice responses.
  “• Conversation flow naturalness • Response latency (targeting sub-2-second responses) • Agent quality and script adherence • Platform availability • Language support quality”summary, stated
• The summary reports Bland.ai latency around 1.5 seconds versus 2.5 seconds or more for other platforms.
  “Bland.ai's self-hosted approach achieved better latency (1.5s vs 2.5s+) and availability but sacrificed flexibility in agent logic and evaluation capabilities.”summary, stated

Alternatives considered (roads not taken)

Maintain a custom STT/LLM/TTS/orchestration stack.

Rejected because: The team found the maintenance complexity too high after building the complete system.

Use Bland.ai's self-hosted platform.

Rejected because: The summary says it had better latency and availability but less flexibility in agent logic and evaluation.

Use Vapi.ai or Retell.ai style platforms.

Rejected because: These provided more LLM control but had latency and availability challenges according to the summary.

Patterns

tool-use-orchestration (voice workflow)

The agent orchestrates speech-to-text, LLM reasoning, text-to-speech, and appointment-flow logic.

eval-driven-development (testing)

The team compared custom and platform solutions using latency, naturalness, script adherence, availability, and language support.

latency-optimization (real-time systems)

Voice-agent platform evaluation centered partly on sub-2-second response latency.

fallback-degradation (build-vs-buy)

The team fell back from custom infrastructure to a platform solution when maintenance complexity outweighed benefits.

Tradeoffs

Infrastructure ownership

Chose Vapi.ai platform over Maintaining a custom voice-agent stack.

Rationale: The platform reduced the complexity of operating the full speech/LLM/orchestration stack.

Latency versus flexibility

Chose More LLM control where needed over Only optimizing for the fastest self-hosted latency.

Rationale: The platform comparison exposed that faster latency could come with less control over agent logic and evaluation.

Gotchas (what bit them)

Custom voice stacks have high operational surface area.

Root cause: Production voice agents combine STT, LLM, TTS, orchestration, latency, availability, and testing concerns.

Resolution: Adopt a third-party platform after learning from the custom build.

Latency and control trade against each other across voice platforms.

Root cause: Self-hosted approaches can reduce latency but limit agent-logic flexibility, while flexible LLM platforms may add latency.

Resolution: Compare platforms across latency, availability, language quality, script adherence, and evaluation support.

Healthcare appointment voice agents require stronger testing than demos.

Root cause: They operate in real-time patient scheduling contexts with language and script-adherence requirements.

Resolution: Use comprehensive testing approaches and platform comparisons before production deployment.

Evaluation

The team evaluated custom and third-party voice-agent approaches across conversation naturalness, latency, agent quality, script adherence, availability, and language support.

Metrics: Conversation flow naturalness · Response latency · Script adherence · Platform availability · Language support quality

Outcomes

• The team built a complete custom voice-agent stack.
• They selected Vapi.ai because maintaining the custom infrastructure was too complex.
• The case produced reusable lessons about latency, availability, flexibility, and evaluation in voice-agent platforms.

The summary reports sub-2-second latency targets and platform comparison figures of about 1.5 seconds versus 2.5 seconds or more.