Building Production AI Agents with Advanced Testing, Voice Architecture, and Multi-Model Orchestration

sierra · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise customer-service agents across voice and chat

Sierra needed to build production customer-service agents for large consumer enterprises where AI behavior is expensive, slow, nondeterministic, and safety-sensitive, while voice channels add real-time latency, interruption, accent, noise, and turn-taking constraints. The platform had to support both no-code business ownership and code-level integrations, with testing and release processes strong enough for enterprise change management.

Constraints:

• AI agents are slower, more expensive, and nondeterministic compared with traditional deterministic software.
• Voice agents must handle background noise, accents, microphone quality, interruptions, pauses, and conversational cues.
• Customer-service agents need deterministic permission boundaries around sensitive data and actions.
• Model and prompt changes require simulations, release processes, and customer-specific quality review before production.
• A single response may require many models, retrieval calls, classification decisions, and reasoning steps under tight latency.
• Enterprise customers need both business-owned no-code workflows and engineering-owned code integrations.

Architecture

Sierra's platform combines a goals-and-guardrails agent architecture, a constellation of parallel model calls, a modular voice stack, simulation-based testing, deterministic access control, and a no-code/code builder. User messages enter an agent runtime that fans out to embedding/RAG lookups, fine-tuned classifiers, frontier reasoning models, speech/transcription/synthesis components, and tools. Voice latency is reduced through speculative execution: retrieval and other likely work can run while a smarter model decides whether it was needed. Quality is controlled through simulations with a user agent, the tested agent, and an evaluator agent, including voice-environment variation, red-team/verbatim simulations, and CI/CD critical simulations. Production transfers and human actions feed a knowledge/SOP learning loop that suggests missing articles and improvements.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  customer-caller-or-chatter["Customer caller or chatter"] --> sierra-agent-runtime["Sierra agent runtime"]
  sierra-agent-runtime --> constellation-model-orchestrator["Constellation model orchestrator"]
  constellation-model-orchestrator --> rag-lookup-tool["RAG lookup tool"]
  constellation-model-orchestrator --> fine-tuned-classification-models["Fine-tuned classification models"]
  constellation-model-orchestrator --> frontier-reasoning-models["Frontier reasoning models"]
  sierra-agent-runtime --> deterministic-access-control-layer["Deterministic access-control layer"]
  sierra-agent-runtime --> voice-transcription-and-synthesis-pipeline["Voice transcription and synthesis pipeline"]
  voice-transcription-and-synthesis-pipeline --> speech-to-speech-model-adapters["Speech-to-speech model adapters"]
  simulation-test-harness["Simulation test harness"] --> user-simulator["User simulator"]
  simulation-test-harness --> sierra-agent-runtime
  simulation-test-harness --> evaluator-agent["Evaluator agent"]
  simulation-test-harness --> abuse-detection-models["Abuse detection models"]
  knowledge-and-sop-learning-loop["Knowledge and SOP learning loop"] --> rag-lookup-tool
  sierra-agent-runtime --> knowledge-and-sop-learning-loop
  business-review-teams["Business review teams"] --> knowledge-and-sop-learning-loop
  business-review-teams --> no-code-and-code-builder["No-code and code builder"]
  no-code-and-code-builder --> sierra-agent-runtime
  no-code-and-code-builder --> enterprise-release-workflow["Enterprise release workflow"]
  enterprise-release-workflow --> simulation-test-harness

Components

Data flow

A customer interacts with a Sierra agent over chat or voice. For voice, audio is transcribed because API/tool calls cannot use voice tokens directly; synthesis produces the response. The agent runtime interprets goals and guardrails, invokes a constellation of models in parallel, and may speculatively run retrieval while another model decides whether retrieval is needed. Access to sensitive data is constrained by deterministic permission systems matching what the logged-in user could see on the website. Before release, simulations create a user-agent/agent-under-test/evaluator-agent loop that can run many times and in parallel, including background noise, accents, adversarial scripts, and customer-specific abuse models. After deployment, transfers to humans and business reviews reveal missing SOPs or knowledge-base gaps, which feed back into drafts and prioritized improvements.

Why this architecture

The architecture exists because production voice/chat agents need both AI-native flexibility and traditional deterministic controls. Simulations replace deterministic unit-test assumptions where model behavior varies; access control stays deterministic where privacy and permissions cannot vary. Voice forced speculative, parallel execution to meet latency, and that same constellation architecture now powers other channels. No-code/code isomorphic tooling lets business teams own goals and guardrails while engineers implement exact integrations.

Models used: LLM agent model · Embedding model · Frontier reasoning model · Fine-tuned classification model · LLM-as-judge evaluator · GPT audio model · Gemini audio model · Speech-to-speech model · Text-to-speech model

Scale

• Simulations may repeat the same conversation 5, 10, or 15 times to account for nondeterminism.
  “We might simulate a given conversation 5 10 15 times”transcript, stated
• The CI/CD simulation system can run 300 simulations in parallel.
  “So if you have 300 simulations that you need to run, you're not waiting for them to run one after the other. They're going off and being executed in parallel.”transcript, stated
• Voice simulation confidence can come from 5/10/15 repeats over 300 to 400 scenarios.
  “if you run it 5 10 15 times over 3 400 simulations, you have a pretty good idea that if those are all passing”transcript, stated
• A single Sierra response may invoke 10, 15, or 20 models.
  “Between me finishing talking and the agent responding, there might be 10, 15, 20 different models that get invoked.”transcript, stated
• Sierra reports working with hundreds of customers.
  “what we've seen working with hundreds of customers now is when we bring together the business side and the technical side”transcript, stated
• Most customers are described as large consumer companies with millions of customers and over a billion dollars in revenue.
  “most of our customers have millions of their own customers. Um most of our customers have over a billion dollars in revenue.”transcript, stated
• Some customers have 10+ full-time people evaluating conversations and improvements.
  “we have customers where they have, you know, 10 plus full-time people evaluating conversations and looking for opportunities for improvement”transcript, stated
• One top-10 U.S. business went from first meeting to live deployment in about three months.
  “one of the 10 largest businesses in the United States. We went from the, you know, basically first meeting with them to live in about 3 months.”transcript, stated
• Voice latency tradeoffs distinguish 5-second responses from 1.5-second responses.
  “in voice, 5 seconds and you know 1.5 seconds are very different.”transcript, stated

Alternatives considered (roads not taken)

Traditional unit tests as the main quality gate

Rejected because: Unit tests pass once in deterministic software, while Sierra needed repeated conversation simulations for nondeterministic agents.

AI-generated adversarial behavior for all red-team cases

Rejected because: The team found models are not good enough at inventing terrible or hijacking behavior, so important abuse cases use verbatim scripted simulations.

Model-mediated access to sensitive data

Rejected because: Sierra keeps sensitive-information access deterministic and gives the model only the sandbox available to the authenticated user.

Speech-to-speech as the default production voice architecture

Rejected because: Sierra believes speech-to-speech is the future but production hallucination remains too high for larger customers.

Serially run every model call in the response path

Rejected because: Serial orchestration would be too slow for voice, so Sierra fans work out in parallel and uses speculative execution.

Decision trees as the literal implementation model

Rejected because: Decision trees are useful conceptual tools, but Sierra says goals-and-guardrails and a web of models better describe the actual agent architecture.

A pure developer platform

Rejected because: Enterprise adoption needed operations/CX teams to own agent behavior, while developers still handled specialized API and tool integrations.

Fine-tune customer-specific or time-sensitive facts into model weights

Rejected because: Weights can regurgitate data and become stale; fine-tuning is reserved for durable, high-leverage style cases.

Patterns

eval-driven-development (evaluation)

Sierra uses repeated conversation simulations, critical CI/CD simulations, red-team simulations, and task-level evaluation to control nondeterministic agent behavior before release.

llm-as-judge (evaluation)

The simulation framework includes an evaluator agent that judges whether a simulated conversation satisfied checklists.

model-routing (orchestration)

Sierra routes parts of a response through embedding, frontier reasoning, fine-tuned classification, and speech models, selecting the right model per task.

latency-optimization (runtime)

Voice latency forces Sierra to run likely work in parallel and speculatively, such as launching retrieval while a smarter model decides whether retrieval is needed.

safety-policy-enforcement (safety)

Sensitive access is enforced by deterministic permission systems, and abuse scenarios are tested with customer-specific abuse models and verbatim simulations.

data-quality-loop (learning-loop)

Sierra learns from human transfers and customer reviews to identify missing knowledge, draft new articles, and prioritize improvements.

human-in-the-loop-gate (quality)

Business reviewers, customer experts, and human callers participate in judging conversation quality, language/dialect fit, and agent improvement opportunities.

fine-tuning-adaptation (model-adaptation)

Sierra fine-tunes selectively for durable, high-leverage voice style and tone, while keeping customer-derived data in customer-specific models.

Tradeoffs

Testing nondeterministic agents

Chose Repeated simulation-based tests with user, agent, and evaluator agents over Traditional single-run unit tests.

Rationale: Conversation behavior varies across runs, so Sierra needs repeated simulations and LLM-judge checklists rather than one deterministic pass.

Protecting sensitive customer data

Chose Deterministic permission systems over Prompt/model judgment about what data may be exposed.

Rationale: Permissions are not an AI-native problem; the model should only receive data/actions the authenticated user already has.

Voice model architecture

Chose Modular transcription/synthesis with production text-to-speech where reliable over Immediate broad speech-to-speech deployment.

Rationale: Sierra believes speech-to-speech will win long term, but hallucination risk keeps text-to-speech as the safer large-customer production path.

Meeting voice latency requirements

Chose Parallel speculative execution across models and tools over Serially deciding and then executing each step.

Rationale: Voice cannot tolerate avoidable seconds of delay, so Sierra launches likely work before all decisions are finalized.

Representing agent behavior

Chose Goals and guardrails with a web of model decisions over Rigid decision trees or legacy chatbot flows.

Rationale: Decision trees help humans reason about flows, but Sierra's runtime behaves more like an interconnected cloud of model/tool decisions constrained by goals and guardrails.

Platform ownership model

Chose Isomorphic no-code plus code tooling over A one-way choice between developer platform and business-user builder.

Rationale: Business teams understand goals and SOPs, engineers understand integrations; Sierra needs both without trapping teams in one mode.

Pricing and quality incentives

Chose Outcome-based pricing over Charging without tying revenue to successful agent completion.

Rationale: The model creates cash-risk when agents fail, but Sierra views that as an earlier signal and stronger alignment with customers.

Where to apply fine-tuning

Chose Selective fine-tuning for durable voice style over Putting broad customer facts or promotions into weights.

Rationale: Style is durable and high leverage, but weights can leak or become stale when business facts change.

Gotchas (what bit them)

Traditional pass-once unit tests do not capture nondeterministic conversation behavior.

Root cause: LLM agents may produce different outcomes across runs, so one deterministic pass/fail execution is too weak.

Resolution: Run simulations multiple times, use user/agent/evaluator agents, and execute critical simulations in CI/CD.

Voice test coverage must include real-world channel noise and user variability.

Root cause: Customers call from cars, streets, different accents, and different microphones, so text-style tests miss voice failures.

Resolution: Add libraries of background noise, voices, accents, and microphone-quality variations to voice simulations.

AI user agents are poor at inventing worst-case abuse.

Root cause: Model safeguards and model behavior make generated adversarial prompts less realistic for the most important attacks.

Resolution: Use verbatim scripted simulations for prompt hijacking and other behaviors that must be prevented.

Sensitive data access cannot be delegated to model judgment.

Root cause: A model can hallucinate or be manipulated, so it must not decide whether protected data is exposed.

Resolution: Use traditional deterministic access controls that expose only the same data the logged-in user could access on the website.

Naive voice turn-taking creates awkward stop/start behavior.

Root cause: Interruption, encouragement, redirection, and long pauses cannot be reduced to a universal millisecond threshold.

Resolution: Use conversational context and customer/channel-specific tuning to decide whether to wait, respond, or handle interruption.

Speech-to-speech models are not yet reliable enough for large-customer production use.

Root cause: Production tests show hallucination levels that exceed Sierra's acceptable risk for larger customers.

Resolution: Keep a modular architecture that supports speech-to-speech but use text-to-speech as the reliable production path where needed.

Benchmarks alone do not reveal the best new model configuration.

Root cause: A model may need different prompting, few-shot examples, or fine-tuning before its true ceiling is visible.

Resolution: Run a suite of evals and iterate for days or weeks before deciding whether a new model beats the previous local maximum.

Outcome-based pricing turns agent failures into immediate business pain.

Root cause: If the agent fails to complete the task, Sierra may spend money without getting paid.

Resolution: Treat the pain as an early warning signal and incentive alignment mechanism that forces issues to be found and fixed before renewal.

Fine-tuning can leak or stale customer-specific content.

Root cause: Information in weights can be regurgitated and time-sensitive offers become outdated when circumstances change.

Resolution: Use customer-specific models for customer-derived data and reserve weight-level adaptation for durable, high-leverage style problems.

Evaluation

Sierra evaluates through simulation, adversarial testing, deterministic permission tests, model-release eval suites, language/dialect human calling, and production learning loops. Simulations are multi-agent: a user agent drives the conversation, the Sierra agent responds, and an evaluator agent judges checklist completion. Critical simulations run in CI/CD, and production reviews/human transfers feed back into knowledge-base and SOP improvements.

Metrics: Conversation simulation pass behavior across 5, 10, or 15 repeated runs. · Data changes caused by simulated conversations. · Checklist completion judged by an evaluator agent. · Parallel critical simulations during merge or release. · Voice robustness across background noise, voices, accents, and microphone quality. · Customer-specific abuse categories such as self-harm, bullying, or hacking. · Language/dialect realism for model tests with human callers. · Task completion and knowledge/SOP gaps after human transfer. · Customer outcomes under outcome-based pricing.

Outcomes

• Sierra built a simulation system that can repeat scenarios and run hundreds of simulations in parallel.
• The voice architecture evolved from back-and-forth conversations to handling interruptions, verbal cues, pauses, and context-dependent turn-taking.
• A modular voice stack lets Sierra support speech-to-speech models while keeping text-to-speech for reliable large-customer production.
• The constellation architecture invokes 10-20 models per message and uses parallel speculative execution to reduce voice latency.
• Post-transfer learning produces draft knowledge articles and prioritized missing-knowledge lists.
• No-code/code isomorphic tooling shifted most agent creation to no-code/customer-owned workflows while retaining code integrations.
• Enterprise change-management integration enabled at least one top-10 U.S. business to go from first meeting to live in about three months.

The transcript claims Sierra works with hundreds of customers, most with millions of customers and over $1B in revenue. It cites simulation repetitions of 5/10/15 across 300-400 scenarios, 300 simulations in parallel, 10-20 models per response, customers with 10+ full-time reviewers, a top-10 U.S. business live in about three months, and voice latency differences between 5 seconds and 1.5 seconds.