Building Production-Grade Customer Experience Agents at Enterprise Scale

sierra · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise customer-experience agents across voice, chat, commerce, support, and loyalty

Sierra needed a production platform for brand-facing customer-experience agents that could serve large enterprises across support, sales, loyalty, payments, voice, chat, multimodal interfaces, and agent-to-agent commerce while staying low-latency, governed, extensible, evaluable, and measurable against business outcomes.

Constraints:

Architecture

Sierra layers a customer-facing agent platform around Agent OS, Agent SDK, no-code Journeys, Ghostwriter, Analyze/Build/Release workspaces, simulations, monitors, Explorer, an Agent Data Platform, memory, voice pipelines, payments isolation, model routing, and enterprise governance. Agent OS translates each turn into prompts and data injection across a constellation of frontier, classifier, and in-house models. Journeys provide a declarative no-code source layer that compiles deterministically to Agent SDK code. Ghostwriter edits journeys, monitors and Explorer find production issues, simulations test changes across many personas and modalities, and release governance keeps humans in the loop before changes reach production.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  analyze-workspace["Analyze workspace"] --> explorer["Explorer"]
  analyze-workspace --> monitors["Monitors"]
  explorer --> ghostwriter["Ghostwriter"]
  monitors --> ghostwriter
  monitors --> simulations["Simulations"]
  ghostwriter --> journeys["Journeys"]
  journeys --> agent-sdk["Agent SDK"]
  journeys --> release-governance["Release governance"]
  agent-sdk --> agent-os["Agent OS"]
  agent-sdk --> tools-and-packages["Tools and packages"]
  agent-os --> model-constellation["Model constellation"]
  agent-os --> agent-data-platform["Agent Data Platform"]
  agent-os --> voice-pipeline["Voice pipeline"]
  agent-os --> payment-infrastructure["Payment infrastructure"]
  agent-os --> external-agents-and-apis["External agents and APIs"]
  voice-pipeline --> transcription-ensemble["Transcription ensemble"]
  voice-pipeline --> model-constellation
  agent-data-platform --> memory["Memory"]
  simulations --> release-governance
  release-governance --> customer-facing-agent["Customer-facing agent"]
  external-agents-and-apis --> customer-facing-agent
  customer-facing-agent --> agent-os
  customer-facing-agent --> analyze-workspace
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Analyze workspace Product area containing Explorer, reports, and monitors for production conversation analysis. Explorer, Monitors, Ghostwriter
Explorer Deep-analysis agent for customer context, conversations, Sierra data, daily questions, and insight discovery. Ghostwriter
Monitors Always-on evaluators that scan conversations, flag review items, create issues, and reduce the volume humans must inspect manually. Ghostwriter, Simulations
Ghostwriter Authoring agent that writes and edits Journeys, suggests fixes from insights, and works under human review. Journeys, Release governance
Journeys Declarative no-code source layer for customer-experience procedures, policies, variables, and agent behavior. Agent SDK, Release governance
Agent SDK Code-based orchestration and context-management layer that Journeys compile into and that customers can extend directly for complex tool or CI/CD needs. Agent OS, Tools and packages
Tools and packages Customer and platform extensions that expose tools, packages, streaming APIs, and other code-backed capabilities to agents. Agent SDK
Agent OS Runtime layer that translates work into prompts and data injection across the model constellation. Model constellation, Agent Data Platform, Voice pipeline, Payment infrastructure, External agents and APIs
Model constellation Mixture of frontier models, in-house models, classifiers, fine-tuned/open models, and speculative low-latency model calls. Voice pipeline, Simulations
Voice pipeline Low-latency voice stack that parallelizes thinking/listening/talking, ensembles transcription providers, and selectively uses voice-to-voice models. Transcription ensemble, Model constellation
Transcription ensemble Parallel transcription selection logic that combines providers, including special handling for silence and accents. Voice pipeline
Payment infrastructure PCI-certified isolated infrastructure for voice payments where payment information does not go to external LLM providers. Agent OS
Agent Data Platform Customer-data and memory layer combining structured customer data, CRM/internal systems, zero-copy integrations, current conversation context, and saved memories. Memory, Agent OS
Memory First-class memory primitive for identifying customers, saving memories implicitly or explicitly, and extracting them in later conversations with authentication policies. Agent Data Platform
Simulations Customer-facing eval product that tests changes across voice, chat, languages, personas, adversarial users, and complex conversation paths. Release governance, Monitors
Release governance Collaboration, review, workspace, change-management, Git/CI/CD, and enterprise release controls. Customer-facing agent
External agents and APIs Integrations to in-house platforms, other Sierra agents, MCP clients/servers, agent-to-agent protocols, and direct API calls. Agent OS, Customer-facing agent
Customer-facing agent Brand agent exposed over voice, chat, ChatGPT apps, websites, or other commerce/support touchpoints. Agent OS, Analyze workspace

Data flow

Enterprise teams supply transcripts, SOPs, knowledge, tools, customer data, and business policies. Operations teams build no-code Journeys directly or with Ghostwriter; those journeys compile down to Agent SDK code and run through Agent OS. At runtime, a customer conversation enters the Sierra agent, which uses Agent OS to route work across frontier models, in-house models, classifiers, tools, memory, the Agent Data Platform, retrieval/knowledge systems, voice pipelines, and optional agent-to-agent/API integrations. Voice paths parallelize thinking, listening, and talking and can ensemble transcription providers. Payment flows route through isolated PCI-certified infrastructure. Production conversations feed monitors, Explorer, simulations, and Ghostwriter-suggested fixes; humans review and release changes through governance workflows.

6 supporting quotes
“platform? >> So, we have a It's basically a web app. There's three main sections. There's analyze, build, and then there's release. Within the analyze section, you have things like our explorer agent, which is kind of the long-running ChatGPT deep research for all of your customer conversations”transcript, stated
“Agent OS, which has our constellation of models. So, translating the tasks that need to be done on the platform into prompts, uh into data injection”transcript, stated
“you have the Agent SDK, which is the code-based layer of agent orchestration and context management. That's kind of where Sierra started, but over the last 18 months, most of the agent development, um pretty much all of the agent development has shifted to”transcript, stated
“no code layer that we call journeys. It compiles down to Agent SDK code deterministically and uh isomorphically”transcript, stated
“Ghostwriter can automatically suggest a fix to an issue. And you can review that issue and push it to your agent.”transcript, stated
“isolated infrastructure where the payment info doesn't go to a large language model, doesn't go to an external large language model”transcript, stated

Why this architecture

Sierra uses a layered platform because customer-experience agents face different constraints than coding agents or simple chatbots: voice latency, multimodal UX, enterprise release governance, high-value outcomes, customer memory, payments, and per-customer process knowledge. The architecture keeps fast customer conversations in Agent OS while moving deeper analysis and authoring into Explorer/Ghostwriter, using simulations and human review before release.

4 supporting quotes
“So, the one constraint we have that Claude code doesn't have is latency. Majority of Sierra conversations are voice.”transcript, stated
“we have customers that have very specific requirements around what clouds they can run in, what models they can use”transcript, stated
“Explorer can ask the same questions proactively. And then partner with Ghostwriter. We currently think of these as kind of two separate agents, the analysis agent and the authoring agent, to say, "Oh, here are some fixes that are suggested to improve." And”transcript, stated
“Most of our customers, they want to review every change that goes into the agent. This is a really important part of their business.”transcript, stated

Models used: Frontier reasoning models · In-house task-specific models · Classifier models · Fine-tuned open models for knowledge · Voice activity detection and transcription models · Voice-to-voice models · Ghostwriter authoring agent · Explorer analysis agent

Scale

Alternatives considered (roads not taken)

A separate product per customer or use case

Rejected because: Sierra uses one extensible platform where agents can be arbitrarily customized for different moments in the customer lifecycle.

“between these different use cases? >> I would say that it's very extensible, and so you can kind of take it in different directions. We like to say, I think it's originally attributed to the one of the creators of the programming language Pearl, but we try to make the easy things easy and the”transcript, stated
“it's not that, oh, there's like a separate product for, you know, one company versus another, but the agents that you can build on it, and we'd like to think of these agents as products onto themselves, can be arbitrarily customized. >> What does it look”transcript, stated

Pure text or prompt-engineering-only no-code authoring

Rejected because: The speaker says raw text creates either nondeterministic compilation or a prompt-engineering task, so Journeys use a structured DSL that compiles deterministically.

“If you're just doing text, you have to choose between this is non-deterministically compiled, which all of the experiments we've done in that direction, you end up I think with more harm than good. Or this is a prompt engineering task, which then puts you in”transcript, stated
“Or this is a prompt engineering task, which then puts you in the realm of engineering teams. And we, you know, are very proud to be more in the realm of operations teams where a lot of that domain specific knowledge resides. The other big piece of it is that”transcript, stated

One voice/transcription/synthesis provider

Rejected because: Sierra chooses modular provider multi-homing and ensembling because no provider is best across all languages, silence cases, accents, synthesis, and speech-to-speech use cases.

“the product design. The other one I would say is just modularity, like I said earlier. Um no one is the best at everything in this space. And when there are, you know, 100-plus languages uh worldwide that, you know, really deliver meaningful results, when many”transcript, stated
“multi-home providers uh across transcription, across synthesis, and across native uh voice-to-voice models. And so on the transcription side, for example, it just turns out uh when you have a thick UK accent from northern UK or at least parts of northern UK.”transcript, stated

Voice-to-voice models for the majority of traffic immediately

Rejected because: The speaker says voice-to-voice is still expensive, less reliable at reasoning/tooling, and currently useful for simpler journeys where naturalism matters more.

“they're still almost an order of magnitude more expensive. They aren't quite as good at reasoning yet. And so the cases where they are live in production, they're not quite as reliable with tool calling and instruction following. The cases where they're live”transcript, stated
“following. The cases where they're live in production, it's cases where we know in advance that the journey is a little bit simpler. And where the naturalism matters even more than usual. And the procedure is not as complex as some other cases. And so it's still I would say a fraction of our”transcript, stated

Automated continual learning that pushes changes without review

Rejected because: Sierra keeps people in the loop today because customers want control over a business-critical agent, with possible future FYI-only flows gated by confidence.

“you can review that issue and push it to your agent. And so, you're still in the loop or people are still in the loop in all of the cases.”transcript, stated
“we also don't want to get ahead of our skis there. Most of our customers, they want to review every change that goes into the agent. This is a really important part of their business. We don't want to pull the future forward too quickly. Um, we want to move at the pace our customers are excited”transcript, stated

Multi-agent decomposition by default

Rejected because: The speaker argues teams often split agents for org-chart or comfort reasons, which can deprive each agent of needed context; Sierra tends toward one brand agent unless jobs are truly separable.

“useful as people think. My thoughts on this would be people should be really thoughtful about why they want a multi-agent system. If you want a multi-agent system so that one team can work on one agent and one team can work on another agent, then you're shipping your org chart. If you want”transcript, stated
“does a task, by building it as a multi-agent system, you're often depriving of the agent doing the task of the information from the triage and depriving the agent doing the triage of all of the procedural information from the task. And that's typically destructive of value. And so, we are often just”transcript, stated

Patterns

model-routing (architecture)

Agent OS routes different tasks in a turn across frontier models, in-house models, classifiers, speculative execution, and provider choices based on reasoning, latency, cost, and customer constraints.

3 supporting quotes
“10 or 15 different models that might be invoked for a given conversation turn.”transcript, stated
“Some of those might be frontier models that need to do, you know, top-tier reasoning.”transcript, stated
“a handful of classifiers that probably don't require that, a handful of speculative execution in the case of voice in particular”transcript, stated

latency-optimization (operations)

Sierra optimizes voice interactions through parallel thinking/listening/talking, speculative retrieval/response work, progress indicators, transcription ensembling, and model/provider selection.

3 supporting quotes
“we are highly optimized for these low latency use cases. There's a ton of parallelism.”transcript, stated
“>> One of the big unlocks for Sierra agents was how to parallelize thinking, listening, and talking. And if this model says it's silent, you trust it.”transcript, stated
“We will often look up answers before we know if we want them. So you'll, you know, before you decide whether this question needs an answer, you'll at least have the answer ready or in parallel with deciding.”transcript, stated

human-in-the-loop-gate (governance)

Monitors and Ghostwriter can detect issues and suggest fixes, but customers review and release changes before they alter production agents.

“Ghostwriter can automatically suggest a fix to an issue. And you can review that issue and push it to your agent.”transcript, stated
“let Ghostwriter run free but still review it before you make any changes.”transcript, stated

eval-driven-development (quality)

Sierra uses simulations, benchmarks, provider evals, monitors, and model-switching evals to guide harness changes and prevent regressions across customer-specific agent behavior.

3 supporting quotes
“have really good evals and a really good harness, a really good architecture, then you should be able to kind of hill climb toward eval performance”transcript, stated
“we built a product specifically for our customers to eval agents called simulations.”transcript, stated
“Internally, simulations is the main way that we eval the actual agents that are going out to production.”transcript, stated

agent-memory (personalization)

Sierra treats memory as a platform primitive for identifying customers, saving memories during conversations, and retrieving them in later interactions, with sensitivity and authentication policies.

3 supporting quotes
“>> Memory is very important to the platform. So, I mentioned the agent data platform earlier, which kind of brings together uh, machine learning data or, you know, big data as you might say about about customers and then marries that with in the moment context.”transcript, stated
“identifying the customer, saving memories either implicitly automatically or explicitly, and then extracting those memories at a future date”transcript, stated
“every business has to think about what the policy is for allowing the extraction of memories”transcript, stated

privacy-preserving-processing (safety)

Payment information is routed through isolated PCI infrastructure instead of external LLM providers, and memory retrieval is constrained by authentication and sensitivity policies.

“payment info doesn't go to a large language model, doesn't go to an external large language model”transcript, stated
“The one thing to be aware of is you have to be really thoughtful about authentication, because often times, if someone calls over the phone, you don't necessarily know 100% from their phone number that it is this person.”transcript, stated

tool-use-orchestration (architecture)

Sierra agents call tools, APIs, other agents, MCP/A2A integrations, and customer platforms while keeping the core conversation loop optimized for customer-experience latency.

“It has access to tools. Um so, there's a lot of similarities. There are also, you know, you can invoke other agents from the Sierra agent.”transcript, stated
“That being said, Sierra agents support the MCP and agent-to-agent protocols. You can kind of install that integration and then your agent can be an MCP client.”transcript, stated

Tradeoffs

Prioritize quality over prompt-cache loyalty and raw cost optimization.

Chose Break cache or spend more when the right context improves outcomes over Strictly preserving prompt cache for speed/cost.

Rationale: Sierra says quality comes first because customer conversations can drive valuable outcomes, making cost less central than correctness and experience.

“we definitely don't want to invalidate a cache for no good reason, but quality comes first.”transcript, stated
“a conversation with a customer could sell a $100 product or a $1,000 lifetime value plan.”transcript, stated

Use a no-code Journey DSL rather than pure text or pure code for most agent authoring.

Chose Declarative Journeys compiled to Agent SDK code over Raw text prompts or engineering-owned code-only workflows.

Rationale: Journeys map to operations teams' SOP knowledge while preserving deterministic compilation and inspectability.

“The person who knows everything about the standard operating procedures should be able to just do that in the product.”transcript, stated
“no code layer that we call journeys. It compiles down to Agent SDK code deterministically”transcript, stated

Keep voice modular instead of going all-in on voice-to-voice models.

Chose A modular speech-to-text/text-to-speech/voice-to-voice architecture with provider ensembling over A single integrated voice-to-voice loop for most traffic.

Rationale: Voice-to-voice models improve naturalism but remain expensive, less reliable for tooling/reasoning, and insufficient across languages and complex procedures.

“they're still almost an order of magnitude more expensive. They aren't quite as good at reasoning yet.”transcript, stated
“if you still want to support all the different languages, you need that modularity for the foreseeable future.”transcript, stated

Meet enterprise model/cloud requirements through model-agnostic architecture.

Chose Comparable-intelligence model portability with eval-guided prompts and eligible model sets over Standardizing all customers on one provider or cloud.

Rationale: Fortune-scale customers have cloud and model constraints, so Sierra built products around switching between models while maintaining eval performance.

“what clouds they can run in, what models they can use, and our company approach is to meet them, you know, on their terms.”transcript, stated
“we can kind of move between models of comparable intelligence without too much heartburn.”transcript, stated

Use outcome-based pricing where agent actions create measurable value.

Chose Pricing aligned to sales, resolution, conversion, loyalty, and other outcomes over Pure seat-based or usage-based pricing for differentiated high-value work.

Rationale: Sierra argues outcome pricing aligns incentives when agents deliver valuable outcomes, while commodity question-answering may fit simpler pricing.

“It aligns the incentives between our company and our customers. And I think the phrase I like to use, which is a little bit cheeky, probably, is if you don't understand the value of outcome-based pricing, your outcomes are probably not that valuable.”transcript, stated
“If your product really like feels a little bit more like a commodity, you'll start to see more usage-based and seat-based pricing”transcript, stated

Keep humans in the review loop for continual improvement today.

Chose Monitor detection, Ghostwriter suggestions, human approval, then release over Fully autonomous self-improving production agents.

Rationale: The platform has primitives for more autonomy, but customers still want authority over changes to business-critical agents.

“all of the primitives are there, it's just around the confidence that people have in the level of control that they want to have.”transcript, stated
“We don't want to pull the future forward too quickly. Um, we want to move at the pace our customers are excited about.”transcript, stated

Gotchas (what bit them)

Voice agents fail user expectations if they inherit a coding-agent latency model.

Root cause: Customer voice interactions require 1-2 second responsiveness, while long-running agent loops can tolerate slower background work.

Resolution: Separate quick-turn voice/customer harnesses from longer Explorer/Ghostwriter-style analysis agents, and use parallelism, speculative execution, and progress indicators in the voice path.

3 supporting quotes
“if you're not responding in 1 or 2 seconds, then people wonder where you went.”transcript, stated
“this is less of a quick-turn conversational agent and more of a longer-term deep analysis agent.”transcript, stated
“You need to be really thoughtful about parallelism. You need to be really thoughtful about what we call progress indicators”transcript, stated

No-code agent authoring can become nondeterministic if it is only free-form text.

Root cause: Raw text either creates nondeterministic compilation or pushes authoring back into prompt-engineering work for engineering teams.

Resolution: Use Journeys as a declarative DSL that compiles deterministically and isomorphically to Agent SDK code while remaining inspectable.

“If you're just doing text, you have to choose between this is non-deterministically compiled”transcript, stated
“It compiles down to Agent SDK code deterministically and uh isomorphically”transcript, stated

Payment agents cannot treat sensitive card data as ordinary model context.

Root cause: External LLM providers were not PCI certified in the needed way, so sending payment information through them would violate the payment architecture.

Resolution: Build isolated PCI-certified payment infrastructure where payment information does not go to external LLMs.

“We were investing in payments before it made sense, I think, and it's a long process, but a few months ago, we announced, you know, we're uh fully PCI DSS level one certified. >> no clue what that means.”transcript, stated
“payment info doesn't go to a large language model, doesn't go to an external large language model”transcript, stated

Memory can leak sensitive or misattributed information if identity is assumed from weak signals.

Root cause: Phone numbers may be shared by offices or families, and memory sensitivity varies from greetings to social-security-number context.

Resolution: Make authentication and memory sensitivity policy explicit for each business before extracting memories into the agent context.

“often times, if someone calls over the phone, you don't necessarily know 100% from their phone number that it is this person.”transcript, stated
“every business has to think about what the policy is for allowing the extraction of memories, and which memories are sensitive versus not so sensitive.”transcript, stated

Customer-facing evals are harder than internal task evals.

Root cause: Real conversations involve background noise, adversarial users, many personas, voice/chat, many languages, and high-dimensional production behavior.

Resolution: Use customer-specific simulations plus monitors and review gates, not just generic provider benchmarks.

3 supporting quotes
“the eval problem is much more complicated and involves things like what happens when there's background noise in voice”transcript, stated
“what if I have an adversarial user and I want to save these 20 personas and run all of my simulations against all 20 of the personas and make sure that it works. And so, you end up with just a more complicated topography because a conversation by nature is very complicated and can go in so”transcript, stated
“simulations is the main way that we eval the actual agents that are going out to production.”transcript, stated

Splitting into many agents can damage quality by separating context from action.

Root cause: A triage agent may lose task procedure details while a task agent loses triage information, so decomposition can ship the org chart rather than improve the user experience.

Resolution: Default to one brand agent with strong context engineering unless jobs are truly separable.

“If you want a multi-agent system so that one team can work on one agent and one team can work on another agent, then you're shipping your org chart.”transcript, stated
“Sierra agents tend to be kind of one agent representing the brand.”transcript, stated

Prompt/context inconsistency can look like model stupidity while actually being system design debt.

Root cause: Conflicting prompt context and lossy compaction can make the model behave incoherently.

Resolution: Use progressive disclosure, avoid incoherent prompt history, and treat hallucinations as a context-engineering signal first.

“when we've seen hallucinations, it's often because one part of the prompt was this, and the other part was this.”transcript, stated
“prompt compaction, you just want to be really thoughtful about not making it lossy”transcript, stated

Evaluation

Sierra evaluates at several layers: internal Agent OS tasks use applied-AI evals, customer agents use simulations across voice/chat/languages/personas, monitors scan production conversations, benchmarks such as TaoBench/MuBench evaluate providers or specific capabilities, and model/provider changes are hill-climbed against eval performance.

Metrics: Customer satisfaction · Resolution rate · Sales conversion · Latency for 1-2 second voice responsiveness · Performance across 20 personas in simulations · Monitor accuracy and review volume reduction · Provider benchmark performance such as MuBench for transcription · Latency-quality-cost tradeoff by model/provider · Regression avoidance across voice, chat, languages, and personas · Outcome value for pricing and renewal alignment

5 supporting quotes
“how can I optimize the metric that I care about, whether that's customer satisfaction or resolution rate or in the case of the customer I mentioned, like sales converted.”transcript, stated
“we built a product specifically for our customers to eval agents called simulations.”transcript, stated
“every change you make is tested against all the assumptions of the platform across voice and chat and many languages and many personas”transcript, stated
“you can set up monitors that run on every conversation and look out for the things that you want to flag either for review or to create issues from”transcript, stated
“Most of the benchmarks we use to evaluate providers more than to evaluate agents.”transcript, stated

Outcomes

The transcript reports Fortune-scale adoption, 10-15 models per conversation turn, nearly 60 voice languages, 100-plus developers in some customer accounts, PCI DSS level one payments, 20-persona simulations, monitor review narrowing from 10,000 conversations to five, and load tests whose annualized rate would be billions of conversations.

5 supporting quotes
“we're working with most of the Fortune 20, something like 40 or 50% of the Fortune 50 or Fortune 100.”transcript, stated
“most of the agent development, um pretty much all of the agent development has shifted to our no code layer that we call journeys.”transcript, stated
“And as of a few months ago, we now have production agents live with the voice-to-voice models. Um and so, they're you know, it's you know, fully end-to-end doing that.”transcript, stated
“That being said, Sierra agents support the MCP and agent-to-agent protocols. You can kind of install that integration and then your agent can be an MCP client.”transcript, stated
“you don't need to transfer to another platform. So, it's a co- cohesive experience throughout checkout.”transcript, stated