Building a Production Voice AI Agent for Customer Support in 100 Days

intercom · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Voice AI customer-support agent

Intercom needed to extend its text-based Fin AI agent into phone support, where customers expect natural real-time conversation, phone support remains widely used and expensive, voice requires different conversation design than chat, and enterprise customers need smooth support-team workflow integration.

Constraints:

Voice support must answer, escalate, and hand off context on live phone calls
Voice latency is less forgiving than chat latency
Long text-style responses do not work when users must listen sequentially
Users initially interact with voice agents like IVR systems
Support teams care about escalation paths and transcript handoff as much as model behavior
Evaluation must include manual and automated call-quality review

Architecture

Intercom built Fin Voice in about 100 days by reusing the existing Fin AI support system and adding a voice channel. The architecture uses speech-to-text, a language model, text-to-speech, OpenAI Realtime API as part of the core stack, RAG over knowledge-base content, existing telephony infrastructure, voice-specific conversation design, escalation paths, transcript-summary handoff, test conversations, internal Streamlit review apps, resolution-rate tracking, and experimental LLM-as-judge transcript analysis.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  caller[Caller] --> fin-voice[Fin Voice]
  fin-voice --> telephony-layer[Telephony Layer]
  fin-voice --> speech-to-text[Speech To Text]
  speech-to-text --> language-model[Language Model]
  language-model --> knowledge-base-rag[Knowledge Base RAG]
  existing-fin-system[Existing Fin System] --> knowledge-base-rag
  realtime-api[Realtime API] --> language-model
  language-model --> text-to-speech[Text To Speech]
  text-to-speech --> fin-voice
  conversation-design-layer[Conversation Design Layer] --> fin-voice
  fin-voice --> escalation-workflow[Escalation Workflow]
  escalation-workflow --> human-agent-handoff[Human Agent Handoff]
  human-agent-handoff --> internal-review-tools[Internal Review Tools]
  evaluation-suite[Evaluation Suite] --> fin-voice
  evaluation-suite --> internal-review-tools
  evaluation-suite --> resolution-metric[Resolution Metric]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Caller`	Customer using phone support for urgent or sensitive issues.	`Fin Voice`
`Fin Voice`	Voice AI agent that answers phone calls, resolves issues, and escalates when needed.	`Caller`, `Telephony Layer`, `Speech To Text`, `Language Model`, `Text To Speech`, `Escalation Workflow`
`Existing Fin System`	Text/chat AI agent platform, conversation analysis, behavior training, testing, and deployment infrastructure reused for voice.	`Fin Voice`, `Knowledge Base RAG`
`Telephony Layer`	Intercom native phone-support infrastructure that puts the agent on customer phone lines.	`Fin Voice`
`Speech To Text`	Converts caller speech into text for the language-model pipeline.	`Fin Voice`, `Language Model`
`Language Model`	Generates responses using RAG context and conversation state.	`Speech To Text`, `Knowledge Base RAG`, `Text To Speech`
`Text To Speech`	Converts generated text back into spoken audio.	`Language Model`, `Fin Voice`
`Realtime API`	Rapid prototyping component that remained part of the evolved core architecture.	`Language Model`
`Knowledge Base RAG`	Retrieval layer for answering questions based on help articles and existing knowledge-base content.	`Existing Fin System`, `Language Model`
`Conversation Design Layer`	Voice-specific latency filler, answer chunking, confirmation prompts, and natural-speech behavior design.	`Fin Voice`
`Escalation Workflow`	Configured paths for handing calls to human support teams.	`Fin Voice`, `Human Agent Handoff`
`Human Agent Handoff`	Transcript summary, recording, call outcome, and context handoff for support agents.	`Escalation Workflow`, `Internal Review Tools`
`Evaluation Suite`	Manual and automated test conversations plus LLM-as-judge transcript analysis.	`Fin Voice`, `Internal Review Tools`
`Internal Review Tools`	Streamlit web apps for logs, transcripts, recordings, troubleshooting, and issue review.	`Evaluation Suite`, `Human Agent Handoff`
`Resolution Metric`	North-star metric for whether calls are resolved without follow-up calls.	`Evaluation Suite`

Data flow

A phone call reaches Fin Voice through Intercom telephony. Speech is transcribed, the language model uses RAG knowledge-base context to answer, text-to-speech returns audio, and complex responses are chunked with confirmation prompts. If needed, Fin escalates to a human agent with transcript summaries and call outcomes. Every major code change runs test conversations, and support/debug tools review logs, transcripts, recordings, resolution outcomes, and LLM-judge findings.

4 supporting quotes

“voice agent ST lm tts uh so speech to text converting uh speech into text lm for uh actually dating the response and text to speech”transcript, stated

“want to mention rag and telefony. So rag obviously is super critical for a lot of agent experiences. Uh but obviously it is important for the uh agent answering questions based on the knowledge base u and then telephony. Um uh so actually being able to put the agent on the phone”transcript, stated

“and then telephony. Um uh so actually being able to put the agent on the phone lines and we had a bit of a head start because our agent on chat already had the rack set up and we already had a native phone support product. So we got some of those things for free.”transcript, stated

“also the context handoff. So after every AI agent call, we generate a transcript summary that gives a bit more context to the human agent that gets the call to what happened on the call. And yeah, these are not like super flashy features, but they were absolutely essential to get this from this demo”transcript, stated

Why this architecture

Intercom chose a controllable STT-LM-TTS pipeline rather than pure voice-to-voice because voice-to-voice may sound more natural but offers less control. Reusing Fin's existing RAG and support-agent system accelerated delivery, and existing telephony gave Intercom a head start. The product layer focused on voice-specific latency, response length, user behavior adaptation, and support-team integration.

4 supporting quotes

“voice to voice voice approach has the benefit of uh potentially faster or natural more sounding speech but also gives you less control over the output.”transcript, stated

“our agent on chat already had the rack set up and we already had a native phone support product. So we got some of those things for free.”transcript, stated

“different um and that voice is not necessarily just chat with sound. So there are like three key differences I want to mention. There's obviously many many more but there's a few that I thought it's worth mentioning. So”transcript, stated

“majority of the feedback wasn't about the voice uh about the model or about the latency was actually about how does it work with the support team workflows”transcript, stated

Models used: speech-to-text · language model · text-to-speech · OpenAI Realtime API · LLM-as-a-judge

Scale

Finn Voice was built in about 100 days.
“to a human agent when needed. And we build this experience in about 100 days. So today in this talk, I'll share what it took to get there. And I'll also talk about why I believe voice is the next big frontier and AI for customer”transcript, stated
Text Fin had over 5,000 customers.
“incredible. Uh we have over 5,000 of of c customers and also in terms of performance is reaching average resolution rate of 56% and for some customer to be 70 80%. Um and this is defined as a percent of interactions”transcript, stated
Text Fin averaged a 56% resolution rate, with some customers at 70-80%.
“performance is reaching average resolution rate of 56% and for some customer to be 70 80%. Um and this is defined as a percent of interactions handled by Finn that are resolved without human intervention.”transcript, stated
Over 80% of support teams still use phone support.
“some of the top level data over 80% of support teams uh still use uh phone support and if you think about all the conversations globally or customer service interactions over onethird of them are uh happening over the phone as well. So it's not a legacy channel”transcript, stated
More than one-third of global customer-service interactions happen over the phone.
“conversations globally or customer service interactions over onethird of them are uh happening over the phone as well. So it's not a legacy channel that's going away it's still widely used and it's also quite costly. Uh, if you think about the average cost of handling”transcript, stated
Human phone support in the US costs about $7-12 per call, while voice AI can be at least five times cheaper.
“a phone call in the US with human support, it's about between seven and 12 dollars. And with voice AI agents, it can be at least five times cheaper.”transcript, stated
The agent supports 30-40 plus languages.
“speech and also multilingual so uh AI agents can support 30 40 plus languages obviously better for the for the users and on the business side major cost savings and also scalability as the business grows or when you need to handle peak times AI agents were much better for that.”transcript, stated
Simple voice queries reached about one-second responses; complex queries could take three to four seconds.
“in terms of our approach for the simple queries uh we got it to about 1 second so we didn't need to do anything extra uh but for more complex queries when they they're running a bit longer three four seconds that we added injected”transcript, stated
“uh but for more complex queries when they they're running a bit longer three four seconds that we added injected filler words I mean like let let me look into this for you let me look it up uh to maintain the conversation flow while we generate the answer in the background”transcript, stated

Alternatives considered (roads not taken)

Use pure voice-to-voice models

Rejected because: They can sound faster and more natural but give less output control than STT-LM-TTS.

“another approach with the voicetovoice models where uh everything is processed directly in audio while skipping the text layer entirely.”transcript, stated

“benefit of uh potentially faster or natural more sounding speech but also gives you less control over the output. Uh so in our approach we did start with realtime API by itself from the get-go and it allowed us to test very very quickly. Uh but eventually we did evolve”transcript, stated

Treat voice as chat plus audio

Rejected because: Voice has different latency, response-length, and user-behavior constraints.

“necessarily just chat with sound. So there are like three key differences I want to mention. There's obviously many many more but there's a few that I thought it's worth mentioning. So latency um on chat it's actually I think okay to”transcript, stated

Let complex answers play as one long response

Rejected because: Voice users cannot skim, so long multi-step responses must be chunked and confirmed.

“wouldn't work for on voice. You don't want to wait there for a minute or two listening to to the agent. So for more complex responses or for responses with multiple steps, we're breaking down the answers into multiple chunks and deliver chunk by chunk and after each we ask the”transcript, stated

“multiple steps, we're breaking down the answers into multiple chunks and deliver chunk by chunk and after each we ask the user to confirm whether they would like to listen to the next step. And this works really well for something like troubleshooting when you have a few”transcript, stated

Price only by usage

Rejected because: Usage-based pricing is predictable but can misalign provider/customer incentives around quality.

“Probably usage is still the most dominant right now. Uh so user based pricing very simple. It's like per minute or per call very predictable but it doesn't necessarily capture the quality of the agent. Um so uh the incentives are not very well aligned between the providers and the customers.”transcript, stated

“it doesn't necessarily capture the quality of the agent. Um so uh the incentives are not very well aligned between the providers and the customers.”transcript, stated

Patterns

`rag-retrieval` (support-knowledge)

Fin Voice answers phone questions using knowledge-base RAG inherited from the existing Fin chat system.

“rag obviously is super critical for a lot of agent experiences.”transcript, stated

“important for the uh agent answering questions based on the knowledge base”transcript, stated

`human-in-the-loop-gate` (support-escalation)

Fin Voice escalates calls to human support agents when needed and includes call context.

“escalates to a human agent when needed.”transcript, stated

“getting configurations for how the calls get escalated to the human support team”transcript, stated

`latency-optimization` (voice-runtime)

Intercom uses about one-second simple responses and filler phrases for three-to-four-second complex queries.

“for the simple queries uh we got it to about 1 second”transcript, stated

“we added injected filler words I mean like let let me look into this for you”transcript, stated

`observability-tracing` (voice-operations)

Internal Streamlit apps let the team inspect logs, transcripts, and recordings for troubleshooting.

“we built some internal the streamed web apps to review the logs the the transcripts the recordings.”transcript, stated

“we can review in detail what happened in the conversation actually troubleshoot it with with the logs.”transcript, stated

`eval-driven-development` (voice-quality)

The team runs manual and automated test conversations on major code changes and tracks resolution rate.

“we had a test a set of uh test conversations that will be running through on every ma major code uh code change.”transcript, stated

“resolution rate. So this is our northstar metric”transcript, stated

`llm-as-judge` (quality-review)

Intercom experimentally uses another model to analyze call transcripts for issues and improvement opportunities.

“LM as a judge. This is more experimental”transcript, stated

“we're using another to analyze the call transcripts to help us identify issues or opportunities for improvement.”transcript, stated

Tradeoffs

Use STT-LM-TTS rather than pure voice-to-voice

Chose speech-to-text, language model, and text-to-speech pipeline over direct audio voice-to-voice models.

Rationale: Voice-to-voice may be faster or more natural but gives less output control.

“voice to voice voice approach has the benefit of uh potentially faster or natural more sounding speech”transcript, stated

“but also gives you less control over the output.”transcript, stated

Reuse existing Fin and phone infrastructure

Chose existing RAG setup and native phone support product over building voice support infrastructure from scratch.

Rationale: The existing chat agent and telephony product gave Intercom major pieces for free and accelerated the 100-day build.

“our agent on chat already had the rack set up and we already had a native phone support product.”transcript, stated

“So we got some of those things for free.”transcript, stated

Chunk complex responses

Chose step-by-step voice answers with confirmation over full long-form answer delivery.

Rationale: Sequential listening makes long answers tiring; chunks work better for troubleshooting.

“we're breaking down the answers into multiple chunks and deliver chunk by chunk”transcript, stated

“this works really well for something like troubleshooting when you have a few steps”transcript, stated

Track resolution rate as north star

Chose resolved-call outcome over only latency or call-volume metrics.

Rationale: Resolution rate measures customer value more directly than whether the voice stack is fast or cheap.

“resolution rate. So this is our northstar metric which actually tells us whether we're delivering value for our customers.”transcript, stated

“this is the main the main metric metric metric we track.”transcript, stated

Gotchas (what bit them)

Voice latency breaks user trust faster than chat latency.

Root cause: A second or two of silence on voice makes users think something has gone wrong.

Resolution: Intercom optimized simple queries to about one second and uses filler phrases for longer complex queries.

“it doesn't really work on voice if the agent goes silence for for a second or two or maybe longer”transcript, stated

“we added injected filler words I mean like let let me look into this for you”transcript, stated

Text-style answer length fails in voice.

Root cause: Users cannot skim spoken answers and do not want to listen for minutes.

Resolution: Complex responses are chunked, with confirmation before the next step.

“it's easy to skim through the answer to find the right information. But again, it wouldn't work for on voice.”transcript, stated

“after each we ask the user to confirm whether they would like to listen to the next step.”transcript, stated

Support workflow integration was the bigger enterprise blocker.

Root cause: Teams cared about escalation and context handoff, not only model quality or latency.

Resolution: Intercom focused on escalation configuration and transcript-summary context handoff.

“majority of the feedback wasn't about the voice uh about the model or about the latency”transcript, stated

“these are not like super flashy features, but they were absolutely essential to get this from this demo stage to the deployment stage”transcript, stated

Users initially treated the AI like an IVR.

Root cause: People are accustomed to phone automation that expects terse commands.

Resolution: Intercom designed more natural agent speech so users learned to use full sentences during the call.

“some customers would interact with invoice like with an old school IBR.”transcript, stated

“actually start using full sentences while they hear the agent using full sentences.”transcript, stated

Evaluation

Intercom evaluates Fin Voice with manual and automated test conversations on major code changes, internal Streamlit review apps for logs/transcripts/recordings, resolution-rate tracking, and experimental LLM-as-judge analysis of call transcripts.

Metrics: resolution rate · call-back within 24 hours · test conversation pass rate · latency · transcript review findings · recording review findings · LLM judge issue/opportunity findings · cost per minute

3 supporting quotes

“test conversations that will be running through on every ma major code uh code change.”transcript, stated

“the user disconnects after hearing at least one answer um and then doesn't call back within 24 hours.”transcript, stated

“The typical cost ranges between three and 20 cents per minute.”transcript, stated

Outcomes

Intercom shipped Fin Voice in about 100 days.
Several enterprise customers used it on their main phone lines.
Fin Voice reused existing Fin RAG and Intercom telephony foundations.
Voice-specific design added filler phrases, response chunking, and user-behavior adaptation.
Support workflow integration added escalation paths and transcript-summary handoff.

Fin Voice was built in about 100 days, deployed to several enterprise customers' main phone lines, extended a Fin system with over 5,000 customers and 56% average text resolution rate, and targeted a phone-support market where over 80% of support teams still use phone support.