Building and Scaling Conversational Voice AI Agents for Enterprise Go-to-Market

thoughtly_/_gladia · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise voice AI go-to-market automation

Thoughtly needed to run production voice agents for inbound support and outbound lead activation at enterprise scale, while coordinating real-time speech recognition, LLM reasoning, text-to-speech, telephony, CRM integrations, compliance, and latency-sensitive conversational behavior.

Constraints:

• voice conversations require sub-second or near-real-time turn-taking to avoid awkward pauses and interruptions
• each call turn incurs cost across STT, LLM, TTS, VoIP, carrier fees, infrastructure, and operations
• customers need workflow-specific routing and external system integrations during the call
• enterprise customers require HIPAA and SOC 2 readiness plus strong multi-tenancy and data privacy
• vendor model quality and latency change quickly, making long custom ML investments risky
• traffic can spike unpredictably for inbound support while outbound calling is easier to forecast

Architecture

Thoughtly built a real-time voice-agent orchestration platform. Audio flows through phone carriers and VoIP providers into Thoughtly, where STT converts speech to text, an LLM finds the caller's step in a customer-defined workflow and generates the next response, optional knowledge-base or CRM/API calls supply fresh context, TTS converts text back into audio, and media returns through VoIP and carrier layers. The team optimizes latency at every layer using provider selection, WebSockets, caching, vector-similarity shortcuts, selective LLM bypass, and customer-adjustable timing controls. Reliability comes from golden-data evals, regression tests, internal benchmarks, defensive scaling, close sales/product/customer feedback, and enterprise compliance controls.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  caller[Caller] --> carrier-network[Carrier network]
  carrier-network --> voip-provider[VoIP provider]
  voip-provider --> thoughtly-orchestration-platform[Thoughtly orchestration platform]
  thoughtly-orchestration-platform --> speech-to-text-provider[Speech-to-text provider]
  speech-to-text-provider --> language-model[Language model]
  thoughtly-orchestration-platform --> workflow-state-engine[Workflow state engine]
  workflow-state-engine --> language-model
  language-model --> external-integrations[External integrations]
  language-model --> text-to-speech-provider[Text-to-speech provider]
  thoughtly-orchestration-platform --> cache-and-vector-scorer[Cache and vector scorer]
  cache-and-vector-scorer --> language-model
  cache-and-vector-scorer --> text-to-speech-provider
  text-to-speech-provider --> voip-provider
  thoughtly-orchestration-platform --> evaluation-and-benchmarking[Evaluation and benchmarking]
  evaluation-and-benchmarking --> product-and-sales-feedback[Product and sales feedback]
  product-and-sales-feedback --> evaluation-and-benchmarking

Components

Data flow

A caller speaks into a phone call. The carrier sends audio through a VoIP provider to Thoughtly. Thoughtly transcribes audio with STT, sends text and workflow context to an LLM for step finding and response generation, optionally retrieves knowledge-base or CRM data, converts the response to audio through TTS, and sends the media back through the VoIP provider and carrier. For deterministic turns, the system may skip the LLM and use vector similarity against cached responses before going directly to TTS. Golden datasets, regression tests, customer feedback, latency metrics, and traffic forecasts feed engineering decisions about models, vendors, caching, scaling, and integrations.

Why this architecture

Thoughtly uses an orchestration architecture because voice agents are real-time multi-provider systems, not single LLM calls. The LLM handles workflow step finding and response generation, while STT/TTS providers, telephony, caches, vector scoring, and APIs solve specialized latency, media, and integration problems. The team avoids self-hosting where commercial providers have already reached better latency/cost tradeoffs, uses deterministic shortcuts only for predictable turns, and invests in evals because rapid model/vendor changes would otherwise make every new provider or configuration risky. Vertical focus lets product and ML improvements compound across customers with similar workflows.

Models used: GPT-4 · GPT-4.1 · GPT-4.1 nano · Gladia speech-to-text · ElevenLabs text-to-speech · Cartesia text-to-speech · language model as judge

Scale

• Thoughtly started in late 2023 and moved from SMBs toward large enterprise and mid-market customers.
  “the company was started in 2023 towards the end of 2023. We were one of the first people uh the first companies in Voice AI”transcript, stated

  “in the beginning we were serving more like SMBs more smaller companies and eventually we kind of moved up the market”transcript, stated
• The platform has handled millions of calls.
  “so far uh today we've done uh millions of calls and we've seen many of our customers be able to win”transcript, stated
• Customers reported 3x conversion and 15x ROI in the webinar claims.
  “we are seeing uh three three times the conversion on traditional methods for contacting leads. Uh and ultimately a 15 time return on investment”transcript, stated
• Thoughtly positions voice outreach as minutes or seconds after intent instead of days.
  “talk to people, talk to your the the callers in minutes and in seconds instead of like the way before where you have to rely on a human. So that used to be days.”transcript, stated
• Gladia's STT support for around 100 languages was a factor for global customers.
  “you guys provide support for the most number of languages. I'm pretty sure you support a hundred different languages”transcript, stated
• Commercial STT latency improved from roughly 400-700 ms to about 50-150 ms.
  “this time last year it's fairly typical to get your to have your speech to text uh take something like 400 to 700 milliseconds”transcript, stated

  “whereas now the the benchmark is is some is a fraction of that right so that's something like 50 milliseconds to 150 milliseconds”transcript, stated
• GPU cold starts were described as taking 10 to 20 minutes.
  “GPUs take typically between 10 minutes to 20 minutes to be popped.”transcript, stated
• Self-hosting for P90/P95/P99 latency can require overprovisioning and add roughly 20-40% infrastructure cost.
  “if you want to have like P90, P95, P99 is to overprovision”transcript, stated

  “you have extra cost in infrastructure which going to cost you like probably 20 to 40% more”transcript, stated

Alternatives considered (roads not taken)

Invoke the LLM for every turn

Rejected because: For deterministic conversational paths, vector similarity and cached responses can skip the LLM and reduce latency without losing required behavior.

Speculatively send partial responses to TTS

Rejected because: It can mask LLM latency, but unexpected caller input can produce an unrelated and jarring response.

Self-host speech models on-premises

Rejected because: Commercial provider latency improved enough that self-hosting no longer justified server hops, cold starts, GPU cost, and overprovisioning.

MCP-style autonomous integration calls during voice conversations

Rejected because: Letting the agent decide when to call MCP servers adds inconsistent latency that can disrupt real-time conversation.

Serve every possible voice-AI vertical horizontally

Rejected because: The team argues vertical focus creates defensibility, repeatable integrations, and ML improvements that compound across similar customers.

Patterns

tool-use-orchestration (agent-runtime)

Thoughtly orchestrates phone carriers, VoIP, STT, LLM step finding, knowledge/API lookups, TTS, CRM/booking integrations, and playback for every conversation turn.

latency-optimization (performance)

The platform optimizes latency across STT, LLM, TTS, VoIP, WebSockets, caching, and deterministic shortcuts because conversational pauses create a bad user loop.

semantic-caching (performance)

For deterministic turns, Thoughtly uses vector similarity against a small response set or cached content to avoid unnecessary LLM calls before TTS.

llm-as-judge (quality)

Thoughtly combines vector scoring with a language-model judge for real-time performance and routing reliability in more deterministic conversations.

eval-driven-development (quality)

The team built internal golden datasets, conversation replay, cosine-similarity regression checks, and benchmarks to compare models and vendors quickly.

model-routing (model-ops)

Thoughtly evaluates provider and model choices such as GPT-4.1 nano, Gladia, ElevenLabs, Cartesia, and on-prem STT against latency, accuracy, quality, and future market trajectory.

safety-policy-enforcement (enterprise)

Enterprise readiness requires HIPAA/SOC 2 controls, multi-tenancy, data privacy, and a lean vendor stack to keep compliance manageable.

Tradeoffs

Real-time turn handling

Chose selective LLM bypass with vector/cached responses over LLM inference for every response.

Rationale: Deterministic steps can be served near-instantly, but only when the state space is small enough to avoid wrong responses.

Speech model hosting

Chose commercial API providers over on-prem self-hosted STT.

Rationale: Provider latency improved quickly enough that self-hosting added cold starts, server hops, and GPU cost without enough benefit.

Integration strategy

Chose direct API integrations over MCP-driven autonomous calls during the conversation.

Rationale: Direct APIs offer more predictable latency for live calls than agent-decided MCP calls.

Market strategy

Chose vertical specialization over horizontal voice AI for every industry.

Rationale: Vertical focus creates reusable workflows, jargon, integrations, and data advantages that compound within a category.

Evaluation investment

Chose internal golden datasets and benchmarks over ad hoc vendor/model switching.

Rationale: Fast-moving providers make every switch tempting, so evals protect against wasting time and quality regressions.

Gotchas (what bit them)

Voice-agent latency failures compound into conversational failure.

Root cause: If the agent waits four or five seconds, callers interrupt, forcing cancellation and restart of partially generated responses.

Resolution: Optimize latency at every layer: STT, LLM, TTS, VoIP, caching, WebSockets, and deterministic response selection.

Speculative response generation can become jarring.

Root cause: A caller may say something unexpected after the system has already started optimizing for a predicted response.

Resolution: Use speculative or LLM-bypass paths only when conversation state is deterministic enough to justify the risk.

Self-hosting speech models can lose to commercial APIs once total latency and operating cost are included.

Root cause: GPU cold starts, network hops, overprovisioning, and infrastructure complexity offset the theoretical benefit of local hosting.

Resolution: Benchmark against current commercial providers and revisit decisions as provider latencies improve.

MCP-style autonomous tool calls can be too variable for live voice conversations.

Root cause: Letting an agent decide when to call an MCP server introduces extra and sometimes inconsistent latency.

Resolution: Use direct API integrations for CRM and booking workflows until tool-call latency is predictable enough.

Compliance gets harder when the provider stack sprawls.

Root cause: Every additional vendor creates more documentation, certification, and risk-assessment burden.

Resolution: Keep infrastructure lean and choose vendors carefully before pursuing enterprise compliance certifications.

Evaluation

Thoughtly evaluates voice-agent changes with internal golden datasets, conversation replay, cosine-similarity regression checks, provider/model benchmarks, latency-by-layer measurements, accuracy/conversation quality checks, business outcomes such as conversion and ROI, and customer/sales feedback used for volume forecasts and feature prioritization.

Metrics: end-of-turn latency · time to first byte · STT latency · LLM latency · TTS latency · conversation navigation accuracy · cosine similarity regression checks · conversion rate · return on investment · call volume · peak traffic forecasts · model/vendor benchmark results

Outcomes

• Thoughtly moved from SMBs toward large enterprises and mid-market customers
• the platform has handled millions of calls
• the webinar reports 3x conversion over traditional contact methods and 15x ROI for customers
• voice agents can contact warm leads within minutes or seconds instead of days
• Thoughtly supports global deployment with multi-country infrastructure and Gladia's broad language coverage
• commercial STT latency improvements reduced the need for on-prem hosting
• internal evals became an important sanity check after the team realized they saved time
• the team prioritized conditional navigation as a customer-driven product feature

Thoughtly operates in production enterprise voice AI with millions of calls, global language needs, HIPAA/SOC 2 requirements, and real-time latency expectations that make per-layer optimization and vendor choice central to unit economics.