Building Low-Latency Voice AI Agents with Workflow Orchestration Trade-offs

gradient_labs · Finance · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Regulated financial-services voice support agents

Gradient Labs had a durable Temporal-based text support agent for regulated financial-services customers, but moving the same agent into live voice exposed a latency mismatch: customers would hang up after a few seconds, while child workflows under the same parent workflow could take seconds just to start.

Constraints:

• Financial-services support requires a high quality bar because banks, lending, fintech, and insurance are regulated industries.
• Voice callers will hang up if they wait two to five seconds for an agent reply.
• Speech-to-speech real-time models were too prompt-injectable and hallucination-prone for regulated use cases.
• The existing text agent's skills, activities, test environments, and debugging model were built around Temporal.
• Temporal's strong child-workflow start guarantees introduced parent-lock startup serialization for many concurrent child workflows.
• The team needed to preserve shared code and Temporal UI debugging while trading some durability guarantees for speed.

Architecture

Gradient Labs reused its Temporal-based text support-agent stack for voice, with LiveKit-style telephony, Deepgram speech-to-text, a text LLM agent, and Cartesia/ElevenLabs text-to-speech. The long-running conversation workflow still coordinates multi-day text conversations and high-level voice lifecycle work. Each voice turn invokes a short-lived turn workflow that composes classification, answer generation, guardrails, translation, LLM calls, API calls, and tools. To avoid child workflows starting sequentially under the parent workflow lock, the team built a fast exec library: the main turn workflow starts detached wrapper workflows from side effects/goroutines, those wrappers signal results back through high-cardinality signal names, and a custom future interface preserves code ergonomics while reducing scheduled-to-start latency.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer[Customer] --> channel-integrations[Channel integrations]
  channel-integrations --> conversation-workflow[Conversation workflow]
  channel-integrations --> speech-to-text-layer[Speech-to-text layer]
  conversation-workflow --> turn-workflow[Turn workflow]
  speech-to-text-layer --> turn-workflow
  turn-workflow --> fast-exec-library[Fast exec library]
  turn-workflow --> agent-skills[Agent skills]
  turn-workflow --> text-to-speech-layer[Text-to-speech layer]
  text-to-speech-layer --> customer
  agent-skills --> activities-and-services[Activities and services]
  agent-skills --> fast-inference-providers[Fast inference providers]
  temporal-server-and-sdk[Temporal server and SDK] --> turn-workflow
  temporal-server-and-sdk --> fast-exec-library
  temporal-server-and-sdk --> temporal-ui[Temporal UI]
  fast-exec-library --> wrapper-workflows[Wrapper workflows]
  fast-exec-library --> temporal-server-and-sdk
  temporal-ui --> debug-workflow-skill[Debug workflow skill]
  post-call-workflows[Post-call workflows] --> conversation-workflow

Components

Data flow

A customer message or voice utterance enters the conversation workflow. For voice, speech-to-text converts audio, then the turn workflow receives conversation state and kicks off work for classification, possible answers, guardrails, and related skills. Fast inference providers, cached/compressed prompts, constrained small models, and redundant concurrent LLM requests reduce LLM latency. Instead of Temporal child workflows being started sequentially through the parent workflow lock, fast exec starts detached wrapper workflows and listens for result signals. The fastest relevant answer returns to the turn workflow, text-to-speech can be started early to overlap latency, and the agent responds before the caller hangs up.

Why this architecture

The architecture is a deliberate compromise. Temporal remained useful for conversation durability, shared text-agent code, debugging, testing environments, and high-level lifecycle orchestration. But the latency-sensitive turn workflow could not wait for parent-locked child workflows to start one by one. Fast exec trades Temporal's strong child-start guarantees for detached starts, wrapper workflows, and signals, preserving most developer ergonomics and UI debuggability while accepting a smaller window of failure because in voice a late response is equivalent to failure anyway.

Models used: Grok · Cerebras models · small open-source models · speech-to-speech real-time models · Deepgram · Cartesia · ElevenLabs · Claude

Scale

• Child workflow scheduled-to-start latency was shown taking three, four, or five seconds.
  “here it's taken a few seconds, three, four, five seconds, let alone to finish the work, but to start some of this work”transcript, stated
• Voice callers would hang up after waiting two to five seconds for a response.
  “If they have to wait two, three, four, five seconds for a response from the agent, they are going to hang up.”transcript, stated
• Text conversations could last days or weeks.
  “if the customer takes a couple of days after each message to come back, this workflow can go on for for days, days, or weeks.”transcript, stated
• Text-agent turns had taken mid-tens of seconds before latency optimization.
  “For our text agent, our turns were taking kind of mid tens of seconds to generate”transcript, stated
• Optimized LLM responses could be just over 300 milliseconds on good runs but double or triple on bad runs.
  “On a good run, we'd be not much over 300 milliseconds. On a bad run, we could be double, triple, or even more than that.”transcript, stated
• Redundant inference sent three requests for each possible question.
  “we would send three requests to the LLMs for each question that the customer could have asked.”transcript, stated
• Temporal was estimated to add about 30 milliseconds per nested level.
  “our rough estimate was around 30 milliseconds per nested thing that we that we had to kick off.”transcript, stated
• The final wrapper-workflow implementation added roughly 100-200 milliseconds.
  “In the final implementation where we had these wrapper workflows, it was uh between kind of 1 and 200 milliseconds.”transcript, stated
• The speaker said the system still left about 100-200 milliseconds of latency on the table.
  “we're leaving a couple of hundred between 1 and 200 milliseconds of latency on the table by using it.”transcript, stated
• ElevenLabs text-to-speech added about 150-300 milliseconds.
  “the text-to-speech side is anywhere from like 150 to 300 milliseconds-ish with 11 Labs.”transcript, stated
• Self-hosted Deepgram speech-to-text added about 200 milliseconds.
  “on the speech-to-text side, we use Deepgram and we're self-hosting that at the moment. That's like maybe like 200 or so milliseconds.”transcript, stated

Alternatives considered (roads not taken)

Use speech-to-speech real-time models.

Rejected because: They were expressive and low-latency but unreliable for regulated financial services due to prompt injection and hallucination.

Use agent-builder voice products.

Rejected because: They provided good UX and speech layers, but not enough flexibility for the AI engineering team to customize the regulated agent behavior.

Remove Temporal from the turn workflow entirely.

Rejected because: This would reduce latency but was hard because activities were only accessible through Temporal workflows and the team wanted shared code, testing, and debugging.

Use normal Temporal child workflows for all concurrent turn work.

Rejected because: Children of the same parent workflow started sequentially because each transfer task acquired the parent workflow lock.

Run classification, answer generation, and guardrails sequentially.

Rejected because: For voice, the team reduced latency by running classification and possible answers concurrently, discarding irrelevant results after classification returned.

Patterns

durable-execution (infrastructure)

Temporal coordinates long-running conversations and the original text-agent turn stack, giving durability and recoverability for multi-day interactions.

latency-optimization (operations)

The team reduced voice latency through faster providers, compression/caching, cached input tokens, constrained small models, redundant concurrent inference, eager workflow start, early TTS kickoff, and detached workflow starts.

tool-use-orchestration (agent-architecture)

A short-lived turn workflow composes classification, answer generation, guardrails, translation, activities, LLM completions, APIs, and tools to decide the next support action.

model-routing (model-operations)

The voice path races multiple identical or provider/model-diverse LLM requests and uses whichever response arrives first.

guardrails-io-filtering (safety)

The agent runs multiple guardrail skills over candidate answers for quality and compliance in regulated financial services.

observability-tracing (observability)

The team values Temporal UI and a debug workflow skill for inspecting what happened in each turn, why the agent responded that way, and which detached wrapper work produced results.

fallback-degradation (reliability)

For live voice turns, the team accepts losing some child-start guarantees because any work that misses the latency window is effectively failed anyway.

Tradeoffs

Trade child-start guarantees for faster voice turns.

Chose detached wrapper workflows started from side effects/goroutines over normal child workflows with strong start guarantees.

Rationale: For live voice, a child workflow that fails to start is not meaningfully worse than one that starts too late; both miss the response window.

Keep Temporal for high-level coordination.

Chose Temporal for conversation lifecycle, outbound batches, calls, pre-call SMS, and post-call actions over removing Temporal from the voice system entirely.

Rationale: Temporal remains valuable where latency sensitivity is lower and lifecycle coordination, retries, and state visibility matter.

Race redundant LLM work.

Chose three requests per possible customer question, sometimes across providers or models over one sequential classify-then-answer request path.

Rationale: The team spent extra compute to reduce variance and average reply latency while cheap cached tokens and pre-launch scale made the cost acceptable.

Reuse the text agent for voice MVP.

Chose speech-to-text into existing text LLM agent, then text-to-speech out over building a new voice-native agent from scratch.

Rationale: Reusing the text system let the team keep existing skills, libraries, test environments, and debugging, but it later required latency-specific re-engineering.

Constrain small models for holding responses.

Chose strict phatic and pragmatic tags for small, cheap, fast models over asking the model to freely generate natural holding responses.

Rationale: Constrained output made it possible to use fast small models without dropping the regulated quality bar.

Gotchas (what bit them)

The text-agent architecture hid a voice-specific latency trap.

Root cause: Temporal was acceptable for asynchronous text where a minutes-late recovery still mattered, but live voice callers leave before recovery is useful.

Resolution: Keep Temporal for high-level lifecycle work while treating latency-sensitive turn execution differently.

Temporal child workflows appeared concurrent but started sequentially.

Root cause: Transfer-task goroutines each acquired the parent workflow lock before creating a child workflow and writing the run ID to parent history.

Resolution: Start former child workflows as detached wrapper workflows and signal results back to the turn workflow.

Activities being reachable only through Temporal created optimization lock-in.

Root cause: The architecture tied bottom-layer activities to workflows, so converting the turn workflow into a plain activity was not straightforward.

Resolution: Use detached wrapper workflows as an intermediate compromise, and treat the coupling as technical debt for possible future refactor.

Side effects and signals made the workflow hard to understand.

Root cause: Detached workflows solved latency but produced side-effect/signal-heavy histories and channel code that confused engineers and Claude.

Resolution: Wrap the pattern in fast exec futures, add side-effect and signal summaries, and link wrapper workflow URLs from the Temporal UI.

Regulated support could not use the lowest-latency speech-to-speech models.

Root cause: The models were vulnerable to prompt injection and hallucination, even when tested with more sophisticated supervisor setups.

Resolution: Use a speech-to-text, text LLM, and text-to-speech cascade so existing guardrails and text-agent controls stay in the loop.

Testing-environment latency symptoms were initially misattributed to weak workers.

Root cause: The problem first appeared in underpowered test environments, making worker scaling look like the likely cause.

Resolution: Reproduce in production and isolate the root cause by starting the same workflows from a standalone Temporal client outside the shared parent.

Evaluation

The case describes operational validation through latency traces, Temporal UI inspection, testing and production comparison, standalone Temporal-client experiments, good-run/bad-run latency observation, and production voice-call outcomes. The team also relies on guardrails for quality/compliance and workflow debugging to understand why the agent responded a certain way.

Metrics: scheduled-to-start latency · customer wait time before hang-up · LLM response latency · latency variance across good and bad runs · Temporal nested workflow overhead · Temporal wrapper workflow overhead · text-to-speech latency · speech-to-text latency · production call success without hang-up · quality and compliance guardrail pass

Outcomes

• Gradient Labs shipped a voice support agent by reusing its text-agent stack while adding speech-to-text, text-to-speech, and fast inference optimizations.
• The team reduced LLM latency through Grok/Cerebras onboarding, conversation compression/caching, cached prompt restructuring, constrained small models, concurrent possible answers, and redundant requests.
• They diagnosed Temporal's parent-workflow lock as the reason concurrent child workflows started sequentially.
• They built fast exec to start detached wrapper workflows and signal results back while preserving a future-like interface for AI engineers.
• The production voice system now replies in a reasonable amount of time instead of letting callers wait and hang up.
• The team still acknowledges 100-200ms of Temporal latency and may eventually remove Temporal from the low-level turn workflow.

The source reports child workflow start delays of three to five seconds, callers hanging up after two to five seconds, text conversations lasting days or weeks, text-agent turns in the mid-tens of seconds before optimization, good voice runs just over 300ms, three LLM requests per possible question, about 30ms per nested Temporal level, 100-200ms final Temporal overhead, 150-300ms ElevenLabs TTS, and about 200ms self-hosted Deepgram STT.