Building Voice-Enabled AI Assistants with Real-Time Processing

bee · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real-time voice assistants and personal AI memory systems

The speakers were exploring how to build voice-enabled assistants that can respond fast enough for conversation, capture personal context continuously, and use memory without becoming unusable, expensive, or privacy-invasive.

Constraints:

• Voice conversations break down when end-to-end latency exceeds normal turn-taking expectations.
• The speech-to-text, LLM, and text-to-speech stages all contribute to latency and cost.
• Continuous wearable capture creates large personal data stores and hard privacy/product-adoption questions.
• Long-lived memory needs decay, consolidation, and application-specific schemas rather than unbounded recall.

Architecture

The meetup describes a composite architecture for personal assistants: a real-time voice loop captures browser or telephony audio, streams it through a voicebot server, speech-to-text, an LLM, and text-to-speech, while wearable devices continuously capture conversations, segment them, summarize them, and make them searchable through memory and retrieval layers.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  browser-audio-client["browser audio client"] --> voicebot-server["voicebot server"]
  telephony-provider["telephony provider"] --> voicebot-server
  voicebot-server --> speech-to-text-service["speech-to-text service"]
  speech-to-text-service --> llm-endpoint["LLM endpoint"]
  llm-endpoint --> text-to-speech-service["text-to-speech service"]
  text-to-speech-service --> voicebot-server
  wearable-capture-device["wearable capture device"] --> conversation-processing-pipeline["conversation processing pipeline"]
  conversation-processing-pipeline --> speaker-verification["speaker verification"]
  conversation-processing-pipeline --> memory-store["memory store"]
  memory-store --> memory-retrieval-layer["memory retrieval layer"]
  memory-retrieval-layer --> llm-endpoint

Components

Data flow

Audio enters through a browser or telephony provider, the voicebot server forwards it to speech-to-text, the transcript is sent to an LLM, the response is synthesized through text-to-speech, and wearable capture pipelines later run larger models, speaker verification, conversation endpointing, summaries, links, and retrieval over stored conversations.

Why this architecture

The architecture separates fast live-turn components from slower after-the-fact processing because voice UX needs sub-second response, while memory quality, speaker attribution, and conversation segmentation benefit from larger models and background processing.

Models used: GPT-3.5 Turbo · GPT-4 · Claude Haiku · speech-to-text models · text-to-speech models · larger post-processing models

Scale

• The voice UX target is below roughly 1.5 to 2 seconds end-to-end before users start talking again.
  “if you go beyond say 1.5 2 seconds aot lot of people will actually say something again”transcript, stated
• Deepgram-style speech-to-text was described around 200 ms, with self-hosting able to push toward 50 ms at higher GPU cost.
  “we have you know around 200 MC latency we can get that down a lot lower if you host yourself”transcript, stated

  “you can get that 200 milliseconds down to about 50 milliseconds uh at the cost of extra GP compute”transcript, stated
• Hosted GPT-3.5/4 calls were estimated around 400 to 600 ms, with GPT-4 hosted endpoints potentially showing multi-second fluctuations.
  “GPD 3.5 turbo or four you probably get you know 400 maybe 600 milliseconds of latency under hosted API”transcript, stated

  “four is going to be a lot slower uh especially if you're using their hosted API endpoint”transcript, stated
• The demo estimated roughly 6.5 cents for a five-minute call at list pricing.
  “you can do a 5 minute call here for about 6 and a half cent”transcript, stated
• The speaker said free credits could cover about 750 hours of post-call transcription or 500 hours of real-time transcription.
  “for post call uh transcription that's about 750 hours uh for real time that's probably about uh 500 hours of real time transcription”transcript, stated

Alternatives considered (roads not taken)

Pure open-source/off-the-shelf speech stack

Rejected because: The speaker argues it may be workable but often will not be fast enough for real-time conversation without managed or optimized services.

Fully managed hosted APIs for every model stage

Rejected because: Managed APIs simplify implementation, but latency and cost can be worse than colocated or self-hosted deployments.

Continuous video as the first capture modality

Rejected because: The session treats video as a future frontier because power and bandwidth make it harder than audio.

Unlimited append-only memory

Rejected because: Long-lived assistants need decay, invalidation, or consolidation because old facts lose relevance and duplicate memories accumulate.

Patterns

latency-optimization (serving)

The voice loop is decomposed into STT, LLM, and TTS latency budgets, with deployment choices such as self-hosting or Azure colocation used to reduce turn latency.

agent-memory (memory)

The assistants use conversational memory, semantic vector memory, knowledge-graph-like memory, profiles, and thread summaries to personalize future interactions.

rag-retrieval (retrieval)

Wearable-captured conversations are made retrievable so users can query past context and jump back to source conversations.

cost-aware-routing (operations)

The system design discusses swapping providers, model choices, and self-hosted GPU compute based on cost and latency.

Tradeoffs

Use faster or colocated services for real-time voice.

Chose Higher-cost GPU/self-hosted or Azure-colocated deployments where necessary over Cheaper hosted endpoints for every component.

Rationale: Latency is visible to users in voice conversations, and cheaper hosted endpoints can have unacceptable delay or variance.

Separate live response from background memory processing.

Chose Fast real-time voice loop plus larger after-call processing over Running every enrichment step inside the live conversation path.

Rationale: Speaker verification, summaries, links, and conversation segmentation are valuable but can happen after the latency-critical turn.

Model memory as multiple specialized stores.

Chose Thread summaries, user profiles, semantic vector memory, and knowledge-graph-style memory over One undifferentiated transcript history.

Rationale: Different applications need different memory schemas, and one memory representation cannot serve every assistant equally well.

Gotchas (what bit them)

Users interpret slow voice responses as conversation failure.

Root cause: A conversational turn has a human latency budget, and the STT, LLM, and TTS chain can easily exceed it.

Resolution: Optimize each stage separately and consider self-hosting or colocated managed services where latency matters.

Conversation endpointing is harder than utterance detection.

Root cause: A pause may be hesitation rather than the end of a semantic conversation, and whole-day capture cannot be stored as one block.

Resolution: Use voice activity, location, topics, and LLM-level signals to segment continuous capture into distinct conversations.

Personal memory accumulates stale and duplicate facts.

Root cause: Long-running capture stores years of facts whose relevance changes, and naive retrieval can surface repetitive memories.

Resolution: Add recency/importance weighting, decay, invalidation, and potentially hierarchical summarization such as RAPTOR.

Technical demos do not guarantee adoption.

Root cause: Wearable assistants impose form-factor, setup, and privacy burdens that can outweigh technical novelty.

Resolution: Treat form factor, cognitive burden, developer setup, and social acceptability as first-class product constraints.

Evaluation

The session evaluates voice systems mainly through latency, cost, accuracy, and qualitative demo behavior rather than a formal benchmark suite.

Metrics: sub-second response target · 1.5 to 2 second user patience threshold · speech-to-text latency · LLM latency · text-to-speech latency and quality · cost per minute or per five-minute call · memory retrieval usefulness

Outcomes

• The session demonstrates a practical architecture for real-time voice assistants with explicit latency and cost budgets.
• Wearable capture prototypes can summarize, segment, search, and cite personal conversations, but the speakers repeatedly frame the product and memory problems as unresolved.
• The memory discussion identifies decay, consolidation, and schema choice as open production design questions.

The strongest scale facts are operating targets and cost estimates: sub-1.5-second turn latency, 200 ms down to 50 ms speech-to-text with more GPU compute, roughly 6.5 cents per five-minute call, and hundreds of hours of transcription credits.