Building a Voice Assistant with Open Source LLMs: From Demo to Production

weights_&_biases · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLMOps productionization

Weights & Biases used a custom voice assistant to demonstrate that moving from an impressive LLM demo to production requires exact outputs, low latency, reproducibility, and evaluation-driven iteration.

Constraints:

• The model had to translate natural speech into exact function calls for predefined skills.
• Voice interactions were sensitive to latency above a few hundred milliseconds.
• LLM development is non-deterministic and experimental rather than linear like ordinary software development.
• Vibes checks could not reliably decide whether small accuracy changes were improvements.

Architecture

The assistant combines Whisper speech recognition, a local Llama/Mistral-style model run through llama.cpp, Python-like function-call generation, and a skill library for music, weather, math, and news. The productionization loop used Weights & Biases experiment tracking, evaluation sets, prompt engineering, model swaps, synthetic example generation, and QLoRA fine-tuning.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  voice-user[Voice User] --> whisper-transcription[Whisper Transcription]
  whisper-transcription --> local-function-calling-model[Local Function Calling Model]
  llama-cpp[llama.cpp] --> local-function-calling-model
  local-function-calling-model --> skill-functions[Skill Functions]
  skill-functions --> external-apis[External APIs]
  synthetic-training-data[Synthetic Training Data] --> local-function-calling-model
  local-function-calling-model --> evaluation-framework[Evaluation Framework]
  evaluation-framework --> experiment-tracker[Experiment Tracker]

Components

Data flow

Speech is transcribed, the local LLM converts the text into a structured function call, the selected skill executes against an API, and accuracy/latency/results are logged. When errors appear, the team updates prompts, switches models, adds examples, fine-tunes, and compares metrics instead of trusting demos.

Why this architecture

The system needed local, fast, open-source inference and exact function-call outputs. The development process needed external experiment tracking because the real production asset was the learning across failed and successful experiments.

Models used: Whisper · Llama 2 · Llama chat · Mistral · ChatGPT · QLoRA fine-tuned model

Scale

• Over 70% of the audience reported LLM applications in production.
  “audience you see 70% of this audience but everywhere I go to give a talk even with companies that you wouldn't necessarily think of as the most forward thinking they are starting to have gen”transcript, stated
• Weights & Biases works with thousands of customers across industries.
  “we have thousands of customers are not Foundation model Builders right there there companies in in nearly every industry”transcript, stated
• The voice-assistant setup was motivated by running a 7B Llama 2 model on roughly $200 Rock Pi-class hardware.
  “showing me that he got um a 7 billion um model uh I think llama 2 running on a rock pie which is like a $200 device similar price to Alexa so I thought okay if you have that maybe we could build an architecture that would um kind of do the things that Alexa does so built a”transcript, stated
• The project improved from 0% to 98% accuracy through iteration.
  “went with um llama 2 at the time and put it in a default prompt and it didn't work 0% accuracy and so every kind of customer goes through this journey to try to improve the accuracy or whatever the real kind of user facing metric is I'll tell you briefly about my journey”transcript, stated

  “that data set um fine-tuning plus mol actually led to 98% accuracy so this is super satisfying project 98% accuracy actually starts to feel like hey maybe we could we could ship this thing um and as a surprising byproduct”transcript, stated
• Production teams may track thousands or tens of thousands of metrics.
  “application we see thousands of metrics we see tens of thousands of metrics we get requests like hey can I do Rex search across my metrics because I have like so many metrics I can't even I can't find you know the the the metrics”transcript, stated

Alternatives considered (roads not taken)

Default prompt with Llama 2

Rejected because: It produced 0% accuracy for exact code/function-call output.

Stop after prompt engineering at 75% accuracy

Rejected because: The system was still too annoying to use in practice.

Rely on vibes for model/prompt changes

Rejected because: Vibes cannot resolve whether an apparent change is a measurable improvement.

Build each step fully before showing users

Rejected because: The speaker calls this an enterprise anti-pattern that delays end-user feedback.

Patterns

eval-driven-development (productionization)

The project used accuracy measurements to decide prompt, model, and fine-tuning changes instead of relying on vibes.

structured-output-contract (tool calling)

The assistant required the LLM to produce exact Python-looking function-call syntax for skills.

fine-tuning-adaptation (model adaptation)

QLoRA fine-tuning on generated examples produced the final improvement to 98% accuracy.

synthetic-data-generation (data)

A larger model generated a couple thousand examples from manually created seeds.

observability-tracing (reproducibility)

Weights & Biases tracking preserved failed experiments and learnings so teammates could continue from them.

latency-optimization (inference)

The system favored small local models because voice assistant latency was central to usability.

Tradeoffs

Productionization strategy

Chose Lightweight prototype plus end-user feedback over Enterprise-style step-by-step perfection before users see anything.

Rationale: The speaker observed teams failing by never getting early versions into users' hands.

Model evolution

Chose Switch from Llama 2 to Mistral when a better model appeared over Continue optimizing the original model only.

Rationale: A same-API model switch gave a small but measurable free improvement.

Evaluation design

Chose Application-specific evals and metrics over Generic advice about whether to use RAG, fine-tuning, or prompt engineering.

Rationale: The right technique should be answerable by the system's own evaluation framework.

Gotchas (what bit them)

Stakeholders may over-trust compelling demos.

Root cause: AI demos can appear astonishing while hiding the distance to production reliability.

Resolution: Expose the system to users early, measure accuracy, and build evals before shipping.

Prompt/model changes can improve some cases while regressing others.

Root cause: LLM changes are experimental and non-deterministic rather than monotonic software improvements.

Resolution: Use eval sets that reveal the difference between competing model/prompt versions.

Reproducibility is difficult but necessary for AI IP preservation.

Root cause: Humans forget to write down every prompt, workflow, and failed experiment.

Resolution: Track experiments passively with outside processes.

Fine-tuning requires a dataset, not just intent.

Root cause: QLoRA is tractable, but the model needs enough good examples.

Resolution: Manually seed examples and use a larger model to generate/filter thousands more.

Evaluation

The talk recommends layered evaluation: must-never-fail tests, fast 20-30 second iteration checks, larger nightly eval suites, and metrics correlated to the actual user experience. The voice assistant itself measured function-call accuracy across prompt/model/fine-tuning variants.

Metrics: Function-call accuracy · Latency · Must-never-fail cases · Fast eval runtime · Nightly eval pass/fail · Metric correlation with user experience · Number of tracked experiments

Outcomes

• The assistant went from 0% to 75%, 79%, and then 98% accuracy.
• Synthetic examples generated by ChatGPT helped create a fine-tuning dataset quickly.
• The project demonstrated that multiple techniques together, not one silver bullet, were needed to approach production quality.

This row shares the same source as the home-project case study; it is a conference example used to generalize LLMOps practices.