Building a Voice Assistant from Open Source LLMs: A Home Project Case Study

weights_&_biases · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Voice assistants and LLMOps

Weights & Biases used a home-built voice assistant to show why LLM demos are easy but production systems require evaluation, reproducibility, latency discipline, and iterative model/prompt/fine-tuning improvements.

Constraints:

• Voice assistants need low latency; waiting more than a few hundred milliseconds is painful.
• Natural speech must be converted into exact function-call code, not merely plausible text.
• Default prompts and models can perform at unusable accuracy.
• The important IP is the experimental learning, including failed prompts/workflows, not just the final prompt or model.

Architecture

The voice assistant uses Whisper for speech transcription, a local open-source LLM running through llama.cpp on consumer hardware to convert natural language into Python-like function calls, and a library of simple skills such as weather, music, math, and news. The project improved from 0% accuracy through prompt engineering, Llama chat, Mistral, synthetic examples, and QLoRA fine-tuning to 98% accuracy.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-speech[User Speech] --> whisper[Whisper]
  whisper --> local-llm[Local LLM]
  llama-cpp-runtime[llama.cpp Runtime] --> local-llm
  local-llm --> skill-library[Skill Library]
  skill-library --> external-apis[External APIs]
  evaluation-dataset[Evaluation Dataset] --> local-llm
  local-llm --> experiment-tracking[Experiment Tracking]
  evaluation-dataset --> experiment-tracking

Components

Data flow

A user's speech is transcribed by Whisper, passed to a local LLM that maps the request into structured Python-like function calls such as weather(location='Boston'), and then the corresponding skill function calls an API. Experiment tracking records prompts, models, datasets, failures, and metrics so future iterations can reproduce the learning.

Why this architecture

The architecture was driven by voice latency and privacy/local-control constraints: speech is transcribed locally, a small local model creates deterministic function-call code, and iterative evaluation determines whether prompt engineering, model switching, or fine-tuning is needed.

Models used: Whisper · Llama 2 · Llama chat · Mistral · ChatGPT · QLoRA fine-tuned model

Scale

• More than 70% of the audience reported having LLM applications in production.
  “ever seen More than 70% so why did you come to this talk you you figured it out right it's not going great not going great all right you want to put it in slideshow”transcript, stated
• Around 30% of the audience reported custom solutions.
  “how many is uh custom Solutions interesting that's going be 30% and how many purchase Solutions Wow way way less what are you what are you purchasing well I put like co-pilot from GitHub in that category of using okay well if you put co-pilot in”transcript, stated
• The home assistant used a 7B model on roughly $200 hardware in the motivating setup.
  “showing me that he got um a 7 billion um model uh I think llama 2 running on a rock pie which is like a $200 device similar price to Alexa so I thought okay if you have that maybe we could build an architecture that would um kind of do the things that Alexa does so built a”transcript, stated
• Default prompt performance started at 0% accuracy.
  “went with um llama 2 at the time and put it in a default prompt and it didn't work 0% accuracy and so every kind of customer goes through this journey to try to improve the accuracy or whatever the real kind of user facing metric is I'll tell you briefly about my journey”transcript, stated
• Prompt/error iteration improved accuracy to 75%.
  “was making looking at the errors incorporating feedback got the accuracy up to 75% so pretty satisfying pretty iterative process probably typical to what you're doing or what your engineers are doing now it's kind of a cool demo my daughter is like mildly impressed if”transcript, stated
• Switching to Mistral improved accuracy to 79%.
  “switch to mistol um mistol came out mid project here um and mol's model for free exact same API to llama two got the accuracy 79%”transcript, stated
• Fine-tuning plus Mistral reached 98% accuracy.
  “that data set um fine-tuning plus mol actually led to 98% accuracy so this is super satisfying project 98% accuracy actually starts to feel like hey maybe we could we could ship this thing um and as a surprising byproduct”transcript, stated

Alternatives considered (roads not taken)

Existing Alexa behavior

Rejected because: It did not remember the child's repeated favorite-song behavior.

Mycroft-style open-source Alexa alternative

Rejected because: The speaker found it out of date and not built around modern LLMs.

Prompt engineering alone

Rejected because: It reached a promising demo but not production-ready accuracy.

Vibes-only testing

Rejected because: Vibes cannot distinguish meaningful improvements such as 75% versus 79% accuracy.

Patterns

eval-driven-development (evaluation)

Accuracy metrics guided each change from default prompt through prompt engineering, model switching, and fine-tuning.

structured-output-contract (tool calling)

Natural language must become exact Python-looking function calls that invoke skills.

fine-tuning-adaptation (model adaptation)

QLoRA fine-tuning on generated and filtered examples provided the final accuracy jump.

synthetic-data-generation (data)

ChatGPT generated thousands of training examples from a small manually created set.

observability-tracing (operations)

The lesson is to passively track experiments, prompts, workflows, failures, and metrics so the learning is reproducible.

latency-optimization (inference)

The project used small local models because voice interaction latency above a few hundred milliseconds felt painful.

Tradeoffs

Inference location

Chose Small local/open-source model through llama.cpp over Larger slower remote inference for every voice request.

Rationale: Voice latency made fast local inference more important than using the largest available model.

Improvement path

Chose Prompt engineering, model switching, and fine-tuning together over Picking a single technique such as prompt engineering or fine-tuning alone.

Rationale: Each technique contributed incremental gains toward production accuracy.

Evaluation cadence

Chose Multiple eval types from fast checks to nightly tests over One vibes-based release judgment.

Rationale: Teams need fast iteration checks, must-never-fail tests, and larger nightly evaluation suites.

Gotchas (what bit them)

Compelling AI demos can be much farther from production than teams expect.

Root cause: LLM behavior is experimental and non-deterministic, so a demo can hide accuracy, latency, and edge-case failures.

Resolution: Build evaluation frameworks and track experiments rather than relying on demos or vibes.

Failed experiments become lost IP if they are not captured.

Root cause: In AI work the learning from prompts, workflows, and failures matters more than the final artifact alone.

Resolution: Passively track everything needed to reproduce runs and preserve learnings across people.

Accuracy that feels okay in a demo remains annoying in daily voice use.

Root cause: A 75-79% function-call success rate produces frequent visible failures for a voice assistant.

Resolution: Continue with model switching, synthetic data, and fine-tuning until accuracy reaches production-feeling levels.

Metrics can fail to reflect user value if they are not connected to the experience.

Root cause: Production ML/LLMOps teams may track thousands of metrics because many failure modes affect user experience differently.

Resolution: Build application-specific eval sets and correlate metrics with the customer value being delivered.

Evaluation

The project evaluated function-call accuracy through successive experiments, tracked all prompts/models/workflows/failures in Weights & Biases, used manually and synthetically generated examples for fine-tuning, and emphasized multiple eval suites for production applications.

Metrics: Function-call accuracy · Latency in hundreds of milliseconds · Prompt/model experiment history · Dataset quality · Fast real-time eval pass/fail · Nightly evaluation results · User-experience correlation

Outcomes

• Default prompting started at 0% accuracy.
• Prompt/error iteration reached 75% accuracy.
• Switching to Mistral reached 79% accuracy.
• QLoRA fine-tuning with generated examples reached 98% accuracy.
• The assistant unexpectedly worked in other languages through the Whisper/LLM stack.

The case is a home project used to illustrate productionization lessons, not a commercial deployment.