Evaluation Driven Development for LLM Reliability at Scale

dosu · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM reliability for open-source maintenance automation

Dosu needed to keep an LLM-powered software-maintenance assistant reliable as it grew from manually inspected responses to thousands of repository installations and many production failure modes.

Constraints:

• LLM prompt changes can improve one domain while regressing another.
• Manual grep and print-log inspection does not scale to thousands of repositories.
• The product includes multiple submodules and LLM calls inside traces.
• Production traffic contains many request types and unforeseen failure modes.
• Prompts need code-review-like discipline and traceability.

Architecture

Dosu adopted evaluation-driven development and LangSmith tracing. The team creates initial evals for new behavior, launches to users, monitors production traces for failure modes, adds examples to offline eval datasets, iterates against those evals, and relaunches. LangSmith traces function inputs, rendered prompts, LLM outputs, metadata, feedback, latency, and errors so engineers can search production behavior and convert failures into evaluation data.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  dosu-production-assistant["Dosu production assistant"] --> langsmith-tracing-sdk["LangSmith tracing SDK"]
  langsmith-tracing-sdk --> trace-search-and-monitoring["Trace search and monitoring"]
  trace-search-and-monitoring --> failure-mode-triage["Failure-mode triage"]
  failure-mode-triage --> offline-eval-datasets["Offline eval datasets"]
  offline-eval-datasets --> prompt-and-model-iteration["Prompt and model iteration"]
  prompt-and-model-iteration --> dosu-production-assistant

Components

Data flow

Dosu instruments LLM-related functions with LangSmith tracing, sends production traces into LangSmith, searches for signals such as feedback, errors, latency, and negative sentiment, identifies failure modes, gathers similar examples, adds them to offline eval datasets, and uses those evals to iterate on prompts, models, and core logic.

Why this architecture

Dosu needed production traceability because manual inspection stopped scaling. Treating prompts as code and converting observed failures into eval examples gave the team a repeatable loop for changing probabilistic behavior while watching for regressions across domains.

Models used: Dosu AI teammate · LLMs · LangSmith · @traceable decorator · Evaluation-driven development

Scale

• Dosu is installed on thousands of repositories.
  “Today, Dosu is installed on thousands of repositories and generates responses at all hours of the day.”article, stated
• The company built dozens of submodules.
  “We've built dozens of submodules to intelligently handle different types of scenarios”article, stated
• Dosu launched in late June 2023 when every response could still be manually inspected.
  “Dosu launched in late June, 2023. Back then, our volume was low enough for us to inspect every single response.”article, stated

Alternatives considered (roads not taken)

Continue manual log inspection.

Rejected because: Manual grep and print statements worked early but became impossible as repository installations and response volume grew.

Treat prompt tweaks as informal product changes.

Rejected because: Small prompt changes can create unpredictable cross-domain regressions.

Use a generic observability tool without code-level traces.

Rejected because: Dosu needed metadata between LLM requests and within a single trace, not just request logs.

Patterns

eval-driven-development (LLM reliability)

Dosu turns production failures into offline eval examples and iterates on prompts, models, and logic against those evals.

observability-tracing (production monitoring)

LangSmith captures function and LLM traces so Dosu can search production behavior and debug failures.

prompt-versioning (change control)

Dosu treats prompts as code and keeps prompt changes in Git with code-level standards.

data-quality-loop (eval dataset creation)

Production failures are searched, grouped, and added to offline eval datasets.

Tradeoffs

Prioritize quality over speed.

Chose Trace and evaluate slow responses over Optimizing only for latency.

Rationale: Response time still matters, but Dosu explicitly says quality comes first and slow responses need diagnosis.

Adopt LangSmith SDK instrumentation.

Chose @traceable decorator and LangSmith UI over Manual logs or less customizable monitoring.

Rationale: The SDK gave fine-grained controls and linked function inputs, prompts, and LLM output in one trace.

Gotchas (what bit them)

Prompt changes create hidden regressions.

Root cause: LLM behavior is probabilistic and domain-coupled rather than deterministic.

Resolution: Use EDD to evaluate every change against known behaviors and failure cases.

Failure modes multiply with input variety.

Root cause: Dosu receives simple questions, upgrade error traces, feature-status requests, and other diverse inputs.

Resolution: Search traces by multiple signals and turn anomalies into eval examples.

Very large user inputs caused slow responses.

Root cause: Users pasted thousands of lines of logs or raw embedding float values.

Resolution: LangSmith trace search exposed the pattern so it could enter the EDD loop.

Evaluation

Dosu creates initial evals, monitors production, identifies failure modes, searches for additional examples, adds them to offline eval datasets, iterates, and relaunches. LangSmith provides trace search over feedback, errors, latency, sentiment, and metadata.

Metrics: Explicit feedback · Recent error incidents · Response time delays · Negative sentiment · Failure-mode examples · Prompt/model regression coverage

Outcomes

• Dosu moved from manual response inspection to scalable production tracing and EDD.
• LangSmith exposed unforeseen failures, including slow responses from huge pasted logs or embeddings.
• The EDD loop lets Dosu improve prompts, models, and logic while guarding against regressions.

The case reports thousands of repository installations, all-hours response generation, and dozens of submodules, but not exact request volume.