Evaluations Driven Development for Production LLM Applications

anaconda · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM debugging assistant reliability

Anaconda needed to improve an AI coding assistant whose initial error-handling success rate was extremely low across model and temperature settings, especially for debugging workflows that dominate assistant usage.

Constraints:

• The fetched source is short and covers the baseline table and EDD motivation; later improvement details are supported by the ZenML summary.
• Initial bug-fix accuracy ranged from 0% to 13% across tested models/settings.
• The assistant had to identify root causes and provide fixes that actually resolve code errors.
• Temperature, model choice, and prompting all affected reliability.
• The team wanted improvements through prompt/eval iteration rather than retraining.

Architecture

Anaconda built an evaluations-driven workflow around an in-house llm-eval framework. The team evaluates assistant responses to code snippets plus error messages, tests model/version/temperature combinations, and uses prompt engineering such as few-shot learning, chain-of-thought prompting, and agentic feedback iteration to improve error-handling reliability.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  code-and-error-prompt["Code and error prompt"] --> anaconda-assistant["Anaconda Assistant"]
  anaconda-assistant --> llm-eval-framework["llm-eval framework"]
  llm-eval-framework --> prompt-optimization-loop["Prompt optimization loop"]
  prompt-optimization-loop --> anaconda-assistant

Components

Data flow

A code snippet and error message are sent to the assistant, the assistant proposes a root cause and fix, llm-eval scores whether the fix resolves the issue, and the results drive prompt/query updates and model-setting comparisons.

Why this architecture

Anaconda used EDD because baseline results made the reliability gap measurable. Comparing models and temperatures gave the team a baseline, and prompt iteration against eval results provided a faster path to improved debugging accuracy than model retraining.

Models used: GPT-3.5-Turbo v0125 · Mistral 7B Instruct v0.2 · Anaconda Assistant · llm-eval

Scale

• Each baseline model/temperature setting was run for 500 iterations.

  "Iterations

Success Rate

GPT-3.5-Turbo (v0125)

0

500

12%"

— article, stated

• The best initial baseline was 13% success.
  “only able to correctly identify the bug and provide a working solution in at most 13% of the test executions.”article, stated
• The summary reports improved success rates of 63-100% after EDD prompt optimization.
  “success rates improved to 63-100% across models”summary, stated
• Mistral 7B reportedly reached 100% accuracy at temperature 1 after optimization.
  “Mistral 7B achieving 100% accuracy at temperature 1”summary, stated

Alternatives considered (roads not taken)

Accept the initial model outputs.

Rejected because: Baseline bug-fix success reached at most 13%, far below production reliability needs.

Optimize only model choice or temperature.

Rejected because: All initial model/temperature combinations performed poorly, so prompt/eval iteration was needed.

"GPT-3.5-Turbo (v0125)

1

500

13%"

— article, stated

Patterns

eval-driven-development (assistant reliability)

Anaconda uses eval results to iteratively improve prompts and model settings for debugging accuracy.

prompt-chaining (prompt optimization)

The summary describes few-shot learning and chain-of-thought prompting as part of the optimization loop.

llm-as-judge (agentic feedback)

LLMs analyze evaluation results to suggest prompt improvements in Anaconda's agentic feedback iteration.

Tradeoffs

Use prompt/eval iteration rather than retraining.

Chose EDD plus prompt engineering over Model retraining as the first lever.

Rationale: The case claims large reliability gains were achieved through systematic prompt optimization.

Run repeated iterations per condition.

Chose 500 executions per model/temperature setting over Single-run anecdotal testing.

Rationale: Repeated runs measure probabilistic performance instead of one lucky or unlucky completion.

"GPT-3.5-Turbo (v0125)

0

500

12%"

— article, stated

Gotchas (what bit them)

Initial debugging success was far too low for production.

Root cause: The assistant could not reliably identify root causes and fixes under the original prompt/model settings.

Resolution: Use llm-eval results to drive prompt engineering and iterative improvement.

Temperature creates reliability tradeoffs.

Root cause: Higher temperature increases diversity but can reduce coherence or factual accuracy.

Resolution: Evaluate model/version/temperature settings instead of assuming one default is best.

Evaluation

Anaconda evaluates debugging responses for whether they correctly identify the root cause and provide a working fix, comparing language models, versions, and temperature settings over repeated executions.

Metrics: Root-cause accuracy · Working fix success · Success rate by model · Success rate by temperature · Iterations per condition

Outcomes

• Baseline success rates were 0-13% across tested model/temperature combinations.
• The summary reports post-optimization success rates of 63-100% across models.
• Mistral 7B Instruct reportedly reached 100% accuracy at temperature 1.
• The case demonstrates substantial reliability gains from EDD-driven prompt optimization.

The fetched source provides the 500-iteration baseline table; the ZenML summary provides the post-optimization 63-100% and 100% Mistral result.