Building a Systematic LLM Evaluation Framework from Scratch

coda · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Productivity software

Coda needed systematic GenAI evaluation after model upgrades changed behavior and broke prompt expectations across AI assistant, table, and RAG features.

Constraints:

• LLM outputs are non-deterministic and model behavior can change outside Coda's control.
• Manual playground workflows did not scale beyond small benchmark sets.
• Prompt-only evaluation diverged from application-code behavior once the app manipulated prompts and payloads.
• Human judgment remained necessary because model upgrades produced unexpected edge cases.

Architecture

Coda evolved its evaluation stack in four phases: OpenAI Playground for early prompt iteration, Coda tables as an internal prompt playground, Vellum for collaborative prompt/dataset experiments, and application-integrated automated evaluations whose scores are uploaded to Braintrust for visualization, trend tracking, and benchmark drilldown.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ai-product-features[AI Product Features] --> application-code[Application Code]
  ai-product-features --> benchmark-dataset[Benchmark Dataset]
  benchmark-dataset --> openai-playground[OpenAI Playground]
  benchmark-dataset --> coda-prompt-playground[Coda Prompt Playground]
  benchmark-dataset --> vellum[Vellum]
  application-code --> automated-eval-runner[Automated Eval Runner]
  benchmark-dataset --> automated-eval-runner
  automated-eval-runner --> braintrust[Braintrust]
  braintrust --> human-reviewers[Human Reviewers]
  human-reviewers --> benchmark-dataset

Components

Data flow

A feature's prompts and benchmark data first moved through manual playgrounds and Coda tables, then through Vellum for shared prompt/dataset runs, and eventually through integration-test-style automated evals close to application code. Completed eval runs upload scores to Braintrust, where engineers inspect aggregate trends and individual benchmark failures.

Why this architecture

Coda moved from playgrounds to app-integrated evals because prompt-only and manual workflows did not reflect production behavior, did not scale, and consumed engineering time that the team wanted to spend improving AI quality.

Models used: DaVinci · GPT-3.5 Turbo · GPT-4 Turbo

Scale

• The initial OpenAI Playground approach did not scale beyond about 10 data points.
  “great for iteration speed it doesn't really scale well Beyond like 10 data points since it requires a lot of manual copy and pasting for it to work to get the eval going and more over it lacks the ability to grade like an AI output and keep tracks of like how it behaves”transcript, stated
• At one point the team had fewer than five engineers.
  “playground uh has increased by a lot we were still pretty lean at that time we only have less than five engineers and instead of spending time maintaining the tool which we spend a lot of time doing at that time we were rather focusing on improving the AI quality uh instead of”transcript, stated
• The mature framework supported more than 15 features.
  “fast forward uh approximately a year we now have about 15 and more uh features supported we have five and more Engineers involved in AI in some Capac capacity and we have it's way more now uh we have more than 50 automated checks so that the amount of mandate work that”transcript, stated
• The mature framework had more than 50 automated checks.
  “we have more than 50 automated checks so that the amount of mandate work that we do right now is uh relatively much less”transcript, stated
• Coda was running more than 500 automated evaluation jobs per week.
  “like evaluation job runs we're basically running more than 500 drops per week and they're all automated now and that's our journey so far um key lessons uh what are the key lessons that uh I guess I knew before I started it um this is a really hard question to answer because a year back”transcript, stated
• The benchmark dataset grew from about 25 examples to more than 5,000 data points.
  “when we first started we only have one feature with uh about 25 examples in a Koda data uh T table and now we have grown to more than 5,000 data points”transcript, stated

Alternatives considered (roads not taken)

Manual OpenAI Playground testing

Rejected because: It was easy to start but required manual copy-paste, lacked grading, and did not track behavior over time.

Maintaining Coda's internal prompt playground as the main evaluation tool

Rejected because: Conversation-model changes increased maintenance work, and the lean team preferred to spend time on AI quality.

Prompt-playground evaluation decoupled from production code

Rejected because: Actual application code assembled payloads differently, so playground passes did not guarantee production behavior.

Snowflake and Mode reporting for eval results

Rejected because: The team started maintaining SQL and reports instead of improving AI quality.

Patterns

eval-driven-development (evaluation)

Coda ran systematic evals close to application code and used score trends and benchmark failures to guide prompt/model changes.

canary-rollout (release)

The team compared model versions and prompt changes through evals before confidently moving to cheaper/faster models.

human-in-the-loop-gate (evaluation)

Human review remained the final edge-case discovery layer even as many checks became automated.

synthetic-data-generation (data)

Coda used seed benchmarks and GenAI to multiply benchmark data over time.

observability-tracing (operations)

Braintrust gave Coda timeline trends, aggregate scores, and drilldowns into individual failing benchmarks.

Tradeoffs

Build versus buy for evaluation tooling

Chose Adopt Vellum and later Braintrust over Continue maintaining internal Coda-based playground/reporting tools.

Rationale: The lean team wanted to spend effort improving AI quality rather than maintaining tools and SQL reports.

Evaluation location

Chose Automated evals close to application code over Prompt-only playground evals.

Rationale: App-integrated evals better reflect what users actually see after production code manipulates prompts and payloads.

Human judgment in eval loops

Chose Humans as final edge-case gatekeepers over Fully automated scoring only.

Rationale: Humans catch failure modes no existing automated check anticipated, then turn observations into new checks.

Benchmark growth strategy

Chose Start small and expand with usage, feedback, and synthetic multiplication over Wait for a perfect comprehensive dataset before evaluating.

Rationale: Coda's dataset investment grew over time from a small baseline into thousands of data points.

Gotchas (what bit them)

A provider model upgrade changed output behavior enough to break existing prompts.

Root cause: GPT-3.5 Turbo became more conversational than instructional, and model behavior changed outside Coda's control.

Resolution: Create systematic evals so model and prompt changes can be measured before release.

Prompt playground results diverged from production application behavior.

Root cause: The playground evaluated prompts in isolation while the actual app manipulated and combined prompts differently.

Resolution: Move evals closer to application code and run them like integration tests.

Manual vibe checks consumed too much time as feature count grew.

Root cause: Vellum still required people to rate outputs by hand for many features and prompts.

Resolution: Automate checks where possible and use humans for novel edge cases.

Eval datasets can miss real-world issues if they do not cover actual user cases.

Root cause: Automated checks and human review cannot compensate for poor benchmark coverage.

Resolution: Use PM/designer input, alpha/beta feedback, customer-facing teammates, and internal usage data to enrich benchmarks.

Evaluation

Coda evaluates AI features with benchmark datasets, app-integrated automated evals, Braintrust timelines and drilldowns, human review for unexpected edge cases, and ongoing dataset curation from usage, product, customer, and synthetic sources.

Metrics: Binary score trends where 1 is good and 0 is failure · Aggregated evaluation-run scores · Individual benchmark pass/fail · Automated check count · Evaluation jobs per week · Benchmark dataset size

Outcomes

• The evaluation framework grew to more than 15 supported features.
• Coda involved five or more engineers in AI evaluation capacity.
• Coda grew to more than 50 automated checks and more than 500 automated eval jobs per week.
• Benchmark data grew from about 25 examples to more than 5,000 data points.

The talk describes roughly a year of evolution from one feature and manual checks to broad automated evaluation coverage.