Building and Testing Production AI Applications at CircleCI

circleci · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production testing for LLM developer tools

CircleCI needed to build AI-assisted developer tooling, such as an error summarizer for CI stack traces, while adapting CI/CD testing practices to LLM outputs that are probabilistic, subjective, potentially biased, costly, and difficult to validate with ordinary deterministic tests.

Constraints:

• LLM outputs are probabilistic and non-deterministic rather than stable database-style results
• quality can involve subjective properties such as helpfulness, tone, toxicity, and whether the answer seems right
• model-graded evals can have model-specific and positional biases
• exhaustive testing is too slow and expensive because LLM calls have latency and token costs
• function-calling and structured-output paths can still produce invalid JSON
• cost scales with usage, prompt length, response length, and model choice
• human review and production feedback remain necessary because test cases are not production

Architecture

CircleCI's concrete use case is an error summarizer that ingests long CI/CD stack traces and produces likely causes and recommendations, with an experimental path toward using the developer diff to propose fixes. Around that product path, the team describes a production testing architecture: model-graded evals for subjective LLM quality, table-driven and mutation-style test sets to offset evaluator bias, critical-user-journey selection instead of exhaustive replay, function-calling error handling and fallback paths, user-feedback capture for a data flywheel, and cost controls through usage reporting, fine-tuning, model routing, and dedicated-capacity decisions.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ci-failure-context[CI failure context] --> error-summarizer[Error summarizer]
  error-summarizer --> llm-provider[LLM provider]
  error-summarizer --> function-calling-parser[Function-calling parser]
  error-summarizer --> evaluation-suite[Evaluation suite]
  error-summarizer --> feedback-collector[Feedback collector]
  evaluation-suite --> model-graded-evaluator[Model-graded evaluator]
  model-graded-evaluator --> llm-provider
  feedback-collector --> evaluation-suite
  evaluation-suite --> human-review[Human review]
  feedback-collector --> human-review
  llm-provider --> cost-router[Cost router]

Components

Data flow

A CI failure produces a stack trace and, for experimental repair, a developer diff. The error summarizer sends that context to an LLM to summarize likely causes and recommendations. During development, generated outputs are evaluated with model-graded eval prompts that ask another model to judge quality, correctness, helpfulness, tone, or non-toxicity. Test suites use multiple table-driven examples and mutation-like rephrasings rather than a single golden answer. In production, explicit feedback such as thumbs up/down and implicit signals such as reprompting, regeneration, or asking for a human feed future evaluation datasets. Cost management watches model-level token bills and can route simple queries to cheaper models, fine-tune to shorten prompts, or reserve expensive models for harder requests.

Why this architecture

CircleCI maps LLM application testing back onto CI/CD habits without pretending LLMs are deterministic services. Model-graded evals provide automated signal for subjective output quality, but table-driven cases, randomized phrasing, mutation tests, and human review offset known biases. Critical user journeys bound cost and latency instead of replaying every possible production request. Function-calling and fallback handling preserve ordinary software resilience. The feedback collector turns real user behavior into use-case-specific eval data, which is more valuable than generic benchmarks for a production developer tool.

Models used: GPT-4 · GPT-3.5 · OpenAI function calling · fine-tuned OpenAI models · Azure hosted GPT-4

Scale

• The error summarizer targets long CI stack traces, including deep Java stack traces.
  “not having to kind of eyeball through a very deep you know Java Java stack Trace”transcript, stated
• The speakers frame exhaustive test replay as too time-consuming for every production request.
  “you would log every single request your application ever gotten the output of it and you would replay it every single time”transcript, stated
• Token billing charges for both prompt input tokens and generated output tokens.
  “you pay for tokens in and you pay for tokens out um is is typically the pricing model there”transcript, stated
• OpenAI cost reporting was described as breaking down usage by GPT-3.5 and GPT-4.
  “open AI recently they their cost reporting they started breaking things down by what's gpt3 po five or GPT 4”transcript, stated
• Routing can target about 50% of the price when a simpler model is good enough.
  “this actually will give you a totally good enough or maybe an even better response for 50% of the price”transcript, stated
• The podcast contrasts shared API pool access with buying dedicated GPT-4 capacity.
  “Azure has an open AI deployment you can buy your own GPT for hosted thing”transcript, stated

  “you can get some dedicated capacity for for certain Foundation models”transcript, stated

Alternatives considered (roads not taken)

Evaluate with string matching or a single golden output

Rejected because: LLM output is probabilistic and language can be correct in many wordings and styles.

Trust one model-graded evaluation result

Rejected because: Models have own-output and positional biases, so one comparison can be misleading.

Test exhaustively against every possible or historical request

Rejected because: LLM evals are expensive and time-consuming, so the team recommends critical journeys and table-stakes datasets.

Use only expensive models for all requests

Rejected because: At scale, cost can be managed by fine-tuning, shortening prompts, routing simple queries to cheaper models, or buying dedicated capacity.

Assume function calling always returns valid JSON

Rejected because: Structured-output function calls can still fail and must be handled with retries or fallback paths.

Patterns

llm-as-judge (evaluation)

CircleCI uses model-graded evals to judge generated LLM output for subjective quality and correctness when string matching is insufficient.

eval-driven-development (quality)

The team frames evals as part of CI/CD: prompts, model choices, and output behavior should be tested before production deployment.

human-in-the-loop-gate (quality)

Human review remains important for subjective quality and edge cases even when model-graded evals are automated.

data-quality-loop (feedback)

User ratings, regenerations, reprompts, and abandonment signals feed future testing datasets and model-performance understanding.

fallback-degradation (resilience)

Structured-output and mediocre-answer failures are handled with retries, fallback paths, or graceful error handling rather than trusted blindly.

model-routing (cost)

CircleCI describes routing easy queries to cheaper models and reserving expensive models for hard queries as usage scales.

cost-aware-routing (cost)

Cost reporting, token accounting, fine-tuning, routing, and capacity choices are treated as a future operational layer for successful AI features.

fine-tuning-adaptation (model-ops)

Fine-tuning is discussed as a way to encode repeated context and shorten prompts, lowering per-request token cost.

Tradeoffs

Evaluation strategy

Chose model-graded evals plus human review over string matching or deterministic unit tests alone.

Rationale: LLM outputs can be semantically correct in many forms and require subjective quality judgment, but automated graders still need human oversight.

Test coverage

Chose critical journey and weighted scenario testing over exhaustive replay of every production request.

Rationale: Exhaustive replay would increase time and cost, while focused journeys target the flows with greatest user value and risk.

Cost scaling

Chose routing and fine-tuning where justified over one expensive model for every path.

Rationale: Routing and fine-tuning can reduce per-request cost, but they add branching behavior and testing complexity.

Provider infrastructure

Chose managed foundation model APIs over building a private AI system from scratch.

Rationale: Managed services provide fast startup and low operational overhead when internal AI-system experts are scarce.

Feedback source

Chose production user behavior as eval data over generic benchmarks alone.

Rationale: Real reprompts, regenerations, thumbs, abandonment, and human requests reveal use-case-specific model performance.

Gotchas (what bit them)

Traditional deterministic tests do not map cleanly to LLM outputs.

Root cause: Language-model outputs are probabilistic, non-deterministic, and can be correct in many styles.

Resolution: Use evals, multiple scenarios, subjective criteria, and statistical confidence instead of exact expected strings.

Model-graded evals can contain hidden evaluator bias.

Root cause: Some models prefer their own outputs, and comparison prompts can have positional bias.

Resolution: Randomize, run multiple examples, use table-driven cases, and understand evaluator behavior before trusting scores.

Testing every possible production case is impractical.

Root cause: LLM calls cost money and time, and high-volume exhaustive replay would be prohibitive.

Resolution: Focus eval spend on critical user journeys, invariants, table-stakes datasets, and differentiated flows.

Function calling does not eliminate structured-output failures.

Root cause: The model can fail to return valid JSON even when asked to call a function with a JSON payload.

Resolution: Treat it as normal error handling: retry, fall back, or route down an alternate path.

Cost optimization adds system complexity.

Root cause: Fine-tuning, routing, model selection, and capacity choices create more knobs and more paths to test.

Resolution: Do not over-optimize prematurely, but understand that successful AI usage creates a future cost-management workload.

Evaluation

CircleCI recommends model-graded evals, table-driven test cases, randomized or weighted scenarios, mutation-style rephrasings, adversarial prompt cases for open chat, critical user journeys, fallback/error testing for function calling, human review, and a production feedback flywheel. The goal is not deterministic proof, but enough confidence to deploy while understanding evaluator bias, model behavior, and use-case-specific failure modes.

Metrics: correctness · helpfulness · non-toxicity · tone/style fit · model-graded score · scenario-weighted aggregate score · positional-bias checks · adversarial prompt handling · valid JSON rate · retry/fallback rate · thumbs up/down · reprompt/regeneration rate · cost by model · token input/output cost

Outcomes

• CircleCI built an error summarizer that condenses long stack traces into likely causes and recommendations
• the team identified automated bug fixing from diffs as a still-experimental extension
• the discussion established model-graded evals as a practical testing path for subjective LLM output
• the team recommends table-driven, mutation-style, and critical-journey tests rather than single examples
• function-calling responses require JSON error handling, retry, and fallback strategies
• user behavior becomes a feedback signal for future evaluation datasets
• cost management through fine-tuning, routing, cost reporting, and capacity choices is treated as a scaling concern

The transcript is an engineering discussion rather than a metrics-heavy case study. It names long Java stack traces, GPT-4 evaluator bias, per-token input/output pricing, possible 50% price savings from routing when a cheaper model is good enough, and managed/dedicated capacity decisions such as Azure-hosted GPT-4 deployments.