AI Error Summarizer Implementation: A Tiger Team Approach

circleci · Tech · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Developer tooling for CI/CD failure comprehension

CircleCI wanted to identify a useful generative-AI feature quickly without starting from a vague mandate to add AI for its own sake.

Constraints:

Architecture

CircleCI treated the first AI feature as a bounded product-discovery exercise: a tiger team interviewed PMs, collected ideas, avoided unnecessary model infrastructure, and shipped an error summarizer through foundation-model APIs and prompt engineering.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ai-tiger-team["AI tiger team"]
  product-manager-interviews["Product-manager interviews"]
  idea-backlog["Idea backlog"]
  foundation-model-api["Foundation model API"]
  prompt-engineering-layer["Prompt engineering layer"]
  error-summarizer-feature["Error summarizer feature"]
  product-feedback-loop["Product feedback loop"]
  ai-tiger-team --> product-manager-interviews
  product-manager-interviews --> idea-backlog
  idea-backlog --> error-summarizer-feature
  foundation-model-api --> prompt-engineering-layer
  prompt-engineering-layer --> error-summarizer-feature
  error-summarizer-feature --> product-feedback-loop
  product-feedback-loop --> ai-tiger-team
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
AI tiger team Small discovery group that learned the AI landscape and scoped a first feature. Product-manager interviews, Idea backlog, Error summarizer feature
Product-manager interviews Collected actual product pain points without asking teams to start from AI ideas. Idea backlog
Idea backlog Held candidate AI opportunities for triage and selection. Error summarizer feature
Foundation model API Provides text generation for the chosen summarization feature. Prompt engineering layer
Prompt engineering layer Shapes CI error context into useful user-facing summaries. Error summarizer feature
Error summarizer feature Adds a narrow AI capability to the existing CircleCI product. Product feedback loop
Product feedback loop Feeds learning back into future AI-enabled feature selection. AI tiger team

Data flow

The team moved from product interviews and an idea backlog into a narrow prompt/API implementation rather than building a custom AI platform first.

“regular software Engineers they're more than capable of you know interfacing with the apis and chaining things together so you can get started right away”transcript, stated

Why this architecture

The architecture stayed small because the team learned that core customer problems mattered more than broad AI infrastructure complexity.

“I don't want you to think about AI when you're answering this question like what are the challenges that your team is facing that you don't really have a solution for”transcript, stated
“you could really kind of drown in all the technical complexities of the stuff but really what you know should have grounded me much more if I had to do it all over again is really kind of a deeper um focus on on those core values”transcript, stated

Models used: foundation text LLMs

Alternatives considered (roads not taken)

Training a custom model or building heavy AI infrastructure first

Rejected because: The team judged that text LLM APIs were enough for the first feature and that a low time-and-cost investment was appropriate.

“chances are pretty good that you do not need to train your own model you know you there's chat GPT there's llama there's D There's You Know audio image video there's so many text llms”transcript, stated
“you don't have to move mountains with this technology it's you can but you don't have to like for a very low investment uh of time and cost”transcript, stated

Starting ideation from a mandate to use AI

Rejected because: The team asked PMs about unsolved challenges without asking them to think about AI first.

“I don't want you to think about AI when you're answering this question like what are the challenges that your team is facing that you don't really have a solution for”transcript, stated

Diving into AI infrastructure complexity before product value

Rejected because: The source explicitly says deeper focus on core values would have avoided unnecessary technical rabbit holes.

“you could really kind of drown in all the technical complexities of the stuff but really what you know should have grounded me much more if I had to do it all over again is really kind of a deeper um focus on on those core values”transcript, stated

Patterns

prompt-chaining (workflow)

The case frames the initial implementation path as regular product engineers calling foundation-model APIs and chaining behavior together rather than building custom ML systems.

“regular software Engineers they're more than capable of you know interfacing with the apis and chaining things together so you can get started right away”transcript, stated

Tradeoffs

How ambitious the first AI-enabled product feature should be.

Chose a subtle, narrow feature inside the existing product over a large standalone AI initiative or custom model.

Rationale: The team argues low-effort additions can produce customer value without moving mountains or training a model.

“you don't have to move mountains with this technology it's you can but you don't have to like for a very low investment uh of time and cost”transcript, stated

Whether to train a custom model for the first feature.

Chose existing foundation models and prompt engineering over training a proprietary model up front.

Rationale: The speaker says most teams can begin with available models, prompts, and possibly later fine-tuning or retrieval if needed.

“chances are pretty good that you do not need to train your own model you know you there's chat GPT there's llama there's D There's You Know audio image video there's so many text llms”transcript, stated

Gotchas (what bit them)

A broad AI mandate can distract the team from actual customer problems.

Root cause: Leadership urgency around generative AI did not initially identify which product pain point mattered.

Resolution: Interview product managers about unsolved problems without asking them to pre-frame answers as AI ideas.

“I don't want you to think about AI when you're answering this question like what are the challenges that your team is facing that you don't really have a solution for”transcript, stated

Engineers can lose time in unnecessary AI-infrastructure rabbit holes.

Root cause: The landscape includes NLP, vector databases, fine-tuning, and custom models even when a simple product feature may not need them.

Resolution: Ground exploration in product value and start with simple foundation-model API usage before adding advanced infrastructure.

“you could really kind of drown in all the technical complexities of the stuff but really what you know should have grounded me much more if I had to do it all over again is really kind of a deeper um focus on on those core values”transcript, stated

Evaluation

The podcast discusses discovery and launch but does not provide a formal evaluation method or user-impact metrics for the summarizer.

Metrics: six or seven weeks on the AI team · about 75 candidate ideas

“we spent was it like six or seven weeks I think on the on the AI team um yeah the first week I want to say was just kind of reading watching videos”transcript, stated

Outcomes

Company podcast; technical implementation details and production quality metrics are limited.

“two Engineers who worked on building a feature that we recently launched which is the AI error summarizer and today we want to talk about what it's like to build AI enabled product”transcript, stated