AI-Assisted Root Cause Analysis System for Incident Response

meta · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Incident response root-cause ranking

Meta responders needed help finding culprit code changes among thousands of monorepo changes at investigation start.

Constraints:

Architecture

Meta’s RCA system uses a two-stage narrowing architecture. A heuristic retriever first reduces thousands of recent code changes to a few hundred using ownership, directory, and runtime-code-graph signals. A fine-tuned Llama ranker then evaluates candidates in context-window-sized groups, uses an election-style aggregation loop, and returns a final top-five candidate list for responders.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  investigation-context["investigation context"]
  heuristic-retriever["heuristic retriever"]
  runtime-code-graph["runtime code graph"]
  candidate-code-changes["candidate code changes"]
  llama-ranker["Llama ranker"]
  election-aggregator["election aggregator"]
  top-five-suggestions["top-five suggestions"]
  investigation-context --> heuristic-retriever
  heuristic-retriever --> runtime-code-graph
  heuristic-retriever --> candidate-code-changes
  candidate-code-changes --> llama-ranker
  llama-ranker --> election-aggregator
  election-aggregator --> top-five-suggestions
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
investigation context Signals available at investigation creation time for a web-monorepo incident. heuristic retriever
heuristic retriever Uses ownership, code/directory signals, and runtime code graph exploration to shrink thousands of changes to hundreds. investigation context, candidate code changes, runtime code graph
runtime code graph Relationship graph of impacted systems used as a retrieval signal. heuristic retriever
candidate code changes Reduced candidate set passed to the LLM ranking stage. heuristic retriever, Llama ranker
Llama ranker Fine-tuned Llama model that ranks candidate changes in batches. candidate code changes, election aggregator
election aggregator Aggregates repeated top-five rankings until only five final candidates remain. Llama ranker, top-five suggestions
top-five suggestions Root-cause candidates shown to responders for investigation. election aggregator

Data flow

At investigation creation time, incident context enters the heuristic retriever, which narrows thousands of possible code changes to hundreds. The Llama ranker scores groups of up to 20 changes, returns top-five candidates per prompt, and the election aggregator repeats/combines votes until five root-cause candidates remain for responders.

4 supporting quotes
“The system incorporates a novel heuristics-based retriever that is capable of reducing the search space from thousands of changes to a few hundred without significant reduction in accuracy”article, stated
“Once we have reduced the search space to a few hundred changes relevant to the ongoing investigation, we rely on a LLM-based ranker system to identify the root cause across these changes.”article, stated
“To rank the changes, we structure prompts to contain a maximum of 20 changes at a time, asking the LLM to identify the top five changes.”article, stated
“The output across the LLM requests are aggregated and the process is repeated until we have only five candidates left.”article, stated

Why this architecture

The design avoids asking an LLM to reason over thousands of changes at once. Heuristic retrieval preserves enough accuracy while fitting the LLM stage into manageable candidate batches, and election ranking handles context-window limits by repeatedly aggregating partial rankings.

4 supporting quotes
“AI offers an opportunity to streamline the process, reducing the time needed and helping responders make better decisions.”article, implied
“The ranker system uses a Llama model to further reduce the search space from hundreds of potential code changes to a list of the top five.”article, stated
“ranking through election was most effective to accommodate context window limitations and enable the model to reason across different changes.”article, stated
“The biggest lever to achieving 42% accuracy was fine-tuning a Llama 2 (7B) model using historical investigations for which we knew the underlying root cause.”article, stated

Models used: Llama 2 7B fine-tuned · Llama

Scale

Alternatives considered (roads not taken)

Rank every possible change directly with the LLM

Rejected because: The search space starts at thousands of changes and exceeds practical context/ranking limits.

“reducing the search space from thousands of changes to a few hundred without significant reduction in accuracy”article, implied

Use a single ranking pass without election aggregation

Rejected because: Election ranking better handled context window limitations and reasoning across many changes.

“ranking through election was most effective to accommodate context window limitations and enable the model to reason across different changes.”article, stated

Patterns

retrieval-ranking (incident-response)

Heuristics shrink the code-change search space before an LLM ranker orders likely culprits.

“The system incorporates a novel heuristics-based retriever that is capable of reducing the search space from thousands of changes to a few hundred without significant reduction in accuracy using, for example., code and directory ownership or exploring the runtime code graph of impacted systems. Once”article, stated

confidence-thresholding (safety)

Low-confidence answers are withheld to favor precision over reach.

“We also rely on confidence measurement methodologies to detect low confidence answers and avoid recommending them to the users – sacrificing reach in favor of precision.”article, stated

Tradeoffs

How broad to make recommendations.

Chose precision-oriented confidence filtering over maximal reach.

Rationale: Meta avoids showing low-confidence guesses even if that means helping fewer investigations.

“We also rely on confidence measurement methodologies to detect low confidence answers and avoid recommending them to the users – sacrificing reach in favor of precision.”article, stated

Gotchas (what bit them)

AI-generated root-cause suggestions can mislead responders.

Root cause: A wrong culprit in an incident can waste mitigation time or send engineers down the wrong path.

Resolution: Prioritize explainability, closed feedback loops, confidence measurement, and reproducibility.

“For instance, it can reduce effort and time needed to root cause an investigation significantly, but it can potentially suggest wrong root causes and mislead engineers.”article, stated

Evaluation

Meta backtested root-cause identification against known investigations and fine-tuned on historical RCA examples.

Metrics: 42% top-five accuracy · ~5,000 SFT examples · thousands to hundreds retrieval reduction

“cause identification during investigations. * Our testing has shown this new system achieves 42% accuracy in identifying root causes for investigations at their creation time related to our web monorepo.”article, stated

Outcomes

AI-assisted ranker rather than autonomous RCA agent; marked non-agentic.

“By integrating AI-based systems into our internal tools we’ve successfully leveraged them for tasks like onboarding engineers to investigations and root cause isolation.”article, stated