LLMs for Cloud Incident Management and Root Cause Analysis

microsoft · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Cloud incident management and root-cause analysis

Microsoft wanted to reduce the manual effort required to diagnose production cloud incidents and recommend mitigations across hyperscale services such as Microsoft 365 and Teams.

Constraints:

• Incident response must quickly detect issues, identify root causes, and mitigate customer impact.
• Recommendations need to work from incident-ticket text such as titles, summaries, error messages, and anomalous behavior descriptions.
• The approach had to be evaluated on real production incidents rather than toy examples.
• The model would face evolving service behavior, requiring awareness of staleness and updated incident context.

Architecture

Microsoft evaluated a GPT-3.x incident recommendation pipeline. Incident tickets provide title and summary fields, fine-tuned GPT-3/GPT-3.5 models generate root-cause and mitigation recommendations, and automated lexical/semantic metrics plus on-call engineer ratings compare the generated text to incident-management ground truth. The next architecture direction adds retrieval over historical incidents, troubleshooting guides, logs, metrics, and dependency graphs for a conversational incident-diagnosis assistant.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Ticket_fields["Incident ticket fields"] --> GPT_model["GPT-3.x recommendation model"]
  Fine_tuning["Fine-tuning dataset"] --> GPT_model
  GPT_model --> Root_cause["Root-cause recommendation"]
  GPT_model --> Mitigation["Mitigation recommendation"]
  Root_cause --> Mitigation
  Root_cause --> Engineer_review["On-call engineer review"]
  Mitigation --> Engineer_review
  Ground_truth["Incident management ground truth"] --> Fine_tuning
  Ground_truth --> Evaluation["Offline evaluation harness"]
  GPT_model --> Evaluation
  Retrieval_corpus["Retrieval corpus"] --> Conversational_assistant["Conversational diagnosis assistant"]
  Conversational_assistant --> Engineer_review

Components

Data flow

An incident author enters a ticket title and summary with diagnostic details. Those fields become LLM input for root-cause and mitigation generation. Historical incident data supports fine-tuning and offline metric evaluation, while on-call engineers provide human usefulness ratings. The proposed RAG extension retrieves incident context from documents and logs before the model participates in the incident discussion.

Why this architecture

The study chose incident-ticket-to-recommendation generation because titles and summaries are already present in the incident workflow, and historical incident records provide both fine-tuning data and evaluation references. Fine-tuning was necessary because generic zero-shot models underperformed on specialized cloud-operations language.

Models used: GPT-3 · GPT-3.5 · Davinci-002 · Curie · Codex · Davinci · RoBERTa · CodeBERT · ChatGPT

Scale

• Microsoft 365 supports hundreds of thousands of organizations.
  “Building highly reliable hyperscale cloud services such as Microsoft 365 (M365), which supports the productivity of hundreds of thousands of organizations, is very challenging.”article, stated
• The evaluation used more than 40,000 incidents from more than 1,000 services.
  “We did a rigorous study on more than 40,000 incidents generated from more than 1000 services and compared several LLMs in zero-shot, fine-tuned, and multi-task settings.”article, stated
• Davinci-002 improved root-cause and mitigation recommendation metrics by at least 15.38% and 11.9% over GPT-3 models.
  “For root cause and mitigation recommendation tasks**, Davinci-002 (GPT-3.5) provided at least 15.38% and 11.9% gains over all the GPT-3 models**, respectively”article, stated
• Fine-tuned GPT-3.5 improved average lexical similarity by 45.5% for root-cause generation and 131.3% for mitigation generation over zero-shot.
  “A fine-tuned GPT-3.5 model improved the average lexical similarity score by 45.5% for root cause generation and 131.3% for mitigation generation tasks over zero-shot”article, stated
• More than 70% of on-call engineers rated the recommendations at least 3 out of 5 for usefulness.
  “More than 70% of on-call engineers gave a rating of 3 out of 5 or better for the usefulness of recommendations in a real-time production setting.”article, stated

Alternatives considered (roads not taken)

Zero-shot inference on pretrained GPT models

Rejected because: Fine-tuning on incident data produced much stronger lexical and semantic performance for root-cause and mitigation tasks.

Use only incident title and summary without richer operational context

Rejected because: The researchers identified discussion entries, logs, metrics, and dependency graphs as needed future context for better diagnosis.

Rely only on automated lexical and semantic metrics

Rejected because: The study also interviewed incident owners because real production usefulness cannot be fully captured by text similarity alone.

Patterns

fine-tuning-adaptation (modeling)

Microsoft fine-tuned GPT-3 and GPT-3.5 on historical incident data to adapt general LLMs to cloud incident root-cause and mitigation language.

eval-driven-development (evaluation)

The model comparison used multiple lexical and semantic metrics plus human incident-owner evaluation before claiming production usefulness.

human-in-the-loop-gate (operations)

On-call engineers evaluated recommendation usefulness, keeping production applicability anchored in practitioner judgment.

rag-retrieval (retrieval)

The forward-looking design adds retrieval augmentation over incident records, troubleshooting guides, logs, metrics, and dependency graphs before LLM diagnosis.

Tradeoffs

Fine-tune incident models

Chose fine-tuned GPT-3/GPT-3.5 on incident data over zero-shot use of pretrained models.

Rationale: Domain fine-tuning substantially improved root-cause and mitigation similarity metrics.

Evaluate with both metrics and practitioners

Chose semantic/lexical metrics plus incident-owner interviews over offline metric-only validation.

Rationale: The system had to show both similarity to historical ground truth and usefulness to on-call engineers.

Move toward retrieval-augmented conversational diagnosis

Chose retrieved documents/logs plus ChatGPT-style incident discussion over standalone generation from title and summary alone.

Rationale: The researchers identified richer context as necessary for improving diagnosis beyond the initial ticket fields.

Gotchas (what bit them)

Incident models can go stale as services and incident distributions change.

Root cause: Cloud services continuously evolve, so historical incident training data may stop representing current operational reality.

Resolution: Plan for frequent retraining with latest incident data and retrieval-augmented context from live operational sources.

Machine-reported and customer-reported incidents do not behave the same for LLM recommendation quality.

Root cause: Machine-reported incidents are more repetitive and structured than customer-reported incidents.

Resolution: Treat incident category as a deployment consideration rather than assuming one model quality profile across all incidents.

Text-similarity metrics alone may not prove operational usefulness.

Root cause: A recommendation can be lexically or semantically close to historical text while still not helping an on-call engineer resolve a live incident.

Resolution: Pair automated metrics with human evaluation from incident owners in real-time production context.

Evaluation

The study compared several GPT-3, GPT-3.5, and encoder baselines on real incident data using lexical and semantic metrics, top-1/top-5 settings, zero-shot/fine-tuned/multi-task variants, and human usefulness ratings from incident owners.

Metrics: BLEU-4 · ROUGE-L · METEOR · BERTScore · BLEURT · NUBIA · Top-1 and Top-5 recommendations · on-call engineer usefulness rating

Outcomes

• Fine-tuned GPT-3.5 outperformed GPT-3 baselines for root-cause and mitigation recommendations.
• Fine-tuning produced large gains over zero-shot settings for both root-cause and mitigation generation.
• More than 70% of on-call engineers rated the recommendations useful enough for a score of 3 out of 5 or better.
• The researchers identified retrieval-augmented conversational diagnosis as the next architecture direction.

The evidence comes from Microsoft Research's evaluation over more than 40,000 real production incidents from more than 1,000 services; the article positions this as promising incident-management assistance rather than full autonomous remediation.