AI-Powered Postmortem Analysis for Site Reliability Engineering

zalando · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: SRE postmortem analysis and infrastructure investment planning

Zalando needed to extract strategic reliability patterns from thousands of postmortems accumulated over two years, where manual reading at 15-20 minutes per document could not answer cross-team datastore failure questions fast enough.

Constraints:

• postmortems vary in depth, clarity, and root-cause assumptions
• manual review scales poorly across thousands of documents
• incident documents contain PII, business metrics, GMV losses, and other sensitive data
• LLMs introduced hallucination, surface attribution, latency, and numerical extraction failures
• human curation remained necessary for trust and final reporting

Architecture

Zalando built a chained LLM pipeline over postmortems, focused first on datastore technologies such as Postgres, DynamoDB, ElastiCache, S3, and Elasticsearch. The pipeline uses specialized stages rather than a single agent: summarization condenses incident reports into five core dimensions with TELeR prompting and no-guessing constraints; classification buckets summaries by datastore technology; analysis converts bucketed summaries into failure fingerprints; pattern detection aggregates fingerprints into a one-page technology-pattern report; opportunity analysis links patterns back to the corpus and internal incident datasets. Human expertise is used at each stage through examination, labeling, quality control, curation, proofreading, and editorial review.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  postmortem-corpus[postmortem-corpus] --> summarization-stage[summarization-stage]
  summarization-stage --> classification-stage[classification-stage]
  classification-stage --> analyzer-stage[analyzer-stage]
  analyzer-stage --> pattern-stage[pattern-stage]
  pattern-stage --> opportunity-stage[opportunity-stage]
  trusted-incident-dataset[trusted-incident-dataset] --> opportunity-stage
  human-curation-loop[human-curation-loop] --> summarization-stage
  human-curation-loop --> classification-stage
  human-curation-loop --> analyzer-stage
  human-curation-loop --> pattern-stage
  human-curation-loop --> opportunity-stage

Components

Data flow

Thousands of postmortem documents enter the pipeline. Each document is summarized into issue, root cause, impact, resolution, and preventive actions. Technology classification maps summaries into datastore buckets. Analyzer stages convert summaries into short digests describing a technology's role in the incident. Pattern stages aggregate digests over time into a report about recurring incident patterns. Opportunity stages combine patterns with the original postmortem corpus and trusted internal incident datasets for investment recommendations. Human curators compare outputs to originals, label correctness, refine prompts/models, and proofread final reports.

Why this architecture

Zalando rejected a no-code agentic prototype after encountering performance limits, inaccuracies, and hallucinations. A hybrid chain of specialized stages made every input and output human-evaluable. The map-fold pattern lets each postmortem be processed independently and then aggregated into higher-level patterns. Claude Sonnet 4 on Bedrock became the current model partly because legal/compliance alignment was necessary for sensitive incident data.

Models used: Claude Sonnet 4 · AWS Bedrock · open source 3B-12B models · LM Studio

Scale

• The corpus contains thousands of postmortems over two years.
  “analyze thousands of postmortems”article, stated
• Thoughtful manual reading takes about 15-20 minutes per postmortem.
  “It takes about 15-20 minutes to thoughtfully read a single postmortem”article, stated
• A dedicated reviewer can process about four postmortems per hour.
  “a dedicated reviewer can process maybe four postmortems per hour assuming a continuous focus”article, stated
• The current Claude Sonnet 4 release processes each postmortem in about 30 seconds.
  “The most recent release, based on Claude Sonnet 4, processes each postmortem in approximately 30 seconds”article, stated
• The pipeline can process annual data analysis in under 24 hours.
  “This has enabled the processing of annual data analysis in under 24 hours.”article, stated
• Early small models showed up to 40% hallucination probability in summary and analysis phases.
  “We have observed up to 40% probability for hallucination at summary and analysis phases.”article, stated
• Human curation started at 100% of output batches and later relaxed to 10-20% samples.
  “During the pipeline development, we conducted 100% human curation of output batches.”article, stated

  “As the system matured, we relaxed human curation to 10-20% of randomly sampled summaries from each output batch.”article, stated

Alternatives considered (roads not taken)

manual strategic postmortem review

Rejected because: Thousands of documents at 15-20 minutes each made company-scale pattern analysis too slow and cognitively expensive.

no-code agentic solution

Rejected because: The prototype was unfeasible due to performance limitations, inaccuracies, and hallucinations.

LLM extraction of numerical business impact

Rejected because: The team could not get reliable accuracy for GMV, EBIT loss, affected customers, and repair time.

Patterns

prompt-chaining (prompting)

The pipeline chains summarization, classification, analysis, pattern detection, and opportunity stages.

batch-offline-processing (operations)

Thousands of postmortems are processed into summaries, buckets, digests, and reports for retrospective analysis.

human-in-the-loop-gate (evaluation)

Human curation labels outputs, checks them against source postmortems, and proofreads final reports.

guardrails-io-filtering (safety)

Summarization prompts forbid guessing, assumptions, and speculation, requiring missing information to be stated explicitly.

eval-driven-development (evaluation)

The team compared generated digests with original postmortems, used labels to refine prompts, and adjusted curation rates as confidence improved.

Tradeoffs

Use a hybrid staged pipeline.

Chose LLM stages with human-evaluable inputs and outputs over no-code agentic automation.

Rationale: The staged design improved confidence after the no-code prototype hit hallucination, accuracy, and performance limits.

Use Bedrock/Claude despite starting with local open-source models.

Chose Claude Sonnet 4 on AWS Bedrock over small local 3B-12B models.

Rationale: Small models had high hallucination rates, while compliance-aligned Bedrock enabled stronger hosted models.

Relax but retain human curation.

Chose 10-20% random sampling after maturity over 100% curation forever or no curation.

Rationale: Initial full curation improved prompts and model selection, but sampling preserved oversight with less human load.

Gotchas (what bit them)

Long-context LLMs do not automatically solve postmortem recall.

Root cause: Large contexts can increase latency, memory usage, and cost while still not guaranteeing perfect recall.

Resolution: Use a chain of specialized stages with single objectives and human-evaluable outputs.

Small models hallucinated plausible incident summaries.

Root cause: Models inferred facts from weak signals such as a technology name appearing in a linked playbook title.

Resolution: Move to stronger models, strict prompt requirements, examples, and human-led curation.

LLMs were unreliable for numerical incident impact extraction.

Root cause: Postmortem text did not yield reliable GMV, EBIT, customer, or repair-time extraction.

Resolution: Use the internal incident dataset as the numerical source of truth for opportunity analysis.

Cloud-hosted LLM use required legal alignment.

Root cause: Postmortems include PII, business metrics, GMV losses, and other sensitive incident data.

Resolution: Treat compliance as a precondition before using Bedrock-hosted Claude.

Evaluation

Zalando evaluates the pipeline through human curation against original postmortems, hallucination monitoring, latency, annual-processing time, sampling rates, strategic pattern usefulness, and downstream investment outcomes.

Metrics: 15-20 minutes manual read time per postmortem · about four postmortems per reviewer hour manually · approximately 30 seconds per postmortem with Claude Sonnet 4 · annual data analysis under 24 hours · 100% initial human curation · 10-20% mature random-sample curation · 25% of subsequent datastore incidents shielded by automated change validation

Outcomes

• The pipeline reduced analysis time from days to hours.
• It processed annual data analysis in under 24 hours.
• It surfaced datastore investment opportunities such as automated change validation.
• Automated change validation shielded Zalando from 25% of subsequent datastore incidents.

The source is strong on operational scale: thousands of postmortems, two years of incident data, 15-20 minutes per manual read, approximately 30 seconds per Claude Sonnet 4 analysis, and annual processing under 24 hours.