Automating Post Incident Review Summaries with GPT-4

canva · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Incident management documentation

Canva needed to reduce reliability-engineer toil and improve consistency in Post Incident Review summaries stored in Confluence.

Constraints:

• PIR summaries must remain blameless
• reports can contain sensitive internal links, emails, and Slack channel names
• summaries need consistent structure without losing incident phase relationships
• generated summaries should be auditable against human edits
• model cost scales linearly with report tokens

Architecture

Canva built a pipeline that fetches PIRs from Confluence, parses HTML into raw text, sanitizes sensitive data, prompts GPT-4-chat with a structured blameless-summary prompt and two example PIR/summary pairs, archives generated summaries to a data warehouse, places summaries on Jira tickets, and uses Jira webhooks to capture manual edits for quality monitoring.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  confluence-pir-store[Confluence PIR store] --> html-parser[HTML parser]
  html-parser --> sensitive-data-sanitizer[Sensitive data sanitizer]
  sensitive-data-sanitizer --> gpt-4-chat-summarizer[GPT-4 chat summarizer]
  gpt-4-chat-summarizer --> summary-archive[Summary archive]
  summary-archive --> jira-ticket-sync[Jira ticket sync]
  jira-ticket-sync --> summary-archive

Components

Data flow

Raw PIRs are fetched from Confluence and parsed from HTML to text. Sanitization removes sensitive links, emails, and Slack names. GPT-4-chat generates the summary using a system prompt plus few-shot examples. The generated summary is archived with metadata and copied to Jira, where human edits are captured back to the warehouse through webhooks.

Why this architecture

Canva chose GPT-4-chat because the chat API could use system messages and role-labeled examples to produce a more consistent summary structure than completion, while fine-tuning lacked enough high-quality examples. Sanitization protects internal data before public-model calls, and Jira/webhook tracking turns human edits into a lightweight quality-monitoring loop.

Models used: GPT-4-chat · GPT completion gpt-4-32k evaluated · GPT fine tune davinci-002 evaluated

Scale

• Canva has more than 150 million monthly active users.
  “We have over 150 million monthly active users with over 17 billion designs across more than 100 languages. Maintaining a smooth user experience requires tracking and analyzing incidents during Canva's development phases.”article, stated
• Canva has more than 17 billion designs across over 100 languages.
  “We have over 150 million monthly active users with over 17 billion designs across more than 100 languages. Maintaining a smooth user experience requires tracking and analyzing incidents during Canva's development phases.”article, stated
• Available fine-tuning data was around 1,500 examples of varied quality.
  “We confine our available training data to existing PIRs and their summaries, amounting to around 1500 examples of varied quality.”article, stated
• A single PIR plus examples can contain up to 10,000 tokens and cost at most about $0.60 per summary with GPT-4.
  “Given that a single PIR, along with sample PIRs and corresponding summaries that inform the GPT model of the expected response format, can contain up to 10,000 tokens, the maximum estimated cost per summary is a manageable $0.6.”article, stated
• After about two months, most AI-generated summaries remained unaltered.
  “After running the project for approximately two months, data indicates that most of the AI-generated PIR summaries remain unaltered by engineers.”article, stated

Alternatives considered (roads not taken)

Fine-tuned davinci-002

Rejected because: The available examples were too few and varied in quality for the detail extraction Canva wanted.

GPT completion API

Rejected because: Completion performed well but was less structurally consistent than chat messages with system guidance and labeled examples.

Manual reliability-engineer summaries

Rejected because: Manual summaries consumed engineer time and became inconsistent.

Patterns

privacy-preserving-processing (governance)

Canva removes internal links, emails, and Slack channel names before sending PIR text to GPT-4.

prompt-versioning (prompting)

The summary prompt defines goals, desired components, constraints, and few-shot examples.

human-in-the-loop-gate (workflow)

Engineer edits on Jira tickets are captured and compared with generated summaries.

eval-driven-development (evaluation)

Canva compared fine-tune, completion, and chat outputs manually before choosing GPT-4-chat.

cost-aware-routing (operations)

Canva explicitly evaluated per-report token cost and judged GPT-4 cost manageable.

Tradeoffs

Use GPT-4-chat

Chose chat API with system messages and role-labeled examples over fine-tuned davinci-002 or GPT completion.

Rationale: Chat gave better structure and critical-detail handling than the alternatives.

Prompt for target length

Chose 600-character target in prompt over hard API response length limit.

Rationale: Prompted length avoids truncation while still discouraging overly long summaries.

Use two example pairs

Chose two PIR/summary few-shot examples over one example summary.

Rationale: Multiple examples avoid rigidly mirroring a single summary structure.

Gotchas (what bit them)

Fine-tuning was not viable with limited noisy examples.

Root cause: The 1,500 available examples were varied in quality and insufficient for detailed extraction.

Resolution: Canva selected GPT-4-chat with prompt examples instead.

Direct API response limits can truncate summaries.

Root cause: Hard output limits can lead to unfinished responses.

Resolution: Canva prompts for a 600-character target and resubmits overly long summaries for condensing.

Blameless incident writing needs prompt-level guardrails.

Root cause: Incident reports can mention individuals, but summaries should avoid blame.

Resolution: The system prompt directs blameless summaries and uses temperature 0.

Evaluation

Canva manually compared about 40 summaries across model options and then monitored human edits after production launch.

Metrics: manual comparison quality · impact duration accuracy · incident phase correlation · human edit rate · per-summary cost

Outcomes

• Most AI-generated PIR summaries remained unaltered after about two months.
• Canva reduced reliability-engineer toil and improved summary consistency.
• Generated and human-modified summaries are archived for reporting and quality monitoring.

The system supports incident reporting at Canva scale: 150M+ monthly active users, 17B+ designs, and 100+ languages.