AI-Powered Incident Investigation for Payment Infrastructure

razorpay · Finance · 2026 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Payment infrastructure incident response

Razorpay needed to reduce the investigation phase of production incidents, where on-call engineers manually correlated evidence across disconnected observability and infrastructure systems before knowing what to fix.

Constraints:

• payment failures create direct customer and revenue risk
• incident evidence spans multiple disconnected systems
• investigation quality varies by on-call engineer experience
• the system was still in shadow mode and not yet the primary decision-maker
• AI investigations need citations, confidence scores, and human review before remediation
• knowledge bases must stay current without polluting future investigations

Architecture

Razorpay built the Oncall Agent as a LangGraph multi-agent incident-investigation workflow. Zenduty alerts go to a Supervisor Agent, which queries two RAG stores for service architecture/dependencies and alert-specific runbooks. The supervisor creates an investigation plan and dispatches specialist agents in parallel across Kubernetes, Coralogix logs, PromQL metrics, AWS infrastructure, deployments, and database health. Specialist findings are written as structured evidence with confidence and supporting data. The supervisor correlates evidence into an incident timeline, scores hypotheses, and sends Slack output with citations, confidence, root-cause assessment, and recommended actions for the human on-call engineer.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  zenduty-alert-source["zenduty-alert-source"] --> supervisor-agent["supervisor-agent"]
  supervisor-agent --> architecture-rag["architecture-rag"]
  supervisor-agent --> runbook-rag["runbook-rag"]
  supervisor-agent --> kubernetes-agent["kubernetes-agent"]
  supervisor-agent --> coralogix-agent["coralogix-agent"]
  supervisor-agent --> promql-agent["promql-agent"]
  supervisor-agent --> aws-agent["aws-agent"]
  supervisor-agent --> deployment-history-agent["deployment-history-agent"]
  supervisor-agent --> database-health-agent["database-health-agent"]
  kubernetes-agent --> evidence-memory["evidence-memory"]
  coralogix-agent --> evidence-memory
  promql-agent --> evidence-memory
  aws-agent --> evidence-memory
  deployment-history-agent --> evidence-memory
  database-health-agent --> evidence-memory
  evidence-memory --> supervisor-agent
  supervisor-agent --> slack-output["slack-output"]
  slack-output --> on-call-engineer["on-call-engineer"]
  evidence-memory --> continuous-learning-loop["continuous-learning-loop"]
  continuous-learning-loop --> architecture-rag
  continuous-learning-loop --> runbook-rag

Components

Data flow

An alert from Zenduty triggers the supervisor. The supervisor retrieves architecture/dependency context and alert-specific runbook context from separate RAG systems, then sends parallel tasks to specialist agents for Kubernetes health, logs, metrics, cloud infrastructure, deployment history, and database checks. Each specialist returns evidence, confidence, and supporting observations into memory. The supervisor orders evidence by timestamp, scores root-cause hypotheses, selects the most likely cause, and posts the completed investigation to Slack for human review. Investigated cases are written back into RAG knowledge stores to improve future investigations.

Why this architecture

The architecture mirrors an expert incident room: a supervisor forms the plan, specialist agents inspect separate systems in parallel, evidence is stored with confidence and citations, and a human engineer keeps remediation authority. Dual RAG grounds the workflow in Razorpay's specific services and runbooks instead of generic LLM advice, while shadow mode lets Razorpay validate accuracy before production reliance.

Models used: large language models · LangGraph

Scale

• Engineers previously spent 20-40 minutes per incident investigating manually.
  “on-call engineers were spending 20-40 minutes per incident manually correlating information across six disconnected observability and infrastructure systems”summary, stated
• The motivating example describes 32 minutes of investigation and nearly 40 minutes of payment failures.
  “a 3 AM production alert where an engineer spent 32 minutes investigating before identifying a bad deployment as the root cause”summary, stated

  “payment failures had already impacted customers for nearly 40 minutes.”summary, stated
• Razorpay handled 15-20 incidents weekly, costing 6-8 hours of repetitive investigation time.
  “With 15-20 incidents occurring weekly, this translated to 6-8 hours of repetitive investigative work.”summary, stated
• Specialist agents complete checks in about 5-8 seconds each.
  “Each specialist agent is domain-focused and completes its specific checks within 5-8 seconds”summary, stated
• After three months in shadow mode, MTTI reportedly fell 80% from 30 minutes to 90 seconds.
  “The MTTI reduction of 80% represents investigations completing in 90 seconds instead of 30 minutes”summary, stated
• MTTR reportedly improved by 50-60%.
  “The 50-60% improvement in MTTR (Mean Time to Resolve) indicates that faster investigation directly accelerates resolution”summary, stated

Alternatives considered (roads not taken)

Manual cross-tool investigation by on-call engineers

Rejected because: It consumed 20-40 minutes per incident, varied by engineer experience, and repeated the same pattern-based evidence gathering.

One general-purpose incident agent

Rejected because: Razorpay used a supervisor plus specialist agents to mirror how expert teams divide investigative work across domains.

Sequential specialist checks

Rejected because: Parallel specialist execution better matches team investigation and supports the 90-second target.

Immediate production authority for the AI

Rejected because: The system stayed in shadow mode while Razorpay validated accuracy across hundreds of incidents before full rollout.

Patterns

multi-agent-orchestration (architecture)

A supervisor agent plans the investigation and dispatches domain specialist agents in parallel for Kubernetes, logs, metrics, cloud infrastructure, deployment history, and database health.

rag-retrieval (retrieval)

The supervisor retrieves application/dependency knowledge and alert-specific diagnostic runbooks before deciding what specialists should inspect.

tool-use-orchestration (operations)

Specialists query operational systems including Grafana, Coralogix, Kubernetes, AWS, deployment history, and database health, then synthesize results for Slack.

human-in-the-loop-gate (control)

The AI performs investigation and recommends actions, but the on-call engineer reviews the analysis and decides remediation.

confidence-thresholding (evidence)

Specialist findings include confidence levels, and the supervisor scores multiple hypotheses before choosing a likely root cause.

agent-memory (state)

The Memory component stores structured investigation evidence, timestamps, confidence, and supporting data for correlation.

data-quality-loop (learning)

Investigated cases are stored back into RAG systems so future investigations benefit from accumulated incident patterns.

Tradeoffs

Automate investigation rather than remediation

Chose AI evidence gathering and root-cause recommendation over AI-driven production fixes.

Rationale: The repetitive, pattern-based investigation work was the bottleneck, while remediation still requires human judgment in payment infrastructure.

Use parallel specialist agents

Chose domain-focused agents running concurrently over one sequential general investigator.

Rationale: Parallel checks improve latency and mirror how expert teams divide incident work.

Use dual RAG stores

Chose separate architecture/dependency and alert-runbook knowledge bases over generic LLM incident reasoning.

Rationale: Grounding the supervisor in Razorpay-specific infrastructure and known diagnostic playbooks reduces generic or irrelevant investigation paths.

Gotchas (what bit them)

Shadow-mode metrics do not prove full production reliability.

Root cause: As of the source, the agent investigates alongside humans and has not yet become the primary incident-response tool.

Resolution: Razorpay targets 80%+ accuracy and plans production rollout only after validation across hundreds of incidents.

Continuous-learning RAG can accumulate bad knowledge if not curated.

Root cause: Writing every investigated case back to RAG improves recall but can also preserve incorrect analyses or outdated infrastructure assumptions.

Resolution: The source identifies knowledge-base pollution and maintenance as an operational challenge, but does not describe the concrete mitigation.

AI incident summaries can be dangerous if confidently wrong.

Root cause: False positives or misleading root-cause claims can delay remediation in payment infrastructure.

Resolution: The design keeps human review and shadow-mode validation before production authority.

Evaluation

Razorpay evaluated the agent in shadow mode over three months, tracking MTTI, MTTR, engineering hours saved, consistency across engineer experience levels, and a target of 80%+ accuracy before production rollout.

Metrics: Mean Time to Investigate · Mean Time to Resolve · weekly engineering hours saved · investigation accuracy target · specialist-agent execution time · investigation consistency

Outcomes

• MTTI reportedly dropped 80%, from 30 minutes to 90 seconds.
• MTTR reportedly improved by 50-60%.
• Razorpay reports 6-8 hours of engineering time saved weekly.
• Junior engineers receive more consistent analysis quality.
• SREs can shift time from repetitive investigation toward proactive resilience work.

The source reports 15-20 incidents per week, 20-40 minutes of prior investigation time per incident, a 32-minute example investigation, specialist checks in 5-8 seconds, three months of shadow mode, 80% MTTI reduction to 90 seconds, 50-60% MTTR improvement, and 6-8 engineering hours saved weekly.