RAG-Enhanced Code Review Bot Using Historical Incident Data

paypay · Finance · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Code review risk detection and incident knowledge reuse

PayPay needed to scale code review across a growing fintech engineering organization by systematically applying lessons from historical incidents and postmortems to new pull requests, instead of relying on individual reviewer memory and ad hoc knowledge sharing.

Constraints:

• Incident knowledge was siloed in projects, people, and postmortem documents.
• Risk assessment varied across reviewers depending on their historical context.
• The system needed to comment automatically on pull requests at very low operating cost.
• Retrieval false positives and embedding quality remained limitations requiring future reranking and model upgrades.

Architecture

PayPay built GBB RiskBot, a RAG-enhanced code review assistant. Historical incidents and postmortems are embedded with OpenAI embeddings and stored in ChromaDB. When a developer opens a pull request, the bot embeds or analyzes the code-change context, runs semantic similarity search against the incident vector database, and uses GPT-4o-mini to generate a contextual review comment that highlights relevant past risks. Planned improvements include upgrading embeddings from ada to text-embedding-3-large and adding a rerank step after cosine similarity search.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  historical-incident-corpus["Historical incident corpus"] --> embedding-pipeline["Embedding pipeline"]
  pull-request-event["Pull request event"] --> embedding-pipeline["Embedding pipeline"]
  pull-request-event["Pull request event"] --> gbb-riskbot["GBB RiskBot"]
  embedding-pipeline["Embedding pipeline"] --> incident-vector-database["Incident vector database"]
  incident-vector-database["Incident vector database"] --> semantic-similarity-search["Semantic similarity search"]
  semantic-similarity-search["Semantic similarity search"] --> gbb-riskbot["GBB RiskBot"]
  semantic-similarity-search["Semantic similarity search"] --> future-reranker["Future reranker"]
  future-reranker["Future reranker"] --> gpt-4o-mini-commenter["GPT-4o-mini commenter"]
  gbb-riskbot["GBB RiskBot"] --> gpt-4o-mini-commenter["GPT-4o-mini commenter"]
  gpt-4o-mini-commenter["GPT-4o-mini commenter"] --> developer-review-thread["Developer review thread"]
  gbb-riskbot["GBB RiskBot"] --> developer-review-thread["Developer review thread"]

Components

Data flow

Incident and postmortem records are collected into a historical incident knowledge base, embedded with OpenAI embeddings, and indexed in ChromaDB. A new pull request triggers GBB RiskBot. The bot performs semantic similarity search between the PR context and historical incidents, retrieves relevant incident examples, and sends the retrieved context to GPT-4o-mini. The model generates a pull-request comment explaining the risk and incident precedent, allowing developers and reviewers to apply organizational incident lessons during code review.

Why this architecture

PayPay used RAG because the useful knowledge already existed in postmortems and incident records, but was not systematically available during review. Semantic search makes historical incidents discoverable from a pull request, while GPT-4o-mini turns retrieved incident context into a developer-facing comment. The low-cost model choice makes the workflow practical across repositories, and planned reranking acknowledges that first-stage similarity search can over-retrieve.

Models used: OpenAI embeddings · ChromaDB · GPT-4o-mini · text-embedding-3-large planned

Scale

• PayPay reports about $0.59 monthly cost for the bot workload.
  “approximately $0.59 USD monthly”summary, stated
• The reported workload covered more than 380 analyses.
  “380+ analyses across 12 repositories”summary, stated
• The reported workload spanned 12 repositories.
  “380+ analyses across 12 repositories”summary, stated
• PayPay launched in 2018 and grew substantially over seven years.
  “PayPay, a financial technology company launched in 2018, has experienced substantial growth over seven years”summary, stated

Alternatives considered (roads not taken)

Manual incident knowledge sharing

Rejected because: Lessons remained siloed and were inconsistently applied across teams and reviewers.

Rely only on current cosine similarity search

Rejected because: PayPay plans a rerank step to improve search accuracy and reduce false positives.

Keep the ada embedding model

Rejected because: The roadmap calls for upgrading to text-embedding-3-large to improve training/retrieval data quality.

Use only RAG long-term

Rejected because: The roadmap includes experiments with CAG and mem0, implying RAG may not be the final architecture for all needs.

Patterns

rag-retrieval (retrieval)

The bot retrieves semantically similar historical incidents and uses them to ground pull-request comments.

retrieval-ranking (retrieval)

PayPay uses semantic similarity today and plans reranking after cosine search to improve relevance.

agent-memory (memory)

Historical incident data becomes organizational memory that is automatically applied during future code review.

cost-aware-routing (cost)

The system uses GPT-4o-mini and reports very low monthly operating cost for hundreds of analyses.

tool-use-orchestration (architecture)

GBB RiskBot orchestrates pull-request triggers, embeddings, vector search, and LLM comment generation.

Tradeoffs

Use historical incidents as review context

Chose RAG over postmortem data over Human-only recall during code review.

Rationale: The bot applies lessons across repositories even when reviewers lack the same incident history.

Use GPT-4o-mini for comments

Chose Low-cost generated review comments over Heavier model calls for each PR analysis.

Rationale: The reported monthly cost suggests the model choice supported broad usage at low cost.

Add reranking later

Chose Initial ChromaDB similarity search first over More complex two-stage retrieval from the start.

Rationale: The current system delivered value, while the roadmap acknowledges precision improvements for false positives.

Gotchas (what bit them)

Incident lessons remain trapped without workflow integration.

Root cause: Postmortem knowledge is manually shared and depends on individual reviewers remembering relevant history.

Resolution: Index historical incidents and retrieve them automatically during pull-request review.

First-stage vector retrieval may create false positives.

Root cause: Cosine similarity alone may retrieve incidents that are semantically close but not actually relevant.

Resolution: Add a rerank step after similarity search to assess relevance more precisely.

System quality depends on incident-report quality.

Root cause: Poorly documented or inconsistent postmortems limit retrieval and comment usefulness.

Resolution: Improve training data and embedding quality; the roadmap includes upgrading embeddings.

Evaluation

The case reports operational scale and cost, but leaves open questions about generated-comment quality evaluation. The roadmap focuses on improving embedding quality and retrieval precision through text-embedding-3-large and reranking.

Metrics: monthly cost · analysis count · repository count · search accuracy · false positive rate · developer reaction

Outcomes

• GBB RiskBot automatically analyzes pull requests against historical incident data.
• The system generated risk comments with GPT-4o-mini using retrieved incident context.
• The reported operating cost was about $0.59 per month for more than 380 analyses across 12 repositories.
• The bot helps democratize incident knowledge across reviewers and teams.

Concrete anchors include $0.59 monthly cost, 380+ analyses, 12 repositories, PayPay's 2018 launch and seven years of growth, OpenAI embeddings, ChromaDB, and GPT-4o-mini.