Building and Deploying Repository-Specific AI Assistants for GitHub

petercat.ai · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Repository-specific developer support

PeterCat.ai wanted every GitHub repository to have a tailored assistant that could answer repository questions, respond to issues, and review pull requests using current repository context.

Constraints:

Repository code and documentation change frequently, making static fine-tuning a poor fit.
The assistant has to integrate naturally with GitHub issues and PRs.
Long repository text must be chunked, embedded, and retrieved without losing relevance.
Open-source adoption required easy installation through a GitHub App.

Architecture

PeterCat.ai built a GitHub Assistant using LangChain agents, GitHub tools, OpenAI-compatible messages, repository chunking, OpenAI embeddings, and Supabase vector storage. Users authorize a GitHub App, select a repository, and the assistant participates in issues and pull-request reviews with repository-specific retrieval context.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  github-app[GitHub App] --> repository-selector[Repository Selector]
  repository-selector --> repository-indexer[Repository Indexer]
  repository-indexer --> embedding-model[Embedding Model]
  embedding-model --> supabase-vector-database[Supabase Vector Database]
  github-assistant[GitHub Assistant] --> github-tools[GitHub Tools]
  github-assistant --> supabase-vector-database[Supabase Vector Database]
  github-app --> github-assistant[GitHub Assistant]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`GitHub App`	Connects PeterCat to repositories after user authorization.	`Repository Selector`, `GitHub Assistant`
`Repository Selector`	Lets users choose which authorized repository receives an assistant.	`GitHub App`, `Repository Indexer`
`Repository Indexer`	Splits repository text into chunks for embedding.	`Embedding Model`, `Supabase Vector Database`
`Embedding Model`	Vectorizes chunks and user inputs using OpenAI embeddings.	`Repository Indexer`, `Supabase Vector Database`
`Supabase Vector Database`	Stores repository vectors and performs similarity search.	`GitHub Assistant`, `Embedding Model`
`GitHub Assistant`	LangChain agent that answers repository questions, issue discussions, and PR reviews.	`GitHub Tools`, `Supabase Vector Database`, `GitHub App`
`GitHub Tools`	Expose repository metadata, issues, and pull-request review actions to the agent.	`GitHub Assistant`

Data flow

A user authorizes the GitHub App, selects a target repository, repository text is split into chunks and embedded into Supabase, a query is vectorized, similar repository content is retrieved, and the LangChain assistant uses GitHub tools to answer, review, or respond.

“Users simply need to authorize the GitHub App like any other application, and PeterCat automatically retrieves their repository list.”article, stated

“When users interact with GitHub Assistant, we search for relevant content within the repository based on their input.”article, stated

Why this architecture

The team chose RAG over fine-tuning because repository knowledge changes constantly and users need up-to-date answers. LangChain supplied agent orchestration, while Supabase embeddings made the assistant repository-specific without expensive retraining.

“Fine-tuning, which involves combining new data with the existing model through additional training, requires significant computational resources and isn’t ideal for frequently changing codebases.”article, stated

“The RAG approach, on the other hand, not only requires fewer resources but also adapts dynamically to repository updates”article, stated

Models used: ChatGPT · OpenAI embeddings · OpenAI-compatible language model service

Scale

The project gained over 850 GitHub stars within three months of open-source release.
“Within just three months, it garnered over 850 stars, with 178 open source projects adopting our AI assistant.”article, stated
178 open-source projects adopted the assistant within three months.
“Within just three months, it garnered over 850 stars, with 178 open source projects adopting our AI assistant.”article, stated

Alternatives considered (roads not taken)

Use a general-purpose ChatGPT experience.

Rejected because: The goal was a repository-aware assistant that understands codebase context and issue history.

“Not just a general-purpose ChatGPT, but a custom-tailored intelligent assistant that knows every corner of your codebase”article, stated

Fine-tune a model for each repository.

Rejected because: Fine-tuning is computationally expensive and mismatched to frequently changing codebases.

“which involves combining new data with the existing model through additional training, requires significant computational resources and isn’t ideal for frequently changing codebases. The RAG approach, on the other hand, not only requires fewer resources but also adapts dynamically to”article, stated

Manual repository support through maintainers only.

Rejected because: The assistant can participate directly in issue discussions and PR reviews after GitHub App installation.

“the AI assistant is officially integrated into the repository. From that point forward, it actively participates in both Issue discussions and PR reviews.”article, stated

Patterns

`rag-retrieval` (retrieval)

Repository text is chunked, embedded, stored, and retrieved dynamically to ground assistant behavior.

“The RAG approach, on the other hand, not only requires fewer resources but also adapts dynamically to repository updates”article, stated

`tool-use-orchestration` (agent tooling)

LangChain agents use GitHub tools for repository metadata, issue response, and code review workflows.

“I decided to leverage LangChain’s toolkit for building the agent — it seemed like the perfect foundation for what I had in mind.”article, stated

`hybrid-search` (retrieval)

The assistant combines repository text chunking with semantic vector retrieval over embeddings.

“The split text chunks are then vectorized using OpenAI’s embedding model, and the resulting vectors are stored in a Supabase database.”article, stated

`human-in-the-loop-gate` (developer workflow)

The system assists with issues and PRs inside GitHub rather than replacing repository maintainers.

“Code Review: The ability to analyze submitted code, checking for compliance with standards and potential bugs.”article, stated

Tradeoffs

Knowledge adaptation

Chose RAG over repository chunks over Fine-tuning for each repository.

Rationale: RAG uses fewer resources and stays current as code changes.

“not only requires fewer resources but also adapts dynamically to repository updates”article, stated

Repository integration

Chose GitHub App authorization over Out-of-band chatbot use.

Rationale: A GitHub App lets the assistant retrieve repository lists and participate directly in repository workflows.

“Users simply need to authorize the GitHub App like any other application, and PeterCat automatically retrieves their repository list.”article, stated

Gotchas (what bit them)

Frequently changing codebases make fine-tuning operationally brittle.

Root cause: Additional training lags repository updates and consumes significant compute.

Resolution: Use RAG so the assistant adapts dynamically to repository updates.

“Fine-tuning, which involves combining new data with the existing model through additional training, requires significant computational resources”article, stated

Repository-specific assistants need installation that maintainers can understand.

Root cause: Direct repository access requires authorization, repository selection, and ongoing integration.

Resolution: Use a GitHub App as the connection bridge.

“We ultimately decided to develop a GitHub App to serve as the connection bridge.”article, stated

Evaluation

The source primarily reports adoption and capability validation through GitHub workflows, not a formal evaluation harness.

Metrics: GitHub stars · Open-source project adoption · Issue response capability · PR code review capability

“Within just three months, it garnered over 850 stars, with 178 open source projects adopting our AI assistant.”article, stated

Outcomes

PeterCat.ai shipped a repository-specific GitHub assistant.
The project gained over 850 stars and 178 adopting projects within three months.
The assistant supports repository Q&A, issue response, and PR code review.

The strongest reported scale is community adoption: over 850 stars and 178 open-source projects within three months.

“From that point forward, it actively participates in both Issue discussions and PR reviews.”article, stated