AI-Powered Similar Issues Detection for Project Management

linear · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Issue tracking and project-management backlog hygiene

Linear needed to reduce duplicate issues across large team workflows before duplicate work accumulated in backlogs, triage inboxes, and support integrations.

Constraints:

• duplicate issues are a persistent problem in large teams with long backlogs
• keyword and property matching could not capture semantic similarity between issue descriptions
• the feature needed to work inside issue creation, Triage, and support integrations
• vector search had to handle tens of millions of existing issues
• the engineering team wanted maintainable infrastructure rather than high-ops specialized databases

Architecture

Linear built Similar Issues around LLM-generated vector embeddings and pgvector-backed cosine similarity search. Issue text is concatenated and converted to embeddings through an API call, stored with workspace/team/status metadata in PostgreSQL on Google Cloud, partitioned by workspace ID across hundreds of partitions, and indexed per partition. The feature surfaces semantically similar issues during issue creation, in the Triage inbox, and directly inside support integrations such as Intercom so support and engineering teams can detect duplicates earlier.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  issue-content-extractor["issue-content-extractor"] --> embedding-api["embedding-api"]
  backfill-task-runner["backfill-task-runner"] --> issue-content-extractor
  embedding-api --> pgvector-embedding-store["pgvector-embedding-store"]
  pgvector-embedding-store --> similarity-search-service["similarity-search-service"]
  similarity-search-service --> issue-creation-surface["issue-creation-surface"]
  similarity-search-service --> triage-inbox-surface["triage-inbox-surface"]
  similarity-search-service --> support-integration-surface["support-integration-surface"]

Components

Data flow

For backfill, Linear iterates through all existing issues, concatenates textual content, calls an embedding API, and stores the resulting vector with workspace, team, and status metadata. At query time, a new issue or incoming support message is embedded and compared against the relevant workspace/team partitions with cosine similarity. Results are shown in the issue composer, Triage, or support integration, including related issue status so teams can mark duplicates or respond without switching tools.

Why this architecture

Linear used embeddings because duplicate issues often use different words for the same concept. It chose pgvector on PostgreSQL after vector-specific databases introduced downtime, cost, or operational complexity; PostgreSQL was already maintainable for a small team and still gave reasonable cosine-query response times. Partitioning by workspace made tens-of-millions-scale indexing workable while preserving the product's natural data boundary.

Models used: LLM embeddings · PostgreSQL · pgvector · Google Cloud Platform · Kubernetes · cosine similarity

Scale

• The feature was rolled out to all Linear workspaces.
  “A few months ago, we rolled out our Similar Issues matching feature to all Linear workspaces.”article, stated
• Linear had tens of millions of issues to embed and index from the start.
  “we have tens of millions of issues from the get-go.”article, stated
• Embeddings were partitioned across several hundred workspace partitions.
  “partitioning the embeddings table by workspace ID across several hundred partitions”article, stated
• The backfill ran jobs in parallel on a Kubernetes cluster.
  “using our task runners to process jobs in parallel on a kubernetes cluster”article, stated
• The database server needed hundreds of GB of memory while index strategy was iterated.
  “even with providing the database server with hundreds of GB of memory.”article, stated

Alternatives considered (roads not taken)

property or keyword-based duplicate detection

Rejected because: It could not match semantically similar issue language such as bug, problem, and broken.

vector-specific databases

Rejected because: They introduced downtime to scale, high cost, or additional operations complexity.

store embeddings only after issue creation

Rejected because: Linear wanted to catch duplicates earlier, including directly beside incoming customer emails.

Patterns

rag-retrieval (knowledge-grounding)

Linear uses embeddings and vector similarity over existing issues to retrieve semantically related records.

batch-offline-processing (operations)

Existing task runners backfilled embeddings for all historical issues in parallel on Kubernetes.

retrieval-ranking (retrieval)

Linear indexes issue embeddings and retrieves nearest matches with cosine similarity for duplicate detection.

Tradeoffs

Use PostgreSQL with pgvector for vector storage.

Chose known maintainable database plus pgvector over specialized vector database providers.

Rationale: The small team preferred operational confidence and reasonable query times over new database operational risk.

Start with embeddings stored as blobs during proof of concept.

Chose primary datastore blob storage for quick iteration over choosing a long-term vector store before validating the idea.

Rationale: The team could evaluate feature behavior and storage options without upfront database migration work.

Gotchas (what bit them)

Naive vector search timed out at product scale.

Root cause: Tens of millions of issue embeddings and no suitable indexing made cosine similarity queries too slow.

Resolution: Partition embeddings by workspace ID across hundreds of partitions and index each partition separately.

Embedding blobs can accidentally bloat ordinary queries.

Root cause: Vectors are much larger than typical database fields.

Resolution: Avoid selecting the vector/blob column unless the query needs it.

Duplicate detection after issue creation was too late for support workflows.

Root cause: The issue already existed by the time Triage saw it.

Resolution: Move similarity detection to support integrations beside incoming customer emails.

Evaluation

Linear evaluated the feature through production rollout to all workspaces, response-time/indexing iteration, customer-experience usage in Intercom, community feedback, and future search-quality improvements.

Metrics: all workspaces rollout · cosine queries timing out or multiple seconds before indexing · tens of millions of issues indexed · support issue consolidation feedback

Outcomes

• similar issues are shown during issue creation
• Triage can surface existing duplicates for external-source issues
• support teams can see related Linear issues directly in Intercom
• customer experience teams spent less time manually aggregating messages
• Linear identified follow-up improvements for integrations, labels, templates, and main search

The main scale facts are all-workspace rollout, tens of millions of issues, several hundred workspace partitions, and Kubernetes-parallel backfill.