Building a Centralized AI-Powered Developer Support System Using RAG

adobe · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal developer support search

Adobe developers needed a centralized way to find answers across wiki pages, software guidelines, troubleshooting guides, and other internal support content without spending time searching across fragmented sources.

Constraints:

• the system must support thousands of internal developers
• developer questions span multiple domains such as CI/CD, environments, and libraries
• retrieval must work across complex multi-domain knowledge sources
• ingestion and content synchronization must scale
• chunking and metadata filtering affect retrieval accuracy

Architecture

Adobe built Unified Support using Amazon Bedrock Knowledge Bases and OpenSearch Serverless vector search. Content is ingested from S3, chunked, embedded with Amazon Titan V2 into 1,024-dimensional vectors, indexed with metadata, retrieved with the same embedding model at query time, filtered by metadata for multi-tenancy, ranked by semantic similarity, and evaluated with document relevance and MRR.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  internal-knowledge-sources[Internal knowledge sources] --> s3-ingestion[S3 ingestion]
  s3-ingestion --> bedrock-knowledge-bases[Bedrock Knowledge Bases]
  developer-query[Developer query] --> bedrock-knowledge-bases
  bedrock-knowledge-bases --> opensearch-vector-engine[OpenSearch vector engine]
  opensearch-vector-engine --> metadata-filter[Metadata filter]
  metadata-filter --> ranked-answer[Ranked answer]
  ranked-answer --> evaluation-framework[Evaluation framework]

Components

Data flow

Source files and metadata are loaded from S3. Bedrock Knowledge Bases chunks and vectorizes the data with Amazon Titan V2, stores vectors in OpenSearch Serverless, embeds developer questions with the same model, retrieves and ranks relevant chunks, and presents top-ranked support information to the developer.

Why this architecture

A managed RAG stack let Adobe improve retrieval precision while keeping ingestion, chunking, vectorization, metadata filtering, and evaluation configurable. Metadata filtering matters because developer support questions span domains and need narrower retrieval than plain semantic similarity alone.

Models used: Amazon Titan V2 embedding model · large language model judge

Scale

• Unified Support is designed to help thousands of Adobe internal developers.
  “This led to the initiative Unified Support, designed to help thousands of the company’s internal developers get immediate answers to questions from a centralized place and reduce time and cost spent on developer support.”article, stated
• Amazon Titan V2 produces 1,024-dimensional chunk vectors.
  “* Vectorization – Each chunk is passed through an embedding model (in this case, Amazon Titan V2 on Amazon Bedrock) creating a 1,024-dimension numerical vector.”article, stated
• Adobe tested context sizes from 1 to 5 retrieved chunks.
  “To assess performance, we measured document relevance and MRR across different context sizes, ranging from 1–5.”article, stated
• The final solution achieved a 20 percent increase in retrieval accuracy.
  “This solution dramatically improved their developer support system, resulting in a 20% increase in retrieval accuracy.”article, stated

Alternatives considered (roads not taken)

Use the initial prototype unchanged

Rejected because: The prototype exposed scale, onboarding, synchronization, and infrastructure-efficiency gaps.

Rely on semantic similarity without metadata filtering

Rejected because: Developers need a limited relevant subset when questions span multiple domains.

Use more complex chunking by default

Rejected because: Fixed-size 400-token chunks proved simplest and most effective in their experiments.

Patterns

rag-retrieval (retrieval)

Unified Support uses Bedrock Knowledge Bases and vector retrieval over internal developer documents.

retrieval-ranking (retrieval)

Retrieved documents are ranked by semantic similarity and evaluated with MRR.

llm-as-judge (evaluation)

Document relevance uses an LLM as an impartial judge for retrieved chunks.

eval-driven-development (evaluation)

Chunking and retrieval choices were compared against ground-truth data using document relevance and MRR.

Tradeoffs

Chunking strategy

Chose fixed-size 400-token chunks over larger fixed, semantic, or hierarchical chunks for this use case.

Rationale: It was the simplest and consistently highest-accuracy option in Adobe's tests.

Retrieval scoping

Chose metadata filtering plus vector similarity over unfiltered vector search.

Rationale: Filtering gives developers semantically relevant information within a defined subset.

Gotchas (what bit them)

Raw semantic retrieval is not enough for multi-domain support.

Root cause: Developer questions span CI/CD, project environments, libraries, and other domains.

Resolution: Use metadata files and filtering to narrow retrieval scope.

Chunking strategy materially changes retrieval quality.

Root cause: Different chunk sizes and hierarchical/semantic approaches change what context is retrieved.

Resolution: Experiment with chunking strategies and evaluate with MRR/document relevance.

Evaluation

Adobe evaluated retrieval with document relevance, LLM-as-judge scores, MRR, multiple top-k context sizes, and different chunking strategies.

Metrics: 20% retrieval accuracy increase · document relevance score 1-10 · MRR · top-k 1-5 · 400-token chunks with 20% overlap

Outcomes

• Adobe achieved a 20 percent increase in document retrieval accuracy.
• The system provides a scalable foundation for developer support across Adobe teams.

The solution is aimed at thousands of Adobe internal developers and uses 1,024-dimensional embeddings with configurable retrieval top-k from 1 to 5.