Building an AI-Powered Browser Extension for Product Documentation with RAG and Chain-of-Thought

reforge · Education · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Product-management documentation assistance

Reforge needed a browser extension that could help product teams draft and improve PRDs and related documents inside their existing work tools while grounding suggestions in Reforge material instead of generic LLM advice.

Constraints:

• The extension had to work inside popular document tools and browsers rather than forcing users into a separate product.
• Simple RAG produced inconsistent outputs and irrelevant recommendations.
• Document type mattered because a PRD and a job description require different suggestions.
• The team had to manage privacy, model cost, analytics, and model/provider flexibility.

Architecture

Reforge built a browser extension that opens alongside tools such as Notion, Google Docs, Confluence, and Coda. It retrieves the user's document and prompt, classifies the document type, retrieves relevant Reforge content from embeddings/Pinecone, asks the LLM in parallel for multiple focused suggestions, streams the suggestions back to the extension, and uses analytics plus Adaline logging to evaluate prompts and outputs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  browser-extension[Browser Extension] --> work-document[Work Document]
  browser-extension --> reforge-backend[Reforge Backend]
  reforge-backend --> document-classifier[Document Classifier]
  document-classifier --> llm-suggestion-generator[LLM Suggestion Generator]
  reforge-backend --> vector-retrieval[Vector Retrieval]
  vector-retrieval --> reforge-corpus[Reforge Corpus]
  reforge-backend --> llm-suggestion-generator[LLM Suggestion Generator]
  reforge-backend --> observability-stack[Observability Stack]

Components

Data flow

The user opens the Reforge side panel in a work document. The system reads the document and prompt, classifies the document type, embeds and searches for relevant Reforge content, retrieves metadata, sends focused prompts and context to an LLM in parallel, then streams three suggestions back to the browser extension.

Why this architecture

Reforge chose an in-workflow browser extension because users already write PRDs in tools like Notion and Google Docs. It moved beyond simple similarity RAG by classifying document type, narrowing context, asking document-specific questions, and grounding suggestions in Reforge content so outputs were more relevant than generic ChatGPT templates.

Models used: OpenAI · OpenAI embeddings · Gemini Pro 1.5 · Anthropic models

Scale

• The extension works across popular browsers and work tools including Chrome, Arc, Google Docs, Notion, Confluence, and Coda.
  “The extension works in most popular browsers (Chrome, Arc, etc.,) and it integrates into popular work tools like Google Docs, Notion, Confluence, and Coda”article, stated
• The system returns three suggestions for each document.
  “* We generated three different suggestions for each document. * For every suggestion, we ran embeddings on the suggestion and the document content.”article, stated
• Initial tracking found less than 50% of responses had no Reforge references.
  “However, once we started tracking this, we found that less than 50% of responses contained no Reforge references. Interestingly, when we upgraded to version 4.0, we began to see a significant increase in references.”article, stated

Alternatives considered (roads not taken)

Simple retrieve-and-generate RAG over document content.

Rejected because: It produced inconsistent outputs and did not solve the core customer problem reliably enough.

Generic ChatGPT or PRD-template assistance.

Rejected because: Reforge wanted content grounded in its own frameworks and embedded in the user's workflow.

Commit to one model provider.

Rejected because: The team wanted to compare outputs across OpenAI, Gemini, and Anthropic models.

Patterns

rag-retrieval (retrieval)

The extension embeds the user's document and prompt, searches Reforge material, and sends retrieved context to the LLM.

prompt-chaining (workflow)

Reforge classifies document type, adapts questions, and then runs focused RAG rather than asking one generic prompt.

eval-driven-development (quality)

The team used distribution of course recommendations and reference rates as proxies for model intelligence and relevance.

observability-tracing (operations)

Adaline stores prompts and logs so the team can inspect LLM inputs and outputs.

model-routing (model-strategy)

Reforge kept model choice open and framed provider comparison as part of the operating model.

Tradeoffs

Product surface

Chose Browser extension embedded in existing tools over Standalone PRD-generation product.

Rationale: The extension reduces copy-paste and tab switching by appearing where users already work.

Retrieval architecture

Chose Classified, focused RAG over Simple similarity-based RAG.

Rationale: Document-aware retrieval made suggestions more relevant to PRDs, job descriptions, and other document types.

Early cost posture

Chose Build for value first while modeling cost over Optimize cost before proving the workflow.

Rationale: The team treated cost as important but secondary to finding product value early.

Gotchas (what bit them)

Simple RAG was not reliably useful enough.

Root cause: Similarity search alone did not tailor suggestions to document type or user task.

Resolution: Add document classification, focused questions, and more specific Reforge content.

The retrieval corpus was incomplete.

Root cause: Not all Reforge program content was stored in Pinecone.

Resolution: Fix corpus coverage and track recommendation distribution as a model-intelligence proxy.

Model outputs needed visibility beyond final user-facing suggestions.

Root cause: Prompt and response behavior was hard to debug without a logging and experiment layer.

Resolution: Use Adaline for prompt storage, logs, and output evaluation.

Evaluation

Reforge evaluated the extension through beta feedback, course-recommendation distributions, reference-rate tracking, prompt/output logging in Adaline, and product analytics in MetaBase, Segment, Amplitude, and Snowflake.

Metrics: Distribution of course recommendations · Share of responses with no Reforge references · Prompt and output logs · User interaction analytics · Average cost per user

Outcomes

• The extension supports product-writing workflows inside common browser-based tools.
• The team moved from simple RAG to classified, focused RAG after reliability problems.
• Reference tracking showed fewer than half of responses lacked Reforge references, with later model upgrades increasing references.

The source reports browser/tool coverage, three suggestions per document, less-than-50% no-reference tracking, and cost modeling per user.