GenAI-Powered Dual-Loop System for Automated Documentation Management

intuit · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: technical documentation management and knowledge discovery

Intuit needed to improve technical documentation so engineers could find current, well-structured, audience-appropriate information quickly instead of searching through inconsistent knowledge-base content.

Constraints:

• documentation quality varied depending on who wrote it
• retrieved documentation might not be up to date
• documents were not necessarily structured for parsing relevant information
• documentation was not always written with target audiences and use cases in mind
• the system needed to improve both knowledge-base content and retrieval/answering quality over time

Architecture

Intuit built a GenAI-powered dual-loop documentation pipeline. The inner loop improves each document before inclusion by analyzing structure/completeness/comprehension against a rubric, rewriting and style-normalizing content, adding semantic context and query links, and augmenting documents with relevant retrieved information. The outer loop embeds improved documents, retrieves relevant chunks with semantic search, and synthesizes answers to user queries. When answers fail, the system updates the Search Plugin and the base documents so both retrieval behavior and content quality improve.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  knowledge-base["Knowledge base"] --> document-analyzer["Document Analyzer"]
  document-analyzer --> document-improvement-plugin["Document Improvement Plugin"]
  document-improvement-plugin --> document-style-guide-plugin["Document Style Guide Plugin"]
  document-style-guide-plugin --> document-discoverability-plugin["Document Discoverability Plugin"]
  document-discoverability-plugin --> document-augmentation-plugin["Document Augmentation Plugin"]
  document-augmentation-plugin --> knowledge-base
  knowledge-base --> embedding-plugin["Embedding Plugin"]
  embedding-plugin --> search-plugin["Search Plugin"]
  search-plugin --> answer-plugin["Answer Plugin"]
  search-plugin --> knowledge-base

Components

Data flow

A document enters the inner loop, where analyzer, improvement, style-guide, discoverability, and augmentation plugins score, rewrite, normalize, contextualize, and refresh it. Improved documents are embedded for similarity search. At query time, the outer loop uses embeddings and semantic search to retrieve relevant chunks, then the answer plugin synthesizes a response. Failed answers feed back into search-parameter updates and missing-information updates in the base documents.

Why this architecture

The dual-loop design attacks documentation quality before retrieval and then improves retrieval behavior from query outcomes. This differs from traditional indexing: documents are actively improved, semantically contextualized, and refreshed before being embedded, while failed queries update both search parameters and missing source content.

Models used: GenAI · LLMs · vector stores · RAG · embeddings · semantic search

Scale

• The inner loop uses five document-improvement plugins.
  “Each plugin looks at a different aspect of the documentation and makes improvements as needed:”article, stated
• The outer loop is described as three plugins: embedding, search, and answer.
  “Embedding Plugin — This tool creates vector representations of documents”article, stated

  “Answer Plugin — This plugin brings the pieces together by synthesizing the information provided by the other two plugins”article, stated
• The approach is intended for broad technical documentation and future content collections beyond software documentation.
  “this type of pipeline can be applied across other collections of content to improve our ability to answer user questions efficiently and accurately across a wide range of domains”article, stated

Alternatives considered (roads not taken)

Traditional indexing when content is added

Rejected because: Indexing alone does not improve document structure, freshness, discoverability, or missing information before retrieval.

Rely only on retrieval-time context

Rejected because: The system also updates base documents with missing information when answers fail, making the corpus itself better rather than only tuning search.

Treat LLMs and vector stores as standalone technologies

Rejected because: Intuit argues the unique value comes from orchestrating them in a pipeline, not from the individual technologies alone.

Patterns

rag-retrieval (knowledge discovery)

The Document Augmentation Plugin uses RAG to refresh documents, while the outer loop retrieves chunks and synthesizes answers from multiple knowledge-base sources.

data-quality-loop (knowledge base quality)

The inner loop scores, rewrites, style-normalizes, contextualizes, and augments documents before they enter or remain in the knowledge base.

retrieval-ranking (search)

The Search Plugin uses semantic similarity to scan likely content areas and choose the most relevant chunks for the current query context.

prompt-chaining (pipeline orchestration)

The system chains specialized GenAI plugins for analysis, improvement, style, discoverability, augmentation, embedding, search, and answer synthesis.

eval-driven-development (quality gates)

The Document Analyzer compares structure, completeness, and comprehension against a custom rubric, using a threshold to decide whether a document proceeds through improvement plugins.

Tradeoffs

Where to improve retrieval quality

Chose pre-process and improve documents before indexing over only index documents as-is.

Rationale: Improving structure, style, freshness, and semantic context before retrieval creates a higher-quality knowledge base for downstream answers.

How to handle failed queries

Chose update search parameters and base documents over only retry retrieval or tune ranking.

Rationale: The source frames failed answers as evidence that both retrieval behavior and missing knowledge may need correction.

How to structure the system

Chose specialized inner and outer loops over one monolithic retrieval pipeline.

Rationale: Separating document improvement from query-time retrieval lets the system improve both the corpus and the answer path.

Gotchas (what bit them)

Documentation quality varies by author.

Root cause: Different writers produce documents with different structure, completeness, comprehension level, and style.

Resolution: Run documents through analyzer, improvement, and style-guide plugins before inclusion.

Search can retrieve stale or poorly structured information.

Root cause: Documents may not be up to date or written in a way that makes relevant information easy to parse.

Resolution: Use the inner loop to keep documents current and structured for extraction before the outer retrieval loop uses them.

Documents may be invisible to the outer retrieval system.

Root cause: Content lacks semantic context or links to the user queries it should answer.

Resolution: Add semantic context and relevant-query links through the Document Discoverability Plugin.

Failed answers require both retrieval and source-content fixes.

Root cause: A failed answer can mean search did not find the right material, but it can also mean the base document lacks missing information.

Resolution: Update both the Search Plugin and the base documents when the system cannot answer successfully.

Evaluation

The article does not report a formal benchmark, but the inner loop includes a custom-rubric document score and threshold gate, and failed query handling serves as operational feedback for search and content updates.

Metrics: document structure score · document completeness score · ease of comprehension score · required threshold for inclusion · failed query events · answer accuracy and comprehensiveness

Outcomes

• Intuit built a dual-loop GenAI pipeline for document management and knowledge discovery.
• The system improves document quality, freshness, style consistency, discoverability, and semantic context before retrieval.
• The outer loop retrieves and synthesizes answers from multiple knowledge-base sources.
• Failed answers update search behavior and missing source information for future queries.
• The approach is positioned for future application to content collections beyond software documentation.

The source does not provide deployment counts, latency, cost, adoption, or accuracy metrics. Its concrete scope is architectural: five inner-loop document plugins, three outer-loop retrieval/answer plugins, and an intended expansion across knowledge domains beyond software documentation.