From Simple RAG to Multi-Agent Architecture for Document Data Extraction

box · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise document data extraction

Box's initial LLM metadata-extraction system worked on simple documents but broke when enterprise customers supplied long documents, poor OCR, multilingual content, hundreds of fields, and confidence-score expectations.

Constraints:

• Box serves large enterprises with very large unstructured-data stores
• documents can exceed the context window
• OCR can fail on crossed-out text, poor scans, or multiple languages
• customers may request hundreds of fields from one document
• generative AI does not naturally produce traditional confidence scores

Architecture

Box rearchitected metadata extraction from a single preprocess-LLM-postprocess path into a multi-agent workflow. Specialized sub-agents group related fields, choose extraction strategies, inspect page images as well as OCR where useful, run quality feedback, and retry with different techniques or model voting when fields look wrong.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  enterprise-documents[Enterprise documents] --> field-grouping-agent[Field grouping agent]
  field-grouping-agent --> extraction-agent[Extraction agent]
  extraction-agent --> quality-feedback-loop[Quality feedback loop]
  quality-feedback-loop --> model-voting[Model voting]
  model-voting --> structured-extracted-data[Structured extracted data]

Components

Data flow

A document and requested fields enter the workflow. Rather than one prompt over preprocessed text, the system breaks the extraction problem into sub-agent tasks: field grouping, data extraction over OCR and page images, quality judging, feedback, and retries using alternative models or techniques.

Why this architecture

The multi-agent abstraction let Box isolate each failure mode instead of piling more preprocessing into a single prompt. It made the system easier to evolve as customers brought new document types and extraction requirements.

Models used: large language model · LLM-as-judge · OCR models

Scale

• Box reports over 115,000 customer companies and tens of millions of users.
  “We have over 115,000 companies, tens of millions of users and our customers have given us uh entrusted us with over an exabyte of their data.”transcript, stated
• Customers have entrusted Box with over an exabyte of data.
  “our customers have given us uh entrusted us with over an exabyte of their data. And in many of these companies uh we are”transcript, stated
• The first simple extraction system immediately processed 10 million pages.
  “We when we did this, we immediately deployed it did 10 million pages. the first uh customer um everything was working and and we got to”transcript, stated
• Customer requests included 300-page documents and 200 to 500 requested fields.
  “This is 300page document that was um well beyond the context windows of the time and uh we're like, "Okay, no problem.”transcript, stated

Alternatives considered (roads not taken)

Single preprocess-LLM-postprocess pipeline

Rejected because: It worked initially but failed on long, noisy, complex enterprise documents.

Heuristic field grouping

Rejected because: Simple field splitting broke when fields had semantic dependencies such as customer and address fields.

Treat confidence as a traditional ML score

Rejected because: Generative AI did not provide confidence scores in the same way old ML extraction systems did.

Patterns

multi-agent-orchestration (agent-design)

Box separated document extraction into specialized sub-agents for grouping, extraction, and quality feedback.

llm-as-judge (evaluation)

The workflow uses a model/judge-style quality loop to assess extracted fields and trigger retries.

fallback-degradation (resilience)

When a field looks wrong, the system can retry using different techniques or model voting.

Tradeoffs

Architecture redesign

Chose multi-agent extraction workflow over incremental preprocessing fixes around one LLM call.

Rationale: The agentic abstraction separated failure modes and made future updates easier.

Quality feedback design

Chose feedback loop with retries and model voting over one-shot extraction confidence.

Rationale: Generative AI lacked old-style confidence scores and needed iterative quality checks.

Gotchas (what bit them)

A simple LLM extraction pipeline failed under enterprise document complexity.

Root cause: Long documents, bad OCR, multilingual pages, hundreds of fields, and confidence expectations exposed separate failure modes.

Resolution: Rearchitect into specialized sub-agents and a quality feedback loop.

Field grouping is semantic, not just mechanical chunking.

Root cause: Related fields like customer and address need to be extracted together for the LLM to preserve context.

Resolution: Use an agentic grouping step that learns which fields belong together.

Evaluation

Box evaluates extraction through quality feedback, judge-style checks, confidence-like signals, retries, and whether fields are extracted accurately enough for customer workflows.

Metrics: pages processed · field extraction accuracy · quality feedback · confidence-like judgment · retry success

Outcomes

• The multi-agent design solved multiple extraction failure modes.
• The architecture became easier for Box to evolve for new document types.

Box reports enterprise-scale usage with over 115,000 companies, tens of millions of users, over an exabyte of data, and an initial 10 million pages processed by the simple system.