Enterprise Data Extraction Evolution from Simple RAG to Multi-Agent Architecture

box · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise unstructured-data extraction

Box needed to scale metadata extraction across enterprise unstructured documents after a simple LLM pipeline that had processed 10 million pages failed on 300-page files, poor OCR, multilingual content, hundreds of fields, and confidence scoring needs.

Constraints:

customers bring over an exabyte of data across enterprise deployments
some documents exceed context-window limits
OCR quality can be poor or language-dependent
single documents can require 200 to 500 fields
the system has to keep evolving for new document categories

Architecture

Box moved from a linear LLM extraction pipeline to a multi-agent architecture. A central workflow splits extraction into specialized sub-agents for grouping fields, selecting extraction modalities such as OCR or page images, judging result quality, and retrying with different models or techniques when feedback identifies problems.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  document-intake[Document intake] --> field-grouping-agent[Field grouping agent]
  field-grouping-agent --> extraction-strategy-agent[Extraction strategy agent]
  extraction-strategy-agent --> quality-judge[Quality judge]
  quality-judge --> retry-and-voting-loop[Retry and voting loop]
  retry-and-voting-loop --> extracted-metadata[Extracted metadata]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Document intake`	Enterprise files and requested metadata fields.	`Field grouping agent`
`Field grouping agent`	Groups semantically related extraction fields before prompting.	`Document intake`, `Extraction strategy agent`
`Extraction strategy agent`	Chooses OCR, page-image inspection, and extraction techniques.	`Field grouping agent`, `Quality judge`
`Quality judge`	Assesses whether extracted fields are likely accurate.	`Extraction strategy agent`, `Retry and voting loop`
`Retry and voting loop`	Retries with different models or techniques when quality feedback flags a field.	`Quality judge`, `Extracted metadata`
`Extracted metadata`	Structured data output from unstructured enterprise documents.	`Retry and voting loop`

Data flow

Documents and requested fields are routed through a field-grouping stage, extraction agents choose how to inspect text or page images, a quality loop judges each field, and retry/model-voting behavior produces structured metadata. This replaces the earlier single-pass preprocess, LLM, postprocess pipeline.

3 supporting quotes

“take your document, uh take the fields that you're looking for, do some pre-processing, um and then some OCR, uh”transcript, implied

“that hard um and so then we ended up rearchitecting from scratch with an agentic approach so rather than just do”transcript, stated

“You know, looks like field number three is wrong. So, all right. Well, let me try again. maybe I'll use different models to vote and to”transcript, stated

Why this architecture

The rearchitecture acknowledges that extraction quality problems are heterogeneous. Specialized agents let Box add or replace strategies for a document type without rebuilding the whole pipeline, while feedback loops handle confidence-like needs that generative AI does not natively satisfy.

“that moment, but because it became easy for us to update. And this really um sort of is the the key of what we”transcript, stated

“when you're thinking of like building these uh u intelligent powered solutions if you do an agentic based approach it's a much cleaner abstraction”transcript, implied

Models used: large language model · LLM-as-judge · OCR

Scale

Box describes itself as serving over 115,000 companies.
“We have over 115,000 companies, tens of millions of users and our customers have given us uh entrusted us with over an exabyte of their data.”transcript, stated
The first deployed extraction system processed 10 million pages.
“We when we did this, we immediately deployed it did 10 million pages. the first uh customer um everything was working and and we got to”transcript, stated
The hard cases included requests for 200 or 500 fields from one document.
“some people were like, "Okay, I want not just 20 pieces of data from this document, but like 200 or 500 different pieces and and that just kind of like”transcript, stated

Alternatives considered (roads not taken)

Keep extending preprocessing around a single LLM call

Rejected because: The team recognized the elegant initial approach no longer worked across customer complexity.

“generative AI because the thing that was working so well that was so awesome that was so elegant just didn't work.”transcript, stated

Rely on OCR-only extraction

Rejected because: Some files needed page-image inspection in addition to OCR.

“these parts of the data, maybe it's going to look at the picture of the of the the pages in addition to just the OCR.”transcript, stated

Patterns

`multi-agent-orchestration` (agent-design)

Specialized sub-agents handle field grouping, extraction strategy, and quality feedback.

“a steps multi-agent architecture where we separated out the problems that we had into a series of sub aents”transcript, stated

`llm-as-judge` (evaluation)

A judge-like step assesses whether extraction appears accurate.

“we'll run an element as a judge and it'll tell you after it's done if it thinks it was accurate or not.”transcript, stated

`fallback-degradation` (resilience)

The system retries with different techniques or models when a field fails quality review.

“maybe I'll use different models to vote and to do other techniques.”transcript, stated

Tradeoffs

System decomposition

Chose specialized sub-agents over one prompt over preprocessed text.

Rationale: The agentic decomposition made the system easier to improve and map to how humans approach extraction.

“think of it like no I've got one document and I've to get through these fields and I'm just going to think of it the way that you would do it as a person”transcript, stated

Quality handling

Chose feedback and retry loop over static confidence score.

Rationale: A feedback loop can actively try different approaches when a field looks wrong.

“give feedback so that the AI could try uh different techniques.”transcript, stated

Gotchas (what bit them)

A successful first deployment created false confidence.

Root cause: The initial 10-million-page result did not cover the harder edge cases customers later supplied.

Resolution: Rebuild around specialized agents and quality feedback.

“then we started to hit the problems”transcript, stated

Generative extraction did not satisfy confidence-score expectations.

Root cause: Customers expected old-ML style confidence scores, but generative AI did not provide them directly.

Resolution: Use judge-style quality feedback and retry behavior.

“generative AI doesn't have confidence scores like old ML models do.”transcript, stated

Evaluation

Evaluation is operational: quality feedback checks extracted fields, flags likely failures, and triggers retry strategies or model voting.

Metrics: extraction quality · field-level feedback · retry outcomes · pages processed

“quality feedback loop, not just to give you confidence, but then also to give feedback”transcript, implied

Outcomes

The agentic design solved immediate document-extraction problems.
It gave Box a cleaner abstraction for evolution and new document-specific agents.

Box reports enterprise-scale deployment context: over 115,000 companies, tens of millions of users, over an exabyte of data, and an earlier 10 million pages processed.

“this approach really solved a lot of our problems”transcript, stated