Enterprise Document Data Extraction Using Agentic AI Workflows

box · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise document data extraction

Box needed to extract structured data from unstructured enterprise documents at scale, but single-shot LLM extraction and OCR preprocessing became brittle on complex, multilingual, scanned, and high-field-count documents.

Constraints:

Box serves large enterprises with massive unstructured content repositories.
Legacy extraction required specialized models, vendors, training data, and content-specific pipelines.
OCR failures propagated bad information into the AI model.
Customers wanted extraction from complex documents with many fields and poor OCR quality.
The architecture needed generic flexibility and scalable processing rather than document-type-specific brittleness.

Architecture

Box moved from OCR plus single-shot extraction to an agentic directed-graph workflow. The agent receives objectives, model background, secure tools, memory, and an orchestrated graph; it searches for relevant data, double-checks results, builds an outline/plan, uses OCR and page-image checks, consults multiple models that can vote, and uses LLM-as-judge feedback for iterative refinement.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  document-input["Document input"] --> ocr-preprocessing["OCR preprocessing"]
  ocr-preprocessing --> agentic-directed-graph["Agentic directed graph"]
  agentic-directed-graph --> search-and-relevance-step["Search and relevance step"]
  agentic-directed-graph --> page-image-verification["Page-image verification"]
  search-and-relevance-step --> multi-model-voting["Multi-model voting"]
  page-image-verification --> multi-model-voting
  multi-model-voting --> llm-judge-feedback["LLM judge feedback"]
  llm-judge-feedback --> extracted-fields["Extracted fields"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Document input`	Unstructured enterprise documents, scans, PDFs, and other content needing field extraction.	`OCR preprocessing`
`OCR preprocessing`	Converts scanned and document content into text and page artifacts.	`Agentic directed graph`
`Agentic directed graph`	Orchestrates extraction steps, planning, checking, and tool/model calls.	`Search and relevance step`, `Page-image verification`, `Multi-model voting`, `LLM judge feedback`
`Search and relevance step`	Finds the relevant parts of the document for requested fields.	`Multi-model voting`
`Page-image verification`	Checks OCR output against pictures of the pages.	`Multi-model voting`
`Multi-model voting`	Uses multiple models from different vendors to cross-check hard answers.	`LLM judge feedback`
`LLM judge feedback`	Provides feedback to keep trying or refine results rather than only accepting/rejecting.	`Extracted fields`
`Extracted fields`	Structured output fields produced from the document workflow.	—

Data flow

Documents enter OCR and preprocessing, then an agentic directed graph searches for relevant fields, checks OCR against page images, calls multiple models, has models vote or judge, and iterates with feedback before emitting extracted fields.

“we still had the same inputs and outputs in document with fields out answers however the approach was an agentic approach”transcript, stated

“we use OCR. We then double check it by looking at pictures of the pages. Um, and and then using multiple models.”transcript, stated

Why this architecture

Box adopted an agentic directed graph because the old generic model call was flexible but fragile: better OCR, regex checks, fine-tuning, or custom models would have reintroduced brittleness and lost generality.

“the fragility of the architecture was one that was uh we weren't really going to be able to solve on our own”transcript, stated

“suddenly all of the genericness of it would be get lost in this process”transcript, stated

Models used: GPT-2/GPT-3-era models · Generic off-the-shelf AI models · Multiple vendor models · LLM-as-Judge · OCR · LangGraph-style directed graph

Scale

Box reports over 115,000 enterprise customers.
“we've got uh over 115,000 enterprise customers.”transcript, stated
Box reports serving two-thirds of the Fortune 500.
“We've got uh twothirds of the Fortune 500.”transcript, stated
Box describes processing demands that include 100 million documents in a day.
“deal with you know 100 million documents that day”transcript, stated

Alternatives considered (roads not taken)

Build specialized AI models and vendors for each content type.

Rejected because: That approach required huge training corpora, custom vendors, custom models, and remained brittle.

“you had to build these specialized AI models uh you had to like focus on specific types of content you had to have this huge corpus of training data”transcript, stated

Use OCR plus single-shot generic LLM extraction.

Rejected because: It worked initially, but OCR, languages, and complex documents caused failure propagation.

“whenever the OCR broke, it would just naturally give bad info to the AI and then um languages were a big pain.”transcript, stated

Patch the old architecture with regex, OCR improvements, or fine-tuning.

Rejected because: Those fixes would not solve the architecture's fragility and would reduce generic flexibility.

“let's just add a post-processing uh regular regular expression checks”transcript, stated

Patterns

`multi-agent-orchestration` (orchestration)

Box uses an agentic directed graph with multiple model/tool steps for extraction.

“a uh full uh directed graph. So the ability to orchestrate it to be able to do things like where you say do this then this.”transcript, stated

`llm-as-judge` (evaluation)

The workflow uses a judge model to provide feedback and tell the extraction process to keep trying.

“the idea of the element as a judge. not just a judge to tell you that this is a um this is the answer, but a judge to tell you uh hey uh here's some feedback, keep trying.”transcript, stated

`eval-driven-development` (evaluation)

Box cross-checks OCR against page images and uses multiple models to validate hard extractions.

“use a set of tools to check and to double check the results.”transcript, stated

`structured-output-contract` (output)

The workflow keeps the same document-in, fields-out contract while changing the internals to agentic orchestration.

“we still had the same inputs and outputs in document with fields out answers however the approach was an agentic approach”transcript, stated

Tradeoffs

Use an agentic directed graph for extraction.

Chose Multiple orchestrated steps with tools, memory, models, and checks. over Single-shot LLM extraction..

Rationale: The directed graph handles complex documents and failure modes while preserving generic field extraction.

“Either it comes up with its own plan or we actually can orchestrate it ourselves because we have knowledge of what we want it to do.”transcript, stated

Separate agentic framework from distributed scaling.

Chose One architecture layer for intelligent workflows and another for generic process scale. over Mixing all extraction logic and scaling concerns together..

Rationale: Box needs both one-document intelligence and massive document-scale processing.

“being able to separate these two systems into like somebody who thinks about the agentic framework and somebody who thinks about the the how to scale a generic process”transcript, stated

Gotchas (what bit them)

OCR errors poison downstream extraction.

Root cause: Single-shot LLM extraction relies on OCR text quality.

Resolution: Double-check OCR using page images and additional model/tool steps.

“whenever the OCR broke, it would just naturally give bad info to the AI”transcript, stated

Specialized extraction pipelines lose generality.

Root cause: Document-specific models, regexes, and fine-tunes solve narrow cases by adding brittle custom logic.

Resolution: Use an agentic abstraction layer with tools, memory, planning, and directed orchestration.

“suddenly all of the genericness of it would be get lost in this process”transcript, stated

Agentic architecture was not obvious to engineers at first.

Root cause: Engineers naturally proposed local improvements to OCR or post-processing rather than a new architecture.

Resolution: Adopt the directed-graph agentic abstraction earlier for complex document workflows.

“this was for us um it was controversial like it was like our engineers like what are you talking about like let's just make the OCR better”transcript, stated

Evaluation

Box evaluates extraction quality by cross-checking OCR, page images, model agreement, and LLM judge feedback before returning structured fields.

Metrics: Model agreement/voting · OCR verification · Field extraction correctness · Judge feedback loops · Enterprise document scale

“Three models from different vendors, two of them think this is the answer. That was probably a good answer.”transcript, stated

Outcomes

Box moved from brittle single-shot extraction to an agentic directed graph.
The architecture supports OCR verification, multiple-model cross-checking, and judge feedback.
The CTO says he would have adopted agentic architecture earlier.

Box cites over 115,000 enterprise customers, two-thirds of the Fortune 500, and the need to handle 100 million documents in a day.

“agentic architecture early. If I go back in time, I would wish to done this sooner”transcript, stated