Production-Scale Document Parsing with Vision-Language Models and Specialized OCR

reducto · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production document parsing, extraction, citation, chart extraction, and form editing

Reducto needed to parse and extract information from highly heterogeneous documents at production scale, where frontier VLMs hallucinate dense tables, traditional OCR misses complex layouts, and downstream RAG/extraction workflows require grounded citations rather than unlocated text.

Constraints:

Architecture

Reducto uses a hybrid divide-and-conquer document pipeline. Layout detection identifies regions, routes simple text to high-precision traditional OCR, and sends complex regions to in-house VLMs or autoregressive layout models. Specialized table, chart, citation, extraction, and edit models run where needed, while pixel-level maps preserve provenance. The team evaluates with internal and public benchmarks, uses metadata when it is reliable, and iterates through production feedback, autoeeval, user-facing Studio controls, and human labeling.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  input-document["Input document"] --> layout-detection-router["Layout detection router"]
  layout-detection-router["Layout detection router"] --> traditional-ocr-models["Traditional OCR models"]
  layout-detection-router["Layout detection router"] --> in-house-vision-language-models["In-house vision-language models"]
  layout-detection-router["Layout detection router"] --> autoregressive-layout-model["Autoregressive layout model"]
  layout-detection-router["Layout detection router"] --> table-parsing-model["Table parsing model"]
  layout-detection-router["Layout detection router"] --> chart-extraction-pipeline["Chart extraction pipeline"]
  traditional-ocr-models["Traditional OCR models"] --> pixel-level-citation-map["Pixel-level citation map"]
  in-house-vision-language-models["In-house vision-language models"] --> pixel-level-citation-map["Pixel-level citation map"]
  autoregressive-layout-model["Autoregressive layout model"] --> pixel-level-citation-map["Pixel-level citation map"]
  table-parsing-model["Table parsing model"] --> pixel-level-citation-map["Pixel-level citation map"]
  chart-extraction-pipeline["Chart extraction pipeline"] --> pixel-level-citation-map["Pixel-level citation map"]
  pixel-level-citation-map["Pixel-level citation map"] --> extraction-endpoint["Extraction endpoint"]
  pixel-level-citation-map["Pixel-level citation map"] --> edit-endpoint["Edit endpoint"]
  extraction-endpoint["Extraction endpoint"] --> autoeeval-benchmark["Autoeeval benchmark"]
  edit-endpoint["Edit endpoint"] --> form-schema-studio["Form schema studio"]
  edit-endpoint["Edit endpoint"] --> autoeeval-benchmark["Autoeeval benchmark"]
  autoeeval-benchmark["Autoeeval benchmark"] --> rd-arena["RD arena"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Input document PDF, scanned form, table, chart, legal document, or other document to parse. Layout detection router
Layout detection router Segments pages and routes regions to OCR, VLMs, autoregressive layout models, or specialized parsers. Input document, Traditional OCR models, In-house vision-language models, Autoregressive layout model, Table parsing model, Chart extraction pipeline
Traditional OCR models High-precision parsing for simple text regions. Layout detection router, Pixel-level citation map
In-house vision-language models Handles complex handwritten, stamped, watermarked, vertical-text, and difficult region parsing. Layout detection router, Pixel-level citation map, Extraction endpoint
Autoregressive layout model Outputs many bounding-box tokens for text-dense pages where traditional detection is insufficient. Layout detection router, Pixel-level citation map
Table parsing model Specialized parser for dense financial tables and multi-format table output. Layout detection router, Pixel-level citation map, Extraction endpoint
Chart extraction pipeline Mixture of strong and weak VLMs that extracts chart components into structured data tables. Layout detection router, Pixel-level citation map, Extraction endpoint
Pixel-level citation map Grounding layer preserving element locations, confidence, and absent-information citations. Traditional OCR models, In-house vision-language models, Autoregressive layout model, Table parsing model, Chart extraction pipeline, Extraction endpoint, Edit endpoint
Extraction endpoint Returns structured extraction with grounded citations for downstream RAG, chatbots, and analysis. In-house vision-language models, Table parsing model, Chart extraction pipeline, Pixel-level citation map, Autoeeval benchmark
Edit endpoint Detects and fills form fields, using schemas and user-adjustable bounding boxes/descriptions. Pixel-level citation map, Form schema studio, Autoeeval benchmark
Form schema studio Human-facing interface for reviewing and modifying form fields, descriptions, bounding boxes, and fill values. Edit endpoint
Autoeeval benchmark Internal evaluation framework for production-distribution document parsing and model deployment decisions. Extraction endpoint, Edit endpoint, RD arena
RD arena Planned user-facing A/B comparison service across Reducto, Textract, and other providers. Autoeeval benchmark

Data flow

A document enters layout detection, which segments pages into regions and chooses routing based on complexity. Simple regions go to OCR; difficult handwriting, stamps, watermarks, dense tables, or charts go to specialized VLM/table/chart models. Parsed elements retain bounding boxes and pixel maps. Extraction/citation models use those grounded elements to produce word- or character-level citations, including absent-information locations. Edit workflows detect form fields, build reusable form schemas, and let users adjust descriptions, bounding boxes, and fill values before repeated form filling.

3 supporting quotes
“Their production pipeline implements a sophisticated divide-and-conquer strategy that begins with layout detection models identifying different regions in a document.”summary, stated
“simple text regions use traditional OCR models optimized for high precision and recall, while complex regions with handwritten text, stamps, watermarks, or vertical text are handled by in-house trained vision-language models.”summary, stated
“Reducto maintains a pixel-level map of every element from the parsing stage onward.”summary, stated

Why this architecture

Reducto avoids a single end-to-end VLM because document parsing failure modes differ by region and document type. Routing lets cheap, precise OCR handle simple text while expensive specialized VLM/autoregressive models handle hard regions. Pixel-level grounding is built into parsing because citations cannot be reliably patched on later. Benchmarks and production-distribution evals keep model choices tied to real document heterogeneity.

3 supporting quotes
“These production challenges motivated their hybrid architecture that combines multiple specialized models rather than relying on a single vision-language model.”summary, stated
“This grounding capability is built into their models from the beginning rather than patched on at the end through prompting.”summary, stated
“Rather than making model deployment decisions based on a handful of labeled samples, they evaluate across their full production distribution”summary, stated

Models used: specialized vision-language models · traditional OCR models · layout detection models · autoregressive layout detection models · YOLO · table parsing models · chart extraction VLM mixture · Rome OCR · frontier models · AWS Textract · Microsoft Azure Document Intelligence · Google Cloud OCR

Scale

Alternatives considered (roads not taken)

Single frontier VLM for all document parsing

Rejected because: Frontier models hallucinate dense tables, repeat patterns, and make temperature tradeoffs between repetition and hallucination.

“end-to-end large language models, while powerful, suffer from systematic failures when applied to document parsing tasks.”summary, stated

Traditional object detection such as YOLO for text-dense layout

Rejected because: Autoregressive models better output hundreds of bounding boxes with high fidelity for the hardest text-dense pages.

“Rather than using traditional object detection models like YOLO, they trained autoregressive models that can output hundreds of bounding box tokens”summary, stated

Uniform VLM patching for chart extraction

Rejected because: Charts require component-level extraction of lines, spikes, axes, and data points using a model mixture.

“Rather than using a single vision-language model that patches the input uniformly, they developed an agentic approach using a mixture of strong and weak vision-language models.”summary, stated

Always parse digitally native PDFs with vision

Rejected because: Reliable PDF metadata can be faster and more accurate for some brokerage-style documents.

“For digitally-native documents like financial brokerage reports, metadata extraction can be significantly faster and more accurate than vision-based parsing.”summary, stated

Send hundreds of page images directly to a model for long-document extraction

Rejected because: Image inputs at that scale produce proliferating hallucinations; parsed text works better for large documents.

“they recommend parsing the entire document first and then working with the parsed text rather than sending hundreds of images to a model.”summary, stated

Patterns

model-routing (architecture)

The pipeline routes document regions to OCR, VLMs, autoregressive layout models, or specialized parsers based on complexity.

“For each region, they route to the appropriate model based on complexity: simple text regions use traditional OCR models optimized for high precision and recall, while complex regions with handwritten text, stamps, watermarks, or vertical text are handled by in-house trained vision-language models.”summary, stated

structured-output-contract (interface)

Parsing and extraction produce structured outputs such as HTML, markdown, JSON, XML, tables, form schemas, and citations.

“They support multiple output formats including HTML, markdown, JSON, and XML, giving users control over representation based on table complexity.”summary, stated

eval-driven-development (quality)

Reducto uses RD-forcebench, RD-table-bench, autoeeval, ablations, and RD-arena-style provider comparison to make deployment decisions.

“Evaluation and benchmarking are central to Reducto's production operations. They built an internal framework called autoeeval that performs automatic evaluation across the wide heterogeneity of document types they encounter in production.”summary, stated

human-in-the-loop-gate (operations)

The Studio lets users modify form schema descriptions, bounding boxes, and fill values, and the team uses an in-house labeling team for difficult capabilities.

“users can modify in their studio interface with full control over descriptions, bounding boxes, and specific fill values.”summary, stated

rag-retrieval (retrieval)

The parsing quality is explicitly optimized as a foundation for downstream RAG and chatbot workflows.

“better document parsing quality directly improves downstream task accuracy, whether for RAG, information extraction, chatbots, or other applications.”summary, stated

Tradeoffs

Use a routed hybrid parser

Chose Traditional OCR plus specialized VLM/layout/table/chart models over Single end-to-end frontier VLM.

Rationale: Different document regions have different cost, latency, and accuracy needs; routing avoids overusing expensive models and reduces hallucination.

“This routing decision is critical for production performance, balancing cost, latency, and accuracy across diverse document types.”summary, stated

Reserve expensive autoregressive layout models for hard cases

Chose Autoregressive bounding-box generation only on challenging text-dense pages over Running slow expensive layout models everywhere.

Rationale: The model is accurate but costly and slower, so production routing limits it to cases where cheaper detectors are insufficient.

“these autoregressive models are expensive to run and slower than alternatives, so they reserve them for the most challenging cases”summary, stated

Use metadata when it is accurate

Chose Hybrid extraction mode that can use PDF metadata over Vision-based parsing for every digitally native document.

Rationale: Metadata can be faster and more accurate for some documents, but only when verified as reliable.

“implementing a hybrid extraction mode that evaluates whether PDF metadata is accurate enough to use.”summary, stated

Compute reusable form schemas

Chose One-time schema creation with user-editable fields over Re-detecting and describing the same form from scratch every time.

Rationale: Repeated form-filling use cases benefit from amortizing field detection and letting users refine schema details.

“The form schema is computed once and then reused, optimizing for production efficiency.”summary, stated

Gotchas (what bit them)

Frontier VLMs hallucinate in dense document structures.

Root cause: End-to-end models struggle with dense tables, repeated patterns, and temperature-sensitive generation.

Resolution: Use specialized table parsers, OCR, and routed VLMs rather than one general model.

“frontier models hallucinating data in dense financial tables, repetition problems when models encounter patterns like sequences of dots”summary, stated

Citation cannot be reliably bolted on after parsing.

Root cause: Generic OCR/VLM output lacks location information tying extracted text back to document pixels.

Resolution: Maintain a pixel-level map of elements from parsing onward and train citation models for location-grounded output.

“Unlike generic OCR services or vision-language models that output text without location information, Reducto maintains a pixel-level map of every element”summary, stated

Form field detection fails on visually sparse or ambiguous PDFs.

Root cause: Many forms are image-only, lack lines, or are questionnaires converted from Word with ambiguous layouts.

Resolution: Collect additional data and train field-detection models on words-only and varied form layouts.

“they discovered through production use that many forms lack clear visual features like lines.”summary, stated

Small benchmark samples can mislead deployment decisions.

Root cause: Production documents are heterogeneous; a handful of labels does not represent the full distribution.

Resolution: Evaluate across the full production distribution with autoeeval and user-submitted RD-arena comparisons.

“a common mistake in building AI applications where developers benchmark on too few examples and fail to account for the heterogeneity that production systems will encounter.”summary, stated

Evaluation

Reducto evaluates parsing and extraction with RD-forcebench for scanned forms, RD-table-bench for dense tables, ablation studies, autoeeval across the production distribution, and RD-arena provider comparisons where users judge outputs A/B.

Metrics: RD-forcebench extraction accuracy · RD-table-bench table parsing quality · ablation study accuracy · provider A/B preference · pixel-level citation confidence · production-distribution autoeeval

3 supporting quotes
“The company maintains an internal benchmark called RD-forcebench focused on information extraction from challenging scanned forms”summary, stated
“For table parsing specifically, Reducto developed RD-table-bench, a publicly available benchmark on Hugging Face”summary, stated
“They built an internal framework called autoeeval that performs automatic evaluation across the wide heterogeneity of document types”summary, stated

Outcomes

The strongest scale anchors are over 1 billion processed documents, hundreds of bounding-box tokens, 100+ page/100+ field extraction, two months of chart iteration, and 15,000 trial credits.

3 supporting quotes
“The company announced their Series B funding and reported processing over 1 billion documents, with most of that volume occurring in recent months, demonstrating rapid scaling of their production infrastructure.”summary, stated
“Their specialized table parsing models outperform these general-purpose services, demonstrating the value of focused iteration on specific document types.”summary, stated
“The system processes over a billion documents with mechanisms for continuous improvement through production feedback, automated evaluation, and rapid iteration”summary, stated