Scaling Document Processing with LLMs and Human Review

vendr_/_extend · Tech · year n/a · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: SaaS contract and order-form processing

Vendr and Extend needed to convert more than 100,000 unstructured SaaS order forms and contracts into reliable structured catalog data.

Constraints:

• Order forms are often PDFs with difficult seller, reseller, supplier, customer, price, and contract-duration relationships.
• Entity mapping errors degrade trust, search relevance, and aggregate analytics.
• Mission-critical extraction could not rely on raw LLM outputs without review and correction.

Architecture

The system combines traditional OCR/document preprocessing, LLM entity extraction, catalog entity mapping, embeddings for document-similarity outlier detection, targeted human review in Extend and Vendr back-office tools, and feedback loops that affirm or correct mappings instead of reviewing every document manually.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Order form PDFs"] --> B["OCR preprocessing"]
  B --> C["LLM extraction prompts"]
  C --> D["Entity mapping workflow"]
  C --> E["Document embeddings"]
  E --> F["Similarity review selector"]
  F --> G["Human review tools"]
  D --> G
  G --> H["Structured document catalog"]
  D --> H

Components

Data flow

Order-form PDFs are converted into document text, LLM prompts extract entity names and other fields, names are mapped into a catalog of customers/sellers/resellers/suppliers, embeddings identify documents that are dissimilar to known clusters, and only targeted cases are routed to human review for corrections or affirmations.

Why this architecture

The architecture separates extraction, mapping, and review because order forms are ambiguous: LLMs can take a first pass, but catalog usefulness depends on canonical entity mapping and targeted human correction of high-risk cases.

Models used: OpenAI embeddings · LLMs for extraction · OCR systems

Scale

• The system processed more than 100,000 documents.
  “hybrid human review has accelerated our ability to digest more than 100,000 documents”transcript, stated
• Without preprocessing and fixing entity data, error rates would exceed 5%.
  “in our case we would have error rates above 5% without pre-processing and fixing the data”transcript, stated
• The short embedding was limited to the first 450 words for one review purpose.
  “we added a short embedding limited to the first 450 words and now use different embedding lengths for different purposes”transcript, stated

Alternatives considered (roads not taken)

Real-time LLM research without preprocessing

Rejected because: The team expected too many mistakes, including unknown unknowns, in complex seller/supplier/customer relationships.

Review every document manually

Rejected because: The corpus was too large, so similarity-based targeting was used to route only suspicious or informative cases.

One embedding representation for all review purposes

Rejected because: Different review questions required different embedding lengths and similarity signals.

Patterns

human-in-the-loop-gate (governance)

Human reviewers correct and affirm LLM-extracted document mappings for targeted cases instead of trusting every extraction automatically.

semantic-caching (retrieval)

Embeddings over document text are reused to compare documents, detect outliers, and focus review on unusual cases.

structured-output-contract (data-extraction)

The pipeline converts order forms into structured catalog fields and canonical entity mappings.

eval-driven-development (evaluation)

Model and prompt choices depend on eval sets to catch regressions and field-level performance changes.

Tradeoffs

Review strategy

Chose targeted human review over manual review of every document.

Rationale: Document similarity can surface likely mistakes and reduce review burden while maintaining confidence.

Extraction boundary

Chose preprocess and fix data before use over ask LLMs to resolve everything in real time.

Rationale: Unknown unknowns and clustered errors made real-time-only reasoning too risky.

Model selection

Chose task-specific model optionality with eval sets over one default model for all document tasks.

Rationale: Accuracy, cost, latency, and visual performance differ by task and model family.

Gotchas (what bit them)

Entity mapping matters as much as text extraction.

Root cause: Supplier, seller, reseller, and customer relationships can be ambiguous or change through acquisitions.

Resolution: Map extracted names into canonical catalog entries and send hard cases through human review.

Incorrect mappings damage user trust and analytics.

Root cause: Misclassified entities hide relevant documents and corrupt aggregate contract-value calculations.

Resolution: Use preprocessing, catalog mapping, and targeted review before exposing structured data.

LLM confidence is not enough for mission-critical extraction.

Root cause: Models can be wrong and self-reported confidence is unreliable.

Resolution: Use logprob-style uncertainty signals, citations, validations, LLM judges, and human review.

Evaluation

The team evaluates extraction quality with eval sets, targeted human-review corrections, validations, uncertainty signals, and document-similarity outlier checks.

Metrics: entity mapping correctness · field-level regressions · human correction rate · document similarity outliers · validation failures · review coverage

Outcomes

• The hybrid LLM and human-review workflow accelerated ingestion of more than 100,000 documents.
• The team reports the approach was substantially faster and more successful than prior human coding.
• Document similarity let the team target review rather than reviewing every document.

The source emphasizes production process design and scale of document ingestion more than exact accuracy percentages after deployment.