Fact-Centric Legal Document Review with Custom AI Pipeline

mary_technology · Legal · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal document review for litigation

Mary Technology needed to automate legal document review for dispute-resolution teams without losing the nuance, provenance, fact disambiguation, and attorney verification required in low-fault-tolerance litigation workflows.

Constraints:

• Legal documents include handwriting, formatting, marginal notes, fragmented facts, ambiguous names, regional date formats, and hearsay/provenance distinctions.
• Standard LLM processing compresses documents through images, text, tokens, embeddings, chunking, and summaries, removing legally important meaning.
• Legal teams cannot rely on a generated document without confidence in the underlying facts and evidence.
• Facts may be scattered across thousands of documents and may support or conflict with each other.
• Lawyers need review and verification UX, not just answer generation.
• The system must integrate with existing legal workflows and downstream AI applications.

Architecture

Mary built a fact-first legal review pipeline. It ingests messy legal documents from email or existing systems, splits and structures bundled files, extracts entities, dates, events, gaps, leaks, timelines, and other key facts, resolves ambiguity such as abbreviations and people, attaches provenance and rationale, presents facts in a review dashboard, lets lawyers inspect source pages and build narratives, and only then pipes the verified fact layer into downstream AI tools such as drafting or analysis interfaces.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  document-intake["Document intake"] --> document-structuring-pipeline["Document structuring pipeline"]
  document-structuring-pipeline --> fact-manufacturing-pipeline["Fact manufacturing pipeline"]
  fact-manufacturing-pipeline --> entity-and-context-resolver["Entity and context resolver"]
  entity-and-context-resolver --> source-evidence-viewer["Source evidence viewer"]
  source-evidence-viewer --> fact-review-dashboard
  fact-review-dashboard --> lawyer-verification-loop["Lawyer verification loop"]
  fact-review-dashboard --> downstream-ai-interface["Downstream AI interface"]

Components

Data flow

Documents enter from email or uploaded tools, are scanned, split, categorized, renamed, and organized; the fact pipeline extracts entities, dates, timelines, gaps, and other key case details; each fact is corrected and rationalized with source provenance; lawyers review facts and narratives in a dashboard; then the trusted fact layer feeds downstream AI applications.

Why this architecture

Mary rejects direct RAG or agentic extraction over raw legal documents because LLMs compress away nuance and cannot by themselves resolve legally meaningful facts. The platform instead creates an auditable fact layer first, then lets lawyers verify that layer and use it in downstream AI.

Models used: Custom fact model · Fact manufacturing pipeline · Document OCR/text extraction · Structured fact layer · Downstream LLM interfaces · RAG and agentic workflows as downstream consumers

Scale

• Mary works with many of the largest firms in Australia, including A&O Shearman.
  “We work with many of the largest firms now in Australia including A&O Sherman who's one of the largest law firms in the world”transcript, stated
• The system is designed for corpora such as thousands of documents in a matter.
  “if that document is placed in between 4,000 other documents”transcript, stated
• Mary reports 75 to 85% reduction in time spent on document review.
  “we have achieved a 75 to 85% reduction in time spent on this, probably the biggest bottleneck in litigation”transcript, stated
• Mary reports a 96 out of 100 NPS score.
  “we've achieved a 96 out of 100 NPS score.”transcript, stated

Alternatives considered (roads not taken)

Direct LLM/RAG over legal documents.

Rejected because: Compression, chunking, and summarization remove the legal nuance and meaning needed for dispute work.

Extract a text snippet and treat it as a ready fact.

Rejected because: Facts need entity resolution, date interpretation, relevance, fragmentation, conflict detection, and provenance.

Generate a polished legal document directly.

Rejected because: Lawyers must be confident in the facts and narrative before filing or sending legal work.

Patterns

structured-output-contract (fact-first legal representation)

Mary represents facts as first-class structured objects with metadata, rationale, and provenance before downstream AI use.

human-in-the-loop-gate (legal verification)

Lawyers review the fact layer, source pages, rationales, and narratives before relying on generated legal work.

data-quality-loop (fact correction)

The pipeline corrects ambiguous facts such as PT/patient into named entities so downstream AI can use them reliably.

Tradeoffs

How to make legal AI reliable.

Chose Process-heavy fact manufacturing over Generalized direct LLM answers.

Rationale: The specialized pipeline preserves legal nuance and evidence at the cost of being less general and more process-heavy.

Where to use RAG or agentic workflows.

Chose After constructing a trusted fact layer over Using RAG or agents directly on the documents.

Rationale: Mary says direct RAG or agentic extraction is not good enough for legally meaningful fact extraction.

Gotchas (what bit them)

LLM compression removes legal meaning before downstream reasoning begins.

Root cause: Document processing collapses pages into images, text, tokens, embeddings, chunks, and summaries.

Resolution: Preserve and structure legal facts before using LLM-based downstream tools.

Legal facts are not equivalent to extracted strings.

Root cause: A fact may require disambiguating names, dates, relevance, fragmented mentions, conflicts, and provenance.

Resolution: Use a process-heavy fact model and manufacturing pipeline.

A seemingly perfect generated legal document is not enough.

Root cause: Attorneys have obligations to verify facts and confidence before acting for clients.

Resolution: Design for fact review, evidence drill-down, and lawyer confidence rather than final-answer delivery.

Evaluation

Mary evaluates the platform through lawyer verification of facts, provenance drill-down, review-time reduction, customer adoption among large firms, and NPS.

Metrics: 75-85% document-review time reduction · 96/100 NPS · Evidence page/location drill-down · Fact relevance rationale · Customer adoption among large law firms

Outcomes

• Mary reports 75-85% less time spent on litigation document review.
• Mary reports a 96/100 NPS score.
• The platform can organize documents, extract entities and dates, summarize matters, detect gaps/leaks, build timelines, and let lawyers review facts.
• The system works with major firms including A&O Shearman and is adding firms weekly.

The source describes global law-firm adoption, large document corpora such as 4,000 documents, review reduced from hours/days to minutes for case grip, and customer metrics across all customers.