LLMs for Investigative Data Analysis in Journalism

propublica · Media & Entertainment · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Investigative journalism data and document analysis

ProPublica needed to use LLMs to accelerate analysis of large investigative datasets and document/media collections without compromising journalistic verification, privacy, or accuracy.

Constraints:

• AI outputs could not be published without human review and confirmation.
• The system had to avoid hallucinated explanations when grant or disciplinary documents did not support a finding.
• Sensitive audio and video material required secure handling rather than casual third-party processing.
• The workflow had to scale across thousands of grant records, large disciplinary-report collections, and hundreds of hours of media.

Architecture

ProPublica used LLMs as investigative analysis tools rather than as writers. For the NSF grants investigation, reporters prompted an LLM to inspect each grant, identify why it might have been flagged, and produce supporting citations while explicitly avoiding guesses. Journalists reviewed every detail before publication. Other workflows used example-guided classification for disciplinary reports and self-hosted open-source AI to transcribe and classify sensitive Uvalde audio/video materials.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Dataset["Investigative dataset"] --> AI_analysis["Prompted AI analysis"]
  Dataset --> Self_hosted["Self-hosted transcription and classification"]
  AI_analysis --> Findings["Citation-backed findings"]
  Findings --> Verification["Reporter verification"]
  Self_hosted --> Verification
  Records["Licensing and public records"] --> Verification
  Verification --> Journalism["Published journalism"]

Components

Data flow

Reporters collect a dataset or media archive, define a focused investigative question, and prompt the AI to classify or explain records with citations and uncertainty guardrails. The AI output becomes a lead-generation or triage layer. Journalists then review details, cross-check external records, seek comment, and write the public story themselves.

Why this architecture

ProPublica treated AI as a lead-generation and analysis accelerator because journalism demands human accountability. The LLM can scan many records and surface patterns, but guardrails, citations, reporter review, and self-hosted processing for sensitive material keep the workflow aligned with investigative standards.

Models used: large language model powering ChatGPT · self-hosted open-source AI software

Scale

• The NSF grant investigation analyzed more than 3,400 grants.
  “ProPublica reporters used an LLM to analyze a database of over 3,400 National Science Foundation grants that Senator Ted Cruz had labeled as promoting "woke" ideology”summary, stated
• One highlighted NSF grant was worth $470,000.
  “Among them: a $470,000 grant to study the evolution of mint plants and how they spread across continents.”article, stated
• The Uvalde investigation involved hundreds of hours of audio and video recordings.
  “This included hundreds of hours of audio and video recordings, which were difficult to sift through.”article, stated
• The misconduct investigation used AI over a large disciplinary-report collection.
  “The investigation relied on a large collection of disciplinary reports, covering a wide range of potential violations.”article, stated

Alternatives considered (roads not taken)

Publish or trust AI outputs directly

Rejected because: ProPublica required staff to confirm every detail and continue seeking comment from named people and agencies.

Let the model guess when evidence is unclear

Rejected because: The prompt explicitly told the model not to guess because hallucination was a known risk.

Use cloud AI services for sensitive Uvalde materials

Rejected because: The team chose self-hosted open-source AI to securely process graphic, sensitive audio/video evidence.

Patterns

human-in-the-loop-gate (governance)

Every AI-generated lead or classification remained subject to journalist review, confirmation, and external comment before publication.

safety-policy-enforcement (quality)

Prompts explicitly told the model not to guess when uncertain, reducing hallucinated investigative claims.

structured-output-contract (integration)

The grant-analysis prompt asked the LLM to produce specific explanations and citations for each grant rather than free-form prose.

privacy-preserving-processing (infrastructure)

For sensitive Uvalde materials, ProPublica used self-hosted open-source AI rather than sending graphic recordings through an external hosted service.

llm-as-judge (classification)

AI was used to review documents and flag whether disciplinary reports or grant descriptions matched investigative categories.

Tradeoffs

Use AI for lead generation, not story authorship

Chose AI-assisted analysis with human-written journalism over AI-generated public stories or headlines.

Rationale: The newsroom wanted speed and pattern discovery while keeping editorial responsibility with journalists.

Use self-hosted AI for sensitive media

Chose self-hosted open-source transcription and classification over outsourced/cloud processing of raw Uvalde material.

Rationale: Sensitive, disturbing material required secure processing while still reducing manual media-sifting burden.

Give the misconduct classifier examples and keywords

Chose few-shot and keyword-guided document review over open-ended classification without reporter-provided examples.

Rationale: Known examples and keywords helped the model focus on the specific investigative category of sexual misconduct.

Gotchas (what bit them)

LLMs can create plausible but unsupported investigative claims.

Root cause: Models are known to hallucinate, which is dangerous in accountability reporting.

Resolution: Prompt the model not to guess, require citations, and have staff review every detail before publication.

AI triage still needs investigative confirmation against authoritative records.

Root cause: A classifier can narrow the search space but cannot substitute for records-based verification.

Resolution: Use two reporters and licensing records to confirm AI-flagged misconduct cases.

Sensitive media can be difficult, graphic, and poorly organized for human-only review.

Root cause: The Uvalde source material contained hundreds of hours of raw audio/video that lacked clear organization and included disturbing content.

Resolution: Use secure self-hosted AI to transcribe and classify material so reporters could reconstruct events.

Evaluation

ProPublica evaluated AI outputs through journalist review, two-reporter confirmation for misconduct classifications, licensing-record cross-checks, named-party comment processes, and editorial publication review rather than automated benchmark metrics.

Metrics: every detail confirmed before publication · two-reporter review of AI-flagged misconduct results · licensing-record confirmation · successful reconstruction of Uvalde events from classified/transcribed material

Outcomes

• The NSF grant analysis helped reveal examples where superficial terms such as diversify or female triggered flags unrelated to the claimed ideology.
• AI-assisted disciplinary-report review helped narrow cases for a sexual-misconduct investigation.
• Self-hosted transcription and classification helped reporters reconstruct the Uvalde response from hundreds of hours of raw media.
• ProPublica maintained a boundary that AI assists reporting but does not write published stories, headlines, newsletters, or takeaways.

The workflow handled more than 3,400 grants, a large disciplinary-report collection, and hundreds of hours of sensitive media, but the final publication process stayed human-reviewed rather than automated.