Automated Data Journalism Platform Using LLMs for Real-time News Generation

realtime · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Media & Entertainment

Realtime needed to keep a data-journalism site updated as public datasets changed faster than journalists could manually inspect, contextualize, and write about every update.

Constraints:

• data sources update continually and often lack push notifications
• generated journalism must avoid unsupported causal claims and visible mistakes
• LLM cost must not scale with page traffic
• readers need transparency and source-level verification

Architecture

Realtime uses a polling and diffing data pipeline to detect new datapoints across public datasets, normalize them into a single format, fetch relevant recent news through NewsAPI, and then construct GPT-4 Turbo prompts from dataset metadata, quantitative update features, and related news. The LLM generates headlines, subheads, and annotated summaries, then a second LLM pass edits for input consistency and style-guide compliance. Visualizations are generated with Vega/Vega-Lite, homepage ranking combines data-change magnitude, recency, and news volume, and reader-facing markup links claims back to data and articles for transparency.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  dataset-polling-pipeline[Dataset polling pipeline] --> normalized-data-store[Normalized data store]
  dataset-polling-pipeline --> newsapi-context-fetcher[NewsAPI context fetcher]
  normalized-data-store --> prompt-builder[Prompt builder]
  newsapi-context-fetcher --> prompt-builder
  prompt-builder --> gpt-4-turbo-generator[GPT-4 Turbo generator]
  gpt-4-turbo-generator --> llm-editing-pass[LLM editing pass]
  gpt-4-turbo-generator --> story-ranking-system[Story ranking system]
  llm-editing-pass --> visualization-layer[Visualization layer]
  story-ranking-system --> visualization-layer
  normalized-data-store --> visualization-layer

Components

Data flow

The system regularly polls each configured data source and diffs the publisher's latest data against known data. When an update appears, Realtime normalizes it, retrieves related NewsAPI articles, and builds a prompt from dataset metadata, latest data features, and recent news. GPT-4 Turbo produces an annotated summary, headline, and subhead; a follow-up LLM edit checks for unsupported or off-style output. The resulting text and data drive Vega/Vega-Lite visualizations, homepage ranking, and reader-facing links that expose the underlying data and news evidence.

Why this architecture

The architecture separates data freshness, LLM generation, quality control, and presentation because each has a different failure mode. Polling and diffing solve unavailable push notifications. Prompt construction grounds generation in structured metadata, numeric data features, and news context. Multi-stage LLM calls improve task performance despite added cost, while post-generation checks and annotations address hallucination and transparency risks. Running generation on data updates, not page views, keeps model spending decoupled from audience growth.

Models used: GPT-4 Turbo

Scale

• Realtime targets datasets that change over time, including economic indicators, political polls, forecasts, environmental data, and sports odds.
  “important datasets that change over time (think economic indicators, political polls and forecasts, environmental data, sports odds, and so on”article, implied
• The data pipeline runs on a distributed cloud platform and can be configured per data feed.
  “This system is scalable (it runs on a distributed cloud platform, and can be configured per-data feed to avoid needless updates)”article, stated
• Generation cost is designed to stay invariant to the number of site visitors.
  “This influenced how we designed our systems, we had to ensure that the cost was invariant to the number of visitors coming to our site.”article, stated

Alternatives considered (roads not taken)

Single monolithic LLM call

Rejected because: Realtime found that splitting work into multiple LLM calls improved performance, even though it increased API cost.

JSON as the primary prompt data format

Rejected because: The team found the extra syntax in JSON adds noise for the model, while YAML-like readable formatting is easier for both humans and LLMs to parse.

Open-source models for lower cost

Rejected because: The article says open-source models are cheaper but did not yet match proprietary accuracy at the time of writing.

Patterns

prompt-chaining (generation-workflow)

Realtime breaks generation and review into multiple LLM calls instead of asking one prompt to perform every task.

structured-output-contract (integration)

The LLM returns typed story fields and annotated summary text that downstream ranking, visualization, and link-rendering code can consume.

guardrails-io-filtering (quality-control)

Prompts forbid unsupported causality and additional LLM calls check for predictable fallacies and style-guide violations.

cost-aware-routing (operations)

Realtime runs LLM generation on data updates rather than per reader request so model spend does not grow with traffic.

Tradeoffs

Multi-stage generation

Chose several focused LLM calls over one broad LLM prompt.

Rationale: Multiple calls improved model performance for generation and review, at the cost of more API calls.

Traffic-invariant generation cost

Chose generate on data updates over generate on every reader request.

Rationale: Decoupling model calls from audience traffic keeps unit economics predictable as readership grows.

Transparent AI labeling and linked evidence

Chose visible generated-text disclosure, visualized data, and linked article/data references over hiding AI generation behind normal editorial presentation.

Rationale: The team considered AI output useful but fallible, so readers need clear provenance and verification affordances.

Gotchas (what bit them)

LLMs overstate causal narratives in news headlines.

Root cause: The model tends to prefer exciting cause-and-effect stories even when the data only supports correlation or temporal co-occurrence.

Resolution: Realtime explicitly forbids unsupported causality and adds guardrail calls for causation, math, and semantic errors.

Prompt data format affects model comprehension.

Root cause: Syntactic characters in machine-friendly formats can become noise for an LLM.

Resolution: Realtime favors human-readable representations such as YAML-like data when passing information to the model.

Generated journalism still inherits input-data quality problems.

Root cause: The LLM can only summarize and contextualize the datasets and news it receives.

Resolution: The system invests in clean, consistently updating datasets and makes source verification visible to readers.

Evaluation

Realtime describes iterative prompt experimentation, follow-up LLM checks for predictable error classes, and reader-facing verification through data visualizations and source links rather than a formal offline benchmark.

Metrics: relative magnitude of data changes · recency of latest data updates · relative volume of news headlines · qualitative performance improvements from multiple LLM calls

Outcomes

• Realtime scaled coverage of changing public datasets into automated stories and visualizations.
• Journalists can spend less time on rote update work and more time on deeper reporting.
• The team reports significant performance improvements from multi-call prompting, while acknowledging continued fallibility.

The article does not give traffic counts, but it explicitly designs generation costs to be invariant to visitor volume and targets many updating public datasets.