Production-Scale RAG System for Real-Time News Processing and Analysis

emergent_methods · Media & Entertainment · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real-time news RAG and contextual news analysis

Emergent Methods needed to provide accurate, timely, source-attributed news answers at million-article-per-day scale, overcoming general-purpose LLM weaknesses around stale information, slow web search, missing global perspectives, and hallucination risk in sensitive news contexts.

Constraints:

• News answers must be current, diverse across regions/languages/sources, and properly attributed.
• The system processes over 1 million articles per day from 50,000 sources in 5-minute buckets.
• Retrieval must filter by timestamp and metadata so current questions do not retrieve stale articles.
• High-volume processing requires independently scalable microservices for embeddings, vector search, inference, and orchestration.

Architecture

Ask News uses a Kubernetes microservices architecture built around context engineering. NewsGrabber supplies articles from 50,000 sources, FlowDapt orchestrates 5-minute ingestion/enrichment workflows, Text Embedding Inference produces embeddings, Qdrant stores dense and sparse vectors with timestamp filters, SPLADE plus dense embeddings provide hybrid retrieval, and vLLM serves on-premise reasoning/generation. The product returns timely answers with citations, coverage trends, region/category filters, and source diversity.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  newsgrabber-source-feed["NewsGrabber source feed"] --> flowdapt-orchestrator["FlowDapt orchestrator"]
  flowdapt-orchestrator["FlowDapt orchestrator"] --> text-embedding-inference-service["Text Embedding Inference service"]
  flowdapt-orchestrator["FlowDapt orchestrator"] --> qdrant-vector-database["Qdrant vector database"]
  flowdapt-orchestrator["FlowDapt orchestrator"] --> vllm-inference-service["vLLM inference service"]
  text-embedding-inference-service["Text Embedding Inference service"] --> qdrant-vector-database["Qdrant vector database"]
  qdrant-vector-database["Qdrant vector database"] --> hybrid-retrieval-layer["Hybrid retrieval layer"]
  qdrant-vector-database["Qdrant vector database"] --> ask-news-application["Ask News application"]
  hybrid-retrieval-layer["Hybrid retrieval layer"] --> vllm-inference-service["vLLM inference service"]
  hybrid-retrieval-layer["Hybrid retrieval layer"] --> ask-news-application["Ask News application"]
  vllm-inference-service["vLLM inference service"] --> ask-news-application["Ask News application"]
  remote-model-api["Remote model API"] --> ask-news-application["Ask News application"]
  ask-news-application["Ask News application"] --> recommendation-profile["Recommendation profile"]

Components

Data flow

Articles arrive continuously from NewsGrabber and are batched into 5-minute ingestion windows. FlowDapt schedules and scales enrichment tasks across Kubernetes, including translation, summarization, custom extraction, and embeddings through TEI. Qdrant stores vectors and metadata for hybrid sparse-dense search with timestamp filtering. A user query is rewritten or clarified when needed, retrieved against fresh article embeddings, and sent to local or remote models for grounded summarization with citations in Ask News.

Why this architecture

Emergent Methods uses modular microservices because news workloads demand freshness, throughput, and easy substitution as LLM infrastructure evolves. Qdrant stays responsible for storage/search, TEI for embeddings, vLLM for inference, and FlowDapt for orchestration. Context engineering replaces rigid pre-LLM pipelines by making translation, summarization, extraction, and output style configurable without rewriting the whole stack.

Models used: LLaMA 2 · vLLM-served on-premise LLMs · OpenAI remote models · Text Embedding Inference · SPLADE · DistilBART · translation models · dense embedding models

Scale

• Ask News adaptively models over 1 million news articles per day.
  “Ask News (asknews.app), adaptively models over 1 million news articles per day”summary, stated
• NewsGrabber gives access to 50,000 sources.
  “Emergent Methods sources their news from NewsGrabber (newscatcher-ai.com), which provides access to 50,000 different sources.”summary, stated
• The system processes news in 5-minute buckets.
  “They process news in 5-minute buckets, continuously ingesting and enriching the latest articles.”summary, stated
• FlowDapt supports local-to-cluster parity for systems spanning 10-100 servers.
  “Local Development Parity: Ensures what developers see on their laptops matches cluster behavior—critical when debugging systems spanning 10-100 servers.”summary, stated
• ChatGPT returned a 25-day-old Gaza article in the motivating example.
  “returning a 25-day-old article when asked for current news on Gaza.”summary, stated

Alternatives considered (roads not taken)

General-purpose LLM with web search for current news

Rejected because: It was slow and returned outdated information for sensitive current events.

Rigid pre-LLM NLP pipeline

Rejected because: Fixed chunking, translation, summarization, and classification pipelines could not adapt quickly to changing product requirements.

All-in-one convenience services

Rejected because: They sacrifice performance and independent scaling compared with single-responsibility microservices.

Vector database-managed embeddings

Rejected because: The team wanted embeddings isolated as a dedicated service for independent resource allocation and scaling.

HyDE for all retrieval

Rejected because: It adds compute cost and may generate synthetic documents from stale model training data rather than current news.

Patterns

context-engineering (context)

Emergent Methods explicitly frames the system as context engineering: parsing, enriching, filtering, and packaging news context for current grounded answers.

rag-retrieval (retrieval)

Ask News retrieves current articles with timestamp, keyword, sparse, and dense filters before generating source-attributed answers.

retrieval-ranking (retrieval)

The system combines sparse SPLADE and dense embeddings to improve retrieval quality and align user queries with document embeddings.

model-routing (architecture)

The platform uses both on-premise models and remote proprietary models depending on capability, cost, and latency needs.

cost-aware-routing (operations)

High-volume article processing uses on-premise vLLM, co-located Qdrant, TEI dynamic batching, and microservice scaling to manage throughput and cost.

Tradeoffs

Use single-responsibility microservices

Chose Purpose-built Qdrant, TEI, vLLM, FlowDapt, and application services over All-in-one LLM/RAG platforms.

Rationale: Independent scaling and substitution are critical in a high-throughput news system where infrastructure changes quickly.

Co-locate Qdrant in Kubernetes

Chose On-premise/in-cluster vector database deployment over Cloud-hosted vector database with external network path.

Rationale: Co-location reduces latency, noisy-neighbor risk, and privacy exposure.

Use hybrid sparse-dense retrieval

Chose SPLADE sparse vectors plus dense embeddings over Dense-only semantic search or BM25 alone.

Rationale: Hybrid search keeps lexical/news specificity while adding semantic retrieval quality.

Use both remote and on-premise models

Chose Hybrid model strategy over Only proprietary APIs or only self-hosted inference.

Rationale: The tradeoff balances frontier capability, infrastructure cost, throughput, and latency.

Gotchas (what bit them)

General-purpose LLMs can be dangerously stale for news.

Root cause: Web-search-augmented LLMs may retrieve old articles and fail recency requirements on sensitive topics.

Resolution: Continuously ingest, timestamp-filter, and retrieve fresh source-attributed news context.

News retrieval must preserve source diversity.

Root cause: Single-region or single-language coverage creates incomplete or biased context for global events.

Resolution: Ingest massive multi-source data and enforce diversity across languages, regions, and sources.

Old documents in a vector database can confuse current retrieval.

Root cause: Accumulated versions or stale articles remain semantically similar to current questions.

Resolution: Use timestamp filtering for news and remove outdated information for slower-moving corpora.

HyDE can import stale model knowledge into search.

Root cause: The hypothetical document is generated from an LLM's training data, not necessarily from current news.

Resolution: Use HyDE cautiously and use prompt engineering for explicit follow-up query generation where needed.

Evaluation

The case emphasizes production quality through retrieval freshness, source attribution, region/language diversity, timestamp filtering, latency optimization, and user-facing product signals such as coverage trends and source citations rather than a named offline evaluation suite.

Metrics: articles processed per day · 5-minute ingestion buckets · timestamp filtering · source citations · coverage trends · region/category filters · latency · throughput · source diversity

Outcomes

• Ask News processes over 1 million articles per day into timely, sourced news context.
• The product supports semantic news search, source attribution, sentiment/region/category filters, coverage trends, story starring, and planned recommendations.
• The architecture demonstrates production context engineering over real-time news rather than static document RAG.

Key scale anchors include 1M+ articles/day, 50,000 sources, 5-minute processing buckets, 10-100 server development parity, timestamp filters, and in-cluster vector/inference services.