Building Modular and Scalable RAG Systems with Hybrid Batch/Incremental Processing

bell · Telecommunications · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise RAG knowledge-base management

Bell needed production infrastructure for RAG systems whose source documents, chunking configuration, embedding models, and vector indexes change over time. The team treated document embedding pipelines and knowledge-base management as an MLOps product rather than a one-off chatbot feature, because dynamic HR, Confluence, and automation knowledge bases create synchronization, lineage, quota, deployment, and race-condition problems that a simple RAG prototype does not solve.

Constraints:

• RAG adds knowledge bases, vector databases, and embedding pipelines beyond a basic LLM chatbot.
• Enterprise knowledge bases are not static; HR policies may update quarterly while Confluence-style sources can change daily.
• Configuration changes to chunking, separators, models, regions, projects, or endpoints can invalidate the existing index and require a full rebuild.
• GCP quotas constrain online embedding traffic, including 100 requests per minute and 30,000 prompts per batch request for the discussed Gecko setup.
• Large document changes can exhaust shared online quotas and affect the production chatbot API.
• Incremental updates need coordination so upload/delete races do not leave stale chunks in the vector database.
• The platform has to support multiple use cases without custom pipeline code each time.

Architecture

Bell built a hybrid RAG knowledge-base platform. A primary batch pipeline, orchestrated by Cloud Composer/Apache Airflow, runs Apache Beam/Dataflow pre-processing, embedding, and post-processing jobs against a structured knowledge-base bucket and a vector database. The knowledge base stores raw documents, processed chunks, processed embeddings, timestamped batch-run folders, and a current folder read by the chatbot API. An optional incremental path adds a Pub/Sub topic on bucket changes and a streaming Dataflow job that processes changed documents, windows events, and keeps the vector database synchronized between batch refreshes. YAML configuration files drive loaders, chunking, embedding models, regions, projects, endpoints, source URIs, and sync targets; a DAG generator turns each YAML into an Airflow DAG so new RAG pipelines can be deployed with little custom code.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  source-systems["Source systems"] --> document-ingestion-workflow["Document ingestion workflow"]
  document-ingestion-workflow --> knowledge-base-bucket["Knowledge base bucket"]
  yaml-configuration-file["YAML configuration file"] --> dag-generator["DAG generator"]
  dag-generator --> cloud-composer-orchestrator["Cloud Composer orchestrator"]
  cloud-composer-orchestrator --> batch-dataflow-pipeline["Batch Dataflow pipeline"]
  batch-dataflow-pipeline --> pre-processing-job["Pre-processing job"]
  pre-processing-job --> embedding-job["Embedding job"]
  embedding-job --> post-processing-job["Post-processing job"]
  post-processing-job --> vector-database["Vector database"]
  knowledge-base-bucket --> pub-sub-topic["Pub/Sub topic"]
  pub-sub-topic --> incremental-dataflow-pipeline["Incremental Dataflow pipeline"]
  incremental-dataflow-pipeline --> vector-database
  knowledge-base-bucket --> rag-chatbot-api["RAG chatbot API"]
  vector-database --> rag-chatbot-api
  rag-chatbot-api --> llm-provider["LLM provider"]

Components

Data flow

Documents enter the raw knowledge-base folder through either a librarian workflow or an automated source-sync pipeline. Batch runs create timestamped intermediate artifacts for raw chunks, pre-processed chunks, and embeddings, then sync the newest run to the current folder that the chatbot API reads. Incremental runs listen to Pub/Sub bucket events, group recent upload/delete/update events, process only the latest action for each changed document, and update the current knowledge base and vector database. When a batch refresh runs, orchestration drains and stops the incremental job, rebuilds the full knowledge base, restarts incremental processing, and then consumes unacknowledged Pub/Sub messages that arrived during the batch window.

Why this architecture

Bell separated batch and incremental paths because configuration rebuilds and large document changes need quota-friendly full processing, while small high-velocity updates need near-real-time synchronization. Apache Beam/Dataflow gave the same programming model for batch and streaming work. Pub/Sub was chosen over isolated cloud functions because grouped event windows can avoid upload/delete races. YAML-driven reusable components and generated DAGs turned the platform into a product that could support many RAG use cases without custom engineering for each one.

Models used: Google Gecko embedding model · Gemini · Google embedding models · Self-hosted LLM model option

Scale

• The discussed GCP setup is constrained by 100 online embedding requests per minute and 30,000 prompts per batch request.
  “you're limited to 100 repress uh 100 requests per minute and for the batch requests you're limited to 30,000 prompts per request”transcript, stated
• The platform was built around the North America Northeast 1 GCP region in Bell's implementation.
  “we're primarily gcp platform uh team so and specifically located in the North americ Northeast one”transcript, stated
• The incremental design can group events in a 60-second sliding window.
  “if we Define a sliding window of 60 seconds or you know x amount of seconds our job will only take the most recent action”transcript, stated
• Low-velocity HR policies were described as quarterly updates, while Confluence-style sources can change daily.
  “our HR policies are a great example of of uh documents that are updated once a quarter”transcript, stated

  “for we have a couple Confluence use cases where people are updating confluences um consistently or by almost like daily”transcript, stated
• Generated DAGs allow a new RAG pipeline to be deployed within minutes.
  “all components are reasonable across pipelines no additional developments needed and this allows you to deploy a pipeline within minutes”transcript, stated
• The platform supports multiple Bell use cases with both librarian-managed and automated-pipeline ingestion already implemented.
  “we've actually implemented both at Bell um so we have a couple Confluence uh solutions that uh are based on the automated Pipeline”transcript, stated

  “we have a couple solutions that are based on the librarian um aspect where people are manually curating your documents”transcript, stated

Alternatives considered (roads not taken)

Use online embedding calls for configuration changes or large document changes.

Rejected because: Bell rejected this for large rebuilds because online processing can consume API quota and interfere with the chatbot API sharing the same project.

Use Cloud Functions or Lambda-style isolated event handlers for bucket changes.

Rejected because: The team chose Pub/Sub plus Beam because isolated upload/delete handlers can race and leave bad chunks synced after a deletion.

Handle large high-velocity document changes on managed online LLM quotas.

Rejected because: The presenters framed this as a rare unicorn case and recommended replacing quota limits with self-hosted serving capacity when it occurs.

Use source loaders for incremental document processing.

Rejected because: Bell focused on document loaders because source loaders process an undefined scope and do not identify the specific changed documents needed for incremental updates.

Preserve unchanged chunks and embed only edited parts of a document.

Rejected because: The team treats a document as the atomic unit because partial chunk preservation creates coordination complexity that is not worth the marginal time savings.

Build a custom pipeline for every RAG use case.

Rejected because: Bell standardized the process and generated DAGs from YAML so reusable components and interfaces can serve new use cases without additional code.

Patterns

rag-retrieval (retrieval)

Bell's system manages the knowledge base, vector index, embedding pipeline, and source-document synchronization layer that makes internal RAG chatbots answer against company-specific documents rather than only model training data.

batch-offline-processing (llmops)

Configuration changes, initial builds, scheduled refreshes, and large document changes are handled by a batch pipeline so the system can rebuild the knowledge base and vector database without burning online embedding quota.

async-message-orchestration (orchestration)

The incremental path uses Pub/Sub messages plus Apache Beam windowing to coordinate document-change events and process only the most recent action, avoiding race conditions from isolated event handlers.

observability-tracing (operations)

The knowledge-base structure keeps raw documents, processed chunks, embeddings, timestamped runs, and current artifacts so teams can trace how documents were processed and why the bot answered the way it did.

data-quality-loop (feedback)

Chunking and retrieval parameters are adjusted through a feedback loop on chatbot responses, with data scientists tuning chunk sizes, overlaps, and vector parameters based on response quality signals.

cost-aware-routing (operations)

The solution matrix routes work to batch, incremental, or self-hosted processing based on document-change size, latency requirements, and quota pressure so embedding jobs do not starve online chatbot traffic.

Tradeoffs

Use a hybrid batch plus incremental architecture.

Chose A primary batch pipeline with optional incremental Pub/Sub/Dataflow updates over Only batch refreshes or only real-time incremental updates.

Rationale: Batch handles initialization, scheduled refreshes, config changes, and large document changes, while incremental processing handles small high-velocity updates; combining them adds orchestration but covers both modes.

Use Pub/Sub and Beam windowing for incremental changes.

Chose Coordinated Pub/Sub events processed by Apache Beam/Dataflow over Cloud Functions, Lambda-style handlers, or isolated atomic event processors.

Rationale: Event windowing and grouping allow the pipeline to resolve upload/delete races by processing the most recent action instead of letting isolated functions finish out of order.

Standardize reusable pipeline services behind YAML.

Chose A generated Airflow DAG and configurable pre-processing, embedding, and post-processing services over Custom pipeline code per RAG use case.

Rationale: The reusable interface lowers deployment effort for new RAG pipelines, but it requires up-front service boundaries, DevOps resources, and a YAML configuration contract.

Use document loaders as the unit of incremental processing.

Chose Specific document URIs and atomic document instances over Source loaders that sweep directories or broad sources.

Rationale: Incremental processing needs to know exactly which document changed; source loaders fit batch jobs but not precise update handling.

Reprocess a changed document as a whole.

Chose Re-chunk and upsert the full changed document over Trying to preserve unchanged chunks and embed only modified sections.

Rationale: Partial chunk preservation was judged too complex relative to the time saved by avoiding a full document reprocess.

Choose between librarian and automated ingestion by source velocity.

Chose Manual librarian curation for slow sources and automated source-sync pipelines for high-velocity sources over One universal ingestion method for every knowledge source.

Rationale: Low-velocity sources are simpler to manage manually, while high-velocity systems like Confluence need automated change detection, landing-zone comparison, and rsync-style synchronization.

Gotchas (what bit them)

Configuration changes cannot be mixed into an existing index piecemeal.

Root cause: Different chunking or embedding configurations make old and new vectors incomparable.

Resolution: Treat configuration changes as full batch rebuild events for the knowledge base and vector database.

Large rebuilds can starve the production chatbot if they share online embedding quota.

Root cause: Managed embedding APIs impose project-level quotas, and high-volume rebuild traffic competes with live user requests.

Resolution: Run large changes through batch processing, rate-limit small incremental updates, and consider self-hosted serving for rare high-velocity large-change cases.

Naive bucket-event functions can process stale documents after a deletion.

Root cause: Upload and delete events run as isolated atomic cloud functions with no coordination, so the slower upload-processing path may finish after the delete.

Resolution: Use Pub/Sub plus Apache Beam windowing/grouping so the incremental job processes only the latest relevant event within the window.

Source loaders are a poor fit for precise incremental updates.

Root cause: A source loader sweeps an undefined source scope and does not know which specific document changed.

Resolution: Focus the pre-processing component on document loaders that operate on specific document URIs or atomic document instances.

Partial re-embedding inside a changed document can become coordination-heavy.

Root cause: Edits in the middle of a document can shift chunks, making it difficult to preserve unchanged chunks while only reprocessing changed sections.

Resolution: Treat documents as the atomic update unit and re-chunk/re-embed the whole document.

The public presentation does not describe a formal RAG-quality evaluation harness.

Root cause: The talk focuses on embedding-pipeline infrastructure, deployment, lineage, and operational synchronization more than answer-quality metrics.

Resolution: Use the described thumbs-up/down feedback loop as a quality signal, but treat formal retrieval/answer evaluation as an unstated gap in this source.

Evaluation

The talk describes engineering validation rather than a formal RAG-answer evaluation suite: modular unit tests, integration tests, CI/CD, lineage/provenance, operational quota reasoning, and a chatbot thumbs-up/down feedback loop that informs chunking and vector-parameter tuning.

Metrics: Unit tests for every component · Integration tests for the overall system · CI/CD deployment of component changes · GCP quota limits such as 100 requests per minute and 30,000 prompts per batch request · Thumbs-up/down response feedback for chunking and vector-parameter iteration

Outcomes

• Bell implemented both librarian-managed and automated-pipeline ingestion modes.
• The same productized components can support multiple RAG use cases without rebuilding a custom pipeline for each one.
• New RAG pipelines can be deployed by uploading YAML that drives DAG generation and component configuration.
• The system can support other teams' data while keeping their knowledge base and vector database in their own GCP project.

The presentation reports a modular Bell-internal platform used for multiple RAG use cases, including HR policy and Confluence-style sources, but it does not publish production traffic, answer-quality, or business-impact metrics.