Production RAG Stack Development Through 37 Iterations for Financial Services

jonfernandes · Finance · year n/a · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production RAG stack for financial-services and on-premise knowledge-base question answering

The presenter needed to move from naive RAG that returned irrelevant answers to a production-ready, privacy-conscious stack that financial institutions could run on-premise while preserving retrieval quality, observability, and evaluation.

Constraints:

• Financial-services customers may require on-premise deployment and cannot always send data to external APIs.
• Naive vector retrieval can return semantically nearby but practically wrong context.
• Prototype tooling can prioritize API convenience, while production requires self-hosted open models and deployable containers.
• RAG systems need monitoring and systematic evaluation rather than one-off manual checks.

Architecture

The final stack uses LlamaIndex orchestration, open embedding models such as BGE for production, Qdrant for vector storage, GPT-4 or API services for prototyping, open Llama/Qwen models served through Ollama or HuggingFace TGI for production, cross-encoder reranking, Arize Phoenix tracing, RAGAS evaluation, and Docker Compose to deploy ingestion, vector database, frontend, model serving, tracing, and evaluation containers. The design separates prototype convenience from production privacy and deployment needs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  data-ingestion-service["Data ingestion service"] --> embedding-model["Embedding model"]
  data-ingestion-service["Data ingestion service"] --> qdrant-vector-database["Qdrant vector database"]
  embedding-model["Embedding model"] --> qdrant-vector-database["Qdrant vector database"]
  embedding-model["Embedding model"] --> rag-orchestrator["RAG orchestrator"]
  qdrant-vector-database["Qdrant vector database"] --> rag-orchestrator["RAG orchestrator"]
  qdrant-vector-database["Qdrant vector database"] --> reranker["Reranker"]
  reranker["Reranker"] --> rag-orchestrator["RAG orchestrator"]
  reranker["Reranker"] --> llm-serving["LLM serving"]
  rag-orchestrator["RAG orchestrator"] --> llm-serving["LLM serving"]
  llm-serving["LLM serving"] --> frontend-application["Frontend application"]
  llm-serving["LLM serving"] --> arize-phoenix-tracing["Arize Phoenix tracing"]
  arize-phoenix-tracing["Arize Phoenix tracing"] --> ragas-evaluation["RAGAS evaluation"]
  ragas-evaluation["RAGAS evaluation"] --> rag-orchestrator["RAG orchestrator"]
  docker-compose-runtime["Docker Compose runtime"] --> data-ingestion-service["Data ingestion service"]
  docker-compose-runtime["Docker Compose runtime"] --> qdrant-vector-database["Qdrant vector database"]
  docker-compose-runtime["Docker Compose runtime"] --> llm-serving["LLM serving"]

Components

Data flow

Documents enter an ingestion service and are embedded by a production-suitable open embedding model. Qdrant stores the vectors and retrieves top candidates for a query. A reranker, prototyped with Cohere and moved toward Nvidia for production, scores the top retrieved documents and selects the most relevant context. The LLM then generates a deterministic answer with temperature 0, Phoenix traces latency and pipeline behavior, and RAGAS evaluates quality across many queries.

Why this architecture

The stack reflects a repeated lesson from 37 failed attempts: production RAG needs a full pipeline, not just embeddings plus a prompt. Financial-services constraints push production toward open/self-hosted models and Dockerized deployment, while retrieval accuracy requires reranking and monitoring. Separating containers lets teams deploy on-premise or cloud while scaling/updating components independently.

Models used: OpenAI text-embedding-ada-002 · OpenAI text-embedding-3-large · BGE-small · Nvidia embedding models · GPT-3.5 Turbo · GPT-4 · Llama 3.2 · Qwen 3 4B · Cohere reranker · Nvidia reranker

Scale

• The stack is presented as the result of 37 failed implementations.
  “This case study is framed as lessons learned from 37 failed implementations, making it a practical guide rather than a theoretical overview.”summary, stated
• Qdrant is described as scaling from a couple of documents to hundreds of thousands of documents.
  “The presenter notes it works well from "a couple of documents to hundreds of thousands of documents."”summary, stated
• The demonstrated reranking pipeline retrieves top 5 documents and reranks to top 2.
  “the top 5 results from the vector database are retrieved, then reranked to select the top 2 most relevant documents before passing to the LLM.”summary, stated
• The Qwen model called out for production is the 4B parameter version.
  “Alibaba's Qwen 3 (specifically the 4B parameter version).”summary, stated
• Production LLM temperature is set to 0 for deterministic factual answers.
  “A notable configuration detail is setting temperature to 0 for the production LLM”summary, stated

Alternatives considered (roads not taken)

Naive RAG with embedding, vector retrieval, and generation only

Rejected because: It produced irrelevant answers and failed to select the best contextual evidence.

API-based closed models for production in financial services

Rejected because: On-premise and data-privacy requirements can make external APIs unacceptable.

Cross-encoders for initial retrieval across the whole corpus

Rejected because: They are more accurate but do not scale because every query-document pair must be processed together.

Single-query manual assessment

Rejected because: One test query cannot establish whether the RAG system maintains quality as data and queries vary.

Patterns

rag-retrieval (retrieval)

The stack uses LlamaIndex plus Qdrant to retrieve candidate context before generation.

retrieval-ranking (retrieval)

A cross-encoder reranking stage improves the top-k retrieved documents before context is sent to the LLM.

model-routing (architecture)

The stack deliberately uses closed/API models for prototyping and open/self-hosted models for production privacy and deployment needs.

observability-tracing (operations)

Phoenix tracing is included as a production container to show latency and behavior across embedding, retrieval, reranking, and generation.

eval-driven-development (quality)

The stack uses RAGAS and multi-query evaluation because learning from failed iterations requires systematic quality measurement.

Tradeoffs

Prototype with closed APIs, deploy with open models

Chose API models for fast experimentation and self-hosted open models for production over Using one model-serving approach across all environments.

Rationale: API models accelerate iteration, while open models satisfy privacy and deployment control requirements.

Use bi-encoders before cross-encoders

Chose Fast bi-encoder retrieval followed by accurate cross-encoder reranking over Cross-encoding every document pair upfront.

Rationale: This keeps broad retrieval scalable while applying expensive semantic scoring only to a small candidate set.

Containerize the RAG stack

Chose Docker Compose with separate services over A single monolithic runtime.

Rationale: Containers make the same system deployable on-premise or in cloud environments and allow components to be scaled or updated independently.

Gotchas (what bit them)

Naive vector similarity can retrieve the wrong practical answer.

Root cause: Bi-encoder retrieval is fast but does not jointly score the query and document for final relevance.

Resolution: Use a cross-encoder reranker on top-k retrieved candidates before generation.

Prototype choices may violate production privacy constraints.

Root cause: API-based closed embeddings and LLMs are convenient, but financial institutions may require on-premise processing.

Resolution: Transition production to open/self-hosted embeddings, LLMs, rerankers, and Dockerized deployment.

RAG quality cannot be diagnosed without component-level traces.

Root cause: Latency or answer errors can originate in embedding, retrieval, reranking, or generation.

Resolution: Run Phoenix or similar tracing alongside the RAG stack.

One successful demo query does not prove production readiness.

Root cause: Knowledge bases grow and query distributions change, so quality can degrade without continuous evaluation.

Resolution: Use RAGAS or similar evaluation across many queries.

Evaluation

The presenter recommends systematic RAG evaluation with RAGAS across many queries, plus Phoenix traces to locate latency and quality problems by pipeline component.

Metrics: RAGAS evaluation · many-query test coverage · latency by component · embedding latency · retrieval latency · reranking latency · generation latency · top-k retrieval quality

Outcomes

• The evolved stack moves from irrelevant naive answers to accurate contextual responses such as correct London St. Pancras assistance information.
• The final design gives financial-services deployments a containerized path for on-premise RAG.
• The case turns 37 failures into a reusable stack of orchestration, retrieval, reranking, tracing, evaluation, and deployment choices.

Key scale anchors include 37 failed iterations, Qdrant's range from a couple of documents to hundreds of thousands, top-5 to top-2 reranking, and multi-container Docker Compose deployment.