Building a Production RAG System for Technical Document Search with Local LLMs

core_marine · Energy · 2026 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Local confidential RAG over offshore engineering project archives

Core Marine needed an internal natural-language search assistant over nearly a decade of highly technical project documents, including OrcaFlex files, while preserving confidentiality through local LLMs and keeping the system fast enough for engineers to use.

Constraints:

• The corpus started as 1 TB of mixed project documents with little organization
• All LLM processing had to run locally for confidentiality reasons
• The first indexing attempts exhausted laptop RAM and froze the operating system
• The default LlamaIndex JSON storage could require days of reprocessing after restarts
• Indexing hundreds of gigabytes locally became GPU-bound
• Original documents were too large to keep on local disk alongside the vector index

Architecture

Core Marine built a local RAG system with Ollama running llama3.2:3b and embeddings, LlamaIndex for retrieval orchestration, ChromaDB backed by SQLite for vector storage, Flask as an API, Streamlit as the user interface, and Azure Blob Storage for original document downloads. The team added aggressive file filtering, batch processing, checkpoints, GPU indexing on a rented RTX 4000 SFF Ada VM, and SAS-token links to serve cited documents directly from Azure while keeping the vector index local.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  streamlit-web-ui[Streamlit Web UI] --> flask-api[Flask API]
  flask-api --> llamaindex-rag[LlamaIndex RAG]
  llamaindex-rag --> ollama-local-models[Ollama Local Models]
  llamaindex-rag --> chromadb-index[ChromaDB Index]
  chromadb-index --> sqlite-storage[SQLite Storage]
  azure-blob-storage[Azure Blob Storage] --> llamaindex-rag
  file-filter-pipeline[File Filter Pipeline] --> batch-indexer[Batch Indexer]
  gpu-vm[GPU VM] --> batch-indexer
  batch-indexer --> llamaindex-rag
  llamaindex-rag --> azure-blob-storage

Components

Data flow

Filtered project files are converted to text where useful, batched through LlamaIndex, embedded with local models, and stored in ChromaDB. Users ask questions in Streamlit, the frontend calls a Flask API, LlamaIndex queries ChromaDB and Ollama, and cited source documents are downloaded directly from Azure Blob Storage through generated SAS-token links rather than copied into local storage.

Why this architecture

The architecture follows from confidentiality and scale constraints. External APIs were off the table, so Ollama handled local models. LlamaIndex simplified RAG orchestration, but default JSON persistence failed at hundreds-of-gigabytes scale, pushing the system to ChromaDB. Original documents stayed in Azure because only the 54GB vector index, 10GB LLM, backend, and frontend needed local disk.

Models used: llama3.2:3b · nomic-embed-text · Ollama

Scale

• The original project archive contained 1 TB of files.
  “I was given access to 1 TB of projects, mixed with technical documentation, reports, analyses, regulations, CSVs, etc.”article, stated
• File filtering reduced the number of files to index by 54%.
  “The result was a 54% reduction in the number of files to index. And of course, my RAM stopped exploding. I could finally start indexing without fear.”article, stated
• The final index covered 451GB of documents.
  “This allowed indexing the 451GB of documents across multiple sessions, with checkpoints, without losing progress on interruptions, without corrupted data.”article, stated
• The completed index held 738,470 vectors and occupied 54GB in ChromaDB.
  “738,470 vectors, 54GB of index in ChromaDB, and a RAG system ready to answer questions. I copied the ChromaDB database, a SQLite file, to my local machine and that was it.”article, stated
• CPU processing of 500 MB took 4-5 hours before GPU acceleration.
  “Processing 500 MB of documents by CPU takes 4-5 hours, not good numbers. I absolutely needed a powerful GPU. In a follow-up meeting, it was decided to rent me a virtual machine with an NVIDIA RTX 4000 SFF Ada, which has 20GB of VRAM.”article, stated
• Indexing on the rented GPU VM took about two to three weeks and cost 184 euros.
  “After several weeks, between 2 and 3, the indexing process finished without failures. 738,470 vectors, 54GB of index in ChromaDB, and a RAG system ready to answer questions.”article, stated

  “I copied the ChromaDB database, a SQLite file, to my local machine and that was it. To the relief of my Sysadmin and Project Manager, we could finally shut down the virtual machine. The cost was 184 euros on Hetzner, not cheap.”article, stated

Alternatives considered (roads not taken)

Use external LLM APIs

Rejected because: Company confidentiality requirements required local models.

Use LlamaIndex default JSON storage

Rejected because: It became unmanageable for hundreds of gigabytes and forced full reprocessing after restarts.

Index all file types directly

Rejected because: Large binary, simulation, backup, CSV, JSON, and other low-value files exhausted memory or added little retrieval value.

Store original documents locally with the index

Rejected because: The vector index and models fit locally, while source documents could remain in Azure and be served by SAS-token links.

Patterns

rag-retrieval (knowledge-access)

Core Marine built a local RAG system that retrieves project-document chunks and returns answers with references to source documents.

privacy-preserving-processing (deployment)

The system keeps language model and embedding processing local because the project corpus is confidential.

batch-offline-processing (indexing)

The indexer processes files in batches, explicitly frees memory, and stores embeddings incrementally.

durable-execution (indexing)

The pipeline saves checkpoints so long-running indexing can resume after outages or restarts.

observability-tracing (operations)

Monitoring scripts expose indexing progress, speed, checkpoint state, and failed files during long-running RAG builds.

Tradeoffs

Use local models

Chose Ollama with local LLaMA models over external LLM APIs.

Rationale: Confidential project data required local processing even though it added hardware and performance constraints.

Move to ChromaDB

Chose a dedicated vector database over LlamaIndex JSON storage and custom checkpointing.

Rationale: ChromaDB enabled batch indexing, resumes across sessions, backup/restore, and large-scale search without corrupt progress.

Filter aggressively before indexing

Chose exclude low-value or dangerous file classes over attempt complete ingestion of the 1 TB archive.

Rationale: Filtering avoided memory blowups and reduced cost while preserving useful technical documents.

Serve source documents from Azure

Chose SAS-token direct downloads from Azure Blob Storage over copying all original documents onto local disk.

Rationale: The local deployment only needed the vector index, model, backend, and frontend; original documents were too large and could remain available on demand.

Gotchas (what bit them)

Naive ingestion of mixed project files crashed the machine.

Root cause: LlamaIndex tried to treat massive binary or low-value files as text and load multi-gigabyte files into memory.

Resolution: The team added extension/name filters and converted only useful office documents to plain text.

Convenient local JSON persistence did not survive production scale.

Root cause: The default LlamaIndex storage format required reprocessing all documents on restart and was not optimized for large searches.

Resolution: Core Marine moved vectors into ChromaDB with SQLite persistence.

Checkpointing is easy to get wrong at large indexing scale.

Root cause: The first custom checkpoint attempt corrupted data and stayed slow.

Resolution: ChromaDB's batch pipeline provided resumable sessions without corrupted progress.

CPU-only indexing made the schedule untenable.

Root cause: The bottleneck moved from memory and storage to embedding throughput.

Resolution: The team rented an RTX 4000 SFF Ada VM and optimized containers for GPU use.

Bad files should not stop an indexing batch.

Root cause: Some PDFs, Word docs, spreadsheets, and other documents fail parsing even after filtering.

Resolution: The pipeline logs failures and continues, leaving manual review for later.

Evaluation

Evaluation was operational and experiential: the author benchmarked indexing speed, monitored RAM/GPU/disk bottlenecks, used progress and failure scripts during long jobs, and validated the final system by successfully indexing the corpus and serving a working Streamlit/Flask RAG application.

Metrics: files filtered · documents indexed · vectors indexed · index size · CPU processing time per 500 MB · GPU VRAM · indexing duration · failed file logs · checkpoint progress

Outcomes

• The team built a local LLM RAG tool for engineers to ask questions over company project history.
• Filtering cut the indexed file set by 54% and stopped RAM crashes.
• The system indexed 451GB of documents into 738,470 vectors and a 54GB ChromaDB index.
• A rented GPU VM reduced an otherwise months-long indexing process to about two to three weeks.
• The deployed app used Flask, Streamlit, LlamaIndex, ChromaDB, Ollama, and Azure Blob Storage links.

The project began with a 1 TB archive, indexed 451GB into 738,470 vectors, produced a 54GB ChromaDB index, used a 20GB VRAM GPU VM, and completed indexing in roughly two to three weeks at a reported cost of 184 euros.