Lessons from Building a Production RAG System: Data Formatting and Prompt Engineering

credal · Tech · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise RAG over complex corporate documents

Credal needed production-grade RAG and routing systems for real corporate documents where naive loaders, vector search, and prompts failed on footnotes, tables, summaries, date reasoning, structured output, and cost-sensitive Slack routing. The core constraint was that LLM attention is limited, so both data representation and prompts had to focus the model on the hardest part of each task.

Constraints:

• the learnings come from more than 250k LLM calls on over 100k corporate documents
• enterprise corpora include long, strangely formatted documents and many user phrasings
• standard document loaders can break semantic relationships such as footnotes and tables
• Claude full-context approaches can cost more than $1 per question and do not scale to large corpora
• Slack routing needed 95%+ accuracy while using GPT-3.5 for cost and latency reasons
• formatting instructions can distract weaker models from the actual classification task
• real systems must handle security, cost, compliance, latency, and user diversity

Architecture

Credal built custom RAG and prompt workflows that restructure documents before retrieval, enrich sections with LLM-generated metadata tags, route summary questions to summary-like sections, inline footnotes near references, convert tables to CSV, and split multi-step prompt work into focused sequential calls. For Slack AI experts, GPT-3.5 first concentrates on matching the user message to the right expert, then a second GPT-3.5 formatting call repairs JSON when needed. Human experts define domain tags and can verify generated answers.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  docs["Corporate document corpus"] --> format["Document formatting pipeline"]
  docs --> tags["Metadata tagging pipeline"]
  curator["Human tag curator"] --> tags
  format --> vector["Vector store"]
  tags --> filter["Tag pre-filter"]
  filter --> vector
  vector --> answer["Answering model"]
  slack["Slack expert router"] --> experts["Expert descriptions"]
  slack --> repair["JSON repair call"]
  repair --> slack
  slack --> domain["Domain-specific AI experts"]
  domain --> answer

Components

Data flow

Corporate documents are ingested through custom formatting rather than naive loaders. Footnotes are inserted at the citation point, tables are converted to CSV, and sections receive LLM-generated tags such as summary/exposition or customer/product-feature concepts. At query time, tags pre-filter the relevant sections before semantic search. The retrieved context goes to an answer model. For Slack AI experts, an LLM triager compares the user message to expert descriptions with examples occupying the bulk of the prompt; if the JSON is malformed, a second GPT-3.5 call focuses only on formatting.

Why this architecture

Credal rejects naive RAG because embedding search cannot infer semantic relationships that live in document structure, such as footnotes attached to citations or tables with implicit rows and columns. Metadata tags give retrieval a way to find summary sections that do not mention the user's words. Prompt splitting keeps GPT-3.5 focused on the high-value classification task while a second cheap call handles formatting if needed. The design follows the principle that model attention is scarce and should be spent on the hard part of the task.

Models used: GPT-3.5 · GPT-4 · GPT-4-32k · Claude · Claude 2 · Pinecone vector store · LangChain components

Scale

• Credal's lessons came from 250k+ LLM calls and 100k corporate documents.
  “Takeaways & lessons from 250k+ LLM calls on 100k corporate docs”article, stated
• The customer environments included enterprises with thousands of employees.
  “we’ve been extremely lucky to work with some amazing Enterprises, with thousands of employees and tons of complex, real-world data”article, stated
• A single long-document Claude query could cost more than $1, and chunked approaches could reach $2 per chunk.
  “If we use Claude on the entire document, each question could cost more than $1 to answer.”article, stated

  “But that can get very expensive; up to a $2 per chunk depending on the model”article, stated
• The Slack AI-expert router needed 95%+ accuracy.
  “we needed an initial step that could triage the user question to the right AI expert with 95%+ accuracy”article, stated
• Calling GPT-4 for every message in a 5,000-person Slack channel was too costly and slow.
  “making a GPT-4 call on every message posted to a 5000 person company Slack channel is... painful”article, stated

Alternatives considered (roads not taken)

Use out-of-the-box LangChain document loaders and parsers

Rejected because: Credal says standard loaders lose structure such as footnote placement and table layout, and LangChain's structured parser distracted GPT-3.5.

Use whole-document context for summary questions

Rejected because: Claude's context window can work for one document but is expensive and does not scale to many documents or very large corpora.

Use LangChain StructuredOutputParser in the triager prompt

Rejected because: The formatting instructions consumed attention and dropped GPT-3.5 matching accuracy to 50%.

Use a single GPT-4 call for routing and formatting

Rejected because: Two focused GPT-3.5 calls were faster and cheaper when JSON repair was needed.

Patterns

rag-retrieval (retrieval)

Credal retrieves focused document sections for user questions, but augments naive vector search with formatting and metadata tags.

data-quality-loop (ingestion)

Credal restructures footnotes, tables, and section metadata before retrieval so the LLM receives semantically meaningful context.

human-in-the-loop-gate (data-curation)

Human experts define the important tags and can verify model outputs, while LLMs perform scalable reading and tagging.

prompt-chaining (prompting)

Credal splits matching and JSON formatting into sequential GPT-3.5 calls with a parsing check between them.

structured-output-contract (integration)

The AI-expert router must return parseable structured data so code can decide which expert to call.

cost-aware-routing (model-selection)

Credal uses GPT-3.5 and focused prompt design for high-volume Slack routing because GPT-4 on every message would be too expensive and slow.

Tradeoffs

Summary retrieval strategy

Chose LLM-generated metadata tags and pre-filtered RAG over plain semantic search over the whole document.

Rationale: The relevant summary section may not contain the words in the user's query, so tags identify the right kind of section before search.

Prompt design for routing

Chose few-shot examples and core matching instructions as most of the prompt over long structured-format instructions.

Rationale: The routing decision was the hard part; JSON formatting mattered but should not consume attention needed for matching.

Model cost strategy

Chose focused GPT-3.5 calls with a second formatting call when needed over one GPT-4 call for every Slack message.

Rationale: GPT-3.5 with focused prompting and a repair call was faster and cheaper while avoiding GPT-4 cost on high-volume channels.

Gotchas (what bit them)

Embedding search cannot recover relationships that were lost in parsing.

Root cause: Footnote content was moved away from the quoted text, so author/citation information was not semantically near the question target.

Resolution: Inline footnote details where the reference appears before indexing or sending context to the LLM.

Bad table formatting causes reasoning failures even with strong models.

Root cause: Raw parsed tables collapse rows and columns into confusing text, making date and count reasoning unreliable.

Resolution: Represent tables clearly, such as converting them to CSV before model input.

Formatting instructions can steal model attention from the actual task.

Root cause: StructuredOutputParser instructions took 10-20 lines and competed with examples for GPT-3.5's limited attention.

Resolution: Remove heavy formatting instructions, embed JSON naturally in examples, and repair malformed JSON in a second focused call.

Full-context summarization can work for one document but fails economically and structurally at enterprise scale.

Root cause: Large context windows are finite and expensive, and enterprise corpora include thousands of documents and huge systems like Drive, Confluence, Slack, and SharePoint.

Resolution: Use RAG plus per-use-case data representation and prompting rather than reading entire corpora.

Evaluation

Credal reports empirical evaluation through answer failures on example questions, model comparisons on table/date reasoning, Slack triager accuracy targets, JSON parsing checks, and human verification for answers where needed.

Metrics: 95%+ routing accuracy target · 80% precision and 95% recall target for the triager · 50% GPT-3.5 accuracy under the structured-output prompt · JSON parse success · answer correctness on summary, citation, and table questions · cost and latency per routing call

Outcomes

• Credal concluded that production RAG requires careful data formatting and per-use-case representation
• inlining footnotes turned an impossible citation question into an easy one
• CSV table representation enabled GPT-4-32k to answer a table question accurately
• LLM-generated metadata tags improved retrieval for summary-style questions
• focused sequential GPT-3.5 calls were cheaper and faster than a single GPT-4 call for routing and formatting

The lessons come from 250k+ LLM calls, 100k corporate documents, enterprise customers with thousands of employees, and high-volume Slack-channel routing.