Building Production-Grade Heterogeneous RAG Systems

rag-retrieval (architecture)

The system retrieves from text knowledge bases, structured data, multimodal image indexes, and caption indexes before answering.

"text-based RAG that retrieves relevant documents and uses an LLM to generate an answer based on textual information." — article, stated "As with traditional RAG pipelines, the retrieval component is the basis of these solutions." — article, stated

tool-use-orchestration (architecture)

The article proposes Bedrock tool use/function calling for routing and code-generation pipelines that run Python or SQL against data stores.

"An alternative approach to routing is to use the native tool use capability (also known as function calling) available within the [Bedrock Converse API](https:" — article, stated "asking the LLM to write Python (if the data is stored in a CSV, Excel, or Parquet file) or SQL" — article, stated

guardrails-io-filtering (safety)

Custom code-generation pipelines can sanity-check generated code and avoid sending large code outputs back to the LLM when that would increase latency or hallucination risk.

"Writing your custom pipelines also allows you to perform some sanity checks on the code" — article, stated "Directly returning the variable to the user removes the risk of hallucination" — article, stated

latency-optimization (operations)

The router uses a smaller model to reduce routing latency, and direct code-output return avoids extra LLM calls for large tabular results.

"To avoid adding too much latency with the routing step, we recommend using a smaller model" — article, stated "the additional call might cause high latency and even risks causing hallucinations." — article, stated

cost-aware-routing (operations)

The multimodal strategy weighs direct multimodal embeddings against multimodal LLM captioning based on speed, cost, detail, and customization needs.

"Multimodal embeddings offer faster ingestion and are generally more cost-effective" — article, stated "though slower and less cost-effective, provides more detailed and customizable results" — article, stated

Tradeoffs

Use an LLM router before retrieval.

Chose Intent detection with a small routing model over Sending all queries to one retrieval pipeline.

Rationale: The router makes sure each data type is processed appropriately, but the routing step must use a small model to avoid adding too much latency.

"In systems with heterogeneous data, this process makes sure each data type is processed appropriately, whether it’s unstructured text, structured tables, or multimodal content." — article, stated "To avoid adding too much latency with the routing step, we recommend using a smaller model" — article, stated

Use generated code for structured data analysis.

Chose Python or SQL generation plus execution over Putting the whole table into the prompt.

Rationale: Code execution handles quantitative filtering, aggregation, and plotting better than raw table prompting for complex or large data.

"On the other hand, LLMs are notably good at code generation; for instance, [Anthropic’s Claude Sonnet 3.5 has 92% accuracy](https:" — article, stated "we can’t reliably use an LLM to analyze tabular data, even when provided as structured format in the prompt." — article, stated

Return some code outputs directly instead of asking the LLM to narrate them.

Chose Direct UI return for large filtered dataframe results over Always sending code output back to the LLM.

Rationale: Direct return is more efficient and avoids hallucination risk when the result is already the requested data.

"In that case, a custom pipeline that directly returns the code output to the UI (the filtered dataframe for the date of June 1st 2024, with oil production greater than 0) would be more efficient than sending it to the LLM for a final answer." — article, stated "Directly returning the variable to the user removes the risk of hallucination" — article, stated

Choose multimodal embeddings for broad visual similarity.

Chose Amazon Titan Multimodal Embeddings in a shared vector space over Caption-first retrieval for every image.

Rationale: Direct multimodal embeddings are faster and more cost-effective for high-level image/text matching.

"Titan Multimodal Embeddings](https://docs.aws.amazon.com/bedrock/latest/userguide/titan-multiemb-models.html), which can embed both images and text into a shared vector space. This allows for direct comparison and retrieval of text and images based on semantic similarity. This simple approach" — article, stated "This simple approach is effective for finding images that match a high-level description" — article, stated

Use caption-first retrieval for detailed visual understanding.

Chose Claude image captions plus Titan Text Embeddings over Only comparing raw image/text embeddings.

Rationale: Generated captions can focus on specific visual attributes and capture diagram complexity, at the cost of an extra LLM call and slower ingestion.

"generated image description is embedded using a traditional text embedding" — article, stated "This would also work better to capture the complexity of diagrams in industrial maintenance." — article, stated

Gotchas (what bit them)

A single retriever will mis-handle heterogeneous data.

Root cause: Tables, documents, images, and multimodal questions require different retrieval and processing logic.

Resolution: Use a router to dispatch to specialized pipelines for text RAG, structured data/code generation, and multimodal retrieval.

"different data types often require distinct retrieval and processing strategies." — article, stated "whether it’s unstructured text, structured tables, or multimodal content." — article, stated

Prompting directly over tables breaks down for complex quantitative analysis.

Root cause: LLMs do not reliably analyze raw tables in prompts unless the data is small and reasoning is simple.

Resolution: Ask the LLM to generate Python or SQL, execute the code, and return the computed result.

"LLMs don’t perform well at analyzing tabular data when it’s added directly in the prompt as raw text." — article, stated "asking the LLM to write Python (if the data is stored in a CSV, Excel, or Parquet file) or SQL" — article, stated

Large tabular outputs can create latency and hallucination risk when re-fed to the LLM.

Root cause: A large filtered dataframe adds another LLM call and large input/output surface.

Resolution: Return code output directly to the UI for large filtering tasks when a natural-language final answer is unnecessary.

"If the filtered dataframe is large, the additional call might cause high latency and even risks causing hallucinations." — article, stated "Directly returning the variable to the user removes the risk of hallucination that can occur with large inputs and outputs." — article, stated

Generated code can damage files or databases if executed blindly.

Root cause: LLM-written Python or SQL may contain operations that modify existing files or databases.

Resolution: Build custom pipelines with code sanity checks before execution.

"perform some sanity checks on the code, to verify, for instance, that the code generated by the LLM will not create issues" — article, stated "Writing your custom pipelines also allows you to perform some sanity checks on the code, to verify, for instance, that the code generated by the LLM will not create issues (such as modify existing files or data bases)." — article, stated

Multimodal embedding and captioning approaches optimize for different things.

Root cause: Direct multimodal embeddings are efficient but less customizable; captions add an LLM call but capture finer, prompted details.

Resolution: Choose direct multimodal embeddings for quick broad matching, and caption-based retrieval for detailed or aspect-specific searches.

"Captions can capture finer details in images, and can be guided to focus on specific aspects such as color, fabric, pattern, shape, and more." — article, stated "Considerations such as latency for different input types, customization needs, and the level of detail required in the output should guide the decision-making process" — article, stated

Evaluation

The article evaluates by decomposing known customer-use-case classes and comparing design choices rather than reporting benchmark results for a deployed system. Its evaluation guidance is qualitative and architectural: route correctness, parseable structured outputs, code sanity, latency, hallucination risk, retrieval fit for each modality, and the comparative cost/speed/detail tradeoffs of multimodal embeddings versus caption-based retrieval.

Metrics: Router category accuracy and debuggability through explanation tags · Clarification rate for ambiguous queries · Latency added by the routing step · Correctness of generated Python or SQL analysis code · Hallucination risk when large code outputs are sent back to the LLM · k-NN retrieval results in OpenSearch · Multimodal embedding ingestion speed and cost · Captioning detail and customization quality · Image-input latency with or without extra caption-generation calls · Suitability for text-to-image, image-to-text, and image-plus-text retrieval

"Prompting the LLM to explain the routing logic may help with accuracy, by forcing the LLM to “think” about its answer, and also for debugging purposes" — article, stated "ask for clarifications in the router, if the LLM finds that the query is too ambiguous." — article, stated "The closest images are then returned. ![This diagram shows the retrieval of images from a user query in a vector database using a multimodal embedding." — article, stated "Multimodal embeddings offer faster ingestion and are generally more cost-effective" — article, stated "as multimodal LLM Same as multimodal embeddings Image Input Latency Faster, no extra processing required Slower, requires extra LLM call to generate image caption Best Use Case General use, quick and efficient data handling Precise searches needing detailed image descriptions ##" — article, stated

Outcomes

• AWS GenAIIC reports building heterogeneous RAG systems across field engineering, oil and gas, financial analysis, industrial maintenance, and ecommerce product search.
• The guide establishes a modular router pattern for routing queries to text, structured data, multimodal, or clarification pipelines.
• The structured-data pattern uses LLM-generated Python/SQL plus execution, with optional direct result return to avoid latency and hallucination.
• The multimodal pattern offers two production options: direct multimodal embeddings or multimodal-caption plus text-embedding retrieval.
• The article does not disclose named customer deployments or quantitative production outcomes.

Concrete scale signals include five use-case families, hundreds of dataframe entries in the oil-and-gas example, 92% HumanEval cited for Claude Sonnet 3.5 code generation, k=5 nearest-neighbor retrieval examples, and three multimodal question categories.

"Use cases we have worked on include: * Technical assistance for field engineers – We built a system that aggregates information about a company’s specific products and field expertise." — article, stated "Building real-world RAG systems with heterogeneous data formats presents unique challenges" — article, stated "By employing techniques like intent detection, code generation, and multimodal embeddings, you can create intelligent systems" — article, stated "enabling seamless integration of images and other media into your knowledge bases." — article, stated "specific company names and quantitative results are not disclosed, which limits the ability to verify claims independently." — summary, stated