Building a Secure Enterprise AI Assistant with RAG and Custom Infrastructure

hexagon · Tech · 2025 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise asset-management AI assistant

Hexagon wanted an AI-powered digital worker for enterprise asset-management products that could answer product questions, support internal workflows, and operate in high-security customer environments without losing domain relevance, tenant isolation, or lifecycle control.

Constraints:

• Enterprise GenAI needed security, accuracy, and sustainability rather than consumer-style broad access
• The assistant had to support high-security customer environments
• Domain-specific product documentation was needed to avoid hallucinations
• Documents contained complex structures such as tables and varied formats
• Generative responses had variable latency and quality compared with traditional APIs
• The platform needed regional failover, multilingual capability, and model lifecycle management

Architecture

Hexagon built HxGN Alix as a digital worker for asset lifecycle intelligence using AWS generative AI services and custom infrastructure. The stack runs open-source Mistral NeMo on Amazon EKS with G6e GPU instances, uses Bedrock Knowledge Bases and S3 for RAG over product documentation, applies Bedrock Guardrails and prompt management, preserves data privacy by not storing user interactions, and uses Amazon Bedrock as a multi-region fallback path for regional failures.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  hxgn-alix[HxGN Alix] --> mistral-nemo[Mistral NeMo]
  hxgn-alix[HxGN Alix] --> bedrock-knowledge-bases[Bedrock Knowledge Bases]
  hxgn-alix[HxGN Alix] --> bedrock-guardrails[Bedrock Guardrails]
  hxgn-alix[HxGN Alix] --> amazon-bedrock[Amazon Bedrock]
  amazon-eks[Amazon EKS] --> mistral-nemo[Mistral NeMo]
  amazon-eks[Amazon EKS] --> g6e-gpu-instances[G6e GPU Instances]
  bedrock-knowledge-bases[Bedrock Knowledge Bases] --> amazon-s3[Amazon S3]
  amazon-bedrock[Amazon Bedrock] --> prompt-management[Prompt Management]

Components

Data flow

A user asks HxGN Alix a product or workflow question. The assistant uses product documentation stored in S3 and retrieved through Bedrock Knowledge Bases, with parsing, hierarchical chunking, FAQs, and special instructions to preserve context. The application routes generation to Mistral NeMo running on EKS/G6e infrastructure, uses Bedrock Guardrails for grounding and safety checks, and can fall back to Bedrock multi-region endpoints if the primary region fails.

Why this architecture

Hexagon chose an enterprise-specific digital worker architecture because consumer GenAI lacked the domain control, security, and integration needed for EAM products. Open-source LLM hosting gave lifecycle and cost control, Bedrock Knowledge Bases supplied managed RAG features for product documentation, Guardrails addressed safety and grounding, and Bedrock fallback reduced regional availability risk.

Models used: Mistral NeMo · Mistral 7B · Amazon Bedrock foundation models

Scale

• Mistral NeMo is a 12-billion-parameter open-source model.
  “Mistral NeMo, a 12-billion parameter open source LLM built in collaboration with NVIDIA and released under the Apache 2.0 license.”article, stated
• The selected model supports a context window up to 128,000 tokens.
  “Mistral NeMo offers a large context window of up to 128,000 tokens and is designed for global, multilingual applications.”article, stated
• Hexagon selected G6e.48xlarge instances with NVIDIA L40S GPUs.
  “We selected G6e.48xlarge instances powered by NVIDIA L40S GPUs—the most cost-efficient GPU instances for deploying generative AI models under 12 billion parameters.”article, stated
• LLM latency ranged from 1 to 60 seconds depending on query.
  “Responses from LLMs can vary significantly in latency, ranging from 1–60 seconds depending on the user’s query”article, stated

Alternatives considered (roads not taken)

Use consumer generative AI patterns

Rejected because: Enterprise use required scalability, security, integration, proprietary data, and control over intellectual property.

Pretrain the model on internal documentation

Rejected because: The team was early in development and lacked enough data, while RAG could ground answers in retrieved context.

Use chat history as the only follow-up retrieval strategy

Rejected because: Context-based retrieval was faster but prompt-based reformulation worked better for large document sets.

Patterns

rag-retrieval (retrieval)

HxGN Alix uses Bedrock Knowledge Bases to retrieve product documentation and ground answers.

guardrails-io-filtering (safety)

Amazon Bedrock Guardrails applies filtering, grounding checks, and safety policies to AI responses.

fallback-degradation (resilience)

The system can fall back to Bedrock multi-region endpoints when regional failures occur.

prompt-versioning (operations)

Bedrock Prompt Management is used for prompt creation, evaluation, versioning, and sharing.

Tradeoffs

Host an open-source LLM

Chose Mistral NeMo on EKS and G6e GPUs over Using only commercial LLM APIs.

Rationale: Open-source hosting gave Hexagon control over lifecycle, customization, licensing, cost, and data privacy.

Use RAG rather than pretraining

Chose Bedrock Knowledge Bases over stored product documentation over Pretraining the model with internal documentation.

Rationale: The team lacked enough early training data and wanted answers grounded in accurate retrieved context.

Choose prompt-based search reformulation for large document sets

Chose Additional LLM call to create a more complete retrieval query over Sending the last five chat messages directly to the knowledge base.

Rationale: Prompt reformulation kept prompts concise and yielded highly relevant results for large document sets, while chat-history retrieval was faster for smaller sets.

Gotchas (what bit them)

Base models hallucinated without product-specific grounding.

Root cause: Generic models had limited knowledge of Hexagon's products.

Resolution: The team implemented RAG over product documentation instead of relying on the base model alone.

Chunking complex documentation could destroy context.

Root cause: Tables and complex structures lost relational information when split into vector chunks.

Resolution: Hexagon used hierarchical chunking in Bedrock Knowledge Bases to preserve important relationships.

Follow-up questions could fail retrieval when processed alone.

Root cause: Short follow-ups such as 'Can you elaborate?' lacked enough standalone context for RAG search.

Resolution: The team implemented mechanisms to maintain conversational context for follow-ups.

Traditional testing did not fit generative AI behavior.

Root cause: Prompts return variable outputs and LLM latency varies by query.

Resolution: Hexagon introduced new testing, QA, monitoring, and optimization processes.

Evaluation

Hexagon evaluated through a proof of concept, internal-user release, backlog discovery, RAG strategy experiments, and adjusted QA/monitoring methods for LLM variability.

Metrics: small internal proof-of-concept release · latency variability from 1 to 60 seconds · comparison of two retrieval approaches · security, accuracy, and sustainability requirements

Outcomes

• HxGN Alix launched as a secure AI-powered digital worker for EAM product workflows
• The proof of concept surfaced security, architecture, network, prompt, and product-integration backlog items
• The team implemented RAG, grounding checks, prompt management, and monitoring changes for production readiness
• The architecture supports regional fallback through Bedrock multi-region endpoints

The system uses a 12B-parameter Mistral NeMo model with a 128,000-token context window, G6e.48xlarge GPU instances, and latency observations ranging from 1 to 60 seconds.