Building an Enterprise RAG-based AI Assistant with Vector Search and LLM Integration

elastic · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise knowledge assistant

Elastic needed an internal GenAI assistant that could answer employee questions from trusted internal data while preserving privacy, search quality, logging, and enterprise deployment controls.

Constraints:

• The assistant needed access to internal wiki and ServiceNow content without exposing user chats indefinitely.
• Retrieval quality depended on semantic search over embedded chunks.
• The organization needed a UI that fit Elastic's product ecosystem and security posture.
• The RAG pipeline had to run as a production service, not a notebook demo.

Architecture

ElasticGPT uses SmartSource, a RAG framework backed by Elasticsearch vector search and storage. LangChain chunks internal Wiki and ServiceNow data, generates embeddings, retrieves relevant context from Elasticsearch, and crafts prompts for GPT-4o or GPT-4o-mini hosted on a dedicated Azure tenant. Kubernetes in Elastic Cloud hosts the application, while Elasticsearch logs chat interactions and deletes user chat data after 30 days.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  employee[Employee] --> elasticgpt[ElasticGPT]
  elasticgpt --> smartsource[SmartSource]
  smartsource --> elasticsearch-vector-store[Elasticsearch Vector Store]
  smartsource --> langchain-pipeline[LangChain Pipeline]
  langchain-pipeline --> openai-azure-tenant[OpenAI Azure Tenant]
  elasticgpt --> elasticsearch-chat-store[Elasticsearch Chat Store]
  elastic-cloud-kubernetes[Elastic Cloud Kubernetes] --> elasticgpt[ElasticGPT]

Components

Data flow

Internal data is chunked and embedded into Elasticsearch. When a user asks a question, SmartSource retrieves relevant context, LangChain builds the prompt, OpenAI models generate the answer, and Elasticsearch stores chat logs and metrics under a retention policy.

Why this architecture

Elastic chose Elasticsearch as both vector/search backbone and chat repository because the assistant's core requirement was retrieval over internal knowledge plus auditable, privacy-aware logging. LangChain supplied the RAG orchestration around chunking, embeddings, retrieval, and prompt construction.

Models used: GPT-4o · GPT-4o-mini · OpenAI embeddings

Scale

• User chat data is deleted every 30 days while metrics are retained.
  “application performance monitoring (APM) keeps tabs on performance, reliability, and resource utilization, ensuring response times stay lightning-fast as adoption scales. User chat data is deleted every 30 days, with only metrics saved, enabling us to retain the most relevant data cost”article, stated
• The assistant is hosted on Elastic Cloud.
  “Hosted on Elastic Cloud, this setup offers the flexibility, scalability, and performance needed to support a growing internal user base.”article, stated
• The knowledge base includes Elastic Wiki and ServiceNow content.
  “SmartSource, our name for our internal model, taps into Elasticsearch’s vector database to store embeddings — numerical representations of our internal data sourced from Elastic’s Wiki, ServiceNow Knowledge Articles, ServiceNow News Articles, and beyond.”article, stated

Alternatives considered (roads not taken)

Use Hugging Face Chat UI as the primary interface.

Rejected because: Elastic switched to EUI to better fit internal product patterns.

Use a generic chatbot without internal retrieval.

Rejected because: The assistant needed SmartSource to retrieve trusted internal context.

Store chat data without a retention boundary.

Rejected because: Elastic chose 30-day deletion for user chat data, retaining only metrics.

Patterns

rag-retrieval (knowledge)

SmartSource retrieves internal context before the model generates responses.

retrieval-ranking (search)

Elasticsearch vector search retrieves relevant chunks from embedded internal data.

privacy-preserving-processing (governance)

Chat content is retained only temporarily while metrics persist.

observability-tracing (operations)

Every interaction is logged in Elasticsearch for analysis and operations.

Tradeoffs

Search backbone

Chose Elasticsearch for vector search and chat storage over Separate vector database and logging store.

Rationale: One platform could serve retrieval and chat repository needs.

Interface stack

Chose EUI-aligned internal UI over Hugging Face Chat UI.

Rationale: Elastic needed a UI consistent with its ecosystem and internal deployment.

Model hosting

Chose Dedicated Azure tenant for OpenAI models over Unscoped public model access.

Rationale: Enterprise use required private, secure model access.

Gotchas (what bit them)

Chat UI choice can become a product-integration problem.

Root cause: A generic UI may not match the organization's design and support expectations.

Resolution: Switch from Hugging Face Chat UI toward Elastic UI.

Enterprise assistant logs need retention limits.

Root cause: Employee chats can contain sensitive internal information.

Resolution: Delete user chat data after 30 days and retain only metrics.

Evaluation

Elastic evaluates ElasticGPT through interaction logs, saved metrics, adoption of internal knowledge retrieval, and operational effects such as reducing redundant IT questions.

Metrics: Chat interaction metrics · IT query reduction · Employee efficiency · Data retention compliance · Retrieval coverage

Outcomes

• ElasticGPT provides internal RAG over Elastic Wiki and ServiceNow data.
• Elasticsearch serves as both vector database and chat-data repository.
• The system supports private GPT-4o/GPT-4o-mini access through a dedicated Azure tenant.

The source reports a 30-day chat retention period, Kubernetes hosting in Elastic Cloud, and internal sources including Elastic Wiki and ServiceNow knowledge/news articles.