Building a Production RAG-based Customer Support Assistant with Elasticsearch

elastic · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support RAG over Elastic product knowledge

Elastic needed its Support Assistant to answer technically deep customer questions from support articles, product documentation, blogs, and internal knowledge while preserving role-based access and keeping retrieval precise enough that the LLM would not use irrelevant context.

Constraints:

• Foundational LLM training was insufficient for Elastic's product depth and breadth
• Fine-tuning would require question-answer pairs that did not match the available unstructured support corpus
• Enterprise support required role-based document access across internal and external audiences
• The knowledge library spanned hundreds of thousands of documents with inconsistent metadata
• The ingestion pipeline had to keep documentation current across Elastic properties and versions
• Long or irrelevant retrieved passages reduced answer accuracy

Architecture

Elastic built the Support Assistant around a search-first RAG knowledge library in Elasticsearch. Support articles, product docs, blogs, labs content, and onboarding guides feed a unified index with document-level security. Crawlee jobs on Google Cloud Run ingest large site maps, ELSER enriches title and summary fields for semantic retrieval, BM25 scores larger content fields, and OpenAI-generated summaries/questions backfill missing metadata. The assistant retrieves concise, role-appropriate passages and sends them to the LLM as context.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  support-assistant[Support Assistant] --> hybrid-retriever[Hybrid Retriever]
  hybrid-retriever --> knowledge-library-index[Knowledge Library Index]
  support-articles[Support Articles] --> knowledge-library-index
  support-articles --> private-context-plugin[Private Context Plugin]
  documentation-crawlers[Documentation Crawlers] --> request-handlers[Request Handlers]
  request-handlers --> knowledge-library-index
  knowledge-library-index --> elser-semantic-search[ELSER Semantic Search]
  elser-semantic-search --> hybrid-retriever
  summary-enrichment-service[Summary Enrichment Service] --> openai-gpt[OpenAI GPT]
  summary-enrichment-service --> knowledge-library-index
  hybrid-retriever --> openai-gpt
  openai-gpt --> support-assistant

Components

Data flow

Support Engineers author knowledge articles and Elastic crawls official documentation/blog sources into Elasticsearch. Request handlers normalize page bodies, document-level security controls which records can be retrieved, ELSER creates semantic embeddings for searchable fields, OpenAI GPT-3.5 Turbo backfills summaries and questions where metadata is missing, and the Support Assistant uses hybrid search over title, summary, content, and generated question fields to assemble compact context for OpenAI.

Why this architecture

Elastic framed the assistant as a retrieval and search-relevance problem, not a model-customization problem. RAG fit existing unstructured knowledge, gave immediate updates, enabled document-level access control, and reused Support Hub search infrastructure. Hybrid retrieval and AI-generated summaries targeted the concrete failure mode the team observed: too much irrelevant context made the LLM less accurate.

Models used: ELSER · OpenAI GPT3.5 Turbo · OpenAI GPT

Scale

• The Elastic technical support knowledge base contained more than 2,800 articles.
  “Even though our technical support knowledge base has over 2,800 articles, we knew that there would be questions that these would not answer for users of the Elastic Support Assistant.”article, stated
• The full knowledge library consisted of over 300,000 documents.
  “Our entire knowledge library at Elastic consists of over 300,000 documents. The documents varied widely in the type of metadata they had, if any at all.”article, stated
• The enrichment service produced over 128,000 AI-generated summaries.
  “of writing, we have vector embeddings for the more than 300,000 documents in our indices and over 128,000 ai-generated summaries with an average of 8 questions per document. Given that we only have ~8,000 technical support articles with human-written summaries, this was a 10x improvement for our”article, stated
• Elastic reported a 10x improvement over roughly 8,000 human-written technical-support summaries.
  “Given that we only have ~8,000 technical support articles with human-written summaries, this was a 10x improvement”article, stated
• Cloud Run hosted four crawler jobs in parallel.
  “Our needs resulted in a total of four jobs running in parallel, each with a base url that would capture a specific category of documents.”article, stated

Alternatives considered (roads not taken)

Fine-tune a Support Assistant model

Rejected because: Elastic's available support data was unstructured and not in question-answer pairs, and the product knowledge needed fast updates and access filtering.

Keep separate Swiftype and AppSearch article stores

Rejected because: The split was technical debt once Elasticsearch could provide the features needed for both internal and customer-facing articles.

Use the first 280 characters as document summaries

Rejected because: The simple heuristic produced poor search relevance, so Elastic used LLM-generated summaries and questions instead.

Crawl broad top-level site roots

Rejected because: Elastic needed technical documents and wanted to avoid duplicate or unhelpful content.

Patterns

rag-retrieval (knowledge-access)

Elastic retrieves concise, role-appropriate passages from a large support and documentation library before sending context to the LLM.

hybrid-search (retrieval-ranking)

The Support Assistant combines semantic retrieval over title/summary fields with BM25 over larger content fields.

privacy-preserving-processing (data-access)

Document-level security filters retrieved context by the user's role before it is sent to OpenAI.

batch-offline-processing (data-enrichment)

Elastic backfilled missing metadata across the knowledge library with Cloud Run jobs and controlled concurrency.

data-quality-loop (retrieval-quality)

Elastic analyzed user trends and search gaps to decide when to add new sources and enrich existing data.

Tradeoffs

Use RAG instead of fine-tuning

Chose retrieval over Elastic knowledge sources over fine-tuning a custom Support Assistant model.

Rationale: RAG matched unstructured data, supported real-time updates, enabled role-based access, and reused Support Hub search infrastructure.

Use Cloud Run crawler jobs

Chose scheduled Cloud Run jobs with Crawlee over Cloud Tasks or PubSub orchestration for this ingestion stage.

Rationale: Cloud Run jobs supported 24-hour timeouts, scheduling, and parallel category crawls that matched Elastic's needs.

Use generated summaries and questions for retrieval metadata

Chose OpenAI GPT-3.5 Turbo summaries/questions over first-280-character summaries.

Rationale: The heuristic was simpler but damaged search relevance; generated summaries improved semantic search at the required scale.

Backfill current product docs first

Chose controlled Cloud Run concurrency and weeks-long backfill over fully saturating the LLM instance for immediate backfill.

Rationale: Elastic avoided starving the Support Assistant while enriching a large corpus.

Gotchas (what bit them)

Large context harmed answer accuracy.

Root cause: The LLM could miss key sentences when Elastic passed larger passages rather than precise context.

Resolution: Elastic tuned retrieval around concise, relevant passages and expanded hybrid search quality.

Crawler body extraction could pollute retrieval with filler text.

Root cause: Elastic's sources had varied document structures and irrelevant page text could become searchable.

Resolution: They created request handlers per document type to parse the relevant body CSS.

Product documentation versions created duplication and version-specific retrieval risk.

Root cause: Users might ask about any Elastic version, while product docs duplicate unchanged pages across versions.

Resolution: Elastic captured enough versioned documentation and tuned queries to default to current docs unless specified.

Metadata gaps limited semantic search over ingested documents.

Root cause: Many documents lacked useful summaries even though ELSER depended on semantic fields such as title and summary.

Resolution: Elastic generated summaries and questions with GPT-3.5 Turbo and stored them in document fields.

Evaluation

Elastic evaluated retrieval quality through its core hypothesis that concise, relevant search context should improve user sentiment and reduce irrelevant-context usage. The team compared summary strategies, monitored search relevance, tracked corpus coverage, and used generated question fields for semantic-search suggestions.

Metrics: positive user sentiment · search relevancy · document count · AI-generated summary count · average questions per document · technical support articles with human-written summaries · context response time

Outcomes

• Elastic established a RAG knowledge library for Support Assistant over support articles, product docs, blogs, labs content, and onboarding guides.
• The support architecture moved from separate article stores to a single role-secured Elasticsearch index.
• Hybrid search used ELSER on title/summary and BM25 on larger content to balance speed and relevance.
• Cloud Run crawler jobs ingested hundreds of thousands of documents from complex Elastic site maps.
• OpenAI GPT-3.5 Turbo generated over 128,000 summaries with about eight questions per document.

The system operated over more than 300,000 indexed documents, including over 2,800 technical support articles, and produced over 128,000 AI-generated summaries with an average of eight questions per document.