Tuning RAG Search for Customer Support Chatbot

elastic · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support RAG search relevance

Elastic's Support AI Assistant had a working RAG chatbot, but search relevance failed for CVEs, product-version queries, semantic matching, and duplicate content, weakening the context available to GPT-4.

Constraints:

• The RAG corpus contained over 300,000 support and documentation documents.
• Only the top 3 search results were passed to GPT-4 because of token constraints and scale.
• CVE and product-version queries needed precise lexical handling.
• Document summaries initially produced poor embeddings.
• Relevance improvements had to be measured against curated test cases.

Architecture

Elastic tuned the RAG retrieval layer with hybrid BM25 plus ELSER semantic search, conditional boosting for CVE/title and version fields, GPT-4-generated field enrichment, collapse-based deduplication, and Precision@K evaluation against curated support queries. Search results fed a GPT-4 generation pipeline through Azure OpenAI.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  support_content_corpus["Support content corpus"] --> elasticsearch_index["Elasticsearch index"]
  support_content_corpus --> gpt_4_enrichment_pipeline["GPT-4 enrichment pipeline"]
  gpt_4_enrichment_pipeline --> elasticsearch_index
  elasticsearch_index --> hybrid_search_query["Hybrid search query"]
  hybrid_search_query --> gpt_4_answer_generator["GPT-4 answer generator"]
  gpt_4_answer_generator --> support_chat_interface["Support chat interface"]
  support_chat_interface --> feedback_analytics["Feedback analytics"]
  feedback_analytics --> evaluation_test_cases["Evaluation test cases"]
  evaluation_test_cases --> hybrid_search_query

Components

Data flow

Elastic crawls support articles, documentation, and blogs into Elasticsearch, enriches fields such as summaries, builds hybrid BM25/ELSER search queries, boosts title or version fields conditionally, collapses duplicate-like content, passes the top 3 results into a system prompt, and sends context plus conversation to GPT-4.

Why this architecture

Elastic treated RAG quality as a search-tuning problem: semantic retrieval alone was not enough for CVE codes, product versions, stale summaries, and duplicate content.

Models used: ELSER · GPT-4 · Azure OpenAI

Scale

• The system indexed over 300,000 documents.
  “over 300,000 documents including Technical Support Knowledge Articles, Product Documentation, and blog content crawled from Elastic's website.”summary, stated
• Only the top 3 results were used as generation context.
  “The limitation to only 3 results was driven by token constraints in their dedicated deployment combined with a large user base.”summary, stated
• Search tuning improved top-3 relevance by about 75%.
  “they achieved a ~75% improvement in top-3 search relevance”summary, stated
• The enrichment generated more than 300,000 AI summaries.
  “generating over 300,000 AI-generated summaries for future applications.”summary, stated

Alternatives considered (roads not taken)

Semantic search alone

Rejected because: CVE and version queries needed targeted lexical and field boosts.

Use crawled first-body-text summaries as embeddings

Rejected because: They poorly represented product documentation content.

Patterns

hybrid-search (retrieval)

Elastic combined BM25 keyword search with ELSER semantic retrieval.

retrieval-ranking (search tuning)

Conditional boosts and collapse deduplication shaped which documents appeared in the top 3.

eval-driven-development (search relevance)

Precision@K against curated test cases guided relevance improvements.

data-quality-loop (field enrichment)

GPT-4 generated better summaries to improve semantic retrieval quality.

Tradeoffs

Use hybrid retrieval.

Chose BM25 plus ELSER over Pure keyword or pure semantic retrieval.

Rationale: Support queries mix exact identifiers, versions, and semantic intent.

Limit generation context to top 3 results.

Chose Small high-precision context over Passing more retrieved documents.

Rationale: Token constraints and user scale made precision in the top 3 essential.

Gotchas (what bit them)

Semantic embeddings can fail on exact identifiers.

Root cause: CVE codes and product versions carry lexical structure that semantic matching may underweight.

Resolution: Detect identifier patterns and boost title/version fields conditionally.

Bad summary fields poison semantic retrieval.

Root cause: Crawler-generated summaries duplicated first body text rather than representing content.

Resolution: Use AI enrichment to generate better summaries.

Small top-k context makes search mistakes expensive.

Root cause: Only three results could fit into the prompt under token constraints.

Resolution: Evaluate and tune top-3 precision directly.

Evaluation

Elastic evaluated search tuning with Precision@K against curated test cases and feedback-derived query patterns.

Metrics: ~75% improvement in top-3 relevance. · Over 300,000 indexed documents. · Over 300,000 AI-generated summaries. · Top 3 search results used as LLM context.

Outcomes

• Top-3 search relevance improved by about 75%.
• The chatbot gained better retrieval for CVE and product-version queries.
• The team generated more than 300,000 AI summaries for future applications.

The system operated over a corpus of more than 300,000 support, documentation, and blog documents.