Building a Context-Aware AI Assistant with RAG for Developer Support

vectorize · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Developer-support RAG assistant

Vectorize needed to answer product-support questions inside its application so users did not have to leave the UI for documentation or wait for human chat support.

Constraints:

• users frequently ask questions that are already answered in documentation
• documentation lives outside the product UI and creates context switching
• docs and support answers change frequently
• retrieval must avoid low-relevance chunks that confuse the LLM
• the generated answer must avoid hallucinating when no relevant documentation exists

Architecture

Vectorize embedded an AI assistant React component into the product UI and backed it with its own RAG pipeline. The pipeline ingests documentation plus approved Discord and Intercom support answers into a vector database, retrieves with the same embedding model, reranks by relevance, adds page topic context to the user question, and sends filtered context to Llama 3.1 70B on Groq with anti-hallucination instructions and feedback analytics.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  knowledge-sources[Knowledge sources] --> vectorize-rag-pipeline[Vectorize RAG pipeline]
  vectorize-rag-pipeline --> vector-database[Vector database]
  product-ui-ai-assistant[Product UI AI assistant] --> topic-context[Topic context]
  product-ui-ai-assistant --> retrieval-endpoint[Retrieval endpoint]
  topic-context --> retrieval-endpoint
  vector-database --> retrieval-endpoint
  retrieval-endpoint --> reranking-model[Reranking model]
  reranking-model --> llm-response-generator[LLM response generator]
  topic-context --> llm-response-generator
  llm-response-generator --> product-ui-ai-assistant
  product-ui-ai-assistant --> feedback-analytics[Feedback analytics]

Components

Data flow

Docs, Discord, and Intercom content flow through a Vectorize RAG pipeline into a vector index. The product UI passes the user's question plus the page topic to the retrieval endpoint. Retrieved chunks can be reranked and filtered before the prompt is sent to the LLM, and thumbs feedback is sent to analytics.

Why this architecture

The assistant was embedded in the UI to remove context switching, used RAG instead of repeated fine-tuning because docs change constantly, and used reranking plus anti-hallucination prompting to keep smaller, faster models reliable.

Models used: Llama 3.1 70B · embedding model · reranking model

Scale

• Vectorize supports dozens of third-party integrations that create support complexity.
  “when you support dozens of integrations with third-party vector databases, AI platforms, and data sources, it’s inevitable that the user will need additional information.”article, stated
• The assistant filters retrieved data below a relevance threshold of 0.5.
  “For the AI assistant, I found by experimenting that a relevance threshold of 0.5 works well. Any data retrieved below that threshold is not sent to the LLM.”article, stated
• The generation model is Llama 3.1 70B hosted on Groq.
  “As for the LLM itself, we have been using the Llama 3.1 70B model hosted on Groq. We have gotten good results from this model, which is lightning fast due to the Groq service and its relatively small size.”article, stated

Alternatives considered (roads not taken)

Standalone chat-with-docs bot

Rejected because: It would still require users to leave the product UI, preserving the same context-switching problem as documentation.

Repeated fine-tuning

Rejected because: The documentation changes constantly, so a RAG pipeline is more practical than repeated model fine-tunes.

Query the vector database directly

Rejected because: Direct vector DB queries would require the application to create embeddings itself; the retrieval endpoint centralizes that work.

Patterns

rag-retrieval (retrieval)

The assistant uses a RAG pipeline over product documentation and support sources to provide context for the LLM.

query-rewriting (retrieval)

Instead of a full LLM rewrite in the first release, Vectorize appended the UI topic to the question to contextualize retrieval.

retrieval-ranking (retrieval)

Retrieved results are scored by a reranking model and filtered before they reach the LLM.

guardrails-io-filtering (safety)

The prompt instructs the LLM to answer only from retrieved content and decline when relevant texts are unavailable.

observability-tracing (operations)

The assistant records user feedback and recommends tracking retrieval relevance over time.

Tradeoffs

Assistant placement

Chose embedded product assistant over standalone chat-with-docs bot.

Rationale: Embedding the assistant keeps the user in the workflow and uses page context for better retrieval.

Knowledge update strategy

Chose RAG over changing documentation and support answers over fine-tuning a foundation model repeatedly.

Rationale: RAG keeps changing docs and support answers fresh without repeated fine-tunes.

Model and retrieval reliability

Chose smaller fast model plus relevance filtering over larger model expected to tolerate bad context.

Rationale: Filtering improves determinism, latency, and cost while keeping generation quality acceptable.

Gotchas (what bit them)

Relevant documentation outside the product UI creates support friction.

Root cause: Users must leave the product and context switch to documentation or chat.

Resolution: Embed the assistant directly into relevant UI pages.

Similarity search can retrieve unrelated chunks with close similarity scores.

Root cause: Vector similarity scores may be tightly clustered even when semantic relevance differs sharply.

Resolution: Use reranking relevance scores and filter below a threshold.

Poor retrieved context makes the generator answer the wrong topic.

Root cause: Low-quality context confuses the LLM even when it has a large context window.

Resolution: Filter low-relevance chunks before generation.

Evaluation

Vectorize used its RAG Sandbox to inspect retrieved chunks and LLM answers, experimented with relevance thresholds, added thumbs-up/down analytics, and planned retrieval relevance monitoring.

Metrics: similarity score · relevance score · 0.5 relevance threshold · thumbs-up/down feedback · retrieval relevance over time

Outcomes

• Appending page topic context improved retrieval quality and stopped unrelated answers.
• Reranking with a 0.5 threshold kept low-relevance chunks out of the LLM prompt.
• Anti-hallucination prompting stopped answers about undocumented development features.

The deployed assistant uses Llama 3.1 70B on Groq, a 0.5 relevance threshold, and support data from documentation, Discord, and Intercom.