Building an AI-Powered Help Desk with RAG and Model Evaluation

vimeo · Media & Entertainment · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support self-service

Vimeo wanted to prototype a generative AI help desk that could answer customer questions from existing Help Center content more directly than search, ticket submission, or an intent-based chatbot.

Constraints:

The source corpus was Zendesk Help Center articles.
The prototype needed to cite and retrieve relevant sections rather than rely on stale model memory.
Model choice had to balance response quality, latency, cost, authentication, and provider redundancy.
Quality assurance is difficult because support questions and generated responses vary widely.

Architecture

Vimeo scraped Zendesk Help Center articles, parsed and chunked the content, embedded the chunks, and indexed them in an HNSWLib vector store with metadata. A LangChain ConversationalRetrievalQAChain rewrites conversation context into a standalone question, embeds it, retrieves similar chunks, and sends those chunks to an LLM. Vimeo tested Google Vertex AI Chat Bison, OpenAI GPT-3.5, GPT-4, and Azure OpenAI before choosing Chat Bison for concise answers and GCP authentication.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  zendesk-help-center[Zendesk Help Center] --> ingestion-pipeline[Ingestion Pipeline]
  ingestion-pipeline --> vector-store[Vector Store]
  documentation-webhook[Documentation Webhook] --> vector-store[Vector Store]
  help-desk-chat-ui[Help Desk Chat UI] --> retrieval-chain[Retrieval Chain]
  retrieval-chain --> vector-store[Vector Store]
  retrieval-chain --> llm-provider[LLM Provider]
  llm-provider --> retrieval-chain[Retrieval Chain]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Zendesk Help Center`	Public support-article source corpus.	`Ingestion Pipeline`
`Ingestion Pipeline`	Scrapes, parses, chunks, embeds, and indexes help articles with metadata.	`Zendesk Help Center`, `Vector Store`
`Vector Store`	HNSWLib-backed embedding index for relevant article chunks.	`Ingestion Pipeline`, `Retrieval Chain`
`Help Desk Chat UI`	Customer-facing prototype surface for support questions.	`Retrieval Chain`
`Retrieval Chain`	LangChain conversational retrieval logic that rewrites questions, retrieves chunks, and calls the LLM.	`Help Desk Chat UI`, `Vector Store`, `LLM Provider`
`LLM Provider`	Configurable model provider such as Vertex AI Chat Bison, OpenAI GPT, or Azure OpenAI.	`Retrieval Chain`
`Documentation Webhook`	Updates, adds, or removes indexed documents when Zendesk content changes.	`Vector Store`

Data flow

Published Zendesk articles are scraped, saved, split into chunks, embedded, and stored with metadata in a vector store. At chat time, the latest user question and chat history are rewritten into a standalone question, embedded, retrieved against the vector store, and passed with relevant documents to the selected LLM for the final answer.

“The Zendesk API exposes all the needed information and metadata, including the article’s content, title, tags, and full URL.”article, stated

“First, we took any available chat history from the current session and combined it with the latest question from the user.”article, stated

Why this architecture

RAG lets Vimeo ground support answers in current Help Center content while keeping the prototype simple. LangChain gives a provider-agnostic retrieval QA chain, HNSWLib is enough for a sub-thousand-article prototype, and metadata links responses back to source documents.

“Filling our vector store with the embeddings for our Help Center articles enables high-speed retrieval and comparison of the embeddings”article, stated

“Langchain provides a simple interface for interacting with embedding and LLM APIs from various AI providers, coordinating multiple prompts to an LLM, which is the chain portion of the conversational retrieval QA chain, as part of a request.”article, stated

Models used: Google Vertex AI Chat Bison · OpenAI GPT-3.5 Turbo · OpenAI GPT-4 · Azure OpenAI GPT-3.5 Turbo · OpenAI embeddings

Scale

The initial prototype used a dataset of less than a thousand Help Center articles.
“which served our purposes for the initial prototype with a dataset of less than a thousand articles. Using a local disk also keeps any sensitive information out of the hands of third parties, which”article, stated
The retriever used five documents in the example LangChain code.
“const chain = ConversationalRetrievalQAChain.fromLLM( aiModel, vectorstore.asRetriever(5), chainOptions ); const response = await chain.call({ question, chat_history }); console.log(response)”article, stated
GPT-4 was more than twice GPT-3.5 Turbo's price per token and dramatically slower in Vimeo's comparison.
“GPT 4 delivers stronger and more concise answers than ChatGPT 3.5 Turbo, but the response speed is dramatically reduced and the price per token is more than doubled.”article, stated
Vertex AI Chat Bison pricing was $0.0005 per 1,000 input and output characters at the time of writing.
“At the time of this writing, the Google Vertex AI Chat Bison model costs $0.0005 per 1,000 characters for both input and output, while OpenAI ChatGPT 3.5 Turbo charges $0.0015 per 1,000 tokens of input and $0.002 per 1,000 tokens of output.”article, stated

Alternatives considered (roads not taken)

Existing Help Center search, tickets, or intent chatbot.

Rejected because: They did not surface relevant domain-restrict embed information at a glance.

“Unfortunately, none of these options provides any relevant information at a glance. Querying the chatbot with this phrase doesn’t yield immediately obvious answers, and it may end up directing the user to submit a ticket to the support team.”article, stated

Larger OpenAI models as the default.

Rejected because: GPT-4 improved answer quality but had materially worse speed and token cost.

“GPT 4 delivers stronger and more concise answers than ChatGPT 3.5 Turbo, but the response speed is dramatically reduced”article, stated

Ask the model to provide document links itself.

Rejected because: ChatGPT returned outdated or nonexistent links from stale training data.

“We attempted to have the AI model provide its own links to the documents, but ChatGPT would regularly return outdated or nonexistent links that were unrelated to the reference docs. After some debugging, we found out that ChatGPT contains an old copy of Vimeo’s Help Center in its training data!”article, stated

Patterns

`rag-retrieval` (retrieval)

Vimeo grounds support answers in retrieved Help Center article chunks rather than model memory.

“The process begins with the standalone question and related documents, which are sent to the LLM to output the final answer.”article, stated

`query-rewriting` (retrieval)

Chat history and the latest user message are rewritten into a standalone question before retrieval.

“The chat transcript was sent to the LLM to rephrase the input as a standalone question”article, stated

`model-routing` (model-strategy)

The prototype can swap LLM and embedding APIs to compare providers and provide redundancy.

“We can easily switch out different LLM and embedding APIs based on the specific needs and complexity of the questions asked.”article, stated

Tradeoffs

Initial vector store

Chose HNSWLib on local disk over External managed vector database.

Rationale: The dataset was small and local disk avoided sending sensitive data to third parties.

“Using a local disk also keeps any sensitive information out of the hands of third parties”article, stated

Model provider for now

Chose Google Vertex AI Chat Bison over OpenAI GPT-3.5, GPT-4, or Azure OpenAI as the default.

Rationale: Bison gave concise answers, efficient processing, cost benefits, and GCP-native authentication.

“We’re going with Google Vertex AI Chat Bison for now, but as we continue to experiment to build the best user experience, we may choose a different LLM”article, stated

Provider abstraction

Chose LangChain with swappable providers over Single hard-coded model and vector backend.

Rationale: The team wanted comparison, redundancy, and the ability to tailor retrieval to different datasets.

“It also helps us compare the performance between different LLMs and APIs, and provides redundancy during outages”article, stated

Gotchas (what bit them)

Model training data can fight retrieved truth.

Root cause: ChatGPT had an old copy of Vimeo's Help Center and could invent stale links.

Resolution: Use metadata source URLs from retrieval instead of model-generated links.

“ChatGPT contains an old copy of Vimeo’s Help Center in its training data!”article, stated

Generated support quality is hard to verify exhaustively.

Root cause: There are unlimited possible customer questions and responses can vary drastically.

Resolution: Set temperature to zero and compare model behavior, while recognizing QA remains difficult.

“Even then, the number of possible responses to an unlimited number of possible questions is challenging to verify by a quality assurance team.”article, stated

Local vector stores are convenient but tied to prototype scale.

Root cause: HNSWLib worked because the initial public help corpus was small.

Resolution: Use the simplest local store for the prototype while keeping the retrieval abstraction swappable.

“Which vector store provider is the best? That really depends on the use case.”article, stated

Evaluation

Vimeo compared Vertex AI Chat Bison, GPT-3.5 Turbo, GPT-4, and Azure OpenAI on response quality, concision, speed, cost, authentication, and operational fit for the help desk prototype.

Metrics: Response quality · Response concision · Response speed · Cost per token or character · Authentication complexity · Safety/moderation behavior

“the ability to switch between LLMs provided us with a way to compare the models.”article, stated

“Ultimately, the choice of model depends on the specific requirements of the application”article, stated

Outcomes

The prototype could answer the domain-restrict embed query with actionable instructions.
Vimeo chose Vertex AI Chat Bison for the current prototype.
The project exposed stale training data and QA challenges as important production risks.

The prototype ran over less than a thousand Help Center articles and used model comparisons with concrete price, speed, and authentication tradeoffs.

“A complete response with actionable instructions for this precise query appears in the chat.”article, stated

“Through careful evaluation, we went with the Google Vertex AI Chat Bison model for this use case”article, stated