Building a Video Q&A System with RAG and Speaker Detection

vimeo · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Video knowledge retrieval

Vimeo wanted viewers to ask natural-language questions of long knowledge-sharing videos and receive grounded answers, playable references, speaker-aware context, and follow-up questions without requiring full-video viewing.

Constraints:

• Initial implementation relied on transcripts only, not visual content.
• Questions require different context sizes: small details, whole-topic summaries, and entire-video understanding.
• Speaker names are not provided in unstructured transcripts and facial recognition was avoided.
• Suggested related questions must remain answerable from the closed-world video context.

Architecture

Vimeo built a transcript-first RAG system for video Q&A. It registers transcripts into a vector database at multiple levels: 100-200 word chunks, 500-word chunks summarized to 100 words, and a whole-video description. At query time, it retrieves relevant context for answer generation, uses a separate quote-finding prompt to produce playable references, detects speaker names through transition-based LLM votes, and generates answerable related questions with RAG.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  viewer[Viewer] --> video-qa-interface[Video QA Interface]
  video-transcript[Video Transcript] --> bottom-up-registration[Bottom Up Registration]
  bottom-up-registration --> vector-database[Vector Database]
  video-transcript --> speaker-detection[Speaker Detection]
  video-qa-interface --> retrieval-pipeline[Retrieval Pipeline]
  retrieval-pipeline --> vector-database
  retrieval-pipeline --> answer-generator[Answer Generator]
  retrieval-pipeline --> reference-finder[Reference Finder]
  retrieval-pipeline --> question-generator[Question Generator]
  answer-generator --> video-qa-interface
  reference-finder --> video-qa-interface
  question-generator --> video-qa-interface

Components

Data flow

A video transcript is chunked into bottom, middle, and video-level representations, embedded with timing metadata into a vector database, queried by the viewer's question, and passed to an LLM for answer generation. A second prompt locates transcript quotes for references, while speaker detection and related-question generation reuse retrieved/transcript context to improve navigation.

Why this architecture

The system uses RAG because video answers must be grounded in private/specific video content and because long transcripts are too slow and expensive to pass wholesale on every question. The multi-level registration exists because different viewer questions require different context windows.

Models used: large language models · ChatGPT 3.5 · vector embeddings

Scale

• The example use case is a 15-minute video that a viewer does not have time to watch.
  “Say you haven’t watched the video before, and you don’t have 15 minutes to watch it now, but you need a summary of the information that the video contains.”article, stated
• Bottom-level chunks are 100-200 words, corresponding to 1-2 minutes of playback.
  “We start by a standard chunking of the transcript to groups of sentences containing 100 to 200 words (corresponding to 1–2 minutes of playback time). These represent the bottom level of processing.”article, stated
• Middle-level chunks are 500 words summarized to 100 words.
  “We then take larger chunks of 500 words and use an LLM to summarize them to 100 words each. We ask the LLM to focus on specific details, numbers, dates, and speaker names and include this in the summary.”article, stated
• Speaker detection uses four prompts to collect several name votes for each speaker ID.
  “If you look closely at Figure 4, you see that we use four different prompts to get several name votes for each speaker ID. The first prompt asks if the LLM can identify the name of Speaker 2 from what Speaker 1 just said, like this: > Speaker 1: “Take it away, Rachel.” >”article, stated

Alternatives considered (roads not taken)

Use visual information in the initial implementation

Rejected because: Vimeo chose transcripts first because captions already existed and knowledge-sharing videos were mostly answerable from speech.

Flat small overlapping chunks only

Rejected because: General and speaker questions require large portions or entire-video context, while detail questions need small context.

Facial recognition for speaker identity

Rejected because: The system intentionally uses transcript/audio transition evidence instead of facial recognition.

One prompt for both answering and finding references

Rejected because: Testing showed better performance by separating answer generation from quote/reference extraction.

Generate related questions from only the user's question

Rejected because: Closed-world video QA needs generated questions to be answerable from retrieved video context.

Patterns

rag-retrieval (retrieval)

Video transcripts are embedded into a vector database and retrieved as context for grounded video answers.

context-compaction (retrieval)

Long transcript context is summarized into middle-level and whole-video representations to support broad questions.

prompt-chaining (generation)

The system uses separate prompts for answering and quote/reference extraction.

confidence-thresholding (speaker identification)

Speaker names are assigned only when aggregated LLM votes reach high confidence; otherwise the name is left unspecified.

synthetic-data-generation (engagement)

The system generates pregenerated and related questions from retrieved video context to guide viewers.

Tradeoffs

Initial video representation

Chose Transcript-only RAG over Immediate multimodal visual understanding.

Rationale: Automatic captions already existed, and transcripts answered most knowledge-sharing video questions.

Context representation

Chose Bottom-up multi-level transcript chunks and summaries over One fixed transcript chunk granularity.

Rationale: Different question types require detail, section, or whole-video context.

Speaker identification precision

Chose Leave speaker names unspecified when confidence is low over Always assign the most likely name.

Rationale: Wrong speaker attribution is worse than missing names for video Q&A trust.

Reference extraction architecture

Chose Answer prompt followed by quote-finding prompt over Single prompt doing both tasks.

Rationale: Separating tasks improved performance and let the second prompt use original transcript text for exact quotes.

Gotchas (what bit them)

A single chunk size cannot answer all video-question types well.

Root cause: Some questions need details from a few sentences, while summaries and speaker questions need section- or whole-video context.

Resolution: Use bottom-up multi-level transcript registration with small chunks, summarized middle chunks, and full-video descriptions.

Speaker names are hard to infer from unstructured transcripts.

Root cause: Transcripts do not say who is speaking, and free speech includes names that may refer to non-speakers or nicknames.

Resolution: Cluster speaker turns, analyze transitions with multiple prompts, aggregate votes, and leave names blank when uncertain.

Combining answering and reference finding in one prompt degraded reference quality.

Root cause: The team suspected ChatGPT 3.5 capacity limits when performing both tasks together.

Resolution: Split answer generation and quote/reference extraction into two prompts sharing the same retrieved context.

Related-question generation can suggest unanswerable questions.

Root cause: A closed-world video chatbot cannot answer questions that are merely related to the user's topic but absent from the video.

Resolution: Use RAG for related-question generation and require questions to be answerable from retrieved context.

Evaluation

Vimeo evaluated the system through experiments on question types, transcript versus visual content, chunking levels, speaker-name voting confidence, answer/reference prompt separation, and related-question answerability in closed-world video context.

Metrics: Question type coverage · Speaker-name identification confidence · Reference quote alignment · Playable moment relevance · Related-question answerability · Prompt-task separation quality

Outcomes

• The system can summarize lengthy videos, link to key playable moments, and suggest related questions.
• The RAG process supports different context lengths and speaker-aware questions over transcripts.
• Vimeo planned to add visual information later.

The article provides architectural and qualitative details but not traffic, latency, or user-adoption metrics.