Building a Scalable Conversational Video Agent with LangGraph and Twelve Labs APIs

jockey · Media & Entertainment · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Conversational video processing agent

Jockey needed to let users search, edit, summarize, and generate text from videos through natural language while coordinating specialized video APIs, complex workflows, state, error recovery, and scalable deployment.

Constraints:

Architecture

Jockey combines LangGraph with Twelve Labs video understanding APIs. A user query enters a LangGraph multi-agent workflow where the Supervisor routes requests, calls a Planner for complex tasks, and delegates to specialized Workers for video search, video text generation, and video editing. Twelve Labs APIs provide native video understanding, LangGraph manages state and routing, and LangGraph Cloud provides scalable deployment, task queues, large state storage, double-texting, background jobs, and Studio debugging.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user[User] --> jockey[Jockey]
  jockey[Jockey] --> langgraph-workflow[LangGraph Workflow]
  langgraph-workflow[LangGraph Workflow] --> supervisor[Supervisor]
  supervisor[Supervisor] --> planner[Planner]
  planner[Planner] --> workers[Workers]
  supervisor[Supervisor] --> workers[Workers]
  workers[Workers] --> twelve-labs-apis[Twelve Labs APIs]
  twelve-labs-apis[Twelve Labs APIs] --> video-foundation-models[Video Foundation Models]
  langgraph-workflow[LangGraph Workflow] --> langgraph-cloud[LangGraph Cloud]
  langgraph-cloud[LangGraph Cloud] --> langgraph-studio[LangGraph Studio]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
User Person issuing conversational video search, editing, summarization, or generation requests. Jockey
Jockey Open-source conversational video agent combining LLMs, LangGraph, and Twelve Labs APIs. User, LangGraph Workflow, Twelve Labs APIs
LangGraph Workflow Graph runtime controlling prompts, LLM calls, state, routing, and node execution. Jockey, Supervisor, Planner, Workers, LangGraph Cloud
Supervisor Central coordinator that routes tasks, manages workflow state, handles errors, and initiates replanning. LangGraph Workflow, Planner, Workers
Planner Creates step-by-step plans for complex user requests. Supervisor, Workers
Workers Specialized agents for video search, video text generation, and video editing. Supervisor, Planner, Twelve Labs APIs
Twelve Labs APIs Video understanding APIs for search, classification, summarization, question answering, editing support, and text generation. Jockey, Workers, Video Foundation Models
Video Foundation Models Native video models that process visuals, audio, on-screen text, and temporal relationships. Twelve Labs APIs
LangGraph Cloud Deployment infrastructure for horizontally scaling servers, task queues, large states, double-texting, and background jobs. LangGraph Workflow, LangGraph Studio
LangGraph Studio Trajectory visualization and debugging tool for rapid iteration. LangGraph Cloud

Data flow

A conversational query enters Jockey. LangGraph analyzes query complexity, the Supervisor either returns a simple text response or invokes the Planner, the Planner breaks complex requests into steps, Workers call Twelve Labs APIs for search/editing/text generation, LangGraph passes controlled state between nodes, and the final video or text result is returned to the user.

3 supporting quotes
“decision-making process from the initial query input, through LangGraph's analysis to determine query complexity”article, stated
“the initial query input, through LangGraph's analysis to determine query complexity, and the subsequent routing to either a simple text response or a more complex chain of video processing steps. This includes retrieving videos from Twelve Labs APIs, combining or segmenting video content as”article, stated
“retrieving videos from Twelve Labs APIs, combining or segmenting video content as needed, and presenting the final results to the user.”article, stated

Why this architecture

Jockey moved to LangGraph because video workflows need granular state and routing control, and Twelve Labs APIs supply native multimodal video understanding without captions as an intermediate representation. The multi-agent split lets Supervisor, Planner, and Workers coordinate complex video tasks while LangGraph Cloud handles production interaction patterns.

3 supporting quotes
“Unlike the legacy LangChain AgentExecutor, LangGraph provides a flexible API for custom cognitive architectures”article, stated
“Powered by advanced video foundation models (VFMs), these APIs work with video natively, without relying on intermediary representations like pre-generated captions. This allows for more accurate and contextual understanding of the full video, including visuals, audio, on-screen text, and”article, stated
“One of the primary advantages of adopting LangGraph is the granular control it provides over each step of the workflow. This allows Jockey to precisely manage which information is passed between nodes and how their responses contribute to the overall state.”article, stated

Models used: Large Language Models · Twelve Labs video foundation models

Scale

Alternatives considered (roads not taken)

Use legacy LangChain AgentExecutor

Rejected because: Jockey migrated to LangGraph for more precise workflow control, scalability, and functionality.

“Originally built on LangChain in v1.0, Jockey now harnesses the power of LangGraph, offering enhanced scalability and improved functionality for both frontend and backend operations.”article, stated
“This transition marks a pivotal advancement in Jockey's architecture, enabling more efficient and precise control over complex video workflows.”article, stated

Use pre-generated captions as video representation

Rejected because: Twelve Labs APIs work with video natively across visuals, audio, on-screen text, and temporal relationships.

“from video content. Powered by advanced video foundation models (VFMs), these APIs work with video natively, without relying on intermediary representations like pre-generated captions. This allows for more accurate and contextual understanding of the full video, including visuals,”article, stated
“This allows for more accurate and contextual understanding of the full video, including visuals, audio, on-screen text, and temporal relationships.”article, stated

Hard-code all future video capabilities

Rejected because: Jockey can add prompts, state fields, and workers to extend behavior without changing the whole system.

“The modular design of Jockey also facilitates customization and extension. Developers can modify prompts, extend the state to handle more complex scenarios, or add new workers to address specific use cases, making Jockey a versatile foundation for building advanced video AI applications.”article, stated
“approach leverages Jockey's language model capabilities to introduce new functionalities without modifying the underlying code. For example, you could create a prompt that instructs Jockey to identify and extract specific types of scenes from videos, such as action sequences or emotional”article, stated

Patterns

multi-agent-orchestration (agent-architecture)

Jockey uses Supervisor, Planner, and Worker agents to route, plan, and execute video tasks.

“the system comprises three main components: the Supervisor, the Planner, and the Workers”article, stated

tool-use-orchestration (workflow)

Workers use available video tools for search, text generation, and editing.

“specialized agents that execute the instructions using their available tools.”article, stated
“The Actual Workers, which are specialized agents that execute the instructions using their available tools.”article, stated

human-in-the-loop-gate (control)

LangGraph's persistence layer supports human approval and editing/resuming actions, even though the Jockey article focuses mostly on automation.

“LangGraph also facilitates human-agent collaboration via its built-in persistence layer, enabling human approval before task execution”article, stated

durable-execution (runtime)

LangGraph Cloud manages task queues, large states, background jobs, double-texting, and resumable interaction patterns.

“The service supports real-world interaction patterns, including double-texting to handle new user inputs on currently-running threads and async background jobs”article, stated

Tradeoffs

Use LangGraph for workflow control

Chose LangGraph over Legacy AgentExecutor.

Rationale: LangGraph gave granular state, prompt, LLM-call, and routing control for complex video workflows.

“Unlike the legacy LangChain AgentExecutor, LangGraph provides a flexible API for custom cognitive architectures”article, stated
“granular control it provides over each step of the workflow”article, stated

Use a multi-agent split

Chose Supervisor, Planner, and Worker roles over One monolithic video agent.

Rationale: The split lets the system route simple requests, plan complex ones, and delegate specialized video actions.

“acts as the central coordinator, responsible for routing tasks between different nodes”article, stated
“is called upon by the Supervisor to create detailed, step-by-step plans for complex user requests.”article, stated

Use native video APIs

Chose Twelve Labs video understanding APIs over Caption-derived intermediate representation.

Rationale: Native APIs preserve visual, audio, text, and temporal context for richer video operations.

“work with video natively, without relying on intermediary representations like pre-generated captions.”article, stated

Gotchas (what bit them)

Caption-only video understanding loses important context.

Root cause: Video content includes visuals, audio, on-screen text, and temporal relationships that captions do not fully represent.

Resolution: Jockey relies on Twelve Labs APIs that process video natively.

“without relying on intermediary representations like pre-generated captions.”article, stated
“including visuals, audio, on-screen text, and temporal relationships.”article, stated

Complex video workflows need more control than a simple agent executor.

Root cause: Search, editing, text generation, planning, and state handoff require precise node-level coordination.

Resolution: The system migrated from AgentExecutor to LangGraph with Supervisor, Planner, and Workers.

“Unlike the legacy LangChain AgentExecutor, LangGraph provides a flexible API for custom cognitive architectures”article, stated
“precisely manage which information is passed between nodes and how their responses contribute to the overall state.”article, stated

Evaluation

The article does not describe a formal evaluation harness; it emphasizes LangGraph Studio debugging, visualizing trajectories, token optimization, state control, and modular extension as the operational feedback mechanisms.

Metrics: token usage · query complexity routing · node trajectory debugging · concurrent user support through LangGraph Cloud

“Such fine-tuned control not only optimizes token usage but also enables more accurate guidance of node responses”article, stated
“a tool for visualizing and debugging agent trajectories, enabling rapid iteration and feedback for developers building agentic applications.”article, stated

Outcomes

The source is a product/architecture article and does not provide usage numbers for Jockey; it does state that LangGraph Cloud handles numerous concurrent users and large states.

“Jockey now harnesses the power of LangGraph, offering enhanced scalability and improved functionality for both frontend and backend operations.”article, stated
“You can deploy Jockey locally for development and testing”article, stated