Rebuilding a Production Chatbot with Direct API Access and Multi-Agent Architecture

langchain · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production documentation chatbot and support engineering workflow automation

LangChain's public chatbot was not reliable enough for support engineers because vector retrieval over chunked documentation lost structure, required constant reindexing, and produced vague citations for complex product/debugging questions.

Constraints:

Architecture

LangChain rebuilt Chat LangChain around two architectures. Fast documentation Q&A uses createAgent with direct API tools for Mintlify docs and Pylon support articles, usually making 3-6 tool calls and returning sub-15-second answers. Complex questions use a Deep Agent with specialized subgraphs for documentation, knowledge base, and codebase search; each subagent filters its own domain and passes refined 'golden data' to a main orchestrator.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Chat LangChain UI"] --> B["LangGraph SDK runtime"]
  B --> C["CreateAgent docs agent"]
  B --> D["Deep Agent orchestrator"]
  B --> M["LangGraph checkpointer"]
  C --> H["Mintlify documentation API"]
  C --> I["Pylon knowledge base"]
  C --> K["Model middleware stack"]
  D --> E["Documentation subgraph"]
  D --> F["Knowledge base subgraph"]
  D --> G["Codebase search subgraph"]
  D --> K
  E --> H
  F --> I
  G --> J["Public codebase tools"]
  C --> L["LangSmith tracing and evals"]
  D --> L
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Chat LangChain UI Public chat interface and conversation surface. LangGraph SDK runtime
LangGraph SDK runtime Handles streaming, state management, thread filtering, and subgraph visibility. Chat LangChain UI, CreateAgent docs agent, Deep Agent orchestrator, LangGraph checkpointer
CreateAgent docs agent Fast default agent for docs and knowledge-base questions. LangGraph SDK runtime, Mintlify documentation API, Pylon knowledge base, Model middleware stack
Deep Agent orchestrator Synthesizes refined insights from specialized subgraphs for complex queries. LangGraph SDK runtime, Documentation subgraph, Knowledge base subgraph, Codebase search subgraph, Model middleware stack
Documentation subgraph Searches full structured documentation pages and extracts relevant documentation facts. Deep Agent orchestrator, Mintlify documentation API
Knowledge base subgraph Scans support article titles and reads promising articles in full. Deep Agent orchestrator, Pylon knowledge base
Codebase search subgraph Uses ripgrep, directory listing, and file reads to find implementation details with line numbers. Deep Agent orchestrator, Public codebase tools
Mintlify documentation API Returns complete documentation pages with structure intact. CreateAgent docs agent, Documentation subgraph
Pylon knowledge base Support article title and content source. CreateAgent docs agent, Knowledge base subgraph
Public codebase tools Pattern search, tree navigation, and file-read tools for code evidence. Codebase search subgraph
Model middleware stack Guardrails, retry, fallback, and caching middleware around model calls. CreateAgent docs agent, Deep Agent orchestrator
LangSmith tracing and evals Conversation traces and evaluation suite used to optimize tool calls and prompts. CreateAgent docs agent, Deep Agent orchestrator
LangGraph checkpointer Stores conversation history and state with a 7-day TTL. LangGraph SDK runtime

Data flow

A user question enters Chat LangChain. Simple questions go to createAgent, which searches full documentation pages and support articles through direct APIs, refines queries if needed, and streams an answer. Complex questions route to a Deep Agent: docs, knowledge-base, and codebase subgraphs search independently with tools such as Mintlify, Pylon, ripgrep, directory listing, and file reads. The orchestrator synthesizes the refined evidence into a cited response while middleware handles guardrails, retries, fallback, and caching.

3 supporting quotes
“The agent searches the docs, checks the knowledge base if needed, refines its query if the results are unclear, and returns an answer.”article, stated
“with three specialized subagents — one for docs, one for knowledge base, one for codebase search”article, stated
“Each subagent operates independently, asking follow-up questions, filtering through information, and extracting only the most relevant insights”article, stated

Why this architecture

LangChain mirrored the workflow their support engineers already trusted: docs for the official story, knowledge base for known issues, and codebase for ground truth. Direct API/tool access preserved structure and citations, while subgraphs prevented raw search results from overwhelming the main agent.

3 supporting quotes
“Docs for the official story. Knowledge base for real-world issues. Codebase for ground truth.”article, stated
“Documentation is already organized. Knowledge bases are already categorized. Codebases are already navigable.”article, stated
“The main orchestrator agent never sees the raw search results. It only receives the refined insights from each domain expert.”article, stated

Models used: Claude Haiku 4.5 · GPT-4o Mini · GPT-4o-nano · Gemini Nano

Scale

Alternatives considered (roads not taken)

Traditional vector embedding RAG over chunked documentation

Rejected because: Chunking broke structure, reindexing was constant, and citations were vague.

“The standard approach to documentation search — chunk docs into pieces, generate embeddings, store in a vector database, retrieve by similarity — works fine for unstructured content like PDFs.”article, stated
“we kept hitting three problems.”article, stated

Single deep agent with all tools and raw results

Rejected because: It overloaded the context window with pages, articles, and snippets.

“When we first built the deep agent as a single system with access to all three tools, it would return everything it found.”article, stated
“The context window would explode”article, stated

Use deep investigation for every query

Rejected because: Deep Agent is thorough but much slower, while most documentation questions need fast answers.

“There's no planning phase, no orchestration overhead — just immediate tool calls and answers.”article, stated
“This deep agent architecture takes longer to run — sometimes 1-3 minutes for complex queries”article, stated

Patterns

multi-agent-orchestration (orchestration)

The Deep Agent delegates docs, knowledge-base, and codebase work to specialized subgraphs that report refined evidence to a main orchestrator.

“For these tasks, we built a Deep Agent with specialized subgraphs: one for documentation search, one for knowledge base search, and one for codebase search.”article, stated

tool-use-orchestration (tools)

Agents use direct APIs, support-article tools, ripgrep search, directory listing, file reads, and middleware rather than relying only on vector retrieval.

“We gave the agent three tools that mirror the workflow from the opening — the same pattern our engineers follow when using Claude Code:”article, stated
“First, it searches the codebase for a pattern using ripgrep. Then it lists the directory structure to understand how files are organized. Finally, it reads the specific file, focusing on the relevant section, and returns the implementation with line numbers.”article, stated

context-engineering (context)

Subgraphs filter raw domain results into golden data before the orchestrator sees them, preventing context overload.

“Each subagent filters and extracts only the "golden data" from its domain before passing the refined insights up.”article, stated

observability-tracing (operations)

LangSmith traces conversations, identifies unnecessary tool calls, and supports prompt/eval optimization.

“The data showed us that most questions could be answered with 3-6 tool calls if we taught the agent to ask better follow-up questions.”article, stated

fallback-degradation (reliability)

Middleware retries API failures and switches between models when a model is unavailable.

“Fallback middleware switches between Haiku, GPT-4o Mini, and Gemini Nano if a model is unavailable.”article, stated

cost-aware-routing (operations)

The system uses CreateAgent for fast/common questions, Deep Agent only for complex cases, and Anthropic caching for expensive calls.

“Caching reduces costs by reusing results for identical queries.”article, stated
“This deep agent architecture takes longer to run — sometimes 1-3 minutes for complex queries — but the thoroughness is worth it. We leverage DeepAgent when the initial response is not addressing the core question.”article, stated

Tradeoffs

Use two architectures rather than one chatbot path.

Chose Fast CreateAgent plus slower Deep Agent over One universal agent architecture.

Rationale: Most docs questions need speed, while complex code-backed issues justify deeper 1-3 minute investigation.

“we quickly realized we needed to combine two different architectures driven by two broad categories of questions.”article, stated
“This deep agent architecture takes longer to run — sometimes 1-3 minutes for complex queries — but the thoroughness is worth it.”article, stated

Replace vector embeddings with direct API access for structured sources.

Chose Direct docs/KB/code tools over Chunking and vector similarity for product docs/code.

Rationale: Direct access preserved structure, avoided reindexing, and improved citation precision.

“The breakthrough was realizing we were solving the wrong problem. Documentation is already organized. Knowledge bases are already categorized. Codebases are already navigable. We didn't need smarter retrieval — we needed to give the agent direct access to that existing structure.”article, stated
“For public Chat LangChain, users get sub-15-second responses with precise citations. They can verify answers immediately because we link directly to the relevant documentation page or knowledge base article. And we no longer spend hours reindexing — the documentation updates automatically.”article, stated

Expose streaming agent activity to users.

Chose Token, tool-call, and state streams over Opaque loading spinners.

Rationale: Streaming makes the agent feel fast and lets users watch it search and build answers.

“Users watch the agent think, search docs, check the knowledge base, and build the response token-by-token. No loading spinners.”article, stated

Gotchas (what bit them)

Support engineers avoided the chatbot because it did not match their real troubleshooting workflow.

Root cause: The original tool used docs-only/vector retrieval when engineers needed docs, support history, and code implementation evidence.

Resolution: LangChain automated the three-step workflow engineers already trusted.

“But we have a confession: our support engineers weren't actively using the LangChain Chatbot. That's where our real learning began.”article, stated
“This three-step ritual worked incredibly well. We watched them do it dozens of times a day and thought: what if we just automated this exact workflow?”article, stated

Chunked vector retrieval broke structured documentation.

Root cause: 500-token chunks lost headers, subsections, and contextual relationships.

Resolution: Agents query Mintlify for full pages and search the existing structure directly.

“Chunking breaks structure. When you chop documentation into 500-token fragments, you lose headers, subsections, and context. The agent would cite "set streaming=True" without explaining why or when. Users had to hunt through pages to find what they needed.”article, stated
“For documentation, we use Mintlify's API, which returns full pages with all their headers, subsections, and code examples intact.”article, stated

A single deep agent with every raw result overloaded context.

Root cause: The main agent received pages, articles, and snippets all at once.

Resolution: Specialized subgraphs summarize/filter each domain before synthesis.

“The main agent would get five documentation pages, twelve knowledge base articles, and twenty code snippets — all at once.”article, stated
“That's when we restructured it with specialized subgraphs.”article, stated

Production agents need middleware, not just clever prompts.

Root cause: Real users hit off-topic queries, API failures, unavailable models, and cost pressure.

Resolution: LangChain added guardrails, retry, fallback, and caching middleware around the agent.

“Even elegant agent designs need production infrastructure to survive contact with real users. We built modular middleware to handle the operational concerns that would otherwise clutter our prompts.”article, stated
“These layers are invisible to users, but they're essential for reliability. They let the agent focus on reasoning while the infrastructure handles failure modes, cost optimization, and quality control.”article, stated

Evaluation

LangChain evaluated through observed support-engineer usage, LangSmith traces, A/B tests of prompts, speed/accuracy measurements, and production adoption of public and internal agents.

Metrics: Sub-15-second public responses · 3-6 tool calls for most docs questions · LangSmith trace tool-call counts · Speed and accuracy from prompt A/B tests · Internal support engineer adoption · Precise citations with pages/articles/line numbers

3 supporting quotes
“LangSmith's evaluation suite let us A/B test different prompting strategies and measure improvements in both speed and accuracy.”article, stated
“Since launching the new systems, we've seen dramatic improvements.”article, stated
“Internally, our support engineers use the Deep Agent to handle the most complex tickets. It searches documentation, cross-references known issues, and dives into our private codebase to find the implementation details that actually explain what's happening.”article, stated

Outcomes

Production public chatbot plus internal support workflow; most docs questions use 3-6 tool calls, Deep Agent can take 1-3 minutes, a sample trace used 7 tool calls over 30 seconds, and conversation state has a 7-day TTL.

3 supporting quotes
“For public Chat LangChain, users get sub-15-second responses with precise citations. They can verify answers immediately because we link directly to the relevant documentation page or knowledge base article. And we no longer spend hours reindexing — the documentation updates automatically.”article, stated
“For public Chat LangChain, users get sub-15-second responses with precise citations. They can verify answers immediately because we link directly to the relevant documentation page or knowledge base article. And we no longer spend hours reindexing — the documentation updates automatically.”article, stated
“The agent doesn't replace our engineers — it amplifies them, handling the research so they can focus on solving the problem.”article, stated