Deploying an AI SDR Chatbot for Lead Qualification with Production-Grade Observability

lubu_labs · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Revenue operations chatbot deployment

Lubu Labs needed to move an AI SDR chatbot from demo to production, where lead qualification, product answers, routing, and CRM writes had real revenue and trust consequences.

Constraints:

The chatbot had to answer from current product positioning rather than stale documents.
It had to qualify intent, collect lead details, route conversations, and write structured downstream outcomes.
Real traffic introduced messy multi-part questions and quality drift.
Tool and model failures were hard to debug without step-level visibility.
Side-effecting CRM operations could not be retried like normal model calls.

Architecture

Lubu Labs modeled the SDR bot as a LangGraph state machine with structured SDRState, Qdrant knowledge retrieval, qualification/product-QA/sales-brief/CRM-write nodes, retry policies, and a PostgresSaver checkpointer. LangSmith traces, datasets, evaluations, and feedback loops supplied production observability. The final hardening separated model retries from side-effecting CRM retries and added idempotency keys based on thread and turn IDs.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  visitor-conversation["Visitor conversation"] --> langgraph-sdr-workflow["LangGraph SDR workflow"]
  langgraph-sdr-workflow --> qdrant-knowledge-base["Qdrant knowledge base"]
  langgraph-sdr-workflow --> crm-write-tool["CRM write tool"]
  langgraph-sdr-workflow --> postgres-checkpointer["Postgres checkpointer"]
  langgraph-sdr-workflow --> langsmith-observability["LangSmith observability"]
  langsmith-observability --> evaluation-dataset["Evaluation dataset"]
  crm-write-tool --> idempotency-key["Idempotency key"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Visitor conversation`	Inbound lead conversation that asks questions and provides qualification details.	`LangGraph SDR workflow`
`LangGraph SDR workflow`	State machine coordinating retrieval, qualification, sales brief, and CRM write nodes.	`Qdrant knowledge base`, `CRM write tool`, `Postgres checkpointer`, `LangSmith observability`
`Qdrant knowledge base`	Retrieval source for product-grounded answers.	`LangGraph SDR workflow`
`CRM write tool`	Side-effecting downstream write for lead notes, scores, tags, and handoff summaries.	`LangGraph SDR workflow`, `Idempotency key`
`Postgres checkpointer`	Persists workflow state for reliability and conversation continuation.	`LangGraph SDR workflow`
`LangSmith observability`	Tracing, evaluation, feedback, and annotation layer for production debugging and regression testing.	`LangGraph SDR workflow`, `Evaluation dataset`
`Evaluation dataset`	Representative conversations used to test routing, extraction, grounding, and side-effect safety.	`LangSmith observability`
`Idempotency key`	Thread-and-turn key preventing duplicate CRM writes across retries.	`CRM write tool`

Data flow

A visitor conversation enters the LangGraph SDR workflow. The graph carries structured state, optionally retrieves product context from Qdrant, qualifies the lead, generates a sales brief, and writes CRM outcomes. Postgres checkpointing preserves state, LangSmith traces record model/tool calls and metadata, evaluations replay representative conversations, and feedback annotations feed production failures back into the test set.

“We modeled the AI SDR chatbot as a LangGraph state machine: each stage is a node, and the conversation state is carried forward between nodes.”article, stated

“LangSmith tracing gave us an end-to-end timeline for each conversation turn: model calls, tool calls, inputs/outputs, latencies, and metadata.”article, stated

Why this architecture

The architecture treats the SDR bot as a stateful, inspectable workflow because production failures involve sequences of model calls, retrieval, tool calls, retries, and CRM side effects. LangSmith made those sequences reproducible; idempotent CRM writes hardened the most business-damaging failure boundary.

“once you have tool calls (CRM write, marketing trigger, scheduling), you need to debug a sequence of model + tool steps, not a single response.”article, stated

“In production, we added an idempotency key (derived from thread_id + turn ID) and made retries explicit so a “retry” could not create a second write.”article, stated

Models used: LangGraph · LangSmith · Qdrant · PostgreSQL PostgresSaver · LLM agent/model (unspecified)

Scale

The system was deployed to production for a loyalty platform.
“We shipped an AI SDR chatbot to production for a loyalty platform.”article, stated
The rare duplicate-action defect produced two lead updates from one user turn.
“Two writes, one user turn.”article, stated
The evaluation dataset covered objections, pricing questions, ambiguous intent, and messy multi-part asks.
“We built a small dataset of representative conversations: objections, pricing questions, ambiguous intent, and “messy” multi-part asks.”article, stated

Alternatives considered (roads not taken)

Treat production quality as better prompting

Rejected because: The article says observability, not better prompts, made the stateful tool-using workflow shippable.

“The difference between a demo and production wasn’t “better prompts”. It was observability.”article, stated

Debug only final assistant messages

Rejected because: The duplicate CRM-write failure was invisible in the final response because the damage was a side effect.

“Nothing “looked wrong” in the final assistant message; the damage was the side-effect.”article, stated

Retry side-effecting tool calls the same way as model calls

Rejected because: Retries are useful for model calls but dangerous for CRM writes unless idempotent.

“Separate thinking from doing. Retries are normal for model calls, but they’re dangerous for side effects.”article, stated

Patterns

`durable-execution` (workflow)

The LangGraph workflow compiled with a PostgresSaver checkpointer so the stateful SDR conversation could persist and recover.

“# Compile with checkpointer for persistence and reliability app = graph.compile(checkpointer=PostgresSaver.from_conn_string(DB_URL))”article, stated

`rag-retrieval` (retrieval)

The SDR workflow used Qdrant retrieval to ground product answers in knowledge-base context.

“Retrieve knowledge base context with error handling and validation from Qdrant vector store.”article, stated

`observability-tracing` (operations)

LangSmith traces captured the sequence of model calls, tool calls, inputs, outputs, latencies, and metadata.

“LangSmith tracing gave us an end-to-end timeline for each conversation turn: model calls, tool calls, inputs/outputs, latencies, and metadata.”article, stated

`eval-driven-development` (evaluation)

A representative conversation dataset and LangSmith evaluation workflows caught routing, grounding, and side-effect regressions.

“Then we used LangSmith’s evaluation workflows to run the agent against that dataset as we iterated.”article, stated

`tool-use-orchestration` (agent)

The workflow orchestrated retrieval, qualification, sales briefing, and CRM writes as graph nodes rather than a single chatbot response.

“The chatbot had to do more than chat. It needed to:”article, stated

Tradeoffs

Model the chatbot as a state machine

Chose LangGraph workflow with explicit state and nodes over A stateless chatbot prompt.

Rationale: The production SDR flow had branching state, retrieval, tool calls, CRM writes, and retry boundaries that needed explicit orchestration.

“each stage is a node, and the conversation state is carried forward between nodes.”article, stated

Invest in observability before launch

Chose LangSmith traces, metadata, feedback, and evaluations over Prompt patches based on anecdotes.

Rationale: Traceability made failures reproducible by run ID and allowed product/sales to compare regressions over conversation sets.

“At that point, the team becomes reactive: you patch prompts based on anecdotes and hope the next deploy is better.”article, stated

Harden the CRM boundary

Chose Idempotent CRM writes keyed by thread and turn over Blindly retrying the side-effecting write.

Rationale: Side-effect retries can create duplicate revenue-operation records even if the final user-visible response looks fine.

“In production, we added an idempotency key (derived from thread_id + turn ID) and made retries explicit so a “retry” could not create a second write.”article, stated

Gotchas (what bit them)

Real traffic caused quality drift.

Root cause: Staging prompts did not cover messy, multi-part production questions.

Resolution: Use representative conversation datasets and regression evaluations.

“Quality drift on real traffic. The same prompt that “worked” in staging started failing once users asked messy, multi-part questions.”article, stated

Tool/model failures looked like generic model weirdness.

Root cause: Without trace-level visibility, teams could not see the exact sequence of retrieval, model, and tool steps.

Resolution: Trace runs by prompt version, context, tool calls, outputs, latency, and metadata.

“When the agent used tools (search, retrieval, CRM write), failures often looked like generic “the model responded weirdly” until you could see the exact sequence.”article, stated

Intermittent CRM duplicate writes escaped manual QA.

Root cause: A timeout plus retry created two successful side-effecting writes for one turn.

Resolution: Make tool calls idempotent and separate model retries from side-effect retries.

“under certain latency spikes, a CRM write timed out, our agent retried, and the second attempt succeeded, but the first one actually completed shortly after.”article, stated

Evaluation

The team built representative conversation datasets, ran LangSmith evaluations during iteration, and fed annotated production examples back into the evaluation set.

Metrics: Route correctness · Required-field extraction · Groundedness · Unsafe side-effect avoidance · No duplicate writes on simulated timeouts

“keep evaluations simple at first (did we route correctly? did we extract required fields? did we stay grounded? did we avoid unsafe side effects?).”article, stated

“add an evaluation case specifically for “no duplicate writes” on simulated timeouts.”article, stated

Outcomes

Moved a loyalty-platform AI SDR chatbot into production.
Made the workflow debuggable, testable, and shippable with LangSmith.
Identified and fixed an intermittent duplicate CRM-write failure.
Created an evaluation and feedback loop for future prompt, model, retrieval, and tool-schema changes.

The source frames this as a production revenue-operations deployment; it mentions a small representative conversation dataset but does not disclose traffic volume.

3 supporting quotes

“LangSmith made the system debuggable, testable, and shippable once we moved from “a chatbot” to “a stateful workflow with tools and side effects”.”article, stated

“This is a great example of why observability is not just for debugging. It’s how you discover what to harden.”article, stated

“If you do only one thing: make your workflow inspectable (traces) and repeatable (evals).”article, stated