AI Chatbots for Customer Service: Production Lessons from 90 Days

edsdev · Tech · 2026 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer-service chatbots

EdsDev found that demo-quality support bots degrade in production unless retrieval, escalation, scope, and review loops are operated carefully.

Constraints:

• real user variance
• retrieval quality
• CSAT/deflection balance
• human escalation

Architecture

The production customer-service bot architecture is deliberately ordinary: scoped RAG, semantically chunked knowledge, source URLs, a reranker before generation, conversation-history limits, real support tools, explicit human escalation, and weekly human review of failed conversations. The source argues that model choice matters less than retrieval quality, escalation quality, logs, and operational ownership over the first 60-90 production days.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  support-chat-interface["support chat interface"]
  vector-search["vector search"]
  semantic-chunks["semantic chunks"]
  reranker["reranker"]
  llm-answer-generator["LLM answer generator"]
  escalation-tool["escalation tool"]
  human-agent["human agent"]
  weekly-review-loop["weekly review loop"]
  support-chat-interface --> vector-search
  vector-search --> semantic-chunks
  vector-search --> reranker
  reranker --> llm-answer-generator
  llm-answer-generator --> support-chat-interface
  llm-answer-generator --> escalation-tool
  escalation-tool --> human-agent
  weekly-review-loop --> semantic-chunks
  weekly-review-loop --> support-chat-interface

Components

Data flow

A customer message enters the support chat, vector search retrieves candidate chunks, a reranker narrows them, and the LLM generates an answer from the support prompt, reranked context, recent history, and tools. If the bot lacks sufficient context or the situation requires a person, the escalation tool hands off to a human agent with context. Weekly review of failures feeds back into chunking, policies, and scope.

Why this architecture

The design optimizes for production support reliability rather than demo impressiveness. The author reports that failures usually came from missing/bad retrieval context, policy contradictions, or weak escalation rather than the base model, so the architecture emphasizes chunk quality, reranking, scope discipline, escalation, monitoring, and weekly human review.

Models used: Claude 3.5 Sonnet · GPT-4o · Cohere rerank-3

Scale

• Some deployed bots handled about 60% of tickets after roughly three months.
  “We’ve had a handful of customer service bots running in production for clients for about three months now. Some are quietly handling 60% of tickets.”article, stated
• Expected real-traffic resolution starts around 30-40% and climbs into the 60s with operational review.
  “Expect resolution rate to start around 30-40% on real traffic and climb into the 60s with weekly review of failed conversations, better chunking, and a real escalation path.”article, stated
• Escalation was used in roughly one in five conversations in shipped bots.
  “It gets called on roughly 1 in 5 conversations in the bots we’ve shipped, and getting that handoff right is what makes the difference between “the bot is helpful” and “the bot is a wall.””article, stated

Alternatives considered (roads not taken)

Optimize only for deflection rate

Rejected because: The author says deflection alone can reward ignoring customers; they track resolution, escalation, response time, CSAT, and sampled human reads instead.

Choose model first and trust evals

Rejected because: Production failures came from retrieval/context problems and needed a 60-90 day improvement curve.

Build custom support AI without operational ownership

Rejected because: The build/buy decision hinges on owning retrieval and log review every week.

Patterns

rag-retrieval (support)

The primary quality driver was retrieval and chunking, not model choice.

human-in-the-loop-gate (support)

A real escalation tool with context handoff is treated as core product behavior.

Tradeoffs

Build versus buy support AI.

Chose build only when someone owns retrieval and logs over build for everyone by default.

Rationale: Without an owner for retrieval operations, a managed product may be safer.

Gotchas (what bit them)

Pre-launch evals overstate production bot accuracy.

Root cause: Real conversations expose missing context, bad chunks, and stale support paths.

Resolution: Plan for weekly log review, retrieval fixes, and scope discipline after launch.

Many apparent hallucinations were actually retrieval failures.

Root cause: The model answered from bad or missing chunks, so model swaps did not fix the underlying product problem.

Resolution: Own the retrieval layer, chunking, reranking, and weekly failed-conversation review.

Evaluation

EdsDev measured deflection with CSAT and reviewed failed conversations weekly.

Metrics: 60% ticket handling · 30-40% to 60%+ resolution · 1 in 5 escalations

Outcomes

• Some bots reached 60% ticket handling; one was shut down after six weeks.

Operational lessons post; marked non-agentic because this is primarily scoped RAG chatbot support.