Building Production-Grade AI Agents with Observability, Evaluation, and Insights

langchain · Tech · 2025 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agent observability and evaluation

LangChain needed LangSmith to support teams moving from prototype agents to production systems with millions of traces, where developers must debug exact agent behavior, discover unknown production patterns, evaluate multi-turn sessions, and turn bad interactions into regression tests.

Constraints:

• Production agents generate large volumes of heterogeneous traces, including non-chat background-agent payloads.
• Teams need to know both known failure modes and unknown patterns in user behavior or agent mistakes.
• Single-turn evals do not capture whole conversations, tool-call trajectories, user sentiment, or session-level cost.
• Offline evals cannot guarantee full coverage of real user behavior, but online evals alone do not replace regression testing.
• Nontechnical stakeholders need access to prompt iteration and experiment workflows.

Architecture

LangSmith combines tracing, offline eval datasets, prompt playground/hub, production trace ingestion, Insights, threads, thread-based evals, online evals, dashboards, automation rules, and trace-to-dataset flows. Tracing captures the exact steps of agent execution. Insights automatically clusters and categorizes production traces to reveal user/product usage, agent errors, hallucinations, wrong-tool patterns, retrieval failures, and unknown question types. Threads group related traces by conversation or session so evaluators and dashboards can judge multi-turn behavior. Offline evals remain regression tests, while online evals run in real time over production traces.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-agent["Production agent"] --> trace-ingestion["Trace ingestion"]
  production-agent --> thread-identifiers["Thread identifiers"]
  production-agent --> online-evaluators["Online evaluators"]
  trace-ingestion --> tracing-debugger["Tracing debugger"]
  trace-ingestion --> insights-engine["Insights engine"]
  trace-ingestion --> dataset-builder["Dataset builder"]
  tracing-debugger --> prompt-playground["Prompt playground"]
  prompt-playground --> prompt-hub["Prompt hub"]
  prompt-playground --> offline-eval-runner["Offline eval runner"]
  prompt-hub --> offline-eval-runner
  dataset-builder --> offline-eval-runner
  insights-engine --> insight-configuration["Insight configuration"]
  insights-engine --> product-usage-analysis["Product usage analysis"]
  insights-engine --> error-analysis["Error analysis"]
  insights-engine --> dashboards["Dashboards"]
  error-analysis --> dataset-builder
  thread-identifiers --> thread-store["Thread store"]
  thread-store --> thread-eval-runner["Thread eval runner"]
  thread-eval-runner --> online-evaluators
  thread-eval-runner --> dashboards
  online-evaluators --> dashboards
  offline-eval-runner --> dashboards
  dashboards --> automation-rules["Automation rules"]
  automation-rules --> dataset-builder

Components

Data flow

An agent or LLM application sends traces to LangSmith, including prompts, model calls, tool calls, payloads, and session identifiers. Tracing lets engineers debug individual runs. Offline eval flows build datasets of known examples and run metrics/evaluators. Production trace streams feed Insights, which categorizes trace patterns for product usage and error analysis, and those discoveries can become new eval monitors or regression datasets. Thread UUIDs group related traces into conversations or sessions, enabling thread-level metrics, sentiment checks, tool-trajectory evals, online production evaluators, dashboards, spot-checking workflows, and automation rules.

Why this architecture

LangSmith pairs discovery and evaluation because production agents create both known risks and unknown interaction patterns. Traces give run-level debuggability, Insights bubbles up patterns teams did not predefine, thread evals capture multi-turn behavior, online evals monitor live traffic, and offline evals preserve known failures as regression tests for future versions.

Models used: LLM-based Insights algorithm · Online evaluators · Offline evaluators · LangSmith agent traces

Scale

• LangSmith customers had moved from v0/v1 agents into production.
  “customers are not just going to the sort of v 0ero to v1 creating their first agent and trying to get it out in front of their users.”transcript, stated
• The platform receives millions of traces per day.
  “we're we're getting these millions of traces a day.”transcript, stated

  “They've got agents in production. They've started to get millions of traces per day.”transcript, stated
• Insights was inspired by work over millions of Claude conversations.
  “over the millions and millions of conversations that people are having with quad what are the general topics of conversation”transcript, stated
• Online evals run in real time as production traces arrive.
  “evaluators that measure your agents performance in real time as traces are coming in.”transcript, stated
• Offline evals do not provide a 100% guarantee of production behavior.
  “offline evals are never going to fully give you like a 100% guarantee that your agent is going to behave the way you want them to.”transcript, stated

Alternatives considered (roads not taken)

Only inspect individual traces manually

Rejected because: At millions of traces per day, customers need automated trend discovery and categorization across production traffic.

Use evals alone for all quality work

Rejected because: Evals test known expectations, while Insights discovers patterns and unknown unknowns that teams did not define ahead of time.

Evaluate only single turns or single traces

Rejected because: Sentiment, tool-call trajectories, repeated tool loops, and full user experience require thread-level context.

Replace offline evals with A/B or online testing only

Rejected because: The team argues offline evals remain useful as regression tests, even though they cannot fully cover production behavior.

Assume the same trace structure as chatbot conversations

Rejected because: Modern agents include background agents and arbitrary payloads, so Insights had to generalize beyond chat history.

Patterns

observability-tracing (operations)

LangSmith traces exact agent execution steps so teams can debug and observe production LLM applications.

eval-driven-development (evaluation)

LangSmith supports offline datasets, online evaluators, thread evals, and regression tests for shipping new versions of agent applications.

llm-as-judge (evaluation)

Evaluators are applied to traces and threads to score sentiment, tool trajectories, performance, and production behavior.

data-quality-loop (learning)

Bad production interactions and discovered failure modes are converted into datasets and future regression evals.

prompt-versioning (development)

Prompt playground and prompt hub allow product and nontechnical stakeholders to iterate on prompts and kick off experiments.

observability-tracing (insights)

Insights categorizes production traces into usage and error patterns so teams can find unknown unknowns at scale.

Tradeoffs

Use Insights for discovery and evals for known checks.

Chose A two-layer loop where unknown patterns become future evals over Treating discovery and evaluation as the same job.

Rationale: Insights finds unknown patterns in traces; evals then monitor specific known behaviors or failures.

Evaluate complete threads when interaction quality spans multiple traces.

Chose Thread-level evals and metrics over Only single-trace evals.

Rationale: Whole-session checks capture sentiment, tool trajectories, loops, and user journeys that are invisible in isolated turns.

Keep offline evals despite incomplete coverage.

Chose Offline regression sets plus online evals over A/B testing and online testing only.

Rationale: Offline evals cannot guarantee full production coverage, but they still protect known important behaviors when shipping new versions.

Make Insights configurable rather than fully automatic.

Chose Built-in modes plus user-exposed algorithm knobs over Opaque automatic categorization.

Rationale: Users know their agent domain and can guide the insight algorithm toward relevant patterns.

Turn production failures into regression data.

Chose Trace/thread-to-dataset flows over One-off manual debugging of each bad interaction.

Rationale: Once a bad production interaction is known, it should guard future releases rather than only fixing the immediate incident.

Gotchas (what bit them)

Trace volume hides the important patterns.

Root cause: Production customers have millions of traces per day, making manual inspection insufficient for usage and error discovery.

Resolution: Add Insights to automatically categorize user behavior and agent failure modes across traces.

Insights over arbitrary agent payloads are harder than chatbot analytics.

Root cause: Agents now include background agents and non-chat payloads without a simple conversation history.

Resolution: Generalize the category-discovery algorithm beyond chatbot conversations and let users provide agent-specific context.

Single-turn evals miss session-level failures.

Root cause: Sentiment, repeated tool calls, progress toward goals, and frustration emerge over multiple related traces.

Resolution: Use threads as the evaluation unit for conversations or user sessions.

Offline evals cannot cover real production behavior completely.

Root cause: Users attempt interactions that differ from what product builders anticipated.

Resolution: Combine offline evals with online evaluators over production data and continuously backfill new bad interactions into datasets.

Discovery output still needs domain guidance.

Root cause: Insights is itself an LLM-based application and produces better results when given clear instructions and context.

Resolution: Expose configuration knobs for agent description and desired insight types.

Evaluation

LangSmith evaluates agents through offline datasets and metrics, online evaluators over live production traces, thread-level evaluators for multi-turn sessions, dashboarded thread/cost metrics, and trace-to-dataset regression loops. Insights is explicitly positioned as discovery that informs later eval creation rather than as a direct substitute for evals.

Metrics: Offline eval metrics over known datasets · Online evaluator scores over production traces · User sentiment across threads · Repeated tool-call loops · Tool-call trajectory quality · Cost per user interaction · Failure-mode frequency · Product-feature usage through copilots · Negative user feedback spot-checks · Regression test pass/fail for new versions

Outcomes

• LangSmith launched Insights to automatically find trace trends for usage and error analysis.
• Insights can surface product areas users engage through an embedded copilot.
• Insights can categorize agent mistakes such as wrong tools, explicit errors, hallucinations, and retrieval failures.
• Thread evals let teams evaluate conversations, sessions, tool trajectories, sentiment, and end-to-end interactions.
• Online evals run against production data while offline evals remain regression tests for new versions.
• The launch aims to reduce the number of unknowns when agents enter production.

The source repeatedly cites customers with agents in production and millions of traces per day, and positions the new features around extracting value from production traces rather than only prototype datasets.