Evaluation Patterns for Deep Agents in Production

langchain · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: production evaluation for stateful long-running agents

LangChain needed evaluation patterns for Deep Agents that go beyond final-response grading because production agents produce trajectories, tool calls, files, memories, and other state over single-step, full-turn, and multi-turn interactions.

Constraints:

Each deep-agent datapoint can have different success criteria.
Regressions may happen at individual decision points rather than final answers.
Stateful agents need clean, reproducible environments per test run.
Multi-turn tests can go off rails if hardcoded follow-up inputs no longer make sense.
External API dependencies can make evals slow, expensive, and difficult to debug.

Architecture

LangChain built evaluation patterns around LangSmith Pytest and Vitest integrations. Tests can assert over agent trajectory, final response, and generated state. Single-step evals interrupt the agent before tool execution for decision checks, full-turn evals inspect end-state artifacts, multi-turn evals use conditional logic to stay on rails, and environment setup uses temporary directories, containers, or mocked/replayed API calls for reproducibility.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  deep-agents-application["Deep Agents application"] --> test-runner["Test runner"]
  reproducible-environment["Reproducible environment"] --> test-runner
  test-runner --> trajectory-and-state-capture["Trajectory and state capture"]
  trajectory-and-state-capture --> assertions-and-judges["Assertions and judges"]
  assertions-and-judges --> langsmith-experiment-log["LangSmith experiment log"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Deep Agents application`	Stateful agent such as a coding agent, in-app assistant, email assistant, or agent builder.	`Test runner`
`Test runner`	Pytest or Vitest integration that executes single-step, full-turn, or multi-turn agent tests.	`Trajectory and state capture`
`Trajectory and state capture`	Records tool calls, tool arguments, final response, files, memories, links, and other artifacts.	`Assertions and judges`
`Assertions and judges`	Applies bespoke code assertions, regex checks, and LLM-as-judge criteria per datapoint.	`LangSmith experiment log`
`Reproducible environment`	Temporary directory, Docker sandbox, mock, or HTTP replay setup reset for each test.	`Test runner`
`LangSmith experiment log`	Stores traces, failures, metrics, token use, and results over time.	—

Data flow

A test case supplies an agent input and success criteria, runs either a single step, full turn, or multi-turn interaction, captures trajectory, final response, and state artifacts, logs the test to LangSmith, and applies custom assertions or LLM-as-judge feedback. Reproducible environments and API replay keep failures debuggable.

“You can make different assertions about the agent’s trajectory, final message, and state for each test case.”article, stated

“This LangSmith integration automatically logs all test cases to an experiment”article, stated

Why this architecture

Deep agents need a layered eval architecture because their correctness is spread across trajectory, state, artifacts, and final response. Single-step tests are fast decision checks, full turns validate end state, and multi-turn tests simulate realistic interaction while conditional logic keeps the test meaningful after deviations.

“Deep Agents breaks this assumption. You’ll want to test more than just the final message.”article, stated

“Think of single-step evals as your “unit tests” that ensure the agent takes the expected action in a specific scenario.”article, stated

Models used: Deep Agents harness · LangSmith Pytest · LangSmith Vitest · LangGraph · LLM-as-judge · Harbor · Docker · vcr

Scale

LangChain shipped four Deep Agents applications in a month.
“Over the past month at LangChain, we shipped four applications on top of the Deep Agents harness:”article, stated
The test patterns cover single-step, full-turn, and multi-turn execution.
“Running a deep agent for a single-step”article, stated

Alternatives considered (roads not taken)

Evaluate only the final response.

Rejected because: Deep agents also produce tool trajectories and state artifacts that may be the real success criteria.

“The “success criteria” may be also more specific to each datapoint, and may involve specific assertions against the agent’s trajectory and state.”article, stated

Run full agent sequences for every regression test.

Rejected because: Single-step evals catch decision-point regressions earlier and save tokens.

“Regressions often occur at individual decision points rather than across full execution sequences.”article, stated

Hardcode multi-turn conversations without checks between turns.

Rejected because: If the agent deviates, later hardcoded user inputs may no longer make sense.

“if you naively hardcode a sequence of inputs and the agent deviates from the expected path, the subsequent hardcoded user input may not make sense.”article, stated

Patterns

`eval-driven-development` (deep-agent testing)

LangChain adds evals for each Deep Agents application and uses multiple run modes to test different behavior surfaces.

“Building and shipping these agents meant adding evals for each of them”article, stated

`observability-tracing` (agent traces)

LangSmith logs test cases and full agent turns as traces for debugging and result tracking.

“you can view traces for a failed test case (to debug what went wrong) and track results over time.”article, stated

`tool-use-orchestration` (trajectory assertions)

Full-turn tests can assert that a required tool was called somewhere in the agent trajectory.

“ensure that a particular tool was called at some point during action, but it doesn’t matter exactly when.”article, stated

`llm-as-judge` (custom assertions)

Some datapoints use an LLM judge with success criteria when code assertions are insufficient.

“use an LLM-as-judge with specific success criteria”article, stated

Tradeoffs

Use single-step evals for decision points.

Chose Interrupt before tool execution over Always executing a complete agent sequence.

Rationale: Single-step tests save tokens and isolate the decision that regressed.

“interrupting before the tool node allows us to inspect the tool call args”article, stated

Use full-turn evals for end-state behavior.

Chose Trajectory, final response, and state assertions over Only testing isolated decisions.

Rationale: Full turns show the complete execution picture and let teams inspect artifacts the agent created.

“Full agent turns give you a complete picture of your agent execution.”article, stated

Gotchas (what bit them)

Deep-agent evals become flaky without environment reset.

Root cause: Stateful long-running agents leave files, memories, and external state behind.

Resolution: Use a fresh clean environment for each eval run, such as temporary directories or containers.

“Deep Agents need a fresh, clean environment for each eval run in order to ensure reproducible results.”article, stated

Live API dependencies make tests slow and expensive.

Root cause: Some agents depend on real external services and state.

Resolution: Record HTTP requests and replay them during test execution.

“Running evals against live services can be slow and expensive. Instead, record HTTP requests into a filesystem and replay them during test execution.”article, stated

Multi-turn test scripts can become invalid after an early divergence.

Root cause: Hardcoded follow-up inputs assume the agent followed an expected path.

Resolution: Use conditional logic after each turn and fail early if the state is not expected.

“We addressed this by adding conditional logic in our Pytest and Vitest tests.”article, stated

Evaluation

LangChain uses bespoke per-datapoint logic, single-step evals, full-agent turns, multi-turn tests with conditional logic, reproducible test environments, and mocked or replayed API calls, all logged through LangSmith experiments.

Metrics: Trajectory assertions · Final response checks · Generated state/artifact checks · Latency · Token use · Tool invocation correctness · LLM-as-judge feedback

“LangSmith makes it really easy to view your full agent turns as traces, where you can see high level metrics like latency and token use”article, stated

Outcomes

LangChain shipped four Deep Agents applications and distilled five evaluation patterns.
The patterns cover custom test logic, single-step tests, full-turn tests, multi-turn tests, and reproducible environments.
LangSmith integrations allow failed tests to be debugged via traces and tracked over time.

The article reports four Deep Agents applications shipped over one month and focuses on reusable testing patterns rather than product usage numbers.

“In this post, we’ll be diving deep into the following patterns for evaluating deep agents.”article, stated