Optimizing Agent Behavior and Support Operations with LangSmith Testing and Observability

podium · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agent observability and support operations

Podium needed visibility into an AI Employee agent that made many LLM calls per interaction, so quality tuning and troubleshooting did not always require engineering intervention.

Constraints:

The agent made 20-30 LLM calls per interaction.
Agent behavior was hard to inspect without observability.
Support specialists needed trace visibility for troubleshooting.
Quality improvement required dataset curation, testing, and fine-tuning.

Architecture

Podium implemented LangSmith for tracing, observability, dataset curation, testing, and fine-tuning, then exposed trace visibility to Technical Product Specialists.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ai-employee-agent["AI Employee Agent"]
  langsmith-observability["LangSmith Observability"]
  dataset-curation["Dataset Curation"]
  fine-tuning["Fine-Tuning"]
  technical-product-specialists["Technical Product Specialists"]
  ai-employee-agent --> langsmith-observability
  langsmith-observability --> dataset-curation
  langsmith-observability --> technical-product-specialists
  dataset-curation --> fine-tuning
  fine-tuning --> ai-employee-agent
  technical-product-specialists --> langsmith-observability

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`AI Employee Agent`	Handles customer/business interactions through multiple LLM calls.	`LangSmith Observability`
`LangSmith Observability`	Captures traces and supports test lifecycle visibility.	`Dataset Curation`, `Technical Product Specialists`
`Dataset Curation`	Feeds testing and fine-tuning from observed agent behavior.	`Fine-Tuning`
`Fine-Tuning`	Improves agent response quality.	`AI Employee Agent`
`Technical Product Specialists`	Troubleshoot support issues using trace visibility.	`LangSmith Observability`

Data flow

AI Employee interactions are traced through LangSmith, datasets and tests are curated from observed behavior, fine-tuning improves response quality, and Technical Product Specialists use traces to troubleshoot without escalating every issue to engineers.

“By implementing LangSmith for observability, dataset curation, and fine-tuning, they established a comprehensive testing lifecycle”summary, stated

“by empowering their Technical Product Specialists with trace visibility, they reduced engineering intervention for troubleshooting by 90%”summary, stated

Why this architecture

LangSmith connected tracing, test data, and fine-tuning so Podium could improve quality and move troubleshooting from engineers to support specialists.

“making 20-30 LLM calls per interaction without clear visibility into agent behavior.”summary, stated

“allowing engineers to focus on development rather than support tasks.”summary, stated

Models used: LangSmith · Fine-tuned AI Employee agent

Scale

Each interaction involved 20-30 LLM calls.
“20-30 LLM calls per interaction”summary, stated
F1 response quality improved from 91.7% to 98.6%.
“improved their agent's F1 response quality from 91.7% to 98.6%.”summary, stated
Engineering intervention for troubleshooting dropped by 90%.
“reduced engineering intervention for troubleshooting by 90%”summary, stated

Alternatives considered (roads not taken)

Engineer-led troubleshooting without traces

Rejected because: It consumed engineering time and hid agent behavior from support specialists.

“without clear visibility into agent behavior.”summary, stated

Quality tuning without a test lifecycle

Rejected because: Podium needed observability, dataset curation, testing, and fine-tuning connected.

“comprehensive testing lifecycle”summary, stated

Patterns

`observability-tracing` (operations)

LangSmith traces made multi-call agent behavior visible to engineers and support specialists.

“Technical Product Specialists with trace visibility”summary, stated

`eval-driven-development` (quality)

A testing lifecycle and dataset curation drove fine-tuning improvements.

“dataset curation, and fine-tuning, they established a comprehensive testing lifecycle”summary, stated

Tradeoffs

Instrument before further tuning

Chose LangSmith observability and tests over Opaque prompt/fine-tune changes.

Rationale: Observability made agent behavior debuggable and testable.

“LangSmith for observability, dataset curation, and fine-tuning”summary, stated

Move troubleshooting to technical product specialists

Chose Trace visibility for support operations over Engineering-only diagnosis.

Rationale: This reduced engineering intervention by 90%.

“reduced engineering intervention for troubleshooting by 90%”summary, stated

Gotchas (what bit them)

Multi-call agents are hard to debug without traces.

Root cause: Each interaction fan-out involved 20-30 LLM calls.

Resolution: Add LangSmith observability and expose traces to support specialists.

“20-30 LLM calls per interaction without clear visibility into agent behavior.”summary, stated

Engineering can become the bottleneck for support troubleshooting.

Root cause: Support teams lacked the trace visibility needed to diagnose agent behavior.

Resolution: Empower Technical Product Specialists with trace visibility.