Leveraging LangSmith for Debugging Tools & Actions in Production LLM Applications

mendable · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: agentic LLM application debugging

Mendable's Tools & Actions feature made assistants capable of API-backed actions, but agent loops with custom tools, just-in-time AI-generated inputs, and production pipelines became difficult to debug without trace-level observability.

Constraints:

• Users can create custom tools through API requests, including AI-filled values generated at request time.
• The agent has access to multiple resources and is embedded in a production pipeline.
• Latency and prompt bloat matter because tool/action runs stream to end users.
• Enterprise deployments require reliable tool schemas, evaluations, and debugging data rather than console logs.

Architecture

Mendable integrated LangSmith into its LangChain/OpenAI-tool-agent stack for Tools & Actions. LangSmith traces expose run hierarchy, inputs, outputs, parameters, response times, feedback, token consumption, and tool inputs. The team used traces to identify a 7.23-second ChatOpenAI call caused by concatenating RAG prompts/sources with tool/action prompts, inspect AI-generated tool query parameters, refine tool descriptions and schemas, and save representative runs into datasets for future evaluation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Mendable_assistant["Mendable assistant"] --> Tools_and_Actions_module["Tools and Actions module"]
  Tools_and_Actions_module --> Custom_API_tools["Custom API tools"]
  Custom_API_tools --> Tool_schema_descriptions["Tool schema descriptions"]
  Tools_and_Actions_module --> LangChain_OpenAI_tool_agents["LangChain OpenAI tool agents"]
  LangChain_OpenAI_tool_agents --> LangSmith_tracing["LangSmith tracing"]
  Mendable_assistant --> LangSmith_tracing
  LangSmith_tracing --> Dataset_builder["Dataset builder"]

Components

Data flow

A user asks a GTM assistant for prospect initiatives. The agent calls APIs such as CRM, Google News or DUNS, and CoreSignal, fills AI-generated tool parameters from the user's question and schema, then synthesizes sales guidance. LangSmith captures each agent run and tool call as a trace, letting Mendable inspect slow model calls, generated query parameters, tool schemas, and selected runs that become evaluation datasets.

Why this architecture

Mendable needed observability at the agent-loop and tool-input level because console logs could not explain why an agent failed to call a tool, generated the wrong tool schema input, or slowed down from prompt bloat. LangSmith fit because it integrated with their existing LangChain agent stack and exposed the exact run hierarchy and metrics needed to debug production agent behavior.

Models used: LangChain OpenAI tool agents · ChatOpenAI · LangSmith · RAG pipeline · CRM API · Google News API · DUNS API · CoreSignal API

Scale

• Mendable equipped about 1,000 customer success and sales users at a company valued over $20 billion.
  “we equipped ~1000 customer success + sales people at a $20+ billion tech company with GTM assistants”article, stated
• The reported deployment achieved $1.3 million in savings over five months and projected $3 million annual savings.
  “In five months, the platform achieved $1.3 million in savings, and it's projected to save $3 million this year”article, stated
• A trace revealed a 7.23-second ChatOpenAI call caused by prompt concatenation.
  “the last call to `ChatOpenAI` took a long time: 7.23 seconds.”article, stated
• Mendable ran the same tool roughly 20 times to inspect output and schema accuracy.
  “we ran the same tool with different queries ~20 times and quickly scrolled through LangSmith making sure the output and schema were accurate”article, stated

Alternatives considered (roads not taken)

Debug with console.log statements

Rejected because: Console logs became messy and did not give visibility into agent behavior, tool calls, or custom-tool correctness.

Concatenate all RAG prompts and sources into tool/action prompts

Rejected because: Tracing showed the combined prompt was massive and delayed streaming.

Assume generated tool inputs are correct

Rejected because: Mendable found tool descriptions were critical to generating correct schemas and inputs.

Patterns

tool-use-orchestration (agent architecture)

Mendable's assistants call CRM, news, DUNS, CoreSignal, and custom user-created API tools to move beyond Q&A into action-based workflows.

observability-tracing (operations)

LangSmith traces expose agent run hierarchy, tool calls, inputs, outputs, latency, feedback, and token consumption for debugging production behavior.

eval-driven-development (quality)

Mendable converts selected traced runs into LangSmith datasets for evaluation after optimization experiments.

latency-optimization (serving)

Tracing identified prompt concatenation as the cause of a slow ChatOpenAI call, leading the team to reduce which prompt chunks Tools & Actions used.

structured-output-contract (tooling)

Custom tools require accurate schemas and AI-generated request parameters, so Mendable inspected and improved tool descriptions to make generated inputs reliable.

Tradeoffs

Make assistants action-capable

Chose Tools & Actions with API calls over Simple question-answering assistants.

Rationale: Mendable saw workflow value in letting assistants gather external data and synthesize it into actions, but accepted more observability and reliability complexity.

Adopt LangSmith tracing

Chose Dedicated LangSmith debugging and evaluation over Console logs and ad hoc manual debugging.

Rationale: LangSmith integrated with their LangChain agent stack and exposed the run data needed to debug slow and incorrect tool behavior.

Gotchas (what bit them)

Agent tools are hard to debug without trace-level visibility.

Root cause: Custom tools, AI-generated values, multiple resources, and production pipelines hide key decisions inside the agent loop.

Resolution: Use LangSmith tracing to inspect run hierarchy, parameters, response times, feedback, tokens, and tool inputs.

Prompt bloat can hide inside a single slow model call.

Root cause: The Tools & Actions prompt had concatenated RAG prompts and sources, creating a massive prompt.

Resolution: Use trace latency to identify the slow call and reduce the prompt chunks used by Tools & Actions.

AI-generated tool inputs depend heavily on tool descriptions.

Root cause: The AI fills request values from the user's question and schema at runtime.

Resolution: Inspect repeated tool runs in LangSmith, annotate failures, improve the AI-generated product surface, and instruct users to provide detailed tool descriptions.

Evaluation

Mendable uses LangSmith traces for debugging and saves selected runs into datasets for further evaluation, with manual inspection of repeated tool executions and schema accuracy.

Metrics: tool input and schema accuracy · response time per model/tool call · feedback · token consumption · dataset coverage for future evaluation

Outcomes

• LangSmith gave Mendable visibility into agent decision-making and tool behavior.
• The team identified and reduced latency from prompt bloat in a 7.23-second ChatOpenAI call.
• The Tools & Actions feature launched and Mendable was still tailoring custom actions with enterprises.

Mendable reports a deployment to roughly 1,000 GTM users at a $20+ billion company, $1.3 million savings in five months, and projected $3 million annual savings; this is self-reported in a partner blog.