Observability and Tracing

Definition

Observability and tracing records an agent's model calls, tool use, retrieval, workflow transitions, and outcomes as a connected execution history that operators can inspect at both single-run and aggregate levels. LinkedIn converts non-deterministic execution into OTel traces and analytics, while Block combines per-session inspection with production conversation and stuck-workflow monitoring. LinkedIn Block

Corpus Support

The corpus counts 329 deduplicated agentic systems (33.2%) across 16 industries. Evidence comprises 182 article, 305 transcript, and 54 summary quotes; 538 are stated and 3 implied. Corpus analysis

Problem It Solves

Agent failures emerge from sequences rather than one function call: a retrieval choice, tool response, model decision, and workflow transition may each look valid while the result fails. Sword logs inputs, retrieved documents, tool invocations, feedback, and evaluations; Prosus wraps raw tracing tools in business-facing annotation views because JSON traces alone were unusable to stakeholders. Sword Health Prosus

Applicability Signals

• Runs contain multiple model, retrieval, or tool steps whose failure must be localized. Parlance Labs
• Long-running workflows can become stuck or non-deterministic. Block
• Operators need aggregate production trends as well as one-run debugging. LinkedIn
• Domain experts must inspect behavior without reading raw telemetry. Prosus

Architecture and Variants

• End-to-end execution traces: correlate prompts, model responses, tool calls, retrieval, and workflow state. Sword Health
• Workflow operations: inspect sessions, errors, stuck work, retries, and lifecycle failures. Block
• Aggregate agent analytics: transform invocation events into OTel traces and production dashboards. LinkedIn
• Expert-facing review surfaces: translate traces into conversations and visual artifacts that business users can annotate. Prosus

Why Teams Choose It

Tracing makes probabilistic behavior inspectable without pretending it is deterministic. It supplies the evidence used to reproduce failures, prioritize fixes, construct evaluation cases, and monitor whether a release changed production behavior. Hex connects internal observability to Context Studio and long-horizon evaluation rather than treating telemetry as a passive log archive. Hex

Tradeoffs

• Rich traces improve diagnosis but can expose customer data; Hex notes that raw agent traces cannot simply be shown for production analytics. Hex
• Generic tracing accelerates instrumentation but often needs custom inspection UX for domain reviewers. Prosus
• Logging every intermediate event increases storage and analysis load, yet sparse logs prevent root-cause analysis. Sword Health

Gotchas and Resolutions

• Classic request metrics miss branching non-deterministic behavior; aggregate agent events into semantic traces. LinkedIn
• Raw JSON traces exclude business experts from review; provide task- and domain-specific annotation surfaces. Prosus
• Durable workflow history can grow while still omitting model-level meaning; retain workflow state and semantic agent events as distinct linked records. Block

When Not to Use It

Do not capture unrestricted prompts, retrieved records, or outputs merely because tracing infrastructure permits it. For sensitive workloads, log the minimum structure needed for diagnosis and provide controlled access or higher-level views. Hex's customer-data constraint and Sword's healthcare setting make that limit concrete. Hex Sword Health

Boundaries and Related Patterns

observability-tracing records what happened. eval-driven-development applies criteria and datasets to decide whether behavior is acceptable. A trace can become an eval case, but telemetry is not a score. durable-execution persists workflow state and retry history; tracing explains the semantic agent behavior across that history. Hex Block

Operational Checklist

• Correlate model, retrieval, tool, workflow, and user-feedback events under one run identity. Sword Health
• Preserve both per-run drill-down and aggregate production views. LinkedIn
• Redact or avoid sensitive payloads and control trace access. Hex
• Give domain reviewers a usable annotation surface rather than raw event JSON. Prosus

Representative Implementations

Methodology and Limitations

This page synthesizes a representative subset of 329 deduplicated systems. Published cases describe what teams capture more often than the privacy, retention, and cost characteristics of their telemetry, so recurrence does not establish an optimal tracing depth. Corpus methodology