Autonomous Observability with AI Agents and Model Context Protocol

pinterest · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production observability and root-cause analysis

Pinterest needed AI agents to reason across fragmented observability silos - metrics, logs, traces, alert data, postmortems, and change events - without forcing a full infrastructure rewrite to OpenTelemetry-era correlated data.

Constraints:

Architecture

Pinterest built an internal observability MCP server as the context bridge for Tricorder Agent. The server exposes controlled tools for metrics, postmortem documents, logs, traces, and alert information, letting an agent gather relevant data from fragmented systems and produce links, hypotheses, suggestions, and next steps. To avoid context overload, early use cases generate filtered dashboard links rather than raw data, tool documentation instructs small-window querying, and the team is testing server-side summarization with a fresh LLM context.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  engineer-alert-input[Engineer alert input] --> tricorder-agent[Tricorder Agent]
  tricorder-agent --> observability-mcp-server[Observability MCP server]
  observability-mcp-server --> metrics-tool[Metrics tool]
  observability-mcp-server --> logs-tool[Logs tool]
  observability-mcp-server --> traces-tool[Traces tool]
  observability-mcp-server --> alert-information-tool[Alert information tool]
  observability-mcp-server --> postmortem-document-tool[Postmortem document tool]
  tricorder-agent --> dependency-graph[Dependency graph]
  tricorder-agent --> filtered-dashboard-links[Filtered dashboard links]
  server-side-summarizer[Server-side summarizer] --> observability-mcp-server
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Engineer alert input Alert link or number supplied by an engineer to begin analysis. Tricorder Agent
Tricorder Agent AI agent that gathers context, follows clues across observability data, hypothesizes causes, and returns guidance. Engineer alert input, Observability MCP server, Filtered dashboard links, Dependency graph
Observability MCP server Central controlled interface that exposes observability tools and context to agents. Tricorder Agent, Metrics tool, Logs tool, Traces tool, Alert information tool, Postmortem document tool, Server-side summarizer
Metrics tool Queries metrics from the time-series database. Observability MCP server
Logs tool Fetches logs for relevant services and time ranges. Observability MCP server
Traces tool Fetches traces for services and time ranges. Observability MCP server
Alert information tool Fetches triggered alert metrics, service identifiers, and time range. Observability MCP server
Postmortem document tool Retrieves incident postmortems when relevant. Observability MCP server
Dependency graph Lets agents explore incoming and outgoing service dependencies during investigations. Tricorder Agent
Filtered dashboard links Return filtered links instead of raw data to conserve context for on-call workflows. Tricorder Agent
Server-side summarizer Planned LLM inside the server that summarizes large data before returning it to the connected agent. Observability MCP server

Data flow

An engineer provides Tricorder with an alert link or number. The agent calls the MCP server to retrieve alert metadata, relevant services, time ranges, metrics, logs, traces, and postmortems. For early workflows it generates dashboard links filtered to the right services and periods; for deeper analysis it can query tools across dependency graphs, build hypotheses, and return suggestions and next steps.

3 supporting quotes
“The engineer can provide the Tricorder with their alert link/number and sit back while it gathers the relevant information for their investigation.”article, stated
“Currently, it provides models with tooling for accessing the following data: * ChangeFeed Events: Finds events related to the service of interest; for instance: deploys, feature flag activations, or experiment rollouts”article, stated
“Tricorder queries our services directly to understand what is going on and hypothesize a cause, providing suggestions and next steps”article, stated

Why this architecture

Pinterest chose MCP as a pragmatic bridge because a full OpenTelemetry-style observability rebuild was infeasible. The MCP server gives agents controlled access to fragmented observability pillars, lets the observability team decide which tools and subsets are exposed, and can grow with plug-and-play tools. Link generation and small-window tool instructions protect model context from Pinterest-scale data volume.

3 supporting quotes
“However, our o11y infrastructure was set up before conventions and tools like OTel were available, and it is not feasible to overturn our entire o11y infrastructure in order to incorporate them into our stack.”article, stated
“MCP was the perfect fit to help us sidestep our limitations and begin to drive Pinterest o11y into a new era as it grants the following:”article, stated
“By providing the MCP server as an interface to our data, we prevent the agents access to everything, allowing us to maintain tighter safety and privacy controls.”article, stated

Models used: Tricorder Agent · LLMs acting as agents · server-side summarization LLM

Scale

Alternatives considered (roads not taken)

Full infrastructure rewrite to modern OTel-style correlation

Rejected because: Pinterest's mature observability stack was too large and fragmented to overturn wholesale.

“However, our o11y infrastructure was set up before conventions and tools like OTel were available, and it is not feasible to overturn our entire o11y infrastructure in order to incorporate them into our stack.”article, stated

Let agents access raw observability data organically

Rejected because: Pinterest-scale metrics/logs/traces would cause agents to query too much and crash their context windows.

“If we allow an agent to organically look through this data, it would end up querying for too much at a time (even if it was only querying for a 15 min window), breaking its context window and causing itself to crash.”article, stated

Give every team independent direct observability access for their agents

Rejected because: A central MCP server preserves observability-team control over access, privacy, and allowed capabilities.

“If not, other teams could have independently developed them on their own, incorrectly accessing our data or giving their agents the ability to alter data that should not be changed.”article, stated

Return full raw datasets to the connected agent

Rejected because: The team is testing server-side summarization to conserve context and avoid returning too much data.

“They are looking at building a similar agent, where we leverage an additional LLM within the server (with a fresh context) to summarize the data.”article, stated
“only return a summary to the agent connected to the MCP server which, in theory, would conserve a lot of context space.”article, stated

Patterns

tool-use-orchestration (agentic)

The MCP server exposes observability tools for metrics, logs, traces, alerts, postmortems, and dependency context.

“provides an agent different tools that it can utilize when working to resolve a request”article, stated
“Metrics:** Queries metrics from our time series database”article, stated

context-compaction (memory)

The design constrains query windows, returns filtered links, and tests server-side summaries to avoid overflowing the agent context.

“we overestimated the amount of information that a model could take while also underestimating the amount of data that we own as a team.”article, stated
“include instructions to only query for a very small period and to call it again until the wanted time period is covered.”article, stated

observability-tracing (operations)

Tricorder follows clues across logs, metrics, traces, alerts, postmortems, and dependency graphs to aid root cause analysis.

“Following clues across logs, metrics, traces, and change events”article, stated
“The agents use tools on multiple parts of the graph, exploring all the incoming and outgoing dependencies”article, stated

guardrails-io-filtering (governance)

The observability team controls which tools and data subsets agents can access through the MCP server.

“we get to decide what information and ability agents interacting with our team’s data actually get”article, stated
“maintain tighter safety and privacy controls.”article, stated

rag-retrieval (retrieval)

The MCP server retrieves postmortem documents and relevant observability data for agent analysis.

“Post Mortem Documents** : After having previously ingested incident Post-Mortem documents into a database, fetches them for analysis when relevant”article, stated

Tradeoffs

Build an MCP bridge instead of replacing the observability stack

Chose central MCP server over existing fragmented systems over complete OpenTelemetry-style infrastructure overhaul.

Rationale: The rewrite was infeasible, while MCP could expose unified context without replacing mature systems.

“it is not feasible to overturn our entire o11y infrastructure in order to incorporate them into our stack.”article, stated
“MCP was the perfect fit to help us sidestep our limitations”article, stated

Return links for early on-call workflows

Chose filtered dashboard links with relevant services and time periods over having the agent parse all raw metrics/logs/traces.

Rationale: For information-gathering use cases, engineers need the right filtered views more than the agent needs all raw data in context.

“For this use case, the agent does not need to parse the raw data. Instead, it only needs to know of relevant time periods and services.”article, stated
“saving an on-call engineer the time spent jumping around all our interfaces and filtering.”article, stated

Keep MCP tool extensibility

Chose plug-and-play tools that can be removed, replaced, or expanded over a fixed one-off observability agent interface.

Rationale: Pinterest expects the server to grow as the observability team adds more tools and agentic use cases.

“The tools within can easily be removed, replaced, and expanded upon without changing the overall system.”article, stated
“our server can easily grow and change with our team”article, stated

Gotchas (what bit them)

Fragmented observability slows root-cause analysis.

Root cause: Logs, metrics, traces, and change events were implemented as separate tools and pipelines without clear cross-pillar matching.

Resolution: Pinterest built an MCP server that lets agents query across data pillars from one controlled interface.

“logs over here, traces over there, and metrics somewhere else.”article, stated
“an on-call engineer must jump around multiple unique interfaces when root causing an issue”article, stated

Raw observability volume can crash agent context.

Root cause: Pinterest processes billions of metrics and terabytes of logs/traces, far beyond what a model context can hold.

Resolution: Early use cases return filtered dashboard links; tool metadata instructs small-window queries; summarization is being tested.

“breaking its context window and causing itself to crash.”article, stated
“the agent generate links to the dashboards containing that information”article, stated

Opening observability data to agents creates privacy and safety risk.

Root cause: Independent direct access could expose too much data or allow agents to alter things they should not change.

Resolution: The o11y team centralizes access through MCP and controls which tools and abilities agents receive.

“incorrectly accessing our data or giving their agents the ability to alter data that should not be changed.”article, stated
“By providing the MCP server as an interface to our data, we prevent the agents access to everything”article, stated

Evaluation

Pinterest evaluates the system through internal availability of the MCP server, Tricorder on-call workflows, context-window failure prevention, hackathon adoption by other teams, and ongoing tests of server-side summarization.

Metrics: 3 billion data points per minute · 12 billion tag key/value combinations per minute · 7 TB logs per day · 7 TB traces per day · 15-minute windows can still overflow context · multiple hackathon teams depended on the MCP server

3 supporting quotes
“The o11y team processes around 3 billion data points per minute, 12 billion keys”article, stated
“7 TB of logs per day, and 7 TB of traces per day”article, stated
“multiple teams developed projects that depended on our MCP server”article, stated

Outcomes

The article is Part 1 and focuses on architecture and data scale rather than final MTTR reduction metrics.

3 supporting quotes
“It is now available internally for Pinterest engineers to use and is a central part of our move towards autonomous o11y.”article, stated
“Faster, clearer root-cause analysis, and actionable guidance for our engineers”article, stated
“creates a hub for engineering teams at Pinterest to be able to access our data for their own agentic needs”article, stated