Autonomous SRE Agent for Cloud Infrastructure Monitoring Using FastMCP

fuzzylabs · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Cloud infrastructure incident diagnosis

FuzzyLabs wanted an autonomous SRE agent that could triage production incidents end to end by inspecting Kubernetes, GitHub, and Slack context, then posting a diagnostic summary without manual tool-driving.

Constraints:

• the implementation is a proof of concept on a deliberately faulted demo application
• Claude Desktop-style clients required user acceptance of tool calls and token limits
• autonomous loops need timeout and stop conditions to avoid runaway cost
• security boundaries become more important if diagnosis expands into remediation
• effectiveness measurement remains an open research question

Architecture

FuzzyLabs built a custom FastMCP client around Claude and three MCP server integrations: Kubernetes for pods/logs, GitHub for source files, and Slack for reporting. CloudWatch or Slack triggers start the loop; the LLM decides which troubleshooting tools to call; the MCP client dispatches calls and returns observations; the agent diagnoses the cart-service incident, fetches RedisCartStore.cs, drafts a fix recommendation, and posts to Slack. Tool caching, filtering, a five-minute timeout, and stop conditioning make the proof of concept operationally bounded.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  cloudwatch-trigger[CloudWatch trigger] --> mcp-client[MCP client]
  slack-trigger[Slack trigger] --> mcp-client
  mcp-client --> claude-llm[Claude LLM]
  claude-llm --> mcp-client
  mcp-client --> kubernetes-mcp-server[Kubernetes MCP Server]
  mcp-client --> github-mcp-server[GitHub MCP Server]
  mcp-client --> slack-mcp-server[Slack MCP Server]
  kubernetes-mcp-server --> online-boutique-app[Online boutique app]
  github-mcp-server --> online-boutique-app
  mcp-client --> tool-cache[Tool cache]
  timeout-controller[Timeout controller] --> mcp-client
  slack-mcp-server --> slack-trigger

Components

Data flow

A CloudWatch CRITICAL 500 alert or Slack trigger sends a prompt to the MCP client. The LLM chooses tools, the client calls Kubernetes, GitHub, and Slack MCP servers, Kubernetes logs identify RedisCartStore.cs, GitHub provides the file, and Slack receives the diagnostic summary and recommendation. Caching, tool filtering, timeout, and stop conditioning constrain cost and autonomy.

Why this architecture

FuzzyLabs built its own MCP client because off-the-shelf AI applications limited autonomy through token gates and manual tool-call acceptance. MCP gives a clean perception-action loop across Kubernetes, GitHub, and Slack, while caching, filtering, timeout, and stop conditioning address cost, security surface, and runaway-loop risks.

Models used: Anthropic Claude · FastMCP client · MCP servers

Scale

• The incident prompt checks the latest 1000 Kubernetes pod logs.
  “The agent prompt specifies: check the latest 1000 Kubernetes pod logs, scroll for errors referencing a file, fetch that file from GitHub, then report findings to Slack.”article, stated
• Tool caching reduced cost per diagnosis by 83%.
  “We found that implementing this caching reduced our cost per-diagnose by 83%. * Tool filtering: We used community build MCP servers but we didn’t need all the tools they provided.”article, stated
• The agent has a five-minute timeout guard.
  “* Enforced agent timeout: we enforce a five-minute timeout on the agent where we stop the agents reasoning loop if doesn't terminate in this time period.”article, stated
• The simulated incident is an HTTP 500 from cart-service.
  “Scenario: The cart-service crashes with an HTTP 500. A user attempting to buy Fuzzy Labs loafers sees the error, which logs a CRITICAL error and triggers the agent to investigate the issue.”article, stated

Alternatives considered (roads not taken)

Use Claude Desktop or an existing MCP client

Rejected because: Token usage gatekeeping and manual tool-call acceptance prevented full autonomous workflows.

Expose all community MCP tools

Rejected because: Unneeded tools increase security risk and token usage.

Let the agent reason until completion with no explicit stop

Rejected because: Runaway reasoning loops waste cost and can get stuck.

Patterns

tool-use-orchestration (incident-response)

The MCP client coordinates Kubernetes, GitHub, and Slack tools from LLM decisions.

semantic-caching (cost)

Tool definitions and message history are cached to reduce repeated LLM cost.

safety-policy-enforcement (runtime)

Tool filtering, timeout, and stop conditioning define boundaries on autonomous action.

observability-tracing (sre)

The workflow begins from CloudWatch errors and inspects Kubernetes logs to diagnose incidents.

human-in-the-loop-gate (operations)

The current system posts diagnosis and recommendations rather than directly remediating production.

Tradeoffs

Build a custom MCP client

Chose FastMCP client directly calling Anthropic over Claude Desktop or other application clients.

Rationale: A custom client removes manual tool-call acceptance and enables autonomous SRE behavior.

Filter available tools

Chose only necessary MCP tools over all tools from community MCP servers.

Rationale: Filtering reduces both attack surface and token usage.

Stop after diagnosis in Slack

Chose diagnostic recommendation over automatic remediation.

Rationale: The team has not yet resolved security, effectiveness, and cost questions for broader authority.

"research areas.

1. Effectiveness: Does the agent actually do what it’s meant to?"

Gotchas (what bit them)

Existing MCP clients blocked full autonomy.

Root cause: They imposed token usage limits and required users to accept tool calls.

Resolution: FuzzyLabs implemented a custom FastMCP client that calls Anthropic directly.

Unbounded agent loops can waste cost and stall.

Root cause: Autonomous reasoning may continue after sufficient work or fail to terminate.

Resolution: The client enforces a five-minute timeout and stops after Slack posting.

Security boundaries become harder when moving from diagnosis to remediation.

Root cause: Giving agents authority to fix systems expands what malicious users or bad calls can exploit.

Resolution: The team keeps security as a research area before expanding remediation authority.

Effectiveness evaluation is unresolved.

Root cause: It is not obvious how to measure whether an SRE agent performs its intended functions meaningfully.

Resolution: FuzzyLabs lists effectiveness measurement as a next research area.

Evaluation

FuzzyLabs evaluates the proof of concept through a fault-injected online boutique microservice incident, a step-by-step CloudWatch-to-Slack diagnosis trace, and operational measurements such as tool-caching cost reduction, timeout behavior, and open research questions around effectiveness/security/cost.

Metrics: latest 1000 Kubernetes pod logs checked · 83% cost-per-diagnosis reduction from caching · five-minute timeout · HTTP 500 cart-service simulated incident · three research areas: effectiveness, security, cost

Outcomes

• The proof of concept diagnosed a cart-service HTTP 500 by inspecting Kubernetes logs and GitHub source.
• The agent posted a Slack diagnosis naming RedisCartStore.cs and recommending a fix path.
• Tool caching cut cost per diagnosis by 83%.
• The team identified effectiveness measurement, security, and cost as unresolved next questions.

This is a controlled proof of concept, not a broad production deployment. The concrete metrics are the latest-1000-log workflow, 83% caching cost reduction, and five-minute timeout.