Building a Production AI Slack Bot with Pydantic AI and Logfire

tiger_data · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI Slack assistant for remote-company knowledge work

Tiger Data needed a production Slack bot that could help employees recover context from company conversations and tools without drowning in Slack history, while remaining reliable, debuggable, scalable, and flexible across LLM providers.

Constraints:

• a production agent needs tool integrations, retries, provider switching, debugging, and logging beyond the LLM call itself
• the agent had to connect to multiple internal MCP servers for Slack, docs, Salesforce, GitHub, Linear, meetings, memory, and reports
• agent failures are hard to debug without visibility into prompts, tool calls, arguments, responses, and service traces
• the system needed to handle thousands of concurrent conversations
• Tiger Data wanted abstraction without locking the whole architecture into an opinionated framework

Architecture

Tiger Data built Tiger Agent for Work as a Slack bot backed by a PostgreSQL work queue, Pydantic AI for LLM/provider/tool orchestration, eight MCP servers for company data, and Pydantic Logfire for distributed tracing. Pydantic AI loads MCP servers from configuration, translates tools across providers, retries tool calls, and supports model switching by environment variable. Logfire instruments database, LLM, MCP, and HTTP calls so each Agent Run is visible as structured trace data queryable with SQL.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  slack-interface[Slack interface] --> postgresql-work-queue[PostgreSQL work queue]
  postgresql-work-queue --> tiger-agent[Tiger Agent]
  tiger-agent --> pydantic-ai-agent[Pydantic AI agent]
  pydantic-ai-agent --> mcp-servers[MCP servers]
  pydantic-ai-agent --> llm-provider[LLM provider]
  tiger-agent --> logfire-tracing[Logfire tracing]
  pydantic-ai-agent --> logfire-tracing
  mcp-servers --> logfire-tracing
  postgresql-work-queue --> logfire-tracing
  logfire-tracing --> logfire-mcp-server[Logfire MCP server]
  logfire-mcp-server --> claude-code[Claude Code]

Components

Data flow

A Slack mention enters Tiger Agent's PostgreSQL-backed work queue. The agent loads MCP server definitions from configuration, creates a Pydantic AI agent with those servers as toolsets, calls the selected LLM provider, invokes MCP tools as needed, retries failures automatically, and returns a Slack response. Logfire captures the database work, LLM prompts, tool invocations, arguments, responses, latency, and HTTP/MCP spans into structured trace data.

Why this architecture

Tiger Data chose Pydantic AI because it abstracted MCP, retries, and provider APIs without forcing a different event model, and chose Logfire because agent debugging required seeing the entire tool/LLM workflow as structured traces. Keeping a PostgreSQL work queue preserved the team's scaling architecture, while environment-based model switching let them handle different model behavior, cost, uptime, and outages.

Models used: Claude · GPT-4o · Claude 4.5 Sonnet · LLMs via Pydantic AI

Scale

• Tiger Agent handles thousands of concurrent Slack conversations.
  “So we built Tiger Agent for Work, a production AI Slack bot that handles thousands of concurrent conversations at Tiger Data with its own memory, context, and production-grade reliability.”article, stated
• Within six weeks, more than half of the company used the bot daily.
  “built Tiger Agent for Work, a production AI Slack bot that handles thousands of concurrent conversations at Tiger Data with its own memory, context, and production-grade reliability. Within 6 weeks, more than half of the company uses it daily.”article, stated
• Tiger Agent integrated eight MCP servers at the time of the article.
  “We integrated eight MCP servers (as of now) into Tiger Agent: Slack search, customer docs, Salesforce cases, GitHub repos, Linear tickets, meeting transcripts, user memory, and progress reports.”article, stated
• A single request may involve logs from at least nine services.
  “We have at least nine services that may all produce logs for any given request, and you’d really like to see them together, in chronological order.”article, stated
• Logfire instrumentation setup took less than one hour.
  “Then we added @logfire.instrument decorators to functions we wanted to trace. Total setup time took less than one hour. ### Traces show everything When Tiger Agent responds to a Slack mention, Pydantic Logfire captures each tool call, its arguments, response, and latency.”article, stated

Alternatives considered (roads not taken)

Build the LLM library and agent plumbing from scratch

Rejected because: That would require thousands of lines of scaffolding for integrations, retries, provider switching, debugging, and logging before solving the Slack problem.

Write protocol handlers and retry logic for every MCP server

Rejected because: Pydantic AI loads MCP servers and handles tool translation/retries automatically.

Keep using Jaeger for traces

Rejected because: Jaeger technically worked but had poor developer experience compared with Logfire's Pydantic AI integration.

Commit to one LLM provider implementation

Rejected because: Provider behavior, reliability, context, cost, and agentic performance vary, and model outages happen.

Patterns

tool-use-orchestration (mcp-integration)

Tiger Agent loads eight MCP servers as toolsets so the Slack bot can search company systems, docs, tickets, repositories, transcripts, memory, and reports.

model-routing (provider-abstraction)

The system can switch between Claude, GPT-4o, and Claude Sonnet by environment variable while Pydantic AI translates provider APIs.

observability-tracing (debugging)

Logfire captures LLM calls, prompts, tool interactions, responses, latency, MCP spans, database calls, and HTTP requests as structured traces.

durable-execution (production-runtime)

Tiger Agent uses a PostgreSQL-backed work queue with atomic event claiming and Pydantic AI retries for failed tool calls.

agent-memory (context)

The Slack bot includes user memory and progress reports as MCP-backed context sources for ongoing company knowledge work.

Tradeoffs

Use Pydantic AI as a library rather than building everything in-house

Chose Pydantic AI for LLM/MCP/provider mechanics over A custom LLM library and MCP plumbing.

Rationale: Pydantic AI avoided low-value scaffolding while still fitting Tiger Agent's PostgreSQL work queue architecture.

Prefer SQL-queryable traces

Chose Logfire spans stored as rows with JSONB attributes over Tracing tools with custom query languages.

Rationale: SQL made debugging accessible to engineers and supported direct queries over agent runs and tool calls.

Use a framework for agent mechanics but keep event architecture

Chose Pydantic AI plugged into a PostgreSQL-backed work queue over A framework that imposes a new event model.

Rationale: The team wanted LLM/agent abstractions without rewriting the production event-processing architecture.

Gotchas (what bit them)

The LLM call is only a small piece of a production agent.

Root cause: Tool integration, retries, provider switching, debugging, and logging dominate the operational scaffolding.

Resolution: Use Pydantic AI for the agent/LLM/MCP mechanics and keep custom code focused on the Slack product.

Debugging agents without tracing is like debugging blind.

Root cause: The final output hides intermediate model decisions, tool calls, tool arguments, tool failures, and cross-service timing.

Resolution: Instrument the AI framework, MCP, database, and HTTP layers with Logfire.

Manual log correlation does not scale across services.

Root cause: A request may produce logs from many services that need chronological correlation.

Resolution: Store spans as structured rows and query traces with SQL.

Model choice cannot be settled once in advance.

Root cause: Providers differ in behavior, uptime, reliability, context window, cost, and agentic capability.

Resolution: Keep provider switching cheap through Pydantic AI's abstraction and environment-variable model selection.

Evaluation

The article emphasizes production observability over a formal offline eval harness: Tiger Data observes Agent Runs, tool calls, args, responses, latency, error spans, service-level failures, and provider behavior through Logfire and SQL queries.

Metrics: tool call arguments, responses, and latency · LLM system prompt, user prompt, tool interactions, and final response · error counts by service · provider uptime/reliability/cost/context behavior · daily adoption by more than half the company within six weeks

Outcomes

• Tiger Agent for Work became a production Slack bot handling thousands of conversations.
• More than half the company used it daily within six weeks.
• Pydantic AI simplified MCP integration, retries, and model/provider switching.
• Logfire made the LLM/tool workflow inspectable and queryable enough to replace custom debugging tooling.

The system handled thousands of concurrent conversations, integrated eight MCP servers, had at least nine services producing logs per request, and instrumented core layers in under one hour.