Building Multi-Agent Systems with MCP and Pydantic AI for Document Processing

deepsense · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Agentic document-processing and MCP integration

DeepSense AI worked for a customer with a document-processing platform and needed to build both an MCP server for that platform and a multi-agent demonstration system that could structure data on demand from large, heterogeneous document sources. The deeper production problem was not only connecting agents to tools, but designing tool schemas, token budgets, model choices, observability, evaluation, validation, and security so the resulting agentic document workflow remained modular, affordable, debuggable, and safe.

Constraints:

• The customer platform handled many file types and hundreds of data sources, including PDFs, Excel sheets, Confluence, Notion, and Google Cloud Storage.
• The user-facing workflow needed to turn an open-ended document-processing goal into an AI-generated table schema and extracted structured data.
• The system had to reuse existing MCP servers, including Databricks MCP, while also exposing the customer's internal platform through a custom MCP server.
• Tool schemas are seen by the LLM primarily through function signatures and docstrings, so weak typing or poor descriptions invite errors.
• Tool listings can consume large context windows; one version took 16K tokens and exceeded a 10K-context model limit.
• Agents should not see every available tool, because broad tool exposure hurts security and tool selection.
• The architecture needed observability, testing, guardrails, and security controls before production use.

Architecture

DeepSense used Pydantic AI as the agent framework, Claude models as the main model family, MCP for external tool integration, and Databricks MCP plus a custom MCP server for the customer's document-processing platform. A user provides a document-processing objective, the system asks clarifying questions, generates a structured table schema, uses Databricks MCP to create the table, invokes the custom platform MCP to process documents from scratch or reuse an existing workflow, writes extracted rows into a Databricks Delta table, and then lets a downstream agent summarize or query the table with SQL. The implementation relies on strongly typed Pydantic models, Pydantic AI tool decorators, dependency injection, optional graph workflows, MCP toolsets, Logfire traces, Pydantic evals, LLM-as-judge evaluation, and guardrail validation with model retry.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-objective["User objective"] --> task-definition-agent["Task definition agent"]
  task-definition-agent --> schema-generation-agent["Schema generation agent"]
  schema-generation-agent --> databricks-mcp-server["Databricks MCP server"]
  databricks-mcp-server --> databricks-delta-table["Databricks Delta table"]
  custom-platform-mcp-server["Custom platform MCP server"] --> document-processing-platform["Document processing platform"]
  document-processing-platform --> databricks-delta-table
  databricks-delta-table --> chat-and-sql-agent["Chat and SQL agent"]
  pydantic-ai-framework["Pydantic AI framework"] --> task-definition-agent
  pydantic-ai-framework --> schema-generation-agent
  pydantic-ai-framework --> chat-and-sql-agent
  pydantic-ai-framework --> pydantic-ai-mcp-toolset["Pydantic AI MCP toolset"]
  pydantic-ai-mcp-toolset --> databricks-mcp-server
  pydantic-ai-mcp-toolset --> custom-platform-mcp-server
  claude-model-provider["Claude model provider"] --> pydantic-ai-framework
  fastmcp-implementation["FastMCP implementation"] --> custom-platform-mcp-server
  pydantic-ai-framework --> logfire-observability["Logfire observability"]
  pydantic-ai-framework --> pydantic-evals["Pydantic evals"]
  pydantic-evals --> guardrail-validation["Guardrail validation"]

Components

Data flow

The user states a goal for a set of files, such as analyzing PDFs in an S3 bucket. An intent/task-definition agent asks clarifying questions and generates a schema with typed fields. After user acceptance or feedback, Databricks MCP creates a target table. The custom MCP server for the document-processing platform starts a new workflow or reuses an existing one, processes the files into LLM-readable format, and stores extracted structured rows. A final agent can summarize findings, show source-backed entries, or run SQL queries against the Databricks table and compose an answer. Observability and evaluation wrap the flow through Logfire traces, token/pricing visibility, test cases, evaluators, and validation guardrails.

Why this architecture

The architecture combines Pydantic AI and MCP because Pydantic AI provides typed structured output, tools, dependency injection, delegation, graph workflows, evals, and observability, while MCP standardizes integrations across tools, languages, and agent implementations. The team used MCP to avoid rewriting the same API integration for each agent, Databricks MCP for table creation, and a custom MCP server for the customer's platform. The lesson set pushes the design toward narrow tool exposure, curated tool payloads, cheaper/faster models for simple work, Logfire tracing, explicit evals, guardrails, and security hardening.

Models used: Claude 3.5 · Claude 4.5 · Claude Haiku · Gemini 2.0 Flash · Pydantic AI · FastMCP · Databricks MCP

Scale

• The customer platform handled huge numbers of file types and hundreds of data sources.
  “huge number of file types like PDFs, Excel sheets, uh hundreds of data sources, uh Confluence, notion, Google cloud storage whatever”transcript, stated
• MCP is described as introduced by Anthropic in November 2024.
  “it was introduced by Antropic. So those from from cloud models in November 2024”transcript, stated
• A single MCP server can be reused by many agents, with 15 agents given as an example.
  “one of those servers can be reused by many for example 15 agents”transcript, stated
• One over-specified tool design used 16K tokens, which exceeded the 10K context limit of the free Claude Code version they wanted to try.
  “it took uh 16 uh K tokens to actually list those tools”transcript, stated

  “clo code free version of CL code who which had only 10K uh context at maximum”transcript, stated
• Unifying tools reduced token usage by 50%.
  “with that way we got 50% off of tokens. It's less expensive and uh it will uh fit the context”transcript, stated
• Claude 3.5/4.5 was the main model choice, while Claude 4.5 was cited as having about 1.88 seconds time to first token without reasoning.
  “we used clothes 3.5 uh as our main model. Now we got 4.5. This is great reasoning.”transcript, stated

  “the latency it's like 1.8 8 seconds time to first token. Not to mention this is without reasoning.”transcript, stated
• The talk summarizes 10 practical lessons for agentic systems.
  “Um so I will present 10 different uh practical tips that you can apply when building agentic uh systems.”transcript, stated

Alternatives considered (roads not taken)

Create separate native tools for every agent and external API.

Rejected because: MCP centralizes each external API or platform behind one reusable server, reducing duplicated integrations and schema-change maintenance.

Expose loosely typed dict-style tool signatures and minimal docstrings.

Rejected because: The LLM mainly sees function signatures and docstrings, so poor typing and vague documentation waste tokens and invite hallucinations or errors.

Return full uncurated JSON payloads from tools.

Rejected because: Full payloads can exceed context limits, raise cost, and make it harder for the LLM to extract the needed information; the speaker recommends selecting only needed fields or compressing when appropriate.

Expose every tool to every agent.

Rejected because: Filtering tools per agent is easier, more secure, and improves multi-agent behavior by avoiding tool overload.

Use the most capable Claude model for every interaction.

Rejected because: The speaker recommends splitting responsibilities: stronger models for creative or complex tasks, and cheaper/faster models such as Claude Haiku or Gemini Flash for routine chat when latency or price matters.

Implement every workflow as a graph.

Rejected because: Graphs can be overkill for mostly linear flows; a linear flow can be easier to read and lower complexity when the process does not need complex looping.

Patterns

multi-agent-orchestration (agent-architecture)

The document-processing demo uses multiple agents for intent clarification, schema generation, platform workflow execution, SQL/query answering, and final summarization rather than a single monolithic agent.

tool-use-orchestration (tooling)

The system exposes platform and Databricks capabilities through MCP servers and Pydantic AI toolsets so agents can create tables, process documents, execute SQL, and call external APIs.

structured-output-contract (extraction)

Pydantic models define typed agent outputs, and the document-processing workflow generates typed schemas with strings, integers, floats, dates, and other fields for extracted tables.

context-compaction (token-management)

The team curates and sometimes compresses tool responses, and unified over-specified data-source tools to cut tool-list token usage by 50%.

model-routing (model-selection)

The guidance routes complex/creative tasks to stronger Claude models and routine conversational work to faster or cheaper models such as Claude Haiku or Gemini Flash.

observability-tracing (operations)

Pydantic Logfire captures nested traces, bottlenecks, token counts, prices, prompts, tool calls, model output, HTTP/MCP/SQL spans, and other debugging information.

eval-driven-development (quality)

The project uses Pydantic evals with named cases, inputs, expected outputs, metadata, and evaluators so prompt changes and agent changes can be tested before production.

llm-as-judge (evaluation)

The speaker describes Pydantic evals using an LLM judge to assess whether structured insights are grounded, directly related, and comprehensive.

safety-policy-enforcement (security)

The talk emphasizes guardrail validation, not trusting agents blindly, filtering tools, and securing MCP integrations against prompt-injection/data-leak risks.

Tradeoffs

Use MCP to expose platform and data tools.

Chose Reusable MCP servers connected through Pydantic AI over Duplicated native tools per agent and per API.

Rationale: MCP reduces duplicate integrations and creates a common language-agnostic JSON-RPC interface, but poorly designed MCP tools can bloat context and damage reasoning.

Unify data-source-specific workflow tools.

Chose One parameterized tool with a literal data-source selector and unified configuration over Separate tools for S3, Google Drive, Azure Blob, and other data-source workflows.

Rationale: Unification cut tool-list tokens by 50% and fit the target context window, but it relies on the agent choosing the right literal/configuration.

Route tasks to models by complexity and latency/cost needs.

Chose Claude for strong reasoning and cheaper/faster models for routine conversation over Using the most capable Claude model for every step.

Rationale: Claude 4.5 offers strong reasoning but has higher pricing and latency, while Haiku or Gemini Flash can handle day-to-day interactions faster and cheaper.

Use a linear flow when graph orchestration is unnecessary.

Chose Readable sequential code for mostly linear demo flows over Pydantic AI graph workflows for every agent process.

Rationale: Graphs support cycles and conditional routing, but they added code complexity when the demo was largely linear.

Validate structured outputs against side effects.

Chose Guardrail checks plus model retry feedback over Trusting the structured response once it matches the Pydantic output schema.

Rationale: Structured output can still describe a database write that did not happen, so the system checks persisted state and asks the model to retry with feedback.

Gotchas (what bit them)

Tool definitions written for humans can fail when exposed to LLMs.

Root cause: The LLM primarily sees the function signature and docstring, not the source code a programmer could inspect.

Resolution: Use explicit input/return types and descriptive but concise docstrings that explain how and when the tool should be used.

Over-specified tool sets can exceed model context before the task starts.

Root cause: Listing many data-source-specific tools inflated the tool schema to 16K tokens, beyond the target 10K context window.

Resolution: Unify related tools and use literal parameters/configuration where the model can reliably choose among supported options.

Tool responses can drown the model in irrelevant JSON.

Root cause: Returning everything from an API may exceed context, raise cost, and make the relevant fields harder for the model to isolate.

Resolution: Curate response fields and compress payloads only when the use case supports it.

All-tools-to-all-agents makes systems less secure and less reliable.

Root cause: An overloaded agent may see tools it should not use and become confused about tool choice.

Resolution: Filter and limit tools per agent and per MCP toolset subset.

Graph workflows can add unnecessary complexity.

Root cause: Mostly linear flows with only minor returns become harder to read when modeled as graph classes and node methods.

Resolution: Prefer a linear flow when the process is fundamentally sequential and does not require complex graph behavior.

MCP can expand the blast radius of prompt-injection attacks.

Root cause: Tooling standards make it easy for agents to act across systems, so malicious tool-visible content can cause data leakage if permissions and validation are weak.

Resolution: Treat MCP security as a first-class design topic, limit tool exposure, validate outputs, and avoid trusting agent actions blindly.

Passing schema validation is not enough; the agent can still hallucinate database effects.

Root cause: Structured outputs only constrain shape, not whether the output reflects actual persisted state.

Resolution: Use guardrail validation that checks database state and raises model retry with feedback if the claim is false.

Evaluation

The talk recommends an explicit evaluation layer using Pydantic evals, named test cases, expected outputs, metadata, custom evaluators, and LLM-as-judge evaluators. It also treats Logfire observability as part of the feedback loop because traces expose prompts, model calls, tools, tokens, prices, and bottlenecks.

Metrics: Expected-output match score in Pydantic evals · LLM-as-judge grounding, relatedness, and comprehensiveness ratings · Input and output token counts · Pricing visible in traces · Latency/time schema and bottleneck traces · Model retry outcomes after guardrail failures

Outcomes

• DeepSense delivered a custom MCP server for the customer's document-processing platform.
• The team reused Databricks MCP to create tables for AI-generated schemas.
• The demo system could ask clarifying questions, generate schemas, process documents, extract table rows, show sources, and answer questions using SQL over the resulting table.
• Unifying over-specified tools cut token usage by 50%.
• The talk distilled 10 practical lessons spanning MCP design, tool filtering, model choice, observability, evals, guardrails, workflow shape, and security.

The source is a technical lessons-learned talk rather than a production KPI report. It quantifies platform breadth as many file types and hundreds of sources, a 16K-to-10K context problem, a 50% token reduction from unifying tools, and model latency around 1.88 seconds time to first token for the cited Claude model.