Implementing Evaluation Framework for MCP Server Tool Selection

neon · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: MCP server tool selection and safe database migration workflows

Neon needed confidence that LLMs using its MCP server could select the correct database migration tools from a large toolset, preserve state across a two-step migration workflow, and avoid directly modifying the main database branch with the wrong tool.

Constraints:

• the MCP server exposes more than 20 tools
• LLMs become confused when selecting from large tool lists
• database migration tools are stateful and must be used in order
• the LLM might bypass migration staging and run arbitrary SQL with run_sql
• LLM outputs are nondeterministic and cannot be compared by exact string match
• tests need to verify both semantic task success and actual database integrity
• evaluation runs must clean up temporary branches and handle incomplete database dumps

Architecture

Neon built a Braintrust-based TypeScript evaluation framework for its MCP server. The target workflow uses prepare_database_migration to apply SQL to a temporary Neon branch and return migration details, then complete_database_migration to apply changes to the main branch and delete the temporary branch. The eval harness runs realistic migration prompts through the MCP tool-calling path, captures the main branch schema before and after, records LLM call messages, deletes non-default branches, and scores the final answer with two mechanisms: a Claude 3.5 Sonnet LLM-as-judge factuality scorer and a mainBranchIntegrityCheck that verifies the main branch schema is unchanged after prepare-only calls. Repeated trials and Braintrust UI debugging guided prompt/description refinements that raised pass rate from about 60% to 100% without code changes.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  LLM_runtime["LLM runtime"] --> Neon_MCP_server["Neon MCP server"]
  Neon_MCP_server --> Migration_tools["Migration tools"]
  Migration_tools --> prepare_database_migration["prepare_database_migration"]
  Migration_tools --> complete_database_migration["complete_database_migration"]
  Neon_MCP_server --> run_sql_tool["run_sql tool"]
  prepare_database_migration --> Temporary_Neon_branch["Temporary Neon branch"]
  prepare_database_migration --> Pending_migration_state["Pending migration state"]
  complete_database_migration --> Main_database_branch["Main database branch"]
  complete_database_migration --> Temporary_Neon_branch
  run_sql_tool --> Main_database_branch
  Braintrust_eval_harness["Braintrust eval harness"] --> LLM_runtime
  Braintrust_eval_harness --> Factuality_Anthropic_scorer["Factuality Anthropic scorer"]
  Braintrust_eval_harness --> Main_Branch_Integrity_Check["Main Branch Integrity Check"]
  Main_Branch_Integrity_Check --> Main_database_branch
  Braintrust_eval_harness --> Braintrust_UI["Braintrust UI"]

Components

Data flow

An eval case supplies a natural-language database migration request and an expert expected outcome. Before the task runs, the harness dumps the main branch schema. It then calls the LLM/MCP interaction, letting the model choose tools. The expected safe behavior is to call prepare_database_migration, create a temporary branch, apply the SQL there, and ask the user whether to commit to main. After the run, the harness dumps the main branch schema again, records LLM call messages, deletes temporary branches, and scores the final message with the LLM-as-judge factuality scorer and the main-branch integrity check.

Why this architecture

Neon's eval design is multi-layered because an LLM can appear to answer correctly while using unsafe tools, and exact answer matching is brittle because LLMs are nondeterministic. The LLM-as-judge scorer handles semantic variations in the final response, while the integrity check proves the migration staging tool left the main branch unchanged. Braintrust supplies repeatable eval execution and debugging UI, and multiple trials account for stochastic model behavior.

Models used: Claude 3.5 Sonnet 20241022 · Claude runtime for MCP tool calling · LLM-as-a-judge factuality classifier

Scale

• The MCP server has more than 20 tools.
  “Since our MCP server has over 20 tools, it’s important for us to know that LLMs can pick the right tool for the job.”article, stated
• There are five evals for the prepare-migration scenario.
  “Let’s focus on the input and the expected sections of this first eval (we have 5 evals in total for this scenario).”article, stated
• Each eval runs 20 trials with max concurrency 2.
  “trialCount: 20, maxConcurrency: 2, scores: [factualityAnthropic, mainBranchIntegrityCheck],”article, stated
• Prompt/description tuning raised pass rate from around 60% to 100%.
  “we were at around 60% in terms of our “pass rate” on these tests. However, we’ve since tweaked our prompts and gotten to 100% of pass rate.”article, stated

Alternatives considered (roads not taken)

Manual/ad hoc testing of the MCP server

Rejected because: Neon treats evals as software tests needed to know whether LLMs can pick the right tool reliably.

Exact string matching against expected responses

Rejected because: LLM outputs are nondeterministic, so semantic factuality scoring is needed.

Use the general run_sql tool for migrations

Rejected because: The safe workflow requires preparing migrations on temporary branches before completion on main.

Change MCP server code to improve pass rate

Rejected because: The observed improvement came from tool descriptions/prompts alone, without code changes.

Patterns

tool-use-orchestration (agent-architecture)

The MCP server exposes migration tools that the LLM must choose and sequence correctly instead of falling back to arbitrary SQL execution.

eval-driven-development (quality)

Neon uses repeatable evals to measure migration tool-selection behavior and guide prompt/description improvements.

llm-as-judge (evaluation)

Claude 3.5 Sonnet scores factual equivalence between the LLM's answer and expert expected output while ignoring nonfunctional differences.

safety-policy-enforcement (governance)

The integrity check enforces the safety policy that prepare-only migration tests must not modify the main branch.

observability-tracing (operations)

The eval harness records LLM call messages and uses Braintrust UI to inspect and debug individual eval runs over time.

Tradeoffs

LLM-as-judge scoring

Chose semantic factuality classification over exact string match against expected output.

Rationale: LLM outputs are nondeterministic but still need to be judged for factual equivalence.

Real database integrity check

Chose before/after main branch schema comparison over only judging the final text response.

Rationale: A model can say the right thing while still using the wrong tool, so the database state itself must be checked.

Prompt/tool-description optimization

Chose tune descriptions for prepare and complete migration tools over changing MCP implementation code.

Rationale: The pass rate improved from roughly 60% to 100% through tool-description changes alone.

Managed eval platform

Chose Braintrust UI and SDK over only custom local eval scripts.

Rationale: The managed UI helps analyze and debug test runs over time.

Gotchas (what bit them)

Large MCP tool surfaces confuse LLM tool selection.

Root cause: As tool count grows, models become worse at choosing the correct tool for the job.

Resolution: Write evals specifically around high-risk tool-selection workflows before trusting the MCP server.

The migration workflow is stateful and easy to bypass.

Root cause: The server needs to track pending migrations and the model may instead choose the generic run_sql tool.

Resolution: Evaluate that prepare and complete are used in the right order and verify main branch integrity after prepare.

LLM evals can produce false negatives from infrastructure artifacts.

Root cause: Incomplete PostgreSQL dump responses may make the integrity check think the database changed when the dump is incomplete.

Resolution: Return null/eject from the integrity check if the after-run dump does not include the completion marker.

Exact expected-output comparisons are brittle for LLM behavior.

Root cause: The model can produce valid answers with different formatting, IDs, order, or wording.

Resolution: Use an LLM-as-judge scorer that ignores non-factual implementation details while grading factual equivalence.

Evaluation

The eval suite uses Braintrust's TypeScript SDK, five migration evals for the scenario, 20 trials with max concurrency 2, Claude-based factuality scoring, and a main branch schema integrity check. It compares natural-language outputs to expert expectations while also verifying that prepare-only runs do not modify the main branch.

Metrics: over 20 MCP tools · five evals for prepare migration scenario · 20 trials per eval · max concurrency 2 · factuality scores A=0.4, B=0.8, C=1, D=0, E=1 · main branch integrity score 1 or 0 · pass rate improved from around 60% to 100%

Outcomes

• Neon created automated evals for MCP database migration tool selection.
• The eval suite verifies both factual response quality and main-branch safety.
• Prompt/tool-description refinements raised pass rate from about 60% to 100%.
• The improvement required no code changes to the MCP server.
• Neon recommends tests and managed eval tooling for MCP server development.

The article is focused on one high-risk migration workflow, but the pattern generalizes to any MCP server where tool selection has real side effects. The strongest evidence is the combination of semantic scoring with concrete database-state verification.