Simplifying Text-to-SQL Agents by Removing 80% of Tools

vercel · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal text-to-SQL data access agent

Vercel needed d0 to answer natural-language analytics questions reliably and quickly, but its original heavily tooled text-to-SQL agent had only 80% success and high maintenance overhead.

Constraints:

• Employees need data answers without writing SQL or waiting on analysts.
• The original agent had many specialized tools, heavy prompt constraints, and custom retrieval scaffolding.
• Every edge case added maintenance cost and every model update required recalibrating constraints.
• The replacement still had to work against the Cube semantic layer and production analytics infrastructure.

Architecture

Vercel replaced an 18-tool specialized d0 agent with a simpler file-system agent: Claude Opus 4.5 runs inside Vercel Sandbox with access to Cube semantic-layer files written as YAML, Markdown, and JSON; it explores context with Unix commands such as grep, cat, find, and ls through a single execute-command tool, then executes SQL through the remaining SQL tool, with Vercel Gateway and Slack/Next.js providing routing and observability.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Slack user question"] --> B["Next.js Slack server"]
  B --> C["Claude Opus 4.5 agent"]
  C --> D["Vercel Sandbox"]
  C --> E["Execute command tool"]
  D --> F["Cube semantic layer files"]
  E --> F
  C --> G["ExecuteSQL tool"]
  G --> H["Analytics infrastructure"]

Components

Data flow

A Slack question reaches d0, Vercel Sandbox receives the semantic-catalog files, Claude browses and greps the Cube semantic layer using a bash command tool, builds SQL from the raw documentation, executes the query, and returns an explanation and results through the Slack-facing service.

Why this architecture

Vercel discovered that the semantic layer itself was strong documentation, so removing custom scaffolding let the model reason directly over raw files with fewer tools, fewer tokens, and better success.

Models used: Claude Opus 4.5 · ToolLoopAgent

Scale

• The old architecture exposed 18 specialized tools.
  “Built multiple specialized tools (schema lookup, query validation, error recovery, etc.)”article, implied

  “Built multiple specialized tools (schema lookup, query validation, error recovery, etc.)”article, stated
• The benchmark used five representative queries.
  “We benchmarked the old architecture against the new file system approach across 5 representative queries.”article, stated
• The new agent averaged 77.4s versus 274.8s before.
  “Avg execution time 274.8s 77.4s 3.5x faster”article, stated
• The new approach used about 61k tokens versus 102k tokens before.
  “Avg token usage ~102k tokens ~61k tokens 37% fewer tokens”article, stated
• The worst old failure took 724 seconds, 100 steps, and 145,463 tokens.
  “The old architecture’s worst case took 724 seconds, 100 steps, and 145,463 tokens before failing.”article, stated

Alternatives considered (roads not taken)

Specialized schema/search/validation tools

Rejected because: The tools encoded assumptions about model limitations, increased maintenance cost, and constrained reasoning.

Hand-coded retrieval for schema context

Rejected because: The Cube files already contained legible definitions and joins, so summarizing them for the model was unnecessary.

Add more patches for every edge case

Rejected because: Each patch created maintenance overhead and model-update recalibration.

Patterns

tool-use-orchestration (agent-architecture)

d0 uses a minimal toolset centered on a generic command tool plus SQL execution instead of many specialized tools.

context-engineering (agent-context)

The agent receives the semantic layer as raw file context and uses Unix search to decide what to read instead of relying on prefiltered summaries.

latency-optimization (performance)

The simplified architecture cut execution time by 3.5x and reduced steps and token usage.

Tradeoffs

Expose raw semantic-layer files.

Chose filesystem access and Unix tools over custom schema lookup and retrieval tools.

Rationale: The files were already readable enough for the model, and direct access improved performance.

Remove most tools.

Chose few generic tools over many specialized tool choices.

Rationale: Each tool encodes a choice for the model, and in this case the model made better choices itself.

Accept simpler architecture with stronger context requirements.

Chose model plus file system plus goal over complete scaffolding for every possibility.

Rationale: The simpler path performs well only if context quality is high.

Gotchas (what bit them)

Over-scaffolding made the agent worse.

Root cause: Vercel constrained reasoning and summarized context the model could read directly.

Resolution: Remove most specialized tools and let Claude inspect the semantic layer with standard file commands.

Simple file access only works when source files are good context.

Root cause: The approach depends on well-structured, consistently named semantic-layer files.

Resolution: Invest in documentation, naming, and semantic-layer quality before handing raw context to the model.

Model updates can invalidate hand-built constraints.

Root cause: The old agent was calibrated around assumptions about model limitations.

Resolution: Build for improving model capability and add complexity only when proven necessary.

Evaluation

Vercel benchmarked the old tool-heavy architecture against the filesystem approach across five representative queries, comparing execution time, success rate, token usage, steps, and qualitative reasoning.

Metrics: average execution time · success rate · average token usage · average steps · worst-case runtime · worst-case token usage

Outcomes

• The filesystem agent achieved 100% success on the five-query benchmark versus 80% before.
• Average execution time improved 3.5x.
• Token usage fell 37% and average steps fell 42%.
• The agent caught edge cases the old scaffolding had not anticipated.

The benchmark was small (five representative queries), but the performance deltas were large and directly tied to a production internal analytics assistant.