Self-Service Data Analytics with Claude-Powered Agents

anthropic · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: self-service business analytics

Anthropic needed Claude-powered analytics agents to answer business questions accurately without creating metric sprawl, stale definitions, or a false sense of precision from letting agents freely query the warehouse.

Constraints:

• Analytics questions are ambiguous because user concepts must map to current entities, metrics, tables, and definitions.
• Schemas, business definitions, and data sources change constantly.
• The search space can include millions of fields, dashboards, notebooks, and prior SQL files.
• Business users may not know enough SQL or data modeling to validate whether an answer is correct.

Architecture

Anthropic built an agentic analytics stack around strong data foundations, governed canonical datasets, semantic-layer-first routing, curated sources of truth, Claude skills that encode procedural analysis workflows, offline evaluations, CI and telemetry, adversarial SQL review, passive monitoring, and active correction harvesting that opens reference-doc PRs for domain owners.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Business user question"] --> B["Claude analytics agent"]
  B --> C["Knowledge skill"]
  B --> D["Unbook analysis skill"]
  C --> E["Semantic layer"]
  E --> F["Canonical data models"]
  C --> G["Curated reference docs"]
  B --> H["Query execution tools"]
  H --> F
  B --> I["Adversarial SQL reviewer"]
  J["Offline eval suite"] --> B
  J --> L["Data repo and CI"]
  K["Correction harvesting agent"] --> G
  K --> L
  F --> L

Components

Data flow

A stakeholder question enters a Claude analytics skill; the skill routes first to the semantic layer, then to curated domain reference files, lineage, dashboards, and governed models; the agent clarifies the question, compiles and runs SQL through managed query tools, performs data-quality and adversarial SQL review, returns a provenance-rich answer, and feeds corrections back into markdown reference docs and offline evals through scheduled correction-harvesting agents.

Why this architecture

Anthropic found that accuracy depends less on giving agents more raw data and more on routing them through governed definitions, curated references, procedural skills, and fixed evals; direct access to thousands of SQL files barely moved accuracy because the hard problem was mapping questions to the right entity.

Models used: Claude · Claude Code

Scale

• Claude automates 95% of Anthropic's business analytics queries with about 95% aggregate accuracy.
  “At Anthropic, 95% of business analytics queries are automated via Claude, with ~95% accuracy in aggregate.”article, stated
• Skills lifted eval accuracy from no more than 21% to above 95% aggregate and around 99% in some domains.
  “Without skills, Claude’s ability to answer analytics questions accurately didn’t exceed 21% on our evals.”article, stated

  “Adding skills gets these numbers consistently above 95% in aggregate and regularly around 99% in certain domains.”article, stated
• Adversarial review improved eval accuracy by 6% but cost 32% more tokens and 72% higher latency.
  “Adversarial review: we’ve found that employing a Claude skill to aggressively challenge all underlying assumptions on a potential final answer increased accuracy by 6%”article, stated
• Direct raw retrieval over thousands of prior SQL files moved accuracy by less than one point.
  “direct grep access to our entire dashboard, transformation, and analyst-notebook SQL (thousands of files).”article, stated

  “Accuracy moved by less than a point in either direction.”article, stated

Alternatives considered (roads not taken)

Point Claude at the warehouse and let agents execute freely

Rejected because: This creates a false sense of precision if concept/entity mapping is wrong.

Expose all prior SQL through raw retrieval

Rejected because: The information was often present, but unstructured access did not help the agent map questions correctly.

Auto-generate metric definitions from raw tables and query logs

Rejected because: The generated definitions encoded the ambiguities Anthropic wanted to remove and hurt eval performance.

Let humans fully own every correction manually

Rejected because: Anthropic uses scheduled agents to draft reference-doc fixes and domain owners to review PRs.

Patterns

tool-use-orchestration (analytics-agent)

Claude skills route the agent through semantic-layer tools, managed query tools, SQL execution, and adversarial review rather than unconstrained warehouse access.

context-engineering (agent-context)

Skills and curated reference docs deliberately narrow what context the agent sees, replacing raw search with procedural and declarative context loading.

eval-driven-development (evaluation)

Every skill structure decision is made against fixed offline evals, CI telemetry, domain launch thresholds, and correction-harvested eval additions.

human-in-the-loop-gate (governance)

Humans own metric definitions, validate generated evals, approve domain launches, and sign off leadership-bound answers.

Tradeoffs

Use governed canonical models and skills.

Chose a smaller curated source space over letting Claude search every table, dashboard, and query.

Rationale: Structure solved ambiguity better than raw information access.

Add adversarial review selectively.

Chose review sub-agents for final SQL answers over cheaper direct answers without review.

Rationale: Review improved accuracy but added substantial token and latency costs.

Generate docs but human-own definitions.

Chose Claude-assisted documentation with human metric ownership over LLM-generated metric definitions.

Rationale: Documentation drafting helps, but definitions need accountable governance to avoid encoding ambiguity.

Gotchas (what bit them)

Raw access to more analytics artifacts did not solve accuracy.

Root cause: The agent could see the relevant information but still failed to map questions to the right governed entity.

Resolution: Curate domain reference docs and procedural skills that route the agent before SQL is written.

Auto-generated semantic definitions encoded ambiguity.

Root cause: LLM-generated definitions from raw tables/query logs mirrored the messy source landscape instead of imposing governance.

Resolution: Use Claude to draft documentation but keep metric definition ownership with humans.

Analytics answers can be silently wrong.

Root cause: A plausible answer may be used without stakeholder correction.

Resolution: Use provenance footers, human sign-off for leadership-bound work, and daily KPI sanity checks against blessed dashboards.

Skills and docs rot as the business changes.

Root cause: Data models, business definitions, schemas, dashboards, and references change constantly.

Resolution: Colocate artifacts in a single repo, enforce CI checks, and harvest stakeholder corrections into reference-doc PRs.

Evaluation

Anthropic uses dashboard-based evals, long-tail evals, CI-integrated suites, telemetry tables with skill/model/git metadata, domain launch gates, adversarial review, passive monitoring, correction harvesting, and standing KPI sanity checks.

Metrics: offline eval accuracy · domain launch threshold · semantic-layer resolution share · correction-language share · token count · wall-clock latency · per-assertion pass/fail

Outcomes

• Claude automates 95% of Anthropic business analytics queries at about 95% aggregate accuracy.
• Skills raised analytics accuracy from no more than 21% to above 95% aggregate and around 99% in some domains.
• A raw retrieval ablation over thousands of SQL files moved accuracy by less than one point, proving structure was the bottleneck.
• Adversarial review added 6% accuracy on evals but at 32% more tokens and 72% higher latency.

The system supports self-service analytics across business surfaces by syncing skills to plugin marketplace, cloud blobs, and MCP-served resources.