Evaluating AI Agent Performance: Skills vs Documentation for Developer Platforms

wix · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-native developer documentation and agent skills

Wix needed to determine whether purpose-built AI agent skills actually outperform agent-optimized documentation, while avoiding a stale parallel documentation layer that could drift from product reality.

Constraints:

Skills were being maintained independently from the documentation they came from.
Developer platform docs increasingly serve AI agents rather than only humans.
Small errors in skills can break builds or send agents down costly paths.
Different task families have different success, latency, turn-count, and token-cost profiles.
Evaluation needed to compare standard docs, optimized docs, and curated skills across controlled runs.

Architecture

Wix ran 250 controlled agent evaluations across CLI-extension and REST-API task families. Each task compared baseline docs, surgically optimized docs, and curated skills or MCP-packaged skills. Agents performed development tasks, then self-evaluated completion and failure causes, while Wix collected deterministic token count, turn count, and wall-clock metrics. The result reframed agent-optimized docs as the backbone and skills as a caching layer for common tasks, with regular evals to detect drift.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  task-corpus["Task corpus"] --> agent-run-harness["Agent run harness"]
  baseline-documentation["Baseline documentation"] --> agent-run-harness
  agent-optimized-documentation["Agent-optimized documentation"] --> agent-run-harness
  curated-skills-layer["Curated skills layer"] --> agent-run-harness
  agent-run-harness --> self-evaluation-and-metrics["Self-evaluation and metrics"]
  self-evaluation-and-metrics --> docs-and-skills-strategy["Docs and skills strategy"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Task corpus`	CLI extension and REST API scripting tasks used for controlled agent evaluation.	`Agent run harness`
`Baseline documentation`	Standard developer docs used as one context condition.	`Agent run harness`
`Agent-optimized documentation`	Docs with targeted fixes such as missing method calls, field-name consistency, and dependency install steps.	`Agent run harness`
`Curated skills layer`	Condensed task guides and MCP-packaged skills tested against the docs conditions.	`Agent run harness`
`Agent run harness`	Runs agents through tasks under each context condition and captures outcomes.	`Self-evaluation and metrics`
`Self-evaluation and metrics`	Captures completion, failure reasons, token counts, turns, and wall-clock time.	`Docs and skills strategy`
`Docs and skills strategy`	Treats optimized docs as backbone and skills as maintained shortcuts for common tasks.	—

Data flow

Wix selects task families, provides one of three context regimes to an agent, lets the agent complete the development task, asks the agent to evaluate its own work, and records deterministic metrics such as completion, tokens, turns, and wall-clock time. Comparative results then feed documentation optimization, skill maintenance, and the decision to keep docs as source of truth.

“We designed a quantitative evaluation across two task families, 250 runs total:”article, stated

“we asked it to change hats and evaluate its own work. Did it complete the task as described? If not, why?”article, stated

Why this architecture

Wix used controlled comparative evals because the popular intuition that skills beat docs was untested. Measuring multiple task families and multiple context regimes exposed when optimized docs outperform skills, when skills save tokens, and when MCP tool fragmentation trades token efficiency for more wall-clock time.

“Skills are intuitive and trendy, but do they really provide agents with an edge over just using the docs and, if so, in what cases?”article, stated

“we decided to question the hype and start measuring.”article, stated

Models used: AI agents · Wix CLI · Wix MCP · Agent skills · Agent-optimized documentation

Scale

The evaluation included 250 controlled runs.
“We ran 250 controlled evaluations comparing how AI agents perform tasks using standard docs, AI-optimized docs, and purpose-built skills.”article, stated
Optimized docs improved CLI task completion from 67% to 87%.
“For CLI tasks, docs optimization alone improved completion from 67% to 87%, while cutting average token usage by 35% and wall-clock time by 9%.”article, stated
Optimized docs beat skills-only runs on CLI completion, 85% to 78%.
“For CLI tasks, docs-optimized runs achieved 85% completion vs 78% for skills-only runs, using 10% fewer tokens”article, stated
Well-aligned skills reduced tokens by 30-50% and time by 30% on specific tasks.
“we saw a 30-50% reduction in tokens and a 30% reduction in time compared to the docs runs.”article, stated
API tasks showed identical 80% completion for docs-optimized and skill-based runs.
“Docs-optimized and skill-based runs achieved identical 80% completion.”article, stated

Alternatives considered (roads not taken)

Treat skills as the primary agent interface.

Rejected because: Small mistakes, drift, and scaffolding mismatches can erase the speed advantage of skills.

“The conclusion: well-defined and accurate skills provide agents with a clear benefit over searching the docs, but misalignments and mistakes in the skills can completely erode this benefit.”article, stated

Leave docs optimized only for humans.

Rejected because: AI agents are now a primary documentation audience, and targeted docs improvements produced large completion and efficiency gains.

“Increasingly, the audience for our docs is shifting from human developers to AI agents.”article, stated

Maintain skills independently from docs.

Rejected because: Product changes can make derived skills stale as APIs, scaffolds, and required fields change.

“the moment the underlying product changes, a scaffold updates, an API gets a new required field, a method is deprecated, any skill derived from stale docs drifts.”article, stated

Patterns

`eval-driven-development` (developer experience)

Wix used controlled agent evals to decide how docs and skills should be designed and maintained.

“we decided to question the hype and start measuring. We ran 250 controlled evaluations”article, stated

`data-quality-loop` (documentation maintenance)

Evaluation findings drive targeted documentation fixes such as missing method calls, field names, and dependency steps.

“The improvements were surgical: adding a missing method call to an API code sample, fixing field name inconsistencies, adding a dependency install step”article, stated

`semantic-caching` (agent skills)

Wix reframes skills as cached shortcuts for common, well-defined tasks derived from the documentation source of truth.

“Skills are a caching layer. They exist to make common, well-defined tasks faster and cheaper.”article, stated

`cost-aware-routing` (agent context)

The study compares token usage, turns, and wall-clock time across docs, optimized docs, skills, and MCP conditions.

“We also collected deterministic data on token count, turn count, and wall-clock time for each run.”article, stated

Tradeoffs

Make optimized docs the backbone.

Chose Agent-optimized documentation as the primary source of truth over Skills as the dominant independent interface.

Rationale: Optimized docs achieved better CLI completion than skills-only while avoiding stale parallel maintenance.

“Agent-optimized docs are the backbone. An agent should be able to use your docs to accomplish any conceivable task with your platform.”article, stated

Use skills for common tasks.

Chose Skills as a caching layer over Expecting agents to search full docs for every repeated task.

Rationale: Accurate, aligned skills can reduce tokens and time for well-defined tasks.

“well-defined and accurate skills provide agents with a clear benefit over searching the docs”article, stated

Optimize across success, tokens, turns, and time.

Chose Multi-metric evaluation over Token-count optimization alone.

Rationale: Skills saved tokens in API tasks but could take more wall-clock time due to tool fragmentation.

“the tradeoff for saving on tokens was an increase in wall-clock time.”article, stated

Gotchas (what bit them)

Skills drift when maintained separately from docs.

Root cause: Skills are derived from docs, but product and API changes can land in one layer and not the other.

Resolution: Wix treats docs as backbone and uses regular evaluations to catch skill underperformance.

“All the skills were maintained independently, without coordination with the documentation they were derived from.”article, stated

Small skill mistakes can wipe out efficiency gains.

Root cause: Scaffold mismatches, missing exports, and best-practice bloat force agents into retries or extra work.

Resolution: Evaluate skills against optimized docs and keep skills aligned with the underlying platform.

“small mistakes in skills wipe out their speed advantage entirely.”article, stated

Token savings can increase wall-clock time.

Root cause: MCP tool fragmentation splits a full reference page into multiple sequential calls, increasing turns and inference latency.

Resolution: Measure tokens, turns, and wall-clock time together rather than optimizing one metric.

“More calls, more LLM inference latency, more turns, even though each call returns a smaller payload.”article, stated

Evaluation

Wix ran 250 controlled evaluations across CLI extension and REST API task families, comparing baseline docs, optimized docs, and curated content. It collected self-evaluation, completion, token count, turn count, and wall-clock time.

Metrics: Task completion rate · Token count · Turn count · Wall-clock time · Self-evaluated failure reasons · Docs vs skills performance

“For each run, after the agent completed its development work, we asked it to change hats and evaluate its own work.”article, stated