Building Eval Forge: A Comprehensive AI Agent Testing and Monitoring Platform

wix · Tech · 2026 · source Agentic: yes · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Testing and monitoring non-deterministic multi-agent systems

Wix needed a way to know whether multi-agent AI systems still worked after changes to prompts, skills, sub-agents, roles, MCP tools, models, or external dependencies. Classical tests worked for deterministic flows, but AI agent behavior was non-deterministic and difficult to trust without scenario-level assertions, side-by-side comparisons, scheduled monitoring, and development workflow enforcement.

Constraints:

Architecture

Eval Forge evolved from scenario testing with assertions and LLM judges into a full AI-agent testing and monitoring platform. Agents are configured with models, temperature, prompts, skills, roles, sub-agents, and MCPs. Scenarios define environments, trigger prompts, and assertions such as skill calls, tool invocations, build results, time limits, LLM judge scores, and API-state checks. Git synchronization and versioning let teams compare versions of skills, roles, MCPs, and sub-agents. Side-by-side runs isolate one changed variable, scheduled suites catch external regressions, dashboards surface pass-rate/cost trends, and GitHub Actions run tagged evaluations on pull requests with merge enforcement.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  eval-forge-platform[Eval Forge platform] --> agent-configuration[Agent configuration]
  eval-forge-platform --> scenario-runner[Scenario runner]
  eval-forge-platform --> git-synchronization[Git synchronization]
  eval-forge-platform --> comparison-runner[Comparison runner]
  eval-forge-platform --> scheduled-monitor[Scheduled monitor]
  eval-forge-platform --> dashboard[Dashboard]
  eval-forge-platform --> github-action[GitHub Action]
  agent-configuration --> skills[Skills]
  agent-configuration --> roles[Roles]
  agent-configuration --> sub-agents[Sub-agents]
  agent-configuration --> mcp-tools[MCP tools]
  agent-configuration --> scenario-runner
  skills --> agent-configuration
  skills --> git-synchronization
  roles --> agent-configuration
  roles --> git-synchronization
  sub-agents --> agent-configuration
  sub-agents --> git-synchronization
  sub-agents --> scenario-runner
  mcp-tools --> agent-configuration
  mcp-tools --> scenario-runner
  mcp-tools --> scheduled-monitor
  scenario-runner --> assertion-engine[Assertion engine]
  scenario-runner --> trace-and-cost-capture[Trace and cost capture]
  assertion-engine --> llm-judge[LLM judge]
  assertion-engine --> trace-and-cost-capture
  trace-and-cost-capture --> dashboard
  git-synchronization --> comparison-runner
  git-synchronization --> github-action
  comparison-runner --> scenario-runner
  comparison-runner --> dashboard
  scheduled-monitor --> alerting[Alerting]
  alerting --> dashboard
  dashboard --> eval-forge-platform
  github-action --> tag-selector[Tag selector]
  github-action --> merge-enforcement[Merge enforcement]
  tag-selector --> scenario-runner
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Eval Forge platform Internal Wix platform for testing, comparing, monitoring, and enforcing quality for multi-agent AI systems. Agent configuration, Scenario runner, Git synchronization, Comparison runner, Scheduled monitor, Dashboard, GitHub Action
Agent configuration Configures models, temperature settings, system prompts, skills, roles, sub-agents, and MCPs. Skills, Roles, Sub-agents, MCP tools, Scenario runner
Skills Description-plus-data units that agents scan and load when relevant to a conversation. Agent configuration, Git synchronization
Roles Reusable execution instructions and constraints provided to agents on every run. Agent configuration, Git synchronization
Sub-agents Predefined task-specific agents invoked by the primary agent in separate contexts. Agent configuration, Git synchronization, Scenario runner
MCP tools External-system interfaces to databases, APIs, GitHub, and other systems used by agents. Agent configuration, Scenario runner, Scheduled monitor
Scenario runner Executes named scenarios with environment setup, trigger prompts, and configured agent architecture. Assertion engine, Trace and cost capture
Assertion engine Evaluates concrete and non-concrete assertions such as skill calls, tool calls, builds, time limits, API state, and judge scores. LLM judge, Trace and cost capture
LLM judge Model-based evaluator for non-concrete scenario outcomes dependent on agent output. Assertion engine
Trace and cost capture Execution trace layer that records step-by-step tool calls, costs, tokens, and duration. Dashboard
Git synchronization Syncs skills, sub-agents, and roles from Git so tests run against production code and can create versions. Comparison runner, GitHub Action
Comparison runner Side-by-side evaluator that holds everything fixed except the changed entity version. Scenario runner, Dashboard
Scheduled monitor Runs scenario suites on demand or on a schedule to catch model, MCP, and dependency regressions. Alerting
Alerting Notification layer for regressions detected by scheduled runs, subject to threshold tuning. Dashboard
Dashboard UI for pass-rate and cost trends, low-pass-rate scenarios, comparison output, and execution details. Eval Forge platform
GitHub Action PR automation that chooses relevant tagged scenarios and runs evaluations before merge. Tag selector, Merge enforcement
Tag selector Maps changed files or documents to relevant scenario tags so PR checks do not run the entire suite. Scenario runner
Merge enforcement Policy layer that blocks skill changes until a corresponding scenario exists and passes. —

Data flow

A developer changes an agent component such as a skill, role, prompt, or MCP. Eval Forge syncs entities from Git, selects tagged scenarios relevant to the change, provisions the scenario environment, runs the agent with the configured architecture, captures traces and costs, and evaluates assertions. Side-by-side comparisons hold all variables constant except the tested version. Scheduled runs execute scenario suites nightly, weekly, hourly, or on demand to catch changes outside the developer's PR. Results flow back to dashboards, alerts, and PR checks that can block merge until a scenario passes.

4 supporting quotes
“Git synchronization was added, allowing every entity—skills, sub-agents, roles—to be synchronized with Git repositories”summary, stated
“Everything remains identical except for the specific variable being tested—perhaps a single skill version. This pinpointed exactly what changed and how that version affected agent execution. The system provides detailed output showing exactly where”summary, stated
“scheduled runs—suites of scenarios could execute on demand, nightly, weekly, hourly, or at any desired frequency.”summary, stated
“This led to implementing automated evaluation on every pull request through a GitHub action. The GitHub action runs evaluation on all relevant PRs. For example, in a repository containing all skills, whenever something changes, the action executes evaluation automatically.”summary, stated

Why this architecture

Wix's architecture separates AI quality into scenarios, assertions, versions, comparisons, schedules, and PR gates because a single manual run or pass/fail score could not establish confidence in non-deterministic agent systems. Versioning and side-by-side comparison isolate exactly what changed; scheduled monitoring catches model and tool drift outside developer control; PR automation makes evaluation part of ordinary development rather than a separate ritual.

4 supporting quotes
“evaluate and assign scores. This initial phase quickly revealed the first major takeaway: scenarios are the unit tests of AI systems. Running scenarios is the easy part; defining appropriate assertions is where the real work lies. A scenario with incorrect”summary, stated
“differences occurred between runs. This comparative capability addressed the second major learning: side-by-side runs provide real certainty. Saying something "feels good," which was the common practice before, doesn't constitute real data. Actually seeing comparison”summary, stated
“Test runners catch what developers break through their changes, while monitoring catches what the external world breaks”summary, stated
“suite. The fourth major takeaway emerged: make the evaluation flow as automated as possible. The harder it is to use, the less it gets used. The goal is minimal effort and procedure—developers write code, evaluation runs automatically, and they simply receive results.”summary, stated

Models used: LLMs · LLM judge · Claude · Gemini Flash · Claude 3.5 Sonnet · Gemini 1.5 Flash · MCP

Scale

Alternatives considered (roads not taken)

Rely on manual AI testing and deterministic classic tests.

Rejected because: That did not provide a systematic way to measure non-deterministic agent quality or catch prompt, model, and dependency regressions.

“product that generated code from user prompts, but they lacked confidence in whether their system truly functioned correctly. While they had deterministic tests for classic flows and some manual AI testing, they had no systematic way to measure quality consistently or catch regressions”summary, stated
“they had no systematic way to measure quality consistently or catch regressions when changes were made to prompts, models, or dependencies.”summary, stated

Judge changes by whether they feel good.

Rejected because: Wix found that side-by-side comparison output produced real data about how a version affected agent execution.

“Saying something "feels good," which was the common practice before, doesn't constitute real data. Actually seeing comparison results between versions does. Beyond simple pass/fail rates or scores, which can obscure what actually changed, every model change,”summary, stated
“something "feels good," which was the common practice before, doesn't constitute real data. Actually seeing comparison results between versions does. Beyond simple pass/fail rates or scores, which can obscure what actually changed, every model change, prompt modification,”summary, stated

Run evaluations only when developers manually trigger them.

Rejected because: Manual runs miss external changes in model behavior, MCPs, and dependencies, so scheduled monitoring was needed.

“things they hadn't known and catching numerous issues missed during development. However, evaluations still only ran when manually triggered. What about changes that happened without explicit awareness? This led to adding scheduled runs—suites of scenarios could”summary, stated
“scheduled runs transformed Eval Forge from purely a test runner into a monitoring system as well. The third major insight emerged: both capabilities are essential. Test runners catch what developers break through their changes, while monitoring catches what”summary, stated

Run the entire scenario suite on every pull request.

Rejected because: Tags allow PR automation to run only relevant scenarios, keeping developer feedback practical.

“Tags play a crucial role here—each scenario can be tagged, allowing the GitHub action to parse which tags are relevant to specific changes”summary, stated
“scenarios. When a particular document updates, the GitHub action determines the relevant tag and executes only applicable scenarios rather than the entire suite. The fourth major takeaway emerged: make the evaluation flow as automated as possible. The harder it is to use, the less”summary, stated

Patterns

eval-driven-development (quality)

Eval Forge makes scenario coverage, assertions, comparison runs, and PR-based evaluations part of the development workflow for agent changes.

“automated evaluation on every pull request through a GitHub action.”summary, stated
“when someone pushes a skill change, the merge is blocked until they add a corresponding scenario.”summary, stated

llm-as-judge (evaluation)

Eval Forge uses LLM judges for non-concrete assertions whose pass/fail status depends on agent output.

“Other assertions were non-concrete, depending on the agent's output, which required an LLM judge to evaluate and assign scores.”summary, stated
“Assertions verified that the correct MCP tool was called and that an LLM judge confirmed 24 items were returned. The demonstration showed the agent's step-by-step execution, including all tool calls, with detailed breakdowns of costs, tokens, and duration for each tool used.”summary, stated

prompt-versioning (change control)

The platform versions skills, sub-agents, roles, MCPs, and system prompts so teams can compare exact versions and track regressions.

“Versioning capabilities were introduced, enabling teams to create versions for any entity and compare them systematically.”summary, stated
“version everything—skills, roles, MCPs, and system prompts—to enable proper comparison and regression tracking.”summary, stated

observability-tracing (monitoring)

Eval Forge records step-by-step tool calls, costs, tokens, durations, pass rates, and low-pass scenarios so teams can inspect failures and monitor trends.

“dashboard providing overall pass rate and cost trends, highlighting scenarios with low pass rates that warrant examination.”summary, stated
“detailed breakdowns of costs, tokens, and duration for each tool used.”summary, stated

canary-rollout (comparison)

Side-by-side runs compare changed versions against a stable baseline with all other variables held constant, creating a canary-style evaluation before merge.

“Everything remains identical except for the specific variable being tested—perhaps a single skill version.”summary, stated
“every model change, prompt modification, or new tool addition requires side-by-side analysis to truly capture the impact.”summary, stated

tool-use-orchestration (agent architecture)

The system evaluates whether agents use skills, MCP tools, API calls, builds, and other external actions correctly within scenario executions.

“Available assertions include whether specific skills were called, whether tools were invoked, whether builds passed, time limits, LLM judge evaluations, and API calls.”summary, stated
“The demonstration showed the agent's step-by-step execution, including all tool calls”summary, stated

Tradeoffs

Use scenarios as AI unit tests.

Chose Scenario definitions with concrete and judge-based assertions. over Manual AI testing or informal confidence..

Rationale: Scenarios give repeatable checks for non-deterministic agents, but only if assertions are carefully designed.

“scenarios are the unit tests of AI systems. Running scenarios is the easy part; defining appropriate assertions is where the real work lies.”summary, stated
“If you cannot properly define when a scenario passes, you also cannot know when it breaks.”summary, stated

Use side-by-side comparison for changes.

Chose Direct comparison runs with one variable changed. over Only comparing aggregate scores or pass/fail rates..

Rationale: Side-by-side output shows exactly where differences occurred and what changed in execution.

“The system provides detailed output showing exactly where differences occurred between runs.”summary, stated
“Beyond simple pass/fail rates or scores, which can obscure what actually changed”summary, stated

Run both test-runner and monitoring modes.

Chose PR/manual test runs plus scheduled scenario suites. over Only running evaluations when developers trigger them..

Rationale: Developer-triggered tests catch local changes; scheduled monitoring catches changes in models, MCPs, and dependencies.

“both capabilities are essential. Test runners catch what developers break through their changes, while monitoring catches what the external world breaks—changes in MCPs, dependencies, or model behavior.”summary, stated
“Test runners catch what developers break through their changes, while monitoring catches what the external world breaks”summary, stated

Use tags to scope PR evaluations.

Chose Tagged scenarios selected by GitHub Actions. over Running every possible scenario on every minor change..

Rationale: Scoped evaluation keeps PR feedback practical while retaining relevant coverage.

“each scenario can be tagged, allowing the GitHub action to parse which tags are relevant to specific changes”summary, stated
“here—each scenario can be tagged, allowing the GitHub action to parse which tags are relevant to specific changes and run only those scenarios. When a particular document updates, the GitHub action determines the relevant tag and executes only applicable scenarios rather than the entire suite. The”summary, stated

Enforce scenario coverage for skill changes.

Chose Merge-blocking policy requiring an accompanying scenario. over Optional testing discipline..

Rationale: Enforcement improves coverage, but the summary notes it can add development overhead if teams see it as bureaucracy.

“requiring anyone changing a skill to create an accompanying scenario demonstrating what didn't work before and now does.”summary, stated
“organizational mandate requiring scenarios for all skill changes represents both a strength and potential friction point—while ensuring coverage, it also adds development overhead. The effectiveness of this approach depends on whether teams view it as valuable guardrails or bureaucratic obstacles.”summary, stated

Gotchas (what bit them)

Bad assertions create false confidence.

Root cause: Running scenarios is easy, but defining what success means is hard; incorrect assertions can say a broken agent passed.

Resolution: Treat assertion design as the core work of AI testing and prefer concrete checks such as tool calls, builds, API state, and time limits where possible.

“Running scenarios is the easy part; defining appropriate assertions is where the real work lies.”summary, stated
“A scenario with incorrect assertions provides false confidence, making developers believe everything works correctly when it doesn't—a situation worse than having no scenario at all.”summary, stated

Testing only the system prompt misses real agent architecture.

Root cause: Real agents include MCPs, sub-agents, roles, and skills that significantly influence behavior.

Resolution: Eval Forge expanded scenario support to represent the full agent architecture in testing.

“Real agents aren't just system prompts—they connect to MCPs, have sub-agents, and have roles”summary, stated
“evaluation wasn't truly comprehensive. The system was expanded to support these additional components, ensuring that the full agent architecture was represented in testing. Following this, Git synchronization was added, allowing every entity—skills, sub-agents, roles—to be synchronized with Git”summary, stated

Manual test runs miss regressions caused by the outside world.

Root cause: MCP tools, dependencies, and model behavior can change without a developer explicitly changing the agent.

Resolution: Scheduled monitoring runs catch regressions from external changes.

“This capability captured changes the team didn't explicitly make, such as subtle model behavior shifts or changes to MCP tools.”summary, stated
“monitoring catches what the external world breaks—changes in MCPs, dependencies, or model behavior.”summary, stated

Alerts are easy to send but hard to define.

Root cause: Small changes in runtime or score may not be meaningful regressions, creating a risk of alert fatigue or missed issues.

Resolution: The platform needed alert thresholds, though the summary notes threshold definition remained challenging.

“the challenging part isn't sending alerts—it's determining when alerts should be sent.”summary, stated
“Not every small change, like a run taking three additional seconds, necessarily indicates a regression requiring immediate attention.”summary, stated

Evaluation friction reduces adoption.

Root cause: Manual update-run-wait loops were lengthy and cumbersome, so developers would avoid evaluations if they were not automated.

Resolution: Wix added PR automation and tag selection so developers write code and receive evaluation results automatically.

“Changing a skill, updating it in the system, running manually, and waiting for results proved lengthy and cumbersome.”summary, stated
“developers write code, evaluation runs automatically, and they simply receive results.”summary, stated

Evaluation

Eval Forge is itself an evaluation system: it runs scenarios with explicit assertions, uses LLM judges for qualitative output checks, compares versions side by side, monitors scheduled suites, tracks pass rates and cost trends, and gates PRs through GitHub Actions. The platform evaluates both concrete system behavior and non-deterministic agent output.

Metrics: Scenario pass/fail based on assertions. · LLM judge scores for non-concrete outputs. · Overall pass rate and low-pass-rate scenarios. · Cost, token, and duration breakdowns per tool. · Side-by-side differences between model, prompt, skill, or tool versions. · PR check success before merge.

4 supporting quotes
“Available assertions include whether specific skills were called, whether tools were invoked, whether builds passed, time limits, LLM judge evaluations, and API calls.”summary, stated
“dashboard providing overall pass rate and cost trends, highlighting scenarios with low pass rates that warrant examination.”summary, stated
“detailed breakdowns of costs, tokens, and duration for each tool used.”summary, stated
“merging is only permitted after results confirm the scenario passes.”summary, stated

Outcomes

The summary does not publish aggregate pass-rate improvements or production incident reductions, but it describes organizational adoption across multiple Wix projects and a policy requiring scenario coverage for skill changes.

4 supporting quotes
“Multiple projects across Wix began using Eval Forge. The platform even became part of a new enforcement policy requiring anyone changing a skill to create an accompanying scenario demonstrating what didn't work before and now does.”summary, stated
“Scheduled Evaluation At this stage, Eval Forge had evolved into a relatively stable system with substantial value, teaching the team many things they hadn't known and catching numerous issues missed during development. However, evaluations still only ran when manually triggered.”summary, stated
“became part of a new enforcement policy requiring anyone changing a skill to create an accompanying scenario”summary, stated
“The platform's growing adoption across Wix and integration into development policies suggests genuine value”summary, implied