AI Agent Evaluation Framework for Travel and Accommodation Platform

booking.com · E-commerce · 2026 · source Agentic: yes · promotional bias: med · summary-only (no fetched source) · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Travel agent evaluation

Booking.com needed a way to evaluate agents that use tools, reason through multi-step travel tasks, and hold multi-turn conversations.

Constraints:

Architecture

Booking.com uses a dual evaluation architecture for AI Trip Planner-style agents. Black-box evaluation sends complete user-agent dialogues, task lists, and product constraints to a judge LLM for conversation-level task completion scoring. Glass-box evaluation inspects the internal trajectory: tool-call validity, tool correctness, argument hallucination, tool-spec reliability, reasoning, memory, and trajectory optimality. A baseline comparison layer weighs task-completion gains against cost and latency before deciding whether agent complexity is justified.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  agent-conversation-logs[Agent conversation logs] --> black-box-judge-llm[Black-box judge LLM]
  task-and-product-constraints[Task and product constraints] --> black-box-judge-llm
  agent-conversation-logs --> glass-box-trajectory-evaluator[Glass-box trajectory evaluator]
  glass-box-trajectory-evaluator --> tool-schema-validator[Tool schema validator]
  glass-box-trajectory-evaluator --> tool-correctness-evaluator[Tool correctness evaluator]
  glass-box-trajectory-evaluator --> advanced-metrics-layer[Advanced metrics layer]
  black-box-judge-llm --> deployment-decision-layer[Deployment decision layer]
  glass-box-trajectory-evaluator --> deployment-decision-layer
  baseline-comparison-harness[Baseline comparison harness] --> deployment-decision-layer
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Agent conversation logs Complete user-agent dialogues from AI Trip Planner and related customer-facing tasks. Black-box judge LLM, Glass-box trajectory evaluator
Task and product constraints Defines expected tasks and acceptable behavior for the judge. Black-box judge LLM
Black-box judge LLM Identifies requested tasks and scores task completion at conversation level. Agent conversation logs, Task and product constraints, Deployment decision layer
Glass-box trajectory evaluator Inspects internal reasoning, tool usage, and action paths. Agent conversation logs, Tool schema validator, Tool correctness evaluator, Advanced metrics layer, Deployment decision layer
Tool schema validator Checks executable tool calls against JSON schemas for names, parameters, types, and formats. Glass-box trajectory evaluator
Tool correctness evaluator Compares used tools to expected tools and flags missing/redundant selections. Glass-box trajectory evaluator
Advanced metrics layer Evaluates consistency, reasoning, memory quality/impact, and trajectory optimality. Glass-box trajectory evaluator
Baseline comparison harness Runs zero-shot, few-shot, deterministic-flow, and agent alternatives through performance/cost/latency comparison. Deployment decision layer
Deployment decision layer Decides whether the agent justifies its cost, latency, and maintenance overhead versus simpler baselines. Black-box judge LLM, Glass-box trajectory evaluator, Baseline comparison harness

Data flow

A conversation with an agent is evaluated in two passes. The black-box path gives the dialogue, expected task list, and product constraints to a judge LLM that identifies requested tasks and marks completion. The glass-box path reads the agent trace and validates tool names, schemas, parameter values, expected tool sets, and tool-spec quality. The resulting metrics feed a deployment comparison against zero-shot, few-shot, or deterministic-flow baselines with cost and latency tracked alongside performance.

4 supporting quotes
“The practical implementation uses a single judge LLM that performs dual functions: identifying which tasks were requested within the conversation and evaluating whether each identified task was successfully completed.”summary, stated
“Glass box evaluation provides visibility into the agent's internal decision-making processes, with particular focus on tool usage patterns.”summary, stated
“When an agent generates a tool call, the system validates multiple aspects: whether the tool name exists in the available tool inventory, whether all required parameters are present, whether any unexpected parameters have been included”summary, stated
“The expectation is that agents should show increased performance with only acceptable increases in cost and latency.”summary, stated

Why this architecture

Booking.com evaluates both final outcomes and internal trajectories because agents are not simple prompt-response systems: they autonomously break problems down, use tools, reason across turns, and may fail invisibly before the final answer. Baseline comparison is part of the architecture because agents are costly, high-maintenance systems that should be deployed only when the measured gain justifies latency and cost.

3 supporting quotes
“While traditional LLMs respond to direct prompts with text generation, agents can autonomously break down complex problems, determine when to use external tools, iterate through reasoning cycles, and engage in multi-turn conversations.”summary, stated
“The company has developed a dual evaluation methodology that examines both the final outcomes (black box) and the internal reasoning processes (glass box) to gain comprehensive insight into agent performance.”summary, stated
“They recognize that agents are complex, high-maintenance systems whose deployment must demonstrate clear performance advantages.”summary, stated

Models used: LLM agents · judge LLMs

Scale

Alternatives considered (roads not taken)

Final-answer-only evaluation

Rejected because: It misses failures in tool use, reasoning, and trajectories that are invisible in the final answer.

“This autonomy creates significant evaluation complexity because the agent's decision-making process involves multiple internal steps that may not be visible in the final output.”summary, stated

Always deploy an autonomous agent

Rejected because: Booking.com compares agents against zero-shot, few-shot, and deterministic-flow baselines before accepting higher maintenance, latency, and cost.

“Their baseline hierarchy progresses from zero-shot LLMs (prompt-based with task instructions only), to few-shot LLMs (adding examples), to LLM-based deterministic flows (chains of LLMs with usage rules).”summary, stated

Split task identification and completion scoring into two judge LLMs

Rejected because: The unified judge was chosen for cost-effectiveness and simplicity because their benchmark showed strong performance.

“While this could theoretically be separated into two distinct judge LLMs, Booking.com chose the unified approach for cost-effectiveness and simplicity”summary, stated

Patterns

llm-as-judge (quality)

Judge models score task completion and agent behavior where exact answers are difficult to label manually.

“Their solution combines black box evaluation (focusing on task completion using judge LLMs) with glass box evaluation (examining internal decision-making, tool usage, and reasoning trajectories).”summary, stated

eval-driven-development (quality)

The framework compares agent behavior against simpler baselines before deciding whether autonomy is worth cost and latency.

“The framework enables data-driven decisions about deploying agents versus simpler baselines by measuring performance gains against cost and latency tradeoffs, while also incorporating advanced metrics for consistency, reasoning quality, memory effectiveness, and trajectory optimality. ## Long Sum”summary, stated

Tradeoffs

Whether to deploy an agent or a simpler baseline.

Chose agent only when measured gain justifies overhead over defaulting to autonomous agents for every task.

Rationale: The framework explicitly considers performance against cost and latency.

“The framework enables data-driven decisions about deploying agents versus simpler baselines by measuring performance gains against cost and latency tradeoffs, while also incorporating advanced metrics for consistency, reasoning quality, memory ef”summary, stated

Agent deployment threshold.

Chose deploy agents only when measured performance gains justify cost and latency over defaulting to agents for complex travel tasks.

Rationale: The framework treats autonomy as an engineering tradeoff that must beat simpler baselines on business-relevant metrics.

“If comparisons don't show clear gains, simpler approaches are preferable due to lower cost, latency, and maintenance effort.”summary, stated

Judge implementation.

Chose one judge LLM for task identification and completion scoring over two separate judge LLM calls.

Rationale: A unified judge reduces cost and complexity while still working well on Booking.com benchmark datasets.

“Booking.com chose the unified approach for cost-effectiveness and simplicity, noting that state-of-the-art LLMs perform excellently on their benchmark datasets.”summary, stated

Gotchas (what bit them)

Final-answer evaluation misses failures inside the agent trajectory.

Root cause: Agents may call tools, maintain memory, or reason poorly even when the final answer looks plausible.

Resolution: Use glass-box metrics for internal decision-making, tool usage, reasoning quality, and trajectory optimality.

“This autonomy creates significant evaluation complexity because the agent's decision-making process involves multiple internal steps that may not be visible in the final output.”summary, stated

Evaluation

Dual black-box/glass-box evaluation for task completion, reasoning quality, consistency, memory, and trajectory optimality.

Metrics: task completion · consistency · reasoning quality · memory effectiveness · trajectory optimality

“es by measuring performance gains against cost and latency tradeoffs, while also incorporating advanced metrics for consistency, reasoning quality, memory effectiveness, and trajectory optimality.”summary, stated

Outcomes

Source fetch failed, so extraction is summary-only.

“Booking.com has developed and deployed LLM-based agents for customer-facing applications including their AI Trip Planner, which assists users with complex travel planning tasks such as destination recommendations, property searches, booking inquiries, and cancellation policy”summary, stated