Building Evaluation Systems for AI-Powered Healthcare at Scale

sword_health · Healthcare · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Healthcare AI, patient-facing LLM systems, and clinical evaluation

Sword Health needed a way to deploy Phoenix, an AI care specialist for physical-therapy and musculoskeletal-care interactions, while keeping the model safe, reliable, useful, clinically bounded, and aligned with clinician workflows. Because Phoenix speaks with patients, responds during sessions, augments clinicians between sessions, and operates in high-stakes healthcare contexts, Sword built a combined offline and online evaluation system rather than relying on informal review or one-time testing.

Constraints:

• Patient-facing healthcare LLMs must remain safe, reliable, and useful before and after release.
• Phoenix must communicate naturally with patients while respecting clinical boundaries.
• The system must fit into clinician workflows and augment clinicians without compromising quality.
• Evaluation datasets must cover messy inputs, long-term outcomes, rare cases, tool calls, RAG, prompt branches, and previously solved issues.
• Automated evaluators must be aligned with expert clinical and product judgment before they are trusted in production.

Architecture

Sword Health built an eval-driven development loop around Phoenix. Offline, the team collects targeted examples, defines binary evaluation criteria, gets clinical-expert or PM annotations, and builds human, code-based, and LLM-as-judge evaluators. LLM-judge prompts are developed with train/development/test splits, human-alignment metrics, mistake analysis, and repeated prompt refinement. Before release, a vibe-check/manual review round catches quality issues not covered by evaluators. Online, the same evaluators and guardrails run in production, A/B tests measure user and acceptance metrics, manual audits inspect production traces, and observability logs inputs, metadata, outputs, retrieved RAG documents, tool invocations, user feedback, and eval results. Production findings feed back into datasets, criteria, and evaluators.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  phoenix-ai-care-specialist[Phoenix AI care specialist] --> patient-session-interface[Patient session interface]
  patient-session-interface --> motion-tracking-data[Motion tracking data]
  motion-tracking-data --> patient-session-interface
  phoenix-ai-care-specialist --> clinician-workflow[Clinician workflow]
  clinician-workflow --> human-annotators[Human annotators]
  phoenix-ai-care-specialist --> offline-evaluation-pipeline[Offline evaluation pipeline]
  offline-evaluation-pipeline --> evaluation-dataset[Evaluation dataset]
  offline-evaluation-pipeline --> evaluation-criteria[Evaluation criteria]
  evaluation-dataset --> evaluation-criteria
  evaluation-dataset --> human-annotators
  evaluation-criteria --> human-annotators
  human-annotators --> human-evaluation-tools[Human evaluation tools]
  human-annotators --> evaluator-suite[Evaluator suite]
  evaluation-criteria --> llm-as-judge-evaluator[LLM-as-judge evaluator]
  evaluation-criteria --> code-based-evaluator[Code-based evaluator]
  evaluator-suite --> human-evaluation-tools
  evaluator-suite --> code-based-evaluator
  evaluator-suite --> llm-as-judge-evaluator
  llm-as-judge-evaluator --> train-dev-test-calibration[Train/dev/test calibration]
  train-dev-test-calibration --> llm-as-judge-evaluator
  offline-evaluation-pipeline --> vibe-check-review[Vibe check review]
  vibe-check-review --> human-evaluation-tools
  phoenix-ai-care-specialist --> production-guardrails[Production guardrails]
  production-guardrails --> ab-test-runner[AB test runner]
  ab-test-runner --> production-observability[Production observability]
  production-observability --> manual-production-audit[Manual production audit]
  production-observability --> production-trace-evaluator[Production trace evaluator]
  manual-production-audit --> evaluation-dataset
  production-trace-evaluator --> evaluator-suite
  production-trace-evaluator --> evaluation-dataset

Components

Data flow

Phoenix conversations produce patient-facing outputs during and between sessions. For offline evaluation, Sword collects real or synthetic examples, adds the same context given to the model plus the Phoenix-patient transcript, defines binary criteria, and gets human annotation from clinical experts or PMs. Evaluators are then built for the criterion: human pairwise comparisons, code-based checks, or LLM-as-judge prompts. LLM judges are calibrated through train/dev/test splits and disagreement analysis. Before release, reviewers perform a manual vibe check. After release, production requests flow through guardrails, A/B tests, production trace logging, manual audits, and production evaluators; discovered failures are fed back into the evaluation dataset and criteria.

Why this architecture

Sword splits evaluation into offline and online loops because healthcare quality cannot be proven by a single release gate. Offline datasets, criteria, annotations, and evaluators provide pre-release confidence; vibe checks catch missing criteria; production guardrails reduce patient-facing risk; A/B tests measure real outcomes; observability and manual audits reveal new failure patterns; and production evaluators feed those discoveries back into the dataset and rubrics.

Models used: Large language models · Phoenix · LLM-as-judge · RAG · Code-based evaluators · Gondola · Argilla

Scale

• Phoenix is evaluated both before and after it reaches patients.
  “models remain safe, reliable, and useful both before and after they reach patients.”transcript, stated
• Phoenix supports patients both during and between sessions.
  “It supports patients both both during and between uh sessions.”transcript, stated
• LLM-as-judge evaluators are positioned as capable of processing thousands of evaluations quickly.
  “they are able to uh process thousands of evaluators uh in a short amount of time.”transcript, stated
• LLM judge calibration uses three data splits: training, development, and test.
  “we divide the evaluation data set into three sets. The um training set, development sets and the test set.”transcript, stated
• Production evaluators may run on all production traces or a subsample depending on criticality.
  “these evaluators can be used to evaluate all production traces or just a subsample.”transcript, stated
• A/B testing can be impractical when data volume forces waits as long as two months.
  “Sometimes your your uh volume of data is not enough to uh make decisions, right? No one wants to wait two months to have the results of an AB test.”transcript, stated

Alternatives considered (roads not taken)

Collect evaluation examples randomly from real data.

Rejected because: Random sampling over-represents simple and common use cases, so Sword deliberately gathers challenging, rare, long-tail, tool-calling, RAG, prompt-branch, and regression examples.

Use free-form or graded outputs for evaluation criteria.

Rejected because: Sword tries to keep criteria binary to reduce ambiguity and improve evaluator alignment.

Use one universal evaluator type for every criterion.

Rejected because: Sword assigns human, code-based, and LLM-as-judge evaluators to different tasks because each has different strengths and failure modes.

Rely only on automated evaluator metrics before release.

Rejected because: A version can improve metrics but fail a manual vibe check, so Sword holds the release and runs another improvement cycle.

Let a failed guardrail simply pass through.

Rejected because: For real-time safety checks, Sword falls back to a static message or retries generation when the guardrail fails.

Evaluate only before release.

Rejected because: The same evaluators continue in production, where pass-rate drops prompt investigation and corrective action.

Patterns

eval-driven-development (evaluation)

Sword's release loop starts with datasets, criteria, human annotation, evaluator suites, and a pre-release vibe check, then reuses those evaluators in production to keep improving Phoenix.

llm-as-judge (evaluation)

Sword uses LLM-as-judge evaluators for complex or subjective checks, with prompts containing task, transcript, rubric criteria, and evaluation instructions.

human-in-the-loop-gate (governance)

Clinical experts and product managers annotate examples, run pairwise comparisons, perform vibe checks, and supply the expert judgment used to validate automated evaluators.

guardrails (safety)

Some evaluators become real-time guardrails that prevent unwanted patient-facing content; medical-advice checks may use LLM-as-judge and failures trigger fallback or retries.

data-quality-loop (data)

Evaluation datasets are continuously improved with user feedback, manual audits, production traces, and hard cases that surface from online monitoring.

observability-tracing (operations)

Sword logs request inputs, metadata, outputs, RAG documents, tool invocations, user feedback, and evals so production traces can be manually audited and automatically evaluated.

model-routing (model-operations)

Evaluations support objective model comparison and cost optimization by checking whether Sword can switch to another or smaller model without degrading performance.

rag-retrieval (context)

RAG is one of the application paths explicitly stressed in dataset construction and logged in production observability, because retrieved documents affect model behavior.

Tradeoffs

Evaluation dataset construction

Chose targeted and synthetic example selection over random real-data sampling alone.

Rationale: Targeted selection covers hard cases and rare long-tail behavior that random sampling misses.

Criterion output design

Chose binary yes/no or pass/fail labels over more ambiguous grading outputs.

Rationale: Binary criteria reduce ambiguity and improve agreement between evaluators.

Evaluator type selection

Chose human, code-based, and LLM-as-judge evaluators by task over a single evaluator mechanism for all criteria.

Rationale: Human review handles expert/tone judgments, code handles objective criteria at scale, and LLM-as-judge sits between them for complex but scalable checks.

Release gating

Chose manual vibe check plus evaluator metrics over automatic release when metrics improve.

Rationale: The vibe check protects against quality issues not yet encoded in criteria, even when evaluation metrics improve.

Production safety response

Chose fallback or retry on guardrail failure over delivering the unsafe generated content.

Rationale: Guardrails are real-time patient-facing safety checks, so failures need an immediate safe response path.

Online evaluation coverage

Chose all traces or subsampling depending on criticality over one fixed production-evaluation cadence.

Rationale: Critical issues justify broader evaluation coverage; less critical cases may use subsampling.

Gotchas (what bit them)

Random dataset sampling hides hard healthcare cases.

Root cause: Randomly collected examples overrepresent simple and common use cases.

Resolution: Deliberately include challenging, rare, long-tail, tool-calling, RAG, prompt-branch, previous-issue, and new-feature examples.

Automated evaluators are only useful if they align with expert judgment.

Root cause: Healthcare criteria depend on clinical judgment, product taste, and tone, which automated evaluators can misread.

Resolution: Use clinical experts or PMs to annotate examples, then validate LLM judges with train/dev/test splits and disagreement analysis.

An evaluator suite can miss full-picture quality problems.

Root cause: Metrics cover defined criteria, while new or subjective quality issues may not yet be represented.

Resolution: Run a manual vibe-check round before production; if metrics pass but the vibe check fails, hold release and repeat an improvement cycle.

A/B tests for LLM releases can be slow or statistically weak.

Root cause: Teams may lack enough data volume, stakeholders may resist holding back a new version, and statistical design still matters.

Resolution: Treat A/B tests as the ultimate production test but interpret them with the usual product-experiment discipline and keep offline/online evals as complementary signals.

Guardrails must account for critical healthcare validations.

Root cause: Some failures, such as medical advice, are higher risk than ordinary latency or formatting issues.

Resolution: Use code-based evaluators for objective real-time checks and LLM-as-judge for critical validations, with static fallback or retry on failure.

Production debugging is weak without broad observability.

Root cause: Failure patterns and root causes require visibility into inputs, outputs, metadata, retrieved documents, tool invocations, feedback, and eval results.

Resolution: Log the full production trace context and use manual audits plus production evaluators to detect issues early and create new evaluations.

Evaluation

Sword evaluates Phoenix through an offline-to-online chain: targeted datasets, binary criteria, human annotations, evaluator suites, train/dev/test calibration for LLM judges, true-positive/true-negative analysis, disagreement review, pre-release vibe checks, production guardrails, A/B tests, manual audits, observability, and production-trace evaluators.

Metrics: Human alignment between evaluator outputs and expert labels. · True positive rate and true negative rate during judge development. · Pairwise human expert scores for major updates. · Code-based checks such as character-count limits. · A/B-test user feedback, usage, and acceptance metrics. · Production evaluator pass rates and threshold-triggered investigations. · Manual audit findings from production traces.

Outcomes

• Sword established a repeatable offline release process for Phoenix that combines datasets, criteria, human annotations, evaluator suites, and manual vibe checks.
• The team uses targeted and synthetic data to cover rare, long-tail, tool-calling, RAG, prompt-branch, and regression cases.
• Human, code-based, and LLM-as-judge evaluators are assigned by task rather than forced into one evaluation type.
• Production Phoenix releases remain under continuous evaluation through guardrails, A/B tests, manual trace audits, observability, and production evaluator pass-rate monitoring.
• The case reports a process architecture for safe healthcare deployment, not a single numeric patient-outcome lift.

The talk frames Phoenix as healthcare-at-scale and patient-facing, but it does not provide exact patient volume. Concrete scale signals include thousands of automated evaluations, three-way judge-calibration splits, production traces evaluated all-or-subsample by criticality, and A/B tests that can take up to two months if data volume is low.