Building Production-Grade LLM Evaluation Systems for HR Tech Interview Intelligence

zebra · HR · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: HR technology, interview intelligence, and production LLM evaluation

Spotted Zebra needed to move an AI-powered interview intelligence product from a slow research feedback loop into production client delivery without losing fairness, transparency, prompt quality, model comparability, auditability, or human control over hiring decisions.

Constraints:

• The product extracts evidence of soft skills, technical skills, and company-value alignment from interview transcripts, but humans remain the decision-makers.
• Occupational psychology experts are the domain experts, but having them manually review every prompt iteration became a bottleneck.
• A prospective client compressed development from informal research pace into formal feedback multiple times per day.
• The team needed to compare prompt/model changes using metrics rather than gut feel, including coverage, accuracy, precision, recall, and F1.
• The hiring domain demands fairness, transparency, compliance, auditability, and preparedness for regulation such as the EU AI Act.

Architecture

Spotted Zebra built an evaluation-centered LLMOps workflow around its Skills Evaluation product. A bot joins interviews, transcribes calls, and passes transcripts to AI models that extract skill/value evidence into structured outputs. Occupational psychology experts frontload judgment into golden examples, which become a gold-standard evaluation set. Prompt files are versioned semantically and executed through a shared model gateway that captures latency, cost, retry logic, prompt versions, model versions, and correlation IDs. Closed tasks use golden examples; open-ended interview-question generation uses LLM-as-judge criteria. Production failures are added to adversarial testing banks, logs are stored/searchable for audit and future fine-tuning data, and evaluation reports are generated for client communication and compliance.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  interview-bot["Interview bot"] --> transcription-pipeline["Transcription pipeline"]
  transcription-pipeline --> evidence-extraction-model["Evidence extraction model"]
  evidence-extraction-model --> golden-example-evaluator["Golden example evaluator"]
  evidence-extraction-model --> human-hiring-decision["Human hiring decision"]
  occupational-psychology-experts["Occupational psychology experts"] --> golden-examples["Golden examples"]
  occupational-psychology-experts --> golden-criteria["Golden criteria"]
  golden-examples --> golden-example-evaluator
  golden-example-evaluator --> evaluation-metrics["Evaluation metrics"]
  prompt-files["Prompt files"] --> model-gateway["Model gateway"]
  prompt-files --> prompt-changelog["Prompt changelog"]
  prompt-files --> production-config["Production config"]
  production-config --> prompt-files
  model-gateway --> api-logging-store["API logging store"]
  golden-criteria --> llm-judge["LLM judge"]
  llm-judge --> evaluation-metrics
  adversarial-testing-bank["Adversarial testing bank"] --> golden-example-evaluator
  api-logging-store --> compliance-reporting["Compliance reporting"]
  evaluation-metrics --> compliance-reporting
  prompt-changelog --> evaluation-metrics

Components

Data flow

An interview bot joins a call and records/transcribes the conversation. The transcript goes through an evidence-extraction model that returns structured evidence for skills, technical skills, and company values. During development, outputs are compared against golden examples created by human domain experts. Prompt files and model parameters are versioned, then executed through a shared model gateway. Metrics and logs capture prompt version, model version, latency, cost, usage, and correlation IDs. For open-ended generation, an LLM judge evaluates generated interview questions against golden criteria. Production failures and edge cases are added to adversarial test banks, and CI/reporting scripts publish evaluation results for clients and compliance.

Why this architecture

The team discovered that evaluation infrastructure gave more client impact than fine-tuning, RAG, or agents because it converted subjective occupational-psychology judgment into repeatable tests, made prompt/model changes measurable, enabled day-one model evaluation, supported rollback and logging, and gave clients evidence that a black-box hiring system was working fairly.

Models used: Interview transcription model · Evidence extraction LLM · LLM-as-judge evaluator · GPT5 · Mistral · Anthropic models · OpenAI models

Scale

• Spotted Zebra defines large clients as typically over 5,000 employees.
  “We define large clients as people with typically over 5,000 employees.”transcript, stated
• The volume-hiring example covers 200 job roles and roughly 10,000 candidates.
  “I've put 200. There we go. 200 job roles. and lots of candidates apply. Maybe it's 10,000 something like that.”transcript, stated
• The executive-hiring example involves five similarly qualified candidates for one senior position.
  “There's one very senior position. You found five fantastic candidates.”transcript, stated
• Hiring managers may conduct five to 10 interviews in a day.
  “They might have five interviews a day. Maybe that's a 10 a day, something like that.”transcript, stated
• The development push compressed about a year's worth of development into a few months.
  “we had a year's worth of development in in a few months.”transcript, stated
• Golden-example stratification included junior roles, senior roles, and a broad spectrum of industries.
  “we stratified our golden example set. We looked at interview transcripts from different role levels. So so junior roles, senior roles.”transcript, stated

  “We looked at a broad spectrum of industries and so on.”transcript, stated
• The company reports ISO 4201 certification in the interview-intelligence category and flags EU AI Act enforcement by 2026.
  “we're actually the first company in the interview intelligence category to get ISO 4201”transcript, stated

  “EU AI act is coming. So if anyone's using AI systems in production, you really should be thinking about this now. It's until 2026”transcript, stated

Alternatives considered (roads not taken)

Keep domain experts in every prompt-iteration loop.

Rejected because: That made each party wait for feedback and became too slow under client-driven iteration pressure.

Hardcode prompts in application code.

Rejected because: Hardcoded prompts make versioning, rollback, metrics, and experiment control more difficult.

Use golden examples for every task.

Rejected because: Golden examples work for closed tasks with correct outputs, but open-ended interview-question generation needs criteria and LLM-as-judge evaluation.

Let engineers call model APIs in their own ad hoc wrappers.

Rejected because: Duplicated API-call logic prevents consistent retries, metrics, prompt execution, and model-gateway behavior across the business.

Treat production errors only as incidents to dread.

Rejected because: The team converted production failure causes into adversarial tests so the same error would not recur.

Patterns

eval-driven-development (quality)

Golden examples, metrics, adversarial banks, prompt comparisons, model tests, and CI reports make development quantifiable rather than gut-feel prompt tuning.

human-in-the-loop-gate (governance)

Human occupational psychologists define gold standards and criteria, while hiring managers retain decision authority from AI-supported evidence.

prompt-versioning (operations)

Prompts live in structured YAML files with semantic versioning, changelogs, config-driven rollout/rollback, provider parameters, system/user prompts, and templating.

llm-as-judge (evaluation)

Open-ended interview-question outputs are evaluated by a separate LLM against golden criteria, enabling development-time and inference-time quality assessment.

observability-tracing (operations)

API logging captures model version, usage, prompt version, correlation IDs, latency/cost, searchable storage, audit history, and future fine-tuning data.

guardrails-io-filtering (safety)

The adversarial test bank includes prompt injection attacks, empty inputs, password-protected PDFs, and production errors that become regression tests.

data-quality-loop (operations)

Production errors and golden-example gaps are fed back into test banks, eval sets, prompt changes, and future fine-tuning datasets.

Tradeoffs

Frontload domain-expert judgment into golden examples.

Chose Gold-standard datasets and automated metric comparisons over Manual expert review of every prompt change.

Rationale: This preserves expert judgment while making iteration faster and more measurable.

Use LLM-as-judge for open-ended tasks.

Chose Golden criteria and a separate model-family judge over Forcing open-ended tasks into fixed golden examples.

Rationale: Criteria are faster to create and can evaluate outputs at inference time, but require hard workshops to define what good means.

Centralize prompt execution through a model gateway.

Chose One shared function/library for prompt file execution, metrics, retries, and API packaging over Each engineer re-implementing provider calls in local code.

Rationale: Centralization reduces duplicated logic and ensures consistent latency/cost/usage/retry instrumentation.

Keep recent prompt versions available in production config.

Chose Config-selectable prompt versions for quick rollback over Deploying one prompt with code changes/restarts for rollback.

Rationale: Fast rollback matters when prompt updates cause production issues.

Move logs out of weak search tooling into queryable storage.

Chose S3-backed/searchable logs over Relying only on LangSmith search.

Rationale: Queryable logs support debugging, auditing, compliance, and future fine-tuning data creation.

Gotchas (what bit them)

Human review of every prompt iteration becomes a development bottleneck.

Root cause: Domain experts and AI engineers wait on each other while client feedback accelerates.

Resolution: Frontload expert judgment into golden examples and use automated metrics for most iteration.

Golden examples do not fit open-ended generation.

Root cause: Interview guide generation can produce thousands of acceptable questions, so there is no single correct answer.

Resolution: Define golden criteria and evaluate generated questions with a separate LLM judge.

Prompt changes are not reproducible if prompts are hardcoded or unversioned.

Root cause: Hardcoded prompt logic hides provider, parameter, system prompt, user prompt, and template differences from experiment tracking.

Resolution: Use structured prompt files, semantic versioning, changelogs, and config-driven prompt selection.

Edge cases become normal at scale.

Root cause: Production traffic exposes prompt injections, empty inputs, password-protected files, and other inputs that are rare in small tests.

Resolution: Collect difficult inputs and production failure causes in an adversarial test bank.

Logs are only obviously necessary after something has gone wrong.

Root cause: Teams cannot predict which metadata they will need during production debugging, audit, or compliance.

Resolution: Log prompt version, model version, usage, correlation IDs, searchable metadata, and store logs in a queryable system.

Evaluation

The evaluation stack combines closed-task golden examples, stratified gold sets, model/prompt version experiments, LLM-as-judge criteria for open tasks, adversarial regression banks, API-call logging, and client/compliance reports. Metrics include coverage, accuracy, precision, recall, F1, role-level slices, industry slices, latency, cost, usage, prompt/model versions, and production failure regressions.

Metrics: Coverage of extracted evidence · Accuracy · Precision · Recall · F1 · Junior versus senior role performance · Industry-stratified performance · Latency · Cost · Usage metrics · Prompt version · Model version · Correlation ID · Adversarial-bank pass/fail · Client fairness/transparency evidence · Compliance evidence for ISO/EU AI Act readiness

Outcomes

• Skills Evaluation reached production as an interview intelligence product.
• Golden examples made prompt/model changes measurable and allowed GPT5 to be tested on day one.
• Prompt versioning, model gateway instrumentation, and prompt config enabled controlled experiments and rollback.
• Adversarial testing banks converted production errors into regression tests.
• Evaluation artifacts supported client communication about black-box behavior, fairness, and compliance, including reported ISO 4201 certification.

The transcript reports large clients over 5,000 employees, hiring examples with 200 job roles and about 10,000 candidates, hiring managers with five to 10 interviews per day, a development push compressing a year's work into a few months, and EU AI Act readiness by 2026.