Training and Deploying Advanced Hallucination Detection Models for LLM Evaluation

patronus_ai · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Hallucination detection for RAG evaluation

Patronus AI needed a reliable hallucination detector for production RAG systems, especially in high-stakes financial and medical domains where closed-source judges were costly, opaque, and sometimes inaccurate.

Constraints:

RAG outputs must be checked against provided documents and factual reality.
Closed-source LLM judges can fail on complex reasoning tasks.
Open-source alternatives historically lagged on evaluation performance.
Training a 70B judge required substantial distributed infrastructure.
The team needed monitoring across training jobs.

Architecture

Patronus fine-tuned Llama-3-70B-Instruct into Lynx-70B-Instruct for hallucination detection using Databricks Mosaic AI, LLM Foundry, Composer, 32 H100 GPUs, FSDP, flash attention, WandB monitoring, and a specialized perturbation-based dataset. Lynx and HaluBench were open-sourced for RAG evaluation research.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  perturbation_training_data["Perturbation training data"] --> llm_foundry_training_pipeline["LLM Foundry training pipeline"]
  llm_foundry_training_pipeline --> composer_training_runtime["Composer training runtime"]
  composer_training_runtime --> h100_training_cluster["H100 training cluster"]
  llm_foundry_training_pipeline --> weights_and_biases_monitoring["Weights and Biases monitoring"]
  h100_training_cluster --> mosaic_ai_training_console["Mosaic AI training console"]
  llm_foundry_training_pipeline --> lynx_hallucination_detector["Lynx hallucination detector"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Perturbation training data`	Provides hallucination identification examples for fine-tuning.	`LLM Foundry training pipeline`
`LLM Foundry training pipeline`	Runs supervised fine-tuning workflow for Lynx.	`Perturbation training data`, `Composer training runtime`, `Lynx hallucination detector`
`Composer training runtime`	Provides FSDP, flash attention, and distributed training optimizations.	`LLM Foundry training pipeline`, `H100 training cluster`
`H100 training cluster`	Supplies compute for 70B model fine-tuning.	`Composer training runtime`, `Mosaic AI training console`
`Weights and Biases monitoring`	Tracks training dynamics and experiment results.	`LLM Foundry training pipeline`
`Mosaic AI training console`	Provides operational visibility for job status and history.	`H100 training cluster`
`Lynx hallucination detector`	Evaluates whether RAG responses conflict with context or factual reality.	`LLM Foundry training pipeline`

Data flow

Training data is constructed for hallucination identification, supervised fine-tuning runs through LLM Foundry and Composer on a 32-H100 cluster, WandB and Mosaic AI track training, and the resulting Lynx model evaluates whether generated answers conflict with source documents.

3 supporting quotes

“The training approach involved constructing specialized training and evaluation datasets for the hallucination identification task using a perturbation process”summary, stated

“The training configuration for supervised fine-tuning on the 70B parameter model utilized 32 NVIDIA H100 GPUs with an effective batch size of 256.”summary, stated

“The team employed several performance optimizations native to Composer, including Fully Sharded Data Parallel (FSDP) for distributed training and flash attention”summary, stated

Why this architecture

Patronus needed an open hallucination judge with domain-specific reasoning ability, so they paired specialized data with large-model fine-tuning and production training infrastructure.

“The key innovation lies in training a model capable of using complex reasoning to identify conflicting outputs between LLM responses and source documents.”summary, stated

“The choice of Databricks Mosaic AI tools, including LLM Foundry, Composer, and the training cluster, was driven by the need for customization options”summary, stated

Models used: Llama-3-70B-Instruct · Lynx-70B-Instruct · GPT-4o

Scale

Lynx was fine-tuned from a 70B parameter model.
“Lynx-70B-Instruct, as a fine-tuned version of Llama-3-70B-Instruct optimized for hallucination detection.”summary, stated
Training used 32 NVIDIA H100 GPUs and an effective batch size of 256.
“utilized 32 NVIDIA H100 GPUs with an effective batch size of 256.”summary, stated
Lynx outperformed GPT-4o by nearly 1% overall and by 7.5% on medical QA.
“Lynx outperforming GPT-4o by nearly 1% in accuracy across benchmark tasks, with even more pronounced improvements in domain-specific tasks like medical question-answering (7.5% improvement).”summary, stated

Alternatives considered (roads not taken)

Use closed-source LLM judges such as GPT-4

Rejected because: They can fail complex reasoning tasks and introduce cost/transparency concerns.

“Even when using top-performing closed-source models like GPT-4, the LLM-as-a-judge approach frequently fails to accurately evaluate responses on complex reasoning tasks.”summary, stated

Use existing open-source judge models

Rejected because: They historically lagged due to lack of challenging domain-specific training datasets.

“Open-source alternatives have historically lagged significantly in performance on evaluation tasks”summary, stated

Patterns

`llm-as-judge` (evaluation)

Lynx is a specialized judge model for detecting hallucinations in RAG outputs.

“The LLM-as-a-judge paradigm has emerged as a popular approach for detecting such inaccuracies due to its flexibility and ease of use.”summary, stated

`fine-tuning-adaptation` (judge training)

Patronus fine-tuned Llama-3-70B-Instruct into a hallucination detector.

“They developed Lynx, a fine-tuned Llama-3-70B-Instruct model trained using Databricks Mosaic AI infrastructure”summary, stated

`observability-tracing` (training monitoring)

WandB and Mosaic AI tracked training dynamics and job status.

“A notable aspect of the LLMOps workflow was the integration of Weights & Biases (WandB) with LLM Foundry for real-time monitoring of training results. This integration enabled the team to track experiments and observe training dynamics through the WandB dashboard.”summary, stated

Tradeoffs

Train an open specialized judge.

Chose Fine-tuned Lynx-70B-Instruct over Relying on closed-source GPT-4 style judges.

Rationale: The specialized open model addressed accuracy, transparency, and cost concerns.

“there are ongoing concerns about the quality, transparency, and cost of relying on closed-source LLMs for evaluation purposes.”summary, stated

Use a 70B model with distributed optimizations.

Chose FSDP and flash attention on 32 H100 GPUs over Smaller or less optimized training setup.

Rationale: A production-grade evaluator needed strong reasoning performance but required substantial infrastructure.

“This represents a substantial computational investment and highlights the infrastructure requirements for training production-grade evaluation models.”summary, stated

Gotchas (what bit them)

General LLM judges can miss complex hallucinations.

Root cause: Complex reasoning and domain-specific facts are hard for generic closed-source evaluators.

Resolution: Train a specialized hallucination detector on targeted data.

“Even when using top-performing closed-source models like GPT-4, the LLM-as-a-judge approach frequently fails to accurately evaluate responses on complex reasoning tasks.”summary, stated

Open judge performance depends on training data specificity.

Root cause: Challenging public datasets were lacking for domain-specific hallucination detection.

Resolution: Construct specialized perturbation-based training and evaluation data.

“Open-source alternatives have historically lagged significantly in performance on evaluation tasks, primarily due to the lack of challenging and domain-specific publicly available training datasets.”summary, stated

“constructing specialized training and evaluation datasets for the hallucination identification task using a perturbation process”summary, stated

Evaluation

Patronus evaluated Lynx on hallucination detection benchmarks including domain-specific medical QA, comparing it with GPT-4o.

Metrics: Nearly 1% accuracy improvement over GPT-4o across benchmark tasks. · 7.5% improvement on medical question-answering. · 32 H100 GPUs and effective batch size 256 during training.

“Results showed Lynx outperforming GPT-4o by nearly 1% in accuracy across benchmark tasks, with even more pronounced improvements in domain-specific tasks like medical question-answering (7.5% improvement).”summary, stated

“medical question-answering (7.5% improvement).”summary, stated

Outcomes

Patronus released Lynx, a specialized hallucination detector.
Lynx outperformed GPT-4o on reported benchmark accuracy.
Both Lynx and HaluBench were open-sourced.

Training used a 70B base model, 32 H100 GPUs, and effective batch size 256.

“Both the model and the HaluBench evaluation dataset were open-sourced to advance RAG evaluation research.”summary, stated