Using Token Log-Probabilities to Detect and Filter LLM Hallucinations in Customer Support

gusto · HR · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Customer support LLM quality control

Gusto needed a production confidence signal for free-form LLM FAQ answers so it could suppress or route hallucinated, vague, or low-quality customer-support responses.

Constraints:

LLMs generate text rather than calibrated class labels.
Incorrect payroll, HR, or benefits guidance can create compliance and customer-trust risk.
The confidence signal had to work without training another model or adding extra inference passes.
Thresholds need recalibration when prompts change.
The approach reduces but does not eliminate bad outputs.

Architecture

Gusto records token log-probabilities from its LLM FAQ service and averages them into a Seq-Logprob confidence score for each generated response. Human support experts label outputs as good or bad, the team correlates those labels with confidence bins, builds precision-recall curves, and uses thresholds to reject poor answers, route low-confidence responses to humans, or collect more information before replying.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  customer-support-faq-service["Customer support FAQ service"] --> llm-answer-generator["LLM answer generator"]
  llm-answer-generator --> seq-logprob-scorer["Seq-Logprob scorer"]
  seq-logprob-scorer --> confidence-monitoring["Confidence monitoring"]
  human-quality-labels["Human quality labels"] --> confidence-monitoring
  human-quality-labels --> precision-recall-calibration["Precision-recall calibration"]
  confidence-monitoring --> precision-recall-calibration
  precision-recall-calibration --> decision-router["Decision router"]
  decision-router --> customer-support-faq-service

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Customer support FAQ service`	Receives customer support questions and invokes the LLM answer generator.	`LLM answer generator`
`LLM answer generator`	Produces support answers and token log-probabilities from internal documentation.	`Customer support FAQ service`, `Seq-Logprob scorer`
`Seq-Logprob scorer`	Averages token log-probabilities into a response confidence score.	`LLM answer generator`, `Confidence monitoring`
`Human quality labels`	Customer support experts label generated answers as good or bad for calibration.	`Confidence monitoring`, `Precision-recall calibration`
`Confidence monitoring`	Tracks confidence distributions and bins response quality.	`Seq-Logprob scorer`, `Human quality labels`, `Precision-recall calibration`
`Precision-recall calibration`	Chooses decision thresholds based on quality and coverage tradeoffs.	`Confidence monitoring`, `Decision router`
`Decision router`	Shows high-confidence answers, rejects poor ones, or sends low-confidence outputs to humans.	`Precision-recall calibration`, `Customer support FAQ service`

Data flow

A customer asks a support question, the FAQ LLM generates an answer from internal documentation, and the API returns token log-probabilities. Gusto averages token logprobs into a sequence score, logs it with the response, calibrates confidence against human quality labels, and applies decision boundaries to show, reject, route, or enrich the response.

3 supporting quotes

“The Seq-Logprob metric is computed as the average of log-probabilities across all tokens in a generated sequence.”summary, stated

“Gusto conducted an experiment with 1,000 support questions processed through their LLM-based FAQ service.”summary, stated

“set thresholds that enable automated actions such as rejecting poor-quality responses entirely, routing low-confidence responses to human experts for verification”summary, stated

Why this architecture

Gusto reused token log-probabilities already produced during generation to create a traditional ML-style confidence threshold for LLM text. That made quality gating cheap enough for production and let the team reason about precision/recall instead of treating hallucination risk as unmeasurable.

“A significant practical advantage is that these scores are "easily obtained as a by-product of generating" a response—no additional inference passes or specialized models are required.”summary, stated

“This brings LLM systems closer to the well-established evaluation paradigms of traditional ML systems.”summary, stated

Models used: OpenAI-style GPT model with token logprobs · Seq-Logprob confidence heuristic

Scale

The validation experiment used 1,000 support questions.
“Gusto conducted an experiment with 1,000 support questions processed through their LLM-based FAQ service.”summary, stated
The top confidence bin reached 76% accuracy.
“the top confidence bin achieved 76% accuracy while the bottom confidence bin achieved only 45% accuracy”summary, stated
The bottom confidence bin reached 45% accuracy.
“the top confidence bin achieved 76% accuracy while the bottom confidence bin achieved only 45% accuracy”summary, stated
The relative accuracy difference between top and bottom confidence bins was 69%.
“a 69% relative difference in accuracy between the most confident and least confident responses”summary, stated

Alternatives considered (roads not taken)

Treat every generated answer as equally trustworthy

Rejected because: Low-confidence responses were substantially more likely to be vague, fabricated, or guideline-breaking.

“Low-confidence responses tended to be "vague or overly broad," were "more likely to make stuff up," and were "less likely to follow prompt guidelines”summary, stated

Use a separate model for confidence estimation

Rejected because: Seq-Logprob is available from generation itself and needs no extra model.

“no additional inference passes or specialized models are required.”summary, stated

Set thresholds once and leave them fixed

Rejected because: Confidence distributions shift when prompts change.

“LLM confidence distribution is sensitive to prompt changes”summary, stated

Patterns

`confidence-thresholding` (quality control)

Gusto thresholds Seq-Logprob confidence to reject, route, or show generated support responses.

“By treating the binary quality label as the target variable and the confidence score as the prediction threshold, teams can visualize and optimize the tradeoff”summary, stated

`human-in-the-loop-gate` (support safety)

Low-confidence responses can be routed to human experts for verification.

“This approach enables them to construct precision-recall curves for their LLM system, allowing automated rejection of poor-quality responses or routing low-confidence outputs to human experts for verification.”summary, stated

`eval-driven-development` (calibration)

Human labels calibrate confidence bins and precision-recall curves for LLM response quality.

“They recorded the Seq-Logprob confidence scores for each generated response and then had customer support experts manually label the outputs as either "good quality" or "bad quality." This created a binary ground truth that could be correlated with the confidence scores.”summary, stated

`observability-tracing` (monitoring)

The system records confidence distributions during normal operation to detect threshold and prompt changes.

“During normal operation, record Seq-Logprob scores for all outputs to understand the expected confidence distribution.”summary, stated

Tradeoffs

Precision-recall thresholding

Chose Reject or route some answers based on confidence over Maximize answer coverage at any quality level.

Rationale: Customer support can trade recall for safer, more precise responses.

“optimize the tradeoff between showing more responses to users (higher recall) and ensuring those responses are correct (higher precision).”summary, stated

Use generation by-product as quality signal

Chose Seq-Logprob from existing API outputs over Additional evaluation model calls.

Rationale: It adds quality control with minimal infrastructure and latency overhead.

“existing infrastructure (API-provided logprobs) can be leveraged for quality control without requiring additional model training or specialized infrastructure.”summary, stated

Gotchas (what bit them)

High-confidence does not mean correct.

Root cause: The best confidence bin still had a substantial residual error rate.

Resolution: Use Seq-Logprob as a filter/routing mechanism, not a guarantee of truth.

“The 76% accuracy in the highest confidence bin still means that roughly one in four responses may be incorrect”summary, stated

Prompt changes invalidate confidence thresholds.

Root cause: Confidence distributions are prompt-sensitive.

Resolution: Recalibrate thresholds whenever prompts evolve.

“if the prompts are modified, the confidence thresholds need to be recalibrated accordingly.”summary, stated

Logprob access is provider-dependent.

Root cause: Not all LLM providers expose token log-probabilities.

Resolution: Teams without logprob access need semantic entropy, multi-sample, or other confidence methods.

“Additionally, the approach relies on access to token log-probabilities, which may not be available for all LLM providers or may be restricted in some API tiers. Organizations using models without logprob access would need alternative approaches.”summary, stated

Evaluation

Gusto evaluated Seq-Logprob against 1,000 manually labeled support answers, compared quality by confidence bins, and used the score to build precision-recall curves.

Metrics: 1,000 support questions · 76% top-bin accuracy · 45% bottom-bin accuracy · 69% relative accuracy difference · Precision-recall curves

“The ability to construct precision-recall curves for the LLM system is a particularly valuable outcome.”summary, stated

Outcomes

Seq-Logprob separated high- and low-quality outputs meaningfully.
Top confidence-bin answers reached 76% accuracy versus 45% for bottom-bin answers.
The method enables poor-output rejection, human routing, or information-gathering before response.
The approach requires ongoing prompt-specific calibration.

The validation experiment used 1,000 generated support answers labeled by customer support experts.

“This substantial gap demonstrates that the confidence metric has meaningful predictive power for identifying potentially problematic outputs.”summary, stated

“The approach is best viewed as a filtering mechanism that reduces but does not eliminate the risk of poor outputs.”summary, stated