Optimizing Generative Retrieval to Reduce LLM Hallucinations in Search Systems

alipay · Finance · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial search generative retrieval

Alipay needed to reduce hallucination in LLM-based generative retrieval for Fund Search and Insurance Search, where irrelevant generated documents undermine production credibility.

Constraints:

The systems serve finance search scenarios.
LLM generative retrieval can generate irrelevant documents.
Retrieval precision and conversion matter in online production.
The solution needed both training-time and post-processing improvements.

Architecture

Alipay combined knowledge-distillation reasoning during generative retrieval training with a decision agent that extends and selects retrieved documents from multiple perspectives.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  generative-retrieval-model["Generative Retrieval Model"]
  reasoning-distillation["Reasoning Distillation"]
  retrieval-extension-model["Retrieval Extension Model"]
  decision-agent["Decision Agent"]
  fund-search["Fund Search"]
  insurance-search["Insurance Search"]
  generative-retrieval-model --> reasoning-distillation
  reasoning-distillation --> generative-retrieval-model
  retrieval-extension-model --> decision-agent
  decision-agent --> fund-search
  decision-agent --> insurance-search

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Generative Retrieval Model`	Generates retrieval candidates for finance search.	`Reasoning Distillation`
`Reasoning Distillation`	Transfers LLM reasoning over query-document pairs into training.	`Generative Retrieval Model`
`Retrieval Extension Model`	Extends retrieved documents for post-processing.	`Decision Agent`
`Decision Agent`	Selects the most relevant results from multiple perspectives.	`Fund Search`, `Insurance Search`

Data flow

LLMs reason over query-document pairs, reasoning data is distilled into the generative retrieval model, retrieved candidates are extended through a retrieval model, and a decision agent selects the most relevant results.

“integrates knowledge distillation reasoning during model training and employs a decision agent for post-processing to improve retrieval precision.”summary, stated

“A decision agent extends retrieved documents through a retrieval model and selects the most relevant results from multiple perspectives.”summary, stated

Why this architecture

Training-time reasoning distillation attacks hallucination at the model level, while the decision agent adds a post-generation precision filter.

“LLMs assess and reason over query-document pairs, with reasoning data distilled as transferred knowledge to the GR model.”summary, stated

Models used: LLM-based generative retrieval · Knowledge distillation · Decision agent

Alternatives considered (roads not taken)

Use generative retrieval without reasoning distillation

Rejected because: LLM-based GR generated documents irrelevant to queries.

“LLM-based GR models generating documents irrelevant to user queries”summary, stated

Use generated results without decision-agent post-processing

Rejected because: The decision agent selects the most relevant results from multiple perspectives.

“selects the most relevant results from multiple perspectives.”summary, stated

Patterns

`rag-retrieval` (retrieval)

The work improves LLM-based generative retrieval for financial search.

“LLM-based generative retrieval (GR) for their Fund Search and Insurance Search systems.”summary, stated

`tool-use-orchestration` (agentic-postprocessing)

A decision agent post-processes retrieved candidates and selects final results.

“employs a decision agent for post-processing”summary, stated

`eval-driven-development` (quality)

Offline experiments and online A/B tests validated search quality and conversion gains.

“offline experiments on real-world datasets and online A/B tests”summary, stated

Tradeoffs

Combine training and post-processing fixes

Chose Reasoning distillation plus decision agent over Only tuning generation prompts or model weights.

Rationale: Hallucination needed both better learned reasoning and final-result selection.

“integrates knowledge distillation reasoning during model training and employs a decision agent for post-processing”summary, stated

Gotchas (what bit them)

Generative retrieval can hallucinate irrelevant documents.

Root cause: LLM-based GR may generate outputs not grounded in user query relevance.

Resolution: Add reasoning distillation and decision-agent post-processing.

“generating documents irrelevant to user queries, which undermines credibility in production applications.”summary, stated

Evaluation

Alipay evaluated with offline real-world datasets and online A/B tests for Fund Search and Insurance Search.

Metrics: Search quality · Conversion gains · Offline retrieval quality · Online A/B performance

“online A/B tests demonstrate improvements in search quality and conversion gains”summary, stated

Outcomes

Offline experiments improved search quality.
Online A/B tests showed conversion gains.
The method applied to Fund Search and Insurance Search.

No traffic volumes are reported; evaluation uses real-world datasets and online A/B tests.

“for both Fund Search and Insurance Search scenarios.”summary, stated