Data Quality Assessment and Enhancement Framework for GenAI Applications

quantumblack · Healthcare · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Unstructured data quality for GenAI and RAG

QuantumBlack needed to assess and remediate unstructured-data problems that cause generative AI and RAG applications to hallucinate, retrieve poorly, leak sensitive information, or waste storage and compute.

Constraints:

Unstructured data comes in formats such as PDFs, images, videos, PowerPoint, and Excel.
Unstructured data quality has less-established frameworks than structured data quality.
Documents may lack metadata, be duplicated, outdated, irrelevant, multilingual, or contain sensitive information.
Downstream hallucination and leakage can be hard to fix after ingestion.
Human review is needed for flagged document-quality issues.

Architecture

QuantumBlack's AI4DQ Unstructured toolkit scans document corpora, scores unstructured data quality, clusters and labels documents with custom embeddings and metadata, identifies duplicates/versioned documents, presents flagged issues to humans for review, and recommends correction strategies so RAG and GenAI systems use recent, relevant, fit-for-purpose data.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  input-corpus["Input corpus"] --> quality-scoring-engine["Quality scoring engine"]
  input-corpus --> clustering-and-labeling-workflow["Clustering and labeling workflow"]
  input-corpus --> deduplication-workflow["Deduplication workflow"]
  quality-scoring-engine --> human-review-app["Human review app"]
  clustering-and-labeling-workflow --> human-review-app
  deduplication-workflow --> human-review-app
  clustering-and-labeling-workflow --> metadata-cache["Metadata cache"]
  metadata-cache --> genai-application["GenAI application"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Input corpus`	Collection of unstructured documents in mixed formats.	`Quality scoring engine`, `Clustering and labeling workflow`, `Deduplication workflow`
`Quality scoring engine`	Generates individual data quality scores and overall corpus readiness view.	`Input corpus`, `Human review app`
`Clustering and labeling workflow`	Trains embeddings, clusters documents, and adds document type and fine-grained metadata tags.	`Input corpus`, `Metadata cache`, `Human review app`
`Metadata cache`	Stores document- and chunk-level tags for search and retrieval.	`Clustering and labeling workflow`, `GenAI application`
`Deduplication workflow`	Finds duplicate/versioned documents and recommends correction strategies.	`Input corpus`, `Human review app`
`Human review app`	Vizro-based interface for reviewers to inspect scores, flagged issues, and remediation recommendations.	`Quality scoring engine`, `Clustering and labeling workflow`, `Deduplication workflow`
`GenAI application`	Downstream RAG or GenAI system that consumes remediated high-quality documents.	`Metadata cache`

Data flow

An organization ingests an unstructured corpus. AI4DQ scores data quality, uses embeddings to cluster documents, labels clusters with document types and tags, optionally repeats labeling at chunk level, detects duplicate/versioned document sets, presents flagged issues to human reviewers, and recommends remediation before documents are used in a GenAI or RAG pipeline.

“AI4DQ Unstructured generates actionable insights to quantify and improve data quality readiness for gen AI applications such as RAG pipelines.”article, stated

“Taking a human-in-the-loop approach, AI4DQ Unstructured scans through the input corpus across the three dimensions and flags the documents that need attention.”article, stated

Why this architecture

The architecture addresses data quality upstream because the article says downstream GenAI failures such as hallucination, information loss, wasted compute, and leakage cannot easily be fixed after the fact.

“If practitioners fail to adequately address data quality issues upstream, there can be significant downstream challenges”article, stated

“AI4DQ Unstructured offers correction strategies to improve unstructured data quality that directly translate into gen AI results that are more accurate and consistent.”article, stated

Models used: Custom embeddings · NLP techniques · GenAI models

Scale

QuantumBlack Labs has more than 250 technologists and supports more than 1,400 data scientists across over 100 locations.
“QuantumBlack Labs has more than 250 technologists dedicated to driving AI innovation and supporting and accelerating the work of its more than 1400 data scientists across over 100 locations.”article, stated
A public-health deployment processed 2.5 GB of data.
“We ingested and processed 2.5 GB of data and identified more than ten high-priority data quality issues”article, stated
The same deployment identified issues across more than 1,500 files.
“Using AI4DQ Unstructured to prioritize the most relevant solutions, we identified issues across the 1,500+ files.”article, stated
Metadata tags improved one RAG pipeline's accuracy by 20 percent.
“One project using AI4DQ saw a 20 percent increase in RAG pipeline accuracy thanks to the addition of metadata tags on document themes.”article, stated

Alternatives considered (roads not taken)

Treat unstructured quality like structured data quality

Rejected because: The article says unstructured quality has less-established frameworks and different failure modes.

“While data quality issues in structured data can be clearly defined, and therefore resolved, there is less of an established framework for unstructured data.”article, stated

Fix GenAI failures after ingestion

Rejected because: Hallucination, information loss, wasted compute, and leakage are hard to resolve downstream.

“there can be significant downstream challenges”article, stated

Patterns

`data-quality-loop` (data)

AI4DQ scans, scores, flags, and remediates unstructured data quality before GenAI consumption.

“AI4DQ Unstructured generates actionable insights to quantify and improve data quality readiness for gen AI applications such as RAG pipelines.”article, stated

`human-in-the-loop-gate` (governance)

Flagged documents and duplicate/version decisions are presented for human review.

“Offering the human in-the-loop a view of potential duplicated/versioned documents and recommending appropriate correction strategies.”article, stated

`rag-retrieval` (retrieval)

Metadata enrichment and chunk-level labeling improve RAG retrieval accuracy.

“Chunk-level metadata can be cached alongside each document for more accurate and targeted search and retrieval.”article, stated

`privacy-preserving-processing` (privacy)

The framework flags sensitive information and inappropriate access as data-quality and compliance risks.

“Sensitive information, such as names and addresses, ingested without proper filtering or access control.”article, stated

Tradeoffs

Assess data quality before building the GenAI application

Chose upstream scoring, tagging, deduplication, and remediation over downstream fixes after the RAG app fails.

Rationale: Many downstream failures cannot easily be fixed after ingestion.

“If practitioners fail to adequately address data quality issues upstream, there can be significant downstream challenges”article, stated

Use human review for remediation

Chose human-in-the-loop correction strategies over fully automatic deletion or correction.

Rationale: Quality scores and duplicate sets require business judgment before remediation.

“Taking a human-in-the-loop approach, AI4DQ Unstructured scans through the input corpus across the three dimensions and flags the documents that need attention.”article, stated

Gotchas (what bit them)

Duplicate or outdated documents can poison downstream answers.

Root cause: Multiple versions of the same document may conflict or provide stale information.

Resolution: Resolve duplicate document entities and remove or correct outdated versions before training/retrieval.

“Document duplication is a common problem where there are multiple versions of the same document in the input corpus with minor differences.”article, stated

Metadata gaps make retrieval weak.

Root cause: Documents lack tags across diverse corpora, making search and retrieval difficult.

Resolution: Cluster documents and chunks, label document types, and add fine-grained tags.

“A lack of metadata tags across a diverse corpus of documents, making search and retrieval difficult.”article, stated

Unfiltered sensitive information becomes a compliance risk.

Root cause: PII or access-controlled content can be ingested without proper filtering.

Resolution: Treat sensitive-information filtering and access controls as part of unstructured data quality.

“Information leakage, such as PII, which creates compliance risk.”article, stated

Evaluation

AI4DQ evaluates unstructured data readiness using document quality scores, metadata coverage, duplicate/version detection, document clusters, chunk-level metadata, high-priority issue counts, and downstream RAG accuracy changes.

Metrics: Data quality score · Document clusters · Metadata tags · Duplicate/versioned documents · High-priority issue count · Storage cost savings · RAG accuracy · Critical information preservation

“A scoring mechanism presents individual data quality scores to give a view of the overall unstructured data quality of the corpus.”article, stated

Outcomes

A project using metadata tags saw a 20 percent increase in RAG pipeline accuracy.
The public-health deployment identified more than ten high-priority data quality issues.
The deployment identified and removed over 100 irrelevant or duplicated documents.
Removing those documents saved 10-15 percent in data storage cost.
The work preserved information otherwise lost for 5 percent of critical policy documents.

The public-health example processed 2.5 GB of data across more than 1,500 files.

“Identification and removal of 100+ irrelevant/duplicated documents, saving 10–15 percent in data storage cost”article, stated

“Preservation of information that would otherwise be permanently lost for 5 percent of critical policy documents.”article, stated