Data Quality Loop

Definition

A data quality loop turns production errors, human feedback, audits, and changing source data into corrections to the knowledge, examples, labels, schemas, or evaluation sets used by an AI system. The defining feature is a closed update cycle rather than one-time cleanup. Sierra learns from human transfers to identify missing knowledge and draft articles; Sword feeds online findings back into datasets and criteria. Sierra Sword Health

Corpus Support

The analysis counts 98 deduplicated agentic systems (9.9%) across 12 industries. Evidence includes 49 article, 107 transcript, and 8 summary quotes; 161 are stated and 3 implied. Corpus analysis

Problem It Solves

Agent quality decays when knowledge, schemas, labels, and user behavior change while the system remains static. Stripe found text-to-SQL errors rooted in table discovery and responded by deprecating low-quality tables and assigning documentation owners; Lexmed updates correction rules and regulatory data from human ground truth and new legislation. Stripe Lexmed

Applicability Signals

• Human escalations reveal missing procedures or knowledge. Sierra
• Retrieval or tool data becomes stale through ordinary business changes. Devin Kearns
• Production traces expose hard cases absent from the initial dataset. Sword Health
• Source quality and ownership, rather than model reasoning, dominate errors. Stripe

Architecture and Variants

• Escalation-to-knowledge loop: mine human transfers, identify gaps, draft fixes, and prioritize review. Sierra
• Continuous knowledge ingestion: collect business events into the retrieval store rather than loading once. Devin Kearns
• Production-to-eval loop: add feedback, audits, traces, and hard cases to evaluation datasets and criteria. Sword Health
• Governed source cleanup: deprecate weak tables and assign human owners to canonical data. Stripe

Why Teams Choose It

The loop improves the system at the layer where recurring failures originate rather than repeatedly patching prompts. It also makes production operations a source of maintained evidence: Faber Labs uses in-session feedback to alter context selection and agent behavior, while Sword converts observed failures into reusable tests. Faber Labs Sword Health

Tradeoffs

• Fast ingestion keeps knowledge current but can propagate unreviewed or adversarial data. Devin Kearns
• Human ownership improves canonical data while creating a maintenance obligation. Stripe
• Production feedback improves relevance but can overfit to recent users and expose privacy-sensitive behavior. Faber Labs

Gotchas and Resolutions

• A one-time vector load silently grows stale; connect source events to reviewed updates. Devin Kearns
• Random production sampling misses rare critical cases; deliberately add hard examples and audits. Sword Health
• Model-looking failures can actually be catalog or schema failures; inspect source selection and ownership before tuning the model. Stripe

When Not to Use It

Do not automatically learn from every user interaction or model output. Feedback can be wrong, private, or adversarial; high-impact updates need provenance and review. Sierra drafts missing knowledge for prioritization rather than silently promoting every human-transfer outcome into production instructions. Sierra

Boundaries and Related Patterns

data-quality-loop changes source data, labels, or evaluation cases. eval-driven-development measures behavior and may generate the feedback that enters the loop. fine-tuning-adaptation changes model parameters using curated data; a quality loop may produce that data but can instead update retrieval, schemas, or rules. Sword Health Stripe

Operational Checklist

• Record the failure, source, correction, reviewer, and affected artifacts. Sword Health
• Separate automatic collection from reviewed promotion. Sierra
• Assign owners and freshness expectations to canonical data. Stripe
• Re-run evaluation after every promoted data or rule change. Sword Health

Representative Implementations

Methodology and Limitations

This page samples 98 deduplicated systems. The corpus often reports that feedback is incorporated without measuring causal improvement, so the page treats the loop as an operational structure rather than proof that every update helps. Corpus methodology