Data Flywheels for Cost-Effective AI Agent Optimization

nvidia · Tech · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise AI agent optimization

NVIDIA needed its internal NV Info Agent router to keep routing employee questions to the right expert agent as data, user preferences, and costs changed, without relying forever on a high-latency 70B model.

Constraints:

Enterprise data and user preferences change over time.
Large models increase inference cost as usage grows.
The router must send queries to the correct expert agent.
Smaller models started with much lower routing accuracy.
Production logs and user feedback need human/SME curation before becoming ground truth.

Architecture

NVIDIA built a data flywheel around NV Info Agent. Employee queries pass through guardrails into a router LLM that selects an expert agent, each backed by RAG. Production inference logs and user feedback are curated with human/SME review into ground truth, evaluated with NeMo Evaluator and LLM-as-judge, used to fine-tune smaller models through NeMo Customizer, and then promoted when accuracy, latency, and cost meet the target.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  employee-user["Employee user"] --> guardrails["Guardrails"]
  guardrails --> router-agent["Router agent"]
  router-agent --> expert-agents["Expert agents"]
  expert-agents --> rag-pipelines["RAG pipelines"]
  router-agent --> production-logs["Production logs"]
  production-logs --> ground-truth-curation["Ground truth curation"]
  ground-truth-curation --> model-evaluation["Model evaluation"]
  model-evaluation --> fine-tuning-pipeline["Fine-tuning pipeline"]
  fine-tuning-pipeline --> promoted-model["Promoted model"]
  promoted-model --> router-agent

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Employee user`	Submits internal support questions to NV Info Agent.	`Guardrails`
`Guardrails`	Protects the interaction for safety, security, and privacy.	`Employee user`, `Router agent`
`Router agent`	LLM router that classifies intent/context and selects the correct expert agent.	`Guardrails`, `Expert agents`, `Production logs`, `Promoted model`
`Expert agents`	Domain-specific agents for HR, finance, IT, product docs, and other employee knowledge domains.	`Router agent`, `RAG pipelines`
`RAG pipelines`	Retrieve relevant enterprise information for the chosen expert agent.	`Expert agents`
`Production logs`	Inference logs and feedback data captured while the router runs.	`Router agent`, `Ground truth curation`
`Ground truth curation`	Human/SME process that labels failures and creates training/evaluation examples.	`Production logs`, `Model evaluation`
`Model evaluation`	NeMo Evaluator and LLM-as-judge process for comparing candidate models.	`Ground truth curation`, `Fine-tuning pipeline`
`Fine-tuning pipeline`	NeMo Customizer process that adapts smaller models for routing.	`Model evaluation`, `Promoted model`
`Promoted model`	Chosen smaller or larger model deployed to power the router.	`Router agent`

Data flow

An employee submits a query. Guardrails protect the interaction, the router agent chooses an expert agent, and the expert retrieves enterprise knowledge. Feedback and production logs feed a flywheel: curate ground truth, analyze failures, evaluate models, fine-tune smaller variants, compare routing accuracy/latency/cost, and promote the best model as the router.

“the router agent depending on the user query understands the intent and context and guides or routes the query to one of these expert agents which further uses the rack pipeline underneath”transcript, stated

“we have a data fly loop set up uh which constantly builds on top of the user feedback and production data inference log that come in when this router is in action.”transcript, stated

Why this architecture

The flywheel exists to convert production feedback into smaller, cheaper models that retain large-model routing accuracy. NVIDIA uses curation, evaluation, fine-tuning, and guardrails as an end-to-end loop rather than treating model choice as a one-time selection.

“this data flywheel cycle triggers a continuous cycle of data curation of ground truth using inference data, business intelligence, user feedback to continuously experiment and evaluate existing and newer models”transcript, stated

“surface efficient smaller models that provide at par accuracy expectations as that of larger language models but which offer lower latency, faster inference and effectively lower cost”transcript, stated

Models used: Llama 3.1 70B · Llama 3.3 70B · 8B model variant · 3B model variant · 1B model variant · LLM-as-a-judge via NeMo Evaluator

Scale

The NV Info Agent covers internal knowledge across HR benefits, financial earnings, IT help, product documentation, and other employee needs.
“agent can help answer queries across a variety of domains from HR benefits, financial earnings, IT help, product documentation or anything everything that an internal employee needs access to.”transcript, stated
The feedback pass produced 1,224 responses, including 729 satisfactory and 495 unsatisfactory responses.
“we were able to curate around,224 data points. 729 of them were incorrect sorry satisfactory responses and 495 of them were unsatisfactory responses.”transcript, stated
The final ground-truth set contained 685 data points.
“we figured out and set up a ground trth data point data set of 8 68 85 data points and we split them into 6040”transcript, stated
The 70B baseline reached 96% routing accuracy with 26-second first-token latency.
“the 70B variant like we saw provided us gave us an accuracy of 96% uh for accurately routing it to the right export agent but it came with an latency of 26 seconds”transcript, stated
A 1B model reached 94% accuracy with 98% lower inference cost and 70x lower latency.
“by deploying let's say a 1B model, you're looking at a 98% savings in terms of lower inference cost and also 10x 70x model size reduction with 70x lower latency.”transcript, stated

Alternatives considered (roads not taken)

Use the 70B router because its baseline accuracy is high

Rejected because: It had high latency and cost compared with smaller models after fine-tuning.

“Hey, we need to go with uh a 70B variant because we're seeing a 96 to a 14% comparison. I don't think even with fine-tuning we would be able to get there.”transcript, stated

Deploy smaller models without customization

Rejected because: The 8B baseline was below 14% routing accuracy before fine-tuning.

“if you see the smaller variants, the accuracy is subpar. The 8b variant is below 14%. And that's not ideal for our use case.”transcript, stated

Patterns

`multi-agent-orchestration` (architecture)

A router agent chooses among multiple expert agents for employee support domains.

“it's basically orchestrated by a mixture of agent architecture underneath. So the router agent depending on the user query understands the intent and context and guides or routes the query”transcript, stated

`data-quality-loop` (data)

Production feedback and logs are curated into ground truth for recurring model improvement.

“using a subject matter expert and human the loop feedback the ground truth is continuously curated.”transcript, stated

`fine-tuning-adaptation` (modeling)

Smaller models are fine-tuned on curated router data to approach 70B accuracy.

“the 8B with a bit of finetuning was able to match the accuracy of a 70B variant”transcript, stated

`model-routing` (orchestration)

The central router model selects the expert agent for each query.

“which of the agents the expert agents should the router route the query to uh and the goal here is to ensure that our agent the routing agents uh accurately routes it”transcript, stated

`cost-aware-routing` (cost)

Model selection explicitly trades routing accuracy against inference cost and latency.

“you can take uh a trade-off between how accurate you want your agent to be versus the cost that you want to manage and the resource that you want resource that you want to manage underneath.”transcript, stated

Tradeoffs

Use a 1B model in some cases

Chose 94% accuracy with much lower cost and latency over maximum 96% routing accuracy from the 70B model.

Rationale: Some missions can accept a small accuracy gap in exchange for 98% lower inference cost and much lower latency.

“the 1B variant. It's got a 94% accuracy. That's not bad. Uh it's just 2% below 70B.”transcript, stated

Use employee feedback as training signal

Chose curated production feedback and logs over static benchmark-only model selection.

Rationale: The production router needs to adapt to real employee queries and failure modes.

“Start thinking about intuitive ways to collect user feedback signals, right? Intuitive user experience, privacy compliance, implicit signals, and even explicit signals”transcript, stated

Gotchas (what bit them)

Large models can hide an unnecessary cost/latency penalty.

Root cause: Baseline comparisons make the 70B model look like the only accurate router before fine-tuning smaller models.

Resolution: Run a data flywheel to curate examples and fine-tune smaller variants before deciding.

“that's exactly where most of the enterprise evaluations get it wrong. So they see a bigger model uh deliver state-of-the-art amazing accuracy of 96%”transcript, stated

Unsatisfactory responses are not all router failures.

Root cause: Only a subset of user-reported unsatisfactory responses were actually due to incorrect routing.

Resolution: Use LLM-as-judge plus SME/manual analysis to attribute failures before training.

“We figured out that 140 of them uh were due to incorrect routing and using a subject matter expert and further manual analysis we found out 32 of them were truly due to this incorrect routing.”transcript, stated

Evaluation

NVIDIA evaluated routing accuracy, first-token latency, inference cost, model size reduction, user feedback, unsatisfactory response attribution, LLM-as-judge results, and SME-confirmed ground truth.

Metrics: Router accuracy · First-token latency · Inference cost · Model size · User satisfaction · Incorrect routing count · Ground-truth data points

“setting up a regular cadence or a mechanism to track your accuracy, latency, um monitoring, uh performance and production logs”transcript, stated

Outcomes

A fine-tuned 8B model matched the 70B router's accuracy.
A 1B model reached 94% accuracy, only 2 points below the 70B baseline.
The 1B option offered 98% lower inference cost and 70x lower latency.
The flywheel produced a curated 685-point ground-truth dataset.
The system established a recurring loop for evaluation and fine-tuning.

The system supports NVIDIA employee knowledge across multiple internal domains and uses production feedback/logs to update the router.

“just with 685 data points the results that we achieved were outstanding and this was possible only because of the data flywheel setup.”transcript, stated