Building and Managing Production Agents with Testing and Evaluation Infrastructure

nearpod · Education · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: K-12 educational content agents

Nearpod needed production agents for teacher-facing question generation while operating in a global K-12 context with cultural, legislative, cost, and quality constraints.

Constraints:

• Educational content must account for students, parents, cultures, and legislation across the world.
• Agent quality depends on a reliable data platform spanning many sources and large row counts.
• Production cost must be estimated from token usage and model choices.
• Agents require thousands of evals and CI confidence before release.

Architecture

Nearpod built on a data-platform foundation covering disparate sources and large datasets, then created specialized agents for input validation and question generation. An agent registry makes agents discoverable across the organization, while thousands of evals in CI measure accuracy, cost, token size, and model choices.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  data-platform[Data Platform] --> input-validation-agent[Input Validation Agent]
  input-validation-agent --> question-generation-agent[Question Generation Agent]
  question-generation-agent --> feedback-loop[Feedback Loop]
  feedback-loop --> data-platform[Data Platform]
  question-generation-agent --> evaluation-framework[Evaluation Framework]
  evaluation-framework --> agent-registry[Agent Registry]
  data-platform --> evaluation-framework[Evaluation Framework]

Components

Data flow

The data platform consolidates sources into reusable data products, an input-validation agent checks requests, a question-generation agent produces educational questions, feedback loops process evaluation data, and CI runs thousands of evals to estimate quality and cost before production use.

Why this architecture

Nearpod narrowed agents into specialized domains because smaller scopes reduce token use, cost, and complexity. The data platform and eval framework are the backbone: they make feedback loops, cost estimation, CI gates, and cross-departmental reuse possible.

Models used: LLMs · cheaper models · specialized agents

Scale

• Nearpod operates in K-12 across the US and globally.
  “of at nearpod uh we are in the edch space for K12 throughout the us but also throughout the world and how do I take my coffee I take my coffee black every time uh yeah I I learned that in college where I was desperate for money and cream was”transcript, stated
• The data platform spans sources with millions to billions of rows.
  “have anywhere from millions of rows to billions of rows so we're talking large scale data uh to some degree not not metat large right but but larger than a than a PC if you will um and so we're able to do this with consistency with reliability and with”transcript, stated
• A prototype that might previously take 60 days or more was built in seven hours.
  “it took us seven hours to build something like this right which in the past is something that's just unimaginable we would have had to take 60 days”transcript, stated
• Nearpod reports 98 to 100% eval accuracy for lower-cost models.
  “that we're getting really accurate results to the tune of 98 to 100% accuracy in our evals for a lower cost because we're able to use better like cheaper models and whatnot and why do you think that is because you're passing in more context”transcript, stated

Alternatives considered (roads not taken)

Broad, general-purpose agents.

Rejected because: The team emphasized narrowing scope as much as possible to reduce cost and complexity.

Use expensive models by default.

Rejected because: Evals showed cheaper models could reach the required accuracy when prompts and tokens were optimized.

Let agent cost remain unknown until production.

Rejected because: Nearpod estimates production cost from evals, usage, token size, and call count before rollout.

Patterns

multi-agent-orchestration (agent architecture)

Input validation and question generation are separated into specialized agents.

eval-driven-development (release quality)

Thousands of CI evals give confidence in each agent's accuracy.

cost-aware-routing (model operations)

Evals and token accounting guide cheaper model use and production cost estimates.

data-quality-loop (feedback)

The data platform processes feedback to understand what should be updated.

Tradeoffs

Agent scope

Chose Narrow specialized agents over Broad general agents.

Rationale: Narrow agents require fewer tokens, run cheaper, and are easier to evaluate.

Model cost

Chose Cheaper models where evals prove quality over Defaulting to expensive models.

Rationale: Nearpod could maintain 98-100% eval accuracy at lower cost.

Release confidence

Chose Thousands of CI evals over Manual confidence alone.

Rationale: A large eval suite gives release confidence in a regulated, global education context.

Gotchas (what bit them)

Educational agents have high-context safety and localization demands.

Root cause: K-12 content varies by students, parents, legislation, language, and culture.

Resolution: Use strict evals and domain-scoped agents rather than generic generation.

Agent costs can balloon with token size and call count.

Root cause: Prompt sentences, number of prompts, and production usage multiply cost.

Resolution: Use evals to estimate production cost and optimize tokens/calls before rollout.

Long agent projects create sunk-cost pressure.

Root cause: Without fast POCs and evals, teams can spend months before discovering a weak agent.

Resolution: Build POCs in hours and use eval evidence to shut down weak ideas early.

Evaluation

Nearpod runs a custom evaluation framework with thousands of CI evals per agent, measuring correctness, cost, token size, call count, and cheaper-model feasibility.

Metrics: 98-100% eval accuracy · Thousands of evals · Token size · Number of calls · Production cost estimate

Outcomes

• Nearpod built agentic question-generation infrastructure for teachers.
• The team reported 98-100% eval accuracy while using cheaper models.
• The data platform and registry made agents reusable across departments.

The source reports global K-12 operation, millions to billions of rows, seven-hour POCs versus 60-day-plus prior timelines, and thousands of evals per agent.