Building Agents for High-Stakes Production Systems with Feature Platform Infrastructure

zipline · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: High-stakes ML feature platform automation

Zipline needed agents to help improve production ML systems such as fraud, trust and safety, and personalization without letting autonomous experiments interfere with production infrastructure or produce unreviewable pipeline code.

Constraints:

• The target systems are high-stakes production ML systems.
• Agents must create training data, serving pipelines, model versions, and dev deployments consistently.
• Experiments must be isolated from production compute, storage, and serving.
• Human reviewers must be able to understand and approve what agents produce.

Architecture

Zipline uses Chronon as an infrastructure abstraction layer for agents. Agents define feature semantics through Chronon's API, while Chronon automates training pipelines, serving pipelines, consistency, branch-based resource isolation, partial-aggregate reuse, shared feature reuse, and human-reviewable outputs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  experiment-agent[Experiment Agent] --> chronon-feature-api[Chronon Feature API]
  experiment-agent --> human-reviewer[Human Reviewer]
  chronon-feature-api --> training-pipeline[Training Pipeline]
  chronon-feature-api --> serving-pipeline[Serving Pipeline]
  chronon-feature-api --> branch-isolated-resources[Branch Isolated Resources]
  chronon-feature-api --> partial-aggregate-cache[Partial Aggregate Cache]
  training-pipeline --> branch-isolated-resources
  serving-pipeline --> branch-isolated-resources
  partial-aggregate-cache --> shared-feature-repository[Shared Feature Repository]
  human-reviewer --> shared-feature-repository

Components

Data flow

An agent proposes or modifies features through the Chronon feature API, Chronon builds isolated dev training and serving infrastructure for the branch, reuses production aggregates where safe, produces a reviewable experiment, and hands the change to a human to decide on A/B testing or production launch.

Why this architecture

Chronon gives agents a semantic, reviewable interface to production-grade ML infrastructure; the alternative would be asking agents to directly write complex data engineering and serving code across many systems.

Models used: LLM agents

Scale

• Chronon originally compressed new feature or model production workflows from months to a day or a couple of days.
  “the workflow to get a new model or a new feature into production went from months to basically a day or a couple of days”transcript, stated
• Airbnb used Chronon in payment fraud and broader production ML systems.
  “the very first use case that we built it to support was payments fraud detection. Um so Airbnb, travel company, but in many ways also a payments company. It moves money across bank accounts, across borders, from guests and hosts um at a very large volume.”transcript, stated
• The compute-reuse example avoids recomputing 100 or 1,000 unchanged features when adding one or a few.
  “the agent to have to recompute, you know, all 100 or 1,000 features that it did not change when it's only trying to add one or a couple.”transcript, stated
• Resource-limit controls are framed for large numbers of concurrent agents, including a 2,000-agent example.
  “2,000, you know, uh agents running at once, basically, the agents are going to get stuck, and that's what practically is going to happen. And then that's going to percolate up uh through the org chart as like the person who's like, "Hey, I'm trying to do this thing, but”transcript, stated

Alternatives considered (roads not taken)

Let agents directly interact with streaming, batch, serving, and training infrastructure.

Rejected because: That would make experiments unsafe, unreviewable, and too complex for agents to manage reliably.

Full autonomy to production

Rejected because: The system intentionally hands experiments to a human for review, A/B test decisions, and production launch decisions.

Full recomputation for every agent experiment

Rejected because: Higher iteration speed would otherwise create unacceptable infrastructure cost increases.

Patterns

tool-use-orchestration (agent-platform)

Chronon gives agents a constrained semantic API for feature definition rather than raw access to many infrastructure systems.

human-in-the-loop-gate (safety)

Agents produce production-ready dev experiments, but humans review and decide whether to A/B test or ship.

privacy-preserving-processing (isolation)

Branch-based resource isolation separates agent experiments from production compute, storage, and serving.

semantic-caching (cost-control)

Chronon reuses partial aggregate caches and production-computed features so experiments avoid recomputing unchanged work.

Tradeoffs

Agent interface to production ML infrastructure

Chose Chronon's semantic feature API over Agent-authored Spark, Flink, service, and serving code.

Rationale: The semantic API is reviewable and lets Chronon automate production-grade infrastructure.

Experiment isolation

Chose Branch-routed isolated compute, storage, and serving over Experiments sharing production infrastructure directly.

Rationale: Isolation lets agents experiment without adding production load.

Reuse policy

Chose Reuse production-computed unchanged features and cached aggregates over Recompute all historical features per experiment.

Rationale: Reuse preserves safe isolation while avoiding cost spikes as agents increase iteration rate.

Automation boundary

Chose Agent prepares dev experiment and human decides launch over Agent launches directly to production.

Rationale: The systems are too high-stakes for unsupervised production changes.

Gotchas (what bit them)

Agent-produced experiments are not useful if humans must rebuild them for production.

Root cause: A demo or research output that is not production-ready fails to accelerate high-stakes ML workflows.

Resolution: Require the agent's dev output to be production-ready and reviewable from the start.

Raw infrastructure access makes agent changes unsafe.

Root cause: Agents would otherwise create training data, serving pipelines, model deployments, and consistency guarantees across many systems.

Resolution: Use Chronon to automate infrastructure while exposing only semantic feature definitions.

More agent experiments can explode infrastructure costs.

Root cause: Higher iteration speed multiplies compute unless unchanged features and aggregates are reused.

Resolution: Make compute reuse a core platform primitive with partial aggregate caching and shared repositories.

Training-serving inconsistency is difficult to debug in high-stakes ML.

Root cause: Feature logic was historically split across batch pipelines, streaming jobs, and services.

Resolution: Use a single feature API and automated engine for both training data and production inference.

Evaluation

Agents run feature experiments, train model versions, evaluate results, and produce a dev pipeline for human review before A/B testing or production launch.

Metrics: Model training results · Experiment evaluation results · Reviewability of the branch · Production-readiness of dev pipelines · Compute reuse and infrastructure cost

Outcomes

• Chronon reduced traditional feature/model production iteration from months to roughly a day or a couple of days.
• The same abstraction becomes a safety layer for agents working on production ML experiments.
• Branch isolation and compute reuse allow more agent experimentation without production interference.

The source discusses large-volume payments fraud, latency/high-QPS use cases, 100-1,000 unchanged features in reuse examples, and a 2,000-agent resource-limit scenario.