Production AI Agents with Dynamic Planning and Reactive Evaluation

hex · Tech · 2023 · source Agentic: yes · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Data science notebook automation

Hex needed production agents that could automate multi-step data analysis workflows without letting open-ended plans, bloated context windows, or compounding errors make the Magic product unreliable.

Constraints:

• Agent plans must be flexible enough for analysis work but bounded enough for production reliability.
• Notebook tasks have dependencies: downstream steps may require upstream outputs and must rerun when users edit upstream results.
• Latency matters because analysts remain in the loop and expect interactive responses.
• Correctness requires domain-specific assertions and executable checks rather than generic LLM quality scores.

Architecture

Hex Magic uses a constrained planning agent for data analysis workflows. The system infers a bounded plan, turns it into a DAG of dependent agent requests, runs independent steps in parallel, and keeps downstream nodes reactive to user edits. It bootstraps tools from proven human-facing capabilities such as SQL generation, prunes context aggressively for GPT-4 Turbo/GPT-3.5 calls, and evaluates with task-specific assertions and execution checks.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  magic-user-interface["Magic user interface"] --> constrained-planning-agent["Constrained planning agent"]
  constrained-planning-agent["Constrained planning agent"] --> dag-execution-planner["DAG execution planner"]
  dag-execution-planner["DAG execution planner"] --> specialized-analysis-agents["Specialized analysis agents"]
  dag-execution-planner["DAG execution planner"] --> reactive-rerun-controller["Reactive rerun controller"]
  specialized-analysis-agents["Specialized analysis agents"] --> human-facing-feature-tools["Human-facing feature tools"]
  specialized-analysis-agents["Specialized analysis agents"] --> notebook-environment["Notebook environment"]
  context-pruning-layer["Context pruning layer"] --> specialized-analysis-agents["Specialized analysis agents"]
  context-pruning-layer["Context pruning layer"] --> gpt-4-turbo-model["GPT-4 Turbo model"]
  context-pruning-layer["Context pruning layer"] --> gpt-3-5-fallback["GPT-3.5 fallback"]
  reactive-rerun-controller["Reactive rerun controller"] --> specialized-analysis-agents["Specialized analysis agents"]
  evaluation-suite["Evaluation suite"] --> specialized-analysis-agents["Specialized analysis agents"]
  evaluation-suite["Evaluation suite"] --> execution-evaluator["Execution evaluator"]
  execution-evaluator["Execution evaluator"] --> notebook-environment["Notebook environment"]

Components

Data flow

A user asks Magic for a data analysis task. The planning step produces a constrained set of allowed steps rather than arbitrary sub-agent behavior. Hex converts that plan into a DAG, uses dependency information to parallelize independent steps and sequence dependent ones, and lets users correct upstream outputs. When an upstream response changes, affected downstream agents rerun without restarting the entire workflow.

Why this architecture

Hex chose bounded plan diversity and DAG reactivity because early open-ended agent planning had too much entropy. The architecture keeps users close to the chain, exposes tools learned from human workflows, and reduces latency by minimizing context rather than relying mainly on smaller models or fine-tuning.

Models used: GPT-4 Turbo · GPT-3.5

Scale

• Hex targets short interactive chains rather than long autonomous runs; the summary contrasts this with a 10-step chain and two-day unattended workflows.
  “if each step is 95% reliable and you chain 10 of them, overall reliability drops dramatically. Brian's response is instructive: keep the chain short.”summary, stated
• The core workflow is intentionally small: get data, transform data, and make a chart.
  “For a data science workflow, the core components are: get data, transform data, make a chart.”summary, stated
• Hex uses GPT-4 Turbo as primary model and GPT-3.5 only on tasks where it has been validated as capable.
  “The team uses GPT-4 Turbo as their primary model, falling back to GPT-3.5 only when they've identified it's capable enough for specific tasks.”summary, stated

Alternatives considered (roads not taken)

Open-ended plan generation with anything-goes sub-agent orchestration

Rejected because: It produced high-entropy behavior that was exciting in demos but unreliable for production.

Ever-expanding libraries of narrow tools

Rejected because: Too-narrow tools require exponentially more tooling, while too-general tools let agents go off the rails.

Stuff more into long context windows

Rejected because: Hex found speed through context minimization, systematically removing unnecessary context.

Generic reusable evaluation metrics only

Rejected because: Many Hex evals are case-specific assertions and execution checks, because correctness is nuanced.

Patterns

multi-agent-orchestration (architecture)

Hex uses a planner plus specialized sub-agents, but constrains the kinds of steps they can execute.

durable-execution (orchestration)

The DAG representation tracks dependencies so downstream work can be rerun after upstream edits.

context-engineering (prompting)

Hex aggressively prunes prompt context to preserve speed while retaining enough information for correctness.

human-in-the-loop-gate (product)

Users remain close to the agent pipeline so they can observe, correct, and steer outputs.

eval-driven-development (evaluation)

AI feature specifications are expressed as evals and per-case assertions that outputs must satisfy.

latency-optimization (operations)

Latency is improved through minimal context construction rather than primarily by smaller models or fine-tuning.

Tradeoffs

Plan constraint level

Chose Bounded general tools and prescribed step types over Either arbitrary tools or an ever-growing set of narrow tools.

Rationale: Bounded generality gives agents enough power without inviting uncontrolled behavior or rigid tool explosion.

Human workflow bootstrapping

Chose Build user-facing capabilities before exposing them to agents over Inventing sub-agent tools first.

Rationale: Human usage teaches the context, display UX, and refinements the agent will need later.

Latency strategy

Chose Context pruning with GPT-4 Turbo primary over Mainly fine-tuning or switching to smaller models.

Rationale: Hex attributes speed to careful context minimization and uses GPT-3.5 only where it is good enough.

Autonomy boundary

Chose Interactive agent pipeline with user correction over Devon-style unattended long-running automation.

Rationale: Keeping users nearby reduces loops and allows course correction.

Gotchas (what bit them)

Open-ended plans were too high entropy.

Root cause: The planner could choose arbitrary structures and sub-agent orchestration patterns.

Resolution: Constrain the types of steps while preserving useful planning ability.

Context stuffing hurts latency and may not improve correctness.

Root cause: Teams tend to add every possible variable, package, history item, or query to the prompt.

Resolution: Start broad, then iteratively remove context until only the necessary pieces remain.

Text-to-SQL fails without tacit institutional context.

Root cause: Schemas and RAG do not capture undocumented caveats like table migrations and business rules.

Resolution: Invest in data dictionaries, known query context, latent relationships, and better retrieval/search.

Reliability compounds poorly across long agent chains.

Root cause: Even high per-step reliability can degrade over many chained steps.

Resolution: Keep the chain short and interactive rather than sending agents away for long unattended runs.

Evaluation

Hex builds a family of evaluators, often with per-case assertions and execution checks, so the desired product behavior becomes the specification for AI features.

Metrics: per-case assertions · toxicity guardrails · extractive statistics checks · vocabulary level · execution correctness · data frame generation

Outcomes

• Hex deployed Magic to automate multi-step analysis workflows in a production notebook product.
• The reactive DAG design allows upstream user corrections to propagate to downstream agent steps.
• The evaluation philosophy turns AI feature specs into concrete eval targets.

No cached source transcript is available, so quantitative production adoption is not independently stated beyond model choices and workflow shape.