Building Observable, Debuggable, and Durable Agentic Systems with Orchestration

union · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Durable orchestration for production agentic systems

Union's talk argues that once agents move from development to production, the hard problem becomes infrastructure-aware durability: agents need secure access to organizational data and tools, variable compute, parallel sub-agent/tool execution, recovery from pod/node/network failures, preservation of context, and observability across infra, logical, semantic, tool, and context layers. Standard agent evals mostly test semantic correctness, so agents often cannot reason about infrastructure failures unless the orchestrator turns failures, resources, state, and traces into usable context.

Constraints:

• Agents may need least-privilege access to organizational data assets, databases, data lakes, and tools.
• A single-node runtime cannot support machine-learning agents that may need to provision compute for training or parallel tool/sub-agent calls.
• Kubernetes pods, nodes, spot instances, and schedulers can kill containers and wipe agent memory or context.
• Failures can occur at infrastructure, network, logical, semantic, tool-execution, and context layers.
• Traditional evals typically test semantic correctness, hallucination, and tool-call formatting rather than network outages, system errors, or retry state.
• Long-running agents with many steps cannot afford to recompute completed work after every failure.
• Some failures require human context or new files when the agent exhausts its iteration budget.

Architecture

Union uses Flyte 2.0 as an orchestration substrate for observable, debuggable, and durable agents. The core building blocks are replay logs for per-run step recovery, global caching for work shared across agent runs, intermediate state persistence to object storage, task environments that specify containers/dependencies/resources/IAM, trace/durability decorators for functions, infrastructure exceptions as context, code-mode and stateless code sandboxes for self-healing, and human-in-the-loop recourse when the agent lacks context. In the Dragonfly case, a tiered ambient research architecture used agent drivers, research coordinators, researchers, and tool-layer replicas to populate a SaaS product knowledge graph over 250K products, with cross-run caching, semantic thread consolidation, checkpoint recovery, full auditability, and thousands of concurrent runs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  agent-rollout["Agent rollout"] --> flyte-2-orchestrator["Flyte 2 orchestrator"]
  flyte-2-orchestrator --> task-environment["Task environment"]
  task-environment --> kubernetes-pod["Kubernetes pod"]
  agent-rollout --> replay-log["Replay log"]
  agent-rollout --> global-cache["Global cache"]
  agent-rollout --> intermediate-state-store["Intermediate state store"]
  kubernetes-pod --> failure-context["Failure context"]
  failure-context --> resource-provisioner["Resource provisioner"]
  failure-context --> code-sandbox["Code sandbox"]
  flyte-2-orchestrator --> human-review-ui["Human review UI"]
  flyte-2-orchestrator --> trace-store["Trace store"]
  dragonfly-agent-driver["Dragonfly agent driver"] --> dragonfly-research-coordinators["Dragonfly research coordinators"]
  dragonfly-research-coordinators --> dragonfly-researchers["Dragonfly researchers"]
  dragonfly-researchers --> dragonfly-tool-layer["Dragonfly tool layer"]
  dragonfly-research-coordinators --> global-cache
  dragonfly-tool-layer --> trace-store

Components

Data flow

An agent runs inside Flyte-managed task environments, invoking tool calls, sub-agents, LLM calls, code execution, and compute-heavy jobs. Each durable function or task records granular step state into a replay log, persists intermediate values to object storage, optionally writes deterministic work into a global cache, and emits traces. When a tool call, pod, node, spot instance, memory limit, dependency, or code path fails, the exception and infrastructure context flow back to the agent loop. The agent can resume from the failed step, increase resources, regenerate code, add dependencies in a sandbox, or request human context through the UI. In Dragonfly, top-level agent drivers coordinate researchers and tool-layer replicas; cached research and checkpointed execution prevent duplicate LLM/tool work and allow failed runs to restart without losing progress.

Why this architecture

The design treats context engineering as an infrastructure problem. If a failure wipes agent state, the prompt and context work inside the agent loop becomes worthless, so the orchestrator must persist state, trace execution, cache deterministic work, surface infra failures, and let agents recover in-line. Plain Python gives agents and humans familiar control flow; task environments add isolation and resources; replay logs and caching make failures cheap; sandboxes let agents repair code or create new tools; and human review becomes the last resort for missing context.

Models used: LLMs · LangChain · Cursor agents · Claude Code · Flyte 2.0 · Pydantic Monty

Scale

• The speaker has worked on Flyte for five years.
  “I've been building and maintaining a open-source ML orchestration product called Flyte for the past 5 years.”transcript, stated
• Flyte has been used with companies including LinkedIn, Stripe, Spotify, and Mistral.
  “helping uh companies like LinkedIn, Stripe, Spotify, and more recently uh Mistral help in build and deploy production-grade ML and AI systems”transcript, stated
• Dragonfly's product-research system covered roughly 250K software products.
  “these were 250k or so software products.”transcript, stated
• Each Dragonfly product agent call had about 200 steps and about 100 LLM calls.
  “Each agent call had about 200 steps.”transcript, stated

  “Each product a little call each product had about 100 LLM calls associated with it.”transcript, stated
• The Dragonfly tiered architecture used 4 agent-driver replicas, 8 research-coordinator replicas, 12 researcher replicas, and 12 tool-layer replicas.
  3 supporting quotes

  “at the top you have the agent driver. So, you have four replicas.”transcript, stated

  “You have research coordinators that have eight replicas.”transcript, stated

  “underneath that are tool 12 replicas of a tool layer that each researcher has access to.”transcript, stated
• Dragonfly reached more than 2,000 concurrent runs.
  “And this got them to 2,000 plus concurrent runs.”transcript, stated
• The reported outcomes included 50% reduction in failure recovery time, 30% higher development velocity, and 12 hours per week saved on infrastructure maintenance.
  “Reduced their recovery failure recovery time by 50%, increased their development velocity by 30%,”transcript, stated

  “they saved 12 hours a week on infrastructure maintenance because of this.”transcript, stated
• Dragonfly onboarded a local Babel agent prototype to production in about an hour.
  “within an hour, they onboarded their local Babel agent prototype into production in about an hour.”transcript, stated

Alternatives considered (roads not taken)

Treat agent failures mostly as semantic or tool-call problems.

Rejected because: Union argues that agents also need to survive infrastructure, network, timeout, retry-state, and system-level failures, which standard evals often ignore.

Aim for failure-proof agents.

Rejected because: The speaker recommends cheap failures, fast recovery, and tight feedback loops instead, because production infrastructure will fail.

Use a domain-specific orchestration language for agent workflows.

Rejected because: The first principle is to use plain general-purpose languages that LLMs and humans know, avoiding DSL surprises while keeping loops, conditionals, exceptions, async, and framework choice available.

Recompute a whole long-running agent after a late failure.

Rejected because: Replay logs record each step and intermediate output so a crashed run can resume from the failed step instead of repeating completed tool and sub-agent calls.

Expose all tool orchestration through formatted tool calls.

Rejected because: Code mode can reduce the context load of tool descriptions and argument formatting by letting the agent write sandboxed Python over a toolbox.

Always build deep multi-tier agent hierarchies.

Rejected because: The speaker says Dragonfly's four tiers were special for scale; most internal use cases should start with one agent and maybe one sub-agent layer.

Patterns

durable-execution (orchestration)

Flyte uses replay logs, checkpointing, intermediate state persistence, and functional durability hooks so long-running agents can recover from crashes, OOMs, spot-preemption, and other infrastructure failures without recomputing completed work.

observability-tracing (operations)

The orchestrator traces every LLM and tool call, records task state and intermediate outputs, and turns failures into auditable context for recovery and debugging.

semantic-caching (cost-optimization)

The system combines per-run replay caching, global cross-run caching, and Dragonfly's semantic consolidation of similar research threads to avoid duplicate work and repeated LLM calls.

human-in-the-loop-gate (recovery)

When an agent lacks context or exhausts its max-iteration budget, Flyte can pause for a human to provide context, upload files, or give a directory before recursively continuing the agent.

tool-use-orchestration (agent-architecture)

Flyte task environments and sandboxes let agents run tools, training jobs, code snippets, imports, limited network IO, and filesystem actions under orchestrator control.

multi-agent-orchestration (scale-out)

Dragonfly's production architecture uses a tiered agent driver, research coordinators, researchers, and tool-layer replicas to scale ambient deep-research work over a large product graph.

fallback-degradation (resilience)

The agent recovery loop catches failures, adjusts resources, adds dependencies, regenerates code, and falls back to human context when automation can no longer self-heal.

Tradeoffs

Optimize for cheap failures rather than no failures.

Chose Replay logs, caching, checkpointing, and quick recovery loops over Trying to make agents failure-proof.

Rationale: Production agents will fail at multiple layers, so the useful objective is fast recovery without recomputing completed work.

Use plain Python as the orchestration surface.

Chose General-purpose language constructs and framework compatibility over A new DSL with hidden semantics.

Rationale: Plain Python gives loops, fan-out, conditionals, try/except, async/sync programming, and familiar constructs for both humans and LLMs.

Cache deterministic work but regenerate creative synthesis.

Chose Cache web search and database reads while disabling cache for the composer sub-agent over Caching every step or caching no steps.

Rationale: Research and IO can be reused, but synthesis may need fresh creativity or entropy on each run.

Use spot instances with checkpoint recovery for cost reduction.

Chose Checkpoint-based recovery on interruptible compute over Avoiding spot/preemptible infrastructure because it can disappear.

Rationale: Dragonfly could reduce costs with spot instances because checkpoint recovery made instance loss a non-issue for progress.

Use sandboxes for self-healing while preserving safety boundaries.

Chose Code-mode and stateless code sandboxes with constrained IO/import/network behavior over Letting agents execute arbitrary code freely or only use fixed tools.

Rationale: Sandboxes give agents enough freedom to fix orchestration/code/dependency issues and write tests, while limiting unsafe side effects.

Gotchas (what bit them)

Semantic evals miss infrastructure failure modes.

Root cause: Evals usually test answer correctness, hallucination, and tool-call format, not network outages, timeouts, system errors, or state across retries.

Resolution: Treat infrastructure state and exceptions as context that the agent and orchestrator can reason over.

Agent context is lost when the infrastructure crash wipes process state.

Root cause: The context engineered into the agent loop is held in execution state unless the orchestrator persists it.

Resolution: Persist intermediate state, replay logs, and lineage artifacts through durable hooks and object storage.

Single-node runtimes create contention for compute-heavy or parallel agent work.

Root cause: ML agents may need to train models on large datasets, run parallel sub-agents, or execute tool calls that outgrow one fixed runtime.

Resolution: Let the orchestration layer provision task-specific containers, resource requests, and compute inside the agent loop.

Tool context and argument formatting can become excessive for orchestration tasks.

Root cause: Tool-call schemas, argument formatting, and result parsing can consume context and restrict how the agent combines operations.

Resolution: Use code-mode sandboxes where the agent writes limited Python over a toolbox, reducing tool-formatting overhead.

Four-tier agent architectures should not be copied blindly.

Root cause: Dragonfly's deep hierarchy was driven by an unusually scaled, non-deterministic research workload with duplicate thread detection needs.

Resolution: Start with one agent and at most one layer of sub-agents unless the workload demands more tiers.

Human context is still needed when the agent lacks the right prompt, files, or information.

Root cause: Some failures are caused by poor prompts or missing context rather than recoverable system or code errors.

Resolution: Pause for human-in-the-loop context injection after max iterations, then continue with the added context.

Evaluation

The talk argues that agent evaluation must extend beyond semantic correctness into infrastructure context, recovery behavior, traces, and failed-run feedback. Flyte traces every LLM/tool call, preserves failures as training data or inline context, and lets agents respond to infra errors such as out-of-memory by changing resources. The Dragonfly deployment reports concrete operational outcomes after adopting these durability mechanisms.

Metrics: Semantic correctness, hallucination, and tool-call argument formatting in conventional evals · Recovery from network outages, timeouts, system-level errors, and state across retries · Every LLM and tool call traced · Failure recovery time reduced by 50% · Development velocity increased by 30% · 12 hours per week saved on infrastructure maintenance · 2,000+ concurrent runs

Outcomes

• Flyte 2.0 is positioned as durable, dynamic, infrastructure-aware orchestration for agents, ML, data science, and ETL.
• Dragonfly onboarded a local Babel agent prototype into production in about one hour.
• Dragonfly reached 2,000+ concurrent runs.
• Dragonfly reduced failure recovery time by 50%, increased development velocity by 30%, and saved 12 hours per week on infrastructure maintenance.
• Checkpoint recovery made heavy spot-instance use effectively non-disruptive.
• The architecture gave full auditability over LLM and tool calls.

The strongest production evidence is the Dragonfly case: about 250K software products, about 200 steps and 100 LLM calls per product agent call, a four-tier replica architecture, 2,000+ concurrent runs, and measurable recovery/development/maintenance gains.