Building Agentic Workflows with Temporal for Data Infrastructure at Scale

instacart · E-commerce · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Durable AI workflow orchestration for data infrastructure

Instacart needed to run LLM and agentic data workflows on production infrastructure where non-deterministic model calls, human review, evals, and large batch processing still required reliability, durability, auditability, and scale.

Constraints:

Instacart already ran mission-critical order workflows on Temporal at very high volume.
LLM calls are non-deterministic and can hallucinate, so they cannot be treated like ordinary deterministic functions.
Some AI-generated changes require human approval for confidence, audit, and compliance reasons.
Batch inference workloads need child-level retries and large-scale data processing semantics.

Architecture

Instacart treats LLM calls as Temporal activities and agent state as Temporal workflows. Human-in-the-loop flows use signals, timers, and workflow updates; eval systems use child workflows for scorers and LLM judges; and batch inference splits large data processing into child workflows with independent retry semantics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  temporal-clusters[Temporal Clusters] --> temporal-workers[Temporal Workers]
  temporal-clusters --> workflow-event-history[Workflow Event History]
  temporal-workers --> agent-workflow[Agent Workflow]
  agent-workflow --> llm-activity[LLM Activity]
  llm-activity --> llm-providers[LLM Providers]
  agent-workflow --> human-reviewer[Human Reviewer]
  agent-workflow --> child-workflows[Child Workflows]
  child-workflows --> eval-judges[Eval Judges]
  eval-judges --> llm-providers
  child-workflows --> batch-data-corpus[Batch Data Corpus]
  agent-workflow --> workflow-event-history

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Temporal Clusters`	Self-hosted orchestration platform running Instacart production and AI workflows.	`Temporal Workers`, `Workflow Event History`
`Temporal Workers`	Worker processes that execute activities, workflow logic, and AI-related orchestration.	`Temporal Clusters`, `Agent Workflow`
`Agent Workflow`	Durable workflow holding agent state and coordinating activities, child workflows, signals, timers, and updates.	`Temporal Workers`, `LLM Activity`, `Human Reviewer`, `Child Workflows`, `Workflow Event History`
`LLM Activity`	Model call wrapped as a Temporal activity with retries, timeouts, heartbeats, and health checks.	`Agent Workflow`, `LLM Providers`
`LLM Providers`	OpenAI, Claude, Google, open-source, and other models used behind Instacart AI applications.	`LLM Activity`, `Eval Judges`
`Human Reviewer`	Person approving, rejecting, or editing LLM-generated changes before they land.	`Agent Workflow`
`Child Workflows`	Parallel workflow units used for eval scorers, judges, and batch inference chunks.	`Agent Workflow`, `Eval Judges`, `Batch Data Corpus`
`Eval Judges`	Deterministic and LLM-based scorers, including ensemble judges, used to assess agent outputs.	`Child Workflows`, `LLM Providers`
`Batch Data Corpus`	Structured or unstructured datasets split into manageable chunks for inference and enrichment.	`Child Workflows`
`Workflow Event History`	Temporal's durable audit and observability record for workflow execution.	`Temporal Clusters`, `Agent Workflow`

Data flow

A workflow owns the durable agent state, invokes LLM calls as retryable activities, waits for human signals when approval is required, forks child workflows for eval scorers or batch chunks, and records audit and observability events through Temporal's event history and integrations.

“if you wrap the LLM call and think of the LLM call as an activity and then you think of the workflow as like the agent state.”transcript, stated

“You have a workflow that starts, you spin up child workflows for your scorers, your scorers come back with some answer”transcript, stated

Why this architecture

Temporal was already Instacart's reliability substrate, and its retries, heartbeats, event histories, signals, child workflows, and workflow updates matched the failure modes of long-running AI workflows better than treating LLM calls as ordinary request/response code.

“the ability to wrap different sophisticated retry policies on top of your activities, the ability to add heartbeats, configure timeouts”transcript, stated

“ability to inject signals or or updates, and pick it up from there. All of those primitives started making more sense”transcript, stated

Models used: OpenAI models · Claude models · Gemini · Open-source models · LLM judges

Scale

Instacart runs close to 56 million Temporal workflows per day.
“Temporal in production. We run close to 56 million workflows a day. Um like I mentioned, we run it all on a self-hosted cluster or actually a couple of clusters.”transcript, stated
Temporal is used across 45 namespaces.
“We run it across 45 different namespaces. So, namespaces is a very rough analogy to different teams that use Temporal and run production work workloads on on our Temporal clusters.”transcript, stated
One large namespace runs close to 200,000 actions per second.
“And then on one of our largest namespace, we run close to 200,000 actions per second. Um and that's where we are seeing a lot of uh so, those are like mission-critical production workflows that run as part of every order that that you place through Instacart.”transcript, stated
These workflows include mission-critical order-path work.
“mission-critical production workflows that run as part of every order that that you place through Instacart. Um so, I've This is my third time speaking at Replay and I've shown this slide at almost all the three times, simply because we learned a lot when we”transcript, stated

Alternatives considered (roads not taken)

Ship LLM-generated changes automatically after tests.

Rejected because: Instacart still wanted human approval for confidence, auditability, and compliance.

“we still want like a human in the process to approve it, not just from like a testing point of view or like a confidence point of view”transcript, stated

“view, but also from like an audit auditability from a compliance point of view. So, that's like a that's like a pretty prominent use case, leverage multiple places across the company.”transcript, stated

Use only deterministic checks for LLM quality.

Rejected because: LLM outputs and business logic require non-deterministic evals and sometimes multiple judges.

“doing. So, it's not just like you want to run a set of deterministic check against the LLM output or not not just LLM output, but also let's say the the business on top of the LLM.”transcript, stated

“Um and then sometimes people don't just stop at running a single judge, you may want to do like an ensemble judge. So I could take, you know, in one of our systems I run an ensemble eval model where I have a Gemini agent, I have a Claude agent and I have a OpenAI agent,”transcript, stated

Use Airflow-style batch processing for LLM enrichment.

Rejected because: Instacart saw LLM batch inference as needing stronger reliability and durability primitives.

“people typically used to do it with Airflow. Not saying that Airflow is not the right solution, but when you talk about LLMs, reliability and durability becomes like key.”transcript, stated

Patterns

`durable-execution` (orchestration)

Instacart maps LLM calls to Temporal activities and agent state to workflows so AI execution inherits retries, heartbeats, signals, event history, and child workflow semantics.

“wrap the LLM call and think of the LLM call as an activity and then you think of the workflow as like the agent state.”transcript, stated

`human-in-the-loop-gate` (safety)

LLM-generated changes wait for human approve/disapprove/edit signals before being checked in or applied.

“you have like a human in in between who is going to just do like an approve disapprove or an edit on that generated description.”transcript, stated

`llm-as-judge` (evaluation)

Instacart runs LLM judges, sometimes as ensembles across Gemini, Claude, and OpenAI, to evaluate agent outputs.

“I run an ensemble eval model where I have a Gemini agent, I have a Claude agent and I have a OpenAI agent”transcript, stated

`batch-offline-processing` (data-processing)

Large corpora are split into child workflows, each with its own retry semantics, for LLM enrichment and model scoring.

“you spin it out into child workflows. Each child workflow is can have its own retry semantics.”transcript, stated

`observability-tracing` (operations)

Temporal event history is used as an observability and audit substrate for increasingly complex AI workflows.

“the event history, the ability to monitor the entire event history uh you know, from a from a observability point of view”transcript, stated

Tradeoffs

Orchestration substrate for agent state

Chose Temporal workflows and activities over A simple request/response chatbot or local agent loop.

Rationale: The workflow complexity grew into retries, heartbeats, child workflows, signals, updates, and observability needs.

“evolving from a request response loop to something like a more streamable interface and then to the point where today we we are talking about agents.”transcript, stated

“All of those primitives started making more sense as and when our the the complexity of the workflow started growing”transcript, stated

Change deployment path

Chose Human approval before applying LLM-generated changes over Direct automated shipping.

Rationale: Instacart wanted human confidence plus compliance and auditability.

“we still want like a human in the process to approve it”transcript, stated

“every change that is going into your system is going through a human, it's going through a set of you know programmatic audits and compliance”transcript, stated

Eval strategy

Chose Multiple scorers and LLM judges over A single deterministic score.

Rationale: Non-deterministic agents need higher-confidence quality assessment for some use cases.

“You you do want to do some kind of non-deterministic testing and that's where this entire concept of evals came about”transcript, stated

Batch inference orchestration

Chose Child workflows with separate retry semantics over One monolithic batch job.

Rationale: Large-scale data processing needs independent failure handling for chunks.

“Each child workflow is can have its own retry semantics. If uh you know, you could use it to generate some kind of scoring”transcript, stated

Gotchas (what bit them)

LLM calls need workflow-level reliability wrappers.

Root cause: The model remains non-deterministic and can hallucinate even as quality improves.

Resolution: Wrap LLM calls as Temporal activities with retries, policies, health checks, and heartbeats.

“although it has improved over the years it is still a non-deterministic model in the back. It can hallucinate”transcript, stated

“wrap it up with like certain kind of retries certain certain policies certain health checks heartbeats and so on.”transcript, stated

LLM workflow quality cannot be treated like deterministic testing only.

Root cause: Agent output evaluation itself may require model-based and domain-specific judging.

Resolution: Use deterministic checks together with domain judges, LLM judges, and ensemble evals.

“you need to make sure that the answer that this non-deterministic API is giving is is is of a higher quality”transcript, stated

“People are talking about not just like running simple evals, they they want to do their own domain judges”transcript, stated

Human approval can take days or months, which ordinary request/response flows handle poorly.

Root cause: Approval is asynchronous and must resume the same workflow context after a long wait.

Resolution: Use Temporal signals, timers, and workflow updates to wait durably and resume after approval.

“the ability to, you know, wait for the human to approve within for days or months or so on, um are like signals.”transcript, stated

Evaluation

Instacart evaluates AI workflows with deterministic checks, domain-specific judges, LLM judges, ensemble evals, and live/offline placement depending on latency and cost requirements.

Metrics: Judge/scorer outputs · Average/min/max ensemble strategies · Latency requirements · Cost requirements · Workflow observability and audit records

“multiple evals at a time through multiple judges and multiple LLM models.”transcript, stated

“you can choose to run it as part of your live LLM conversations, or you can run it as like an offline thing, just depending on your latency requirements, your cost requirements”transcript, stated

Outcomes

Instacart generalized existing Temporal infrastructure into a pattern for production AI workflows.
Human-in-the-loop, eval, and batch inference use cases all reuse Temporal primitives.
The architecture supports auditability, observability, and durable resumption for AI changes.

Instacart runs close to 56 million Temporal workflows per day across 45 namespaces, with one namespace near 200,000 actions per second.

“this thesis exactly has kind of played out uh pretty well over the years as well as and when our our not just our scale has increased”transcript, stated

“We run close to 56 million workflows a day.”transcript, stated