Large-Scale LLM Batch Processing Platform for Millions of Prompts

instacart · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale LLM batch processing infrastructure

Instacart needed a shared way for internal teams to run millions of LLM prompts for catalog cleanup, item enrichment, fulfillment routing, search relevance, and ML training workflows. Real-time provider APIs caused rate limits, throttling, delays, duplicated team-specific pipelines, fragmented solutions, and high cost. The platform had to batch requests, manage files, track costs, handle provider failure modes, support batch and real-time providers, and let teams process massive prompt jobs without becoming LLM infrastructure experts.

Constraints:

• many workflows require millions of LLM calls
• provider batch files are limited to 50,000 prompts or 200MB
• large jobs with one million prompts require at least 20 separate batches
• provider batch workflows require encoding, upload, polling, download, parsing, and retrying failed prompts
• real-time APIs are rate-limited and some providers do not offer batch interfaces
• large files created storage, memory, and processing limitations
• jobs must resume after exceptions without losing work or wasting money on partial completion

Architecture

Instacart built Maple, a reusable service that accepts CSV or Parquet input plus a prompt and returns an output file merged with AI responses. Maple orchestrates batching, provider-format encoding and decoding, file upload/download, job polling, retry handling, result merging, cost tracking, and progress tracking. It sits between internal applications and external model providers, proxies requests through Instacart's AI Gateway, logs cost and usage through Cost Tracker, stores inputs/outputs on S3, uses Parquet and PyArrow for efficient large-file processing, and relies on Temporal for durable execution. For providers without batch APIs, Maple uses the same interface while adding parallelization, exponential backoff, retry policies, and failure tracking over real-time APIs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  workflows["Internal AI workflows"] --> maple["Maple service"]
  maple --> rpc["RPC API"]
  maple --> temporal["Temporal workflows"]
  maple --> storage["S3 Parquet storage"]
  storage --> pyarrow["PyArrow processing"]
  maple --> gateway["AI Gateway"]
  gateway --> tracker["Cost Tracker"]
  gateway --> providers["External LLM providers"]
  providers --> failures["Failure handling policies"]
  failures --> temporal
  providers --> gateway
  gateway --> maple
  maple --> storage

Components

Data flow

Internal applications submit a CSV or Parquet file and prompt to Maple. Maple splits large inputs into smaller Parquet batches on S3, encodes each batch into provider-specific request files, uploads them through the AI Gateway, polls provider jobs, downloads results to S3, matches responses back to inputs, creates per-batch Parquet result files, and merges them into a final output. Cost Tracker records spend by job and team. Temporal preserves long-running job state so failures can resume from the last completed step. Failed task classes such as expired, rate-limited, refused, or invalid-image requests flow into tailored retry policies.

Why this architecture

Maple centralizes LLM bulk processing because repeated team-owned scripts were creating duplicated effort and inconsistent operations. Batch processing is cheaper and better suited to million-prompt workloads than real-time APIs, but provider batch interfaces are operationally complex. Temporal is used because long-running jobs need durable resume semantics. S3 and Parquet replace database task storage because large files need cheaper storage, compression, faster load/save, and non-linear access.

Models used: LLM provider batch APIs · LLM provider real-time APIs · AI Gateway · Temporal · PyArrow · Parquet · orjson

Scale

• Instacart uses LLMs across millions of items, customers, and deliveries.
  “a grocery platform that cuts across millions of items, customers, and deliveries”article, stated
• Many workloads require millions of LLM calls.
  “Many of these tasks require millions of LLM calls”article, stated
• Provider batch limits are 50,000 prompts or 200MB.
  “Each batch is limited to 50,000 prompts or 200MB”article, stated
• A one-million-prompt job requires at least 20 separate batches.
  “large jobs, like those with a million prompts, require at least 20 separate batches.”article, stated
• Parquet compression reduces file size up to 25x compared to CSV.
  “out-of-the-box compression reduces file sizes up to 25x compared to CSV.”article, stated
• The performance sample covered about 580 batches with 40-50K tasks per batch.
  “stats from a sample set of ~580 batches with 40–50K tasks per batch”article, stated
• LLM batches averaged 2.6 tasks per second in the sample.
  “LLM batches average 2.6 tasks per second.”article, stated
• Most batches finish in under 12 hours, with some almost a full day.
  “most batches complete in under 12 hours, with occasional batches that take almost a full day”article, stated
• Maple scaled to 10M+ prompt jobs.
  “These optimizations allowed Maple to scale efficiently to 10M+ prompt jobs.”article, stated
• Maple saves up to 50% on LLM costs compared with standard real-time calls.
  “saving up to 50% on LLM costs compared to standard real-time calls”article, stated

Alternatives considered (roads not taken)

Use real-time LLM APIs for million-prompt workloads

Rejected because: Real-time APIs were rate-limited, required throttling, introduced delays, and were not designed for that scale.

Let each team build its own LLM batch workflow

Rejected because: That produced duplicated effort, fragmented solutions, and repeated implementation of provider batch mechanics.

Store task data in the database

Rejected because: Database storage became inefficient at scale, so Maple moved task data to S3 Parquet files.

Only support providers with batch APIs

Rejected because: Instacart needed the same simple interface for providers that only expose real-time APIs.

Patterns

batch-offline-processing (processing)

Maple turns large CSV/Parquet prompt sets into provider batch files, processes millions of prompts asynchronously, and merges results back into output files.

durable-execution (reliability)

Temporal lets long-running LLM batch jobs resume exactly where they left off after exceptions or provider failures.

cost-aware-routing (cost)

Maple routes through batch APIs when available, tracks spend per job and team, and saves up to 50% relative to real-time calls.

fallback-degradation (failure-handling)

Maple uses tailored retry policies for expired, rate-limited, refused, and invalid-image failures, including provider-specific real-time fallback handling.

observability-tracing (operations)

Maple exposes job progress, AI Gateway cost logging, per-job/team spend, and failure tracking so bulk prompt jobs remain traceable.

tool-use-orchestration (platform)

Maple orchestrates provider batch APIs, real-time APIs, AI Gateway, Cost Tracker, S3, Parquet, PyArrow, Temporal, retries, and result merging behind one interface.

Tradeoffs

LLM processing mode

Chose Batch-first processing through Maple over Direct real-time API calls for bulk workloads.

Rationale: Batch APIs fit million-prompt workloads better and can reduce LLM costs by up to 50%, while real-time calls hit rate limits and throttling.

Storage layer

Chose S3-backed Parquet files over Database-backed task storage.

Rationale: S3 and Parquet reduce database operations, improve load/save speed, compress files, and support fast access for huge datasets.

Failure handling policy

Chose Failure-class-specific retries over One generic retry policy for every provider error.

Rationale: Expired and rate-limited tasks can retry indefinitely, while refused tasks often repeat and are capped at two retries.

Provider abstraction

Chose Same Maple interface for batch and real-time providers over Require teams to adopt different workflows per provider.

Rationale: Teams can keep CSV input/output workflows while Maple handles real-time parallelization, backoff, retries, and future migration to batch interfaces.

Gotchas (what bit them)

Provider batch APIs are cheaper but operationally involved.

Root cause: Batch workflows require encoding, upload, monitoring, download, parsing, and retrying failed prompts.

Resolution: Centralize those mechanics in Maple instead of making each team implement them.

Large prompt files can turn database task storage into a bottleneck.

Root cause: High-volume jobs hit storage, memory, and processing limits as internal clients sent larger inputs.

Resolution: Move task data to S3 Parquet files, stream processing, and faster JSON handling with orjson.

Failures are inevitable in tens-of-millions prompt processing.

Root cause: Network issues, provider failures, bugs, expiration, rate limits, refusals, and invalid images can occur mid-run.

Resolution: Use Temporal for resume semantics and separate retry policies by failure class.

Invalid images create a retry-cost tradeoff.

Root cause: Checking every image in a large batch adds significant overhead, but bad URLs can cause request failures.

Resolution: Retry invalid-image tasks optionally, and only check image existence on the second attempt.

Provider interface differences can leak complexity to product teams.

Root cause: Some providers expose batch APIs and others only expose real-time APIs.

Resolution: Hide provider differences behind the same Maple CSV input/output interface and switch providers transparently when batch support appears.

Evaluation

Instacart evaluates Maple through operational throughput and completion metrics, cost tracking, progress tracking, provider-failure handling, and real-world batch statistics collected from hundreds of large batches.

Metrics: prompt processing speed · batch completion time · job size versus completion time · cost usage per job and team · up to 50% LLM cost savings · ~580-batch sample · 40-50K tasks per batch · 2.6 average tasks per second · most batches complete under 12 hours · 10M+ prompt job support

Outcomes

• Maple became part of Instacart's AI infrastructure backbone and a universal bulk LLM provider for multiple teams.
• Teams can process millions of prompts without writing custom batch-processing infrastructure.
• Maple reports up to 50% savings on LLM costs versus standard real-time calls.
• Some processes dropped from hundreds of thousands of dollars per year to thousands of dollars per year.
• Optimizations with S3, Parquet, streaming, and orjson let Maple scale to 10M+ prompt jobs.
• The platform accelerates experimentation and lets teams ship LLM-powered features faster.

The source reports millions of items/customers/deliveries, million-prompt workflows, 50,000-prompt or 200MB batch limits, at least 20 batches for a million-prompt job, 25x Parquet compression compared with CSV, ~580 observed batches of 40-50K tasks, 2.6 tasks/sec average throughput, most batches under 12 hours, 10M+ prompt jobs, up to 50% LLM cost savings, and annual process costs reduced from hundreds of thousands to thousands.