Semantic Data Processing at Scale with AI-Powered Query Optimization

docetl · Research & Academia · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Semantic data processing over unstructured documents

DocETL addresses the problem of running natural-language semantic operators over large unstructured corpora while controlling cost, accuracy, latency, and user steerability.

Constraints:

• Unstructured data such as transcripts and reports requires reasoning that traditional relational systems do not handle directly.
• Running state-of-the-art LLMs over long documents and large corpora can be prohibitively expensive.
• Semantic operator plans vary in accuracy as well as cost and latency, unlike traditional database plans.
• Users need interfaces to understand data, refine specifications, and catch scale failures.

Architecture

DocETL represents text workloads as semantic operators such as map, reduce, filter, and join; rewrites them using a registry of rewrite directives instantiated by LLM agents; searches candidate plans with cost, accuracy, and latency estimates; applies task cascades and decomposition rewrites to reduce expensive model calls; and pairs the optimizer with Doc Wrangler, an interface that helps users inspect outputs, convert notes into better specifications, and trigger decomposition when complex prompts fail.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Semantic operator pipeline"] --> B["Plan search optimizer"]
  B --> C["Rewrite directive registry"]
  C --> D["LLM rewrite agent"]
  D --> E["Candidate query plans"]
  E --> F["Sample evaluation harness"]
  F --> B
  A --> G["Doc Wrangler interface"]
  G --> H["Expensive LLM judge"]
  H --> A

Components

Data flow

Users author semantic-operator pipelines over document collections, the optimizer selects a pipeline to rewrite, LLM agents instantiate rewrite directives with prompts, schemas, chunk sizes, or synthesized operators, candidate plans are sampled for cost and accuracy estimates, and Doc Wrangler lets users inspect outputs, record feedback, revise specifications, and decompose complex operations before running at corpus scale.

Why this architecture

DocETL borrows database optimizer ideas because semantic pipelines need plan search, but it extends them with LLM-instantiated rewrites and accuracy modeling because LLM plans vary in quality as well as cost.

Models used: LLM agents · LLM judges · proxy models · state-of-the-art LLMs

Scale

• Target workloads include tens of thousands of transcripts with hundreds of pages per transcript.
  “if you're going to run this on tens of thousands of transcripts with each transcript hundreds of pages long”transcript, stated
• DocETL reports task-cascade rewrites reducing cost by 86% on average while staying within a 90% target accuracy.
  “we're able to reduce cost compared to the pipeline before rewriting to 86% on average, which is awesome. Staying within a 90% target accuracy.”transcript, stated
• Doc Wrangler users can accumulate 25 to 30 feedback notes while inspecting examples.
  “in practice, people can go up to like we've seen 25 30 notes. People just really look at their examples, provide feedback.”transcript, stated
• DocETL's optimizer can produce plans with up to 2x the accuracy of baseline systems in the reported workloads.
  “often it can be 2x times the accuracy of what the other systems can find.”transcript, stated

Alternatives considered (roads not taken)

Run the original semantic operator plan with the most expensive model

Rejected because: The original plan is treated as the Oracle fallback but is too expensive to apply to every document when cheaper proxy or decomposed tasks can resolve many items.

Let a general GPT agent search for the best pipeline without DocETL rewrite directives

Rejected because: The reported GPT5 agent baseline had access to the query engine and documentation, but DocETL's designed rewrite directives found more accurate and cheaper plans.

Ask users to manually specify perfect prompts up front

Rejected because: Doc Wrangler assumes users discover requirements while inspecting data and therefore need assisted specification.

Patterns

query-rewriting (optimization)

DocETL rewrites semantic operator pipelines using rewrite directives for decomposition, fusion, cascades, and operator substitution.

tool-use-orchestration (agent-architecture)

LLM agents are used as tooling inside the optimizer to instantiate prompts, schemas, and rewrite choices for new semantic operators.

cost-aware-routing (optimization)

Task cascades route inputs through cheaper proxy tasks first and only escalate uncertain cases to the original expensive plan.

llm-as-judge (evaluation)

Doc Wrangler runs an expensive LLM judge on samples to identify inaccurate attributes and suggest decomposition.

Tradeoffs

Use task cascades before the Oracle plan.

Chose cheap proxy tasks followed by escalation to the original plan over running the expensive original plan for every document.

Rationale: Task cascades reduce cost when proxy confidence can preserve target accuracy.

Spend money on rewrite exploration.

Chose instantiate and evaluate candidate rewrite directives over avoid exploration cost and keep the initial pipeline.

Rationale: Some rewrites can guarantee accuracy probabilistically while reducing full-run cost enough to justify exploration.

Use an interactive UI as part of the system.

Chose Doc Wrangler output inspection, feedback, and assisted prompt updates over pure backend optimization without user steering.

Rationale: Users discover missing requirements by inspecting data, so steering tools are part of production quality.

Gotchas (what bit them)

Semantic operators can become too expensive at corpus scale.

Root cause: Naively running state-of-the-art models over tens of thousands of long transcripts is costly.

Resolution: Search rewritten plans, use task cascades, and route confident cases through cheaper proxy tasks.

Traditional optimizer assumptions break for LLM pipelines.

Root cause: Semantic plans vary not only by cost but by output accuracy and quality.

Resolution: Estimate cost, accuracy, and latency for candidate semantic plans instead of optimizing cost alone.

Prompts that work on small samples can break at full corpus scale.

Root cause: Users may overpack multiple attributes into one LLM call or miss edge cases in the data.

Resolution: Use Doc Wrangler feedback, LLM judging, and decomposition suggestions to split complex operations.

Evaluation

DocETL evaluates candidate plans by sampling for cost and accuracy, comparing against baselines such as Lotus, Palyst/Abacus, and a GPT agent baseline, and using LLM judges in Doc Wrangler to detect inaccurate operation outputs.

Metrics: cost · accuracy · latency · target accuracy · Pareto frontier · sample pass rates

Outcomes

• Task-cascade rewrites reduced cost by 86% on average within a 90% target accuracy.
• DocETL's optimizer reportedly found the most accurate plan across the evaluated real workloads.
• Some DocETL plans reached up to 2x the accuracy of baseline semantic-operator systems.
• Doc Wrangler supported interactive specification and decomposition for users working with document pipelines.

The motivating workloads include tens of thousands of long transcripts and real document-processing users such as public defenders and medical analysts.