Building Production LLM Applications with DSPy Framework

alixpartners · Consulting · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production LLM application development with DSPy programs, document workflows, tool calling, and prompt optimization

AlixPartners needed a way to build many different LLM applications as maintainable Python programs rather than hand-tuned prompt strings, while preserving structured outputs, tool use, model transferability, observability, and optimization across consulting use cases such as contract analysis, file routing, multimodal document processing, web research, and time-entry correction.

Constraints:

• Consulting teams encounter varied domains, so the application structure must transfer across legal investigations, process improvement, contract review, and internal AI deployments.
• Applications need typed inputs and outputs, but the framework also has to remain flexible enough for exploratory document analysis and multimodal workloads.
• Model capabilities and model prices change quickly, so the program logic must survive movement between large, small, and multimodal models.
• Prompt optimization only helps when teams define useful metrics and have enough representative examples.
• Productionizing workshop prototypes requires stronger classification, monitoring, and guardrails than the quick examples demonstrate.

Architecture

The architecture centers LLM applications as DSPy programs. Signatures declare typed inputs and outputs; modules compose signatures with Python business logic; tools expose ordinary Python functions to React-style agents; adapters translate signatures into model-facing prompt formats; model configuration and context managers route calls to different LLMs; caching and LiteLLM usage tracking reduce iteration cost; Phoenix and inspect-history tracing expose prompts, responses, tool trajectories, and token usage; and optimizers use metrics and datasets to compile improved DSPy modules. The workshop applies those primitives to sentiment classification, SEC Form 4 processing, contract summarization and boundary detection, multimodal file classification, web-research agents, and time-entry correction.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  application-use-case[Application use case] --> dspy-signature[DSPy signature]
  dspy-signature --> dspy-module[DSPy module]
  dspy-signature --> adapter[Adapter]
  dspy-module --> tool-functions[Tool functions]
  dspy-module --> model-configuration[Model configuration]
  attachments-processor[Attachments processor] --> dspy-signature
  adapter --> llm-runtime[LLM runtime]
  model-configuration --> llm-runtime
  llm-runtime --> cache[Cache]
  llm-runtime --> observability-stack[Observability stack]
  tool-functions --> trajectory-inspector[Trajectory inspector]
  observability-stack --> trajectory-inspector
  observability-stack --> evaluation-metrics[Evaluation metrics]
  evaluation-metrics --> optimizer[Optimizer]
  optimizer --> dspy-module

Components

Data flow

A developer expresses the task as a DSPy signature, optionally backed by Pydantic types and field descriptions. A module wraps the signature and any deterministic business logic, then invokes a DSPy primitive such as Predict, ChainOfThought, or React. Adapters render the signature and runtime inputs into the prompt format required by the selected model. Tools are supplied as Python functions when the React module needs external actions. Inputs such as PDFs or images are normalized through Attachments before entering the signature. DSPy dispatches model calls through LiteLLM, records usage, caches unchanged calls, and exposes execution history and trajectories for debugging. For optimization runs, examples and metrics drive algorithms such as MIPRO/JEPA to produce a compiled module that can be saved, loaded, and reused.

Why this architecture

The framework-based architecture was chosen because the presenter wants stable program structure, explicit typed contracts, model interchangeability, and quantitative optimization rather than ad hoc prompt editing. DSPy's level of abstraction lets developers keep Python control flow and business logic visible while still letting LLMs fill the semantic steps.

Models used: DSPy · LiteLLM · GPT-4.1 · GPT-4.1 nano · GPT-4.1 mini · Gemini 2.5 Flash · Phoenix · Attachments · Pydantic · MIPRO optimizer · JEPA optimizer

Scale

• One motivating legal use case involved searching 10,000 contracts for a clause or paragraph.
  “Maybe we need to look through 10 10,000 contracts to identify a particular clause um or or paragraph.”transcript, stated
• The React tool-calling example bounded the agent loop to five rounds.
  “I'm giving it the get weather and the search web tools and I'm saying, okay, do your thing, but only go five rounds”transcript, stated
• The time-entry correction use case involved hundreds of thousands of entries.
  “they had hundreds of thousands of time entries and they needed to make sure that they all adhere to the same format.”transcript, stated
• The optimization example uses 10 to 100 input/output examples as an indicated practical range.
  “you don't need thousands and thousands of examples necessarily, but as long as you have enough, maybe 10 to 100 of inputs and outputs.”transcript, stated
• In one transfer example, a smaller model dropped to about 70% before optimization recovered roughly 87%.
  “Can I try it with 41 nano? Okay, maybe it's at 70% whatever it might be. But I run the optimizer on 41 nano and I can get the performance back up to maybe 87%.”transcript, stated
• A concrete MIPRO optimization example improved a time-entry model from 86 to 89.
  “You can see the actual performance on um, the basic versus the optimized model. In this case, went from 86 to 89.”transcript, stated

Alternatives considered (roads not taken)

Ad hoc prompt editing instead of a DSPy program

Rejected because: The presenter explicitly frames the goal as building real Python programs with LLM calls, not iterating on prompt strings.

Using LangChain as the primary abstraction

Rejected because: The presenter says DSPy sits at a level of abstraction that gets in the way less for his use cases, while acknowledging other libraries can still be good choices.

Fine-tuning before prompt optimization

Rejected because: The talk recommends trying DSPy optimizers first because they can improve performance without fine-tuning infrastructure.

Always using one model for all workloads

Rejected because: The file-processing example routes images to Gemini and other work to GPT-style models based on workload fit.

A fully autonomous file classifier for production

Rejected because: The presenter marks the workshop example as quick-and-dirty and says a production use case would need stronger or multiple classification passes.

Patterns

structured-output-contract (interface-contract)

DSPy signatures and Pydantic output types define the semantic contract for model calls, including field names, field descriptions, typed outputs, and literal choices.

tool-use-orchestration (agent-control)

DSPy React modules expose Python functions as tools, cap the number of rounds, and expose trajectories so developers can inspect tool arguments and observations.

model-routing (runtime-routing)

The file-processing program defines regular, small, and image-oriented model configurations, then routes specific workloads such as image recognition or Form 4 extraction to different models.

semantic-caching (cost-control)

DSPy automatically reuses results when signatures and inputs have not changed, speeding local testing and avoiding unnecessary live calls.

observability-tracing (operations)

Phoenix, inspect-history, token usage, and trajectory inspection provide visibility into prompts, responses, model usage, tool calls, and optimizer behavior.

eval-driven-development (optimization)

The optimizer workflow depends on explicit metrics and example data, then reports metric-level performance so teams can decide whether to adjust the metric or improve the program/data.

cost-aware-routing (runtime-economics)

The talk uses DSPy optimization and model transfer as a cost-control tactic: move work to a smaller model, recover quality through optimization, and keep the program structure stable.

fine-tuning-adaptation (model-adaptation)

The talk positions prompt optimization as a lighter-weight adaptation path to try before fine-tuning, especially when teams want performance gains without fine-tuning infrastructure.

Tradeoffs

Represent LLM behavior as DSPy signatures and modules

Chose Declarative typed programs with composable modules over Hand-written prompt strings and direct response parsing.

Rationale: The presenter values program structure, typed input/output guarantees, and transferability across model changes more than low-level control over every prompt string.

Use adapters to vary prompt format

Chose Swappable JSON/BAML/XML-style adapters over One fixed prompt rendering format.

Rationale: Adapters let the same program experiment with more readable or token-efficient formats that may improve performance without rewriting the business logic.

Try prompt optimization before fine-tuning

Chose DSPy optimizers and metrics over Immediately setting up fine-tuning infrastructure.

Rationale: Optimization can recover acceptable performance on cheaper models while avoiding the operational overhead of fine-tuning.

Route workloads to specialized models

Chose Different model configurations for regular, small, and image-heavy work over A single global model configuration for all tasks.

Rationale: The examples use smaller models for some structured work and Gemini for image recognition, suggesting model choice should follow task requirements.

Mix LLM calls with deterministic Python business logic

Chose Custom modules that interleave model predictions and hard-coded rules over Pure prompt-only applications.

Rationale: DSPy modules can encapsulate LLM calls while still allowing normal Python control flow and business rules where determinism is appropriate.

Gotchas (what bit them)

DSPy can look deceptively simple while hiding prompt-shaping details inside field names and adapters.

Root cause: The same typed Python fields that make signatures ergonomic also become part of the prompt, so naming and descriptions still matter.

Resolution: Treat signatures as both code contracts and prompt surfaces; inspect the generated prompt history when behavior is surprising.

Workshop-grade examples are not automatically production-grade systems.

Root cause: The examples are deliberately quick demonstrations with small category sets and obvious files.

Resolution: Add more stringent or multi-pass classification and production checks before applying the same pipeline to ambiguous real-world inputs.

Optimization quality depends on examples and metrics, not on the optimizer alone.

Root cause: Optimizers optimize against the provided metric and dataset; poor metrics or low-quality data can steer the compiled prompt toward the wrong behavior.

Resolution: Invest in well-structured input/output examples and inspect metric-level breakdowns to decide whether metrics should be decomposed or changed.

Tool-calling agents can spin beyond the intended scope without explicit bounds.

Root cause: React-style agents are given tools and allowed to act, so the developer needs to set operational limits.

Resolution: Bound the loop explicitly, as in the example that allows only five rounds, and inspect trajectories for unexpected tool use.

Framework abstraction can obscure whether a problem really needs DSPy or a simpler structured-output call.

Root cause: DSPy does enforce structured output, but the presenter notes that calling it only structured outputs is reductive and that alternatives can solve similar tasks.

Resolution: Use DSPy when composability, model routing, optimization, tool orchestration, and program structure matter; simpler structured-output APIs may suffice for narrow calls.

Evaluation

The workshop evaluates DSPy programs by defining task-specific metrics, running optimizers against example inputs/outputs, inspecting before/after performance, breaking results down by metric, and using observability tools such as inspect-history, token usage, Phoenix traces, and React trajectories to understand behavior.

Metrics: Optimization performance before and after MIPRO/JEPA runs · Metric-level breakdowns for capitalization, punctuation, and formatting requirements in the time-entry example · Tool trajectory inspection for React calls · Prompt/response inspection through DSPy inspect-history · Token usage tracking from LiteLLM · Cache hit behavior during repeated development runs

Outcomes

• DSPy let the presenter and team rapidly iterate on multiple LLM applications as Python programs.
• The file-processing examples routed SEC filings, contracts, patents, city infrastructure images, and other inputs to different processors.
• The contract example detected main-document and schedule boundaries across a 13-page PDF.
• The time-entry optimizer improved the demonstrated metric from 86 to 89.
• The model-transfer example describes recovering a smaller model from about 70% to about 87% while reducing cost by multiple orders of magnitude.
• Tracing, cache behavior, usage tracking, and trajectory inspection gave developers visibility into model calls and tool use.

The talk is a workshop rather than a deployed product case, but it contains concrete scale signals: 10,000-contract search, hundreds of thousands of time entries, five-round tool-call bounds, 10-100 optimization examples, a 13-page contract boundary demo, and optimization results from 86 to 89 or 70% to 87%.