Scaling AI Applications with LLMs: Dynamic Context Injection and Few-Shot Learning for Order Processing

choco · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Food supply-chain order processing

Choco needed to improve LLM-based order extraction across heterogeneous restaurant order formats, including shorthand texts, semi-structured PDFs, and voicemail transcripts, while avoiding brittle fine-tuning cycles and enabling customer-success teams to feed new examples back into the system.

Constraints:

• Order formats vary significantly by customer, distributor, channel, and layout.
• Extraction errors include wrong columns, ambiguous information, abbreviations, product identifiers, and ASR mistranscriptions.
• Voicemail transcription errors are visible to users and feed downstream extraction, leaving little room for mistakes.
• Fine-tuning is slower, less precise for targeted fixes, and schema changes require retraining.
• The system needs to keep model flexibility as newer LLMs become available.

Architecture

Choco built an in-context learning system that labels common order formats, retrieves relevant examples at inference, injects distributor and customer context, and lets customer-success teams add new labels from the order inspection tool. Embedding-based semantic retrieval supplements metadata matching, while dynamic context injection provides order history, distributor assortment, and product catalog hints for transcription correction and extraction.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Incoming orders"] --> B["Information extraction step"]
  C["Customer success labeling interface"] --> D["Labeled examples store"]
  D --> E["Example retrieval system"]
  E --> B
  F["Dynamic context injector"] --> B
  B --> G["Structured order output"]
  G --> C

Components

Data flow

Incoming text, PDF, or voicemail orders enter an information-extraction step that converts product names, quantities, and restaurant details into structured JSON. The system retrieves labeled examples by metadata and semantic similarity, injects recent order history and distributor context, and passes the enriched prompt to the LLM. Customer-success annotations and corrections become new labels, evaluation data, and future retrieval context.

Why this architecture

Choco chose prompt-based in-context learning and dynamic context injection because they support targeted, reversible improvements without retraining every time a schema or failure mode changes. Semantic retrieval handles layout similarity that metadata alone misses, and customer-success labeling turns operational corrections into reusable examples.

Models used: LLMs · ASR models · embedding models · smaller models · fine-tuned models

Scale

• The system handles multiple order modalities, including text messages, PDF attachments, emails, and voicemail orders.
  “Some restaurants place orders using shorthand text messages with abbreviations, while others rely on third-party tools that send semi-structured orders as PDF attachments via email.”article, stated
• Choco injects only the last few weeks of order history to keep repeat-order context useful.
  “Since restaurants often place repeat orders, we injected only the last few weeks of order history.”article, stated
• The labels serve four downstream purposes: fine-tuning, evaluation, new-label generation, and autonomous error judging.
  “They serve as: 1. A foundation for fine-tuning.
  2. A dataset for evaluating and refining our LLM components.”article, stated

Alternatives considered (roads not taken)

Metadata-only example matching

Rejected because: Metadata matching breaks down with layout variation, ambiguous senders, and unseen formats, so Choco added semantic retrieval.

Generic catch-all transcription correction prompt

Rejected because: The generic prompt improved some transcriptions but struggled with brand names, uncommon products, and colloquial ordering language.

Fine-tuning as the primary improvement path

Rejected because: Fine-tuning is slower, less precise, and requires retraining when output schemas change.

Patterns

context-engineering (context)

Choco dynamically injects distributor categories, prior orders, popular products, and instructions for how the LLM should use that context.

rag-retrieval (retrieval)

Choco indexes labeled examples as embeddings and retrieves similar order formats at inference so the prompt receives relevant few-shot examples.

human-in-the-loop-gate (governance)

Customer-success staff annotate orders and add labels when new formats or failure modes appear, feeding human expertise into the system.

structured-output-contract (data)

The extraction task converts heterogeneous orders into structured JSON required by downstream processing.

eval-driven-development (evaluation)

Choco treats labeled examples as evaluation data and uses operational labels/corrections to refine LLM components.

Tradeoffs

Prompt-based improvement versus fine-tuning first

Chose in-context learning and dynamic context injection first over fine-tuning as the primary improvement mechanism.

Rationale: Prompt methods let Choco make targeted changes, keep control over inserted context, avoid regressions, and switch models as newer LLMs appear.

Example retrieval strategy

Chose semantic retrieval over labeled example embeddings over metadata-based matching alone.

Rationale: Embedding retrieval can match structurally similar orders even when metadata or sender information differs.

Human expertise in the loop

Chose customer-success annotation interfaces over ML-only ownership of new labels.

Rationale: Customer-success teams can label real order formats and failure modes directly, reducing the need for deep ML expertise.

Gotchas (what bit them)

Few-shot quality depends on retrieving the right examples.

Root cause: Metadata matching alone misses layout variations, ambiguous senders, and unseen formats.

Resolution: Choco added embedding-based semantic retrieval over labeled examples.

Generic prompts fail on domain-specific product names.

Root cause: Restaurant orders use uncommon products, brand names, and colloquial terms such as bottle-cap color names.

Resolution: Inject distributor assortment, recent order history, popular products, and guidance on how to use the context.

Fine-tuning slows targeted fixes and schema changes.

Root cause: Adding training examples does not guarantee the exact generalization needed, and output schema changes require retraining.

Resolution: Prioritize prompt-based in-context methods for fast, surgical improvements while keeping fine-tuning complementary.

Evaluation

Choco evaluates and improves components through labeled examples, customer-success annotations, corrections from failure modes, semantic retrieval feasibility tests, and datasets that support evaluation and refinement of LLM components.

Metrics: Information extraction accuracy · Similarity threshold for retrieved examples · New order formats and failure modes labeled · Mistranscription correction quality · Fine-tuning and evaluation dataset growth

Outcomes

• Choco built a dynamic few-shot prompting system for order extraction.
• Customer-success teams can annotate orders directly in the inspection tool.
• Embedding retrieval can select structurally similar labeled examples at inference.
• Dynamic context injection improved handling of voicemail mistranscriptions and product-name ambiguity.
• Prompt-based methods allowed faster iteration and model flexibility while preserving fine-tuning as a complementary path.

The article does not disclose order volume, but it describes production handling across multiple order modalities, distributor/customer-specific formats, recent-weeks order history, and reusable labels for fine-tuning, evaluation, new-label creation, and autonomous error judging.