Advanced Prompt Engineering Techniques for Production LLM Applications

instacart · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production prompt engineering

Instacart needed prompt techniques that made LLM outputs more reliable for internal productivity tools.

Constraints:

• Hallucinations
• Inconsistent formats
• Context limits
• Model-specific behavior

Architecture

Instacart describes an internal productivity-tooling pattern library for GPT-4/GPT-3.5 applications rather than one monolithic app. The architecture wraps an internal OpenAI/LLM proxy and task workflows with prompt techniques: ReAct delegates tool calls to application software, Room for Thought decomposes generation into intermediate steps, Monte Carlo samples several options before finalizing, self-correction asks the model to critique its own output, a classifier endpoint guarantees valid outputs with logit bias, and puppetry pre-fills response form. The system is a production prompt-control layer around internal tools.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  internal-productivity-tools[Internal productivity tools] --> prompt-technique-library[Prompt technique library]
  internal-productivity-tools --> internal-llm-proxy[Internal LLM proxy]
  prompt-technique-library --> gpt-models[GPT models]
  internal-llm-proxy --> gpt-models
  prompt-technique-library --> react-action-executor[ReAct action executor]
  react-action-executor --> internal-productivity-tools
  prompt-technique-library --> draft-synthesis-loop[Draft synthesis loop]
  draft-synthesis-loop --> gpt-models
  internal-llm-proxy --> classifier-endpoint[Classifier endpoint]
  classifier-endpoint --> gpt-models
  few-shot-examples[Few-shot examples] --> prompt-technique-library

Components

Data flow

Internal productivity workflows send task prompts through Instacart's LLM proxy to GPT-4 or GPT-3.5. Depending on the task, the prompt may request an outline before final generation, ask the model to call tools that application software executes and feeds back, sample multiple drafts and synthesize a final answer, critique and revise its own output, or route through a classifier endpoint that guarantees valid output.

Why this architecture

Instacart adopted a library of prompt-level controls because production LLM work failed in multiple small ways: context limits, hallucinations, task failure, invalid formats, and weak first drafts. Different workflows needed different control tactics, so the architecture keeps prompt techniques composable around an internal proxy and software actions.

Models used: GPT-4 · GPT-3.5

Scale

• The techniques were used in internal productivity tooling rather than described as a single public product.
  “This article will be an exploration of prompt techniques we’ve used for our internal productivity tooling at Instacart. These techniques are a combination of industry and academic research along with our own small-scale internal development efforts.”article, stated
• The techniques were implemented with GPT-4 and some with GPT-3.5.
  “all of these prompt techniques have been implemented and used with GPT-4. Some have also been used with GPT-3.5.”article, stated

Alternatives considered (roads not taken)

Ask the model for the final artifact in one pass

Rejected because: Room for Thought and Monte Carlo split intermediate reasoning or options before producing the final answer.

Let the model freely choose classifier labels

Rejected because: The internal proxy endpoint guarantees valid output and can use logit bias to nudge valid labels.

Require the model to start in exactly the desired format unaided

Rejected because: Puppetry reduces formatting burden by telling the assistant it has already started responding.

Patterns

prompt-chaining (prompting)

Instacart uses multi-step prompt structures that let the model reason, act, and self-correct.

tool-use-orchestration (prompting)

The ReAct-style prompt lets software execute external actions and resume the model task.

Tradeoffs

How to improve generated writing quality.

Chose intermediate outlines, multiple samples, and self-correction over single-pass final generation.

Rationale: Instacart found that forcing intermediate structure or sampling several options improved the final output for complex writing tasks.

How to keep classifier outputs valid.

Chose a proxy endpoint with constrained valid output and logit bias over unconstrained natural-language classification.

Rationale: Production code benefits from guaranteed label shape, while logit bias nudges ambiguous choices toward allowed outputs.

Gotchas (what bit them)

LLMs need prompt scaffolding to reduce hallucination and format issues.

Root cause: The models are powerful but constrained by context and consistency challenges.

Resolution: The team used named prompt techniques rather than one-shot prompting.

Evaluation

The article is a technique writeup and does not provide formal production metrics.

Outcomes

• The techniques were deployed in Instacart's internal Ava productivity tooling.

The article says the techniques were implemented and used with GPT-4.