Pitfalls and Best Practices for Production LLM Applications

humanloop · Tech · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production LLM application development and LLMOps platform practice

Humanloop observed that teams building LLM applications often treat prompts, retrieval choices, model selection, evaluation, and production feedback less rigorously than traditional software artifacts, which causes teams to abandon viable use cases, thrash between approaches, or deploy costly and hard-to-monitor systems.

Constraints:

• LLM applications combine normal software with model, prompt, and data-selection decisions that all affect quality.
• Prompt changes can cause large performance differences and require experimentation to assess.
• Outputs are subjective and non-deterministic, making evaluation harder than traditional software testing.
• Production teams need feedback and monitoring before they can safely iterate without regressions.
• Prompt artifacts must be accessible to domain experts as well as engineers.
• Large models can be too costly or too slow for some production settings.
• Fine-tuned models can create unresolved access-control risks when training data is private to a subset of users.

Architecture

Humanloop frames a production LLM app as traditional application software wrapped around an LLM block: a base model, a prompt template, and a data-selection or retrieval strategy. The recommended operating system around that block includes prompt-management infrastructure, structured evaluation during development and production, user-feedback capture, regression checks before prompt/model changes, and selective fine-tuning when prompt engineering and retrieval have reached their limits. For sensitive fine-tuned data, the talk treats access isolation and adapter-style separation as an open design area rather than a solved pattern.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Traditional_application_shell["Traditional application shell"] --> LLM_block["LLM block"]
  LLM_block --> Base_or_fine_tuned_model["Base or fine-tuned model"]
  LLM_block --> Prompt_template["Prompt template"]
  LLM_block --> Context_selection_strategy["Context selection strategy"]
  Context_selection_strategy --> Prompt_template
  Prompt_template --> Prompt_management_system["Prompt management system"]
  Prompt_management_system --> Evaluation_and_monitoring_loop["Evaluation and monitoring loop"]
  Evaluation_and_monitoring_loop --> Production_feedback_capture["Production feedback capture"]
  Production_feedback_capture --> Evaluation_and_monitoring_loop
  Evaluation_and_monitoring_loop --> Fine_tuning_data_flywheel["Fine-tuning data flywheel"]
  Fine_tuning_data_flywheel --> Base_or_fine_tuned_model

Components

Data flow

A user request enters traditional application code; the app selects relevant context through retrieval or another selection strategy; the selected data is inserted into a prompt template; the configured base or fine-tuned model generates an output; production feedback, implicit actions, corrections, and logs feed evaluation and monitoring; successful feedback can later be filtered into fine-tuning data for task-specific models.

Why this architecture

The architecture separates the pieces that are easy to conflate: model choice, prompt design, retrieval/context selection, feedback capture, evaluation, and deployment control. Humanloop argues that teams succeed when they measure these parts explicitly, preserve prompt and experiment history, start simple with strong models and prompts, and only introduce chains, agents, or fine-tuning after evidence shows they are needed.

Models used: GPT-4 · Claude · Anthropic models · OpenAI models · GPT-3.5 · custom fine-tuned open source models · Flan-T5 · Flan-UL2 · Falcon

Scale

• Humanloop had been operating its LLM developer platform for over a year at the time of the talk.
  “human Loop is a developer platform that makes it easier for people to build reliable applications on top of large language models we've been doing this for over a year now”transcript, stated
• Humanloop had observed several hundred LLM projects succeed or fail in production.
  “we've seen you know several hundred projects both succeed and not succeed in production”transcript, stated
• GitHub Copilot is described as having well over a million users.
  “GitHub co-pilot you know most people are familiar with coding assistant one of the most successful applications of llms out there well over a million users”transcript, stated
• The talk cites three feedback types as useful production signals: votes, actions, and corrections.
  “capture these three types of feedback uh votes are like the simplest one that we you know you'll see in applications things like thumbs up thumbs down but also actions”transcript, stated
• Humanloop reports that hundreds of annotated data points can be enough to start fine-tuning and thousands can produce strong performance.
  “sometimes hundreds of data points is enough for them to get started and thousands of data points can get you really good performance”transcript, stated
• One customer saw a smaller fine-tuned model outperform GPT-4 on its task using about 3,500 data points.
  “when they did it outperforms gpd4 a much smaller model for their tasks quite quickly I think they did three and a half thousand data points”transcript, stated

Alternatives considered (roads not taken)

Eyeball prompt behavior in playgrounds without objective evaluation

Rejected because: Humanloop says this causes teams to give up too early or thrash among retrieval systems, chains, agents, and prompts without knowing whether quality is improving.

Store prompts only in ad hoc documents, spreadsheets, notebooks, or code

Rejected because: Ad hoc stores lose experiment history, while code-only prompt management creates friction for domain experts who need to collaborate on prompt design.

Start production design with complicated chains or agents

Rejected because: Humanloop recommends exhausting prompt engineering and strong base models first, because early chains and agents make failures harder to isolate and maintain.

Rely indefinitely on the largest hosted model instead of task-specific fine-tuning

Rejected because: For narrow tasks, Humanloop says smaller fine-tuned models can reduce cost and latency while using proprietary feedback data to improve task performance.

Share one fine-tuned model across users with different training-data permissions

Rejected because: The speaker treats cross-permission fine-tuned model sharing as unsolved and suggests separate models or adapter-style isolation as possible mitigations.

Patterns

eval-driven-development (evaluation)

Humanloop recommends making objective evaluation and regression checks the organizing principle for prompt iteration, production monitoring, retrieval quality, and model changes.

prompt-versioning (prompt-ops)

The talk treats prompt templates as production artifacts that need experiment history, model configuration, access control, collaboration, and deployment management.

observability-tracing (operations)

Production feedback, logs, telemetry, implicit actions, and corrections are treated as operating signals for monitoring quality and improving the system.

rag-retrieval (retrieval)

The talk identifies retrieval-augmented generation as a common, legitimate LLM chain where query-relevant context is selected before generation.

fine-tuning-adaptation (model-adaptation)

Humanloop recommends considering fine-tuned smaller models after prompt engineering reaches its limit, especially when proprietary feedback data can improve a narrow task.

safety-policy-enforcement (safety)

For telemetry and private fine-tuning data, the talk puts user permission and access boundaries ahead of model reuse convenience.

Tradeoffs

Use objective evaluation and feedback capture instead of rapid eyeballing.

Chose component-level evaluation and production feedback signals over playground eyeballing and unstructured logs.

Rationale: Structured evaluation prevents false negatives, unmeasured thrash, and regression-prone prompt/model changes.

Adopt prompt management tooling that includes non-engineers.

Chose prompt systems with experiment history and cross-functional access over Git-only or document-only prompt storage.

Rationale: Git gives version control, but domain experts need a usable way to collaborate on prompt engineering without losing deployment discipline.

Start with best available models and prompt engineering before complex orchestration.

Chose strong base models and simple prompt/retrieval workflows first over early multi-step chains and agents.

Rationale: Simple baselines are easier to evaluate and maintain; complex chains are reserved for cases where reasoning needs justify the added moving parts.

Fine-tune smaller models for narrow tasks when data exists.

Chose task-specific fine-tuned models over continued reliance on the largest general hosted model.

Rationale: Narrow fine-tunes can use proprietary feedback data to improve cost, latency, and performance on the actual task rather than paying for broad general capabilities.

Treat telemetry and private fine-tuning data as consent and access-control problems.

Chose explicit permission and access separation over silent telemetry capture or shared fine-tuned models across permission boundaries.

Rationale: Feedback improves services, but only when users understand and permit telemetry, and private training data should not become indirectly available through model weights.

Gotchas (what bit them)

Teams abandon or thrash viable LLM applications when they lack objective evaluation.

Root cause: Prompt and retrieval changes are judged by eyeballing examples rather than by component-level metrics and production feedback.

Resolution: Evaluate during prompt iteration, evaluate components such as retrieval separately, monitor production outcomes, and use feedback to avoid regressions.

Prompt work loses institutional memory when it lives in playgrounds, notebooks, spreadsheets, or scattered documents.

Root cause: Teams treat prompts as informal text instead of production artifacts with history, model configuration, and collaboration controls.

Resolution: Use prompt management infrastructure that records experiments and makes the history accessible to technical and non-technical contributors.

Premature chains and agents make production failures harder to diagnose.

Root cause: Multiple prompts and steps introduce combinatorial complexity before teams know whether a simpler model, prompt, or retrieval setup works.

Resolution: Start with the strongest available model and push prompt engineering first; move to chains or agents only when the task truly requires reasoning and simpler alternatives have been tested.

Teams underestimate fine-tuning because frontier models feel sufficient.

Root cause: Teams assume small models need too much data or cannot compete with GPT-class hosted models on narrow production tasks.

Resolution: Try fine-tuning smaller models when proprietary examples and feedback data exist for a narrow task, then compare cost, latency, and task performance.

Fine-tuning on private data can blur access-control boundaries.

Root cause: Training data may be memorized or embedded in a shared model's weights even when not everyone should access the underlying documents.

Resolution: Avoid sharing fine-tuned models across unauthorized groups; consider separate fine-tunes or adapter-style access separation while recognizing the problem is not fully solved.

Evaluation

The talk argues for evaluation at three levels: development-time prompt iteration, production monitoring of user outcomes, and regression checks before prompt/model changes. It highlights GitHub Copilot's production feedback loop as a concrete exemplar.

Metrics: End-user acceptance of model suggestions · Whether accepted code remains in the codebase at later time intervals · Explicit votes such as thumbs up or thumbs down · Implicit user actions · User corrections or edited generated text · Component-level retrieval quality · Regression checks after prompt or model changes · Cost, latency, and task performance for fine-tuned models

Outcomes

• Humanloop distilled lessons from several hundred production LLM projects into four main pitfalls: missing evaluation, weak prompt management, premature chains/agents, and underused fine-tuning.
• GitHub Copilot is presented as a successful example of production feedback-driven evaluation at more than one million users.
• Phind is presented as a developer-search example where a task-specific fine-tuned open-source model improved cost, latency, and performance for its niche.
• The talk concludes that LLM systems need LLM-specific versions of software practices such as version control, evaluation, and CI/CD rather than copying traditional tools unchanged.

The talk is advisory rather than a single deployment report, but it draws from Humanloop's year-plus of platform operation and several hundred customer projects. Its strongest concrete scale examples are GitHub Copilot's million-plus users and a fine-tuning case using about 3,500 data points.