Panel Discussion: Best Practices for LLMs in Production

various · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Generative AI production readiness and LLMOps practice

The panel addresses how teams move generative AI prototypes into production despite easy prototyping, hard deployment, cost/latency/quality tradeoffs, hardware constraints, evaluation uncertainty, modular failure modes, UX risk, and unclear RAG-versus-fine-tuning decisions.

Constraints:

• The field is new enough that few practitioners have long production experience.
• LLM prototypes are easy to build while production systems are expensive and difficult to operate.
• Production deployments need observability before users encounter unexpected behavior.
• Evaluation must reflect user value rather than only easy technical counters.
• RAG, fine-tuning, prompts, data, indexes, and UX all change over time and need versioning or monitoring.
• Hardware scarcity and heterogeneous accelerator options complicate serving and fine-tuning.

Architecture

The panel recommends a production lifecycle rather than a single model trick: start with APIs to test value, whiteboard system architecture before over-investing, instrument the modular stack from day one, capture user feedback and product metrics, version code/data/indexes/prompts, use RAG for knowledge injection and fine-tuning for output style, protect users with guardrails and UX recovery paths, and iterate continuously after launch.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  API_prototype["API prototype"] --> System_design_review["System design review"]
  API_prototype --> Production_observability["Production observability"]
  System_design_review --> RAG_retrieval_layer["RAG retrieval layer"]
  System_design_review --> Fine_tuning_track["Fine-tuning track"]
  System_design_review --> UX_and_guardrails["UX and guardrails"]
  Production_observability --> User_feedback_loop["User feedback loop"]
  Production_observability --> Evaluation_dataset_and_metrics["Evaluation dataset and metrics"]
  Production_observability --> Modular_LLM_stack["Modular LLM stack"]
  Modular_LLM_stack --> RAG_retrieval_layer
  Modular_LLM_stack --> Fine_tuning_track
  Modular_LLM_stack --> Prompt_and_artifact_versioning["Prompt and artifact versioning"]
  Modular_LLM_stack --> UX_and_guardrails
  RAG_retrieval_layer --> Prompt_and_artifact_versioning
  Fine_tuning_track --> Evaluation_dataset_and_metrics
  Prompt_and_artifact_versioning --> Evaluation_dataset_and_metrics
  UX_and_guardrails --> User_feedback_loop
  User_feedback_loop --> Evaluation_dataset_and_metrics
  User_feedback_loop --> System_design_review

Components

Data flow

A generative AI feature starts as an API-backed prototype, moves through system-design review, launches with observability across modular components, captures user feedback and top-line metrics, and feeds evaluation results into prompt, retrieval, fine-tuning, data, index, and UX iterations.

Why this architecture

The panelists argue that model choice alone is not the production system. LLM applications are modular, non-deterministic, and expensive, so teams need system design, observability, evaluation, versioning, UX protection, and iteration from the first deployment.

Models used: GPT-4 · OpenAI models · Open source LLMs · RAG systems · Fine-tuned models

Scale

• The panel occurred after roughly six to nine months of intense generative AI/LLM attention.
  “we have been having so much talk about gen and llm over the past I don't know like six months nine months”transcript, stated
• The panelists note production experience is still short; nobody has five years of LLM production experience.
  “I don't think that there are like anyone like with with five years of experience in putting llms in production”transcript, stated
• The recommended fast prototype window is one to two days.
  “take one to two days to get the quickest dirtiest prototype that you can”transcript, stated
• HELM is cited as listing about 59 evaluation metrics.
  “Helm list like I think 59 different metrics of course you don't want to like sit there and try to optimize all over the place”transcript, stated

Alternatives considered (roads not taken)

Start by fine-tuning or hosting custom models

Rejected because: The panel recommends first testing traction with API access because fine-tuning and hosting introduce major costs.

Treat production as the finish line

Rejected because: Panelists frame production as the start of iteration, where interesting failures and improvements happen.

Optimize token usage and latency because they are easy to measure

Rejected because: The panel warns that easy metrics may not reflect user value or experience.

Use fine-tuning for fresh factual knowledge

Rejected because: Panelists distinguish RAG for knowledge injection from fine-tuning for style, terminology, and output behavior.

Patterns

observability-tracing (operations)

The panel recommends observability before production so teams can trace user interactions and locate failing components in modular LLM stacks.

eval-driven-development (evaluation)

The panel emphasizes user feedback, ground-truth datasets, and use-case-specific metrics as the basis for production iteration.

model-routing (model-operations)

Teams are advised to prototype with API providers, then decide whether to fine-tune, prompt open-source models, or use other providers.

rag-retrieval (retrieval)

RAG is positioned as the right mechanism for adding factual knowledge and current context to model outputs.

prompt-versioning (prompting)

Prompt, data, index, and code versioning are treated as non-negotiable for production LLM systems.

guardrails-io-filtering (safety)

The panel recommends input and output guardrails to protect users from bad outputs and prevent PII leakage to public endpoints.

Tradeoffs

Prototype with model APIs before investing in custom models.

Chose API-backed validation over Immediate fine-tuning and self-hosting.

Rationale: APIs reveal whether users want the product before the team pays for larger GPUs, tuning, and hosting.

Treat production as an iteration point.

Chose Launch with observability and improvement loops over Waiting for a perfect pre-production model.

Rationale: Real users reveal prompt classes, failures, and interaction patterns that cannot be fully anticipated in the lab.

Use one primary optimization metric with supporting metrics.

Chose Focused use-case-specific evaluation over Optimizing dozens of benchmark metrics at once.

Rationale: A single north-star metric plus supporting indicators is easier to manage and align with the user experience.

Gotchas (what bit them)

Prototype ease hides production difficulty.

Root cause: LLMs are computationally expensive and force cost-latency-quality tradeoffs that are less severe in traditional ML.

Resolution: Prototype quickly, but plan system architecture, infrastructure, observability, and hardware strategy before scaling.

Teams may optimize irrelevant metrics.

Root cause: Token usage and latency are easy to measure but may not show whether users receive value.

Resolution: Pick two or three metrics tied to user experience, plus supporting metrics across the stack.

User experience is often treated as an afterthought.

Root cause: Teams focus on fine-tuning and model plumbing while forgetting that non-deterministic outputs can be offensive, dangerous, or confusing.

Resolution: Design recovery UX and guardrails so users can get unstuck and stay protected.

RAG/fine-tuning confusion wastes effort.

Root cause: Teams treat RAG and fine-tuning as alternatives instead of tools for different failure modes.

Resolution: Use RAG when context is missing or stale; use fine-tuning when output style, terminology, or format is wrong.

Evaluation

The panel recommends combining human user feedback, use-case-specific product metrics, supporting stack metrics, ground-truth prompt datasets, model-assisted critique where useful, and frameworks such as HELM/RAGAS without optimizing every available metric.

Metrics: Thumbs up/down feedback · User interaction counts · Edit distance for summarization · Refresh rates for interactive experiences · Semantic similarity to example sentences · Levenshtein distance for output diversity · Top-line use-case metric plus supporting indicators · Prompt drift against production prompt datasets

Outcomes

• The panel produced a consensus that production LLM work begins after launch rather than ending there.
• Panelists emphasized API prototyping, observability, user feedback, versioning, guardrails, RAG/fine-tuning separation, and hardware-aware deployment as production practices.
• No single deployed product outcome is reported because this is a cross-practitioner discussion.

This is a multi-speaker 2023 panel, so the extraction captures practitioner consensus and cautions rather than a single system's production metrics.