Building Production-Grade LLM Applications: An Architectural Guide

github · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Reference architecture for production LLM applications

GitHub’s guide addresses how developers should scope, select, customize, deploy, and evaluate LLM-powered applications when outputs are probabilistic, context must be retrieved and packaged, unsafe inputs/outputs must be filtered, and online user behavior must feed product iteration.

Constraints:

• LLM applications are fundamentally different from conventional software because outputs are probabilistic and do not produce the same predictable outcomes.
• Teams need a scoped use case that is small enough to iterate on but valuable enough to matter to users.
• Model choice must account for licensing, size, performance, speed, cost, and offline evaluation before customization.
• Production architecture needs user input handling, input enrichment, prompt construction, caching, filters, and telemetry rather than a raw model call.
• Online evaluations must measure real user interaction, such as acceptance and retention/edit rates.

Architecture

The guide proposes a five-step LLM application lifecycle: focus on one problem, choose the right pretrained LLM, customize it with in-context learning/RLHF/fine-tuning, set up the application architecture, and run online evaluations. The runtime architecture includes an LLM API/host, UI/router, speech-to-text where needed, data sources, an embedding model, vector database, prompt optimization, a data filter, an LLM cache, content classifier/filter, and telemetry service. A hypothetical ISP assistant illustrates the flow: Dave’s spoken Wi-Fi issue is routed through the UI, translated to text, enriched with vector-search context from company complaints/solutions, filtered for authorized data, packaged into a prompt, answered by the LLM, moderated by safety tooling, cached when appropriate, and evaluated through telemetry.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  problem-scoping["Problem scoping"] --> llm-selection["LLM selection"]
  llm-selection --> llm-customization["LLM customization"]
  llm-customization --> llm["LLM"]
  user["User"] --> ui-and-router["UI and router"]
  ui-and-router --> speech-to-text["Speech-to-text"]
  speech-to-text --> embedding-model["Embedding model"]
  data-source["Data source"] --> embedding-model
  embedding-model --> vector-database["Vector database"]
  vector-database --> prompt-optimization-tool["Prompt optimization tool"]
  data-filter["Data filter"] --> prompt-optimization-tool
  prompt-optimization-tool --> llm-api-and-host["LLM API and host"]
  llm-api-and-host --> llm-cache["LLM cache"]
  llm-api-and-host --> llm
  llm --> content-classifier["Content classifier"]
  content-classifier --> user
  llm --> telemetry-service["Telemetry service"]
  telemetry-service --> online-evaluation["Online evaluation"]
  online-evaluation --> problem-scoping

Components

Data flow

A user enters through a UI or voice channel. The UI/router sends the request to the right LLM assistant path, and speech-to-text converts voice into text when needed. Input enrichment embeds the query, retrieves relevant context from a vector database backed by company data, filters unauthorized data, and uses prompt optimization to prioritize and order context. The LLM API/host generates an answer, optionally reusing cached outputs for similar queries. A content classifier/filter blocks harmful responses, and telemetry captures user behavior for online evaluation and product improvement.

Why this architecture

The guide recommends a layered architecture because LLM output quality depends on more than the model call. The app must start from a focused problem, pick a commercially appropriate and measurable model, enrich user input with retrieved context, block unauthorized data, package prompt context, reduce latency and variability with caching, filter harmful responses, and use telemetry/online evaluations to keep improving the user experience.

Models used: Pre-trained LLMs · Ada-scale models around 350 million parameters · 7B-13B parameter open models · 175B parameter large models · Embedding models · Reward models for RLHF · Fine-tuned task-specific models · Content classifiers

Scale

• The guide defines five major steps for building an LLM application.
  “we’ll cover five major steps to building your own LLM app”article, stated
• The model-size examples range from 350 million to 175 billion parameters.
  “The size of LLMs can range from 7 to 175 billion parameters—and some, like Ada, are even as small as 350 million parameters.”article, stated
• The guide says most LLMs at publication time were 7-13B parameters.
  “Most LLMs (at the time of writing this post) range in size from 7-13 billion parameters.”article, stated
• The RLHF example uses an 80% acceptance probability as an acceptable-output threshold.
  “if there’s an 80% probability that a user will accept an output, then it’s fine to generate.”article, stated
• The online evaluation example uses acceptance rate and retention/edit rate.
  “online evaluations for GitHub Copilot are measured through acceptance rate”article, stated

  “as well as the retention rate (how often and to what extent a developer edits an accepted completion).”article, stated

Alternatives considered (roads not taken)

Attempt to solve a broad class of user problems immediately.

Rejected because: GitHub recommends starting with a single scoped problem, citing Copilot's initial focus on coding functions in the IDE instead of all developer problems.

Pick the largest model by default.

Rejected because: The guide notes that smaller models are faster and cheaper, and improved smaller-model performance challenges the bigger-is-always-better assumption.

Fine-tune before measuring the base model.

Rejected because: The guide says teams should evaluate the pretrained model's speed, consistency, and desired output before customization.

Use only prompt/in-context learning for adaptation.

Rejected because: The guide presents in-context learning, RLHF, and fine-tuning as different adaptation mechanisms with different supervision and labeling costs.

Generate every response fresh from the model.

Rejected because: Caching similar-query outputs can reduce latency, computational cost, and variability.

Patterns

rag-retrieval (architecture)

The assistant enriches the user query with company-specific context retrieved from embeddings stored in a vector database.

retrieval-ranking (architecture)

Prompt optimization prioritizes relevant embeddings and orders context snippets before sending the prompt to the LLM.

guardrails-io-filtering (safety)

The architecture includes a data filter to block unauthorized/PII input and a content classifier to prevent harmful or offensive responses.

semantic-caching (operations)

The LLM cache reuses outputs for repeated or similar queries to lower latency, cost, and response variability.

observability-tracing (operations)

Telemetry captures production user behavior so developers can evaluate and improve the LLM app.

eval-driven-development (quality)

The guide recommends offline evaluations before customization and online evaluations after deployment, using metrics like acceptance and retention/edit rate.

fine-tuning-adaptation (modeling)

Fine-tuning is presented as a supervised customization technique for task-specific behavior when labeled examples are available.

Tradeoffs

Start with a narrow problem scope.

Chose A focused use case that supports quick iteration and visible user value over Trying to address all user problems with AI from the start.

Rationale: A narrow scope makes iteration and measurement more practical while still enabling expansion later.

Consider smaller models for production apps.

Chose Smaller/faster/cheaper models when quality is sufficient over Automatically choosing the largest model.

Rationale: The guide says smaller-model performance is improving and may be viable where large models are out of scope.

Use RLHF when exact labels are unavailable but user preference feedback exists.

Chose Reward-model learning from accept/reject behavior over Supervised fine-tuning with exactly labeled outputs for every sample.

Rationale: RLHF expands acceptable-output criteria without requiring exact labels, while fine-tuning can be powerful but labeling-intensive.

Cache similar outputs for efficiency.

Chose LLM cache reuse for repeated or similar queries over Always generating a fresh answer.

Rationale: Caching reduces latency and cost but must be paired with filters and telemetry for responsible behavior.

Host locally for tinkering but use cloud hosting for volume.

Chose Cloud hosting for production call volume, local hosting for low-cost experimentation over One hosting strategy for every lifecycle stage.

Rationale: The guide distinguishes production scale from experimentation cost management.

Gotchas (what bit them)

LLM apps cannot be engineered like deterministic compiled software.

Root cause: Developers must navigate datasets, embeddings, and parameter weights, and outputs are probabilistic.

Resolution: Use evaluation, retrieval, prompt construction, guardrails, telemetry, and iterative product scoping around the model call.

Starting too broad slows learning and makes evaluation vague.

Root cause: A general-purpose AI ambition does not provide the focused iteration loop needed for measurable progress.

Resolution: Choose a focused problem first, as Copilot initially did with coding functions in the IDE.

Commercial model licensing can block deployment if ignored during model selection.

Root cause: Some models/APIs may not be licensed for commercial use.

Resolution: Include licensing as an explicit LLM selection criterion before building a sellable application.

Context retrieval can leak unauthorized or personal data into the LLM.

Root cause: The prompt construction layer pulls company data into model context.

Resolution: Add a data filter that blocks unauthorized data and PII before prompt construction.

Unsafe user behavior can produce unsafe model behavior if outputs are not filtered.

Root cause: Users may be frustrated or abusive, and the automated assistant might respond in kind.

Resolution: Use a content classifier or filter on the response path.

Evaluation

The guide uses both offline and online evaluation. Offline evaluation measures how well, fast, and consistently a model generates known desired outputs before customization. Online evaluation measures production user interaction, with GitHub Copilot examples including acceptance rate and retention/edit rate.

Metrics: Offline latency · Offline accuracy · Contextual relevance · Consistency of desired output · Incremental scoring/partial correctness · Acceptance rate · Retention rate · Extent of edits to accepted completions · Telemetry-driven user activity · Safety/filtering behavior

Outcomes

• The article defines a reusable five-step lifecycle for LLM app development.
• It provides a reference architecture for user input, enrichment/prompt construction, and responsible/efficient AI tooling.
• The ISP assistant example illustrates how vector retrieval, prompt packaging, data filtering, caching, content filtering, and telemetry combine in an LLM app.
• It ties evaluation to both offline model testing and online user-interaction metrics.
• The source is a guide rather than a measured deployment report, so it does not provide production outcomes for the hypothetical ISP assistant.

Concrete scale references include five lifecycle steps, model sizes from 350M to 175B parameters, a common 7-13B parameter range at publication time, an 80% acceptance-probability RLHF example, and Copilot online metrics of acceptance and retention/edit rate.