Building and Scaling LLM Applications at Discord

discord · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Consumer-scale LLM application development

Discord needed a repeatable way to identify, prototype, evaluate, and scale LLM-powered features for a large consumer platform while balancing quality, latency, safety, language support, throughput, cost, privacy, and infrastructure effort.

Constraints:

Architecture

Discord describes a staged LLM development process: identify use cases suited to generative AI, define product requirements, prototype with advanced commercial models such as GPT-4, evaluate prompts using traditional metrics plus AI-assisted critic prompts, limited-release the feature through A/B tests, and deploy at scale behind a production architecture. The production architecture prepares inputs into tested prompts, sends them to an LLM inference server, hydrates the output, runs content-safety checks with Legal/Safety partners and trust-and-safety models, and returns results to users. For production model serving, Discord weighs commercial APIs against self-hosted Llama/Mistral-style models, Triton/vLLM servers, GPUs, batching, token-length patterns, and fine-tuning needs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  use-case-filter[Use case filter] --> product-requirements-checklist[Product requirements checklist]
  product-requirements-checklist --> prototype-llm[Prototype LLM]
  prototype-llm --> prompt-evaluator[Prompt evaluator]
  prompt-evaluator --> a-b-learning-loop[A/B learning loop]
  a-b-learning-loop --> production-api-gateway[Production API gateway]
  a-b-learning-loop --> metrics-and-cost-monitor[Metrics and cost monitor]
  production-api-gateway --> prompt-preparation[Prompt preparation]
  prompt-preparation --> llm-inference-server[LLM inference server]
  llm-inference-server --> content-safety-service[Content safety service]
  content-safety-service --> production-api-gateway
  content-safety-service --> legal-and-safety-partners[Legal and Safety partners]
  llm-inference-server --> self-hosted-inference-stack[Self-hosted inference stack]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Use case filter Checks whether generative AI is appropriate for unstructured-content analysis, massive scaling, or tasks that resist rules/traditional ML. Product requirements checklist
Product requirements checklist Defines latency, task complexity, prompt length, quality, safety, language support, and estimated QPS before prototyping. Use case filter, Prototype LLM
Prototype LLM Advanced commercial model, such as GPT-4, used to validate whether the idea is tractable before infrastructure investment. Product requirements checklist, Prompt evaluator
Prompt evaluator Traditional metrics plus AI-assisted evaluation using a critic prompt on a stronger model. Prototype LLM, A/B learning loop
A/B learning loop Limited release process that collects user satisfaction, engagement, hallucinations, latency, throughput, errors, and cost data. Prompt evaluator, Production API gateway, Metrics and cost monitor
Production API gateway Gateway that receives user requests, requests LLM inference, hydrates outputs, and routes through content safety before response. A/B learning loop, Prompt preparation, Content safety service
Prompt preparation Transforms inputs into tested and evaluated prompts for the inference server. Production API gateway, LLM inference server
LLM inference server Commercial API or self-hosted model server that produces generated outputs from prepared prompts. Prompt preparation, Content safety service, Self-hosted inference stack
Content safety service Trust-and-safety layer that scans generated content before it reaches users. Production API gateway, LLM inference server, Legal and Safety partners
Legal and Safety partners Cross-functional teams that define mitigations, privacy principles, and safety requirements. Content safety service
Metrics and cost monitor Tracks satisfaction, engagement, incorrect answers, hallucinations, latency, tokens per second, error rates, and token spend forecasts. A/B learning loop
Self-hosted inference stack Optional Llama/Mistral-style stack with Triton or vLLM, GPUs, batching, queues, and task-specific tuning. LLM inference server

Data flow

A candidate use case is filtered through Discord's suitability criteria and requirements checklist. The team prototypes with a strong commercial LLM, iterates on the prompt, and sends outputs to a separate critic prompt on a best-in-class model for yes/no or rating-style evaluation. Once quality is sufficient, the feature ships as a limited A/B test, collecting user satisfaction, hallucination/problem reports, latency, tokens per second, error rates, and cost forecasts. At scale, a user request passes through an API gateway and prompt-preparation layer to an LLM inference server, returns an LLM output, is hydrated and scanned by content-safety services, and only then reaches the user. If commercial APIs are unsuitable, a self-hosted inference stack with load balancing, queues, batch scheduling, GPUs, and model servers becomes the serving path.

4 supporting quotes
“Starting with product ideation and defining requirements, we first need to figure out what we’re building and how it can benefit users.”article, stated
“AI-assisted evaluation consists of 2 separate prompts: one for your task and another to evaluate your results.”article, stated
“Once we are sufficiently confident in the quality of the results our prompt generates, we roll out a limited release (e.g. A/B test) of our product”article, stated
“Inputs to the inference server are prepared into a prompt that we’ve tested and evaluated on a robust set of examples.”article, stated

Why this architecture

Discord starts with commercial frontier models because they provide the fastest tractability test: if GPT-4 cannot solve the prototype well enough, the problem may not be addressable with current generative AI. AI-assisted evaluation turns subjective prompt iteration into a measured loop, and A/B tests provide real in-situ user and system data before full launch. The production gateway plus content-safety layer reflects Discord's consumer-platform need for privacy, safety, and quality. Self-hosting is treated as an optimization path for cost or privacy only when the team accepts the infrastructure and performance tradeoffs.

4 supporting quotes
“if problems can't be adequately solved with state-of-the-art foundational models like GPT-4, then more often than not, those problems may not be addressable using current generative AI tech.”article, stated
“By using metrics to guide our changes, we know we are moving the needle on the quality of our results. To do this, we employ a technique known as AI-assisted evaluation, alongside”article, stated
“Because we care deeply about the user experience, privacy, and safety, we work with cross-functional partners like Legal and other Safety teams”article, stated
“Using commercial LLMs is great because it gives us access to top-notch models and we don't have to worry about setting up the tech ourselves”article, stated

Models used: GPT-4 · ChatGPT · OpenAI GPT APIs · Llama · Mistral · trust and safety ML models

Scale

Alternatives considered (roads not taken)

Use generative AI for any product engagement opportunity

Rejected because: Discord explicitly filters for problems where an LLM is the right tool rather than applying AI when other tools fit better.

“generative AI is a tool — and one that shouldn’t be applied when other tools are a better fit. When it comes to identifying where generative AI can make an impact, we dig into challenges that commonly: * Involve analysis, interpretation, or review of”article, stated

Invest in ML infrastructure before proving the idea

Rejected because: Discord first validates with commercial LLMs so engineering can focus on product iteration if the model can solve the problem.

“We generally lean towards picking more advanced commercial LLMs to quickly validate our ideas and obtain early feedback from users.”article, stated
“concentrate on iterating on our product rather than diverting engineering resources towards building and maintaining machine learning infrastructure.”article, stated

Commercial LLM APIs for full-scale production

Rejected because: They offer quality without setup burden, but cost can scale quickly and privacy may favor in-house processing.

“Using commercial LLMs is great because it gives us access to top-notch models and we don't have to worry about setting up the tech ourselves, but the expenses can add up quickly.”article, stated
“For privacy reasons, we may also prefer to process full-scale data completely in-house. A solution is to self-host an open-sourced or custom fine-tuned LLM. Opting for a self-hosted model can reduce costs dramatically - but with additional development time,”article, stated

Self-hosted open-source or fine-tuned models

Rejected because: Self-hosting can reduce costs or improve privacy, but it adds development time, maintenance overhead, GPUs, server tuning, and possible performance implications.

“Opting for a self-hosted model can reduce costs dramatically - but with additional development time, maintenance overhead, and possible performance implications.”article, stated
“We need a dedicated model server for running model inference (using frameworks like Triton or vLLM), powerful GPUs”article, stated

Always choose the largest model

Rejected because: For self-hosted paths Discord recommends starting smaller and moving up only if quality requires it.

“We've found it's best to start with smaller models and only move up to bigger ones if needed for quality reasons.”article, stated

Patterns

eval-driven-development (quality)

Discord uses AI-assisted evaluation, traditional metrics, limited-release A/B tests, user feedback, system health metrics, and cost metrics to decide whether to iterate or deploy.

“To do this, we employ a technique known as AI-assisted evaluation, alongside traditional metrics for measuring performance.”article, stated
“If it is, then we can proceed to the next step: deploying at scale. If not, then we look to take our learnings, iterate on the system, and try again.”article, stated

llm-as-judge (evaluation)

A stronger critic model such as GPT-4 automatically evaluates the inference model's output against criteria.

“AI-assisted evaluation uses best-in-class LLMs (like GPT-4) to automatically critique how well the AI's outputs match what we expected”article, stated
“The task prompt is passed to the inference model whereas the critic prompt is passed to the more advanced critic model. ### Launch and Learn Once we are sufficiently confident in the quality of the results our prompt generates, we roll out a limited release (e.”article, stated

model-routing (model-ops)

Discord chooses among commercial APIs, smaller models, larger models, open-source models, and fine-tuned/self-hosted options based on quality, latency, privacy, and cost.

“The product requirements we define then play into our selection of which off-the-shelf LLM we'll use for our prototype.”article, stated
“start with smaller models and only move up to bigger ones if needed for quality reasons.”article, stated

guardrails-io-filtering (safety)

Generated content is scanned by content safety filters and trust-and-safety models before reaching users.

“we chose to incorporate content safety filters to the output of the inference server to identify undesired material before it reaches the user.”article, stated
“We can leverage in-house or third-party trust and safety ML models to detect inappropriate content.”article, stated

safety-policy-enforcement (governance)

Discord includes safety, prompt hacking, privacy, data minimization, Legal, and Safety partners in the product architecture.

“Safety: How important is it to sanitize user input or prevent the generation of harmful content and prompt hacking?”article, stated
“we work with cross-functional partners like Legal and other Safety teams to ensure we’ve implemented thoughtful mitigations”article, stated

structured-output-contract (integration)

Discord treats consistently structured LLM output as necessary for minimizing parsing errors and using outputs robustly in services.

“LLMs sometimes have trouble generating output in a consistently structured format, which is crucial for minimizing data parsing errors and ensuring the output is robustly usable in our services.”article, stated

canary-rollout (release)

Discord uses limited releases and A/B tests to observe real behavior before full deployment.

“we roll out a limited release (e.g. A/B test) of our product and observe the system’s performance in situ.”article, stated

Tradeoffs

Prototype strategy

Chose advanced commercial LLMs over building ML infrastructure first.

Rationale: Frontier APIs quickly reveal whether the idea is tractable and let engineers iterate on the product before maintaining infrastructure.

“We generally lean towards picking more advanced commercial LLMs to quickly validate our ideas and obtain early feedback from users.”article, stated
“concentrate on iterating on our product rather than diverting engineering resources towards building and maintaining machine learning infrastructure.”article, stated

Latency-quality balance

Chose explicit requirements tradeoff between bigger models, compute, and smaller models over assuming the highest-quality model is always acceptable.

Rationale: Bigger models can deliver better outcomes but run more slowly, forcing a choice among compute cost, latency, and quality.

“a bigger, more capable LLM usually delivers better outcomes but operates more slowly during inference owing to the model’s larger size.”article, stated
“either incurring higher costs (e.g. by having more available compute) or accepting a drop in quality by using smaller models.”article, stated

Evaluation design

Chose GPT-4 critic prompts with simple ratings over manual review of every prompt output.

Rationale: Simple yes/no or scaled critic outputs make prompt iteration automatable while still using a stronger model's judgment.

“Automating evaluation allows us to quickly see what's working well and what needs to be tweaked in our prompts, without having to manually check everything.”article, stated
“we design prompts that ask for simple yes or no answers or rate the outputs on a scale”article, stated

Serving model

Chose commercial APIs when quality and speed to launch matter over self-hosting by default.

Rationale: Commercial APIs avoid setup burden and provide top models, while self-hosting is reserved for privacy, cost, or custom quality reasons.

“Using commercial LLMs is great because it gives us access to top-notch models and we don't have to worry about setting up the tech ourselves”article, stated
“For privacy reasons, we may also prefer to process full-scale data completely in-house.”article, stated

Model size for self-hosting

Chose start with smaller models and scale up only for quality over defaulting to larger models.

Rationale: Smaller models reduce serving burden when they satisfy quality requirements.

“We've found it's best to start with smaller models and only move up to bigger ones if needed for quality reasons.”article, stated

Gotchas (what bit them)

Generative AI can be misapplied when a simpler tool fits better.

Root cause: AI excitement can obscure whether the problem actually needs an LLM.

Resolution: Filter use cases for unstructured content, massive scale, or problems difficult for rules and traditional ML.

“generative AI is a tool — and one that shouldn’t be applied when other tools are a better fit.”article, stated

Prompt iteration becomes ambiguous after many wording changes.

Root cause: Manual inspection alone makes it hard to tell whether prompt adjustments improve results.

Resolution: Use metrics and AI-assisted evaluation to guide prompt changes.

“after a lot of adjustments, it's often difficult to tell if these changes are actually improving our results.”article, stated
“By using metrics to guide our changes, we know we are moving the needle on the quality of our results.”article, stated

LLM features can pass prototype quality but fail in real user experience.

Root cause: Real users expose hallucinations, incorrect answers, engagement issues, latency, structured-output failures, and cost behavior.

Resolution: Run limited A/B releases and iterate from satisfaction, system health, and cost metrics before full deployment.

“This feedback is critical to identify areas for improvement, including highlighting incorrect answers or instances where LLM hallucinations might be causing a strange user experience.”article, stated

Structured output inconsistency breaks downstream service use.

Root cause: LLMs sometimes fail to emit a consistently parseable format.

Resolution: Track system health and plan post-hoc processing needed to productionize the capability.

“LLMs sometimes have trouble generating output in a consistently structured format, which is crucial for minimizing data parsing errors”article, stated
“Insights here can inform how much post-hoc processing might be needed to fully productionize this capability at scale.”article, stated

Self-hosting shifts cost savings into engineering and performance complexity.

Root cause: Dedicated model servers, GPUs, batching, token-length tuning, and maintenance become the team's responsibility.

Resolution: Start with commercial models for validation, then self-host only when privacy or cost justifies the overhead.

“Opting for a self-hosted model can reduce costs dramatically - but with additional development time, maintenance overhead, and possible performance implications.”article, stated

Evaluation

Discord evaluates LLM features with AI-assisted critic prompts, traditional metrics, yes/no or rating-based outputs, limited-release A/B tests, user satisfaction and engagement, incorrect-answer and hallucination reports, latency, throughput in tokens per second, error rates, cost forecasting, and structured-output robustness checks.

Metrics: critic yes/no judgments · critic rating scores · user satisfaction · direct user feedback · user engagement · incorrect answers · hallucination reports · response latency · tokens per second · error rates · token consumption and forecast cost · structured-output parsing errors

4 supporting quotes
“This helps us pick the prompts that lead to better quality outputs, making the end product more appealing to users.”article, stated
“For consumer-facing applications, we similarly focus on measures of user satisfaction - direct user feedback, user engagement measures, etc.”article, stated
“we also pay attention to system health metrics, such as response speed (latency), throughput (tokens per second), and error rates.”article, stated
“We look at how many tokens per second we're using in our initial limited release to predict the costs of a complete launch”article, stated

Outcomes

The case is a methodology article for a large consumer platform. The concrete operating concerns are hundreds of millions of users, estimated QPS, tokens-per-second cost forecasting, safety review before user exposure, and potential self-hosted inference infrastructure.

3 supporting quotes
“Finally, we fully launch and deploy our product at scale. In this post, we will dive deeper into each stage of this process.”article, stated
“All these elements put together form a system that taps into the capabilities of LLMs efficiently while monitoring the content's quality and safety”article, stated
“Tuning the inference servers for optimal performance is task-specific - the best configuration depends on the models we’re using”article, stated