Large-Scale AI Assistant Deployment with Safety-First Evaluation Approach

discord · Tech · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Consumer-scale AI assistant safety and evaluation

Discord needed to ship Clyde AI, an agentic RAG chatbot, to a massive consumer audience while preventing safety, legal, policy, and abuse failures from surfacing in production. At Discord scale, rare failures become common enough to matter, so the central problem was measuring and mitigating risk before launch while keeping evaluations usable by product engineers.

Constraints:

• Clyde AI launched to over 200 million Discord users
• one-in-a-million failures can happen hundreds of times at Discord scale
• safety, legal, security, and policy stakeholders repeatedly blocked launch until risk was quantified
• users actively try to jailbreak and abuse LLM applications
• evaluations need to be simple and fast enough for developers to run repeatedly
• privacy constraints prevented a clean feedback loop from production data into eval datasets
• prompt changes, model choices, and long conversations can affect behavior and safety

Architecture

Discord built Clyde AI as an agentic RAG chatbot and wrapped it in a safety-first evaluation workflow. The team treated evals like unit tests: fast, local, deterministic checks for individual components such as moderation, tool triggering, retrieval summarization, and end-to-end behavior. Promptfoo provided local declarative evaluation and red-teaming workflows, while Datadog carried LLM observability metrics alongside product metrics. Safety assessment included predeployment risk views, harmful-input tests, application-specific jailbreak generation with attacker and judge LLMs, and coverage for PII leaks, hallucination, hijacking, and political opinions. Configuration and prompts were managed with Git and Retool, and lower-powered models were occasionally steered back on track by injecting GPT-4 responses in long conversations.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  clyde["Clyde AI chatbot"] --> evals["Component eval suites"]
  evals --> promptfoo["Promptfoo CLI"]
  evals --> pr["Pull request workflow"]
  clyde --> obs["Runtime observability"]
  obs --> online["Model-graded online evals"]
  clyde --> config["Prompt and configuration store"]
  promptfoo --> risk["Risk assessment view"]
  risk --> redteam["Adversarial red-team loop"]
  redteam --> promptfoo

Components

Data flow

A Clyde interaction flows through the assistant's agent/RAG path, where components such as moderation, tool use, retrieval triggering, and summarization are evaluated separately. Developers run local promptfoo eval suites from declarative configs and attach eval evidence to pull requests or CI/CD. Production and online eval metrics flow into Datadog. For safety, promptfoo risk views and red-team loops generate or mutate adversarial inputs before launch, while public dogfooding/social-media examples feed new eval cases when privacy blocks direct production-data reuse.

Why this architecture

Discord chose a developer-first evaluation architecture because safety risk, not model capability, was the repeat launch blocker. Unit-test-like evals were understandable, local, deterministic, and cheap enough to run frequently, while predeployment red teaming reduced the chance that vulnerabilities would first appear at 200-million-user scale. Existing observability and simple Git/Retool configuration kept the system operationally familiar.

Models used: Clyde AI · RAG chatbot · GPT-4 · GPT-3.5 · Llama · attacker LLM · judge LLM · model-graded evals

Scale

• Clyde AI launched to more than 200 million Discord users.
  “launched to over 200 million users on Discord”transcript, stated
• At Discord scale, a one-in-a-million failure can occur about 200 times.
  “if you have a one in a million uh sort of occurrence uh it will happen 200 times at at Discord scale”transcript, stated
• The speaker says simple deterministic personality checks got more than 80% of the value for about 1% of the work.
  “gets us more than 80% of the way there for like 1% of the work”transcript, stated
• The target developer workflow was dozens of eval runs per day.
  “it really should be easy for devs to do dozens of of evals per day”transcript, stated
• The team used GPT-4 as an occasional steering response in long conversations with lower-powered models.
  “we would occasionally drop in a GPT 4 response um just literally randomly um which would kind of act as like a bowling alley bumper”transcript, stated

Alternatives considered (roads not taken)

Complex LLM graders or classifiers for every behavior check

Rejected because: Discord found simple deterministic checks could capture most of the desired signal with much less effort.

Large end-to-end evals as the main testing strategy

Rejected because: The team preferred small component-level evals and only one broad end-to-end eval where needed.

Live databases or live web search in every eval

Rejected because: For retrieval-generation testing, the speaker recommends static context to isolate summarization from triggering and external-system variation.

Specialized LLM observability platform

Rejected because: Discord used Datadog because it was already in the product monitoring stack and could accept the relevant metrics.

Close the eval feedback loop directly with live production data

Rejected because: Privacy constraints prevented Discord from incorporating live production data into eval datasets.

Patterns

eval-driven-development (evaluation)

Discord made evals part of normal development hygiene, expecting each PR to include an eval and preferring local unit-test-style checks.

safety-policy-enforcement (safety)

The launch process focused on quantifying and mitigating security, legal, safety, policy, and harmful-content risk before production.

guardrails-io-filtering (safety)

Discord used predeployment risk assessment, harmful-input testing, and application-specific jailbreak testing to expose safeguard failures.

observability-tracing (operations)

LLM metrics and online production evals were fed into Datadog alongside other product metrics.

prompt-versioning (configuration)

Discord used Git as the source of truth for prompt versioning and Retool for configuration toggles.

rag-retrieval (retrieval)

Clyde is described as an agent and RAG chatbot, and the evaluation strategy includes splitting retrieval triggering from static-context summarization.

Tradeoffs

How sophisticated eval metrics should be

Chose Simple deterministic component evals over Fancy LLM graders and complex metrics for every behavior.

Rationale: Simple checks were understandable, fast, local, and got enough signal to support high-frequency developer use.

How to test RAG or web-search behavior

Chose Separate triggering tests from static-context summarization tests over End-to-end live retrieval tests for every eval run.

Rationale: Splitting the tests isolates decisions and avoids live database or live web-search instability.

Which observability stack to use

Chose Datadog and in-house simple online evals over A specialized LLM observability platform.

Rationale: The team could put LLM metrics into the observability tool already used for product metrics.

Where to focus safety effort

Chose Predeployment risk assessment and red teaming over Relying mainly on live filtering after launch.

Rationale: The speaker says commercial attention skews toward filtering, but predeployment risk assessment was more important for Discord.

Gotchas (what bit them)

Safety failures appear in production if they are not quantified before launch.

Root cause: Rare failures become frequent at Discord's 200-million-user scale, and users actively probe LLM weaknesses.

Resolution: Make risk measurement and mitigation a predeployment requirement with safety evals and red-team tooling.

Prompt bloat from fixing every discovered failure can produce negative returns.

Root cause: Teams are tempted to keep adding special cases to prompts after each eval failure.

Resolution: Remove prompt clutter and give the model room to do the reasonable thing.

Prompts do not transfer cleanly across model families.

Root cause: Teams calibrate prompts around GPT behavior, but Claude and Llama-style models need model-specific tuning.

Resolution: Treat prompts as vendor-specific artifacts and spend time re-tuning when switching models.

Lower-powered models can drift away from the system prompt in long conversations.

Root cause: Long chats caused models like Llama and GPT-3.5 to revert toward vanilla chat behavior.

Resolution: Discord experimented with occasional GPT-4 responses as bumpers to steer the conversation back on track.

The ideal production-data feedback loop may be blocked by privacy.

Root cause: Live user data could not be freely incorporated back into evaluation datasets.

Resolution: Use dogfooding, public posts, and publicly visible failures or wins as substitute eval examples.

Evaluation

Discord evaluated Clyde through local deterministic component tests, PR-level eval requirements, optional CI/CD integration, promptfoo red-team runs, risk-category views, online production evals in Datadog, and public/dogfood-derived regression examples.

Metrics: component-level moderation behavior · tool-triggering behavior · static-context summarization correctness · end-to-end assistant behavior · risk categories such as brand and legal risk · harmful-content refusal · PII leak checks · hallucination checks · prompt-hijacking checks · political-opinion checks

Outcomes

• Discord shipped Clyde AI to more than 200 million users with safety and legal risk as the main launch concern.
• The team built promptfoo as an open-source local eval and red-teaming CLI.
• Simple deterministic evals became the team's preferred development mechanism.
• Prompt bloat was reduced after diminishing and negative returns.
• Datadog carried LLM observability and online eval metrics alongside product metrics.
• The team could not close the ideal production-data feedback loop because of privacy constraints.

The talk repeatedly anchors risk in Discord scale: 200 million users, one-in-a-million failures occurring about 200 times, and a developer workflow aiming for dozens of local eval runs per day.