Automated Evaluation Framework for LLM-Powered Features

slack · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise collaboration

Slack needed a scalable way to measure and improve subjective LLM features such as summarization and natural-language search across many unique user experiences.

Constraints:

• summary quality is subjective and context-dependent
• product changes need representative evaluation beyond a handful of examples
• some evaluation methods are too expensive to run on every output
• privacy limits engineers' access to underlying validation-set data

Architecture

Slack's evaluation framework decomposes broad LLM-output quality into specific automated metrics for summarization and search, including accuracy, coherence, relevance, extrinsic hallucinations, citation correctness, and Slack-specific formatting of users, channels, and messages. Expensive LLM-as-judge methods are combined with smaller natural-language-inference models and sampling. The development cycle runs through a golden set for fast debugging, a larger 100-500-sample validation set where automated metrics substitute for raw-data inspection, and A/B tests that use the same metrics to evaluate production experiments.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  generative-slack-products[Generative Slack products] --> golden-set[Golden set]
  generative-slack-products --> validation-set[Validation set]
  generative-slack-products --> a-b-testing[A/B testing]
  quality-metric-library[Quality metric library] --> llm-evaluator[LLM evaluator]
  quality-metric-library --> nli-evaluator[NLI evaluator]
  llm-evaluator --> composite-scoring[Composite scoring]
  nli-evaluator --> composite-scoring
  composite-scoring --> golden-set
  composite-scoring --> validation-set
  composite-scoring --> a-b-testing

Components

Data flow

A proposed change to a Slack generative feature is first tried against a small golden set where engineers can inspect the source messages and outputs. If it looks promising, the change moves to a larger validation set of about 100-500 representative samples where automated metrics score outputs without exposing underlying data. Metrics may be produced by LLM evaluators, NLI models, and Slack-specific format checks, then combined into composite or broken-down quality scores. Finally, A/B testing uses those quality metrics to decide whether the experiment improves the product.

Why this architecture

Slack's architecture treats evaluation as a system rather than a single score because LLM product quality is both subjective and multi-dimensional. Decomposed metrics make hallucination, citation, formatting, relevance, and coherence measurable enough to guide engineering decisions. The staged golden-set, validation-set, and A/B-test lifecycle balances developer visibility, representative scale, privacy, and live-product confidence. Using both LLM judges and smaller NLI models lets Slack trade evaluation sophistication against compute cost.

Models used: large language models as evaluators · natural language inference models

Scale

• Slack focuses the framework on two generative products: message summarization and natural-language search.
  “our focus is primarily across two generative based products that were uh building the first is around sort of summarization of user messages”transcript, stated
• The validation set usually contains about 100-500 samples.
  “the next is a much larger and more representative validation set typically comprising around 100 to5 500 samples”transcript, stated
• The talk frames the lifecycle at a 10,000-foot level rather than disclosing production user counts or thresholds.
  “highlight what is our sort of llm development life cycle kind of look at at a 10,000 sort of foot level and sort of to set the stage our”transcript, implied

Alternatives considered (roads not taken)

Evaluate on ten examples and ship

Rejected because: Slack considers each user experience random and unique, so a tiny static set cannot support confident product improvement.

One monolithic hallucination metric

Rejected because: The team argues specific evaluation measures yield better outcomes, so hallucinations are broken into product-relevant subcategories.

Run large LLM judges for every evaluation need

Rejected because: Large evaluator models may be too compute-heavy to run at the desired scale, so Slack also uses sampling and NLI models.

Patterns

eval-driven-development (evaluation)

Slack uses automated quality metrics as stage gates from prototyping through validation and A/B testing.

llm-as-judge (evaluation)

Slack uses large LLMs as evaluators where their cost can be managed through sampling.

guardrails-io-filtering (quality-control)

Metrics check extrinsic hallucinations, incorrect citations, and malformed Slack IDs so generated content integrates safely with Slack.

Tradeoffs

Staged evaluation lifecycle

Chose golden sets, validation sets, and A/B tests over one evaluation phase for every model change.

Rationale: The staged process lets teams debug quickly, then gain representative confidence, then verify impact in production.

Specific metrics over broad labels

Chose narrow hallucination, citation, relevance, coherence, and formatting metrics over single broad quality or hallucination labels.

Rationale: Specific measures are more tractable and more aligned with product needs.

Mixed evaluator stack

Chose LLM evaluators plus NLI models and sampling over large LLM evaluation everywhere.

Rationale: The mixed stack preserves richer evaluator judgment where useful while making high-scale checks practical.

Gotchas (what bit them)

LLM output quality is subjective and context-dependent.

Root cause: Different users can legitimately prefer different summary lengths and detail levels.

Resolution: Slack breaks broad quality into multiple metrics and evaluates in context rather than forcing a single binary definition.

Small example sets create false confidence.

Root cause: Generative product experiences are random and unique across users.

Resolution: Slack pairs small golden-set debugging with larger validation sets and A/B testing.

LLM judges may be too expensive at full evaluation scale.

Root cause: Large evaluator models require significant compute.

Resolution: Slack uses sampling with LLM evaluators and smaller NLI models for higher-scale quality questions.

Evaluation

Slack evaluates generative features through metric decomposition, small inspectable golden sets, 100-500-sample validation sets, and A/B testing, using LLM judges, NLI models, and product-specific checks.

Metrics: accuracy · coherence · relevance · extrinsic hallucination · citation correctness · Slack user/channel/message ID formatting · composite or broken-down quality score

Outcomes

• The framework allowed Slack to evaluate extractive summarization as preprocessing for large-context summarization.
• Automated metrics showed quality tradeoffs and format-capability improvements.
• The standardized lifecycle supports rapid prototyping and fail-fast product development.

The validation stage usually uses 100-500 representative examples, while A/B testing moves evaluation into live experiments.