Building Robust Evaluation Systems for Auto-Generated Video Titles

loom · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Evaluation of LLM-generated user-facing content

Loom needed a reliable way to evaluate whether LLM-generated video titles were good for real users, not merely whether the model followed a prompt.

Constraints:

• Video titles must convey the main idea, be concise, engaging, readable, non-hallucinated, and follow Loom's style.
• Some quality checks are objective and should not require an LLM judge.
• Subjective qualities such as relevance, engagement, and clarity still need model-based evaluation.
• Generic quality scorers can add noise unless tailored to the feature.
• Scorers must be calibrated on examples before running larger online evaluations.
• The process should help teams ship AI features faster without analysis paralysis.

Architecture

Loom evaluates its auto-generated video title feature with Braintrust evals and custom scoring functions. The process starts from the feature, asking how humans judge titles and what great examples look like. The team identifies task-specific traits, checks common quality measures, implements deterministic code-based scorers for objective checks such as exactly one emoji at the end or required JSON keys, and uses LLM-as-a-judge scorers for subjective qualities such as relevance, conciseness, engagement, clarity, and correct language. They calibrate scorers on around 10 to 15 examples, inspect rationales from chain-of-thought judge outputs, refine prompts/scorers/datasets, and then run online evaluations at scale.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  video-input[Video input] --> title-generator[Title generator]
  video-input --> quality-criteria[Quality criteria]
  title-generator --> braintrust-evaluation-run[Braintrust evaluation run]
  quality-criteria --> code-based-scorers[Code based scorers]
  quality-criteria --> llm-judge-scorers[LLM judge scorers]
  code-based-scorers --> braintrust-evaluation-run
  llm-judge-scorers --> braintrust-evaluation-run
  llm-judge-scorers --> judge-rationale[Judge rationale]
  judge-rationale --> scorer-calibration[Scorer calibration]
  braintrust-evaluation-run --> scorer-calibration
  scorer-calibration --> online-evaluations[Online evaluations]

Components

Data flow

The video-title feature receives video input and generates a candidate title. Loom defines feature-specific quality criteria by asking how humans evaluate good titles. Deterministic code-based scorers check objective requirements such as emoji or schema compliance, while LLM-as-a-judge scorers evaluate subjective qualities such as relevance, conciseness, engagement, clarity, and language. The team runs about 10 to 15 examples through Braintrust, reads scorer outputs and rationales, refines prompts, scorers, and datasets, then enables larger online evaluations once the scorers are calibrated.

Why this architecture

Loom separates objective and subjective evaluation because code-based checks are cheaper, faster, and more consistent for deterministic requirements, while LLM judges are better suited for semantic judgments. The architecture is feature-first rather than model-first, so scorers measure whether the output is ideal for users. Small calibration runs prevent teams from scaling unreliable scorers, while online evaluations provide confidence for broader feature rollout.

Models used: LLM for auto-generating video titles · LLM-as-a-judge scorers with chain-of-thought rationales

Scale

• Loom is used by millions of professionals worldwide.
  “Loom is a video communication platform used by millions of professionals worldwide.”article, stated
• Initial scorer calibration uses around 10 to 15 test examples.
  “feeding in around 10-15 test examples to get a feel for how the scorers are performing”article, stated
• Once scorers are calibrated, Loom runs larger online evaluations.
  “they begin running evals at scale by configuring online evaluations.”article, stated
• Scoring granularity can be binary, 3-point, or 5-point depending on the quality dimension.
  “Sometimes a binary (yes/no) score is enough. Other times, a 3- or 5-point scale is necessary for more nuance.”article, stated

Alternatives considered (roads not taken)

Ask whether the model followed instructions

Rejected because: That optimizes for model compliance rather than whether the feature output is ideal for users.

Use LLM judges for objective checks

Rejected because: Deterministic code scorers are cheaper, faster, and less variable when the requirement is objectively checkable.

Apply generic scorers unchanged

Rejected because: Loom tailors scorers to each feature to remove prompt noise and produce more actionable results.

Build a perfect evaluation system before shipping

Rejected because: Loom starts small, iterates quickly, and avoids analysis paralysis.

Patterns

eval-driven-development (quality)

Loom structures AI feature development around custom Braintrust evals, scorer calibration, online evaluations, and a define-implement-evaluate-refine loop.

llm-as-judge (evaluation)

Loom uses LLM-as-a-judge scorers for subjective qualities and enables rationale output to calibrate the judge.

structured-output-contract (validation)

Objective response-format rules such as required JSON keys or a single final emoji are checked with deterministic code.

guardrails-io-filtering (safety)

Common quality measures include hallucination, factuality, bias, toxicity, brand safety, and language checks before outputs reach users.

data-quality-loop (iteration)

Loom refines prompts, scorer definitions, and datasets together until the evaluation captures essential output qualities.

Tradeoffs

Evaluate the feature outcome

Chose User-centered title quality over Prompt instruction-following.

Rationale: Feature quality is about whether the generated title helps users, regardless of how the output was produced.

Use deterministic code scorers where possible

Chose Code-based checks for objective measures over LLM-based judging for every quality dimension.

Rationale: Code scorers are cheap, fast, consistent, and avoid LLM variability.

Use LLM judges for subjective title qualities

Chose Relevance, conciseness, engagement, clarity, and language scorers over Trying to express every quality as deterministic code.

Rationale: Semantic and subjective qualities require judgment beyond straightforward format checks.

Start small before online evaluation

Chose 10 to 15 examples for scorer calibration over Immediately scaling uncalibrated scorers.

Rationale: Small calibration runs let the team inspect and refine scorer behavior before larger production evaluations.

Gotchas (what bit them)

Model-compliance metrics can miss whether the feature is good for users.

Root cause: Instruction-following evaluates the prompt/model relationship rather than the end-user outcome.

Resolution: Frame scorers around how real users evaluate the generated output.

LLM judges are wasteful and variable for objective checks.

Root cause: Checks like emoji placement or JSON keys have deterministic answers and do not need semantic judgment.

Resolution: Use code-based scorers whenever possible.

A multi-aspect scorer hides which quality dimension is failing.

Root cause: One scorer trying to measure factuality, style, and formatting at once produces muddier optimization signals.

Resolution: Make each scorer focus on one distinct aspect of the output.

Scorers can be hard to debug without rationale.

Root cause: A raw judge score does not explain whether the judge measured the intended quality dimension.

Resolution: Enable chain-of-thought rationale for LLM-as-a-judge scorers and use it to calibrate.

Evaluation

Loom's evaluation method identifies traits of great outputs, checks common quality measures, implements deterministic code scorers for objective checks, creates LLM judge scorers for subjective criteria, calibrates them on 10 to 15 examples, then scales through Braintrust online evaluations while iterating prompts, scorers, and datasets.

Metrics: Main idea conveyed · Conciseness and descriptiveness · Engagement potential · Readability and grammar/spelling · Hallucination absence · Single relevant final emoji · Relevance or faithfulness to transcript · Correct language · Weighted averages when some aspects matter more

Outcomes

• Loom established a repeatable evaluation system for shipping AI features faster and more confidently.
• The team can run large-scale evaluations more quickly through Braintrust online evaluations.
• The system combines objective and subjective measures so teams can identify both technical and quality issues.
• The article does not report title acceptance rates, user engagement lift, latency, cost, or production sample size.

Concrete scale includes Loom's millions of professional users, initial calibration on around 10 to 15 examples, and later online evaluations at scale. No production metric values are disclosed.