Systematic LLM Evaluation Framework for Content Generation

canva · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM evaluation for creative content generation

Canva needed to evaluate whether LLM-generated transformations of messy design inputs into summaries, emails, songs, and other content were actually good, even though creative outputs lack simple ground-truth accuracy labels.

Constraints:

Generated content is subjective and non-deterministic.
Quality includes information preservation, intent, semantic order, tone, and format.
Prompt changes can improve one metric while regressing another.
LLM-as-judge scores needed validation before being trusted.
The system had to support regression testing for prompt engineering.

Architecture

Canva defined quality criteria, converted them into metrics, implemented rule-based and LLM-as-judge evaluators, scored generated outputs on continuous 0-1 scales, and used ablation checks plus regression testing to guide prompt engineering for Magic Switch-style content transformations.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user_design_input["User design input"] --> llm_content_generator["LLM content generator"]
  user_design_input --> metric_score_modules["Metric score modules"]
  llm_content_generator --> metric_score_modules
  evaluation_criteria_modules["Evaluation criteria modules"] --> metric_score_modules
  metric_score_modules --> rule_based_evaluators["Rule-based evaluators"]
  metric_score_modules --> llm_as_judge_evaluators["LLM-as-judge evaluators"]
  llm_as_judge_evaluators --> regression_testing["Regression testing"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User design input`	Messy multimodal or text-rich design content transformed by Magic Switch.	`LLM content generator`, `Metric score modules`
`LLM content generator`	Produces summaries, emails, songs, or other transformed content.	`User design input`, `Metric score modules`
`Evaluation criteria modules`	Define quality criteria for each output type.	`Metric score modules`
`Metric score modules`	Collect output, output type, and input context for scoring.	`User design input`, `LLM content generator`, `Evaluation criteria modules`, `Rule-based evaluators`, `LLM-as-judge evaluators`
`Rule-based evaluators`	Score objective properties such as format where deterministic checks are suitable.	`Metric score modules`
`LLM-as-judge evaluators`	Score subjective criteria such as tone, intent, semantic order, and information preservation.	`Metric score modules`, `Regression testing`
`Regression testing`	Compares prompt changes across metrics to prevent localized improvements from causing broader regressions.	`LLM-as-judge evaluators`

Data flow

A user design and target output type are passed to an LLM content generator. Evaluation criteria modules define quality dimensions, metric score modules capture the output and input context, rule-based and LLM judges score each metric, and prompt engineers compare metric changes across iterations.

4 supporting quotes

“The solution involved defining quality criteria upfront, converting them into measurable metrics (information score, intent score, semantic order score, tone score, and format score)”summary, stated

“using both rule-based evaluators and LLM-as-a-judge approaches to score outputs.”summary, stated

“Evaluation Criteria Modules: Capture relevant quality criteria for each output type (summary, email, song)”summary, stated

“Evaluation Metric Score Modules: Capture necessary information (output content, output type, LLM input) and score based on criteria instructions”summary, stated

Why this architecture

Canva decomposed quality into multiple measurable criteria because creative LLM outputs are subjective; no single accuracy score can capture whether the output preserves information, follows intent, sounds right, and uses the right format.

3 supporting quotes

“Unlike traditional ML tasks where there's often a ground truth to compare against, LLM-generated creative content (summaries, emails, songs) exists in a subjective space”summary, stated

“LLMs provide creativity and non-deterministic outputs, but don't guarantee correctness, objectivity, or determinism—making evaluation particularly challenging”summary, stated

“ensuring prompt improvements for specific metrics didn't negatively impact other quality dimensions.”summary, stated

Models used: LLM content generator · LLM-as-judge

Scale

LLM judge scores are normalized on a continuous 0-1 scale.
“instructions ask the model to provide a score between 0 and 1, where 0 means criteria are not followed and 1 means all criteria are fully met.”summary, stated

Alternatives considered (roads not taken)

Traditional accuracy-style metrics

Rejected because: Creative generated content does not have a single objective ground truth.

“Unlike traditional ML tasks where there's often a ground truth to compare against, LLM-generated creative content (summaries, emails, songs) exists in a subjective space”summary, stated

Trust LLM judges without validation

Rejected because: The team explicitly ran ablation checks to see whether judge scores dropped when outputs were degraded.

“A critical question in LLM-as-judge approaches is: "Can we trust the scores generated by LLMs?" The team addressed this through ad-hoc ablation score checks”summary, stated

Patterns

`eval-driven-development` (quality)

Canva built explicit LLM output metrics and regression tests to guide prompt iteration.

“The evaluation system enabled effective prompt engineering with regression testing capabilities, ensuring prompt improvements for specific metrics didn't negatively impact other quality dimensions.”summary, stated

`safety-policy-enforcement` (quality constraints)

The evaluation framework encoded format, tone, information, intent, and semantic-order expectations as enforceable scoring criteria.

Tradeoffs

Use LLM judges for subjective criteria.

Chose LLM-as-judge evaluators over Pure rule-based scoring.

Rationale: Subjective qualities such as tone and semantic order require judgment beyond deterministic checks.

“using both rule-based evaluators and LLM-as-a-judge approaches to score outputs.”summary, stated

Score on a continuous 0-1 scale.

Chose Continuous normalized metric scores over Binary pass/fail labels.

Rationale: A continuous score supports aggregation and comparison across prompt iterations.

“The presenter clarified this is a continuous scale (not binary), normalized for easy aggregation and comparison across iterations.”summary, stated

Gotchas (what bit them)

Creative output quality is not reducible to a single correctness label.

Root cause: Summaries, emails, and songs are subjective and non-deterministic outputs.

Resolution: Define multiple quality criteria and score each separately.

“LLM-generated creative content (summaries, emails, songs) exists in a subjective space where traditional accuracy metrics don't apply.”summary, stated

Prompt improvements can regress other quality dimensions.

Root cause: Optimizing one metric can change output behavior along tone, format, intent, or information dimensions.

Resolution: Use regression testing across the metric suite.

LLM-as-judge reliability must be tested.

Root cause: Subjective judges may appear plausible without responding correctly to known degradations.

Resolution: Run ablation checks that deliberately degrade outputs and verify score drops.

“A critical question in LLM-as-judge approaches is: "Can we trust the scores generated by LLMs?" The team addressed this through ad-hoc ablation score checks—deliberately degrading outputs and verifying that scores dropped appropriately:”summary, stated

Evaluation

Canva evaluated generated content with information, intent, semantic order, tone, and format metrics, using rule-based and LLM judges plus ablation validation.

Metrics: Information score. · Intent score. · Semantic order score. · Tone score. · Format score. · 0-1 normalized judge scores.

“converting them into measurable metrics (information score, intent score, semantic order score, tone score, and format score)”summary, stated

“• Format validation: Changed output format to be less desirable, verified format score dropped”summary, stated

Outcomes

Canva created a systematic framework for evaluating creative LLM transformations.
Prompt engineering gained regression testing across quality dimensions.
Ablation checks increased confidence in LLM-as-judge scoring.

The summary describes metric modules and score normalization but does not report production volume for the feature.

“This case study demonstrates a mature approach to LLMOps evaluation that recognizes the unique challenges of creative content generation”summary, stated