Synthetic Data Generation for Privacy-Preserving Search Evaluation

canva · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Private search evaluation and synthetic data

Canva needed to evaluate private design search for a massive user base without viewing real user queries or designs, while avoiding slow, low-power manual checks and risky online experiments.

Constraints:

• Private user designs and queries could not be inspected for labeling.
• Offline tests needed reproducible results independent of user state.
• Search evaluation had to reflect production behavior.
• Engineers needed fast local iteration before online experiments.
• Synthetic data had to mimic realistic design length, query difficulty, and corpus behavior.

Architecture

Canva used GPT-4o to synthesize realistic private-search datasets: designs, titles, content, relevant and nonrelevant items, and queries at varying difficulty levels. Engineers run those static datasets through a local Testcontainers replica of the search pipeline, calculate recall and precision, and inspect results in a Streamlit visualization tool.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  aggregate_privacy_statistics["Aggregate privacy statistics"] --> gpt_4o_synthetic_design_generator["GPT-4o synthetic design generator"]
  gpt_4o_synthetic_design_generator --> synthetic_query_generator["Synthetic query generator"]
  gpt_4o_synthetic_design_generator --> synthetic_search_corpus["Synthetic search corpus"]
  synthetic_query_generator --> synthetic_search_corpus
  synthetic_search_corpus --> local_testcontainers_search_pipeline["Local Testcontainers search pipeline"]
  local_testcontainers_search_pipeline --> evaluation_module["Evaluation module"]
  evaluation_module --> streamlit_visualization_tool["Streamlit visualization tool"]

Components

Data flow

GPT-4o generates synthetic designs from realistic topics, sampled design types, titles, and content lengths informed by aggregate privacy-preserving statistics. Queries are generated programmatically and rewritten by GPT-4o, then the local Testcontainers search pipeline indexes the synthetic state and computes evaluation metrics for engineers.

Why this architecture

Canva used LLMs for offline dataset generation rather than runtime search because synthetic static datasets solve privacy and reproducibility needs while avoiding latency, randomness, and API-cost issues during evaluation runs.

Models used: GPT-4o · Search ranking models

Scale

• Canva had 200M monthly active users.
  “In October 2024, Canva celebrated the milestone of 200M monthly active users (MAUs).”article, stated
• Customers had over 30 billion designs and created almost 300 new designs every second.
  “Our customers have over 30 billion designs on Canva and create almost 300 new designs every second.”article, stated
• The offline evaluation runs more than 1000 test cases in under 10 minutes.
  “Our offline evaluation tool outputs results on our more than 1000 test cases in less than 10 minutes.”article, stated
• The local evaluation enables more than 300 offline runs in the time one online experiment would take.
  “achieving 300x faster iteration cycles compared to traditional A/B testing while maintaining strong correlation with online experiment results”summary, stated

Alternatives considered (roads not taken)

Label real user queries and designs

Rejected because: Privacy policy made viewing personal Canva designs strictly off limits.

Rely on a few manual offline queries and online A/B tests

Rejected because: Manual tests had low statistical power and online tests risked exposing users to bad search behavior.

Patterns

synthetic-data-generation (evaluation data)

Canva generated synthetic private-search documents and queries so engineers could evaluate search without seeing user data.

eval-driven-development (quality)

Engineers used local recall and precision evaluations to decide whether search changes should advance to online experiments.

Tradeoffs

Use static synthetic datasets for evaluation.

Chose Generated reusable datasets over Calling an LLM during every evaluation.

Rationale: Static datasets keep evaluation deterministic and fast while avoiding runtime LLM latency and randomness.

Run search locally before online experiments.

Chose Testcontainers local search pipeline over Immediate online A/B testing.

Rationale: Local evaluation protects users from poor search behavior and enables much faster iteration.

Gotchas (what bit them)

Privacy can make standard search evaluation impossible.

Root cause: Private designs and queries could not be inspected to build labeled evaluation datasets.

Resolution: Generate entirely synthetic content and queries and validate offline-online alignment.

LLMs can violate synthetic-data instructions in subtle ways.

Root cause: GPT-4o sometimes refused long titles, repeated or hallucinated misspellings, and missed strict title constraints.

Resolution: Treat generated data as engineering output that must be inspected and refined, not automatically trusted.

Synthetic evaluation is useful only if it correlates with online behavior.

Root cause: Offline metrics can become misleading if synthetic cases do not reflect production search changes.

Resolution: Run experiments comparing offline and online results and track alignment for improvements and regressions.

Evaluation

Canva evaluates search changes with recall and precision over synthetic test cases, visualizes results in Streamlit, and validates offline results against online experiments.

Metrics: Recall and precision. · More than 1000 test cases. · Under 10 minute local evaluation. · Offline-online positive correlation.

Outcomes

• Engineers can evaluate search changes locally with reproducible synthetic datasets.
• Evaluation runs finish in under 10 minutes.
• The workflow enables more than 300 offline evaluations during one online experiment cycle.
• Offline results showed robust alignment with online experiment outcomes.

Canva reports 200M MAUs, over 30B designs, almost 300 new designs per second, more than 1000 synthetic test cases, and under-10-minute local evaluation.