LLM Feature Extraction for Content Categorization and Search Query Understanding

canva · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Content discovery feature extraction

Canva needed faster and cheaper feature extraction for search query categorization and content page clustering without building separate bespoke NLP pipelines for every text shape and taxonomy.

Constraints:

• Traditional intent classification required tens of thousands of labels plus training and deployment infrastructure.
• Content pages varied from short metadata to long-form informational articles, requiring many feature extraction methods.
• Jobs ran weekly over hundreds of thousands of inputs, while search-query volume could reach millions per day before aggregation.
• External LLM API costs still had to be managed through aggregation, task selection, and possible open-source alternatives.

Architecture

Canva evaluated LLM APIs and embeddings as a middle-layer feature extractor. For search query categorization, a few-shot prompt with fewer than 10 examples outputs predefined intent classes at inference time, avoiding custom classifier training. For content page categorization, LLM embeddings on plain page text replace scattered keyword extraction, summarization, keypoint extraction, and open embedding combinations, producing semantic representations for clustering pages into topics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Search_query_aggregation["Search query aggregation"] --> Few_shot_intent_prompt["Few-shot intent prompt"]
  Few_shot_intent_prompt --> LLM_API_classifier["LLM API classifier"]
  LLM_API_classifier --> Intent_feature_output["Intent feature output"]
  Intent_feature_output --> Downstream_categorization["Downstream categorization"]
  Content_page_corpus["Content page corpus"] --> LLM_embedding_extractor["LLM embedding extractor"]
  LLM_embedding_extractor --> Clustering_evaluator["Clustering evaluator"]
  Clustering_evaluator --> Downstream_categorization

Components

Data flow

Aggregated search queries or content pages enter the feature-extraction workflow. For query categorization, a few-shot prompt maps text to predefined classes through an LLM API. For content pages, raw text is embedded directly with LLM embeddings, then grouped by semantic similarity into topic clusters. The outputs feed downstream categorization and discovery systems.

Why this architecture

The LLM-as-feature-extractor approach reduced training/deployment overhead and unified scattered text transformation pipelines while improving operational metrics for straightforward text tasks.

Models used: LLM APIs · few-shot learning · LLM embeddings · open-source embeddings · sentence-transformers · BERT embeddings · single-purpose text intent classifier

Scale

• The traditional query classifier took about four weeks to build.
  “four weeks for us to develop this single purpose classifier”transcript, stated
• The LLM API approach took about four to five days end to end.
  “end-to-end development time for weeks and that of lm's apis as well for eight to five days”transcript, stated
• The traditional classifier cost about 100 per month while the LLM API cost less than five dollars per month.
  “single purpose intense classifier it costs about 100 per month while that of um the LMS apis is roughly less than five dollars per month”transcript, stated
• The scheduled job ran weekly with hundreds of thousands of inputs.
  “it is a scheduled jobs running weekly with hundreds and thousands of inputs”transcript, stated
• Search queries were at the magnitude of millions per day before aggregation.
  “our search queries are also at you know at the magnitude of like Millions per day”transcript, stated
• LLM embeddings grouped 89% of content pages into relevant categories in the content-page case study.
  “it grouped 89 of the pages into relevant categories which is the highest among all the other methods”transcript, stated

Alternatives considered (roads not taken)

Single-purpose text intent classifier

Rejected because: It required large labeled data, training infrastructure, deployment infrastructure, four weeks of development, and higher monthly cost.

Multi-step traditional NLP feature extraction for content pages

Rejected because: The pipeline required scattered keyword extraction, summarization, keypoint extraction, and embedding methods.

External LLM API at full raw query volume

Rejected because: For millions-per-day scale, Canva reduced volume through aggregation or considered open-source LLM infrastructure to control external API cost.

Patterns

semantic-caching (cost-control)

Canva aggregated high-volume search queries before LLM processing so repeated or grouped query patterns could be categorized at lower volume.

eval-driven-development (evaluation)

The team compared feature extraction approaches with development time, cost, accuracy, balance, completion, and coherence metrics.

Tradeoffs

Use LLM APIs for straightforward text intent classification.

Chose Few-shot inference over Training a bespoke classifier.

Rationale: The LLM path cut development time and operating cost while improving accuracy in the case study.

Aggregate high-volume queries before LLM processing.

Chose Preprocessing and aggregation over Running the LLM on every raw query.

Rationale: Aggregation reduces millions-per-day query volume before downstream LLM feature extraction.

Gotchas (what bit them)

A simple classifier still carried a full ML lifecycle burden.

Root cause: Traditional text intent classification required labels, training infrastructure, inference development, and deployment.

Resolution: Use few-shot LLM APIs where predefined classes and straightforward text tasks made fine-tuning unnecessary.

Content-page text variation made hand-built feature extraction brittle.

Root cause: Pages ranged from short metadata to long-form articles and required multiple NLP transformation methods.

Resolution: Embed plain page text directly with LLM embeddings and cluster in semantic space.

Evaluation

Canva evaluated query categorization with development time, operating cost, scale, and accuracy; content page categorization with balance, completion, and coherence.

Metrics: development time · operational cost · weekly scheduled job inputs · accuracy · balance · completion · coherence

Outcomes

• The LLM API approach reduced query-intent development from weeks to days.
• The LLM API case lowered operational cost from about 100 per month to less than five dollars per month.
• LLM embeddings on plain text produced the best content-page clustering metrics and grouped 89% of pages into relevant categories.

The query job ran weekly over hundreds of thousands of inputs, while raw query volume could be millions per day before aggregation.