Building and Scaling AI-Powered Visual Search Infrastructure

figma · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Design search infrastructure

Figma needed users to find visually or semantically similar designs and components across large teams and organizations, even when frames were unlabeled or component names did not contain the user's keywords.

Constraints:

• search must work across designs, published components, screenshots, layer selections, and text descriptions
• private Figma files and customer data could not be used for embedding-model training
• unpublished frames inside Figma files were not readily enumerable through existing APIs
• early beta users still required their whole teams' data to be indexed for the product to work
• the system had to generate and search billions of embeddings without runaway compute and OpenSearch cost
• vector search had to coexist with existing lexical search expectations

Architecture

Figma built AI-powered search around CLIP-style multimodal embeddings. Server-side asynchronous jobs run a headless C++ Figma editor to identify indexable frames, persist metadata and embeddings in DynamoDB, render thumbnails to S3, generate batched embeddings through AWS SageMaker, and write vectors plus metadata into OpenSearch. Search queries from text, screenshots, or Figma selections are embedded on demand and matched against the vector index. Component search runs lexical and embedding searches in parallel, normalizes and boosts scores, and interleaves results. Scaling work focused on C++ preprocessing, CPU thumbnail rendering, four-hour indexing debouncing, autoscaling, index pruning, vector quantization, and OpenSearch bug fixes.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  search-ui[Search UI] --> query-embedding-generator[Query embedding generator]
  query-embedding-generator --> clip-embedding-model[CLIP embedding model]
  query-embedding-generator --> opensearch-vector-index[OpenSearch vector index]
  frame-identification-job[Frame identification job] --> dynamodb-metadata-store[DynamoDB metadata store]
  frame-identification-job --> thumbnail-renderer[Thumbnail renderer]
  thumbnail-renderer --> s3-thumbnail-store[S3 thumbnail store]
  s3-thumbnail-store --> sagemaker-embedding-endpoint[SageMaker embedding endpoint]
  clip-embedding-model --> sagemaker-embedding-endpoint
  sagemaker-embedding-endpoint --> opensearch-vector-index
  dynamodb-metadata-store --> opensearch-vector-index
  opensearch-vector-index --> hybrid-result-ranker[Hybrid result ranker]
  lexical-component-index[Lexical component index] --> hybrid-result-ranker
  hybrid-result-ranker --> search-ui
  indexing-scheduler[Indexing scheduler] --> frame-identification-job
  indexing-scheduler --> opensearch-vector-index

Components

Data flow

For indexing, an asynchronous job uses the server-side C++ editor to enumerate searchable frames in a Figma file, persists frame metadata to DynamoDB, renders thumbnails to S3, and queues embedding generation. SageMaker receives batches of thumbnail URLs, downloads and normalizes images in parallel, and emits embeddings that are persisted and indexed in OpenSearch with file, project, team, and organization metadata. At query time, Figma converts a text prompt, screenshot, or layer selection into an embedding and performs nearest-neighbor vector search; component search also runs fuzzy lexical search and interleaves normalized results.

Why this architecture

Figma chose multimodal embeddings because users can express visual intent through words, screenshots, or Figma selections, and CLIP maps those modalities into one vector space. The pipeline is split into asynchronous stages because frame enumeration, thumbnailing, embedding generation, and indexing need different batching and retry controls. OpenSearch was reused because it already powered Figma search, while hybrid ranking preserved exact lexical expectations. Cost controls were built into preprocessing and index freshness because indexing billions of entries made preprocessing and OpenSearch footprint more expensive than the model alone.

Models used: CLIP · fine-tuned CLIP variant for component search

Scale

• The AI search system must generate embeddings for and index billions of entries.
  “Enabling AI-powered search at Figma’s scale requires generating embeddings for and indexing billions of entries.”article, stated
• Indexing freshness was debounced to at most every four hours.
  “if we debounced indexing to be at most every four hours, we would only have to process 12% of the data!”article, stated
• The four-hour debounce reduced processing volume to 12% of the original data.
  “if we debounced indexing to be at most every four hours, we would only have to process 12% of the data!”article, stated
• Removing draft files, duplicates, and unchanged copies cut the vector index in half.
  “We ended up removing draft files, duplicate designs within files, and files that were copies without new changes, and this cut the index in half.”article, stated
• OpenSearch's default kNN representation stores each embedding dimension as a four-byte float.
  “By default, OpenSearch’s kNN plugin represents each element of the embedding as a four byte float”article, stated

Alternatives considered (roads not taken)

Textual JSON representation of a Figma selection

Rejected because: Image embeddings returned better results and shared the screenshot-search code path.

RDS for AI search metadata

Rejected because: The feature needed high-throughput key-value reads/writes, not transactions or relational constraints.

GPU thumbnail rendering on older AWS instances

Rejected because: CPU software rendering with llvmpipe on newer instances was cheaper and faster for this workload.

Trigger indexing on every file change

Rejected because: Frequent editing sessions made immediate reindexing wasteful; four-hour debouncing preserved freshness while cutting work.

Pure semantic component search replacing lexical search

Rejected because: Figma retained valuable exact and fuzzy string matches and interleaved them with semantic results.

Patterns

hybrid-search (retrieval)

Component search runs lexical and embedding indexes in parallel, normalizes scores, boosts exact lexical matches, and interleaves results.

retrieval-ranking (search)

Figma uses nearest-neighbor vector search over embeddings with faceted metadata and a score normalization layer for hybrid results.

batch-offline-processing (indexing)

Frame identification, thumbnailing, embedding generation, and search indexing are separate asynchronous jobs with batched SageMaker inference.

privacy-preserving-processing (data-governance)

The embedding models were not trained on private customer files; fine-tuning used public Community UI content.

cost-aware-routing (operations)

The indexing pipeline reduces work through debounce windows, C++ preprocessing, CPU rendering, autoscaling, index pruning, and vector quantization.

Tradeoffs

Model input representation

Chose image-based embeddings for selections over textual JSON representations of selected layers.

Rationale: Image embeddings produced better results and reused the screenshot search path.

Vector infrastructure

Chose OpenSearch over introducing a new specialized vector database.

Rationale: OpenSearch was already widely deployed at Figma for search, making it operationally attractive despite later kNN quirks.

Search-result strategy

Chose hybrid lexical plus semantic search over semantic-only component search.

Rationale: Hybrid search kept valuable exact and fuzzy lexical results while adding semantic understanding.

Index freshness

Chose debounced indexing at most every four hours over reindexing on every file change.

Rationale: Users make many edits in a session, so fast refreshes had little value compared with the processing cost.

Index footprint

Chose vector quantization over full four-byte float embeddings everywhere.

Rationale: Quantization reduces memory required for storage and search while accepting a small nearest-neighbor accuracy reduction.

Gotchas (what bit them)

The expensive part of embedding search was not necessarily model inference.

Root cause: Identifying meaningful designs inside Figma files and rendering thumbnails dominated compute costs.

Resolution: Rewrite identification from Ruby to C++, remove JSON serialization, and move thumbnailing to cheaper CPU software rendering.

A tiny beta cohort could still require nearly all teams to be indexed.

Root cause: For any one user to experience search correctly, the user's entire team data had to be indexed, and Figma has many small teams.

Resolution: Make indexing and backfilling cost-efficient before broad rollout through preprocessing and index-size optimizations.

Batching inference past a threshold increased latency linearly.

Root cause: The SageMaker embedding endpoint did not keep delivering sublinear batching benefits at larger batch sizes.

Resolution: Experiment to find the optimal batch size and parallelize image download, resizing, and normalization.

Replica shards returned nondeterministic vector results.

Root cause: OpenSearch replica queries hit a type-casting error in the delete path on clusters using segment replication.

Resolution: Partner with AWS OpenSearch to diagnose the bug and ship an upstream fix.

Removing embeddings from OpenSearch _source broke metadata updates.

Root cause: OpenSearch used _source to apply document updates, so updated documents lost embeddings that had been removed from _source.

Resolution: Re-fetch embeddings from DynamoDB on updates while keeping the _source optimization for normal query responses.

Evaluation

Figma relied on end-to-end testing, batch-size experiments, cost/latency analysis, and debugging of distributed OpenSearch behavior. The source emphasizes operational testing more than offline relevance metrics.

Metrics: embedding batch latency · indexing data volume · OpenSearch result determinism · index size · compute cost · query latency

Outcomes

• Figma launched AI-powered Search for designs and improved Search for components in early beta
• users can search by text description, screenshot, or layer selection
• component search gained semantic matching while retaining lexical search
• four-hour debounce reduced processing to 12% of original volume
• index-pruning cut the vector index in half
• Figma and AWS resolved an OpenSearch kNN replica nondeterminism bug upstream

The system was built for billions of embeddings and organization/team-wide indexing, with continued rollout planned after Config 2024.