Integrating LLMs and Diffusion Models for Website Design Automation

wix · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Automated website and graphic-design layout generation

Wix had AI tools for website copy and imagery, but professional layout design still required design expertise, time, templates, or human designers. The challenge was to generate unique, personalized layouts that users could control by conditioning on selected component attributes while preserving high-quality graphic-design structure.

Constraints:

• website design requires balancing text, visual content, titles, and relevant visuals
• professional-looking layouts require design expertise and considerable time
• pre-designed templates lack personalization
• real layouts mix continuous attributes such as position and size with discrete attributes such as category
• real-world applications need user interaction and conditioning on partial component attributes
• additional attributes such as color, text style, and content matter in production design
• website creation ultimately needs business functionality beyond layout design

Architecture

Wix developed the Diffusion Layout Transformer (DLT), a conditioned layout-generation framework built around a Transformer encoder and a joint discrete-continuous diffusion process. Layouts are represented as sets of components with category, position, and size attributes. Practitioners can condition generation by fixing component classes, positions, sizes, or other subsets, and the model generates the remaining layout. Training randomly masks components or attributes so the model learns diverse conditioning scenarios. During inference, component embeddings and a timestamp flow through diffusion iterations to output clean coordinates and component classes.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Wix_website_building_workflow["Wix website-building workflow"] --> Wix_AI_Text_Creator["Wix AI Text Creator"]
  Wix_website_building_workflow --> Text_to_image_visual_generator["Text-to-image visual generator"]
  Wix_website_building_workflow --> Diffusion_Layout_Transformer["Diffusion Layout Transformer"]
  Text_to_image_visual_generator --> Diffusion_Layout_Transformer
  Conditioning_inputs["Conditioning inputs"] --> Layout_component_representation["Layout component representation"]
  Layout_component_representation --> Diffusion_Layout_Transformer
  Diffusion_Layout_Transformer --> Transformer_encoder["Transformer encoder"]
  Transformer_encoder --> Joint_diffusion_process["Joint diffusion process"]
  Joint_diffusion_process --> Continuous_attribute_diffuser["Continuous attribute diffuser"]
  Joint_diffusion_process --> Discrete_attribute_diffuser["Discrete attribute diffuser"]
  Joint_diffusion_process --> Combined_loss["Combined loss"]
  Continuous_attribute_diffuser --> Generated_layout_output["Generated layout output"]
  Discrete_attribute_diffuser --> Generated_layout_output
  Generated_layout_output --> Wix_website_building_workflow
  Benchmark_evaluation_datasets["Benchmark evaluation datasets"] --> Diffusion_Layout_Transformer

Components

Data flow

Website-building workflows already use LLM-based text creation and text-to-image models such as DALL-E for content and visuals. DLT adds the layout layer: layouts are represented as component sets with discrete classes and continuous geometry. Training masks some components or attributes and optimizes a combined discrete-continuous loss. At inference time, the user can provide no conditions, component types, sizes, positions, or other subsets; DLT receives component embeddings and a timestamp, runs diffusion iterations through a Transformer encoder, and produces clean component coordinates and classes.

Why this architecture

Wix needed a design generator that could work with partial user intent rather than a fixed template. The Transformer encoder enables non-autoregressive generation and flexible conditioning; the joint discrete-continuous diffusion process matches the mixed nature of layout data; and random masking during training makes the model robust to many conditioning scenarios, such as fixing component classes, positions, or sizes while generating the rest.

Models used: Large Language Models for Wix AI Text Creator · DALL-E-style text-to-image models · Diffusion Layout Transformer · Transformer encoder · joint discrete-continuous diffusion model

Scale

• DLT was presented at ICCV 2023.
  “presented at ICCV 2023, we introduced the Diffusion Layout Transformer (DLT)”article, stated
• Wix describes three common conditioning settings in the results section.
  “let’s look at the diffusion process output using three common conditioning settings.”article, stated
• The evaluation used PubLayNet, RICO, and Magazine benchmark datasets.
  “including annotated document images (PubLayNet), Android UI screens (RICO), and digital images (Magazine).”article, stated

Alternatives considered (roads not taken)

Pre-designed templates

Rejected because: Templates are suitable suggestions but do not create unique, highly personalized layouts.

Professional designer for every custom layout

Rejected because: Professional design remains valuable, but it adds time and expertise requirements that Wix wanted to reduce for users.

Layout methods that model only continuous or only discrete attributes

Rejected because: Real layouts include both geometric and categorical attributes, so DLT models both jointly.

Sequential autoregressive component generation

Rejected because: Wix chose a Transformer encoder that can generate components non-autoregressively and support flexible inference conditioning.

Patterns

structured-output-contract (interface)

DLT represents layouts as structured component sets with category, position, and size attributes, then outputs clean coordinates and classes.

eval-driven-development (quality)

Wix evaluated generated layouts on multiple benchmark datasets and design aesthetic metrics before reporting DLT's performance.

tool-use-orchestration (workflow)

Wix combines LLM text generation, text-to-image models, and layout generation as separate AI tools in the website-building workflow.

latency-optimization (modeling)

The Transformer encoder makes layout generation non-autoregressive, allowing the model to generate all components without sequential dependency.

Tradeoffs

Build custom layout generation rather than template recommendation.

Chose conditioned Diffusion Layout Transformer over suggesting suitable layout templates.

Rationale: The custom model broadens creative options by generating unique and personalized layouts.

Use a Transformer encoder.

Chose non-autoregressive Transformer encoder with flexible conditioning over previous layout-generation methods with less flexible conditioning.

Rationale: The encoder supports simultaneous component generation and inference-time conditioning.

Model discrete and continuous attributes jointly.

Chose joint discrete-continuous diffusion process over approaches using either continuous or discrete attributes alone.

Rationale: Real layouts combine geometric coordinates with component categories, so one unified generative process better matches the data.

Train with random masking.

Chose hide components or specific attributes during training over training only on complete unconditioned layouts.

Rationale: Random masking teaches robustness across diverse conditioning scenarios users will create at inference time.

Gotchas (what bit them)

Text and image generation did not solve layout design.

Root cause: Professional website layout requires balancing visual and textual elements, which remains a design-expertise problem.

Resolution: Wix built DLT specifically for custom conditioned layout generation.

Templates limit personalization.

Root cause: Pre-designed templates constrain users to existing layout options rather than generating unique designs.

Resolution: DLT aims to create unique, highly personalized layouts instead of merely suggesting templates.

Layout data mixes incompatible attribute types.

Root cause: Positions and sizes are continuous, while component categories are discrete.

Resolution: DLT uses a joint discrete-continuous diffusion process with a combined loss.

Real design tools need partial user control.

Root cause: Users often know some attributes, such as a button class or image position, but need the model to complete the rest.

Resolution: DLT supports conditioning on any subset of component attributes.

Layout generation alone is not a full website product.

Root cause: Website creation also requires SEO, analytics, payments, and other business functionality.

Resolution: Wix frames layout generation as one part of a broader website-creation system.

Evaluation

Wix evaluated DLT on layout datasets for different graphic-design tasks, including PubLayNet, RICO, and Magazine, using common design aesthetic metrics. The reported result is that DLT outperformed existing approaches on layout synthesis and editing while maintaining computational complexity comparable to previous methods.

Metrics: common design esthetic metrics · layout synthesis performance · layout editing performance · computational complexity compared with prior approaches · PubLayNet benchmark · RICO benchmark · Magazine benchmark

Outcomes

• Wix introduced the Diffusion Layout Transformer for conditioned layout generation.
• DLT supports unconditioned generation, conditioning on component classes, conditioning on sizes, and partial layout completion.
• The model outperformed existing solutions on layout synthesis and editing tasks in the reported evaluation.
• DLT maintained computational complexity on par with previous approaches.
• The framework is positioned as applicable beyond websites to mobile UI, slides, magazines, papers, infographics, indoor scenes, and other graphic-design layouts.

The source reports ICCV 2023 publication, three common conditioning settings, and evaluation on PubLayNet, RICO, and Magazine. It does not report production rollout volume, latency, user adoption, or online business metrics for DLT.