Production-Scale Generative AI Infrastructure for Game Art Creation

playtika · Media & Entertainment · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production generative image infrastructure for game art and marketing assets

Playtika needed an internal generative art platform that could fine-tune studio-specific diffusion models, serve multiple image-generation actions with low latency and stable GPU utilization, and reduce art production time without requiring R&D involvement for every deployment.

Constraints:

• Each game studio and asset category has distinct visual style requirements.
• DreamBooth produced better quality initially but created deployment proliferation and large model-weight footprints.
• Diffusion serving requires GPUs, large NVIDIA dependencies, and careful container/image optimization.
• GPU utilization metrics can be misleading unless Tensor Core and DCGM-level metrics are monitored.

Architecture

Playtika built a use-case factory for generative art. Internal users fine-tune and evaluate models through a training UI, tag best outputs with studio clients, register chosen configurations, and deploy models for production image generation. The platform evolved from DreamBooth-per-use-case deployments on Stable Diffusion 1.5 to SDXL plus LoRA adapters, with a shared base model, lightweight per-use-case add-ons, deferred model loading from a model registry, OneFlow acceleration, FP16 memory reductions, DCGM monitoring, and MIG GPU partitioning.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  studio-art-request["Studio art request"] --> training-ui["Training UI"]
  training-ui["Training UI"] --> fine-tuning-pipeline["Fine-tuning pipeline"]
  training-ui["Training UI"] --> studio-client-review["Studio client review"]
  fine-tuning-pipeline["Fine-tuning pipeline"] --> sdxl-base-model["SDXL base model"]
  fine-tuning-pipeline["Fine-tuning pipeline"] --> lora-adapters["LoRA adapters"]
  fine-tuning-pipeline["Fine-tuning pipeline"] --> model-registry["Model registry"]
  studio-client-review["Studio client review"] --> model-registry["Model registry"]
  sdxl-base-model["SDXL base model"] --> inference-service["Inference service"]
  lora-adapters["LoRA adapters"] --> inference-service["Inference service"]
  model-registry["Model registry"] --> inference-service["Inference service"]
  inference-service["Inference service"] --> oneflow-acceleration["OneFlow acceleration"]
  inference-service["Inference service"] --> mig-gpu-partition["MIG GPU partition"]
  oneflow-acceleration["OneFlow acceleration"] --> dcgm-monitoring["DCGM monitoring"]
  mig-gpu-partition["MIG GPU partition"] --> dcgm-monitoring["DCGM monitoring"]

Components

Data flow

A studio defines an art use case such as character, background, or element generation. The training UI generates captions, lets users tune parameters, runs fine-tuning, and supports inference testing. Studio clients tag preferred outputs, and the best model configuration is registered. At inference time, a service loads the SDXL base model and requested LoRA adapter from the model registry, performs text-to-image, image-to-image, or inpainting, and runs on optimized GPU partitions monitored through DCGM metrics.

Why this architecture

Playtika's architecture follows the tension between creative quality and serving efficiency. DreamBooth won initial quality tests but multiplied heavy deployments across studios, categories, and actions. SDXL plus LoRA later met the quality bar while shrinking per-use-case artifacts and allowing adapter switching. GPU observability, acceleration, FP16, and MIG then addressed the remaining serving bottlenecks.

Models used: Stable Diffusion 1.5 · SDXL 1.0 · DreamBooth · Textual Inversion · LoRA · SDXL Turbo · LoRA LCM · OneFlow · TensorRT · AITemplate

Scale

• The business objective was a 50% reduction in art production time.
  “The business objective was ambitious: achieve a 50% reduction in art production time while simultaneously increasing the volume of art assets”summary, stated
• Generation clicks produce a fixed batch of 4 images.
  “In their use case, batch size was fixed at 4 (each "Generate" click produces 4 images).”summary, stated
• OneFlow reduced latency by 40%.
  “achieving a 40% reduction in latency time.”summary, stated
• nvidia-smi showed 100% GPU utilization while Tensor Core utilization was only 45.5%.
  “GPU utilization appeared at 100% while Tensor Core utilization was only 45.5%”summary, stated
• The best MIG setup used two A100 slices: MIG 3g.20gb and MIG 4g.20gb.
  “The most effective configuration involved two slices: MIG 3g.20gb and MIG 4g.20gb.”summary, stated

Alternatives considered (roads not taken)

Generalized models across many studio use cases

Rejected because: Specific models per use case produced significantly better quality.

DreamBooth-per-use-case production deployments

Rejected because: It created many gigabyte-scale deployments across studios, categories, and actions with poor GPU utilization.

Bake models into Docker images

Rejected because: Generative model images already require gigabytes of dependencies, and baking models would further slow scaling and deployment.

Use nvidia-smi as the main utilization signal

Rejected because: It reported busy-kernel time rather than actual computational utilization.

Use NVIDIA MPS for GPU sharing

Rejected because: The MPS server crashed frequently and caused quality-of-service degradation.

Patterns

fine-tuning-adaptation (model-adaptation)

Playtika fine-tunes diffusion models and later LoRA adapters for studio-specific art styles and use cases.

model-routing (architecture)

The system selects action-specific services and request-time LoRA adapters rather than deploying a separate full model for every combination.

cost-aware-routing (operations)

GPU partitioning, FP16, LoRA switching, and lean image construction are used to reduce resource waste and serve more users.

observability-tracing (operations)

The team moved from nvidia-smi to DCGM metrics to understand real GPU and Tensor Core utilization.

human-in-the-loop-gate (quality)

Studio clients tag the most accurate and relevant outputs before configurations are registered for production.

Tradeoffs

Start with DreamBooth for quality

Chose DreamBooth full model fine-tuning over LoRA or Textual Inversion in the initial Stable Diffusion 1.5 era.

Rationale: DreamBooth produced the best image quality even though it created scaling pain.

Switch to SDXL plus LoRA

Chose One base model with many compact adapters over Many full DreamBooth deployments.

Rationale: The newer base model made LoRA good enough while dramatically improving model management and service utilization.

Select OneFlow for acceleration

Chose OneFlow over TensorRT, AITemplate, LoRA LCM, or SDXL Turbo.

Rationale: OneFlow met support requirements while preserving quality better than other acceleration options they evaluated.

Use MIG despite small latency cost

Chose A100 MIG partitions over Unpartitioned or MPS-managed GPU sharing.

Rationale: MIG served more users reliably, while MPS crashed and unpartitioned GPUs left concurrency/utilization headroom unused.

Gotchas (what bit them)

Quality-first fine-tuning can create deployment explosion.

Root cause: DreamBooth created full model weights for each studio/category/action combination.

Resolution: Move to SDXL plus LoRA once LoRA met the production quality bar.

Large generative-AI containers slow scale-up.

Root cause: GPU serving requires NVIDIA dependencies and large base images before model weights are even considered.

Resolution: Use a lean CUDA runtime image, deferred model loading, and a mounted model registry.

GPU utilization dashboards can lie.

Root cause: nvidia-smi reports busy kernel time, not actual computational use or Tensor Core saturation.

Resolution: Use DCGM metrics through dcgm-exporter and optimize against Tensor Core/utilization gaps.

MPS GPU sharing harmed quality of service.

Root cause: The MPS server frequently crashed and the root cause was not identified.

Resolution: Discontinue MPS and use MIG partitioning, accepting small latency tradeoffs.

Evaluation

Playtika evaluates generated art through studio-client tagging of outputs, load testing for throughput and latency, DCGM/Tensor Core monitoring for GPU efficiency, and quality checks when considering acceleration or fine-tuning changes.

Metrics: studio output quality tags · throughput · latency · GPU utilization · Tensor Core utilization · batch size 4 generation · latency reduction · quality-of-service degradation

Outcomes

• The platform enables multiple game studios to produce text-to-image, image-to-image, sketch-to-image, and inpainting assets using internal fine-tuned models.
• SDXL plus LoRA reduced deployment proliferation by keeping one base model with many compact adapters.
• OneFlow reduced latency by 40%.
• MIG partitioning allowed more users to be served with only a few additional seconds of latency.

The case states a 50% art-production-time goal rather than a proven outcome. Concrete technical scale anchors include batch size 4, 40% latency reduction, 100% vs 45.5% GPU/Tensor Core utilization mismatch, and MIG 3g.20gb/4g.20gb A100 partitioning.