Real-Time Generative AI for Immersive Theater Performance

university_of_california_los_angeles · Media & Entertainment · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Near-real-time generative AI for live immersive performance

UCLA OARC and REMAP needed a live-performance AI media pipeline that could turn audience phone sketches into 2D images and 3D meshes fast enough for a theater show, without failing during performances.

Constraints:

• The system had to support a minimum concurrency of 80 mobile-phone users per performance.
• Mean round-trip time from sketch to media presentation had to stay under 2 minutes.
• Graceful degradation was not acceptable because system failure would directly affect the live audience experience.
• The workload was bursty, with inference surges lasting about 15 minutes rather than a steady always-on pattern.
• Large Python/PyTorch dependencies exceeded Lambda layer limits.

Architecture

UCLA used a serverless-first, event-driven AWS architecture. Mobile sketches came through Firebase and MacOS workstations into SQS, a Lambda helper sorted messages by pipeline, a processing Lambda validated and orchestrated pre-processing, SageMaker/Bedrock inference, and post-processing, and SNS/DynamoDB/SQS delivered results back to on-premises workstations for human review and display on Unreal Engine-powered LED shrines.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Mobile sketch clients"] --> B["Firebase orchestration layer"]
  B --> C["MacOS workstations"]
  C --> D["SQS ingress queues"]
  D --> E["Lambda queue sorter"]
  E --> F["Inference sub-queues"]
  F --> G["Processing Lambda"]
  H["EFS dependency volume"] --> G
  P["CodeBuild deployment"] --> H
  G --> I["SageMaker AI endpoints"]
  G --> J["Amazon Bedrock models"]
  I --> N["S3 asset bucket"]
  N --> G
  G --> K["SNS completion topic"]
  K --> L["DynamoDB analytics table"]
  K --> M["Final asset queue"]
  M --> C
  C --> O["Performance media servers"]

Components

Data flow

Audience sketches and metadata enter through Firebase/on-premises workstations and are posted to SQS. Lambda sorts them into inference-specific sub-queues. A processing Lambda validates the request, manages concurrency, runs pre-processing and inference through SageMaker AI endpoints and Bedrock models, posts completion events to SNS, updates DynamoDB, and places final work on a queue polled by MacOS workstations that retrieve assets for media presentation.

Why this architecture

The architecture fits a bursty live-event workload: queues and Lambda isolate pipelines, SageMaker hosts custom GPU models without managing 20+ EC2 instances, Bedrock reduces operational load for managed foundation models, and human operators retain control through dashboards and review loops.

Models used: DeepSeek VLM · Anthropic Claude 3.5 Sonnet · Amazon Nova Canvas · Stable Diffusion 3.5 · SDXL · SPAR3D · ControlNet · Yamix-8 · CSGO · IP Adapter · InstantID · antelopev2

Scale

• The system supported 7 performances with about 500 total audience members.
  “designed and implemented a solution for 7 performances”article, stated

  “with about 500 total audience members, up to 65 at a time co-creating media during the performance.”article, stated
• The minimum concurrency requirement was 80 mobile-phone users per performance.
  “strict minimum concurrency requirement of 80 mobile phone users for each performance”article, stated
• Mean round-trip time had to be under 2 minutes.
  “The mean round-trip time (MRTT) from mobile phone sketches to media presentation had to be under 2 minutes”article, stated
• The production deployment used 24 SageMaker AI endpoints across 8 g6.12xlarge and 16 g6.4xlarge instances.
  “Inference requests were handled by 24 SageMaker AI Endpoints”article, stated

  “using 8 g6.12xlarge and 16 g6.4xlarge instances.”article, stated
• Average SageMaker processing took 20-30 seconds on g6.12xlarge and 40-60 seconds on g6.4xlarge.
  “ranged from 40-60 seconds on the g6.4xlarge instances, and 20-30 seconds on the g6.12xlarge instances.”article, stated
• The inference surge window lasted about 15 minutes.
  “This surge of requests only lasted 15 minutes”article, stated

Alternatives considered (roads not taken)

Deploy many on-demand EC2 GPU instances

Rejected because: Obtaining enough on-demand instances was challenging and maintaining 20+ hosts created too much burden.

Use EKS for model hosting

Rejected because: The team considered EKS but preferred SageMaker AI because it was faster to configure for this project.

Automated post-processing validation for generated images

Rejected because: The initial deployment relied on human review due to time constraints, while automated guardrail/object-detection validation was deferred.

Manual code deployment to Lambda

Rejected because: CodeBuild automation reduced manual deployment/update risk across environments.

Patterns

tool-use-orchestration (orchestration)

Lambda orchestrates multiple model services and specialized inference pipelines to transform sketches into images, textures, and 3D meshes.

human-in-the-loop-gate (governance)

On-premises workstations and dashboards supported human review and intervention before generated assets reached the stage displays.

cost-aware-routing (operations)

The team selected smaller/larger GPU endpoints by pipeline need and added scheduled endpoint shutdown to control SageMaker costs.

observability-tracing (operations)

The management dashboard surfaced SQS and CloudWatch metrics so operators could see job queues and purge messages.

fallback-degradation (reliability)

The system was designed around fault tolerance and high availability because graceful degradation was not acceptable for the live show.

Tradeoffs

Use SageMaker AI for most custom model hosting.

Chose SageMaker AI endpoints over Managing 20+ EC2 instances or using EKS.

Rationale: SageMaker AI met reliability, configuration, deployment, and load-balancing criteria with less maintenance burden for the live event.

Combine custom SageMaker models with managed Bedrock models.

Chose Hybrid SageMaker AI plus Bedrock inference over All-custom model hosting or all-managed foundation models.

Rationale: The pipeline needed fine-grained control for open-source/custom models and managed serverless access for foundation models.

Generate lower-resolution images before upscaling.

Chose Low-resolution generation plus Bedrock/Stable Diffusion upscaling over Generating final-resolution assets directly on larger endpoints.

Rationale: The staged approach reduced inference time and allowed smaller SageMaker instance types without sacrificing final quality.

Gotchas (what bit them)

Large ML dependencies exceeded Lambda layer limits and caused cold-start latency.

Root cause: PyTorch and related dependencies grew to about 5GB, requiring EFS-mounted dependencies at runtime.

Resolution: The team mounted EFS to Lambda and accepted startup latency rather than changing the development process late in the project.

Manual infrastructure configuration created avoidable deployment error risk.

Root cause: Many AWS services were configured manually during a rapid creative/academic project.

Resolution: The postmortem recommends CloudFormation or similar infrastructure-as-code automation for future iterations.

Idle GPU endpoints can dominate project spend.

Root cause: SageMaker AI endpoints were expensive when left running during rehearsals/development or idle windows.

Resolution: Nightly EventBridge/Lambda shutdowns stopped unused endpoints and improved cost predictability.

Evaluation

The deployment was evaluated against live-performance constraints: concurrency, mean round-trip time, SageMaker processing time, availability during performances, and operational manageability through dashboards and rehearsals.

Metrics: Minimum 80-user concurrency · Mean round-trip time under 2 minutes · SageMaker job initiation to asset-return time · Successful performances · Cost share and endpoint idle cost · Operator visibility into queues/logs

Outcomes

• The system supported seven live Xanadu performances and related rehearsals/playtests.
• Audience and performer sketches were transformed into AI-generated 2D images and 3D meshes for live scenery.
• The architecture met near-real-time live-performance constraints with a serverless/event-driven managed-service stack.
• The team identified concrete phase-2 improvements around infrastructure-as-code, guardrails/object detection, cold starts, and cost optimization.

The deployment supported 7 performances, about 500 total audience members, up to 65 audience members at once plus 12 performers, 80 mobile users required per performance, 24 SageMaker endpoints, 20-60 second SageMaker processing windows, and under-2-minute sketch-to-display MRTT.