Using Evaluation Systems and Inference-Time Scaling for Beautiful, Scannable QR Code Generation

modal · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Generative image quality and QR-code reliability

Modal needed to turn visually impressive QArt-code generation into a production system that reliably produced QR codes that both looked good and scanned.

Constraints:

• The system had competing objectives: aesthetic quality and scan reliability.
• Original demo-style generation worked inconsistently across prompts and scenes.
• Automated evaluation had to align with human scanning and aesthetic judgments.
• Production latency needed to stay under 20 seconds.
• The service-level objective was a 95% scan rate.

Architecture

Modal engineered QArt generation around evaluation and selection. Human scanning of thousands of images seeded ground truth, automated QReader and aesthetic predictors were validated against human judgment, Weave logged experiments, parameter sweeps explored generation settings, and production inference generates eight candidates in parallel. Lightweight online evaluators score scan probability and aesthetic quality, then select the best QR code before returning it under the latency budget.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  prompt-and-url-input["Prompt and URL input"] --> qart-generator["QArt generator"]
  qart-generator --> parallel-candidate-generator["Parallel candidate generator"]
  parallel-candidate-generator --> scan-evaluator["Scan evaluator"]
  parallel-candidate-generator --> aesthetic-evaluator["Aesthetic evaluator"]
  scan-evaluator --> candidate-ranker["Candidate ranker"]
  aesthetic-evaluator --> candidate-ranker
  candidate-ranker --> production-response["Production response"]
  qart-generator --> experiment-logging["Experiment logging"]
  scan-evaluator --> experiment-logging
  aesthetic-evaluator --> experiment-logging

Components

Data flow

A prompt and URL enter the generator, which produces multiple QR-code candidates in parallel. Each candidate is evaluated by scan-detection and aesthetic-scoring models. The service ranks candidates by the validated eval criteria, selects the best scannable/aesthetic result, and returns it to the user, while experiment logs support offline parameter sweeps and metric analysis.

Why this architecture

Modal treated scan rate as the production KPI and used evaluation-driven selection to manage the probabilistic generator. Because evaluation networks were cheaper than generation, running multiple candidates and selecting the best improved reliability without blowing the latency budget.

Models used: ControlNet · Stable Diffusion · QReader · Aesthetic rating predictor

Scale

• Modal manually scanned thousands of QR codes during ground-truth collection.
  “having team members physically scan thousands of QR codes with iPhones while recording results and aesthetic judgments.”summary, stated
• The eval-alignment experiment used approximately 2,000 prompt-URL pairs.
  “The team ran alignment experiments on approximately 2,000 prompt-URL pairs to ensure their automated metrics correlated with human assessments.”summary, stated
• Parameter sweeps generated tens of thousands of images in minutes.
  “they could sweep over configuration parameters by generating tens of thousands of images and calculating performance metrics in minutes”summary, stated
• Production inference generated eight QR candidates per request.
  “By generating eight QR codes in parallel for each request”summary, stated
• The production SLO was a 95% scan rate.
  “Modal's achievement of a 95% scan rate service-level objective while maintaining sub-20-second response times”summary, stated
• The response-time target was under 20 seconds.
  “maintaining sub-20-second response times demonstrates that rigorous LLMOps engineering can bridge the gap between research demos and production systems.”summary, stated

Alternatives considered (roads not taken)

Optimize aesthetics and scanability without a clear priority

Rejected because: Modal prioritized scan rate as the KPI and treated aesthetics as a constraint.

Use manual evaluation for every iteration

Rejected because: Manual scanning created ground truth, but automated evals were needed for scalable parameter sweeps.

Generate one QR code per request

Rejected because: Parallel generation plus evaluation selection improved scan probability with acceptable latency cost.

Patterns

eval-driven-development (evaluation)

Modal used human-aligned automated scan and aesthetic evaluations to drive parameter sweeps and production candidate selection.

llm-as-judge (evaluation)

Although the evaluator was not a general LLM, Modal used learned evaluators as production judges to score generated QR outputs before serving.

batch-offline-processing (optimization)

Modal ran large parameter sweeps over generated images once automated evaluations were trusted.

latency-optimization (serving)

The service used parallel candidate generation and lightweight evaluators to improve reliability while staying under the response-time target.

Tradeoffs

Primary scan-rate objective

Chose Optimize scan rate first while preventing aesthetic regression over Treat scanability and aesthetics as equal objectives.

Rationale: A beautiful QR code that does not scan fails the product's core function.

Inference-time compute scaling

Chose Generate eight candidates and select the best over Return the first generated QR code.

Rationale: Parallel sampling improved scan probability substantially while adding only sublinear latency.

Online evaluation in production

Chose Score generated candidates at request time over Use offline evaluations only for development.

Rationale: Real-time ranking let the production service consistently choose higher-quality outputs.

Gotchas (what bit them)

Viral AI demos did not translate into reliable production behavior.

Root cause: Stylized QR generation was probabilistic and failed across diverse prompts and scenes.

Resolution: Modal built validated automated evaluations and production candidate selection.

Automated evaluators themselves needed evaluation.

Root cause: Scan and aesthetic metrics only help if they correlate with human judgment.

Resolution: Modal ran alignment experiments against human scanning and aesthetic judgments.

Optimizing two objectives without hierarchy can stall production decisions.

Root cause: Aesthetic appeal and scanability can conflict.

Resolution: Modal made scan rate the primary KPI and kept aesthetics as a non-regression constraint.

Evaluation

Modal combined human scanning and aesthetic judgments, QReader scan detection, an aesthetic predictor, alignment experiments, parameter sweeps, Weave experiment tracking, online ranking, and service-level scan-rate/latency metrics.

Metrics: 95% scan-rate SLO · Sub-20-second response time · Approximately 2,000 prompt-URL alignment pairs · Eight generated candidates per request · Scan probability and aesthetic score

Outcomes

• The system reached a 95% scan-rate SLO.
• Responses stayed under 20 seconds.
• Parallel generation and evaluation improved scan probability.
• Automated evaluations enabled parameter sweeps over tens of thousands of images.

The system moved from thousands of manually scanned QR codes and 2,000-pair eval alignment to production inference with eight candidates per request and online scoring.