Building and Testing a Production LLM-Powered Quiz Application

google · Education · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM-powered quiz generation

Google's Quizaic team needed to turn a static trivia app constrained by fixed English-only content into a dynamic quiz system that could generate questions, languages, images, and hosted multiplayer sessions on demand, while handling the unreliability, latency, malformed outputs, hallucinations, model drift, and validation burden introduced by generative AI.

Constraints:

• the original Open Trivia Database source had limited categories, questions, formats, language coverage, and no imagery
• LLM outputs are non-deterministic and can be malformed, empty, or inaccurate
• quiz generation and image generation are slower than ordinary API calls
• users need responsive progress indicators during long generation steps
• commercial model safety filters can reject legitimate topics
• quality validation must test semantic correctness, not only JSON syntax
• model upgrades can change behavior and require regression testing

Architecture

Quizaic uses a Flutter client for web/mobile quiz creation and play, a Python Flask API server on Cloud Run, Firestore for document-shaped quiz/session/result data and real-time browser updates, and Vertex AI for Gemini quiz generation plus Imagen topic-image generation. The API prompts Gemini for structured JSON quiz content, post-processes malformed responses, and starts quiz and image generation in parallel. A validator pipeline uses an LLM to judge generated quiz assertions after first measuring validator accuracy on Open Trivia data. Defensive engineering wraps the system with placeholders, retries, fallback imagery, model-version pinning, prompt/version pairing, batching, caching, and regression testing across model changes.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  flutter-client[Flutter client] --> cloud-run-api[Cloud Run API]
  flutter-client --> firestore[Firestore]
  cloud-run-api --> vertex-ai-gemini[Vertex AI Gemini]
  cloud-run-api --> vertex-ai-imagen[Vertex AI Imagen]
  cloud-run-api --> firestore
  vertex-ai-gemini --> json-post-processor[JSON post-processor]
  json-post-processor --> cloud-run-api
  vertex-ai-imagen --> fallback-image-handler[Fallback image handler]
  fallback-image-handler --> cloud-run-api
  cloud-run-api --> validation-pipeline[Validation pipeline]
  validation-pipeline --> vertex-ai-gemini
  validation-pipeline --> open-trivia-benchmark[Open Trivia benchmark]
  session-host[Session host] --> flutter-client
  session-host --> firestore
  player-clients[Player clients] --> flutter-client
  player-clients --> firestore
  model-version-gate[Model version gate] --> vertex-ai-gemini
  model-version-gate --> validation-pipeline

Components

Data flow

A creator chooses a topic, difficulty, question count, language, and generator. The Flutter UI creates a placeholder quiz while the Cloud Run Flask API calls Vertex AI Gemini for quiz JSON and Imagen for a topic image in parallel. Firestore stores quizzes, sessions, participants, and results, then streams real-time updates to the host and players during synchronous play. Post-processing strips non-JSON prefixes or markdown wrapping before parsing. For quality checks, generated multiple-choice questions are decomposed into truth assertions, batched, shuffled, and sent to a validator LLM. The validator's answers are compared with known true/false labels to estimate quiz confidence and can feed background validation or model/version regression reports.

Why this architecture

The architecture keeps ordinary application concerns ordinary: Flutter handles cross-platform UI, Cloud Run hosts containerized services with autoscaling and logs, and Firestore maps naturally to quiz documents while streaming session state. Generative AI is isolated behind Vertex AI calls and defensive parsing because the app cannot trust structured output. Quiz and image generation run independently so image failures do not block play, and validation is separated from interactive generation because users should not wait for a second LLM pass. The validator exists because syntactic checks only prove the quiz parses; correctness requires measuring whether answers are true.

Models used: Palm · Gemini Pro · Gemini Ultra · Gemini 1.5 Flash · Imagen version 1 · Imagen version 2

Scale

• The original 2016 data source had only about 25-30 categories.
  “there was maybe 25 or 30 categories and that was it”transcript, stated
• The Open Trivia Database had on the order of a few thousand questions.
  “on the order of a thousand or more few thousand maybe uh but it's not huge by uh web standards”transcript, stated
• The live demo reached over 100 players, with 104 visible during a question.
  “68 of 104 oh we did get over 100 players that's awesome”transcript, stated
• Players had 15 seconds to answer each question in the demo.
  “I think I gave you each 15 seconds so you're going to run out of time if you haven't responded”transcript, stated
• The team observed model churn from Palm to Gemini Pro, Gemini Ultra, and likely Gemini Flash within roughly one year.
  “just within one year we went from Palm to Gemini um Pro to Gemini Ultra and now we will probably use Gemini flash”transcript, stated
• A prompt that was 98% accurate still failed about 1 in 50 times.
  “even though I got the prompt to be 98% accurate I still need to add the code to to make sure it does the right thing”transcript, stated
• Palm judged Open Trivia answers correctly about 80% of the time, while Gemini Ultra judged about 94% correctly.
  “Palm the model that we use initially uh it was around 80% accuracy”transcript, stated

  “Gemini Ultra the latest one we use now it's it's around 94%”transcript, stated
• Gemini Pro-generated quizzes scored roughly 70% accuracy, while Gemini Ultra-generated quizzes reached 91%.
  “the generator Gemini Pro was roughly 70% accurate in terms of its quizzes”transcript, stated

  “quizzes generated by Gemini Ultra has jumped up to 91%”transcript, stated
• A 10-question multiple-choice quiz becomes 40 validation assertions.
  “for a quiz a 10 question quiz you'd have 40 assertions and you'd probably shuffle them to avoid locality bias”transcript, stated
• The LLM-enabled prototype took about seven weeks after an idea that had been blocked for seven years.
  “what would have taken seven years took me seven weeks or took us s weeks”transcript, stated

Alternatives considered (roads not taken)

Continue with Open Trivia as the content source

Rejected because: It was limited in categories, question count, language, format, images, and required tedious manual curation to expand.

Generate only the quiz before requesting the image

Rejected because: Quiz and image generation are independent slow LLM calls, so starting both at the same time reduces wait time.

Trust prompted JSON output directly

Rejected because: The model sometimes wrapped or prefixed JSON, so parsing needed post-processing and defensive checks.

Use the LLM for free-form answer grading

Rejected because: Traditional fuzzy matching was faster, cheaper, and good enough for the user-interface path.

Use the LLM for logo image editing and profanity filtering

Rejected because: Conventional image editing and standard profanity filters were more appropriate than slow, costly LLM calls.

Always use direct provider SDKs

Rejected because: The team found API churn across Palm, Gemini, and other providers made a higher-level abstraction such as LangChain worth considering.

Patterns

structured-output-contract (interface)

The quiz generator prompts Gemini to return JSON with questions, responses, and correct answers, then pairs that prompt with output parsing logic.

fallback-degradation (resilience)

Image generation can fail without failing the quiz; the app falls back to the Quizaic logo so the core workflow continues.

prompt-versioning (prompt-ops)

Prompts are treated as code and versioned together with the parsing logic that expects their output shape.

eval-driven-development (quality)

The team measures prompt and model changes with validator accuracy, generated-quiz accuracy, and regression test reports before adopting new models.

llm-as-judge (evaluation)

An LLM validator checks generated quiz assertions and produces a confidence signal for quiz accuracy.

batch-offline-processing (throughput)

Validation decomposes quiz questions into assertions and batches them into one API call, and the app batches prompts where possible.

latency-optimization (performance)

The app uses placeholders, progress indicators, background validation, and parallel quiz/image generation to mask slow model calls.

model-routing (model-ops)

The app kept Palm, Gemini Pro, Ultra, Flash, and Imagen variants available for testing and quality comparison as models evolved.

safety-policy-enforcement (safety)

The team explicitly reviews model safety settings because commercial safety filters can block legitimate quiz topics.

Tradeoffs

Dynamic content generation

Chose Gemini-generated quizzes and Imagen-generated images over static Open Trivia content.

Rationale: Generative models removed constraints on topic, language, format, and imagery that made the original app hard to expand.

Image failure handling

Chose fallback image and keep quiz flow over fail the whole quiz when image generation fails.

Rationale: The generated quiz is core functionality; the image improves experience but is not required to play.

Validator latency

Chose background confidence scoring over interactive validation before showing the quiz.

Rationale: A second LLM pass improves confidence but would make users wait, so it should attach scores after generation.

Abstraction layer

Chose consider LangChain-style standardization over provider SDKs everywhere.

Rationale: Direct libraries give more control, but provider and model churn created repeated interface work.

Model upgrades

Chose pin and advance model versions after tests over always use latest model alias.

Rationale: Model changes can improve quality or break behavior, so upgrades should be treated like software dependencies.

Gotchas (what bit them)

A precise prompt still occasionally produced non-JSON output.

Root cause: LLMs can add conversational prefixes or markdown despite explicit JSON-only instructions.

Resolution: Post-process and parse defensively instead of trusting the prompt contract alone.

The same prompt and model can return different answers.

Root cause: LLM calls are non-deterministic and models can also be updated under the same name.

Resolution: Expect inconsistent output, pin model versions, and run tests before adopting new model versions.

More prompt rules sometimes made quiz quality worse.

Root cause: The model did not monotonically improve as the prompt became more verbose.

Resolution: Find the minimal prompt that works, then measure every additional constraint before keeping it.

Multilingual prompting was fragile.

Root cause: Placing the language phrase in the wrong part of the prompt made the model apply it to the topic instead of the quiz.

Resolution: Treat prompt wording and placement as production logic and test generated output in target languages.

Syntactic validation does not prove quiz correctness.

Root cause: A quiz can parse as JSON and still be off-topic, inaccurate, or have wrong answers.

Resolution: Use semantic validation with LLM-as-judge assertions and regression reports.

LLM calls can create user-visible latency and failure modes.

Root cause: Generation calls are slow and may return empty results, false answers, no output, or safety-filter rejections.

Resolution: Use progress UI, retries, background work, fallback images, and safety-setting review.

Evaluation

Quizaic validates at three levels: syntactic checks for parseable JSON, structure checks for expected question/answer counts and image presence, and semantic checks where a validator LLM judges generated quiz assertions. Validator quality is first measured against Open Trivia known answers; generated quizzes are decomposed into true/false assertions, batched, shuffled, and compared with expected labels. The team intends to run background confidence scoring and regression suites across model or app versions.

Metrics: parseable JSON · expected quiz structure · image exists · validator accuracy on Open Trivia · Palm validator accuracy around 80% · Gemini Ultra validator accuracy around 94% · Gemini Pro generated quiz accuracy around 70% · Gemini Ultra generated quiz accuracy around 91% · confidence score · model-version regression report

Outcomes

• Quizaic moved from fixed Open Trivia content to on-demand Gemini-generated quizzes
• the app added topic-specific image generation with Imagen
• Flutter, Cloud Run, Firestore, and Vertex AI formed a production-style two-tier application stack
• Firestore real-time updates enabled synchronous hosted quiz sessions
• multilingual quiz generation became possible by prompt changes, though quality still required testing
• Gemini Ultra-generated quizzes validated at about 91% accuracy compared with about 70% for Gemini Pro
• the team identified production lessons around defensive parsing, model pinning, retries, placeholders, batching, caching, and not overusing LLMs
• a prototype that had been blocked for years became feasible in weeks

The demo ran with more than 100 live participants and 15-second answer windows. The project moved from roughly 25-30 fixed categories and a few thousand static questions to arbitrary generated topics, languages, formats, and imagery. Validation numbers included around 80% Palm answer-judging accuracy, 94% Gemini Ultra judging accuracy, 70% Gemini Pro quiz accuracy, and 91% Gemini Ultra quiz accuracy.