LLM Testing Framework Using LLMs as Quality Assurance Agents

various · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Conversational AI quality assurance

Alaska Airlines and Bitra needed a scalable way to test customer-facing generative-AI chatbots against happy-path, adversarial, and brand-risk conversations before release, because manual probing found issues but could not cover hundreds of conversational paths or ongoing regressions.

Constraints:

• Customer-facing travel chatbots can be manipulated into offensive, off-brand, or legally risky responses.
• Manual adversarial testing produced useful examples but did not scale beyond a small number of conversations.
• The framework had to support many personas, conversation depths, goals, attitudes, and success criteria.
• The tested chatbot and the testing agent both needed to remain swappable as new LLMs appeared.
• Business users, product managers, QA staff, and engineers all needed ways to inspect or run tests.
• Automated testing still needed human review for low scores, score changes, and sampled successful cases.

Architecture

The team built CARL, an LLM-powered QA Response Liaison that simulates user personas, converses with a model under test, scores the conversation, stores logs and metrics, and routes risky or changing results into human review. The production architecture uses configurable JSON/storage/CI inputs, persona-tuned models in a model registry, Vertex AI endpoints, a conversation coordinator running as containers on GCP Autopilot, Cloud Storage for metrics, AlloyDB plus Vertex AI extensions for text analysis, Cloud Composer smoke tests, and Cloud Build CI/CD integration.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Test configuration interface"] --> F["Conversation coordinator"]
  B["Persona-tuned CARL models"] --> D["Vertex AI endpoints"]
  C["Model registry"] --> B
  C --> D
  D --> F
  E["Model under test"] --> F
  F --> G["Cloud Storage metrics store"]
  F --> H["AlloyDB conversation store"]
  F --> I["Config update module"]
  I --> J["Human review GUI"]
  H --> J
  K["Cloud Composer smoke tests"] --> A
  L["Cloud Build CI"] --> A
  G --> M["Reporting dashboards"]
  H --> M

Components

Data flow

A test configuration supplies tags, topics, depth, personas, goals, attitudes, and success criteria. Persona-tuned or base LLMs are served through Vertex AI or other LLM APIs, the conversation coordinator proxies messages between CARL and the model under test, evaluation criteria score each conversation, logs and metrics go to Cloud Storage and AlloyDB, and low scores, score changes, or sampled successful conversations are surfaced to humans for review and configuration/prompt updates.

Why this architecture

CARL uses an LLM to test another LLM because the team needed scalable conversation generation and insight synthesis across many personas, not a static manual test list. The architecture deliberately abstracts LLM backends, test configuration, orchestration, storage, and review so the framework can run in CI/CD, smoke tests, product workflows, and future model integrations without rebuilding around a single provider API.

Models used: Gemini · Dialogflow with Gemini integration · persona-tuned LLMs · general LLM for summarization

Scale

• The framework was designed to cover hundreds of conversational paths.
  “we needed something that could scale up we needed hundreds of different conversational paths”transcript, stated
• CARL scores every conversation from 1 to 10.
  “rates every conversation on a scale from 1 to 10 and cites any reasons for deductions”transcript, stated
• Smoke tests run every couple of hours.
  “one being our smoke test which is running sort of every couple of hours it's just always running”transcript, stated
• The demo used six-message conversations before evaluation.
  “it's running six messages through each but it takes a little while to run behind the scenes”transcript, stated
• The production effort started with one data scientist, then added an ML engineer and one production engineer.
  “it started with just one started with just me uh and then we eventually brought in an ml engineer to help make it a little bit more robust”transcript, stated

Alternatives considered (roads not taken)

Manual adversarial testing by a developer

Rejected because: Manual probing surfaced examples quickly but could not scale to hundreds of conversation paths or trend analysis.

Build CARL around one LLM API

Rejected because: The LLM landscape changed too quickly, so CARL had to plug into multiple APIs and models on both the tester and model-under-test sides.

Send every conversation to human review

Rejected because: Reviewing everything would defeat the automation goal, so only low scores, score changes, and sampled 10-score conversations go to humans.

Overfit prompt fixes to every discovered failure case

Rejected because: The team looked for common threads across failures and made global system-prompt changes instead.

Patterns

llm-as-judge (evaluation)

CARL uses an LLM-driven evaluator to score chatbot conversations, cite deductions, summarize interactions, and flag whether the model under test behaved acceptably.

eval-driven-development (evaluation)

CARL is wired into smoke tests, CI/CD, score distributions, and regression tracking so chatbot changes are evaluated before beta or customer exposure.

multi-agent-orchestration (architecture)

The system coordinates a tester LLM and a model-under-test LLM through a conversation coordinator, with configurable personas and success criteria driving the interaction.

human-in-the-loop-gate (operations)

Human reviewers inspect low scores, score changes, and sampled high-scoring conversations, while non-technical users can read conversations and flag issues in a GUI.

model-routing (architecture)

The design keeps both the CARL persona model and the target chatbot backend swappable across Gemini, Dialogflow, and other LLMs rather than binding to one API.

Tradeoffs

Use an LLM to test another LLM

Chose A generative-AI testing agent that simulates personas and scores conversations over Manual QA scripts and one-off adversarial conversations.

Rationale: The team needed scaled conversation generation and synthesis across loose human prompts, which is exactly the capability LLMs provide.

Make model interfaces pluggable

Chose Support any LLM for both CARL and the model under test over Optimize only for one provider API.

Rationale: The team expected model APIs and capabilities to change quickly, so lock-in would make the framework obsolete.

Store results in both Cloud Storage and AlloyDB

Chose Cloud Storage for metric artifacts and AlloyDB for text outputs plus Vertex AI extension analysis over A single generic results store.

Rationale: Conversation metadata and text analysis had different access patterns: reporting needed metrics, while text summaries benefited from database-backed LLM extensions.

Sample human review rather than reviewing every run

Chose Route low scores, score changes, and sampled 10-score conversations to humans over Sending all conversations to the human-in-the-loop queue.

Rationale: Reviewing every conversation would undermine the point of automation, but selective review still checks failures and validates apparent successes.

Gotchas (what bit them)

Manual adversarial testing gives a false sense of coverage.

Root cause: A human tester can find a few edge cases quickly, but the space of personas, intents, and conversation paths is much larger.

Resolution: Use CARL to generate hundreds of simulated conversations across goals, attitudes, depths, and success criteria.

Open-ended LLM-to-LLM conversations can run too long and hit rate limits.

Root cause: The prototype originally looped without a depth limit, causing long-running conversations and rate-limit problems.

Resolution: Add a depth parameter controlling how many turns happen before evaluation.

Base LLMs may not simulate nuanced personas well enough for useful QA.

Root cause: Some base models could not carry the desired persona conversations properly.

Resolution: Fine-tune persona models with sample conversations from product teams and customer testing.

Fixing every failure individually can overfit the chatbot prompt.

Root cause: Edge cases often share broader motivational patterns, but treating each failed conversation as a unique patch bloats the system prompt.

Resolution: Look globally across failures and make common-thread prompt changes, such as framing the bot's purpose as a mission it should not deviate from.

Evaluation

CARL evaluates chatbot behavior through configurable simulated conversations, scores each conversation from 1 to 10, stores logs and metrics, tracks score distributions by intent/topic, runs happy-path and bad-path tests through smoke tests and CI/CD, and samples both low-score and high-score cases for human review.

Metrics: Conversation score from 1 to 10 · Score distributions across conversations and intent types · Score changes from smoke-test and CI/CD baselines · Latency and rate-limit hits · NPS lift in beta testing · Human-reviewed low-score and sampled high-score conversations

Outcomes

• CARL caught the travel chatbot adopting pirate speech even though the conversation still produced legitimate travel recommendations.
• The system supported Dialogflow, Dialogflow with Gemini integration, and direct Gemini testing with conversation context.
• The team reported significant NPS lift in beta testing after using CARL.
• The framework became the team's CI/CD-like safety layer for deploying new generative-AI customer experiences.
• Future roadmap items included multilingual, multimodal, third-party call, and Looker dashboard support.

The production design is built for repeated operation rather than a one-time demo: scheduled smoke tests every few hours, CI/CD checks on code and business-logic changes, hundreds of conversation paths, metrics in Cloud Storage, and score-change queries in AlloyDB.