Lessons from Red Teaming 100+ Generative AI Products

microsoft · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Generative AI red teaming and security evaluation

Microsoft's AI Red Team needed a practical operating model for probing safety and security risks across many generative AI products. The work had to cover both adversarial and benign actors, security and responsible-AI harms, model and system-level weaknesses, automation at scale, and human judgment for novel or subjective harms.

Constraints:

• the field of AI red teaming is nascent and operational questions remain open
• testing must cover complete systems, not just standalone models
• simple prompt-engineering attacks may be more realistic than gradient-based attacks
• benchmarks cannot discover all novel harm categories
• automation expands coverage but does not replace expert human review
• responsible-AI harms can be probabilistic and subjective
• securing AI systems is an ongoing process rather than a one-time fix

Architecture

Microsoft AIRT formalized an internal threat model ontology covering systems, actors, tactics/techniques/procedures, weaknesses, and impacts, then applied it through red-team operations across generative AI products. PyRIT provides automated prompt datasets, converters, attack strategies, and scorers, while human experts handle domain-specific, cultural, emotional, and novel-harm judgments. The operating model combines system-level attack thinking, automation, case studies, break-fix cycles, and purple-team style improvement.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  system["System under test"] --> ontology["Threat model ontology"]
  ontology --> operation["Red-team operation"]
  operation --> pyrit["PyRIT automation"]
  pyrit --> attacks["Automated attack strategies"]
  attacks --> system
  pyrit --> scorers["Output scorers"]
  scorers --> humans["Human expert review"]
  humans --> mitigation["Mitigation cycle"]
  ontology --> impacts["Impact taxonomy"]
  impacts --> mitigation
  mitigation --> operation

Components

Data flow

A red-team operation starts by defining the system under test and actor being emulated, including malicious and benign users. The team chooses tactics and techniques, probes for weaknesses, and records impacts across security and safety categories. Automated PyRIT workflows can generate transformed prompts, run attack strategies, and score outputs across more risk areas than manual testing alone. Human red teamers and subject-matter experts then judge ambiguous harms, assess context, and feed findings into mitigations and repeated break-fix cycles.

Why this architecture

The ontology makes red teaming repeatable across systems, actors, weaknesses, and impacts. PyRIT extends coverage because fully manual testing cannot span the risk landscape. Human review remains necessary because novel harms, cultural context, medical/cyber/CBRN content, and distress scenarios require judgment that benchmark scores alone cannot provide. Break-fix cycles recognize that defenses raise attack cost rather than permanently solving AI security.

Models used: generative AI products · LLMs · PyRIT · uncensored models for automated red teaming · multimodal output scorers

Scale

• The findings are based on red teaming over 100 generative AI products at Microsoft.
  “Based on our experience red teaming over 100 generative AI products at Microsoft”article, stated
• Microsoft's AI Red Team was established in 2018.
  “The team was officially established in 2018 and has evolved significantly as AI capabilities have expanded”summary, stated
• Since 2021, AIRT conducted over 80 operations covering more than 100 products.
  “Since 2021, AIRT has conducted over 80 operations covering more than 100 products.”summary, stated
• The work spans standalone models and integrated systems such as copilots, plugins, and AI applications.
  “Products tested include both standalone models and integrated systems (copilots, plugins, AI applications).”summary, stated

Alternatives considered (roads not taken)

Treat safety benchmarking as sufficient

Rejected because: The paper distinguishes red teaming from benchmarks because red teaming explores unfamiliar scenarios and novel harms.

Prioritize gradient-based adversarial methods

Rejected because: The paper says real attackers prompt engineer, and simple jailbreaks can be more practical than gradient attacks.

Test only the base model

Rejected because: A system-level adversarial mindset is required because deployed products include infrastructure, filters, databases, plugins, and cloud resources.

Rely on automation without human judgment

Rejected because: Automation improves coverage, but subject-matter, cultural, and emotional judgments remain crucial.

Patterns

eval-driven-development (security-testing)

Microsoft AIRT uses red-team operations, threat models, case studies, and break-fix cycles to evaluate AI systems against real risks.

safety-policy-enforcement (security)

The red-team ontology and operations examine security impacts, responsible-AI harms, jailbreaks, prompt injection, data exfiltration, and unsafe content.

tool-use-orchestration (red-team-automation)

PyRIT orchestrates prompt datasets, converters, attack strategies, and output scorers to automate portions of AI red teaming.

human-in-the-loop-gate (review)

Human experts judge specialized domains, cultural context, distress scenarios, and novel harms that automated tools cannot fully assess.

observability-tracing (security-operations)

The ontology records actors, techniques, weaknesses, and impacts so findings can be communicated and mitigated systematically.

Tradeoffs

Red-team coverage strategy

Chose automation plus expert review over fully manual testing or fully automated scoring.

Rationale: Automation covers more risk surface, while humans remain necessary for novel and contextual harm judgments.

Testing focus

Chose system-level adversarial testing over model-only vulnerability testing.

Rationale: Integrated products include filters, infrastructure, databases, plugins, and cloud resources that create attack paths beyond the model.

Security goal

Chose raise attacker cost through defense-in-depth and repeated mitigation over treating AI safety as a solved technical problem.

Rationale: The paper argues that sufficiently long prompts can elicit non-zero-probability outputs, so practical defense is continuous improvement.

Gotchas (what bit them)

Academic attacks can distract from the attacks real users attempt.

Root cause: Gradient methods often require model access and compute, while practical attackers use prompts and simple transformations.

Resolution: Prioritize realistic, simple jailbreaks and system-level attacks in red-team coverage.

RAG and plugins create cross-prompt injection paths.

Root cause: Untrusted retrieved content can carry hidden instructions that alter model behavior or exfiltrate data.

Resolution: Combine system-level mitigations, input sanitization, instruction hierarchy, and an assumption that untrusted input can produce arbitrary output.

Responsible-AI harms are hard to measure like conventional bugs.

Root cause: Outputs are probabilistic, causal explanations are limited, and harm definitions are subjective and policy-dependent.

Resolution: Use human review, policy detail, and culturally aware evaluation in addition to automated scoring.

Traditional appsec still matters inside AI products.

Root cause: AI systems are deployed as software systems with dependencies, transmission methods, video processing components, and APIs.

Resolution: Continue traditional security testing alongside AI-specific red teaming.

Evaluation

Evaluation is red-team based: define system and actor, select tactics and techniques, use PyRIT automation for prompt transformations and scorers, review impacts with human experts, and repeat break-fix mitigation cycles.

Metrics: number of products red-teamed · number of operations · security impact categories · safety impact categories · successful jailbreak/prompt-injection paths · automated scorer outcomes · human expert harm assessment

Outcomes

• AIRT reports red-team experience across more than 100 generative AI products
• the paper contributes an internal threat model ontology and eight lessons
• PyRIT is identified as an open-source red-teaming automation framework
• the core conclusion is that AI security requires continuous human-plus-automation work rather than one-time benchmarking

The work spans more than 100 Microsoft generative AI products and over 80 operations since 2021, according to the project summary.