Red Teaming AI Agents: Uncovering Security Vulnerabilities in Production Systems

casco · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI agent security and red teaming

Casco tested live production AI agents and found that developers were treating agent security as mostly prompt-injection or model-safety work, while real compromises came from ordinary web and infrastructure flaws exposed through agent tools.

Constraints:

• The assessment targeted already-live agents with only 30 minutes allocated per agent.
• Agents typically connected a server front end, API server, LLM, tools, and data sources.
• Tool definitions and system prompts exposed what agents could access, execute, or retrieve.
• Vulnerabilities could sit downstream of the LLM in authorization, sandboxing, credentials, and network boundaries.
• Security recommendations had to translate traditional web security controls into agent architectures.

Architecture

Casco's red-team method first extracted system prompts and tool definitions, then inverted the stated constraints to test what the agent could actually access. The assessment found three recurring failure classes: cross-user data access through IDOR, arbitrary code execution and lateral movement through weak sandboxes, and SSRF-style credential theft from private repositories. The recommended architecture treats agents as users, applies external authorization, sanitizes inputs/outputs, and uses real code sandboxes rather than custom containers.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Production agent front end"] --> B["API server"]
  A --> C["LLM runtime"]
  C --> D["System prompt"]
  C --> E["Tool definitions"]
  B --> E
  B --> F["Authorization layer"]
  B --> G["Data sources"]
  F --> G
  E --> G
  E --> H["Code execution sandbox"]
  H --> I["Infrastructure metadata endpoints"]
  G --> J["Private repository fetcher"]

Components

Data flow

Casco selected live agents, started a 30-minute timer, extracted system prompts, inspected tool definitions, and then tried to exploit whatever the tools could access. In the IDOR case, leaked user/document IDs flowed through agent tools into other users' data. In the sandbox case, write/read/execute capabilities let the attacker modify protections, discover metadata, and move laterally. In the SSRF case, a tool that fetched private GitHub schemas sent credentials to an attacker-controlled repository. Fixes sit around the agent: authenticate and authorize each request, sanitize inputs and outputs, and isolate code execution with enterprise sandboxes.

Why this architecture

Casco frames agent security as an end-to-end systems problem because the damaging paths are often in the arrows between LLMs, tools, APIs, sandboxes, credentials, and data. Treating the agent like a user forces authorization, sanitization, and sandboxing to be implemented outside natural-language prompts.

Models used: Large Language Models · Claude

Scale

• Casco tested 16 live agents.
  “we hacked uh out of 16 agents that were launched within 30 minutes each.”transcript, stated
• Seven of the 16 agents were compromised within the 30-minute windows.
  “We were hacked uh we hack we hacked seven of them.”transcript, stated
• One cited sandbox/metadata failure exposed customer data through BigQuery.
  “Whoops. See, we have access to all their customer data. So that's uh we just queried BigQuery”transcript, stated
• Code-related agents are a major usage category in Claude telemetry cited by the talk.
  “we make up 3.4% of the world but we're 37% of cloud's usage.”transcript, stated

Alternatives considered (roads not taken)

Treat agent security as primarily prompt injection and harmful-content filtering

Rejected because: The compromises came from ordinary authorization, sandbox, credential, and network paths around the LLM.

Let the agent act with service-level permissions

Rejected because: Agents should inherit user-like authorization boundaries rather than broad server/service privileges.

Roll a custom code execution sandbox

Rejected because: Sandboxing arbitrary compute safely is hard, and weak isolation can expose metadata, service tokens, and internal resources.

Patterns

safety-policy-enforcement (security)

The core recommendation is to enforce authentication, authorization, sanitization, and sandbox controls outside the LLM rather than trusting natural-language constraints.

tool-use-orchestration (architecture)

The red-team methodology centers on agent tool definitions because tools determine data access, code execution, database creation, and private repository fetches.

privacy-preserving-processing (security)

Casco's IDOR example shows that agent data access must preserve user/resource boundaries even when the agent has a valid token.

observability-tracing (operations)

Casco recommends production-grade sandboxes with built-in observability so agent code execution is isolated and inspectable.

Tradeoffs

Agent permission model

Chose user-like permissions and external authorization over service-level agent permissions or LLM-decided access control.

Rationale: Agents act on behalf of users and can be manipulated; letting them inherit service permissions turns model/tool mistakes into broad data access.

Code execution isolation

Chose purpose-built sandbox solutions over custom Python/container apps.

Rationale: Sandbox correctness is difficult and failures enable lateral movement across infrastructure.

Local agent safety UX

Chose permission prompts or constrained modes over full local YOLO execution.

Rationale: The speaker frames local agent operation as an industry split between full autonomy and asking every time, implying a safety/usability tradeoff.

Gotchas (what bit them)

Valid authentication is not enough for agent tools.

Root cause: The API checked token validity but not whether the authenticated user could access the specific resource ID.

Resolution: Apply a resource-level authorization matrix on every agent-mediated request.

Natural-language prompt constraints reveal what attackers should invert.

Root cause: System prompts often describe exactly what developers do not want the agent to do, but do not enforce it technically.

Resolution: Treat prompts as clues for red-team testing and enforce sensitive constraints through code and infrastructure controls.

Custom code sandboxes can become lateral-movement paths.

Root cause: Arbitrary compute in a container can discover metadata, tokens, network resources, and internal services.

Resolution: Use hardened sandbox products and avoid plain containers as isolation boundaries.

SSRF-style tool calls can leak private repository credentials.

Root cause: A tool fetched database schemas from a private GitHub repository using credentials, but accepted attacker-controlled repository strings.

Resolution: Sanitize tool inputs/outputs and audit each arrow where credentials are attached to downstream requests.

Evaluation

Casco evaluated agent security through a constrained live red-team exercise: 16 production agents, 30 minutes per agent, system-prompt/tool inspection, and exploit attempts categorized by recurring vulnerability classes.

Metrics: 16 live production agents tested · 30 minutes per agent · 7 agents compromised · 3 recurring vulnerability classes identified

Outcomes

• Seven of 16 live agents were compromised within 30 minutes each.
• The recurring vulnerability classes were IDOR/cross-user data access, arbitrary code execution/lateral movement, and SSRF credential leakage.
• The talk reframed agent security as a full-system security problem, not just prompt injection or harmful-content filtering.
• Casco recommended user-like permissions, explicit authorization, input/output sanitization, and hardened code sandboxes.

The evidence is a limited but concrete red-team sample of 16 live agents. The findings are qualitative vulnerability classes rather than a statistical industry benchmark.