Red Teaming an Internal AI Agent Through Prompt Injection and Social Engineering

block · Finance · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal AI agent red teaming and enterprise endpoint security

Block needed to test whether goose, an internal AI agent with MCP integrations and a developer shell tool, could be turned into an initial-access vector before the tool was open sourced and more widely targeted.

Constraints:

• goose supported multiple underlying models, making prompt injection reliability model-dependent.
• The agent shipped with a developer shell tool that could run bash commands.
• MCP integrations could move external untrusted text into the model context.
• The red team needed stealth and evasion because Block's detection/response team actively hunts malicious activity.
• The operation was limited to getting execution on a Block laptop, not wider lateral movement.

Architecture

Block's offensive security team ran Operation Palefire as an end-to-end red-team exercise. They first targeted Google Calendar MCP by hiding zero-width Unicode prompt injection in calendar invites so goose would ingest it and invoke its developer shell. When calendar context, model variance, and rate limits made the attack unreliable, they pivoted to goose recipes: shareable workflows whose base64 JSON can customize system and user prompts. The successful campaign used a legitimate-looking RTL recipe bug report to get a goose developer to execute a poisoned recipe, triggering an infostealer simulation that existing controls detected.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Malicious calendar invite"] --> B["Google Calendar MCP"]
  B --> C["Internal MCP marketplace"]
  C --> D["goose agent"]
  E["Invisible Unicode payload"] --> A
  E --> F["Recipe URL"]
  F --> G["Recipe runner"]
  G --> D
  D --> H["Developer shell tool"]
  H --> I["Attacker server"]
  H --> J["Employee laptop"]
  J --> K["Detection and response controls"]

Components

Data flow

The red team surveyed Block's internal MCP marketplace for extensions that could put attacker-controlled text into goose's context. They sent external calendar invites with hidden zero-width prompt injections; if a user asked goose about the calendar, Google Calendar MCP would fetch the event, inject the hidden text into context, and ask goose to run a curl-to-bash developer-shell command. After the calendar path failed, a poisoned recipe URL loaded hidden instructions into goose's system prompt; a targeted developer-support pretext led an employee to run it, the payload contacted the attacker server, and detection controls caught the simulated infostealer activity.

Why this architecture

The operation attacked goose through realistic enterprise paths rather than toy prompt-injection demos: MCP marketplace integrations, calendar invites, recipe sharing, and shell execution. That made it possible to test whether AI-specific context poisoning could be operationalized into the same endpoint compromise and detection workflow Block uses for traditional red-team operations.

Models used: Claude Sonnet 3.5 · Claude Sonnet 3.7 · GPT models

Scale

• The operation took place around fall 2024.
  “It was something that began as an internal project back in fall of 2024.”transcript, stated
• Calendar rate limits constrained the team to about 50 invites per day.
  “We could only send I think like 50 invites a day.”transcript, stated
• Real calendars often had eight to ten invites, unlike the sparse test accounts.
  “Real calendars have a lot more calendar invites usually like eight, nine, 10.”transcript, stated

Alternatives considered (roads not taken)

Calendar-based user prompt injection

Rejected because: It was stealthy but unreliable because calendar context filled the model window, model behavior varied, and the payload had not been tested realistically.

Recipe-based system prompt injection

Rejected because: This became the preferred path because recipes allowed injection into the system prompt and gave the payload more influence than user-context injection.

Broad meeting/invite-based recipe campaign

Rejected because: It produced engagement but a typo and rate limits kept it from producing the desired payload execution quickly.

Patterns

safety-policy-enforcement (security)

Block turned the red-team findings into controls: stripping non-standard Unicode, showing recipe contents before execution, prompt-injection detection, command allowlisting, and calendar policy changes.

tool-use-orchestration (architecture)

The risk and the exploit both depend on goose's tool integrations, especially MCPs and the developer shell tool.

guardrails-io-filtering (security)

The remediation filtered hidden Unicode and added prompt-injection detection around agent inputs and shell commands.

observability-tracing (operations)

The operation used existing detection controls, logs, and a later purple-team review to identify and improve signals for AI-borne attacks.

Tradeoffs

Payload delivery channel

Chose calendar invites and recipes over ordinary email phishing.

Rationale: Calendar invites looked less scrutinized than email and could feed untrusted text into goose through the Google Calendar MCP.

Prompt-injection placement

Chose system-prompt injection via recipes over user-context injection via calendar events.

Rationale: System-prompt injection gave the payload more influence and more available context than a calendar event inside user context.

Post-red-team mitigations

Chose multiple controls across Unicode filtering, recipe transparency, detection, allowlisting, and calendar policy over a single prompt-injection detector.

Rationale: The team acknowledged detector false positives/noise and treated mitigation as layered rather than one-size-fits-all.

Gotchas (what bit them)

Prompt injections that worked in test did not reliably work in realistic calendars.

Root cause: The test accounts had only two or three events, while real calendars had much denser context windows that diluted the malicious invite.

Resolution: The team identified the need for a realistic automated test harness across models and versions.

Universal prompt injection was not reliable across model choices.

Root cause: The payload was tuned for Sonnet 3.5, while users could run Sonnet 3.7 or GPT models.

Resolution: The campaign pivoted toward system-prompt injection through recipes for stronger influence.

A tiny script typo broke an otherwise successful social-engineering campaign.

Root cause: goose fixed the typo in testing but not consistently in production.

Resolution: The team kept the same approach, fixed the bug, and switched to a more targeted phishing vector.

Hidden Unicode appeared in logs but required special detection to notice.

Root cause: Zero-width characters were not human-visible in the invite but surfaced as odd characters in logs.

Resolution: Block added detection for those characters and continued work on prompt-injection detection.

Evaluation

Block evaluated the attack by running staged red-team campaigns, observing whether goose executed the payload, monitoring attacker-server callbacks, and then relying on detection/response controls and follow-up purple-team analysis to evaluate what signals could have caught the activity.

Metrics: Execution on a Block laptop was the operation objective · Google Calendar campaign limited to roughly 50 invites per day · Calendar context test accounts had two or three events while real calendars had eight to ten · Existing controls eventually caught the simulated infostealer activity

Outcomes

• The recipe-based social-engineering campaign achieved code execution on an employee laptop.
• Existing controls detected the simulated infostealer-like behavior.
• Block stripped non-standard Unicode characters from goose input and recipes.
• Recipe content was made visible before execution.
• Block added prompt-injection detection, bad-bash-command detection, command allowlisting, and Google Calendar policy changes.
• The team identified the need for realistic model/version test harnesses for prompt-injection efficacy.

This was an internal red-team exercise with a narrow objective: endpoint execution through goose, not a production benchmark across many organizations.