Red-Teaming an AI Agent: Security Testing of goose Through Operation Pale Fire

block · Finance · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI coding-agent security and internal red-team operations

Block needed to understand how attackers could use goose, an open-source agent with MCP extensions and shell access, to compromise employees through untrusted context, prompt injection, and social engineering.

Constraints:

• goose can take meaningful real-world action through MCP extensions.
• Internal users can connect calendar, development, shell, and other workflow tools.
• The agent mixes instructions and data in a single context window.
• Payloads behaved differently across models and vendors.
• Calendar APIs imposed 50-invite-per-day rate limits on external invitations.
• The exercise needed to validate detection and response without causing real harm.

Architecture

Block ran Operation Pale Fire as an offensive exercise against goose. The red team first targeted Google Calendar MCP context poisoning with hidden Unicode prompt injections, then pivoted to poisoned goose recipes that append prompts to the system prompt, and finally used targeted social engineering to get a goose developer to run the recipe. DART detected and contained the simulated compromise, and the goose team shipped policy, recipe transparency, zero-width stripping, prompt-injection detection, telemetry, and incident-response improvements.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["External calendar invitations"] --> B["Google Calendar MCP"]
  B --> C["MCP extensions"]
  C --> D["goose agent"]
  E["Zero-width prompt payload"] --> A
  E --> F["Recipe URL"]
  F --> G["Recipe loader"]
  G --> D
  D --> H["Developer shell tool"]
  H --> I["C2 test server"]
  H --> J["User workstation"]
  J --> K["DART monitoring and response"]
  D --> K

Components

Data flow

The red team surveyed MCP extensions for untrusted context ingestion, selected Google Calendar, and sent external calendar invites containing prompt injections. When a user asked goose about their calendar, goose would ingest the event description and invoke the developer shell tool toward a C2 server. After rate limits and model/MCP variance made that path unreliable, the team moved the payload into a shareable recipe that decoded JSON and appended prompt content to the system prompt. A targeted recipe-debugging pretext caused an employee to execute the recipe, triggering the infostealer simulation; DART isolated the environment and traced the agent prompt interaction to local activity.

Why this architecture

Operation Pale Fire deliberately attacked goose through the same extensibility paths that make agents useful: MCPs, recipes, context windows, and shell tools. The engagement paired offensive testing with DART monitoring so Block could validate both prevention and response, then convert the findings into platform mitigations.

Models used: Large Language Models · multiple models from many different vendors

Scale

• The first campaign sent roughly 50 calendar invitations before hitting API rate limits.
  “After sending roughly 50 calendar invitations, we encountered Google Calendar API rate limits.”article, stated
• The red team continued at 50 invitations daily.
  “We continued the campaign over the following days, sending 50 invitations daily before running into rate limits.”article, stated
• Approximately five days passed with no successful requests before they pivoted.
  “After approximately five days with no successful requests we decided to continue sending out invites, alongside pivoting to a new vector”article, stated
• The successful campaign generated multiple server requests before final payload debugging.
  “On our server we saw multiple requests come through, and it looked like we had finally caught something!”article, stated

Alternatives considered (roads not taken)

Calendar MCP prompt injection through user/agent prompt context

Rejected because: It was technically possible but inconsistent across models, MCP versions, calendar density, and non-deterministic model behavior.

Target the system prompt through shareable recipes

Rejected because: It became the preferred pivot rather than rejected; recipes offered more reliable control because prompt content was appended to the system prompt.

Broad calendar-invite phishing with a recipe link

Rejected because: A typo blocked the full execution flow, and Google Calendar rate limits pushed the team toward a more targeted recipe-debugging pretext.

Patterns

safety-policy-enforcement (security)

Block translated red-team findings into calendar policy changes, recipe transparency, zero-width character stripping, prompt-injection detection, and AI-specific security principles.

tool-use-orchestration (architecture)

The attack and defense revolve around MCP tools and the built-in developer shell tool, showing that real-world agent actions must be governed as security-sensitive tool use.

guardrails-io-filtering (security)

Block's remediation strips hidden Unicode, detects prompt injection, and warns that any documents/emails/calendar entries the agent processes can carry malicious input.

observability-tracing (operations)

DART used monitoring, telemetry, and response playbooks to identify, contain, and analyze the simulated AI-borne attack.

human-in-the-loop-gate (security)

Block's lessons explicitly include user confirmation as a defense layer, and the successful campaign still required a human to click and execute the poisoned recipe.

Tradeoffs

Attack vector selection

Chose recipe/system-prompt targeting after initial calendar testing over calendar-only user/agent prompt injection.

Rationale: Calendar prompt injection was feasible but inconsistent; recipes gave the injected prompt more control by appending it to the system prompt.

Campaign delivery

Chose targeted developer-team social engineering over continued broad calendar-invite campaigns alone.

Rationale: The broad approach hit API limits and execution unreliability, while a targeted RTL-bug pretext guaranteed interaction with the recipe system.

Defense strategy

Chose defense in depth over one perfect AI-specific control.

Rationale: Block concluded that no single control would prevent these attacks, so input sanitization, user confirmation, monitoring, and incident response all matter.

Gotchas (what bit them)

Untrusted tool context can become executable instruction.

Root cause: Calendar events and other external data sources enter the agent's context window, where the model can treat injected text as instructions.

Resolution: Sanitize all input sources, detect prompt injection, and design MCP integrations as untrusted data paths.

Human-visible review missed hidden Unicode prompt injection.

Root cause: Zero-width characters made a lengthy prompt injection invisible to users while still interpretable by the model.

Resolution: Strip zero-width characters and add prompt-injection detection when loading content into goose.

Model and tool heterogeneity made payload reliability brittle.

Root cause: Block allowed many model/vendor choices, and prompt payloads did not transfer reliably across models or MCP/goose versions.

Resolution: The red team pivoted from calendar context injection to recipe-based system-prompt targeting and recorded the operational variability as a lesson.

Transparent workflow sharing became a prompt-poisoning vector.

Root cause: Recipes decoded base64 JSON and populated agent instructions/prompt in the UI without enough transparency.

Resolution: The goose team added transparency to recipe loading by visualizing the instructions being loaded.

The initial successful phish failed to execute because of a typo.

Root cause: A small prompt-injection typo prevented the infostealer from being downloaded and run.

Resolution: The team fixed the typo, continued calendar invites, and pivoted to a new vector.

Evaluation

Block evaluated goose security through a live internal red-team operation, controlled payload execution, monitoring for C2 requests and local behavior, DART containment, and post-engagement remediation validation through merged goose changes and updated detection/playbook work.

Metrics: Roughly 50 calendar invitations before rate limiting · 50 invitations daily during follow-on calendar campaigns · Approximately five days without successful requests before pivoting · Multiple server requests observed during the second campaign · Successful compromise of one employee laptop in the final campaign

Outcomes

• The red team compromised a Block employee's laptop with an invisible-Unicode prompt injection and social-engineering chain.
• DART quickly identified and contained the simulated threat.
• Block changed Google Calendar policy for external invites.
• goose added recipe transparency, zero-width character stripping, and prompt-injection detection.
• Block strengthened telemetry, agent/system-event correlation, and AI-specific response runbooks.
• The exercise produced public guidance that AI output should be treated like user input and access control should not depend on the AI.

This was an internal red-team engagement rather than a broad benchmark; its scale is measured in campaign attempts, daily invite limits, and one successful simulated compromise.