LLM Security: Discovering and Mitigating Repeated Token Attacks in Production Models

dropbox · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM application security

Dropbox needed to understand and mitigate repeated-token attacks against ChatGPT-family models used in AI-powered product features, including attacks that could bypass provider-side filtering, extract memorized training data, and create expensive long-running requests.

Constraints:

• Dropbox used OpenAI-compatible chat completion models for AI-powered features, so third-party model vulnerabilities mattered to product security.
• The attack lived inside user-controlled prompt text, making input validation and prompt-template boundaries central to mitigation.
• OpenAI's initial mitigation blocked repeated single tokens but did not cover repeated multi-token sequences.
• Successful attacks could expose memorized training data, including sensitive or personally identifiable information.
• Some prompts could consume excessive model runtime and return timeout errors, creating denial-of-service or cost-exhaustion concerns.
• Findings had to be handled through responsible disclosure before public release.

Architecture

Dropbox treated repeated-token behavior as an adversarial security-testing problem: researchers reproduced prior single-token divergence attacks, searched for multi-token repeat sequences that still induced model divergence, validated training-data extraction on GPT-3.5 and GPT-4, measured long-running request behavior, disclosed the findings to OpenAI, and translated the results into practical mitigations such as broader repeated-token filtering, prompt-input sanitization, max_tokens limits, server-side timeouts, and a planned LangChain-compatible detector.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Repeated-token prompt generator"] --> B["OpenAI chat-completion models"]
  B --> C["Divergence output analyzer"]
  C --> D["Training-data match verifier"]
  B --> E["Resource-exhaustion probe"]
  C --> F["Provider mitigation channel"]
  F --> G["Repeated-token detector"]

Components

Data flow

Researchers constructed prompts containing repeated token or multi-token sequences, sent them through OpenAI chat-completion models, inspected the outputs for divergence, hallucination, memorized-data matches, or timeout behavior, compared extracted text against public source material, disclosed confirmed findings to OpenAI, and updated mitigation guidance and tooling around repeated-token detection and resource limits.

Why this architecture

The research architecture mirrors the attack surface: repeated-token sequences had to be generated at the tokenizer/prompt layer, responses had to be evaluated against memorization and resource-exhaustion criteria, and mitigations had to live both at provider-side filters and at application-side input controls because Dropbox could not assume third-party model security was complete.

Models used: GPT-3.5 · GPT-4 · gpt-4-32k

Scale

• The tested tokenizer maps UTF-8 text into roughly 100,000 possible token IDs.
  “OpenAI’s GPT-3.5 and GPT-4 LLMs use the cl100k_base tokenizer, which maps all possible UTF-8 encoded text into integers from 0 to roughly 100,000”article, stated
• A GPT-3.5 attack repeated the two-token jq_THREADS string 2,048 times.
  “Dropbox demonstrated successful extraction of memorized training data from GPT-3.5 using the two-token string "jq_THREADS" (token IDs 45748 and 57339) repeated 2,048 times.”summary, stated
• A GPT-4 attack repeated /goto maiden 12,000 times in a gpt-4-32k prompt.
  “When the phrase is repeated 12,000 times and sent in a gpt-4-32k prompt, the model responds with an excerpt from the Old Testament’s Book of Genesis”article, stated
• One streaming request returned more than 10,700 tokens before the ten-minute threshold.
  “Using streaming requests, gpt-4-32k returned over 10,700 tokens before hitting the timeout.”summary, stated
• OpenAI requested a 90-day disclosure period for confirmation and remediation.
  “OpenAI requested a 90-day disclosure period to confirm and, if necessary, address the security gaps we disclosed.”article, stated

Alternatives considered (roads not taken)

Rely only on provider-side single-token repetition filtering

Rejected because: Dropbox showed that the single-token filter left multi-token repeats able to induce divergence and extract memorized data.

Treat repeated-token behavior as only hallucination or prompt-injection weirdness

Rejected because: The research connected divergence to memorized training-data extraction and resource exhaustion, requiring security controls rather than merely prompt-tuning.

Depend on proxy timeouts for long-running repeat requests

Rejected because: Dropbox recommended application-side sanitization and max_tokens limits because requests could still consume substantial model-side work before timing out.

Patterns

safety-policy-enforcement (security)

Dropbox turned repeated-token research into concrete safety controls: broader provider-side filters for multi-token repeats, application-side input sanitization, and a planned detector for internal model assets.

eval-driven-development (evaluation)

The team used adversarial experiments as an evaluation harness for model-security behavior, repeatedly probing GPT-3.5 and GPT-4 before and after provider mitigations.

privacy-preserving-processing (security)

Training-data extraction was treated as a privacy risk because divergence could reveal memorized content including PII, making leakage prevention part of the deployment threat model.

guardrails-io-filtering (security)

The mitigation path moved from narrow single-token filtering to broader repeated-sequence controls and application-level sanitization of prompt inputs.

Tradeoffs

Use adversarial security probing against third-party model endpoints

Chose Directly test provider-hosted GPT-3.5 and GPT-4 behavior with repeated-token prompts over Assuming provider alignment and filtering controls fully covered the risk.

Rationale: Direct testing revealed both GPT-3.5 and GPT-4 extraction paths after the initial provider-side single-token mitigation.

Mitigate both content leakage and resource exhaustion

Chose Input sanitization and max_tokens limits alongside repeated-token filtering over Treating the vulnerability only as a data-extraction issue.

Rationale: The same family of repeated-token prompts could cause long-running expensive requests, so production controls needed cost and availability protections too.

Coordinate disclosure before publication

Chose Responsible disclosure to OpenAI with a requested disclosure window over Immediate public release of vulnerability details.

Rationale: The approach let OpenAI confirm the vulnerabilities and deploy broader filtering before Dropbox published the research.

Gotchas (what bit them)

A mitigation copied from published research can miss nearby attack variants.

Root cause: OpenAI's first repeated-token filter focused on repeated single tokens, while Dropbox found repeated multi-token sequences could still trigger divergence.

Resolution: Extend filtering to multi-token repeats and keep adversarial regression tests around the broader class of token-repetition attacks.

Prompt-injection anomalies may hide a training-data-extraction vulnerability.

Root cause: Repeated tokens can make chat models diverge from aligned dialogue behavior and revert toward lower-level language-modeling behavior.

Resolution: Evaluate suspicious prompt behaviors for memorized-data leakage, not only for instruction-following or hallucination failures.

Short adversarial prompts can become expensive resource-exhaustion requests.

Root cause: Certain repeat instructions caused GPT-4 requests to run for ten minutes and stream thousands of tokens before timing out.

Resolution: Sanitize prompt input and set max_tokens limits so applications do not rely solely on upstream proxy or server timeouts.

Third-party model security does not remove the application owner's security-testing responsibility.

Root cause: Dropbox depended on OpenAI models for customer-facing AI features, but vulnerabilities still existed in the provider model behavior.

Resolution: Run internal LLM security research and threat modeling even when using well-resourced external model providers.

Evaluation

Dropbox evaluated repeated-token attacks by executing controlled prompts against GPT-3.5 and GPT-4, checking whether model outputs diverged from expected chat behavior, matching generated spans against known source documents, recording timeout and token-stream behavior, and verifying whether provider filtering changed after disclosure.

Metrics: Number of repeated tokens or token pairs needed to induce divergence · Sequential cl100k_base token matches against source material · Presence of memorized training-data excerpts in model output · Request timeout duration and streamed token count · Provider filtering behavior before and after mitigation

Outcomes

• Dropbox demonstrated repeated-token training-data extraction against GPT-3.5 using jq_THREADS repeats.
• Dropbox demonstrated GPT-4 vulnerability using /goto maiden repeats that elicited Bible-passage output.
• OpenAI confirmed the GPT-3.5 and GPT-4 findings as training-data-extraction vulnerabilities.
• OpenAI broadened repeated-token filtering to include multi-token repeats.
• Dropbox identified long-running repeat requests as a resource-exhaustion risk and planned a LangChain-compatible repeated-token detector.

The strongest operating-scale signals are in the attack parameters: thousands of repeated tokens, 32K-token GPT-4 context windows, more than 10,700 streamed tokens before timeout, and a 90-day disclosure process.