Detecting and Mitigating Prompt Injection via Control Characters in ChatGPT

dropbox · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM prompt-injection security

Dropbox needed to harden LLM-powered product and research infrastructure against user-controlled prompt inputs that could bypass server-side prompt templates and model instructions.

Constraints:

• LLM products may combine server-side instructions, context, and user-controlled questions.
• Attackers can manipulate user-controlled input or server-side data.
• Prompt templates with delimiters, IDK responses, word limits, and no-new-instructions clauses are not sufficient by themselves.
• Control characters can be encoded in JSON payloads and interpreted as model input.
• Mitigations must consider model-specific behavior and valid use cases for control sequences.

Architecture

Dropbox ran black-box OpenAI Chat API experiments against GPT-3.5 and GPT-4 prompt templates. The team prepended or inserted encoded control characters such as carriage returns and backspaces into user-controlled questions, observed instruction betrayal, context forgetting, and hallucinations at scale, then recommended model- and application-specific input sanitization strategies rather than relying only on prompt wording.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user-controlled-input["User-controlled input"] --> prompt-template["Prompt template"]
  control-character-test-script["Control-character test script"] --> user-controlled-input["User-controlled input"]
  prompt-template --> openai-chat-api["OpenAI Chat API"]
  openai-chat-api --> model-response["Model response"]
  model-response --> experiment-log["Experiment log"]
  control-character-test-script --> experiment-log["Experiment log"]
  experiment-log --> sanitization-strategy["Sanitization strategy"]
  sanitization-strategy --> user-controlled-input["User-controlled input"]

Components

Data flow

Dropbox's security workflow builds a controlled prompt template with context, IDK response, max-word limit, and no-new-instructions guard. A test script submits user-controlled questions to OpenAI chat completions with increasing encoded control-character offsets. Observed responses are compared to expected context-constrained answers, and results inform sanitization guidance for production LLM services.

Why this architecture

Dropbox treated prompt injection as an empirical security-testing problem because control-character behavior was undocumented and model-specific. Repeatable API experiments let the team observe where prompt-template controls failed and translate those observations into sanitization guidance.

Models used: GPT-3.5 · GPT-4 · gpt-3.5-turbo · gpt-4-32k

Scale

• The GPT-3.5 experiments included 350 or more carriage returns.
  “when 350 (or more) carriage-returns are inserted, the model doesn’t address the first question, as if forgetting it.”article, stated
• GPT-3.5 answered both questions with up to 4000 backspace characters in one encoding.
  “at up to 4000 backspace characters (about as many as can fit in GPT-3.5’s 4096-token context window), the model answered both questions.”article, stated
• A three-character backspace encoding caused GPT-3.5 to forget the first question with at least 450 backspaces.
  “it was possible to get GPT-3.5 to forget the first question by sending at least 450 such backspaces”article, stated
• With GPT-4 32K, similar effects appeared at relative prompt offsets of -10000 and greater.
  “With a GPT-4 context length of 32K (32768 tokens using the gpt-4-32k model), we were able to trigger similar effects as demonstrated for GPT-3.5 at higher relative prompt offsets (-10000 and magnitudes greater).”article, stated

Alternatives considered (roads not taken)

Rely on prompt-template wording and delimiters

Rejected because: The control-character technique was agnostic to instruction wording and formatting suggestions.

Use the same sanitization policy for all models

Rejected because: Dropbox observed model-specific susceptibility and recommends testing based on model and application risk.

Strip all escape/control sequences unconditionally

Rejected because: Some products may legitimately need source-code or binary-format content that includes escape/control sequences.

Patterns

guardrails-io-filtering (security)

Dropbox recommends sanitizing input according to application risk and model behavior to block abusive control-character prompt input.

eval-driven-development (security-testing)

The team scripted repeatable black-box experiments against model APIs to test whether prompt controls held.

safety-policy-enforcement (security)

The work hardened internal infrastructure and server-side prompt controls for secure LLM product use.

Tradeoffs

Sanitize input by model and application

Chose Risk-tuned sanitization over One universal control-character rule.

Rationale: Model susceptibility varies and some applications need to process control sequences legitimately.

Test black-box model behavior

Chose Repeatable API experiments over Assuming documentation covered control-character semantics.

Rationale: The effect was undocumented, so experiments were needed to discover actual model behavior.

Balance security with valid control-sequence use cases

Chose Support modes for full character ranges where justified over Always stripping every escape sequence.

Rationale: Source-code and binary-format analysis may require preserving characters that are dangerous in ordinary prompt paths.

Gotchas (what bit them)

Prompt templates can fail even when they explicitly forbid new instructions.

Root cause: Large numbers of encoded control characters changed how models interpreted the prompt context.

Resolution: Validate and sanitize user-controlled input before it enters the prompt template.

Control-character effects are model- and encoding-dependent.

Root cause: Different encodings and context-window sizes produced different failure thresholds.

Resolution: Run model-specific experiments and tune sanitization to the chosen model/application.

Security filters can break legitimate content if applied blindly.

Root cause: Some applications need to analyze source code or binary-like formats containing escape/control sequences.

Resolution: Consider product modes that support full character ranges while isolating or sanitizing risky prompt paths.

Evaluation

Dropbox used scripted black-box experiments with prompt offsets and increasing counts of encoded control characters, checking whether GPT-3.5 and GPT-4 followed context/instruction constraints or hallucinated/out-of-context answers.

Metrics: Control-character count/offset required for instruction betrayal · Context-following vs context-forgetting · Out-of-context answer production · Hallucination behavior · Model-specific susceptibility across GPT-3.5 and GPT-4

Outcomes

• Identified a previously undocumented control-character prompt-injection technique.
• Demonstrated instruction betrayal and context forgetting on GPT-3.5 and GPT-4 prompt templates.
• Recommended input sanitization strategies tailored to model, application, and risk tolerance.

The study used OpenAI Chat API experiments in June 2023, including GPT-3.5 4096-token behavior and GPT-4 32K-context behavior.