Security Learnings from LLM Production Deployments

nvidia · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM application security

NVIDIA's security talk explains that production LLM applications introduce new trust-boundary and data-flow risks because RAG stores, plugins, logs, external documents, and model-visible secrets all become application inputs or leakage paths.

Constraints:

• External data sources and RAG stores can be attacker- or insider-controlled.
• Guardrails are useful for moderation but insufficient as a security boundary.
• Plugins can turn prompt injection into SQL execution, SSRF, data exfiltration, or destructive actions if over-permissioned.
• If sensitive data enters model context, the model may be induced to leak it.

Architecture

The talk recommends treating LLM applications as application-security systems: identify trust and security boundaries, trace where data comes from and goes, treat RAG and plugin inputs as tainted application inputs, minimize output and privileges, isolate secrets from model context, avoid logging sensitive prompts/responses, constrain RAG sources to authoritative documents, and parameterize/sanitize plugins.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["User input"] --> B["Guardrails"]
  A --> C["Embedding model"]
  C --> D["RAG document store"]
  D --> E["Prompt template"]
  E --> F["LLM runtime"]
  B --> F
  F --> G["Plugin layer"]
  G --> H["Downstream systems"]
  F --> I["Conversation history and logs"]

Components

Data flow

User input enters guardrails and/or standardization; RAG flows send the request to an embedding model and document retrieval service, insert retrieved chunks into a template, and send that context to the LLM; tool-use flows let the LLM choose plugins that may call databases, URLs, or code execution; conversation history and logging may store prompts/responses; security review traces all external documents, plugin inputs, logs, and outputs as potentially tainted data.

Why this architecture

The talk argues that standard application security still applies, but LLM systems expand the input surface through RAG, plugins, logs, external documents, and probabilistic model outputs; therefore security review must follow data flows rather than trust the model or guardrails as the boundary.

Models used: LLMs · embedding models

Scale

• The talk frames RAG as the most common production framework the speaker sees deployed.
  “retrieval augmented generation this is probably the most common framework that we see deployed these days”transcript, stated
• Security review starts from two broad questions: where data comes from and where it goes.
  “we always sort of look for two main things where does the data come from and where does the data go”transcript, stated

Alternatives considered (roads not taken)

Rely on guardrails to keep data safe

Rejected because: Guardrails are weaker as security tools and can be bypassed with encoding or timing side channels.

Let plugins expose broad capabilities to the LLM

Rejected because: Prompt injection can make an LLM invoke destructive or sensitive plugin actions.

Give an HR/RAG bot all enterprise documents

Rejected because: The talk recommends limiting retrieval to authoritative documents for the use case.

Log prompts and responses containing sensitive data

Rejected because: Logs can become a separate leakage path whose access controls may not match the underlying documents.

Patterns

rag-retrieval (retrieval)

The talk analyzes RAG as a common production architecture where retrieved enterprise/external documents are inserted into model context and become security-relevant inputs.

tool-use-orchestration (tooling)

The talk treats LLM plugin/tool calls as security-critical downstream actions that must be parameterized, validated, and least-privileged.

safety-policy-enforcement (security)

The recommended controls are least privilege, output minimization, data-flow tracing, source restriction, and isolation of secrets from model context.

Tradeoffs

Use guardrails for moderation, not data security.

Chose supplementary content controls plus data isolation over relying on guardrails to keep secrets safe.

Rationale: Guardrails can be bypassed and should not see sensitive data as the primary defense.

Constrain plugins rather than trusting prompts.

Chose parameterized, sanitized, least-privilege tools over letting LLM-generated text control SQL, URLs, or destructive operations.

Rationale: Prompt injection can make broad plugin permissions dangerous regardless of guardrails.

Limit RAG source scope.

Chose authoritative, purpose-specific documents over broad enterprise or user-writable document access.

Rationale: Source limitation reduces insider poisoning and accidental retrieval of irrelevant or malicious documents.

Gotchas (what bit them)

RAG document stores become an application input surface.

Root cause: External documents or insiders with write access can poison retrieved context and induce model behavior.

Resolution: Control write access, limit retrieval sources, and treat external/RAG documents as tainted input.

Guardrails can leak information or fail under encoded prompt injection.

Root cause: LLMs can handle encodings such as Base16/Base64/ROT13 that evade superficial guardrail checks.

Resolution: Treat guardrails as supplementary moderation controls, not the primary data-security boundary.

Logs can violate document-level access assumptions.

Root cause: Prompt/response logs may contain retrieved sensitive documents, but logging-system permissions may not match each document's permissions.

Resolution: Avoid logging prompts/responses with sensitive data or apply durable opt-out and restrictive access controls.

Prompt injection can turn a model into a plugin exploit path.

Root cause: If the model can generate SQL, URLs, or code for plugins, injected instructions can redirect those tool calls.

Resolution: Parameterize plugins, hardcode unsafe parts such as URLs, sanitize parameters, and restrict plugin permissions.

Sensitive data in context is fundamentally leakable.

Root cause: Model output is probabilistic and adversarially steerable, so model-visible secrets can be reproduced.

Resolution: Isolate API keys, passwords, secret documents, and other sensitive data from the LLM whenever possible.

Evaluation

The talk describes AI red-team/product-security assessment as data-flow and trust-boundary review: enumerate inputs, external data sources, logs, plugins, output sinks, and sensitive data propagation.

Metrics: input vectors · data-flow sinks · trust/security boundaries · plugin permissions · logging exposure · sensitive data reachability

Outcomes

• The talk provides a practical security checklist for production LLM apps: trace data, identify trust boundaries, restrict plugins, limit RAG sources, minimize outputs, and isolate secrets.
• It reframes RAG and external documents as application inputs that require sanitization and access control.
• It argues guardrails should be supplementary moderation, not a substitute for data isolation and least privilege.

The talk is based on product-security and AI red-team reviews of deployed LLM-enabled applications, with RAG described as the most common framework observed.