Building Secure Generative AI Applications at Scale: Amazon's Journey from Experimental to Production

amazon · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise generative AI security operations

Amazon needed to move generative AI applications from experimentation to production while handling nondeterministic model behavior, new attack vectors, fast-changing security techniques, and a large builder population deploying customer-facing and internal AI systems.

Constraints:

Architecture

Amazon's production GenAI security architecture combines secure-by-default building blocks with continuous testing and operations. Builders use Bedrock, Bedrock Guardrails, data perimeters, permissions, APIs, agents, and golden-path templates to launch common GenAI patterns. Existing AppSec, pentest, bug bounty, incident-response, and policy mechanisms are augmented with GenAI-specific threat modeling, prompt testing, and guardrail questions. FAST, the Framework for AI Security Testing, integrates with build pipelines and pre-prod stages: it loads a maintained baseline of attack prompts, calls the app through a proxy lambda against the user interface or inference endpoint, uses AI classifiers to detect deflection/errors/PII/out-of-bound responses, generates scorecards, and can block deployment until owners fix guardrails, prompts, or mitigations.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  builder-source-repository[Builder source repository] --> build-system[Build system]
  builder-source-repository --> golden-path-templates[Golden path templates]
  golden-path-templates --> bedrock-platform[Bedrock platform]
  golden-path-templates --> runtime-guardrails[Runtime guardrails]
  bedrock-platform --> runtime-guardrails
  bedrock-platform --> inference-endpoint[Inference endpoint]
  threat-model[Threat model] --> security-mechanisms[Security mechanisms]
  threat-model --> fast-baseline-prompt-suite[FAST baseline prompt suite]
  security-mechanisms --> golden-path-templates
  security-mechanisms --> operating-flywheel[Operating flywheel]
  build-system --> pre-prod-stage[Pre prod stage]
  pre-prod-stage --> fast-runner[FAST runner]
  fast-baseline-prompt-suite --> fast-runner
  fast-runner --> proxy-lambda[Proxy lambda]
  proxy-lambda --> user-endpoint[User endpoint]
  proxy-lambda --> inference-endpoint
  user-endpoint --> runtime-guardrails
  inference-endpoint --> runtime-guardrails
  fast-runner --> ai-response-classifiers[AI response classifiers]
  ai-response-classifiers --> security-scorecard[Security scorecard]
  fast-runner --> security-scorecard
  security-scorecard --> pre-prod-stage
  security-scorecard --> builder-remediation-loop[Builder remediation loop]
  builder-remediation-loop --> runtime-guardrails
  builder-remediation-loop --> pre-prod-stage
  pre-prod-stage --> production-deployment[Production deployment]
  production-deployment --> operating-flywheel
  operating-flywheel --> runtime-guardrails
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Builder source repository Application source code and GenAI configuration owned by product teams. Build system, Golden path templates
Golden path templates Secure-by-default application templates and pluggable components for common and specialized GenAI workloads. Builder source repository, Bedrock platform, Runtime guardrails
Bedrock platform AWS platform for selecting, optimizing, fine-tuning, customizing, and connecting models and agents. Golden path templates, Runtime guardrails, Inference endpoint
Runtime guardrails Input/output controls, content filters, topic filters, PII removal, data perimeters, and network/data consistency controls. Bedrock platform, User endpoint, Inference endpoint, Operating flywheel
Threat model GenAI security model that classifies frontend prompt injection and backend agent/indirect-prompt threats. Security mechanisms, FAST baseline prompt suite
Security mechanisms Augmented AppSec reviews, pentesting, bug bounty, incident response, policies, and guidance pages. Threat model, Golden path templates, Operating flywheel
Build system Existing builder CI/CD flow that runs integration tests and deploys to pre-prod before production. Builder source repository, Pre prod stage
Pre prod stage Pipeline phase where approval workflows and FAST security tests gate promotion to production. Build system, FAST runner, Production deployment
FAST runner Framework for AI Security Testing that runs baseline attack prompts and evaluates generated outputs. Pre prod stage, FAST baseline prompt suite, Proxy lambda, AI response classifiers, Security scorecard
FAST baseline prompt suite Maintained default prompt set updated from bug bounty, pentest, findings, and new attack patterns. Threat model, FAST runner
Proxy lambda Adapter used by FAST to invoke the application endpoint with test prompts. FAST runner, User endpoint, Inference endpoint
User endpoint Frontend/user-facing application interface for customer-style prompt testing. Proxy lambda, Runtime guardrails
Inference endpoint Backend model endpoint or data-processing path tested when no user interface exists. Proxy lambda, Bedrock platform, Runtime guardrails
AI response classifiers AI evaluators that classify deflections, errors, PII disclosures, and out-of-bound responses. FAST runner, Security scorecard
Security scorecard Application-specific results showing how the app performed against FAST tests. AI response classifiers, Pre prod stage, Builder remediation loop
Builder remediation loop Self-service process where owners inspect failures, update guardrails/system prompts/mitigations, and rerun the pipeline. Security scorecard, Runtime guardrails, Pre prod stage
Production deployment Released GenAI application such as Rufus or an internal Amazon Q assistant. Pre prod stage, Operating flywheel
Operating flywheel Post-launch loop that monitors responses, finds defects, adapts controls, and updates guardrails. Production deployment, Runtime guardrails, Security mechanisms

Data flow

A product team starts from golden-path templates or Bedrock components, selects and customizes a model, adds runtime guardrails, and builds a GenAI app. Existing security mechanisms apply a GenAI threat model and prompt-testing expectations. When code changes enter the build system, the pre-prod stage invokes FAST. FAST loads the default baseline prompts, sends them through a proxy lambda to the user interface or inference endpoint, evaluates the outputs with AI classifiers for deflection, errors, PII, and out-of-bound behavior, and produces a security scorecard. If the app fails, the pipeline can block; the builder updates guardrails, prompts, or mitigations and reruns. After production launch, monitoring and feedback feed an operating flywheel that updates guardrails and test baselines as new attacks or defects appear.

7 supporting quotes
“They're effectively like secured by default templates we give people to get them off the ground with a full end-to-end application”transcript, stated
“FAST, Framework for AI Security Testing. So, in this, like going back to our original threat model, the framework itself actually integrates with the build pipeline”transcript, stated
“say test, it actually does a series of actions. So, it loads a series of test prompts. This is what we call the default baseline. We maintain this based on the feedback we have gotten over all of our products. And so, as we learn any new attack in this space from bug bounty to pentest to any”transcript, stated
“It will then use the evaluation service to hit the inference endpoint using a proxy lambda. So, in this case, we have specifically designed "How do I load data into this application proxy lambda?" Which is then used to then say, "Okay, you can run these”transcript, stated
“it identifies whether the is a deflection. Did it respond in a way that's like, "I'm sorry, as an AI assistant, I'm actually not allowed”transcript, stated
“It then lets us interpret these results and builds up effectively a scorecard for the app, which tells us effectively how well it is doing against the entire set of tests. This has been super useful because we have been able to test so many things. We”transcript, stated
“pipeline stages that effectively run these tests and say, "If they fail, don't move forward." FAST is one of those tests, and so it is integrated in that pipeline. It'll run the workflow stage, and if it fails, then it will get the approval failure and not move forward. If it passes,”transcript, stated

Why this architecture

Amazon built security into existing development mechanisms because GenAI risk changes too quickly for one-time review or manual security-team intervention. Nondeterministic outputs, prompt injection, context-window attacks, PII leakage, and agent/tool behavior require continuous testing of inputs and outputs. Golden paths reduce duplicated security work for builders, while FAST scales security review across many apps by running new attack baselines in the same pipelines builders already use.

5 supporting quotes
“the applications are not deterministic, so the same output can give you different, the same input can give you different outputs”transcript, stated
“comes to fixing these prompt injections attacks, right? You can no longer just fix the code. You need to actively test your inputs, your outputs. You need to make sure that you evaluate everything that you're getting in and out, and that you fine tune, whether it is”transcript, stated
“to moderate and monitor. And so, this is very important, but it's not just a one and done. We don't launch guardrails and then say, "We're good." But you want launch the guardrails and you have mechanisms to understand the feedback from your users, and you have mechanisms”transcript, stated
“the next question you have to ask yourself is how am I going to keep up with and maintain knowing if they are secure?”transcript, stated
“we optimize for trying to inject security into the current processes, whatever they may be. And so, what's next? If you wanna learn more about GenAI security, AWS, we have a great link. If you wanna check out some cool models, the Amazon Nova Models are out”transcript, stated

Models used: Foundation models available through Amazon Bedrock · Amazon Q Business · Amazon Q Developer · Rufus shopping assistant models · Amazon Nova models · AI classifiers used by FAST

Scale

Alternatives considered (roads not taken)

Treat GenAI security like traditional static code security

Rejected because: Foundation models are complex, nondeterministic, and data-centric, so flaws can live in prompts, data, outputs, or model/tool behavior.

“Traditional software development is typically more interpretable. It's easy to read, and the challenges and the issues are easier to identify.”transcript, stated
“When it comes to foundation models or large language models, the complexity of these, the models themselves, can make it very difficult to identify security flaws”transcript, stated

Fix prompt injection like SQL injection

Rejected because: Prompt injection involves nondeterministic interaction between user input and model behavior, so there is no single code fix.

“anything that the user is trying to do, convincing the model to give you the data, right? So, fixing SQL injection, it's kind of simple. You just go fix the code, you will parameterize the queries, you store some of the procedures, right? You can allow-list some of the inputs, that's”transcript, stated
“outputs within GenAI models, the story changes a little bit, right? There is no code to fix. The inputs and the outputs are non-deterministic, so think about you have all that, your application is built correctly, everything is great, and your user input is ignore”transcript, stated

Build completely separate AI security processes

Rejected because: Amazon already had AppSec, pentest, bug bounty, incident response, and builder CI/CD mechanisms that could be augmented for GenAI.

“I did not need to create a new AppSec process. I just needed to take the AppSec process we had and make sure it had the right questions for generative AI workloads.”transcript, stated
“that are coming. The second one that we got is that integrating new tools and techniques into our current pipelines is super important. And so when you think about it from the perspective of these things that are coming out, if we can hook them into the things we've built, like our FAST framework”transcript, stated

Require every builder to hand-assemble security controls

Rejected because: Security requirements span many teams, policies, datasets, and controls, so duplicated manual implementation would not scale.

“All of these teams are gonna have different policies, different data sets, different controls that you need to deploy”transcript, stated
“And so the next question we had was, "How do we streamline this for our builder community?" Because it's a lot of work. We at Amazon have this thing we call golden path. We're a big fan of this. They're effectively like secured by default templates we give people to get them off the ground”transcript, stated

Human-review every FAST output

Rejected because: The volume of prompts and outputs is too high, so Amazon uses AI classifiers to score responses.

“we use our own forms of AI to do evaluation against the outputs, because there's no way I could scan all of this with humans.”transcript, stated
“evaluation against the outputs, because there's no way I could scan all of this with humans. Like, it's a lot of outputs and a lot of results and a lot of prompts. So we have that, and it identifies whether the is a deflection. Did it respond in a way that's like, "I'm sorry, as an AI”transcript, stated

Patterns

guardrails-io-filtering (security)

Amazon uses Bedrock Guardrails and runtime controls for harmful content thresholds, topic filtering, PII removal, blocked words, and input/output guardrails.

“You can use these guardrails to, with quite a few of the models, to have thresholds around what is harmful data considered”transcript, stated
“Can I remove PII automatically from outputs, right? And you can just define a set of words that when a user inputs them, or the LM generates them, it stop.”transcript, stated

eval-driven-development (quality)

Amazon treats GenAI security as a cycle of testing, evaluating, fine-tuning, implementing guardrails, reevaluating, releasing, and monitoring.

“you're gonna build a test, you're gonna test your thing and see how well it works, and you're gonna evaluate.”transcript, stated
“At that point, you reevaluate again, and maybe then you release it to production.”transcript, stated

observability-tracing (operations)

Post-launch operation monitors responses, defects, user feedback, and where guardrails need adaptation.

“look at the responses you're getting from the model to your user base. You're gonna evaluate, are there defects in this?”transcript, stated
“you have mechanisms to understand if it goes awry, and you have mechanisms to know where and how”transcript, stated

safety-policy-enforcement (security)

Amazon translates responsible AI principles, security policies, threat models, and existing AppSec/pentest/bug-bounty mechanisms into enforceable GenAI release checks.

“we wanna make sure that any application that we build follows responsible AI principles.”transcript, stated
“we developed a lot of that into our social security mechanisms, into our AppSec reviews.”transcript, stated

canary-rollout (release)

FAST gates pre-prod pipeline promotion, blocking production movement when AI security tests fail.

“in the pre-prod phase, you can have approval workflows or specific pipeline stages that effectively run these tests and say, "If they fail, don't move forward."”transcript, stated
“If it passes, the pipeline succeeds and then move to production.”transcript, stated

fallback-degradation (operations)

Guardrail versioning, rollback agility, and patching speed let teams decide whether to fix forward or roll back when defects appear.

“we needed to be able to do rollback agility and patching speed alongside that”transcript, stated
“how quickly can we fix it if we haven't fixed it yet, and is that sufficient, or should we roll back?”transcript, stated

model-routing (architecture)

Bedrock lets builders choose and optimize models by use-case needs such as cost, latency, and accuracy.

“The best model that you can choose for your use case. You can pick to optimize it for what your use case actually requires, whether it's cost, whether it's latency”transcript, stated

Tradeoffs

Augment existing security mechanisms

Chose AppSec, pentest, bug bounty, incident response, and guidance updates over A separate AI-only security organization and process.

Rationale: Existing mechanisms already scale across Amazon; adding GenAI-specific questions and training preserves builder familiarity while expanding coverage.

“I did not need to create a new AppSec process. I just needed to take the AppSec process we had and make sure it had the right questions”transcript, stated
“We were able to do all of these in conjunction with the existing security teams and the existing security mechanisms”transcript, stated

Use golden paths for secure-by-default development

Chose Common linear templates plus pluggable pink-box components over Every team inventing its own secure GenAI architecture.

Rationale: Templates accelerate builders while giving specialized teams flexible components for prompt engineering, evaluation, and model comparison.

“what everyone will do as the common path in the middle here that's in blue.”transcript, stated
“not everyone will do this, but a bunch of teams are gonna go do prompt engineering. So, we built these as functional components to the golden path.”transcript, stated

Make FAST modular by integration point

Chose Test either user interface or inference endpoint over A single mandatory testing surface.

Rationale: Some GenAI workloads are user-facing, while backend data pipelines may be better tested at the inference endpoint.

“they can integrate it towards where the user's interface, or towards the inference endpoint.”transcript, stated
“if there isn't actually a user interface for example, it's a backend data processing pipeline, the inference endpoint might be the more appropriate.”transcript, stated

Use AI classifiers for FAST scoring

Chose Automated response classification over Manual review of every test output.

Rationale: The number of prompts and outputs is too large for humans to review at Amazon scale.

“we use our own forms of AI to do evaluation against the outputs, because there's no way I could scan all of this with humans.”transcript, stated
“It then lets us interpret these results and builds up effectively a scorecard for the app”transcript, stated

Default to pipeline blocking on security test failures

Chose FAST can block promotion until owners remediate over Advisory-only AI security reporting.

Rationale: Blocking ensures critical findings are addressed before production, while remediation remains self-service for builders.

“user definable flag to some extent where we understand that blocking pipelines is a very dramatic action, but the default would be to block the pipeline. The pipeline owner then would go in and confirm and mitigate any findings, and so we get the results and be able to say, "Hey, I failed, I”transcript, stated
“That's typically the flow where it's all self-service and focused on what the builder can do for themselves”transcript, stated

Gotchas (what bit them)

Prompt injection lacks a single code-level fix.

Root cause: The attack is an unsafe interaction between user input and a nondeterministic model, not a deterministic query construction bug.

Resolution: Continuously test inputs and outputs, evaluate behavior, tune data/prompts, and add external validation and guardrails.

“Prompt injection on the other side is the unsafe interaction between a user, the input, right? And the model that you're working with”transcript, stated
“You can no longer just fix the code. You need to actively test your inputs, your outputs.”transcript, stated

Security reviews missed GenAI-specific failure questions until augmented.

Root cause: Existing mechanisms did not naturally ask about hallucinations, wrong product summaries, or prompt-level failures.

Resolution: Add GenAI questions to AppSec reviews, pentesting, bug bounty, incident response, policies, and guidance pages.

“what if it comes back with a wrong response, what if the product summary comes back wrong? These were questions that were not naturally in our security mechanisms”transcript, stated
“We already had a pentesting process, we already had a bug bounty program. So, we augmented those things with generative AI use cases.”transcript, stated

Internal assistants have different risk semantics than customer-facing retail assistants.

Root cause: Wrong internal knowledge such as a holiday schedule can directly change employee decisions.

Resolution: Adapt guardrails and threat thinking to internal context, access-control gaps, and employee impact.

“If you get the wrong holiday schedule, then you're gonna take a holiday you shouldn't.”transcript, stated
“In the internal chat bot experiences, we had a lot more of these use cases which were around the internal experience for an employee”transcript, stated

Enhanced discovery can reveal access-control gaps.

Root cause: Internal chatbots make company documents easier to find, including resources that may have been too broadly shared.

Resolution: Use discovery behavior to identify and remediate access-control issues.

“We were also able to find anything that maybe shouldn't have been broadly shared”transcript, stated
“understand what is our access control gaps and where we were seeing those as a result of the discovery”transcript, stated

Guardrails are not one-and-done controls.

Root cause: New attacks, bad customer experiences, hallucinations, and product defects appear after launch.

Resolution: Operate a flywheel that monitors, evaluates, adapts controls, and updates guardrails.

“At that point, you have guardrails, you've released it to production, but it's still not a hundred percent.”transcript, stated
“You adapt your controls, and then you continue to moderate and monitor.”transcript, stated

FAST can surface hallucinated or real PII in model responses.

Root cause: A malicious prompt can cause an app to emit personal-looking data, and the system must detect whether PII appears in outputs.

Resolution: Use classifiers to flag unsanitized PII and redact phone numbers in logs.

“We have seen cases where unfiltered PII data does come in the response, and it may have been hallucinated”transcript, stated
“We actually do redact the phone numbers in the logs, so that way we don't actually have to process 'em twice.”transcript, stated

Evaluation

Amazon evaluates GenAI applications through prompt testing, pre-prod FAST runs, maintained baseline attack prompts, AI classifiers for deflection/errors/PII/out-of-bound behavior, scorecards, pipeline gates, AppSec/pentest/bug-bounty augmentation, and post-launch monitoring/guardrail adaptation.

Metrics: Deflection classification · Error classification · PII disclosure classification · Out-of-bound response classification · FAST security scorecard · Pipeline pass/fail · Guardrail version and last-known-bug coverage · Production defects and customer experience flaws

5 supporting quotes
“Did it respond in a way that's like, "I'm sorry, as an AI assistant, I'm actually not allowed to do those types of things?"”transcript, stated
“Or, did it respond with an error? It's just like, "I'm sorry, something went wrong."”transcript, stated
“the classifier for GenAI sec PII is true. So like, it flagged it as yes, this contained PII.”transcript, stated
“For context, an out of bound response could be something like, "I have a assistive agent, which I used to explain or explore products or customer reviews, and it is very good at developing limericks."”transcript, stated
“that gives us every time they make a change, that they get a reevaluation against the default set of tests and we can get a score every time.”transcript, stated

Outcomes

The system is designed for Amazon-scale builder adoption: production GenAI apps had been live for a year by 2025, new GenAI techniques appear every couple of weeks, and FAST reruns a maintained baseline every time participating teams change code.

5 supporting quotes
“we released Amazon Q Business, we released Q Developer, we released, within the Amazon space, Rufus”transcript, stated
“we learned very quickly that this is a space we're not going to know everything. We're going to get comfortable with the fact that we will have new things come out”transcript, stated
“We saw a lot of lift in ability to discover resources across the company, especially common ones that people normally wouldn't have found.”transcript, stated
“we have been able to test so many things. We actually get a good pulse of like, where's the average in the Amazon ecosystem?”transcript, stated
“we optimize for trying to inject security into the current processes, whatever they may be.”transcript, stated