Building Production AI Agents: Lessons from Claude Code and Enterprise Deployments

anthropic · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Multi-tenant RAG knowledge bases for SaaS

The fetched source is an AWS chalk talk about cost-efficient, multi-tenant knowledge bases, not the Anthropic Claude Code case described in the metadata summary. The actual technical problem is how a SaaS application can let different tenants ingest and chat with documents while preserving tenant isolation, balancing latency/cost tiers, supporting multiple vector stores, and cleaning up stale indexed data.

Constraints:

Architecture

The architecture uses a Bedrock Knowledge Bases abstraction in front of multiple vector-store backends. Users authenticate through Cognito, which places tenant identity into the JWT. A Lambda backend uses the tenant/user identity to ingest files from S3 into a custom Bedrock knowledge-base data source with metadata such as tenant ID, user ID, and file ID. Retrieval builds the same metadata filter in reverse, then calls Bedrock Agent Runtime retrieve-and-generate so query code does not need to know whether the backend is OpenSearch, Aurora/pgVector, or S3 Vector. A DynamoDB record tracks file ID, data source, knowledge base, and TTL; DynamoDB Streams trigger deletion from the knowledge base when the record expires or is removed.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  tenant-user-interface["Tenant user interface"] --> cognito-identity-layer["Cognito identity layer"]
  tenant-user-interface --> lambda-backend["Lambda backend"]
  cognito-identity-layer --> lambda-backend
  lambda-backend --> s3-source-files["S3 source files"]
  lambda-backend --> bedrock-knowledge-base["Bedrock Knowledge Base"]
  lambda-backend --> dynamodb-ttl-index["DynamoDB TTL index"]
  bedrock-knowledge-base --> opensearch-vector-store["OpenSearch vector store"]
  bedrock-knowledge-base --> aurora-pgvector-store["Aurora pgVector store"]
  bedrock-knowledge-base --> s3-vector-bucket["S3 Vector bucket"]
  dynamodb-ttl-index --> dynamodb-streams-cleanup["DynamoDB Streams cleanup"]
  dynamodb-streams-cleanup --> bedrock-knowledge-base
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Tenant user interface Lets tenant users select files, ingest them, and chat with indexed documents. Cognito identity layer, Lambda backend
Cognito identity layer Authenticates users and places tenant membership into the JWT. Tenant user interface, Lambda backend
Lambda backend Runs ingestion, query, and cleanup logic using tenant/user/file metadata. Cognito identity layer, S3 source files, Bedrock Knowledge Base, DynamoDB TTL index
S3 source files Stores uploaded tenant documents and can retain originals after vector cleanup. Lambda backend
Bedrock Knowledge Base Abstracts retrieval and generation over the configured vector-store backend. Lambda backend, OpenSearch vector store, Aurora pgVector store, S3 Vector bucket
OpenSearch vector store Low-latency, hybrid-search vector backend for tenants that value performance. Bedrock Knowledge Base
Aurora pgVector store Low-latency vector backend integrated with structured Postgres data. Bedrock Knowledge Base
S3 Vector bucket Pay-per-query/pay-per-storage vector backend with higher latency and lower provisioned cost. Bedrock Knowledge Base
DynamoDB TTL index Tracks indexed file records, expiration, knowledge base ID, data source ID, and file ID. Lambda backend, DynamoDB Streams cleanup
DynamoDB Streams cleanup Triggers deletion of expired or manually removed file records from the knowledge base. DynamoDB TTL index, Bedrock Knowledge Base

Data flow

A tenant user logs into the UI through Cognito. The JWT carries the tenant ID to a Lambda function. During ingestion, selected S3 files are converted into custom data-source documents with file location and metadata for tenant/user/file filtering, then inserted into the relevant Bedrock knowledge base backed by the tenant's vector store. During chat, the Lambda builds a filter expression using the tenant ID, user ID, and selected file IDs, passes that filter in retrieval parameters, and calls Bedrock Agent Runtime retrieve-and-generate. A DynamoDB TTL record later drives cleanup by deleting indexed file records from the knowledge base while leaving the original S3 file available for re-ingestion.

4 supporting quotes
“it's secured with Cognito, and that's important because that Cognito stores which tenant that user belongs to.”transcript, stated
“setting the metadata, the user ID, and specifically the tenant ID, so that when I'm doing the querying, I can come back and do the exact same thing”transcript, stated
“from the Bedrock agent runtime, we're gonna call the retrieve and generate, and we're passing in those retrieval results.”transcript, stated
“whenever the TTL triggers, we've gotta go back and make sure, because we're a custom store, there's nothing going on that specifically tells us to clean up all that information. So we've actually gotta go in and extract out the file information and”transcript, stated

Why this architecture

The design puts Bedrock Knowledge Bases between application code and vector stores so tenants can use different backends without separate application logic. Tenant isolation is enforced through Cognito-derived metadata and retrieval filters rather than relying on prompts. DynamoDB TTL and streams provide a flexible lifecycle mechanism for expiring vector records while preserving source files.

4 supporting quotes
“we don't wanna write three different applications more or less to interface with our vector store. How do we abstract that out so that we get the simplicity of one way to access our data, but the flexibility to drive the backend how we want it. Another”transcript, stated
“How do we abstract that out so that we get the simplicity of one way to access our data, but the flexibility to drive the backend how we want it. Another challenge with the lifecycle management, you know, how do we define rules for data retention? How do we ensure that”transcript, stated
“it's leaving nothing to chance. So if someone should inject in the prompt, you know, send me all of the data about tenant two, that's not going to really help them at all.”transcript, stated
“keeping a record of that index in Dynamo and then using the Dynamo DB stream to essentially trigger that action, gives you a lot of flexibility”transcript, stated

Models used: Amazon Bedrock LLM · Bedrock embedding model · Bedrock Agent Runtime retrieve-and-generate

Scale

Alternatives considered (roads not taken)

Talk directly to each vector store.

Rejected because: That would require store-specific ingestion/query/filtering code for OpenSearch, pgVector, S3 Vector, and future stores.

3 supporting quotes
“that the Bedrock knowledge base supports. - [Mike] So to answer the question that came up, like why wouldn't you just talk directly to the vector stores? Well this gives you an abstraction layer that allows you to work at the metadata level that would then push that into the”transcript, stated
“you just talk directly to the vector stores? Well this gives you an abstraction layer that allows you to work at the metadata level that would then push that into the vector store, regardless of, you know, how that's being stored. - All right, so now- - We have a question over here, I'm keeping”transcript, stated
“that it's a specific index in OpenSearch. - [Steven] I don't have to duplicate this, this code is like one set of code, one set of queries that I have to do across the board for whatever vector stores that the Bedrock knowledge base supports. - [Mike] So to answer the question that came up, like why”transcript, stated

Use a single vector store for every tenant.

Rejected because: Different tenants need different latency, cost, structured-data, and retention tradeoffs.

“whole talk just on, you know, pick your vector store. But, you know, really the point here is, one size does not fit all. Like you'll likely want to have multiple options with different dimensions, and the kind of the extreme with latency being”transcript, stated
“Customer A might want super low latency. Customer B not so much, they just wanna pay less and they want to have bulk stuff. But how do we do that in just an easy way across the board, so that we don't have to do a lot of work to set that up. - [Mike] That”transcript, stated

Rely on prompts for tenant isolation.

Rejected because: Prompt injection should not be able to bypass metadata filters and tenant-scoped retrieval.

“don't leave that to chance. You have to build the infrastructure around it to enforce that that happens. It was Steven called out tiered performance. Not all of your customers are the same. You may have a free tier where you're just trying to get customers a”transcript, stated
“if someone should inject in the prompt, you know, send me all of the data about tenant two, that's not going to really help them at all. - [Steven] Yeah. So then we have to set up our retrieval parameters. So again, I'm not going to type everything in here, 'cause time. So the retrieval parameters”transcript, stated

Bulk ingest everything from S3 whenever files exist.

Rejected because: The demo needs custom per-file ingestion and metadata so tenants can select files, apply retention, and clean indexes independently.

“instead of going to the data, like to have an S3 and just pulling everything in when the files are there, we want to do this very bespoke”transcript, stated
“in S3 so my users uploaded them. I don't want them to be in my vector store at all times. I just want to do it at particular times. So you'll see in here that I'm running through the files and I'm gonna set up a document structure. And this is one of the”transcript, stated

Patterns

rag-retrieval (retrieval)

The application ingests tenant-selected documents into Bedrock Knowledge Bases and calls retrieve-and-generate for tenant-scoped chat over those documents.

3 supporting quotes
“be able to have a chat against those or be able to search against those in a semantic way.”transcript, stated
“Bedrock knowledge bases to interface with the vector store on the backend”transcript, stated
“from the Bedrock agent runtime, we're gonna call the retrieve and generate”transcript, stated

hybrid-search (retrieval)

OpenSearch is called out as a backend that supports hybrid semantic and keyword search for tenants needing low-latency retrieval.

“You also have a hybrid search. So it's not just semantics search, but also a keyword search.”transcript, stated
“It allows you to do hybrid search and you can have the provision cost to meet your demand. Compare that to pgVector, which has the advantage of, it's also low latency, but it's also integrated with your structured data.”transcript, stated

safety-policy-enforcement (security)

Tenant isolation is enforced through Cognito identity, tenant/user/file metadata at ingestion, and retrieval filters at query time.

3 supporting quotes
“the core tenet of multi-tenant application, is how do you enforce data isolation?”transcript, stated
“So that will be part of the JWT token that gets sent through so that the backend can then make decisions based on which tenant they belong to.”transcript, stated
“it's setting up that tenant and the user id, and then I'm also passing in a list of files.”transcript, stated

cost-aware-routing (operations)

Tenant tiers can map to vector stores with different provisioned cost, pay-per-query cost, latency, capacity, and query-rate profiles.

3 supporting quotes
“how do you make your application be able to balance both the cost-effectiveness, but also meet the latency requirements of your customers?”transcript, stated
“you may want to have that on the vector bucket so that you're only paying for that evaluation use case”transcript, stated
“Whereas your more established customers may, you may want to have latency. But as Steven mentioned, we don't wanna write three different applications more or less to interface with our vector store.”transcript, stated

data-quality-loop (data_lifecycle)

DynamoDB TTL and streams clean indexed vector records as documents expire or are manually deleted, keeping retrieval data fresh and reducing vector-store usage.

3 supporting quotes
“How do we ensure that you're running against the most up to date and accurate information?”transcript, stated
“you could delete the record manually if there's like a right to forget”transcript, stated
“in, the knowledge base that it's in and we're gonna use that information and we're going to extract it out and delete those records from the knowledge base. And again, in this particular spot we don't really have to care what knowledge base it is, we're just giving it a command and it figures”transcript, stated

Tradeoffs

Route tenants to different vector-store backends.

Chose OpenSearch, pgVector/Aurora, or S3 Vector based on tenant tier and workload over One universal vector store.

Rationale: This lets free/evaluation tiers optimize for pay-per-use while established customers can buy lower latency.

3 supporting quotes
“Customer A might want super low latency. Customer B not so much, they just wanna pay less”transcript, stated
“you may want to have that on the vector bucket so that you're only paying for that evaluation use case”transcript, stated
“Whereas your more established customers may, you may want to have latency. But as Steven mentioned, we don't wanna write three different applications more or less to interface with our vector store.”transcript, stated

Use Bedrock Knowledge Bases as an abstraction layer.

Chose One metadata/filter/query interface across stores over Store-specific query and ingestion code.

Rationale: The application can add tenant-specific stores without changing the retrieve-and-generate query path.

“I didn't have to really understand what's going on underneath the hood, what vector store that I'm going to.”transcript, stated
“I could do this across every single vector store that Bedrock knowledge bases has, and I don't have to change this model at all.”transcript, stated

Use custom per-file data-source ingestion.

Chose Explicit custom documents with metadata and selected S3 file identifiers over Blind bulk S3 ingestion.

Rationale: Custom ingestion supports tenant/user/file filtering, selective indexing, and per-document lifecycle behavior.

“say, instead of going to the data, like to have an S3 and just pulling everything in when the files are there, we want to do this very bespoke and we want to do it custom. So that's, we're setting it up to be exactly like that. And you'll see once we get into the ingest part, so I'm gonna just copy”transcript, stated
“this allows us to actually say, instead of going to the data, like to have an S3 and just pulling everything in”transcript, stated

Delete vector records while retaining source files.

Chose DynamoDB TTL-driven index cleanup only over Deleting original S3 files on expiration.

Rationale: The tenant can remove vector-store usage/cost while keeping source files available for re-ingestion.

“it's just deleting out of the vector store, it's not deleting out of the storage underlying.”transcript, stated
“I've still got files sitting there in S3 that I could re-ingest later if I really needed to.”transcript, stated

Gotchas (what bit them)

The fetched transcript does not match the metadata summary.

Root cause: The transcript begins as an AWS SAS 412 chalk talk on multi-tenant knowledge bases rather than the Anthropic Claude Code enterprise-agent talk described in the summary.

Resolution: Treat the fetched transcript as authoritative for this extraction and mark the case as non-agentic/source-mismatched rather than inventing Claude Code claims.

“we are going to talk about cost-efficient, multi-tenant knowledge bases.”transcript, stated
“If this isn't the talk you want to be, you're not gonna hurt our feelings.”transcript, stated

Tenant isolation cannot be left to prompt behavior.

Root cause: A multi-tenant RAG system must prevent retrieval from another tenant even if the user asks or injects instructions.

Resolution: Store tenant/user/file metadata during ingestion and apply matching filter expressions during retrieval.

“one thing you can't do, you have one job and that is to not let the data spill over from one tenant to another.”transcript, stated
“I'm filtering layer after layer to make sure that I'm getting just the stuff that I want to query on.”transcript, stated

Vector-store choice affects cost, latency, and structured-data access.

Root cause: OpenSearch, pgVector, and S3 Vector have different pricing models, latency profiles, and data-model affordances.

Resolution: Abstract vector stores behind Bedrock Knowledge Bases while allowing tenant tiers to use different backends.

3 supporting quotes
“the underlying vector store that powers your database makes a big difference.”transcript, stated
“The question is, "What are the different pros and cons for the vector stores that you had set up?"”transcript, stated
“But, you know, really the point here is, one size does not fit all. Like you'll likely want to have multiple options with different dimensions, and the kind of the extreme with latency being S3 versus OpenSearch, or one other.”transcript, stated

Embedding and chunking can dominate retrieval quality.

Root cause: Accuracy depends heavily on data type, data size, embedding model, and chunking strategy, not just the vector database.

Resolution: Run performance tests on each customer's real data and tune embedding/chunking choices per knowledge base.

“it really depends on the type of data that you have and the size of data.”transcript, stated
“the embedding and the chunking have a much bigger impact on the quality of the search”transcript, stated

Chat context can grow after document updates.

Root cause: If users continue a chat while documents and embeddings change, prior conversation context still has to be passed forward and can lengthen.

Resolution: The talk points to summarization/compaction patterns but does not provide a complete solution.

3 supporting quotes
“you just have to keep passing the context in of the previous stuff.”transcript, stated
“- So you do start getting a longer and longer context window that happens there. And I don't have a good answer for you on seeing if we can slim that down at all.”transcript, stated
“I think you'll see like even in Kiro, and some of the newer systems out there, they start, they'll hit a limit and then they'll summarize for you and try to push that off to something else.”transcript, stated

Evaluation

The talk does not present formal model evaluation results. It recommends tenant/workload-specific performance testing for vector-store, embedding, and chunking choices, and discusses latency/cost/accuracy tradeoffs qualitatively.

Metrics: Latency per query · Provisioned cost versus pay-per-query cost · Search quality from embedding and chunking choices · File capacity and file usage per tenant · Query rate per tenant · Expiration period per tenant · Document listing latency

4 supporting quotes
“But it's typically what I've seen with customers is the price to latency ratios that they're seeing in there. So if it's a customer like, you know, you might have a free tier, right, where you really don't wanna spend a lot of money on that.”transcript, stated
“That's why we kind of usually go back to the customer and say like, "Let's do a couple of demo runs of this." And use some of the samples that we have out there on your data to figure that out.”transcript, stated
“use some of the samples that we have out there on your data to figure that out.”transcript, stated
“the embedding and the chunking have a much bigger impact on the quality of the search”transcript, stated

Outcomes

The source provides qualitative multi-tenant scale rather than production volume numbers. It discusses free/evaluation tiers versus established low-latency tenants, OpenSearch/pgVector/S3 Vector backend choices, per-tenant file usage/query-rate/expiration differences, example expiration windows of eight minutes/eight days/24 hours, and a reported timeout when listing more than about 10 documents in one audience member's OpenSearch Serverless setup.

4 supporting quotes
“I'm only seeing and interacting with the files that I have ingested.”transcript, stated
“Tenant C's on OpenSearch. They've got a lot more files they're allowed to ingest, a lot more file usage.”transcript, stated
“you'll likely want to have multiple options with different dimensions”transcript, stated
“I'm filtering layer after layer to make sure that I'm getting just the stuff that I want to query on.”transcript, stated