Multi-Layered Caching Architecture for AI Metadata Service Scalability

salesforce · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI inference metadata infrastructure

Salesforce's AI Metadata Service introduced latency and availability risk into every AI inference request because tenant-specific configuration had to be fetched synchronously from shared backend dependencies.

Constraints:

Every Agentforce and AI inference request required tenant-specific metadata.
Metadata retrieval added about 400ms at P90.
Backend database failures could halt inference workflows platform-wide.
Different metadata types have different freshness requirements.

Architecture

Salesforce implemented a two-layer caching architecture: L1 client-side caching inside the AIMS client integrated with AI Gateway, and L2 service-level caching inside AIMS to avoid database and downstream-service calls while preserving resilience during backend outages.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ai-gateway["AI Gateway"]
  aims-client-l1-cache["AIMS Client L1 Cache"]
  aims-service-l2-cache["AIMS Service L2 Cache"]
  backend-metadata-services["Backend Metadata Services"]
  observability-hooks["Observability Hooks"]
  ai-gateway --> aims-client-l1-cache
  aims-client-l1-cache --> ai-gateway
  aims-client-l1-cache --> aims-service-l2-cache
  aims-service-l2-cache --> aims-client-l1-cache
  aims-service-l2-cache --> backend-metadata-services
  backend-metadata-services --> aims-service-l2-cache
  observability-hooks --> aims-service-l2-cache
  observability-hooks --> aims-client-l1-cache

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`AI Gateway`	Handles inference requests requiring tenant metadata.	`AIMS Client L1 Cache`
`AIMS Client L1 Cache`	Provides local metadata cache hits without network calls.	`AI Gateway`, `AIMS Service L2 Cache`
`AIMS Service L2 Cache`	Caches longer-lived metadata inside the service and buffers backend outages.	`AIMS Client L1 Cache`, `Backend Metadata Services`
`Backend Metadata Services`	Provide canonical tenant configuration and CDP admin data.	`AIMS Service L2 Cache`
`Observability Hooks`	Track cache age, usage, invalidation, and backend-health signals.	`AIMS Service L2 Cache`, `AIMS Client L1 Cache`

Data flow

AI Gateway requests tenant metadata through the AIMS client. The client checks L1 first; on misses, AIMS checks L2 before calling backend services or the shared database. Background refresh and observability keep cache entries fresh and visible.

“The first layer consists of local caching within the AIMS Client, which is integrated into the AI Gateway service.”summary, stated

“The second layer implements service-level caching within the AIMS itself, designed to avoid unnecessary database and downstream service calls.”summary, stated

Why this architecture

Caching worked because tenant AI metadata changes infrequently but is required on every inference request. Layered caches reduce latency, protect availability, and preserve freshness flexibility with configurable TTLs.

“The approach recognized that AI metadata changes infrequently, making caching a highly effective strategy once properly implemented.”summary, stated

“The team made TTLs configurable per use case, recognizing that different types of metadata have different freshness requirements.”summary, stated

Models used: OpenAI ChatGPT · Salesforce internal models · Agentforce

Scale

Metadata retrieval contributed about 400ms to P90 latency.
“metadata retrieval alone contributing approximately 400 milliseconds to P90 latency per request.”summary, stated
End-to-end inference latency reached 15,000ms at P90 before the fix.
“contributing to overall end-to-end latency that reached 15,000 milliseconds at the P90 percentile.”summary, stated
A database incident disrupted metadata fetches for about 30 minutes.
“resulting in approximately 30 minutes of disrupted metadata fetches that halted inference workflows across the platform.”summary, stated
The cache reduced retrieval latency from 400ms to sub-millisecond and improved end-to-end request latency by 27%.
“reducing metadata retrieval latency from 400ms to sub-millisecond response times and improving end-to-end request latency by 27%”summary, stated
The system maintained 65% availability during backend outages.
“maintaining 65% availability during backend outages.”summary, stated

Alternatives considered (roads not taken)

Fetch metadata synchronously for every inference request

Rejected because: It introduced a 400ms P90 bottleneck and exposed every request to backend outages.

“every AI inference request required real-time metadata retrieval from AIMS, which depended on a shared backend database and multiple downstream services”summary, stated

Use one fixed cache policy for all metadata

Rejected because: Model preferences and trust attributes have different freshness requirements.

“Model preferences might remain static for weeks, while trust attributes may require more frequent updates.”summary, stated

Patterns

`observability-tracing` (operations)

Cache age, usage, invalidation, and L2 reliance are monitored to detect backend problems early.

“Observability hooks were integrated to monitor cache age, track usage patterns, and enable proactive cache invalidation when necessary.”summary, stated

`model-routing` (architecture)

Tenant metadata determines AI provider, model, tuning parameters, routing, and context application.

“Each tenant may use different AI providers (OpenAI's ChatGPT or Salesforce's internal models) with unique tuning parameters”summary, stated

`safety-policy-enforcement` (governance)

Tenant-specific trust and configuration metadata shape request routing and context application.

“tenant-specific configuration for every inference request.”summary, stated

Tradeoffs

Serve stale-but-valid metadata during outages

Chose L2 cache resilience over Failing closed on every backend outage.

Rationale: Many configurations change infrequently, so limited staleness preserves AI availability.

“it can safely serve potentially stale but valid responses during backend failures.”summary, stated

Standardize caching through annotations

Chose SmartCacheable annotations in the Scone framework over Each team implementing custom caching logic.

Rationale: Standardization gives teams shared guardrails and easier adoption.

“SmartCacheable annotations (both reactive and non-reactive variants) that enabled other teams to adopt caching capabilities without implementing custom caching logic.”summary, stated

Gotchas (what bit them)

Metadata services can become hidden inference bottlenecks.

Root cause: Every AI request synchronously depended on metadata retrieval and backend database health.

Resolution: Move frequently used metadata into L1 and L2 caches with background refresh.

“This created a synchronous dependency chain that introduced significant performance penalties.”summary, stated

Shared databases create noisy-neighbor risk for inference.

Root cause: AIMS shared a database with other CDP services, so unrelated load degraded AI metadata performance.

Resolution: Use caching as a resilience buffer and add alerts for shifts toward L2 usage.

“Because AIMS shared its database with other CDP services, resource contention from high usage by other services frequently degraded AIMS performance.”summary, stated

Evaluation

Salesforce measured retrieval latency, end-to-end latency, outage availability, cache usage, and cache age.

Metrics: P90 metadata latency · P90 end-to-end latency · availability during outages · cache age · L2 usage

“reducing metadata retrieval latency from 400ms to sub-millisecond response times and improving end-to-end request latency by 27% while maintaining 65% availability during backend outages.”summary, stated

Outcomes

Metadata retrieval dropped from about 400ms to sub-millisecond.
End-to-end request latency improved by 27%.
The system preserved 65% availability during backend outages.

The AIMS cache protects multi-cloud, multi-tenant AI inference requests including Agentforce.