Building a Comprehensive AI Platform with SageMaker and Bedrock for Experience Management

qualtrics · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Experience-management AI platform

Qualtrics needed an internal platform that let researchers, scientists, engineers, knowledge workers, and application teams build, deploy, monitor, and consume ML, LLM, and agentic capabilities for the Qualtrics Experience Management suite.

Constraints:

• Qualtrics serves over 20,000 clients globally
• internal users need different workbenches for research, data, deployment, and app integration
• LLM access must span multiple providers and hosting environments
• capacity, rate limits, cost attribution, and billing codes must be controlled
• agentic features need orchestration, prompt lifecycle management, and observability

Architecture

Qualtrics built Socrates on AWS managed services. The platform includes Science Workbench for research, AI Data Infrastructure for data management, AI Playground for prototyping, model deployment and capacity management, a Unified GenAI Gateway for LLM/embedding access across SageMaker, Bedrock, and other providers, Managed Inference APIs, Prompt Lifecycle Management, and a LangGraph-based agent platform.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  science-workbench[Science Workbench] --> ai-data-infrastructure[AI Data Infrastructure]
  ai-data-infrastructure --> ai-playground[AI Playground]
  science-workbench --> model-deployment[Model deployment]
  model-deployment --> managed-inference-apis[Managed Inference APIs]
  ai-playground --> unified-genai-gateway[Unified GenAI Gateway]
  unified-genai-gateway --> managed-inference-apis
  unified-genai-gateway --> prompt-lifecycle-management[Prompt Lifecycle Management]
  prompt-lifecycle-management --> agent-platform[Agent Platform]

Components

Data flow

Researchers and scientists build or train models in Socrates workbenches over governed data. Models are deployed through SageMaker-backed inference infrastructure and exposed through Managed Inference APIs. LLM and embedding requests flow through the Unified GenAI Gateway, which handles provider access, authentication, attribution, rate limiting, and cost controls. Agentic applications use LangGraph and shared observability/prompt infrastructure.

Why this architecture

Socrates abstracts common AI infrastructure so application teams can use models without owning deployment, capacity, provider integration, authentication, attribution, or observability. The gateway and managed inference layers also let Qualtrics control cost and rate limiting across many internal users and use cases.

Models used: Amazon SageMaker models · Amazon Bedrock models · OpenAI GPT · Meta Llama 3 8B

Scale

• Qualtrics serves over 20,000 clients globally.
  “As the creators and stewards of the Experience Management (XM) category, Qualtrics serves over 20,000 clients globally, bringing humanity, connection, and empathy back to businesses across various industries, including retail, government, and healthcare.”article, stated
• SageMaker inference features reduce FM deployment costs by 50 percent and latency by 20 percent on average.
  “reduce FM deployment costs by 50% on average and latency by 20% on average”article, stated
• Auto-scaling improvements reduce scaling times by up to 40 percent and detection speed six times for Meta Llama 3 8B.
  “reduced auto scaling times by up to 40% for models like Meta Llama 3 and increased auto scaling detection speed by six times faster.”article, stated

Alternatives considered (roads not taken)

Application teams integrate directly with each model/provider

Rejected because: The gateway centralizes access, authentication, attribution, rate limiting, and provider complexity.

Self-manage every deployment path

Rejected because: Managed Inference APIs handle deployment, scaling, maintenance, and production SLAs.

Patterns

model-routing (model-ops)

The Unified GenAI Gateway provides one interface across LLM and embedding providers.

prompt-versioning (prompt-ops)

Prompt Lifecycle Management maintains prompt security and governance.

multi-agent-orchestration (agent-design)

Socrates includes a LangGraph-based agent platform for graph-based agentic features.

semantic-caching (optimization)

The gateway plans semantic caching to optimize inference performance.

Tradeoffs

Platform abstraction

Chose centralized gateway and managed inference over each team building direct model integrations.

Rationale: Centralization simplifies provider churn, cost attribution, rate limits, and product integration.

Inference modes

Chose both asynchronous and synchronous managed APIs over one inference mode for all workloads.

Rationale: Different application use cases have different latency and traffic needs.

Gotchas (what bit them)

Provider sprawl creates integration and governance overhead.

Root cause: LLMs and embedding models run across multiple providers and hosting environments.

Resolution: Route access through a Unified GenAI Gateway.

Unmanaged inference capacity can break cost and reliability goals.

Root cause: Multiple consumers share model capacity with different traffic patterns.

Resolution: Monitor usage and set rate limiting and auto-scaling policies.

Evaluation

Qualtrics monitors deployment cost, latency, auto-scaling time, detection speed, rate limits, resource usage, production SLAs, and integration reliability through Socrates and SageMaker Inference improvements.

Metrics: 50% FM deployment cost reduction · 20% latency reduction · 40% auto-scaling time reduction · six-times faster detection · production SLAs

Outcomes

• Socrates provides unified model access, managed inference, prompt governance, and agent orchestration for Qualtrics teams.
• SageMaker Inference optimizations reduced cost, latency, and auto-scaling times for generative AI workloads.

Qualtrics serves over 20,000 clients globally and runs a multi-persona internal platform for researchers, scientists, engineers, and knowledge workers.