Building Enterprise-Grade GenAI Platform with Multi-Cloud Architecture

coinbase · Finance · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise GenAI platform operations

Coinbase needed to operationalize GenAI across employee and customer applications, but production use quickly exceeded the simple cost-versus-accuracy model of LLM selection. The platform had to handle frequently changing model quality, trust and safety guardrails, cost controls, latency, provider availability, capacity quotas, RAG grounding, self-service builders, and emerging agentified workflows across multiple clouds and model providers.

Constraints:

• LLM leaderboards change frequently, so model choice must remain flexible across use cases.
• There are no clear universal trust-and-safety benchmarks, so Coinbase needed its own guardrails.
• Model costs vary dramatically by provider, model size, and GPU requirements.
• Latency can range from seconds to tens of seconds, and conversational use cases are latency-sensitive.
• Provider demand and GPU shortages can create rationed quotas and quality-of-service issues for large or bursty deployments.

Architecture

Coinbase built CB-GPT as a unified GenAI platform spanning AWS Bedrock, GCP VertexAI, Azure GPT, and open-source LLMs. A routing and decision layer selects models for each use case based on accuracy, cost, latency, availability, and safety needs. Platform services include an internal LLM evaluation framework, rate limiting, usage tracking, billing dashboards, semantic caching, load/latency benchmarks, RAG over enterprise search, web search, vector storage, input/output guardrails, API access for engineers, Studio for no-code assistants, and experiments with self-hosted open-source and agentified LLM chains.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  cb-gpt-platform[CB-GPT platform] --> model-routing-layer[Model routing layer]
  cb-gpt-platform --> rag-layer[RAG layer]
  cb-gpt-platform --> guardrails-layer[Guardrails layer]
  cb-gpt-platform --> cb-gpt-api[CB-GPT API]
  cb-gpt-platform --> cb-gpt-studio[CB-GPT Studio]
  cb-gpt-platform --> agentified-workflow-layer[Agentified workflow layer]
  model-routing-layer --> aws-bedrock[AWS Bedrock]
  model-routing-layer --> gcp-vertexai[GCP VertexAI]
  model-routing-layer --> azure-gpt[Azure GPT]
  model-routing-layer --> open-source-llms[Open-source LLMs]
  model-routing-layer --> decision-framework[Decision framework]
  model-routing-layer --> load-and-latency-benchmarks[Load and latency benchmarks]
  internal-llm-evaluation-framework[Internal LLM evaluation framework] --> model-routing-layer
  internal-llm-evaluation-framework --> decision-framework
  rate-limiting-and-usage-dashboards[Rate limiting and usage dashboards] --> cb-gpt-platform
  semantic-cache[Semantic cache] --> cb-gpt-platform
  rag-layer --> enterprise-search[Enterprise search]
  rag-layer --> web-search[Web search]
  rag-layer --> vector-storage[Vector storage]
  guardrails-layer --> cb-gpt-platform
  cb-gpt-api --> cb-gpt-platform
  cb-gpt-studio --> cb-gpt-platform
  agentified-workflow-layer --> data-extraction-agent[Data extraction agent]
  data-extraction-agent --> data-analysis-agent[Data analysis agent]
  data-analysis-agent --> report-generation-agent[Report generation agent]

Components

Data flow

Applications and CB-GPT Studio users send requests through CB-GPT's unified API or no-code interface. The platform applies a decision framework, routes the request to cloud-hosted or open-source LLMs, retrieves grounding data from enterprise search, web search, or vector stores when RAG is needed, and applies guardrails to both inputs and outputs. Usage, costs, latency, load, and model performance are tracked through platform dashboards and benchmarks. For agentified workflows, specialized agents can be chained so one extracts data, another analyzes it, and another generates reports.

5 supporting quotes

“Coinbase We recognized the power of this technology and have built a platform (CB-GPT) that serves as a unified interface for all GenAI-powered applications at Coinbase. The key characteristics of CB-GPT that allow us to solve for the challenges above include: ### 1. Multi-Cloud, Multi-LLM”article, stated

“truly multi-cloud (across AWS Bedrock, GCP VertexAI, Azure GPT and open-source LLMs) with different use cases routed to the appropriate destination. ”article, stated

“we extensively use a technique called RAG (Retrieval Augmented Generation) to ensure our responses are anchored in reliable sources of truth.”article, stated

“applications, it is crucial to design a solution architecture with appropriate guardrails. These guardrails should evaluate both input and output information to ensure that the LLM's responses adhere to the 'Three H Principle' (Helpful, Harmless, and Honest). ### 4. CB-GPT API”article, stated

“one agent might handle data extraction, another might focus on data analysis, and a third might generate reports based on the analyzed data.”article, stated

Why this architecture

Coinbase chose a multi-cloud, multi-LLM platform because model quality, price, latency, safety behavior, and capacity change frequently. A single provider would expose CB-GPT to vendor lock-in, quota limits, and model-leaderboard volatility. RAG, guardrails, semantic caching, evaluation, and self-service API/Studio surfaces make GenAI usable across many teams while keeping production controls centralized.

Models used: AWS Bedrock · GCP VertexAI · Azure GPT · Open-source LLMs · OpenAI GPT-4 · OpenAI GPT-3.5 · Anthropic Claude 3 Opus · Anthropic Claude 3 Haiku · ChatGPT · Google Gemini

Scale

• Coinbase's customer-facing conversational LLM chatbot served all US consumers after launching in June 2024.
  “Coinbase’s customer-facing conversational LLM chatbot, serving all our US consumers, was launched in June, 2024”article, stated
• CB-GPT was designed as the unified interface for all GenAI-powered applications at Coinbase.
  “serves as a unified interface for all GenAI-powered applications at Coinbase.”article, stated
• GPT-4 was cited as 10x more expensive than GPT-3.5.
  “OpenAI's GPT-4 is 10x more expensive than GPT-3.5”article, stated
• Claude 3 Opus was cited as 60x the price of Claude 3 Haiku.
  “Anthropic's Claude 3 Opus is 60x the price of Claude 3 Haiku.”article, stated
• Model latency ranged from a few seconds to tens of seconds.
  “The latency of different models can vary from a few seconds to tens of seconds”article, stated
• Several dozen use cases had been built by non-ML teams with CB-GPT.
  “Till date, several dozen use cases have been built by non-ML teams, using CB-GPT.”article, stated
• The platform supported AWS Bedrock, GCP VertexAI, Azure GPT, and open-source LLMs.
  “across AWS Bedrock, GCP VertexAI, Azure GPT and open-source LLMs”article, stated

Alternatives considered (roads not taken)

Optimize only the accuracy-cost tradeoff.

Rejected because: Production GenAI required additional dimensions: trust and safety, scalability, latency, model availability, and changing model quality.

Tie the platform to one cloud vendor or model provider.

Rejected because: Frequent model-leaderboard changes and quota/availability issues required provider flexibility.

Use expensive frontier models for every task.

Rejected because: Coinbase matched model capability to task complexity, using cheaper models for simple tasks and stronger models for complex ones.

Expose GenAI only through engineering APIs.

Rejected because: Coinbase wanted technical and non-technical employees to build relevant assistants, so it added no-code Studio alongside APIs.

Use only third-party hosted LLM APIs.

Rejected because: Coinbase began working with self-hosted open-source LLMs for cost management and Coinbase/crypto-specific fine-tuning, despite higher technical difficulty.

Patterns

model-routing (platform architecture)

CB-GPT routes different use cases across AWS Bedrock, GCP VertexAI, Azure GPT, and open-source LLMs using a decision framework that accounts for model quality, price, latency, and availability.

rag-retrieval (grounding)

Coinbase built RAG into CB-GPT so enterprise answers could be anchored in Help Center articles, enterprise search, web search, and vector-backed bespoke data sources.

guardrails-io-filtering (safety)

The platform evaluates inputs and outputs to reduce hallucination, jailbreaking, and responses that violate helpful, harmless, and honest behavior.

semantic-caching (cost optimization)

CB-GPT uses semantic caching to serve previously asked questions without a new LLM invocation, reducing cost for repeated or similar requests.

eval-driven-development (quality)

Coinbase built internal evaluation and load/latency benchmarks for Coinbase- and crypto-specific use cases, using them to monitor performance across the platform's model options.

cost-aware-routing (operations)

The platform tracks usage and billing and routes simpler tasks to cheaper LLMs while reserving more capable models for higher-value work.

multi-agent-orchestration (agent architecture)

Coinbase describes agentified LLM solutions as chains of specialized agents for extraction, analysis, and report generation on tasks too complex for one model alone.

Tradeoffs

Build a multi-cloud, multi-LLM platform.

Chose CB-GPT across AWS Bedrock, GCP VertexAI, Azure GPT, and open-source LLMs. over A single cloud or model provider..

Rationale: The multi-cloud architecture reduced lock-in and allowed Coinbase to change models as leaderboards, costs, latency, and availability shifted.

Use task-appropriate model tiers.

Chose Cheaper LLMs for simple tasks and stronger, more expensive LLMs for complex tasks. over Using the highest-capability model for everything..

Rationale: Cost differences were large enough that model choice had to align with task value and accuracy needs.

Ground enterprise answers with RAG.

Chose RAG over reliable sources such as Help Center articles, enterprise search, web search, and vector stores. over Ungrounded model responses..

Rationale: Coinbase needed responses anchored in reliable sources of truth, especially for customer and enterprise use cases.

Expose both API and no-code Studio interfaces.

Chose CB-GPT APIs for engineers plus CB-GPT Studio for no-code assistant creation. over Serving only technical users through APIs..

Rationale: The platform needed to support diverse job functions and let non-ML teams create assistants rapidly.

Explore self-hosted open-source LLMs.

Chose Open-source LLMs hosted inside Coinbase infrastructure. over Only third-party hosted LLMs charged per API call..

Rationale: Self-hosting was technically harder but promised lower cost and higher accuracy through Coinbase- or crypto-specific fine-tuning.

Gotchas (what bit them)

The initial cost-versus-accuracy framing missed production constraints.

Root cause: Enterprise-grade LLM systems also need trust and safety, scalability, evolving model choices, latency, and availability.

Resolution: Coinbase expanded CB-GPT into a platform with routing, guardrails, benchmarks, caching, RAG, and availability planning.

Provider capacity and quality-of-service became an unexpected blocker.

Root cause: High demand and GPU shortages left model providers oversubscribed and quotas rationed.

Resolution: Coinbase considered availability and burst handling as first-class routing concerns and negotiated capacity with AWS/GCP for larger use cases.

Model leaderboards and model versions changed too quickly for a static model choice.

Root cause: Major providers release new model versions every few months, changing the relative quality and cost landscape.

Resolution: CB-GPT kept a multi-cloud, multi-LLM architecture and internal evaluation framework so selections could be revisited by use case.

Trust and safety could not rely on a universal benchmark.

Root cause: The article says clear trust-and-safety benchmarks for LLMs were lacking, and different LLMs needed different tuning and prompting.

Resolution: Coinbase built its own guardrails to evaluate inputs and outputs against hallucination, jailbreaking, and Three H behavior.

Customer-facing conversational use cases could not be optimized only for average model capability.

Root cause: Latency, user experience, high load, and burstiness affect chatbot quality of service.

Resolution: The platform added load and latency benchmarks and treated availability as part of use-case routing.

Evaluation

Coinbase's evaluation approach combines an internal LLM evaluation framework for Coinbase- and crypto-specific use cases, load and latency benchmarks for all platform models, guardrail checks over inputs and outputs, and decision-framework inputs such as cost, accuracy, latency, and availability.

Metrics: Internal LLM evaluation framework monitors performance across Coinbase and crypto use cases. · Load and latency benchmarks cover all LLMs available on the platform. · Cost comparisons include GPT-4 at 10x GPT-3.5 and Claude 3 Opus at 60x Claude 3 Haiku. · Latency varies from a few seconds to tens of seconds. · Several dozen non-ML team use cases had been built using CB-GPT.

Outcomes

• CB-GPT became a unified interface for GenAI-powered applications at Coinbase.
• A customer-facing conversational LLM chatbot serving all US consumers launched in June 2024.
• Several dozen use cases had been built by non-ML teams using CB-GPT.
• The platform supported multiple clouds and model sources, including AWS Bedrock, GCP VertexAI, Azure GPT, and open-source LLMs.
• Coinbase implemented robust safeguards while continuing to adopt new LLM solutions for employees and customers.

The article reports broad organizational adoption through API and Studio surfaces, several dozen non-ML team use cases, and a US-consumer-facing chatbot, but it does not publish detailed accuracy, safety, cache-hit, cost-savings, or latency SLA results.