Building a Multi-Provider GenAI Gateway for Enterprise-Scale LLM Access

grab · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise GenAI gateway and governance

Grab needed a central gateway so internal teams could safely and efficiently access many GenAI providers without each team handling provider authentication, infrastructure, cost tracking, quota management, and governance review themselves.

Constraints:

• providers use different authentication schemes and APIs
• reserved capacity can be wasted if services churn or demand shifts
• production use cases need privacy, cybersecurity, and AI governance review
• batch and online workloads compete for the same provider quotas
• rapid provider and model innovation creates continuous maintenance pressure

Architecture

Grab AI Gateway is a lightweight reverse-proxy layer across OpenAI, Azure, AWS Bedrock, Google Vertex AI, in-house OSS models, and other providers. It uses API keys for exploration and production access, request-path authorization, provider-key substitution, rate limits, a unified OpenAI-style API interface, dynamic routing across reserved capacity and regions, auditing into Grab's data lake, per-request cost attribution, and platform integrations with Chimera notebooks and Catwalk deployments.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  internal-applications[Internal applications] --> grab-ai-gateway[Grab AI Gateway]
  slack-bot[Slack bot] --> api-keys[API keys]
  api-keys --> grab-ai-gateway
  governance-review[Governance review] --> grab-ai-gateway
  grab-ai-gateway --> unified-api-handler[Unified API handler]
  unified-api-handler --> provider-reverse-proxies[Provider reverse proxies]
  provider-reverse-proxies --> external-ai-providers[External AI providers]
  grab-ai-gateway --> dynamic-routing[Dynamic routing]
  dynamic-routing --> external-ai-providers
  grab-ai-gateway --> audit-and-cost-data-lake[Audit and cost data lake]
  audit-and-cost-data-lake --> cost-dashboards[Cost dashboards]
  grab-ai-gateway --> rate-limiter[Rate limiter]
  ml-platform-integrations[ML platform integrations] --> internal-applications
  ml-platform-integrations --> api-keys

Components

Data flow

A Grabber requests an exploration or production key, with production use cases going through mini-RFC/checklist review and sometimes AI Governance review. Applications call the gateway using either provider-specific APIs or a unified OpenAI-style API. The gateway authenticates the internal key, authorizes the path, swaps in the provider key, translates unified payloads to provider-specific schemas when needed, routes to a provider or similar reserved-capacity model, records request/response metadata and costs in the data lake, enforces rate limits, and returns the provider response with minimal processing.

Why this architecture

Grab chose a minimalist reverse-proxy gateway because it needed central control without slowing provider innovation or forcing app teams to rewrite provider integrations. The gateway acts like the provider from the user's perspective, while centralizing provider keys, authorization, quotas, audit logs, cost attribution, model switching, and governance.

Models used: OpenAI · Azure OpenAI · AWS Bedrock · Google VertexAI · in-house open source models · vLLM via Catwalk

Scale

• The gateway has onboarded more than 300 unique use cases.
  “With more than 300 unique use cases onboarded and many of those making it to production, AI Gateway has gained popularity since its inception in 2023.”article, stated
• More than 3,000 Grabbers requested exploration keys.
  “Exploration keys are highly popular, with more than 3,000 Grabbers requesting the key to experiment with APIs.”article, stated
• The catalogue problem spans over 50 AI models.
  “With over 50 AI models each suited for a specific task type, finding the correct model to use is becoming complex.”article, stated
• Exploration keys are valid for a few days with staging-only access.
  “The keys are short-lived with a validity period of a few days, stricter rate limit restrictions, and access limited to only the staging environment.”article, stated
• Grab operates across 700 cities in eight countries.
  “Grab offers a wide range of on-demand services in the region, including mobility, food, package and grocery delivery services, mobile payments, and financial services across 700 cities in eight countries.”article, stated

Alternatives considered (roads not taken)

Each team integrates providers directly

Rejected because: Provider-specific authentication, infrastructure setup, reviews, cost tracking, and quotas would be duplicated across teams.

Reserved capacity per service

Rejected because: Pre-purchased capacity can be wasted when services deprecate or demand shifts, so Grab uses a shared pool and global usage view.

Provider-specific SDK logic everywhere

Rejected because: A unified API lets users switch providers by changing a model parameter and makes fallback/dynamic routing easier.

Treat the reverse proxy as fully transparent and maintenance-free

Rejected because: Provider SDK edge cases still required integration tests and missing path/configuration fixes.

Patterns

model-routing (model_operations)

The gateway exposes multiple providers and can route specific model requests to similar models, reserved instances, or regions.

cost-aware-routing (operations)

Grab uses shared capacity pools, usage trends, per-request cost calculations, dashboards, and alerts to control GenAI spend.

observability-tracing (operations)

Every call's request, response, token usage, URL path, and model are archived for audit, security inspection, and usage reporting.

guardrails-io-filtering (safety)

The gateway roadmap includes built-in input/output guardrails and prompt-injection defenses so app teams do not each build their own.

safety-policy-enforcement (governance)

Production use cases pass through mini-RFC/checklist review and sometimes AI Governance review before deployment.

fallback-degradation (reliability)

Unified routing supports fallback logic and load balancing across providers or regions when quota or availability constraints arise.

Tradeoffs

Use a minimalist reverse proxy

Chose Lightweight provider-facing gateway with minimal response processing over A heavy abstraction layer that normalizes all provider behavior deeply.

Rationale: The minimalist design kept operations manageable while providers and models changed rapidly.

Offer both provider-specific and unified APIs

Chose Provider-specific access plus OpenAI-style unified interface over Only one canonical provider API shape.

Rationale: Provider-specific compatibility reduces migration friction, while unified access lowers the barrier to experimentation and routing.

Use exploration keys before production approval

Chose Short-lived staging-only keys for experimentation over Full production access before review.

Rationale: Exploration keys reduce friction for learning and prototypes while keeping production use under governance review.

Gotchas (what bit them)

Provider-specific auth and setup slow internal adoption.

Root cause: Each provider has different keys, roles, credentials, infrastructure, and SDK behavior.

Resolution: Centralize provider setup behind gateway keys, base URLs, and provider-key substitution.

Batch traffic can interfere with online service uptime.

Root cause: Use cases with different SLOs may share the same underlying provider quota.

Resolution: Use gateway-level fair distribution by criticality and RPS, plus Async APIs where acceptable.

Reverse-proxy compatibility breaks on provider edge cases.

Root cause: Provider SDK features can depend on paths or configurations missing from the gateway.

Resolution: Catch issues with SDK integration tests before deploying gateway changes.

The model catalogue becomes hard to navigate as provider count grows.

Root cause: Users struggle to compare model capabilities, latency, cost, quotas, and modalities across more than 50 models.

Resolution: Build a catalogue with metadata such as modality, token limits, quota, pricing, and references.

Evaluation

The article describes operational governance and platform metrics rather than an ML benchmark: usage is tracked through audit logs, cost calculations, rate limits, integration tests, dashboards, and governance review.

Metrics: number of onboarded use cases · number of exploration keys requested · request/response metadata and token usage · per-request and daily async cost calculations · rate-limit behavior by key · SDK integration test results · cost threshold alerts

Outcomes

• Grab AI Gateway became the central access path for multiple GenAI providers.
• More than 300 unique use cases were onboarded, with many reaching production.
• More than 3,000 Grabbers requested exploration keys.
• The gateway powers real-time audio signal analysis, content moderation, menu-description generation, translation, text-to-SQL, incident automation, and support bots.
• The platform reduced duplicated provider setup while adding governance, auditing, rate limits, dynamic routing, and cost showback.

The source reports more than 300 onboarded use cases, more than 3,000 exploration-key requests, over 50 AI models in the catalogue problem, and Grab's broader operating footprint across 700 cities in eight countries.