Building a Unified GenAI Platform for Hundreds of Production Use Cases

karrot · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Marketplace platform AI infrastructure

Karrot needed a unified GenAI platform after independent team adoption created API-key sprawl, fragmented cost visibility, experimentation bottlenecks, and inconsistent reliability for production AI services.

Constraints:

Architecture

Karrot built three connected platforms: LLM Router centralizes provider access, credentials, model abstraction, rate limits, and cost attribution; Prompt Studio lets teams create, test, evaluate, deploy, version, and operate prompts/agents; and KarrotChat exposes Prompt Studio agents and models to employees, including DANA, a BigQuery-backed data analyst agent.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  product-services[Product Services] --> llm-router[LLM Router]
  llm-router --> model-providers[Model Providers]
  llm-router --> cost-management[Cost Management]
  prompt-studio[Prompt Studio] --> model-providers
  prompt-studio --> prompt-studio-api[Prompt Studio API]
  prompt-studio-api --> product-services
  prompt-studio --> mcp-hub[MCP Hub]
  mcp-hub --> agent-builder[Agent Builder]
  prompt-studio --> agent-builder
  agent-builder --> karrotchat[KarrotChat]
  karrotchat --> dana[DANA]
  dana --> bigquery[BigQuery]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Product Services User-facing and internal services that consume LLM APIs for writing, images, try-on, analytics, and other features. LLM Router, Prompt Studio API
LLM Router Central API gateway for provider abstraction, service IDs, credentials, rate limits, quotas, and cost attribution. Product Services, Model Providers, Cost Management
Model Providers OpenAI, Anthropic, Google, self-hosted vLLM, image, and video models behind a normalized interface. LLM Router, Prompt Studio
Cost Management Infrastructure cost platform receiving per-service model usage attribution. LLM Router
Prompt Studio No-code prompt/model testing, evaluation, deployment, versioning, fallback, and agent-building platform. Model Providers, Prompt Studio API, MCP Hub, Agent Builder
Prompt Studio API Stable integration point that lets UI-selected prompt versions serve production traffic. Prompt Studio, Product Services
MCP Hub Registry for internal MCP tools such as data querying and document fetching. Prompt Studio, Agent Builder
Agent Builder No-code agent creator that combines instructions, models, and MCP tools and captures traces. Prompt Studio, MCP Hub, KarrotChat
KarrotChat Internal agent platform where teams discover, run, and share agents and conversations. Agent Builder, DANA
DANA Data analyst agent combining BigQuery MCP, data-model documentation, and prompts. KarrotChat, BigQuery
BigQuery Analytical data warehouse queried by DANA through MCP tooling. DANA

Data flow

Product services call LLM Router through a standard OpenAI SDK interface using service IDs, Prompt Studio stores and evaluates prompts and agents, MCP Hub supplies internal tools, Agent Builder orchestrates tool loops with traces, and KarrotChat lets users invoke agents such as DANA over BigQuery-backed data documentation.

4 supporting quotes
“We built LLM Router to solve these challenges. The core concept: funnel all AI API calls through a single gateway. API keys and accounts are managed centrally by LLM Router, while individual services authenticate with just a service ID — no separate credentials required.”article, stated
“Engineers integrate the Prompt Studio API only once. After that, teams ship prompt improvements directly from the UI — no code changes needed.”article, stated
“Prompt Studio handles the orchestration loop automatically and captures every Agent execution as a Trace — enabling step-by-step debugging: “Why did the LLM call this tool?” “What was the tool response?””article, stated
“DANA — Data Analyst of NA Product team — exemplifies this transformation. It combines the BigQuery MCP, detailed documentation of the data model, and carefully crafted prompts. Team members now get answers and insights without understanding the underlying schema.”article, stated

Why this architecture

Karrot split the platform into gateway, prompt/agent development, and consumption layers because provider abstraction, experimentation, reliability, and organizational knowledge sharing became distinct scaling problems once GenAI moved from experiments to hundreds of production use cases.

3 supporting quotes
“While the architecture may seem complex at first glance, LLM Router, Prompt Studio, and KarrotChat each evolved to solve distinct problems. Let’s dive into what each platform addresses and how we built them.”article, stated
“Teams adopt AI independently while the platform provides company-wide visibility and governance. Effective prompts and Agents naturally propagate across the organization.”article, stated
“With multiple services depending on shared infrastructure, we invest heavily in reliability, observability, and security.”article, stated

Models used: OpenAI models · Anthropic models · Google models · vLLM self-hosted models · claude-4.5-sonnet · gpt-5.2 · gemini-2.5-flash · GPT-5 · GPT-Image · Sora · NanoBanana

Scale

Alternatives considered (roads not taken)

Per-team provider accounts and API keys

Rejected because: Credential sprawl created provisioning, security, rate-limit, and cost-visibility problems.

“In the early days of LLM adoption, teams provisioned their own accounts and API keys whenever they needed AI capabilities. With models from OpenAI, Anthropic, Google, and others, accounts and keys proliferated rapidly — and so did the management overhead. Several pain points emerged:”article, stated
“With numerous keys in circulation, tracking which team used which credentials for what service became increasingly difficult, raising security concerns.”article, stated

Provider-specific client code

Rejected because: Different provider formats would force code changes whenever a model changed.

“For LLM Router to serve as the universal gateway, it needed to support OpenAI, Google, Anthropic, and even models we self-host via vLLM. The challenge: each provider has different API formats and capabilities. Without abstraction, client code would need changes every time we switched models.”article, stated

Engineer-dependent prompt iteration

Rejected because: Every prompt experiment required code and engineering support, slowing idea validation.

“Validating an idea — “Can this prompt solve this problem?” — meant first requesting engineering support. How well an approach works, how to craft the prompt, which model fits best — these vary by use case and require iterative experimentation.”article, stated

Feature-by-feature reliability handling

Rejected because: Handling retries and fallback independently would be costly and inconsistent.

“Handling reliability per-feature is costly and inconsistent. Prompt Studio provides multiple layers of resilience: Retry: Transient failures trigger automatic retries with exponential backoff.”article, stated

Patterns

model-routing (platform)

LLM Router normalizes access to OpenAI, Anthropic, Google, and self-hosted models behind the OpenAI SDK interface.

“All models — regardless of provider — are accessible through the OpenAI SDK.”article, stated

cost-aware-routing (operations)

Every request is attributed to its originating service and fed into infrastructure cost management.

“Every request through LLM Router is attributed to its originating service.”article, stated

fallback-degradation (resilience)

Prompt Studio handles retries, regional fallback, model fallback, and circuit breakers for AI services.

“Prompt Studio provides multiple layers of resilience:”article, stated
“The system blocks requests to that model immediately, routing directly to fallback.”article, stated

eval-driven-development (quality)

Prompt Studio batch-evaluates prompt versions over uploaded test sets before production deployment.

“Prompt Studio’s Evaluation feature lets you upload test sets — hundreds to thousands of examples — generate results in batch, and measure performance quantitatively.”article, stated

tool-use-orchestration (agents)

Agent Builder combines MCP tools, instructions, models, orchestration loops, and traces without code.

“Combine tools from the MCP Hub to create Agents without code. Write task instructions, select a model and MCPs, and you’re done.”article, stated

observability-tracing (agents)

Agent executions are captured as traces for step-by-step debugging.

“captures every Agent execution as a Trace — enabling step-by-step debugging: “Why did the LLM call this tool?” “What was the tool response?””article, stated

Tradeoffs

Provider abstraction

Chose OpenAI-compatible interface for all models over Provider-specific client integrations.

Rationale: Teams can switch models by name and adopt new models without code changes.

“you simply change the model name.”article, stated

Prompt ownership

Chose No-code Prompt Studio with UI-controlled production versions over Engineer-owned prompt deployments.

Rationale: Prompt improvements can ship directly from the UI after the one-time API integration.

“Selecting which version serves production traffic is a single click.”article, stated

Reliability location

Chose Central Prompt Studio resilience layers over Each feature implementing its own retry/fallback behavior.

Rationale: Centralization makes fallback behavior consistent and reduces per-feature effort.

“Handling reliability per-feature is costly and inconsistent. Prompt Studio provides multiple layers of resilience:”article, stated

Analytics democratization

Chose DANA as an agent over BigQuery and documented schema knowledge over Requiring SQL and schema expertise for every analysis question.

Rationale: Team members can query data through natural language while the agent chooses tables, joins, and aggregations.

“No SQL knowledge required. No schema expertise needed.”article, stated

Gotchas (what bit them)

Credential sprawl made GenAI adoption hard to govern.

Root cause: Teams independently created provider accounts and API keys.

Resolution: Centralize provider credentials in LLM Router and let services authenticate with service IDs.

“API keys and accounts are managed centrally by LLM Router, while individual services authenticate with just a service ID”article, stated

Per-account rate limits caused uneven availability.

Root cause: Independent accounts meant some teams exhausted quotas while others had spare capacity.

Resolution: Centralize rate limit monitoring and quota adjustment in the infrastructure team through the gateway.

“AI APIs typically enforce per-account rate limits. With teams operating independently, some accounts had plenty of headroom while others hit their limits”article, stated

Prompt iterations bottlenecked on engineering support.

Root cause: Early feature validation required code changes for every prompt/model experiment.

Resolution: Provide a no-code Prompt Studio UI with model selection, evaluation, deployment, versioning, and rollback.

“Enter a prompt, select a model, click run. If results don’t match expectations, revise and run again”article, stated

Single-model dependencies can take critical AI features down during provider outages.

Root cause: Rate limits, latency spikes, and cloud API disruptions affect external AI services.

Resolution: Layer retries, region fallback, model fallback, and circuit breakers centrally.

“When a critical feature depends on a single model, any service interruption impacts that feature directly.”article, stated

Evaluation

Karrot evaluates prompts in Prompt Studio by uploading test sets, generating outputs in batch, and measuring performance quantitatively; Agent Builder adds traces for debugging tool decisions and responses.

Metrics: Prompt performance across varied scenarios · Batch evaluation results · Provider rate-limit/error behavior · Fallback/circuit-breaker health · Per-service model usage and cost · Agent trace steps and tool responses

3 supporting quotes
“A few successful examples don’t guarantee consistent quality across diverse inputs.”article, stated
“generate results in batch, and measure performance quantitatively.”article, stated
“captures every Agent execution as a Trace — enabling step-by-step debugging”article, stated

Outcomes

Karrot reports the platform handles hundreds of millions of requests and that DANA is used in daily workflows; the article is a company-authored platform writeup.

“This enabled PMs and other non-engineering roles to build AI features independently, iterating as fast as they needed.”article, stated
“Since launch, DANA has become part of the team’s daily workflow — helping everyone make faster, more confident data-driven decisions.”article, stated