Building an AI API Gateway for Streamlined GenAI Service Development

deliveryhero · E-commerce · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise GenAI API gateway

Woowa Brothers needed centralized infrastructure for generative AI services because prompts, credentials, providers, invocation logic, and logging were fragmented across teams and applications.

Constraints:

Multiple providers had different credential and invocation formats.
Teams needed prompt and parameter changes without repeated code deployments.
Services had to handle PII, logging, quotas, rate limits, cost visibility, and provider-specific pricing.
The gateway had to integrate with existing infrastructure and workflows.

Architecture

The AI Platform Team built an AI API Gateway as the first GenAI platform component. It centralizes credential management, template and prompt versioning, proxy definitions, simple multi-step pipelines, provider integrations for AWS Bedrock, Azure OpenAI, GCP VertexAI/Imagen, NCP, a LangChain-compatible SDK, and dashboards for request volume, token usage, image counts, and cost trends.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  service-developer[Service Developer] --> ai-api-gateway[AI API Gateway]
  service-developer --> ai-studio-web-view[AI Studio Web View]
  ai-api-gateway --> credential-manager[Credential Manager]
  ai-api-gateway --> template-manager[Template Manager]
  ai-api-gateway --> proxy-manager[Proxy Manager]
  proxy-manager --> template-manager[Template Manager]
  proxy-manager --> credential-manager[Credential Manager]
  ai-api-gateway --> pipeline-manager[Pipeline Manager]
  pipeline-manager --> proxy-manager[Proxy Manager]
  proxy-manager --> provider-integrations[Provider Integrations]
  ai-api-gateway --> usage-dashboard[Usage Dashboard]
  ml-sdk[ML SDK] --> ai-api-gateway[AI API Gateway]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Service Developer`	Builds GenAI features and calls the gateway instead of embedding provider-specific logic.	`AI API Gateway`, `AI Studio Web View`
`AI API Gateway`	Central infrastructure layer for credentials, templates, proxies, pipelines, provider calls, and observability.	`Service Developer`, `Credential Manager`, `Template Manager`, `Proxy Manager`, `Pipeline Manager`, `Provider Integrations`, `Usage Dashboard`, `ML SDK`
`Credential Manager`	Gateway feature for registering, modifying, deleting, and selecting provider credentials.	`AI API Gateway`, `Provider Integrations`
`Template Manager`	Versioned prompt hub with dynamic inputs and shared prompt engineering knowledge.	`AI API Gateway`, `Proxy Manager`
`Proxy Manager`	Defines simplified proxy request formats from credentials, templates, parameters, and provider choices.	`AI API Gateway`, `Template Manager`, `Credential Manager`, `Provider Integrations`
`Pipeline Manager`	Chains consecutive GenAI calls for simple scenarios by passing one proxy output into the next.	`AI API Gateway`, `Proxy Manager`
`Provider Integrations`	Connections to AWS Bedrock, Azure OpenAI, GCP VertexAI/Imagen, and Naver Cloud Platform.	`Credential Manager`, `Proxy Manager`, `AI API Gateway`
`ML SDK`	LangChain-compatible gateway client for Python services.	`AI API Gateway`, `Service Developer`
`Usage Dashboard`	Dashboard for request volume, input/output tokens, image counts, and cost trends by project/model/task.	`AI API Gateway`
`AI Studio Web View`	Planned UI for managing projects, credentials, templates, and service requests.	`Service Developer`, `AI API Gateway`

Data flow

A service registers credentials, templates, proxies, and optionally pipelines in the gateway. At runtime the client calls a simplified proxy or pipeline endpoint, the gateway fills dynamic template variables, selects provider credentials when needed, invokes the configured GenAI provider, logs inputs/outputs/parameters, and exposes usage/cost dashboards.

“Ultimately, credentials, prompts and appropriate parameter values are required to apply Gen AI API to a service.”article, stated

“When configuring a pipeline, a user can assign specific output from the previous proxy to be delivered to the subsequent proxy.”article, stated

Why this architecture

The team prioritized the gateway because existing GenAI services were already using multiple providers, while prompts and invocation logic were fragmented. A central gateway let service developers move faster while platform teams handled credential policy, prompt/template management, provider abstraction, security, quota handling, logging, and cost observability.

“After consolidating feedback from AI platform developers, data scientists, and project managers, we decided to prioritize the development of an API Gateway.”article, stated

“We also understood the urgency in building a Hub to manage credentials and prompts before working on the development of RAG Pipeline, Prompt Experiment, and Experiment Feedback as the prompts and invocation logics of existing Gen AI-based services were fragmented.”article, stated

Models used: AWS Bedrock · Azure OpenAI · GCP VertexAI Imagen · Naver Cloud Platform

Scale

The organization already used multiple GenAI providers: AWS Bedrock, Azure OpenAI, and GCP Imagen.
“The fact that Woowa Brothers already adopts various Gen AI providers, such as AWS Bedrock, Azure OpenAI, and GCP Imagen, was a key factor in this decision.”article, stated
Text generation cost is proportional to input/output tokens and image generation cost to generated image count.
“pricingis proportional to the number of input/output tokens while that of the image generation**AI API is proportional to the number of images generated”article, stated
The gateway supports multiple credentials per proxy to handle high-volume traffic or sudden traffic surges.
“multi-credential capability that helps address issues such as call rate limits caused by high-volume traffic or sudden traffic surge.”article, stated

Alternatives considered (roads not taken)

Use external integrated API-management products such as Kong or Dify.ai.

Rejected because: Woowa Brothers wanted custom workload support, flexibility, long-term cost savings, security compliance, and infrastructure integration.

“Although various solutions, such as Kong and Dify.ai, already support integrated management”article, stated

Keep credentials in AWS Secrets Manager and modify deployments for changes.

Rejected because: Credential growth and change tracking made the process harder to manage, and gateway-managed credentials could reflect immediately.

“users can modify/delete through the gateway for immediate reflection on services without having to go through Secrets Manager.”article, stated

Manage prompts and provider parameters in code or resource files.

Rejected because: Model, API version, prompt, temperature, and provider changes required code modifications or deployments.

“In order to modify settings (api_key, api_version, model) or parameter values (prompt, temperature), or when there is change in the calling Gen AI services (Azure OpenAI -> AWS Bedrock), service developers need to modify codes or make deployments.”article, stated

Use only proxy calls for chained GenAI scenarios.

Rejected because: Some scenarios need repeated or sequential calls where one proxy output feeds another.

“if there is no separate server in operation and only the gateway is used to make consecutive Gen AI API calls, the proxy feature alone will have limitations.”article, stated

Patterns

`model-routing` (platform)

The gateway abstracts calls across Bedrock, Azure OpenAI, GCP VertexAI/Imagen, and NCP so services can change providers through gateway configuration.

“you may easily change the Gen AI API provider by modifying service=auzre_openai to service=aws_bedrock.”article, stated

`prompt-versioning` (platform)

Templates support dynamic inputs, versioning, rollback, and a shared prompt hub.

“Along with the generation of a new template, versioning is provided to keep track of the existing details when changes are made.”article, stated

`cost-aware-routing` (operations)

The public dashboard tracks provider-specific pricing units such as tokens and generated image counts.

“a dashboard that reflects pricing units should be provided to help understand the cost trend at a glance.”article, stated

`fallback-degradation` (resilience)

Multiple credentials can be registered so the gateway can select an appropriate credential under rate-limit or surge conditions.

“If multiple credentials are registered**while creating a proxy,**the gateway will select and use an appropriate credential”article, stated

`privacy-preserving-processing` (security)

The gateway requirements include rejecting or masking sensitive data before external GenAI APIs see it.

“requests containing sensitive data should be rejected or have the sensitive information masked before being passed to external Gen APIs.”article, stated

Tradeoffs

Gateway ownership

Chose Build and operate an internal gateway over Adopt an external integrated API gateway product.

Rationale: Internal ownership better matched custom workload support, flexibility, security requirements, cost, and existing infrastructure integration.

“Need the flexibility to quickly modify and scale the gateway in line with the rapidly evolving AI technologies”article, stated

Prompt management

Chose Versioned gateway templates with dynamic variables over Static prompts in code or resource files.

Rationale: Data scientists and GenAI developers could change prompts without engineering deployments.

“Templates use dynamic inputs and generate the final prompt based on these inputs (query).”article, stated

Pipeline boundary

Chose Simple gateway-managed consecutive calls over Making every chained scenario run in a separate server.

Rationale: The gateway could support simple chained calls while acknowledging it would not cover all possible cases.

“Although it does not support all possible cases, the gateway provides a pipeline feature to make consecutive Gen AI API calls for simple scenarios.”article, stated

Gotchas (what bit them)

Provider credentials multiply quickly.

Root cause: Each GenAI provider uses different credential parameters and every new service can add more secrets to manage.

Resolution: Move credential registration, modification, deletion, and provider integration into the gateway.

“The number of credentials to be managed continue to increase as more credentials are created for each service.”article, stated

Duplicate platform work appears in every GenAI service.

Root cause: PII handling, logging, quota errors, caching, and standard IO are needed beyond a simple external API call.

Resolution: Centralize these cross-cutting concerns in the gateway so service teams focus on business logic.

"real service applications require the following considerations and developments:

Personal Identifiable Information (PII)"

“service developers can focus on core business logics, which in turn, is expected to greatly improve the overall system stability, security, and efficiency.”article, stated

Prompt and provider changes cause deployment churn.

Root cause: Prompts, model deployment names, API versions, and temperatures were embedded in code or resource files.

Resolution: Use dynamic routing and internal template/resource management so changes can happen without server code changes.

“the gateway implements dynamic routing and internal resource (e.g., template) management features”article, stated

Evaluation

The team gathered requirements through employee interviews and planned operational evaluation through gateway logs, request-volume dashboards, token/image pricing metrics, and feedback systems for GenAI outputs.

Metrics: Request volume by model and task · Request volume by LLM service, project, and model · Input/output token counts · Generated image counts · Cost trends · User feedback on GenAI outputs

“We conducted interviews with employees actively using Gen AI to understand the challenges they face and identify essential features”article, stated

“Request volume by LLM service**(project, model)**and number of input/output tokens”article, stated

Outcomes

Woowa Brothers prioritized the API Gateway as the first GenAI platform component.
The gateway centralizes credential, template, proxy, pipeline, provider, SDK, and usage-dashboard capabilities.
The platform is expected to improve service-development productivity and support NoCode GenAI service creation.

The article cites multiple providers, multiple credential formats, high-volume traffic/rate-limit scenarios, token/image pricing dashboards, and a planned AI Studio web view.

“the gateway introduced in this post to significantly improve the productivity of the service development teams using Gen AI.”article, stated

“We are committed to creating an environment where Gen AI services are built using NoCode”article, stated