Gateway Pattern for Managing Multi-Agent MCP and LLM Traffic

solo_io · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: agent gateway traffic governance

Solo.io needed to govern, secure, observe, and optimize traffic from growing internal AI agents and coding tools to multiple MCP servers and LLM providers without modifying every agent, MCP backend, or LLM integration.

Constraints:

• multiple MCP servers needed to be aggregated behind one interface
• authentication, timeout, JWKS, audience, and resource metadata policies should not be duplicated in every MCP backend
• tool usage needed user/product attribution without modifying MCP backends
• engineers lacked per-user and per-model LLM cost visibility when accessing Anthropic models through Vertex AI
• LLM traffic required authentication, authorization, model routing, organization-level tracking, and Grafana visibility

Architecture

Solo.io inserted agentgateway as a central mediation layer for both MCP and LLM traffic. Multiple MCP backends are configured behind a single virtual MCP listener, while centralized policies add timeouts, strict MCP authentication, JWKS, audience/resource metadata, and observability. The same gateway pattern fronts LLM access through Vertex AI, enforcing Solo.io user authentication, routing Google and partner models based on request details, transforming model names, and emitting metrics for per-user, per-organization, and per-model token and cost tracking.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  internal-ai-agents["Internal AI agents"] --> agentgateway["Agentgateway"]
  support-agent["Support agent"] --> agentgateway
  agentgateway --> mcp-virtual-server["MCP virtual server"]
  agentgateway --> policy-layer["Policy layer"]
  agentgateway --> telemetry-layer["Telemetry layer"]
  agentgateway --> llm-routing-layer["LLM routing layer"]
  mcp-virtual-server --> documentation-mcp-backend["Documentation MCP backend"]
  mcp-virtual-server --> slack-mcp-backend["Slack MCP backend"]
  telemetry-layer --> grafana-dashboard["Grafana dashboard"]
  llm-routing-layer --> vertex-ai["Vertex AI"]

Components

Data flow

An internal support agent, engineer using Claude/Cursor, or coding agent sends requests to the multiplexed agentgateway endpoint. The gateway applies authentication, authorization, timeout, routing, and transformation policies, then forwards MCP calls to documentation, Slack, codebase, GitHub, or internal-tool MCP backends, or routes LLM calls through Vertex AI to Google or partner models. Metrics and logs enrich each tool or LLM call with user, product, organization, model, token, and cost dimensions for Grafana analysis.

Why this architecture

The gateway architecture separates cross-cutting governance from agents and backends. Solo.io can aggregate MCP services, enforce auth and policy, add observability, route models, and track cost centrally while agents, MCP servers, and LLMs remain unchanged.

Models used: agentgateway · MCP · kagent · agentregistry · agentevals · Claude Code · Cursor · Anthropic models · Vertex AI · Google models · Opus 4.7 · Sonnet 4.6 · Grafana

Scale

• Solo.io created several open-source projects since the previous year.
  “Since last year, we have created several open source projects: agentgateway, kagent, agentregistry, and agentevals.”article, stated
• Three of those projects were donated to vendor-neutral Linux Foundation foundations.
  “We also donated agentgateway, kagent, and agentregistry to vendor-neutral foundations under the Linux Foundation.”article, stated
• The example virtual MCP server exposes multiple stateless MCP services on port 3000.
  “the configuration below exposes multiple stateless MCP services as a unified virtual MCP server on port 3000”article, stated
• The support-agent MCP configuration includes documentation and Slack MCP backends.
  “host: http://docs.mcp.svc.cluster.local:3003/mcp”article, stated

  “host: http://slack.mcp.svc.cluster.local:8000/mcp”article, stated
• MCP timeout policy examples set request and backend request timeouts to 300 seconds.
  “requestTimeout: 300s”article, stated

  “backendRequestTimeout: 300s”article, stated
• The LLM routing example is scoped to the engineering organization using an X-Solo-Org header.
  “a simple routing configuration for the engineering organization consuming models via Vertex AI.”article, stated
• Token usage is visualized by organization, model, user, and time window.
  “we can visualize token usage by organization in our Grafana dashboard, as well as tokens per model.”article, stated

  “view token usage per model by each user over any time window.”article, stated

Alternatives considered (roads not taken)

configure each MCP backend separately for auth, timeout, and metadata

Rejected because: The gateway centralizes these policies without code changes, rebuilds, or redeployments for every backend.

expose each MCP server independently to agents and engineers

Rejected because: As MCP server count grew, Solo.io needed a single interface that aggregates services behind one listener.

let engineers access LLMs through Vertex AI without gateway telemetry

Rejected because: That path lacked visibility into usage patterns and per-user, per-model costs.

static MCP capability configuration for future agent workloads

Rejected because: Solo.io wants progressive disclosure so agents can discover capabilities dynamically through search and execution.

Patterns

tool-use-orchestration (agent infrastructure)

Agentgateway mediates tool traffic between agents and MCP backends so support agents and engineers can reach Slack, documentation, codebase, internal tools, and GitHub issue tools through one endpoint.

multi-agent-orchestration (agent infrastructure)

The gateway centralizes governance for multiple internal agents, support agents, coding agents, and engineers consuming shared MCP and LLM services.

model-routing (routing)

Agentgateway dynamically selects and transforms model requests based on request path, organization headers, and whether the target is a Google or partner model.

observability-tracing (operations)

The gateway adds logs and metrics for MCP tool calls and LLM usage, including user, product, organization, model, token, and cost dimensions.

cost-aware-routing (efficiency)

Per-user, per-organization, per-model usage and spending metrics help Solo.io decide whether subscription or pay-as-you-go models are more economical.

safety-policy-enforcement (governance)

The gateway enforces authentication, authorization, timeouts, resource metadata, and verified Solo.io-user access across MCP and LLM traffic.

Tradeoffs

Traffic governance location

Chose central agentgateway configuration over per-agent and per-MCP-server code changes.

Rationale: Centralized configuration lets Solo.io add policies, telemetry, multiplexing, and model routing without modifying agents or backends.

MCP exposure model

Chose single multiplexed endpoint over direct access to many separate MCP endpoints.

Rationale: A unified virtual server gives agents and engineers one endpoint while preserving multiple backend services behind the gateway.

LLM cost management

Chose gateway-level usage and spend tracking over provider access without per-person cost attribution.

Rationale: The gateway's telemetry enables subscription-versus-pay-as-you-go decisions and organization/model/user-level visibility.

Future MCP capability discovery

Chose progressive disclosure through search and execution over static configuration only.

Rationale: Dynamic discovery can let agents find capabilities as needed, while static configuration becomes limiting as MCP services grow.

Gotchas (what bit them)

MCP server sprawl made agent access harder to manage.

Root cause: Solo.io developed multiple MCP servers for internal agents, and the count continued to grow.

Resolution: Put multiple MCP backends behind a single agentgateway listener and virtual MCP server.

Cross-cutting auth and timeout logic can leak into every backend.

Root cause: Without a gateway, each MCP backend or application code path would need timeout, JWKS, audience, and resource metadata configuration.

Resolution: Handle those concerns centrally in agentgateway configuration while keeping MCP servers unchanged.

LLM usage through cloud providers hid per-user and per-model costs.

Root cause: Engineers initially accessed Anthropic models through Vertex AI without sufficient usage-pattern and cost visibility.

Resolution: Route LLM traffic through agentgateway and enrich logs/metrics by user, organization, model, tokens, and spend.

Static tool configuration may not scale to richer agent workloads.

Root cause: Agents need to discover capabilities dynamically instead of relying only on preconfigured server lists.

Resolution: Explore progressive disclosure of MCP servers through search and execution.

Evaluation

Solo.io evaluates the gateway through operational telemetry: tool usage distribution, user/product attribution, token usage by organization, tokens per model, per-user model usage over time, and cost comparisons between subscription and pay-as-you-go LLM access.

Metrics: tool usage distribution · user associated with each tool call · product associated with each tool call · usage and spending by individual · usage and spending by organization · token usage by organization · tokens per model · token usage per model by each user over any time window

Outcomes

• Solo.io gained a single mediated path for MCP and LLM traffic.
• Internal agents and engineers can securely access all MCP servers through a multiplexed endpoint.
• The team can track LLM usage and spending at individual and organization levels.
• Agentgateway provides a cleaner path to scale agentic workloads safely and predictably without backend modifications.

The article describes several internal AI projects, multiple MCP backends behind one listener, a virtual MCP server on port 3000, 300-second timeout policies, organization-level model routing through Vertex AI, and Grafana dashboards for token usage by organization, model, user, and time window.