Implementing MCP Gateway for Large-Scale LLM Integration Infrastructure

anthropic · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise MCP gateway infrastructure for LLM tool integrations

Anthropic needed to standardize a rapidly growing set of internal and external LLM tool integrations after custom endpoints, duplicated authentication, incompatible interfaces, outbound network access, and credential handling created integration chaos across LLM-backed services.

Constraints:

Architecture

Anthropic standardized LLM integration around MCP and built an MCP Gateway as shared infrastructure. Engineers call connect_to_mcp with a URL plus organization/account identity and receive an MCP SDK client session. The gateway performs URL-based routing to internal or external MCP servers, centralizes credential management and OAuth flows, authenticates internal callers with signed tokens, applies rate limiting and observability, and exposes standardized JSON-RPC streams over transports such as WebSockets for internal use and standard MCP transports for external servers. This creates a pit of success: teams reuse the gateway instead of building custom /call-tool and /get-context endpoints, while security and policy controls sit in one place.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  LLM_backed_services["LLM-backed services"] --> MCP_client_library["MCP client library"]
  MCP_client_library --> MCP_Gateway["MCP Gateway"]
  MCP_client_library --> MCP_SDK_client_session["MCP SDK client session"]
  Signed_token_authentication["Signed token authentication"] --> MCP_Gateway
  MCP_Gateway --> URL_router["URL router"]
  MCP_Gateway --> Credential_store["Credential store"]
  MCP_Gateway --> OAuth_flow_helpers["OAuth flow helpers"]
  MCP_Gateway --> Internal_WebSocket_transport["Internal WebSocket transport"]
  MCP_Gateway --> External_MCP_transport["External MCP transport"]
  MCP_Gateway --> Policy_and_audit_layer["Policy and audit layer"]
  URL_router --> Internal_MCP_servers["Internal MCP servers"]
  URL_router --> External_MCP_servers["External MCP servers"]
  Credential_store --> OAuth_flow_helpers
  Credential_store --> External_MCP_servers
  Internal_WebSocket_transport --> MCP_SDK_client_session
  External_MCP_transport --> MCP_SDK_client_session
  MCP_SDK_client_session --> JSON_RPC_message_spec["JSON-RPC message spec"]
  JSON_RPC_message_spec --> Policy_and_audit_layer
  Policy_and_audit_layer --> Internal_MCP_servers
  Policy_and_audit_layer --> External_MCP_servers
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
LLM-backed services Internal services such as PR review bots, Slack management tools, and other agents that need external context or tools. MCP client library, MCP Gateway
MCP client library Developer-facing package exposing a simple connect_to_mcp call and returning an MCP SDK client session. LLM-backed services, MCP Gateway, MCP SDK client session
MCP Gateway Shared gateway that routes MCP connections, centralizes credentials, handles authentication, and provides observability/rate limiting/policy hooks. MCP client library, URL router, Credential store, OAuth flow helpers, Internal WebSocket transport, External MCP transport, Policy and audit layer
URL router Routes connection requests to internal or external MCP servers based on URL. MCP Gateway, Internal MCP servers, External MCP servers
Credential store Centralized storage and retrieval for user credentials used by external integrations. MCP Gateway, OAuth flow helpers, External MCP servers
OAuth flow helpers Gateway helper functions for starting and completing provider-specific OAuth flows. MCP Gateway, Credential store
Signed token authentication Internal authentication mechanism so services do not pass raw user credentials around. LLM-backed services, MCP Gateway
Internal WebSocket transport Anthropic's selected internal transport for sending JSON-RPC messages over bidirectional streams. MCP Gateway, MCP SDK client session
External MCP transport Standardized external transport layer for remote MCP services. MCP Gateway, External MCP servers
MCP SDK client session Standard SDK object returned to the caller so new protocol features arrive through package upgrades. MCP client library, Internal WebSocket transport, External MCP transport
Internal MCP servers Anthropic-owned context/tool providers behind the gateway. URL router, Policy and audit layer
External MCP servers Remote third-party MCP services that require external connectivity and authentication. URL router, Credential store, External MCP transport
Policy and audit layer Central place for rate limiting, observability, prompt-injection defenses, malicious-server blocking, content classification, and audit. MCP Gateway, Internal MCP servers, External MCP servers
JSON-RPC message spec Standardized MCP message format for tools, resources, prompts, sampling, and context exchange. MCP SDK client session, Policy and audit layer

Data flow

A consuming LLM-backed service imports Anthropic's MCP client library and calls the gateway with a URL and identity context. The internal service authenticates to the gateway with a signed token. The gateway routes the URL to the proper internal or external MCP server, retrieves stored user credentials when needed, creates an authenticated MCP SDK client, and pipes read/write streams between the chosen transport and the client session. Tool execution, resource, prompt, and context messages all pass through standardized MCP JSON-RPC streams, giving Anthropic a centralized point for credentials, rate limits, observability, policy checks, audit, and future MCP feature adoption.

3 supporting quotes
“we designed a service which is just a piece of shared infrastructure called the MCP gateway that provides a single point of entry”transcript, stated
“the code that we have here we just made some client libraries where we just MCB gateway connect to MCP uh we pass in a URL an org ID account ID”transcript, stated
“We're hitting a websocket connection to MCP gateway uh with O token provided as headers to that”transcript, stated

Why this architecture

Anthropic centralizes MCP access because tool integration plumbing is not a differentiator, but duplication creates security, reliability, and developer-experience problems. MCP standardizes the message contract, while the gateway makes the secure path the easiest path: one client call, one credential layer, one place for rate limiting and observability, and one upgrade point for future protocol features.

3 supporting quotes
“being boring on stuff like this is good. It's not a competitive advantage to be really good at making Google Drive talk to your app.”transcript, stated
“if you make the right thing to do, the easiest thing to do, then everyone in your org kind of falls into it.”transcript, stated
“we handle all the credential management automatically because you don't want to be implementing OOTH five times in your company.”transcript, stated

Models used: Claude models with tool calling · MCP-compatible model clients

Scale

Alternatives considered (roads not taken)

Custom endpoints per integration

Rejected because: Service-specific /call-tool and /get-context endpoints caused duplicated functionality, incompatible interfaces, and multi-week rewrites.

“Custom endpoints start proliferating for every use case.”transcript, stated
“nothing really works the same you have an integration that works really well in service A but then you want to use it in service B but you can't”transcript, stated

Each service manages its own credentials and OAuth

Rejected because: Duplicated OAuth implementations and direct credential access create security and auditing problems.

“you don't really want all of those services to be going and accessing user credentials uh because that ends up being being kind of a nightmare.”transcript, stated
“auditing becomes really complex.”transcript, stated

Standardize only on external MCP transport details

Rejected because: The presenter says the message specification contains most of MCP's value, and internal transport can vary by organization.

“most of the juice of MCP is in this the message specification and the way that the uh servers are interacting.”transcript, stated
“whatever it takes inside your organization is great.”transcript, stated

Avoid standardization until future requirements are known

Rejected because: MCP already contains solutions for likely future needs such as sampling, billing, token limits, and usage tracking.

“one of the things you get with MCP is that it solves problems that you haven't actually run into yet.”transcript, stated
“MCP has uh already has sampling primitives so you can build your integration”transcript, stated

Expose outbound network connectivity everywhere

Rejected because: Anthropic wanted a central gateway instead of broad external connectivity and credentials across many internal services.

“don't want uh outbound external network connectivity everywhere.”transcript, stated
“a single point of ingress egress and standardized message formats”transcript, stated

Patterns

tool-use-orchestration (agent-architecture)

The MCP gateway standardizes how LLM-backed services connect to tools, resources, prompts, external services, and internal context providers.

“your model could go and call your Google Drive and then it could call your maps and then it could send a text message to people.”transcript, stated
“MCP is really just JSON streams and how you pipe those streams around your infrastructure is a small implementation detail.”transcript, stated

safety-policy-enforcement (governance)

Centralizing MCP traffic creates a policy layer for credential access, prompt-injection risk, malicious-server blocking, content classification, and auditing.

“there's some papers written on MCP prompt injection attacks. There is a general risk of uh models going and having access to Google Drive and deleting everything in Google Drive.”transcript, stated
“You might want to be able to like ban malicious servers. Um do some content classification on the request, see what's coming in, kind of given audit.”transcript, stated

privacy-preserving-processing (governance)

The gateway keeps stored credentials in one place and prevents internal services from directly handling user tokens.

“the credentials are portable. If you have a batch job that you're kicking off, um your users don't have to reauthenticate to that.”transcript, stated
“You're just calling the same MCP with your internal user ID and they get everything added correctly. You're also internal services don't have to worry about your tokens.”transcript, stated

observability-tracing (operations)

The gateway gives a central place for rate limiting and observability across tool integrations and model context flows.

“we handle all the credential management automatically because you don't want to be implementing OOTH five times in your company. Uh, gives you a centralized place for rate limiting and observability.”transcript, stated
“a central place for all of your context that your models are asking for and all the context that's flowing into your models for your org.”transcript, stated

model-routing (serving)

URL-based routing lets one gateway call connect to internal or external MCP servers through the same client abstraction.

“we used URL based routing to route to external servers, internal servers, it doesn't matter. It's all the same call.”transcript, stated
“You just update your MCP packages internally. You get those features across the board. Everything works great. The same code seamlessly connects to internal external integrations.”transcript, stated

cost-aware-routing (operations)

MCP sampling primitives provide a reusable answer for billing, token limits, and usage tracking across products with different models.

“You might have four products that have four different billing models uh for reasons because you're building fast.”transcript, stated
“you might have a bunch of different token limits. You might have different ways of tracking usage.”transcript, stated

Tradeoffs

MCP standardization

Chose one internal standard for model context/tool integration over custom endpoints and bespoke service interfaces.

Rationale: A standard message contract reduces repeated plumbing, improves reuse, and lets engineers focus on product work.

“You can spend your cycles working on interesting problems instead of trying to figure out how to plum uh integration.”transcript, stated
“you should standardize on something, anything. I think MCP is a good idea. If you don't think it's a good idea, like just pick something.”transcript, stated

Gateway centralization

Chose centralized MCP Gateway for routing, credentials, observability, and rate limiting over each service connecting directly to internal and external tools.

Rationale: Centralizing shared problems reduces duplicated OAuth work, external connectivity sprawl, and credential exposure.

“centralizing at the correct layer. So solving some shared problems off and external connectivity once allows you to spend your time working on uh more interesting problems”transcript, stated
“don't want uh outbound external network connectivity everywhere. Um auditing becomes really complex.”transcript, stated

Internal WebSocket transport

Chose WebSockets for Anthropic's internal MCP transport over a single mandated transport for all environments.

Rationale: The presenter argues internal transport is an implementation detail; WebSockets fit Anthropic, while other orgs might choose gRPC, Unix sockets, or something else.

“We've got the standardized transport for connecting to external MCP servers. Um, but really just picking the best thing for your org. So we went and picked uh websockets for our internal transport.”transcript, stated
“you might want to do this with gRPC because you want to wrap these in some multipplexed transport”transcript, stated

Pit-of-success developer interface

Chose simple connect_to_mcp client API over requiring engineers to understand credential, transport, and provider details for every integration.

Rationale: Making the secure standardized path easiest encourages adoption across a fast-moving organization.

“we're trying to make that as simple as possible uh because that way people will use it if it's the easiest thing to do.”transcript, stated
“make the right way to do a thing the easiest way to do a thing and then everyone just falls into doing the right thing naturally.”transcript, stated

Protocol bet

Chose MCP as the industry-aligned standard over a private Anthropic-only protocol.

Rationale: The presenter argues major AI labs and ecosystem demand make MCP a good bet, and future protocol features can be adopted once at the gateway.

“there's a large coalition of engineers and organizations that are all involved in building out the standard. Uh all of the major AI labs are represented in that.”transcript, stated
“as new model capabilities start to be developed uh those patterns will be added to the protocol”transcript, stated

Gotchas (what bit them)

Rapid tool-calling adoption creates integration chaos.

Root cause: Teams independently build custom endpoints, authentication, and context interfaces for each use case.

Resolution: Standardize on MCP and provide a gateway that makes the standard path easier than custom integrations.

“Custom endpoints start proliferating for every use case. there's a lot of like services popping up with like slashcall tool and slash like get context”transcript, stated
“this kind of led to some integration chaos where you're duplicating a bunch of functionality around your org”transcript, stated

Credentials become a security and audit nightmare when every service accesses them.

Root cause: Many internal agents and services need external integrations, but direct credential access broadens risk and complicates audit.

Resolution: Centralize OAuth, credential storage, signed-token service authentication, and audit at the gateway.

“you don't really want all of those services to be going and accessing user credentials uh because that ends up being being kind of a nightmare.”transcript, stated
“We set up this unified authentication model where we're handling ooth at the gateway”transcript, stated

Interface mismatch destroys reuse between LLM services.

Root cause: Integrations are useful in one service but cannot be reused elsewhere without adapting to a new interface.

Resolution: Return a standard MCP SDK client session from the gateway so new features and integrations work across consumers.

“you want to use it in service B but you can't because it's going to take you three weeks to rewrite it to talk to the new interface”transcript, stated
“this call returns an MCP SDK object which means that when new features get added to protocol. You just update your MCP packages internally.”transcript, stated

Transport debates can distract from the real standardization value.

Root cause: Organizations may focus on global transport details even though the JSON-RPC message specification carries most of the reusable contract.

Resolution: Standardize messages via MCP, then choose the transport that fits the internal environment.

“most of the juice of MCP is in this the message specification and the way that the uh servers are interacting.”transcript, stated
“And so it truly doesn't matter like whatever it takes inside your organization is great.”transcript, stated

Centralizing MCP traffic creates a potential single point of failure and shared-capacity concern.

Root cause: The gateway becomes the aggregate ingress/egress layer for many LLM integrations.

Resolution: The source does not detail the availability design; future review should check gateway reliability, disaster recovery, and capacity planning.

“you have the operational simplicity of having a single point of ingress egress and standardized message formats”transcript, stated
“here's a a gateway in the middle. This is kind of the thing. Just one more box will solve all our problems.”transcript, implied

Evaluation

The talk does not describe a formal benchmark or eval suite for the gateway. The operational validation is architectural: the gateway reduces integration rewrites, centralizes credential management, supports internal and external integrations through one interface, exposes a common observability/rate-limit layer, and makes adding MCP support to new services as simple as importing a package.

Metrics: 20 years large-scale-systems background from presenter · three-week rewrite avoided for cross-service integration reuse · five OAuth implementations avoided by central credential management · support for internal and external MCP integrations through same call · multiple internal language packages supported · central place for rate limiting and observability

3 supporting quotes
“you have an integration that works really well in service A but then you want to use it in service B but you can't because it's going to take you three weeks”transcript, stated
“adding MCP support to new services is as simple as possible. You just go and import a package.”transcript, stated
“we handle all the credential management automatically because you don't want to be implementing OOTH five times in your company. Uh, gives you a centralized place for rate limiting and observability.”transcript, stated

Outcomes

The talk is an architecture and infrastructure case rather than a quantified product outcome study. Its strongest reusable lesson is the gateway-as-pit-of-success pattern: standardize non-differentiating tool plumbing, centralize risky shared concerns, and expose a simple client API so fast-moving LLM teams naturally choose the secure maintainable path.

3 supporting quotes
“You just update your MCP packages internally. You get those features across the board. Everything works great. The same code seamlessly connects to internal external integrations.”transcript, stated
“adding MCP support to new services is as simple as possible. You just go and import a package. Uh it doesn't matter what language you're in.”transcript, stated
“you get future features for free as the protocol involves you get all of that work uh naturally”transcript, stated