Skip to content
Running AI in production· Guide 1 of 6Intermediate7 min readUpdated

LLM gateway vs MCP gateway ​

An LLM gatewayLLM gateway: A service between applications and model providers that gives them one API endpoint and adds per-client keys, routing, spend limits, rate limits and checks on prompts and answers. (often called an AI gateway) sits between your applications and the model providers: it decides which model a caller may use, with which key, how often and at what cost. An MCP gatewayMCP gateway: A service that sits between MCP clients and MCP tools and serves them through one endpoint, adding per-client credentials, access rules, limits and a log of every call. sits between AI agents and the tools they call over the Model Context ProtocolModel Context Protocol (MCP): An open protocol that lets AI applications discover and use tools, data and prompt templates from separate servers through one standard set of JSON-RPC messages.: it decides which tools a client may see and call, with which arguments, and records each call. One governs what the model is asked; the other governs what the agent does. If MCP is new to you, start with what MCP is.

Two kinds of traffic ​

An AI agent sends two kinds of requests, and they go to different places.

  1. Model calls. The agent sends the conversation and its tool definitions to a model provider and gets back text, or a request to call a tool. This is a chat completion: HTTP to a provider such as OpenAI, Anthropic, Gemini or a local Ollama server.
  2. Tool calls. When the model asks for a tool, the agent runs it. With MCP, that is a tools/call request to an MCP server, which then reads a database, opens a ticket or pushes a commit.
Where each gateway sitsAn agent or app in the middle. Model calls go up through an LLM gateway to model providers. Tool calls go down through an MCP gateway to MCP servers.Agent or appLLM gatewayModel providersMCP gatewayMCP servers (tools)model calls: prompt, model name, tokenstool calls: tool name, arguments, result
The two gateways sit on different paths. The LLM gateway sees what the model is told about a tool call; only the MCP gateway sees the call itself.

The model never runs a tool. It only names one. So a gateway on the model path sees the model asking for a tool, and later the result the application chooses to send back. A gateway on the tool path sees the call that actually happens. That difference decides what each one can control.

What an LLM gateway controls ​

An LLM gateway is one endpoint that applications use instead of calling each provider directly. Most speak an OpenAI-compatible APIOpenAI-compatible API: An HTTP API that accepts the same requests as OpenAI's API (such as /v1/chat/completions), so any client built for it can switch servers by changing the base URL., so any OpenAI SDK works by changing its base URL. Typical controls:

  • Routes. Callers ask for a public model name, such as fast-chat. The gateway maps it to a provider, an upstream model and a stored key. Swapping the provider behind a name does not touch any application.
  • Keys per caller. Each application gets its own gateway token. Provider keys stay on the gateway, so one leaked app token can be revoked without rotating the provider key for everyone.
  • Budgets and rate limits. Spend or token caps per caller, team or project, and calls per minute. Without them, one runaway loop can spend a month's allowance; OWASP lists this as LLM10, Unbounded Consumption.
  • Model allowlistsModel allowlist: A list of the providers and models an organization permits. Calls to any other model are refused.. Which models may be called at all, for example only models your contracts or data rules allow.
  • Checks on text. Optional checks on the prompt before it leaves and on the answer before it returns.
  • Usage records. Tokens, cost, latency and outcome per request, for chargeback and incident review.

What it cannot do is stop a tool from running. If the model answers with a tool_calls entry, the gateway passes it back and the application runs the tool itself. The gateway can block the whole answer. It sees a tool's result only if the application sends it back to the model on the next request, and it never sees what the call changed in the other system.

What an MCP gateway controls ​

An MCP gateway is an MCP server that sits in front of other MCP servers. Clients such as IDE agents, desktop assistants and command-line agents connect to it over Streamable HTTPStreamable HTTP: The MCP transport for servers reached over a network: every client message is an HTTP POST to one endpoint, and the server answers with a JSON body or a server-sent events stream., and it forwards their calls to the real servers. Typical controls:

  • Toolsets. A named group of tools served at one endpoint. A support team's agents get the ticket tools; a release bot gets the deploy tools. A client only sees the toolsets it was given.
  • A token per client. Each client authenticates with its own token. The gateway holds the upstream servers' credentials and adds them itself, so clients never receive a code host or database secret. The MCP security guidance forbids the shortcut where a server accepts a token that was not issued to it and forwards it downstream ("token passthrough"): "MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server."
  • Rules per tool call. Allow, deny or require a human approval for a tool, and, in stricter gateways, for particular argument values (deny push to a production repository, hold refunds above an amount for approval).
  • Budgets and rate limits on tool calls. The MCP specification says servers MUST "Rate limit tool invocations"; a gateway is one place to do it for every server behind it.
  • Telemetry and audit. Which client called which tool, when, with what outcome.

What it cannot do is limit the agent's own model spend or choose its model. It never sees the prompt that led to the call.

Where the names get muddled ​

"AI gateway" is used for both, and some products bundle both behind one admin screen. These questions sort a product into one column or the other:

QuestionLLM gatewayMCP gateway
What protocol do callers speak?A model API, usually OpenAI-compatible chat completionsMCP (JSON-RPC over Streamable HTTP)
What does one request ask for?A completion from a named modelA call to a named tool with arguments
What does a rule decide?Which model, how much spend, what text may passWhich tool, which arguments, who must approve
Whose credentials does it hold?Model provider keysCredentials for databases, SaaS APIs, code hosts
What does a refusal look like?An HTTP error such as 429A JSON-RPC error or a tool error the agent reads

When a model-side gateway talks about governing tools, check which side it means. Filtering tool_calls in model answers is useful, but it is not the same as deciding each tool call with its real arguments at the point where it runs.

Do you need both? ​

It depends on what leaves your network and who calls what.

  • Apps call models, no agents with tools: an LLM gateway is enough.
  • Agents in IDEs or desktop apps call shared tools, and each person brings their own model subscription: an MCP gateway is the part you control. Model calls go from the client to its provider, outside your gateway.
  • Your own agents call models and tools: you need both. The LLM gateway caps spend and picks models; the MCP gateway decides what the agents may do with the answers.

If you run both, give each caller one identity across them. A cap per client then means the same thing on both paths, and an incident review can follow one client from the prompt to the tool call.

Both in one self-hosted deployment ​

L2M has both, and both are optional: nothing is served until you configure it.

  • The /v1 AI gateway serves an OpenAI-compatible API (/v1/chat/completions, /v1/embeddings, /v1/models). Admins define route sets that map public model names to providers and stored keys. Each request is checked against the client's grant, its rate limit, budgets and the model allowlists before the provider is called, and writes a usage row with tokens and estimated cost, never the prompt or answer. A used-up block budget answers 429 with code insufficient_quota.
  • The MCP gateway serves toolsetsToolset: A named, curated group of tools, drawn from several MCP servers and workflows, served at one MCP endpoint so each client sees only the tools it needs. at /mcp/<toolset> over Streamable HTTP, speaking MCP protocol versions 2025-11-25, 2025-06-18 and 2025-03-26. A toolset can hold workflows exposed as tools and tools of registered MCP servers. Calls to server tools go through MCP tool policy and get the server's secret from the gateway; a workflow tool runs its workflow, whose agents check policy in turn. Calls that reach a tool write a telemetry row. Budgets and per-client rate limits are optional: a used-up block budget refuses the call with JSON-RPC error -32032, and a rate limit with -32029.

The two share one set of client tokens, created under MCP → Clients. A client's rate limit is one allowance across both paths, and a budget with the MCP client scope counts its /v1 usage and its gateway usage together. The reverse direction also works: an agent inside an L2M workflow can send its model calls to an existing corporate gateway through the AI Gateway Chat Model node.

bash
# Model traffic: any OpenAI SDK, pointed at L2M
curl https://l2m.example.com/v1/chat/completions \
  -H "Authorization: Bearer $L2M_CLIENT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model": "fast-chat", "messages": [{"role": "user", "content": "Hello"}]}'

Try it in L2M ​

Follow Run a self-hosted MCP gateway to put one toolset in front of a registered server and connect an IDE agent with its own token. Then add a route set with one route and its provider key under MCP → AI routes, and tick it under AI routes in the same client's Governance section (MCP → Clients) to send its model calls through /v1. To watch an agent call MCP tools without installing anything, open the support agent in the live demo.

Further reading ​

Last updated:

This website (not the L2M product) uses Google Analytics, only after you accept cookies. Cookie settings