LLM gateway
A service between applications and model providers that gives them one API endpoint and adds per-client keys, routing, spend limits, rate limits and checks on prompts and answers.
Also called an AI gateway. Without one, every application holds its own provider keys, picks its own models and reports its own spend, if it reports it at all. A gateway moves those decisions to one place. Applications call the gateway, usually through an OpenAI-compatible APIOpenAI-compatible API: An HTTP API that accepts the same requests as OpenAI's API (such as /v1/chat/completions), so any client built for it can switch servers by changing the base URL., so existing SDKs work by changing the base URL and the key.
What it usually does
- Keys. Each application gets its own gateway token. Provider keys stay in the gateway and can be rotated without touching applications.
- Routing. A public model name such as
fast-chatmaps to a provider and model, so the backend can change without a code change. - Limits. Per-client budgetsBudget: A cap on model spend or tokens for one scope, such as a project, a user or a client, over a period, which warns or blocks new work when it is reached. and rate limitsRate limit: A cap on how many requests a caller may make in a period, so one client cannot flood a service or run up its costs., the controls OWASP lists against unbounded consumptionUnbounded consumption: OWASP LLM10: an application that lets users or agents run up model calls, tokens or compute without limits, which costs money and can deny service to others. (LLM10).
- Checks and records. Allowed models, checks on prompts and answers, and a usage record per call.
Example
Three internal apps call fast-chat. Finance gets a $200 monthly budget, the chatbot a rate limit of 60 calls a minute. When the provider raises prices, the gateway route moves fast-chat to another model; no app is redeployed.
Common confusion
An LLM gateway governs model calls: prompts in, completions out. An MCP gatewayMCP gateway: A service that sits between MCP clients and MCP tools and serves them through one endpoint, adding per-client credentials, access rules, limits and a log of every call. governs tool calls. An agent uses both: the model call goes through one, each tool call through the other.
In L2M
The optional AI gateway serves an OpenAI-compatible API at /v1: routes map public model names to providers, clients use L2M client tokens, and budgets, rate limits, model allowlists and content rules apply. With no route set up, it stays off.
Further reading
- OWASP LLM10:2025 Unbounded Consumption (owasp.org)