Stop Hard-Wiring LLMs Into Your Code: The Case for an AI Mediation Layer via an AI Gateway | WSO2

Skip to main content

CONTACT

Register

Login

Login to Customer Portal

Login to Partner Portal

Blog

Share on social media

TL;DR: AI mediation is the layer an AI gateway adds between client applications (or AI agents) and backend LLMs, where it translates, routes, secures, and optimizes traffic that behaves very differently from ordinary web requests. A capable AI gateway gives you a single endpoint and key in front of providers like OpenAI, Anthropic, Azure OpenAI, Mistral, and AWS Bedrock; multi-provider load balancing and failover; semantic caching and prompt compression to cut token cost and latency; guardrails for PII masking, prompt safety, and content moderation; and token-based and cost-based rate limiting. The AI gateway is one piece of a larger picture: a full platform pairs it with an API gateway for REST, GraphQL, gRPC, and SOAP and an event gateway for streaming and event traffic, all governed from one control plane rather than one physical gateway.

Traffic to a language model does not look like traffic to a web service. It is priced per token, it is slow and bursty, its responses are non-deterministic, and a “bad” reply is a safety or compliance problem rather than a stack trace. Point a few applications at a few model providers directly and you end up with vendor SDKs hard-wired into every codebase, no shared view of cost, and no consistent place to enforce safety. That is the gap an AI gateway fills, and the work it does in that gap is what people mean by AI mediation.

Figure 1: The AI gateway as a mediation layer between client apps or agents and the models, tools, and APIs behind them.

An agent is an LLM in a loop

It helps to be precise about what an “AI agent” actually is, because it explains why the gateway has to govern two kinds of traffic at once. An agent is not a program that reasons on its own. It is a loop wrapped around an LLM. The framework code people call “the agent” is plumbing: it holds the goal, tracks state, and passes messages back and forth. The reasoning, deciding which tool to use, with what arguments, and what the result means, happens in the LLM.

That is why having tools available does not remove the need to call the model. A tool is a fixed function someone wrote ahead of time. It executes; it does not decide. Give an agent a getOrder tool and a refundOrder tool and it still cannot handle “the customer says their order arrived broken, sort it out” without an LLM to interpret the request, pick the tools, read the results, and choose the next step. Each step in that loop is another call to the model, interleaved with calls to the tools.

Figure 2: The agent holds state; the LLM reasons; the tools execute. Every turn of the loop is another model call plus tool calls.

So a single agent request is not one call. It is a burst: reason, act, observe, reason again, act again. The reasoning calls go to an LLM provider through the AI gateway; the action calls go to APIs and tools through whichever gateway fronts them. Governing an agent means governing both, which is why the AI gateway sits alongside the rest of the platform under one control plane rather than off to the side.

The mediation layer: One front door to many models

The first job of mediation is to stop applications from talking to model providers directly. A good AI gateway lets an administrator define a provider connection once, including the upstream credential and the details of how to read each provider’s usage and model metadata. Applications then call the gateway’s endpoint with an inbound key the gateway issues, and never touch the real provider key.

That indirection is what makes everything else possible. Because the gateway sits in the middle, it can standardize inputs with prompt templates and decorators (for example, prepending “Act as a support assistant” so tone and role stay consistent across apps), apply policy on the way in and out, and present one stable contract to developers even as the providers behind it change. The strongest designs separate two layers: an organization-wide provider configuration that sets the baseline, and per-application proxies that inherit it and add their own rules. An application can tighten the baseline, but it can never bypass it.

Figure 3: The two-layer model: One shared provider sets the organization baseline; each app’s proxy inherits it and layers its own rules on top.

The diagram above shows three app LLM proxies, /assistant, /codegen, and /supportbot, each add app-specific prompt templates and guardrails, and all inherit from one shared LLM provider that holds the upstream credential, organization-wide guardrails, and budget and access controls, connecting to the LLM backend.

The payoff shows up the first time you switch models or a key leaks. With the gateway in front, you rotate one credential or repoint one provider and every application...