What a gateway actually buys

What breaks without one?

Four things, and each becomes visible at the worst moment.

Attribution. One organisational API key means every call looks the same. When spend triples, the question “whose workload is this” has no answer, and finding out takes a day of archaeology.

Revocation. With a shared key, cutting off one misbehaving application means rotating the key for everything, so nobody does it during an incident.

Evidence. If nothing recorded what was asked and what came back, an investigation has nothing to examine. This is also the gap that makes an exposure impossible to assess afterwards.

An off switch. Without a single route, disabling a model or a tool means changing configuration in every application that calls it, coordinated across teams, while the incident continues.

What it is not

It is not a router. Routing between providers is the feature most vendor material leads with and the one an SDK already handles. Selling a gateway on routing is selling the least valuable thing in the box.

It is also not a guardrail. Detection can sit at the gateway and benefits from being there, but the gateway’s job is control and evidence, and conflating the two leads to buying a filter and believing it is a platform.

Why does redaction belong here?

Because a prompt is data, and it is about to be stored.

Tracing products capture prompts and responses by default, and several put masking behind a paid tier. If sensitive fields are removed at the gateway before anything is written, the client’s obligations are satisfied regardless of which observability product is licensed later, and switching tools does not reopen the question.

The alternative — capture everything, mask on read — means the sensitive data is already in a store, and the control is a query rather than a boundary.

Capability	Without a gateway	With one
Attribution	One key, no owner	Virtual key per team and per agent
Budget	Provider invoice, monthly	Hard cap at the key; deviation alarm continuously
Evidence	Whatever each application chose to log	One trace format, redacted at capture
Shutdown	Coordinated redeploys	One configuration change
Model inventory	Discovered during an incident	An allowlist, and an alert when something new appears

The objection that deserves an answer

Routing everything through one component creates a single point of failure. That is true, and pretending otherwise is how the objection becomes an outage.

The mitigations are ordinary: more than one instance, a fallback route that has been tested rather than documented, an explicit decision per application about what happens when the gateway is unavailable, and a rehearsal of a provider outage before one occurs. What the centralisation buys in exchange is the only place where budgets, revocation and evidence can be enforced at all.

Why we build it in the client’s environment

Because of a promise we make at the start of these engagements: if the client stops paying us, the controls keep working. That rules out designs where the control plane belongs to someone else, and products whose licensing makes continued operation conditional. It is a narrower set of choices, and it is the set that survives the end of a relationship.