Building an AI gateway

What is a gateway for?

The word suggests routing, and routing is the part an SDK already handles. A gateway earns its place for four other reasons: it is where identity is attached to a model call, where budgets are enforced, where a record of what was asked and answered is created, and where a model can be switched off in one change.

Without those, an organisation running AI in production has no way to attribute spend, no way to revoke access to a single team’s workload, no evidence for an investigation, and no faster remedy during an incident than asking application teams to redeploy.

Cost is derived, and the derivation matters

Token counts are returned with the response. Money is not. Every cost figure in an AI system is a token count multiplied against a price table that someone has to maintain as providers change their pricing.

Provider dashboards do report spend, at provider granularity and on a delay. The delay is the problem. Published accounts of runaway agent costs share a shape: a loop between two components, no step cap, no per-conversation budget, and a bill that had already accumulated before anything in the reporting chain showed a change. These accounts are practitioner write-ups rather than audited incident reports, so the dollar figures should be treated as self-reported. The mechanism they describe is consistent enough to design against.

A monthly ceiling is the wrong control for that failure. It triggers after the spend has happened, which is too late to matter. What catches a loop is deviation from a trailing baseline: an alarm on spend running at many times the recent average, evaluated continuously, backed by a hard cap at the key as a last resort and retry limits so a loop fails closed rather than spinning.

What does a production gateway carry?

Six things, in roughly this order of importance.

Identity. Virtual keys issued per team and per agent rather than one company key shared everywhere. Attribution, billing and revocation all depend on this, and it is the hardest thing to retrofit.

Budgets and limits at the key. Per-team budgets, rate limits, retry caps, and deviation alarms on top of them.

An approved model and tool list. The inventory of what may be called. New models appearing in traffic without approval is one of the more useful signals a gateway produces, because it shows where the organisation is moving faster than its own governance.

Redaction before storage. Sensitive fields removed at the gateway, before a tracing tool ever receives a prompt. Prompts are data, and most tracing products do not mask them by default; several put masking behind a paid tier. Moving redaction upstream is better practice regardless of which product is licensed.

Traces with cost attached. Per-request traces including token counts, with cost derived at capture time so the number exists when somebody asks at two in the morning.

An off switch that has been tested. One configuration change that cuts a model, a tool or a key within minutes, plus a fallback route for provider outages. Untested, this is a claim rather than a control.

Shape	Strengths	Costs	Suits
Application-level gateway	Virtual keys, per-team budgets, spend attribution, guardrail hooks, broad provider coverage	Application runtime under concurrency; another service to operate	Organisations that need product-level controls now
Infrastructure gateway	Proxy-grade routing and rate limiting; fits existing mesh operations	Fewer product features; assumes platform maturity	Kubernetes-native platforms with an operations team
Managed edge service	No operations; fastest to adopt	Traffic crosses a third party; control plane is not yours	Teams already on that provider, with data policy that permits it

The objection worth taking seriously

Centralising every model call creates a single point of failure. That is true, and dismissing it is how the objection becomes an outage.

The mitigations are ordinary: run more than one instance, keep the fallback route tested rather than documented, make the failure mode explicit for each application, and rehearse a provider outage before one happens. What the centralisation buys in exchange is a single point of control, which is the only place where budgets, revocation and evidence can actually be enforced. The trade is worth making, and it should be made with the failure modes written down.

A note on telemetry conventions

OpenTelemetry’s conventions for generative AI now model an agent run as a span tree rather than a series of isolated model calls, which is the right shape. They remain in development status, and in June 2026 they moved into a dedicated repository alongside the MCP conventions.

The practical consequence for anyone instrumenting a system this year is to pin the instrumentation library version and record a dated snapshot of the conventions in use. Building on an unstable specification is reasonable; building on it without recording which version was used is how a telemetry pipeline breaks quietly during an upgrade.

What we build, and why it stays

Agile Labs deploys gateways in the client’s own environment on a permissively licensed stack, wired to their identity provider. The reason is a promise we make at the start of these engagements: if the client stops paying us, the controls keep working.

That constraint rules out designs where the control plane belongs to somebody else, and it rules out products whose licensing makes the client’s continued operation conditional. It is a narrower set of choices, and it is the set that survives the end of a relationship.