Running costs at scale

What does a working system actually cost?

The question has no general answer, which is why vendor pricing pages are unhelpful. It has a specific answer per system, and the unit is one completed piece of work: one answer returned, one document processed, one case triaged.

That figure is measurable from the first week of a pilot. Multiply it by real volume and the monthly cost stops being a projection.

Four things drive the bill

Driver	What moves it	What holds it
Context length	Passages retrieved per question, conversation history	Reranking, and a cap on passages passed
Turns per task	Agents that reason in loops	A step limit per conversation
Model choice	Routing every request to the strongest model	Routing by task, measured on the graded set
Retries	Failures that re-run the whole call	A retry budget, and idempotency

Context length is the one most often underestimated, because each turn in an agent conversation carries the accumulated context forward. Cost per turn therefore rises as the conversation lengthens, which is why an unproductive exchange is more expensive than a productive one of the same length.

Who is the bill attributed to?

Without cost attribution there is one number for the organisation, and no way to answer whether a use is worth its cost. With it there is a figure per team, per feature and per workflow, and the conversation changes from “AI is expensive” to “this workflow costs eleven hundred a month and saves an adviser four hours a week”.

Attribution is a build decision, and a cheap one if taken early. Every call carries an identity through the gateway, and the gateway records which team, which feature and which model answered.

Limits that hold

A monthly ceiling notices after the money is gone. What catches a loop while it is still running is a per-step limit on the conversation, a per-key budget with a deviation alarm against the trailing baseline, and a hard cap at the key as the backstop. The first is the detector. The last stops the bill.

Published runaway-cost accounts share a shape: no step cap, no per-key budget, and a monthly invoice as the first signal.

What do we do about it?

Unit economics are established during the pilot rather than estimated before it. Per-team budgets sit at the gateway with deviation alarms against the trailing baseline, and every call is attributed to the team and feature that made it.

The monthly report carries cost per team against the previous month, so a change in the bill has an owner before it has a surprise.