How to tell a provider changed the model

Can a model change without the version changing?

Yes, and it has. At least one laboratory has published a postmortem describing serving faults that degraded output quality while the version string clients were pinning stayed the same. Independent measurement has also shown a single named version behaving differently across months.

None of that requires bad faith. Serving stacks change, routing changes, safety layers change, and quantisation or hardware differences shift behaviour at the margin. The version string identifies the weights, not the system that serves them.

For anyone operating an AI system, the consequence is specific: a change you did not make, did not approve, and will not be told about can alter what your users receive.

What does not detect it

Reading the changelog. Providers publish deprecations and new versions. Serving-level regressions surface, when they surface at all, after users report them.

Uptime monitoring. The endpoint is up, latency is normal, error rate is flat. The answers are worse.

User complaints. These do eventually arrive, and by then the system has been producing worse answers for as long as it took someone to care enough to report it. In an internal tool, that can be months, because people quietly stop using it instead.

A one-off benchmark from the vendor. It measures their test set on their configuration on the day they ran it.

What does detect it?

The same questions, asked on a schedule, against the live system, with the answers compared to a marked expectation.

That is the whole mechanism, and its value comes from the discipline rather than the sophistication. A fixed set of cases drawn from the organisation’s real work, graded once by someone who knows the domain, then run weekly. When the score moves, something changed: the model, the prompt, the retrieval corpus, or the traffic. All four are worth knowing about.

Between runs, cheap statistical monitors catch faster movements. Refusal rate is the most useful single one, because a safety-layer change usually shows up there first. Output length distribution and tokens per request are almost free to compute and move visibly when something underneath has shifted.

Signal	Detects	Cadence	Cost
Fixed question set, graded	Quality movement of any cause	Weekly	Moderate; the set is the asset
Refusal rate	Safety-layer and policy changes	Continuous	Near zero
Output length distribution	Serving or prompt changes	Continuous	Near zero
Tokens per request	Retrieval or context changes, and cost drift	Continuous	Near zero
Deprecation calendar	Announced end of life	Weekly review	Near zero, and often ignored

The part people skip: the calendar

Announced deprecations are the easy case and still catch organisations out. Model lifecycles have compressed, and a migration handled with ninety days’ notice is routine engineering work. The same migration handled with a fortnight’s notice is an incident with a deadline.

Tracking end-of-life dates for every model in use, at least ninety days ahead, costs an hour a month. It converts the most predictable disruption in this field into scheduled work.

What we run

In a monitoring engagement, the fixed set runs weekly against the live system rather than a staging copy, the statistical monitors run continuously, and the deprecation calendar is reviewed with the monthly report. When the weekly score moves, the report says what moved and what we think caused it, which is a different document from one that says the system was available 99.9% of the time.