Monthly AI Oversight Report

What ran, what it cost, what was blocked, what changed upstream, and the one decision this month needs.

SPECIMEN. This is a layout sample showing the form of the report the retainer produces each month. The client, the systems, the teams and every figure in it are invented. It is not a record of any organisation’s activity, and nothing in it may be cited as evidence.

Client

xxxxxxxxxxxxxxxxxxxxxx · financial services, Singapore

Period

1–31 August 2026 · retainer month 7

Systems covered

Customer support assistant · internal knowledge assistant · claims triage agent

Instrumented at

The client’s own gateway. All traces, budgets and keys sit inside their environment

Watch

Machines continuously; people on a weekly, monthly and quarterly cadence. No twenty-four-hour staffed watch is offered or implied

Distribution

Confidential · head of technology, CISO, risk committee pack

The month at a glance

Every figure below is a trend against the trailing baseline, not a snapshot. Twelve-week lines.

Model spendS$18,420 11% above the twelve-week mean. One anomaly caught, section 2.

Calls1.42M Rising with support volume. Cost per call fell 4%.

Blocked3,108 0.22% of traffic. False-positive rate 0.4%, within the agreed band.

Answer quality91.4% Down 4.1 points in week 3, recovered. Cause in section 4.

1 What ran, and what appeared unapproved

The inventory delta from the gateway logs. This table is the shadow-AI register.

Tool or model	Owner	Status	Share of calls
Support assistant · primary model	Customer operations	Approved	68%
Knowledge assistant · primary model	Technology	Approved	24%
Claims triage agent · four tools	Claims	Approved	7%
Vision model, first seen 12 August	xxxxxxxxx, marketing	New, unapproved	0.9%
Consumer-tier assistant reached from two laptops	Unattributed	Outside the gateway	Not measurable

Two appeared this month that were not in July’s register. The vision model came in through an approved key and is visible; the consumer-tier assistant was reached from two managed laptops and does not pass through the gateway at all, so nothing about what it was sent can be recovered. Both are carried into section 5.

2 Spend against the trailing baseline

Cost is computed per call as it happens and reconciled monthly against provider invoices, because providers report spend a day late.

Team	August	Twelve-week mean	Movement
Customer operations	S$11,940	S$11,210	+6.5%, tracking volume
Technology	S$4,180	S$4,320	−3.2%
Claims	S$2,300	S$1,090	+111%, see below

The anomaly, caught on 19 August at 02:14

A retry loop in the claims triage agent ran at fifty-one times the trailing average for eleven minutes. The deviation alarm fired at four minutes, paged the named on-call in Claims and the Agile Labs duty engineer, and the retry cap on the key stopped it at the eleven-minute mark. Exposure: S$1,180. A monthly ceiling would not have noticed until 31 August, at roughly xxxxxxx times that figure.

The fix — a bounded retry with jitter on the claims tool — shipped 21 August and is tested weekly. Reconciliation against the provider invoice for August closed at a 0.7% variance, within tolerance.

3 What was blocked

Signal	August	Trend	Note
Refusals, ordinary	2,742	Flat	Policy working as configured
Jailbreak attempts	319	Up 38%	211 from one account, section 5
Data-loss blocks, outbound	47	Down 12%	Mostly pasted customer identifiers
False positives, sampled	0.4%	Flat	Agreed band is under 1%

The false-positive figure is reported every month alongside the block count, because a control that refuses ordinary work is switched off within weeks and the block count on its own would look like success.

4 What changed upstream, and what deprecates next

The fixed question set runs weekly against the live system. Providers change models underneath a running system without notice; comparing the same answers week to week is the only way to see it.

Week	Question set score	What moved
Week 1	95.5%	Baseline
Week 2	95.2%	Within noise
Week 3	91.4%	Provider served a revised model with no version change. Twelve questions on policy citation regressed together
Week 4	95.1%	Prompt adjusted 27 August; recovered

Deprecation calendar, next ninety days

The support assistant’s primary model retires xx xxxxxxx xxxx — 74 days. A replacement has been scored against the question set and sits 0.6 points above the incumbent.
An embedding model used by the knowledge assistant moves to a new version in October, which requires the retrieval corpus to be re-indexed. Half a day, scheduled.
No change announced for the claims agent’s tools.

Model lifecycles are compressing toward six to twelve months, so migration is routine work planned inside the retainer rather than an incident handled outside it.

5 What is outstanding

Item	Owner	Raised	Status
Consumer-tier assistant reached from two managed laptops, outside the gateway	Client, endpoint team	14 Aug	Open
Unapproved vision model to be approved or withdrawn	Client, marketing	12 Aug	Open
211 jailbreak attempts from one internal account; identity review requested	Client, CISO	22 Aug	Open
Support assistant model migration before the retirement date	Agile Labs	28 Aug	Scheduled
Quarterly runbook rehearsal and re-attack	Agile Labs	—	Due September

6 The decision this month

One decision, for the monthly meeting

The consumer-tier assistant reached from two managed laptops

It does not pass through the gateway, so nothing it was sent can be recovered and nothing about it appears in any figure in this report. It is the only thing under watch that cannot be measured. Three routes, and the choice is the client’s:

Block the domain at the endpoint. Fastest, and it will surface whatever the two people were using it for as a support request within days.

Bring it inside the gateway on a commercial tier. Costs more, keeps the workflow, and it becomes measurable next month.

Accept it in writing, with the date and the named owner, and it moves from an open finding to an accepted risk on the register.

Recommended: B, because the traffic pattern suggests real work rather than curiosity, and A returns it as an unplanned support request in the same week.

Every report ends with exactly one decision. It is what keeps the monthly meeting a working session rather than a presentation.

For supervisory use. The five sections above map to the four dimensions of the MAS consultation on Guidelines on AI Risk Management: inventory and materiality to section 1; third-party and model risk to sections 2 and 4; controls and monitoring to section 3; governance and oversight to sections 5 and 6. Prepared with reference to the proposed guidelines, which remain in draft. This report is not a compliance opinion and does not claim conformance with guidelines that have not been issued.