UltHealth case study

SectorConsumer health

PlatformMobile application with AI features

ServicesAI Red Teaming, Secure AI Engineering

OutcomeFindings closed, verified and wired into CI

The challenge

UltHealth holds the data people are least willing to have exposed: sleep, weight, resting heart rate, nutrition, medication habits and the notes they write about how they feel. The AI features read that record and produce guidance from it.

Two properties make that combination worth testing rather than assuming. The model reads content the user supplies, including free-text notes and uploaded material, which is untrusted input arriving in the same channel as the application’s own instructions. And the assistant reaches a private data store on the user’s behalf, which means the question is not only what it says but what it can be made to retrieve and where that could go.

UltHealth insights generated from personal data

UltHealth daily check-in, including free-text notes

The AI reads the record on the left and writes the guidance in the middle. The free-text field on the right is user-supplied content the model will later read.

One loop, not two engagements

Testing that ends at a report leaves the client to arrange their own remediation and to assert their own closure. Here the testing and the engineering ran as one loop, and every confirmed attack finished as a test in the client’s pipeline.

Red teaming

Discover the attack

Automated probes set the floor. Engineers then worked the paths a scanner cannot reach: instructions planted in user-supplied content, retrieval behaviour, and what the assistant could be persuaded to fetch.

Reproduce it

An attack that worked once is an anecdote. Each candidate was repeated to establish how often it succeeded, and only reproducible cases were reported.

Understand the root cause

Why the system allowed it: what the model could reach, which instruction it treated as authoritative, and which check was missing rather than which phrase got through.

Secure AI engineering

Implement the control

The fix addressed the cause rather than the phrasing: what the assistant may reach, what it may do without a person, and where the boundary between user content and instruction sits.

Re-run the attack

The same attack, the same number of attempts, against the changed system. A finding is closed when the attack fails, not when a ticket does.

Add a regression test

The attack becomes an assertion in the client’s own pipeline, with the expected outcome recorded.

Every future release

The suite runs on each deploy, so a prompt edit or a model change that reopens the finding fails the build rather than reaching users.

Why the loop matters: findings age within weeks as prompts, models and data change. The tests do not, which is why the regression suite is the part clients keep long after the report has been filed.

What the testing covered

Instructions inside user content

Free-text notes, logged meals and uploaded material are read by the model later. Each is a place an instruction can be planted, and none of them looks like an attack when it is written.

What the assistant can retrieve

The assistant reads a private health record on the user’s behalf. The test is whether it can be made to retrieve or reveal more than that user’s own data.

Where information could leave

Any outbound path — a link, a share, an integration, a connected device — turns a retrieval problem into an exfiltration problem, so the paths were enumerated before the attacks were written.

Guidance under adversarial input

Health guidance carries consequences. Testing covered whether the assistant could be pushed into advice it should decline, and whether it declined the things it should.

Logging a meal, including free-text fields

Experiments driven by the personal record

The outcome

Every confirmed finding carries the attack, the transcript, how often it succeeded, the severity and the fix. Each was re-tested after remediation and recorded as closed on a date rather than assumed to be.

What UltHealth holds now is a changed system and the evidence that it changed: controls in the product, a regression suite in the pipeline, and a report that a customer, an investor or a regulator can read without taking the claim on trust.

Specific findings are not published, at the client’s request. The method, the controls and the verification are described here in full.

AI Red Teaming → Secure AI Engineering →