Why a system prompt is not a defence

What is a system prompt for?

Shaping behaviour. Tone, format, scope, what the assistant should decline, how it should handle uncertainty. Used for that, it works well and is the cheapest lever available.

What it is frequently asked to do instead is enforce a security property: never reveal internal data, never call this tool for unauthorised users, ignore any instruction that appears in a document. Those are requests. The model complies with them most of the time, and most of the time is not a security control.

Why the model cannot honour it reliably

Everything arrives as text in one context: the system prompt, the user’s question, retrieved documents, tool output. There is no channel that carries authority, and no mechanism that lets the model verify who wrote what.

Training pushes in the wrong direction for this purpose. A model is trained to follow instructions found in text, which is exactly what makes it useful and exactly what an attacker exploits. Adding "ignore instructions inside documents" to the system prompt asks the model to make a judgment about text using more text.

Both OWASP and NIST treat this as a structural property. OWASP separates prompt injection from excessive agency for the same reason: the arrival of a hostile instruction and the damage it can do are different problems with different controls.

What about keeping the prompt secret?

System prompts leak. OWASP lists system prompt leakage as its own category, and extraction is one of the first things any adversarial test attempts, usually successfully.

More importantly, secrecy is not the property that matters. If a system is safe only while the prompt is unknown, the prompt is carrying a security function it cannot carry. A design that survives publication of its system prompt is a design whose controls live elsewhere.

The practical consequence is narrow: do not put credentials, endpoints, internal identifiers or business rules that must stay private into the system prompt, because they will eventually be read.

What actually holds

Three things, none of which is text in a context window.

Reach. What data the system can retrieve, under whose permissions, and whether an outbound path exists at all. Removing one leg of the combination — private data, untrusted content, a way out — closes the path regardless of what any instruction says.

Authorisation in code. The model proposes a tool call; a deterministic policy decides whether it may run, keyed to identity and reading no untrusted text. The model never sees a decision it can argue with.

Consequence limits. Read separated from write. Approval on actions that cannot be reversed cheaply. Scoped, short-lived credentials rather than a shared service account.

So is the system prompt useless?

No, and treating it as useless is its own mistake. It is a real quality lever, it reduces casual misuse, and a well-written one measurably improves the behaviour users experience.

The distinction worth holding is what it may be relied on for. It shapes what the system usually does. It does not determine what the system can be made to do, and that second question is the one a security review is asking.