Why AI pilots fail to reach production

What do the failure statistics actually measure?

Two numbers dominate discussion of enterprise AI. The first, that 95% of pilots produce no measurable return, comes from a single report published by MIT’s Media Lab NANDA initiative in August 2025. Its method is stated openly: 52 structured interviews, 153 survey responses collected at four conferences, and a review of roughly 300 publicly disclosed initiatives, gathered between January and June 2025. That is a study of a self-selected population, and the report has been criticised on exactly those grounds.

The second number, that more than 80% of AI projects fail at about twice the rate of other IT projects, appears in RAND’s 2024 report. Read the sentence in context and it is framing rather than finding: RAND cites the estimate, then investigates the causes.

Neither figure supports a claim about how often AI projects fail. Both are useful for something narrower and more interesting, which is where the failures sit.

The causes are documented, and they are mostly organisational

RAND interviewed 65 data scientists and engineers with at least five years of experience and grouped what they described into five root causes. Leadership misunderstanding or miscommunicating the problem. Data that could not support the intended use. Technology chosen before a problem was identified. Underinvestment in the infrastructure needed to deploy and operate a model. And, last, applying AI to problems the state of the art cannot yet solve.

Only the fifth is a limit of the technology. Among industry interviewees, 84% named leadership-driven issues as the primary cause, which places the decisive failure well before any engineering begins.

This is not new. Sculley and colleagues at Google described the same shape in 2015, when they observed that the model occupies a small box in a much larger system, and that the surrounding code, data dependencies, feedback loops and unrecorded consumers generate the ongoing cost. Language models changed what sits inside the box. They did not shrink the diagram.

What separates a demo, a pilot and a production system?

The three are often treated as stages of the same thing, differing in scale. They differ in what they establish.

A demo shows that the model can produce an acceptable output on inputs chosen by the person demonstrating it. A pilot shows that acceptable outputs hold across a bounded set of real cases with a small group of users who know they are testing something. Production means an organisation depends on the system while the provider changes the model underneath it, usage patterns move, costs move with them, and someone has to answer for the result on an ordinary Tuesday.

The risk a pilot removes is capability risk. In most engagements that is the smallest of the risks present, which is why a successful pilot so often fails to predict anything about the system that follows it.

Three states, three questions

DemoCan the model do this at all?

Inputs chosen by the presenter. Nobody depends on the answer.

PilotDoes it hold across real cases?

Bounded scope, known users, tolerant conditions, usually open permissions.

ProductionCan the organisation operate it?

Real permissions, changing models, monitored cost, named owner, incident path.

The gap between a pilot and a service is ordinary engineering

Almost everything missing from a stalled pilot is work an engineering team already knows how to do. It is skipped because the pilot did not need it.

The definition of a correct answer is the first omission and the most expensive. Without a written statement of what an acceptable answer or action looks like for this task, agreed with the people who own the work, there is no way to say whether the system is good enough, and no way to notice when it stops being good enough. Evaluation depends on that definition; so does the decision to launch.

Permissions are the second. Pilots commonly run under a service account with broad access to a document store, because that is the fastest way to make retrieval work. Production runs under the permissions of the person asking, which changes the answers as well as the risk. Retrofitting this after launch tends to be a redesign rather than a configuration change.

Cost and control come third. A pilot with twenty users and a company credit card gives no signal about what a thousand users cost, and provider dashboards report spend on a delay long enough for a retry loop to run up a substantial bill before anyone sees it. Routing model calls through a single controlled point, with identity, budgets, logging and a way to switch a model off, is what makes the system operable at all.

The last omission is ownership. A production system has a person with budget and the authority to make decisions, a measurement that runs on a schedule, and a documented path for the day something goes wrong at two in the morning. A pilot with none of these cannot be handed to anyone, and often is not handed to anyone, which is one way a project ends without a decision ever being taken.

What the delivery evidence adds

DORA’s 2025 research found AI adoption now associated with higher delivery throughput and, at the same time, with continued instability. Their reading is that generated volume outpaces the review and deployment capacity available to absorb it.

Applied to AI projects specifically, the pattern is consistent with what the RAND interviews describe. Capability is cheap and rising. The constraint has moved to the systems that decide whether what was produced is safe to run, and those systems are slower to build than a prototype.

What this means before a project starts

Most of the questions that determine whether a pilot becomes a service can be asked in the first week, when they are cheap to answer.

• What does a correct answer look like for this task, in writing, agreed by the people who own the work?
• Which real cases will the system be measured against, and who marks them?
• Whose permissions does the system act with?
• What does it cost at expected volume, and what stops a runaway?
• Who owns it after launch, and what do they receive each month?

A pilot that cannot answer these is still worth running, provided everyone understands that it is answering a capability question and nothing else. The failure mode worth avoiding is the one where a successful demonstration is mistaken for evidence that the system is nearly ready.