The question I kept being asked
Between 2022 and 2026, I was involved in or conducted post-incident reviews for eleven AI proof-of-concepts that did not reach production, or that reached production and were shut down within six months.
The question that came after each one was the same: what went wrong? The answer was almost never the model. This research is an attempt to describe, with specificity, what actually went wrong.
The sample
Eleven PoCs across five industries: financial services (three), media and content (two), healthcare administration (two), manufacturing (two), and public sector (two). Organisation sizes ranged from five hundred to fifty thousand employees. All had received board-level approval for AI investment. All had technical teams capable of building the systems.
Of the eleven, nine had a model that met its evaluation target. The system worked, in the evaluation environment. None of those nine reached sustained production use.
The seven failure patterns
1. Evaluation environments that did not represent production
The most common failure pattern, present in eight of eleven cases. The model was evaluated on clean, structured, internally-sourced data. Production data was messier, more diverse, and structured differently. The model’s performance on evaluation data did not predict its performance on production data.
The specific gap varied: missing fields, encoding inconsistencies, upstream data pipeline changes, edge case distributions that were absent from the evaluation dataset but common in production. In two cases, the evaluation data had been cleaned specifically for the evaluation, which was not disclosed to the team building the model.
2. No named owner after handoff
Present in seven of eleven cases. The team that built the PoC did not become the team that owned the production system. The handoff was to an operational team that had not been involved in the build, did not understand the system’s failure modes, and had no clear accountability for its outputs.
When the system produced a wrong output, the question of who was responsible could not be answered. The system was eventually shut down because the accountability vacuum made incident resolution impossible.
3. Infrastructure budget not planned
Present in six of eleven cases. The PoC was approved with a build budget. When the team requested a separate infrastructure budget for monitoring, retraining and on-call coverage, the request was declined or deferred. The system was launched without the infrastructure it needed to detect and correct failures.
In two of these cases, the infrastructure gap was known before launch and was escalated to project leadership, which decided to proceed anyway.
4. Pilot success misread as production readiness
Present in five of eleven cases. The pilot produced strong results. The project team presented those results as evidence of production readiness. Project leadership approved production deployment without requiring the team to address the difference between pilot conditions and production conditions.
The systems that were shut down within six months were all in this category. They reached production, produced wrong outputs at an unacceptable rate, and were shut down before achieving meaningful use.
5. No defined success metric for production
Present in five of eleven cases. The PoC had a clear evaluation metric: accuracy on a labelled test set, precision and recall on a classification task. The production system had no equivalent metric, because the outcomes in production were harder to label.
Without a production success metric, the team could not demonstrate that the system was working, and could not detect when it stopped. The system ran, but nobody knew whether it was running well.
6. Volume assumptions that were wrong
Present in four of eleven cases. The PoC was built and evaluated at a scale of hundreds of examples. Production required processing thousands or tens of thousands. The system was not architected for the production volume and required complete re-engineering before it could serve production load.
7. Regulatory review not planned for
Present in three of eleven cases (all in financial services and healthcare). The PoC reached a production-ready state before the team discovered that the system’s outputs required regulatory approval that had not been planned for. The regulatory review added between six months and a year to the timeline. Two of the three systems were abandoned before the review was complete.
What the pattern suggests
The failures are not random. They cluster around two root causes: inadequate planning for the difference between evaluation and production conditions, and inadequate organisational preparation for accountability and ownership.
Both can be addressed before the PoC begins. The organisations that address them before building have materially better production rates than those that address them after a failure.
Implications for PoC design
A PoC that is designed to reach production should include, at minimum: an evaluation dataset that is drawn from real production conditions, a named production owner who participates in the PoC evaluation, a documented infrastructure plan for monitoring and maintenance, and a defined production success metric agreed before the system launches.
None of these are engineering requirements. All of them are organisational requirements. The engineering team can build the system; only the organisation can create the conditions for the system to succeed.