Why AI Pilots Stall, and Five Decisions That Get Them to Production
Most AI pilots don’t fail because the model is bad. They stall because of decisions nobody wanted to make at the start. Here are the five we now insist on before anyone writes a line of code.

On this page 9 sections
- The demo went well. Then nothing happened.
- Why demos and production are different animals
- Decision 1: Who owns the outcome?
- Decision 2: What data is it allowed to touch?
- Decision 3: How will we know it’s good enough?
- Decision 4: Where does it fit in someone’s day?
- Decision 5: Who runs it after launch?
- Warning signs your pilot is stalling
- What this looks like in practice
The demo went well. Then nothing happened.
If you have been anywhere near an enterprise AI programme in the last two years, you will recognise this story.
A small team builds a pilot. Maybe it summarises customer complaints, or drafts responses to supplier queries, or answers questions about internal policies. The demo is genuinely impressive. Someone senior says, “This is exactly what we need.” There is a round of applause, a follow-up meeting is booked, and everyone leaves feeling good.
Six months later, the pilot is still a pilot. It runs on a laptop, or in a sandbox nobody else can reach. The team that built it has been pulled onto something else. The business case slide is still in a deck somewhere, but nobody has updated it. If you ask what happened, you get a shrug and a vague answer about “integration” or “security” or “priorities changing.”
We have seen this pattern many times, and we have been part of it more than once in earlier careers. What we have learned is that the model is rarely the problem. The model usually works well enough. What is missing is a handful of decisions that a production system needs and a demo can comfortably avoid.
Why demos and production are different animals
A demo has an easy life. It runs on a curated set of examples. It doesn’t need to know who is asking. It doesn’t write anything back to a real system. Nobody has to support it at 3am, and if it gets something wrong in front of the steering committee, someone laughs and says, “Well, it’s a prototype.”
A production system has none of those luxuries. It has to cope with the messy long tail of real inputs. It has to respect who is allowed to see what. It has to fit into the tools people already use, without making their day harder. It has to be watched, measured, updated and, occasionally, switched off.
None of that is especially hard technically. What makes it hard is that every one of those requirements forces a decision about budget, risk, ownership or accountability. Those decisions are uncomfortable, so they get postponed. And they keep getting postponed until the pilot has to become real, which is exactly the moment it stops.
So we changed our approach. Before building anything that matters, we ask for five decisions. They take a week or two, not months. And they change the odds dramatically.
Decision 1: Who owns the outcome?
Not who owns the model. Who owns the outcome.
A pilot owned by an innovation team is an experiment. A pilot owned by the head of customer operations, with her name next to “reduce average handling time”, is a commitment. When things get difficult, and they always do, the second one gets the attention it needs: data access gets approved, the security review gets prioritised, and the people who will use it actually turn up to the testing sessions.
So in the first week we ask a simple question: which business leader will put their name against the number this system is meant to move? We also ask what that number is today, and what “good” would look like in six months.
If nobody will own it, don’t treat that as a failure. It is useful information. It usually means the use case is interesting but not yet important. Better to find that out now than after three months of engineering.
The model usually works well enough. What is missing is a handful of decisions that a production system needs and a demo can comfortably avoid.
Decision 2: What data is it allowed to touch?
Demos run on exported spreadsheets and a handful of sample documents. Production systems run on live data, with access rules, retention policies, audit requirements and, often, regulators who will ask questions later.
The gap between those two worlds is where many projects quietly lose months. The team discovers, late, that the documents contain personal data that cannot leave a certain region. Or that half the knowledge base is restricted to specific roles, so the assistant can’t simply show everything to everyone. Or that the data the pilot used was a one-off extract that nobody is allowed to refresh.
We agree the data boundary early, in plain language:
- Which sources the system may read, and which it may write to.
- Who may see which answers, and how that maps to existing permissions.
- Where the data is processed, and whether any of it leaves your environment.
- What gets logged, and how long it is kept.
And we bring your security and data-protection colleagues into that conversation at the start, not at the go-live review. In our experience they are far more helpful as designers than as gatekeepers. Most of them would much rather shape something early than block it late.
Decision 3: How will we know it’s good enough?
“It looks good” is not an acceptance criterion. Neither is “the users liked it in the demo.”
Before building, we sit down with the people who know the work best and create a graded set of real examples: questions, documents or cases, along with what a good answer looks like. Typically a few hundred is enough to start. Then we agree three things: the quality bar the system has to clear on those examples, the response time users will tolerate, and the cost per request the business case can afford.
This does two useful things. First, it turns subjective debates (“I don’t think it’s quite there yet”) into measurable ones. Second, it gives you a regression test. Every time someone changes a prompt, swaps a model or updates the data, you can check it against the same examples before it reaches a single user.
It also tends to surface disagreements early. It is not unusual to discover that two experienced people in the same team would answer the same question differently. Far better to resolve that before the AI is blamed for it.
Decision 4: Where does it fit in someone’s day?
People don’t adopt AI tools. They adopt better ways of doing their job.
A brilliant assistant in a separate browser tab will usually lose to a modest one that appears inside the system people already use, at the moment they need it. If a claims handler has to copy text out of one system, paste it into another, wait for an answer and then paste that back, they will try it twice and then go back to the old way. Not because they are resistant to change, but because they are busy.
So we map the workflow before we design the system. Where does the work arrive? Who touches it, in what order? Where are decisions made, and where are they recorded? Which steps are slow, error-prone or simply tedious? Then we decide where AI should sit, and, just as importantly, which steps still need a person to look at the result and approve it.
This is also where you find the real value. It is often not the step everyone assumed. The biggest win might be assembling the information a person needs before they make a decision, rather than making the decision for them.
Decision 5: Who runs it after launch?
AI systems are not finished when they go live. They are closer to a new team member than a new piece of software. They need watching, especially in the first few months.
Data drifts. A new product line launches and the assistant has never heard of it. A model provider releases an update that subtly changes behaviour. Usage grows faster than expected and the monthly bill raises eyebrows in finance. Someone finds an edge case that produces a genuinely embarrassing answer.
None of this is a crisis if someone is clearly responsible and has the means to respond. It becomes a crisis when everyone assumes someone else is watching.
So before launch we agree who will own the system day to day, whether that is an internal team, a partner, or a mix. And we build the hand-over into the plan from the start: documentation that people will actually read, runbooks for common problems, dashboards for quality, cost and usage, and training for the people who will own it.
Warning signs your pilot is stalling
Stalled pilots rarely announce themselves. They just get quieter. If you are not sure whether yours is on track, these are the signs we look for:
- The demo hasn’t changed in weeks. The same five examples keep being shown, because nobody trusts what happens with the sixth.
- “Integration” is always next sprint. The system still doesn’t talk to the tools people actually use.
- Nobody can say what it costs to run. Not the build cost, the monthly running cost at full scale.
- Security hasn’t seen it yet. Or they have, and their questions are sitting unanswered.
- The intended users haven’t used it. Feedback has come from sponsors and enthusiasts, not from the people whose job it is meant to change.
- The business owner has stopped coming to reviews. This is the most reliable sign of all.
Two or more of these and it is worth pausing the build for a week to go back to the five decisions. It feels like a step backwards. It almost always saves time.
What this looks like in practice
In our engagements, these five decisions become a short written charter, usually two or three pages. The business owner, the technology lead and someone from risk all sign it. It is not glamorous work, and nobody has ever applauded a charter in a steering committee.
But it changes the conversation. Instead of “Can AI do this?”, the question becomes “What will it take to run this safely, and who is accountable?” That is a much better question, and a much easier one to answer in week two than in month six.
If you have a pilot that has stalled, it is worth asking which of these five decisions was never made. In our experience, it is rarely more than two, and once they are made, things tend to move surprisingly quickly.
Key takeaways
- Ask a business leader to own the outcome and the number it should move.
- Agree data access, privacy and residency in plain language before building.
- Judge quality, speed and cost against a graded set of real examples.
- Put AI inside the tools people already use, at the moment they need it.
- Decide who runs it after launch, and build the hand-over into the plan.
Case studies & examples
References & further reading
- Hidden Technical Debt in Machine Learning SystemsSculley et al., NeurIPS 2015
- AI Risk Management Framework (AI RMF 1.0)NIST, 2023
- People + AI GuidebookGoogle PAIR
- AI Index ReportStanford HAI, annual
Links checked on 1 November 2026. External sources are listed for further reading; Delphilens is not affiliated with them.