A striking proportion of enterprise AI initiatives stall after the pilot. The common explanation is that the technology was not ready. That is rarely the real reason. The prototype usually worked. What was missing was everything that has to exist around it before a business can depend on it.
A demo proves possibility, not reliability
A demo is run by the person who built it, on inputs they chose, in front of an audience predisposed to be impressed. Production is run by people who did not build it, on inputs nobody anticipated, in front of customers who will notice a wrong answer. These are different problems, and succeeding at the first tells you remarkably little about the second.
The question that separates them is simple and uncomfortable: what is the accuracy, on a representative sample, measured against a known-correct answer? If there is no number, there is no pilot result — only an anecdote.
Build the evaluation harness first
On an AI engagement the first deliverable should not be a model. It should be a golden dataset: a few hundred real examples with correct answers, assembled with the people who do the work today. It is tedious, and it is the highest-leverage fortnight of the project. It is how we intend to start every AI build we take on.
Once it exists, every subsequent decision becomes measurable. Does a different model improve things? Does a cheaper one cost accuracy? Did last week’s prompt change quietly regress a category nobody was watching? Without the harness these are opinions. With it they are numbers, and arguments end quickly.
If you cannot state your system’s accuracy as a number against a known dataset, you do not have a pilot result. You have an anecdote.
Decide what happens when it is wrong
Every useful AI system is wrong sometimes. Designing as if that is an edge case is how pilots fail in production. The question to answer at design time is what the system does when it is not confident, and that answer has to be built rather than assumed.
- Confidence thresholds that route uncertain cases to a person, with the work pre-filled rather than started from scratch
- Schema validation so malformed output fails loudly and immediately instead of propagating downstream
- Grounding in retrieved sources with citations, so a reviewer can verify a claim in seconds
- Logging complete enough that a decision can be reconstructed months later for audit
Budget for operations, not just the build
An AI system is not finished at launch in the way a conventional application can be. Model providers deprecate versions. Your data distribution shifts. Prompts that worked degrade as usage patterns change. Someone has to own that, and if nobody is named, the system decays quietly until it is quietly switched off.
Plan for evaluation runs on a schedule, drift monitoring, cost observability per feature, and a named owner. This is ordinary operational engineering. It is also the part that separates the initiatives still running after eighteen months from the ones that produced a good slide deck.
Choose a narrower first use case than feels ambitious
The instinct is to start with the transformational use case. The better move is to start with a narrow, high-volume, low-consequence process where the data is already clean and the correct answer is unambiguous. It delivers value sooner, and it builds the organisational muscle — evaluation, governance, change management — that the ambitious use case will require later.
The organisations succeeding with AI are rarely the ones that started with the boldest idea. They are the ones that shipped something small, learned how to operate it, and then went further.
Written by the Voxalone team. If this raised a question about your own systems, get in touch — you will get a straight answer from an engineer, with no obligation attached.