Skip to content
Where not to

Where AI Does Not Belong: A Decision Checklist

Editorial desk·9 min read··Composite scenario, no client described
The decision to take

Five conditions under which the right answer is not to deploy, why each one is fatal rather than fixable, and how to check for them in an afternoon.

Executive summary

Most AI review meetings ask whether a system can be built and never whether it should be. Deciding not to deploy is a named risk response, not a failure of nerve. Five conditions make a project unsuitable regardless of model quality, and each one is visible during scoping if somebody is looking for it.

A proposal reaches a review meeting. The question in the room is whether the team can build it, and the answer is almost always yes. Nobody asks the other question, which is whether the organisation should operate the result for the next five years.

That second question has no natural owner. Engineering answers feasibility. Procurement answers price. The business sponsor answers value, usually optimistically, because the sponsor proposed it. Suitability sits between all three, and things that sit between functions do not get decided, they get assumed.

This matters because unsuitability is not a defect you fix later. A project can recover from a bad model, a bad vendor and a bad first release. It cannot recover from being the wrong shape for the process it was pointed at. That property is fixed at scoping, and it is visible at scoping to anyone whose job is to look.

Declining to deploy is a decision, not a failure

It is worth naming this plainly, because organisations treat a stopped project as an embarrassment and therefore avoid stopping. Formal risk practice does not agree. The NIST AI Risk Management Framework organises the work into four functions, Govern, Map, Measure and Manage, and its Map function exists to establish context and decide whether a system is appropriate at all, before anything is measured. Avoidance appears in Manage as one of the standard responses to an identified risk, alongside mitigation, transfer and acceptance.

In other words, the framework treats not building as an ordinary option with equal standing. Most review processes do not, which is why they produce so few refusals.

Where the case genuinely holds

To be fair to the other side, there is a shape that works, and it is worth stating so the checklist below reads as a filter rather than as a prejudice. Repetitive work at high volume, where a wrong answer is cheap to catch and correct, where the input arrives in a stable format, and where somebody already reviews the output. Under those conditions the economics are usually favourable and the failure modes are usually survivable.

Everything below is a way of establishing that you are not in that situation.

Five conditions that should stop a project

1. The accuracy you need is higher than the approach can reach

Every system has a rate at which it is wrong. That is not a defect to be engineered away, it is a property of the approach. The question is not whether errors occur but who absorbs them.

If a wrong answer is caught by the next step in the process, the error rate is an efficiency question. If it reaches a customer, a patient or a regulator, the error rate is a liability question, and the two are priced completely differently. Teams get into trouble by evaluating on the first framing and deploying into the second.

The practical test is uncomfortable and quick. Ask what happens to the hundredth wrong output, not the first. The first gets noticed and fixed. The hundredth is the actual operating condition, and if nobody can describe what it costs, the accuracy requirement has not been established and the project is not ready to be approved.

2. The process is not written down anywhere

Automating an undocumented process does not automate the process. It automates whatever the person building it assumed the process was.

Undocumented work is usually undocumented for a reason: it contains judgement, exceptions and local knowledge that never made it into a procedure. Those are precisely the parts a system will handle badly, and precisely the parts nobody will notice are missing until an exception arrives and is handled as though it were routine.

This condition is the most recoverable on the list, because the fix is to write the process down. But that fix is a project in itself, it takes weeks rather than days, and it should be sequenced before the AI work rather than discovered during it.

3. Nobody will own the system after launch

This is the condition most likely to be waved through, and the one with unbounded cost.

A model in production degrades without anyone touching it. Inputs drift, an upstream team renames a field, a vendor updates a version, and the outputs remain plausible while becoming wrong. There is no exception thrown and no alert raised, because from the system’s point of view nothing failed.

The expense is not the eventual repair. It is every decision made on bad output during the months before anyone noticed, and that quantity has no ceiling. The condition is binary: either a named person is accountable for the system in production, with time allocated for it, or the correct decision is not to deploy. We have written separately about which pilots to stop in the first month, and unowned systems sit at the top of that list.

4. The domain is legally high-risk and you are not resourced for it

Some applications carry obligations that have nothing to do with how good the system is. Under the EU AI Act, Annex III lists the domains treated as high-risk, including biometrics, critical infrastructure, education, employment, access to essential services, law enforcement, migration and the administration of justice. Systems falling into those categories attract requirements for risk management, data governance, technical documentation, record keeping, transparency and human oversight, and a provider that judges its Annex III system not to be high-risk must document that assessment rather than simply proceed.

None of that makes deployment impossible. It makes it a compliance programme with an engineering component, which is a different undertaking with a different budget and a different owner. The failure mode is starting it as an engineering project and discovering the rest at the point of launch, when the work is done and the obligations are not.

5. The data cannot go where the system needs it

Sometimes the blocker is simply that the information required for the system to be useful cannot be exposed to it, whether because of contractual terms with customers, sector rules, or a commitment already made in a privacy notice.

The tempting response is to build with a reduced dataset. Occasionally that works. More often it produces a system that is technically live and operationally pointless, because the removed data was the part that carried the signal. Establish what the system needs before designing around what it can have, and if those two sets do not overlap enough, the answer is no rather than a smaller yes.

How to run the check

None of these require a pilot to detect. Each one is answerable during scoping by a person willing to ask an unwelcome question, and all five together take an afternoon.

  • Accuracy. What does the hundredth wrong output cost, and who pays it. If nobody can answer, the requirement is undefined.
  • Documentation. Does a written description of the current process exist. If not, that is the first project, not this one.
  • Ownership. Name the person accountable in production and the hours allocated. A team name is not an answer.
  • Regulatory class. Does the use case appear in Annex III or an equivalent regime. If yes, budget the compliance programme before approving the build.
  • Data. List what the system needs, then what it is permitted. If the gap removes the signal, stop.

A project that fails one of these is worth pausing. A project that fails two is worth cancelling, because the failures compound: an undocumented process with no owner produces errors that nobody has the context to recognise.

The decision

The reason these questions go unasked is structural rather than intellectual. Everyone in the room has an incentive pointing towards yes, and the cost of a wrong yes lands eighteen months later on somebody who was not in the meeting. Fixing that requires a person whose standing does not depend on the project proceeding, which is a governance arrangement rather than a technical one. It also requires the organisation to accept that a refusal is a successful outcome of a review, not a failed one.

It is worth noting what this discipline saves beyond money. The systems described in the Google research on hidden technical debt in machine learning systems accumulate maintenance obligations through entanglement, undeclared consumers and shifting data dependencies. Every system you decline to build is a set of those obligations you never take on, and unlike the savings from a successful deployment, that benefit is permanent.

One concrete step for this week. Take the AI proposal closest to approval and run the five questions above against it in a single meeting, with someone present who is allowed to say no. Write the answers down. If two or more are blank, the honest output of that meeting is a pause rather than a plan.

Deciding who holds that authority is its own question, and it is usually the harder one, which we cover in our note on decision rights when technology and operations disagree. If you want that judgement applied to a specific proposal rather than discussed in the abstract, that is the work we do.

Frequently asked questions

Is deciding not to deploy AI a failure of the project?

No. Formal risk practice treats avoidance as one of the standard responses to an identified risk, alongside mitigation, transfer and acceptance. A review that never produces a refusal is not exercising judgement, it is processing paperwork.

Can a better model fix an accuracy problem?

Only if the requirement is close to what the approach can reach. If a wrong answer reaches a customer or a regulator, the constraint is liability rather than model quality, and improving the model does not change who absorbs the error. Establish the cost of a wrong output before evaluating any system.

What if the process we want to automate is not documented?

Write it down first, as a separate piece of work. Undocumented processes carry judgement and exceptions that never reached a procedure, and those are exactly the parts a system handles badly. This is the most recoverable condition on the list, but the fix takes weeks and belongs before the build, not during it.

How do we know whether our use case is legally high-risk?

Under the EU AI Act, Annex III lists the domains treated as high-risk, including biometrics, critical infrastructure, education, employment, essential services, law enforcement, migration and justice. If your use case falls there, the deployment is a compliance programme with an engineering component, and it needs the budget and owner that implies.

How many of the five conditions can a project fail and still proceed?

One is worth pausing for. Two is usually worth cancelling, because the conditions compound. An undocumented process combined with no named owner produces errors that nobody in the organisation has the context to recognise, which is the situation in which damage accumulates silently.

Sources