How to Prepare for an AI Pilot That Actually Ships

Here's how to do that sorting before you spend money.

Start with the bottleneck, not the technology

AI made producing certain kinds of work cheap: drafts, code, ad variants, reports, summaries, classifications, data extraction, and routine analysis.

That doesn't necessarily make the business faster.

In many workflows, the constraint simply moves downstream. Engineering teams see this when developers generate code faster than experienced people can review it. Marketing teams can produce more copy than anyone has time to approve. Operations teams can automate data entry only to discover that exceptions still pile up waiting for someone who understands what they mean.

But review isn't the only bottleneck worth attacking.

A good AI pilot starts by finding expensive friction in an existing workflow. That might be:

• a queue waiting for expert review;
• repetitive data entry between systems;
• employees searching through documents for answers;
• information trapped in emails, PDFs, notes, or voice recordings;
• routine decisions that follow recognizable patterns;
• customer or employee questions that repeatedly consume expert time; or
• a process that works only because one experienced employee knows how to handle all the exceptions.

The important part is to start with the constraint rather than the technology.

Don't begin with "Where can we use AI?"

Begin with "Where does work slow down, become expensive, or depend unnecessarily on scarce expertise?"

The people closest to that work usually know the answer better than the CIO does.

Find where the judgment lives

One of the most common ways to underestimate an AI project is to begin with: "Give us your data and your rules."

The data is usually messy. The rules are worse.

In an established business, important decisions rarely come entirely from a procedure manual. They also depend on the person who knows which customers always pay late, which supplier's "in stock" means three weeks, which project manager uses a field differently from everyone else, and which compliance flag is routine versus which one needs immediate attention.

Some of that knowledge can eventually be captured. Some can't.

So before automating a workflow, separate it into three parts:

What is repetitive?
These are the steps AI or conventional automation may be able to handle reliably.

What requires judgment?
These are the ambiguous cases, exceptions, or consequential decisions where an experienced person still matters.

What evidence does that person use?
Documents, transaction history, emails, customer records, policies, previous cases, or simply experience.

That third question is particularly important. If the expert cannot explain what information they use to make the decision, the AI team probably cannot build a reliable system around it yet.

The goal is not necessarily to automate the expert. Often the better system automates the routine work around the expert.

Instead of reviewing 300 items, the expert reviews 30 uncertain ones. Instead of searching five systems before making a decision, the relevant evidence is assembled for them. Instead of manually entering information from an email, they approve a structured record extracted from it.

Before your pilot starts, identify the people whose judgment makes the current process work. Make sure they have both time and reason to participate.

A pilot without access to the people who understand the exceptions is usually a demo.

Decide what kind of AI pilot you're actually running

Not every useful AI system is an agent, and not every pilot should be measured the same way.

Most business AI pilots fall into a few practical categories.

Generation: create drafts, code, reports, proposals, marketing material, or other content.

Extraction and structuring: turn emails, documents, images, notes, or speech into usable records.

Retrieval: find the right information across documents, databases, or internal systems.

Prediction or classification: prioritize, categorize, detect anomalies, estimate outcomes, or identify items requiring attention.

Workflow automation: take information from one system, apply rules or reasoning, and perform actions in another.

Decision support: assemble evidence, suggest an action, and leave the final judgment to a person.

A pilot may combine several of these, but you should know which problem you're solving.

That determines what success looks like.

A retrieval system should be measured on whether it finds the right information. An extraction system should be measured on accuracy and exception rates. A workflow agent should be measured on successful task completion and intervention rates. A generation system should be measured on whether its output survives review and is actually used.

Calling all of them "AI agents" hides useful differences.

"Faster" is not a result

Ask people how AI is helping and they'll often say it makes them more productive.

Maybe. But "productivity" is easy to claim and surprisingly difficult to turn into a business case.

A person can generate a document with AI and send it to another person who uses AI to summarize it. Both people saved time. The company may have gained nothing.

Time saved matters only if something useful happens with the time.

So don't stop at:

"This will save ten hours per week."

Ask what those ten hours change.

Perhaps the same team can process 30% more orders. Perhaps an engineer can handle more customer implementations. Perhaps invoices go out two days earlier. Perhaps a compliance team can examine every transaction instead of sampling them. Perhaps the company can grow without immediately adding another employee.

The business result usually connects to one or more of four things:

Revenue or capacity gained. More customers, transactions, projects, or work can be handled.

Cost avoided or reduced. Less outside labor, rework, processing cost, or additional hiring.

Risk reduced. Fewer errors, missed obligations, compliance failures, or costly exceptions.

Service improved. Faster response, shorter cycle times, fewer delays, or better availability.

Time saved can be an important measurement. It just shouldn't be the end of the calculation.

Before the pilot starts, write down the operational metric you expect to change and how that change connects to the business.

If you can't describe that connection, it will be difficult to defend the project when the budget gets tight.

Measure the AI and the workflow separately

A pilot can have an impressive model and still fail as a business system.

Suppose an extraction model is 96% accurate. That sounds good. But if the remaining 4% consists of the most consequential transactions, the system may still require someone to inspect everything.

Or suppose an AI assistant answers 90% of questions correctly, but employees don't trust it and continue calling the same expert. Technically, the model works. Operationally, nothing changed.

Measure both layers.

For the AI itself, that may include:

• accuracy;
• retrieval quality;
• task completion rate;
• false positives and false negatives;
• escalation rate;
• correction rate; and
• cost per transaction.

For the workflow, measure what happened around it:

• cycle time;
• queue size;
• expert review time;
• throughput;
• rework;
• errors;
• adoption; and
• the percentage of AI output that is actually accepted or used.

Establish the baseline before the pilot starts. Otherwise, you'll finish with a collection of anecdotes and no way to tell whether anything improved.

Recognize the vendor checklist

Most advice on AI pilots eventually includes three recommendations:

Secure budget authority beyond the pilot. Start with one problem in one business unit with one cooperative expert. Don't postpone integration until the end.

All sensible advice.

But read from the other side of the table, it's also exactly what a software vendor needs to expand an account.

Budget authority means somebody can sign. A contained business unit creates a manageable first deployment. Early integration makes the product more valuable—and makes switching away from it more expensive.

None of this is sinister. It's how enterprise software works.

It simply means your architecture and purchasing decisions should account for the vendor's incentives as well as your own.

One contradiction is especially worth noticing. Companies are frequently warned not to become dependent on a single AI platform because models, prices, and ecosystems will change. The same companies are then told to stop buying software and start "hiring agents."

Agents are software.

They are still built, hosted, priced, upgraded, and sometimes discontinued by somebody.

So decide explicitly which parts of the system you need to control.

Your evaluation cases are particularly valuable. They describe what "good" means for your business. The instructions, business rules, integration logic, and data access layer may also be worth keeping portable.

That doesn't mean every prompt must live in a text file you own or that every component must be vendor-neutral. Sometimes the fastest and cheapest solution is deliberately proprietary.

The question is whether you understand the dependency you're accepting.

Before committing to a platform, ask:

What happens if the underlying model changes?

Can we export our configuration and evaluation data?

Where does the system's learned behavior actually live?

Can another model or vendor access the same business context?

What would we have to rebuild if we left?

Vendor lock-in isn't automatically bad. Lock-in you didn't know you were creating is.

Test with exceptions, not demos

Happy-path demos are easy.

Real workflows aren't.

If you're testing an invoice-processing system, don't give it twenty clean invoices. Give it the supplier that puts the purchase order number in the wrong field, the invoice with two currencies, the credit memo, the duplicate, the handwritten note, and the customer who somehow manages to break the process every month.

If you're testing internal search, don't ask the ten questions whose answers appear verbatim in the employee handbook. Ask the ambiguous questions employees actually escalate.

If you're testing an agent that takes actions, deliberately create situations where it should refuse, ask for clarification, or escalate.

Build this test set before the pilot if possible.

It becomes your evaluation set: a collection of representative normal cases, difficult cases, known exceptions, and high-consequence failures.

Then rerun it whenever the model, prompts, business rules, or underlying system changes.

That gives you something much more useful than a polished demonstration: a repeatable definition of acceptable behavior.

Check the statistics you're handed

Two claims illustrate why this matters.

The first is the widely repeated statement that "95% of AI pilots fail."

It traces to 2025 research associated with MIT's Project NANDA. The underlying research described a much more specific set of findings about organizations experimenting with generative AI, their progression from evaluation to pilots and deployment, and the difficulty many had demonstrating meaningful business impact.

That's worth paying attention to.

But turning it into a universal law that exactly 95% of AI pilots fail removes most of the context needed to interpret the research. The useful lesson isn't the precision of the headline number. It's that moving from an impressive AI demonstration to measurable operational value is difficult.

The second is the radiology story.

It's sometimes presented as proof that AI didn't replace radiologists but instead made them more productive, causing hospitals to hire more of them.

Radiologist demand has indeed remained strong, but AI is only one part of a much larger story. Imaging volumes have been increasing while the supply of radiologists faces demographic and training constraints. American College of Radiology workforce update AI is being introduced into an environment that already has substantial demand for scarce expert capacity.

The useful lesson is narrower.

Radiology illustrates an important pattern for AI deployment: machines can help prioritize work, surface information, and assist with parts of a complex process while qualified professionals remain responsible for consequential decisions.

That's a much less exciting story than "AI transformed radiology."

It's also much more useful when designing a pilot.

Whenever a statistic or anecdote is being used to justify an AI investment, look up the original source and ask whether it actually supports the conclusion being sold.

It may be the cheapest ten minutes of due diligence in the project.

The preparation checklist

Before the first line of a pilot gets written:

1. Name the bottleneck. Identify the expensive delay, repetitive work, information problem, or dependency on scarce expertise you're trying to change.

2. Map the workflow. Separate routine work, judgment, exceptions, and the information people use to make decisions.

3. Name the expert. Make sure the people who understand the exceptions are available and have a reason to participate.

4. Define the type of AI system. Generation, extraction, retrieval, classification, workflow automation, decision support—or a combination.

5. Establish the baseline. Measure the current cycle time, cost, error rate, throughput, queue, or expert effort before introducing AI.

6. Write the business outcome in one sentence. Connect the operational improvement to revenue, capacity, cost, risk, or service.

7. Build an evaluation set. Include normal cases, difficult cases, known exceptions, and failures that would actually matter.

8. Decide what you need to own. Especially your evaluation data, business rules, integrations, and other components that would be expensive to reconstruct.

9. Set a fixed measurement period. Sixty days is often enough to discover whether a narrow workflow is improving. Measure both AI performance and the business process around it.

10. Keep the first deployment narrow. One workflow, one accountable owner, clear users, and predefined success criteria. Expand after you know what worked and why.

A successful AI pilot doesn't need to prove that AI is transformative.

It needs to prove that one specific business process works measurably better with AI in it.

If it does, you'll know what to expand and why.

If it doesn't, a well-designed pilot should still leave you with something valuable: a clearer map of the workflow, a test set of real cases, baseline measurements, and a much better understanding of where your organization's judgment actually lives.

That's considerably more useful than another impressive demo.