AI Safety for Business: What to Require Before Deployment

I’d be interested. I’d also have a few questions before handing over the keys.

What happens when the email is confusing? Can the software change something it shouldn’t? If it makes a mess of an order on Friday afternoon, who helps straighten it out?

You can ask all of those questions without understanding how an AI model works. You understand your business. You know what a wrong delivery date can do to a production schedule, or how much fun it is to send a crew to a job where the materials haven’t arrived.

That knowledge belongs in the conversation before the software starts changing your records.

First, what exactly are we letting it do?

“We’re adding AI to purchasing” sounds promising, but I wouldn’t approve a project on that description alone.

Let’s use a hypothetical example throughout this article. You want software to read supplier order confirmations and suggest changes to delivery dates. A purchasing employee checks the suggestion against the original email and approves it. The software cannot change prices, substitute materials, or place orders.

Now we have a job we can discuss.

If a supplier writes that a shipment will arrive Friday, the software needs to find the correct order and propose the correct date. If the message mentions three orders, it needs to keep them straight. If the supplier writes “probably Friday, but we’ll confirm,” I’d want that uncertainty passed along to the employee.

Write down that job before adding more features. It is easy for a conversation about updating dates to become a conversation about letting AI resolve purchasing problems. Somewhere along the way, you may have given it permission to spend money.

I’m discussing office and business-record workflows here. If your proposed system controls machinery or affects physical safety, bring in people qualified to assess that operation. These purchasing tests won’t answer those questions.

“Show me” is a useful phrase

In the rogue-agent article, we looked at what can happen when an AI acts beyond its assignment. For your own project, ask the vendor to show you where the limits are enforced.

If the software is supposed to update delivery dates, have someone try to change a price using test records. If an employee can see only one project, try requesting information about another. Watch what the business system actually allows.

OWASP, a software-security organization, recommends restricting AI applications to the tools and permissions they need, enforcing authorization in the systems they use, and requiring human approval for high-impact actions. Those are things a vendor should be able to demonstrate. OWASP’s guidance on excessive agency

Then follow one permitted update all the way through. Show me the supplier’s message, the order it matched, the proposed change, and the employee’s approval. Finally, show me the changed record in the purchasing system.

If the AI says “Done,” but the order still shows the old date, we have something to investigate. A cheerful answer on the screen doesn’t get the materials to the job site.

Where does the information actually go?

“Your data is private” is a good beginning to an answer. I’d like to hear the rest of it.

In our voice-enabled inventory application, the audio is processed on the mobile device and stays there. The information extracted from the request is used to work with a database in the cloud. When someone requests a report in natural language, a model on the server processes that request.

So yes, the voice stays on the phone. That doesn’t mean the entire application runs on the phone, or that no business information travels anywhere.

Ask your vendor to walk through the same explanation for one of your supplier emails. Which parts go to the AI provider? Does the attachment go too? Where are copies kept, including records used for troubleshooting? Who can read them, when are they deleted, and can they be used to train a model?

Get answers for the product and account you would actually buy, including any other companies that process the information. A general statement on a website may not settle those details.

And ask why the software needs each kind of access. I’d be curious to hear why something that updates delivery dates needs to look at payroll.

Sort that out before putting confidential information into a trial.

Give it some of your less cooperative paperwork

A clean supplier email is a reasonable place to start a demonstration. Bring a few awkward examples as well: a revised confirmation, a missing order number, two different dates, or a message about several deliveries.

Use a separate test environment and agree beforehand on what the software should do. With an unclear message, “Please check this” may be the correct result.

NIST’s guidance supports testing under conditions similar to actual use and warns against drawing broad conclusions from a few narrow examples. In our purchasing example, that means including the emails employees have to puzzle over. NIST’s Generative AI Profile, Measure 2.3 and 2.5

Try an interruption too. The order updates, but the connection drops before the application receives confirmation. Someone clicks again. Does the software check what already happened?

Keep a record of the mistakes and their consequences. An awkward sentence and an update to the wrong order deserve different attention, even if both count as one error in a vendor’s accuracy figure.

Passing these tests won’t guarantee every future result. It gives you a more useful basis for deciding what to try next.

Put the person doing the checking in front of the screen

“A human will approve it” sounds straightforward. Let that human try it.

For our purchasing example, show the order number, supplier, item, old date, and proposed date together, with the original message easy to find. The employee should be able to correct or reject the proposal. If its details change after approval, it needs another review.

Then watch how long the checking takes. Does the employee have to open five windows to work out what happened? Are they reviewing something clear enough to judge, or trying to reconstruct the software’s reasoning?

If proposals arrive faster than anyone can check them, you have created a backlog. At some point, clicking “Approve” on everything may begin to look attractive. I’d want to discover that during the trial.

We had our own small lesson in keeping an application useful. Originally, we wanted our inventory software to speak its answers as well as accept voice input. It sounded like a nice feature. In practice, it slowed things down. A user could glance at “23 widgets in the Boise warehouse” faster than listen to the answer.

We dropped the spoken response. Sometimes improving an AI application means taking something out.

I would bring that same attitude to the review screen: make the employee’s job easier, and measure whether it actually is.

Try the stop button while there is nothing expensive to break

Before relying on a “kill switch,” ask to see what it stops.

Using test orders, pause the purchasing automation while some proposals are waiting for approval and an update is underway. Find out what finished, what is still pending, and whether anything keeps running in the background. Then restart it and check what happens to the unfinished work.

There is another part to this exercise. Stopping the software won’t put an incorrect delivery date back. And fixing the date won’t necessarily tell the planner who has already rearranged tomorrow’s work around it.

Work through that recovery too. Who identifies the affected orders? Who corrects them? Who tells the people using the information?

During the pilot, employees should still be able to use the existing purchasing process. They need somewhere to report problems and someone responsible for responding. Agree on the vendor’s part in that arrangement before you need help on that Friday afternoon.

Ask what happens when the vendor changes the software or model. The UK National Cyber Security Centre’s guidance calls for ongoing monitoring and testing updates because changes to models, data, or instructions can change behavior. NCSC’s guidance on secure operation and maintenance

Ask how significant changes are communicated and how you can check them before depending on them.

What would make the trial worth continuing?

For this purchasing project, I would begin with one team and one type of supplier message. Start with test data. Once the access restrictions and data-handling questions are settled, try a limited amount of live work, with an employee approving each change.

Agree on a review date before starting. Also decide what would stop the trial early. An update made without required approval, or information shown to someone who shouldn’t see it, would be a reason to pause the affected workflow and investigate.

At the review, start with whether the work is easier. Include time spent checking and fixing mistakes. Count messages the software missed, not just those it processed. Add the software charges and support time, and set usage or spending limits where available.

Be curious about claims such as “95% accurate.” What happens in the other 5%? An unclear message sent back for review and a wrong order quietly changed are very different outcomes.

You may decide to expand the trial. You may discover that it needs a simpler task, a better review screen, or more work before it saves anyone time. Stopping is also a reasonable result if the benefit doesn’t justify the cost and trouble.

And if updating delivery dates works well, enjoy that success before giving the software permission to place orders. That is another decision, with another set of consequences.

When I would say, “Not yet”

I would wait if the vendor couldn’t explain where our information goes, demonstrate the access restrictions, or show how to investigate and correct a mistake. I would also wait if nobody in the company had time to check the work or take responsibility for the process.

You can often keep working on the idea with made-up data or a smaller assignment while those gaps are closed. There is no need to expose live records to find out whether a supplier’s approach is promising.

Across this series, we have considered some very large questions about AI. A business owner still has to make a much more specific decision: can we trust this application with this part of our work?

I’d come back to the purchasing employee. Can they see what the software proposes, catch a mistake, and get help when something goes wrong? Can you establish that it saves enough work to be worth keeping?

Those answers would give me a reason to proceed. The impressive demonstration got us interested; now we have something on which to base a decision.

If you have a workflow in mind, discuss it with BusinessForward.AI. Tell us what your employees do today and which part you would like to make easier. That is a good place to start.

Previous: AI and Entry-Level Jobs: What the Evidence Means for Hiring and Training · Return to the AI risk series guide