← Journal

Why every automation I build has a human in the loop

“Can it just do it automatically?” is the first question most owners ask about AI. The honest answer is: some of it, once you have seen it work. Here is how I decide how much.

The problem with “fully automatic”

Modern AI models are very good at reading messy documents, emails and forms and turning them into tidy data. They are also, occasionally, confidently wrong. A misread supplier name or a date in the wrong format is a minor nuisance on one invoice. Repeated quietly across a few hundred, it becomes a reconciliation headache at year end.

So the useful question is not “can AI do this?” but “which parts can it do on its own, and how will we know when it is unsure?”

What a confidence threshold is

Each time the automation reads a document, it records not just its answer but how sure it is about each part, as a score between 0 and 1. That score comes from a mix of checks: did the supplier match one already in your accounts, do the line items add up to the total, is the date in a sensible range, did two independent reads agree?

The threshold is the line we draw. Above it, the automation prepares a draft and moves on. Below it, the item goes into a review queue for a person, with the uncertain fields highlighted.

# how the threshold trades effort against risk (illustrative)

threshold   to a person   handled alone   risk of a slip
0.99        most          few             very low
0.90        some          most            low
0.70        few           nearly all      noticeable

Start strict, then relax with evidence

Every automation I build starts with the threshold set high, and often with a period where everything is checked by a person regardless of score. During that time we record how often a person changed the automation's answer, and on which kinds of document.

After a few weeks you have real evidence: “invoices from regular suppliers above 0.9 were never corrected; new suppliers were corrected often”. Then we relax the threshold for the safe cases only. That decision is yours, made on your data, not a setting I pick on day one.

Where I keep a person in charge permanently

• Anything that moves money: payments, refunds, credit notes.
• Anything a customer reads, unless it is a simple, pre-approved acknowledgement.
• Anything with a legal or regulatory consequence: contracts, HR matters, health or financial advice.

In these cases the automation still saves most of the time. It gathers the information, drafts the reply or the entry, and lays it out for a quick check. Approving a good draft takes seconds; typing it from scratch takes minutes.

What it costs you to do it this way

A little more build time, mainly for the review screen and the log, and a few weeks of supervised running before you see the full saving. In exchange, you can explain to your accountant, your staff or a customer exactly what the automation did and why. For a small business, that trust is worth more than squeezing out the last few per cent of automation.

The short version

• Every item gets a confidence score; a threshold decides what a person sees.
• Start strict, measure corrections, relax only where the evidence supports it.
• Money, customers and legal matters always get a human check.