The Good CFO
Insights · Humanism in the Age of AI

Humans as verifiers, not exception handlers

Almost every finance automation gets built the same way: the machine does the volume, and a person handles whatever it kicks out. That design guarantees the person never sees the errors that matter.

By Matthew Everitt · Founder & CEO · 7 min read

An exception queue only contains the items the system already knew it could not handle. The expensive errors are the ones it processed with confidence, and by construction those never enter the queue. So a human whose job is the queue is not checking the automation. They are tidying up after the part of it that works.

This is the least glamorous and most consequential design decision in a finance automation project, and it usually gets made by default. Somebody builds the rules, the rules cover most cases, the rest goes to a person, and everyone calls that the human in the loop. The loop is real. It just does not close anywhere near the risk.

01 · The default

The queue at the end of the pipe

Here is the shape of it, whatever the tooling. Transactions arrive. A rule set, a model, or some combination of the two assigns each one a category, a client, a project, an approver. Most get handled. A residue does not: the vendor nobody has seen before, the description that reads like three things at once, the amount that trips a threshold. That residue lands in a queue, and somebody works the queue.

It is an appealing design because it is measurable. Queue depth, time to clear, items per hour. You can put it on a dashboard by Friday. Most of the agency finance automation we are asked to review is built exactly this way, and the throughput numbers are usually excellent.

The trouble is what the design assumes. It assumes that the system's own confidence is a reliable guide to where the errors are. Sort into handled and unhandled, watch the unhandled pile, and the handled pile takes care of itself. Deming spent thirty years arguing the general version of this point: you cannot inspect quality into a product, and inspecting only what has already been rejected tells you nothing about what was accepted. Nothing about that changed when the sorting got done by software.

02 · The problem

The errors you care about are confident

Think about how a categorization rule actually goes wrong at an agency. It is almost never a case the rule found confusing. It is a case the rule found easy and got wrong for a reason nobody encoded.

A freelance editor invoices through an LLC whose name looks like a software vendor, so eighteen months of production cost sits in subscriptions. A pass-through media buy gets coded to revenue because the deposit looks like every other client deposit, and the gross-to-net picture quietly stops being true. A retainer client's out-of-scope work gets coded to the retainer project because that is where their work always goes, so the overrun never appears and the renewal gets priced off a fiction. Every one of those is processed at high confidence, on the first pass, without hesitation. None of them ever reaches the queue.

Say a categorizer processes four hundred transactions a month, flags twelve for a person, and handles the other three hundred and eighty-eight itself at ninety-six percent accuracy. Sixteen of those are wrong, and none of them reach the queue. Your exception handler is looking at the twelve the system flagged, of which maybe two were genuinely hard. And a rule error does not stay one error, which is the part that gets missed. A person miscoding by hand makes a mistake. A rule miscoding does the same thing every month until somebody notices, which is why automation multiplies rather than repairs.

So the honest description of a well-run exception queue is that it is a measure of the system's self-awareness. Useful. It is not a control.

03 · The difference

Four things that separate a verifier from a handler

Often the same person, at the same desk, on the same salary. The distinction is in how the seat is designed, not in who is sitting in it.

A verifier looks at the output the system was sure about

The handler sees only what was flagged. The verifier pulls a sample of what was not, at random rather than by any rule that correlates with the automation's own logic, and checks it against the standard. That single change is the difference between measuring the error rate and hoping there isn't one.

A verifier owns the standard, not the throughput

Ask what the person is measured on. If the answer is queue depth, you have a handler, and you have also told them that a fast wrong answer beats a slow right one. A verifier is measured on whether the output is correct and on whether the error rate is falling, which are the two things the owner actually needs to be true before reading the report.

A verifier can change the rule

This is the one that gets skipped, usually for good reasons: the rules live in a system somebody else administers, and changing them feels like engineering. But a person who can identify a systemic error and cannot fix it will stop identifying it. Not out of spite. Because reporting a problem you are not allowed to solve is just adding work to your own day, so the reporting quietly stops and you lose the signal.

A verifier's learning goes back into the system

When a handler resolves an item, the knowledge that resolved it leaves with them at the end of the shift. When a verifier resolves the same item, the resolution becomes a rule, an exception list, or a line in the close checklist, and the class of error stops recurring. Same fifteen minutes of work. One of them compounds.

04 · The design

How to build the seat

None of this requires better technology, which is fortunate, because most agencies already have more automation than they have verification. It requires four decisions made on purpose.

Write down what correct means, before you automate anything. Not the process. The output. What a correctly coded media buy looks like, what belongs in project cost and what does not, when a timesheet entry is billable. If that document does not exist, verification is a matter of opinion and the loudest opinion wins. This is the part of controller work that survives every wave of tooling, and it is the part agencies most often skip because the bookkeeping appeared to be working.

Sample the confident output on a fixed schedule. Twenty-five transactions a month, drawn at random, checked against the standard, error rate written down. It takes under an hour and it is the only thing in this article that will tell you whether your automation is any good. Sample size matters far less than doing it every month, because the value is in the trend and in seeing the same category fail twice.

Give the verifier authority over the rule, with a record. They change it, they note what changed and why, the change gets reviewed at month end. Authority without a record is how you get undocumented drift. A record without authority is how you get a very tidy list of problems nobody fixed.

Watch whether the queue is shrinking. A healthy exception queue gets smaller over time, because each exception either becomes a rule or reveals one that was wrong. A queue that holds steady month after month is not a workload. It is a design that has stopped learning, and the person working it has been assigned to absorb the difference indefinitely.

05 · What it does to the org

The seat this creates is senior

The uncomfortable implication is that the exception-handler role should not exist as a job, and in most agencies it currently is one, held by somebody capable who has been given a queue instead of a scope. That is a bad outcome for the agency and a worse one for the person. It is all residue, no ownership, and no route to being able to fix the thing generating the work.

The verifier seat is different in kind. It needs someone who can hold a standard, judge whether a number is credible before it goes in front of an owner, and reason about why a rule failed rather than just that it did. At a digital marketing agency running pass-through media, that judgment is the difference between a gross revenue figure you can bank a hiring decision on and one you cannot. It is not a data entry job that got automated. It is the judgment layer that was always underneath the data entry, and it was easier to see back when a person touched every transaction.

Which is why the honest answer to "will automation reduce my finance headcount" is: it will change what the headcount is for. The processing hours compress. The verification and standard-setting hours do not, and they get more valuable as volume rises, because a bad rule now runs at scale. An agency that cuts the seat because the transactions stopped needing hands has automated away its only remaining check. What it should be doing is moving that seat up, and putting the recovered hours where the CFO layer can use them: pricing, capacity, the client-level profitability nobody has had time to look at.

06 · In practice

Where to start

When a finance function already has automation running, the exception queue is the wrong place to start. The better first pass is a sample of the confident output against whatever written standard exists, and where no written standard exists, that is the finding that matters most. It is a dull exercise and it is where the real errors are, because the queue has usually been worked diligently for months by someone who was never given anything else to work with.

Then build the loop: a standard, a monthly sample, a named person with the authority to change the rule, and the error rate on the same page as the throughput. That is Plan-Do-Study-Act applied to the part of the process everybody assumed was finished, and it is the study step that nearly always turns out to be missing. Most automations we inherit have a plan and a do, and a dashboard proving the do is fast.

The reason to care goes past accuracy. A person who verifies is a person whose judgment the system depends on and improves from, which is a role worth holding. A person who handles exceptions is a queue with a name, and that fact eventually shows up in the data, because the process shapes the behavior and not the other way round. If the aim is a system where the people in it are the point, then how you place the human relative to the machine is not an HR question. It is the design.

FAQ

Common questions

What does human-in-the-loop actually mean in finance automation?

In most implementations it means the system processes what it is confident about and routes the rest to a person, who resolves the queue. That is a human in the loop in the narrow sense, but the person is only ever seeing the items the system already knew it could not handle. A stronger version gives the human three things instead: a sample of the output the system was confident about, the authority to change the rule that produced it, and a written standard for what correct looks like.

What is the difference between an exception handler and a verifier?

An exception handler works a queue of items the system rejected. Their measure is how fast the queue clears, they cannot change the process that fills it, and what they learn resolving each item is discarded. A verifier owns the question of whether the output is right, samples the confident output as well as the flagged output, holds the standard, and has the authority and the obligation to change the rule when the standard is missed. Same person, often the same salary, completely different job.

If the AI does not flag an error, how would anyone catch it?

By sampling. Pull a fixed number of confidently processed items each period, chosen at random rather than by a rule that correlates with the automation logic, and check them against the standard. You are not trying to catch every error, which is impossible and was never possible under manual processing either. You are trying to measure the error rate and see its pattern, so that the rule can be corrected. A small honest sample beats a full review nobody has time to do.

Does automating the bookkeeping eliminate the controller role?

It changes what the role is for. The transaction-processing part of the job compresses, sometimes sharply. The part that decides what correct means, owns the close calendar, judges whether a number is credible before it goes to the owner, and changes the system when it is not, does not compress at all. In our experience it becomes more valuable, because the volume flowing through the rules is higher and a bad rule now scales.

The bottom line

Every automation has a human in it somewhere. The only question is whether that human is positioned where the errors are or where the system happened to leave the leftovers. Handling exceptions puts them at the end of the pipe, working the residue, measured on speed, with no authority to change what fills their day. Verifying puts them above it, holding the standard, sampling what the machine was confident about, and closing the loop the automation cannot close on itself.

It is the same person and roughly the same hours. One of those designs gets better every month. The other one gets faster.

Let's Talk

Has anyone checked what your automation got right?

Twenty-five transactions, drawn at random, against a written standard. An hour of that tells you more about your books than a month of the exception queue.

Schedule a Conversation