It's Not How Often It's Right. It's What It Costs to Find Out.

We know how to fix broken software, and we know how to supervise fallible people. AI takes the first playbook away and quietly breaks the second, because every control we’ve built for human error is a bet on where mistakes cluster.

Machine failure is a gift.

That sounds wrong until you look at what a bug actually hands you. It’s loud. It’s reproducible. Same input, same broken output, every time. You get a stack trace, a root cause, a patch, and a regression test that guarantees this specific failure never happens again. The bug is reproducible, explainable, fixable, and the fix is permanent.

That’s why we hand machines work we’d never hand a person. Not because machines are smart. Because they’re boring. A calculator is trusted with arithmetic that would need a second set of eyes if a person did it, and it never occurs to us to check.

AI doesn’t fail that way. Same prompt, different answer. Nothing to reproduce, no root cause, no test that proves it’s fixed. So the debugging playbook is gone, and the natural move is to reach for the other one — the one we use on work done by people. Sampling QA, four-eyes review, reconciliations, sign-offs, escalation paths. Accounting has centuries of practice at it.

That’s the move I’m here to scrutinize, and here’s the part that took me a while to see. Every one of those controls is a bet on where mistakes live.

You don’t check everything. You can’t. You check where mistakes cluster — the fortieth page, the late-afternoon hours, the tedious reconciliation at the end, the first quarter someone works a new book. You skim the parts that rarely go wrong. That isn’t laziness. It’s the whole economics of review. Sampling works because errors are patterned.

You lean on a second signal too. People telegraph doubt: they hedge, they ask, they flag the footnote they weren’t sure about. Imperfectly, but enough that confidence carries information.

AI breaks both, and the second one is worse than it sounds.

Ethan Mollick’s jagged frontier covers the first: these systems are excellent at some hard things and fail at some trivial ones, and the boundary has nothing to do with what people find difficult. Stanford’s 2026 AI Index has a version I can’t get out of my head. A model took gold at the International Mathematical Olympiad. A highly capable model on the market reads an analog clock correctly 50.1 percent of the time — a coin flip, against 90 percent for people with no particular training. And the field experiment that coined the phrase, 758 consultants at BCG, found the ones using AI did worse on tasks just past the boundary, because a confident wrong answer is persuasive enough to take.

And you don’t correct for it on your own. A preregistered experiment put 577 people in front of an AI tool, salted its output with errors — sometimes on tasks that were easy, sometimes on ones that were hard — and measured what they’d pay to keep using it. The researchers expected the easy-task errors to spook people more, because those are the ones that should look wrong. They didn’t. People discounted the tool for making mistakes and not for making the wrong kind of mistakes. The signal that should have set off the alarm didn’t register as a signal at all.

It’s wrong the way a person is wrong. It just isn’t wrong where a person is wrong. And where is the only thing supervision has ever been built on.

Sampling against errors that don’t cluster isn’t less effective. It doesn’t work. Check ten percent, catch ten percent, ship the rest.

So check all of it, then.

Which lands you on a five-page paper from 1983 that keeps getting rediscovered. Lisanne Bainbridge’s Ironies of Automation argued that the better the automated system, the harder the human’s job gets — not easier. Vigilance decays fast. The system’s logic is opaque to whoever is watching it. And the operator ends up accountable for catching failures the machine can’t flag and they have no reliable way to spot. She was writing about industrial control rooms. It describes a person reviewing machine-generated work with uncomfortable precision.

What that looks like in practice is two failures with very different shapes.

Take a support desk. The old one had a script and a decision tree, and asking it something off the tree failed the way software fails — visibly, immediately. I’m sorry, I didn’t understand that. Annoying, unmistakable, and it routed you to a human. The failure announced itself, and the business learned about it the same second the customer did.

Replace it with an AI assistant that can hold a conversation and something better happens almost every time. It also, occasionally, explains a refund policy the company doesn’t have. Fluently. In brand voice. Without hedging, because it isn’t uncertain — it’s wrong, which is a different thing. The customer believes it, acts on it, and the company finds out when there’s a dispute. The old failure cost you a transfer to a rep. The new one is a commitment made on your behalf that you didn’t know you’d made.

Now the version I live with: A fund publishes a list of every loan it holds, updated quarterly, a few hundred rows long, and somebody has to get that list into a database. The old parser meets a new document layout and breaks. It breaks on every document in that layout, immediately and completely. Operations opens the file and sees zero rows. Someone writes a ticket, someone writes a fix, someone writes a test, and that failure is gone for good. Total damage: a bad afternoon.

The new extractor reads the same document with no code change at all, which is the pitch, and the pitch is real. It also misses a footnote on one loan — the one saying the borrower stopped paying interest in cash and started adding it to the balance instead. So it books income the fund isn’t receiving. No error. No flag. No hedge. The same confident tone it used for the four hundred rows it got right. That number rolls into the yield calculation, into the quarterly numbers, into a client report.

Which failure would you rather have in production — the one that stops the line, or the one that ships with 99.9% accuracy but a highly impactful error?

Notice what happened to the team in both cases. Their job used to be exception handling: the system told them what it couldn’t do, and they worked the queue. Now it’s verification — deciding, one item at a time, whether a confident answer is true. That’s a harder job needing more expertise, not less. It’s also the job the business case said you were eliminating.

But what about the one place that’s unequivocally been an AI success story?

Coding agents. They’re the clearest win anyone has, and they should be a problem for everything I’ve just written. So why do they work?

Because code is the rare output that checks itself. The generator is probabilistic, but the artifact is deterministic — it compiles or it doesn’t, the tests pass or they don’t, and the tests keep passing forever at no marginal cost. You’re not sampling the process. You’re running the product against something that never gets tired at 4:45.

That’s not a counterexample. It’s the rule stated the other way round: AI pays wherever checking is cheaper than making. Code sits at the far end of that. Extraction sits at the other, where verifying the footnote means reading the footnote, which is the work. So the cost never falls.

Which is where the business case comes apart.

Automation ROI has always been labor cost minus machine cost, with a third term that never made the spreadsheet because it rounded to zero: verification. You don’t staff a team to spot-check the ATM. Determinism wasn’t a feature of the product, it was the product.

Which is why I’d argue machines never really replaced people. They replaced procedures. The calculator, the spreadsheet, the ATM — each took work where certainty was the entire value and did it the same way every time. The machine wasn’t a better human. It was a different kind of thing, and the substitution worked precisely because it was nothing like us. Automation substitutes for tasks, not jobs; the people who study this for a living call the opposite assumption the substitution myth.

AI is the first machine sold as a substitute for judgment, and it’s built out of the exact error profile that made judgment need supervision in the first place. So verification cost never reaches zero, and it doesn’t stay flat either — it scales with volume. Every thousand extra outputs is a thousand more things a person has to decide about.

None of which is an argument against using it. It’s an argument about what you’re signing up for. You get a transfer, not a replacement. The cost moves from doing to checking, and checking is the part that demands the expertise you were hoping to remove. I wrote in June that the artifact was never the job, and last month that the cheap number on the chart is cheap because a person accepted the work. Same seam, third angle.

So, three things. Stop writing bug tickets for a system whose failures don’t recur, and start writing controls — sampling with teeth, defined tolerances, an escalation path, someone’s name on it. Put verification on the ledger permanently, as a line that grows with volume, staffed by people who know the work well enough to catch a plausible wrong answer. And, perhaps most importantly, push determinism to the edges wherever you can find it: make the totals foot, reconcile against a second source, flag anything outside a plausible range. Every rule you enforce in code is one fewer judgment call you’re asking a tired person to make at the end of a long day.

The question worth asking about any of this was never how often it’s right. It’s what it costs you to find out.

Where that’s cheap, push forward. Accelerate. Where it isn’t, you’re not buying capacity. Be honest and account for the workload shift from the maker to the checker.