Let’s talk about the person who used to be able to explain how the whole thing worked, and the day they found out they couldn’t anymore.
Picture a pump station on the edge of a small town. Not a dramatic place. A brick building, a concrete reservoir, a fence somebody keeps meaning to fix.
There’s a man there who has run it for twenty-two years. Ask him anything and he’ll walk you through it. Which pump is tired. Which valve sticks in cold weather. Why the pressure drops on the east side of town every second Thursday when the dairy does its washdown. He can’t show you a document for most of that. He just knows it, the way you know your own kitchen in the dark.
Then the upgrades arrive, one at a time, each with a business case.
A predictive maintenance system that listens to the pumps and tells you which bearing is going. A cloud platform that trends the flow and flags anomalies. An optimiser that decides pump scheduling so the town pays less for electricity. Each one works. Each one came with a manual, a dashboard, a support line and a monthly invoice.
And then one Tuesday at three in the morning the reservoir runs low, and nobody can tell the council why.
The foreman can explain his part. The vendor can explain their part. The integrator can explain the wiring. There are three documents, four dashboards, and not one person on earth who can hold the whole chain in their head and say: here is what happened, and here is why.
That’s the thing I want to talk about. Not a failure. A disappearance.
We have spent twenty years separating the act of running a system from the act of understanding it. And we did it without ever deciding to, because understanding was never on the invoice.
Think about how these systems get bought. Every component is evaluated on capability, price, reliability, support. Legibility. Can it do the job, what does it cost, who do I call when it breaks. All reasonable. All answerable.
Nobody puts a line item in the budget for “the account of the whole.” There’s no procurement category for it. It doesn’t have a SKU. So it doesn’t get bought, and after enough years of not being bought, it stops existing as a role.
The old engineer who understood the plant was, in a very real sense, an unpaid feature. He was redundant on paper right up until the morning he was the only person who could tell you anything true.
Now, this is not a story about technology being bad. The optimiser probably saved that town real money. The maintenance system probably caught a bearing. I’d take the upgrades.
The problem is what the upgrades did to the one thing every accountability system on earth depends on: a human who can reconstruct the reasoning.
Look at how professional responsibility actually works. A doctor signs a file. An engineer certifies a bridge. A director signs the financials. A founder tells a client, on a call, why the number moved. In every case the signature means the same thing: I can account for this. I can be asked a question I haven’t prepared for, and I can answer it, because I understand what happened.
Take that away and the signature doesn’t become a lie. It becomes a formality. Something a person does because the process requires a name, while the actual reasoning sits in a box they can’t open, owned by a company they can’t compel.
And here’s where it gets commercially interesting.
If only the vendor can explain the system, then the vendor is the expert witness in every dispute about its own product. The explanation arrives as a report from the party being questioned. The regulator gets a summary drafted by the defendant. The court gets a reconstruction produced by the people who built the thing that’s on trial.
That is not a conspiracy. It’s just what happens when comprehension is scarce and one party holds the supply. Opacity becomes an asset. Not because anybody planned it, but because the alternative is expensive and nobody is forcing them to give it away.
And the market doesn’t correct this, because the failure is invisible until the incident. The system works. Ninety-nine days out of a hundred it does exactly what it should, and on those days the question “who understands this” sounds like a philosophical indulgence from someone who doesn’t want to ship.
Comprehension only gets priced after the accident. And by then the person asking the question has a lawyer.
Now add AI, and this gets worse in a specific way that I don’t think we’ve absorbed.
A traditional black box is honest about being one. It gives you a number, or a switch, or a light. You know you don’t understand it, because there’s nothing there pretending to be an explanation.
A language model gives you a paragraph. Fluent, structured, confident, plausible. It sounds exactly like understanding, and that is the problem. Because a model can generate a causal story about behaviour nobody actually traced. It doesn’t need to have observed the mechanism. It needs to have read enough text that resembles an explanation of a mechanism like this one.
So you get the worst combination available: a system you don’t understand, and a system that will happily explain itself in a voice that makes you stop asking.
The old foreman knew what he didn’t know. That’s a skill, and it’s being trained out of us. We’re replacing the honest gap with a confident paragraph, and the paragraph costs nothing to produce, which means it gets produced constantly.
That’s the shift. Not that machines took the knowing. That they took the not-knowing, and filled it with something that reads better than the truth.
So what do you actually do about it, if you run something?
Four things. None of them clever.
One. Write the plain-language account yourself, before anything goes wrong. One page. No vendor language. If this output is wrong, here’s how we’d find out, and here’s what we’d do. If you cannot write that page, you don’t understand your own operation yet, and no amount of dashboards will fix that. The act of writing it is the act of finding out.
Two. Name one person who owns the whole chain. Not a committee, not a rota, not “the team.” One human being who can be asked a question nobody anticipated about the system end to end, and who is expected to go and find the answer. If that person doesn’t exist, you don’t have oversight. You have a signature policy.
Three. Test with a new question. Not “does it work.” Everyone tests that, and it always works, that’s why you bought it. Ask instead: why did it do that on Tuesday. Why this decision and not the other one. A system that can’t be interrogated on a specific case is a system whose judgment you’re renting.
Four. Keep the ability to run it manually, degraded, for a day. Not because you’ll need to. Because the ability to do it is the difference between operating a system and subscribing to one. If the only way to keep the town’s water running is for a vendor portal to be up, you don’t run a pump station. You have a relationship with a company that does.
And if you’re the person in the middle, the one who used to be able to explain it and now can’t: say that out loud. In writing. “I can account for this part. I cannot account for that part, and here’s who can, and here’s what I’ve asked them.” That document is worth more than a year of confident meetings. It’s the only thing that will save you, and it’s also the only thing that will actually fix the gap, because someone has to name it before anybody buys the missing piece.
There’s a temptation here to turn this into a case for slowing down, or hiring more engineers, or going back to the way it was. I don’t buy that. The complexity is going to keep coming, and most of it is genuinely better than what it replaced.
The thing to protect isn’t simplicity. It’s the human capacity to give an account. Not a complete account, because nobody has ever had one of those. A true one. This is what I did, this is what I don’t know, this is who I asked, this is what we’ll do if it goes wrong again.
That’s an old human act, older than engineering. It’s what a responsible person says when something breaks in their care. And it only works when the person saying it actually holds enough of the thing to mean it.
You don’t have to understand everything. Nobody does, and pretending otherwise is how you end up with a signature and no account. You do have to be able to find out, and to know the difference between finding out and being told something fluent.
That difference is going to be worth a lot in the next few years. More than the tools, probably.