Let’s talk about who gets to decide whether an algorithm is dangerous.
Because that decision — that single judgment — happens before every rule, every appeal, every human in the loop that the new laws are built on. And right now, in most of the world, it’s being made by the wrong person.
Picture a credit committee in a mid-sized bank. Not the board. The quiet room three floors down where the score gets set. They’re rolling out a new engine that decides, from your spending, your sleep, your phone use, who gets the loan and at what rate.
Nobody in that room is evil. They’re just under a deadline and a budget. So when the compliance form arrives and asks: is this a high-risk system, under the new rules?
The person filling it in has already been told, politely, what the right answer is.
Because here’s the thing the headlines about the AI Act keep missing. The regime is genuinely big — the high-risk obligations started enforcing in August, the public registry is real, deployers now carry the liability. That is a real step forward.
But every one of those protections only switches on after a system has been classified as high-risk. And that classification — in a large part of the architecture — starts with the company itself. Self-assessment. You look at your own engine, you decide how dangerous it is, and you file that.
That is not a loophole someone hacked. It’s structural.
So think about the incentives. A bank that admits its scoring engine is high-risk now has to run impact assessments, document the data, log the decisions, stand up human oversight that can actually say no, and take liability if it discriminates.
A bank that decides its engine is low-risk — because, the argument goes, a human reviews the output, so the machine is really just a suggestion — does almost none of that.
The commercial pressure is not subtle. It’s cheaper to believe your machine is a helper than to treat it as a decision-maker. And the company gets to be the one who believes it.
Now follow the human consequence. Not the company — the person on the other side of that engine. They get the rejection, the higher rate, the flagged file. They may never even know a machine made the call.
But even if they did, under a self-classification system, they have no way to contest the first and most important decision of all: whether the machine that judged them is even subject to the rules.
You can appeal a decision. You cannot appeal a classification you were never shown. And the thing being classified — the engine, its training, its version, its actual behavior — is exactly what the company calls a trade secret.
So the loop closes. The party with the strongest financial reason to call the machine harmless is the only one who gets to look under the hood and decide.
That’s the hole. And it’s the hole that matters most, because everything else — the registries, the audits, the human override — is downstream of this one label.
Get the label wrong, and the rest of the law is a museum. Lovely to visit. Nothing runs.
The fix is not more paperwork. The fix is moving the classification away from the self-interested party — some form of independent, evidence-based determination, with the burden of proof on the deployer to show why a consequential system is low-risk, and a real right for the affected person to see and challenge the label that governs their treatment.
A system that quietly decides who gets credit, who gets hired, who gets flagged — that should have to prove it’s harmless. Not merely assert it.
Because sovereignty is not decided at the appeal. It’s decided at the classification.
And if we let the machine grade its own homework — with the human who depends on the grade never allowed to see the paper — then all the grand laws in the world are just furniture in a house where the alarm is always set to “fine.”
The question isn’t whether AI will be regulated. The question is whether the most important regulatory decision gets made by the one party that can’t be trusted to make it.
We’re still human. So let’s start insisting on the right to know how dangerous the thing is — before it decides how dangerous we are.
You’ve got this.