Let’s talk about why every AI safety score you have ever been shown was produced by the party being scored.
Start with a person, because it always starts with a person.
Someone gets a claim denied. Medical, insurance, a loan, a refund, doesn’t matter which. The denial is short, the reason is vague, and it arrives fast. So they appeal. And the appeal comes back in minutes with a slightly different phrasing of the same decision.
That is not an appeal. That is the same machine having a second opinion about itself.
Now zoom out from that person to the industry building the tools. Because we are in the middle of something bigger, and it is happening in the least glamorous room in the house — the verification room. The place where somebody is supposed to check the work.
And in AI, that room is being furnished by the companies that live in it.
Follow the money and the conveniences for thirty seconds.
Model evaluations are now routinely scored with models. Labs publish safety cards and capability benchmarks assembled with heavy internal tooling, chosen internally, run internally, reported internally. Agent platforms ship “audit” features where the same model that took the action writes the review of the action. Enterprises are buying AI risk scores and safety ratings from the labs whose products carry the risk. And there is a genuinely new cottage industry of AI auditors who fly in, run inference on the client’s cloud, using the client’s model weights, and hand over a report with a logo on it.
None of that is fraud. That is the part people get wrong. Very few people in that chain are lying. Most are competent and sincere.
The problem is structural, not moral.
Here’s why this should bother you more than a scandal would.
We solved this problem once already, in other industries, and the solution was expensive and annoying and it worked. Auditors can’t be employed by the company they audit. Drug trials get blinded and run by someone who has no stake in the molecule working. Meat inspection doesn’t get outsourced to the abattoir’s own quality team, because everyone understood, a hundred years ago, that self-report is not a report.
It is not that we believed those people were dishonest. It’s that we accepted a plain fact about incentives: the person who benefits from the answer cannot be the person who certifies the answer.
Somehow AI has been granted the opposite arrangement, on the quiet, in about two years.
And it’s not only about lying. This is where most people arguing about AI governance miss the actual mechanism. Even a completely honest self-evaluating system fails, for three boring reasons.
One — the test set gets picked by the party being tested. Whatever the model is bad at becomes out of scope, not by malice, just by the natural gravity of what gets measured.
Two — the judge learns the taste of the room. If your evaluator is a model trained on the same distribution, tuned by the same objectives, and prompted by the same team, it will agree with the house style. It will grade generously for the kind of output the house produces. That’s not corruption. That’s convergence.
Three — failures are defined by the failures you already know about. The ones nobody has seen yet are, by definition, not in the test.
So the report comes back green, and the green is real, and the green is worthless.
Now the operator lesson, because this is where it gets practical and where most founders are currently exposed.
If you are buying AI, the question was never “is this model accurate.” The question is “who wrote the test, who ran it, and could the subject have changed the result if it wanted to.”
Ask those three questions about every dashboard, every vendor safety claim, every “97% accuracy on our benchmark” slide, and watch how fast the room goes quiet.
Then do the unglamorous work. Keep a test set of your own — real cases from your own operation, with the correct answers, held somewhere the vendor can’t see and can’t tune to. Log your own runs so that your evidence of what the machine did doesn’t live inside the machine. And when someone sells you a governance report, price the report by the independence of its author, not the thickness of the PDF.
If the only proof your product works is the dashboard you built to prove it works, you don’t have proof. You have marketing with a chart on it.
And if you build the thing — build the verification as if it will be attacked, because it will. The moment your system denies somebody something, you have created a person who needs to see the reasoning, and a person whose only recourse is to shout at a web form. That person is going to find the seams.
There’s a human consequence here that is bigger than tooling, and I want to say it plainly.
We are moving appeal, review, hiring, credit, claims, moderation, and eligibility into systems that only take a second pass through the same weights. That means the last human defence in the chain — the reviewer, the manager, the appeal — is being quietly converted into a rerun.
The whole thing mercy ever came through was a decision made by someone with no stake in saying no.
It’s worth naming what we’re losing before we finish automating it away.
A verification layer that depends on the cooperation of the thing being verified isn’t verification. It’s a permission slip. And a society that runs on permission slips has stopped testing its own machinery — it’s just reading the label the manufacturer supplied.
So here’s the line, and I’d hold it hard.
You cannot verify power with the instruments power supplied. Test sets you didn’t write, judges with nothing invested in the answer, and an appeal that goes to somebody whose name you could, in theory, write down. Those aren’t bureaucracy. Those are the load-bearing walls.
The machine grading its own homework isn’t a scandal you can expose once. It’s a design choice being made right now, by people who mostly mean well, inside rooms with no windows.
The human part isn’t going along with it.