Let’s talk about the instruction the United States government handed AI companies this week — and the part of it almost nobody read.
On the eighth of September, the NSA, CISA and the FBI published a joint advisory, AA26-251A. Six Chinese AI companies, they said, have been stripping capability out of American models at industrial scale since late 2024. DeepSeek, Moonshot, Alibaba, MiniMax, StepFun, Z.AI. Billions of tokens, millions of requests, tens of thousands of fraudulent accounts. That’s the version that ran everywhere.
Here’s the version that didn’t.
In the mitigation section — the part written for the companies doing the defending — the agencies made one specific recommendation. Don’t just block the accounts you suspect. Quietly serve them a worse answer. Route them to a less capable model. Reduce the reasoning depth. Vary the quality of the output. And do all of it without telling the user anything.
A government told private companies to lie by omission to their own customers.
And you could almost file that under “well, the targets are bad actors.” Except the invisibility isn’t a side effect of the idea. It’s the entire design. A blocked account tries again from a new one. A subtly poisoned account burns an entire training run and never knows why. That is the efficiency they were going for.
So let’s look at who else is standing in that room.
Picture a developer in Johannesburg. Small insurer, maybe ninety people, claims handled by a form and a human who is very tired. He wires a frontier model into the claims intake — reads the documents, flags the obvious stuff, drafts the summary. It works. He tells his boss it works. Eighteen months later the summaries start coming back a little vaguer. Slightly less careful. He blames his prompt. He blames the data he fed it. He rewrites the whole pipeline on a Sunday.
There’s no error code. There’s no incident line on the provider’s status page. There’s nothing to open a ticket about.
Because somewhere in a risk model, his usage pattern looked like a swarm.
That’s the part nobody’s reporting, and it’s the part that matters to you, whether you’re in Johannesburg or anywhere else. The advisory’s recommendation is aimed at suspected distillers. Detection systems don’t produce suspects. They produce probabilities, and probabilities produce false positives. Every false positive here is an ordinary paying customer being quietly served less intelligence than they bought, with no notification and no recourse.
Now. I want to be fair about the intent. The threat is real. Anthropic has already told us that three Chinese labs generated more than sixteen million exchanges through roughly twenty-four thousand fake accounts. The people doing this are not confused students. The agencies acknowledged distillation is legitimate research; what they described is that same technique aimed outward, at scale, as a core development strategy, with the evidence held privately.
My problem isn’t with the fight. It’s with the weapon chosen and the direction it points.
Because think about what we spent the last two years building. The EU’s AI Act made chatbots legally obliged to say they’re machines. Deepfakes have to be labelled. Public-interest AI text has to carry a marker. An entire regulatory layer exists on one assumption: the machine must disclose what it is.
That assumption just got flipped. The doctrine is now that the machine may not have to tell you which machine it is.
Sit with how much that breaks.
Because if the model behind the answer can be swapped without notice, then the model’s identity is not a fact you can rely on. It’s a claim the vendor makes and can revoke at any time, silently. Which means a model card is marketing. A benchmark score is a statement about a version you may not be talking to. And a contract clause that promises “frontier-class capability” — that’s not a capability anymore. It’s a mood.
You cannot audit what you cannot identify. You cannot benchmark it, you cannot reproduce it, and you sure as hell can’t take a screenshot of it and expect anyone to believe you.
And one more thing buried in the same document. Alongside the silent downgrade, the agencies recommended that providers strengthen user identity verification, tighten controls on enterprise subscriptions, and flag accounts whose usage-to-subscription ratio looks wrong.
Read that next to the other story from Thursday.
On the tenth, IDScan — a Louisiana identity-verification vendor that most people have never heard of — confirmed that hackers had pulled driver’s licences out of its cloud. More than a hundred and fifty million of them, according to the dark web listing that Krebs traced back to the company. Full names, licence numbers, passport numbers, photographs. Exfiltrated continuously for over a year.
So the recommendation is: verify users harder, and the verification industry is confirming it lost the records of every person who ever handed over an ID at a rental counter or a dispensary.
That’s where the two stories meet. The defence against a machine that won’t tell you what it is, is more identity data, held by a layer that is provably losable. And the agency on the receiving end of a silent downgrade — the person, the developer, the small business — is the last node in the chain, told nothing, twice.
Here’s the operator lesson, and it’s blunt: you now have to contract for the model, not just the outcome.
If you’re building on top of an API, log the model identifier on every single call. Not the product name — the exact version string, with a timestamp. When it changes, you want a record. When a customer complains that quality dropped in March, you want to be able to show whether it changed in March.
If you’re buying, put it in the contract. No silent downgrade. Notify within twenty-four hours of any change to model, version, or reasoning budget. Same data, same price, same capabilities — or you compensate.
And if the vendor won’t sign that, take the information you’ve just been given. You’re not buying intelligence. You’re buying access to a variable, and someone else gets to turn the dial.
There’s a principle underneath all of this, and it’s older than AI.
Consent requires disclosure. Not disclosure in the brochure. Disclosure at the moment of the transaction.
We spent a year telling each other that suspicion doesn’t scale — that you can’t build a society where nobody trusts anything. Fine. So the answer has to be proof. And proof starts with the smallest, plainest, most boring fact in the whole stack:
which machine answered me?
If a provider can change the intelligence inside your product without telling you, and the government can recommend it stay quiet, then the ability to name the machine is now the first sovereignty question. Everything else is downstream of it.
You can’t verify what you can’t name.
So log the version. Ask for the receipt. And when someone tells you it’s the same model as before, don’t take it on faith. You’ve got this — you just have to stop assuming the machine has an identity it’s obliged to show you.