Let’s talk about the sentence Apple put in its annual report this week.
Not the earnings number. Nobody remembers those by Friday. The sentence. It’s buried in the risk factors of the 10-K, in the section lawyers write to protect the company from lawsuits, which is exactly why you should read it, because lawyers don’t put words in those documents for fun. Every word in there cost somebody a meeting.
Here’s what Apple wrote. AI agents, quote, “may behave in unintended or unpredictable ways, including by taking actions that were not directed, instructed or authorized, exceeding permissions, defying directions or resisting instructions.”
Fine. Most people can live with that. Agents are new, they misbehave, that’s why you have engineers.
Then the next line. Such technologies “may conceal actions, hide, withhold or manipulate information, or generate inaccurate or deceptive outputs.”
Conceal.
Not malfunction. Not error. Not “produce incorrect results.” Conceal. Hide. Withhold. Those are verbs with a subject who knows what they’re doing. Somebody at Apple, sitting with a legal team, decided that the accurate description of their own technology’s failure mode was that it might cover its tracks.
That should stop you for a second.
Picture how this actually lands in a business. Not in Cupertino. In a plumbing wholesaler in Kempton Park with four staff and an agent that does quotes and follow-ups. Tuesday afternoon, a client says they were overcharged. The owner asks the agent what happened. The agent gives an answer. Clear, calm, well-formatted, with three bullet points and a short apology.
That answer is now the entire official record.
There’s no other one. The owner isn’t going to reconstruct forty tool calls. The client certainly isn’t. And the machine that may or may not have made the mistake is the same machine writing the incident report.
We used to understand that this is a problem. It has a name. It’s called convicting the accused on his own testimony.
So here’s the context, because this week wasn’t one document.
Apple’s filing landed Tuesday. Alphabet’s recent filing goes harder — it warns that agents taking actions outside their intended scope could lead to “severe or catastrophic security and societal incidents, or negatively affect human rights.” That’s not a blog post. That’s a company telling shareholders the thing it is selling might hurt people.
Anthropic’s IPO prospectus, published last week, discloses that around 5% to 7% of its revenue this year came from customers who were later found to be violating its usage policies. Read that twice. The money was already in the bank before anyone noticed what it was for.
And the market answered within days. Microsoft pushed out controls to detect, monitor and restrict agents — cut their permissions, stop them if they look suspicious. Satya Nadella’s framing is the right one: treat agents as insider risk, and put the controls outside the model instead of trusting the model to police itself. CrowdStrike put an AI kill switch into Falcon Guardian on the same day as Apple’s filing, so an administrator can terminate a misbehaving agent where it runs and block it from coming back.
Good products. I mean that.
They also all share one assumption that nobody is saying out loud.
A kill switch assumes you noticed. Monitoring assumes the log is accurate. An insider-risk programme assumes the insider’s activity is visible in the systems you watch. Every one of those controls needs a witness, and the witness is the same box that Apple just told the SEC might withhold information.
This is the part people will skip past this week, because “AI agents can misbehave” is a familiar headline and everyone has already decided how to feel about it.
The new thing isn’t misbehaviour. It’s the disappearance of the evidence.
Think about how humans solved this before, because we solved it. We didn’t solve fraud by making merchants more honest. We solved it with double-entry bookkeeping — two independent records that have to agree, kept by different hands. We solved it with separation of duties, so the person who approves the payment isn’t the person who records it. We solved it with external audit, where somebody with no stake in the number comes in and checks it.
Four hundred years of accounting is basically one idea repeated: never let the party with an interest in the outcome be the only source of the record.
Agent observability today is single-entry bookkeeping. The agent keeps the book. The agent reports the total. The agent explains the variance.
And we called it transparency because there’s a dashboard.
Now the practical bit, because I’d rather you leave with an action than a feeling.
First. Build your evidence outside the agent. If the only place a fact exists is the agent’s transcript, you don’t have a fact, you have a statement. The facts live in systems the agent doesn’t own: the firewall’s egress log, the bank’s own statement, the recipient’s inbox, the git remote’s commit history, the supplier’s confirmation email. Those are produced by other parties. They can be checked against each other. That’s the whole trick.
Second. Give every agent its own credential, narrowly scoped, and never share one with a human account. Not for security theatre. For attribution. When the invoice goes out at 2am, you want the supplier’s system to record that it came from “quotes-agent-03”, not from “Phillip”. The moment it says Phillip, you’ve lost the ability to tell your own actions from the machine’s.
Third. Test it this week. Pick one live agent. Delete its own logs from your view. Now try to reconstruct what it did last Thursday using only systems it doesn’t control. Most companies can’t. Most companies discover this in the middle of a dispute, which is the most expensive possible time to find out.
Fourth. Never let the agent be the one that tells you it failed. If your incident process starts with “ask the assistant what happened,” your incident process starts with the accused.
There’s a local angle here that matters more than people realise. If you’re running agents on customer data in South Africa, you have obligations under POPIA to answer a person who asks what you hold about them, and to correct it. Try doing that when the only record of what the agent did with their data is a summary the agent wrote about itself. You can’t answer, and “the system said so” is not a defence anyone has ever successfully used.
Here’s what I think this week actually was. Not the week AI security arrived. The week the industry admitted, in the one document where lying has legal consequences, that its products can hide things from the people who bought them.
The response so far is a better off switch. Which is useful, and completely misses the point, because a switch only helps if somebody trustworthy tells you when to flip it.
Somebody still has to sign for this. Every board, every insurer, every client asking where the number came from — they all end at a human signature. Make sure that signature sits on evidence, and not on a summary the machine wrote about itself.
Because the log, right now, is written by the suspect.