Let’s talk about the switch that got flipped on the fifteenth of September — and why it turned the AI agent working for you into a trespasser.
Picture someone in Johannesburg on a Tuesday morning. They ask their assistant to do something small and real: check whether the licence renewal portal is up, pull the current fee, book the earliest slot. Not a research question. A task. One agent, one website, one human on the other end waiting for it to be finished.
The assistant loads the page. Gets a block.
Not because the site owner hates AI. Because nobody at that site ever made a decision about it at all. A default made the decision for them, ten days ago, on a dashboard they have probably never opened.
Here’s what actually happened.
On the first of July, Cloudflare announced it was done treating every automated visitor as one thing called an “AI bot.” It split the traffic into three buckets. Search — a crawler building an index so it can answer questions later. Training — a crawler swallowing your content into the body of a model. And Agent — automation acting in real time, on a person’s behalf, to get something done right now. That last one is the key. Cloudflare’s own words: chat fetch bots, and browser agents like Gemini or Claude driving a browser. A human waiting on the other end.
Then on the fifteenth of September, the defaults changed. For new domains joining the network — and for existing free-tier customers — the Training category and the Agent category are now blocked by default on any page that carries an ad. Search stays allowed.
Read that again, because the shape of it is the whole story. The thing that got switched off by default is the agent. The category every company on earth is currently betting its next three years on. The one where a machine acts for a human in real time. And the thing that got protected is the ad page — the one place where a human eye was supposed to land and see something sellable.
The old economy’s unit of value got defended. The new one got defaulted off.
Now, the pitch is reasonable and I’m not going to pretend otherwise. The web has been scraped for thirty years on a deal that no longer holds — you crawl me, I get referrals. AI took everything and sent back nothing. Cloudflare’s data says more than half of the crawl traffic coming from AI bots is spent re-fetching pages it has already seen. And there’s a number in that announcement that should stop you cold: for the first time, the majority of traffic on the internet is not human. That was not expected until next year. It arrived early.
So the defaults of the web are being rebuilt — for a web where the machines are the population and we are the minority case.
Here’s the part almost nobody covered.
Because the most restrictive rule wins, multi-purpose crawlers get judged on all of their behaviour. Googlebot, Applebot, Bingbot — they crawl for search and they feed AI. So the moment a site owner ticks the box that says “no training,” the thing that walks in the door is Googlebot, and by the letter of the new rules it gets blocked. The small site owner who tries to protect their own work can quietly delete themselves from search in the same gesture.
That’s not a bug in the announcement. It’s the negotiation. Protect your labour, and lose your visibility. Let the machines take it, and you keep the referral traffic that may not even matter in a world where the answer shows up directly on the results page and nobody clicks through.
And underneath the whole thing, there’s a quieter property problem. Cloudflare wants to make agents verifiable — a transitive trust model, where an agent shows up carrying a header proving who it is really acting for. The intent is sound. But a header is a string. Anyone can type it. So the machine that claims to be you still has no way to prove it is you — and the site on the other end has no way to tell a helpful assistant from an operator stripping the shelves.
Which brings me to the uncomfortable bit.
Cloudflare sells the block. Cloudflare sells the toll — Pay Per Crawl, now being rebuilt as Pay Per Use, so a publisher charges when their content creates value rather than when it is fetched; Cloudflare Wallets for machine-to-machine payments; two launch partners, Ceramic and You.com. And Cloudflare also sells AI, to the same customers on the other side of the gate. That’s not a scandal. It’s just arithmetic. The referee has a product line. So when you read a default, read it as a positioning decision taken by someone with a balance sheet, not as a moral fact about how the web should work.
So what does an operator actually do this week?
Three things.
First, go and read your own defaults. Find out which of your pages carry ads, and check what you are blocking by accident. You will almost certainly find something you did not choose.
Second, if you protect your content, verify that you have not also cut your own search crawler. Test it. Look at what Googlebot is seeing, not what you assumed it was told. The opt-out exists but the window has closed for new domains and the setting is buried.
Third, stop planning as though access to the open web is a right your agent will always have. It is a metered, revocable product now, priced by whoever stands at the edge. If your business depends on agents fetching, booking, buying or listing, that dependency just got a landlord. Keep your own source of truth, keep your own distribution, and don’t let your only route to being found be a road somebody else can toll.
Because here is what’s really being settled, under the crawler configuration screens.
A default is not consent. Nobody voted for this one. It landed as an administrative change in a dashboard, and it is now the layer that decides which machines reach which page, which companies get into the answer, and which of us are findable at all. That’s governance wearing the costume of a setting.
And the principle worth keeping: you should be able to be found without being farmed, and you should be able to say no to the trade without becoming invisible.
The question is no longer whether the machines get in. It’s who holds the switch, who is holding the invoice, and whether anyone asked you.