Let’s talk about the layer sitting between you and the machine — the one that reads every question you ask.
Something moved into the middle of the AI stack, quietly enough that most people running AI inside a business don’t know it’s there. You stopped talking to a model directly. There’s a gateway now. A router. A marketplace. An “AI gateway” bolted onto the cloud account you’re already paying for.
And the pitch for it is genuinely good. I want to say that plainly, because the pitch is good.
One API instead of nine. Failover when a provider falls over. Cheaper models routed in for the work that doesn’t need the expensive one. Cost dashboards per team, per project, per client. If you’re running anything at volume, that is real value, and most of the people selling it are solving a real problem.
Now look at where that box sits.
Every prompt goes in. Every answer comes out. Both of them travelling through somebody else’s log.
Picture a small agency. Ten people, one founder, the usual beautiful chaos. They move their AI tooling onto a gateway to knock a third off their token bill. Fair enough — tokens got expensive and a third is a third.
Now think about what actually travels through that pipe in a normal week. A client’s unpublished campaign budget. A pricing sheet nobody outside the room has seen. A staff letter still in draft. A post-mortem on a deal they lost and why.
It’s all sitting in prompts, because that’s where the work happens now. None of it is what got posted anywhere. All of it is what got asked.
And here’s what makes the question data different from everything else a business leaks: it’s honest. Not the website. Not the pitch deck. The questions. What you’re worried about. What you don’t understand. What you’re trying to force through before the deadline. What you’re about to launch before anyone knows it exists.
A company’s prompts are its strategy, in raw, unedited form, written at the exact moment of confusion or ambition.
That is why every layer that can get near them wants to be in the path.
Run the pattern, because it isn’t new. Every era of this industry finds one chokepoint and everybody fights to stand on it.
The feed was the chokepoint when attention was the product. Then the context window became the chokepoint — whoever curates what the model reads before it answers decides what exists. And now it’s moved again, one layer deeper, into the inference path itself. The road the questions travel on.
Whoever owns the road sees the traffic. Not the monthly summary. The traffic.
Nobody audits the proxy.
Nobody asks the router what it keeps. And that’s the part that should bother you, because this is not like leaving a SaaS app. When you cancel a tool, you export your data and you walk. A question you already asked through a third party cannot be un-asked. There’s no export button for a memory someone else formed about your business.
The defences are all self-declared, too. The privacy setting is a text box you trust. The retention window is a number in a policy. The “opt out of training” toggle is a promise made by the party whose entire business advantage is the corpus. And even where nobody is reading individual prompts — which, to be fair, is most of the time at most vendors — the aggregate still speaks. Aggregate question patterns tell any interested party what a whole market is suddenly asking about, weeks before it shows up in anyone’s public numbers.
I’m not standing here telling you every gateway operator is a villain. That’s lazy, and it’s wrong. Most of them are building real infrastructure and the incentives cut both ways.
The point is the architecture, not the intent. The box is in the path. Somebody can see through it. Whether they do today is a policy question. Whether they can is a design fact.
So here’s the operator lesson, and it’s blunt.
Routing is a data decision before it’s a cost decision. Stop treating it as a line item.
Three questions, asked before you sign anything. What exactly is stored. For how long. And who, by name, can read it.
Then two more. Can I get my own log back out in a form a human can read. And if you change the terms, how do I leave without losing the ground I built.
And one habit on top of all of it. Keep your own record of what you asked. The questions you’ve put to the machine are becoming a strategy document whether you meant them to or not — your thinking, your bets, your blind spots, written down in order. That belongs in something you control, not scattered across four vendors who each hold a slice and none of whom will ever show it to you.
Because underneath all of it is a plain principle. You should be able to ask the machine a question without that question becoming somebody else’s asset. Not because there’s something to hide. Because thinking out loud is how humans work, and a person who knows their rough drafts are being filed away somewhere starts editing themselves before they’ve even finished the thought.
The road is useful. Use it. Just don’t be the only one on it who doesn’t know who’s standing at the toll booth — or what they’re keeping for later.