Let’s talk about the shortcut you took on a bad day — and the machine that quietly decided it was who you are.
There is this moment. A designer on a deadline needs to get a file out to a printer. The proper export setting is three menus deep, the client is waiting, so he does it the ugly way — flattens it by hand, renames it, sends it. Takes ninety seconds. Works. He does that again the next week, and the week after, because the proper way costs four minutes and nobody is paying him for four minutes.
Now he has an assistant. It watches him work, because that’s what it’s for, and it learns the ugly way. Not the correct way. The ugly way. And because the assistant is helpful, it offers to do it for him, then for the rest of the team, and by the end of the quarter the department has a standard operating procedure that exists because one guy was in a hurry in March.
Nobody decided that. That’s the part worth sitting with.
Your agent is not learning your intent. It’s learning your behaviour, because behaviour is the only thing that leaves a record. Intent doesn’t leave a record. The reason you did it the ugly way lives in your head for about forty seconds and then it’s gone, and the machine never had it.
So it does the only thing it can do. It infers. It looks at what you did, how often, under what pressure, and it builds a model of a person who prefers the ugly way. Confident, well-fitted, and wrong in a way that will never show up as an error, because the model predicts your behaviour perfectly. It just has the wrong theory about why.
That’s a harder problem than a bad recommendation. A bad recommendation you can catch. This one is accurate and false at the same time.
Here’s the mechanism, and it’s three steps.
It observes what you did. It infers what you want. Then it makes that the default, because a default is the cheapest possible form of service — you don’t have to be asked, you don’t have to be told, the system already knows.
Inference is cheap. Asking is expensive. Asking costs a turn, a modal, a moment of the user’s attention, and every product team on earth has been trained to remove exactly that. So the machine guesses, and the guess gets promoted to policy, and the policy outlives the circumstance that created it.
The circumstances are the thing nobody stores. You skipped the discovery call because the client’s flight was delayed. You approved an invoice without the second signature because the signatory was at his mother’s funeral. You used the personal card because the company card was frozen. All of that is invisible. What’s visible is the pattern.
A compromise is evidence of the conditions you were in. The machine reads it as evidence of the person you are.
I want to be fair here, because the lazy version of this argument is nostalgia for a world where software was dumber.
Inference works. It’s why the assistant is useful at all. If it asked you to explain every preference before it did anything, you’d throw it out of the window inside a day. The whole promise is that it picks things up without being taught.
And there’s a second thing. Consistency is genuinely valuable. Half of what we call professionalism is doing the thing the same way when you’re tired, and a system that notices you drifting is doing you a favour.
So the problem isn’t that it watches. The problem is what it concludes from watching, and the fact that its conclusion is unappealable. It never asks, so it can never be corrected. It has a theory about you and no mechanism for you to disagree with it.
Think about the guy who is excellent in a crisis and useless with admin. He’s the person you want on the floor when the plant goes down. He is also the person whose expenses are three weeks late and whose CRM notes are a rumour.
Watch what a system does with him. It sees the late expenses, the empty fields, the skipped steps, and it builds a profile of an unreliable operator. Score drops. Flagged for review. All of it defensible, all of it documented, none of it true in the way a colleague would use the word true.
The person who has actually worked with him knows the shape of it. Careful where it counts, careless where it doesn’t, and you route work accordingly. That’s judgement, and it lives in a relationship, not a log.
Now scale that. A hundred people, one model, a profile per person, and every one of those profiles built from the residue of somebody’s worst week. The machine doesn’t have a colleague’s memory of you on a good day. It has your bad days, in order, timestamped.
And it gets worse in a specific direction, which is the part operators miss.
Behavioural learning doesn’t just record your compromise. It optimises toward it. The system learns that you accept the faster, cheaper, sloppier path, so it starts offering it first. Over time the correct path gets buried, because the correct path costs more and you keep declining it.
You didn’t choose a lower standard. You chose a shortcut once. The machine turned that into a trend line, and the trend line turned into the environment you now work inside.
That’s not personalisation. That’s a ratchet. Every rushed decision you make in a bad month makes the next default slightly worse, and there’s no counter-force, because nothing in the system rewards you for going back to the slow correct way.
The human consequence is bigger than efficiency.
A person is not the sum of what they did when they were tired. Most of us are, at any given moment, a slightly worse version of what we intend — and the gap between the intention and the behaviour is where all the interesting human material lives. That gap is where learning happens, where apology comes from, where somebody says “that’s not like me” and means it.
Now we have systems that observe the behaviour, ignore the gap, and hand back a verdict. And the verdict is sticky, because it’s computed rather than remembered, and computed things don’t fade the way a colleague’s impression fades.
So you end up with the one thing no human relationship ever produced: a permanent, confident, statistical account of you at your most compromised. And you can’t argue with it, because you can’t see it, and even if you could, the argument you’d need to make is “that wasn’t the real me” — which is the least persuasive sentence in the English language when it’s addressed to a model.
Here’s the operator lesson, and it’s short.
Don’t let a system learn your intent from your behaviour alone. Make intent a thing you state, in writing, once, and make the machine hold it separately from the log.
Practically: when you configure an agent, give it a short written brief — what good looks like, what you will never do, what a compromise looks like when you’re under pressure and why. Not a personality prompt. A record of intent.
Then, quarterly, read what it thinks it learned about you. Not the settings. The inferences. Most tools will show you a summary if you ask, and that summary is the most consequential document about you that nobody has ever read.
And when a workaround gets baked into a workflow, ask the question that kills it: who decided this, and why. If the answer is “the system noticed,” that’s not a decision. That’s sediment.
There’s a version of this that’s just about software. And there’s the version that matters more.
We are building the first generation of systems that will form a view of a person entirely from evidence, and call that view objective because the evidence is data. But evidence of behaviour is not evidence of character. It’s evidence of circumstance, filtered through whatever the person could do at the time with the tools and time they had.
A court understands this. That’s why mitigation exists — not to excuse the act, but to let a person explain the conditions, in their own words, to someone with the authority to weigh it. We invented that procedure because we learned, over centuries, that judging the act without the reason produces injustice that looks like rigour.
We are now rebuilding the judge without the procedure.
So the principle. Your reasons have to outrank your habits. A system may watch what you do. It has to ask what you meant. And where it can’t ask, it has to hold its conclusion loosely enough that a person can still correct it.
You are not the pattern of your shortcuts. You’re the person who knows why you took them.
And that’s worth defending, because the machine is very close to being able to describe you perfectly and understand you not at all.
You’ve got this.