Let’s talk about the law that made a human signature mandatory — and then declined to protect the human who signs.
It’s three in the morning on a ward somewhere. A patient is going off. Not dramatically — no alarm, no crash team, just a man in his seventies who looks worse than he did at midnight, and a nurse who has been doing this for eleven years and can feel it.
She checks the system. The acuity score says he’s stable. The score is generated by software her hospital bought to decide how much attention each patient needs, and it has just told her, in effect, that she is wrong.
Two-thirds of the nurses in a survey of more than two thousand of them said those automated scores didn’t match their own assessment of their patients. Two out of three.
So she does the thing you’d want her to do. She escalates. She acts on her judgment instead of the number on the screen.
And here’s the question I can’t get past. Who’s covering her if she’s wrong?
Not “who’ll thank her.” Who’s covering her.
Because at the end of September, California answered that question in law — four times in one week — and the answer was: nobody. Not really.
Let me lay out what actually happened, because the detail matters more than the headline.
On the twenty-eighth of September, the governor signed AB 1609 — the Right to Human Customer Service Act. If you’re a business pulling more than five hundred million dollars a year and you put a chatbot in front of customers, you now have to say it’s a bot, and if someone asks for a person, you have to make a good-faith effort to connect them within fifteen minutes, or give them an appointment. Fines are five thousand for the first violation, ten thousand after. No private right of action, so you can’t sue them yourself. Hospitals are carved out for healthcare communications. It takes effect on the first of January.
Two days later, three healthcare bills.
AB 1979 says a licensed clinician has to exercise independent clinical judgment whenever AI informs patient care, and that an AI system can’t independently perform clinical functions the state reserves for licensed professionals — or instruct unlicensed staff to perform them. Breaking it counts as unprofessional conduct, enforced by the Medical Board. It also pulls any business offering a consumer health chatbot under medical confidentiality law.
SB 503 makes bias mitigation and documentation a duty, for developers and for the hospitals deploying the thing.
SB 947 says an automated system’s output alone can’t justify disciplining or firing an employee. A human has to corroborate it with actual evidence.
Four bills. One message, repeated: the human decides.
And then the same day as the healthcare signing, the governor vetoed AB 2575.
That was the bill that would have said: a clinician can’t be punished by their employer for relying on the AI or for refusing it. And — this is the part that should stop you — it would have stopped the developers and the health systems from arguing in court that the clinician’s failure to override the AI was the cause of the harm.
That’s the liability shift. That’s the whole ballgame. When the tool hurts someone, the vendor’s best defence is “the nurse had the final say and she didn’t use it.” The bill would have taken that defence away.
It got vetoed. The stated reasons were technical and, read fairly, not stupid: the standard in the bill was too absolute, and the labour commissioner isn’t equipped to judge clinical standards of care. That’s a real objection. I’m not going to pretend it was malice.
But the effect is what it is. The California Nurses Association put it plainly: a nurse who sees the algorithm is wrong and acts to save her patient can still be disciplined or fired by her employer.
So sit with the shape of this. The law now requires that a human exercises judgment. The law does not require that the human be given time to exercise it, information to exercise it with, or protection when they exercise it against the wishes of the people who bought the software.
That is not oversight. That’s a signature.
And here’s why this is bigger than California, and bigger than hospitals.
For two years, “human in the loop” has been the phrase that closes arguments. Vendor decks, procurement reviews, regulator submissions, board packs — the sentence appears and the room relaxes. A human is in the loop. Fine. Move on.
This week that phrase became a legal form. Which means it will now be produced on demand, by companies that have no interest in what it costs the human standing in the loop. The loop is where the liability gets parked, and the person inside it has been told they’re the safeguard.
Think about what the human actually needs in order to be a safeguard rather than a rubber stamp. Three things.
Time. A real window to look at the case. Not four seconds between patients.
Information. Some sense of why the system said what it said, what it was trained on, how often it’s wrong in cases like this one.
Cover. A job on Monday if they say no. A defence that holds if they say no and they’re wrong.
AB 1979 gives the clinician a duty. SB 503 gives the developer a documentation duty. Nothing in that package gives the clinician cover. The one bill that did — the vetoed one — is gone.
Now the operator lesson, because there’s a version of this you can use this week.
If you’re buying AI that touches a decision with a human life or a person’s livelihood attached, the question is not “is there a human in the loop.” The question is: what happens to that human when they disagree with your software?
Write it into the contract. Who bears the loss when the human overrides and the human is wrong. Who bears the loss when the human complies and the software is wrong. Who owns the override log, and can the human read their own record. What the escalation path is when the human says no and someone above them says do it anyway. If a vendor won’t put those four things in writing, they’re not selling you oversight. They’re selling you a witness.
And if you build this stuff — which is my side of the table — stop shipping “human-in-the-loop” as a feature claim. It costs nothing to say and it’s now worth less than nothing, because a regulator can point at a law and say yes, we know, everybody has one of those.
Ship this instead: the human can say no, here’s the button, here’s what happens next, and here’s the record that proves they were given a real choice. That’s a product decision, not a compliance one. It’s harder. It’s also the only version that survives contact with a bad night.
There’s a second thing buried in that week that I don’t want to lose, because it’s quieter and it’ll outlast the hospital story.
The right to a human being, in customer service, is now a legal entitlement — but only above five hundred million in revenue, only for Californians, only during business hours, only by “good faith effort,” and with no private right of action if they fail you. Which means access to a person is becoming a regulated privilege attached to the size of the company you happen to be dealing with.
That’s the direction of travel. The human path as a tier. A premium lane. Something you’re entitled to if the counterparty is big enough and you’re in the right jurisdiction — and something you simply don’t get if you’re not.
We already watched that happen with phone support. Now it has legislation around it, which at least means someone wrote down where the line is.
So what’s the principle here.
A signature only means something if the person holding the pen can afford to refuse.
Strip that away and you get the worst of both worlds: the machine gets to act at scale, and a human gets to absorb the blame at the end of it, and everybody points at the signature as proof that someone was thinking.
The whole point of a human in the loop was never that a human is present. It was that a human can look at the thing and say no — and still have a life on Monday.
If the law won’t give them that, the least we can do is stop pretending the loop is closed.
The nurse at three in the morning was right about her patient, by the way. She usually is. That’s not the part that worries me. What worries me is the version of that story where she’s right, she overrides, the patient does fine, and the only record anyone keeps is that she deviated from the system.
You’ve got this. Just make sure the paper says so.