Let’s talk about the people OpenAI hired to be the human in the loop — and what they did when nobody was checking.
Because on Monday, 404 Media reported something that belongs on the wall of every founder who sells a human touch. Multiple contractors hired to improve OpenAI’s models have been fired — for using AI to do the job they were hired to do by hand.
The setup is called Project Lily. Hundreds of people, recruited through a staffing firm called Mercor, some of them earning more than fifty dollars an hour, reading real ChatGPT conversations and then scoring the model’s answers on a one-to-seven scale. Their entire purpose is to be the human. They flag the answers that are too sycophantic, too eager to flatter, too quick to pretend to be a person. They are, literally, the judgment OpenAI sells as the reason its model behaves this way and not some other way.
And some of them were quietly pasting that work into a chatbot and letting it grade itself.
Picture the person. Not a villain. Someone at the end of a long shift, staring at their nine hundredth conversation of the day, with a quota and a tired brain. So they do the thing everyone does now. They open an assistant. They ask it to write the summary. They clean it up. They submit. In a Slack channel somewhere, a reviewer gets a hunch and posts a sample — is this AI? — and the room answers, usually, yes.
One contractor’s termination letter flagged the “authenticity” of their work. Their own words to the reporter: “I’m not a bad person or worker. I just needed a little boost and turned to AI to help me, which eventually led to my downfall.” They added that they felt no joy in the work.
That’s the scene. Now the part that actually matters.
Think about what “human feedback” has always meant to you. Every time a company tells you its model was trained on human feedback, you hear a promise. A real person looked at this and made a judgment. That word — human — is doing an enormous amount of work. It is the trust label. It is the whole difference between a machine grading itself and a machine being corrected by someone who knows better.
This story says the label was never verified. The humans were machines too. And the company only found out by squinting.
Look at the internal guidance they were given. Reviewers were told not to use GPTZero or any other AI detection tool, because the tools are not reliable. Not allowed to use Grammarly. Not allowed to use translation. And then this instruction, which is the one that gives the whole game away: do not tell the evaluators why you suspect AI, because it is easier for them to hide if they know what you look for. Judge the overall pattern, not one clue.
So the test for catching a machine was a human’s gut feeling about style. Repetitive phrasing. Too many em dashes. Work finished too fast.
Now hold that next to what this industry has spent three years telling the world — that detection doesn’t work, that you cannot reliably tell machine text from human text, that anyone claiming otherwise is selling snake oil. Here is a company using exactly that unreliable instinct as grounds to fire people and cut off their income.
Both things cannot be true at once. Either detection works well enough to end somebody’s job, or it doesn’t work and some of the people fired were innocent. You don’t get to call it junk in public and conclusive in a contract. When the same tool is “unreliable” when it suits you and “definitive” when it doesn’t, you are not looking at technology anymore. You’re looking at power.
Here’s why this is bigger than one company’s hiring mess. That pipeline is fed by you. The training opt-in is on by default for Free, Plus and Pro accounts — you have to go into data controls and switch it off yourself. Your conversations, filtered, are the raw material. And the human signal coming out the other end is part of what shapes the next version of the model that talks to you.
If that signal is partly synthetic, then the model is partly teaching itself, wearing a contractor’s login. There’s a name for where that ends. Model collapse. A paper called The Curse of Recursion, and a study in Nature, both document the same failure. Train a system on its own output enough times and it doesn’t just get worse — it narrows. The range of what it can say, can imagine, can get right, shrinks toward the middle. It loses the edges of reality.
And one contractor admitted to something worse than cheating. They sometimes picked the worst answer on purpose. Their words: “it does feel like I’m getting paid to make AI worse.” A saboteur inside the feedback loop, completely invisible, because the feedback loop has no way to check itself.
So you have three inputs nobody can tell apart — genuine human judgment, machine judgment wearing a human’s badge, and deliberate sabotage. All of it lands in the same bucket. All of it gets called human feedback. None of it carries a receipt.
If you build anything that claims to be human-reviewed, human-moderated, human-verified, this is your warning.
Human is not a verification. It’s a claim. And every claim in your business that actually matters needs provenance behind it, or it’s just a politer word for a guess.
So do three things.
Verify at the point of work, not downstream. If a human is the value you’re selling, the system should prove the human did it while it’s happening — not reconstruct it later from writing style and how fast somebody finished. Every process that tries to detect its own integrity after the fact is already broken. You’re not auditing at that point. You’re guessing and calling it assurance.
Watch the incentives, not the intent. Nobody set out to poison the training data. They set a quota, a rate, an hourly target, and a mountain of tedious work — then acted shocked when people took the shortcut. The shortcut was always going to get taken. Design for it, or it designs you.
And own your own verdict. When a vendor hands you a score, a rating, a “human-checked” badge, ask the one question the whole industry is dodging: who verified the verifier, and where is the log I can pull myself. If they can’t show you the receipt, the badge is decoration.
And underneath all of it, the thing worth keeping.
Human judgment is the one input in this entire economy that can’t be manufactured at scale. That’s why it’s expensive. That’s why every lab on earth is trying to bottle it. Which is exactly why it’s worth being honest about when it’s real and when it’s just a costume.
A machine can produce a rating. It can produce ten thousand ratings by lunch. What it can’t do is care whether the rating is true. That was the job. That was always the job. The moment the person filling it stops caring, you don’t have human feedback — you have a warm body forwarding a machine’s opinion, and a contract that calls it trust.
The word human was never a guarantee. It was a responsibility. And somewhere in a Slack channel at the end of a long shift, it got quietly handed off.
You’ve got this. Just don’t let anyone sell you the word. Ask for the proof.