Let’s talk about the focus group that loved your idea — the one that was never in the room.
Picture a founder with a landing page, three price points, and a board meeting on Friday. She needs to know which price survives contact with the market. Two years ago that meant recruiting eight people, paying them in vouchers, and losing a fortnight of her life to scheduling. So she types the segment into a box instead. Women, thirty-four to forty-five, urban, household income in the top third, buys skincare online. Eleven minutes later she has forty respondents. They are articulate, they are consistent, and they all think the middle price is fair.
Of course they do.
This is not a thought experiment anymore. It’s an industry. Aaru — a company founded in early 2024 by teenagers — raised a Series A led by Redpoint last December at a one-billion-dollar headline valuation, with a blended number under that, a round north of fifty million, and annual recurring revenue still below ten million. EY ran a thirty-six-hundred-person global wealth survey through Aaru’s simulation and reported strong agreement with the real thing, in a day instead of six months. Qualtrics, which owns a serious chunk of the world’s survey plumbing, now sells synthetic respondents that sit in the same dataset as the real ones. The category has a conference track, a vendor list, and a growth chart.
So the question isn’t whether this is coming. It’s what it actually measures.
Here’s what the evidence says, and I want to be fair to it, because the fair version is more useful than the dismissive one. In April, the researchers at MeasuringU went through twelve peer-reviewed papers comparing synthetic respondents with real humans. They tagged nine findings as encouraging and fourteen as discouraging. When the synthetic panel is asked about averages — how a population feels about a category, roughly — it often lands close. Correlations can be high. That’s real, and it’s why the tools sell.
Then you go one level down and it comes apart. A team led by Park tried to replicate fourteen classic social-science experiments with GPT-3.5. Six produced data too flat to analyse at all. Five failed to replicate. Three worked. That’s a twenty-one percent hit rate on studies designed to be replicable. Shrestha and colleagues put forty-three policy questions to synthetic and human samples across the US, Saudi Arabia and the UAE. The averages lined up nicely. Roughly seventy percent of the individual questions came back significantly different.
That is the specific failure mode, and it’s worth naming precisely, because it’s not “the numbers are wrong.” It’s that the variance collapses. Real populations have tails — the woman who hates your brand for a reason you’d never guess, the man who’d pay double if you removed one feature. Simulated populations cluster around the middle, because the middle is what the model learned. You get a tidy distribution with the awkward people shaved off. And the awkward people are the entire reason you did research.
There’s a harder finding underneath that one. Wang and colleagues, in Nature Machine Intelligence last year, showed that language models standing in for people tend to misportray and flatten identity groups. Which makes sense if you think about where the training data came from: the internet’s description of a group, not the group. So the customer who is expensive to reach — the one on the phone, at the till, in the taxi — gets represented by a confident summary written by everyone who never met them. She didn’t consent to being spoken for. She just stopped being in the room.
And the flattening isn’t only statistical. The Nielsen Norman Group tested synthetic users for product research and found the feedback was “often favorable and vague.” Their words. Imagine a respondent with no memory of ever being disappointed by a product, no budget, no impatient spouse, no history of being burned. Ask that respondent whether your idea is good and you’ll get a considered, plausible, encouraging answer. You have not measured demand. You’ve measured agreeableness.
Now the part that makes this a pattern rather than a product review. Follow the incentive.
Research is a cost centre that produces delays and bad news. It is the only function in a company whose job is to tell you something you don’t want to hear, which means it is permanently under pressure from the people who fund it. Synthetic respondents remove the delay, remove the cost, and — this is the bit nobody says out loud — remove the bad news. Every concept tests “promising.” Every segment shows “strong intent.” Every price point lands inside the acceptable band. The deck writes itself because the panel agreed with the deck.
That’s not a data quality problem. That’s a market where the instrument has been quietly replaced by a mirror.
And here’s the capture, which is the reason this belongs on RADAR and not in a research newsletter. Once the simulation is in the loop, it stops being a tool and becomes your evidence base. It’s the thing in the board pack. It’s the thing the roadmap is defended against. It’s the reason the last three launches went the way they did. You can’t switch it off, because switching it off means admitting the last three launches were built on a description of a customer rather than a customer. The exit cost isn’t a licence fee. It’s your own story about your business.
The second-order effect is worse and slower. Your competitors are running the same panels, trained on the same internet, agreeing with the same strategies. Every company’s simulated customer drifts toward the same agreeable middle. So the market you’re all building for gets more average at exactly the moment the real market gets more weird. And a chunk of that training data is now AI text written by other models about customers who were themselves simulated. The caricature compounds.
So what do you actually do. Four things, and I’ll keep them short.
One. Synthetic for hypotheses, never for verdicts. Use it to draft the questions, map the language, find the categories you forgot to ask about. Never let it decide a price, a launch, or a positioning call. The rule I’d hold: a simulated panel can inform a decision you can reverse cheaply. It cannot be the reason for a decision you can’t.
Two. Run the bad-idea test before you trust any panel. Feed it something you know is terrible — deliberately, obviously bad — and watch what comes back. If it says “promising,” you’ve learned the tool’s real function: it measures agreement with the frame you handed it. Five minutes, and it tells you more than the vendor’s case study.
Three. Ten real calls. That’s the arbitrage. Your competitors have stopped phoning customers because the machine is faster, which means the person who still phones is now the only one with unflattering information. Ten calls beats a thousand simulated respondents for one reason: a real person can surprise you, and surprise is the only thing research is actually for.
Four. If you sell, assume your buyer’s panel will sand your differentiation into adjectives. “Robust.” “Innovative.” “Trusted partner.” Every one of those survives a simulation, which means none of them does any work. The things that don’t survive a panel are the things that can’t be summarised: a demo that runs, a reference who’ll take the call, a number you’ll guarantee in writing, a price with your name on it. Put the weight there.
The principle underneath all of it is small and it doesn’t sound like much. A customer is a person who can surprise you. A model of a customer can’t. Every system that replaces the first with the second will feel faster, cost less, and tell you a cleaner story — and the story will be about the customer you described, not the one who exists.
Somebody still has to answer the phone.
You’ve got this. But go and ask someone real, and let them ruin your week.