When Synthetic Data Works in Research (and When It Doesn't)

Legitimate synthetic data research uses are narrower than the marketing suggests: hypothesis generation, pilot-testing discussion guides, red-teaming analysis, and carefully flagged gap-filling in quantitative datasets. None of these involve replacing real respondents in a study. The honest direction for the field is more real respondents at lower cost, not fewer replaced by statistical averages of the training distribution.

Key Takeaways

• Synthetic data has four narrow legitimate uses; none involve substituting simulated voices for real participants in a study.
• LLM-generated personas are useful as prompts for discussion guide design, not as evidence of customer opinion.
• Pilot-testing a draft guide against a simulated respondent catches obvious problems before you field with humans.
• To evaluate any synthetic respondents product, run a head-to-head study against real respondents and compare what each surfaces.
• Real respondents win these comparisons decisively, even at much smaller sample sizes than the synthetic run.

The four legitimate synthetic data research uses

Hypothesis generation is the cleanest case. Before fielding, having an LLM brainstorm what a segment might say sharpens discussion guide development. You treat the output as prompts for your thinking, not as data.

Pilot-testing guides is the second: running a simulated interview against a draft catches obvious problems, ambiguous wording, leading probes, dead-end branches, before a real participant ever sees it.

Red-teaming analysis is the third: asking a model to generate counter-arguments to your findings ("What would a skeptic say? What alternative reading fits the same quotes?") stress-tests conclusions without adding new evidence.

The fourth, completing demographic gaps in quantitative datasets, is a specialist statistical application with specific methodological guardrails, and it sits well outside the qualitative use case most vendors are pitching.

AI persona research limitations, stated plainly

Synthetic personas regenerate the training distribution. They produce fluent, plausible answers that do not correspond to any actual person. That is fine as a thinking aid, dangerous as evidence. The deeper problem is that an LLM cannot tell you what it does not know: it will never surprise you with the thing you needed to hear, because it has no access to the lived experience that generates surprise. It smooths ambivalence into preferences, because the training data rewards clear answers. It misses silences, hesitation, the participant who changes the subject when you ask about family. Reframed in our platform evaluation guide, this is the difference between rehearsal and research.

Synthetic vs real respondents: How to evaluate a vendor

If you are evaluating a synthetic respondents product, insist on a head-to-head comparison against real respondents on a topic you care about. Run the same study both ways. Read both outputs carefully and ask which one produced an insight you did not already know. In the comparisons we have seen, real respondents win decisively, even at much smaller sample sizes. The synthetic deck reads smoothly and confirms priors; the real deck contains the unexpected quote that changes the brief. Enumerate's asynchronous AI-moderated interviews exist precisely to make that "more real respondents at lower cost" path economically viable for studies that previously could not afford the depth.

When to use synthetic research data, and when not to

Use it inside the research process, as tooling. Use it to draft, to rehearse, to challenge. Do not use it as a substitute for the people whose decisions you are trying to understand. The category will settle, through market discipline, disclosure requirements, or industry standards, but practitioners do not need to wait for that correction to make the right call now.

Want to see what real respondents at scale looks like in practice? Book a demo with Enumerate.