Can AI answer your survey for you? What a Marketing Science study found
Asking a chatbot to play your customers is cheap and tempting. Mixed in carelessly, those answers make your results worse. Used with a correction, they can cut the number of real people you need to ask.
Pooling AI-generated answers with real customer answers can bias a preference study. A correction method kept the AI’s help and saved up to 80% of real data.
Marketing in 3 · Issue 35 · Week of 7 September 2026
Each tick box is a preference. The question here is whether a machine can tick them for a customer. Photo: Zulfugar Karimov on Unsplash.
The short answer
Not on its own. In a study of vaccine choices, answers generated by ChatGPT models did not match what real people chose, and simply pooling them with real answers pushed estimates further from the truth. A statistical correction learned from a small real sample fixed this and saved between 25% and 80% of the real data needed, depending on how many real answers were available. AI answers are a supplement to real respondents and not a stand-in for them.
Key takeaways
AI-generated and human answers differ in systematic ways. In this study, newer models and cleverer prompts did not close the gap, and step-by-step reasoning prompts often widened it.
Adding AI answers the obvious way, by pooling them with human ones, usually made estimates worse for the stronger models.
The authors' correction compares AI and human answers on a small real sample, then adjusts a much larger batch of AI answers. It reduced error in every setting tested.
On vaccine choices it saved 24.9% to 79.8% of the real data needed. On sports car choices the savings were 5.1% to 60.3%.
What the researchers did
Market researchers want to know how much customers value each feature of a product. The standard tool is conjoint
analysis. People choose between made-up products that differ in price, features and so on, and a model works out which
features drive their choices. It works, and it is slow and costly, because every answer comes from a recruited person.
Large language models offer a shortcut. Ask one to act like a random shopper and it will answer thousands of choice
questions for almost nothing. Earlier studies found these answers look plausible, and also differ from real human answers in
ways that can mislead.
Mengxin Wang (UT Dallas), Dennis Zhang (Washington University in St. Louis) and Heng Zhang (Arizona State) asked a practical
question. Is there a way to use AI answers without being misled by them? They built a statistical method, which they call
an AI-augmented estimator, and tested it on two real conjoint studies.
Main test
A national survey on COVID-19 vaccine preferences, 1,971 U.S. adults, each choosing between two hypothetical vaccines five times
Second test
A sports car study with 200 people, each choosing between three cars ten times
AI used
Several ChatGPT models, with plain prompts, step-by-step reasoning prompts, example-based prompts and a model fine-tuned on real answers
Method
50 repeated experiments per setting, mixing a small set of real choices with 1,000 AI-generated ones
Funding
OpenAI Research funding supported the work
Why simply pooling the answers goes wrong
The tempting move is to put real and AI answers into one pile and analyse them together. The authors show why that fails.
Take the vaccine data. Using the full human survey as the truth, they built small real samples and added 1,000 AI answers. When
they pooled the AI answers in directly, the estimates did not settle toward the truth as more AI data arrived. In most
cases they drifted further away. Adding more real data did the opposite and shrank the error toward zero.
The pattern held across model versions. Stronger models and fancier prompts did not fix it, and sometimes made things
worse. The authors suspect that when a model reasons step by step, it becomes too rational. It weighs efficacy and side effects the way a
textbook would, and real people are messier. Fine-tuning on real answers helped, without closing the gap.
The authors' summary is that AI-generated data is informative but not a direct substitute for people.
The fix: learn the gap, then correct it
The method works in plain steps. First, collect a small number of real answers. Then ask the AI the same questions, so
each real answer has an AI answer beside it. A statistical model learns how the AI's answer relates to the real one for
each type of choice. Next, that model corrects a much larger batch of AI answers, so they look more like human answers.
Finally, everything is analysed together.
The idea is to use the small human sample twice. It supplies real data, and it teaches the system how far the AI strays.
The authors prove the estimates stay sound as the sample grows, which the naive pooling approach cannot offer.
The correction also improves when the AI answers are better. With a fine-tuned model it did best. The plain pooling
approach shows no such pattern.
They compared it with another known technique for mixing AI predictions with real data. The two performed about the same
with few product features, and the new method did better when there were many.
How much real data it saved
The test was how many real answers a researcher can skip. The authors found the amount of real data you would need
to reach the same accuracy without AI help, and compared.
With only 50 real choices, which is about ten people at five choices each, the correction saved 72% to 80% of the data,
depending on the model. With 100 or 150 real choices the saving was 28% to 58%. With a fine-tuned model it ranged from
25% to 80% across sizes. The saving shrinks as the real sample grows, because there is less left to add.
The sports car study gave a similar picture with smaller numbers. Savings ran from 5% to 60%, and were 47% to 60% with
a very small real sample. The authors note that sports car preferences are highly subjective, which they say is why
newer models and cleverer prompts helped less here.
Share of real survey answers saved, by size of real sample
Share of real survey answers saved, by size of real sample
Group
Value (%)
50 real choices
80
100 real choices
58
150 real choices
50
200 real choices
25
Table 8, COVID-19 vaccine study, fine-tuned GPT-4o with the correction method, averaged over 50 runs. Each respondent made five choices, so 50 choices is about ten people.
What it means for your research
Read this as a method for stretching a real sample, not a way to avoid one. The correction needs real answers to learn
from, and the more varied they are, the better it works.
For a small firm, the practical gain is the early stage. When budgets allow ten or twenty real respondents, a correction
step can make that small sample go much further. It is also cheap to repeat. Preferences shift, so conjoint studies need
repeating, and AI answers cost almost nothing to regenerate.
There is a risk in the other direction. Anyone can run an AI survey and get tidy-looking numbers. The study shows
those numbers can be confidently wrong, and the error can get worse as you ask the AI more questions.
Marketing in 3 steps
Ready. Set. Market.
1
ReadyDecide and prepare
Collect a small real sample first. Survey at least 20 real customers, each making five product choices. Keep the exact product descriptions and the answer each person gave.
Ask an AI tool the same questions. Paste each choice into a chat tool with: "Act like a random shopper. Which option would you choose? Reply with one letter." Run the same 100 choices and record each answer beside the human one.
2
SetBuild it
Count where they disagree. Work out the share of choices where the AI picked a different option from the human. Then look at which feature, such as price or a brand, the AI seems to over-weight.
Keep the AI out of your headline results. Report the human results as your main finding. Use AI answers only to plan, for example to shortlist features or draft questions, unless an analyst applies a statistical correction.
3
MarketLaunch and measure
Ask for a correction, not an average. If you need a bigger sample than you can afford, ask your analyst to use a method that learns the gap between AI and human answers and adjusts for it. Do not average the two sets.
Test it before you trust it. Split your real sample in half. Run the corrected method using only one half, and compare its estimates with those from the full human sample. Use it only if it lands closer than the half-sample alone.
What to expect
A cheaper first pass. Corrected AI answers can stretch a few dozen real respondents a long way when budgets are tight.
Savings that shrink. The more real data you have, the less AI adds. Expect a big saving on small samples and a small one on large samples.
Plausible but wrong. A pure AI survey can look clean and be off. Always keep a real sample to check it against.
Repeat studies get easier. Re-running a preference study each year gets cheaper, because the AI answers cost almost nothing.
Where it does not work
Two products only. Vaccines and sports cars are two of many categories. Results may differ for everyday goods, services or luxury.
One model family. All the AI answers came from OpenAI models, and the work was funded by OpenAI Research. Other models may behave differently.
Models were not the newest. The newest model available at the time was left out, and the authors expect the pattern to hold, but they did not test it.
Possible overlap with training data. The authors cannot be sure the AI had not seen the survey data. They argue this would have helped pooling, which still failed.
Questions marketers ask
Can AI replace survey respondents in a conjoint study?
Not safely. In this study AI-generated choices differed from real ones in consistent ways, and pooling them with real answers often made estimates worse (Wang, Zhang & Zhang, 2026, Marketing Science).
Do better prompts fix AI survey answers?
Not reliably. Step-by-step reasoning prompts often widened the gap, perhaps because the model became too rational. Examples and fine-tuning helped somewhat, though neither closed the gap.
How does the correction method work?
It uses a small set of real answers that have an AI answer beside them to learn how the AI differs from people. It then adjusts a large batch of AI answers using that pattern before analysing all the data together.
How much real data can it save?
In the vaccine study, between about a quarter and four fifths, with the most saved when the real sample was smallest. In the sports car study, between about 5% and 60%.
Is it worth it for a small business?
It can be, if you have at least a few dozen real respondents to learn from and an analyst who can apply the method. Without a real sample, there is nothing to correct against.
AI agents can now complete online surveys by reading the screen, and old attention checks miss them. The author builds questions that are easy for people and hard for vision models. Tested on 1,007 people and 526 AI agents, they caught 97.1% of agents and flagged 4.1% of people, against 2.3% of agents caught by traditional checks. If you pay survey respondents, add checks like these.
International Journal of Research in Marketing · 2026
A step-by-step guide for putting a generative AI chat inside a survey platform, with four worked examples: a search assistant, an automated interviewer, a co-creation tool and personalised messages. It also offers a decision framework for when AI adds something over standard methods, and shows how AI coding tools can write the connecting code. Start with one of the templates.
Source
Wang, M., Zhang, D. J., & Zhang, H. (2026). Large Language Models for Market Research: A Data-Augmentation Approach. Marketing Science, 45(4), 728-751. https://doi.org/10.1287/mksc.2025.0009
This brief is written for practitioners from the published article. Numbers are as reported by the authors; read the article for full methods.
A language model guessed the star rating of 28,900 reviews from the text alone. When it missed, human readers missed too, and found the review less helpful.
In four experiments, AR try-on previews raised return intentions after a flawed product arrived. Positive reviews and realistic rendering reduced the effect.
Bruno Schivinski is Associate Professor of Marketing at Gdańsk University of Technology. He studies how consumers engage with brands in digital and social media.
If this issue saved you a meeting or sharpened a campaign, say thanks with a euro, or buy me a coffee or a pint if it really helped. It keeps Marketing in 3 free, weekly and ad-free.
