12 September 2022 1037 words, 5 min. read

17% error rate: the truth about social network surveys

By Pierre-Nicolas Schwab PhD in marketing, director of IntoTheMinds
Realizing a survey requires real technical expertise. The easy access to captive respondents on social networks has pushed some market research institutes to propose methodological shortcuts whose effects are now clearly visible. Research published in Nature shows all the limitations of Facebook surveys. A sample […]

Realizing a survey requires real technical expertise. The easy access to captive respondents on social networks has pushed some market research institutes to propose methodological shortcuts whose effects are now clearly visible. Research published in Nature shows all the limitations of Facebook surveys. A sample of 250,000 respondents to a Facebook survey led to an error of 17%. The size amplifies the sample bias. This is the Big Data Paradox.

Summary

If you only have 30 seconds

  • Surveys realized through social networks have biases that can lead to large errors.
  • Research on Facebook surveys realized at the time of Covid-19 shows up to 17% error in estimates
  • The errors in estimates are due to the sampling strategy. Samples on social networks are rarely representative of the target population.
  • When large sample sizes are biased, the estimation error will be even larger.
  • Identify the variables that influence the behavior of the target population before deciding on your sampling strategy.
  • Do not use social media surveys for B2B market research.

Introduction

During the Covid pandemic, numerous surveys were launched to understand the population’s willingness to be vaccinated. We were asked by the authorities to study, by survey, the reasons for non-vaccination. The objective was to reduce non-vaccination factors and increase herd immunity.

Surveys were realized on social networks, and Facebook even launched a “Covid-19 Information Centre”. On this occasion, in the United States, nearly 250,000 people participated each week in a survey to assess their propensity to receive the first dose of vaccine.

As the 3 authors of the research show, this survey realized by Delphi and Facebook delivered estimates that only got worse as the epidemic progressed.

facebook covid information center

Facebook has created a permanent section dedicated to information about the Covid-19 pandemic.


Social network surveys less accurate than standard methods

The authors compare 3 surveys with the same objectives: to assess Americans’ propensity to receive the first vaccine against Covid-19.

The first survey was realized on Facebook, the second by email / SMS, and the third via a traditional methodology (panels):

  • Facebook survey: 250,000 respondents per week
  • Survey by email and SMS: 75,000 respondents per week
  • Classic panel survey: 1000 respondents per week

The survey administration method (Facebook in the first, email and SMS in the second) introduces a bias. The first two surveys have the particularity of not controlling who answers. In other words, there are no quotas based on the respondents’ profiles.

On the other hand, the panel method allows us to obtain a much better precision despite a sample size much smaller than that of the other two surveys (“Big Data Paradox”). To be convinced, you have to look at the graph below, which shows the drift of the different survey’s month after month.

Erreurs dans les estimations de vaccination au Covid-19 de la population américaine.

Errors in Covid-19 vaccination estimates of the US population. Comparison of 3 surveys to observed data (CDC Benchmark). Source: Bradley et al. (2021).

While in January 2021, the errors remain minimal, the accuracy of the surveys based on very large samples continues to decrease until May 2021. While the observed vaccination rate is about 60%, the Facebook survey overestimates the rate by 17% and the second survey by 14%. In contrast, the survey with 1000 people with the quota method allows us to limit the error to about 3-4%.


When large samples are biased, the estimation error will be greater.



A larger sample size does not guarantee a smaller margin of error

In survey research, it is paradoxical that a large sample size does not necessarily lead to better precision. Indeed, the size will only amplify the sample’s bias.

So, where do the huge errors come from? From the stratification of the sample.
The 3 surveys did stratify by age and gender. But the biases are deeper than that.

The Facebook survey polls only Facebook users. This is logical but deadly because a Facebook user is not necessarily representative of the population as a whole.

The second survey only considers respondents who can be contacted by email or SMS. This means that 19% of the American population is ignored.

The third survey ensures a wide diversity of respondent profiles. In addition to representativeness by age and age group, quotas are also set:

  • education level
  • ethnic origin
  • political opinions
  • respondents with neither email nor cell phone are included (about 1% of the sample).


A sample of 250,000 people can have the same inaccuracy as a sample of 10 people.



How to improve the accuracy of your surveys?

The survey is the most common method of market research. However, it is important to master the technical aspects to avoid drawing the wrong conclusions.

The sample size is not everything, and as we have seen, a sample of 250,000 people can have the same inaccuracy as a sample of 10 people. It’s all in the selection of respondents.

To improve the accuracy of your survey, you must select your respondents carefully and understand the criteria for choosing them. In the case of the Covid-19 vaccination, previous research has shown the influence of education and political affiliation. Not controlling for these variables could only lead to statistical disaster.

Before launching a survey, it is imperative to do literature research to discover which factors are likely to influence the behavior you are studying. Based on these factors, you must then decide on your sampling strategy. We realize that this is a complicated subject. So, if you need help, don’t hesitate to contact us.



Do surveys via social networks have any value?

So, the question is worth asking? What is the purpose of surveys via social networks, and can we trust them? The answer is simple: it depends.

It all depends on the purpose of the survey.

If your survey is about social network users, it makes sense.
If 100% of your target population uses social networks, it makes sense.
But you are lost if, by chance, some of your target respondents are not on social networks.

One last piece of advice for the road. Don’t consider surveying social networks if you want to research a B2B market. It wouldn’t make sense.



Posted in Research.

Post your opinion

Your email address will not be published. Required fields are marked *