Opinion surveys: beware of these 4 types of statistical error

Opinion surveys are indispensable tools in market research. To obtain reliable results, you need to avoid 4 types of statistical error. In this article, I explain each error in detail: coverage, sampling, non-response, and measurement errors. If you have any questions, please post them in the comments. And, of course, if you would like us to conduct a survey, don’t hesitate to contact IntoTheMinds.

Contact the IntoTheMinds research agency

Coverage errors

Coverage errors occur when your target population is not fully represented in the survey. This is an important bias. This type of error can distort the results of a survey.

For example, a survey conducted online without considering individuals without Internet access can lead to coverage error, as it excludes a segment of the population who may have different opinions or characteristics from those online.

Coverage error is common when certain target segments are more difficult to access (see example above). I refer you here to my article on the diverse types of sampling.

Here are a few ways of solving the problem:

• Pre-survey analysis: this is my golden advice. Conduct a pre-analysis to identify potential coverage problems and adjust the survey design accordingly. How do you conduct this pre-analysis? With desk research, of course. In this case, I always favor academic sources to find results related to various customer segments.
• Broaden Access: use multiple modes of survey distribution (e.g., online or CAWI, telephone or CATI, mail) to include segments of the population that might be missed by a single method.
• Targeting: identify and conduct efforts to include under-represented groups in the survey sample. A survey must be representative.

Sampling errors

Sampling errors occur when only a subset, not all, of the survey population is polled, even if the selection is random. This error is fairly common; you can also see the link with the non-coverage error (previous paragraph).

The size of the sampling error can be estimated and is inversely proportional to the square root of the sample size, which means that larger samples generally tend to have smaller sampling errors. I stress “in general,” as this also depends on the statistical distribution.

Without going into too much technical detail, here are some ideas for avoiding sampling errors:

• Increase Sample Size: when feasible, increase the sample size to reduce sampling error. However, please remember that a sample, even a large one, will be useless if your sampling technique is wrong. In this blog, I’ve highlighted the dangers of convenience sampling, particularly the fact that surveys conducted via social networks are generally unreliable.
• Stratified sampling: use stratified sampling to ensure that important population sub-groups are proportionally represented.
• Clear reporting: report the margin of error and confidence intervals in the survey results to provide context for sampling error.

Non-response errors

Non-response errors are quite difficult to detect. It occurs when individuals who respond to a survey differ substantially from those who do not. As a result, respondents’ opinions may not accurately represent those of the entire sampled population. Factors such as the length of the survey, the sensitivity of the subject, and the mode of administration of the opinion poll can influence response rates and the extent of non-response error.

Here are some suggested solutions to this problem:

• Improve engagement: if you want everyone to respond, your surveys must make people want to take them. Without going as far as gamification, layout, colors, and general design should be considered. Remember to create questionnaires that can be easily viewed on smartphones.
• Follow-up: encourage non-respondents to participate. For our customers, we propose to generate unique URLs for CAWI surveys. This makes checking who has yet to respond and conducting targeted reminders possible.
• Adjustment and weighing: use statistical techniques to adjust for observable differences between respondents and non-respondents.

Measurement errors

Measurement errors occur when survey questions are poorly formulated or presented, so respondents provide inaccurate or uninterpretable answers. I can’t stress enough that creating a good quantitative questionnaire is complicated. Free tools make you think it’s easy. But as the old saying goes, “The devil is in the detail.”

This mistake is extremely easy to make, especially when amateurs try it. Here are just a few of the mistakes that come to mind:

• Ambiguous questions
• Leading questions,
• Overly complex language
• Inadequate scale

To avoid measuring just anything, my advice is to contact a Market Research agency.

However, I understand that this is not for everybody. Here are a few workable solutions:

• Ask someone to read your questionnaire: have someone you know read it. Stand next to them and observe their reactions. You need to rework the questionnaire if you can see from their face that they are stumbling over a question.
• Simplification: use clear, simple language and avoid technical jargon or complex question structures. I constantly repeat the rule to my colleagues: “One question = one topic.”
• Pre-tests: Conduct a survey to identify and correct problem questions. This is generally called a “soft launch” in the survey world. The survey is sent to a small percentage of respondents (5%, for example), and then the results are abstracted to detect any anomalies.

By Pierre-Nicolas Schwab Pierre-Nicolas is de directeur van de marketing agentschap IntoTheMinds. Hij heeft speciaal interesse voor e-commerce, retail, HoReCa en supply-chain projecten. Hij is ook een onderzoeker aan de Vrije Universiteit Brussel en werkt ook als coach voor meerde publieke organisaties. Hij verwelkomt Linkedin uitnodigingen : gelieve gewoon een paar woorden toe te voegen en uit te leggen waarom U wilt connecteren.