An opinion survey involves much more than simply asking many people for their opinions. I’ve talked about the choice of sampling method, and today, I will talk about the measurement itself. What is a good measurement? How can you determine the quality of the measurement based on the results obtained? If you have any questions, don’t hesitate to ask them in the comments. I’ll be happy to answer them.
Contact IntoTheMinds Survey Agency
Quantitative research aims to research phenomena by gathering quantifiable data and performing statistical techniques. Therefore, such research results must be valid, reliable, and generalizable. At least, that’s what our customers expect. In the article below, I outline the 4 characteristics that define good measurement in quantitative research. In particular, I illustrate these 4 characteristics with a diagram (below), which makes them much easier to understand.
The 4 main principles I outline below are not reserved for academic research. They apply to in-depth research, and every self-respecting survey company should consider them for every project.
1: the measurement is neither valid nor reliable
2: the measurement is confirmed but not reliable
3: the measurement is reliable but not valid
4: the measurement is reliable and valid
1. Validity (accuracy)
Validity refers to the extent to which a concept, conclusion, or measurement is well-founded and corresponds accurately to the real world. It is the precision of the research measure.
Several types of validity need to be considered:
- Content validity: ensures that the measurement covers the scope of the concept it is intended to assess. This means ensuring that the measure reflects the specific content domain intended.
- Convergent validity: demonstrates that measures of concepts that are supposed to be related are related. This means showing that your measure correlates well with other measures of the same construct.
- Discriminant validity: ensures that concepts or measures that are supposed to be unrelated are🙂. This type of validity shows that your measure does not correlate too strongly with measures of different constructs.
- Predictive validity: indicates that the measure can accurately predict future outcomes or behaviors. The measure can predict related outcomes in the future.
- Nomological validity: involves demonstrating that the measure fits into a theoretical framework that explains how the measure is related to other variables and measures in a predictable way.
2. Reliability (consistency)
Reliability refers to the consistency of a measurement. A measure is considered reliable if it produces the same result under the same conditions.
- Internal consistency: assesses whether the elements of a test are related to each other. It is often measured with Cronbach’s alpha, indicating the extent to which a set of items measures a single unidimensional latent aspect of individuals.
- Inter-rater reliability: measures the extent to which different observers or raters agree in their assessment decisions. This is crucial to ensure that the measure is not subjective or dependent on the interpretation of a single observer.
- Test-retest reliability: evaluates the stability of a measurement over time. It involves administering the same test to the same subjects twice and correlating the scores.
If you want your measurements to be reliable, use confirmed measurement scales. Whatever construct you would like to measure, there’s a 99% chance that a psychometric scale already exists. They can be found in so-called “scale books” or via a literature review. I want to stay within the subject, but I can only reiterate the need to conduct a literature review before creating your survey.
Contact IntoTheMinds to conduct your B2B and B2C surveys
3. Generalizability (the measurement can be used in different contexts)
A good measurement should be generalizable and can be applied in different contexts and populations. In other words, this is what we call extrapolation.
This characteristic ensures that the research results can be extended to a wider context. In the case of an opinion poll (during an election, for example) or a survey for a company, the ability to generalize the results ensures that the company can base its decisions on a solid foundation. Who would want to make strategic decisions on a country-wide scale when the results of a survey, in fact, only concern a sub-component of that country?
Market research agencies often hide behind the “margin of error” concept. However, the ability to generalize a result does not necessarily depend on this margin of error. I want to stress here that a “margin of error” can only be calculated if you know the distribution of your sample. This is rarely the case upstream.
4. Sparse (shorter measurements are preferable)
I can’t stress enough that, regarding surveys, “less is more.” In other words, the questionnaire should be as economical as possible. This is what I mean by the concept of parsimony.
Parsimony in measurement implies that simpler, shorter measures are preferable as long as they don’t compromise the validity and reliability of the results. A parsimonious measurement is efficient, avoiding unnecessary complexity that could make the measurement difficult to interpret or apply in various contexts.
Here are some practical tips:
- Limit the number of questionsto avoid the phenomenon of “fatigue,” which tends to affect the validity of measurements adversely
- Opt for brief wording to avoid differences of interpretation between respondents (another validity problem).
- Make sure questionsare worded clearly and unambiguously. This will prevent respondents from experiencing comprehension problems (a reliability problem).
Posted in Marketing.
