The Net Promoter Score (NPS) is a simple method to measure customer satisfaction globally. But you must apply it correctly. We regularly discover, through the customer satisfaction audits we do, that the results are distorted by incorrect use of the software. The example we are presenting today comes from Lufthansa. The error in implementing the NPS is not obvious, but it still taints the credibility of the results. We conclude this article by presenting the 2 most common types of errors in the design of customer satisfaction questionnaires.
NPS: use by Lufthansa
The screenshot below shows you an NPS-type formulation sent to a Lufthansa customer. The question’s title (in German) corresponds well to the one formulated by Reichheld: “How likely is it that you would recommend Lufthansa to a friend or colleague?
Even if you don’t speak German, 2 aspects differ from the NPS we are used to seeing:
- the NPS scale is reversed: the maximum score (10: very likely) is on the left side of the scale, and the minimum score (0: very unlikely) is on the right side.
- a green color has been used to delimit the “promoters” zone (scores 9 and 10) and red color for the detractors (0 to 6).
By defining the zones between promoters and detractors in this way, Lufthansa introduces at best uncertainty, at worst, bias.
What are the problems with the NPS as used by Lufthansa?
In this paragraph, we analyze the 2 errors we found in Lufthansa’s use of the NPS.
Inverting the NPS scale
Inverting the scale is not in itself a mistake. It is even a good practice when a respondent is subjected to a complex questionnaire. Inverting the scale makes it possible to verify that the respondent is paying attention by checking that they do not always mark the same answers (on the left of the scale, in the middle, or on the right). Inverting the scale is particularly relevant when the same question is asked in two places in the questionnaire: it allows us to check the consistency of the answers.
In the case of the NPS, this inversion does not seem essential.
Adding colors to the NPS scale
Adding colors to the NPS is a different story. By delimiting the zones between promoters and detractors, Lufthansa introduces at best uncertainty, at worst, a bias. The colors unconsciously indicate to the respondent what is good (green) and bad (red). The NPS scale, which is deliberately neutral, is superimposed with a reading grid to guide the responses. A direct consequence will undoubtedly be that there will be fewer answers 7 and 8. Respondents will try to “position” themselves concerning their satisfaction or dissatisfaction. Let’s not even talk about the choice of grey (a neutral color, for example) for the “neutral” category choices.
Therefore, this modification of the NPS is a gross error that taints the measurement of customer satisfaction and makes it unreliable. This error falls into the broader category of “persuasion bias” that we find in many companies during our audits.
Customer satisfaction: why so many mistakes?
As we mentioned in the preamble, we realize many customer satisfaction audits, and sometimes we find major errors and always minor ones.
Beyond this difference in terms of severity, there are 2 types of errors that can occur when setting up a customer satisfaction measurement instrument:
- formulation error, which introduces a bias and increases the margin of error
- design error invalidates the results by orienting the responses more or less deliberately. This error is extremely serious and was theorized by Goodhart.
Let us reassure you that most of the errors we find among our customers are of the first type. The people who designed the satisfaction questionnaires did not have all the technical knowledge to avoid errors. They, therefore, introduced unintentional biases.
Unfortunately, we also encounter much more serious errors that influence the customers’ answers. Paradoxically, these errors are mostly made in large companies. These companies are supposed to have the best-trained staff, but within them, the impact of customer satisfaction is also felt the most. Customer satisfaction measurement is indeed an indicator that can go all the way up to top management and on which bonuses sometimes depend. All the conditions are, therefore, present for the results to be distorted. Several examples come to mind as I write these lines:
- car dealerships that tell you what to answer to the satisfaction questionnaire you will receive by email after picking up your car
- a bank that advertises its satisfaction rate but measures it using a biased questionnaire
- an insurance company that measures its performance against overly permissive benchmarks
Posted in Marketing.
