18 April 2017 479 words, 2 min. read

How the Goodhart’s law makes customer satisfaction surveys fail

By Pierre-Nicolas Schwab PhD in marketing, director of IntoTheMinds
“When a measure becomes a target, it ceases to be a good measure.”. This is the Goodhart’s law and it explains in my opinion why why an awful lot customer satisfaction measures fail. Let me take two concrete examples that […]

“When a measure becomes a target, it ceases to be a good measure.”. This is the Goodhart’s law and it explains in my opinion why why an awful lot customer satisfaction measures fail.
Let me take two concrete examples that I’ve experienced myself : Audi and Beflius.

in this section I’d like to discuss two concrete examples of customer satisfaction measurement that becomes organizational objectives and therefore cease to be good measurement instruments.

The Belfius case : when you judge your own performance it can only backfire

I already discussed the Belfius case recently and expressed doubts years ago about their satisfaction measurement. The slides of their press conference are available here and if you read them you may notice that the satisfaction claims are a communication exercise. On page 17 for instance they compare themselves to Volvo. Volvo is associated to “safety” and Belfius should be associated to “customer satisfaction”. While this claim is perfectly possible, the reliability of the claims differ largely. The Volvo’s safety claims are based on results provided by an independent testing organism, using the same testing method for all manufacturers, hence allowing this claim to become verifiable for the car manufacturing market. No such homogeneous measure exist for customer satisfaction, hence making Belfius’ claims theoretical and not verifiable.

At Belfius the performance is probably measured by those who are also assessed. This is a very serious bias. The Goodhart’s law applies: the measures becomes a target; hence the risk exists of a biased measure that will eventually deliver the expected results.

The Audi case : when you incentize employee based on customer satisfaction, it can also backfire

The Audi case is even more extreme. Audi dealerships are incentivized on their performance and most particularly on the customer satisfaction scores they obtain. I’ve experienced all the dangers of this when I brought my -rather old- Audi (replaced in the meantime by a Toyota Prius) to maintenance some 5 years ago. When I picked up the car after the regular maintenance I was asked to fill in a satisfaction survey. The employee literally put me under pressure to tick the “extremely satisfied” box, explaining me that he was assessed on this score and that “it was important for him to collect as many very satisfied assessments as possible”.
Needless to say that the measure was completely biased and useless.

Conclusion

These 2 examples show that customer satisfaction surveys must be handled for what they are : measurement instruments the accuracy of which needs to be assessed. Academics spend a great deal of time developing reliable scales. This isn’t an easy task.
Yet the challenge doesn’t end when the scale is built. It must still be administrated in a way that avoid bias. In the case of Belfius, linking salaries to satisfaction scores was probably not the best decision to take to ensure that results wouldn’t be biased by employees’ behaviors.

Image: shutterstock



Posted in Marketing.

Post your opinion

Your email address will not be published. Required fields are marked *