Mystery shopping and mystery calling have become widely used techniques by market research agencies to assess customer satisfaction. This technique is mainly applied in a B2C context. While the methodological approach seems justified, it is based on several assumptions and limitations, which we decipher in this article.
Entrust us with your customer satisfaction measurement project
Mystery shopping and customer satisfaction: All you need to know in 30 seconds
- The use of mystery shopping/mystery calling techniques assumes that measurements taken by professionals will be consistent with those taken by real customers.
- This assumption has been challenged by scientific research in the Netherlands.
- Mystery shoppers tend to assess the NPS (Net Promoter Score) more severely than real customers. The difference is 0.5 points.
- Real customers have a significantly higher level of gratitude than mystery callers.
- There are no significant differencesin satisfaction between real customers and mystery callers.
Mystery shopping, and its telephone form “mystery calling,” is a widely used market research technique. It aims to assess and improve customer service quality. The technique is used in various sectors, including the hotel and restaurant industry, financial services, transport, retail, and health services. This method enables companies to:
- Understand customer experiences
- Measure customer satisfaction
- Identify areas for improvement
- Ensure compliance with established standards
The mystery shopping/mystery-calling industry has grown steadily over the years. Today, it represents a $2 billion industry employing 1.5 million people worldwide.
The hypothesis that calls into question the value of “mystery” approaches
The value of mystery shopping and mystery calling for measuring customer satisfaction is based on a dual hypothesis:
- mystery callers” and “mystery shoppers” would faithfully represent the behavior of real customers
- They would provide representative assessments of how consumers feel about their satisfaction.
Of course, this assumption needs to be clearly stated.
Companies that opt for mystery shopping/calling are, therefore, unaware of it. Beyond the customer, it’s a safe bet that the research firm itself isn’t even aware of it.
What if this hypothesis isn’t confirmed? What if mystery shoppers underestimate or overestimate customer satisfaction? This is what we propose to discuss today, based on scientific research conducted at the University of Groningen (Netherlands).
The advantages of mystery shopping as a market research technique
Mystery techniques make it possible to:
- Assess customer contact experiences
- Take advantage of participatory observation
- Experience a service situation first-hand
- Monitor how processes and procedures are applied
These are undeniable advantages.
But there are others:
- Customers are often unable to remember details
- Customers don’t often give feedback
- Mystery customers are trained to be objective
- Mystery callers can gather factual information during a call
Do mystery callers resemble real customers?
The authors of the research conducted two studies in a call center. They, therefore, focus on mystery calling as a market research technique, leaving mystery shopping to one side.
The first research aimed to compare the results of customer satisfaction as assessed by real customers with the scores awarded by “mystery callers.” This assessment was conducted based on 3 criteria:
- Overall customer satisfaction
- Gratitude
- Word-of-mouth assessment based on the Net Promoter Score.
The second research aimed to research a group of mystery callers and determine the origin of differences in judgment.
There are significant differences between mystery callers and real customers
The first research delivered an unexpected result, uncovering significant differences between real customers and mystery callers regarding “gratitude” and “word-of-mouth.” Specifically, mystery callers systematically underestimated these factors. They were, therefore, harsher than real customers. Regarding word-of-mouth, the difference was relatively large (over ½ point on the NPS scale).
As far as measuring customer satisfaction is concerned, the research shows no significant differences. This means that mystery calling delivers results in line with what real customers think. This is good news.
Mystery callers differ from one to another
The 2nd research sought to understand whether the origin of the variations was within the mystery caller group itself. Here, too, the results were surprising. The following factors significantly influence the satisfaction scores given by the mystery callers:
- Gender
- Experience
- Age
- Whether the mystery callers were themselves customers of the company being assessed
When mystery callers were the company’s customers, they tended to be more positive in their judgment than their non-customer colleagues. If you’re running a mystery calling campaign, it’s a good idea to check this with the people making the calls.
Use mystery calls with discretion
Results show that mystery callers are biased regarding word-of-mouth and gratitude. As a result, the overall customer satisfaction score may need to be more representative. Remember that around 70% of companies that measure customer satisfaction extensively use NPS. This research shows a difference of almost half a point between real customers and mystery callers.
Our advice is to consider the various limitations of mystery calling when interpreting the results.
Posted in Marketing, Research.
