The Wall Street Journal just revealed that Netflix may increase the size of its “Netflix Preview Club” to tens of thousands of test customers from 2,000 today. These privileged customers see the movies in preview. Their scores help to improve them substantially, like market research that would be conducted on a product still in the prototype stage.
This research aims to be as close as possible to subscribers’ expectations in a market where huge budgets ($14 billion for Netflix) are being swallowed up for titles with an increasingly short lifespan. This larger panel could allow Netflix to better segment its subscribers’ reactions and propose different versions according to their expectations.
Contact our market research firm
Netflix Preview Club: movies and series previewed but not discussed
As explained by a member of the Reddit forum, to be part of the Netflix Preview Club, you have to receive an invitation and sign an NDA (Non-Disclosure Agreement). In return, the members of the Netflix Preview Club have one week to watch the movie or series and give their opinion via a quantitative questionnaire. At this stage, the work is still being edited, and adjustments can be made to respond to the panelists’ criticisms.
This approach allows adjustments to be made to a prototype and thus increases its chances of success on the market. Amazon and Hulu propose similar initiatives. These are Amazon Preview and Hulu Brain Trust.
If this idea of using a panel to increase the chances of success of a television production may seem innovative, this practice has existed for over 50 years.
The idea of the panel for audiovisual production was born with Sesame Street
The idea of improving an audiovisual production by researching the reactions of its future viewers has been introduced previously. As early as the end of the 1960s, the producers of Sesame Street used “panels” of children to research their reactions, measure their attention, and improve each episode. Only episodes that captured more than 80% of the attention were retained. The CTW Research program was undoubtedly a pioneer in the quasi-scientific research of television pre-production.
This iterative, little Lean approach has helped Sesame Street episodes achieve 85% or higher attention rates. In other words, a child watching Sesame Street would only take his eyes off the screen for a maximum of 15% of the viewing time. Even as an adult, some sequences like the one above remain particularly enjoyable to watch. Let’s not sulk in our pleasure.
Other famous examples of viewing tests exist. For example, the movie “I am Legend” with Will Smith was changed because its final scene did not please the public who had seen it in previews. The video below shows the two alternative endings for this movie. The first one is the one that was retained, and the second one is the one that was rejected.
Digitization has enabled the industrialization of pre-tests
The difference between Sesame Street in the early 1960s and the Netflix Preview Club in 2022 is scale. Where Sesame Street episodes were tested using qualitative methods on a few dozen children at most, Netflix can count on millions of potential “guinea pigs.” Digitization has increased the pool of simultaneous testers and made the research results more reliable. Larger samples mean smaller margins of error. At the same time, the marginal cost of these tests is a fraction of what CTW was paying for developing Sesame Street.
This trend toward content optimization is not new. Advertisers use pre-and post-tests. The former is qualitative, the latter quantitative.
Website managers use A/B testing to determine the most effective design (UX). The effectiveness is then measured in different ways:
- Conversion for websites
- Bounce rate
Why is Netflix increasing the size of the Netflix Preview Club?
There are several reasons for the increased number of testers in the Netflix Preview Club:
Lower margin of error
With 2000 participants, the margin of error of a survey is 2.2%. With 10,000, it is only 0.98%. That one percent difference can mean much money in the race for ratings.
Low marginal cost
The second reason is that the marginal cost of growing to tens of thousands of members is negligible. Netflix Preview Club members don’t get paid and are excited to preview movies and give feedback.
You can bet that Netflix has automated the entire data collection (from the invitation to join the club to the analysis of the viewing data and the post-viewing survey). So, scaling up doesn’t cost anything extra and can only yield more responses and data for better decision-making.
Better segmentation for different versions of the same production?
Finally, increasing the number of testers could allow Netflix to better segment the results. By increasing the sample size, sub-segments could be identified that respond differently to the proposed content. This could prompt Netflix to propose different versions of the same content based on whether a subscriber belongs to one segment or another.
To conclude
The Netflix Preview Club allows Netflix to make its productions as close as possible to the audience’s expectations. This maximizes satisfaction and loyalty, which is particularly important in a subscription-based business model.
The shift from 2,000 to tens of thousands of testers revealed by the Wall Street Journal is undoubtedly motivated by an expected better accuracy of results and a low marginal cost. Moreover, this change of scale will allow us to identify populations of viewers who react differently. Why not propose versions of the film or series even more adapted to their expectations?
Posted in Strategy.