American Customer Satisfaction Index (ACSI) data for the retail industry was released this week, and the folks over at Consumerist noticed something, well, odd. Scores for Sears, JC Penny, and Macy's took huge leaps in 2016--despite the fact that those traditional department stores have been closing stores in the face of sustained sales declines and changing consumer tastes. In addition, Abercrombie & Fitch, a speciality clothing retailer, also posted a big ACSI gain despite struggling to actually sell stuff.
The idea that customer satisfaction scores would jump as the companies are losing customers seems counterintuitive to say the least. The ACSI analyst gamely suggested that shorter lines and less-crowded stores are leading to higher customer satisfaction and, yeah, I'm not buying it.
I'd like to suggest some alternate hypotheses to explain why these failing retailers are posting improved ACSI scores:
Theory 1: There's No There There
Before trying to explain why ACSI scores might be up, it's worth asking whether these companies' scores are actually improving, or whether it might just be a statistical blip.
Unfortunately, ACSI doesn't provide much help in trying to answer this question. In their report (at least the one you can download for free) there's no indication of margin of error or the statistical significance of any changes. They do disclose that a total of about 12,000 consumers completed their survey, but that's not helpful given that we don't know how many consumers answered the questions about Sears, JC Penny, etc.
With this kind of research there's always a temptation to exaggerate the statistical significance of any findings--after all, you don't want to go through all the effort just to publish a report that says, "nothing changed since last year." So I'm always skeptical when the report doesn't even make a passing reference to whether a change is meaningful or not.
It could be that these four companies saw big fluctuations in their scores simply because they don't have many customers anymore and the sample size for those companies is very small. There's nothing in the report to rule this possibility out.
Theory 2: Die-Hards Exit Last
We know that even though surveys like ACSI purport to collect opinions about specific aspects of customer satisfaction, consumer responses are strongly colored by their overall brand loyalty and affinity.
So as these shrinking brands lose customers, we expect the least-loyal customers to leave first. That means the remaining customers will, on the whole, be more loyal and more likely to give higher customer satisfaction scores than the ones who left.
In other words, these companies' survey scores are going up not because the customer experience is any better, but because only the true die-hard fans are still shopping there.
If this is the case, then the improved ACSI scores are real but not very helpful to the companies. They are circling the drain with an ever-smaller core group of more and more loyal customers.
This is a hard theory to test. If ACSI has longitudinal data (i.e. they survey some of the same customers each year) then it might be possible to tease out changes in customer populations from changes in customer experience.
Theory 3: ACSI Has Outlived its Usefulness
Finally, it's worth asking whether the ACSI is simply no longer relevant. The theory behind ACSI is that more-satisfied customers will lead to more customer loyalty and higher sales, all else being equal. But the details are important, and the specific methodology of ACSI was developed over 20 years ago, based on research originally done in the 1980's.
I know that ACSI has made some changes over the years (for example, they now collect data through email), but I don't know if they've evolved the survey questions and scoring to keep up with changes in customer expectations and technology. Back in 1994 when ACSI launched, not only did we not have Facebook and Twitter, but Amazon.com had only just been founded, and most people didn't even have access to the Internet.
So if the index hasn't kept up enough, it's possible that ACSI is putting too much weight on things that don't matter to a 21st century consumer, and missing new things that are important.
Interpreting Survey Data Is Hard
I'm only picking on ACSI because their report is fresh. The fact is that interpreting survey data is hard, and it's important to explore alternate explanations for the results. Even when the data perfectly fits your prior assumptions you may be missing something important without looking at competing theories.
It's entirely possible that ACSI did exactly that, tested all three of my alternate theories and others, and they have some internal data that supports their explanation that, "Fewer customers can lead to shorter lines, faster checkout, and more attention from the sales staff." But if they went through that analysis there's no evidence of it in their published report.
When the survey results are unexpected, you really need to explore what's going on and not just reach for the first explanation that's remotely plausible.