After the results of the 2016 presidential election came in, the first reaction of many people was that the polls were wrong. A more detailed analysis seems to show that the polls in 2016 were about as accurate (or inaccurate) as they usually are, but many people treated them as more precise than they really are.
I think the surprise (to many) election of Donald Trump serves as an important reminder of the limitations of survey research. Surveys aren't a precision instrument. That's partly because of inaccuracies and biases in sampling, but it's also because surveys are trying to measure opinions, and opinions are inherently fuzzy, malleable, and context-dependent.
In fact, given the limitations of surveys, it's remarkable that political polls are as accurate as they are. Predicting the outcome of an election is easily the most challenging application for a survey, given that you are trying to predict the future behavior of the general population, races are often decided by margins smaller than the margin of error, and you don't get credit for being close if you predicted the wrong winner.
This year's campaign should serve as an important reminder to be humble when interpreting survey results. A solid voice-of-the-customer program isn't as challenging as election forecasting, but customer surveys can still have important biases and inaccuracies. Keep in mind that:
- Low response rates mean that you are more likely to hear from customers with strong opinions.
- The survey process may be excluding some customer segments that have different experiences than the customers who can take the survey.
- If you're giving employees bonuses for hitting survey targets, they may be trying to manipulate the survey.
- Customers may be interpreting the survey questions differently than you intended.
Keep this in mind, and you will be less likely to make the mistake of being too confident when trying to understand what your customers are trying to tell you.