We talk a lot about the margin of error in a survey, as well as the sample bias inherent in the survey method. The former is the inherent uncertainty because you can't survey 100% of customers, and the latter is a systematic bias coming from the way participants are selected or the survey is administered.
There are lots of other ways surveys can be wrong, though. For any survey there's a certain percentage of surveys which don't get filled out correctly.
With a survey where the participant fills out the answers directly, mistakes include:
- Misunderstanding the question
- Understanding the question, but accidentally choosing the wrong option
- Being forced to answer a question which doesn't apply (if the survey forces a participant to answer a question to continue when the question doesn't apply, most people will just make something up)
These errors can be very difficult to quantify, since there's usually no (economical) way to go back to the participants and ask them what they really meant. As a rule of thumb, we generally figure that a couple percent of questions are answered incorrectly--and we work hard to try to craft a survey which minimizes participants' confusion.
When there's a live interviewer, you can get a better handle on whether a survey question is confusing or doesn't apply, but there are other kinds of mistakes:
- Accidentally marking a survey answer differently than the participant answered * Changing the wording of a survey question
- Prompting the participant for a particular answer
- Failing to clarify an ambiguous answer
If there's a recording of the interview (and there should always be a recording of the interview!), most of these mistakes can be caught by carefully reviewing the recording. One advantage of a live interview is that you actually can go back and find mistakes, rather than just having to guess. While it isn't practical to review every interview, by reviewing a sample of recordings and tracking the number of mistakes, you can get a handle on how often various kinds of mistakes are made.
You can never expect 100% perfection in a survey, but it's important to understand where there might be errors and what the limitations of the data are.