One of our clients is setting up a survey to calculate customer service scores for its employees, based on the answers to several different questions in the survey. These scores will be used for, among other things, calculating bonuses, so it's important to get them right.
There's a wrinkle, though. Not every question is asked on every survey, and the client (understandably) wants to exclude unanswered questions from score calculations completely--but that means there's more than one way to calculate the average score for each employee:
Calculate the number of points possible and the number of points earned on each individual survey, then calculate a percentage score for each individual survey, and average the percentage scores across all surveys.
This method weights each customer equally in the final score, no matter how many questions the customer answered.
Calculate the total number of points possible on all surveys and the total number of points earned on all surveys, the calculate an overall percentage score of possible points earned.
This method weights each question equally in the final score, so interactions where the customer gets a shorter survey form are weighted less.
It's important to clarify which method to use, since there's a large block of questions which will be asked on some surveys but not others.
To put this in concrete terms, let's suppose there's a sales rep, Joe, and only two survey questions, Q1 and Q2. All customers answer Q1, but only half the customers answer Q2. Joe gets 75% of possible points on Q1, but only 25% of possible points on Q2.
Using method (a), Joe's overall score is 62.5%.
Using method (b), Joe's overall score is 58.3%, because of the extra weight assigned to those customers who were asked Q2.
There are good reasons to use method (b): it's a simpler claculation, and the client may want to weight more complex customer interactions (which may also be the ones with the longer survey form) more heavily. In practical terms, it might not actually change scores much (depending on patterns in the data). On the other hand, I think most people who notice this subtlety would want to use method (a), even though (b) is the more likely default choice.