The Customer Service Survey


by Peter Leppik on Mon, 2007-05-14 01:00

Survey analysis is an important step in going from raw data to improvements in service levels, and a key part of the analysis is looking for relationships in the data.

Most frequently, the relationships are in the form of correlations. A correlation simply means that X and Y tend to go together in the data: for example, customers who report that they didn't get what they needed during the call tend to have longer calls on average.

Correlations are never perfect, both because the relationships between different variables are very complicated, and because survey data is inherently noisy and imprecise. The quality of a correlation is typically measured as a "p" value, the probability that a given correlation between two variables is the result of random chance. By tradition, a p value of less than 0.05 is usually considered statistically significant.

This creates a problem, though, since automated data analysis makes is possible to test vast numbers of relationships in the data.

Our own analysis tool tests hundreds of thousands of possible correlations in a typical survey, which means that it's likely to "find" thousands of bogus correlations which meet the usual test for statistical significance. Because of that problem, we don't flag a correlation as "likely" unless the p value is less than 0.01, and we distinguish the relationships with p < 0.001 as especially likely.

We tend to find several classes of correlations:

1. Obvious and Trivial Correlations

The first group are the correlations which we expect to find, and which we'd be surprised to not find. For example, people tend to answer "satisfaction" and related questions very similarly: someone who is "very satisfied" with the call overall is likely to respond that he's "very satisfied" with the customer service representative, and "very likely" to remain a customer. At some level, these questions are all asking for the same thing, and customers often don't distinguish between them even when there are qualifiers attached (such as "how satisfied were you with the efficiency of the call?").

2. Interesting but Useless Correlations

We also uncover relationships which aren't necessarily obvious (and which can lend some insight into the data), but which are hard to do anything about. One common correlation is that when we're doing live interviews, the length of the interview call often correlates to the customer's satisfaction. At first this seems weird, until you listen to a few interview recordings and realize that when a customer has a bad experience he usually want to describe it at length. Good experiences, on the other hand, don't usually elicit such a response. This is an interesting result, but it doesn't help us much when we're looking to improve satisfaction.

3. Interesting and Useful Correlations

The really helpful stuff is in the relationships which aren't obvious and which point to ways to improve service levels. For example, when satisfaction is correlated to the type of call (sales vs. support vs. billing), there may be a problem with how certain calls are handled. Or when an expected relationship fails to materialize, that can also point to problems--such as when the customer's stated reason for calling doesn't correlate to the agent queue, that suggests that the IVR is doing a poor job of routing calls.

Sorry...This form is closed to new submissions.