Vocalabs Newsletter: Quality Times


In This Issue:

Keys to Survey Success

By Peter Leppik

Customer surveys are a staple of marketing and quality control. Four factors determine the success of any customer survey, and a well-designed survey will take all four into account:

1) How Many Customers you Survey

The number of survey responses determines the margin of sampling error, often just called the "margin of error" even though other factors like the sampling method can create large errors in the survey data. It takes hundreds of responses to get a reasonable sampling error (400 responses gives a 5% error), but there's a law of diminishing returns: to cut the sampling error in half, the number of responses needs to be quadrupled. The rule applies to subsets of the data, too, so while a survey with 1,600 responses has a sampling error of 1.25%, statistics calculated for the 400 California residents in the survey will have a 5% sampling error.

2) Which Customers you Survey

In most businesses it isn't practical to survey every single customer, so there has to be some method of choosing which customers to survey. If the method of choosing survey participants only selects satisfied customers for example, then the survey results won't paint a true picture of how customers feel. The most accurate method is to randomly choose from among all customers, but this is harder than it sounds because the time, place, and method of surveying will influence which customers can take the survey. For example, surveying at the end of a customer service call tends to exclude people who had a bad call, since those customers are likely to hang up early.

3) How you Survey the Customers

The mechanics of a survey can heavily influence the survey results, both by influencing who takes the survey, and also by influencing how customers answer the questions. Subtle biases in a question and the order in which options are presented will change survey answers, and words may mean different things to different people. The word "Satisfied" often means different things to a company and its customers. Social factors, like wanting to please the interviewer, not make a fuss, or not get an employee into trouble, may also influence survey answers.

4) What you Do With the Data

A survey, by itself, doesn't do much. The data has to be used in some way to promote the business goals of the company. The most effective method is to use the customer survey as part of an iterative process of looking for ways to improve, implementing changes, validating that the change had the desired effect, then looking for additional improvements. On the other hand, if the survey simply sits on the shelf, or gets a cursory review, then there may be little benefit, and the survey itself may lose credibility since the results don't seem to matter.


By Rick Rappe

The results of our June 2006 SectorPulse studies of mobile phone companies and financial services companies are in.

During the three months ending June 30, T-Mobile posted the best results, remaining at an "A" in Caller Satisfaction and a "B" in Call Completion, a measure of the percent of callers able to complete their business with a single phone call.

Over the nearly three years we have been tracking these companies, T-Mobile and Verizon have regularly traded first place position but this quarter Verizon Wireless dropped from an "A" to a "B" in Satisfaction while retaining a "B" in Completion.

In past studies, Cingular scores were affected by the merger with AT&T Wireless, but the study participants defining themselves as customers of AT&T has dwindled, and we have ceased to include those individuals. Cingular scores were noticeably improved earning "B"s in both categories and putting them in a second place tie with Verizon.

While all the carriers have shown substantial improvement over the last 11 quarters, Sprint remains in last place dropping from "B" to "C" in Satisfaction and retaining a "D" in Completion.

We note that the percent of calls handled entirely by IVR technology has increased among all carriers at the same time overall performance scores have risen, lending evidence to our belief that it is not self service technology consumers dislike, just poor technology that fails to meet their needs.

We have been tracking the performance of financial service companies and PayPal for some time as well. Unfortunately we have not seen any across the board statistically significant improvement like we have among the cell phone companies. Washington Mutual has consistently garnered the highest Satisfaction scores, and both PayPal and Wells Fargo have improved in overall caller satisfaction as well, but only enough to rise to the "B" level. All financial service companies we track have consistently scored poorly in single call completion.

In this most recent study, Washington Mutual, besides being the only entity that has measurably increased the percent of calls handled entirely without the need for a live service representative, also improved in single call completion, moving from a "D" to a "C". But while the decrease in overall satisfaction among Washington Mutual customers was slight, it was enough to cross the threshold from an "A" to "B" in Satisfaction. Thus, PayPal's improvements from "C"s to "B"s in both categories put it in the top spot for the first time.

Newsletter Archives