The Customer Service Survey

What's Wrong With the Rank Order Question?

by Peter Leppik on Tue, 2007-10-16 01:00

Time for a little descent into survey geekery, with a short rant about the rank order question.
What's a rank order question? It's a question where the survey participant is asked to rank a number of different choices from best to worst, for example:

Please rank these kinds of vegetables from 1 (favorite) to 4 (least favorite):

____ Carrots
____ Celery
____ Artichokes
____ Arugula

Rank order questions don't appear very often on customer service related surveys, but you do see them from time to time in market research. So why don't I like them?

1) They're more confusing than other kinds of survey questions.

2) They force the participant to make a choice even if s/he doesn't have an opinion (most rank order questions don't allow the participant to specify a "tie"), which can lead to the illusion of stronger data than what really exists.

3) They only work in writing: it is difficult to craft a rank order question which doesn't bog down hopelessly in an interview or (even worse!) an IVR survey.

4) A rank-order question actually provides less data than a series of Likert-scale questions (for example: How much do you like carrots? A lot / somewhat / a little / not at all), and the Likert-scale questions are easier to understand, quicker to complete, and translate better into different media.

So next time you're tempted to write a rank order question, try writing it as a series of Likert-scale questions instead. You'll probably be happier with the results.

Sorry...This form is closed to new submissions.