MaxDiff vs ranking questions vs pairwise comparison

When you want people to put a list of options in order, you have three common methods. A ranking question asks them to sort the whole list. MaxDiff shows a few options at a time and asks for the best and the worst. Pairwise comparison shows two at a time and asks for one. Each suits a different kind of list.

Ranking questions

A ranking question shows the full list and asks the person to drag the options into order, or to number them.

  • Strengths: easy to build in any survey tool, and the answer is a complete order from each person.
  • Weaknesses: long lists are hard work. Qualtrics writes that ranking lists longer than 10 quickly become meaningless. A MeasuringU study of ranking questions with 4 to 30 items found that each item added about five seconds to the first ranking question, and that how familiar and how alike the items are also matters.
  • Best for: short lists of text options, where people can hold every option in mind at once.

MaxDiff

MaxDiff, short for maximum difference scaling, shows a small set of options in each question and asks for the best and the worst. Sawtooth Software describes a typical study: 4 or 5 items per question, 8 to 15 questions per person, and about 15 to 40 items in total. The result is a score for each item on a common scale.

  • Strengths: handles long lists, and choosing best and worst is easier than sorting everything. Sawtooth says its scores separate items more clearly than rating scales do.
  • Weaknesses: you need a tool that builds the question sets and does the analysis, and each person answers many similar questions.
  • Best for: long lists of text items, such as features, messages or benefits, in a formal study.

Pairwise comparison

Pairwise comparison shows two options and asks for one. Repeated over many pairs, the picks turn into a ranking. Comparing every pair by hand is the paired comparison method. Tools can cut the number of pairs, for example with a knockout or a random sample of pairs.

  • Strengths: each choice is as simple as it gets, and it works when the options are images or videos. Two pictures or two clips side by side are easy to compare. Six at once are not.
  • Weaknesses: comparing every pair grows fast, 120 pairs for 16 options. Shortcuts such as a knockout give up a full order for each person.
  • Best for: visual options, quick decisions by a group, and lists where small differences matter.

Side by side

Ranking question MaxDiff Pairwise comparison
What a person sees The whole list A few options at a time Two options at a time
What they do Sort the list Pick the best and the worst Pick one
Effort per person Grows with each item Many short questions Many quick picks
What you get Each person’s full order A score for each item Wins or points for each item
Best for Short text lists Long text lists in a study Images, videos and close calls

Which one to pick

  • A short list of text options: a ranking question is quick to set up and easy to read.
  • A long list of text options, or a study you’ll analyse: MaxDiff, with a tool built for it.
  • Images or videos: pairwise comparison. People judge visuals best side by side.

Pairpoll does only the last one. You upload 2 to 16 images or videos, share one link, and each person picks the better of two until their top 1, 3 or 5 is settled. It doesn’t run MaxDiff or ranking questions and doesn’t take text options. How voting in pairs works explains its method. For text options with pairs, OpinionX offers pairwise ranking as well as MaxDiff.

All guides

Make your own poll

Free. Upload 2 to 16 images or videos and share one link. Voters don't sign up.

Create a poll

Pairpoll uses Google Analytics cookies to see which pages people visit, only if you accept. Details are on the privacy page.