Preference testing vs A/B testing

Both tests compare versions of something, such as a design, an ad, a thumbnail or a headline. A preference test asks people which version they prefer. An A/B test shows each version to different people in real use and measures what they do. They answer different questions, and they work best one after the other.

What a preference test is

People see two or more versions and choose the one they prefer, usually in answer to a question such as “Which would you click?” You can run it before anything is built or published, with a mockup, an exported image or a rough cut of a video. It takes a few dozen people and a day or two.

A preference test measures what people say when they look on purpose. That is useful, but it is not the same as what they do.

What an A/B test is

An A/B test splits real traffic. Some people see version A, others see version B, and you compare a result such as clicks, sign-ups or sales. Ad platforms, website testing tools and YouTube’s test and compare for titles and thumbnails all work this way.

An A/B test measures behaviour, which is what you care about in the end. It needs the versions to be live, enough traffic to tell a real difference from chance, and time.

The difference at a glance

Preference test A/B test
Question it answers Which version do people prefer? Which version makes people act?
What it measures Stated preference Behaviour in real use
What you need The versions as images or videos, and a few dozen people Live versions and enough traffic
How many versions Many. Comparing in pairs keeps a long list manageable Usually two to four, because each one takes a share of the traffic
How long it takes Hours to a few days Days to weeks, depending on traffic
Main weakness People judge more carefully than they would while scrolling Weak versions get real traffic, and low traffic gives noisy results

When to use a preference test

  • Before you build or publish, while changes are still cheap.
  • When you have many options and want to drop the weak ones.
  • When you have no traffic yet: a new product, a new channel, a video that isn’t out.
  • When the choice is about first impressions, such as a thumbnail, a cover, a logo or a product photo.

When to use an A/B test

  • When the versions are live and you have enough traffic to split.
  • When the result you care about is an action: a click, a sign-up or a sale.
  • When people can’t predict their own behaviour, as with prices, form length or the order of a page.

How to use both

  1. Make several versions that differ in one thing, such as the image, the headline or the opening of a video.
  2. Run a preference test with people like your buyers or viewers, and drop the versions that lose.
  3. Run an A/B test on the two or three that remain.
  4. Keep the winner, note what made it win, and use that in the next round.

An A/B test splits traffic between versions, so every extra version needs more traffic and time. Cutting the list first means the live test spends traffic only on serious options. How to test ads before spending walks through this for ads.

If the two tests disagree, trust the A/B test for the result you measured. The preference test still tells you what people like when they look closely, and that is worth knowing for the next version.

Common mistakes

  • Treating a preference test as proof of behaviour. “I would click this” is not a click.
  • Calling a small lead a win. With few voters or little traffic, a narrow gap is often chance. How many voters do you need? shows how much.
  • Asking the wrong people. Your team knows the product and judges it differently from strangers.
  • Changing everything between versions. If the image, the copy and the layout all differ, a winner teaches you nothing you can reuse.

Running a preference test with Pairpoll

Pairpoll runs preference tests in pairs. Upload 2 to 16 images or videos, share one link, and each person picks the better of two until their top 1, 3 or 5 is settled. You get each voter’s list and an overall ranking. For videos, you can make each one play once with no replay, closer to how people meet a video in a feed. Voters need no account, and Pairpoll is free.

Pairpoll doesn’t run A/B tests on live traffic. Use your ad platform, your website testing tool or YouTube Studio for that step.

All guides

Make your own poll

Free. Upload 2 to 16 images or videos and share one link. Voters don't sign up.

Create a poll

Pairpoll uses Google Analytics cookies to see which pages people visit, only if you accept. Details are on the privacy page.