Guide · Measurement
A/B Testing for Websites: A Simple Plan Any Team Can Run
A/B testing is a way to compare two versions of a web page and see which one gets more of the visitors to do what you want. This guide gives a simple plan: pick one change, split your traffic, fix how long you will run the test, and read the result without fooling yourself.
What A/B testing is
You make two versions of a page. Version A is what you have now. Version B has one change. You show each version to a random half of your visitors. Then you count which one gets more of the action you care about, such as sales or form fills.
The idea is old. Harvard Business Review notes that the method is almost 100 years old. Jakob Nielsen wrote in 2005 that it had been a classic method in direct mail for decades, where firms split their mailing lists. He added that the web makes it easy to show different page versions to different visitors.
Some tests also change several things at once. That is called a multivariate test. This guide sticks to the plain two-version kind.
Sources: Nielsen Norman Group, Putting A/B Testing in Its Place, 2005; Harvard Business Review, A Refresher on A/B Testing, 2017
Pick one change and one goal
Start with one question. For example: will a shorter contact form get more enquiries? Write it down before you build anything.
Then pick one main goal. Nielsen says A/B testing works best when a project has one clear, all-important goal that a computer can count. Examples he gives are sales, newsletter sign-ups, and white paper downloads.
Change only one thing per test. If you swap the headline, the photo and the button together, you will not know which change made the difference.
You can add a few extra goals to watch for side effects. A test that lifts sign-ups but cuts purchases is not a win. Practical Ecommerce suggests tracking both your main goal and smaller steps, so you can spot unintended effects.
Sources: Nielsen Norman Group, Putting A/B Testing in Its Place, 2005; Practical Ecommerce, Google Optimize: Setting Up Experiments, 2017
Split the traffic and set up the test
Send half your visitors to A and half to B, at random. A visitor should keep seeing the same version on later visits. Nielsen notes that sites usually use a cookie for this, so people do not see the page flip back and forth.
In 2021, a small team can do this without a developer. Google Optimize has a free version. You create an experiment, build a variation in its editor, and choose an objective. Objectives can be a purchase, a completed contact form, or a goal you already set up in Google Analytics.
One limit to know about: in the free version, you pick your objectives before you start. You cannot add new ones while the test runs.
Before you go live, open the page on a phone and in a private browser window. Check that the variation shows up and that your goal is being counted.
Sources: Nielsen Norman Group, Putting A/B Testing in Its Place, 2005; Practical Ecommerce, Google Optimize: Setting Up Experiments, 2017
Decide how long to run it
This is where most teams go wrong. They check the dashboard every day. The moment it says the new version is winning, they stop.
Evan Miller explains why that fails. Significance numbers assume you fixed the sample size in advance. If you stop as soon as you see a win, you get false wins much more often. In his example, a test that you think has a 5 percent error rate can wrongly find a winner 26.1 percent of the time.
His advice is simple: decide on a sample size in advance, and do not stop early because the numbers look good. He offers a free sample size calculator, and a rule of thumb you can use if you like maths.
Some tools work differently. Practical Ecommerce notes that Google Optimize uses Bayesian statistics, so it does not ask for a fixed sample size. For those tools, the writer suggests running at least two business cycles. That gives people who start buying early time to finish. The same writer says not to go beyond four full weeks, because the same person can be counted more than once.
A practical rule for any tool: write down your end date before you start, and keep to it.
Sources: Evan Miller, How Not To Run an A/B Test, 2010; Practical Ecommerce, Google Optimize: Setting Up Experiments, 2017
Read the result honestly
When the test ends, ask three plain questions.
- Did B beat A on the goal you picked at the start?
- Is the gap big enough to matter to the business, not just to the chart?
- Did anything else get worse, such as another goal you tracked?
If there is no clear winner, that is a result too. It tells you the change did not matter much. Go back to your list and test a bolder idea.
Be careful about short-term numbers. Nielsen warns that a quick win can hide a slow loss. A cluttered page may sell more today and still make people less likely to come back. Look at overall results, not only the one item you promoted.
Also remember that A/B testing only works once a design is fully built. Nielsen suggests using quick paper prototypes and user tests first, then A/B testing the best idea at the end.
Sources: Nielsen Norman Group, Putting A/B Testing in Its Place, 2005; Evan Miller, How Not To Run an A/B Test, 2010
A one-page checklist
Print this and tape it to the wall before the next test.
Step What to write down Question One sentence: what do we think will change, and why Change One difference between A and B Main goal One action a computer can count Side goals One or two numbers that must not get worse Sample size or end date Fixed before launch, and no early stopping Check Both versions load on phone and desktop, and the goal is counted Decision Ship B, keep A, or test a bigger idea Teams that follow these steps learn something from every test, even the ones that do not win.
Sources: Nielsen Norman Group, Putting A/B Testing in Its Place, 2005; Evan Miller, How Not To Run an A/B Test, 2010; Practical Ecommerce, Google Optimize: Setting Up Experiments, 2017
Sources