Blog Archives

June 2, 2015 - No Comments!

The 9 steps to a successful A/B test.

Anyone can run an A/B test. Start your A/B test software, build a test and launch it. The result? Probably bad.
Running a successful test takes time and preparation. Follow the following steps and let your tests be a success.

1. Research

Research is the most important aspect of A/B testing. You need to know what to test. Testing random changes is a waste of time and money. The idea behind your A/B test is very important, you need to learn from your tests. This is why you should research before testing. Find the problems on your website, get to know your customers and talk to real people. At the end of the research you'll have a list of "leaks" of your website, which will be needed for the next step.

2. Hypothesis

The next step is to turn your problems into hypothesis. Ask yourself, why are these problems here? Why do my visitors leave my website on this page? Turning your problems into hypothesis will enable you to learn from your A/B tests. Raising your conversionrate is fun, but won't teach you anything.

3. Prioritizing

After putting together your hypothesis, it's time to prioritize them. Not every hypothesis is as useful or easy to test. Many experts us the PIE method to do this. An example of PIE is shown below.


PIE is a shortcut for Potential, Importance, Ease. Each of these three criteria gets a number appointed between 1 and 10, where 10 is the highest.  The higher the grade, the more important it is.  An hypothesis could get the following grades: P=7, I=8 and E=1. This means that the  hypothesis is very potential and important, but hard to develop as A/B test. By counting the total and dividing it by 3, you get the PIE score. The higher the score, the better the hypotheses is. These hypothesis should be tested first.

4. Designing

After finding your best hypothesis, it's time to design your tests. You know the problem and how to solve it, but how is the solution going to look like? Your test can be as easy as changing a few elements, or as hard as adding a countdown or changing form fields. This step is very important, because solving the problem with the wrong visual elements can lead to a unsuccessful test. So put a lot of thought in how your A/B test variants are going to look like.

5. Test development

After finishing the previous four steps, it's finally time to open your A/B Test software and start building your test. Most changes can be done using the visual editor, but your solution can also be hard to implement. If this is the case, you probably need a jQuery developer who can manually code your test variants. Finding these developers can be hard and long task. If you find yourself in this position, check out my startup I launched this week. Validfit turns your A/B test ideas into code.

6. Setting your goals

To measure how well your A/B test is doing, you need to set goals. These goals can be the number of forms filled in or people reaching your thanks you page, but also time on site, reaching the bottom of a page or the time a video has been watched. During and at the end of the test, these goals will give you information about how well your A/B test has performed.

7. Running the test

After completing all above steps, it's time to set your test live. Normally you'd set your traffic allocation to 100%, but most experts don't do this. They set the traffic to 10% of the total, which means that 90% of the visitors see the normal variant and only 10% the test variant. This trick could save you a lot of money. Should there be any problems with your A/B test, than only 10% of your visitors would see the broken test, instead of half of your visitors. When, after two to three days no problems occur, it's time to set your traffic to the normal values.

8. Ending the test

Every test has an end. There area few important conditions that need to be met, before you can end your test successfully:

  1. More than 95% significant.
  2. More than 150 conversion per variant.
  3. Ran a full business cycle.

Not meeting these conditions will results in bad data and will lead to winning variants, that actually or not the real winners.

9. Analyzing the results

The last step is analyzing your test results. It's easy to see which variant has won, but the most important step is to understand why it has won. Doing this will answer your hypothesis and result in learning about your customers. These learning can be used in the following A/B test to even get more successful results.


Published by: admin in A/B Testing

Leave a Reply