Introduction to A/B Testing

When & How did A/B testing started

A/B testing was dated back in BC, it’s mentioned in the bible. Back in the 1700s dutch people were going on big ships to the east and tried to understand what people got which disease or not. Of course, that doesn't make sense in the internet era, right? So, Internet came back in the early 90s and there is no concept of A/B testing on the web. But there used to be A/B testing in coupon codes in advertisements of local newspapers and phamplets. First browsers came back in 95 and still, A/B testing is non-existent. In’s the year the 2000s we were able to dig through log files of website and tried to understand behaviour. Do a couple of changes and observe again, whether if there is any shift in user behaviour. Here, we are comparing week 1 with week 2 or 3. In the mean time, external factors might change. So, it's still not perfect A/B testing. Later in the 2000s, we started doing meta redirects using javascript browsers. We aren’t using cookies. So, there is a 50% chance that you might end up in the wrong variation. Still, it is not perfect A/B testing.

When to use A/B testing?

We can use A/B-testing for several reasons, but mainly for two reasons.

  1. For new Deployments: If you are deploying something on your website, could be a new feature, could be an update or anything. You should deploy it as an experiment. Say, something like 5% of traffic will go to this new deployment at the start and slowly scale it up to 95% with A/B testing. We dos this because we wanted to make sure this new deployment does not have any negative impact on the main North star metric. If there is no impact, you can go ahead and deploy it. If there is a negative impact, you should go back to the drawing-room and relook at the customer research.
  2. Research: We will also use A/B testing for research. Imagine, you have a specific product page, with some description and buttons. You can do A/B testing by deleting few elements from the page. you wanted to know which elements make an impact and which does not. So, pick one area and start. Also, if social proof works on your website, make a fly-in stating, “15 people bought this in the last 24hrs.” This could be annoying, but it works. Sometimes they could even lower your conversions. So, this is not website optimization, here you are learning which motivation factors, impact the customer buying behaviour. This research is to learn, whether you have an impact or no impact or flat line.

How to determine, if you have the right amount of data for A/B test?

For determining if you have the right data for A/B testing we use something called as ROAR model.

ROAR Model



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sai Ananth

Sai Ananth

Read my thoughts on Happiness, Productivitry & Creativity