How To Do A/B Testing to Increase Conversion Rate?

A/B Testing is a must if you’re a startup. After all, you don’t have a tried-and-tested website model yet that produces a lot of conversions. You need to know what works and what doesn’t. A/B Testing gives you this exact information with hard statistical data. But do you know HOW to do A/B Testing yourself?

If you’re new, probably not. Don’t worry, I’ll walk you through the entire process in this article. 

If you’re not A/B testing, you’re doing conversion rate optimization (CRO) wrong. 

There you go. That’s the blunt truth. 

I have an article on increasing the conversion rate of your site that has 19 different tips. But if you don’t follow this one tip, i.e. a/b testing, you’ll be doing everything else wrong. 

A CRO tool performs the very important function of telling you where your conversion rate is dropping. A CRO tool does not tell you how to fix the problem. Why? Because there isn’t ONE way to fix the problem at all. 

There are no tried-and-tested formulas that are guaranteed to work. It’s all a process of trial and error. 

You need to see for yourself what works and what doesn’t. And this is exactly why you need the experimental method of a/b testing. 

Does a/b testing sound too scary and technical? Don’t worry, it’s not rocket science. You just have to do it right. 

What is A/B testing?

A/B testing is the process of showing two different versions of your webpage to two different segments of visitors. This experiment is done to see which version of the webpage is performing better. 

“Performing better” is defined in terms of business metrics. Statistical analysis of the data from a/b testing shows you which version gives you a greater conversion rate. 

The days of “I think this might work” are gone. A/B testing gives you hard data. Either it does work or it doesn’t. There is no guesswork required after a/b testing. 

Why do you NEED to do A/B testing?

Your site has hundreds of variables that you can tweak around. Should you spend all day wondering, “Maybe a green CTA button will work better than a red one”?

Obviously not. 

Should you trust the data that is already floating around the internet? 

Again, a big no.

What worked out for them might not work out for you, after all. Everyone has a different audience and your aim is to please YOUR audience. Who cares what anyone else thinks? Do an a/b test and figure out what your audience thinks. 

Also, here’s the tea: A lot of companies don’t perform a/b testing regularly. And of course, they suffer for it.

According to Econsultancy, the conversion rates of websites have experienced a consistent downwards spiral in the last 4 years. Only 22% of the companies are actually happy with their conversion rates. 

But you know which company DOES use a/b testing extensively? Google. Yes. Google ran 7000+ a/b tests in the year 2011 alone. Do I really have to elaborate upon how well it has worked out for them? Netflix, Ebay and Amazon are addicted to a/b testing as well. 

So, as you can see- the winners know the formula. The losers don’t. A/B testing is the key to CRO and you must run a significant number of tests to see the effects. 

Now, let’s get to the real question: 

How To Do A/B Testing to Increase Conversion Rate? 

The “How” of anything is always the difficult part. But I’ll try to simplify it for you. 

First, let me tell you the basics of the process, and then we’ll move over to the tools. 

Step 1: Figure Out If You Are Eligible

Not all websites can perform a/b testing. Why? Because this is a statistical process and you need a sufficient number of page visitors to be eligible. 

If you don’t have enough web traffic, your tests cannot reach statistical significance. 

What is statistical significance? 

Statistical significance determines whether your result is an effect of chance or of some factor of interest. 

Simply speaking: If you have very few people to test on, your sample might produce random results. These results might not be reflective of a larger pattern that you can base your further changes on. 

Let’s say you’re trying to figure out what kind of chocolate people like. And you can a group of only 10 people to test on. 8 of them like 99% pure cocoa chocolates. Should you start manufacturing thousands of chocolates based on that data? Of course not. Maybe those 8 people are just a fraction of your total buyers who got packed together coincidentally. Their preference doesn’t reflect the preference of your larger audience base at all. 

90-95% statistical significance is necessary to obtain reliable results from a/b testing. And for this, you need high traffic. 

So, how do you figure out if you have enough website traffic to obtain statistical significance? 

You have to find a sample size calculator for a/b testing. Optimizely’s Sample Size Calculator is a great option. It shows you whether or not you have enough website traffic to gain statistical significance. But additionally, it also shows you how much more traffic you need to get if you are not yet eligible. 

Step 2: Decide What To Test

As I’ve said before, there are a million variables that you can test. 

The layout, the design, the placement of the title, the wording- everything!

So, where should you start?

The first mistake that many make is not basing this decision on data. Every step of a/b testing is supposed to depend on statistical data- not mere guesswork. So, how can you expect reliable results if you have no data to base your tests on? 

Do you know which of your pages is performing the best? Do you know which is performing the worst? 

No? Then you should start with an analytics tool. Sure, Google Analytics will work. But you can also try out something like BigCommerce if you have an eCommerce site. Heatmapping tools like Hotjar also work in this regard. They show you the cold spots on your page that people don’t frequent. You can experiment with the layout of this page based on your heatmap data. 

Wrytx has an analytics tool that allows you to access all of these features. I’ll elaborate upon this when I start my discussion on a/b testing tools. 

The most obvious choice for a/b testing is the most visited page of your site. 

This is the one that draws in the most traffic. Hence, it’s crucial to optimize the performance of this page. 

This can be your Homepage or your About page. Since most people read these pages, you can expect the maximum number of conversions here. 

But you cannot reach higher than a certain point with these pages. On the other hand, focusing on the worst-performing pages can produce a drastic change in your conversion rate. 

You have no risk of screwing these pages up with changes. They cannot get any worse so they’ll probably get better. People have often seen 5-10x improvement in these low-performing pages after running a single a/b test. 

There are many other variables that you can and should test. I’ll discuss them in detail once I finish explaining how to do a/b testing.

Step 3: Create a Hypothesis

As any high school science teacher will tell you, creating a hypothesis is essential. Why? Because if you don’t remember what you set out to test, the results will be futile.

This hypothesis, as I mentioned in Step 2, should be based on data. Not on complete guesswork like “I’ll just tweak things around a bit and see what works.”

Hypotheses can be of various kinds. 

“A green CTA button will perform better than a red one.”

“An exit-pop will convert more than a scroll-triggered pop-up.”

“Long-form content will perform better than short-form content.”

It can be anything. The important part is to write it down before you start testing. Otherwise, you won’t know what you set out to test. 

This becomes especially difficult with split testing. 

Though a/b testing and split testing are often used interchangeably, they are not the same. 

In a/b testing, you change any one element. In split testing, you compare two completely different designs. 

This is why it’s often difficult to understand which specific factor fuelled the conversion in split testing. Hence, being armed with your hypothesis is especially necessary in this case. 

Step 4: Choose a Control and a Challenger. 

Again, high school science classes must have taught you what a control is. 

A control represents the base condition against which the challenger is supposed to perform. Your existing webpage can act as a control whereas your new webpage design can act as your challenger. 

Unless your page is performing terribly, no one pits one challenger against the other. If your home page is performing well enough, change one variable and create the challenger. Don’t shuffle everything around needlessly. 

For example, let’s say your hypothesis is: “An exit pop-up will increase the conversion rate.”

Then, your control will be your original page without the exit pop-up. And, your challenger will be your page with the added exit-pop up. 

Step 5: Avoid Sample Pollution

This is probably the most important step of all because this is what people mess up the most. 

If you’re using an a/b testing software, the sampling process is mostly taken out of your hands. This is a good thing because you don’t run the risk of choosing biased samples. 

However, biased samples can be accidentally chosen based on your test running time. 

For example, if you run the a/b test solely on the weekdays, you’re missing on out the weekend trend. If you run the a/b test during the holiday season, you’re missing out on the year-wide trend. And so on and so forth. 

But these are not the only ways in which you can pollute a sample. 

Samples can be polluted by internal AND external reasons. 

Say your competitor’s website has a sale going on when you’re running the test. All the customers are going to rush to that site and bring down your conversion rate. Does this reflect the general trend of your site? Of course not. 

The same problem occurs when you’re having a sale while running the a/b test. Your conversion rate is going to be higher than general. This won’t be reflective of your conversions on a regular day either. 

Another popular mistake that people make is counting the same visitor multiple times. Make sure whichever software you are using can pick out unique users instead of counting the number of visits.

There are some kinds of sample pollutions that are harder to avoid than others. For example, cross-browser or cross-device sample pollution is almost impossible to avoid. 

Your aim should be to conduct the tests for no less than 2 weeks and no more than 4 weeks. And you should conduct multiple tests in a year so that you don’t get swayed by the seasonal drifts.

Step 6: Reach Statistical Significance 

I’ve already mentioned this in step 1 but I must reiterate this. 

Statistical Significance is crucial. 

Most a/b testing software will only stop testing once they’ve reached your specified level of statistical significance. 

90-95% is what I suggested. But does that mean you cannot stop at 70%? Wouldn’t that be faster? 

It will be and you can stop here but you shouldn’t. I’ll explain why. 

Stopping at a 70% statistical significance is like being satisfied with 70% certainty. 

“I’m 70% certain that making this major change to our site will work.” Have you ever heard say something like that? No, right? This is why you cannot stop at 70. 

“But what is it is a really teeny-tiny change?” you might ask. “Do I still need to wait for 95% certainty?”

Yes, you do. 

One popular misconception is this. If you’re making a tiny change, statistical significance does not matter as much. That’s actually the exact opposite of the truth. 

If you’re making a huge change, even a lower statistical significance will show you the overall trend. 

If you’re making a tiny change, it’ll take way more weight to sway the scales. 

Get it now? The more minuscule the change, the higher the statistical significance you need to reach.

Step 7: Analyse Your Results

This is the very last step. 

Look at the results that you have obtained and figure out if your hypothesis was correct. If it was, replace the control with the challenger. If it wasn’t, keep the control and come up with a new hypothesis. 

Remember this statistic. Only 1 out of 8 a/b tests show any significant change in your conversion rate. You need to keep running tests to get results. 

However, don’t make the mistake of running more than one a/b test at once. Even if your conversion rate increases, you’ll have no idea which experiment fuelled it. 

What Are The Best A/B Testing Software?

Optimizely

Experimentation and Personalization are Optimizely’s keywords. 

Optimizely is the main player in town when it comes to a/b testing software. Why? Because it’s extremely user-friendly. Anyone can use it, even beginners. 

As I’ve told you, Optimizely holds your hand from the very first step and guides you through the process. If you’re serious about experimentation, Optimizely is for you. From Microsoft to IBM to Ebay, everyone uses Optimizely. 

It’s super easy to run multiple tests on the same page with Optimizely. You can even run tests on the mobile interface. 

However, Optimizely is expensive. Really expensive. This is why you should check out the next few alternatives. 

VWO

This is an Indian company and hence you should totally give it a try. Sorry, but had to specify that detail.

VWO focuses mainly on graphics and design. This is why their Visual Editor is the most popular feature. Through VWO, you can perform a/b testing, multivariate testing, and split testing. 

Though it’s less expensive than Optimizely, prices tend to rise with the rise in website traffic. However, if graphics are your main focus- do go for the 30-day free trial period. 

AB Tasty

The specialty of this one is that it allows funnel testing along with a/b, split and multivariate testing. However, this tool is not really for beginners. 

If you are an enterprise marketer, AB Tasty is for you. It is more affordable than Optimizely but the actual pricing is only available on request. 

Google Optimize

If you’re a beginner, you’re most probably looking for a free a/b testing tool. 

There’s no point in singing Optimzely’s praises to you because it’s most likely out of your budget. 

This is why I bring to you- Google Optimize. 

Just like Google Analytics, this is free. It lets you do all of the basic tests. A/B, split, multivariate- you can do it all here. 

However, if you want to run multiple tests at once or make multiple personalizations- switch to Google Optimize 360. This is the paid version but it allows you to do a lot more. With this, you can run 100+ experiments at once and test up to 36 combinations in the multivariate test. 

Just one problem, Google Optimize 360 costs a LOT more than the free  Google Optimize. 

This is why I’d like to suggest the next option. 

Wrytx

Wrytx analytics lets you do heatmap tracking, record visitor sessions, and do a/b testing all at once. 

The best part? It is well and truly FREE FOREVER for beginners. 

And as you can see from the payment plan below, there’s also a paid plan available for larger businesses

Best part? There’s a 7-day No Questions Asked Money Back Guarantee. Give it a try. If you love it, great! If you don’t, there’s nothing to lose! 

Which Variables Should You A/B Test?

Headline and Subject Line

The first thing anyone notices about your site is the headline. If you get this wrong, this will also end up being the last thing that they notice. 

The top of the sales funnel is a very open space that lets multiple potential customers escape. Hence, you should have the right hook to hold on to them. 

You already know the basics- use keywords, pay attention to SEO, etc. But everyone else does those too. What sets you apart? It can be something as trivial as the font or as essential as the wording. 

Why do so many online headline generators exist? Because we aren’t always capable of coming up with the perfect headlines by ourselves. However, what if you have more than one headline that is competing for the top spot? What if you want to see whether changing the existing headline can get you more clicks? 

You should run these through the a/b test. 

The same is valid for email subject lines. Whether or not someone will click on the email depends completely on the subject line. Open rates of emails range from 25-47%. So, say you get lucky and achieve the very highest percentage of open rate on this scale. You still can reach out to less than 50% of your audience. 

This is very getting the email subject line right is crucial. 

Now, I can give you super generic advice on “Don’t use clickbaity subject lines” etc. But that’s not really going to help you. You’ll never know what works and what doesn’t till you actually try it out. 

This, for example, is definitely clickbaity. Especially because when I opened it I realized it was just a blatant ad with no discounts or anything. But I did click, didn’t I? 

That’s exactly the thing. Unless you a/b test your email subject line, you won’t know what actually appeals to people. What really intrigues them. 

Also, very importantly, do a/b test your sub-headline too. This is also visible at first glance. And this gives you an opportunity to pique the audience’s interest further.

Content Depth

Long-form or short-form. What works?

Both. Neither. 

I’m not trying to spin riddles here. That’s the actual answer. There’s no steady formula for what works. An in-depth article or an infographic. 

Certain people prefer to skim through the article quickly. If they find a hugely long-form article that can be scrolled down for miles, they’ll bounce. 

On the other hand, there can be people who actually want in-depth information. Surface-level infographics are a complete waste of time for them. If they see a 500-word article, they’ll bounce. 

How to solve this problem?

Figure out which kind makes up the majority of your audience base. There’s no other way to do this besides a/b testing. Sure, you can slap on a quiz through a pop-up. But often people themselves don’t know what they’ll prefer till they are presented with the actual options. 

Also, you cannot take the results of one a/b test and keep applying it for years on end. Why? Because separate topics require separate content depths. If they already know about something in detail, they might want a simple info-graphic checklist. If they are trying to learn about something in detail, they will want a deep dive. 

You can never assume what people might want. You have to put your content through the test. Write one long-form and one short-form article. Put it through the a/b test and see which one is performing better. 

Only then can you understand what your audience really wants. 

Navigation

If there’s a problem with your website’s navigation, you most probably won’t even realize it. You’ll just see the conversion rate dropping silently. 

There’s a very simple reason for this. If you live in a maze-like mansion, you yourself will be too accustomed to it to know it’s a maze. You’ll always know the exact route to the kitchen. 

But what happens when a guest stays over and gets lost? That’s exactly what faulty navigation can do to your website. Turn your website into a maze.

Your site must have a clear path that even newcomers can walk through. 

Your homepage is the drawing-room. This is where your guests get seated before they can make a choice to venture off elsewhere. You must ensure that your homepage links out to all your other pages. And not just this, there must be alleys of linkages between the other pages as well. 

Internal linking is something that Wire does pretty well, I think. It carefully places the links of other related articles within the post that you are already reading. Since these topics are related, there’s a good chance that you’ll click on the other link too. 

The above internal link was provided within an article on the caste-based labor in Rajasthan’s prisons. Completely interlinked. 

But how did they know this is the optimal position of the internal link? How did they know exactly which article should be interlinked to which other? You guessed it- a/b testing. 

If you’re running an eCommerce site, navigation is of crucial importance to you. There are thousands of items listed on your site. If your user cannot find them, they cannot buy them. 

Myntra has this wonderful drop-down menu that collates every kind of saleable item that they have. 

By placing everything in neat buckets, it makes everything easily available to the buyer. The buyer no longer has to go scavenging for what they want. 

But how can you know the optimal placement of all these items? How can you know which filters or recommendations you should place around an item?

A/b testing, yes. 

Also, a/b testing is not only important for CRO, it is super important for marketing as well. I’ve already spoken of testing subject headlines of marketing emails earlier. You should also a/b test your Facebook and other social media ads to see which ones are working.

CTA button

There’s no way to figure out “Which is the best kind of CTA button?” besides a/b testing. 

The Call-to-Action button is important. It’s very, very important. This is what you want your audience to see. It is what you want your audience to click. This is what is going to complete the conversion process. 

So, obviously, there are a lot of questions surrounding this precious button. 

What colour should the CTA button be? Should it be large? If so, how large should it be? And most importantly, where and how often should it be placed? 

You’re going to get a million tips about this on the internet. Red works, green doesn’t. Big works, small doesn’t. 

It’s futile to base such an important decision on hearsay. This is exactly why I won’t tell you how your CTA button should look like. 

YOU figure it out through an a/b test. First, pick out the top options and then test them out. No one can know your audience better than you can. 

Social Proof

Testimonials and reviews are essential to maintain the credibility of a website. However, that doesn’t answer all the other questions that follow. 

How many testimonials should you display? 

Where should you display them? 

How should you display them? 

How you decide to show off your testimonials matters. Why? Because you don’t want to overshadow the features of your actual product with an overwhelming bulk of testimonials. That actually undermines the value of your product and your testimonials both. 

Some companies have a revolving banner of testimonials. Other companies just have a board displaying the logos of their important clients. Some companies get creative like this: 

Point is, there are multiple routes that you can take. Which one should you choose? 

See what works on your customers of course. Just because you find an idea cool, doesn’t mean everyone else will. So test it out! 

Design and Layout

As I’ve already mentioned, VWO specializes in this. Their Visual Editor is extremely popular. 

I probably didn’t need to tell you about this one because it’s pretty obvious. Design and layout are two things that you cannot use any guesswork with- at all. 

Often, we face the problem of adding too much information to the homepage. The reason is pretty self-evident. We get the maximum number of visitors here. So, we don’t want them to miss out on anything. 

However, as any high school English teacher will tell you- editing is the key. You have to edit out some information to prevent your homepage from looking cluttered. 

So what should you delete? And how can you be sure that you’re not losing customers for this change? 

A/B testing is the answer. 

Let your audience choose which display they want to see. Let them choose which layout they’re the most comfortable with. 

One test obviously won’t cut it for these kinds of variables. You need to keep a/b testing to see how close to perfect you can get.

According to Brian Dean of Backlinko, you should go big when you’re making these changes. Your challenger can be drastically different from your control. Once you figure out which version performs better, you can start tweaking the minor details. 

Pop-Up Placement

Entry pop-ups are bad, they’ll tell you. They annoy the audience into leaving, they’ll reason. 

Then why, pray tell, are squeeze pages glorified? Why do landing pages often look like glorified entry pop-ups? And most importantly, why do so many successful businesses use entry pop-ups? 

If they really didn’t work, they’d go out of fashion, right?

The difference between “good” and “bad” pop-ups is hard to pin down. 

A good pop-up is not meant to annoy you. But that would disqualify every other pop-up besides the exit-intent pop-ups. After all, all other kinds of pop-ups interrupt your browsing session, don’t they? 

See, again- there is no golden rule with regards to pop-ups. Something might or might not work depending upon the audience, the content, the discount offer- anything. 

This is why you’ve got to perform the a/b test to understand if a pop-up is good or bad. 

After all, who can predict what annoys the audience more than the audience themselves? 

In Conclusion

The primary concern of a/b testing is doing away with baseless guesswork. Yes, there is still guesswork involved when you’re trying to create a hypothesis. But a/b testing takes away the element of uncertainty from the final step of the process. 

There’s no “maybe” about the situation anymore after the statistical results of your a/b test are in your hands. Your challenger either works or it doesn’t. If it doesn’t you can move on to other experiments. If it does, congratulations. Now you need to keep improving. 

So, there’s no point in asking “When to use A/B Testing?” The answer is: always. A/B testing is based on data science. And data always changes. If your last A/B test was performed in 2017, that data no longer has any value.

A/B Testing is an essential process that you NEED to incorporate into your conversion rate optimization routine. Hopefully, by now, you have an idea of how you can do a/b testing yourself. You also know the most popular a/b testing software and the variables that you can a/b test. 

So, are you prepared to conduct the experiment now? 

If you have any further questions, do let me know in the comments below!

Frequently Asked Questions