AddThis Academy

An ever-growing library of resources to help you become a better online marketer.

How to Successfully Run Your First Website A/B Test [WEBINAR RECAP]

Update: As of June, 2019, AddThis has retired the A/B Testing feature. If you would like some more information you can find it here.

You’ve probably heard the term “A/B test” several times during marketing discussions, but do you know how to use them to drive success?

In our recent webinar “How to Successfully Run Your First Website A/B Test,” we shared the complete run-down on how to make A/B testing work for you. Thue Madsen, the Marketing Operations Manager of Kissmetrics, dove deep into the inner workings of A/B testing and showed our attendees that the proof is in the pudding.

What is A/B Testing?

If you’re unfamiliar with A/B testing, no worries — we’ll explain. A/B testing, simply put, is a method for discovering the best online marketing strategies for your business. It involves taking two versions or portions of a landing page, email etc., and pitting them against each other to determine which one performs better.

In A/B testing, data enables you to evaluate design, copy changes or different calls-to-action to improve your conversion rate and conversion funnel. Whichever of the two variants (A or B) receives better results—potentially in the form of click-through rates or conversions—is the winner.

Which variants you put to the test will depend on your goals. For example, if your goal is to increase email collection, you might test the types of fields in a signup form, display of privacy policy, your call-to-action, and other dimensions. If your goal is to see which headline on a blog post receives a better click-through rate, you can test two versions of a headline to identify which one draws a larger audience.

The ultimate goal of A/B testing is to figure out what gets the most visitors to click or convert, or what deters them. As long as you’re testing the appropriate elements, you’ll receive the right data and you’ll be able to improve your click-through or conversion rates.

How Does it Work?

A/B testing starts with deciding what to test. The control, or “Version A” of your test is what you normally use as your landing page, email, call-to-action, etc. The variation, or “Version B”, has the changes you’re testing.

ab test control example

It might be tempting to peek at the data during the test and jump to conclusions, but it’s important to wait until you receive statistical significance before making any decisions. After you’ve collected enough data, you’ll learn whether the variation you tested performed better or worse than the control.

You can also use multivariate testing to determine which variation of several performs the best. Multivariate testing uses the same core mechanism as A/B testing, but compares a higher number of variables. As in an A/B test, traffic to a page is split between different versions of the design.

multi variate ab testing

Multivariate testing is a great option to help you redesign and test separate features of a web page at once to ensure the highest impact. This is especially useful when building landing pages for marketing campaigns, as the data collected on a design element’s effectiveness can be applied to future campaigns.

Why Use A/B Testing?

When you use A/B testing your decisions are based on data rather than guesswork. Rather than wondering which color, design, copy, or signup form will work best, A/B testing provides clear answers. By gathering data from users, you’ll continually learn more about what people want from your website.

The Basics of a Successful Testing Program

Now that you understand the purpose of A/B testing, it’s time to do your homework. Research is a very important step. Learn as much as you possibly can about A/B testing, including what elements to test in order to see the greatest improvements for your particular goals.

After you’ve researched, but before building your test, form a hypothesis. For example: “By doing X, Y will happen, because of Z.” In other words, you’re conjecturing, “I think that by changing A to B, it will result in this impact.” By performing the A/B test, you can examine to what extent your assumptions were on the money: whether the variant had the impact you expected. Plus, you gain insight on the preferences of your target audience.

After you run a test, dive into the data and analyze your results to find your biggest areas of opportunity. If you notice instances where people aren’t clicking or converting, diagnose the problem. What’s stopping them? Is the call to action strong? Are the visuals clear? Are there any technical problems? These are all potential opportunities for improvement.

You can also further investigate to gain a better understanding by surveying users. Ask them to explain what’s preventing them from converting. You’ll get direct results and a better understanding of how to correct the problem. The data you receive from surveys like the one below will lead you in the right direction.

user survey example

Building the Test

If you’re A/B testing portions of your website, Optimizely can help you to experiment with different variations of copy and design.

optimizely ab test example

Here are some things to consider when building your test:

  • Start small with your testing. Identify an element that won’t be too complicated to change and measure. Make sure the element you choose has the potential for improving conversion rates.
  • Think outside the box—if visitors aren’t clicking “Call us today!,” the answer isn’t always to bold the font and add fire around it.
  • Get an outside opinion. As the designer of the page, you might not notice what someone else would consider a flaw.

When your test is officially launched, you’ll be able to find out how your control plays against your variation. Now it’s time to see what really works.

Document and Share Your Findings

It’s important to document your findings during your preparation, launch, and final results. Besides preventing you from spinning your wheels and forgetting what features fail and what works, your documentation allows you to easily share what learnings are available from current and past A/B tests so your team doesn’t repeat old mistakes.

Wash, Rinse, Repeat

After you’ve analyzed your test results and you know what changes to implement, you’ll want to repeat the process as often as possible. Why? Because a website is never 100% perfect. The best thing you can do is to keep improving step-by-step. Not all tests are winners, either. The more learnings you can gather, the higher the chances of significantly improving your conversions.

Tooltips and Burgers

Choosing the right optimization tools are kind of like making your own burger. We all understand that not everyone likes their burger the same way. We want our burger made only with the ingredients we find delicious, appetizing, and savory. (I promise this relates back to optimization.)

Think of it this way: when looking for the right platform for analytics, surveys, testing, and others, you’re going to select from a variety. Which tools does your company need? Maybe you’d add the company Wiki but ditch the micro surveys. It’s okay to ‘have it your way.’

Thue’s optimization stack consists of an analytics tool to get insights on how his customers behave, user surveys to collect feedback, a company wiki to document his process and learnings, and a testing platform to build tests and collect data.


Once you get started, don’t quit A/B testing if you don’t see significant results right away. Test the elements that have the biggest impact on ROI before testing all of the finer details of your web site. Done right, A/B testing and conversion optimization will improve the ROI of your web site.

A/B Testing with AddThis

AddThis also offers A/B testing capabilities with our Audience Targeting tool. This feature enables you to compare two or more Audience Targeting rules to see which one yields the greatest increase in conversions, whether your goal is collecting emails, getting social shares, or driving traffic to a specific link or page on your website. Learn more about A/B testing with AddThis Audience Targeting and start improving your website’s performance today.

Last modified:  July 23rd, 2019