Subscribe to Internet Marketing Secrets | Article Marketing | SEO Tips | Affiliate Marketing |Email MarketingNews FeedSubscribe to Internet Marketing Secrets | Article Marketing | SEO Tips | Affiliate Marketing |Email MarketingComments

Browse > Home / Archive: April 2009

An Introduction to A/B Testing and Multivariant testing

April 29, 2009 by  
Filed under Internet Marketing

Leave a Comment

Design is very subjective in the print world and on the web.  Fortunately, internet marketing folks have it easier than print designers because they have the ability to test web pages they publish on the internet.  

A/B testing and multivariant testing is one of the primary tools in any data-driven environment.  You can think of it as a big cage match. Send in your champion versus several other challengers and out comes a victor.  Of course, on the web there’s less blood and more statistics, but the principle remains the same: how do you know who will win unless you force them to fight to the death?

A/B Testing lets you compare several alternate versions of the same web page simultaneously and see which produces the best outcome, e.g., increased click-through, engagement, or any other metric of your choice.

Ok, What is A/B Testing, Really?

A/B Testing is a way of conducting an experiment where you compare a control group to the performance of one or more test groups by randomly assigning each group a specific single-variable treatment.  Let’s break this down.

First, you decide on an experiment.  Maybe you’re building a web application that forces users to register and you want to experiment on your landing page.  You want to see if you can improve the percentage of people who register.

For example, if 1000 people visit your landing page today and 200 of those people register then you have a conversion rate of 20%.  All else being equal, the landing page with the higher conversion rate is better.

Building Treatments

Once you know what you want to test you have to create treatments to test it.  One of the treatments will be the control, i.e., your current landing page.  The other treatments will be variations on that.  Here are some things worth testing:

  • Layout. Move the registration forms around. Add fields, remove fields.
  • Headings. Add headings. Make them different colors. Change the copy.
  • Copy. Change the size, color, placement, and content of any text you have on the page.

You can have as many treatments as you want, but you get better data more quickly with fewer treatments.  I rarely conduct A/B tests with more than four treatments.

Randomization Means Control

You can’t just throw up one landing page on Tuesday and another landing page on Thursday and compare the conversion rates — there’s no reason to believe that the conversion rate for users who visit on a Tuesday is the same for users who visit on a Thursday.  In fact, they’re probably not.

A/B testing solves this by running the experiment in parallel and randomly assigning a treatment each person who visits.  This controls for any time-sensitive variables and distributes the population proportionally across the treatments..

An Example

Say we are testing mortgage loan messages and we’re conducting an experiment on our landing page.  Our goal is to improve the conversion rate by at least 10%. When a new visitor arrives on the landing page we randomly assign them one of three treatments: the control, Treatment A, or Treatment B.

Let’s also say these treatments involve the headline copy.  For example, the control treatment’s headline copy might be “Record low mortgage rates. Apply Now.” One of the experimental treatments might have “4.3% 30 yr morgage rates— click here.”

You run the experiment for a few days and get the following data:

A/B Testing Example Data for the Mortgage Loan message
Treatment Visitors Loan Applications Conversion Rate
Control 1,339 328 24.49%
Treatment A 1,428 359 25.01%
Treatment B 1,377 432 31.37%

From the data above you’d conclude that Treatment B is the winner, but you have to be careful — if the conversion rates were closer or if your sample size were smaller you wouldn’t be able to tell which treatment won.  For example, can you say for certain that Treatment A is better than the control treatment, or could it just be due to chance?

The bottom line is this, multi variant testing is important, and it can positively impact your bottom line.  If you’re a small business owner or large company not taking action, this is the time to knock out your competitors and grow your business.   Take the results up above. If you sell insurance products, loans, or just collect leads off the internet, wouldn’t you like to increase your conversion rate by 6%?  You decide…

Landing Page Optimization Resources

Interested in landing page optimization?

This directory is meant as a comprehensive guide to companies, products, and services that can help you.  If you feel like we’ve left off a web site resource, please leave the URL in the comment section below.

Conversion Testing Companies – These companies focus primarily on conversion rate improvement through testing of alternative website designs.

  • SiteTuners – Specializes in very-large-scale testing. As Google Website Optimizer Authorized Consultant can implement A-B Splits or Multivariate/Taguchi testing. Larger tests are done with proprietary TuningEngineSM technology. Choice of flat fee or pure performance based payment options.
  • Kefta – Recently acquired by Acxiom Corporation (Nasdaq: ACXM). Kefta focuses on testing an personalized content delivery for Global 2000 (very large) enterprises.
  • Optimost – Uses A-B split and Multivariate/ Taguchi testing. Makes unrealistic claims about possible test sizes on their website (“billions” of possible versions in a multivariate test). Charges by getting a client into an ongoing subscription of tests. Recently acquired by Interwoven.
  • Omniture Test&Target – Focus is on larger e-tailers. Charges upfront fees and then per-page for content served even after the test is completed. Recently acquired by Omniture.
  • Conversion Rate Experts – U.K. based conversion consulting company. Google Website Optimizer Authorized Consultants.
  • TaguchiNow – Site seems very hype-heavy and talks about some special secret about how to make Multivariate/Taguchi testing work for online marketing – there is no secret.
  • Widemile – Taguchi testing for interactive agencies.
  • Maxymiser – UK-based testing company.

Conversion Testing Tools – These companies primarily provide tools for conversion testing.

  • OnDialog – Visual creation of landing pages with SiteTuners TuningEngineSM automated testing system built in.
  • Google Website Optimizer – FREE A-B Split and Mutlivariate tool available to all AdWords users in their account.
  • SiteSpect – Testing tools intercept traffic after it leaves the site and shows the end-user the correct content. Does not require I.T. support or changes to your original pages. Designed for larger companies.
  • Verster – A-B Split and Multivariate testing tool provider – full-service engagements are also available.
  • Hiconversion
  • SplitAnalyzer

Targeting & Personalization – Companies that improve conversions via dynamic personalization of individual visitor content.

Information & Resources – These websites provide articles, blogs, and tips on conversion tuning.

Web Analytics & Tracking – Most web analytics software companies offer support for basic A-B split or multivariate testing. Some also offer specific support for testing engagements.

  • Google Analytics – Very comprehensive and free in the ASP version. Can also buy server licenses to run on your own machines.
  • Omniture – Web analytics company focusing on larger clients. Recently acquired Offermatica.
  • WebSideStory
  • WebTrends
  • FireClick – Owned by software industry e-commerce company Digital River. Offers comprehensive web analytics suite.

Conversion & Usability Consultants – These companies focus primarily on consulting or complete site redesigns meant to improve conversions.

General Online Marketing – These companies focus on several aspects of online marketing, but offer some kind of landing page testing.

Market Research & Case Studies – These companies publish reports and case studies involving landing page optimization.

General Usability Resources – These companies or organizations have a broad focus on usability.

Tags: