Bloomberg Anywhere Remote Login Bloomberg Terminal Demo Request


Connecting decision makers to a dynamic network of information, people and ideas, Bloomberg quickly and accurately delivers business and financial information, news and insight around the world.


Financial Products

Enterprise Products


Customer Support

  • Americas

    +1 212 318 2000

  • Europe, Middle East, & Africa

    +44 20 7330 7500

  • Asia Pacific

    +65 6212 1000


Industry Products

Media Services

Follow Us

Bloomberg Customers

When Multivariate Testing Is Better than A/B Testing

Posted by: Today's Tip Contributor on July 26, 2010

Marketers know that testing and tracking different elements of their campaigns is the best way to understand what works with their intended audience. Common methods for running controlled experiments range from simple A/B testing to sophisticated multivariate testing. You can test pretty much anything, particularly in the online world—from e-mail to video to mobile content to advertising. For the purposes of this piece, I’ll focus on testing Web content.

In A/B testing, one or more new versions of a page or single site element compete against the original (control) version to see which one is more persuasive. A/B testing is quite effective for answering certain types of marketing questions, including creative bake-offs where the decision has come down to two competing versions. But for more complex questions, often those that arise earlier in the testing cycle, multivariate testing is often the way to go.

That’s because while A/B testing allows you to test just one factor at a time, multivariate testing enables you to test many changes simultaneously. For example, two alternate product images, plus two alternate headlines, plus two alternate product copy texts, for a total of 27 possible combinations (including the original control versions).

What’s important to understand about multivariate testing is that it not only shows you which combination of elements generate more sales or pull more leads, but it also reveals which individual elements influence visitor behavior vs. those that do not. Evaluating the impact of combinations of factors and variations often reveals significant interaction effects that can have a dramatic impact on your conversion goals.

Multivariate testing enables you to understand whether variations in the product image, for example, influenced visitor behavior more, less, or the same as the copy. Understanding how each site element causes visitors to interact with your site is the essence of a "test-learn-repeat" process that marketers can use to synthesize new ideas and continually improve their site’s ability to achieve and exceed their marketing goals.

Every website idea, whether it be content, functionality, or campaign-related, should be put to the test to determine if the idea helps or hurts the visitor experience. While some new ideas lift conversions, others fail, sometimes significantly. But even with these failures, definable knowledge is gained that helps you learn what to avoid the next time. The ability to test a new idea and look before you leap is an unmistakable advantage that breaks the constraints on marketing innovation. Only once a solid testing capability is in place, and the impact of any site change can be quantified, can marketers truly optimize the effectiveness of their site.

Kim Ann King
Chief Marketing Officer

Post a comment



Want to improve the way you run your business? Entrepreneurs, academics, and consultants from diverse industries offer practical advice on a variety of topics each business day.

To submit a tip for consideration, first check our archive of previous tips to make sure you're not repeating a tip someone has already contributed. Then send the tip to Small Business channel contributor Michelle Dammon Loyalka. Because of the volume of material she receives, she may not respond to each individual.

BW Mall - Sponsored Links

Buy a link now!