LearnThe Business Case for A/B Testing


writes on February 17, 2010

Does design of a sales page matter? Traditional reasoning says that the product always remains the same no matter how you dress it up on the sales page. So, one should focus on making the product more awesome rather than investing time to make it look awesome. Well, the reasoning sounds plausible in theory but the data says it is not well grounded.

Case Study

This post is about a recent A/B split testing case study where a redesign of a sales page resulted in 20% increase in sales. AquaSoft is a leading software company in the area of digital photo presentation. They have a complete portfolio of (desktop) software products related to photos – slideshow creation, desktop publishing, photo books, etc. Their software products can be downloaded for a free limited duration trial, following which a customer pays for the full version.

AquaSoft undertake more than 10 A/B tests per month using Visual Website Optimizer, an online A/B testing tool. They recently finished a test in which they tested their existing sales page against a new redesigned sales page. Note that this page wasn’t the free trial download page; rather it was a sales page where customers visit to actually buy the software.

Here is the control version of the page (click to view full version):

And here is the (final) redesigned version (click to view full version):

What’s different?

The pages are in German language, so let me point out a few key differences between the new and the control version:

  • The new design gave a modern, fresher vibe on the page and had better contrast.
  • The old design had two calls to action per product – one for instant download and other for purchasing software in a shrink wrapped box. The new design reduced the choice to primarily one option (instant download) and the call-to-action was also transformed into a snazzy button (as compared to a simple text link in old design).
  • The new design was optimized to build trust in the customer. The sidebar in new design contained different kinds of guarantees:
    • Free lifetime updates and bonus software
    • Lifetime guarantee on the software
    • 24h/365 days support
    • Free customer login id
  • The new design also incorporated a seal made out of their logo to provide authenticity to the guarantees

A key point to note is that AquaSoft did not change their product features neither their guarantee policy. The details were simply not present (or highlighted) on the old sales page.

Two phase testing

AquaSoft carried the A/B test in two phases. In the first phase, old design was pitted against a variation of the new design. Improvements started with about 50% but settled on about 35-40% after some time. They were very happy with the result but craved for more details because in this test they were measuring clicks into the shopping cart, not the actual sales.

So they measured the sales which showed an increase of 17.7% of actual sales. It also showed that the bounce rate within the shopping cart was somewhat higher in the winning combination but the overall increased click rate outtakes the bounce rate.

In the second phase, AquaSoft refined the winning variation by doing minor changes such as reducing whitespace and placing software names above their respective box shots. This resulted in a further increase of 10% in clicks on shopping cart button and the overall increase in sales settled to about 20% (up from 17.7% in previous phase). All results were statistically significant, so the 20% increase in sales is real.

What can we learn form this?

This case study has several lessons for all of us who are serious about optimizing online sales:

  1. Design matters! If your sales page has a boring design, expect boring sales.
  2. A/B split testing is totally worth it! AquaSoft invests time, money and effort into doing as many as 10 split tests per month. And that is because they understand what impact a successful A/B test can have on the bottom-line.
  3. Some customers do abandon the checkout process! The next logical step for AquaSoft would be to optimize their checkout process but their payment system is integrated into a third party so it won’t be easy to set up a split test there.
  4. A/B testing is best done in phases. Small improvements in multiple phases ultimately add up to give a significant boost to conversions.

AquaSoft summarized the key take-home lesson from this A/B test:

“Don’t think your website is good as it is. Always test, always improve. It’s a slow process but worth doing. Our concrete test results show that a clear modern design improves sales. Especially in the sales process, trust building is very important.”

For AquaSoft, A/B testing required one-time investment of design resources but the 20% increase in sales will hopefully continue for months to come. If AquaSoft can manage to get such a fantastic ROI, so can you. Think about it – is your sales page optimally designed? What elements can you test on it?

Beta Invite

To try out Visual Website Optimizer enter invite code ‘carsonified‘ when signing up.


Learning with Treehouse for only 30 minutes a day can teach you the skills needed to land the job that you've been dreaming about.

Get Started

6 Responses to “The Business Case for A/B Testing”

  1. Does design of a sales page matter? — on my experience.. yes it is.

  2. Anonymous on October 12, 2010 at 6:46 pm said:

    Once again thinkvitamin.com is a god send. http://www.Pricefalls.com is looking into A/B testing. We have been insanely pressed for time lately and testing has somewhat fallen by the way side. It continues to come into my head at the most random times and having read the successes here I am sending this link to my bosses. If testing is not done regularly time can be severely wasted.

  3. Thank you for this great article. Any ideas on how many users are need on each version of the test to get a meaningful measurement ?

  4. Very interesting article and I had not come across Visual Website Optimizer before. I will be sure to give it a try.

  5. I’ve never really done much A/B testing, mostly because I feel like I don’t have time. But after hearing Ryan emphasize it on the Web 2.0 show podcast interview, I’m starting to look into it more. This post is a great example that I can reference as I form a strategy for some A/B testing on some of the sites I run. Thanks for the example!

  6. Affiliate Millionaire by Andrew Fox is a true bonus to any online marketer.. This is a physical product that will be delivered right to your front door! Is this course something that will put your affiliate earnings on auto pilot or some kind of cpa network system? I will be filling you in on all the aspects of the course and telling you exactly what you will be getting. Of course no course is perfect and I will be looking it over carefully and I will find its shortcomings and fill you in on all the details.
    Affiliate Millionaire Bonus

Leave a Reply

You must be logged in to post a comment.

man working on his laptop

Are you ready to start learning?

Learning with Treehouse for only 30 minutes a day can teach you the skills needed to land the job that you've been dreaming about.

Start a Free Trial
woman working on her laptop