Beethoven, Obama, and the Perils of A/B-testing

What is A/B-testing?

In web design, A/B-testing means comparing two different designs to see which one works better. This could be about getting more people to complete a task, sign up for a newsletter, or buy a product.

A Classical Example: The Obama Campaign

A famous use of A/B-testing is from Obama’s 2008 election campaign.

The campaign team wanted more people to sign up for a newsletter on their website. They tested eleven different designs with 300,000 people. One design got a lot more sign-ups, and the team believed it helped them raise an extra $60 million.

The Catch

$60 million sounds great, but there’s a problem. If you look at the design above, it asked for an email and ZIP code but had a button saying “Learn more”. This is odd because you don’t typically give your email just to get more information.

Other buttons that didn’t perform as well had more logical text like “Join us Now” or “Sign up”. Even though they made more sense, they didn’t get as many sign-ups.

So what’s happening here? There is a conflict of interest going on:

• the campaign team wanted people to sign up right away,
• however, visitors just wanted more info first.

The “learn more” button was a very unfortunate “solution” for this conflict: it essentially tricked people into signing up when they weren’t ready.

This is not a use case for an A/B-test. The campaign team either didn’t see the conflict, or they did but decided not to do anything about it. Either way, that’s not great.

The Statistical Unpower of A/B-testing

The fundamental issue with A/B-testing is that it’s too simple. Design involves thousands of decisions about words, colors, layout, and typography. How can you meaningfully choose just two designs from the millions of possible combinations? You can’t.

A/B-testing can reveal problems, but it is no substitute for design skills. The designer’s expertise is what really matters.

Beethoven’s A/B-test

Let’s take music as an analogy. A lot of famous music, from J.S. Bach to Abba, was composed in isolation, without feedback from listeners.

Did Beethoven ask orchestras across Europe to test different opening bars for his famous 5th Symphony? He didn’t. But his sketchbooks show that he tried several motifs before deciding on the now-famous “fate motif”.

Apparently, he was able to find this motif, and understand its qualities, without the help of an audience.

Beethoven didn’t need to A/B-test, because as a skilled craftsman, he simply knew what worked.