Beethoven, Obama, and the Perils of A/B-testing

Are A/B-tests useful? Here’s what they can and can’t do for you.

Beethoven holding up examining an Erlenmeyer flask in a lab for examination
Beethoven in his lab. AI-generated image by DALL-E

What is A/B-testing?

In web design, A/B-testing means comparing two different designs to see which one works better. This could be about getting more people to complete a task, sign up for a newsletter, or buy a product.

A Classical Example: The Obama Campaign

A famous use of A/B-testing is from Obama’s 2008 election campaign.

The campaign team wanted more people to sign up for a newsletter on their website. They tested eleven different designs with 300,000 people. One design got a lot more sign-ups, and the team believed it helped them raise an extra $60 million.

Photo of the Obama family, with slogan 'Change We Can Believe in'
The winning design. © Optimizely.

The Catch

$60 million sounds great, but there’s a problem. If you look at the design above, it asked for an email and ZIP code but had a button saying “Learn more”. This is odd because you don’t typically give your email just to get more information.

Other buttons that didn’t perform as well had more logical text like “Join us Now” or “Sign up”. Even though they made more sense, they didn’t get as many sign-ups.

'Join us now' button
'Sign up' button
'Sign up Now' button
The calls to action that didn’t make it into the winning design. © Optimizely.

So what’s happening here? There is a conflict of interest going on:

The “learn more” button was a very unfortunate “solution” for this conflict: it essentially tricked people into signing up when they weren’t ready.

This is not a use case for an A/B-test. The campaign team either didn’t see the conflict, or they did but decided not to do anything about it. Either way, that’s not great.

The Statistical Unpower of A/B-testing

The fundamental issue with A/B-testing is that it’s too simple. Design involves thousands of decisions about words, colors, layout, and typography. How can you meaningfully choose just two designs from the millions of possible combinations? You can’t.

A/B-testing can reveal problems, but it is no substitute for design skills. The designer’s expertise is what really matters.

Beethoven’s A/B-test

Let’s take music as an analogy. A lot of famous music, from J.S. Bach to Abba, was composed in isolation, without feedback from listeners.

Did Beethoven ask orchestras across Europe to test different opening bars for his famous 5th Symphony? He didn’t. But his sketchbooks show that he tried several motifs before deciding on the now-famous “fate motif”.

The opening bars of Beethoven's Fifth Symphony, in music notation.
The iconic opening bars of Beethoven's Fifth Symphony

Apparently, he was able to find this motif, and understand its qualities, without the help of an audience.

Beethoven didn’t need to A/B-test, because as a skilled craftsman, he simply knew what worked.

Stay informed of new posts?

Sign up below and you will receive a weekly digest of new posts.

If you change your mind, you can unsubscribe at any time. You will not be spammed, and we will keep your email address private.

Article index