Rely on data, Not HiPPOs

A/B Testing is when you have two different versions of a site and send 1/2 of your traffic to the original version (A), and 1/2 of the traffic to the new version (B) to see which one produces the most positive results (clicks on link, buys an item, views a video, etc!). A/B Testing proves which version is most effective. It is a bad idea to judge a website based on your personal preferences like the pointy haired boss does in the Dilbert comic above. This comic exemplifies the concept of HiPPO. Some companies make the mistake of skipping testing and rely on the Highest Paid Person’s Opinion (HiPPO!) for how a site should look.  Using the data and numbers you can derive from A/B testing will increase the amount of conversions made on your website just by listening to your users though the data. Testing is must more trustworthy than the HiPPO.

Before starting an A/B test, it is important to ensure that you are only testing one variable.For example, in an article by Microsoft, a Foot Care company did A/B Testing where they changed multiple things on their website such as adding a “Continue Shopping button, “Add Coupon Code” button, and a few other minor changes.. They went with the new version because it had the greatest order magnitude, however, once them implemented, the site lost 90% of revenue. When people saw the spot to add a coupon, they began thinking that there was a coupon out there that they did not have and thought twice before buying. Getting rid of the “Add Coupon” spot was an easy fix, but in order to target specific changes and to achieve the greatest ROI (return on investment), it important to change only one variable at a time.

Amelia Showalter worked as the director of analytics for Barrack Obama’s reelection campaign in 2012 and used A/B testing to entice voters to donate money to help get him reelected. They did this in all sorts of ways, for example, by testing which email subject line caught donor attention and enticed them to donate the largest sum of money. The team first sent out a variety of different email subject lines to a small sample of people on their email list, and the one that performed most successfully in the focus group they used to send to their entire email list. The difference between the top contender and the runner up was huge- $2,540,866-1,914,371= nearly $600,000 sum of donations that could have otherwise been lost. By using data to choose subject lines, the color of donation buttons, and even the greeting used on the email Showalter and her team were able to get Obama reelected.

The Obama campaign learned many things through A/B testing. They learned that the ugly formatting and attention grabbing ugly highlighting actually worked. More people donated when the campaign used attention grabbing formatting on important keywords or sentences. They also learned that no matter how many emails they sent out, receivers did not unsubscribe! This is not something normal businesses should depend on. With the Obama campaign, people will only receive those emails for a matter of months, maybe a year at most. It is also something that many people will feel passionate about. The people who subscribe to Obama’s campaign emails are people who support Obama in his mission to get elected. They are willing to put up with the emails for a short-time frame until the election happens. There is a set end date for when the emails will end. Other companies will continue sending out emails forever, or until you unsubscribe (or they go out of business).

Other companies such as the lingerie company Adore Me use A/B Testing not on subject lines and buttons, but on models. Multiple shots are taken of models in different poses, and the two that are deemed best are used on the website. 1/2 the customers see picture A where the model is standing upright with her hands on her hips, and the other half see picture B where the model is sitting down playing with her hair. Over time, the company will see that one picture sells the lingerie better than other and will take down one picture and send 100% of the traffic to the other one.

The lingerie industry found that the right model controls buyers more than the price. If there is a model that buyers like, they are more likely to buy the item (even if it is expensive) than if that same item is on a model they do not like (even with a price cut!).

Half-way through reading the article, I began to wonder if they they were going to discuss if / how the models race impacted sales. At the end of the article, the CEO is quoted saying, “ethnicity doesn’t mix and match with how it sells: We had some super strong sellers on the African American model, and we had some super bad sellers on the African American model, it was all about the way she behaves in the picture. That’s what makes the beauty of the A/B testing. It doesn’t cancel out an ethnicity and a genre; it’s all about the emotion.” The emotion on the model, and the emotion that the person viewing the picture feels. If the person buying it does not feel like the model is giving off a vibe they would want to give off if they were the one wearing the item, they would not buy the item, and the other way around is true too. Using A/B testing can increase purchases by interpreting how buyers react to pictures. If the item is bought- the purchaser clearly reacted well to the image.

When you implement A/B testing, never expect it to work perfectly right off the bat. There are neutral and negative results that occur just as frequently as positive results occur, and it’s important not to expect either version to work best until there is data to back it up. A/B testing can work in favor of your business, but don’t expect to get vast improvement immediately, or ever. Performing the test might get you a little extra on every page, but that will add up over time.

Sources: This Lingerie Company A/B Tests the World’s Hottest Women To See Who Makes You Click Buy,      The Problem with A/B Testing Success Stories,      The Science Behind Those Obama Campaign E-Mails,      The Obama Campaign’s Legacy,      Practical Guide to Controlled Experiments on the Web,      What is A/B Testing?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s