The Future of Split Testing and Conversion Rate Optimization

I’ve been fortunate enough to see and experience first hand the evolution of the Internet, from even before the web till today.

I’ll spare you a lengthy history lesson explaining how we’ve gone from brochureware sites to where we are today, but I do want to share some thoughts and perspective on where I think things are going.

When marketers started to understand the potential of dynamic web sites, there were two terms everyone was throwing around:

Personalization & Customization.

Fast forward to today (2011). The user experience is still exactly the same for all visitors (other than on a handfull of sites such as

For the most part, web site Personalization has failed. Sure it sounds good in theory, but trying to tailor the web site experience at the individual level is extremely difficult. It is difficult both from a technological perspective but mostly by trying to create an optimal user experience based on data from a single individual.

There is no doubt in my mind that in the future (and to some extent today) the user experience when visiting a web site will be created dynamically based on what gets the best results, but based on “anonymous” information which is common to large groups of visitors, and not based on a single person.

This reminds me of the concept of Psychohistory from the science fiction series “Foundation” by Isaac Asimov.
Wikipedia explains it better than I can:

The premise of the series is that mathematician Hari Seldon spent his life developing a branch of mathematics known as psychohistory, a concept of mathematical sociology (analogous to mathematical physics). Using the law of mass action, it can predict the future, but only on a large scale; it is error-prone on a small scale. It works on the principle that the behaviour of a mass of people is predictable if the quantity of this mass is very large. The larger the number, the more predictable is the future.

I also like to think of this in terms of what usually happens at (successful) brick and mortar stores.

When you walk into a store, the salesperson probably doesn’t know you personally, but will probably try to help you based on certain public traits such as gender, age, if you’re by yourself or with someone else, etc.

Which brings me back to what actually prompted me to write this article in the first place :)

While I’ve been split testing since 2005 in order to improve conversion rates, the majority of the time, it’s still about what works best for the site as a whole, opposed to split testing together with segmentation (which is what we really want).

Until recently, there haven’t been many options out there to achieve this level of targeting and testing (at least not priced for small to mid sized businesses) but over the past few months, I’ve been starting to see more and more startups trying to bring this level of sophistication to the masses.

While I haven’t had a chance to use any of these services first hand, there is no doubt in my mind that business that truly embrace this level of targeting and split testing will eventually lead the pack and leave most one-size-fits all web sites in the dust.


3 thoughts on “The Future of Split Testing and Conversion Rate Optimization

  1. Totally agree with you Ophir, but one thing I am “worried” about for the smaller sites is that if you combine personalization and testing you might end up with segments that are too small, ie with too little traffic to get significant results.

    Do you use any rules of thumb for determining whether personalization will likely be effective?

    • Hi Michael,

      The truth of the matter is that most sites (especially smaller ones) don’t segment or split test at all, let alone split test based on segments :)

      It’s hard to say if segmentation (I’m purposely using that term opposed to personalization) will be worth the effort without factoring in the specifics of a web site.

      My general feeling is that most sites still have quite a bit of low hanging fruit just in terms of optimizing for the lowest common denominator (ie. making site wide optimizations) before continuing to segmentation and then ultimately split testing based on segments.

      One of the cool features of most split testing platforms is the ability to see how different segments performed. This should provide some insight as to the value of actually serving up different content to different segments.

  2. Also, in smaller markets such as Australia, it’s practically unfeasible due to – as you note – fewer people. But I definitely agree that using other attributes will help push personalization along.

    I think using a person’s history from things like Facebook connect is definitely achievable and feasible if you get enough information that’s *relevant*. You should checkout the BT buckets and Facebook integration. Really cool stuff there and its all free.

    I don’t see why content sites aren’t using Facebook connect to tailor initial content recommendations yet!

    By the way, I’ve found that linking GA and your testing platform can help you identify nice segments too.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s