Test Fatigue – Conversion Optimization’s Dirty Little Secret

I’m going to expose to you a phenomenon that’s fairly common when split testing, but no one seems to be talking about it (other than veteran split testers) and I don’t think it’s ever been blogged about (please add a comment if I’m wrong).

It has to do with the question:
“Will the lift I see during my split test continue over time”?

Let’s start by looking at a scenario commonly used by practically everyone in the business of split testing.

Your web site currently is currently generating $400k a month is sales which has been steady for the past few months. You hire a conversion optimization company, which does a split test on your checkout page.

After running the test for 3-4 weeks, the challenger version provides a 10% lift in conversion and RPV at a 99% statistical confidence level. The conversion rate company turns off the test and you hard code the winning challenger.

First of all – Wooohoo!!! (Seriously, that’s an excellent win.)

A 10% lift from $400k a month is an extra $40k a month. Annualized that amounts to an extra $480k a year. So your potential increased yearly revenue from using the winning checkout page is almost half a million dollars. Sounds pretty good to me.

Here’s the problem.

All things being equal, by using the winning version of the checkout page and not your old checkout page, there is a good chance you won’t be making an extra $480k in the next 12 months.

Don’t get me wrong. You will indeed be making more money with the winning checkout page than with the old one, but in all likelihood, it will be less than simply annualizing the lift from during the test itself.

The culprit is what I like to call “Test Fatigue” (a term I think I just coined).

Here’s what often happens if instead of stopping your split test after 3-4 weeks you could let it run for an entire year. There is a phenomenon that I’ve often, but not always seen with very long running split tests; after a while (this might be 3 weeks or 3 months) the performance of the winning version and the control (original) version start to converge.

They usually won’t totally converge, but that 10% lift which was going strong for a while with full statistical confidence is now a 9% lift or an 8% lift or a 5% lift or maybe even less.

As I mentioned before this doesn’t always happen and the time frame can change, but this is a very real phenomenon.

Why is does this happen?

Please read my next posting – Why Test Fatigue Happens where I provide some explanations on why this happens.

Also, I’d love to hear if you have also seen this phenomenon with your own tests and what your personal theories are as to why it happens.


How can I help you with conversation optimization?

I just realized it’s been almost six months since I last posted on this blog. While I have plenty of ideas for posts, I figured it might be best to ask you – my readers (all three of you) how I can help you. Specifically there are two major ideas I’ve had in my head for a while and I’m debating between which one to write about next.

The first idea is a technical overview of how the web works, going into detail on web analytics and split testing. Everything someone who is not a techie needs to know in order to gain a better understanding of what the data really means from a technical perspective as well the implications on how technical decisions impact business decisions.

The second idea is making conversion rate optimization more of a science and less of an art. I’ve read just about every book out there that deals with site and page optimization. I’ve also conducted countless split tests and have analyzed more sites than I can remember. What I’ve found is that there seems to be a major gap in the process where what to do next and how to do it becomes more of an art and less of a science.

Plenty of smart marketers can see a web page and know intuitively that it won’t convert well. Often it’s even easy to identify specific elements which are “broken” and need to be fixed, but more often than not (at least for me), it’s usually not so simple to explain the internal thought process of converting an OK page into a great one. This is something I’d like to address.

So, my loyal readers, please let me know what I should write about. Even if it’s something other than the two topics I’m thinking about let me know.


The 3 Levels of Conversion Rate Optimization Maturity

If you’re reading this article, I hope you realize that split testing is no longer optional if you want to increase the performance of your web site. So, what is split testing? It’s simply presenting different versions of content to different visitors and measuring which version of your content gets the most desired results. Here are what I consider to be the three levels of conversion rate optimization maturity with an emphasis on split testing.

1 – Lowest Common Denominator

The first level of conversion rate optimization maturity is what I like to call “Lowest Common Denominator Split Testing”. This means treating all of your visitors the same. You simply split test all of your traffic together and see what performs best.

Not so long ago, if you were doing any split testing, you were ahead of the game since most of your competitors weren’t. That’s not the case anymore. I’m willing to bet the vast majority of the top 100 eCommerce sites are already doing some form of split testing.

The problem with lowest common denominator split testing is that your visitors are not all the same. While you’ve found what works best for the group as a whole, you are not taking advantage of obvious differences in terms of why they came to your site and what will get them to take action.This brings us to the second level of maturity.

2 – Segmentation

The second level of conversion rate optimization maturity takes advantage of “standard” information you know about your visitors. For example, where they came from – was it search (organic or paid), direct traffic (they typed in your url directly), a referral (a link from another site) or maybe an internal email. Have they come to your site before (new or returning users). Where are they located? What Browser are they using? etc.

This is the type of information you’ll usually find in a web analytics tool. Most of it is what’s available to you at the time of the visit itself.

This type of targeting is also known as segmentation. Basically, instead of putting everyone in a single bucket, you can now segment your visitors into several buckets. Instead of split testing all of your site traffic together you can measure the difference in behavior for each segment and more importantly serve up different tests to different segments.

The analogy I like to use is that of a sales person at a store who greets someone who just walked into the store. A good salesperson will try to put that visitor into a “segment” such as male, female, age, income, etc and propose products that person will most likely be interested in. If you’re not segmenting, it’s like having a blind and deaf salesperson.

Using segments together with split testing is way better than just split testing on it’s own, but you’re still treating everyone in each segment the same. What if you could actually treat every visitor as an individual? This brings us to the next level.

3 – Profiling

The final level of conversion rate optimization looks at each visitor as an individual. While segmenting on it’s own takes advantage of what you know about a visitor at the time of the visit, profiling also takes advantage of everything a visitor previously did as well as everything all of your other visitors have done.

Profiling gives each visitor a history that you can fully take advantage of. If a visitor bought shoes on their last visit, show them a banner for socks. You can even track what type of shoes they purchased in order to know what type of socks to offer.

Going back to the salesperson analogy, using segmentation on it’s own is like never having the same salesperson at your store. Every visit for every visitor has a different salesperson. Profiling is like having one single super-salesperson that remembers everything every visitor ever did.

Profiling can also be automated. If most people who purchased products A & B also bought product C, then automatically show product C to anyone who purchases products A & B. While profiling on it’s own is very powerful, but the ultimate in optimization is profiling together with split testing.

Netflix and Amazon are two examples of companies that are already doing profiling (and split testing). Wouldn’t you like to be like them?

As always, please leave questions and comments in the comments section.

Thank you

New Job at Adobe

Just a quick note to announce that I am now an Optimization Manager at Adobe.

What does that mean to you?

While analytics and optimization go hand in hand, going forward the emphasis will be more on the optimization side of things.

Also, now that I’m primarily using Test&Target as a split testing tool, I will be able to pass on any cool tips or ideas that I come up with on Test&Target.

If you have any Test&Target related question, feel free to ask me directly on this blog.

– Ophir

Conversion Tips from a Crying Toddler

I can’t help but think about conversion rate optimization in my day to day life outside of work.

A friend of mine who studied film (making movies) in college once told me that he can no longer watch a movie without dissecting it in his head.

Every know and then I have a similar experience (usually as part of an “a-ha” moment) in my life where I realize something about human behavior – specifically how to get someone to do something.

I have quite a few of these in my head which I’ll start sharing on my blog.

Today it’s a lesson I learned from a crying toddler.


David is a 3 year old boy. He has a cat called Muffins. David and Muffins get along very well, but if Muffins bothers him at night while he’s sleeping, it scares him and we wakes up.

Last night when David started to cry, I saw Muffins run out of his room and immediately understood what happened. I went into his room and tried to calm him down.

David: Crying
Me: It’s OK David, Muffins won’t hurt you
David: Crying
Me: I’ll close the door so Muffins won’t come back. Daddy’s here.
David: Crying
Me: Did Muffins scare you?
David: Yes (and stops crying)

David calmed down at that point and quickly fell back to sleep.

As you can see, I was only able to get through to David once I acknowledged his feelings. This is parenting 101 but the point here is:

Acknowledging someone’s feelings is a very powerful way to get through to them.

Some examples off of the top of my head:

  • Do you suffer from high blood pressure?
  • Are you frustrated by your child’s behavior?
  • Worried about your debt?

Personally, I’m not crazy about using “informercial” type headlines, but the reason they are so widely used is that they usually work. Of course like any good marketer you are testing your copy so ultimately you know what works best for your audience.

Four New Additions to the Google Analytics API

For those of you not following the GA API change log Google just added four new data points:


  • ga:dayOfWeek


  • ga:percentVisitsWithSearch
  • ga:visitsWithEvent
  • ga:eventsPerVisitWithEvent

All of the new data points are essentially “calculated metrics”, meaning you could calculate this yourself if you were to download the data and do the calculations offline, but still, I applaud Google for continuing to make it easier to get the data without having to resort to offline processing.

Personally, I’m most existed about the dayOfWeek dimension. If you’ve never segmented your traffic by day of week, you really should. Do you know what day of the week has the highest conversion rates? Maybe you should be sending out your emails that morning :)

3 Easy Ways to Improve Your Conversion Rates

I’ve read my share of articles on “101 things to test to improve conversion rates”.

While most of the suggestions are usually sound, I find that these lists are often overwhelming and you don’t know where to start.

So here’s how to start with a a simple but often overlooked problem –  your links / link visibility.

Specifically, do your links look like links?  Do visitors know what will happen after they click on a link?

This goes back to one of my main mantras in conversion rate optimization – Don’t make me think.

Visitors don’t read web pages, they skim. And when skimming, you should make these two points very obvious:

  1. What elements on a page are a link?
  2. What will happen when I click on that link?

While the answers to the above questions are obvious to you – the site creator, they aren’t always obvious to a first time site visitor.

Here’s how you can actually fix any issues your links might have.

First of all, print out your homepage (or other page you want to test). Take the printout to someone who has never seen your site before, if possible, someone who is similar to your target audience.

Now ask them to circle the links on the page with a pen or highlighter. For extra credit, use two pens. A blue one for elements they’re pretty sure are a link and a red one for elements they think are a link but aren’t sure.

This alone should unveil any major issues where visitors aren’t sure what actions they can take on page.

Next, ask them to mark any links where they aren’t 100% sure what will happen once they click on the link.

For example, a link labeled “HOT” might be confusing where “Most Popular Items” would not be.

Lastly, people know a link is a link based on two different criteria.

  1. What it says
  2. What it looks like

When viewing a page, what a link looks like will be the first thing a visitor notices. Is it a different color? Does it have an underline? etc.

Only after reading the link text will they factor in what it says. For example, “Click Here”, “More Info” or “Add to Cart”.

In order to make sure visitors can find links based purely on what they look like, we’ll use the “Greek Link Test”. The idea is to translate all of a page’s text to Greek and then see if people know what’s a link and what isn’t.

First thing is to go to Google Translate – http://translate.google.com/ choose English to Greek and enter the URL of your page.

For example, here’s what my blog looks like in Greek: http://goo.gl/EKwya

Now print the page (now in Greek) and do the same exercise as before. Ask someone who is not familiar with the site to mark all of the links on the page.

What’s Next?

Now that you’ve identified problematic links on your page, you have one of two possibilities.

Your best option is to actually split test problematic links with ones that look more like a link. This will tell you conclusively the effect of improving link visibility, it will look like you’re getting instant likes on Instagram. The first metrics you should look at are bounce rate (or exit rate), page views per visit and time on site per visit. You should also look at the conversion rates for your site’s main goals, but it will probably take longer to get statistically significant data.

Please note that if time on page goes down, this is NOT a bad thing. Sometimes increasing link visibility makes it easier for visitors to find what they’re looking for and they stay less time on a page.

Even if you can’t split test the links, I would still suggest trying to improve them by making them visually stand out more or improve the link text itself. Then repeat the above exercises and see if there is any improvement.

What are your thoughts?