Internetland is changing often. Those who are very serious about testing use it often, much more than every year. Some would consider a day to be a long time.

I agree that pro-actively raising the point isn’t the best idea, but on the other hand I have personally been asked by clients why isn’t the lift from the test we did 3 months ago (and implemented the winner) showing up in their bottom line?

It’s not about doing something about it so much as being aware of the phenomenon.

The fourth reason is that the underlying mechanisms that are responsible for the observed change are complex (as they are in the case of a website with multiple page elements, people from different backgrounds with different intentions, genders, preferences, expectations, motivations and so different levels of susceptibility to the marketing messages who visit the website. Disambiguating these confounders is usually impossible in this sphere – as the information is not always available. We often do not know the gender or age of the visitors who did not convert, and people can use the same keyword phrase with very different intent (and as we do not have visibility to their prior searches, we cannot glean this from the data we have). Often it is not feasible to run a test long enough to gather sufficient data to slice and dice the demographics finely enough to spot the real gems. Such is life – we must always operate with incomplete information. On the other hand, we are incredibly lucky to have the wealth of data we can get our hands on, so quickly and relatively cheaply. Before the internet, this would have been an incredibly expensive exercise both financially and temporally.

Regards,

Salvatore

Interesting observation. This is a common problem with (using and interpreting) statistics – and applies to every science where an underlying mechanism cannot be demonstrated by controlled experiments. Unless the changes to the winning page are due to better psychology (so they apply to the vast majority of normal humans – if such a beast exists – for example: providing a story that demonstrates product benefits that pander to the underlying human need for acceptance will win over a list of features, and will probably win forever, by a significant margin), then there is a a possibility that the raise is not due to anything more than a passing fashion preference (and this is why so many web pages are so easy to optimise – they look old fashioned, so changing them to look more modern fits with what people expect to see, so they are more likely to buy). Sorry for the parenthesis and convoluted sentence – too much caffeine and not enough time for edits! ;-)

Regards,
Salvatore

With all tests, you need to ensure there is no channel bias as for example if PPC (paid traffic) is driving the conversion, if/when this activity is switched off this will obviously alter the Conversion %. Always look at channel split for all tests.

@Ophir, look forward to seeing some output in part 2.

I think this is an area where optimisers need collaboration.

Great comments and I agree that while we often think we have a big enough sample rate, we really don’t.

I’ll be incorporating some of your comments in part two.