Just a quick post about a phenomenon I’ve personally seen happen but don’t recall ever seeing mentioned in split testing articles.
I’ll start by saying that ideally you should always look at the results from any split test by segmenting your visitors.
It’s not enough to know that overall version X did better than version Y. Ideally you should check how the different versions performed for various visitor segments. For example, users from organic search might behave differently than visitors from a referring site or direct traffic.
There is one segment though where merely the fact that you’re doing a split test can have an impact on the results:
New vs. Return visitors
Even if you weren’t doing a split test, you would probably see a difference between the two segments based purely on the fact that return visitors already know something about your product, service or site.
I’m talking about a different phenomenon though. The “something has changed” effect.
For new visitors, your site will be new regardless of which version of a test page they see.
For return visitors who have some level of familiarity with your site, if they see something new or changed on the site, they’ll probably pay more attention to it – merely because it’s different.
For example, if you’re homepage does not currently have any video on it and you test a new version with some video on it, return visitors who get the version with the video might watch the video simply because it’s something new.
Conclusion: Always segment visitors by new and return visitors.
If both groups show the same preference, it’s safe to say that you have a winner. If you’re seeing a large variance between new and return visitors, it might be worth it to let the test run for a while to see if the variance changes over time as more of the return visitors first visited the site after the split test started.
I was just thinking that this would be a feature that split testing tools can / should support. Segmenting not only new vs return users, but return users who’s first visit was before the split test started vs return users who’s first visit was after the split test started.
Heck – you should be able to only include visitors who’s first visit was after the test started if you want to. Are you listening optimizely, visual website optimizer, and the rest of the gang?
3 thoughts on “Split Testing and Return Visitors”
Great blog post! I couldn’t agree with you more. During the Obama campaign we used segmentation extensively and it made a huge impact. This is something I’d love to be able to offer with Optimizely. We’re working on building a great user experience for this feature and expanding our reporting functionality to be able to support this. Let me know if you are interested in a look at our early designs. We certainly would appreciate your feedback!
Co-Founder & CEO
We already support this feature. Have a look at this blog post: http://visualwebsiteoptimizer.com/split-testing-blog/tag/segmentation/
Is this what you were referring to in this post?
Director, Visual Website Optimizer
Great post. I’m setting up a test on a bank’s homepage (which rarely changes and has a large proportion of return visitors). I’m very curious to see how it impacts behaviour.
Assuming visitor recency is relatively high, perhaps it maybe worth running the test for just over 30 days or however long it takes for repeat visitors to adapt. Should I consider a follow up test?
I’ll admit I’m not sure how to approach this other than by warning the client.