A few reactions that crept up while reading:
1a. part of a strong optimization culture should be to value learning-through-losing. i.e. team members should know it’s ok, even encouraged, to get a losing test result. so long as you, in Avinash’s words, “fail quickly.”

1b. the optimization culture needs to be extended to “partners” as well. trying to run tests is exceedingly difficult when the SEM/Design/Research/Technology partners aren’t on board or even aware of the efforts and goals of optimization

2. a bit troubled by your positioning of personas as “non-analytical.” I understand this is often the case, but to be fair, personas done well are based on tons of quantitative data!

Only thing I would add is that it should be a constant process, too many people do some work, get results, pat themselves on the back and then the results start to ebb and they revisit to try to regain the results they achieved. End-users habits evolve and so does the optimisation process.

I’d like to propose a seventh point: measurement of success doesn’t stop at the ‘final’ conversion. Say you’re testing an online store’s product page and the winning variation doesn’t display the store’s service hotline or any other way of contacting costumer support. Perhaps as a result this variations causes more returns and therefore reduces profit due to the cost of returns. Actually, the cost for returns might be even higher than the additional profit generated by the variation. If one doesn’t measure these ‘post-conversion activities’ one can’t say that the winning variation actually increases the store’s profit.

Obviously, merging data from the testing software with CRM data is challenging for a bunch of reasons (technical and political reasons, for example). If one wants to focus on a store’s profit it’s necessary though.

A related case is the tracking of conversions made via telephone. Say you’re testing a landing page: the control variation displays a telephone number where you’re able to order the product too, the variation doesn’t display this number. To fully judge whether the variation performed better, one has to compare the online conversions from the control variation + ‘telephone conversion’ on the one hand to the online conversions from the variation on the other hand. If one doesn’t measure the ‘telephone conversions’, one misses part of the picture.

Do you think this issue deserves its own point or should be under one of your point above?

You make some excellent points!

In terms of segmentation and targeted user experiences, it really doesn’t make sense to target a single individual per-se, but use group behavior to provide a targeted experience.

How big or how small those groups are *does* depend on how much traffic you are getting.

Like any efforts to improve a business, you need to factor in the point of diminishing returns.

Also, I’m not saying every single decision should be split tested and optimized – just the important ones :)

A realistic process that somewhat addresses this issue would be to setup a testing schedule. For example, plan to launch one test every month.

This forces you to prioritize your efforts based on maximum ROI and also ensures your optimization efforts keep momentum.

I do have clients that are launching a test every week, but they didn’t start that way – they have been testing for over 3 years.

Also, since consumer behavior is in constant change from outside influencers at what point do you retest to see if the same optimization in the past have the same effect in the present?