Web Analytics

Split Testing and Return Visitors

/ Ophir Prusak / 3 Comments

Just a quick post about a phenomenon I’ve personally seen happen but don’t recall ever seeing mentioned in split testing articles.

I’ll start by saying that ideally you should always look at the results from any split test by segmenting your visitors.

It’s not enough to know that overall version X did better than version Y. Ideally you should check how the different versions performed for various visitor segments. For example, users from organic search might behave differently than visitors from a referring site or direct traffic.

There is one segment though where merely the fact that you’re doing a split test can have an impact on the results:

New vs. Return visitors

Even if you weren’t doing a split test, you would probably see a difference between the two segments based purely on the fact that return visitors already know something about your product, service or site.

I’m talking about a different phenomenon though. The “something has changed” effect.

For new visitors, your site will be new regardless of which version of a test page they see.

For return visitors who have some level of familiarity with your site, if they see something new or changed on the site, they’ll probably pay more attention to it – merely because it’s different.

For example, if you’re homepage does not currently have any video on it and you test a new version with some video on it, return visitors who get the version with the video might watch the video simply because it’s something new.

Conclusion: Always segment visitors by new and return visitors.

If both groups show the same preference, it’s safe to say that you have a winner. If you’re seeing a large variance between new and return visitors, it might be worth it to let the test run for a while to see if the variance changes over time as more of the return visitors first visited the site after the split test started.

[UPDATE]
I was just thinking that this would be a feature that split testing tools can / should support. Segmenting not only new vs return users, but return users who’s first visit was before the split test started vs return users who’s first visit was after the split test started.

Heck – you should be able to only include visitors who’s first visit was after the test started if you want to. Are you listening optimizely, visual website optimizer, and the rest of the gang?

Advertisement

The Problem with Monthly Reporting

/ Ophir Prusak / 2 Comments

Just a short blog entry about something that’s been bothering me for many years.

I think the best way to explain the problem is by starting with a simple riddle.

My site had visits 31,000 for one month, and then 30,000 visits the next month but the overall trend for visits was up.

How could that be?

The answer is simple.
Not every month has the same number of days.

If I had 31,000 visits in January (31 days) and 30,000 visits in February (28 days) I’d say the overall trend is up.

July / August and December / January are the only consecutive months that actually do have the same number of days in the month.

All this means is that when comparing metrics for two different months, be sure to use relative number and not absolute ones.
For example, comparing average visits per day for the month is better than comparing visits for the entire month.

To make matters even worse, most sites will see a large variance between weekdays and weekends.
Therefore the BEST way to compare month to month will factor this in as well.

I’ve seen two different ways to handle this:

1 – Compare the month in question to a time frame exactly 28 days (4 weeks) prior to the first and last days of the month you’re looking at (yes, you’ll probably have a few days which are overlapped, but that’s OK).

For example, we can compare May which is 31 days to the time frame of April 3rd (May 1st minus 28 days) to May 3rd (May 31st minus 28 days).

2 – Use metrics like “visits per weekday” or “visits per weekend day”. Not perfect since there is still some variance between specific days, but this is better than “visits per day” for the entire month.

– Ophir

Web Analytics Career Advice

/ Ophir Prusak / 4 Comments

The past couple of months I’ve been seeing more and more discussion about breaking into web analytics as a career.

There are plenty of articles out there with great career advice.
Jim Sterne recently put together a list of resources and links for starting a career in web analytics.

I don’t want to repeat what others have already said, but I do want to add some fodder for thought that I don’t seem to be seeing other people mentioning.

As I mentioned in my previous post about selling web analytics, web analytics is just a tool.
A means to an ends.

Think of a carpenter’s tool box – hammer, saw, screwdriver, drill, etc.
Yes, you need to be proficient using these tools to be a good carpenter, but the point is that you’re using these tools to make something!

Which brings me to my point.

In order to be successful in web analytics, you need to “make something with your tools”.

You should be using web analytics in order to provide value, specifically providing actionable information.

In order to provide information that is actionable, you need to have a good understanding and some real world experience in the realm you’re trying to improve.

While many online activities can be optimized using web analytics, here are some categories that are commonly optimized using web analytics.

  • Website Conversion Rate Optimization
  • Paid Search
  • Search Engine Optimization
  • Social Media
  • Email Marketing
  • Online to Offline Marketing

Each of the above topics can and should be using data in order to improve the results.

A common “life-cycle” using web analytics to improve something is:

  1. What happened (make sure you’re measuring everything that needs to be measured)
  2. Why it happened (analysis of the data together with domain expertise)
  3. What’s next (domain expertise on how to make improvements)
  4. Start over again

For example:

What happened
The bounce rate on our homepage is 68%. Domain experience tells you that this is abnormally high and can/should be reduced.

Why it happened (analysis of the data together with domain expertise)
Looking at the homepage I see it’s way to busy. Two column layout with equal size columns makes it hard know what to focus on. No clear headline and multiple competing calls to action.

What’s next (domain expertise on how to make improvements)
Split test different layouts, calls to action, headline, amount of content.
Ideally you should be able to provide recommendations for the actual variations to be tested, and not just say you should test this.

So, if you want a career in web analytics, you need to become proficient in one or more of the above topics so you’ll know how to answer the questions “why it happened” and “what’s next”.

It’s not enough to just tell people what’s broken –
You need to tell them how to fix it.

As always, comments are greatly appreciated.

– Ophir

Adwords Search Funnel Update

/ Ophir Prusak / Leave a comment

I just noticed a tweet from analyticspros:

Jeff gillis from @googleanalytics announces new updates to adwords search funnels: up to 90 days back, actual query, unique paths #emetrics

This is GREAT news!

Previously search funnels only showed data back 30 days, which is adequate for many sites, but if your conversion event often happens after 30 days (which is often the case with large item purchases and B2B) you weren’t getting the full picture.

I have not see this mentioned anywhere else, so it’s probably hot off the “press” at eMetrics.

Update [Oct 6]: I just noticed that the official adwords blog has this update

– Ophir

Selling Web Analytics

/ Ophir Prusak / 12 Comments

How to Sell Web Analytics

While I was at the Google Analytics Certified Partner summit this year, someone came up to me and asked:

How do you sell web analytics?

This is a question I deal with quite often, both internally within the agency I work for and when trying to convey the value of web analytics to clients.

The short answer is that you shouldn’t be trying to sell web analytics.

Web analytics is a tool, a means to an ends.

It has no inherent value by itself. It’s only through the analysis of data and providing actionable insights that it creates value.

You should be selling the value that can be gained by a proper web analytics implementation and actionable analysis.

Marketing people talk about selling the benefits, not the features.

Web analytics is a feature.
Improving the bottom line is a benefit.

A simple analogy is HTML (the code used to create web pages).
Imagine if you tried to pitch HTML services to a business. They would probably be scratching their heads as to why they need your services.

Now imagine pitching web site creation services, providing real world examples on how businesses have improved their bottom live with their web site.
Now they’re listening.

When selling services that include web analytics, try starting with the end result and then work your way backwards. If you start with the prize, people will usually pay more attention.

Here’s an example:

  1. I can help you make an additional $80,000 a year
  2. It will cost you a one time investment of $25,000 investment and $1,000 a month
  3. You’re currently doing 3,000 sales a year
  4. I will get you 400 additional sales a year (average sale value is $200)
  5. The additional sales will come from improving your overall conversion rate by 13.3%
  6. Improving the overall conversion rate will coming from decreasing bounce rates, increasing “add to cart” rates and decreasing cart abandonment.
  7. The above improvements will come from making changes on the web site
  8. We’ll test a few options until we find something that works (split testing)
  9. We’ll know what changes to test based on data that tells us what people are doing on your web site
  10. In order to get the data that we need, we have to install web analytics on the site and then analyze the data

The problem is that many web analytics businesses start their pitch with step 10 and then work their way to step 1.

When people ask me what I do for a living, the answers have changed over the years, but now I usually answer along the lines of:

I help web sites do better at whatever they’re try to do.
Specifically, I learn the business objectives and then do some detective work, finding issues that can be improved and then I improve them.

As always, comments and suggestions are appreciated.

Ophir

The Missing Video Metric

/ Ophir Prusak / 3 Comments

waitI’d like to share an “a-ha” moment I recently had while viewing an online video.

As part of my job as a web analyst, I often do site audits.

This entails analyzing the web analytics data (usually Google Analytics, Omniture or WebTrends) in conjunction with viewing the site itself.

While the numbers tell the story of what happened, I try to figure out why it happened by also examining the site and it’s contents.

For example, a site might be tracking video views with data on how many people started viewing each video, how many people viewed at least half of the video and how many people saw the entire video.

In a scenario where there is a large drop-off from people starting to view the video to people viewing half of it, there are several factors which could account for the drop-off.

It could be content related (the video doesn’t hold the viewer’s interest) or it could be technical (the video was stuttering or took to long to start playing) people should know that in order to catch everyone’s attention the video needs to be made by video production companies phoenix or a similar company.

I’ll usually check the minimum bandwidth required to play the video smoothly to see if I think it’s a load time issue, but that’s just circumstantial evidence.

Today, while viewing a video that was constantly buffering (and annoying to watch) I left the page and then realized:

Buffering events should be tracked as a metrics

The following numbers should be tracked

  • How long from clicking on play to when the video actually started to play
  • Every time the video automatically pauses for buffering, you should track the time-stamp (relative to video start) and for how long the video was paused.

This information will help you get from what happened to the why it happened, conclusively telling you if the viewing was abandoned due to load times or something else!

Get more than 5 custom variables in Google Analytics

/ Ophir Prusak / 26 Comments

A few months ago Google Analytics released an awesome new feature – Custom Variables.

If you’re new to custom variables, you should read the official Google help page:
http://code.google.com/apis/analytics/docs/tracking/gaTrackingCustomVariables.html

And check out the webinar they did recently:
http://www.youtube.com/watch?v=UmQTfqmoSyk

While custom variables are great – you only get 5 of them, or so Google says. There is an undocumented feature that basically gives you as many custom variables as you want. It’s as simple as setting the number of maximum custom variables as you want to use. Here is the Google Analytics function to use:

[important update July 2, 2010]
Omar brought to my attention that when drilling down via the standard GA interface, when you pick a custom variable after the 5th one (in order that is was set, not in the order on the interface) you get an error saying:

An Error Has Been Detected
Please try again. Thank you for your patience.

My gut tells me that this is ONLY an issue with the interface and that internally the data is still being recorded, but I can’t know for sure.
In any case it seems that from a functional standpoint creating more than 5 custom variables is fairly limited since currently you can only see the variable names and not the values for slots 6 and up.

Continue reading →

adCost Bug in Google Analytics API

/ Ophir Prusak / 3 Comments

Recently I’ve been using the Google Analytics API to automate some of the report generation I’m doing.

After some serious hair pulling I finally realized that there is a bug in the API when one of the metrics you request is adCost.

Specifically, when requesting adCost with other specific metrics, the results for adCost are always zero.

After some research I found this post which confirms that Google is aware of the adCost issue.

Unfortunately the post doesn’t really help much in terms of finding out which combinations trigger the bug, so I tried out some of the combinations myself found that using any traffic source dimension with these metrics will trigger the bug:

  • entrances
  • exits
  • exit %

If you are aware of other metrics that trigger this bug, please let me know and I’ll add them to the list.

I should add that this bug also affects custom reports!

If you’re using any traffic source dimension and adCost together with any of the above metrics in a custom report you’ll get zero for the adCost data.

Hopefully this saves someone else from some major hair pulling.

Beyond the Page View

/ Ophir Prusak / Leave a comment

I started using web analytics software back in 1996, before it was even called web analytics. At the time we were measuring mostly “hits” and page views.

This is before JavaScript includes became the defacto standard in web tracking. Way before web 2.0. Before JavaScript support was a given. Before Ajax. Before embedded audio and video.

Basically, every web page was a more or less a static page with no on-page interaction.

Now, almost 15 years later, the web is a different place. The “lets measure page views” model for tracking and measuring online activity crumbled a long time ago.

Today a single web page can be a totally interactive experience in itself. We have flash, dynamic HTML with JavaScript, Ajax, embedded videos, etc.

A few years ago Google Analytics added event tracking in order to address the shortcoming of trying to measure everything as a page view.

While this is better than nothing, the problem is that it’s too generic to label everything other than a page view as an event.

Every type of user interaction has it’s own characteristics, and should be measured accordingly.

For example: Forms.
Forms are an entire world in terms of tracking user interaction: Was the form submission successful? What fields were filled out? How long did the user spend on each field? etc.

http://www.ClickTale.com actually does a great job of measuring all of this, but my point is:

We need a new standard dictionary for what elements we can measure and what attributes each element has.

eCommerce transactions are already standard in all web analytics solutions. Most of the higher end analytics solutions are already addressing the overall  issue by adding dedicated event types for video views, downloads, or outbound links.

I’d like to see dedicated events for things like clicks, scrolls, even a “still on the page” event (this is for more accurate time on page measurement)

There will always be always be room for a generic type of event, but in order to reach the next generation of web analytics adoption, we (the industry) really need to expand our horizons beyond page views and events.