I’ve always suspected there is a fine line between engaging people and wasting their time, especially when you’re dealing with B2B sites. After all, when I’m looking for information as part of my workday, I don’t want to spend a lot of time on a site. I don’t want to view a lot of pages: I want to find the answer to my question right away. Even micro-conversions don’t necessarily mean that my visit was successful: maybe I’m printing out pages or emailing myself a link because I don’t have time to wade through the confusion right now.

I was recently doing an analysis for a site, and was curious about the “engagement” level of two important customer segments. I looked at time on site, content pages per visit, bounce rate, navigation to content ratio (i.e. for a ratio of 2, it would mean that on average, for every content page viewed, visitors must view 2 navigation pages), % visits that contained email to friend actions, and % visits that contained at least one print-ready view. I couldn’t factor  in other potential engagement/loyalty metrics (visit frequency, etc.) because of the large number of shared computers and accounts for this particular site, which is heavily used in a workplace setting. Here’s what I found.

• Segment B had half the bounce rate of Segment A (although both were pretty low).
• Segment B spent 30% more time on the site.
• Segment B viewed 30% more content.
• Segment B viewed print-ready pages in 4% more visits, close enough that I’d consider their usage of this function to be roughly equivalent.
• Segment B needed to go through 8% fewer navigational pages in order to find content.
• Usage of the email function was similar for both segments, just slightly higher for Segment A.

By all measures except the two conversions, I would have considered Segment B to be a good bit more “engaged” than Segment A. Even on the print and email conversions they were roughly the same. But my satisfaction surveys told a different story. I used the responses to our “Were you able to find the information you were looking for?” question to double-check overall satisfaction scores. What did I find?

• Segment B scored 2 points lower than Segment A on overall satisfaction (on a 0-100 scale).
• Segment B respondents were 7% less likely to answer “Yes” to the question about whether they found what they are looking for.

I don’t know if the above differences are statistically significant, but what I absolutely do know is that higher performance on engagement-related metrics did not mean that Segment B customers were happier or more satisfied with the site. If anything, it’s just the opposite.

The best thing you can do for a content site (and other sites, too, IMHO) is to install continuous surveys. That’s the only way you will ever be able to assess the quality of your visitors’ experience online. If you can’t afford one of the for-pay surveys that can be well-customized (from ForeSee Results or iPerceptions), there are still a number of customer satisfaction tools, like 4Q and Kampyle, that allow you to ask your customers straight out whether or not their visit was successful.

I assume that most web analytics tools identify the IE version by looking for MSIE X.Y in the browser string. However, this is no longer valid for IE8. This is because the IE8 user agent string will include MSIE 7.0 when in compatibility mode. The difference between the “real” IE7, and IE8 in compatibility mode is the word Trident, which is included in both variants of IE8:

Example of a regular IE8 user agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Example of IE8 in compatibility mode: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Literally thousands of web sites are not compatible with IE8. A list of more than 3,000 incompatible sites is maintained by Microsoft. This list can be downloaded by IE8 users so that the browser can automatically switch itself into compatibility view when a site is encountered that has previously been identified by IE8 users as incompatible. Many more sites are not compatible, but are not on the list because they have lower traffic levels.

Because a visitor can have multiple user agents in one visit, this raises a number of questions:

• Does your analytics tool keep the user agent string from each individual page view, or do they associate one browser with the entire visit?
• If browser is associated with the entire visit, which browser is recorded? If they keep the string on the entry page, then IE8 is likely represented correctly in your data, but you won’t know if users are resorting to compatibility mode in order to view your site. If your analytics tool keeps the last browser string encountered in the visit, then your numbers are likely biased toward IE7 unless your tool is properly grouping this traffic as IE8.
• If browser is associated with page views instead of the visit, then adding up visits in your browser report would give you more than the total visits for your site. In other words, browser visits would not be “summable” the way they are when one can assume that each visit has only one browser. This is not the end of the world, just something to be aware of because it’s not intuitive.
• Does your analytics tool properly group the browsers with both MSIE 7.0 and Trident as IE8? If not, do they expose the entire string so you can do the calculations yourself to see if your site has IE8 issues?
• If you are doing logfile analysis without cookies, sessionization is probably based on IP + User Agent. For sites where I’ve transitioned from logfiles to tags in the same tool, my experience has been that IP/User Agent sessionization tends to over-count visits: this issue will increase that inflation even more. Bear in mind that many tag-based tools resort to IP/UA when cookies are blocked, so there could be a small inflation effect regardless of the type of data-collection you use.

I examined a few of my sites and found the percentage of visits with IE8 to be roughly between 5% and 15%, depending on the site. My B2B sites tend to have lower IE8 penetration, while sites that attract high-tech users will tend to show a higher percentage of the latest browsers.

If your web analytics tool exposes the entire browser string (Google Analytics does not), I recommend you search through your user agent strings looking for Trident, and see for yourself if this is an issue for the sites you analyze. One metric I’m looking at is the percentage of my Trident browser visits that also contain MSIE 7, assuming that sites that are not compatible with IE8 will show a higher percentage of users resorting to compatibility mode. For a site with known IE8 issues I calculated 25% , while another site I randomly chose calculated to 12%. I haven’t examined enough sites yet to know if that means the second site also has IE8 issues, or if it just means it’s “normal” for a certain percentage of IE8 users to surf in compatibility mode. Clearly I have more work to do.

Update: Last night I received an email from a colleague who had read this post, asking why should they care? It’s a fair question so I thought I’d answer it publicly.

First, if you’re asking then you probably aren’t in a situation where you need to care. That’s OK: the lowly browser report isn’t the most important report in your web analytics tool, not by a long shot.

But I can think of a couple of situations where it’s important:

1. When deciding whether or not to fund development changes to enable compatibility with certain browsers, “fewer than 5% of our visits use that browser” is a lot different than “nearly 10% of our visits use that browser”.  The numbers you use for those decisions should be as accurate as practical.

2. Your customer service department may receive emails or phone calls from visitors complaining that they are unable to perform certain tasks on your site (like complete a transaction). When they receive multiple complaints that sound similar but are unable to reproduce the problem in house they may ask you, the analytics ninja, for help defining the scope of the problem. These intermittent issues are difficult to troubleshoot because they’re often environment-related. One starting point is to examine the user experience through that transaction — transaction page views per visit is sometimes sufficient, or you may want to look at a funnel chart for the process — and segment it by different browser versions. If the issue is due to a browser incompatibility, you can sometimes pinpoint it quickly with this type of analysis.

If Fark is to be believed, the Internet is all about porn anyway.

For the record, I don’t analyze porn sites for a living. While I admit I have done analysis for at least one adult-oriented site in the past, this is different. This is the effect of sexually-oriented search terms on websites that have little or nothing to do with sex, websites that I would happily show to my mother. But if you analyze a wide enough variety of sites, you will find that fetishes come in a surprising variety of shapes and sizes, and you’ll be surprised where they, um, pop up.

There are three ways that these “thrill-seekers” may affect your data.

1. By causing a one-time traffic spike. This is more likely to happen for a blog or a news site, when an article mentions something sexual in a fairly innocuous way. For example, this article contains plenty of keywords that may attract traffic that is not part of my target audience (and if you haven’t bounced by now, welcome to the world of web analytics!). This can happen on news or magazine sites that run features on a variety of subjects, and it can often catch the web analyst off guard. For example, consider the more-or-less legitimate — if somewhat sensational — news articles that were all the rage a couple months ago, talking about teens sending naked pictures of themselves to each other on cell phones. When you mention “teens” and “sex” and “naked pictures” in the same article, you’re bound to attract some of that kind of traffic.

This usually only becomes an issue when the traffic spike for a single article is large enough to influence aggregate numbers for the entire week or month. Any sudden spike (or dip) in traffic should always be investigated: it may have been due to a simple editorial choice instead of that awesome marketing campaign that your HiPPO designed.

2. By inflating search engine visits long-term. Perhaps “inflating” isn’t the best term, since the traffic is real, it’s human, and it’s coming from search engines. This situation happens when there are articles or images on your site that are intended for one audience but end up attracting another audience – the kind that’s not likely to become a customer – and it can wreak havoc with your conversion rates. A prime example is a site that publishes medical information intended for a professional medical audience. A thorough enough site will likely contain pictures of certain body parts or descriptions of rare medical procedures, and a glance through some of your top search terms can yield insights into the human psyche that you wish you didn’t know.

Always look past the “Top X” keyword report that is spit out of your web analytics package by default. Look for terms that seem over-represented on a site like yours. Pay careful attention to image searches, and ensure that you can separate image search keywords from text search keywords if necessary.

3. By logging visits that never really happened. This is fairly rare, and you will likely only catch it if a) your analytics are based on server logs instead of JavaScript tags, and b) your site contains one or more unprotected redirect URLs, “pages” that contain a URL as a value in the query string. The symptom is a sudden appearance in your keyword reports of sexually-oriented phrases that have absolutely nothing to do with your site. The cause is a search engine ranking hack, where a site-of-ill-repute manages to get themselves indexed by means of your redirect URLs, using your site’s good reputation to increase their rankings. You can confirm by looking at the entry pages for the offending terms to see if they are the redirect pages.

As with any traffic that is obviously unqualified, you very likely want to segment out the perverts from some of your conversion rate calculations, especially if you are doing optimization efforts on one or more areas of your site. Unqualified traffic volume can be more than enough to skew results and mask changes to real customer behavior. However, I don’t recommend you filter this traffic from your entire data set. If your linking, advertising, or SEO efforts are bringing in the wrong kind of traffic, this is something you really need to know.

How bad is it? As usual, it depends. A site where people tend to visit on a daily basis will see more inflation from cookie-deletion than a site that is only visited once per month. In the former example, one person may count toward the monthly total as many as 30 or so times, while in the latter, even a frequent cookie-deleter would only count once.

Let’s pretend that we know something about the actual people visiting a site, and see if we can determine by how much our web analytics numbers might be affected by cookie-deletion.

In order to make the calculations easy, I have assumed people only visit or delete their cookies on daily, weekly, or monthly boundaries, and I am only considering a one month time frame. However, the same logic could be applied to more granular data. It ultimately boils down to a matrix algebra problem, but I doubt many of us are eager to get into that level of detail.

An example

Consider 10,000 people: not 10,000 “cookies” and not 10,000 “unique visitors,” but 10,000 real-life, carbon-based beings. Suppose we are able to observe these people in such a way that we know “the truth” about their online behavior. Suppose also we have observed that, on average, 10% of our people delete their cookies every day, 15% delete once per week, and the remainder delete their cookies monthly or less frequently.

We have also observed that 20% of these people visit our website every day, 30% visit once per week, and the remainder only visit once in a given month. There’s no reason for these people to login to our website — all visits are anonymous — and we count Unique Visitors using a cookie.

How bad is it?

Our first step is to find out how many different cookies each person will receive over the course of the month, based on the number of times they visit our site and how often they delete their cookies. For simplicity’s sake, we’ll assume each month has 30 days and 4 weeks.

Daily deleters will receive a different cookie each time they visit, so daily visitors will log 30 cookies and weekly visitors log 4. Weekly deleters who visit every day will log 4 different cookies over the course of the month, as will weekly deleters who visit once per week. Everybody else’s activity will be logged with one cookie. We can summarize as shown below.

Using the above factors, we can determine the cookie-contribution from each set of people:

Wow. Our 10,000 people are being represented as 18,950 unique visitors: the Unique Visitors number is inflated by 90%!

Visitor loyalty reports are affected, too

Unique Visitors isn’t the only number that’s affected by cookie-deletion. Any visitor-based number is going to be off, so you have a lot of trouble understanding visitor loyalty. You can tell when your efforts to improve loyalty are working, since the numbers will move in the right direction, but the magnitude of change will be misleading.

For the above example, we know that 20% of our people visited daily, 30% visited weekly, and 50% visited monthly, so the number of visits in a month works out to 77,000 (2000 x 30 + 3000 x 4 + 5000). Visits aren’t affected by cookie-deletion to any great extent (a good argument for visit-based analysis!) – so our tool will also report 77,000 visits.

This means our people averaged 7.70 visits (77000/10000) over the course of the month, but our web analytics tool will only report 4.06 (77000/18950) visits per visitor because the visitors are inflated. Our average visits per visitor are under-reported by 47%!

If you prefer to view loyalty using a histogram (# visitors who visited once, twice, three times, etc.), then we need to determine to which bin each visitor’s cookies will be credited.

The first thing that jumps out of this table is that the majority of the cookies are only encountered once, regardless of how many times someone actually visited. This explains why visitor loyalty graphs, regardless of the tool used, are often overloaded with so many one-time visitors.

visitor duration graphs

Applying the frequencies in the histogram table to the numbers in the calculations table show us how our visitor retention graph is affected by cookie-deletion.

visitor frequency corrections

Again, wow! While 20% of our people visited the site every day, with cookie-based visitor counting, only 8% appear in this super-loyal segment. The majority of the “visitors” that were added to the site via cookie-deletion appear in the 1 visit bin, inflating that number by almost a factor of 3.

Adding Authentication

If 100% of the people to the above site authenticated, and the authenticated visitor identifier were used to count unique visitors, then the number would be pretty accurate (ignoring shared logins, etc.). But most sites don’t require authentication to see certain pages, so the likelihood of 100% authentication is low except in special cases, like intranets.

For our example above, what if half the visitors authenticated? Half of the 10,000 people would be more-or-less accurately represented, while our cookie-deletion calculations would apply to the remaining 5,000. The unique visitor multiplier factor decreases with increasing percentage of authenticated people.

Assumed: 10/15/75 cookie deletion and 20/30/50 visiting frequency (daily/weekly/monthly).

Is it always that bad?

Not necessarily, it could be worse or it could be better. The above examples assumed that 20% of people visited the website every day, and 30% visited weekly. This was an arbitrary example meant to make calculations easier. In real life, you may have far fewer daily visitors (or more, it just depends on the site). Running the numbers assuming 5% daily and 50% weekly visitors, for example, results in a unique visitor inflation of 1.5 instead of the 1.9 calculated in our example.

I’ve attached my spreadsheet so you can run your own what-ifs.