I’ve always suspected there is a fine line between engaging people and wasting their time, especially when you’re dealing with B2B sites. After all, when I’m looking for information as part of my workday, I don’t want to spend a lot of time on a site. I don’t want to view a lot of pages: I want to find the answer to my question right away. Even micro-conversions don’t necessarily mean that my visit was successful: maybe I’m printing out pages or emailing myself a link because I don’t have time to wade through the confusion right now.

I was recently doing an analysis for a site, and was curious about the “engagement” level of two important customer segments. I looked at time on site, content pages per visit, bounce rate, navigation to content ratio (i.e. for a ratio of 2, it would mean that on average, for every content page viewed, visitors must view 2 navigation pages), % visits that contained email to friend actions, and % visits that contained at least one print-ready view. I couldn’t factor  in other potential engagement/loyalty metrics (visit frequency, etc.) because of the large number of shared computers and accounts for this particular site, which is heavily used in a workplace setting. Here’s what I found.

• Segment B had half the bounce rate of Segment A (although both were pretty low).
• Segment B spent 30% more time on the site.
• Segment B viewed 30% more content.
• Segment B viewed print-ready pages in 4% more visits, close enough that I’d consider their usage of this function to be roughly equivalent.
• Segment B needed to go through 8% fewer navigational pages in order to find content.
• Usage of the email function was similar for both segments, just slightly higher for Segment A.

By all measures except the two conversions, I would have considered Segment B to be a good bit more “engaged” than Segment A. Even on the print and email conversions they were roughly the same. But my satisfaction surveys told a different story. I used the responses to our “Were you able to find the information you were looking for?” question to double-check overall satisfaction scores. What did I find?

• Segment B scored 2 points lower than Segment A on overall satisfaction (on a 0-100 scale).
• Segment B respondents were 7% less likely to answer “Yes” to the question about whether they found what they are looking for.

I don’t know if the above differences are statistically significant, but what I absolutely do know is that higher performance on engagement-related metrics did not mean that Segment B customers were happier or more satisfied with the site. If anything, it’s just the opposite.

The best thing you can do for a content site (and other sites, too, IMHO) is to install continuous surveys. That’s the only way you will ever be able to assess the quality of your visitors’ experience online. If you can’t afford one of the for-pay surveys that can be well-customized (from ForeSee Results or iPerceptions), there are still a number of customer satisfaction tools, like 4Q and Kampyle, that allow you to ask your customers straight out whether or not their visit was successful.

I assume that most web analytics tools identify the IE version by looking for MSIE X.Y in the browser string. However, this is no longer valid for IE8. This is because the IE8 user agent string will include MSIE 7.0 when in compatibility mode. The difference between the “real” IE7, and IE8 in compatibility mode is the word Trident, which is included in both variants of IE8:

Example of a regular IE8 user agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Example of IE8 in compatibility mode: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Literally thousands of web sites are not compatible with IE8. A list of more than 3,000 incompatible sites is maintained by Microsoft. This list can be downloaded by IE8 users so that the browser can automatically switch itself into compatibility view when a site is encountered that has previously been identified by IE8 users as incompatible. Many more sites are not compatible, but are not on the list because they have lower traffic levels.

Because a visitor can have multiple user agents in one visit, this raises a number of questions:

• Does your analytics tool keep the user agent string from each individual page view, or do they associate one browser with the entire visit?
• If browser is associated with the entire visit, which browser is recorded? If they keep the string on the entry page, then IE8 is likely represented correctly in your data, but you won’t know if users are resorting to compatibility mode in order to view your site. If your analytics tool keeps the last browser string encountered in the visit, then your numbers are likely biased toward IE7 unless your tool is properly grouping this traffic as IE8.
• If browser is associated with page views instead of the visit, then adding up visits in your browser report would give you more than the total visits for your site. In other words, browser visits would not be “summable” the way they are when one can assume that each visit has only one browser. This is not the end of the world, just something to be aware of because it’s not intuitive.
• Does your analytics tool properly group the browsers with both MSIE 7.0 and Trident as IE8? If not, do they expose the entire string so you can do the calculations yourself to see if your site has IE8 issues?
• If you are doing logfile analysis without cookies, sessionization is probably based on IP + User Agent. For sites where I’ve transitioned from logfiles to tags in the same tool, my experience has been that IP/User Agent sessionization tends to over-count visits: this issue will increase that inflation even more. Bear in mind that many tag-based tools resort to IP/UA when cookies are blocked, so there could be a small inflation effect regardless of the type of data-collection you use.

I examined a few of my sites and found the percentage of visits with IE8 to be roughly between 5% and 15%, depending on the site. My B2B sites tend to have lower IE8 penetration, while sites that attract high-tech users will tend to show a higher percentage of the latest browsers.

If your web analytics tool exposes the entire browser string (Google Analytics does not), I recommend you search through your user agent strings looking for Trident, and see for yourself if this is an issue for the sites you analyze. One metric I’m looking at is the percentage of my Trident browser visits that also contain MSIE 7, assuming that sites that are not compatible with IE8 will show a higher percentage of users resorting to compatibility mode. For a site with known IE8 issues I calculated 25% , while another site I randomly chose calculated to 12%. I haven’t examined enough sites yet to know if that means the second site also has IE8 issues, or if it just means it’s “normal” for a certain percentage of IE8 users to surf in compatibility mode. Clearly I have more work to do.

Update: Last night I received an email from a colleague who had read this post, asking why should they care? It’s a fair question so I thought I’d answer it publicly.

First, if you’re asking then you probably aren’t in a situation where you need to care. That’s OK: the lowly browser report isn’t the most important report in your web analytics tool, not by a long shot.

But I can think of a couple of situations where it’s important:

1. When deciding whether or not to fund development changes to enable compatibility with certain browsers, “fewer than 5% of our visits use that browser” is a lot different than “nearly 10% of our visits use that browser”.  The numbers you use for those decisions should be as accurate as practical.

2. Your customer service department may receive emails or phone calls from visitors complaining that they are unable to perform certain tasks on your site (like complete a transaction). When they receive multiple complaints that sound similar but are unable to reproduce the problem in house they may ask you, the analytics ninja, for help defining the scope of the problem. These intermittent issues are difficult to troubleshoot because they’re often environment-related. One starting point is to examine the user experience through that transaction — transaction page views per visit is sometimes sufficient, or you may want to look at a funnel chart for the process — and segment it by different browser versions. If the issue is due to a browser incompatibility, you can sometimes pinpoint it quickly with this type of analysis.

If Fark is to be believed, the Internet is all about porn anyway.

For the record, I don’t analyze porn sites for a living. While I admit I have done analysis for at least one adult-oriented site in the past, this is different. This is the effect of sexually-oriented search terms on websites that have little or nothing to do with sex, websites that I would happily show to my mother. But if you analyze a wide enough variety of sites, you will find that fetishes come in a surprising variety of shapes and sizes, and you’ll be surprised where they, um, pop up.

There are three ways that these “thrill-seekers” may affect your data.

1. By causing a one-time traffic spike. This is more likely to happen for a blog or a news site, when an article mentions something sexual in a fairly innocuous way. For example, this article contains plenty of keywords that may attract traffic that is not part of my target audience (and if you haven’t bounced by now, welcome to the world of web analytics!). This can happen on news or magazine sites that run features on a variety of subjects, and it can often catch the web analyst off guard. For example, consider the more-or-less legitimate — if somewhat sensational — news articles that were all the rage a couple months ago, talking about teens sending naked pictures of themselves to each other on cell phones. When you mention “teens” and “sex” and “naked pictures” in the same article, you’re bound to attract some of that kind of traffic.

This usually only becomes an issue when the traffic spike for a single article is large enough to influence aggregate numbers for the entire week or month. Any sudden spike (or dip) in traffic should always be investigated: it may have been due to a simple editorial choice instead of that awesome marketing campaign that your HiPPO designed.

2. By inflating search engine visits long-term. Perhaps “inflating” isn’t the best term, since the traffic is real, it’s human, and it’s coming from search engines. This situation happens when there are articles or images on your site that are intended for one audience but end up attracting another audience – the kind that’s not likely to become a customer – and it can wreak havoc with your conversion rates. A prime example is a site that publishes medical information intended for a professional medical audience. A thorough enough site will likely contain pictures of certain body parts or descriptions of rare medical procedures, and a glance through some of your top search terms can yield insights into the human psyche that you wish you didn’t know.

Always look past the “Top X” keyword report that is spit out of your web analytics package by default. Look for terms that seem over-represented on a site like yours. Pay careful attention to image searches, and ensure that you can separate image search keywords from text search keywords if necessary.

3. By logging visits that never really happened. This is fairly rare, and you will likely only catch it if a) your analytics are based on server logs instead of JavaScript tags, and b) your site contains one or more unprotected redirect URLs, “pages” that contain a URL as a value in the query string. The symptom is a sudden appearance in your keyword reports of sexually-oriented phrases that have absolutely nothing to do with your site. The cause is a search engine ranking hack, where a site-of-ill-repute manages to get themselves indexed by means of your redirect URLs, using your site’s good reputation to increase their rankings. You can confirm by looking at the entry pages for the offending terms to see if they are the redirect pages.

As with any traffic that is obviously unqualified, you very likely want to segment out the perverts from some of your conversion rate calculations, especially if you are doing optimization efforts on one or more areas of your site. Unqualified traffic volume can be more than enough to skew results and mask changes to real customer behavior. However, I don’t recommend you filter this traffic from your entire data set. If your linking, advertising, or SEO efforts are bringing in the wrong kind of traffic, this is something you really need to know.

1. If the data-pulling part of the analysis can be done in the tool, ask for instructions.

One thing you should know up front: If the answer to your business problem is something that can be found in the tool, you will save a lot of money by doing it yourself. However, not everyone has the time or the expertise to pull and/or interpret the numbers themselves. If you know (or suspect) that this is the case, ask your consultant to include step by step instructions on how to repeat the analysis on your own at a later date, and to document why she reached the conclusions she did. This may increase your estimate by a few hours to cover documentation costs, but you will save in the long run by being able to repeat the analysis as desired at a later date, to establish trends or to see how making changes to your site affects the outcomes. You will also have gained some valuable insight into how an “expert” approaches your particular business problem.

2. Tell your consultant what business problem you’re trying to solve.

Make sure you clearly communicate the specific business problem you are trying to solve. If you know of one methodology that will get the answer you are looking for, don’t be afraid to share. But it is usually best not to over-specify the methodology to be used, even if you think you know exactly what you want. Your analysis tool has a maze of tables and relationships and data under the covers, and there is often more than one way to tease out the same business insight. Allowing the consultant the flexibility to suggest other methodologies could result in significant cost savings, or in added insight for the same amount of hours. For example, asking your consultants for “all paths” that lead to purchase should set off some alarm bells with your consultant. A request for a list of “all” anything – unless the data is intended to be moved into another database for further analysis – is usually a sign that the business question hasn’t been well-formed. A good consultant will try to guide you to the real reason behind your question instead of providing you with a data dump: Were you trying to find out whether users were more likely to search or to navigate before making a purchase? Or were you trying to figure out whether the new tools you added in your product view area led to a higher likelihood to purchase?

3. Beware of “canned” analysis.

I am not talking about your consultant re-using queries and methodologies from prior engagements – that’s smart business, and reduces the time it takes to get answers and the likelihood of errors – but about “analysis” menus from which you can order as though you are in a restaurant. These are seldom tailored to your site, and are based on a one-size-fits-all methodology. These analyses are not all bad – they can serve as a training tool for your own in-house analysts if they include instructions on how to replicate the analysis yourself – but you should expect to pay accordingly.

4. Meet with the consultant before you ask for a quote.

When you first approach your vendor for services and speak with the consultant, insist on a web conference where you can call up your actual website and point to specific areas of concern. Taking time to do this up front helps to avoid misunderstandings, helps your consultant understand any company-specific terminology you use when referring to activity on your site, and should result in a more accurate estimate and more satisfactory analysis.

5. Clearly define — in writing — what the analysis engagement is about.

Finally, don’t be afraid to ask for a “solutions design” document to specify exactly what you’re paying for. Don’t expect the design document to include all of the details: it is hard to run a business when a consultant executes most of the engagement up front, only to have the client decide not to spend the money after all. However, the document should include enough detail about the methodology for the consultant to know how much work is involved, and to minimize any disconnects between what you asked for and what she delivers.

When you contract with your vendor for professional services, you are paying for special access to your site’s dataset, a breadth of knowledge that allows the consultant to apply lessons learned on a variety of sites, and for the business acumen to understand something about your particular business and what makes your site different from everyone else’s. Make the best of the opportunity!