How can a UX professional – and indeed, the entire business – benefit from web analytics? There are a number of ways.

Analytics can help UX focus their efforts where they can have the most impact. When site changes are suggested or when user pain points are uncovered, look at your analytics data to see how many users are affected. This helps the user testing team (and the business owner, too) allocate scarce resources to the areas of the site where they can make the most impact. Changes that affect only a small number of users, or are within a section of the site that is not used very often, should take lower priority over improvements that can affect more users unless there is a specific reason (e.g. the handful of dissatisfied users represent your most profitable customers).

Look for user stumbling blocks that you don’t even know you have. Scenario analysis (a/k/a funnel analysis) shows progression through a process on the site. Finding disproportionately large abandonment on certain steps of the process can point to user issues with the process. Similarly, errors on the site and in forms also point to areas where users are not having a good experience.

Help to formulate questions that can be answered through user testing. For example, analytics can identify little-used tools or sections of a website, but it can’t tell us why those sections are underutilized. User testing, however, can tease out whether it’s because those sections are poorly designed and implemented, or if they’re just not part of why people come to the site in the first place. In the former case, improving the design or content is justified. In the latter, the business can avoid spending resources improving features your users don’t care about.

Ensure test subjects are representative of actual site users. If a website contains links to information for different audiences (e.g. links for students vs. teachers, or resources for small businesses vs. large businesses), you can often get an idea of the breakdown of those populations by looking at relative usage in each area.  For sites requiring authentication, web analysts also often have access to demographic information in addition to the behavioral data, and can help profile the different segments that should be represented in the user tests.

Last, but certainly not least, web analytics can help quantify and document how UX adds value to a project. Before a redesign, UX and WA should work together to determine which metrics show that a visit is or isn’t successful. These metrics can be collected (or pulled historically) for several weeks or months before the redesign, and compared with the same metrics after the redesign to provide concrete evidence of what works and what doesn’t, and how much impact the changes made.

I’ve always suspected there is a fine line between engaging people and wasting their time, especially when you’re dealing with B2B sites. After all, when I’m looking for information as part of my workday, I don’t want to spend a lot of time on a site. I don’t want to view a lot of pages: I want to find the answer to my question right away. Even micro-conversions don’t necessarily mean that my visit was successful: maybe I’m printing out pages or emailing myself a link because I don’t have time to wade through the confusion right now.

I was recently doing an analysis for a site, and was curious about the “engagement” level of two important customer segments. I looked at time on site, content pages per visit, bounce rate, navigation to content ratio (i.e. for a ratio of 2, it would mean that on average, for every content page viewed, visitors must view 2 navigation pages), % visits that contained email to friend actions, and % visits that contained at least one print-ready view. I couldn’t factor  in other potential engagement/loyalty metrics (visit frequency, etc.) because of the large number of shared computers and accounts for this particular site, which is heavily used in a workplace setting. Here’s what I found.

• Segment B had half the bounce rate of Segment A (although both were pretty low).
• Segment B spent 30% more time on the site.
• Segment B viewed 30% more content.
• Segment B viewed print-ready pages in 4% more visits, close enough that I’d consider their usage of this function to be roughly equivalent.
• Segment B needed to go through 8% fewer navigational pages in order to find content.
• Usage of the email function was similar for both segments, just slightly higher for Segment A.

By all measures except the two conversions, I would have considered Segment B to be a good bit more “engaged” than Segment A. Even on the print and email conversions they were roughly the same. But my satisfaction surveys told a different story. I used the responses to our “Were you able to find the information you were looking for?” question to double-check overall satisfaction scores. What did I find?

• Segment B scored 2 points lower than Segment A on overall satisfaction (on a 0-100 scale).
• Segment B respondents were 7% less likely to answer “Yes” to the question about whether they found what they are looking for.

I don’t know if the above differences are statistically significant, but what I absolutely do know is that higher performance on engagement-related metrics did not mean that Segment B customers were happier or more satisfied with the site. If anything, it’s just the opposite.

The best thing you can do for a content site (and other sites, too, IMHO) is to install continuous surveys. That’s the only way you will ever be able to assess the quality of your visitors’ experience online. If you can’t afford one of the for-pay surveys that can be well-customized (from ForeSee Results or iPerceptions), there are still a number of customer satisfaction tools, like 4Q and Kampyle, that allow you to ask your customers straight out whether or not their visit was successful.

I assume that most web analytics tools identify the IE version by looking for MSIE X.Y in the browser string. However, this is no longer valid for IE8. This is because the IE8 user agent string will include MSIE 7.0 when in compatibility mode. The difference between the “real” IE7, and IE8 in compatibility mode is the word Trident, which is included in both variants of IE8:

Example of a regular IE8 user agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Example of IE8 in compatibility mode: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Literally thousands of web sites are not compatible with IE8. A list of more than 3,000 incompatible sites is maintained by Microsoft. This list can be downloaded by IE8 users so that the browser can automatically switch itself into compatibility view when a site is encountered that has previously been identified by IE8 users as incompatible. Many more sites are not compatible, but are not on the list because they have lower traffic levels.

Because a visitor can have multiple user agents in one visit, this raises a number of questions:

• Does your analytics tool keep the user agent string from each individual page view, or do they associate one browser with the entire visit?
• If browser is associated with the entire visit, which browser is recorded? If they keep the string on the entry page, then IE8 is likely represented correctly in your data, but you won’t know if users are resorting to compatibility mode in order to view your site. If your analytics tool keeps the last browser string encountered in the visit, then your numbers are likely biased toward IE7 unless your tool is properly grouping this traffic as IE8.
• If browser is associated with page views instead of the visit, then adding up visits in your browser report would give you more than the total visits for your site. In other words, browser visits would not be “summable” the way they are when one can assume that each visit has only one browser. This is not the end of the world, just something to be aware of because it’s not intuitive.
• Does your analytics tool properly group the browsers with both MSIE 7.0 and Trident as IE8? If not, do they expose the entire string so you can do the calculations yourself to see if your site has IE8 issues?
• If you are doing logfile analysis without cookies, sessionization is probably based on IP + User Agent. For sites where I’ve transitioned from logfiles to tags in the same tool, my experience has been that IP/User Agent sessionization tends to over-count visits: this issue will increase that inflation even more. Bear in mind that many tag-based tools resort to IP/UA when cookies are blocked, so there could be a small inflation effect regardless of the type of data-collection you use.

I examined a few of my sites and found the percentage of visits with IE8 to be roughly between 5% and 15%, depending on the site. My B2B sites tend to have lower IE8 penetration, while sites that attract high-tech users will tend to show a higher percentage of the latest browsers.

If your web analytics tool exposes the entire browser string (Google Analytics does not), I recommend you search through your user agent strings looking for Trident, and see for yourself if this is an issue for the sites you analyze. One metric I’m looking at is the percentage of my Trident browser visits that also contain MSIE 7, assuming that sites that are not compatible with IE8 will show a higher percentage of users resorting to compatibility mode. For a site with known IE8 issues I calculated 25% , while another site I randomly chose calculated to 12%. I haven’t examined enough sites yet to know if that means the second site also has IE8 issues, or if it just means it’s “normal” for a certain percentage of IE8 users to surf in compatibility mode. Clearly I have more work to do.

Update: Last night I received an email from a colleague who had read this post, asking why should they care? It’s a fair question so I thought I’d answer it publicly.

First, if you’re asking then you probably aren’t in a situation where you need to care. That’s OK: the lowly browser report isn’t the most important report in your web analytics tool, not by a long shot.

But I can think of a couple of situations where it’s important:

1. When deciding whether or not to fund development changes to enable compatibility with certain browsers, “fewer than 5% of our visits use that browser” is a lot different than “nearly 10% of our visits use that browser”.  The numbers you use for those decisions should be as accurate as practical.

2. Your customer service department may receive emails or phone calls from visitors complaining that they are unable to perform certain tasks on your site (like complete a transaction). When they receive multiple complaints that sound similar but are unable to reproduce the problem in house they may ask you, the analytics ninja, for help defining the scope of the problem. These intermittent issues are difficult to troubleshoot because they’re often environment-related. One starting point is to examine the user experience through that transaction — transaction page views per visit is sometimes sufficient, or you may want to look at a funnel chart for the process — and segment it by different browser versions. If the issue is due to a browser incompatibility, you can sometimes pinpoint it quickly with this type of analysis.

If Fark is to be believed, the Internet is all about porn anyway.

For the record, I don’t analyze porn sites for a living. While I admit I have done analysis for at least one adult-oriented site in the past, this is different. This is the effect of sexually-oriented search terms on websites that have little or nothing to do with sex, websites that I would happily show to my mother. But if you analyze a wide enough variety of sites, you will find that fetishes come in a surprising variety of shapes and sizes, and you’ll be surprised where they, um, pop up.

There are three ways that these “thrill-seekers” may affect your data.

1. By causing a one-time traffic spike. This is more likely to happen for a blog or a news site, when an article mentions something sexual in a fairly innocuous way. For example, this article contains plenty of keywords that may attract traffic that is not part of my target audience (and if you haven’t bounced by now, welcome to the world of web analytics!). This can happen on news or magazine sites that run features on a variety of subjects, and it can often catch the web analyst off guard. For example, consider the more-or-less legitimate — if somewhat sensational — news articles that were all the rage a couple months ago, talking about teens sending naked pictures of themselves to each other on cell phones. When you mention “teens” and “sex” and “naked pictures” in the same article, you’re bound to attract some of that kind of traffic.

This usually only becomes an issue when the traffic spike for a single article is large enough to influence aggregate numbers for the entire week or month. Any sudden spike (or dip) in traffic should always be investigated: it may have been due to a simple editorial choice instead of that awesome marketing campaign that your HiPPO designed.

2. By inflating search engine visits long-term. Perhaps “inflating” isn’t the best term, since the traffic is real, it’s human, and it’s coming from search engines. This situation happens when there are articles or images on your site that are intended for one audience but end up attracting another audience – the kind that’s not likely to become a customer – and it can wreak havoc with your conversion rates. A prime example is a site that publishes medical information intended for a professional medical audience. A thorough enough site will likely contain pictures of certain body parts or descriptions of rare medical procedures, and a glance through some of your top search terms can yield insights into the human psyche that you wish you didn’t know.

Always look past the “Top X” keyword report that is spit out of your web analytics package by default. Look for terms that seem over-represented on a site like yours. Pay careful attention to image searches, and ensure that you can separate image search keywords from text search keywords if necessary.

3. By logging visits that never really happened. This is fairly rare, and you will likely only catch it if a) your analytics are based on server logs instead of JavaScript tags, and b) your site contains one or more unprotected redirect URLs, “pages” that contain a URL as a value in the query string. The symptom is a sudden appearance in your keyword reports of sexually-oriented phrases that have absolutely nothing to do with your site. The cause is a search engine ranking hack, where a site-of-ill-repute manages to get themselves indexed by means of your redirect URLs, using your site’s good reputation to increase their rankings. You can confirm by looking at the entry pages for the offending terms to see if they are the redirect pages.

As with any traffic that is obviously unqualified, you very likely want to segment out the perverts from some of your conversion rate calculations, especially if you are doing optimization efforts on one or more areas of your site. Unqualified traffic volume can be more than enough to skew results and mask changes to real customer behavior. However, I don’t recommend you filter this traffic from your entire data set. If your linking, advertising, or SEO efforts are bringing in the wrong kind of traffic, this is something you really need to know.

How bad is it? As usual, it depends. A site where people tend to visit on a daily basis will see more inflation from cookie-deletion than a site that is only visited once per month. In the former example, one person may count toward the monthly total as many as 30 or so times, while in the latter, even a frequent cookie-deleter would only count once.

Let’s pretend that we know something about the actual people visiting a site, and see if we can determine by how much our web analytics numbers might be affected by cookie-deletion.

In order to make the calculations easy, I have assumed people only visit or delete their cookies on daily, weekly, or monthly boundaries, and I am only considering a one month time frame. However, the same logic could be applied to more granular data. It ultimately boils down to a matrix algebra problem, but I doubt many of us are eager to get into that level of detail.

An example

Consider 10,000 people: not 10,000 “cookies” and not 10,000 “unique visitors,” but 10,000 real-life, carbon-based beings. Suppose we are able to observe these people in such a way that we know “the truth” about their online behavior. Suppose also we have observed that, on average, 10% of our people delete their cookies every day, 15% delete once per week, and the remainder delete their cookies monthly or less frequently.

We have also observed that 20% of these people visit our website every day, 30% visit once per week, and the remainder only visit once in a given month. There’s no reason for these people to login to our website — all visits are anonymous — and we count Unique Visitors using a cookie.

How bad is it?

Our first step is to find out how many different cookies each person will receive over the course of the month, based on the number of times they visit our site and how often they delete their cookies. For simplicity’s sake, we’ll assume each month has 30 days and 4 weeks.

Daily deleters will receive a different cookie each time they visit, so daily visitors will log 30 cookies and weekly visitors log 4. Weekly deleters who visit every day will log 4 different cookies over the course of the month, as will weekly deleters who visit once per week. Everybody else’s activity will be logged with one cookie. We can summarize as shown below.

Using the above factors, we can determine the cookie-contribution from each set of people:

Wow. Our 10,000 people are being represented as 18,950 unique visitors: the Unique Visitors number is inflated by 90%!

Visitor loyalty reports are affected, too

Unique Visitors isn’t the only number that’s affected by cookie-deletion. Any visitor-based number is going to be off, so you have a lot of trouble understanding visitor loyalty. You can tell when your efforts to improve loyalty are working, since the numbers will move in the right direction, but the magnitude of change will be misleading.

For the above example, we know that 20% of our people visited daily, 30% visited weekly, and 50% visited monthly, so the number of visits in a month works out to 77,000 (2000 x 30 + 3000 x 4 + 5000). Visits aren’t affected by cookie-deletion to any great extent (a good argument for visit-based analysis!) – so our tool will also report 77,000 visits.

This means our people averaged 7.70 visits (77000/10000) over the course of the month, but our web analytics tool will only report 4.06 (77000/18950) visits per visitor because the visitors are inflated. Our average visits per visitor are under-reported by 47%!

If you prefer to view loyalty using a histogram (# visitors who visited once, twice, three times, etc.), then we need to determine to which bin each visitor’s cookies will be credited.

The first thing that jumps out of this table is that the majority of the cookies are only encountered once, regardless of how many times someone actually visited. This explains why visitor loyalty graphs, regardless of the tool used, are often overloaded with so many one-time visitors.

visitor duration graphs

Applying the frequencies in the histogram table to the numbers in the calculations table show us how our visitor retention graph is affected by cookie-deletion.

visitor frequency corrections

Again, wow! While 20% of our people visited the site every day, with cookie-based visitor counting, only 8% appear in this super-loyal segment. The majority of the “visitors” that were added to the site via cookie-deletion appear in the 1 visit bin, inflating that number by almost a factor of 3.

Adding Authentication

If 100% of the people to the above site authenticated, and the authenticated visitor identifier were used to count unique visitors, then the number would be pretty accurate (ignoring shared logins, etc.). But most sites don’t require authentication to see certain pages, so the likelihood of 100% authentication is low except in special cases, like intranets.

For our example above, what if half the visitors authenticated? Half of the 10,000 people would be more-or-less accurately represented, while our cookie-deletion calculations would apply to the remaining 5,000. The unique visitor multiplier factor decreases with increasing percentage of authenticated people.

Assumed: 10/15/75 cookie deletion and 20/30/50 visiting frequency (daily/weekly/monthly).

Is it always that bad?

Not necessarily, it could be worse or it could be better. The above examples assumed that 20% of people visited the website every day, and 30% visited weekly. This was an arbitrary example meant to make calculations easier. In real life, you may have far fewer daily visitors (or more, it just depends on the site). Running the numbers assuming 5% daily and 50% weekly visitors, for example, results in a unique visitor inflation of 1.5 instead of the 1.9 calculated in our example.

I’ve attached my spreadsheet so you can run your own what-ifs.

In a nutshell: Eric thinks the WAA definition sucks, the WAA should adopt the IAB definitions, all of the web analytics tools should immediately change their Unique Visitors metric to Unique Cookies, and the WAA Standards Committee is being asinine (his word, not mine) because they don’t change direction and adopt the IAB metrics.

The WAA Standards Committee did not seek out a urinary Olympiad with either the IAB or with Eric Peterson, but there are a lot of points out there regarding the IAB’s Audience Reach Guidelines and the WAA Standards that are not being addressed through Eric’s bully pulpit. As long-time co-chair of the WAA Standards Committee — first with Jason Burby, and more recently with Judith Pascual, both from ZAAZ — I would like to offer some more insights. Some of this is my own perspective as a practitioner, but most of it has been discussed ad nauseam in our committee meetings over the past three years and represents the committee’s position on the subject.

Redefining a metric that has been in use for a decade, and is unlikely to change in a lot of tools, will only increase confusion.

Unique visitors suck, and a better metric should be available. But as long as we are using the same terminology as the sucky metric we’ve always used, we’re not fixing anything. This is really my major beef with the IAB specification, as I applaud their efforts to invent a metric that does a better job measuring people.

Unique visitors has been a technology-based metric for over a decade. It seems unlikely that vendors are going to change either their technology or its terminology en masse. This means that as an analyst, you will still be stuck trying to figure out what it means when your vendor reports unique visitors, except you’ll have a lot more variations thrown into the mix.

A different name for an improved metric (unique people? unique individuals? reach?) would provide little doubt that the tool in question is reporting something different, since the new metric wouldn’t even exist in non-compliant tools.

Data from two different IAB-compliant tools still won’t be comparable.

Why? Because the IAB spec doesn’t tell you how to correct the numbers, just that you’re supposed to do it. A “scientific” method must be used, and it must be based on information obtained directly from people, and beyond that you are relying on the mathematical and/or statistical acumen of the people reporting the numbers. The way the spec is written, they can correct the data however they want as long as they can make an argument that it’s people-based. And you are likely to be comparing panel-based corrections with cookie-deletion-algorithm-based corrections with registration-based corrections as you go from site to site.

Should you correct detail data? Or aggregate data?

This is not made clear in the IAB’s spec, either, so it’s yet another way that numbers reported by two different tools or companies could be different.

Web analytics data is collected on a record-by-record basis. Someone makes a request on a website that is instrumented for analytics, and a record of that request, along with any other information your tool collects (cookies, user ID, referrer, user agent, custom data, etc.), is made. This record is put into one or more tables in the database that makes up the guts of your web analytics tool. The majority of what happens when you “do” web analytics is based on queries of that database.

Should correction for actual people take place at the detail level, on those records in the database? Or is there a fudge factor that is applied to the monthly (weekly, quarterly, whatever) data you report from the tool?

And would you use the same terminology for a metric corrected at the detail level as for a metric corrected in aggregate?

Privacy is important to our customers, so it is important to us.

Privacy is the 800 pound gorilla sitting right in the middle of this discussion, and if we’re not careful he’s going to start flinging poo. In creating a standard metric, we cannot force sites to identify the people that visit their site, if those people do not want to be identified.

We have to balance our need for accurate metrics with the privacy considerations of our visitors and ease of use of our websites. Many Internet users simply do not want to be identified, either for privacy reasons or for convenience, and for many sites there really isn’t any benefit to the user of logging in.

There are even legal issues for sites used by children under 13 (Children’s Online Privacy Protection Act, or COPPA) and for sites that deal with medical information, like insurance or medical advice sites (Health Insurance Portability and Accountability Act, or HIPPA). These laws and others like them don’t prohibit login — indeed, sometimes it’s necessary — but they do throw another business consideration into the mix, and often it’s easier for the business to go out of their way to ensure they *can’t* identify individuals, so they don’t have to deal with any potential Personally Identifiable Information (PII).

I have read suggestions about using unique device identifiers instead of cookies to measure Unique Visitors so that no PII would be required. In fact this is what is used for some types of mobile analytics, as mobile devices have a unique identifier. From a privacy standpoint, this could spell disaster for our industry. From a user standpoint, a unique device identifier is the equivalent of a 3rd party cookie that is tied to you and only you: one that you can never delete. Your online behavior can be tracked from site to site to site. Claims about the “anonymity” of a unique device identifier are going to be irrelevant to privacy advocates, since a user’s search data can be and has been used to tie “anonymous” information back to individuals.

The audience for the IAB’s standard is unclear, and attempts to clarify it have failed.

Eric’s posts about the IAB spec have put a spotlight on the confusion over just who the IAB specification was written for, and who would be expected to comply. Unfortunately, the IAB (Joe Laszlo and his colleague Sherril Mane) gave me a different answer than the answer Eric wrote about in his post: I was told that the definitions most likely would be applicable to web analytics vendors.

So who were IAB requirements written for? Your guess is as good as mine.

On the same phone call, Joe and Sherril also told Jodi and me that the IAB didn’t intend to change anything about their standard, regardless of any input they might receive. The document was “thoroughly vetted” by their stakeholders, and that was the final say. Which brings me to my next point…

If you are reading this post, odds are the IAB doesn’t represent you.

Does that make them bad? No. Does that make them irrelevant? No. But they represent a different industry and a different set of stakeholders than the WAA, and your interests as a web analyst are not even a tiny part of their consideration.

Companies pay anywhere from $5,000 (for non-voting associate membership) to upwards of $300,000 per year to be members of the IAB. These are the interests they represent. You may or may not work for someone who “has a say,” and there are no individual members.

The WAA, on the other hand, is comprised of practitioners, consultants, and vendors in the Web Analytics field. This is who we represent. A recent survey by the WAA Research Committee (login required) reported that over half of the members responding have individual, not corporate membership. Many of us individual members, myself included, pay the ~$200 WAA membership fee out of our own pockets.

Why does this matter? Because we on the Standards Committee work in the web analytics field, we strive to consider how the work we do affects real practitioners analyzing different types of websites. Over the years, we have had significant participation by practitioners working “in the trenches” at companies as diverse as Disney, Saks Fifth Avenue, IBM, Reed Elsevier, FedEx, The Motley Fool, ANSI, Adobe, CNN, Ford, Intuit, AOL, Yahoo!, and more. Many of these people are individual members and many of them have been with us for the better part of our tenure as a committee, even as they moved from one company to another and were able to bring in experience from multiple perspectives.

We don’t write standards only for sites that buy or sell advertising. A definition that applies to all sites must actually work for all site types. This includes e-commerce sites, portal sites, subscription-based sites, pure content sites, lead-generation sites, and self-service sites in addition to ad-based sites. And standards that we write must benefit the analyst more than they confuse him or her.

Because we don’t currently have a good way of measuring people that is applicable to all sites, or to even a majority of sites, we have not defined one. Instead, we have attempted to capture the most common practices of today.

Unique Visitors (per the WAA definition) and Unique Cookies are not the same thing.

Eric has suggested more than once that the easy solution to this issue is for web analytics tools to relabel their Unique Visitors numbers as Unique Cookies . I quote: “Web analytics practitioners (and theoretically the vendors) will learn to use ‘Unique Cookies’ since that is a technically correct and 100% accurate description of the data.”

But it isn’t accurate. Using cookies is indeed the most prominent method of identifying Unique Visitors, but it isn’t the only one. Calling that a 100% accurate description of the data is a huge error. Analysts who care about more accurate unique visitor counts are very likely not using cookies to count unique visitors/users. They are probably using registration data from purchases, or user IDs from logins. This is allowed, and preferred, by the WAA standard.

Even if you are using cookies to count your Unique Visitors, they are probably not being used for 100% of the count. Some of your visitors block the JavaScript that sets the cookie. Your WA tool more likely than not has a hierarchy of calculations they use to estimate the “anonymous” (or “guessed”) unique visitors.

If your tool analyzes logfiles, and your site does not drop tracking cookies itself, then the Unique Visitors metric in your analytics tool counts the number of different IP address/User Agent (browser + operating system) combinations it finds. This isn’t even close to counting Unique Cookies.

Changing the name of a bad metric in the tools, to a metric that doesn’t describe what the tool does, doesn’t fix a thing.

What is a “standard,” anyway?

Eric’s reply to me on the Yahoo Group defines what is probably the crux of the issue:

“Perhaps we disagree on what a ’standard’ is. I think a standard should describe the way things ** should be ** not the way things (unfortunately) are.”

Yes, we definitely disagree on that point, and perhaps that is what makes this discussion so difficult. I would call the way things should be a requirement, not a standard. It’s no doubt true that standard can be interpreted either way.

The dozens of people – practitioners, consultants, and vendors — who have been meeting every other week for the last three plus years to help establish the WAA standards have chosen to use the standards to describe reality as we know it, so that’s the way I think of a standard. We think that, at this point, it is more important for analysts to understand what they are measuring than to change what they are supposed to measure, and that has been the focus of our committe’s efforts to date. This doesn’t mean we will never define the future as we think it should be, only that the industry is young, and we believe this is the appropriate first step.

And I really couldn’t let this one slide.

Finally, in one of his posts Eric says “…considering the fact [emphasis mine] that the WAA’s definition is wrong…”

I know I should probably ignore that statement. The issues here are a matter of interpretation: the WAA is not wrong, and Eric is not wrong (as long as he stays away from black-and-white, I’m-right- and-you-are-wrong arguments), and — despite my and the committee’s disagreement — the IAB is not necessarily wrong. We represent disparate opinions about what terms and definitions will most benefit those in our respective industries. But I can’t help but point out:

The WAA definition of Unique Visitors describes how nearly every single analytics tool that exists calculates Unique Visitors. The WAA definition of Unique Visitors is also essentially in agreement with the JICWEBS definition of Unique Users. (For the US-centric among us, JICWEBS is the Joint Industry Committee for Web Standards, a standard-setting body from the UK and Ireland. It includes ABCe and the IAB UK.)

On the other hand, the IAB definition takes an admittedly flawed metric and completely redefines it. It represents the way almost no one calculates unique visitors or unique users.

And yet, it’s a fact that the WAA is wrong? Oh really?

Photo credits:

O RLY Owl: http://www.hjo3.net/orly/gallery1.htm

Gorilla: User noladoc30 on stock.xchng (http://www.sxc.hu/photo/1112963)

Commands used in this post:

xargs - allows you to use a list generated by one command as arguments for another command

mv - moves or renames a file

ls - lists files

tar - an archive utility that lets you store and extract multiple files; although it can be used to write files to tape, one very useful property is that it can also write output to a regular file, which is extremely handy if you need to transfer a bunch of files to your PC (WinZip can handle tarfiles)

Let’s say you have a lot of files in your directory named test1.txt, test2.txt, test3.txt, and so on, and you want to append .old to all of them. You could run a mv (move) command on each individual file, or you could rename all of them at once using xargs as follows:

ls -1 test* | xargs ABC mv ABC ABC.old

• ls -1 (that’s dash-”one”, not a lowercase “L”) lists the files so that there is only one file per line

• Each file name is temporarily held in a variable called ABC, and the rest is just substitution (mv test1.txt test1.txt.old, etc.).

I ran into a situation last week with a batch job that spits out thousands of reports, each into its own subdirectory. After all the reports had run, an issue was found that affected several hundred of them, which were then rerun. Because the reports had already been archived and moved to their final location (not difficult, but time-consuming due to sheer volume), we only wanted to re-archive the reports that had been rerun that day. This was relatively easy to do thanks to xargs. Here’s the command I used:

ls -l */*.out | grep “Jan 21″ | sed ‘s/^.*Jan 21 ..:.. //’ | xargs tar cvf reportreruns.tar

• ls -l (this time it’s a lowercase “L”, not a “one”) does a long listing, which includes permissions, size, owner, date stamp, etc. The first few lines for the listing look something like this:

-rw-r–r– 1 owner group 1234567 Jan 18 22:07 subdir1/report.out
-rw-r–r– 1 owner group 1234567 Jan 21 05:46 subdir2/report.out
-rw-r–r– 1 owner group 1234567 Jan 19 09:31 subdir3/report.out
-rw-r–r– 1 owner group 1234567 Jan 21 05:57 subdir4/report.out

• The grep command (see prior post for more information) pulls out only those lines that include the string “Jan 21″.

• The sed command (see prior post for more information) strips off everything up to the subdirectory and file. Quite literally, it’s substituting everything from the beginning of the line (^) to “Jan 21″, a space, two characters (. is a wildcard for a single character), a colon, two more characters (that just took care of the time stamp), and another space – and replaces it with nothing.

• The xarg command takes all of those files and passes them to the tar command. Notice I didn’t use the variable (ABC) this time: xargs will automatically put the arguments at the end of the following command, so you only need to use a variable if the arguments appear elsewhere in the command.

• The cvf modifier to tar tells it to create (”c”) the archive, see verbose (”v”) messages (it’ll output a message as each file is added: it’s optional), and write the archive to a file (”f”) named reportreruns.tar. The tar command needs to be followed by a list of files to put into the archive, but xargs has already taken care of that.

Here is an introduction to some of my favorite commands:

grep – used to find lines that contain a certain string or regular expression (note that regular expressions are not fully supported in the default grep command for some Unix systems)

cut – used to pull out specific columns from a file based on a specified delimiter; most logs are space-delimited

awk – a programming language that can parse through text files; short pieces of code can be used on the command line

sort – sort the output; the –n modifier sorts numerically; can use a –t modifier to sort on something other than the beginning of the line

uniq – eliminate duplicate lines; the –c modifier shows a count of how many times each line appears

In order to make “cut” work, you need to know which fields contain your data of interest. If you use the “combined” log format, the following table lists the fields where data is located. Cutting out cookie data can be a bit more difficult: we’re using cut with a space delimiter, but spaces can be contained in the user agent field so pulling out cookie values takes a little more work.

In Unix environments, you are allowed to view your results page by page on the screen, or to save them to a file. To page through the results on screen, pipe the command through “more” as shown:

command | more

To save the results to a file, redirect your output to a file of your choice using the greater than symbol:

command > outputfile

To open a logfile, use gunzip –c (the –c will only gunzip it to the screen instead of uncompressing and saving your file) if your file is ends in a .gz, which indicates it is compressed. Use the “cat” command if the logfile is not compressed. To take a peek at one of your logfiles, you would do the following:

gunzip –c file.gz | more or cat file | more

The remainder of our examples assume we are examining a compressed logfile.

To pull out all records from one IP address (1.2.3.4, for example):

gunzip –c file.gz | grep “1.2.3.4” > outputfile

To pull out all records from any IP address that begins with 12:

gunzip –c file.gz | grep “^12.” > outputfile

Notes:

• A caret (^) is the how you specify the beginning of a line with a regular expression
• The backslash () tells the regular expression you are looking for an actual period instead of a wildcard

To look at the requests made by one IP address (1.2.3.4, for example):

gunzip –c file.gz | grep “^1.2.3.4 “ | cut –d’ ‘ –f7 > outputfile

Pull out “page” requests only (status code = 200, and not an image, css, or javascript file):

gunzip –c file.gz | grep “ 200 “ | grep –v “.jpg “ | grep –v “.gif “ | grep –v “.png “ | grep –v “.css “ | grep –v “.ico “ | grep –v “.js “ > outputfile

Notes:

• You can exclude any other file extensions you wish by piping another grep –v into your command; ending the grep string ends with a space ensures you will only eliminate lines where those extensions are the request, and not embedded in a query string value.
• If you do a lot of logfile parsing, you may wish to put all the grep –v commands into a script so you don’t have to type all the commands every time you want to limit your output to pages.

Make a list of the most popular referrer fields for the /index.html page:

gunzip –c file.gz | grep “GET /index.html” | cut –d’ ‘ –f11 | sort | uniq –c | sort –nr > outputfile

Notes:

• The output will be a sorted list of lines with a number and a URL; the number is how many times the referrer occurred, and the URL is the referrer
• The uniq command must be executed on sorted input, which is why we sort the output first
• The second sort command lists the output by most to least popular referrer; -n is numeric and –r is reverse order

Pull out all the records from userid “angie”, and sort them by timestamp:

gunzip –c file.gz | grep “ angie “ | sort –t’ ‘ +3 > outputfile

Note:

The sort command is modified as follows: -t’ ‘ says the input is space-delimited, while the +3 says to sort on the fourth column (defaults to first column, but we need to move it over three columns)

Find all requests that are more than 1000 characters long:

Very long requests are often a sign that something is wrong: they can indicate a problem with your website’s code or they can be indicative of someone trying to hack into your website (especially if the requests contain any SQL code words).

gunzip –c file.gz | awk ‘length > 1000’ > outputfile

Stay tuned for more posts with additional commands.

1. If the data-pulling part of the analysis can be done in the tool, ask for instructions.

One thing you should know up front: If the answer to your business problem is something that can be found in the tool, you will save a lot of money by doing it yourself. However, not everyone has the time or the expertise to pull and/or interpret the numbers themselves. If you know (or suspect) that this is the case, ask your consultant to include step by step instructions on how to repeat the analysis on your own at a later date, and to document why she reached the conclusions she did. This may increase your estimate by a few hours to cover documentation costs, but you will save in the long run by being able to repeat the analysis as desired at a later date, to establish trends or to see how making changes to your site affects the outcomes. You will also have gained some valuable insight into how an “expert” approaches your particular business problem.

2. Tell your consultant what business problem you’re trying to solve.

Make sure you clearly communicate the specific business problem you are trying to solve. If you know of one methodology that will get the answer you are looking for, don’t be afraid to share. But it is usually best not to over-specify the methodology to be used, even if you think you know exactly what you want. Your analysis tool has a maze of tables and relationships and data under the covers, and there is often more than one way to tease out the same business insight. Allowing the consultant the flexibility to suggest other methodologies could result in significant cost savings, or in added insight for the same amount of hours. For example, asking your consultants for “all paths” that lead to purchase should set off some alarm bells with your consultant. A request for a list of “all” anything – unless the data is intended to be moved into another database for further analysis – is usually a sign that the business question hasn’t been well-formed. A good consultant will try to guide you to the real reason behind your question instead of providing you with a data dump: Were you trying to find out whether users were more likely to search or to navigate before making a purchase? Or were you trying to figure out whether the new tools you added in your product view area led to a higher likelihood to purchase?

3. Beware of “canned” analysis.

I am not talking about your consultant re-using queries and methodologies from prior engagements – that’s smart business, and reduces the time it takes to get answers and the likelihood of errors – but about “analysis” menus from which you can order as though you are in a restaurant. These are seldom tailored to your site, and are based on a one-size-fits-all methodology. These analyses are not all bad – they can serve as a training tool for your own in-house analysts if they include instructions on how to replicate the analysis yourself – but you should expect to pay accordingly.

4. Meet with the consultant before you ask for a quote.

When you first approach your vendor for services and speak with the consultant, insist on a web conference where you can call up your actual website and point to specific areas of concern. Taking time to do this up front helps to avoid misunderstandings, helps your consultant understand any company-specific terminology you use when referring to activity on your site, and should result in a more accurate estimate and more satisfactory analysis.

5. Clearly define — in writing — what the analysis engagement is about.

Finally, don’t be afraid to ask for a “solutions design” document to specify exactly what you’re paying for. Don’t expect the design document to include all of the details: it is hard to run a business when a consultant executes most of the engagement up front, only to have the client decide not to spend the money after all. However, the document should include enough detail about the methodology for the consultant to know how much work is involved, and to minimize any disconnects between what you asked for and what she delivers.

When you contract with your vendor for professional services, you are paying for special access to your site’s dataset, a breadth of knowledge that allows the consultant to apply lessons learned on a variety of sites, and for the business acumen to understand something about your particular business and what makes your site different from everyone else’s. Make the best of the opportunity!

Many larger companies have self-service websites where employees research their benefit offerings, reconcile their expense accounts, and book their own travel. And even smaller companies are embracing the intranet: a place where employees can find company policies, view documentation, open trouble tickets, and/or find employee contact details. These sites have one thing in common that requires a different approach from your external websites: You are paying the salaries of your site’s visitors, and their time spent equals your company’s money.

It’s often hard to justify advanced analytics or A/B testing on an intranet. If the site is instrumented at all, it’s often the case that someone “slapped some tags” on the site, and examination of the data ends at whatever standard reports are generated. However, if the company is large enough, then a good financial case can often be made for performing the same types of optimization efforts you practice on your external site. The larger the company, and the more expensive the workforce, the more compelling the case becomes.

Let’s consider a company with 5,000 employees — “unique visitors” in your web analytics tool — who use your intranet. Assume that on average, these employees visit twice per day, with an average visit duration of 1 minute. On a monthly basis, intranet usage represents over 3300 labor hours – not an insignificant amount. To put it in perspective, that’s roughly the equivalent of 20 full-time employees doing nothing but surfing the intranet!

Hours per Month Calculation

The dollar value put on those labor hours varies depending on company size, workforce skill and experience level, overhead costs, etc. If you don’t know the “wrap rate” (fully burdened rate that includes overhead, administration, etc. – usually 2-3 times an average hourly pay) used to estimate projects at your company, then $100/hour is good rule of thumb. It’s not unusual for very large companies to run higher than that, or for small/medium companies to run much lower, but it’s a nice round number that makes calculations easy.

Now let’s set a goal to shave 10% off the total visit hours each month by improving search or navigation on the site. For our example above, that’s a savings of 330 hours, or $33,000 per month. Potential savings of nearly $400,000 per year helps makes a compelling ROI argument indeed!

Note: For this analysis, it is especially important to know whether your analytics tool eliminates “bounced” (single-page) visits from your average visit duration calculation. That’s because the intranet is set as the home page for the default browser install within many companies, leading to a very high percentage of visits where only one page is viewed. I recommend you eliminate these visits from your calculations, since they don’t represent an optimization opportunity since people are just passing through on their way to another site.

Photo courtesy stock.xchng

Thomas Bosilevac Says:
December 28th, 2008 at 12:02 am

Great article! I manage the analytics service for the Intranet sites of a large multinational myself. Yes, just Intranet sites. You are right, in some cases this may only warrant the need to “slap a tag on it” and it does not get as fun as multi-variant testing, heat maps and campaign analysis. For organizations who spend millions (probably at least the fortune 1000) on their Intranet strategy, however, I can easily argue that a bit more care MUST be made, especially considering most organizations cannot use free tools such as Google Analytics. So let’s keep preaching for better Intranet stats!
Your article does a great job describing the start of Intranet valuation. How many people and how long do they spend. I place great caution though on the example to decrease this holistically. In a time where “shaving 10%” never meant more, this is a dangerous single metric to make into a KPI. The Intranet, if used wisely, can be an extremely valuable and cost-saving asset to the company. Instead of shaving off, perhaps increasing usage of such apps as client communication templates, call Center Deflection, etc is in the best interest to the bottom line.
While search may be an area where “time to find result” warrants a decrease, we have found many areas that we are now promoting due to the cost savings vs. “traditional” models involving too much people, paper and process.
Sorry for the rant, I was very excited to see a blog category specifially for Intranet analytics I had to comment away!

angiebb Says:
December 28th, 2008 at 3:34 pm

Thomas, thanks for you insights. It’s great that your organization does analytics on your intranet sites. That’s one area where many companies fall down, and terribly.

I completely agree with you that there are times when increasing time spent is actually better, and your examples are good. When I wrote the post, I was imagining a company that does little to no analytics on their intranet, and TOS is a quick and dirty way to get someone’s attention: enough to know that this stuff definitely IS worth analyzing, and it might not take much in terms of improvement to completely recoup whatever investment is made in analytics capabilities (people and/or tools).

I appreciate your comments!

Trackbacks

1. Times are rough – what are your troops doing? « Hear – Listen – Communicate