Unique Visitors suck. That’s why we shouldn’t change the definition.

Last week saw a lot of back-and-forth about the IAB audience reach definitions vs. the WAA definition of Unique Visitors. Jodi McDermott tried to explain the difference between the WAA’s UV definition and a completely different definition of the term put out there by the IAB. Her post was met by some rather harsh criticism by Eric Peterson, who was then sternly lectured by April Wilson (who is a WAA Board director) and then apologized to Jodi down in the comments of his post. I weighed in on the subject in the Yahoo Group, then Eric replied to me.

In a nutshell: Eric thinks the WAA definition sucks, the WAA should adopt the IAB definitions, all of the web analytics tools should immediately change their Unique Visitors metric to Unique Cookies, and the WAA Standards Committee is being asinine (his word, not mine) because they don’t change direction and adopt the IAB metrics.

The WAA Standards Committee did not seek out a urinary Olympiad with either the IAB or with Eric Peterson, but there are a lot of points out there regarding the IAB’s Audience Reach Guidelines and the WAA Standards that are not being addressed through Eric’s bully pulpit. As long-time co-chair of the WAA Standards Committee — first with Jason Burby, and more recently with Judith Pascual, both from ZAAZ — I would like to offer some more insights. Some of this is my own perspective as a practitioner, but most of it has been discussed ad nauseam in our committee meetings over the past three years and represents the committee’s position on the subject.

Redefining a metric that has been in use for a decade, and is unlikely to change in a lot of tools, will only increase confusion.

Unique visitors suck, and a better metric should be available. But as long as we are using the same terminology as the sucky metric we’ve always used, we’re not fixing anything. This is really my major beef with the IAB specification, as I applaud their efforts to invent a metric that does a better job measuring people.

Unique visitors has been a technology-based metric for over a decade. It seems unlikely that vendors are going to change either their technology or its terminology en masse. This means that as an analyst, you will still be stuck trying to figure out what it means when your vendor reports unique visitors, except you’ll have a lot more variations thrown into the mix.

A different name for an improved metric (unique people? unique individuals? reach?) would provide little doubt that the tool in question is reporting something different, since the new metric wouldn’t even exist in non-compliant tools.

Data from two different IAB-compliant tools still won’t be comparable.

Why? Because the IAB spec doesn’t tell you how to correct the numbers, just that you’re supposed to do it. A “scientific” method must be used, and it must be based on information obtained directly from people, and beyond that you are relying on the mathematical and/or statistical acumen of the people reporting the numbers. The way the spec is written, they can correct the data however they want as long as they can make an argument that it’s people-based. And you are likely to be comparing panel-based corrections with cookie-deletion-algorithm-based corrections with registration-based corrections as you go from site to site.

Should you correct detail data? Or aggregate data?

This is not made clear in the IAB’s spec, either, so it’s yet another way that numbers reported by two different tools or companies could be different.

Web analytics data is collected on a record-by-record basis. Someone makes a request on a website that is instrumented for analytics, and a record of that request, along with any other information your tool collects (cookies, user ID, referrer, user agent, custom data, etc.), is made. This record is put into one or more tables in the database that makes up the guts of your web analytics tool. The majority of what happens when you “do” web analytics is based on queries of that database.

Should correction for actual people take place at the detail level, on those records in the database? Or is there a fudge factor that is applied to the monthly (weekly, quarterly, whatever) data you report from the tool?

And would you use the same terminology for a metric corrected at the detail level as for a metric corrected in aggregate?

Privacy is important to our customers, so it is important to us.

Privacy is the 800 pound gorilla sitting right in the middle of this discussion, and if we’re not careful he’s going to start flinging poo. In creating a standard metric, we cannot force sites to identify the people that visit their site, if those people do not want to be identified.

We have to balance our need for accurate metrics with the privacy considerations of our visitors and ease of use of our websites. Many Internet users simply do not want to be identified, either for privacy reasons or for convenience, and for many sites there really isn’t any benefit to the user of logging in.

There are even legal issues for sites used by children under 13 (Children’s Online Privacy Protection Act, or COPPA) and for sites that deal with medical information, like insurance or medical advice sites (Health Insurance Portability and Accountability Act, or HIPPA). These laws and others like them don’t prohibit login — indeed, sometimes it’s necessary — but they do throw another business consideration into the mix, and often it’s easier for the business to go out of their way to ensure they *can’t* identify individuals, so they don’t have to deal with any potential Personally Identifiable Information (PII).

I have read suggestions about using unique device identifiers instead of cookies to measure Unique Visitors so that no PII would be required. In fact this is what is used for some types of mobile analytics, as mobile devices have a unique identifier. From a privacy standpoint, this could spell disaster for our industry. From a user standpoint, a unique device identifier is the equivalent of a 3rd party cookie that is tied to you and only you: one that you can never delete. Your online behavior can be tracked from site to site to site. Claims about the “anonymity” of a unique device identifier are going to be irrelevant to privacy advocates, since a user’s search data can be and has been used to tie “anonymous” information back to individuals.

The audience for the IAB’s standard is unclear, and attempts to clarify it have failed.

Eric’s posts about the IAB spec have put a spotlight on the confusion over just who the IAB specification was written for, and who would be expected to comply. Unfortunately, the IAB (Joe Laszlo and his colleague Sherril Mane) gave me a different answer than the answer Eric wrote about in his post: I was told that the definitions most likely would be applicable to web analytics vendors.

So who were IAB requirements written for? Your guess is as good as mine.

On the same phone call, Joe and Sherril also told Jodi and me that the IAB didn’t intend to change anything about their standard, regardless of any input they might receive. The document was “thoroughly vetted” by their stakeholders, and that was the final say. Which brings me to my next point…

If you are reading this post, odds are the IAB doesn’t represent you.

Does that make them bad? No. Does that make them irrelevant? No. But they represent a different industry and a different set of stakeholders than the WAA, and your interests as a web analyst are not even a tiny part of their consideration.

Companies pay anywhere from $5,000 (for non-voting associate membership) to upwards of $300,000 per year to be members of the IAB. These are the interests they represent. You may or may not work for someone who “has a say,” and there are no individual members.

The WAA, on the other hand, is comprised of practitioners, consultants, and vendors in the Web Analytics field. This is who we represent. A recent survey by the WAA Research Committee (login required) reported that over half of the members responding have individual, not corporate membership. Many of us individual members, myself included, pay the ~$200 WAA membership fee out of our own pockets.

Why does this matter? Because we on the Standards Committee work in the web analytics field, we strive to consider how the work we do affects real practitioners analyzing different types of websites. Over the years, we have had significant participation by practitioners working “in the trenches” at companies as diverse as Disney, Saks Fifth Avenue, IBM, Reed Elsevier, FedEx, The Motley Fool, ANSI, Adobe, CNN, Ford, Intuit, AOL, Yahoo!, and more. Many of these people are individual members and many of them have been with us for the better part of our tenure as a committee, even as they moved from one company to another and were able to bring in experience from multiple perspectives.

We don’t write standards only for sites that buy or sell advertising. A definition that applies to all sites must actually work for all site types. This includes e-commerce sites, portal sites, subscription-based sites, pure content sites, lead-generation sites, and self-service sites in addition to ad-based sites. And standards that we write must benefit the analyst more than they confuse him or her.

Because we don’t currently have a good way of measuring people that is applicable to all sites, or to even a majority of sites, we have not defined one. Instead, we have attempted to capture the most common practices of today.

Unique Visitors (per the WAA definition) and Unique Cookies are not the same thing.

Eric has suggested more than once that the easy solution to this issue is for web analytics tools to relabel their Unique Visitors numbers as Unique Cookies . I quote: “Web analytics practitioners (and theoretically the vendors) will learn to use ‘Unique Cookies’ since that is a technically correct and 100% accurate description of the data.”

But it isn’t accurate. Using cookies is indeed the most prominent method of identifying Unique Visitors, but it isn’t the only one. Calling that a 100% accurate description of the data is a huge error. Analysts who care about more accurate unique visitor counts are very likely not using cookies to count unique visitors/users. They are probably using registration data from purchases, or user IDs from logins. This is allowed, and preferred, by the WAA standard.

Even if you are using cookies to count your Unique Visitors, they are probably not being used for 100% of the count. Some of your visitors block the JavaScript that sets the cookie. Your WA tool more likely than not has a hierarchy of calculations they use to estimate the “anonymous” (or “guessed”) unique visitors.

If your tool analyzes logfiles, and your site does not drop tracking cookies itself, then the Unique Visitors metric in your analytics tool counts the number of different IP address/User Agent (browser + operating system) combinations it finds. This isn’t even close to counting Unique Cookies.

Changing the name of a bad metric in the tools, to a metric that doesn’t describe what the tool does, doesn’t fix a thing.

What is a “standard,” anyway?

Eric’s reply to me on the Yahoo Group defines what is probably the crux of the issue:

“Perhaps we disagree on what a ’standard’ is. I think a standard should describe the way things ** should be ** not the way things (unfortunately) are.”

Yes, we definitely disagree on that point, and perhaps that is what makes this discussion so difficult. I would call the way things should be a requirement, not a standard. It’s no doubt true that standard can be interpreted either way.

The dozens of people – practitioners, consultants, and vendors — who have been meeting every other week for the last three plus years to help establish the WAA standards have chosen to use the standards to describe reality as we know it, so that’s the way I think of a standard. We think that, at this point, it is more important for analysts to understand what they are measuring than to change what they are supposed to measure, and that has been the focus of our committe’s efforts to date. This doesn’t mean we will never define the future as we think it should be, only that the industry is young, and we believe this is the appropriate first step.

And I really couldn’t let this one slide.

Finally, in one of his posts Eric says “…considering the fact [emphasis mine] that the WAA’s definition is wrong…”

I know I should probably ignore that statement. The issues here are a matter of interpretation: the WAA is not wrong, and Eric is not wrong (as long as he stays away from black-and-white, I’m-right- and-you-are-wrong arguments), and — despite my and the committee’s disagreement — the IAB is not necessarily wrong. We represent disparate opinions about what terms and definitions will most benefit those in our respective industries. But I can’t help but point out:

The WAA definition of Unique Visitors describes how nearly every single analytics tool that exists calculates Unique Visitors. The WAA definition of Unique Visitors is also essentially in agreement with the JICWEBS definition of Unique Users. (For the US-centric among us, JICWEBS is the Joint Industry Committee for Web Standards, a standard-setting body from the UK and Ireland. It includes ABCe and the IAB UK.)

On the other hand, the IAB definition takes an admittedly flawed metric and completely redefines it. It represents the way almost no one calculates unique visitors or unique users.

And yet, it’s a fact that the WAA is wrong? Oh really?

Photo credits:

O RLY Owl: http://www.hjo3.net/orly/gallery1.htm

Gorilla: User noladoc30 on stock.xchng (http://www.sxc.hu/photo/1112963)