Estimating the effects of cookie-deletion

There are differing opinions on how to label the metric historically known as “Unique Visitors”. On one side of the fence are those who think it should be relabeled “Unique Cookies”, since that is the most popular method used for calculations. On the other side of the fence are others who think the metric is a catch-all for the “best available” measurement (authenticated visitors, cookies if those aren’t available, IP/UA combination if neither is available) and should be replaced with a different term if/when a better, standardized way to measure people comes along. What we all agree on, though, is that a Unique Visitor metric measured with cookies is terribly inaccurate.

How bad is it? As usual, it depends. A site where people tend to visit on a daily basis will see more inflation from cookie-deletion than a site that is only visited once per month. In the former example, one person may count toward the monthly total as many as 30 or so times, while in the latter, even a frequent cookie-deleter would only count once.

Let’s pretend that we know something about the actual people visiting a site, and see if we can determine by how much our web analytics numbers might be affected by cookie-deletion.

In order to make the calculations easy, I have assumed people only visit or delete their cookies on daily, weekly, or monthly boundaries, and I am only considering a one month time frame. However, the same logic could be applied to more granular data. It ultimately boils down to a matrix algebra problem, but I doubt many of us are eager to get into that level of detail.

An example

Consider 10,000 people: not 10,000 “cookies” and not 10,000 “unique visitors,” but 10,000 real-life, carbon-based beings. Suppose we are able to observe these people in such a way that we know “the truth” about their online behavior. Suppose also we have observed that, on average, 10% of our people delete their cookies every day, 15% delete once per week, and the remainder delete their cookies monthly or less frequently.

We have also observed that 20% of these people visit our website every day, 30% visit once per week, and the remainder only visit once in a given month. There’s no reason for these people to login to our website — all visits are anonymous — and we count Unique Visitors using a cookie.

How bad is it?

Our first step is to find out how many different cookies each person will receive over the course of the month, based on the number of times they visit our site and how often they delete their cookies. For simplicity’s sake, we’ll assume each month has 30 days and 4 weeks.

Daily deleters will receive a different cookie each time they visit, so daily visitors will log 30 cookies and weekly visitors log 4. Weekly deleters who visit every day will log 4 different cookies over the course of the month, as will weekly deleters who visit once per week. Everybody else’s activity will be logged with one cookie. We can summarize as shown below.

Using the above factors, we can determine the cookie-contribution from each set of people:

Wow. Our 10,000 people are being represented as 18,950 unique visitors: the Unique Visitors number is inflated by 90%!

Visitor loyalty reports are affected, too

Unique Visitors isn’t the only number that’s affected by cookie-deletion. Any visitor-based number is going to be off, so you have a lot of trouble understanding visitor loyalty. You can tell when your efforts to improve loyalty are working, since the numbers will move in the right direction, but the magnitude of change will be misleading.

For the above example, we know that 20% of our people visited daily, 30% visited weekly, and 50% visited monthly, so the number of visits in a month works out to 77,000 (2000 x 30 + 3000 x 4 + 5000). Visits aren’t affected by cookie-deletion to any great extent (a good argument for visit-based analysis!) – so our tool will also report 77,000 visits.

This means our people averaged 7.70 visits (77000/10000) over the course of the month, but our web analytics tool will only report 4.06 (77000/18950) visits per visitor because the visitors are inflated. Our average visits per visitor are under-reported by 47%!

If you prefer to view loyalty using a histogram (# visitors who visited once, twice, three times, etc.), then we need to determine to which bin each visitor’s cookies will be credited.

The first thing that jumps out of this table is that the majority of the cookies are only encountered once, regardless of how many times someone actually visited. This explains why visitor loyalty graphs, regardless of the tool used, are often overloaded with so many one-time visitors.

visitor duration graphs

Applying the frequencies in the histogram table to the numbers in the calculations table show us how our visitor retention graph is affected by cookie-deletion.

visitor frequency corrections

Again, wow! While 20% of our people visited the site every day, with cookie-based visitor counting, only 8% appear in this super-loyal segment. The majority of the “visitors” that were added to the site via cookie-deletion appear in the 1 visit bin, inflating that number by almost a factor of 3.

Adding Authentication

If 100% of the people to the above site authenticated, and the authenticated visitor identifier were used to count unique visitors, then the number would be pretty accurate (ignoring shared logins, etc.). But most sites don’t require authentication to see certain pages, so the likelihood of 100% authentication is low except in special cases, like intranets.

For our example above, what if half the visitors authenticated? Half of the 10,000 people would be more-or-less accurately represented, while our cookie-deletion calculations would apply to the remaining 5,000. The unique visitor multiplier factor decreases with increasing percentage of authenticated people.

Assumed: 10/15/75 cookie deletion and 20/30/50 visiting frequency (daily/weekly/monthly).

Is it always that bad?

Not necessarily, it could be worse or it could be better. The above examples assumed that 20% of people visited the website every day, and 30% visited weekly. This was an arbitrary example meant to make calculations easier. In real life, you may have far fewer daily visitors (or more, it just depends on the site). Running the numbers assuming 5% daily and 50% weekly visitors, for example, results in a unique visitor inflation of 1.5 instead of the 1.9 calculated in our example.

I’ve attached my spreadsheet so you can run your own what-ifs.