I assume that most web analytics tools identify the IE version by looking for MSIE X.Y in the browser string. However, this is no longer valid for IE8. This is because the IE8 user agent string will include MSIE 7.0 when in compatibility mode. The difference between the “real” IE7, and IE8 in compatibility mode is the word Trident, which is included in both variants of IE8:

Example of a regular IE8 user agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Example of IE8 in compatibility mode: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; Media Center PC 5.0; .NET CLR 3.5.21022)

Literally thousands of web sites are not compatible with IE8. A list of more than 3,000 incompatible sites is maintained by Microsoft. This list can be downloaded by IE8 users so that the browser can automatically switch itself into compatibility view when a site is encountered that has previously been identified by IE8 users as incompatible. Many more sites are not compatible, but are not on the list because they have lower traffic levels.

Because a visitor can have multiple user agents in one visit, this raises a number of questions:

• Does your analytics tool keep the user agent string from each individual page view, or do they associate one browser with the entire visit?
• If browser is associated with the entire visit, which browser is recorded? If they keep the string on the entry page, then IE8 is likely represented correctly in your data, but you won’t know if users are resorting to compatibility mode in order to view your site. If your analytics tool keeps the last browser string encountered in the visit, then your numbers are likely biased toward IE7 unless your tool is properly grouping this traffic as IE8.
• If browser is associated with page views instead of the visit, then adding up visits in your browser report would give you more than the total visits for your site. In other words, browser visits would not be “summable” the way they are when one can assume that each visit has only one browser. This is not the end of the world, just something to be aware of because it’s not intuitive.
• Does your analytics tool properly group the browsers with both MSIE 7.0 and Trident as IE8? If not, do they expose the entire string so you can do the calculations yourself to see if your site has IE8 issues?
• If you are doing logfile analysis without cookies, sessionization is probably based on IP + User Agent. For sites where I’ve transitioned from logfiles to tags in the same tool, my experience has been that IP/User Agent sessionization tends to over-count visits: this issue will increase that inflation even more. Bear in mind that many tag-based tools resort to IP/UA when cookies are blocked, so there could be a small inflation effect regardless of the type of data-collection you use.

I examined a few of my sites and found the percentage of visits with IE8 to be roughly between 5% and 15%, depending on the site. My B2B sites tend to have lower IE8 penetration, while sites that attract high-tech users will tend to show a higher percentage of the latest browsers.

If your web analytics tool exposes the entire browser string (Google Analytics does not), I recommend you search through your user agent strings looking for Trident, and see for yourself if this is an issue for the sites you analyze. One metric I’m looking at is the percentage of my Trident browser visits that also contain MSIE 7, assuming that sites that are not compatible with IE8 will show a higher percentage of users resorting to compatibility mode. For a site with known IE8 issues I calculated 25% , while another site I randomly chose calculated to 12%. I haven’t examined enough sites yet to know if that means the second site also has IE8 issues, or if it just means it’s “normal” for a certain percentage of IE8 users to surf in compatibility mode. Clearly I have more work to do.

Update: Last night I received an email from a colleague who had read this post, asking why should they care? It’s a fair question so I thought I’d answer it publicly.

First, if you’re asking then you probably aren’t in a situation where you need to care. That’s OK: the lowly browser report isn’t the most important report in your web analytics tool, not by a long shot.

But I can think of a couple of situations where it’s important:

1. When deciding whether or not to fund development changes to enable compatibility with certain browsers, “fewer than 5% of our visits use that browser” is a lot different than “nearly 10% of our visits use that browser”.  The numbers you use for those decisions should be as accurate as practical.

2. Your customer service department may receive emails or phone calls from visitors complaining that they are unable to perform certain tasks on your site (like complete a transaction). When they receive multiple complaints that sound similar but are unable to reproduce the problem in house they may ask you, the analytics ninja, for help defining the scope of the problem. These intermittent issues are difficult to troubleshoot because they’re often environment-related. One starting point is to examine the user experience through that transaction — transaction page views per visit is sometimes sufficient, or you may want to look at a funnel chart for the process — and segment it by different browser versions. If the issue is due to a browser incompatibility, you can sometimes pinpoint it quickly with this type of analysis.

If Fark is to be believed, the Internet is all about porn anyway.

For the record, I don’t analyze porn sites for a living. While I admit I have done analysis for at least one adult-oriented site in the past, this is different. This is the effect of sexually-oriented search terms on websites that have little or nothing to do with sex, websites that I would happily show to my mother. But if you analyze a wide enough variety of sites, you will find that fetishes come in a surprising variety of shapes and sizes, and you’ll be surprised where they, um, pop up.

There are three ways that these “thrill-seekers” may affect your data.

1. By causing a one-time traffic spike. This is more likely to happen for a blog or a news site, when an article mentions something sexual in a fairly innocuous way. For example, this article contains plenty of keywords that may attract traffic that is not part of my target audience (and if you haven’t bounced by now, welcome to the world of web analytics!). This can happen on news or magazine sites that run features on a variety of subjects, and it can often catch the web analyst off guard. For example, consider the more-or-less legitimate — if somewhat sensational — news articles that were all the rage a couple months ago, talking about teens sending naked pictures of themselves to each other on cell phones. When you mention “teens” and “sex” and “naked pictures” in the same article, you’re bound to attract some of that kind of traffic.

This usually only becomes an issue when the traffic spike for a single article is large enough to influence aggregate numbers for the entire week or month. Any sudden spike (or dip) in traffic should always be investigated: it may have been due to a simple editorial choice instead of that awesome marketing campaign that your HiPPO designed.

2. By inflating search engine visits long-term. Perhaps “inflating” isn’t the best term, since the traffic is real, it’s human, and it’s coming from search engines. This situation happens when there are articles or images on your site that are intended for one audience but end up attracting another audience – the kind that’s not likely to become a customer – and it can wreak havoc with your conversion rates. A prime example is a site that publishes medical information intended for a professional medical audience. A thorough enough site will likely contain pictures of certain body parts or descriptions of rare medical procedures, and a glance through some of your top search terms can yield insights into the human psyche that you wish you didn’t know.

Always look past the “Top X” keyword report that is spit out of your web analytics package by default. Look for terms that seem over-represented on a site like yours. Pay careful attention to image searches, and ensure that you can separate image search keywords from text search keywords if necessary.

3. By logging visits that never really happened. This is fairly rare, and you will likely only catch it if a) your analytics are based on server logs instead of JavaScript tags, and b) your site contains one or more unprotected redirect URLs, “pages” that contain a URL as a value in the query string. The symptom is a sudden appearance in your keyword reports of sexually-oriented phrases that have absolutely nothing to do with your site. The cause is a search engine ranking hack, where a site-of-ill-repute manages to get themselves indexed by means of your redirect URLs, using your site’s good reputation to increase their rankings. You can confirm by looking at the entry pages for the offending terms to see if they are the redirect pages.

As with any traffic that is obviously unqualified, you very likely want to segment out the perverts from some of your conversion rate calculations, especially if you are doing optimization efforts on one or more areas of your site. Unqualified traffic volume can be more than enough to skew results and mask changes to real customer behavior. However, I don’t recommend you filter this traffic from your entire data set. If your linking, advertising, or SEO efforts are bringing in the wrong kind of traffic, this is something you really need to know.

Here is an introduction to some of my favorite commands:

grep – used to find lines that contain a certain string or regular expression (note that regular expressions are not fully supported in the default grep command for some Unix systems)

cut – used to pull out specific columns from a file based on a specified delimiter; most logs are space-delimited

awk – a programming language that can parse through text files; short pieces of code can be used on the command line

sort – sort the output; the –n modifier sorts numerically; can use a –t modifier to sort on something other than the beginning of the line

uniq – eliminate duplicate lines; the –c modifier shows a count of how many times each line appears

In order to make “cut” work, you need to know which fields contain your data of interest. If you use the “combined” log format, the following table lists the fields where data is located. Cutting out cookie data can be a bit more difficult: we’re using cut with a space delimiter, but spaces can be contained in the user agent field so pulling out cookie values takes a little more work.

In Unix environments, you are allowed to view your results page by page on the screen, or to save them to a file. To page through the results on screen, pipe the command through “more” as shown:

command | more

To save the results to a file, redirect your output to a file of your choice using the greater than symbol:

command > outputfile

To open a logfile, use gunzip –c (the –c will only gunzip it to the screen instead of uncompressing and saving your file) if your file is ends in a .gz, which indicates it is compressed. Use the “cat” command if the logfile is not compressed. To take a peek at one of your logfiles, you would do the following:

gunzip –c file.gz | more or cat file | more

The remainder of our examples assume we are examining a compressed logfile.

To pull out all records from one IP address (1.2.3.4, for example):

gunzip –c file.gz | grep “1.2.3.4” > outputfile

To pull out all records from any IP address that begins with 12:

gunzip –c file.gz | grep “^12.” > outputfile

Notes:

• A caret (^) is the how you specify the beginning of a line with a regular expression
• The backslash () tells the regular expression you are looking for an actual period instead of a wildcard

To look at the requests made by one IP address (1.2.3.4, for example):

gunzip –c file.gz | grep “^1.2.3.4 “ | cut –d’ ‘ –f7 > outputfile

Pull out “page” requests only (status code = 200, and not an image, css, or javascript file):

gunzip –c file.gz | grep “ 200 “ | grep –v “.jpg “ | grep –v “.gif “ | grep –v “.png “ | grep –v “.css “ | grep –v “.ico “ | grep –v “.js “ > outputfile

Notes:

• You can exclude any other file extensions you wish by piping another grep –v into your command; ending the grep string ends with a space ensures you will only eliminate lines where those extensions are the request, and not embedded in a query string value.
• If you do a lot of logfile parsing, you may wish to put all the grep –v commands into a script so you don’t have to type all the commands every time you want to limit your output to pages.

Make a list of the most popular referrer fields for the /index.html page:

gunzip –c file.gz | grep “GET /index.html” | cut –d’ ‘ –f11 | sort | uniq –c | sort –nr > outputfile

Notes:

• The output will be a sorted list of lines with a number and a URL; the number is how many times the referrer occurred, and the URL is the referrer
• The uniq command must be executed on sorted input, which is why we sort the output first
• The second sort command lists the output by most to least popular referrer; -n is numeric and –r is reverse order

Pull out all the records from userid “angie”, and sort them by timestamp:

gunzip –c file.gz | grep “ angie “ | sort –t’ ‘ +3 > outputfile

Note:

The sort command is modified as follows: -t’ ‘ says the input is space-delimited, while the +3 says to sort on the fourth column (defaults to first column, but we need to move it over three columns)

Find all requests that are more than 1000 characters long:

Very long requests are often a sign that something is wrong: they can indicate a problem with your website’s code or they can be indicative of someone trying to hack into your website (especially if the requests contain any SQL code words).

gunzip –c file.gz | awk ‘length > 1000’ > outputfile

Stay tuned for more posts with additional commands.