Commands used in this post:

xargs - allows you to use a list generated by one command as arguments for another command

mv - moves or renames a file

ls - lists files

tar - an archive utility that lets you store and extract multiple files; although it can be used to write files to tape, one very useful property is that it can also write output to a regular file, which is extremely handy if you need to transfer a bunch of files to your PC (WinZip can handle tarfiles)

Let’s say you have a lot of files in your directory named test1.txt, test2.txt, test3.txt, and so on, and you want to append .old to all of them. You could run a mv (move) command on each individual file, or you could rename all of them at once using xargs as follows:

ls -1 test* | xargs ABC mv ABC ABC.old

• ls -1 (that’s dash-”one”, not a lowercase “L”) lists the files so that there is only one file per line

• Each file name is temporarily held in a variable called ABC, and the rest is just substitution (mv test1.txt test1.txt.old, etc.).

I ran into a situation last week with a batch job that spits out thousands of reports, each into its own subdirectory. After all the reports had run, an issue was found that affected several hundred of them, which were then rerun. Because the reports had already been archived and moved to their final location (not difficult, but time-consuming due to sheer volume), we only wanted to re-archive the reports that had been rerun that day. This was relatively easy to do thanks to xargs. Here’s the command I used:

ls -l */*.out | grep “Jan 21″ | sed ‘s/^.*Jan 21 ..:.. //’ | xargs tar cvf reportreruns.tar

• ls -l (this time it’s a lowercase “L”, not a “one”) does a long listing, which includes permissions, size, owner, date stamp, etc. The first few lines for the listing look something like this:

-rw-r–r– 1 owner group 1234567 Jan 18 22:07 subdir1/report.out
-rw-r–r– 1 owner group 1234567 Jan 21 05:46 subdir2/report.out
-rw-r–r– 1 owner group 1234567 Jan 19 09:31 subdir3/report.out
-rw-r–r– 1 owner group 1234567 Jan 21 05:57 subdir4/report.out

• The grep command (see prior post for more information) pulls out only those lines that include the string “Jan 21″.

• The sed command (see prior post for more information) strips off everything up to the subdirectory and file. Quite literally, it’s substituting everything from the beginning of the line (^) to “Jan 21″, a space, two characters (. is a wildcard for a single character), a colon, two more characters (that just took care of the time stamp), and another space – and replaces it with nothing.

• The xarg command takes all of those files and passes them to the tar command. Notice I didn’t use the variable (ABC) this time: xargs will automatically put the arguments at the end of the following command, so you only need to use a variable if the arguments appear elsewhere in the command.

• The cvf modifier to tar tells it to create (”c”) the archive, see verbose (”v”) messages (it’ll output a message as each file is added: it’s optional), and write the archive to a file (”f”) named reportreruns.tar. The tar command needs to be followed by a list of files to put into the archive, but xargs has already taken care of that.

Here is an introduction to some of my favorite commands:

grep – used to find lines that contain a certain string or regular expression (note that regular expressions are not fully supported in the default grep command for some Unix systems)

cut – used to pull out specific columns from a file based on a specified delimiter; most logs are space-delimited

awk – a programming language that can parse through text files; short pieces of code can be used on the command line

sort – sort the output; the –n modifier sorts numerically; can use a –t modifier to sort on something other than the beginning of the line

uniq – eliminate duplicate lines; the –c modifier shows a count of how many times each line appears

In order to make “cut” work, you need to know which fields contain your data of interest. If you use the “combined” log format, the following table lists the fields where data is located. Cutting out cookie data can be a bit more difficult: we’re using cut with a space delimiter, but spaces can be contained in the user agent field so pulling out cookie values takes a little more work.

In Unix environments, you are allowed to view your results page by page on the screen, or to save them to a file. To page through the results on screen, pipe the command through “more” as shown:

command | more

To save the results to a file, redirect your output to a file of your choice using the greater than symbol:

command > outputfile

To open a logfile, use gunzip –c (the –c will only gunzip it to the screen instead of uncompressing and saving your file) if your file is ends in a .gz, which indicates it is compressed. Use the “cat” command if the logfile is not compressed. To take a peek at one of your logfiles, you would do the following:

gunzip –c file.gz | more or cat file | more

The remainder of our examples assume we are examining a compressed logfile.

To pull out all records from one IP address (1.2.3.4, for example):

gunzip –c file.gz | grep “1.2.3.4” > outputfile

To pull out all records from any IP address that begins with 12:

gunzip –c file.gz | grep “^12.” > outputfile

Notes:

• A caret (^) is the how you specify the beginning of a line with a regular expression
• The backslash () tells the regular expression you are looking for an actual period instead of a wildcard

To look at the requests made by one IP address (1.2.3.4, for example):

gunzip –c file.gz | grep “^1.2.3.4 “ | cut –d’ ‘ –f7 > outputfile

Pull out “page” requests only (status code = 200, and not an image, css, or javascript file):

gunzip –c file.gz | grep “ 200 “ | grep –v “.jpg “ | grep –v “.gif “ | grep –v “.png “ | grep –v “.css “ | grep –v “.ico “ | grep –v “.js “ > outputfile

Notes:

• You can exclude any other file extensions you wish by piping another grep –v into your command; ending the grep string ends with a space ensures you will only eliminate lines where those extensions are the request, and not embedded in a query string value.
• If you do a lot of logfile parsing, you may wish to put all the grep –v commands into a script so you don’t have to type all the commands every time you want to limit your output to pages.

Make a list of the most popular referrer fields for the /index.html page:

gunzip –c file.gz | grep “GET /index.html” | cut –d’ ‘ –f11 | sort | uniq –c | sort –nr > outputfile

Notes:

• The output will be a sorted list of lines with a number and a URL; the number is how many times the referrer occurred, and the URL is the referrer
• The uniq command must be executed on sorted input, which is why we sort the output first
• The second sort command lists the output by most to least popular referrer; -n is numeric and –r is reverse order

Pull out all the records from userid “angie”, and sort them by timestamp:

gunzip –c file.gz | grep “ angie “ | sort –t’ ‘ +3 > outputfile

Note:

The sort command is modified as follows: -t’ ‘ says the input is space-delimited, while the +3 says to sort on the fourth column (defaults to first column, but we need to move it over three columns)

Find all requests that are more than 1000 characters long:

Very long requests are often a sign that something is wrong: they can indicate a problem with your website’s code or they can be indicative of someone trying to hack into your website (especially if the requests contain any SQL code words).

gunzip –c file.gz | awk ‘length > 1000’ > outputfile

Stay tuned for more posts with additional commands.