apache - how to monitor HTTP response codes

I am looking for a way to monitor the count of each HTTP response
codes from apache reverse proxy servers. As a quick proof of concept
I wrote the ksh script below to scan the HTTP response codes and show
how many of each type occurred every hour. This code is very
inefficient and would be much better if it were written in perl or
something else that doesn't have to re-read the file so many
times...

Ultimately I would like the ability to poll the the count of each HTTP
response code without having to parse access logs period. I checked
into using /server-status, but this doesn't contain that type info...
Is there something easy I am overlooking that wouldn't require parsing
log files to get the information on a 10 minute basis?

-Inet




#!/usr/bin/ksh

FILE="$1"

echo " 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20
21 22 23"
for CODE in 200 301 302 304 400 403 404 500; do
print -n "$CODE "
for HOUR in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17
18 19 20 21 22 23; do
COUNT=`cat $FILE | grep "2008:$HOUR" | awk '{print $9}' | grep
$CODE | wc -l`
print -n "$(( COUNT )) "
done
echo
done

On Tue, 6 May 2008 08:49:23 -0700 (PDT), inetquestion
<inetquestion@hotmail.com> wrote:

>I am looking for a way to monitor the count of each HTTP response
>codes from apache reverse proxy servers. As a quick proof of concept
>I wrote the ksh script below to scan the HTTP response codes and show
>how many of each type occurred every hour. This code is very
>inefficient and would be much better if it were written in perl or
>something else that doesn't have to re-read the file so many
>times...
>
>Ultimately I would like the ability to poll the the count of each HTTP
>response code without having to parse access logs period. I checked
>into using /server-status, but this doesn't contain that type info...
>Is there something easy I am overlooking that wouldn't require parsing
>log files to get the information on a 10 minute basis?
>
>-Inet

I'm sure there is nothing you can do with Apache
configuration to solve this.

>#!/usr/bin/ksh
>
>FILE="$1"
>
>echo " 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20
>21 22 23"
>for CODE in 200 301 302 304 400 403 404 500; do
> print -n "$CODE "
> for HOUR in 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17
>18 19 20 21 22 23; do
> COUNT=`cat $FILE | grep "2008:$HOUR" | awk '{print $9}' | grep
>$CODE | wc -l`
> print -n "$(( COUNT )) "
> done
> echo
>done

Try something with:
awk 'print --select key fields here--' $1 \
| sort \
| uniq --count \
| awk '--some script to transpose results--'

Another possibility is to prepend every page with a
small php script that registers the hit in a database.

By the way,
cat $FILE | grep "2008:$HOUR"
is useless, you can directly grep the file:
grep "2008:$HOUR" $FILE

HTH
--
( Kees
)
c[_] If televison's a babysitter, the Internet
is a drunk librarian who won't shut up. (#301)

Following perl script gives ratio of 404s to all other traffic.




#!/usr/bin/perl

if (my $filename = shift @ARGV) {
open(DATA, "< $filename") || die ("Could not open file: $filename");
*STDIN = *DATA;
}

while (<STDIN>) {
chomp ($_);
if (/.* 404 [0-9]*$/) {
$http404+=1;
}
$entriesTotal+=1;
}

$ratio=$http404*100/$entriesTotal;
printf "%2.6f\n",$ratio;

On Wed, 7 May 2008 10:05:20 -0700 (PDT), inetquestion
<inetquestion@hotmail.com> wrote:

>Following perl script gives ratio of 404s to all other traffic.
>
>#!/usr/bin/perl
>
>if (my $filename = shift @ARGV) {
> open(DATA, "< $filename") || die ("Could not open file: $filename");
> *STDIN = *DATA;
>}
>
>while (<STDIN>) {
> chomp ($_);
> if (/.* 404 [0-9]*$/) {
> $http404+=1;
> }
> $entriesTotal+=1;
>}
>
>$ratio=$http404*100/$entriesTotal;
>printf "%2.6f\n",$ratio;

Cute.
--
( Kees
)
c[_] Quantum Mechanics: The dreams stuff is made of.
(Steven Wright) (#187)

inetquestion wrote:
> I am looking for a way to monitor the count of each HTTP response
> codes from apache reverse proxy servers.

if you have a high volume of traffic or serve content from multiple machines,
the log-grepping approach is a pain.

for a quick summary of any http response codes in your live apache logs you can
run this (its one unix command line and assumes combined log format):

cat logfile1 logfile2 logfileN | perl -ane 'print qq#$3-$2-$1 $4\t$5\n# if
m#\[\s*?(\d+)/(.*?)/(\d+):(\d+):.*? HTTP/.*?"\s+(\d+)#;' | sort | uniq -c

to produce output like this:

HITS DATE HOUR CODE
291 2008-May-05 10 200
1 2008-May-05 10 301
6 2008-May-05 10 302
1 2008-May-05 10 401
13 2008-May-05 13 200
1 2008-May-05 13 301
1 2008-May-05 13 404
4 2008-May-05 14 200

if you really want to run this every 10 minutes on live logs, you won't need the
year, month, date, just each 10-minute period you're summarizing:

cat logfile1 logfile2 logfileN | perl -ane 'print qq#$1:${2}0\t$3\n# if
m#\[\s*?\d+/.*?/\d+:(\d+):(\d).*? HTTP/.*?"\s+(\d+)#;' | sort | uniq -c

resulting in:

HITS TIME CODE
21 09:20 200
2 09:20 302
31 09:30 200
3 09:30 302
20 10:00 200
1 10:00 301
1 10:00 401
240 10:10 200
3 10:10 302
31 10:20 200
3 10:20 302
7 12:00 200
6 12:10 200

> Ultimately I would like the ability to poll the the count of each HTTP
> response code without having to parse access logs period. I checked

for a longer-term solution, any of the log-based web analytics tools will
summarize http response codes for you, here are a few of them:

http://webalizer.org/ (hits by response code)
http://analog.cx/ (status code report)
http://awstats.sourceforge.net/ (http status codes)

analog in particular is fast and configurable, you can probably automate it to
run only the status code report as frequently as you need, and generate
machine-readable output that you might use in a monitoring system like nagios.

--sean

--
sean dreilinger - http://durak.org/sean/