realtime log analyzer: difficult to write?

i've checked out http://www.softngine.com/ seems like nice product.
i've wondered if i should try to write my own so i'm not stuck with an
x domain license. anyone know if there's something similar in the
open-source community? i checked sourceforge but didn't find
anything. thanks for any ideas...

4MLA1FN

4MLA1FN <eff1fan@mail.yahoo.com> wrote in
news:e6da515ld5ocffvh951lkhf4hi55gkr8gn@4ax.com:

> i've checked out http://www.softngine.com/ seems like nice product.
> i've wondered if i should try to write my own so i'm not stuck with an
> x domain license. anyone know if there's something similar in the
> open-source community? i checked sourceforge but didn't find
> anything. thanks for any ideas...

http://freshmeat.net/search/?q=web+l...ction=projects


AWStats is very popular, especially if the target is nontechnical users,
like bosses or clients (because it's very pretty):

http://awstats.sourceforge.net/

Don't even think about using this (or any other log analysis program)
without requiring authentication to view the results. Take pains to make
sure it's not easily accessible from the Internet. Exploits routinely
appear and you're not likely to upgrade it constantly. Log analysis can
also leak sensitive information about a site.

Be aware that fully featured log analysis programs can put a heavy load
on a server (especially when resolving IP addresses). If you don't need
all of that information, it's easy to write your own script in [insert
language of choice] that returns the data you want.

Here's a bash one liner to check how often a specific file was requested
today, to get you started:

egrep "`date +%d/%b/%Y`.*favicon.ico" /var/log/apache/access_log | wc -l

Run this right before midnight to get today's hit count:

egrep "`date +%d/%b/%Y`" /var/log/apache/access_log | wc -l

This one will run a quick check for today's missing pages:

egrep "`date +%d/%b/%Y`.* 404 " /var/log/apache/access_log

Once you understand your site's log format, it's trivial to write a
perl/python/whatever script that ranks your top pages by number of
requests.

On Thu, 07 Apr 2005 14:15:25 GMT, Jorey Bump <devnull@joreybump.com>
wrote:

>AWStats is very popular...

many thanks for the reply. i'm currently using awstats. i didn't
realize it had a real-time component like the softngine product.
(with softngine's product, you put a snippet of javascript in the
files you want to track. the javascript just calls a logger on the
server. when you check the site stats, it shows up to the minute
access data. the one main omission is that it'll never show 404s.
(for that i guess i could just periodically grep the logfile.)

thanks for the freshmeat link. dunno why i didn't check their first.
i found "probe" which seems to work on a similar principle. i'll
check that out.

>Don't even think about using this (or any other log analysis program)
>without requiring authentication to view the results....

my current stats pages is behind some authentication; done solely to
prevent general users for seeing this info. i hadn't thought that it
could be a source of sensitive info, though no i see that is clearly
so. great advice.

thanks again.


4MLA1FN

[Jorey Bump wrote in comp.infosystems.www.servers.unix]
> AWStats is very popular,
[...]
> http://awstats.sourceforge.net/

But really, really make sure you have an up to date version
if you use it. Quite a nasty exploit was discussed at length
on the Bugtraq mailing list recently.

From the front page of the awstats site :

"
Warning, a security hole was recently found in old AWStats
versions (from 5.0 to 6.3) when AWStats is used as a CGI: A
remote user can execute arbitrary commands on your server
using permissions of your web server user (in most cases
user "nobody").
"


This isn't supposed to be a vote against awstats, as it's a
really good tool, more a warning to make sure yours is safe.



--
http://fotoserve.com/ - Prints, Slides, Posters, Mugs, T-shirts,,
Calendars, Jigsaws, Tableware, Caricatures, Greetings cards, Picture
bags, Photo Album and Book covers, Canvas Prints, tissues and more
..... from your own digital images.