apache compromised to send spam, need way to check file access

First of all, I am not a linux expert. I have done linux for a few years
and have managed to get a successful server/gateway/firewall with samba
networking for my home LAN. I an not a true "newbie" but do not understand
all of the tools at my disposal and would appreciate some tolerance and
real help and guidance now, I really need your help. If you could please be
"helpful and nice", this would go a long way, thank you very much. I can
take a little criticism but pure criticism and no actual help would be
rather discouraging. <quite sincere here>

I started this conversation in comp.mail.sendmail and after finding out
that this is apache spamming the world at large via sendmail, it was
suggested that I take my security question here by Susan. My original
thread can be found as "sendmail compromised - Somebody help me!" in the
newsgroup, comp.mail.sendmail, if anyone needs further information. Sue's
summary of my post:

<short summary: OP runs webserver on a DSL-connection. Recently spam
injected by his webserver is clogging his mailqueue. Distribution is
RedHat 9 and no, he has not heard about fedoralegacy>

I have a redhat 9 linux server on a 24/7 ADSL connection that acts as a
server/gateway/firewall for my small, 3 PC home LAN. The redhat box has 3
FQDNs on it with 3 virtual hosts that are hosted in the user_dirs. One site
is my personal site with virtually no content, I use for an http file
server for friends. The second site is a family website that runs phpbb
2.0.6, the entire site is behind an .htacccess file that passwords it. I
also run coppermine photo gallery 1.2.1 on it. Coppermine and phpbb can
both send mail. The third site is an Outlook Express stationery website for
a friend and it now has a guestbook, openbook version 1.2.2. The guestbook
can send mail when a user makes an entry, it will send mail to myself and
the webmaster. My ISP blocks port 25 traffic so the server cannot receive
any email from the world but it can send mail out by using the smarthost
feature of the sendmail.mc file. I have enabled this feature in order to
send mail and not be rejected by the dnsbl lists that reject all mail from
anyone on a DUN list. Apparently anyone running a mail server on an IP
range that falls into a cable, dial up, or DSL connection is black listed
so I used the smarthost to send mail through my ISP's mail server, back in
the days before port 25 was truly "stealthed" by my ISP for everyone with
an Internet account. Cannot send mail on port 25 now, not even to another
mail server mail account, only through my ISP's mail server.

I know that redhat 9 is old now and should be replaced with a newer distro,
perhaps FC3 or so. I cannot do this now, I have years of installs, setups,
and tweaks on this server and cannot bring it down for weeks or months
while I install a new distro, but I am seriously considering it and will
get to it, someday this year, I hope. For now, the redhat 9 box has to stay
and something must be done to stop the spam emails coming from apache.

I believe the spam to be coming from apache as this email that I got from
my redhat box via pop3 showed up in my inbox:

---------------------------------------------------------------------

Return-Path: <apache@ohmster.com>
Received: from ohmster.com (localhost.localdomain [127.0.0.1])
by ohmster.com (8.12.8/8.12.8) with ESMTP id j3J0bADa030038
for <root@ohmster.com>; Mon, 18 Apr 2005 20:37:17 -0400
Received: (from apache@localhost)
by ohmster.com (8.12.8/8.12.8/Submit) id j3J0b8wP030036
for root; Mon, 18 Apr 2005 20:37:08 -0400
Date: Mon, 18 Apr 2005 20:37:08 -0400
From: Apache <apache@ohmster.com>
Message-Id: <200504190037.j3J0b8wP030036@ohmster.com>
To: root@ohmster.com
Subject: Account compromised
Status: O
X-SpamSubtract-Analysis: other: not to or cc me
X-SpamSubtract-Analysis: user moved message to inbox.

This account has been compromised, please clean it

---------------------------------------------------------------------

Looking through the maillogs shows that apache has sent a large amount of
spam email all over the world. This is very troubling and has to be
stopped. What I believe and most everyone in the sendmail newsgroup
believes is that a php, pl, or cgi file that can send mail has been
exploited and is being used to spam the world at large via apache. Some
version info on my setup:

[root@ohmster mail]# uname -a
Linux ohmster.com 2.4.20-31.9 #1 Tue Apr 13 18:04:23 EDT 2004 i686 i686
i386 GNU/Linux
[root@ohmster mail]# rpm -q sendmail
sendmail-8.12.8-9.90
[root@ohmster mail]# rpm -q httpd
httpd-2.0.40-21.17.legacy
[root@ohmster mail]#

I had an incident today and in the past, where the redhat box has slowed
way down to a crawl and by ssh'ing over to the box from my desktop
computer, I ran top and discovered a runaway perl process, owned by apache,
that was chowing down on 99% of the CPU usage of the machine. This activity
continued until I killed the perl process with a kill -9. Here is the very
top line of the top screen:

16313 apache 25 0 2092 468 240 R 99.7 0.0 1884m 0 perl

I could not even restart httpd until this perl process was killed, see
results:

[root@ohmster mail]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: (98)Address already in use: make_sock: could not bind to
address 0.0.0.0:443
no listening sockets available, shutting down
[FAILED]
[root@ohmster mail]# kill -9 16313 (The runaway perl process)
[root@ohmster mail]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: [ OK ]
[root@ohmster mail]#

Something had took over the socket(?) and was pretty much running things
over here on my end with a perl process, a web mail exploit, perhaps? Would
there be any logs of use to investigate this incident a little further,
perhaps?

What I desperately need is to find a way to see what web directory files
are being unusually accessed a lot, with regard to the rest of the site.
Since all three web sites are such low traffic, anyone using a mail capable
file would stick out like a sore thumb. How can I check when files in a web
directory are accessed in a short period of time, let's say a few days or a
week? I cannot find a way to make ls do this. I am not sure it can be done
with ls. Is there any kind of script that I can run with cron to watch all
of the files in a directory, and it's subdirectories, to see when they are
accessed and report a date and time for each incident of access, and log it
to a file that can be examined? There has to be a way to monitor these
public_html directories for file access for a short time and see which
files are being used to spam the world with apache. I need to do this for a
short time to find the culprit so that I can put a stop to it.

Sue is right, this is now a security question so I have to seek help from
the linux security community on this so that is why I am here. Please help
me to find a way to find out which perl files are being used in this
exploit so that it can be stopped. I will watch this thread closely and
will respond to any requests for further information and tests. Thank you,
I really need your help now.

--
~Ohmster
ohmster at newsguy dot com

> What I desperately need is to find a way to see what web directory files
> are being unusually accessed a lot, with regard to the rest of the site.

Sorry, I just skimmed through your post. Remember that you really can't
trust any tools you use (even as root) on a compromised host.

But I would be tempted to try using 'lsof' (install it if you don't already
have it) to see what files are currently in use. As the web server fetches
files, they will be visible by lsof in real-time.

--
Jem Berkes
Software design for Windows and Linux/Unix-like systems
http://www.sysdesign.ca/

On Mon, 25 Apr 2005 05:28:15 +0000, Jem Berkes wrote:

>> What I desperately need is to find a way to see what web directory
>> files are being unusually accessed a lot, with regard to the rest of
>> the site.
>
Jem is a smart guy who has helped me more than once before (and right
here!) What he says has value.

> Sorry, I just skimmed through your post. Remember that you really can't
> trust any tools you use (even as root) on a compromised host.

It is regrettable that you find yourself in this predicament. Please
don't let it happen to you again. As complex as your requirements are,
you should be using an IDS like tripwire, for early alerts for intrusions.
That's 20/20 hindsight, but probably too late and not what you need most
right now.

http://www.tripwire.org/

Get some tools that you can trust, so you can see what has really
happened. Many would say to immediately disconnect, and they would have
good reasons. Whether you immediately disconnect or not might be your own
choice in the short term. As long as you don't disconnect, whoever may
have access to your systems may continue to infect and restrict your
efforts.

Often, the best immediate forensic response is to boot from a "read only"
OS like Knoppix (CD-ROM).

www.knoppix.org

http://cart.cheapbytes.com/cgi-bin/c...ml?id=en9NQIQV

> But I would be tempted to try using 'lsof' (install it if you don't
> already have it) to see what files are currently in use. As the web
> server fetches files, they will be visible by lsof in real-time.

If you cannot promptly control this, them please disconnect. I wish you
well and good luck.

On Mon, 25 Apr 2005 04:05:30 +0000, Ohmster mumbled something like this:

> The second site is a family website that runs phpbb
> 2.0.6

phpbb has had many vulnerabilities recently and is now at version 2.0.14.
This could well be the point of entry.

When you have sorted your current problems out, I would recommend you to
subscribe to the bugtraq mailing list and keep all applications up to date.

--
Rinso
/\
/ \
/wizz\
~~~~~~~~~~~~

Jem Berkes <jb@users.pc9.org> wrote in
news:Xns964351028A9Ejbuserspc9org@130.179.16.24:

> But I would be tempted to try using 'lsof' (install it if you don't
> already have it) to see what files are currently in use. As the web
> server fetches files, they will be visible by lsof in real-time.

Ohhh, that would have helped yesterday when the machine ran away with a
perl process by apache. Was using 99% CPU per top and would not quit. Had
to kill it -9. I would have killed to know what that perl process was! I
have never used lsof but that would have been the time, let me try it
now...

Whoa! Tons of stuff, just scrolling by! Okay this is going to take some
figuring, will have to whittle the output down to files opened by apache
and then maybe grep the output for php, pl, or cgi. Might have to wait for
it to happen again, the perl proc I have seen run away twice over several
months, might be more frequent, will have to keep and eye on it.

That is a helpful tool, thanks for the advice, Jem.

--
~Ohmster
ohmster at newsguy dot com

Newsbox <nospam_for_me_please@thanks.invalid> wrote in
news:3MadnU0a2YcmDPHfRVn-hA@acadia.net:

> Jem is a smart guy who has helped me more than once before (and right
> here!) What he says has value.

Yeah no kidding, his advice was very valuable, I am writing it on the
wall so as not to forget it, or at least will print it and tape it up.

>> Sorry, I just skimmed through your post. Remember that you really
>> can't trust any tools you use (even as root) on a compromised host.

Not really sure this is a compromised host, just a form mail or other
apache spam email exploit. No evidence of being compromised, other than
that.

>
> It is regrettable that you find yourself in this predicament. Please
> don't let it happen to you again. As complex as your requirements
> are, you should be using an IDS like tripwire, for early alerts for
> intrusions. That's 20/20 hindsight, but probably too late and not what
> you need most right now.
>
> http://www.tripwire.org/

I have tripwire installed, I used to get email from it every day, long as
hell emails that were very difficult to understand. That was years ago.
Tripwire is still there but I have not gotten email from it for a really
long time. Not sure what happened to it though. Will have to invistigate
that. Had to put in all kinds of pass phrases for it.

[ohmster@ohmster ohmster]$ rpm -q tripwire
tripwire-2.3.1-17.2.legacy.9

>
> Get some tools that you can trust, so you can see what has really
> happened. Many would say to immediately disconnect, and they would
> have good reasons. Whether you immediately disconnect or not might be
> your own choice in the short term. As long as you don't disconnect,
> whoever may have access to your systems may continue to infect and
> restrict your efforts.

Not disconnecting immediately, but will keep a very close eye on it and
stop services or connection if more evidence reveals an actual threat,
other than spam email from apache. Not taking this lightly though, will
really watch it for mal activity.

>
> Often, the best immediate forensic response is to boot from a "read
> only" OS like Knoppix (CD-ROM).
>
> www.knoppix.org

Have it, used it to copy both the / hard drive with swap and boot
partitions over to a larger one and the /home drive to a larger one.
Actually copied the drives with ghost and then used knoppix to boot and
mount the new / drive, mount it, chroot it, and then reinstalled grub on
it. Worked very well. Good stuff that knoppix.

>
> http://cart.cheapbytes.com/cgi-bin/c...se=502/tf=titl
> e.html?id=en9NQIQV
>
>> But I would be tempted to try using 'lsof' (install it if you don't
>> already have it) to see what files are currently in use. As the web
>> server fetches files, they will be visible by lsof in real-time.

Yeah yeah, this for sure. Will be learning a lot about lsof now and how
to see what apache is doing and how to watch it.

>
> If you cannot promptly control this, them please disconnect. I wish
> you well and good luck.

Taking control, thank you for your advice and for your help.

--
~Ohmster
ohmster at newsguy dot com

"Dave {Reply Address in.sig}" <noone$$@llondel.org> wrote in
news:5702848.IE1brrojB9@robinton.llondel.org:

> [snip]
>
> Do you have any apache log files? Often found in /var/log/httpd.
>
> If so, have a look through them and see if any .pl, .php, .cgi, etc
> files are being accessed. Check your cgi-bin directory (on RH9 it may
> be /var/www/cgi-bin) for such files. Of course, if your machine has
> been compromised rather than just leaving a mail-capable executable
> where others can find it, then you've got bigger problems that require
> a complete re-install.
>
> --
> Dave

Yeah. Tried that yesterday but I cannot figure out how to grep the output
of "cat *" in /var/log/httpd or at least each set of logs at a time such
as "grep access*" and then filter out specifically for .pl, .cgi, or
..php. Just using the letters gave me far too much irrelevent output,
these logs are freaking huge and there are lots of them. Trying to grep
with the . such as .pl did not seem to work. There must be a better
way...

Don't believe the machine has been compromised as no evidence of it,
other than apache spamming the public now and apache would do this with
any one of several files it can use in the www roots to email with? Wish
there were more evidence in the mails as to what file or file(s) are
doing this. Have to watch this now more closely. Have not totally ruled
out a compromised host, will watch very closely, not taking your advice
lightly. If you have anymore suggesgtions, please feel free to express
them, I am watching.

Thanks Dave.

--
~Ohmster
ohmster at newsguy dot com

Rincewind <rinso@unseen.edu> wrote in
news:pan.2005.04.25.11.13.54.238396@unseen.edu:

> phpbb has had many vulnerabilities recently and is now at version
> 2.0.14. This could well be the point of entry.
>
> When you have sorted your current problems out, I would recommend you
> to subscribe to the bugtraq mailing list and keep all applications up
> to date.

This I just found out, what leans me away from this is that all of the
phpbb files are in an .htaccess protected directory for the family
website. One cannot access those directories unless one knows the
log/pass for that directory and it's subs. Still worth checking though.
Can grep through the apache logs for all of the user/passes to see who
has been in there recently.

phpbb is such a freaking pain in the bloody arse to update though,
especially if one has mods applied to it like the calendar, etc. I did
this once, ideally you download the patch, apply it, watch the output,
and all is well. Not so in real life. It fails all over the place to
update and then you have to do it by hand, file by file, line by line,
and it takes a long time. Yeah but you are right, I should do this. I
wanted to know what to do and you are making this quite evident, my
friend.

bugtraq, bugtraq. This is the second time I hear about bugtraq in the
past few months with regards to this distro. Might have to go for it now,
this is pretty serious. Okay bugtraq and a phpbb update are on the todo
list, the apache spam threat is on top but these two items are at the top
as well, they could turn out to be more related than I think right now.
Thanks for your help, Rinse.

--
~Ohmster
ohmster at newsguy dot com

Newsbox <nospam_for_me_please@thanks.invalid> wrote in
news:3MadnU0a2YcmDPHfRVn-hA@acadia.net:

>
> http://www.tripwire.org/

Tripwire is still in the cron queue as per webmin. It runs every day at 4
in the morning. Wonder why it does not email me anymore? Will have to
investigate this further. I have not gotten any tripwire mails in a long,
long time, close to two years, I bet. Hard to understand them, maybe I
became complacent. Need to figure this out, saving all posts for future
review and to do that I say I will do. Thank you, Newsbox.

--
~Ohmster
ohmster at newsguy dot com

On Mon, 25 Apr 2005 12:28:43 +0000, Ohmster mumbled something like this:

> Yeah. Tried that yesterday but I cannot figure out how to grep the output
> of "cat *" in /var/log/httpd or at least each set of logs at a time such
> as "grep access*" and then filter out specifically for .pl, .cgi, or .php.

Try:

grep [.pl][.cgi][.php] *access*


--
Rinso
/\
/ \
/wizz\
~~~~~~~~~~~~