Re: sendmail compromised - Somebody help me!

On Tue, 26 Apr 2005 16:23:20 +0000, Bev A. Kupf wrote:

[...]

I could not say it better.
What is a smale ?

On Tue, 26 Apr 2005 15:13:21 -0400,
Newsbox (nospam_for_me_please@thanks.invalid) wrote:

> What is a smale ?

Something that when you get a whole bunch of becomes a grate.

More seriously, its from "The Parson's Tale" (which is one of
the stories in the collection called The Canterbury Tales)

Beverly
--
Many a smale maketh a grate -- Geoffrey Chaucer

"Bev A. Kupf" <bevakupf@myhome.net> wrote in
news:slrnd6sqnn.hk9.bevakupf@myhome.net:

> I would urge you to give the matter some serious consideration, before
> sticking with your decision to just disable sendmail, and leave the
> box online.

Sure, no problem, doesn't matter that this is something that I
absolutely, and have made quite clear, *need*. I will just throw it out.
No problem, right on top of the trash heap. I mean, that is what I came
here for, right? Not for serious help, solving or tackling this spam over
apache issue, it was to throw out my much needed property. No problem
there, just throw it right on out. Hey, the tail pipe is the Wife's nice
Honda is rattling a bit when the car is at a stoplight. I could come here
for help with that too, right? Sure, 2,000 Honda Civic, tail pipe
rattling? You know the deal ohmster, throw that car right out. Besides,
your wife will get much needed exercise that way, what with all of the
walking and all. That is what the security newsgroup is for, getting you
to throw your property away. Yeppers. Hey, my bathtub takes forever to
drain, Roto Rooter cannot do anything about it but the security newsgroup
can help with that, right? Just throw that bathtub right on out. Use
Handi Wipes or something in the future, much more secure, you see? Heck,
I could have the world just mail me request for web pages and then I
could print them out and mail them to them or something. Talk about
security! And for work, when I need a technical document on my webserver
from home, I could just tell the customer to wait, zoom about 70 miles
back home (Thank God I did not come here for help with *both* cars!),
print out the doc, and then zoom right back there again, hey no problem
at all. Won't have many customers after that, not any income to speak of,
but think of all of the blissful security that I can bask in at night,
sleeping in the woods.

We could fix DSW, they just had their computer system broken into and the
hackers got, what was it, 1.3 million credit card numbers? Why pay
millions on security for this problem, the comp.os.linux.security
newsgroup has the answer for that problem. Just run all over the
building, unplugging all of your equipment, and shut it down and pull the
plug. See? Problem solved. No security risk at all anymore. What a
Godsend this newsgroup is, guaranteed to fix any security problem; shut
everything down, unplug it, and never reconnect it again, or at least
wait several years until you get the kind of software and hardware that
we approve of, then maybe we will let you get back to earning a living
and supporting your family. In the meantime, eat grass and sleep under
trees.

Bev, please don't take this personally. I had to blow off steam. I came
here for help with a security issue and I started getting good help and
now all I get is "pull the plug". Not helpful, not appreciated, not
wanted, save your breath. I will figure it out myself somehow to get me
past the duration until I can wipe the drive and install Fedora Core and
NO, I am not going without servers until I can do this. Appreciate your
concern, good day.

Here is output of rootkit, I don't see anything so terrible, do you? I
will listen, go ahead, and I promise to be respectful. :)
http://www.chkrootkit.org/

ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `date'... not infected
Checking `du'... not infected
Checking `dirname'... not infected
Checking `echo'... not infected
Checking `egrep'... not infected
Checking `env'... not infected
Checking `find'... not infected
Checking `fingerd'... not infected
Checking `gpm'... not infected
Checking `grep'... not infected
Checking `hdparm'... not infected
Checking `su'... not infected
Checking `ifconfig'... not infected
Checking `inetd'... not tested
Checking `inetdconf'... not found
Checking `identd'... not infected
Checking `init'... not infected
Checking `killall'... not infected
Checking `ldsopreload'... not infected
Checking `login'... not infected
Checking `ls'... not infected
Checking `lsof'... not infected
Checking `mail'... not infected
Checking `mingetty'... not infected
Checking `netstat'... not infected
Checking `named'... not infected
Checking `passwd'... not infected
Checking `pidof'... not infected
Checking `pop2'... not found
Checking `pop3'... not found
Checking `ps'... not infected
Checking `pstree'... not infected
Checking `rpcinfo'... not infected
Checking `rlogind'... not infected
Checking `rshd'... not infected
Checking `slogin'... not infected
Checking `sendmail'... not infected
Checking `sshd'... not infected
Checking `syslogd'... not infected
Checking `tar'... not infected
Checking `tcpd'... not infected
Checking `tcpdump'... not infected
Checking `top'... not infected
Checking `telnetd'... not infected
Checking `timed'... not found
Checking `traceroute'... not infected
Checking `vdir'... not infected
Checking `w'... not infected
Checking `write'... not infected
Checking `aliens'... no suspect files
Searching for sniffer's logs, it may take a while... nothing found
Searching for HiDrootkit's default dir... nothing found
Searching for t0rn's default files and dirs... nothing found
Searching for t0rn's v8 defaults... nothing found
Searching for Lion Worm default files and dirs... nothing found
Searching for RSHA's default files and dir... nothing found
Searching for RH-Sharpe's default files... nothing found
Searching for Ambient's rootkit (ark) default files and dirs... nothing
found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/perl5/5.8.0/i386-linux-thread-multi/.packlist
/usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi/auto/GD/.packlist
/usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-
multi/auto/fb_c_stuff/.packlist /usr/lib/nvu-0.90/.autoreg /usr/lib/qt-
3.0.5/etc/settings/.qtrc.lock /usr/lib/qt-3.0.5/etc/settings/.qt_plugins_
3.0rc.lock /usr/lib/qt-3.0.5/etc/settings/.kstylerc.lock
/usr/lib/openoffice/share/gnome/net/.directory
/usr/lib/openoffice/share/gnome/net/.order
/usr/lib/openoffice/share/kde/net/applnk/OpenOffice.org/.directory
/usr/lib/openoffice/share/kde/net/applnk/OpenOffice.org/.order
/usr/lib/transgaming_winex3/.transgaming
/usr/lib/transgaming_winex3/.transgaming
Searching for LPD Worm files and dirs... nothing found
Searching for Ramen Worm files and dirs... nothing found
Searching for Maniac files and dirs... nothing found
Searching for RK17 files and dirs... nothing found
Searching for Ducoci rootkit... nothing found
Searching for Adore Worm... nothing found
Searching for ShitC Worm... nothing found
Searching for Omega Worm... nothing found
Searching for Sadmind/IIS Worm... nothing found
Searching for MonKit... nothing found
Searching for Showtee... nothing found
Searching for OpticKit... nothing found
Searching for T.R.K... nothing found
Searching for Mithra... nothing found
Searching for LOC rootkit... nothing found
Searching for Romanian rootkit... nothing found
Searching for HKRK rootkit... nothing found
Searching for Suckit rootkit... nothing found
Searching for Volc rootkit... nothing found
Searching for Gold2 rootkit... nothing found
Searching for TC2 Worm default files and dirs... nothing found
Searching for Anonoying rootkit default files and dirs... nothing found
Searching for ZK rootkit default files and dirs... nothing found
Searching for ShKit rootkit default files and dirs... nothing found
Searching for AjaKit rootkit default files and dirs... nothing found
Searching for zaRwT rootkit default files and dirs... nothing found
Searching for Madalin rootkit default files... nothing found
Searching for Fu rootkit default files... nothing found
Searching for ESRK rootkit default files... nothing found
Searching for anomalies in shell history files... nothing found
Checking `asp'... not infected
Checking `bindshell'... not infected
Checking `lkm'... chkproc: nothing detected
Checking `rexedcs'... not found
Checking `sniffer'... eth1: PF_PACKET(/usr/sbin/pppoe,
/usr/bin/natmonitord)
ppp0: not promisc and no PF_PACKET sockets
Checking `w55808'... not infected
Checking `wted'... chkwtmp: nothing deleted
Checking `scalper'... not infected
Checking `slapper'... not infected
Checking `z2'... chklastlog: nothing deleted
Checking `chkutmp'... The tty of the following user process(es) were not
found
in /var/run/utmp !
! RUID PID TTY CMD
! root 1661 ttyS0 powerd
chkutmp: nothing deleted

--
~Ohmster
ohmster at newsguy dot com

On Tue, 26 Apr 2005 23:45:16 GMT,
Ohmster (notareal@emailaddress.com) wrote:

There are two ways for you to take advice from professionals in the
field. The first is to consider that the opinion offered _may_ just
be a well considered opinion, especially if it comes from more than
one source.

The second is to throw a hissy-fit. The facts are that _your_
incompetence led to _your_ box possibly being compromised. As a
sysadmin, its _your_ job to keep your box current with whatever
patches are offered for your system.

Let's say that your box is used to launch attacks on several other
boxes. Is it right that someone else has to spend their time
(minimally) redesigning a firewall, because _you_ were incompetent?

And btw, I've cut most of your rant out, but no one has suggested
that you throw the box out. Now a sensible sysadmin would have
a backup of all the data. And it would take less than a day to
reload a _secure_ operating system, and then restore the data from
backup (we are after all talking about a single box here). But
everything that you've displayed of yourself here indicates that
sense is something that doesn't come easily for you. So, off
course you don't have any backups.

> Sure, no problem, doesn't matter that this is something that I
> absolutely, and have made quite clear, *need*. I will just throw it out.
> No problem, right on top of the trash heap. I mean, that is what I came
> here for, right?

You came here for advice. Whether you take what is offered to you
or not is your choice. Too bad you don't like tough medicine if
that's what is called for.

You've been given other advice. Find out which scripts are being
accessed repeatedly from Apache's access_log. Did you do that?
Heck, no. Put plainly, you're _incompetent_. And stupidity is
its own reward.

Beverly
--
Many a smale maketh a grate -- Geoffrey Chaucer

"Bev A. Kupf" <bevakupf@myhome.net> wrote in
news:slrnd6tn8p.i7a.bevakupf@myhome.net:

> On Tue, 26 Apr 2005 23:45:16 GMT,
> Ohmster (notareal@emailaddress.com) wrote:
>
> There are two ways for you to take advice from professionals in the
> field. The first is to consider that the opinion offered _may_ just
> be a well considered opinion, especially if it comes from more than
> one source.

I got quite a bit of good advice. Searching through the apache logs for
files that can email and are being used more often than should be. I got
a great tip from Jem to use lsof to snap a list of open files when the
incident occurs. Got advice about tripwire and it is installed, but I am
not sure if that would be of much help now. The barn door and all. Mike
sent me a detailed pdf file, analysis of the security, some of it pretty
scary, some quite minor. I appreciate and an working on "the list". Got
tips about phpbb2 from Michael from bugtraq. All really good advice, all
helpful, either now or for future consideration. There were a few that
insisted that the box come offline, without any reasonable expectation of
getting it back online again. That really was not helpful. I use this
server every day for work and cannot simply take it down like that.

> The second is to throw a hissy-fit. The facts are that _your_
> incompetence led to _your_ box possibly being compromised. As a
> sysadmin, its _your_ job to keep your box current with whatever
> patches are offered for your system.

Hissy fit? Ma'am, I made it clear that I could not take the box offline
for an extended period, but could do things now like shutdown sendmail,
as that seems to be a threat to others and myself. I will listen to any
other reasonable suggestions that do not bring down the servers, even
removing the formmail pages. How many times must one stress the same
point, about taking down the servers, only to have it ignored and to
request that the box be taken offline indefinitely and immediately? I was
frustrated, that is all. The "hissy fit" was directed at no one in
particular and I made it very clear that it was not directed at you, Bev,
unless you misunderstood "Not helpful, not appreciated, not wanted, save
your breath." to be a personal attack. It was not. I meant "save your
breath for something important like breathing" other than to repeat the
same advice that I cannot take in it's entirety and had already made
quite clear, several times.

> Let's say that your box is used to launch attacks on several other
> boxes. Is it right that someone else has to spend their time
> (minimally) redesigning a firewall, because _you_ were incompetent?

Of course not, that is why I shutdown sendmail and stopped it from
starting at boot time. The "attacks" of spam are a result of some
security issue with a formmail exploit of apache. I cannot shut down
apache or take the box down for days, weeks, or longer, but I can stop
sendmail and the spam will cease because of it, until I can find the real
root of the formmail exploits. I even ran a chkrootkit to be sure as was
suggested from the professionals. ...again with the "incompetent" word?

> And btw, I've cut most of your rant out, but no one has suggested
> that you throw the box out. Now a sensible sysadmin would have
> a backup of all the data. And it would take less than a day to
> reload a _secure_ operating system, and then restore the data from
> backup (we are after all talking about a single box here). But
> everything that you've displayed of yourself here indicates that
> sense is something that doesn't come easily for you. So, off
> course you don't have any backups.

Uh, yeah, I have the data backed up on the original hard disks. The /home
disk and the /, swap, and /boot partitions are on a second disk, this was
just done recently and the data is there. The data is not so "mission
critical" that I need daily backups of it. It took me a long time to get
this system running like it is, I could not install and configure
everything for a new system in a day. There is firewall and NAT, there
are a few mysql databases, there is the web server and virtual hosts,
there is samba and shares, there are personal and custom tweaks. I could
install a new distro and get it running in a day though, but I would have
to re-install and re-configure everything to work as it did for a new
distro and that could not be done in a day, at least by me, I am not a
professional at this, but I do enjoy it. It is difficult to be
"incompetent" when one claims no excellence in the first place. Yes, I
have the backup that I need.

> You came here for advice. Whether you take what is offered to you
> or not is your choice. Too bad you don't like tough medicine if
> that's what is called for.

Tough medicine? Look, I came here to root out an apache formmail exploit,
you were the one that actually directed me here. I got good medicine.
That kind of "tough medicine" might be called for in your book, had I
discovered a serious security flaw in the ckrootkit, I would have taken
the box down. I won't run a compromised box where I know that root access
has been compromised, that would be just plain foolish. There is no
indication of that level of compromise. Just a non-privileged daemon
mailing out spam from a formmail exploit. Yeah it bites the big one and
all but it can be contained until something can be done to make it more
secure.

> You've been given other advice. Find out which scripts are being
> accessed repeatedly from Apache's access_log. Did you do that?

Sure, searching through the logs for files accessible to apache that can
email was good advice. But there is over a hundred megs of text logs.
That is a lot of searching. I tried but got bleary eyed trying. Dave
showed me how to use regex in grep to help with the searching. Maybe I am
not searching properly but I am working on refining the search. It will
take a while though. Sendmail is shut down in the meantime.

> Heck, no. Put plainly, you're _incompetent_. And stupidity is
> its own reward.

Again with that word. If I came here claiming to be a professional
sysadmin, then you could probably say "incompetent" and you would be
right. I never said that. I am a casual linux user and I learn more as
time passes. I setup and use the servers on my linux machine and I need
them. I am very sad that there is a formmail exploit on my system because
I really do need the servers. I never insulted you personally, Bev, and
took the time to say this in my "hissy fit". I did not come here to
exchange insults, I just wanted some tips or pointers in the right
direction. I got some, by some very well meaning, professional
individuals. The problem is not solved yet but I have things to work on
now. I do believe that you were the one to point out that this is a
formmail exploit in the first place. What is the point in swapping
insults? I am sure you are competent enough to do your job and I
appreciate your time.

> Beverly

--
~Ohmster
ohmster at newsguy dot com

On Wed, 27 Apr 2005 01:41:23 +0000, Ohmster wrote:

> "Bev A. Kupf" <bevakupf@myhome.net> wrote in
> news:slrnd6tn8p.i7a.bevakupf@myhome.net:
>
>> On Tue, 26 Apr 2005 23:45:16 GMT,
>> Ohmster (notareal@emailaddress.com) wrote:
>>
>> There are two ways for you to take advice from professionals in the
>> field. The first is to consider that the opinion offered _may_ just be
>> a well considered opinion, especially if it comes from more than one
>> source.
>
> I got quite a bit of good advice. Searching through the apache logs for
> files that can email and are being used more often than should be. I got
> a great tip from Jem to use lsof to snap a list of open files when the
> incident occurs. Got advice about tripwire and it is installed, but I am
> not sure if that would be of much help now. The barn door and all.

If you haven't seen a (tripwire) report in a couple of years then it's not
really doing you any good, installed or not. Forget about tripwire for
now, unless of course you think it might be useful to know why it stopped
sending you those long e-mails... Maybe your box was cracked in 2003?

> Mike
> sent me a detailed pdf file, analysis of the security, some of it pretty
> scary, some quite minor. I appreciate and an working on "the list".

You should have been scanning your sites yourself regularly right along,
and you might have been able to take care of those "scary" things long
ago. You set up what seems to be an audacious site, with mail and web
servers and dog knows what else, all connected to the public; by
comparison my own local systems are very simple and minimal, with _no_
_servers_, a good and tested firewall, etc, etc, etc, and I still scan my
externals regularly. My mails and domains are hosted by external
providers with their own staffs of dedicated, knowledgeable professionals.
How much did you save by putting all your (essential) stuff on a box in
your house? $10/month, $20, $30, what??? And are all the rest of us in
the world supposed to take our lumps from your box because you wouldn't
hire someone who knew what s/he was doing, and who would do all the
routine but necessary things to keep it running and safe, the same things
that you obviously didn't know enough about or care enough about to learn
how to do?

Look Ohmster, I really don't want to be unnecessarily harsh or unkind. You
continue to answer in mostly good ways, and it appears you are a serious
and in many ways responsible person. I have read and understood what you
have written. In several ways you sound like a man who is looking for
good alternatives. I know that you want to be convinced that all your
troubles are just because of a bad formmail script, and that is even still
possible. But from what you have written, you wouldn't know if your box
were vectoring an attack on a nuclear power station, or on the FBI or NSA.
Honestly and truly, your refusal to disconnect your server is just simply
not responsible, is just simply not acceptable. Read below.

HERE IS A GOOD SUGGESTION:

Outsource your servers to a qualified hosting concern, and _THEN_
disconnect and rebuild your own home-based server. There are hundreds, if
not thousands of good, reasonably priced hosting companies easily found
with common, free search engines. It shouldn't take more than a day or
two at most for the DNS changes to be reflected worldwide. You can just
FTP your web pages to their servers, set up you e-mail with them, and then
you can take you time to deal with your wife's rattling tailpipe. Doesn't
that sound like a good alternative?

To make it easy for you (this is not a "plug", dog knows they don't need
any more "problem customers"), here is one that I know is good. They have
excellent e-mail support (no telephone support) during their US Pacific
Coast working hours.

https://www.tigertech.net/

Starts at $6.25 per month per domain. Doesn't that sound like a cheap and
easy way out of your immediate problems? And you can almost immediately
save all the rest of us in the world from the assaults that your
home-based server might spew. Hope you take this advice immediately.
Really, really do.

> Got
> tips about phpbb2 from Michael from bugtraq. All really good advice, all
> helpful, either now or for future consideration. There were a few that
> insisted that the box come offline, without any reasonable expectation
> of getting it back online again.

Would you drive a car knowing it had bad brakes and might kill someone?
Would it be the other motorist's concern that you didn't know how long it
would be before you could see your way clear to fixing your brakes? Get
real. Park it. Take a cab.

> That really was not helpful. I use this server every day for work and
> cannot simply take it down like that.

You certainly can, and simply, and without disrupting anything essential.

Hire a remote server. It's cheap and easy. You can get lots of GOOD help
here and elsewhere at no cost to you. But if you don't take the good
advice just because it is free (even though it comes the same from
multiple known good sources) then you can vent and complain all you want,
the good advice won't help you. People are trying to make a living
providing exactly the services that you need, and for affordable prices,
and you should stop complaining and listen. Then hire some of those good
and competent people to do what you need done. Throw your bias and your
hubris on the trash, not your server hardware. Would you know the
difference?

>> The second is to throw a hissy-fit. The facts are that _your_
>> incompetence led to _your_ box possibly being compromised. As a
>> sysadmin, its _your_ job to keep your box current with whatever patches
>> are offered for your system.
>
> Hissy fit? Ma'am, I made it clear that I could not take the box offline

You certainly can and you certainly should.

> for an extended period, but could do things now like shutdown sendmail,
> as that seems to be a threat to others and myself.

You don't know clue #1 what you machine is doing.

> I will listen to any
> other reasonable suggestions that do not bring down the servers,

See the above

> even
> removing the formmail pages.

So, who cares about your formmail pages except you. We care about
compromised machines being connected to the public, and about intransigent
know-nothings who insist on keeping bad, unmaintained systems connected.

> How many times must one stress the same point, about taking down the
> servers, only to have it ignored and to request that the box be taken
> offline indefinitely and immediately?

You have no right to keep a compromised machine connected. It is way,
_WAY_ worse than rude to suggest otherwise.

> I
> was frustrated, that is all. The "hissy fit" was directed at no one in
> particular and I made it very clear that it was not directed at you,
> Bev, unless you misunderstood "Not helpful, not appreciated, not wanted,
> save your breath." to be a personal attack. It was not. I meant "save
> your breath for something important like breathing" other than to repeat
> the same advice that I cannot take

As outlined above, there is absolutely no reason you should not take the
good advice given you.

> in it's entirety and had already made quite clear, several times.

It has been made quite clear to you that it is not acceptable to leave a
compromised system connected. Are we clear enough, yet? Or not?


>> Let's say that your box is used to launch attacks on several other
>> boxes. Is it right that someone else has to spend their time
>> (minimally) redesigning a firewall, because _you_ were incompetent?
>
> Of course not, that is why I shutdown sendmail and stopped it from
> starting at boot time. The "attacks" of spam are a result of some
> security issue with a formmail exploit of apache.

I really do not want to be unnecessarily unkind. But you are acting in a
really dense manner. As loose as your system is, was and has been, you
really don't have any credibility to say what (else) has or has not been
compromised. The only safe assumption is that your box has been trashed,
raped and plundered. See my above good advice, and then disconnect your
machine.

> I cannot shut down
> apache or take the box down for days, weeks, or longer,

Yes you can.

> but I can stop
> sendmail and the spam will cease because of it, until I can find the
> real root of the formmail exploits.

We all have SPAM filters. SPAM is bad, but it is not what we are thinking
about most. If your box is cracked, it could be vectored to blow up a
NUKE. Smaaten up; U wouldn't know if it was or not. Disconnect it!

> I even ran a chkrootkit to be sure
> as was suggested from the professionals. ...again with the "incompetent"
> word?

If you had been competent, you would not have needed to ask the
"professionals" in the first place


>> And btw, I've cut most of your rant out, but no one has suggested that
>> you throw the box out. Now a sensible sysadmin would have a backup of
>> all the data. And it would take less than a day to reload a _secure_
>> operating system, and then restore the data from backup (we are after
>> all talking about a single box here). But everything that you've
>> displayed of yourself here indicates that sense is something that
>> doesn't come easily for you. So, off course you don't have any
>> backups.
>
> Uh, yeah, I have the data backed up on the original hard disks.

If you are determined to run such an audacious web presence in your home,
get yourself a CD writer (if you don't already have one) and do your
backups to removable media. If you box is cracked, all the data on your
hard drives has been parsed, and changed to suit the crackers' desires and
whims. Your backups should be designed for easy automated restores, so
that you can, in fact restore your entire system in a few hours,
automatically, except for switching CD's.

> The
> /home disk and the /, swap, and /boot partitions are on a second disk,
> this was just done recently and the data is there. The data is not so
> "mission critical" that I need daily backups of it. It took me a long
> time to get this system running like it is, I could not install and
> configure everything for a new system in a day.

Find out where all this wonderful stuff that you treasure is saved and
back it all up every time you tweak it, and in a way that makes it easy,
fast and automatic to restore. That's what a system administrator does,
among other things. If you don't know how to do this, hire a hosting
company. They are competent, you are not. Your machines are a hazard to
everyone else in the world.

> There is firewall and
> NAT, there are a few mysql databases, there is the web server and
> virtual hosts, there is samba and shares, there are personal and custom
> tweaks. I could install a new distro and get it running in a day though,
> but I would have to re-install and re-configure everything to work as it
> did for a new distro and that could not be done in a day, at least by
> me, I am not a professional at this, but I do enjoy it.

You may enjoy it better when you know how to do it properly.

> It is difficult
> to be "incompetent" when one claims no excellence in the first place.

You claimed "excellence in the first place" when you put your audacious
servers onto the internet. If that is not what you intended to do, then
just plain take them down.

> Yes, I have the backup that I need.

The backups that you have aren't what you need if you cannot restore your
system over a weekend. Fact, Jack.


>> You came here for advice. Whether you take what is offered to you or
>> not is your choice. Too bad you don't like tough medicine if that's
>> what is called for.
>
> Tough medicine? Look, I came here to root out an apache formmail
> exploit, you were the one that actually directed me here. I got good
> medicine. That kind of "tough medicine" might be called for in your
> book, had I discovered a serious security flaw in the ckrootkit, I would
> have taken the box down. I won't run a compromised box where I know that
> root access has been compromised, that would be just plain foolish.
> There is no indication of that level of compromise. Just a
> non-privileged daemon mailing out spam from a formmail exploit. Yeah it
> bites the big one and all but it can be contained until something can be
> done to make it more secure.

Well, maybe you are correct that only one script has been penetrated, and
then again maybe not. You are clearly trying, but with the level of
knowledge and expertise and care that you have demonstrated, you have no
reason, right or business to be running public servers. Take your box down
and hire some professionals. It doesn't cost a lot. They (may) know what
they are doing; you certainly do not.


>> You've been given other advice. Find out which scripts are being
>> accessed repeatedly from Apache's access_log. Did you do that?
>
> Sure, searching through the logs for files accessible to apache that can
> email was good advice. But there is over a hundred megs of text logs.
> That is a lot of searching. I tried but got bleary eyed trying. Dave
> showed me how to use regex in grep to help with the searching. Maybe I
> am not searching properly but I am working on refining the search. It
> will take a while though. Sendmail is shut down in the meantime.

D-uh. If it takes you a couple of years to notice that tripwire isn't
sending you daily e-mails, how serious are you about the tools that you
claim to have working?

>> Heck, no. Put plainly, you're _incompetent_. And stupidity is its own
>> reward.
>
> Again with that word. If I came here claiming to be a professional
> sysadmin, then you could probably say "incompetent" and you would be
> right. I never said that. I am a casual linux user and I learn more as
> time passes. I setup and use the servers on my linux machine and I need
> them. I am very sad that there is a formmail exploit on my system
> because I really do need the servers. I never insulted you personally,
> Bev, and took the time to say this in my "hissy fit". I did not come
> here to exchange insults, I just wanted some tips or pointers in the
> right direction. I got some, by some very well meaning, professional
> individuals. The problem is not solved yet but I have things to work on
> now. I do believe that you were the one to point out that this is a
> formmail exploit in the first place. What is the point in swapping
> insults? I am sure you are competent enough to do your job and I
> appreciate your time.
>
>> Beverly

You got some tips and pointers. Get a professional hosting service and
most of these problems will disappear. Disconnect your box until you know
more and our concerns will level right off. Quid pro quo.

Best wishes.

ps. Hope you appreciate how much time and effort it has taken us to get a
very simple result.

> I got quite a bit of good advice. Searching through the apache logs for
>> files that can email and are being used more often than should be. I got
>> a great tip from Jem to use lsof to snap a list of open files when the
>> incident occurs. Got advice about tripwire and it is installed, but I am
>> not sure if that would be of much help now. The barn door and all.

If you need your server maybe you should buy a cheap system and use it
while you fix the other system. You can get low end systems for as
little as $150. You would still be able to "do your work" without doing
in others. The suggestion to outsource your server temporarily is also a
good one.

You could also try tightening up your iptables to restrict out going
traffic which might keep the malware caged a bit. Still taking the
system off the air is best until you fix the problem. As Bev said it
really shouldn't take that long to reload your system and get it back
up. I have done it many times for friends who have gotten in trouble
both with Windows and Linux. To reinstall Linux should not take more
than an hour. To backup and restore data might take a little longer but
all told it should not take more than one long day.

As Bev said "hard medicine". I know a days lost work is hard to take but
sometime it is needed.

"Barton L. Phillips" <bartonphillips@sbcglobal.net> wrote in
news:DyPbe.1956$zu.1600@newssvr13.news.prodigy.com :

> If you need your server maybe you should buy a cheap system and use it
> while you fix the other system. You can get low end systems for as
> little as $150. You would still be able to "do your work" without
> doing in others. The suggestion to outsource your server temporarily
> is also a good one.
>
> You could also try tightening up your iptables to restrict out going
> traffic which might keep the malware caged a bit. Still taking the
> system off the air is best until you fix the problem. As Bev said it
> really shouldn't take that long to reload your system and get it back
> up. I have done it many times for friends who have gotten in trouble
> both with Windows and Linux. To reinstall Linux should not take more
> than an hour. To backup and restore data might take a little longer
> but all told it should not take more than one long day.
>
> As Bev said "hard medicine". I know a days lost work is hard to take
> but sometime it is needed.

This is all very good advice, Baron. I have decided to install a new
distro. I looked into the http logs, trying to match up php, cgi, or pl
files that could do email, against web logs to see if there is a higher
frequency of any emailable files in the logs but the challenge is
daunting. A cat and grep through a single 7Mb log files for cgi reveals
nothing, pl yields only a "search this site" script that cannot mail,
but php is a horse of a different color. Just one log file scrolls
screen after screen of php pages, you can literally sit for several
minutes, watching it all go by. Stopping to analyze it reveals general
php traffic. This is only 1 of many http logs to go through. To actually
examine and extract php file names, put them in an ordered list, count
them, then examine which ones can do email, there are many, is a very
daunting task in itself. It could be done but in the end, I get a "long
in the tooth" redhat 9 machine that is getting older all of the time and
really, just cannot be updated that much anymore.

I have sendmail disabled as an immediate stop gap and will have a new
distribution in place A.S.A.P.. All I really need is a modern distro
running with NAT and firewall in place, that should not take long, then
get my critical work files on a limited server. The rest I can take my
time with and eventually, get the system that I want this way. If I
cannot do this soon enough, I might just take one of the older computers
sitting around, I think I have an HP Pavilion in the other room, and do
you like you said, get a modern, cheap system in place for now. Yeah my
firewall needs to be tightened up, I took firestarter because it was
easy. Might go with shorewall this time as it has a webmin module now.
But all of this is old news. it is time for a new distro. I am leaning
to Fedora Core. I don't really need convincing anymore, after going
through log after log and realizing I still have an outdated machine was
enough for me.

Good advice Baron, thanks. I got work to do now, outta here, buddy!
--
~Ohmster
ohmster at newsguy dot com

Ohmster wrote:
> This is all very good advice, Baron. I have decided to install a new
> distro. I looked into the http logs, trying to match up php, cgi, or pl
> files that could do email, against web logs to see if there is a higher
> frequency of any emailable files in the logs but the challenge is
> daunting. A cat and grep through a single 7Mb log files for cgi reveals
> nothing, pl yields only a "search this site" script that cannot mail,

Did you try to look for formmail perl scripts in your cgi-bin directory
?
What version is used ?
-job

Ohmster wrote:
> "Barton L. Phillips" <bartonphillips@sbcglobal.net> wrote in
> news:DyPbe.1956$zu.1600@newssvr13.news.prodigy.com :
>
>

SNIP


> firewall needs to be tightened up, I took firestarter because it was
> easy. Might go with shorewall this time as it has a webmin module now.
> But all of this is old news. it is time for a new distro. I am leaning
> to Fedora Core. I don't really need convincing anymore, after going
> through log after log and realizing I still have an outdated machine was
> enough for me.
>
> Good advice Baron, thanks. I got work to do now, outta here, buddy!

Cool. You'll enjoy FC3, and FC4 is right around the corner.

Me.