Rootkit and system malfunction, reinstall from scratch?

Hello!

We caught a rootkit on one of out production servers
(Red Hat 7.3, Apache, Sendmail, named and some other services).
Now, I observe many malfunctions:
- cannot start sshd
- cannot run some console programs (mc, top, probably other...)

I have localized and killed intruder's processes, deleted
alien files, changed passwords and disabled some rarely used
services.

Is it a rule that in this situation I should reinstall from the scratch?

--
Regards
Michal

The problem with troubeshooting is that
sometimes the trouble shoots back

"ML [Lodz]" <lederbergANTISPAMMER@poczta.onet.pl> writes:

> Hello!
>
> We caught a rootkit on one of out production servers
> (Red Hat 7.3, Apache, Sendmail, named and some other services).
> Now, I observe many malfunctions:
> - cannot start sshd
> - cannot run some console programs (mc, top, probably other...)
>
> I have localized and killed intruder's processes, deleted
> alien files, changed passwords and disabled some rarely used
> services.

Pretty idiotic. Never delete anything on a compromised box, you'll be
shooting your own forensic ability in the head. And there is a reasonable
expectation someone may well know both your old *and* new password schemes
by now.

> Is it a rule that in this situation I should reinstall from the scratch?

Yes. Cast in stone. Take a backup of the data and reinstall everything from
scratch. And this time, do it properly - no more packages than you *need*,
no more services listening than you require, an uptodate distribution[1],
proper stateful iptables firewall[2], run a *daily* upgrade cycle[3], and
preferably a GRsecurity[4] kernel as well. Install some IDS and nIDS - AIDE
and snort for preference.

While you're putting your asbestos underwear back on, protect yourself
against some DoS as well - use rate-limiting in xinetd and in iptables,
both on outgoing udp and icmp packets and on the packets you log. Maybe
even on the incoming NEW packets you accept to services you provide, too,
if you want.

~Tim


Footnotes:
[1] TBH I wouldn't use RedHat anything for a production server; RPM update
hell for 6 months and then reinstall with the next version or wait 2 years
and become "unsupported". Give me Debian or Gentoo, any day, for *ease of
maintenance* in the long run.

[2] My starter script, easily adapted to a single server running a handful
of services, is at <http://spodzone.org.uk/packages/security/iptables.sh>.
Do *not* just rely on whatever RedHat's default configuration is like -
their idea of a sane firewall policy was pretty crap in 7.x for starters.

[3] get it to update indexes and mail you a list of what's available for
upgrade; then at 9am every day, log in and perform such upgrades as are
required.

[4] or LIDS or SElinux according to taste

--
21:48:28 up 111 days, 6:28, 0 users, load average: 0.10, 0.04, 0.01
piglet@stirfried.vegetable.org.uk |Vescere bracis meis
http://spodzone.org.uk/cesspit/ |

Tim Haynes wrote:
> "ML [Lodz]" <lederbergANTISPAMMER@poczta.onet.pl> writes:
>
>
>>Hello!
>>
>>We caught a rootkit on one of out production servers
>>(Red Hat 7.3, Apache, Sendmail, named and some other services).
>>Now, I observe many malfunctions:
>>- cannot start sshd
>>- cannot run some console programs (mc, top, probably other...)
>>
>>I have localized and killed intruder's processes, deleted
>>alien files, changed passwords and disabled some rarely used
>>services.
>
>
> Pretty idiotic. Never delete anything on a compromised box, you'll be
> shooting your own forensic ability in the head.

I mean I deleted it after I backed it up.

> And there is a reasonable
> expectation someone may well know both your old *and* new password schemes
> by now.

I ma aware of that.
>
>
>>Is it a rule that in this situation I should reinstall from the scratch?
>
>
> Yes. Cast in stone. Take a backup of the data and reinstall everything from
> scratch. And this time, do it properly - no more packages than you *need*,
> no more services listening than you require, an uptodate distribution[1],
> proper stateful iptables firewall[2], run a *daily* upgrade cycle[3], and
> preferably a GRsecurity[4] kernel as well. Install some IDS and nIDS - AIDE
> and snort for preference.
>
> While you're putting your asbestos underwear back on, protect yourself
> against some DoS as well - use rate-limiting in xinetd and in iptables,
> both on outgoing udp and icmp packets and on the packets you log. Maybe
> even on the incoming NEW packets you accept to services you provide, too,
> if you want.
>
> ~Tim
>
>
> Footnotes:
> [1] TBH I wouldn't use RedHat anything for a production server; RPM update
> hell for 6 months and then reinstall with the next version or wait 2 years
> and become "unsupported". Give me Debian or Gentoo, any day, for *ease of
> maintenance* in the long run.
>
> [2] My starter script, easily adapted to a single server running a handful
> of services, is at <http://spodzone.org.uk/packages/security/iptables.sh>.
> Do *not* just rely on whatever RedHat's default configuration is like -
> their idea of a sane firewall policy was pretty crap in 7.x for starters.
>
> [3] get it to update indexes and mail you a list of what's available for
> upgrade; then at 9am every day, log in and perform such upgrades as are
> required.
>
> [4] or LIDS or SElinux according to taste
>
Thank you for this really comprehensive reply!

--
Regards
Michal

The problem with troubeshooting is that
sometimes the trouble shoots back

Tim Haynes wrote:

> "ML [Lodz]" <lederbergANTISPAMMER@poczta.onet.pl> writes:
>
>
>> Hello!
>>
>> We caught a rootkit on one of out production servers
>> (Red Hat 7.3, Apache, Sendmail, named and some other services).
>> Now, I observe many malfunctions:
>> - cannot start sshd
>> - cannot run some console programs (mc, top, probably other...)
>>
>> I have localized and killed intruder's processes, deleted
>> alien files, changed passwords and disabled some rarely used
>> services.
>
>
>
> Pretty idiotic. Never delete anything on a compromised box, you'll be
> shooting your own forensic ability in the head.


I mean I deleted it after I backed it up.

> And there is a reasonable
> expectation someone may well know both your old *and* new password
schemes
> by now.

I do NOT use password schemes.
>
>> Is it a rule that in this situation I should reinstall from the scratch?
>
>
>
> Yes. Cast in stone. Take a backup of the data and reinstall
everything from
> scratch. And this time, do it properly - no more packages than you
*need*,
> no more services listening than you require, an uptodate distribution[1],
> proper stateful iptables firewall[2], run a *daily* upgrade cycle[3], and
> preferably a GRsecurity[4] kernel as well. Install some IDS and nIDS
- AIDE
> and snort for preference.
>
> While you're putting your asbestos underwear back on, protect yourself
> against some DoS as well - use rate-limiting in xinetd and in iptables,
> both on outgoing udp and icmp packets and on the packets you log. Maybe
> even on the incoming NEW packets you accept to services you provide, too,
> if you want.
> ~Tim
>
>
> Footnotes: [1] TBH I wouldn't use RedHat anything for a production
server; RPM update
> hell for 6 months and then reinstall with the next version or wait 2
years
> and become "unsupported". Give me Debian or Gentoo, any day, for *ease of
> maintenance* in the long run.
> [2] My starter script, easily adapted to a single server running a
handful
> of services, is at
<http://spodzone.org.uk/packages/security/iptables.sh>.
> Do *not* just rely on whatever RedHat's default configuration is like -
> their idea of a sane firewall policy was pretty crap in 7.x for starters.
>
> [3] get it to update indexes and mail you a list of what's available for
> upgrade; then at 9am every day, log in and perform such upgrades as are
> required.
>
> [4] or LIDS or SElinux according to taste
>
Thank you for this really comprehensive reply!

--
Regards
Michal

The problem with troubeshooting is that
sometimes the trouble shoots back

"ML [Lodz]" <lederbergANTISPAMMER@poczta.onet.pl> writes:

> Tim Haynes wrote:
[snip]
> > Pretty idiotic. Never delete anything on a compromised box, you'll be
> > shooting your own forensic ability in the head.
>
> I mean I deleted it after I backed it up.

Hmmm, better, but still not perfect. Once compromised, you can no longer
trust anything - find, tar, cpio coming particularly to mind - to find
everything.

Say you discover rootkit A with files A1, A2, A3. Say you were also cracked
by rootkit B which modifies files B1, B2, B3 but you don't find this out
today. Say that B2 = A2, e.g. /bin/ps. Say that B uses an LKM to divert
open() calls separately to exec() and mask the existence of /bin/ps.real.
Your backups will be in a state of disarray.

> > And there is a reasonable expectation someone may well know both your
> > old *and* new password schemes by now.
>
> I do NOT use password schemes.

The passwords themselves may still well be known whether you have a scheme
or no. Exploits I've seen have involved mailing /etc/shadow to various
addresses before now.

[snip]
> > [4] or LIDS or SElinux according to taste
> >
> Thank you for this really comprehensive reply!

No worries. I'm out to protect my own boxes; one less box likely to be used
as a source of scans and attacks is a step in the right direction. :)

<http://www.sans.org/resources/idfaq/>

<http://www.cert.org/tech_tips/win-UNIX-system_compromise.html>

HTH,

~Tim
--
22:37:56 up 111 days, 7:17, 0 users, load average: 0.10, 0.09, 0.05
piglet@stirfried.vegetable.org.uk |Sapere aude
http://spodzone.org.uk/cesspit/ |

ML [Lodz] <lederbergANTISPAMMER@poczta.onet.pl> wrote:

> Is it a rule that in this situation I should reinstall from the scratch?

Much as I hate to have to post a "Me too" article, in this case it's
merited: Listen to Tim Haynes. The man knows his stuff!

--
Cheers,
Rick Moen Ban the Bomb.
rick@linuxmafia.com Save the world for conventional warfare.

Rick Moen <rick@linuxmafia.com> writes:

> ML [Lodz] <lederbergANTISPAMMER@poczta.onet.pl> wrote:
>
>> Is it a rule that in this situation I should reinstall from the scratch?
>
> Much as I hate to have to post a "Me too" article, in this case it's
> merited: Listen to Tim Haynes. The man knows his stuff!

Thank you kindly. :)


Every so often, I figure it helps to leave an article for google-groups to
learn from, even if that means typing out an entire security policy or
something.

~Tim
--
Morning dawning / |piglet@stirfried.vegetable.org.uk
With life abounding |http://www.photoboxgallery.com/timhaynes

Tim Haynes wrote:
> Rick Moen <rick@linuxmafia.com> writes:
>
>
>>ML [Lodz] <lederbergANTISPAMMER@poczta.onet.pl> wrote:
>>
>>
>>>Is it a rule that in this situation I should reinstall from the scratch?
>>
>>Much as I hate to have to post a "Me too" article, in this case it's
>>merited: Listen to Tim Haynes. The man knows his stuff!
>
>
> Thank you kindly. :)
>
>
> Every so often, I figure it helps to leave an article for google-groups to
> learn from, even if that means typing out an entire security policy or

Tim,

thanks a lot for the tips. I have my server reinstalled at the moment,
I tried to tighten the security by proper package selection, I have
installed a firewall and have it running. I had no time for any IDS
though.
But the best thing is that you made me conscious of such things like
Gentoo distribution which can ease my administration in the future.
I will try it and I think, implement promptly.

--
Regards
Michal Leder

The problem with troubeshooting is that
sometimes the trouble shoots back

"ML [Lodz]" <lederbergANTISPAMMER@poczta.onet.pl> writes:

[snip]
> thanks a lot for the tips. I have my server reinstalled at the moment,
> I tried to tighten the security by proper package selection, I have
> installed a firewall and have it running. I had no time for any IDS
> though.

OK. Oh, while I'm at it, some log-analysis would be sensible too - keep an
eye on what people are doing to your firewall with fwlogwatch.

> But the best thing is that you made me conscious of such things like
> Gentoo distribution which can ease my administration in the future.
> I will try it and I think, implement promptly.

This is the thing: a little initial overhead, maybe 2-3 days to get a
cranking solid base installation, and you're away. One of the tricks you
could consider is if you have a farm of servers with near-enough identical
hardware - then you can set up one base gentoo installation, clone it
across the lot of them, and designate one box for building and testing
packages, and you copy the .tbz2 packages it builds across to the others -
hey presto, your servers don't need a compiler any more and treat it as a
binary distribution, while you gain the benefits of optimization and
flexible configure options (and address obfuscation) and testing on a
separate box prior to running package-upgrades live. I'm not sure it comes
much better. ;)

~Tim
--
17:16:47 up 118 days, 1:56, 0 users, load average: 0.01, 0.04, 0.06
piglet@stirfried.vegetable.org.uk |A big sky above me,
http://spodzone.org.uk/cesspit/ |West winds blow.

Tim Haynes wrote:
> "ML [Lodz]" <lederbergANTISPAMMER@poczta.onet.pl> writes:
>
[cut]

>>But the best thing is that you made me conscious of such things like
>>Gentoo distribution which can ease my administration in the future.
>>I will try it and I think, implement promptly.
>
>
> This is the thing: a little initial overhead, maybe 2-3 days to get a
> cranking solid base installation, and you're away. One of the tricks you
> could consider is if you have a farm of servers with near-enough identical
> hardware - then you can set up one base gentoo installation, clone it
> across the lot of them

As my target is to install two servers and I have got 2 identical
machines for each implementation (4 in total), I think I could arrange
it like mirror installs. Testing on one, launching on the second one
(or the first :-)).
Thank you once again !!

--
Regards
Michal Leder

The problem with troubeshooting is that
sometimes the trouble shoots back