My Linux server got hacked last night -- please help!

It looks as though my Linux server (running RedHat Fedora Core 3) was
hacked last night.

I see the following files in my /lib directory (note modification
times, permissions and sizes)

?---rwS--T 2200 4249291143 4170711954 4253155062 Dec 20 1974
libc-2.3.3.so
?--x-wx--T 1467 4252107961 4180869466 84017700 Jan 6 1973
libnss_nis-2.3.3.so
?--xr-s-w- 809 4223534637 4167107119 99548634 Jun 9 1972 libblkid.so.1
?-wx--x--- 666 65210227 4197645536 114950169 May 10 1972
libnss_nis.so.2
?rw-rw-rwT 1088 4200988799 4227794193 3080127 Aug 30 1971
libdevmapper.a.1.00
?--SrwSrwT 228 43058577 4228381127 2593258783 May 20 1971
libcidn-2.3.3.so
?---rwxr-x 282 42925887 4284678677 4287692964 Apr 25 1970
libNoVersion-2.3.3.so
?r-srwsrwT 65486 4286578997 4270783980 17891147 Mar 20 1970
libdevmapper.a
?rwxrw-rwt 439 4223794553 4277798468 2687893457 Mar 12 1970
libdevmapper.so.1.00
?-wSr-s-wT 64569 102040035 17627963 15990883 Jan 1 1969
libblkid.so.1.0
?-wS--S--x 64693 42663644 62192531 4269276205 Jul 21 1968
libnss_hesiod-2.3.3.so
?rwSrwS-wT 64087 38338406 60292326 4270063399 Nov 3 1967
libnss1_dns-2.3.3.so
?-wS-wsr-T 64295 4286970048 112657123 52232677 Nov 2 1966
libdevmapper.so
?-wS-wSrwT 64115 32897306 144572815 4179361569 Aug 15 1966 libe2p.so.2
l-w-r----t 63784 112655096 150224719 38339193 Jul 7 1966
libSegFault.so

I can't chmod or chown these files, even as root.

The following is in my /var/log/secure from last night:

Nov 29 04:55:02 andromeda sshd[32300]: Invalid user admin from
::ffff:210.212.85.11
Nov 29 04:55:02 andromeda sshd[32300]: error: Could not get shadow
information
for NOUSER
Nov 29 04:55:02 andromeda sshd[32300]: Failed password for invalid
user admin
from ::ffff:210.212.85.11 port 58496 ssh2
Nov 29 04:55:09 andromeda sshd[32304]: Invalid user admin from
::ffff:210.212.85.11
Nov 29 04:55:09 andromeda sshd[32304]: error: Could not get shadow
information
for NOUSER
Nov 29 04:55:09 andromeda sshd[32304]: Failed password for invalid
user admin
from ::ffff:210.212.85.11 port 58599 ssh2
Nov 29 04:55:19 andromeda sshd[32306]: Invalid user user from
::ffff:210.212.85.11
Nov 29 04:55:19 andromeda sshd[32306]: error: Could not get shadow
information
for NOUSER
Nov 29 04:55:19 andromeda sshd[32306]: Failed password for invalid
user user
from ::ffff:210.212.85.11 port 58726 ssh2

I'd appreciate any advice on
1) How to cleanse my system
2) How to avoid this type of attack in future.

Right now I've powered off the server. I'll reboot using a RedHat
install CD in rescue mode. Does anyone know how to force RedHat to
reinstall all packages without repartitioning my hard drive?

Thanks,

S

sarah chang wrote:
....
>
> I'd appreciate any advice on
> 1) How to cleanse my system

You MUST reinstall completely.

> 2) How to avoid this type of attack in future.

You have given NO information... so no answer.
Let us know all about the configuration and what
it was running and that will help somewhat.

sarah chang wrote:
> It looks as though my Linux server (running RedHat Fedora Core 3) was
> hacked last night.
> [...]
> I can't chmod or chown these files, even as root.

The first thing I would do, after taking the system off any network, is to
thoroughly check for filesystem damage, e.g. with "shutdown -rF", the "F"
option forcing an fsck upon reboot.

If you are unable to use chmod or chown against the files, when you have
restarted with the rescue-CD (and you know that you are in fact executing
the commands from that immutable CD), then this strongly implies to me a
filesystem failure, rather than a "hack."

Sundial Services <info@sundialservices.com> writes:

]sarah chang wrote:
]> It looks as though my Linux server (running RedHat Fedora Core 3) was
]> hacked last night.
]> [...]
]> I can't chmod or chown these files, even as root.

]The first thing I would do, after taking the system off any network, is to
]thoroughly check for filesystem damage, e.g. with "shutdown -rF", the "F"
]option forcing an fsck upon reboot.

]If you are unable to use chmod or chown against the files, when you have
]restarted with the rescue-CD (and you know that you are in fact executing
]the commands from that immutable CD), then this strongly implies to me a
]filesystem failure, rather than a "hack."

Could be but first do lsattr filename and see if the i bit is set
man lsattr
man chattr


Then do
rpm -Vf /complete/name/of/file/with/path

sarah chang <sarahd00d@yahoo.co.uk> wrote:
>It looks as though my Linux server (running RedHat Fedora Core 3) was
>hacked last night.

Unfortunate.

>The following is in my /var/log/secure from last night:
>Nov 29 04:55:02 andromeda sshd[32300]: Invalid user admin from
>::ffff:210.212.85.11

No way to tell if this was pre- or post- breakin, or just part of a script to
gain access, which failed (but then some other part succeeded). Look at all
logs for suspicious things, ESPECIALLY for programs or services you haven't
updated recently.

>I'd appreciate any advice on
>1) How to cleanse my system

Format and reinstall from clean media. It's the only way to be sure.

>2) How to avoid this type of attack in future.

Keep software up-to-date, use a hardware firewall, turn off services you don't
need, make sure passwords are resistant to guessing.

>Right now I've powered off the server. I'll reboot using a RedHat
>install CD in rescue mode. Does anyone know how to force RedHat to
>reinstall all packages without repartitioning my hard drive?

You want to reformat the drive. Reinstalling all packages will not remove
backdoors which do not conflict with any package. Use the rescue CD to get
any data files saved, then nuke it.
--
Mark Rafn dagon@dagon.net <http://www.dagon.net/>

Bill Unruh spilled the following:

> Sundial Services <info@sundialservices.com> writes:
>
> Could be but first do lsattr filename and see if the i bit is set
> man lsattr
> man chattr
>

Yup, but if the machine has been compromised then it's time to format those
hard disks and reinstall from a known good backup / scratch.

>
> Then do
> rpm -Vf /complete/name/of/file/with/path

Unless you have an offline backup of your rpm database it's unwise to rely
on it as reference for IDS.

C.

Colin McKinnon wrote:
> Bill Unruh spilled the following:
>> Sundial Services <info@sundialservices.com> writes:
>> Could be but first do lsattr filename and see if the i bit is set
>> man lsattr
>> man chattr
> Yup, but if the machine has been compromised then it's time to format
> those hard disks and reinstall from a known good backup / scratch.

The operative word here is "if." There is a nonzero chance that the disk
drive has thrown an error, and part of the rather-complicated filesystem
data structures have been damaged, perhaps causing files to be immutable.
Or maybe they've always /been/ immutable, e.g. as a defensive measure, but
we had no prior motive to discover this.

>> Then do
>> rpm -Vf /complete/name/of/file/with/path
> Unless you have an offline backup of your rpm database it's unwise to rely
> on it as reference for IDS.

The first thing to establish, I think, is "what" happened. The high
capacity disk drives we use today are not /nearly/ as reliable as their
less-capacious predecessors. Many peculiar logins are attempted, for users
like "news," at odd times of the morning for perfectly valid, scheduled
reasons. When something happens that you don't expect or understand it's
easy to jump to the conclusion that "I've been hacked!" That may or may
not prove to be so.

dagon@dagon.net (Mark Rafn) writes:

]sarah chang <sarahd00d@yahoo.co.uk> wrote:
]>It looks as though my Linux server (running RedHat Fedora Core 3) was
]>hacked last night.

]Unfortunate.

]>The following is in my /var/log/secure from last night:
]>Nov 29 04:55:02 andromeda sshd[32300]: Invalid user admin from
]>::ffff:210.212.85.11

]No way to tell if this was pre- or post- breakin, or just part of a script to
]gain access, which failed (but then some other part succeeded). Look at all
]logs for suspicious things, ESPECIALLY for programs or services you haven't
]updated recently.

]>I'd appreciate any advice on
]>1) How to cleanse my system

]Format and reinstall from clean media. It's the only way to be sure.

]>2) How to avoid this type of attack in future.

]Keep software up-to-date, use a hardware firewall, turn off services you don't
]need, make sure passwords are resistant to guessing.

]>Right now I've powered off the server. I'll reboot using a RedHat
]>install CD in rescue mode. Does anyone know how to force RedHat to
]>reinstall all packages without repartitioning my hard drive?

]You want to reformat the drive. Reinstalling all packages will not remove
]backdoors which do not conflict with any package. Use the rescue CD to get
]any data files saved, then nuke it.

No No. You have to burn the whole computer and any CDs DVDs that were in
the house at the time. Even if it was only a Cinderella DVD, you never
know. No sense in not being careful.

Buy a new computer and start again.

A backdoor is only a backdoor if it can be opened from outside.

sarahd00d@yahoo.co.uk (sarah chang) wrote in
news:24d1fc75.0411291116.57cfad5b@posting.google.c om:

> I can't chmod or chown these files, even as root.

Their scripts use chattr to lock the files from actions.
If I remember right its
chattr -i (file)
to change it back.

> I'd appreciate any advice on
> 1) How to cleanse my system
> 2) How to avoid this type of attack in future.
>
> Right now I've powered off the server. I'll reboot using a RedHat
> install CD in rescue mode. Does anyone know how to force RedHat to
> reinstall all packages without repartitioning my hard drive?

SSH attacks are getting common. I never get anything against my telnet or
ftp anymore.

I could tell you how to clean your system but are you SURE you want to?
Its rarely recommended and the results are always questionable.


Gandalf Parker
-- email me
Gandalf@Community.Internet
(without the Inter of course)

In article <24d1fc75.0411291116.57cfad5b@posting.google.com >,
sarahd00d@yahoo.co.uk says...
> The following is in my /var/log/secure from last night:
>
> Nov 29 04:55:02 andromeda sshd[32300]: Invalid user admin from
> ::ffff:210.212.85.11
> Nov 29 04:55:02 andromeda sshd[32300]: error: Could not get shadow
> information
> for NOUSER
> Nov 29 04:55:02 andromeda sshd[32300]: Failed password for invalid
> user admin
> from ::ffff:210.212.85.11 port 58496 ssh2
> Nov 29 04:55:09 andromeda sshd[32304]: Invalid user admin from
> ::ffff:210.212.85.11
> Nov 29 04:55:09 andromeda sshd[32304]: error: Could not get shadow


I don't see anything that indicates a break-in. I do see things that
might indicate a filesystem problem, though.

If you want ti reinstall, you have to format and reinstall everything.
If your system trully is compromised, you have no way of knowing what is
and isn't safe.