MD5 checksum changed

I've got a box running Mandrake 9. It e-mailed me a warning
yesterday that a file had changed (this is from Mandrake's
msec security check).

This box is behind a hardware firewall, and the only
open ports are smtp, pop3, http (and maybe https, I can't
remember whether I closed that again after playing with it).

It's running the Apache it came with, postfix for smtp
and qpopper for pop3. It doesn't do anything else: it's
just a web and mail server. There are no other machines
on that internal network (behind that firewall).

AFAIK, I haven't missed any security patches, I read
comp.security.announce for updates, but it's possible I
missed something.

Anyway, yesterday msec e-mailed me to say:
Security Warning: These files belonging to packages have changed of
status
on the system :
- Newly modified : /usr/lib/postfix/pickup

I couldn't get to the machine until today, but today I did
$ rpm -V postfix
and, if I'm interpreting what it says correctly, the file
size hasn't changed, but the MD5 checksum has changed.

I can't find any *other* evidence of possible intrusion,
there is nothing strange in any of the logs (lots of the
usual probes, lots of spam being rejected normally etc.)

That doesn't mean there isn't any, but how likely is it
that the file has got corrupt? The machine doesn't have
a high real workload, but the disc is kept busy logging
all that spam being rejected and it's been running
continuously for some months now (the machine is about
four years old, but has only been pressed into service
as the mail/web server for a few months).

If it was cracked, and they hid their traces so well,
how come they missed just the one file? And why *that*
file? There's security, and there's paranoia.

What I'd like to do is simply replace that single file
from the original postfix rpm package, to see if it
changes again, but can't see how in the rpm man pages
(I'd have expected it to be easy, but haven't used rpm
for anything complex).

What do you all think? Wipe the disc, and reinstall
everything? Upgrade to the latest version of postfix?

Oh, and it's probably time to think about a new version
of Linux anyway. I chose Mandrake because I got it
as cover discs from a Linux mag, and I knew that it
would either be very easy to install or impossible
because I've tried Mandrake 7 and 8 in the past, but
for this particular use, it may be overkill. Suggestions
for a minimalist small-server Linux welcome. It doesn't
particularly need an X Server or a Desktop, but free
would be nice, and easy to set up essential :-).

Sorry this is so long.

Jonathan

--
Use jlc1 at address, not spam.

spam@softluck.plus.com (Jonathan L Cunningham) writes:

]I've got a box running Mandrake 9. It e-mailed me a warning
]yesterday that a file had changed (this is from Mandrake's
]msec security check).


]Anyway, yesterday msec e-mailed me to say:
] Security Warning: These files belonging to packages have changed of
]status
]on the system :
]- Newly modified : /usr/lib/postfix/pickup

]I couldn't get to the machine until today, but today I did
] $ rpm -V postfix
]and, if I'm interpreting what it says correctly, the file
]size hasn't changed, but the MD5 checksum has changed.

That means the contents have changed. Did you recently install a new
version of postfix?


]I can't find any *other* evidence of possible intrusion,
]there is nothing strange in any of the logs (lots of the
]usual probes, lots of spam being rejected normally etc.)

]That doesn't mean there isn't any, but how likely is it
]that the file has got corrupt? The machine doesn't have
]a high real workload, but the disc is kept busy logging
]all that spam being rejected and it's been running
]continuously for some months now (the machine is about
]four years old, but has only been pressed into service
]as the mail/web server for a few months).

]If it was cracked, and they hid their traces so well,
]how come they missed just the one file? And why *that*
]file? There's security, and there's paranoia.

]What I'd like to do is simply replace that single file
]from the original postfix rpm package, to see if it
]changes again, but can't see how in the rpm man pages
](I'd have expected it to be easy, but haven't used rpm
]for anything complex).

Replace the package withe the
--force
option.



]What do you all think? Wipe the disc, and reinstall
]everything? Upgrade to the latest version of postfix?

rpm -Va|grep '^..5'>/tmp/verify
to see if there is something else suspicious.



]Oh, and it's probably time to think about a new version
]of Linux anyway. I chose Mandrake because I got it
]as cover discs from a Linux mag, and I knew that it
]would either be very easy to install or impossible
]because I've tried Mandrake 7 and 8 in the past, but
]for this particular use, it may be overkill. Suggestions
]for a minimalist small-server Linux welcome. It doesn't
]particularly need an X Server or a Desktop, but free
]would be nice, and easy to set up essential :-).

]Sorry this is so long.

]Jonathan

]--
] Use jlc1 at address, not spam.

On 17 Aug 2004 23:40:42 GMT, unruh@string.physics.ubc.ca (Bill Unruh)
wrote:

>spam@softluck.plus.com (Jonathan L Cunningham) writes:

>]on the system :
>]- Newly modified : /usr/lib/postfix/pickup
>
>]I couldn't get to the machine until today, but today I did
>] $ rpm -V postfix
>]and, if I'm interpreting what it says correctly, the file
>]size hasn't changed, but the MD5 checksum has changed.
>
>That means the contents have changed. Did you recently install a new
>version of postfix?

Thanx for your suggestions.

No, just changed the "aliases" file (so the rpm -V postfix
told me about that, too) but I haven't installed a new
version for a while.

>]What do you all think? Wipe the disc, and reinstall
>]everything? Upgrade to the latest version of postfix?
>
>rpm -Va|grep '^..5'>/tmp/verify
>to see if there is something else suspicious.

Ok, I'll try that. But I think Mandrake's msec checks all
the packages -- that's how it told me about the postfix file.

The documentation also says it does a check for rootkits, but
I guess that's like anti-virus software, you need the latest
version?

(Oh, I also used netstat -a, to see if anything obviously
wrong there. Since I'm not using a software firewall, I guess
it would be much harder for any cracker to hide *all*
evidence, because they'd have to be talking through existing
services without breaking them.)

Jonathan

--
Use jlc1 at address, not spam.

Jonathan L Cunningham wrote:
> (Oh, I also used netstat -a, to see if anything obviously
> wrong there. Since I'm not using a software firewall, I guess
> it would be much harder for any cracker to hide *all*
> evidence, because they'd have to be talking through existing
> services without breaking them.)

This is not true. The function of a rootkit is to hide these things. If
there is a rootkit it may have replaced the netstat binary with one that
hides the process and socket the attacker is using.

There are also kernel rootkits that do the same thing at a lower level.

If you assume your system has been compromised then you can trust no
binary on it - you have to boot from trusted media (boot cd, rescue cd
and such) and examine the status from there.

hth,
Nils

Nils Juergens <ju@isf.rwth-aachen.de> writes:

> Jonathan L Cunningham wrote:
>> (Oh, I also used netstat -a, to see if anything obviously wrong there.
>> Since I'm not using a software firewall, I guess it would be much harder
>> for any cracker to hide *all* evidence, because they'd have to be
>> talking through existing services without breaking them.)

*Ahem*. If you had iptables on the machine, you could be restricting access
to the services running.

> This is not true. The function of a rootkit is to hide these things. If
> there is a rootkit it may have replaced the netstat binary with one that
> hides the process and socket the attacker is using.

Agreed.

While I have been known to spot a rootkit's presence by netstat(1) behaving
differently (complained about no `-p' option despite being on a RH6.2 box),
I would prefer to rely on an external scan of all ports from another box,
e.g. with nmap.
Note that this will generally not find a port-knocker daemon if that's how
the rootkit works.

~Tim
--
There can be only one! |piglet@stirfried.vegetable.org.uk
|http://pig.sty.nu/Pictures/

unruh@string.physics.ubc.ca (Bill Unruh) writes:

[snip]
> ]What I'd like to do is simply replace that single file from the original
> ]postfix rpm package, to see if it changes again, but can't see how in
> ]the rpm man pages (I'd have expected it to be easy, but haven't used rpm
> ]for anything complex).
>
> Replace the package withe the --force option.

No no no no NO!!!!

Destroying evidence like this is sheer lunacy. Investigate until you find
the answer.

Here's two more ideas to consider:
a) hardware cockup leading to filesystem corruption manifesting itself as a
screwup in that binary - run a fsck on the partition in quesiton in
single-user mode;

b) Are you running `prelink'? See
<http://www.google.com/groups?hl=en&lr=&ie=UTF-8&threadm=86k6yjoyfr.fsf%40potato.vegetable.org.uk &rnum=1&prev=/groups%3Fq%3D%2B%2522Paul%2BBlack%2522%2B-moonlight%2Bgroup:uk.comp.os.linux%2Bauthor:Haynes %26hl%3Den%26lr%3D%26ie%3DUTF-8%26scoring%3Dd%26selm%3D86k6yjoyfr.fsf%2540potato .vegetable.org.uk%26rnum%3D1>

~Tim
--
In a big country dreams stay with you |piglet@stirfried.vegetable.org.uk
Like a lover's voice fires the mountainside |http://pig.sty.nu/about.html
Stay alive |

On Thu, 26 Aug 2004 10:14:38 +0100, Tim Haynes
<usenet-20040826@stirfried.vegetable.org.uk> wrote:

>unruh@string.physics.ubc.ca (Bill Unruh) writes:
>
>[snip]
>> ]What I'd like to do is simply replace that single file from the original
>> ]postfix rpm package, to see if it changes again, but can't see how in
>> ]the rpm man pages (I'd have expected it to be easy, but haven't used rpm
>> ]for anything complex).
>>
>> Replace the package withe the --force option.
>
>No no no no NO!!!!
>
>Destroying evidence like this is sheer lunacy. Investigate until you find
>the answer.

In an ideal world, I could replace that machine with another, copy all
the binaries off it, reinstall everything again from the original
installation disks, copy all those off somewhere else, then do a diff
on all the files.

But what would that tell me that would justify my time? (That's a
serious question: I'd *like* to know what the cause was, but if, say,
I could just install the latest version of RedHat instead of using
a (few months old) version of Mandrake, mere curiosity is not
sufficient justification.)

Meanwhile, we still need to send and receive e-mail, and it would be
nice if the web-site remained up.

>Here's two more ideas to consider:
>a) hardware cockup leading to filesystem corruption manifesting itself as a
>screwup in that binary - run a fsck on the partition in quesiton in
>single-user mode;

Yes, that seems like a good idea: and it won't take so long I need to
install a "spare" machine while I'm doing it.

>b) Are you running `prelink'? See
><http://www.google.com/groups?hl=en&lr=&ie=UTF-8&threadm=86k6yjoyfr.fsf%40potato.vegetable.org.uk &rnum=1&prev=/groups%3Fq%3D%2B%2522Paul%2BBlack%2522%2B-moonlight%2Bgroup:uk.comp.os.linux%2Bauthor:Haynes %26hl%3Den%26lr%3D%26ie%3DUTF-8%26scoring%3Dd%26selm%3D86k6yjoyfr.fsf%2540potato .vegetable.org.uk%26rnum%3D1>

Sorry, I followed this link and I couldn't make see what you are
referring to. I see in your other reply in this thread that you
mentioned iptables there, too. I'm not running iptables on this
machine, but have been relying on the separate hardware firewall.

I assume there is some additional benefit I would get? But you'd
need to explain v-e-r-y s-l-o-w-l-y in words of one syllable what
I'd need to do. We're not big enough to hire a full-time security
consultant, so I have to try and fit this stuff into all my other
work. (Anyone who explains that therefore we deserve to go out
of business will be politely ignored.)

Jonathan

--
Use jlc1 at address, not spam.

On Thu, 26 Aug 2004 10:05:12 +0100, Tim Haynes
<usenet-20040826@stirfried.vegetable.org.uk> wrote:

>Nils Juergens <ju@isf.rwth-aachen.de> writes:
>
>> Jonathan L Cunningham wrote:
>>> (Oh, I also used netstat -a, to see if anything obviously wrong there.
>>> Since I'm not using a software firewall, I guess it would be much harder
>>> for any cracker to hide *all* evidence, because they'd have to be
>>> talking through existing services without breaking them.)
>
>*Ahem*. If you had iptables on the machine, you could be restricting access
>to the services running.

Could you expand, please? You seem to be suggesting that there is some
additional benefit in running iptables as well as the hardware
firewall. I can sort-of imagine that might be the case, but not in
enough detail to guess what specifically you are suggesting.

At present, for historic reasons, we have two subnetworks connected
to a fairly dumb ADSL modem. Each of these has its own hardware
firewall. There is only the one machine in one of them: it accepts
connections on SMTP, POP3 and HTTP ports. Apart from maybe restricting
access to the POP3, I can't see what other restrictions to impose:
we want anybody to access our website (if they're interested) and
similarly, we need to be able to accept genuine e-mails from anyone.

(I recently had to turn off some of the spam-filtering checks --
the reverse DNS lookup -- because some guy in a big, well-known
company phoned to complain we were bouncing his mail. I had the
same problem once with a much smaller company too -- but their IT
guys fixed it instead of getting annoyed.)

Jonathan

--
Use jlc1 at address, not spam.

spam@softluck.plus.com (Jonathan L Cunningham) writes:

>>> Replace the package withe the --force option.
>>
>>No no no no NO!!!!
>>
>>Destroying evidence like this is sheer lunacy. Investigate until you find
>>the answer.
>
> In an ideal world, I could replace that machine with another, copy all
> the binaries off it, reinstall everything again from the original
> installation disks, copy all those off somewhere else, then do a diff on
> all the files.
>
> But what would that tell me that would justify my time? (That's a serious
> question: I'd *like* to know what the cause was, but if, say, I could
> just install the latest version of RedHat instead of using a (few months
> old) version of Mandrake, mere curiosity is not sufficient
> justification.)

It would tell you how secure the box was - a clean installation off new
trusted media (CDs) with audited data only, uptodate, firewalled, only
running what it needs, with IDS and nIDS and regular log-watching - this is
well-defined.
Something that's been running a while and you've lost the certainty of
integrity ain't the same. And, more to the point, if you go around
replacing bits of the machine willy-nilly without checking for or tackling
underlying (potential) problems, that certainty is *never* going to come
back.

If you find the box to be clean, all to the good; if you find it's been
cracked, you might have a clue for things to look out for in the future
(firewalling, minimalism, whatever config-changes enhance the box's
security to avoid the class of problem that lead to a crack). Them's the
benefits.

> Meanwhile, we still need to send and receive e-mail, and it would be
> nice if the web-site remained up.

Sure. The logistics are up to you - it may be time to gain certainty with a
clean installation and do it properly; it may be there's a simple
explanation (prelink or HD corruption), if you're lucky. Either way you
should resolve the changed binary, find *why* it happened or otherwise
guarantee that the machine is clean.

>>Here's two more ideas to consider: a) hardware cockup leading to
>>filesystem corruption manifesting itself as a screwup in that binary -
>>run a fsck on the partition in quesiton in single-user mode;
>
> Yes, that seems like a good idea: and it won't take so long I need to
> install a "spare" machine while I'm doing it.

Hope so :)

>>b) Are you running `prelink'? See
>><http://www.google.com/groups?hl=en&lr=&ie=UTF-8&threadm=86k6yjoyfr.fsf%40potato.vegetable.org.uk &rnum=1&prev=/groups%3Fq%3D%2B%2522Paul%2BBlack%2522%2B-moonlight%2Bgroup:uk.comp.os.linux%2Bauthor:Haynes %26hl%3Den%26lr%3D%26ie%3DUTF-8%26scoring%3Dd%26selm%3D86k6yjoyfr.fsf%2540potato .vegetable.org.uk%26rnum%3D1>
>
> Sorry, I followed this link and I couldn't make see what you are
> referring to.

Ah, it was an interesting thread on uk.comp.os.linux a while ago; Paul had
a box where tripwire started saying a binary had changed, transpired he was
running this prelink thing (<http://freshmeat.net/projects/prelink/>) which
modified it...

> I see in your other reply in this thread that you mentioned iptables
> there, too. I'm not running iptables on this machine, but have been
> relying on the separate hardware firewall.

A simple iptables firewall is so easy to stick on a box, I'd say use one
network-wide thing *and* per-machine.

> I assume there is some additional benefit I would get? But you'd need to
> explain v-e-r-y s-l-o-w-l-y in words of one syllable what I'd need to do.

Firewalling a machine individually? My method:

* Ascertain how the existing firewall system works - whether it uses a raw
script or an iptables-save output. Look in /etc/init.d/ and trace it back
from there, what config files the appropriate service script references.

* Grab my firewall script -
<http://spodzone.org.uk/packages/secure/iptables.sh> - and edit it. There
are sections for handling return packets for things we've asked for, and
for services we provide, and then the rest is dropped. (If you want to run
ssh, leave that line in; if you run mail and web-servers, clone twice and
replace 22 with 25 and 80, etc.)

* Run it, check that it works - you should get different output in `iptables
-nL' that looks vaguely like the rules, packets to various ports should be
permitted or dropped, from various places. Test a few.

* If appropriate, maybe try
iptables-save > /var/lib/iptables/rules-save
(on Gentoo, at least). Either that or make your version of my script run on
startup by fair meansa or foul.

That's, erm, it. :)

> We're not big enough to hire a full-time security consultant, so I have
> to try and fit this stuff into all my other work. (Anyone who explains
> that therefore we deserve to go out of business will be politely
> ignored.)

You do need to dedicate some time or resource to it - read-up on linux
security (<http://www.linuxsecurity.com/> being a possible start) and
firewalling (<http://www.netfilter.org/> also). If you don't have the time,
hire a temporary consultant.
Try to persuade your boss that if you don't invest the time now in a nice
solid lump (day or two, maybe a new machine) and get a regular routine
(apply package updates and read logfiles for the first 10mins of every
day), then the future upstream costs *when* a crack really happens and have
have to do damage assessment and limitation and repairs will take over a
week of your time, being caught with your knickers down with no spare
machine having to do a reinstallation while your web & mail servers are
offline as you sweat your guts up.

~Tim
--
16:58:06 up 10 days, 32 min, 4 users, load average: 0.05, 0.04, 0.06
piglet@stirfried.vegetable.org.uk |The light of the world keeps shining,
http://spodzone.org.uk/cesspit/ |Bright in the primal glow

spam@softluck.plus.com (Jonathan L Cunningham) writes:

> On Thu, 26 Aug 2004 10:05:12 +0100, Tim Haynes
> <usenet-20040826@stirfried.vegetable.org.uk> wrote:
>
>>*Ahem*. If you had iptables on the machine, you could be restricting
>>access to the services running.
>
> Could you expand, please? You seem to be suggesting that there is some
> additional benefit in running iptables as well as the hardware firewall.
> I can sort-of imagine that might be the case, but not in enough detail to
> guess what specifically you are suggesting.

Five things:

a) what happens when there's a bug in your hardware firewall, or it flakes
out, or your rules aren't well-enough designed, and somehow it lets packets
through?

b) the internal guy; since you say you have two networks, maybe there are
combinations of ports and people who shouldn't be talking to each other.

c) if you had more machines, you could have different policies for them
each.

d) Analysis: what things did the bad-ass do prior to that attempt to send
spam? There are several analysis tools for iptables logs in syslog.

e) dynamic firewalling; I don't personally believe in this, but if you
drink enough coffee maybe you'll like the idea of putting a trap in the
website such that if a bot follows a polluted link, their IP# is added to a
bad-list for mail purposes... this is one rather crude way to reduce spam,
depending whether you like it / think it'll solve much of the problem /
won't lend itself to auto-DoS attacks, etc.

Add those up, maybe it comes out as `individual firewall is a good idea'. :)

> At present, for historic reasons, we have two subnetworks connected to a
> fairly dumb ADSL modem. Each of these has its own hardware firewall.
> There is only the one machine in one of them: it accepts connections on
> SMTP, POP3 and HTTP ports. Apart from maybe restricting access to the
> POP3, I can't see what other restrictions to impose: we want anybody to
> access our website (if they're interested) and similarly, we need to be
> able to accept genuine e-mails from anyone.

Well, I'd consider ssl-wrapping pop3. Yes, otherwise, I agree that
webservers and mail-servers should accept mail. These are things I'd
recommend opening wide in iptables.sh; you may feel similarly about my
suggested "10/8 on eth0 for the LAN" line, with subtle variations, or you
may not want to trust everyone so widely.

> (I recently had to turn off some of the spam-filtering checks -- the
> reverse DNS lookup -- because some guy in a big, well-known company
> phoned to complain we were bouncing his mail. I had the same problem once
> with a much smaller company too -- but their IT guys fixed it instead of
> getting annoyed.)

Unfortunately yes, messed-up reverse-DNS is rife. It's not one of my checks
either; we have spamassassin with bayes, razor2 and pyzor engines, that's
quite enough for me atm. :)

~Tim
--
Can you tell me how to get, |piglet@stirfried.vegetable.org.uk
How to get to Sesame Street? |http://www.photoboxgallery.com/timhaynes