any way to confirm break-in?

On 30 Apr 2007, in the Usenet newsgroup comp.os.linux.security, in article
<1177930718.337989.300510@n76g2000hsh.googlegroups .com>, yosato_uk wrote:

>The inevitable conclusion seems to be it is not so straightforward
>even just to ascertain a break-in, if the attacker's as clever as is
>expected, though there is strong circumstantial evidence...

One technique with a reasonable chance of success is to compare the
file system with a known clean version. The original version of this
type of test is 'tripwire' currently unsupported - and the derivatives
such as 'aide'. Briefly, you run this application on a clean box to get
checksums and hashes of EVERYTHING. Thereafter, you take the file system
off line, and run the same checksums and hashes using a "trusted" copy
of an operating system, and compare these with the original data. Is
anything different? The two problems with this is that you have to
create the reference data from a known clean installation, AND keep it
up to date with any intentional changes (such as security updates),
and that you should run the tests with a known secure O/S (which
might be a floppy that also has the needed tools) rather than the
"normal" O/S (which may have been compromised).

>And I've got the feeling that ssh is *not* the route through which the
>attacker gained access, if they indeed did --- my passwds are much
>stronger than the names tried in the failed attempts,

What else is running on the system? A web server, especially one with
dynamic content/scripting is FAR more likely. Is everything up to date?

>and I'm sure that they would have remove *all* the log files.

It's terribly easy to edit logs - and a common skript-kiddy technique.

>So... exluding pure coincidence, my guess is the same attacker that
>tried the ssh route found some other route.

Or someone else did - there's more than a few skript kiddiez out there.

>Now I'm not so much interested in hunting for the culprit as
>preventing a further problem. I will try and enhance the security the
>various ways suggested,

Good

>but the remaining worry is that there might be some malware or worm
>that is left on the system. If the usual tools may fail to detect them,
>is there any better way?

Depends. My usual trick is to use the package manager to check the
system ('/bin/rpm -V' for an rpm based system, 'debsums -s' for a
Debian based system --> even though the debsums man-page depreciates
this mode). First, check the tool by substituting some other file for
a binary that the intruder would normally alter:

/bin/mv /bin/ps /bin/ps.original
/bin/mv /bin/less /bin/ps
rpm -Vf /bin/ps or debsums -s procps
/bin/mv /bin/ps.original /bin/ps

DON'T NEGLECT THAT LAST STEP!!! Did your package manager freak out
when it discovered that /bin/ps doesn't look like /bin/ps _ought_ to
look like? If no, you're toast. If yes, then move on to the next step
in the Jedi Mind Games.

Now, if you _had_ followed the last paragraph in the CAVEATS section of
the 'debsums' man page

If you are looking for an integrity checker that can run from safe
media, do integrity checks on checksum databases and can be easily
configured to run periodically to warn the admin of changes see other
tools such as: aide, integrit, samhain, or tripwire.

you might have other options. Isn't hindsight wonderful? ;-)

>If there's no way to be absolutely sure, I'd in fact clean-reinstall
>the system altogether and recover the backed up data.

That is the most reliable solution. Ideally, the backed up data was
also clean. I make backups when _I_ change things on the server, not
as an automatic function. Thus, my backups should reflect that last
known "good" situation (I hope). ;-)

>But I presume there still would remain some risk, if some malware's
>mixed into the data directory...

Your "clean install" should reduce a lot of this risk - unless this is
some scripting/database/what-ever that might be executable as part of
(for example) your web page. You also have the issue that you haven't
corrected what-ever hole was exploited.

>so probably a rather naive question again: any (fast enough) way to
>transfer data during which you can verify safety?

The integrity checkers like aide and so-on can do this, BUT they need
a "clean" starting point. By the way, now you may know why our publicly
visible systems all run from "read-only" media, and have only the
minimum "stuff" installed to do their job. No bells, no whistles, no
frilly knickers.

Aren't "learning experiences" wonderful?

Old guy

Thank you for the very interesting post!

On Mon, 30 Apr 2007 19:51:27 -0500
ibuprofin@painkiller.example.tld (Moe Trin) wrote:
....
> So - what's the firewall solution? Do not _allow_ everything by
> default. I don't know about you, but I don't offer SSH to everyone
> in the world. For SSH at home, I accept connections from exactly 1545
> IP addresses (a /22. two /24s, and nine specific hosts) ONLY.

What is your opinion about port knocking and/or single packet authorization?
I have checked links available at portknocking.org and it seems the
majority of projects are in a stale condition (except fwknop and a
couple of projects at alpha stages).

Regards,
Mikhail

On 30 Apr, 16:47, Unruh <unruh-s...@physics.ubc.ca> wrote:
> yosato_uk <yosat...@gmail.com> writes:
> >Now I'm not so much interested in hunting for the culprit as
> >preventing a further problem. I will try and enhance the security the
> >various ways suggested, but the remaining worry is that there might be
> >some malware or worm that is left on the system. If the usual tools
> >may fail to detect them, is there any better way? If there's no way to
> >be absolutely sure, I'd in fact clean-reinstall the system altogether
> >and recover the backed up data. But I presume there still would remain
> >some risk, if some malware's mixed into the data directory... so
> >probably a rather naive question again: any (fast enough) way to
> >transfer data during which you can verify safety?
>
> As a minimum scan the backup for suid programs. There almost certainly
> should be none. At least examine any very carefully.
> find /backup -perm /6000 -ls

erk, IMHO not up to your usual quality of response, U

If one strongly suspects their machine has been compromised and / or
its got trashed, a format / re-install is called for. This is where
you start wishing you'd been running an IDS like tripwire or L5. By
all means mount the drive read-only on an already running, clean
system and have a poke around for setuid files / rootkit detectors,
but the quickest route to getting your system back to a trusted state
is probably going to be by vetting stuff you carry forward to the new
installation, not trying to find what's been tampered with in the old
one - there's just too much to check and too much risk of missing
something.

OP: you can see this flurry of log entries - but it does not
necessarilty follow that this corresponds with when your machine was
compromised - you should do a clean install and only restore your own
files from backup (not ones which were from the previous install) and
you should check them for interference using a rootkit detector.

C.

On 2007-04-29, yosato_uk <yosato16@gmail.com> wrote:

> This may be a very basic and vague question, but any help would be
> appreciated. I'd be happy to provide further info if needed.

There may be some left on the newsstand, if you hurry. But if not,
you might consider ordering this back issue of Linux-Pro magazine.

http://www.linux-magazine.com/issue/77

I don't often buy periodicals anymore, especially pricey one's like
L-P, but this issue had some very good info on back doors and
real-time IDS techniques (lsof). Maybe a friend has an issue.

nb

"C." <colin.mckinnon@gmail.com> writes:

>On 30 Apr, 16:47, Unruh <unruh-s...@physics.ubc.ca> wrote:
>> yosato_uk <yosat...@gmail.com> writes:
>> >Now I'm not so much interested in hunting for the culprit as
>> >preventing a further problem. I will try and enhance the security the
>> >various ways suggested, but the remaining worry is that there might be
>> >some malware or worm that is left on the system. If the usual tools
>> >may fail to detect them, is there any better way? If there's no way to
>> >be absolutely sure, I'd in fact clean-reinstall the system altogether
>> >and recover the backed up data. But I presume there still would remain
>> >some risk, if some malware's mixed into the data directory... so
>> >probably a rather naive question again: any (fast enough) way to
>> >transfer data during which you can verify safety?
>>
>> As a minimum scan the backup for suid programs. There almost certainly
>> should be none. At least examine any very carefully.
>> find /backup -perm /6000 -ls

>erk, IMHO not up to your usual quality of response, U

>If one strongly suspects their machine has been compromised and / or
>its got trashed, a format / re-install is called for. This is where
>you start wishing you'd been running an IDS like tripwire or L5. By

It would really really help if you read both the post and the response.
This comment was in response to the suggestion to reinstall and then put in
place the backup of /home. That backup itself should be scanned.

>all means mount the drive read-only on an already running, clean
>system and have a poke around for setuid files / rootkit detectors,

Sorry, you are of the impression that files can run themselves? Why do you
want to mount the drive on clean system ro?

My adivce would be.
a) reinstall the operating system.
b) scan all of the stuff that you did NOT reinstall for suid/sgid files.
All. /tmp, /var/* /home, /usr/local.
I once found a file /tmp/banana which was a root suid shell.


>but the quickest route to getting your system back to a trusted state
>is probably going to be by vetting stuff you carry forward to the new
>installation, not trying to find what's been tampered with in the old
>one - there's just too much to check and too much risk of missing
>something.

Precisely what I was saying-- vet the stuff you carry forward.


>OP: you can see this flurry of log entries - but it does not
>necessarilty follow that this corresponds with when your machine was
>compromised - you should do a clean install and only restore your own
>files from backup (not ones which were from the previous install) and
>you should check them for interference using a rootkit detector.

Well, first you should try to make sure that your machine actually was
complrimised. If you reinstall every time something suspicious appears in a
log file, you will never use your computer for anything worthwhile. It will
be a daily excercise.

If you have a rpm based system, first do
rpm -Va>/tmp/verify
and then look through those files for chnged files. If you find one that
you cannot explain, you are probably comprimised.

Then look for suid file. Look everywhere. Make sure that those file should
be suid files. If you find an suid root file in /dev, /tmp, etc, then you
have been comprimised.

Look at the log files for suspicious blanks.

>C.

On Tue, 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
<20070501074148.15276527.invalid_muxaul@lenta.ru >, Mikhail Zotov wrote:

>Thank you for the very interesting post!

You are welcome!

>ibuprofin@painkiller.example.tld (Moe Trin) wrote:
>...
>> So - what's the firewall solution? Do not _allow_ everything by
>> default. I don't know about you, but I don't offer SSH to everyone
>> in the world. For SSH at home, I accept connections from exactly 1545
>> IP addresses (a /22. two /24s, and nine specific hosts) ONLY.

>What is your opinion about port knocking and/or single packet authorization?

]] If I'm going on a trip and need to allow access from other addresses,
]] I can do so

Port knocking is another layer - and relatively simple to add to the
non-standard server port number. From my Sunday post to get some numbers
to use here:

[compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column
93285798 057134614 41126395029913 67274730
03984161308677 379491685 36299082976038 146914677
68815375230003 4144452 90087622526 34724849==
[compton ~]$

The knocking scheme I've used is crude - listen for packets on port
39841 (which is closed), and on the firewall seeing such packet, open
port 13086 (where the SSH server is hiding) for the IP address that
had hit the first port for one minute. Also set two "trip" ports just
above and just below the SSH port - if the knocking address hits those
ports, close 13086 under the premise that a port scanner hit the knock
port, then continued to scan. The trip ports will ensure that the port
scanner closes the SSH port before he gets to it unless the scanner is
hitting random numbers, and the Gods of Chance are frowning at me.

PORT KNOCKING IS NOT A SUBSTITUTE FOR PROPER AUTHENTICATION. For those
with a knee-jerk reaction to "hiding things" being security through
obscurity, shove it! Port knocking, like moving the server to an obscure
port number, or using firewall rules to harshly restrict access to a
minimum number of addresses, is an _addition_ to what ever security
functions (proper authentication, using 'non-dictionary/non-phone-book'
usernames, strong authentication keys, etc.) that you _should_ have
had in place before reading this. All you are trying to do is to make
it more difficult. Grabbing numbers out of the sky - assume there is a
single chance in a thousand of the bad guy guessing the right username
(you DO NOT use 'root' or 'admin' or your 'public_username'), Assume
one chance in a thousand or the back guy independently guessing your
non-dictionary password. Thus, a robot has one chance in a million
of guessing both. But this will do the robot little good if it must
also guess which of the sixty-thousand ports you've hidden the server
at - and port knocking merely makes it harder for the bad guy to stumble
across your hidden server. Yes, I am aware of packet sniffing, and
there are additional steps that can be taken to reduce that risk.

The one problem you want to avoid is making what ever scheme you use to
complicated. You _will_ screw up, and lock yourself out - and you won't
be the first person to do so. I encountered a situation several years
ago where the simple port-knock wouldn't work. I gave up after the third
try, but later investigation of the logs showed the knock was being
intercepted by a proxy server, while my SSH attempt was going straight
through - the proxy server had (naturally) a different IP address.

>I have checked links available at portknocking.org and it seems the
>majority of projects are in a stale condition (except fwknop and a
>couple of projects at alpha stages).

I've looked at a number of the projects (though not recently) and tend
to agree. The scheme used doesn't have to be complicated - my original
knocking "daemon" was a firewall rule to log to a special file any
connection attempts to port $FOO, and a cron script that looked for
that file - and on finding it, parsed the IP, set a firewall rule for
port $BAR from that IP, and set a timer to clear that rule a minute
later. The firewall 'ESTABLISHED" rule allowed the "conversation to
continue. Security does not have to be complicated, or cute. Simple
steps _can_ do a lot for you as long as you use some common sense. Oh,
and you _do_ remember to change that password (and username if you are
ultra-paranoid) on a regular basis, right? Just don't over do it.

Old guy

On 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
<1178020943.501703.59150@l77g2000hsb.googlegroups. com>, C. wrote:

>Unruh <unruh-s...@physics.ubc.ca> wrote:

>> As a minimum scan the backup for suid programs. There almost
>> certainly should be none. At least examine any very carefully.
>> find /backup -perm /6000 -ls

>erk, IMHO not up to your usual quality of response, U

Well, he did say it was the very minimum - but yes it's preferable not
to have to recover from the contaminated backups. Doing automated
nightly or weekly backups may not be the best scheme. Backing up
immediately after _you_ make some change is safer, and safer still
is having those originals created on an isolated box, with the
server running from 'read-only' copies of that source.

>If one strongly suspects their machine has been compromised and / or
>its got trashed, a format / re-install is called for. This is where
>you start wishing you'd been running an IDS like tripwire or L5.

Agreed

>By all means mount the drive read-only on an already running, clean
>system and have a poke around for setuid files / rootkit detectors,

though this is for educational purposes (how did they do it) only.

>but the quickest route to getting your system back to a trusted state
>is probably going to be by vetting stuff you carry forward to the new
>installation, not trying to find what's been tampered with in the old
>one - there's just too much to check and too much risk of missing
>something.

Talking about a public server here - a work station should not be
running public severs, and a server should not have users allowed
to log in - the data for the server should be developed/tested on
a separate box, whether it be web pages, files for a file server, or
what-ever. Getting the system back to the trusted state then only
involves a clean install of the operating system and applications,
and then copying the data from the development system. Running the
server application such that the data is owned by a specific
non-privileged user AND GROUP allows additional "does this smell right"
checks to be run.

Old guy

ibuprofin@painkiller.example.tld (Moe Trin) writes:

>On 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
><1178020943.501703.59150@l77g2000hsb.googlegroups .com>, C. wrote:

>>Unruh <unruh-s...@physics.ubc.ca> wrote:

>>> As a minimum scan the backup for suid programs. There almost
>>> certainly should be none. At least examine any very carefully.
>>> find /backup -perm /6000 -ls

>>erk, IMHO not up to your usual quality of response, U

>Well, he did say it was the very minimum - but yes it's preferable not
>to have to recover from the contaminated backups. Doing automated
>nightly or weekly backups may not be the best scheme. Backing up
>immediately after _you_ make some change is safer, and safer still
>is having those originals created on an isolated box, with the
>server running from 'read-only' copies of that source.

The breakin could have happened a month ago and was just utilized/screwed
up last night. If you are broken into, do not trust anything, including
your backups. That is why scanning your backups, no matter when made, is a
minimum.
Note that I am NOT advocating just doing this.

yosato_uk <yosato16@gmail.com> (07-04-30 03:58:38):

> The inevitable conclusion seems to be it is not so straightforward
> even just to ascertain a break-in, if the attacker's as clever as is
> expected, though there is strong circumstantial evidence...

As abstract as I like to be, the problem is that for a computer, there
is no distinction of `compromised' vs. `not compormised'. Everybody has
a different idea of where this distinction occurs. This is why there
can't be a perfect rootkit scanner or break-in detector.

In other words: To check for a break-in, you need to minimize the scope
of the effect of a possible break-in (like changed files, BIOS, system
time, etc.) to a certain area of your computer (like all executable
files on your hard-disk), learn how this area works, and check it
yourself. Because this is impractical and error-prone, you'll do better
just reinstalling your operating system on all possibly affected hosts,
backing up all important data.


> And I've got the feeling that ssh is *not* the route through which the
> attacker gained access, if they indeed did --- my passwds are much
> stronger than the names tried in the failed attempts, and I'm sure
> that they would have remove *all* the log files. So... exluding pure
> coincidence, my guess is the same attacker that tried the ssh route
> found some other route.

Perhaps your system was compromised long before that word-list attack
happened. Perhaps it has been compromised later. Probably it hasn't
been compormised at all. Sometimes system updates, which include OS
components, break things.


> Now I'm not so much interested in hunting for the culprit as
> preventing a further problem. I will try and enhance the security the
> various ways suggested, [...]

You will want to use proper authentication methods, and restrict,
restrict, restrict everything you can for people not supposed to access
your hosts. In the best case, a random attacker (i.e. script kiddie)
doesn't even know your network exists.


> [...] but the remaining worry is that there might be some malware or
> worm that is left on the system. If the usual tools may fail to detect
> them, is there any better way? If there's no way to be absolutely
> sure, I'd in fact clean-reinstall the system altogether and recover
> the backed up data.

Exactly. There is no way you can know, besides checking manually, which
is impractical. So it's best to do drop your current installation
completely.


> But I presume there still would remain some risk, if some malware's
> mixed into the data directory... so probably a rather naive question
> again: any (fast enough) way to transfer data during which you can
> verify safety?

No automated way. Firstly you need to separate functional data from
non-functional data. Shell scripts and often configuration files are
functional. A JPEG image generally isn't (although with a suitable
viewer and EXIF, it might).

Whether data is functional or not is not always as obvious. Take care,
and if uncertain, always assume functional data. For example, you
wouldn't consider your Quake save-games to be functional, but in fact
they are! Functional data is not necessarily executable.

For all functional files, you need to check manually whether they have
been manipulated. Depending on the number of functional files you have,
this is going to take a long time. After you've finished checking, make
sure you use secure versions of viewer programs.

Of course this is überparanoid and certainly extremely expensive and
time consuming, but it's the only way to make sure. Always assume that
the attacker has had compromised your system long before you noticed
there's something wrong.


Regards,
Ertugrul Söylemez.


--
From the fact that this CGI program has been written in Haskell, it
follows naturally that this CGI program is perfectly secure.

On Tue, 01 May 2007 14:58:31 -0500
ibuprofin@painkiller.example.tld (Moe Trin) wrote:

> On Tue, 1 May 2007, in the Usenet newsgroup comp.os.linux.security, in article
> <20070501074148.15276527.invalid_muxaul@lenta.ru >, Mikhail Zotov wrote:
....
> >What is your opinion about port knocking and/or single packet authorization?
>
....
> Port knocking is another layer - and relatively simple to add to the
> non-standard server port number. From my Sunday post to get some numbers
> to use here:
>
> [compton ~]$ head -2 /dev/random | mimencode | tr -d 'A-z/+' | column
....

Cute :-)

....
> The one problem you want to avoid is making what ever scheme you use to
> complicated. You _will_ screw up, and lock yourself out - and you won't
> be the first person to do so.

LOL :-) This happened one, this happened twice... How more times will
this happen?

> ... Security does not have to be complicated, or cute. Simple
> steps _can_ do a lot for you as long as you use some common sense. Oh,
> and you _do_ remember to change that password (and username if you are
> ultra-paranoid) on a regular basis, right? Just don't over do it.

Thank you! It's a big pleasure to study your posts!

Regards,
Mikhail