How rockus.at got hacked

"You what?," I thought, and tried again.

rockus@andoril:~$ su
Password:
su: Authentication failure
Sorry.
rockus@andoril:~$

Hmmm, something wasn't right with my systems.

Then I checked the login logs and discovered successful logins from very unususal geographic areas: Romania. At least unusual for my servers as I didn't have any users that frequent that area.

Not a funny feeling. But as soon as I realised, or feared, what had happened, I went out to my server rack and switched off the net connection to the outside world. Hard, without any notice to anyone. From that point on, I was offline and had bought myself some time to investigate.

Some months ago I read a very interesting report of how a sys admin of a commercial network traced back an intruder. I wasn't at a point where I could investigate on the where's and who's, though. And I was offline. So I were on my own.

The one main point I remember from that story mentioned above, apart from it being a good read, was that the intruder installed trojan commands that hide special files.

I had luck! The intruder wasn't experienced. As I later found out, it was a script that found it's way into my system through a weak password of one of my users.

Since it was only a script, and a not very intelligent one as well, I still had the login times in my logs and tried to find files on my system that had a suitable change date and time. And there were lots.

For a short time I thought of re-installing the system and setting it all up again - but I only had done that a few days earlier when moving servers around and switching hardware. I wasn't in the mood to lose all my setups and data and files and everything so I started to try to find out exactly which files where compromised.

To do that I had a closer look on the other servers in my net and discovered that there weren't any logins into these systems - or at least I didn't find any. But judging from the way the script worked (and I was sure it was a script, since commands where issued rapidly following each other, sometimes as many as 6 per second), I was reasonably sure that the script only tried to phone home and didn't do anything else.

I first copied over good binaries from the other system. 'ps' and 'ls' being the most important ones.

With them I browsed through the files and found two different change mechanisms at work:

First, files were changed between login and logoff of the suspicious users.

And, second, system files where owned by a user, but an unknown group. Really suspicious as well.

Also, there was a daemon running on my system which responded to a port in the upper 6000s. Unfortunately for it, my firewall blocked these ports.

First case of "Bah! I'm better than you!". That was a good feeling. The first one after switching off my net connection. From there on it was work, but not too unpleasant. I'm a night person anyway, so that really wasn't a problem as well. I only had to get up in the morning for a presentation at work, but that was of no concern to me at that point.

What I do remember, though, is that the key to success was having access to known good binaries for ls and ps. The way the script worked was to replace these (and others) to hide various daemons and tasks running in the system, as well as hiding them from a directory listing.

Being armed with tools that actually do display these tasks and files, it was a pure matter of browsing through directories replacing binaries, after having killed all suspicious tasks.

I also made sure to copy over a valid apt-get (and assorted libraries) to forcibly upgrade the system packages on my debian system. Having made a list of corrupted files before, I re-checked after the forced upgrade, or rather side-grade, that all of these files really did have the correct length, permissions and ownership set.