I've been thinking recently (again) about how to securely connect to a machine you have to administer.

For users, this isn't a major problem. You can have random passwords for each host you need to connect to as long as you set up ssh keys with passphrases and alternatively, ssh-agent to deal with the annoying parts of actually connecting to the machine. As long as you have "trusted" hosts which house the private ssh keys, you're good.

For admins, it's different.

I've been slowing gearing up for a major security kick as of late. So much bad, and not enough good. The SSH Trust Web idea is just part of it. Later today, probably after I get some sleep, I'll write a "secure" server policy, which details mount points, kernel security settings (grsecurity, etc) and the like.

Last week I scheduled downtime for all the production servers at work, as they all need reboots for kernel upgrades.

If I get the security policy written in time for a cursory approval from some of the more security conscious people I know, I'll reinstall the LAN firewall following it. I already know it's going to be somewhat of a pain, as keeping all suid binaries on their own partition tends to be a minor annoyance. However, like all things, it's easily scripted around.

A few months ago I played around with grsecurity, but at the time didn't care enough to consider implementing it on real machines. I guess I care now, and recompiled eos.int.walnutfactory.org's kernel, enabling just about every option that looked sane. I'm curious to see how usable the machine is, for day-to-day use, but none of the PWF kids currently use the machine for anything (as its still new to the Factory).

There are a few things I need to dig into with regards to grsecurity, the most interesting being the "learning mode" for ACLs.

Speaking of which, the most time-intensive aspect of enable grsecurity is going to be writing a sane ACL policy, assuming I don't let it figure it out on its own, and then actually turn ACLs on. In one respect, it's good that the majority of my machines are Debian GNU/Linux. Of course, generally speaking, completely homogenous networks are not the love, but it sure does making it easy when rolling out new technologies scripting for system administration.

Considering the amount of documentation I'm going to have to produce in a short amount of time, I really should consider starting to write it all in LaTeX.

After a few minutes of looking around, I've discovered quite a bit of useful documentation, most of it describing the use of shiny things which Just Work.

I've been using ssh forever, and yet never knew about the ForwardAgent feature... because I've never really thought to use ssh-agent. Well, it's nice to think "Hm, it'd be useful if..." and have it already done.

This series of articles written by the Gentoo Linux Chief Architect describe a few nifty tricks with ssh and ssh-agent, using the Gentoo-supplied keychain program.

This howto has a few useful aliases in it for starting keychain on login and sourcing the env vars so you can get access to your agents identities. Don't forget the "-q" switch so you don't get that Gentoo-default green and blue spam screen. What's with Gentoo and green and blue console messages, anyway?

At any rate, all of this will be included in my eventual HOW-TO for work, which will be released here as well.

So much useful software, and so easy to use.

Spent the majority of the day yesterday hanging out with Kyle and Pete at Factory. Went to lunch at the supergood Mexican place on ~9th and Washington, then chilled at the space until 2000 or so.

Got SMTP-TLS working on the Factory mailserver, did a little more work on gate, the new firewall, and spent a couple hours reading bash.org.

With regards to SMTP-TLS, a couple years ago I waded through getting Postfix TLS and sasldb for a machine at work. This run through, I just used The Perfect Setup HOWTO and was done with it.

The only thing that really bugs me was having to use a couple backports for libsasl2, which possibly I didn't need (since I'm using the pwcheck daemon, authenticating against /etc/shadow), but I didn't think about it too much.

There are also a few useful notes here.

Around 2000, Ian showed up with his friend Mike and Samid. Ian had his LinuxWorld swag, including a copy of Sun's Java Desktop System, which we've all been very interested in seeing.

Shortly after that, Pete and I took off, as it was getting late and it was already hovering around 0 degrees out.

Ah, Pennsylvania winters. How I love you like truck.

Sometime yesterday afternoon, the test fileserver's netatalk install decided to stop displaying directories in the root share. I'd stayed home to code, so when my co-worker called and informed me of this, of course I had just put my clothes (all of them) in the wash.

So a few hours later, when I had pants to wear, I headed down to Factory to see what was up. After some screwing around, I couldn't determine if I'd managed to fix the problem or not. So I had to come into work (where I am now, half-awake and reading Perl docs), getting in just before it started snowing in earnest.

The problem was caused, I think, by having shares within shares (we have a root volume, with "client" volumes underneath, which are just directories containing jobs for a specific client). Pretty sure it made the .AppleDB databases sad in some way. Unfortunately I don't know enough about netatalk (I suppose I could take eniak's approach and read the source, but, gar, reading C gives me a headache) to be sure.

Luckily the resource forks (which lives in .AppleDouble) didn't explode.

Volumes that didn't have shares below them didn't exhibit any problems, so my solution was to move the .AppleDB directory out of the way, let it get recreated, and remove all the sub-shares.

As I'm looking to move primary fileserver to Linux/netatalk, hopefully we won't be running into too many of these issues...

My favorite part is where I google for the error I'm getting, and all I find is some German bulletin board.

Lovely, that.

Machines get compromised. Pretty much just the way things are, out here on the Internet. However, hastur getting owned was, to the best of my knowledge, the first time one of my UNIX machines has been popped.

hastur runs mirrorshades.org/net, foreword.com, amongthechosen.com, mail and DNS for all of it. It was a lame install, about two years old, before I started enacting filesystem-level security measures (half a dozen partitions, locked down mount options, filesystem checking utilities like AIDE). It was running Snort, but Snort can only detect so much, and looking back at the logs (which are emailed to me every morning -- obviously not the best solution, as they can be munged by an attacker who gains root), I don't see anything that would suggest the attack.

Which isn't Snort's fault, as this was an application-level fault.

But let's back up.

For the past several weeks I've been working as an admin for the metawire.org project, a free shell/hosting service. It's been an interesting experience so far.

I saw the undeadly.org story and signed up. zerash remembered TDYC! and the happybox, which I mentioned in my signup application, and we got to talking. I wrote a couple quick specs for a planned upgrade, and have been helping out with administration tasks since.

It's something of a challenge. The machine is running OpenBSD (3.5, as zee upgraded it over the weekend), and overall is set up okay. They're running custom user admin utilities, which we're slowly working on re-writing to be more abstract and portable (which reminds me, I need to get working on Unix::Admin this week, it's still very larval). I wrote a "hardening" script for OpenBSD, and we ran that on the box, locking file perms down pretty tightly.

metawire has a couple dozen domains attached to it, so users have a pretty good choice of "where" they want their stuff served from. We aren't doing actual virtual domains for mail yet, but that'll come along in a few weeks. There are some issues with CGI and PHP (namely, it's not running as CGI), but those will also be fixed as we have more time to work on the machine. It's already very popular (I'd guess ~100 applications a day), and now it's just a matter of defining where the issues lie and repairing them.

The challenge for me really comes in once you take the users into account. About 80% of the kids are using the machine to good purpose, but the rest are punks from various countries around the world. The majority of them are just script kiddies, but there have been one or two with some amount of skill. Finding the smart ones has proven to be a bit of a luck thing, which bothers me. The kiddies are simple to find. They all use the same stupid tricks, and seem to go from not having a clue how to use a shell to downloading exploits and running them (after some work) against remote hosts.

There've been a few problems with mailbombing as well, which annoys the crap out of me. Luckily Postfix is love, so it throttles and just keeps on trucking no matter what you throw at it.

I think zee, blister, mjc (someone also new to the project I suggested be added as an admin) and I will eventually start working on a known-sploit finder. mjc had the idea of doing binary checks for shellcode, which is a good idea, but might be sort of slow considering the number of files we're going to have to be checking (~2500 users on the box, perhaps a tenth who actually use it on a regular basis; that's still a lot of users). My idea was to just maintain an archive of MD5 sums of found exploit code and binaries. There's a lot of problems with this method, unfortunately. I can't think of anything better without figuring out how to do fuzzy matching, and I'm pretty damn sure I'm not smart enough for that. O'Donnell will have some good suggestions, I'm sure.

Anyway, I've been meaning to write about this for a while, but hadn't been able to find the time. If you're interested in shell communities at all, check out #metawire on irc.metawire.org, and sign up for an account.

Try to make sure that your signup application reason doesn't involve running BNC or "learning Linux", and you should be okay. ;-)

The planned upgrade is going to take some donations, which we seem to be doing okay on. If after a few weeks you find the service useful, try to drop us a few bucks to make it better. Jordan also had the idea of throwing a logo contest and start selling metawire.org wares, which is a pretty good idea. That's on-going. I haven't seen any of the submissions yet, but hopefully someone will hook us up with something good.

I'm enjoying working on metawire; it's going to get me to actually write useful software, I think, and it's a big boon to my actually learning stuff I haven't had a lot of access to in the past for whatever reason.

Anyway, check it out.

Spent three hours this afternoon trying to install OpenBSD on an Ultra10. There's a known issue where the damn things don't like booting floppies. So I grab the OpenBSD boot CD, and try to boot it. It refuses. So instead of trying the obvious (as Harry eventually did with one of his own U10s) and swapping the CD-ROMs out -- the first thing I would have done on x86 -- I screw around with it for hours. If nothing else, at least I got openboot flashed and all up to date and happy. This machine is actually pretty nice. 450Mhz, gig of RAM, two 20GB drives (though one of those will be pulled, as it's unneeded here).

Yeah, anyway. Grabbing the install sets now. I sure feel pretty stupid.

This is really my first time working on a Sun workstation form factor, and it sort of weirded me out that I had to turn the box upside down to pull the case off. Odd.

I wrote a little script to generate a postfix-style virtual table file (as opposed to the Sendmail-style/alias I had been using) yesterday (and had a couple problems with hash assignments... note that list and hash context? Yeah. Important!) and I assume his interest piqued Eric installed Postfix and a random front-end he found to play around with Postfix.

Three (going on four) years ago I wrote this horrible front-end for administering Postfix, Apache and FrontPage (on Apache! guh!) in PHP, feeding into a MySQL backend, with a scary, scary (my first big) Perl script to generate the flatfiles. It's pretty horrible. The high5 postfixadmin app blows it out of the water (considering the simplicity there, that should describe how scary my app is).

(I tried several times to re-write the scary "web panel" app, in PHP, but it never went anywhere because -- in my opinion -- writing big applications in PHP is just too annoying. Writing it in Perl with CGI::Application, Class::DBI and the Template Toolkit would be almost trivial, though.)

That front-end also comes with a HOW-TO, which details installing Postfix+MySQL+IMAP. Decent howto, it looks like. Postfix+IMAP is sort of old hand to me now, though I still view the whole thing as being slightly magickal, though that's entirely due to the eight thousand ways to do auth for the POP/IMAP daemon. Not deep voodoo, just kind of obnoxious, I think.

(The whole high5.net project seems to be pretty cool, in fact.)

Not sure how I feel about throwing my virtual tables into a relational database. The lookups overhead would, I think, tax the machine unduly (though I sort of suspect that Postfix is smart enough to do caching -- I haven't really looked into it, but Postfix hasn't even done anything that made me think it was in any way stupid). The current mailserver at work gets hit enough (what with the spam processing) to skew the clock without ntpd running (this didn't use to happen when the machine was a webserver).

The reason I installed OpenBSD on that Sparc, in fact, was to be a backup mailserver while I reinstalled our current mailserver at work, which is a three-year-old mess. It's sort of amazing the things you can learn about processes, automation, programming, and systems in three years.

The most important thing I've learned, though, is how much there is still to learn...

The majority of my mailservers to date have run Debian and Postfix, and most of the machines running local MTAs have dnscache bound to the loopback. So earlier tonight I noticed an OpenBSD machine I had installed last week hadn't been sending me logcheck reports after I had moved it from the internal network (where I do installs) to the DMZ.

I go check, and it appears that's it's unable to do DNS lookups to get MX records. After a few minutes of screwing around, I notice this error:

May 25 05:22:14 clortho postfix/postfix-script: warning: /var/spool/postfix/etc/resolv.conf and /etc/resolv.conf differ

Easily fixed, and then I go look at Debian's Postfix init script:

FILES="etc/localtime etc/services etc/resolv.conf etc/hosts \
etc/nsswitch.conf"
for file in $FILES; do
[ -d ${file%/*} ] || mkdir -p ${file%/*}
if [ -f /${file} ]; then rm -f ${file} && cp /${file} ${file}; fi
if [ -f ${file} ]; then chmod a+rX ${file}; fi
done

...yup. Debian is awesome, because it does so much for you. And Debian is bad, because it can make your brain lazy, leaving you to wonder why something you haven't had any issues with previously us suddenly acting weird.

This is why I'm really starting to like OpenBSD, I think. No coddling without being obnoxious.

Harry brought a mailserver down to Factory last night, and we spent two hours getting it installed. The problem is something to do with the ISP we get our connection from... there is some incredible weirdness going on.

The layout is like this: We have an uplink from the ISP plugged into our external switch (who has recently been acting up --- some ports have been dying on any packets larger than 206 bytes; kudos to Eric on figuring that one out last week), a firewall with three NICs: WAN, LAN, DMZ.

The ISP has kindly given us a number of IPs... the problem is if you start swapping machines and addresses, whatever upstream hub/switch/router from our switch seems to not refresh its ARP cache. Ever. So when Harry brought his mailserver down, we found that none of our remaining IPs would route past our switch.

Andrew and I spent several hours the other night trying to figure this one out, and somehow managed to get it to work, when we installed the new firewall (OpenBSD 3.5 box). It could have just been coincidence, I really don't know. You can sit there watching arp traffic and you'll see the router ask "Who has $x?" and the host respond "Me!", and then the router proceeds to ask again. The host in question can sit there and see some traffic from the network downstairs (upstream), like netbios, other arp traffic... but not all. I suspect that's because there's some device segmenting the network, and the router and other core stuff is on one side of that, and random other stuff (like workstations and printservers) is on our side.

It's the weirdest thing and we don't have access to the ISP's equipment (obviously) to fix it.

So last night Harry and I ran into this problem, and eventually I just gave up on trying to force a refresh on the router, or whatever the hell is upstream of us (I wish I were cool enough to figure out timing to accurately map a network of transparent devices...), and just plugged his mailserver into the (so far unused, though this will change once we figure out the arp thing) DMZ port of the firewall, and just port forwarded for it. So ghetto, but it worked fine, and I would have felt worse if he had had to take his machine home.

The only hitch was me forgetting the following rule while hacking out the new pf rules:

pass in log on $wan_if inet proto tcp from any to 10.1.1.2/32 port $smtp_services keep state

Because I'd forgotten how pf translated IPs, or more specifically, when.

And then Harry ran into one or two problems getting the ldap server on the machine up... back he quickly fixed that.

It was pretty nice debugging with Harry, actually. He knows stuff, and we swapped position at my laptop to work on the various problems without any issues. Sort of like eXtreme systems administration or something. :)

Just wrote a very bare-bones HOWTO (if you can even call it that) for installing Snort on OpenBSD 3.5.

Harry brought up the uid issue (my useradd statement will just add _snort as a user, and not within the daemon uid range, typically the 500s), so I checked out Postfix's pkg INSTALL script:

useradd \
-g =uid \
-c "Postfix Daemon" \
-d /nonexistent \
-s /sbin/nologin \
-u 507 _postfix

The fact that it's hard-coded suggests to me that there's a daemon to uid map for OpenBSD somewhere, but I'll be damned if I know how to find it.

I'll ask mjc (a fellow metawire.org admin and OpenBSD monkey).

Spent yesterday fighting with our crappy backup staging server (where things go before they're taped, and stay live for a period of time) at work. The thing is a junk Gateway "server" box that was super cheap (always a prevailing concern for purchasing hardware there if we can't offset the cost somehow), but has since proven to be an enormous pain in the butt (get what you pay for).

The machine has, at various points, had its motherboard replaced, its RAM, its CPU, and finally I get the thing working, and the primary IDE bus blows out. Pretty awesome, but a minor fix, as it has three (getting that third one to work is something I really should have documented; it was a pain, iirc).

So initially the machine was running Linux with Reiserfs on a software RAID across three IDE disks on dedicated busses. Slow as hell, but it worked.

So the reiser journals blew their trees all over the place, and since the data is taped anyway, I figured I'd give OpenBSD's software RAID (RAIDFrame, ported from NetBSD) a whirl.

I unplugged the RAID disks (habit) so I wouldn't get confused during the install ( a good habit), pointed the installer at an FTP mirror and went and did other work for a while.

After the machine installed itself (not counting download times, about 10 minutes of work... Much less-than-three to OBSD) I recompiled the kernel, pinning wd0 to the first channel of the secondary IDE bus (it wanted to boot off the , and started fighting with getting the machine to get the third IDE bus recognized (the BIOS and bootloader saw the drive on it fine, but the kernel refused to see it). I spent an hour or so trying to figure out how to get the bus's device attributes (what device number it is, etc), and failed pretty badly. I don't remember how I got it for the previous Linux install, though I did try booting Debian as I have vague recollections of the default Debian kernel seeing it.

Eventually gave up on that and threw another PCI IDE card in the machine. Pinned the RAID disks in place (config -e with -o is pretty great) so they couldn't move around on me ever, and started setting up the array as described in raidctl(8). Pretty simple stuff, though I have to admit that it seemed odd (at first) that I needed to have a FS_RAID type disklabel on the array's drives.

Get the RAID device formatted, mount it, and... it's somehow managed to lose about 100GB of space. There are three drives: two 160s, and one 120. So there should be somewhere in the vicinity of 410GB usuable space, 440 total.

Machine would only see 330GB tops. I pulled the 120 out, fed it another 160, rebuild the array... yeah. Pulled the 160 out, build the array up, and it would only see 300... There's an error stating that it's "truncating" the last disk, but googling for the warning returns nothing.

Next stop is mailing lists and looking at the raidframe code to see what causes it to happen.

Obviously I'm doing something incredibly stupid here. My initial thought was block size, but... That seems unlikely considering the amount of space involved here.

So yeah. If anyone has any ideas on this one, I'd appreciate it before I just reinstall Linux on the box (tomorrow, heh, as I'm tired of not having an easy place to do backups to).

This is somewhat depressing. I took a few minutes this morning to play with SpamStats on work's mailserver.

These stats start at 0630 Jun 02:

Total number of emails processed by the spam filter : 60038
Number of spams : 43548 ( 72.53%)
Number of clean messages : 16490 ( 27.47%)
Average message analysis time : 3.61 seconds
Average spam analysis time : 3.37 seconds
Average clean message analysis time : 4.17 seconds
Average message score : 7.02
Average spam score : 10.43
Average clean message score : -1.20
Total spam volume : 102 Mbytes
Total clean volume : 69 Mbytes

It's also a default, non-tweaked install of SpamAssassin, so I would wager somewhere between a third and half of those "clean" messages really aren't.

My next step is going to be finally throw a.mx at our colo and have it dump anything over a 7, then relay the rest to b.mx at our offices.

Meh.

Spent yesterday working on moving the services living on hastur.mirrorshades.net to ligur.mirrorshades.net. Considering some of the deals that ServerBeach offers, it was a pretty simple decision to make for Dan and I.

I had some initial issues with the machine... like the kernel I compiled didn't take, then grub's fallback mechanism didn't kick in. Apparently the NIC driver I used (eepro100, which is the module the current, vendor-supplied kernel is using) wasn't happy and thus networking didn't come back up. I had planned on writing a little at job to run when the machine comes up to check if it can get 'net, and reboot into a known-good kernel if it can't, but hit a wall with regards to available time.

Yesterday was crazy at work, so the migration only got a few minutes here and there of my time. Overall it was a simple process, and took maybe an hour an a half of my actual attention. Go UNIX. Try doing this shit with Windows, eh.

I kept a relatively information-less log of what I did for the migration (this being what, the fourth time? for this machine) if anyone cares that much.

As always, the easiest portion of the migration involved Postfix. So much love for that piece of software.

Anyway, I should probably shower and get ready for another twelve hour day at work...

I just spent the last hour fighting with raidframe on another machine (the new production backup server).

The error message:

raidlookup on device: /dev/wd0a failed !

The config:

START array
# numRow numCol numSpare
1 2 0

START queue
fifo 100

All looks okay, right?

Wait... look at the error message again...

"wd0a "

...argh.

Adam O'Donnell: i would patch the source
Adam O'Donnell: and submit the patch.
Bryan Allen: http://monkey.org/openbsd/archive/misc/0010/msg01366.html
Bryan Allen: FOUR YEARS AGO
Adam O'Donnell: because no one wrote the patch
Adam O'Donnell: i will do it with you sometime soon if you want
Adam O'Donnell: there is no "strip()" in C.
Bryan Allen: I know. Or =~ s/^\s+|\s+$//gs;

heh. :)

Why is mail such a pain in the ass?

MTA, MDA, MUA.

SMTP. POP. IMAP.

So many pieces for something that is actually relatively simple. And the problem is that the pieces themselves are actually relatively well designed. (In the case of say, Postfix, exceedingly well-designed.)

But taken as a whole, it suddenly becomes something that requires a flowchart with various colors denoting things.

And that doesn't even get us to the point where we're talking about being able to authenticate users sixteen different ways bi-directionally or relaying mail between machines or talking about backups or anything remotely interesting.

For something so simple, it sure does get complex quickly.

That said, it took roughly an hour to get IMAP-SSL running on the new company mailserver today. That includes compiling. And spending a half hour not working on it. So.

Easy. But it still seems way too involved for some reason.

Anyway.

These last two weeks have not been my superhappyfuntime.

The company I work at is merging with another company and their IT guy, who was not only lazy and shall we say, somewhat cavalier with regards to his duties as a systems administrator, but well... the emphasis here is was.

So at the moment I sort of have two jobs. I'm getting tired of 12-16 hour days.

Not to mention the two 24 hour days.

The biggest problem I have is that their entire shop is Windows-based, except for two Macs in pre-press and two in design. That leaves about two dozen Windows workstations, half of which are infected with various forms of viruses, and three Windows servers.

Including, for some as-yet-to-be-determind, an MSSQL machine.

I suppose that explains the "FixBlaster.exe" binary on the PDC's desktop.

I'm just not used to a Windows environment, I think. Tuesday, network connectivity was being super spotty; "Crap," I think. "That 486 junk firewall I replaced their horrible SonicWall with is dying on me." So I go and steal a disk out of a machine whose processor fan had recently failed, install OpenBSD on it, and waste half an hour of bandwidth and a half hour of my time (counting interruptions to deal with other stuff) that the network is still thrashing.

"Bloody Hell," I says, watching it take six packets to get anything anywhere. "hm, are my pf rules screwy?" pf is turned off and the connection is again happy. "Well, I suppose that hopefully rules out the NICs and the hardware," I thinks to meself.

So I finally do what I should have done in the first place:

tcpdump -eni ep0

Oh.

"Golly gee, that's a lot of 135 and 445 traffic going to 192.168.0.0/16 space... space that doesn't exist. Invalid subnets. Damnit!"

And let's not forget the 6667 traffic fleeing outbound to the world, doing gods know what...

So I quickly block all egress traffic save for a few required ports, and connectivity is somewhat happier, though hardly not at all. So I ponder to myself, "Ponder ponder, what's the probl-- oh. Queues."

Yes indeedy. It was taken half a dozen to a dozen packets to fall up the goddamn stack and get routed. Luckily I'm a complainer and Andrew quickly suggested that I just block all non-valid traffic on the internal interface, so the junk never gets processed.

Word to Andrew.

tcpdump -c 50000 -eni xl0 src net 192.168.1.0/24 and dst net 192.168.0.0/16 and dst net \!192.168.1.0/24 > infected_hosts ; awk '{print $6}' infected_hosts |sed -e /.....$/s///|sort|uniq

(My regexp sucks so much.)

That was an adventure!

And not the only one for that day, but the only one that I can remember, because it involved me being stupid. And I always remember those stories.

Today was also pretty awful, but I got a lot done. It's funny how that works. I spent about an hour swapping machines because one of the managers decided to upgrade a piece of software on an operating system that doesn't support... something or other the new version of the application needs. So yesterday Adam installed it on a Win2k box, which is what it wanted.

Only the guy forgot to mention that some printers needed to sort of be hooked up to that box... "Looks like a job for bda!"

So this new machine is actually one of our old ones, but it's been at the new building for maybe three months. And it was caked with dust. And the older box that I was swapping out? Oh. I think at one point, it was probably that stupid tan color old machines all are. But it was grey.

And my clothes? Well, they were black when I started. By 1130, though, they were white.

Yay!

Luckily I still had a box of Christmas clothes in my cube at the other building, so I could change and not be covered in goddamn dust all day. Whee!

The only thing I feel even remotely good about is that the new mailserver appears to be operating optimally. There was some issue with IMAP and Mail.app... namely, if you create a folder on the IMAP server, then add a message to it... delete the message... and then delete the folder, Mail.app cries. "Can't SELECT!" Because it doesn't refresh after deleting and before opening again. And it was connecting way too much.

But I realized I was blocking the UDP ports IMAP wants on the box, and that seems to have fixed the issue. I didn't look too much into it... tomorrow I'll see exactly why that might be. It seems... odd to use UDP for those operations. But what do I know.

I'm not even going to get into the dozen or so "omfg!" fires that people came to be about, causing me to not clean the infected Windows machines. argh. You'd think that'd be my priority, and it is, but it still hasn't happened. Gods willing, I'll get to that tomorrow morning and afternoon.

What else, what else.

Apparently the previous IT guy's default responses to anything anyone ever asked him to do were:

• No.


• I can't do that.

• Deal with it.

And pleasant.

And useful.

And they seem truly astounded perhaps not by our annoyance and the broken state of affairs, but by our wanting to make things better.

For instance! Two sales guys have a printer in their office, a big HP 8500. Nice printer. It speaks JetDirect. The two designers, who use Macs find it with no problem. Humans ask the "sysadmin" if they can print to it. He tells them, "No, you can't. Windows can't print to that printer."

A week after his ass gets canned, the matter is brought to Adam's attention, who says "wtf?" and yesterday asks me to take care of it today.

I poke around for a few minutes, having absolutely no idea how to get a printer without a real printserver to work on Windows. In OS X-land, it's trivial to get it working (and, I assume, just as trivial with AppleShare/AppleTalk in OS 9 or whatever, as that's what the designers use). However, I am a somewhat astute observer of human behavior, so I check the "sysadmin's" WindowsXP workstation, which I have access to.

Lo and behold, he has the printer added. I check to see how it's configured, and apparently you add the thing as a local printer, then configure the port via IP... pretty silly, I think to myself, but exceedingly straight-forward.

I go to add the printer on the sales guy's workstations, and one of them tells me that the "sysadmin" had told him once: "Yeah, you can use that printer. You just have to install the drivers and figure out the IP. I'm sure you can do it." And walked away.

This is, of course, while the machine I was touching was pulling the drivers off the fucking printer and installing them.

In all, this process took perhaps fifteen minutes, five of which I had spent poking at the printer itself like a retarded monkey with a dopamine problem.

(And then a Mac OS 9 box ate its "Volume Header", which I presume to be some sort of MBR analogue, and after I screwed with Open Firmware for ten minutes, I got someone to bust out a Norton Utilities CD and that fixed it right up.)

So that's what I'm up against. Years of that kind of "administration." The place is an enormous mess, and I think it's going to drive me insane. That was just an example. I could go into detail about the problems with the network itself... but it would all be stupid stuff like the gateway's IP being 192.168.1.8.

The only lights on my horizon at this point is that I've been promised an Xserve and a terabyte XRAID, with which I can get rid of the NT4 PDC and manage both the Macs (which will outnumber the Windows boxes once my company finally gets into the new building) and the Windows boxes.

Joy. Network authentication and control and gods willing some form of remote patch management.

Also, Hunter and the company librarian (the guy who deals with backups) and I finally managed to get together and have a nice productive meeting about Archivist, the NetBackup replacement (and job management, and archiver, and possibly some form of remote data access and preview functionality stuff) I designed and started writing months ago... and then stopped because this merger started happening. But with Hunter coding, it should actually get somewhere, and become useful, and with Adam driving Hunter, it should get done. The backend stuff is all designed following the Postfix model... which is to say, the UNIX model, which is to say... Hopefully I won't fuck up a good thing.

And it'll be OSS. Yay.

During this meeting the owner was sitting at his desk (his office is in the conference room) and was half-listening to us. At one point we were talking about system failures and he said "Woah, I don't want to hear that talk?" "What?" "I don't tolerate system failures." "No, you plan for them."

"Bah!" says he.

And now? Now I'm going to sleep. Because I deserve it.

(I realize the above examples seem somewhat trivial and probably childish. But fuck you. It's obnoxious. It's Windows. I am a UNIX ADMINISTRATOR DAMNIT. I'll whine if I want to while I'm getting all this Microsoft garbage shoved down my throat.)

/* Oh. And I'm missing HOPE because the things I mentioned are 15% of what I wanted to get done this week, and because I want to start moving into my new apartment with my friend Pete this weekend. Which is also something to look forward to. To put it mildly. */

It really annoys the hell out of me when an application has an "Import" function, but refuses to let you point it at an arbitrary directory.

Say if you're swapping machines for a user, and want to get them off Outlook.
You have to run Outlook first, copy the pst file to the correct location, and then have Thunderbird import. That's asinine.

I realize I could have just used some other tool to convert the pst to mbox and dump the files into the Thunderbird directory, but... why? Why not just let me say "Import THIS file"?

Grr.

I'm sure Engler posted about this at some point (as he knew what I needed when I was bitching earlier), but: FileMon.

Useful for when you have an application that's writing temp data to some random directory, you have no idea where, but seems to require Admin group privs on the local box.

I hate Windows. But Windows with lsof is slightly less obnoxious.

Sort of.

Many other useful tools on sysinternals as well.

Got bored today and decided to install Solaris 10 beta 5 on some boxes. Keeping in mind that my experiences with commercial UNIX has always left a sour taste in my mouth (IRIX, AIX), and that I have very specific ideas about what UNIX is, you greybeards may want to take this with a shot of J.D. or something. Also keep in mind that this is beta software.

10:20 <@bda> 916 qtimageser 79.2% 0:23.08 1 36 104 344M+ 2.80M
189M- 1.94G
10:20 <@bda> wtf is that.
10:20 <@bda> Oh.
10:20 <@bda> Jesus fuck.
10:20 <@bda> I love how Mac OS will continue to thumbnail files even after
you've LEFT THE DIRECTORY.
10:20 <@bda> So unmounting the volume is unpossible.
10:20 <@bda> This is such bullshit.
10:20 <@bda> Finder--
10:21 < mdxi> this is what happens when you put the user first
10:21 <@bda> Yes.
10:21 * bda kills it.

So we're a pre-press shop. Everyone uses OS9 or Classic with OS X.

I get a call this morning from one of the operators who tells me that "Classic crashed, and won't start again."

So I go over there and mess with the machine for three or four hours. I copy System Folders from other boxes, none of them get recognized as being bootable.

Then:

16:15 < solios> copy FROM the running OS 9 box TO the share.
16:15 < solios> do it the other way around and you'll get Pain in your face.

And that works. OS9 and Classic are happy. Unfortunately all the prefs, serial numbers, etc, from the original System are angry.

So I start copying crap around, becoming more and more annoyed with the situation.

And then Mark, another operator comes back from a smoke and says:

"You know, this used to happen... and we would just copy System Folder:System from another machine and replace the local copy and it would be okay again."

So yeah. I hate computers.

(Also, Norton 7.0 will destroy symlinks in / for OS X. Just a heads up. Fixing them is easy enough: Just re-symlink them from /private. Annoying, but hey. How often do you get to see single-user mode, eh, Mac guy?)

I've always wondered about this, and I needed to know this morning.

mount -u -rw /

Muchos gracias to ejp for taking five minutes to read the man page while I was putting out local fires.

I could install a cluster of UNIX machines with a pair of tweezers and a magnet faster than I can do a single Windows workstation install.

I'm going to laugh pretty hard if this machine gets owned by some host on the LAN I missed in last week's annual Trojaned Bitch sweep before I get it updated.

...and then I'd go the hell home.

I have an ext3 drive with 119G of junk on it that I'd like to feed onto this XRAID we just got. Of course, doing that over a network will take ages, so being able to read ext2 from Mac OS X would be pretty keen.

Enter ext2fsx.

Thanks to ejp for the link.

Fighting with OS X Server is something I have decided I do not enjoy.

There are two problems:

1. The tools suck.


2. I don't entirely understand how OpenDirectory is interfacing with those tools. Because changes made in one tool doesn't seem to actually propagate into Actual Use, even though it's Apparently Working. So not cool.

So who's stupid? Me or the software?

A little from column A, a little from column B...

Totally awesome.

Just ran into this.

I am doing the most ghetto backup solution I think I have ever done. Details later.

I've been fighting with the OS X Server since we got it. Getting it backed up has proven to be damn near unpossible within the context of our current backup system.

This is an email I just threw together after I spent the weekend troubleshooting the machine and various pieces of software that have mashed together.

Just installed OpenBSD on a box. Hardware: one Adaptec 29160, one 3WARE IDE RAID controller. Made sure that the controller I wanted to boot off got detected by the BIOS first (the Adaptec) and boot off floppy35.fs (as it supports both the Adaptec and the 3WARE card, as well as the awful, awful onboard Broadcom NIC -- it's pretty awesome that all three of those random devices are supported in GENERIC).

Do the install, no problems. It detects the single drive on the Adaptec chain as sd0, as expected. Reboot the machine.

Comes up complaining that rsd0* has not been configured. I say whadafuh and get a sneaking suspicion...

Sure enough, dmesg confirms that the damn thing swapped the Adaptec and 3WARE cards, so the RAID card is now sd0. What the hell?

Mount the filesystems, edit /etc/fstab, and one %s/sd0/sd1/g line later, all is well.

The whole day has been like this, though.

First the IDE bus in that machine dies, and then...

Man. I don't even want to talk about it any more. I just want to go home and play X-Men Legends.

Use NFS.

That's really all I have to say.

If you're a printshop, you probably have a bunch of idiotic characters in your filenames. Some of these idiotic filenames have ":" or possibly even "\" in them, and the ":"? They're probably actually part of some stupid character's hex analog. And you can't just mangle the filename on copy because it'll cause linking problems with say, Quark.

So what do you do?

You stop using Samba and use NFS.

Just note that you'll need to use the -P option with mount_nfs on the OS X box.

I'll have a wrapper script for ditto by tomorrow probably so it'll do update syncs as opposed to simply copying every damn thing all the time (gah). Considering how slow this already is, I can only imagine that it's going to get a lot slower. :\

From a default OS X install:

[bda@10-1-2-74]:[~]$ grep NFS /etc/daily
# Clean up NFS turds. May be useful on NFS servers.

bahah...

[via esch]

Here's that ditto wrapper script I mentioned.

Re-wrote it this morning as I had no Internet access (I'm at Steve's, in Plattsburgh, for his wedding) and therefore actually got work done. Funny how that works.

The comments at the top explain it all pretty well, I think.

Now to see if it actually works in production. The tests were fine.

Nick Holland wrote a nifty FAQ on upgrading OpenBSD from 3.5 to 3.6, focusing on keeping system configuration in sync with the release.

I got my CDs a couple weeks ago but have yet to have reason to put install it on anything, or upgrade any machines.

Lots of runaway disk-bound processes on our OS X Server earlier, so I killed them. The machine killed my shell and network latency started oscillating between 1ms and 9000ms. Then it rebooted itself.

21:24 <@bda> [root@sobek]:[~]# vim
21:24 <@bda> E575: viminfo: Illegal starting char in line: b0VIM 6.2
21:24 <@bda> Hit ENTER or type command to continue
21:24 <@ejp> rjbs: bda has shame?
21:24 <@bda> That's new.
21:24 <@rjbs> bda: delete viminfo
21:24 <@ejp> (moded cows)++
21:25 <@bda> I did.
21:25 * bda isn't dumb. :(
21:25 <@bda> I just don't know what that means is all.
21:25 <@rjbs> (modded cows)==
21:25 <@rjbs> bda: it means you need to delete viminfo
21:25 <@bda> k.
21:25 <@rjbs> bda: viminfo gets corrupt every once in a blue moon
21:25 <@bda> ah.
21:25 <@bda> Well. The machine rebooting itself would definitely cause that.
21:25 <@bda> Though, unfortunately, it does it far more often that once in a blue moon.
21:32 < solios> :|
21:33 <@ejp> maybe you shouldn't pee on it every full moon then?
21:37 <@bda> Naw. It needs it.
21:38 <@bda> Or it won't grow.
21:39 < solios> hahah
21:40 <@ejp> and suddenly I undestand all your computer problems.

Updated selene (my PowerBook, PB12A) last night. Seemed fine. Also updated helios (Mystic, dual G4 500), and it has displayed no weird behavior.

This morning, I woke selene up to check email and do some admin stuff. I was connected via wlan via AirPort. Had Keychain Access open, and was in the process of opening TextEdit. They both SPOD'd, which is really, really weird (never seen it happen before), so I killed them, restarted KA -- and all my keychains were gone. KA keeps references to files, and doesn't actually try to manage the files unless you insist, so the actual keychain files had not been touched. So I went to re-add login.keychain (which is kind of important), and it failed silently. At this point I'm more than a little annoyed, so I close all my apps and go to log out.

The machine boots itself into single user mode. Awesome.

After failing to log in a couple times, I hardboot it and it comes back up. I reconnect to our wlan network, and kick open some apps. They start displaying the same behavior. I kill them, turn off AirPort, and plug in a wire. Everything is fine.

So either someone has something unreported and is spamming at my machine via wlan, or Apple managed to fuck up AirPort somehow with 10.3.6.

As Adam said, "Occam's Razor."

sigh.

I was whining about OS X and caching, and Rik bothered to think for two seconds:

<@rjbs> lookupd -flushcache

Yay.

Had to crack a Win2k box this afternoon as it was "appropriated" and no one knew the Administrator password.

I used Austrumi, which in turn uses (I think) ntpasswd to do the actual password changing.

I'm sure I've made posts similiar to this in the past, but I never actually bothered doing any of this (generally speaking, any workstations we would have gotten from elsewhere are riddled with viruses, trojans, and random stupid crap users install on machines, so it's easier to just reinstall most of the time).

Anyway, it worked well.

pyopenbsd, a set of Python classes for interfacing with OpenBDS and associated libs.

<@newsham> awesome. now you dont have to be a C programmer to enjoy the diverging APIs of unix systems!!

Had an idea to do the same thing with a set of Perl modules. Got so far as registering the namespace on the CPAN before getting distracted.

This was, of course, six months ago.

sigh.

Dynamic filtering with pf.

Finally got around to "installing" dovecot.

I say "installing" because it was just a make install, then editing the config file to change the available daemons from the default (imap,imaps) to imaps only.

That was very possibly the most painless piece of server software I have ever installed.

It's funny when you enable pf, because it has no state table.

Friend of mine asked me to pull his data off a busted Windows install, so I said sure.

Finally get around to actually plugging the drive into my workstation (which is a Mac) and then go looking for Linux PPC LiveCDs that don't suck. Well. They all had problems. The Knoppix images I tried didn't even boot all the way. The Gentoo image would boot, then start throwing "vt: argh data_driver is NULL !" errors. Booting with noapic fixed it half the time, at least. But then, of course, NTFS wasn't supported by the kernel and I had just about no interest in recompiling their kernel and re-burning the image for it.

So I do a quick google run for third-party NTFS apps for Mac OS X and hit the Mac OS X Filesystems list, which seems very comprehensive.

And of course OS X has NTFS read-only support. So I boot and start pulling data off to a network drive. It's really not the fastest thing in the world, but it appears to be mostly working (there are some I/O errors getting thrown around; I dunno if that's physical or driver, though).

Anyway, it's saved me a trip to Factory or work in 23 degree weather, so I'm okay with it taking all damn day if it wants to.

Stayed at work late last night with the intention of rebuilding the LAN firewall, and replacing the router with an OpenBSD box.

Unfortunately I had to move the mx to get at the firewall... and the mx has a really twitchy root drive, which finally decided to kill itself. Manged to pull the passwd file and some of the postfix configs off.

The upshot of this is that I spent about five hours building machines and migrating users and data.

• Wrote a really lame Linux (Sixth Edition) password file to BSD master.passwd converter. I had been up since 0630 could barely see by this point, which was kind of fun. The awk line was ripped off from this.
• Documented my default actions during an OpenBSD install.

Got everything up and running (and users shouldn't notice any changes, except being asked to save a new cert if they're using pop3s) around 0330.

Walked to the train station and saw more creepy people in Camden last night than I think I have in four years of late nights. Got home around 0445 and slept for six hours before my next door neighbor decided it was a good time to start shooting aliens and woke me up.

(Good news is today is Sophy and Adam's potluck.)

Let me know what you think about the obsd install doc (though it's more script than doc). I'm not sure about the harden_obsd.pl script any more, but.

An addendum... I forgot to make a seperate /var/mail partition for mailspools as I've gotten used to delivering to Maildir in the user home directories. But I've gotten in the habit, lately, of having spillover drive space mounted to /vol/scratch for just such occurrances.

# umount /vol/scratch
# postfix stop
# mv /var/mail /var/mail.foo
# mkdir /var/mail
# disklabel -E /dev/wd0c

Kill the scratch partition, add /var/mail, re-add /vol/scratch...

# newfs /dev/wd0{$x}
# newfs /dev/wd0{$y}
# mount /vol/scratch
# mount /var/mail
# mv /var/mail.foo/* /var/mail
# rm -r /var/mail.foo
# vi /etc/fstab

Yay.

So earlier today I noticed that a 'pop3' process on our (newly installed, thanks to the old one's root drive burning itself out) user mailserver eating CPU and thrashing.

After a few minutes investigation (and a suggestion from ejp), it seems that the mbox index cache that dovecot builds got corrupted, and spun out of control. Blowing the cache away fixed the problem.

I also upgraded the user's MUA from Thunderbird 0.9 to 1.0, though I figure it's sort of unlikely that was the cause.

Worries me that dovecot will do that in the first place, though, and I suspect this may be one of the things Harry was talking about when he said that dovecot doesn't scale...

(Note: You can turn caching off.)

• Some jackass turned off the machine that serves the estimating software.
• Had to troubleshoot that fun dovecot bug.
• Co-workers machine got owned and was hammering random machines on the Interweb with ssh brute force attacks. Common forensics software doesn't see shit (linux), but I'm not surprised.
• Had to set up the firewall rules for reflection, as I forgot to do it on Friday.
• Discovered that rsync_hfs apparently does not work with netatalk2, in that it does not keep resource forks. Which is its entire purpose for being. psync, ditto, CpMac, etc, all work as expected. Rewrote wrapper scripts to use psync. Last two weeks of data have no resforks. Will be very entertaining when someone asks for a restore. Note to self: Test all software more thoroughly after upgrades instead of just assuming because it doesn't cause OS X to reboot itself doesn't mean it's more gooder.
• It was so windy and cold this morning that the side of my neck which took the brunt of the wind is still red, seven hours later.
• Haven't got to work on my graphing project at all, which is annoying but fair.
• Need to set up OpenVPN on the company firewall.
• Need to set up OpenVPN on PWF's firewall and set up tunnels to CCCP and the Hasty Pastry. Need to read up on routing protocols (or just leave it up to porkchop who seems way more interested in it than I do).

Yeah.

Someone spammed this to misc@openbsd yesterday. Unattended OpenBSD install media. Awesome. Will definitely be playing with this once I get home.

Pulling the config based on system stuff is definitely something I might be interested in working on as well.

I just added the metawire.org Apache logs to newsyslog.conf:

find /var/www/logs -name "*_log" |sort |sed 's/$/ root:daemon 640 10 * 24 Z "apachectl stop ; apachectl start"/' >> /etc/newsyslog.conf

newsyslog -v -f /etc/newsyslog.conf

And it took a good few minutes, as they've never been rotated and weighed in around 1.2G. The loadavg kicked up to 80 while the files were being compressed, which was pretty entertaining.

A more sane solution to the apachectl command above would be a script that stops Apache, waits until any httpd-related ports aren't being returned by netstat, and then start it back up.

The other day I refactored the PWF network, changed internal addressing, set up a DMZ off the firewall, etc. I also stole conduit Mk I (the PWF mailserver) and reinstalled it for use as the fileserver Kyle has been talking about for a while. Did another install on some other random box for conduit Mk II. Spent a few hours down here, made sure it all worked, then went home around 2200...

Kyle msg'd me this afternoon and told me he couldn't get to conduit Mk II. As everything was working fine last night, I was somewhat confused as to why it would be broken. But yeah, it was inaccessable. Figuring it was a hardware problem, I became suitably annoyed, since I have yet to set up decent automated OpenBSD install stuff. Showered and got to Factory around 1520, plugged a keyboard and head into the box, logged in. Everything seemed fine. I let Kyle know, then went out to Sev for some soda.

Came back and noticed that conduit's power light was blinking. I tried to ping the box, and hey... nothing. So I smacked the keyboard that was still plugged into it. Display woke up, but still no network. Power light went solid. The damn thing was sleeping.

Rebooted, fixed the BIOS...

I don't think I've ever had that happen before. Possibly that's because I usually removed APM stuff from the Linux boxes I admin (unless it's required for P4 HT, but those are all server motherboards anyway), and most of the OBSD installed I've done have been on previously known-good hardware. This was just one of the random Andrew-hoarded junk PCs we have laying around here.

I took some pictures. That first one is my bedroom. The rest are of Factory. Ian noted that the date on the camera is wrong. It reset after I put new batteries in it, and I guess I wasn't paying enough attention while licking the buttons. Pressing. Pressing the buttons.

Couple hours later Bryce came down and we cleaned up a bit. By "clean" I suppose I mean we cleaned off the desks and Bryce did some organizing. I suppose I should take some "after" pictures, but that seems pointless.

My next few Factory projects: mail filter box, list server, VPN...

Oh.

And the latch on my PowerBook broke. That sucks.

Harry bitched at me for making that Red Hat joke the other day, so just to be an ass I went ahead and downloaded the Fedora Core 3 ISOs. Finally got around to installing it on a machine today:

Dual P3 900-something, 512MB RAM, SCSI, Ensoniq something or other, NVidia something, Intel EEPro.

It's a Penguin Computing workstation, so all the parts are pretty much guaranteed to work, or they're bad.

Anyway, it booted, saw stuff, installed.

So far it's not awful. I just went with the Workstation install, just to screw with it, since it's just a toy to me. The up2date tool is nice. The fact that it's in the menubar at launch is good stuff. I just pulled a 130MB of updates just now and it's installing.

Netfilter defaults to on, as does SELinux, so there's actual Workstation Security stuff going on, which is pretty awesome.

GNOME 2.8 is fast. The menu layout still sucks. After using OS X for the past two years, I'm not used to wading through menus to get at things anymore, especially not simply configuration/preferences. There should be some sort of central location for that stuff. (Is there? gconf doesn't count.)

The keychain icon in the toolbar when you auth to root is good stuff as well.

Overall, thus far, I would say it's a pretty good product.

That said, some people have run into problems with the install or various other things. Perhaps I'll hit those, but probably not before I install something else on the machine. :)

It should also be noted that the Windows Browser thing is still broken. I've never seen a distro where it actually does work, though, so you can't really hold it against RH (I guess).

I had initially intended on getting some RH server action and doing a real review, and how it stood up against other server OSes (Sol10, OpenBSD, etc), but obviously I can't get at the RH Enterprise bits, and reviewing FC3:Server against those just doesn't really seem fair.

I would reccommend it to someone who just wants a workstation, anyway.

I just got done fumbling around creating a ccd on OpenBSD; spent about an hour on it, or a little more.

Background: This is a machine I'm sure I've complained about in the past. Gateway "server" with three dead IDE busses. In its current iteration, it's meant to be used as a mirror of our production data and server backups. These will get taped off nightly.

I "repurposed" a 200G SCSI drive that had been hanging off the O2000 a couple months ago. But it'd been laying on the server room floor (sigh) for a while, so it was up for grabs. I didn't realize it was 200G until I mounted /vol/scratch, though. Bit of a shock.

Anyway, creating a ccd is super trivial. It's in GENERIC, so there's no need to recompile. By default, you have four available ccd's (ccd0-ccd3).

First, create disklabels on the component devices. Make sure your track offset is 2. This is what bit my ass for over an hour, because I wasn't thinking.

I had to read this to actually get it. And then it was all made clear.

Anyway, this machine was meant to eat four 200G IDE drives, but there's no way I can fit the fourth drive in there; the IDE cables just won't have it. If I had some velcro I could ghettohack it, but I haven't got any. So, anyway.

Once you have your diskabels made, it's just a matter of:

ccdconfig creates a non-zero partition table... "c", which is usually used to symbolize the whole disk is in this case a whole partition encompassing the full disk.

If you want to cut the ccd up into smaller partitions:

and use the "z" command to zero the partitions and then create your partitions as you normally would. The FAQ fails to mention this, and it was not immediately obvious to me (but that's probably simply because I'm stupid and miss the obvious as times). ccd(4) and ccdconfig(8) do not mention it either, though, so...

Anyway, once you have your partitions set up:

Pretty easy.

So I'm installing the machine that will replace both hastur and ligur, named ligur Mk II. I'm installing postfix, and when it pulls the tls/ipv6/pf patch, it throws a checksum error. "What the hell," says I, and grab an md5 of the file. Sure enough, it doesn't match the checksum listed in distinfo. So I go check on another box, and sure enough... so then I uncompress the two patches, get digests, and they're the same. I copy the patches to a third machine and diff the "bad" and known good patches. No differences.

Same filesize, same chars, same digest. So I recompress the "good" patch on the working box, and copy it over the new box. Same checksum error.

After a few minutes of screwing around, I think to myself...

[bda@selene]:[~]$ touch foobar ; gzip foobar ; md5 foobar.gz
MD5 (foobar.gz) = 36b0031ef3f51c3ceaa0700d8546de41
[bda@selene]:[~]$ rm foobar.gz; touch foobar ; gzip foobar ; md5 foobar.gz
MD5 (foobar.gz) = 997d552d8d6835a6f2b4ea719ba350d5

Apparently gzip flips bits as part of its compression algo. Useful so you know if a file has been recompressed (which must have happened on the mirror I pulled the patch from originally).

As an ADC member, I get access to the latest Mac OS 10.4 builds. I can't talk about them due to that whole NDA thing, but they cause me to reinstall my machines on a fairly regular basis.

8a393 came out today and I installed it on my laptop when I got home. I went to install it on my PowerMac (a dual G4 gigE, "Mystic"), and the DVD-RAM in it pretty much said "No. Piss off." This is pretty common with the junk-ass drive, so I thought about it for a few minutes.

First I figured I would just dd the Tiger image onto my spare 2g 10G iPod. This didn't work so well.

Then I realized... you can boot off firewire drives. So it stands to reason you can fucking install onto them. I have a firewire enclosure.

A few minutes of screwing around in the Mystic's insides later, I had the root drive out, attached to the enclosure, and plugged into my laptop. A reboot later, and I installed off the DVD. The install rebooted and asked me if I had another Mac I wanted to sync off of for this new install... as a matter of fact, I did: The laptop's boot volume. Ten minutes later (sigh, slow drives), I had a nice mirror of my laptop.

And now? Booted off the firewire drive with my keychain, my Mail, iChat, Safari, Terminal, etc, etc, settings all happy.

Sure it's just a matter of ditto/cpMac'ing files around... but damn. When it's so easy I don't have to think about it, just say "Yes, do that thing" and it works?

Well, that's why all my workstations are Macs these days.

Word up.

So Pete brought home a copy of WoW for me to waste my life on. Super, I think, but me being me and unable to just leave well enough alone, I don't want to wait for my MacMini to arrive. So I go to install it on my laptop, as my PowerMac a) has no CD-ROM and b) is a dual G4 500, not something WoW will be happy on.

Unfortunately WoW says "wtf is this Tiger bullshit? You ADC membership-having sucker, I won't play nice with this", so I figure, fine, 8a414 came out last week, I need to update anyway. I'll just dump my data onto my workstation and install Panther before going to bed, then burn the new Tiger build tomorrow and install it on another partition.

Ha!

I boot my laptop into target disk mode, as that will be much faster than copying 25G over ethernet. No big. Then I hear the enclosure my workstation's root drive is in go "Pop!" and suddenly get real quiet. Anything not currently in memory on my workstation stop working. "Awesome!" I think, and reboot.

It boots my laptop OS. "Not awesome! But whatever!"

So figuring I'll just go ahead and continue on with dumping my data onto the PowerMac's data drives, I roll my chair back to grab the power cord for the laptop... and crack. I roll over it. "Fucking damnit!" says I, and look at it. It looks sad, but plugs in and charges the machine happily. "Whoo," I think.

So now I am copying all my crap off my laptop into my workstation which has no OS drive on it. Likely I just blew the enclosure and not the drive itself (which seems to spin up okay from what I can hear), so I'll just shove it back onto IDE later.

Well, now that the machine has free IDE. I spend the majority of yesterday moving data around... the new mirrorshades.net box (ligur Mk II) now has two 200G Maxtors in it, concatenated into around 355G. Yay.

I like that all of this happens at midnight. I really should know better by now...

Well, the ligur to crowley migration seems to be mostly over. Just a few little things left to do but it's answering DNS, serving web pages, and eating mail, which in my book means it's pretty much done.

As a few of the users aren't used to BSD (crowley runs OpenBSD) but Linux, there are some issues there. :)

Installing amavis was just as big a pain in the ass as I remember, but I just dug out my link to the Fairly Secure Anti-Spam Wiki and ran with it. Some modifications to their stuff... I need to clean up the script I generated to actually install the stuff, but eh.

Took about four hours to set up the new box and move everything over, I think (data had already been getting sync'd). Meh.

Migrating the POP accounts was quite painless. Now, getting away from that commercial webmail client with the obfuscated format... that kind of sucked.

• Converting Mbox mailboxes to Maildir format
• Maildrop configuration

The gutted PowerMac on the floor *was* running a Tiger beta, which had some interesting issues (DNS would stop resolving. AFP enjoyed eating a CPU and not responding to requests -- taking any other hosts which had it mounted with it), so I figured I would reinstall it tonight.

Realized that I would either have to find a DVD drive, or pull the root disk out and plug it into one of my other Macs and install it via firewire...

Figured it was all too much of a pain in the ass and OpenBSD is just so much easier to deal with.

I love OS X, but freakin' Finder should have been replaced in 10.4. Punkass bitches.

From the BitTorrent FAQ.

On some unices, BSD libc has a bug that causes BitTorrent to be very processor intensive. Run the client with the "--enable_bad_libc_workaround 1" option to fix this.

Apparently OS X/Darwin is not one of those libcs, but OpenBSD is. Good to know.

Really tired of this bug in OS X where if I sleep my laptop, wake it up, sleep it again without authenticating, the next time I wake it and do auth, it will go back to sleep. Usually it resets the brightness to the lowest level as well.

If someone could fix it, that'd be super kthx.

Someone emailed in response to this misc@openbsd post asking for pointers on getting AV and spam filtering running on OpenBSD. I've gone ahead and cleaned my notes up slightly and dumped them in my scripts dir...

Here are my amavis install on OpenBSD notes.

As I've said before, I've used the Fairly Secure Anti-Spam Wiki as a basis.

Like I told Charles... YMMV. :-)

I've become a big proponent of TRAC in the last month or so. It's a very simple, very efficient project management system and svn client. It's good stuff. Many projects (including Catalyst) have adopted it.

I got bored this morning and decided to install a personal copy on mnet, which required installing mod_python and setting up a bunch of other junk for it.

So here are some more "Installing stuff on Op