-- William Gibson, All Tomorrow's Parties
So last night I was at work until around 2230, doing various sysadmin things that because of this merger have been neglected. I spent an hour or so at our colo, updating kernels, rebooting machines, testing applications, etc. I got home around 2330, talked to Nancy for a couple hours, and then crashed.
At 1000 I get a call from Adam saying that osiris, the production database/fileserver is acting weird and none of the applications that use it are working. So I log in, note that the load average is 150 and rising, and notice that a) many many rsyncs are running (there is a live mirror of all the data on another machine), a lot of pgsql maintenance processes. In fact, I notice that anything that tries to talk to the RAID array is hanging. So I check the system log and sure enough, there are all sorts of SCSI errors.
I think to myself, could this becaused by 2.6's new Adaptec 7k driver? Naw... maybe I hit the cable or something last night (ignoring the fact that I tested the apps after the kernel upgrade).
So I spend a half hour updating the backup server (because save for syncing the database data, dump files, and the production files themselves, I haven't really set this box up as a secondary server) with everything, changing all the application information (which amounts to changing "foo-db" entries in /etc/hosts and changing NFS mounts to point at isis, the secondary), and it all seems good. In fact, the clients say that it's noticably faster. Considering that osiris (the sad primary) is a dual Opteron 240 with a gig of RAM, and isis (the secondary) is a P4 something with half the RAM and a slower drive, I'm inititally confused. Then I think about it for two minutes and realize it makes perfect sense, though I still don't really care about it.
Anyway, then I get a call stating that anubis, the pre-press backup mirror (which eats backups from the production pre-press server via an NFS mount and ditto_sync.pl) is dead. In fact, it's dumped to the debugger. Awesome. First time I've seen OpenBSD eat itself.
Then I get a call saying that sobek, the pre-press Xserve with attached XRAID, is being "really slow".
Note to self: Update ditto_sync.pl to check what it's writing to. If it's trying to write to something that is not an NFS mount, exit. Otherwise / will suddenly be very, very full, and the machine will be running very, very slowly.
It's a problem, but one easy to fix.
Amusingly, while OS X continued to operate more or less just fine, the GUI completely stopped responding and wouldn't wake up. Go BSD "subsystem".
So to fix sobek I have to go into work. Adam and I then go out to the colo to fix osiris, or rather just reboot it and deal with it later, deciding that isis is now the new primary until we can re-architect the whole data asset management network thing into something that isn't a thing but a thing that doesn't suck. I head back into the city just as solios is getting off his train at 30th St Station. He takes a cab into the city and manages to get to my apartment before me.
We then spend the day walking around the city, eating about four lunches with various people, two dinners with other various people, and see very amusing things, like:
- Arab cab driver with a bullhorn out his window stating that if you love liberty, you will vote for Bush.
- On Christian or Carpenter, a line of row houses with a hole in the middle of the block, with a sign out front: "Another P.H.A. Mess. The neighborhood thanks you."
And other things I am sure I am forgetting.
We head to dinner at Gusto with Adam and Sophy, then down to B&N and Borders to get a copy of Pattern Recognition for solios, which apparently they simply do not have in Pittsburgh. Apparently Neuromancer has been re-released, so Adam and I, the fanboys that we are, picked up a copy a piece of that. Awesome. Hopefully I can get that signed if Gibson does a tour for the new book (whenever the hell that'll be out... next year hopefully).
We hung out at their place for a while, until Nick and Fred showed up, then we split up, agreeing to meet up at Liquid shortly.
I showed solios what the cover of system will be, in the unlikely even that I ever, ever get that far.
Tonight there was a Moqita show which we didn't actually geto to see Moqita play at (we got there two hours late). Two other bands were supposed to play tonight as well, but both ditched for some reason. By the time solios and I swung by Factory to get my shit, back to my apartment to drop said shit off, and got to Liquid, JG and Billy were incredibly drunk. JG complimented solios on his Crass shirt, stating that now that someone wearing that band's shirt had come to one of his shows, he was done. His career was complete. Liquid was pretty beat, which sucks. Last time there was a decent crowd up until around 0100.
And solios hasn't even heard any Moqita yet.
We then hit up the Diner for the second time today so Nick and Fred could get some food. I had some tasty pie.
I showed solios the Korean War Memorial, explaining about the whole Losing My Shit episode, and now he's crashed out on the couch and I'm about to burn out as well.
Tomorrow is Pumpcon.