"That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace."
-- William Gibson, All Tomorrow's Parties
May 2, 2007

Nothing to see here, citizen. Move along. Or we ask for your papers.

Next up: Freedom Hotline. Inform on your fathers. Your mothers.

(The State is your family.)

4:11 AM | Life
May 5, 2007

Charles Stross had a short story collection republished last year: Toast. It's stuff from the 90s, and overall it's some great stuff. The first entry in the collection (Antibodies) is definitely my favorite, though. Also has the story that became the first chapter of Accellerando (a novel I definitely did not care for; but the spiritual sequel, Glasshouse, got pretty damn awesome if you let it). Ex-KGB sapient space lobsters. No. Really. It works much better standalone, I think.

I ordered copies for random people. I should do that with Meiville's Looking For Jake which is also full of some amazing stories.

I think I need to go back and re-read Stanislaw Lem's The Cyberiad. I was relating one of the stories to #215 and managed to get it completely backwards. Oof.

Old age. Memory is sliding even further into nowhere.

3:52 AM | Books

11:13 PM | Linkwhore
May 7, 2007

Sweet.

I will be taking copies of Fight Club and his (newly delivered to me) Rant

You should come.

11:47 AM | Books
May 8, 2007
May 9, 2007
May 12, 2007

Ben Rockwood responds to Paul Boutin talking smack? maybe? about sysadmins.

My own experience with becoming a system administrator did not involve drawing Sun logos onto my Trapper Keepers or memorizing IBM hardware line-ups instead of important Civil War battles. I wasn't even really aware of those things. No, I fell into it from the bottom-up. The first UNIX box I touched was Linux, and I didn't even really understand there was a whole ecosphere of UNIXes out there for a couple years after that. I knew they existed, I suppose, but they were like funny birds you hear about in far-off countries.

Looking back on the (almost) ten years of my "career", it's only now that I actually feel I'm edging up onto the ramp of competency. It's more an understanding of all the things I don't know than a pride in the things I do, though. In some respects that's heartening, because it means I'm becoming good enough to know what I'm not good at, instead of simply being blindly ignorant. It's also disheartening, though, because there's a great deal of areas in which I know I need a great deal of improvement.

When I get down about this, I'll break out my (rather beat up) copy of Hagakure and read the following excerpt:

A certain swordsman in his declining years said the following:

In one's life, there are levels in the pursuit of study. In the lowest level, a person studies but nothing comes of it, and he feels that both he and others are unskillful. At this point he is worthless. In the middle level he is still useless but is aware of his own insufficiencies and can also see the insufficiencies of others. In a higher level he has pride concerning his own ability, rejoices in praise from others, and laments the lack of ability in his fellows. This man has worth. In the highest level a man has the look of knowing nothing.

These are the levels in general. But there is one transcending level, and this is the most excellent of all. This person is aware of the endlessness of entering deeply into a certain Way and never thinks of himself as having finished. He truly knows his own insufficiencies and never in his whole life thinks that he has succeeded. He has no thoughts of pride but with self-abasement knows the Way to the end. It is said that Master Yagyu once remarked, "I do not know the way to defeat others, but the way to defeat myself."

Throughout your life advance daily, becoming more skillful than yesterday, more skillful than today. This is never-ending.

It doesn't necessarily make me feel better, but it usually makes me hate the world less. Getting back to even, maybe.

My job is often stressful, and mainly seems to be ever more rare islands of "ok, now this is cool" amongst a sea of frustrations. I wish I could say this condition makes it easy to lose sight of why I started down this path -- but like I said: I fell into it. I'm still here because I don't know anything else. I suppose it's enjoyable enough, and typically pays well enough, that the Irish genes kick in.

Like Colin Sullivan says in The Departed:

I'm fucking Irish, I'll deal with something being wrong for the rest of my life.

(Still not sure what it means, being California Irish by way of pretty much everywhere.)

And lately, anytime something ridiculously stupid occurs, it's harder to treat it as nothing more than a challenge. Now it's just another reason to stumble off this path and find another.

Good luck with that, me.

7:05 PM | Systems Administration
May 18, 2007

I've been spending a lot of time working on consolidating our services. It's tedious, because we have been a Linux shop since the company was started: there are many Linux and GNUisms. I have yet to question the decision to move as much as I can to Solaris 10, however. Consider:

We have, in the past, had two (more or less) dedicated systems for running MySQL backups for our two production databases. These replicas would stop their slave threads, dump, compress, and archive to a central system. Pretty common. But they were both taking up resources and rackspace that could otherwise be utilized.

Enter Solaris Zones. There's no magic code required for mysqldump and bzip2, so moving them was trivial. The most annoying part of building any new MySQL replica is waiting on the import. But, if you admin MySQL clusters you're probably already really, really used to that annoyance.

So I built a new database backup zone to run both instances of MySQL. Creatively, I named it dbbackup. It ran on zhost1 (hostnames changed to protect the innocent). Unfortunately, zhost1 also runs all our development zones (bug tracking, source control, pkgsrc and par builds) as well our @loghost. Needless to say, the addition of two MySQL dbs writing InnoDB pretty much killed I/O (this is on an X2100 M1 with mirrored 500GB Seagate SATA 3.0Gb/s drives), making the system annoying to use.

This week I deployed two new systems to act as zone hosts, one of which is slated for non-interactive use. So last night I brought down the database backup zone and migrated it over.

This document details the process, which is ridiculously trivial. No, really. The most annoying part was waiting on the data transfer (60GB of data is slow anywhere at 0300).

My one piece of extra advice is: Make sure both systems are running the same patch level before you start. PCA makes this pretty trivial to accomplish.

This is a sparse-root zone, but there are two complications:

  • I delegate a ZFS dataset to the zone, so there are a bunch of ZFS volumes hanging off it. However, they all exist under the same root as the zone itself, so it's not really a big deal.
  • I have a ZFS legacy volume set up for pkgsrc. By default pkgsrc lives in /usr/pkg, /usr is not writable since it's a sparse zone, and I don't really want to deal with moving it. It needs to be mounted at boot time (before the lofs site_perl mounts which contain all our Perl modules in the global zone), however, and after a little bit of poking I couldn't figure out how to manipulate zvol boot orders. Legacy volumes get precedence over lofs, though, so. Ghetto, I know.

The volume set up looks like this:

[root@dbbackup]:[~]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
data 85.3G 348G 24.5K /data
data/zones 41.2G 348G 27.5K /export/zones
data/zones/dbbackup 40.4G 348G 135M /export/zones/dbbackup
data/zones/dbbackup/tank 39.9G 348G 25.5K none
data/zones/dbbackup/tank/mysql 39.9G 348G 8.49G /var/mysql
data/zones/dbbackup/tank/mysql/db2 24.5G 348G 24.5G /var/mysql/db2
data/zones/dbbackup/tank/mysql/db1 6.92G 348G 6.92G /var/mysql/db1

So, my process?

First, shut down and detach the zone in question.

[root@zhost1]:[~]# zlogin dbbackup shutdown -y -i0
[root@zhost1]:[~]# zoneadm -z dbbackup detach

Make a recursive snapshot of the halted zone. This will create a snapshot of each child hanging off the given root, with the vanity name you specify.

[root@zhost1]:[~]# zfs snapshot -r data/zones/dbbackup@migrate

Next, use zfs send to write each snapshot'd volumes to a file.

[root@zhost1]:[~]# zfs send data/zones/dbbackup@migrate > /export/scratch/dbbackup@migrate
[root@zhost1]:[~]# zfs send data/zones/dbbackup/pkgsrc@migrate > /export/scratch/dbbackup-pkgsrc\@migrate
[root@zhost1]:[~]# zfs send data/zones/dbbackup/tank@migrate > /export/scratch/dbbackup-tank\@migrate
[root@zhost1]:[~]# zfs send data/zones/dbbackup/tank/mysql@migrate > /export/scratch/dbbackup-tank-mysql\@migrate
[root@zhost1]:[~]# zfs send data/zones/dbbackup/tank/mysql/db2@migrate > /export/scratch/dbbackup-tank-mysql-db2\@migrate
[root@zhost1]:[~]# zfs send data/zones/dbbackup/tank/mysql/db1@migrate > /export/scratch/dbbackup-tank-mysql-db1\@migrate

Now, copy each of the dumped filesystem images to the new zone host (zhost2), using scp or whatever suits you. Stare at the ceiling for two hours. Or catch up on Veronica Mars and ReGenesis. Whichever.

Once that's finished, use zfs receive to import the images into an existing zpool on the new system.

[root@zhost2]:[/export/scratch]# zfs receive data/zones/dbbackup < dbbackup\@migrate
[root@zhost2]:[/export/scratch]# zfs receive data/zones/dbbackup/pkgsrc < dbbackup-pkgsrc\@migrate
[root@zhost2]:[/export/scratch]# zfs receive data/zones/dbbackup/tank < dbbackup-tank\@migrate
[root@zhost2]:[/export/scratch]# zfs receive data/zones/dbbackup/tank/mysql < dbbackup-tank-mysql\@migrate
[root@zhost2]:[/export/scratch]# zfs receive data/zones/dbbackup/tank/mysql/db2 < dbbackup-tank-mysql-db2\@migrate
[root@zhost2]:[/export/scratch]# zfs receive data/zones/dbbackup/tank/mysql/db1 < dbbackup-tank-mysql-db1\@migrate

Before I could attach the zone, I needed to set the mountpoints for the dataset and legacy volumes properly.

[root@zhost2]:[~]# zfs set mountpoint=legacy data/zones/dbbackup/pkgsrc
[root@zhost2]:[~]# zfs set mountpoint=none data/zones/dbbackup/tank

Also, since the zone was living on a different network than the host system, I needed to add a default route for that network to the interface. I talked about this earlier, and it's a workaround that should be going away once NIC virtualization makes it into Solaris proper from OpenSolaris (I would guess u5?).

[root@zhost2]:[~]# ifconfig nge0:99 A.B.C.D netmask 255.255.255.224
[root@zhost2]:[~]# route add default A.B.C.1
add net default: gateway A.B.C.1
[root@zhost2]:[~]# ifconfig nge0:99 0.0.0.0 netmask 255.0.0.0

Now, create a stub entry for the zone with zonecfg.

[root@zhost2]:[~]# zonecfg -z dbbackup
dbbackup: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:dbbackup> create -a /export/zones/dbbackup
zonecfg:dbbackup> exit

And that's pretty much it.

Attach the zone and boot it and you're done.

[root@zhost2]:[~]# zoneadm -z dbbackup attach
[root@zhost2]:[~]# zoneadm -z dbbackup boot

Once it was up, I logged in, made sure the MySQL instances were replicating happily and closed the ticket for moving the zone.

This level of flexibility and ease of use is key. In addition to the other technologies included in Solaris 10, you'd be crazy not to be utilizing it. (Even with the annoying bits still lurking in Sol10, it's absolutely worth the effort.)

And it's only going to get better.

8:36 AM | Systems Administration
May 19, 2007

Peter Watts previews:

I've just finished reading a draft of R. Scott Bakker's soon-to-be-released Neuropath. Holy shit.

...

No. He sets it a mere decade into the future, in the context of a serial killer police procedural. Instead of aliens and freaks he uses sexy FBI agents and divorced psychologists. This guy is basically writing about Blindsight-type issues, but is aiming them squarely at a da Vinci Code audience. He is dealing with the same existential questions, but has rendered them accessible for beach readers. He has done exactly what I would have done, if only I'd been smart enough.

At least Blindsight came out first. I can cling to that. Because trust me: when Neuropath hits the shelves, it's gonna be "Peter who?"

Peter who? Peter fuckin' awesome Watts, is who. You should all read his work.

12:24 AM | Books

Just a little exercise from a couple weeks ago. Minor amount of editing. Lots of obvious thievery.

There is only the vaguest appreciation for death while it's happening. It's only later, after, that you can make any sense of it. Attempt to comprehend the enormity of loss as your short reLife is either ripped or trickles away from you. That you can consider the millennia filled with people who went through this same experience, in its myriad, fractal manifestations. So you'll be perhaps better prepared, the next time.

The field around me is full of crying soldiers who should know better by now. Not that I can see them, my eye sockets filled with flaking flesh and ash. The bugs on this world have what amounts to a laser grenade. One of their older, unadapted, pre-Contact weapons. No doubt it worked wonders on each other, incoherent light cauterizing any portion of their sight-bands. 75% of your body being covered in optics, all of which you consider crucial, makes for a lot of area lacking armor. The bug version of a land mine.

Humans (or anyway, those of us currently inhabiting vessels derived from human stock), though, with comparatively tiny optical surface area makes us inherently less vulnerable to such a peculiar weapon. Our helmet visors do well enough blocking the majority of the effect, and the lasers are nowhere near powerful enough to make our armor even tingle in retaliation. So all in all, the cornea bombs are pretty ineffective against us, unless you happen to have taken a couple rounds to the face, shattering your visor but miraculously not your brainpan.

While I was on my knees, screaming, eyes wafting away on the wind, one of the damn bugs came up behind and impaled me with one of those long spikes they favor for close-in combat. The ghostbeam managed to dig enough holes in my skull to compromise the neural mesh, screwing up my communications link and no doubt some as yet undiscovered autonomic functions. So I'm blind and mute, armor holding up like paper against the bug's blade.

So here I lay, modified blood already clotted the wounds, but enough major organs and their redundancies have been shredded that it doesn't matter. I lay here, listening to the screams of my fellow hitchhiking soldiers and the inhuman cries of our enemies, and I wait to die.

Of course command wouldn't want you to be able to pull your own plug. You'd have guys popping off as soon as they lose an arm or a leg, or had their lungs liquified while their exchangers are still happily oxygenating what passes for their blood. It's just pain, they tell us. Use it. Get over it. Take the fucking hill.

But laying here, I can feel the meanest part of my intelligence trying to cram itself into the entangled uplink wrapped around this body's enhanced spine. Desperately wanting to find itself back in the safety of our true home, the quantum womb we so rarely inhabit these days. Knowing full well that any respite will be brief but not caring; knowing the reBirth will be worse than this, this laying here and waiting to die.

Again.

2:38 PM | Writing

Videos of the recently announced Starcraft II.

woot.

I spent a ridiculous amount of time playing the first one. From the videos, it looks like they didn't jack with the concept at all, just updated the engine, units and whatnot.

Warcraft 3 was ... really awful, compared to previous titles, so not jacking around with Starcraft makes me a happy little nerd.

Here's hoping SC2 is ++.

10:55 PM | Gaming
May 20, 2007

Jimmy Carter Is Displeased With You, President Bush.

Recant Your Evil Ways Or Face the Consequences.

I mean, look at that photo. Would you want President Carter making that unhappy face at you? He is usually so happy and smiling and positive.

9/11, Iraq, Katrina, our civil liberties... And now Bush has made Jimmy Carter sad.

What next, he kills a kitten with a baby's stolen candy?

1:47 AM | Linkwhore
May 23, 2007

The trailer for The Golden Compass, first book of the His Dark Materials series (which is definitely one of the best sets of books I've ever read), just dropped.

Actors look bang-on. Art direction looks awesome.

I am tentatively hopeful that it will not suck!

(And Lyra really does look like Lyra. Yay!)

1:01 AM | Movies

< bda> pkg_add: Where did our dependency go?!
< confound> haha
< bda> Where DID our dependency go?
< bda> This is not my $PATH!
< bda> This is not my beautiful package management system!
< bda> How did I get $PWD?
< rjbs> letting the links go by
< rjbs> inodes flowing under names

11:20 AM | irk
May 26, 2007

An actual complete, finished story. Could no doubt do with another hour of editing, but I'm tired and have already celebrated its being typed up with several bottles of boozahol.

It was actually written out several weeks ago, but my handwriting is so atrocious that I couldn't get over the dread of deciphering it to type it up.

It may also be that it somehow ties into this and this, but stands well enough on its own (I hope) to let it be.

9:49 PM | Writing
May 31, 2007

< bda> Every block is checksummed / every block is duplicated / when a block is corrupted / zpool becomes .. not very irate at all, actually.
< rjbs> there are hurds in this world
< rjbs> there are reisers
< rjbs> there are ext3's and coda's and then
< rjbs> there are those who worship solaris
< rjbs> but i've never been one of them
< rjbs> oh, I'm a x f esser
< rjbs> and I have been since before I was fscked
< rjbs> if there's one thing they say about big blue
< rjbs> it's that there is no firing risk

10:26 AM | irk

Peter Watts, whose works I seem unable to shut up about it, is up for a John W. Campbell award.

I've read a handful of the books on the list, and have looked at most of them on amazon. Social consumerism is really creepy.

That list also reminds me that I'm really looking forward to the next book in the Jump 225 series.

7:47 PM | Books

One of my major blocking tasks right now is to rebuild our Perl tree (about 600 modules) as PARs for easy distribution. I'm not using SRV4 or pkgsrc packages for them because I want to have one build system for both our Linux and Solaris systems. Much of our code relies on version-specific behaviors, so the only way for me to actually get stuff ported (without going totally batshit) from the entrenched Linux systems to the Solaris boxes is to rebuild all those modules, at those versions.

Yesterday, I spent most of the day compiling one Perl module (Math::Pari, which relies on the pari math libraries, and engages me in an indecent amount of skullfuckery whenever I try to build it. This particular adventure into stupidity was caused mainly by braindead pkgsrc dependencies... pari relies on teTeX -- to build its documentation -- which relied on X11. I spent a good portion of time trying to get it working the way it wanted until finally just ripping the X bits out. Guess how long that took. Yup. Bare minutes.). Finally I just built pari by hand to /opt and linked Math::Pari against that. Could have saved hours and hours...

Spent all of today building the rest of our Perl modules. Got down from ~600 to 67. Pulling the modules was easy enough was hdp mentioned the by-module listing on the CPAN. The vast majority of modules were well-behaved; it was simply a matter of iterating over the modules, running

perl Makefile.PL --skipdeps && \
make && make test && \
cd blib \
zip -r $DEST/$MODULE-i386-solaris-5.8.8.par *

and then installing it. Not a big deal. Some of them were tenacious and obnoxious, though, and ate up a lot of time. We can theoretically (this has not proven to be completely true) skip deps as any dependencies should exist in our local tree. I'm sure once I'm done I'll have to check to make sure all the modules actually have their deps, but the vast majority should.

Tomorrow I get to finish this up and then maybe get some working code running in some zones. Huz-freakin'-zah.

8:05 PM | Systems Administration