"That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace."
-- William Gibson, All Tomorrow's Parties
Another broken morning.

So this morning has been... annoying.

A box was rebooted and didn't come back up. Network came up (pingable) but not ssh. Based on previous idiocy with this system, I suspected it had something to do with filesystems not being able to mount at boot. I shot off a mail to the NOC monkeys, not expecting much (and four hours later, still no response from them), and then started trying to get into the system myself to fix it.

The box in question is a Sun X4150; a really nice system (though now that I've had a T5120 for a while, I have to say I really do much prefer SPARCs simply for ease of administration), with a really lame-ass LOM (ELOM). But: Whatever. So I go to start the console via the LOM... no joy. Apparently console is not redirecting. So, ok, I should be able to get at glass (thanks for the reminder, dlg) via the web interface.

Of course there's no VPN at that site. So I kick open netcat and don't have much in the way of luck. After a few minutes of screwing around with it, I give up and download haproxy. In about three minutes I have it compiled, configured, and forwarding :80 and :443 for me.

listen proxy1 0.0.0.0:80
mode http
balance roundrobin
server test 192.168.11.10:80
contimeout 3000
clitimeout 150000
srvtimeout 150000
maxconn 60000
redispatch
retries 3
grace 3000
option forwardfor
option httplog
option dontlognull

listen ssl-relay 0.0.0.0:443
option ssl-hello-chk
balance source
server inst1 192.168.11.10:443

I log into the LOM, start the redirection Java app, and... nothing.

And... Mac OS X Java bullshit.

So I start an old Parallels OpenSolaris image I had laying around, connect to the LOM that way and... get an I/O connection error. Figuring that the KVM was running on another port, I sniffed off my firewall and discovered that yes, it wanted :8890 as well.

listen ssl-relay 0.0.0.0:8890
option ssl-hello-chk
balance source
server inst1 192.168.11.10:8890

Did that, got into the box and discovered the problem was...

[20080813-05:43:12]:[root@brood]:[~]# tail -2 /etc/vfstab
/dev/zvol/dsk/data/zones/lb-arc/root /dev/zvol/rdsk/data/zones/lb-arc/root /zones/lb-arc ufs 1 yes logging
/dev/zvol/dsk/data/zones/lb-arc/root /dev/zvol/rdsk/data/zones/lb-arc/root /zones/lb-arc ufs 1 yes logging

ugh.

A svcadm clear filesystem/local later, and all was well.

sigh.

August 13, 2008 5:43 AM