"That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace."
-- William Gibson, All Tomorrow's Parties
November 16, 2004

pyopenbsd, a set of Python classes for interfacing with OpenBDS and associated libs.

<@newsham> awesome. now you dont have to be a C programmer to enjoy the diverging APIs of unix systems!!

Had an idea to do the same thing with a set of Perl modules. Got so far as registering the namespace on the CPAN before getting distracted.

This was, of course, six months ago.


November 30, 2004
December 18, 2004

Stayed at work late last night with the intention of rebuilding the LAN firewall, and replacing the router with an OpenBSD box.

Unfortunately I had to move the mx to get at the firewall... and the mx has a really twitchy root drive, which finally decided to kill itself. Manged to pull the passwd file and some of the postfix configs off.

The upshot of this is that I spent about five hours building machines and migrating users and data.

  • Wrote a really lame Linux (Sixth Edition) password file to BSD master.passwd converter. I had been up since 0630 could barely see by this point, which was kind of fun. The awk line was ripped off from this.
  • Documented my default actions during an OpenBSD install.

Got everything up and running (and users shouldn't notice any changes, except being asked to save a new cert if they're using pop3s) around 0330.

Walked to the train station and saw more creepy people in Camden last night than I think I have in four years of late nights. Got home around 0445 and slept for six hours before my next door neighbor decided it was a good time to start shooting aliens and woke me up.

(Good news is today is Sophy and Adam's potluck.)

Let me know what you think about the obsd install doc (though it's more script than doc). I'm not sure about the harden_obsd.pl script any more, but.

January 3, 2005

Someone spammed this to misc@openbsd yesterday. Unattended OpenBSD install media. Awesome. Will definitely be playing with this once I get home.

Pulling the config based on system stuff is definitely something I might be interested in working on as well.

January 4, 2005

I just added the metawire.org Apache logs to newsyslog.conf:

find /var/www/logs -name "*_log" |sort |sed 's/$/ root:daemon 640 10 * 24 Z "apachectl stop ; apachectl start"/' >> /etc/newsyslog.conf

newsyslog -v -f /etc/newsyslog.conf

And it took a good few minutes, as they've never been rotated and weighed in around 1.2G. The loadavg kicked up to 80 while the files were being compressed, which was pretty entertaining.

A more sane solution to the apachectl command above would be a script that stops Apache, waits until any httpd-related ports aren't being returned by netstat, and then start it back up.

January 26, 2005

I just got done fumbling around creating a ccd on OpenBSD; spent about an hour on it, or a little more.

Background: This is a machine I'm sure I've complained about in the past. Gateway "server" with three dead IDE busses. In its current iteration, it's meant to be used as a mirror of our production data and server backups. These will get taped off nightly.

I "repurposed" a 200G SCSI drive that had been hanging off the O2000 a couple months ago. But it'd been laying on the server room floor (sigh) for a while, so it was up for grabs. I didn't realize it was 200G until I mounted /vol/scratch, though. Bit of a shock.

Anyway, creating a ccd is super trivial. It's in GENERIC, so there's no need to recompile. By default, you have four available ccd's (ccd0-ccd3).

First, create disklabels on the component devices. Make sure your track offset is 2. This is what bit my ass for over an hour, because I wasn't thinking.

I had to read this to actually get it. And then it was all made clear.

Anyway, this machine was meant to eat four 200G IDE drives, but there's no way I can fit the fourth drive in there; the IDE cables just won't have it. If I had some velcro I could ghettohack it, but I haven't got any. So, anyway.

Once you have your diskabels made, it's just a matter of:

[root@dua]:[~]# cat /etc/ccd.conf
# $OpenBSD: ccd.conf,v 1.1 1996/08/24 20:52:22 deraadt Exp $
# Configuration file for concatenated disk devices
# ccd ileave flags component devices
#ccd0 16 none /dev/sd2e /dev/sd3e
ccd0 16 none /dev/wd0a /dev/wd1a /dev/wd2a

[root@dua]:[~]# ccdconfig -C
[root@dua]:[~]# ccdconfig -g
ccd0 16 8 /dev/wd0a /dev/wd1a /dev/wd2a

ccdconfig creates a non-zero partition table... "c", which is usually used to symbolize the whole disk is in this case a whole partition encompassing the full disk.

If you want to cut the ccd up into smaller partitions:

disklabel -E ccd0

and use the "z" command to zero the partitions and then create your partitions as you normally would. The FAQ fails to mention this, and it was not immediately obvious to me (but that's probably simply because I'm stupid and miss the obvious as times). ccd(4) and ccdconfig(8) do not mention it either, though, so...

Anyway, once you have your partitions set up:

[root@dua]:[~]# newfs /dev/ccd0c
[root@dua]:[~]# mount /dev/ccd0c /vol/backups/dam
[root@dua]:[~]# df -h |grep dam
/dev/ccd0c 550G 2.0K 522G 0% /vol/backups/dam

Pretty easy.

January 27, 2005

So I'm installing the machine that will replace both hastur and ligur, named ligur Mk II. I'm installing postfix, and when it pulls the tls/ipv6/pf patch, it throws a checksum error. "What the hell," says I, and grab an md5 of the file. Sure enough, it doesn't match the checksum listed in distinfo. So I go check on another box, and sure enough... so then I uncompress the two patches, get digests, and they're the same. I copy the patches to a third machine and diff the "bad" and known good patches. No differences.

Same filesize, same chars, same digest. So I recompress the "good" patch on the working box, and copy it over the new box. Same checksum error.

After a few minutes of screwing around, I think to myself...

[bda@selene]:[~]$ touch foobar ; gzip foobar ; md5 foobar.gz
MD5 (foobar.gz) = 36b0031ef3f51c3ceaa0700d8546de41
[bda@selene]:[~]$ rm foobar.gz; touch foobar ; gzip foobar ; md5 foobar.gz
MD5 (foobar.gz) = 997d552d8d6835a6f2b4ea719ba350d5

Apparently gzip flips bits as part of its compression algo. Useful so you know if a file has been recompressed (which must have happened on the mirror I pulled the patch from originally).

April 23, 2005

Well, the ligur to crowley migration seems to be mostly over. Just a few little things left to do but it's answering DNS, serving web pages, and eating mail, which in my book means it's pretty much done.

As a few of the users aren't used to BSD (crowley runs OpenBSD) but Linux, there are some issues there. :)

Installing amavis was just as big a pain in the ass as I remember, but I just dug out my link to the Fairly Secure Anti-Spam Wiki and ran with it. Some modifications to their stuff... I need to clean up the script I generated to actually install the stuff, but eh.

Took about four hours to set up the new box and move everything over, I think (data had already been getting sync'd). Meh.

May 20, 2005

The gutted PowerMac on the floor *was* running a Tiger beta, which had some interesting issues (DNS would stop resolving. AFP enjoyed eating a CPU and not responding to requests -- taking any other hosts which had it mounted with it), so I figured I would reinstall it tonight.

Realized that I would either have to find a DVD drive, or pull the root disk out and plug it into one of my other Macs and install it via firewire...

Figured it was all too much of a pain in the ass and OpenBSD is just so much easier to deal with.

I love OS X, but freakin' Finder should have been replaced in 10.4. Punkass bitches.

May 21, 2005

From the BitTorrent FAQ.

On some unices, BSD libc has a bug that causes BitTorrent to be very processor intensive. Run the client with the "--enable_bad_libc_workaround 1" option to fix this.

Apparently OS X/Darwin is not one of those libcs, but OpenBSD is. Good to know.

July 16, 2005

Someone emailed in response to this misc@openbsd post asking for pointers on getting AV and spam filtering running on OpenBSD. I've gone ahead and cleaned my notes up slightly and dumped them in my scripts dir...

Here are my amavis install on OpenBSD notes.

As I've said before, I've used the Fairly Secure Anti-Spam Wiki as a basis.

Like I told Charles... YMMV. :-)

I've become a big proponent of TRAC in the last month or so. It's a very simple, very efficient project management system and svn client. It's good stuff. Many projects (including Catalyst) have adopted it.

I got bored this morning and decided to install a personal copy on mnet, which required installing mod_python and setting up a bunch of other junk for it.

So here are some more "Installing stuff on OpenBSD" docs:

Installing mod_python on OpenBSD
Installing TRAC on OpenBSD

If you find any issues with them, drop me a line.

July 19, 2005

dmesg getting filled with SCSI media error garbage and screwing with line output can cause pretty funny things:

SCOpenBSD 3.6-stable (GENERIC) #1: Thu Jan 13 07:57:07 EST 2005

September 5, 2005

Couple security fixes in OpenSSH 4.2 so it was time to go on an update spree. I have:

  1. breen
  2. gordon
  3. kleiner
  4. citadel
  5. philtered
  6. ghetto
  7. valve
  8. conduit
  9. punchclock
  10. hyperion
  11. gibson
  12. hastur

A few of those are still running 3.6, and OSSH 4.2 hit 3.6 and 3.8 a few days ago, so they were already updated. But overall? 10 minutes to update those hosts (counting cvsup time), manually, with no script (which would be trivial to do).

Nowhere near the number of machines I had while working at DCI, but there I would have just scripted the updates.

And of course now I have to wait for the few Debian boxes I still maintain, whenever the debsec team releases a package... grr.

[root@kleiner]:[~]# cvsup -g /etc/cvs-supfile
[root@kleiner]:[~]# cd /usr/src/usr.bin/ssh
[root@kleiner]:[/usr/src/usr.bin/ssh]# make clean && make depend \
 && make && make install
[root@kleiner]:[/usr/src/usr.bin/ssh]# cp ssh_config sshd_config /etc/ssh
[root@kleiner]:[/usr/src/usr.bin/ssh]# pkill -f /usr/sbin/sshd
[root@kleiner]:[/usr/src/usr.bin/ssh]# /usr/sbin/sshd

If you made changes to the ssh config files you might want to do a little diff action.

And test.

[bda@eos]:[~]$ ssh kleiner
Last login: Mon Sep  5 23:50:48 2005 from
OpenBSD 3.7-stable (GENERIC) #0: Thu Aug 25 16:30:04 EDT 2005

[bda@kleiner]:[~]$ ssh -V
OpenSSH_4.2, OpenSSL 0.9.7d 17 Mar 2004

Teh yay.

January 21, 2006

pkg_find is a nice little shell script from Michael Erdely which lets you search packages for a given string and returns a list of matching packages (one of which you can then choose to install). It keeps a local copy of the index, and updates it every n days. There is a port tarball available.

It's only a couple hundred lines, and replaces the ghetto manual index grep I've been doing for a while. I kept meaning to write something exactly like this, but yay apathy. According to the comments on the post, he wrote it to get away from exactly that. :)

The next version of pkg_add will include -i, which will apparently do the same thing. Marc Espie has been kicking ass with the package system.

I should probably start tracking -CURRENT somewhere.

January 22, 2006

untaring src.tar.gz into /usr instead of /usr/src kind of sucks.

It sure is a good thing OpenBSD's ftp is statically compiled. Just had to grab base38.tgz to another box (gzip likes to link to libraries more than directories; blowing out /usr/lib kind of sucks), uncompress it, copy it over to the hosed box, and untar it in /tmp. Copy /usr/local to /tmp/usr/local, as local shouldn't have been touched, then just get rid of the hosed /usr and copy over /tmp/usr.

The other solution would be to build the new /usr on a fresh partition (I like to have one or two spare at the end of the root disk), and then swap them.

Note to self: Have a static sshd and ssh handy. And start using screen-static instead of screen. :)

So for a while now I've been trying to find a decent of rotating virtual host logs... finally tonight I got around to spending the twenty minutes to write a couple little scripts to deal with it for me.

There's a few problem with Apache logs, especially if you've got logs dumping on a per-vhost basis, and are running some form of stat generation against them. Previously I was doing an extraordinarily lame ghetto hack, with every vhost having two entries in /etc/newsyslog.conf, one for access log, the other for error, and doing a svc -t /service/httpd after every log rotation (svc is part of djb's daemontools, -t signifies you want to HUP the service). Needless to say, that's pretty crap; more or less the same as doing a graceful n times a night.

The rotatelogs program seems to be far from awesome, seeing as how it doesn't seem to want to fork() per-vhost, and anyway it looks like it bases rotation time from server restart... which is totally useless for cron jobs. Maybe I'm wrong. I suspect I don't care. newsyslog slipped me a twenty, so.

The solution I came up with is rather simple. I have the following script generate a newsyslog.conf for my Apache configs:


use strict;
use warnings;

use vars qw/ @ARGV @domains /;

unless ($ARGV[0]) { die; }

my $file = $ARGV[0];

open (FILE,"<$file");

while (<FILE>) {
push @domains,$_;

close FILE;

my $log_string = <<EOF;
/var/www/logs/access_log root:daemon 640 10 * \$D0 Z
/var/www/logs/error_log root:daemon 640 10 * \$D0 Z
/var/www/logs/ssl_engine_log root:daemon 640 10 * \$D0 Z
/var/www/logs/suexec_log root:daemon 640 10 * \$D0 Z

for my $domain (@domains) {
$log_string .= <<EOF;
/var/www/logs/vhosts/$domain/access_log root:daemon 640 10 * \$D0 Z
/var/www/logs/vhosts/$domain/error_log root:daemon 640 10 * \$D0 Z

print $log_string;


# /root/bin/apache_newsyslog_build.pl /root/etc/domains.txt \< /root/etc/apache-newsyslog.conf

And then in root's crontab:

00 0 * * * newsyslog -f /root/etc/apache-newsyslog.conf && /usr/local/bin/svc -t /service/httpd && /root/bin/run_webalizer.pl

Where run_webalizer.pl is:


# This script expects webalizer configs to be in the format:
# domain.com.conf

use strict;
use warnings;

my $www = "/var/www";
my $vhosts = "$www/vhosts";
my $vhosts_logs = "$www/logs/vhosts";
my $access_log = "access_log.0";

my $webalizer = "/usr/local/bin/webalizer";
my $webalizer_confs = "/root/etc/webalizer";
my $gunzip = "/usr/bin/gunzip";
my $gzip = "/usr/bin/gzip";

opendir (DIR,$webalizer_confs) or die ("Couldn't open $webalizer_confs: $!\n");
my @configs = grep (!/^\..*/, readdir (DIR));
closedir (DIR);

foreach my $domain (@configs) {
$domain =~ s/\.conf$//;
if (-d "$vhosts_logs/$domain") {
print "Found $vhosts_logs/$domain.\n";
if (-d "$vhosts/$domain") {
print "Analyzing $domain... ";
unless (-d "$vhosts/$domain/stats") {
print "Creating stats dir. ";
mkdir("$vhosts/$domain/stats"); chmod 0755, "$vhosts/$domain/stats";
if (-f "$vhosts_logs/$domain/$access_log.gz") { my $decompress = `$gunzip $vhosts_logs/$domain/$access_log.gz`; }
my $analyze = `$webalizer -c $webalizer_confs/$domain.conf`;
if (-f "$vhosts_logs/$domain/$access_log") { my $compress = `$gzip $vhosts_logs/$domain/$access_log`; }
print "Done.\n";
else { print "Skipping $domain.\n"; }

Pretty simple.

Anyway, I guess we'll see how well it works tonight. :)

May 30, 2006

Friday I installed OBSD 3.9 on two Dell 1850s and configured CARP and pfsync. It was amazingly trivial. If you need failover systems of pretty much any sort, this is the way to go.

To quote the OpenBSD FAQ page:

CARP is the Common Address Redundancy Protocol. Its primary purpose is to allow multiple hosts on the same network segment to share an IP address. CARP is a secure, free alternative to the Virtual Router Redundancy Protocol and the Hot Standby Router Protocol.

It takes about five minutes to set up, and about fifteen minutes playing "plug/unplug the systems and watch the ifconfig state change, tee-hee!". Kind of like that episode of the Simpsons where Homer keeps pulling on the pig's tail.

"Curly! Straight! Curly! Straight!"

Only CARP just does what it does instead of biting your face off like a certain piggy.

pfsync is, simply, a way to sync your firewall state tables to a group of hosts on a trusted network of some sort. So when your primary firewall/proxy/whatever dies, and a backup kicks in, your users don't notice anything -- they don't lose their sessions. Quite awesome.

Firewall Failover with pfsync and CARP

PF: Firewall Redundancy with CARP and pfsync

The PF page there is pretty much all you need. Getting it working is maddenly easy and it Just Works.

August 21, 2006

So I was putting together a test backup server using rdiff-backup last week, and I wanted to (for some strange reason) backup up the various OpenBSD machines I have installed since starting there.

It's pretty trivial:

pkg_add popt
pkg_add -i python

wget http://easynews.dl.sourceforge.net/sourceforge/librsync/librsync-0.9.7.tar.gz
wget http://savannah.nongnu.org/download/rdiff-backup/rdiff-backup-1.0.4.tar.gz

tar -xzf librsync-0.9.7.tar.gz
cd librsync-0.9.7
make all check
make install

tar -xzf rdiff-backup-1.0.4.tar.gz
cd rdiff-backup-1.0.4
python setup.py install --prefix=/usr/local --librsync-dir=/usr/local

If you are using 64-bit hardware, you'll need to use use --with-pic for librsync

The next step is to involved a hacked up version of the littlest backup wrapper script that could, resync 0.3, and bang, done.

rdiff-backup is pretty sweet. Check out the examples, this howto on unattended backups, maybe this arstech article, and this here wiki.

I need to clean up resync a bit (getting it back in VCS will give me an excuse to try out git, too) and then I'll throw it up on code.

November 23, 2006

I don't really want to get into why I did this (let's just say today has sucked), but you can change MAC addrs on OBSD since 3.8 without digging up `sea.c`.

# ifconfig bge0 lladdr 0a:0b:0c:0d:0e:0f


November 3, 2007

onlamp has an interview with the OpenBSD devs on what's new 4.2. Basically: Lots.

The highlights for me are doubled pf performance, IP load-balancing with CARP, and layer 7 hostated support (with HTTP/SSL hackery). Marc Espie's continuing improvements to the package management system are also no doubt going to continue making my life easier.

All in all, a very exciting release! Go buy your CD!

June 6, 2008