"That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace."
-- William Gibson, All Tomorrow's Parties
February 8, 2007

Continuing on my "pkgsrc is pretty awesome" schtick, here's how easy it is to get it running on OS X. The most annoying part is downloading XCode (if you haven't got it already).

Once you have the DevTools/XCode installed, you'll need to create a case-sensitive volume for pkgsrc. Until somewhat recently you couldn't resize volumes, and this would have been far more annoying. However, these days it's pretty trivial.

Another option would be to create a disk image and run pkgsrc out of that. The documentation suggests this course. It has the added bonus of being portable (pkgsrc on your iPod?); I'm just doing this on my laptop and don't want to have to deal with managing a dmg, so I'm going the resize route.

[root@selene]:[~]# diskutil list
/dev/disk0
#: type name size identifier
0: GUID_partition_scheme *74.5 GB disk0
1: EFI 200.0 MB disk0s1
2: Apple_HFS selene 74.2 GB disk0s2
[root@selene]:[~]# diskutil resizeVolume disk0s2 70G "Case-sensitive Journaled HFS+" pkgsrc 4.2G

Once it's done resizing the volume, you'll need to reboot.

The reboot is required because we're monkeying around with the boot volume. If you're doing this on an external disk, you can just refresh diskarbitrationd with disktool -r.

[root@selene]:[~]# diskutil list

/dev/disk0
#: type name size identifier
0: GUID_partition_scheme *74.5 GB disk0
1: EFI 200.0 MB disk0s1
2: Apple_HFS selene 70.0 GB disk0s2
3: Apple_HFS 4.1 GB disk0s3

Well, leetsauce, our volume exists. But it's not mounted, because resizeVolume doesn't actually format it.

[root@selene]:[~]# diskutil eraseVolume "Case-sensitive Journaled HFS+" pkgsrc disk0s3
Started erase on disk disk0s3

Erasing

Mounting Disk

Finished erase on disk disk0s3 pkgsrc

And now it shows up happily:

[root@selene]:[~]# mount
/dev/disk0s2 on / (local, journaled)
devfs on /dev (local)
fdesc on /dev (union)
on /.vol
automount -nsl [182] on /Network (automounted)
automount -fstab [189] on /automount/Servers (automounted)
automount -static [189] on /automount/static (automounted)
/dev/disk0s3 on /Volumes/pkgsrc (local, journaled)

And is it indeed case-sensitive:

[root@selene]:[/Volumes/pkgsrc]# touch foo Foo
[root@selene]:[/Volumes/pkgsrc]# ls -l ?oo
-rw-r--r-- 1 root wheel 0 Feb 8 02:09 Foo
-rw-r--r-- 1 root wheel 0 Feb 8 02:09 foo

If you care to, you can change the mountpoint using Netinfo Manager, but I'm lazy and just symlinked /Volumes/pkgsrc -> /usr/pkg. You could also change --prefix=/Volumes/pkgsrc, but as I'm using it on other OSes, I like having it all in the same place. Personal preference.

(When Netinfo Manager stops SPODing at startup I'll probably change the mountpoint.)

Anyroads:

[root@selene]:[~]# cd /usr/pkg
[root@selene]:[/usr/pkg]# curl -O ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-2006Q3/pkgsrc-2006Q3.tar.gz
...
[root@selene]:[/usr/pkg]# tar -xzf pkgsrc-2006Q3.tar.gz
[root@selene]:[/usr/pkg]# cd pkgsrc/bootstrap
[root@selene]:[/usr/pkg/pkgsrc/bootstrap]# ./bootstrap

And off it goes.

...
===> bootstrap started: Thu Feb 8 02:37:50 EST 2007
===> bootstrap ended: Thu Feb 8 02:40:15 EST 2007
[root@selene]:[/usr/pkg/pkgsrc/bootstrap]# mkdir ../../etc/
[root@selene]:[/usr/pkg/pkgsrc/bootstrap]# cp /usr/pkg/pkgsrc/bootstrap/work/mk.conf.example ../../etc/mk.conf

Prepend /usr/pkg/bin:/usr/pkg/sbin to your $PATH.

The main reason I did this was to upgrade vim to 7.x, because I want tab support (I never really got the hang of managing buffers, but to the dismay of all my Elite Vim hax0r Friends).

So:

[root@selene]:[/usr/pkg]# cd pkgsrc/editors/vim
[root@selene]:[/usr/pkg]# bmake package

And after downloading a few billion patches and compilation...

[root@selene]:[/usr/pkg/pkgsrc/editors/vim]# which vim
/usr/pkg/bin/vim
[root@selene]:[/usr/pkg/pkgsrc/editors/vim]# vim --version
VIM - Vi IMproved 7.0 (2006 May 7, compiled Feb 8 2007 02:59:09)

Whoo.

June 2, 2008

The other day I ran into an issue where bootstrapping pkgsrc 2008q1 would hang while running bmake regression tests.

The fix is here.

August 27, 2008

Recently I moved our x86-64 pkgsrc build zone to another system. When I did so, I had forgotten I had built the original zone as full, to get around an annoying install(1M) bug. Basically, when you tried to build a package, it would attempt to recursively mkdir /usr/pkg. On sparse zones, /usr is shared read-only from the global zone.

So the install would fail, because it couldn't create /usr for obvious reasons. At the time, I thought I had tried various install programs, but given that the problem was being re-addressed and I didn't feel like reprovisioning a zone, I figured I would tackle it again.

After some minor discussion on #pkgsrc and grepping through mk/ I "discovered" the following variable:

TOOLS_PLATFORM.install?= /usr/pkg/bin/ginstall

Added to mk.conf and all is good. Mainly because ginstall actually uses mkdir -p, so...

The contents of pkgsrc/mk/platform/ are very useful if you aren't on NetBSD.

March 18, 2009

So Linux has a history of hosed db interfaces. Apache worked around this about ten years ago by including their own SDBM in their distribution.

pkgsrc separates their Apache packages into DSOs. So mod_perl, mod_fastcgi, mod_ssl, etc, are built as separate packages. However, when you compile Apache1 with no SSL, it disables SDBM, so mod_ssl (which requires some sort of DBM) fails.

The PR is here.

My workaround was to do this:

ap-ssl$ bmake patch

ap-ssl$ vi /usr/pkg/pkgsrc/www/ap-ssl/work/mod_ssl-2.8.31-1.3.41/pkg.sslmod/libssl.module

Search for the first instance of APXS.

Add the following two lines above it:

APXS_MODE="yes"

my_rule_SSL_SDBM="yes"

And ap-ssl will compile happily.

July 1, 2009

Someone on Sun managers asked for advice on moving from Linux to Solaris and tips on living with Solaris in general. I guess I kind of have a lot to say about it, actually..

One thing I forgot to mention is using SMF. You may have two software repositories (Sun's and pkgsrc), but you only want one place to manage the actual services. Write SMF manifests! It's easy, and you can use puppet to manage it all.

From: Bryan Allen <bda@mirrorshades.net>
To: Jussi Sallinen
Cc:
Bcc:
Subject: Re: Looking for tips: Migrating Linux>Solaris10
Reply-To: bda@mirrorshades.net
In-Reply-To: <20090624113312.GA32749@unikko>
AIM: packetdump

+------------------------------------------------------------------------------
| On 2009-06-24 14:33:12, Jussi Sallinen wrote:
|
| Im new to Solaris and about to start migrating Linux (Gentoo) based E450 server
| to V240 Solaris 10.
|
| Currently running:
|
| -Apache2
| -Postfix
| -Dovecot
| -MySQL
|
| About 70 users using WWW and email services.
|
| So, to the point:
| In case you have tips and tricks, or good to know stuff please spam me with
| info regarding migration.

A quick note: I work for a company where I migrated all our services from Linux
on whiteboxes to Solaris 10 on Sun hardware. It was a major effort, but
garnered us many benefits:

* Consolidation. Thanks to the faster harder and Zones, we are down from 50+
Linux boxes to a dozen Sun systems. And for honestly not that much money.
* Much greater introspection (not just only mdb or DTrace; the *stat tools are
just that much better)
* Before ZFS, we were mostly sitting on reiserfs (before my time) and XFS
(which I migrated as much as I could to before getting it on ZFS). ZFS has
been a huge, huge win in terms of both reliability and availability.

This turned out to be quite an article, but here are some "quick" thoughts on
using Solaris particularly, and systems administration in general:

* Read the System Administrator Guides on docs.sun.com if you are new to
Solaris
* No, seriously. Go read them. They are incredibly useful and easy to parse.
* Follow OpenSolaris development, either via the mailing lists or #opensolaris
on freenode. This gives you a headsup and stuff that might be getting into
the next Solaris 10 Update, so you can plan accordingly.

* Use a ZFS root instead of UFS (text installer only, but you really want to
use JET -- see below)
* Use rpool for operating system and zoneroots only
* Set up a tank pool on seperate disks
* Delegate tank/filesystems to zones doing the application work

This minimizes the impact of random I/O on the root disks for data and vice
versa (just a good practice in general, but some people just try to use a
single giant pool).

It also negates the issue where one pool has become full and is spinning
platters looking for safe blocks to write to impacting the operating system or
application data.

* Use Marin Paul's pca for patching

The Sun patching tools all suck. pca is good stuff. You get security and
reliability patches for free from Sun; just sign up for a sun.com account.

You don't usually get new features from free patches (you do for paid patches),
but regardless all patches are included in the next system Update.

* Learn to love LiveUpgrade

With ZFS roots, LiveUpgrade became a lot faster to use. You don't have a real
excuse anymore for not building an alternative boot environment when you are
patching the system.

Some patches suck and will screw you. Being able to reboot back into your
previous boot environment is of enormous use.

* Use NetBSD's pkgsrc

Solaris 10 lacks a lot of niceties you and your users are going to miss.
screen, vim, etc. You can use Blastwave, but it has its own problems. pkgsrc
packages will compile basically everything without a problem; they are good
quality, easy to administer, and easy to upgrade.

If you aren't doing this on a single box, but several machines, you would have
a dedicated build zone/host, and use PKG_PATH to install the packages on other
systems. Since you are using a single machine, see below about loopback
mounting the pkgsrc directory into zones: Compile once, use everywhere.

The services you listed are available from pkgsrc and work fine. The one thing
you might want to consider instead is using Sun's Webstack and the MySQL
package, as they are optimized for Solaris and 64bit hardware.

In addition to the above, we use pkgsrc on our (dwingling number of) remaining
Linux hosts. It means we have a *single version* of software that may be
running on both platforms. It segments the idea of "system updates" and
"application updates" rather nicely with little overhead.

* Use Solaris Zones

Keep the global zone as free of user cruft as possible. If you segment your
services and users properly, zones make it incredibly easy to see what activity
is going on where (prstat -Z).

It also makes it easy to manage resources (CPU, RAM) for a given set of
services (you can do this with projects also, but to me it's easier to do at
the zone level).

Install all your pkgsrc packages in the global zone and loopback mount it in
each zone. This saves on space and time when upgrading pkgsrc packages. It also
means you have one set of pkgsrc packages to maintain, not N. It's the same
concept as...

* Use Sparse Zones

They are faster to build, patch and manage than full root zones. If you have
recalcitrant software that wants to write to something mounted read-only from
the global zone, use loopback mounts within the global zone to mount a zfs
volume read-write to where it wants (e.g., if something really wants to write
to /usr/local/yourface).

I also install common software in the global zone (e.g., Sun's compiler,
Webstack or MySQL) and then loopback mount the /opt directory into each zone
that needs it (every zone gets SSPRO).

* Delegate a ZFS dataset to each zone

This allows the zone administrator to create ZFS filesystems inside the zone
without asking the global admin. Something like rpool/zones/www1/tank. It's
easier to manage programmically too, if you are using something like Puppet
(see below) to control your zones. You only have to edit a single class (the
zones) when migrating the zone between systems.

* Use ZFS Features

No, really. Make sure your ZFS pools are in a redundant configuration! ZFS
can't automatically repair file errors if it doesn't have another copy of the
file.

But: ZFS does more for you than just checksumming your data and ensuring it's
valid. You also have compression, trivial snapshots, and the ability to send
those snapshots to other Solaris systems.

Writing a script that snapshots, zfs sends | ssh host zfs recvs is trivial. I
have one in less than 50 lines of shell. It gives you streaming, incremental
backups with basically no system impact (depending on your workload,
obviously).

Note that if disk bandwidth is your major bottleneck, enabling compression can
give you a major performance boost. We had a workload writing constantly
rewriting 30,000 sqlite databases (which reads the file into memory, creates
temp files, and writes the entire file to disk -- which are between 5MB and
2GB). It was incredibly slow until I enabled compression, which gave us a 4x
write boost.

You can also delegate ZFS filesystems to your users. This lets them take a
snapshot of their homedir before they do something scary, or whatever.

* Use the Jumpstart Enterprise Tool

Even though you only have one Solaris system, if you're new to Solaris, the
chances are you're going to screw up your first couple installs. I spent months
trying to get mine just the I wanted. And guess what, installing Solaris is
time-consuming and boring.

Using JET (a set of wrappers around Jumpstart, which can also be annoying to
configure), you have a trivial way of reinstalling your system just the way you
want. I run JET in a virtual machine, but most large installs would have a
dedicated install VLAN their install server is plugged into.

Solaris installs have a concept of "clusters", which define which packages are
instaled. I use RNET, the smallest one. It basically has nothing. I tell JET to
install my extra packages, and the systems are configured exactly how I want.

You use the finish scripts to do basic configuration after the install, and
to configure the *rest* of the system and applications, you...

* Use a centralized configuration management tool

I use Puppet. It makes it trivial to configure the system programmically,
manager users and groups, and install zones. It's a life and timesaver. In
addition to making your system configuration reproducible, it *documents* it.

Puppet manages both our Solaris and Linux boxes, keeping each in a known,
documented configuration. It's invaluable.

I also store all my user skel in source control (see next), and distribute them
with Puppet. Users may be slightly annoyed that they have to update the
repository whenever they want to change ~/.bash_profile, but it will be the
same on *every* host/zone they have access to, without them doing any work,
which will make them very happy.

* Store your configs in a source control manager

Both your change management and your system configuration should all be
versioned. Usefully, you can use your change management to manage your system
configs!

We have an internal directory called /sw where we deploy all our software to.
Older services have configs hard-coded to other locations, so we use Puppet to
ensure symlinks exist as appropriate. We deploy to /sw with a script that
checks the tree out of git and rsyncs it to all machines. It's pretty trivial,
and very useful if you have more than, say, two hosts.

/sw is also a loopback mount into every zone, and read-only. It enforces the
idea that all config changes must go into the repository, *not* be changed
locally... because developers can't write to /sw just to fix something quickly.

* Solaris Sucks At: Logging, by default

The default logging setup is awful. Install syslog-ng from pkgsrc, and write
your logs to both a remote syslog server and the local disk (enable compression
on your logs ZFS filesystem!)

* Solaris Sucks At: Firewalling

ipf is a pain in the butt. Unless you absolutely have to do host-based
firewalling, set up an OpenBSD system and use pf.

...

I'm sure I could think of quite a lot more (DTrace, Brendan Gregg's DTrace
Toolkit, RBAC, mdb), but it's dinnertime. :)

Hopefully the above will prove somewhat useful!
--
bda
cyberpunk is dead. long live cyberpunk.


August 14, 2009

Co-worker asked for this. After a few minutes poking at the Makefile, I just googled and hit this page which gave me what I needed.

Yay for lazyweb.