"That which is overdesigned, too highly specific, anticipates outcome; the anticipation of outcome guarantees, if not failure, the absence of grace."
-- William Gibson, All Tomorrow's Parties
January 22, 2006

So for a while now I've been trying to find a decent of rotating virtual host logs... finally tonight I got around to spending the twenty minutes to write a couple little scripts to deal with it for me.

There's a few problem with Apache logs, especially if you've got logs dumping on a per-vhost basis, and are running some form of stat generation against them. Previously I was doing an extraordinarily lame ghetto hack, with every vhost having two entries in /etc/newsyslog.conf, one for access log, the other for error, and doing a svc -t /service/httpd after every log rotation (svc is part of djb's daemontools, -t signifies you want to HUP the service). Needless to say, that's pretty crap; more or less the same as doing a graceful n times a night.

The rotatelogs program seems to be far from awesome, seeing as how it doesn't seem to want to fork() per-vhost, and anyway it looks like it bases rotation time from server restart... which is totally useless for cron jobs. Maybe I'm wrong. I suspect I don't care. newsyslog slipped me a twenty, so.

The solution I came up with is rather simple. I have the following script generate a newsyslog.conf for my Apache configs:


use strict;
use warnings;

use vars qw/ @ARGV @domains /;

unless ($ARGV[0]) { die; }

my $file = $ARGV[0];

open (FILE,"<$file");

while (<FILE>) {
push @domains,$_;

close FILE;

my $log_string = <<EOF;
/var/www/logs/access_log root:daemon 640 10 * \$D0 Z
/var/www/logs/error_log root:daemon 640 10 * \$D0 Z
/var/www/logs/ssl_engine_log root:daemon 640 10 * \$D0 Z
/var/www/logs/suexec_log root:daemon 640 10 * \$D0 Z

for my $domain (@domains) {
$log_string .= <<EOF;
/var/www/logs/vhosts/$domain/access_log root:daemon 640 10 * \$D0 Z
/var/www/logs/vhosts/$domain/error_log root:daemon 640 10 * \$D0 Z

print $log_string;


# /root/bin/apache_newsyslog_build.pl /root/etc/domains.txt \< /root/etc/apache-newsyslog.conf

And then in root's crontab:

00 0 * * * newsyslog -f /root/etc/apache-newsyslog.conf && /usr/local/bin/svc -t /service/httpd && /root/bin/run_webalizer.pl

Where run_webalizer.pl is:


# This script expects webalizer configs to be in the format:
# domain.com.conf

use strict;
use warnings;

my $www = "/var/www";
my $vhosts = "$www/vhosts";
my $vhosts_logs = "$www/logs/vhosts";
my $access_log = "access_log.0";

my $webalizer = "/usr/local/bin/webalizer";
my $webalizer_confs = "/root/etc/webalizer";
my $gunzip = "/usr/bin/gunzip";
my $gzip = "/usr/bin/gzip";

opendir (DIR,$webalizer_confs) or die ("Couldn't open $webalizer_confs: $!\n");
my @configs = grep (!/^\..*/, readdir (DIR));
closedir (DIR);

foreach my $domain (@configs) {
$domain =~ s/\.conf$//;
if (-d "$vhosts_logs/$domain") {
print "Found $vhosts_logs/$domain.\n";
if (-d "$vhosts/$domain") {
print "Analyzing $domain... ";
unless (-d "$vhosts/$domain/stats") {
print "Creating stats dir. ";
mkdir("$vhosts/$domain/stats"); chmod 0755, "$vhosts/$domain/stats";
if (-f "$vhosts_logs/$domain/$access_log.gz") { my $decompress = `$gunzip $vhosts_logs/$domain/$access_log.gz`; }
my $analyze = `$webalizer -c $webalizer_confs/$domain.conf`;
if (-f "$vhosts_logs/$domain/$access_log") { my $compress = `$gzip $vhosts_logs/$domain/$access_log`; }
print "Done.\n";
else { print "Skipping $domain.\n"; }

Pretty simple.

Anyway, I guess we'll see how well it works tonight. :)