-- William Gibson, All Tomorrow's Parties
Harry brought a mailserver down to Factory last night, and we spent two hours getting it installed. The problem is something to do with the ISP we get our connection from... there is some incredible weirdness going on.
The layout is like this: We have an uplink from the ISP plugged into our external switch (who has recently been acting up --- some ports have been dying on any packets larger than 206 bytes; kudos to Eric on figuring that one out last week), a firewall with three NICs: WAN, LAN, DMZ.
The ISP has kindly given us a number of IPs... the problem is if you start swapping machines and addresses, whatever upstream hub/switch/router from our switch seems to not refresh its ARP cache. Ever. So when Harry brought his mailserver down, we found that none of our remaining IPs would route past our switch.
Andrew and I spent several hours the other night trying to figure this one out, and somehow managed to get it to work, when we installed the new firewall (OpenBSD 3.5 box). It could have just been coincidence, I really don't know. You can sit there watching arp traffic and you'll see the router ask "Who has $x?" and the host respond "Me!", and then the router proceeds to ask again. The host in question can sit there and see some traffic from the network downstairs (upstream), like netbios, other arp traffic... but not all. I suspect that's because there's some device segmenting the network, and the router and other core stuff is on one side of that, and random other stuff (like workstations and printservers) is on our side.
It's the weirdest thing and we don't have access to the ISP's equipment (obviously) to fix it.
So last night Harry and I ran into this problem, and eventually I just gave up on trying to force a refresh on the router, or whatever the hell is upstream of us (I wish I were cool enough to figure out timing to accurately map a network of transparent devices...), and just plugged his mailserver into the (so far unused, though this will change once we figure out the arp thing) DMZ port of the firewall, and just port forwarded for it. So ghetto, but it worked fine, and I would have felt worse if he had had to take his machine home.
The only hitch was me forgetting the following rule while hacking out the new pf rules:
pass in log on $wan_if inet proto tcp from any to 10.1.1.2/32 port $smtp_services keep state
Because I'd forgotten how pf translated IPs, or more specifically, when.
And then Harry ran into one or two problems getting the ldap server on the machine up... back he quickly fixed that.
It was pretty nice debugging with Harry, actually. He knows stuff, and we swapped position at my laptop to work on the various problems without any issues. Sort of like eXtreme systems administration or something. :)