This week is sysadmin week, in which I catch up on a few months of accumulated system administration headaches.
On Monday I went down to Peer 1 Networks' colocation facility in New York, where the main Joel on Software server lives. Peer 1 provides free bandwidth and a wee shelf for Joel on Software, for which I am extremely grateful. Michael and I installed the original server there about a year and a half ago, and it's been running fine ever since, down only because of Windows Updates. (Don't get me started.) Sometimes the server didn't come back up properly after one of the reboots required for patching Windows, so we installed a remote controlled power strip, which has a web interface allowing us to power cycle the server. There's supposed to be such a thing built into the server itself, something Dell makes called RAC, but it crashes more often than the server, requiring a full power cycle to get it back to life, which defeats the purpose…
Anyway the reason I went down on Monday was to slide in another 1U Dell server into the rack which will serve as a “hot backup” in case the main server dies. I'm going to set up some simple replication from the main server to the hot backup so we should be able to switch back and forth between the main server and the backup server without more than a few seconds of downtime. The replication will use robocopy for files and log shipping for SQL databases like the database behind the discussion group.
Peer 1, by the way, is doing incredibly well. When I installed the server there last winter they only had two rows of racks, mostly empty. On Monday when I went down there the whole data center was crammed with racks and they were turning away new customers until they could arrange for a bigger data center. Joe Cooper, the NY manager, told me they had gone from 20% to 90% capacity in their colo facility and were trying to reserve the remaining 10% for existing customers. A nice problem to have. I couldn't be happier with their hosting services and they're the nicest people, so even though I'm completely tainted since they host my site for free, I most heartily recommend them if you're looking for colocation (or wicker furniture, har dee har har). [Joel on Software]