The disaster that wasn't
You may recall some spectacular disasters in recent years where power or cooling went out in a large colocation datacenter, taking down dozens of important internet services for up to a day or more.
In 2007, the 365 Main datacenter in San Francisco experienced 45 minutes of power outage after backup generators failed to start. Even though the power came back on, the necessity to correct any file system corruption and restart complex systems consisting of multiple servers kept services like Craigslist, Technorati and TypePad down for much of the day.
In 2008, Rackspace experienced a series of outages due to power and air conditioning failures which also brought down highly visible web properties including 37signals. Some customers left Rackspace, even after being offered large discounts.
So you can imagine my panic when I was in our datacenter, hosted at CoreSite, a week ago and the lights went out. So did the rumbling CRAC units (computer room air conditioning) that keep the servers cool. A partial silence descended on the datacenter but to my relief, the servers kept running. Almost immediately, the space began to heat up, but then, within 30 seconds, the cooling and lighting came back on, and the servers never missed a beat.
This time, all the backup systems worked as expected, and as I stood there waiting for something to go wrong and it didn't, I gave thanks for CoreSite and the care they put into our datacenter when they recently upgraded it from it's original WorldCom design.
I found out later that a truck had struck a high-tension line, cutting off power to the datacenter, and that all the backup systems kicked in perfectly to carry the load while the line was being repaired.
TrackBack URI for this entry
