You may recall some spectacular disasters in recent years where power or cooling went out in a large colocation datacenter, taking down dozens of important internet services for up to a day or more.

In 2007, the 365 Main datacenter in San Francisco experienced 45 minutes of power outage after backup generators failed to start.  Even though the power came back on,  the necessity to correct any file system corruption and restart complex systems consisting of multiple servers kept services like Craigslist, Technorati and TypePad down for much of the day.

In 2008,  Rackspace experienced a series of outages due to power and air conditioning failures which also brought down highly visible web properties including 37signals.   Some customers left Rackspace, even after being offered large discounts.

So you can imagine my panic when I was in our datacenter, hosted at CoreSite, a week ago and the lights went out.  So did the rumbling CRAC units (computer room air conditioning) that keep the servers cool.   A partial silence descended on the datacenter but to my relief, the servers kept running.   Almost immediately, the space began to heat up, but then, within 30 seconds, the cooling and lighting came back on, and the servers never missed a beat.

This time, all the backup systems worked as expected, and as I stood there waiting for something to go wrong and it didn't, I gave thanks for CoreSite and the care they put into our datacenter when they recently upgraded it from it's original WorldCom design.

I found out later that a truck had struck a high-tension line, cutting off power to the datacenter, and that all the backup systems kicked in perfectly to carry the load while the line was being repaired.

Lesson 3, "Behind The Scenes in Determining the Costs of Cloud Computing" continues with this discussion of why the exact relationship between cloud performance and cost cannot be known, which makes comparing vendors somewhat difficult.  This lesson builds on the previous ones.   Please see the overview in Lesson 3.1 if you have not already read it.

Cloud vendors, like any company that has to buy computer equipment over time, cannot (and don't want to) buy the same models of servers year after year. Instead, they take advantage of improving server cost and performance over time by buying the latest models so that they can pass those improved economies to their customers or offer higher levels of performance. This causes a problem in giving the cloud customer a known amount of CPU power per dollar, since CPU power keeps rising over time. This is where the “hardware abstraction” promise of cloud computing breaks down.

Let's look at an example. Suppose that a cloud vendor promises a price of $1 per CPU-hour based on an older model server, and they buy a new server with a 20% faster CPU. They could simply offer 80% of the new CPU at the same price, and customers would get the same performance whether their software was running on the old or new server. However, this creates a fragmentation problem: what to do with the remaining 20% of the new server's CPU? To solve this problem, fixed or quantized-instance vendors create a “virtual CPU” with a certain speed rating that doesn't change over time. Amazon, for example, calls a “core” equivalent to a 2006 1-1.2 GHz Opteron. However, if the cloud vendor buys a server that doesn't have a CPU with a multiple of the power of the virtual CPU, they are faced with a choice: either divide the extra computing power among the virtual CPUs on the server, or waste the extra power.

If the vendor divides the extra resources among customers' virtual CPUs, the customer gets something near but not exactly equal to the number of virtual CPUs they have ordered. As long as the customer never uses any specialized performance measurement software, they will not know if they're getting a little more or less than they have been promised, depending on how the cloud vendor deals with the fragmentation. However, customers will always have slightly different amounts of CPU while paying the same for them.

This problem occurs for the variable resource allocation vendors as well, but they have more options in resolving the fragmentation, since at any time, some of their customers will request very small instances that can potentially use the extra CPU. However, variable resource allocation vendors have an additional problem as well: since customer's instances can be moved to new servers at any time, they have to allow for those servers to have slightly different CPU power, which may potentially also change the customer's CPU allocation slightly. One possible solution is that the vendor rates CPU power in Gigahertz, like the processor vendors (AMD, Intel) do, but this presumes that different CPUs with the same clock frequency perform equivalently, which is generally not true.

Ultimately, the customer has no easy way of validating how serious this problem is, or even if it is a problem. Vendor transparency, and the trust it creates, is the only solution to this problem.