You may recall some spectacular disasters in recent years where power or cooling went out in a large colocation datacenter, taking down dozens of important internet services for up to a day or more.

In 2007, the 365 Main datacenter in San Francisco experienced 45 minutes of power outage after backup generators failed to start.  Even though the power came back on,  the necessity to correct any file system corruption and restart complex systems consisting of multiple servers kept services like Craigslist, Technorati and TypePad down for much of the day.

In 2008,  Rackspace experienced a series of outages due to power and air conditioning failures which also brought down highly visible web properties including 37signals.   Some customers left Rackspace, even after being offered large discounts.

So you can imagine my panic when I was in our datacenter, hosted at CoreSite, a week ago and the lights went out.  So did the rumbling CRAC units (computer room air conditioning) that keep the servers cool.   A partial silence descended on the datacenter but to my relief, the servers kept running.   Almost immediately, the space began to heat up, but then, within 30 seconds, the cooling and lighting came back on, and the servers never missed a beat.

This time, all the backup systems worked as expected, and as I stood there waiting for something to go wrong and it didn't, I gave thanks for CoreSite and the care they put into our datacenter when they recently upgraded it from it's original WorldCom design.

I found out later that a truck had struck a high-tension line, cutting off power to the datacenter, and that all the backup systems kicked in perfectly to carry the load while the line was being repaired.

This posting is a summary of Lesson 3 in the Cloud 101 Class, in which different methods of allocating cloud computing resources to meet customer demand were discussed, and how those methods affected costs.  You can return to the first posting in Lesson 3 here.

• There are three major ways that vendors allow customers to request resources from the cloud: fixed instance size allocation, quantized instance size allocation, and variable instance size allocation. Each method allows customers to request computing resources according to their needs, which potentially offers significant cost savings compared to purchasing or leasing computer hardware.
• Some vendors use a fourth method in which instance size varies based on the number they pack onto their servers, which does not offer the guaranteed resources that people expect from the cloud, though it is often offered at a fixed price. This is more of a 'hosting' offering than true cloud computing.
• Fixed instance allocation such as that used by Amazon EC2 offers low instance cost, but encourages customers to over-buy instance sizes to avoid performance problems, which increases the actual cost to the customer. Responding to changes in demand requires allocating new instances, which is complex and costly. Third party vendors have arisen to solve some of these problems.
• Quantized instance allocation solves some of the problems with fixed instance allocation by giving customers more flexibility in selecting instance sizes.
• Variable instance allocation allows customers to request exactly the resources they need from the cloud, but requires them to know exactly what those requirements are. Most customers are not familiar with the techniques necessary to understand their true requirements.
• Each allocation method requires customers to understand how much resources their applications need at a far more detailed level than ever before, potentially creating the risk of performance problems or unexpected charges.
• Automatic scaling, either by adding instances or by resizing existing ones, can simplify the requirement of understanding how much resources an application needs to run successfully.  However, it can result in unpredictable bills.
• Because hardware is always changing, cloud vendors may not be able to guarantee an exact performance level per unit of cost. How serious this problem is depends on the policies of the vendor, which may not be transparent to the customer.
• Cloud vendors have to keep unused inventory of servers available for peak demand, which can increase costs. The exact amount of these costs is not generally known.
• Total Cost of Deployment, the actual cost the customer must pay to run their application, is difficult to calculate for cloud deployments (though simpler than TCD calculations for building your own datacenter.) This difficulty threatens the success of cloud computing, since customers are looking a simple way to understand their costs, not just find the lowest costs.

We often receive calls from potential customers who are looking to create a multimedia-based application or website. To our surprise, almost all of them think that they need a CDN (Content Delivery Network) to provide the multimedia file service to their customers. However, this is generally not true, in large part because the media players on modern operating systems are capable of compensating for all but the worst network latencies and performance – and the worst latencies are often in the “last mile” connection between the viewer and the internet, which cannot be addressed by the CDN. The fact that CDNs are so popular is a testament to the power of their marketing, not the necessity of their technology. Let’s take a look at an example of why a CDN won’t deliver any benefit in many cases.

In our picture below, we have a typical multimedia user in New York City, watching content served by our servers in the MAE-West NAP Network Access Point in San Jose, California. We chose MAE-West for our data centerbecause it is an internet hub and has the best connections to other hubs around the country and the world and minimizes the latency (delay) for data to get back and forth from the user’s computer to our servers. This is important for interactive applications where the client PC has a lot of short conversations with the server. Our experience building the data center for NetSuite has borne this out: they still have only one datacenter to serve their customers worldwide after 10 years in operation. We shall see that this advantageous location on the internet is part of the reason why our customers can usually do without a CDN.

The user is connected to the internet through their ISP by a typical 384kbps DSL line. Other types of ISPs which offer faster service may improve the viewing experience, but we have found that despite claims of tens of megabytes of download speed, most ISPs cannot actually deliver that speed during prolonged media downloads. So, unless the internet is severely congested, the speed of the download is determined by the “last mile” connection – the users’s connection to the public internet through their ISP. Internet congestion is relatively rare, since the internet protocol allows data to travel through multiple paths to get to a destination

To start viewing the video, the user clicks “play” on the hosted website, and a request for the media travels across the internet to our server at our point of presence (POP), which takes a few moments to find the media and start sending it. Then the first packet wends its way back across the internet to the users’s computer. This process adds up to 0.052 seconds. After that, the media player software on the users’ computer receives packets of the video until it has enough stored up (“buffered”) until it decides that it can play the video without any interruptions, at 20.91 seconds. This delay is usually determined by a desired maximum and minimum buffer size that the player calculates. After this point, playback begins, but the buffer continues to fill while the “whoa” message makes its way back to the server and the stream stops. Then, playback empties the buffer until the minimum size is reached, and the player requests more video data from the server. This goes on until all the video data is sent.

Now, what does the CDN bring to the table? A CDN places the video on a server “near” the user, often in the same data center as the ISP equipment serving the user. Actually, since the CDN doesn’t know where the next user will be, it has to place the video on many servers around the country, which becomes expensive very quickly since it will store the same data many times.

When the user clicks “play”, the request is directed to the nearby CDN server, which can get the first packets of data to the user’s computer without the delays of sending it across long stretches of the internet: the CDN has reduced the latency dramatically. However, reducing the latency doesn’t improve the user’s experience very much: since most of the delay before play begins is the 20.8 seconds to fill the buffer, the video begins to play only imperceptibly faster with the CDN than with a single-point hosted solution such as ENKI, colocation, or another cloud vendor.

If the latencies of the internet connection are sufficiently large, for example when the provider is isolated from NAP (network access points) such as MAE-West (represented by the larger network hubs in the picture), then they can lead to buffer underrun, which causes the video playback to pause.  The different latencies from a variety of vendors, and their consequences, are compared in the chart below.

Some media player software uses adaptive buffer sizing to reduce the chance that this can happen: it watches how the buffer is filling up: if it’s filling too slowly or latencies are large, it will buffer more of the video before playing; if the buffer fills quickly or latencies are small, the software will buffer less video before starting playback.

The CDN, by bypassing the public internet, can also reduce the effects of public internet congestion. Excess congestion lowers the transfer speed and can cause buffer underruns. However, as mentioned above, we see congestion relatively rarely. A more common problem that slows downloads is that the ISP’s infrastructure is overloaded due to excessive cost-cutting and lack of investment. This is especially prevalent with cable internet services, where everyone shares a “party line.” The CDN cannot help with this problem since it occurs between the CDN’s server and the user.

Since a CDN can be very expensive – 10 times your hosting cost or more – when does it become a necessity? As your service grows, you will first run into bandwidth limitations if your software architecture runs the multimedia stream through a single server or database. However, this is not a reason to switch to a CDN, since you can rearchitect your application to use a federated file storage or delivery method with multiple servers collaborating to provide the streams. The next barrier to cross is the bandwidth limitation of your provider’s POP. Many services have solved this problem by rewriting their software again to allow deployment to multiple POPs, spreading the media files among the different locations, but sharing directory information so that the user’s browser can find the file in the correct location. However, if your total volume grows beyond being able to be served by a few POPs, you will need to move the files closer to the users. And that’s when a CDN becomes imperative.

For most of our customers, this will never happen, and even for those who it may happen to, they will likely be able to go for years without needing a CDN. As an example, we have hosted a multimedia file sharing company that grew to 200,000 users with 600Mb/sec average throughput. They were able to serve all these users through two 4-core virtual instances (one running Java and one running PostgreSQL.) Their cost was a tiny fraction of what a CDN would have charged them, and they served customers throughout North America, Latin America, and even Europe. (Note that since media players are relatively insensitive to latency, even serving customers on another continent from a single POP is quite feasible.)

Great article in Network World. The article shows why Cloud Computing with a service component is the wave of the future. Not every company is going to have someone who is equipped to manage the internal infrastructure in the cloud. However in order to compete in today's business environment, virtualization is important. The companies that are succeeding are doing more with less. Which then points to a virtual infrastructure with a little added bonus, support.
The new breed of cloud provider offers the service component which enables any company to run as efficiently as a nimble web 2.0. Enjoy the article. By Denise Dubie , Network World , 05/20/2009
LAS VEGAS -- Despite the reported benefits of virtualization, a majority of IT managers polled at Interop this week say they experience problems with the technology and don't always realize the cost benefit.
Network Instruments polled 120 network managers, engineers and IT executives at Interop to learn how IT organizations are putting server and desktop virtualization technology to use. Fifty-five percent reported they virtualized mission-critical servers, including e-mail and Web servers, and another 50% said they run DNS and DHCP servers on virtual machines. And nearly 40% have already extended virtualization to their desktop environments.
Yet 55% told the network analysis vendor they experience more problems than benefits with the technology, while the remaining 45% said they had realized the benefits of virtualization. Among the problems were a lack of visibility and tools to troubleshoot performance problems in virtual environments for 27% of respondents. More than one-fourth of those polled at Interop cited a lack of training on virtual infrastructure and 21% expressed concern over an inability to secure the infrastructure.
For nearly 60%, the primary problem with virtualization was a lack of experience to appropriately manage the technology and nearly 50% said that technology implementation costs were too high, according to Network Instruments.
Managing the technology doesn't seem to have become less challenging for network managers. Last year, nearly 40% of 117 network managers polled at Interop also listed virtualization as the emerging technology that represents the "greatest monitoring challenges," according to a joint survey conducted by Network Instruments and NetQoS.
"Not surprisingly, a high number of companies have deployed critical network services on virtual machines," said Charles Thompson, product manager at Network Instruments. "The number of organizations without appropriate monitoring tools, however, definitely caught my attention. Without proper tools, application performance can unnecessarily degrade and network teams waste hours troubleshooting."

Lesson 3, "Behind The Scenes in Determining the Costs of Cloud Computing" continues with this discussion of how unused capacity affects the cost of cloud computing.  This lesson builds on the previous ones.   Please see the overview in Lesson 3.1 if you have not already read it. 

So far, we’ve discussed how the cost to the cloud computing vendor is affected by how their application instances are allocated on the vendor’s servers. However, there is an even larger factor that affects the vendor’s costs, which is how many unused servers they keep in their inventory. For example Amazon EC2 rose to fame with their now-famous example of how they digitized nearly 100 years of microfiches using 100 virtual servers in one day. Now, customers routinely use EC2 to provide thousands of machines for short periods to perform medical or scientific calculations. However to provide these burst of computing, Amazon has to keep a large inventory of unused servers around, by some estimates as much as 66% of their total on the average. This of course adds greatly to their costs for data center space, equipment cost, and repairs, even if the servers are turned off when not in use. This isn’t generally discussed by cloud providers, but if you examine the quarterly reports of the cloud providers that are public companies, you may be able to see some interesting facts.  One would expect that over time, these costs would have to be passed on to the customers, even if the vendor keeps the prices low for a while to buy market share.   In other words, you can expect cloud computing costs to rise over time as vendors establish themselves.

While the bulk of business and web computing varies in load over time, the variation is typically constrained and in the range of two to three times the minimum usage, allowing for an average load across many customers that varies far less than what Amazon must build their datacenters to accommodate. If a cloud vendor is willing to give up on taking business from customers who want thousands of servers for short periods of time, they should be able to manage their costs by keeping a much smaller inventory of unused equipment in stock, and in turn passing the savings on to their customers.

For vendors that choose to supporty widely varying loads, the latest "innovation" is allowing customers to reserve capacity so that they can be sure the computer power will be there when they need it (which Amazon doesn't guarantee, otherwise.)  Especially in the case of prepaid reservations, this allows the cloud vendor to get help from the customers in purchasing hardware which otherwise would simply be kept around at a loss.

Lesson 3, "Behind The Scenes in Determining the Costs of Cloud Computing" continues with this discussion of why the exact relationship between cloud performance and cost cannot be known, which makes comparing vendors somewhat difficult.  This lesson builds on the previous ones.   Please see the overview in Lesson 3.1 if you have not already read it.

Cloud vendors, like any company that has to buy computer equipment over time, cannot (and don't want to) buy the same models of servers year after year. Instead, they take advantage of improving server cost and performance over time by buying the latest models so that they can pass those improved economies to their customers or offer higher levels of performance. This causes a problem in giving the cloud customer a known amount of CPU power per dollar, since CPU power keeps rising over time. This is where the “hardware abstraction” promise of cloud computing breaks down.

Let's look at an example. Suppose that a cloud vendor promises a price of $1 per CPU-hour based on an older model server, and they buy a new server with a 20% faster CPU. They could simply offer 80% of the new CPU at the same price, and customers would get the same performance whether their software was running on the old or new server. However, this creates a fragmentation problem: what to do with the remaining 20% of the new server's CPU? To solve this problem, fixed or quantized-instance vendors create a “virtual CPU” with a certain speed rating that doesn't change over time. Amazon, for example, calls a “core” equivalent to a 2006 1-1.2 GHz Opteron. However, if the cloud vendor buys a server that doesn't have a CPU with a multiple of the power of the virtual CPU, they are faced with a choice: either divide the extra computing power among the virtual CPUs on the server, or waste the extra power.

If the vendor divides the extra resources among customers' virtual CPUs, the customer gets something near but not exactly equal to the number of virtual CPUs they have ordered. As long as the customer never uses any specialized performance measurement software, they will not know if they're getting a little more or less than they have been promised, depending on how the cloud vendor deals with the fragmentation. However, customers will always have slightly different amounts of CPU while paying the same for them.

This problem occurs for the variable resource allocation vendors as well, but they have more options in resolving the fragmentation, since at any time, some of their customers will request very small instances that can potentially use the extra CPU. However, variable resource allocation vendors have an additional problem as well: since customer's instances can be moved to new servers at any time, they have to allow for those servers to have slightly different CPU power, which may potentially also change the customer's CPU allocation slightly. One possible solution is that the vendor rates CPU power in Gigahertz, like the processor vendors (AMD, Intel) do, but this presumes that different CPUs with the same clock frequency perform equivalently, which is generally not true.

Ultimately, the customer has no easy way of validating how serious this problem is, or even if it is a problem. Vendor transparency, and the trust it creates, is the only solution to this problem.  

I recently read an interesting article about how a number of cloud providers' services were studied by researchers at the University of New South Wales in Australia, and found to have significant performance problems.   This isn't too surprising: virtualizing a highly variable mix of loads on shared infrastructure is a hard problem.   Despite this, cloud computing is still a major growth industry, with Garner predicting that it will grow from $46Bn last year to $150Bn.  While these numbers are debatable (especially since everyone has a different idea of what Cloud is and I suspect Gartner used the broadest one), they do indicate a growing market.

However within this market, there is definitely room for a variety of approaches to meet different needs for cost and performance.  The largest cloud vendors, starting with Amazon, use commodity hardware and achieve surprisingly good results with it, but still limited by the realities of commodity hardware and other optimizations they have made to deliver service at the lowest possible price.  As a result, customers who are attempting to use these services outside their design capabilities are in for a big surprise, as these Australian researches discovered.   One discovery they made was that "response times on the services also varied by a factor of twenty depending on the time of day the services were accessed."  At a recent Cloud Camp, I saw the resulting frustration, often as a result of customers having to overprovision to meet their performance goals, and as a result, being disappointed that cloud deployment costs exceeded what they expected.  The researchers also brought up the lack of measurement and monitoring tools that allow customers to verify their performance in a production environment, something that we at ENKI consider to be absolutely essential to any business use of the internet.

ENKI was founded not on providing the best price, but rather the best value to its customers, which we measure as TCO (Total Cost of Operations), to which performance is a contributing factor.  To this end, over the years we've added enterprise grade monitoring and have been improving the performance of our cloud to match our customers' expectations.  Recently we've developed a higher-end cloud computing product that has the same performance as a purpose-built datacenter would have, called PrimaCloud .  I'll be writing more about PrimaCloud in a future article.