Storage has always been a major challenge for ENKI in building a high-performance cloud: how do we achieve reliability AND speed?  

For worry-free reliability, we have consistently chosen not to place storage on the compute instances, unlike Amazon or Rackspace's sliced-dedicated-server clouds.  If the storage is on the instance, any speed gains are offset by the possibility of data loss if the instance fails.   On the other hand, concentrating the storage demands of many instances onto a common storage infrastructure gets you persistent instances with full failover restartability, but it requires the centralized storage to be very high speed and connected to the instances over fast networking.  In other words, expensive.  To solve these problems, we've so far chosen Infiniband networking coupled with SANs that accelerate access to storage using SSD caching for commonly-used data, offering a large fraction of full-SSD storage wthout the price.

But now, Amazon and other cloud providers are upping the available storage speed in their clouds by placing SSD storage into their servers.  Do these increased speeds - necessary for today's cloud database and transactional loads - warrant building in even further incentives to decentralizing storage?   I don't think so.  

First of all, today's applications - in order to reach the highest levels of transaction processing speed - are highly parallelized, meaning they run on multiple servers that have to exchange data in real-time.   So "landlocking" their data on the SSD inside a server actually serves to slow down the application, unless the storage can be fully fragmented (often called "sharded.")   In fact, with many applications the necessity for high speed synchronization between servers becomes so extreme that the networking speeds have to approach the memory access speeds to allow applications to scale linearly.   Very few cloud providers are putting that kind of networking in place because it's expensive.  

Second, placing SSD on the server doesn't solve the failover problem.  In fact it makes it worse in practice because even more of the client's access-speed critical data will be placed on the server.   The cloud provider who places SSD on the server is essentially dangling an irresistable treat in front of their customers, tempting them to leap off a dangerous cliff of unreliability.

There have been a few proposed distributed storage architectures which maintain access-critical data in local SSD cache on the server, but over time, these solutions have all become unidirectional storage products, used for media servers and such, because they didn't synchronize data between the separate local caches.   There haven't yet been products offered that offer distributed cache coherency for storage, especially because the customers with a full Infiniband or similar network structure tying their clouds together just haven't existed.

I think the solution to offering cloud with true enterprise-grade performance still remains with centralized storage: making SSD available as a shared resources, accessed over dedicated, fast networking.    This is the approach that we are going to be offering for our new Santa Clara datacenter cloud cluster in Q2 '13.    It also offers the additional benefit that the SAN can dynamically move the data to the most appropriate storage type (disks or SSD) depending on load, which reduces the overall cost of SSD storage for the cloud customer.