A number of us from ENKI went to the Web 2.0 Conference last week.  I wanted to share the single most important take-away that I got from the conference with you.

Last year's conference floor was almost a venture capitalist's fire sale.  There were numerous startup companies exhibiting their concepts and beta sites, vying for mindshare and potentially funding.  However, despite my careful attention to who was walking the aisles, I saw very few investors interested in the startups.  Instead, it seemed that most of the attendees were entrepreneurship fans looking to absorb some of the glory, knowledge, and experience that they would need to start their own companies.  In a many of the conference sessions, the Q&A was - rather irritatingly - taken up by newbs asking the most basic questions of the presenting entrepreneurs to their evident frustration.  It was at times almost like attending a Robert Kiyosaki real estate investment seminar, where the not-so-hidden question is always "how do I make my first 10 million?"

This year, the conference seemed to have learned from last year's experiences, and more of the booths and sessions were tailored to assist people in creating their own Web startup companies.  But the most impressive change is that the industry as a whole understands that creating your own Web startup is not only desirable to a large audience, but also possible with the technologies reaching maturity today. 

What I saw was a large number of products designed to reduce the barrier to going live with your own Web application.  The most obvious were companies such as Google or the many hosted application development systems that create a limited, simplified environment for you to create and deploy your application.  By constraining the environment, the vendor can support your development effort more easily with prewritten code for user interface creation and basic system functions, as well as simplifying deployment into their own application hosting framework.  For many with a simple Web application in mind, these new development and deployment frameworks make life easy.

But there's also help on the way for the more traditional Web applications, developed from scratch in one of the popular implementation systems such as Java, Ruby on Rails, PHP, etc.  I saw a number of standalone application development environments (often based on the versatile Eclipse IDE) that simplify the coding process (perhaps in some case over-simplifying it.)  The most impressive was Liquid Apps, which will generate code for multiple languages, and import specifications such as UML as well.

On the deployment side, I saw a number of new options for entrepreneurs to deploy to the Cloud, beyond the old standby of Amazon's AWS.    Even AWS was dressed up with third parties offering deployment and management solutions that take some of the edge off Amazon's much maligned peccadillos.  What was missing was easy one-touch deployment solutions for the application development environments that would eliminate entrepreneurs' having to become IT experts.

Development and deployment seem to be heavily on the minds of entrepreneurs. I spent much of my time representing ENKI and 3Tera in the 3Tera booth, and I must have fielded hundreds of questions about how to set up a Web application with little or no IT knowledge.   Fortunately, this is the need that ENKI was founded to address, so I had some answers!

However, the answers that I had usually involved ENKI's operations services.  And, I believe there's a good reason for this, which will take a long time for the industry to address.  If I look at what we do for our customers, it will take significant technology advances to automate these operations - if they can be automated at all:

• application-related virtual data center design
• middleware and operating system tuning 
• optimal software release processes
• application-level monitoring and response
• optimal database design
• low down-time management processes
• design-for-test

Granted, many smaller Web application deployments don't need this level of expertise - but how many of those smaller sites are owned by entrepreneurs who want to grow them to be The Next Big Thing?

So my take-away is that this year's Web 2.0 Conference was all about lowering the barriers to creating and deploying your own Web app - just what many entrepreneurs were looking for.  But what's still missing is a simple end-to-end answer for deployment and management. 

Over the last couple years that we've been growing ENKI, we've seen how important it is for us to pick vendors that we're really excited about doing business with - vendors we can see as partners in our business that help it grow.  As a small business, our partners have a huge effect on our business and the overall impression we can make on our clients, so picking good ones is critical for us.  I was talking to Dave about this at lunch the other day and we realized there's a pattern to good and bad vendors that we could see early on and would help us pick partners that naturally fit in with our win/win business approach in the future. 

First of all, I'm not going to name many names here - that's not the point.  But if I rave about a vendor, you can take that as a recommendation :)  The best vendors are ones where there isn't any drama in our relationship: we can trust them to deliver their product or service and we never feel betrayed or taken advantage of by them, nor do we feel we have to keep asking (or begging) for some sort of change in our relationship in order to run our business.  We can always enthusiastically recommend them.  The worst vendors keep us constantly looking out for a replacement vendor, or wishing we could because we're locked in to a relationship that we can't easily extricate ourselves from.  It's hard to recommend them to others.  Unfortunately, despite our best efforts so far, the majority of our vendors fall into the group which we'd have trouble recommending.  (If you read my Small Business CRM Challenge blog articles, you can see how frustrating this can be.)

So by now you're probably wondering what the patterns are that we've seen, and how you can use them to pick your vendors.  The strongest and most determining pattern seems to be where the vendor lies on the scale of expedience versus complacency.  Both states seem to stem from their views of their own financial success.   The expedient vendors will put on a happy face and say what it takes to get your business, since they are on the financial edge and need business badly to stay afloat.  In their mind, doing anything for the sale is justified, but you rapidly become cynical of their promises since they can't live up to them - often because they're just too strapped to do so.  Just as troubling are the complacent ones, who have a history of success or a lock-in business model and think they're entitled to your business as a result, without working to earn it.  These vendors betray your trust by acting unilaterally in their own interest, since they think they're the only game in town. 

So how do you see these patterns in advance?  In my experience, the expedient vendors are visible up-front because they offer large discounts or a complex pricing structure with lots of add-on packages that obscure the final cost to you behind a low buy-in price.  In other words, they're thinking, "just sign the contract!!!"  The complacent vendors are more difficult to spot: you'll probably need to do some research on their existing customer base, perhaps by searching online discusion groups.  One telltale is a high buy-in price or inflexibility in negotiating a contract.  They're thinking, "we're the only game in town." It's not always easy to see these patterns, but I'm learning to be a more feel-my-way-through-it kind of guy and if I feel cynical or betrayed in my initial contacts with the vendor, I stay away, since these feelings are harbingers of what is to come.  What's going on here in either case is that the vendor or their representative is avoiding some sort of humiliating experience of loss by the way they're dealing with you - and that's a powerful motivator to keep on practicing this dysfunctional behavior.  So the ultimate thing to look out for is loss-focus (or its littermate, gain-focus.)

What about the opposite - a vendor you can't say enough good things about? If I'm to believe my own analysis, these are vendors who are neither complacent nor expedient.  What does that look like?  I thought about this for a long time and realized it meant that the preoccupation with the humiliation of loss isn't foremost in their minds.  They're having fun, they're passionate about their work and product, and they're connected to their source of internal strength and resources rather than in a relationship of dependence with their customers.  The one word I could think of to describe this was "Courage."  Good vendors exemplify courage.  I know this sounds like one of those kitschy motivational posters you sometimes see in executive offices, but there's some truth to it.  If your vendor is courageous, they aren't focused on milking your business like a cow, but rather building a partnership that benefits both parties: a win/win relationship.  The relationship stops being about them and how they can survive, and starts being about exploring success together because business is about drawing on each others' strengths.  This aligns with the practices that most highly successful executives embody.

OK, so how do you recognize a courageous vendor?  In my experience, they're willing to listen and then act on that listening - not just look like they're acting. This takes a lot of humility - knowing that you don't have all the answers - and that's rare. In our lives, we usually get humility from humiliation and given that we spend most of our time avoiding it, it's a dearly-learned state!   Lately, businesses have realized that looking like you're listening is important, and have created customer relationship management systems, customer surveys, and customer service agents dripping with faux humility to give the impression they listen.  Don't get me wrong: these are good practices, but they have to be backed up with the courage to truly listen and act.  As a small or medium-sized business principal, you know the value of relationships.  So it pays to create a personal relationship up-front and see if your vendor is listening.

I had an experience this week with a vendor I already am enthusiastic about: Kerio Technologies.  Their product just works, and is easy to use.  Their support is effective and doesn't waste your time.  Their pricing is reasonable and easy to understand.  We went to their offices to talk about sales synergies and while we were standing in the hall, their CEO, Scott Schreiman, walked by on his way to get a soda.  He stopped, and listened.  Really listened.  And then he asked us into a conference room to listen some more, without making our time together about him or his company.  He wasn't selling or trying to get rid of us, even though we're a small customer of his.  He was relaxed and open to what was transpiring.  It just reinforced my desire to do more business with his company.  I know that whatever partnership I create with him, even if it isn't working at the time, will be able to move forward because of that orientation to listen.

Finally, my personal observation is that Gandhi's famous words, "Be the change you want to see in the world," still apply more than ever to your business dealings.  If you can be courageous, you'll attract similar partners to yourself since you will all know what's important in a relationship.

Introduction

In response to requests from customers, I'm going to start a series of irregular blog articles on making the most efficient use of cloud computing resources, such as those provided by ENKI's Computing Utility.  I will use examples from our customer base, as well as from the storehouse of experience the staff at ENKI have from working at start-up and enterprise companies.

Ultimately, making efficient use of resources in the Cloud, where computing resources are utility-billed, is very similar to simply dotting your Is and crossing your Ts with respect to efficiency in in any computing environment: you want to use your allocated resources as effectively as possible.  But there are also some interesting new problems and opportunities that come with being able to resize or re-allocate your computing resources on demand.  Today I am going to write about bursty loads, such as those involved with media transcoding, image processing, and database-intensive tasks. 

Many of the applications I created in the past, and those of customers who come to ENKI, are developed and possibly deployed on fixed hardware such as a PC under their desk, or a physical server in a data center.  In this case, the hardware is typically dedicated to the application and runs all parts (or tiers) of the application.  If the application is too slow, the choices are to improve the efficiency of the code (which Dave wrote about in Data Center Power Consumption, Part III: The software), buy a bigger server, or split the application into parts, each running on a different server.  With ENKI's Computing Utility service, you can increase the size of your server on demand, but it won't increase the efficiency of your application, only your monthly bill!  Splitting the application is where you can get some big efficiency gains from utility computing.

Before we dig into the numbers, please remember that this discussion involves a tradeoff between the cost of the computing service and the user experience.  If you make the wrong choices, you may no longer have a business to optimize!

A bursty batch processing example 

Let's take a look at a sample application which transcodes stored video files into a customer-selected format for download.  We will start by assuming that a standard server can process 1000 videos an hour.  If you deploy this server to transcode a video when a user requests it, the user would have to wait (60*60/1000) = 3.6 seconds to be able to start downloading it, which is not a problem. Suppose we deploy the application this way and observe how the users use it.  For most of the day, requests come in once a minute on the average, which means that our server is only at 6% utilization.  However, in the evenings, there are peaks where users are submitting a request about once every 2 seconds for periods of up to 15 minutes. In this case, after 15 minutes, the last requestor has to wait for their video an amount of time equal to the number of backlogged requests submitted times the processing backlog on each video: 15*30*(3.6-2) = 720 seconds or 12 minutes.

Whether this 12 minute delay is a problem for your business or not depends on how you set your users' expectations.  If they have to sit and watch a spinning film reel, they'll go away and not come back.  If you tell them you'll send them an email when their processing job is ready to download, they will most likely accept that.  The very successful (until the RIAA killed it) music download service, AllofMP3, allowed you to transcode your music this way, and their customers were delighted.  Similarly, NetSuite submits long-running reports for background execution, notifying you when they are complete.

So, how can we change the deployment of this application to save you money during most of the day when your server is only being used at 6%?  The trick is to move the transcoding to a separate virtual machine instance (called an "Appliance" at ENKI) which is dedicated to processing only the transcoding tasks, and implementing a processing queue to allow the instance to manage its backlog.  You can then set the amount of CPU allocated to the instance to control your maximum queue length at peak periods.  A side benefit of doing this is that you can reduce the resources allocated to the Appliance with the remaining tasks in your application (UI, downloading, etc.) because it no longer has to process transcoding jobs quickly.

For example, if you chose to set your CPU allocation to 12% of one standard CPU for your queue processing Appliance, your processing time for one video would increase to (1/0.12)*3.6 or 30 seconds.  In this case, using the math above, users would have to wait a maximum of 210 minutes at a peak usage time.  However, most of the day their wait would be 30 seconds.  In return for this increased delay, you save 88% on your resource charges.  You could even look at the queue length and offer them the option of waiting or getting an email when it was done if they had to wait longer than 30 seconds.

Making the cost vs. speed tradeoff 

This is a tradeoff that you will have to make based on the best information you have about your business.  You need to know what kind of user experience is required for your business to succeed before you can analyze performance numbers. You may want to conduct a focus group or user feedback study. There may be automatic ways to check if performance is unacceptable, such as correlating abandoned sessions to changes in resource allocation, for example.   If you are an internet startup company or an internal IT department rolling out a new application, you only have one chance to make a first impression, so it's probably wise to start with a generous allocation and reduce it over time while checking back with your users to make sure they aren't too frustrated.  However, by setting the users' expectations correctly, such as with the email-on-completion method listed above, you can take a lot of pressure off of your application.

In the next article, I'll write about another technique for making efficient use of utility-billed computing, which is varying the breadth (the parallelism) of your application as deployed in the Cloud in response to increasing load. 

There's been a lot written about Cloud Computing being the wave of the future, but here at ENKI we're seeing that it can save our customers big money right now and enable them to sign a larger number of profitable deals.  We're seeing an increasing number of SaaS (Software as a Service) companies interested in our Computing Utility service, many of whom offer middleware services like data translation, data communications services, or data aggregation and reporting.  They sell their software as a service to end customers.  Most of them have large customers, so their typical sales cycle has been:

1. Sell and sign the business
2. Engage their professional services and IT departments to build out their own data center or a new data center for the customer
3. Set up the application software on the new new data center
4. Activate the software and deliver it to the end customer

This has worked well for them, so they haven't been too focused on cost or time to market; but as you can see provisioning a data center or adding significantly to one can take weeks to months, and involve a great deal of labor and capital costs.  However, they have been unable to profitably address smaller end customers: imagine using the process above if all you need is a one and a quarter server's worth of computing power!  So, they were focusing on a few larger deals and leaving the many smaller deals on the table (the infamous "Long Tail", named for the asymptotically dropping curve of price versus quantity of customers.)  And now, with the economy slowing, this ponderous process is beginning to look too expensive to SaaS providers, even when they provision an application for a large customer, due to the large capital outlays required to sign business that might not be around long enough to pay off the data center.

However, using our pay-as-you-go cloud computing technology based on AppLogic, ENKI can reduce the sales cycle for SaaS to look like the following:

1. Sell and sign the business OR use a portal where the customer can sign up and pay via self-service
2. Automatically provision a new copy of the application in a virtual data center sized to the customer's needs
3. Automatically activate the software and deliver it to the end customer 

By eliminating manual steps including hardware and software provisioning, the turnaround on the sales cycle drops from months to as little as minutes.  Also, the cost - both of the sales cycle and of provisioning the resources - drops dramatically due to the reduced labor component and the fact that resources are allocated without waste (no extra capacity is needed to provide for future growth or redundancy for reliability!)  This allows SaaS providers to profitably sign customers in the Long Tail and greatly increase their market.

I've found that the biggest obstacles they encounter in switching to cloud computing are internal resistance from departments who are used to doing things the old way, and switching their financial models from lump-sum expenditures to flow-through cash-basis, often paid with credit cards or ACH rather than invoice and check.  But this is a small price to pay for increasing your market penetration and profits.

If you'd like to learn more about cloud computing, there's an excellent summary in a blog article here.  

One of the questions I get asked most often by prospective customers is how they will be able to release new versions of their custom software to their websites under AppLogic or VMWare.  Generally, ENKI does not allow customers to have root access to their own production websites if they place responsibility for keeping them up onto us, so that we can keep track of any changes that are made in order to provide maximum uptime and minimize flailing about to find out who changed what.   Database access is provided, however.  I'll describe our release processes in this context.  If you have your own Cloud installation, you'll want to partition responsibility in a similar way between your developers and your operations staff in order to have reliable and short release windows.

Our Operations as a Service (OaaS) basic release processes are described below, but bear in mind that the actual process has to be flexibly defined depending on the structure and design of your application, and may require a custom process.

No Database Migration Needed

1. We create a copy of the production site to make a test environment.
2. The customer copies the release candidate code via scp/ssh into the test environment and tests it.
3. When the testing is complete, the customer stops the production environment and notifies us that the test environment is ready to be released.
4. We rsync any changes to the live site data from the production environment into the test environment where appropriate.
5. We alter the IP addresses to make the test environment be the new production environment and restart it to complete the release. 

Database Migration Needed

1. We create a copy of the production site to make a test environment.
2. The customer copies the release candidate code via scp/ssh into the test environment, migrates the data, and tests the release candidate.
3. The customer creates a patch process document describing the necessary steps to update the code and run the migration, and provides an SCP server for ENKI to retrieve the updated code - usually in tarball form - from (which may just be the test environment.)
4. When the testing is complete and the release is determined to be ready, the customer notifies ENKI, which stops the production environment and restarts it on a temporary IP address.  A site-down server is started to notify users.
5. ENKI deploys the code updates and runs the migration according to the patch process document (or, alternatively, the customer runs the migration).
6. Customer tests the production site and notifies ENKI when they are satisfied.  ENKI alters the IP addresses of the production environment to bring it live and  restarts it to complete the release. 

Minor patch

Customer notifies ENKI and provides an scp server with the patch on it, together with patching instructions. We stop the live site, perform the patch and restart the live site.

Minimizing Down Time

Clearly there are are quite a few variations that can be made to these processes, depending on the customer requirements.  For example, if the live site cannot be allowed to go down for more than an absolute minimum, the migration-based process could be modified to include keeping transaction logs on the live site while the patching and migration took place, followed by a database update from the logs while the live site was down. This of course depends on selecting a database that can reliably work with transaction logs.

We're well aware that many of our customers are in the volatile social networking, SaaS, or mobile application space where downtime can cause a permanent loss of customers due to impatience or bad publicity, so it makes sense to create a custom release process that minimizes the risk.  One of the great benefits of AppLogic or VMWare when combined with on-demand or utility computing service is that you can work out all the kinks in your release process on an instantly-provisioned copy of your live site while customers are still using the previous software version.

Well here it is six months after I wrote the first article about ENKI's search for a CRM system, and we finally did choose one and started our implementation.  After a lot of looking around and testing, we ended up choosing NetSuite, a SaaS service.  It wasn't an easy choice: I was quite certain that none of the systems I looked at would meet 100% of our needs, but NetSuite with its full spectrum of features (from web store to CRM to accounting) together with script-based customization, seemed like it would fit the bill best.  I haven't changed that opinion after 5 months of using it, but I'm also not completely happy with our purchase.  Read on...

After living with NetSuite for 5 months, in which we moved our accounting records for 2007 to it, switched to using it for sales automation, and have started to use it for support automation, I have to say that I'm very impressed with what it can do out of the box.  If you've read this site carefully, you will know that I worked there earlier in its history, but even though I was in charge of the development department, I never had a chance to experience the full range of its capabilities.  In most cases, whatever I want to do is much easier than QuickBooks, and offers more capability.  Reports, for example, offer a great deal more customizability, and the ability to monetize almost any interaction with the customer is incredible.

However, it hasn't been a honeymoon.  Like most lower-end ERP and CRM products, NetSuite has a built-in business process for everything, which unfortunately doesn't match the processes my business needs to succeed.   Or, it has no process at all and you're stuck with having to remember how to do something right.  Unlike Fortune-500 ERP and CRM systems, Netsuite isn't built on general purpose software frameworks that support the underlying essential "atomic" elements of business process automation such as event handling, event scheduling, and conditional event flow. This leads to trouble in getting it to do what you want, even with it's scripting customization capability.

There are two main areas that I am concerned with.  The first is billing: we bill our customers on a pay-as-you-go basis for services rendered and resources consumed each month.  This neither fits into NetSuite's memorized transactions or their Advanced Billing(available on an upcharge) models, since the billing is recurring, but always different.  The way around it is for us to write custom software in NetSuite's Javascript customization language that calculates the customer's bill and creates an invoice using NetSuite's web services access.  This is in process, and I have hope that it will work out.  For now I stay up late at the end of each month doing billing manually!

The second area is our support process.  We offer a variety of service level agreements (SLAs) to our customers, some of which have guaranteed response times of as little as two hours.  During that time, the system has to capture the customer's request, notify a responsible on-duty support engineer, and find an alternate if there is no acknowledgement.  However, NetSuite's capabilities for measuring elapsed time and acting on time-based settings is only accurate to hours, not minutes, and they only check the status of time-based tasks every half hour.  The result is that the process I described above, under worst-case conditions, can take up to 1 1/2 hours to notify an alternate support engineer, which could cause us to miss our guaranteed response window.   Worse yet, there are bugs in NetSuite that keep the case escalation from working as documented so we don't reliably receive email notification that our cell phones can react to and start beeping for attention. So for now, we at ENKI can't take any time away from our cell phones, and have to check them constantly for customer support cases.

Also, some critical functionality we need to access and respond to cases with our cell phones via email is not part of the product.  I've written some JavaScript to work around this, but there are more bugs that keep that from working as expected.  With NetSuite's poor record on fixing bugs that might be critical to only a small segment of their customer base (according to posts on their users' group bulletin board), I despair of any quick resolution.  These problems are so severe that with ENKI's continued growth, we may need to replace NetSuite just to be able to deliver our core value proposition to our customers.  This would be a shame, since in general I really like the service - and I don't have time for another migration and implementation. 

We've also had heart-stopping excitement from time to time in other areas.  A few days before Christmas, I wanted to use a NetSuite Campaign to send my customers a Christmas thank-you card.  Unfortunately, I missed un-setting the "new customers are unsubscribed to campaigns by default" checkbox when I set our implementation up, and now I can't mail my customers, and I can't change the setting for existing customers due to NetSuite's ill-considered anti-spam policies.  I had to export the list to excel, and then use a third-party mail program.  The cards went out on time.  I just didn't get any sleep.

Since our implementation, one of our partners has suggested Enterprise Wizard as a solution to our problems.  A quick tour through that service showed me that they're focused on implementing their customers' unique business processes through an easy graphical editor.  This would be perfect for us, but it's missing NetSuite's financial aspects and its webstore, which I've started to use to sell our off-the-shelf virtual private server products.  Writing an accounting system in Enterprise Wizard isn't my idea of fun!

This experience has shown me that even with today's Software-as-a-Service ERP and CRM applications, automating your company's processes is still a difficult, expensive, and frustrating process - as it was 10 years ago.  When it starts working, however, the benefits can be incredible, saving you money on staffing and allowing big-company marketing, sales, and support services that you couldn't achieve any other way. 

Some learnings I take away from this humbling experience so far include:

1. Don't be in a hurry.  Sticking with what you have a little longer is always better than switching systems if you feel under pressure.  The switch itself is a lot of work, and working around an incomplete evaluation is expensive or could even cause your roll-out to fail.
2. No system will do everything you want, or if it does, it's oriented for a budget that SMBs can't afford.
3. Ask your vendors and customers how they solve the problems you're considering buying this software or service to address.  Use networking opportunities as well, such as online groups, professional groups, and associations.
4. If you get a system that does 90% of what you need, be prepared to hire consultants for extensive programming to achieve the last 10%, or spend some time learning today's web2.0 technologies to do it yourself.
5. Be creative.  If it turns out the system doesn't do something you need, there's usually a workaround if it allows you to export data or access it remotely via web services.  You may be able to change your business processes to match those of the system as well.
6. Implement as much of your needs as you can in your test drive or trial of the product before buying.  However, don't let a vendor charge you for any sort of testing or application consulting prior to purchase: it sets a bad precendent for your future relationship.
7. Pick your implementation partners well, or hire your own programmer for customization.  It doesn't make any sense to implement a customization five times because your partner doesn't understand your business.
8. Document your actual and/or desired business processes prior to buying.  This will help you with your evaluation, and you may be able to extract a money-back guarantee from the system vendor's sales staff that they can meet these requirements.  This is not easy - but you'll need to do it sooner or later anyway if you are going to hire consultants to customize your system for you, or if you want to go for SAS70 or ISO certification.

As usual, I'd love to hear from you about your experiences, or advice!

I came across an interesting blog article today about making a reliable MySQL database cluster on EC2.  The author goes over the pain of working around two limitations that EC2 has: lack of immutable (unchangable) storage associated with an EC2 instance, and the lack of a static IP address for an EC2 instance.  However, EC2 isn’t the only game out there for utility computing.  Services such as ENKI's Computing Utility, based on 3Tera's AppLogic grid operating system, can provide scalable utility-billed cloud computing at higher levels of reliability than EC2.

With AppLogic, you get 3-nines “right out of the box”, and 4 or 5-nines can be achieved using some of the techniques mentioned in the blog I was reading. AppLogic’s 3 nines comes from its ability to automatic restart on a different, working server any virtual servers that were hosted on failed hardware. AppLogic is oriented around “Applications” which are networks of Appliances in a private address space - essentially a virtual private data center. Even if an Appliance is restarted after a failure, it gets the same DNS name inside the Application that it had before (as maintained by AppLogic) so other Appliances in the Application can continue to refer to it successfully. The automatic switchover usually takes just a few minutes after a server fails.

Where AppLogic goes beyond EC2 is that virtual server images, called “Appliances” in AppLogic, are object-oriented in the sense that the configuration information for the server (application, setup files, patches, etc) is maintained separately from the server instance itself, so if the server fails, it’s restarted from its parent “class.” Clustering or parallel instances are supported because each Appliance's unique configuration information (including config file settings entered from the AppLogic user interface) are loaded into the Appliance when it restarts, yet all the Appliances in a cluster can share a single NAS Appliance and hence share a single deployment of code or data.  To get better performance in, for example, a Java/Tomcat environment you need only add another Tomcat instance to your Application.

User data is stored in a RAIDed volume or on a NAS appliance with its own RAIDed volumes within the grid of redundant servers that AppLogic runs on, which means that it is available without interruption even if a server goes down.  Again, this storage is more reliable (and available at a lower cost) than Amazon's S3 offering - and it's within the same grid of hardware, connected by gigabit links for better performance.  

There are currently 3 ways to use AppLogic. and ENKI can help you get started with any of them. You can purchase utility computing from ENKI on a pay-as-you-go basis which allows you to use a virtual private data center (AppLogic Application) and scale it on demand. In this case, you don’t need to know anything about AppLogic to use it since we manage it for you. You can also lease a stack of servers from a managed service provider offering AppLogic and manage it yourself or hire us to manage it for you.  Or, if you’re working for a big enough company, 3Tera will lease AppLogic licenses to you for use in your data center, and we can supply training and support.

People often ask me, "what's so great about AppLogic?" I tell them that aside from it's 5 minute mean-time-to-repair for hardware failures (like a server going down) one of the big advantages is ease of system administration and the ability to avoid unnecessary downtime.  This is why ENKI chose it as the application framework for delivering our Computing Utility product.  I don't usually do much with AppLogic myself, since I am responsible for Sales and Marketing at ENKI.  However, I am also responsible for our corporate website.  So today, I got a demonstration of what AppLogic can do for you - on my own company website - and I wanted to share it with you.

Our company website is hosted in a Virtual Private Data Center, which consists of a number of appliances (virtual servers connected to each other in a private name space): Apache, MySQL, a NAS, a logging server, and a firewall server.  MySQL and Apache mount their data from the NAS.

We had been running our website on Joomla 1.0.12 for a while now.  I noticed recently that our site was getting attacked with exploits that might breach 1.0.12.  Also, there were some bugs in 1.0.12 that I wanted to fix by upgrading to a newer version.  It was time to upgrade Joomla.  Here's how I did it:

- I created a copy of my running VPDC and started it.  This also copied all my data, so the new VPDC had an exact copy of our company website.  My website was still up on the "old" copy.

- I installed the new Joomla onto my copy (by loading it into the appropriate directory on the NAS)

- I tested the new VPDC with the new Joomla in it.

- I altered the IP address of the new VPDC to be that of our main website

- I shutdown the old VPDC and restarted the new one.

The total down time for our website was under a minute, and I had deployed a fully working and tested upgrade!  Joomla is a PHP application, and installing it is basically just a file download and un-tar, followed by some setup work.  But installing a new version of MySQL would have been very similar: in that case, I would simply have instantiated a newer version of the MySQL appliance in my VPDC and restarted it, since all appliances are instances of appliance-objects.

AppLogic's easy copying of a VPDC has other benefits as well.  You can automatically create and provision a copy in order to meet the demands of new customers for a SaaS (software as a service) application that need their own VPDC.  Or, you can copy the VPDC to another physical data center for hot or cold standby purposes.

The biggest obstacle to enjoying these AppLogic advantages is the mental barriers one has around how system administration has always been done, and what operations are possible in an object-oriented virtual operating system.  I often find new ways to shortcut operations that might have been risky or expensive, once ask myself "how is this done in AppLogic?"  To allow our customers to shortcut the need to learn AppLogic, ENKI offers these advantages to our customers without their having to learn AppLogic, since we manage their VPDCs for them.

I just got off the phone with a potential customer for our virtual private servers, and he suggested I put the discussion we had into this blog.  He's technically capable, but not trained in all of the details of maintaining his web application, and he wanted to know how much support he could receive from ENKI if he purchased our virtual private server product.

ENKI's VPS products are primarily aimed at people such as this customer, who don't need extensive handholding, yet want a partner to be available in case something goes wrong with their hosted environment.  For non-technical customers requiring more support or even fully outsourced operations, we offer our Outsourced Operations (OaaS) services.  But this customer just wanted the occasional helping hand.  At ENKI, our primary concern is that our customers' businesses are successful, since we see our own success flowing from theirs.  As a result, we've avoided having too restrictively defined "packages" of services that might put off potential business or even worse, prevent a customer from calling for help if they needed it.  We rely on negotiation and saying what we think to each other, rather than restrictive contracts.  But his question was valid: what is the right level of support for a leased server?

As I see it, to succeed with a leased server powering your website, you will need three types of services: application programming and maintenance, provided by a programmer; system administration, provided by a system administrator; and hardware maintenance, provided by your hosting/colocation company.  At ENKI, our hardware is essentially self-maintaining, healing itself within minutes after any failure.  So our customers don't need to have staff available to service hardware failure requests.  We also expect our customers to have application programmers available to them, or we can provide them as part of a consulting or outsourced operations package.  This makes sense, since your business runs on your software, which you would want maintained or extended by someone who understood your business in detail.

So what remains is system administration, which your hosting or colocation provider should be able to provide if you need it.   But what's the right balance between administration that you do and they do on your server?   I think that the general guideline for making such decisions is that if it relates to your particular business or setup, you should either find a trusted partner that takes the time to understand your business or do the work yourself.  This way, you know what's happening with your server, and you know how to set it up to meet your needs.  Otherwise, if the setup or management has nothing in particular to do with your business, the service provider should do it for you.  I have some thoughts on specific cases:

- Operating system and middleware installation: Your provider should offer you the option of pre-installing this software so you can get started faster.

- Operating system and middleware (including database) updates: your provider should perform this work, but only when you ask for it.  You should be able to have the option of doing it yourself if you prefer. 

- Backups: Since you need to set the policies (when and what gets backed up) the provider should only give you access to backup software and assist you in operating it.  You should retain responsibility for doing backups.

- Installing new application software: If the software is middleware (mail servers, etc.) the provider should offer you the option of doing it for you.  Otherwise, you should do it yourself since you will be operating it, and you'll need it configured for your specific needs.  Again, your mileage may vary: if you're not software-literate, perhaps you can work with your provider to do it together.

- Application software updates: This should be done either by you, or by your operations provider, when you ask for it.  In the case of production systems, it makes sense to set up a policy where the application software update is tested in a "sandbox" environment first, then moved to the production server when it is ready (and so are you!)

- Performance and security problems: This is almost always a joint exploration between you and your vendor, and they should be able to help you with this when you request it.

ENKI's support policies for virtual private servers mirror this philosophy.  As always, I welcome your thoughts on the matter.

The Pain of it All

I'm seeing an increasing number of customers coming to me who are "accidental tourists" in the land of hosting and information technology, suddenly finding themselves trying to operate and administer one or more colocated servers - essentially a data center - in order to grow their business to the next level.

One is a web design firm that was repeatedly asked by their customers not to just design their web pages, but to install and maintain their web sites for them.  They went out and on the advice of one of their employees, leased colocated servers and installed the Plesk hosting application, ending up with over 1000 hosted customers in less than a year.  However, there was a large gap between the skills of their staff and the support their colocation provider was willing to give for the servers, and they experienced long periods of downtime when they tried to increase the power of their servers to match demand.  Can you imagine hundreds of customers calling in, furious?  The web design firm just wants their computer systems to work so their clients will be happy, but they can't find anyone to do it for them.  Their CEO said to me, "how can I run this business without any computers or geeks?"

Another customer is getting ready to roll out a media-rich web2.0 application.  They are expert programmers and visionaries, but have little experience preparing a "factory" of servers and related staff for the large computing loads that are on the way with a successful launch of their web site.  Their existing hosting provider can't seem to explain why their software runs faster on the development servers than it does in their data center of colocated servers.  With time running out until their launch date, this customer was desperate to find a solution, and was considering leasing space in a colocation facility, hiring staff, buying servers, and running their data center themselves.  But they didn't have enough time - or money - to do that.

Another customer had been hosting their web site through an online marketing company that itself has little to no technical expertise.  As long they didn't need any changes to their website, all was well. But now they wanted to add a web store and their host, the marketing company, was unable to help them: each time a change was needed, they would call *their* colocation provider for help.  Not surprisingly, the 3-way connection led to lots of mistakes.  Not being technically savvy themselves, this customer didn't even know where to turn for help.

I have run across lots more examples, but I'm seeing an increasing number of small to mid-size companies that are in the position of needing the services of enterprise-level information technology just to stay in business, and yet are unable to find a way to get it within the constraints of their business, either through their own efforts or from a hosting company or managed services provider.  We even get calls from large enterprises that want to keep the  fixed overhead and startup costs of a new IT project to a minimum while ensuring its success.  As a former director of development at a web2.0 company, and with our founder Dave as a CIO at multiple web-based businesses, we both saw how complicated, expensive, and risky it is to develop a strong in-house IT capability, whether the hardware is purchased or leased from a provider.     There had to be another solution for these companies to get the IT services they needed without this level of risk - which is why we founded ENKI.

What IT Services You Need To Run Your Business 

What capabilities does a web-based or internet-based business really need to have its IT just work? Quite a bit, actually. Here's what we at ENKI have as our collective checklist of essentials:

As you can see, building these capabilities for your small to mid-sized company or department is going to be a challenge.   But what alternatives are there?  There is no one right answer, since the best strategy depends on your business.

Strategies For Meeting Your IT Operations Needs

- Do it all yourself: buy or lease equipment, and hire skilled staff to provide IT services.  This approach is the time-honored one, and was the only one available to small and mid-sized companies and startups until recently.  Today, this approach only cost-effective if your business has specialized needs such as extreme security or performance requirements that a vendor or partner can't address.  However, many companies still choose this path because it's comfortable - even if they can't afford it.

- Hire a consultant, and then do the rest yourself.  This helps to avoid the long start-up times in setting up your own IT services and staffing up, but you are paying for the consulting as well as your own staff and hardware.  In the long run, companies often keep the consultant "on staff" because they can never find staff with all the skills they need.

- Create a stable of partners, each serving one need.  For example, a colocation or hosting partner, a system administration partner, one or more partners for specialized needs such as security or business continuance services, and application programmers as necessary.  The challenge here is coordinating all the services to make sure they work together and there aren't any "holes" of missing responsibility that will interfere with your business' success later.  A requirements document for everything you need and clearly documented responsibilities for each vendor are essential, as well as a software system to facilitate group collaboration among your vendors (document sharing, ticketing, helpdesk, etc.)  Many vendors simply don't want to get into the complexity of shared service delivery, in part because it's not possible to assure good quality of service - nobody is responsible because everyone is responsible!  Companies with good vendor management processes can be very successful with this approach - but it takes work.

- A combination of partners and your own staff.  This works well for many companies.  The partners provide services to you that are difficult for to provide with your own full time staff such as tasks that are infrequent or bursty and would require a large staff that was underutilized, or functions that are generic and have little to do with running your business.  General consulting, data center setup, system administration, 24x7 incident response, and cross-disciplinary strategic planning or systems design are good examples.  Partners - in this case MSPs - are also an excellent way to avoid cost and risk of owning your own hardware, especially if they have a pay-as-you-go utility computing offering.  You retain staff that understand your business and that you can keep busy, such as application programmers or program managers who are also responsible for coordinating your vendors.

- Find a partner who can handle all of your IT operations service needs on an outsourced basis.  Many companies are simply too small or too focused to productively provide their own operations services.  They need a partner who can set up and manage their IT operations for them, yet is invested in personally understanding their business needs so that they aren't giving up the advantages of a custom IT operations solution.  However, few vendors exist to provide this service, for one basic reason: it's very expensive.  If you need outsourced operations, you're going to face a large bill from an operations partner.  Most outsourced IT customers react by stepping back and reconsidering one of the alternatives listed above even if they are more expensive in the long run, since the total bill isn't visible up-front and people tend to assume that they are smart and can figure out a way to provide the service for themselves at lower cost.  My experience from working in both small and large enterprises was that this isn't a correct assumption, but I fell victim to it myself many times. 

Utility Billing For Risk Sharing - A Possible Solution 

Because of the investment a vendor will have to make in serving your business' IT operations needs, they typically charge you up-front for a year of service, producing a large bill that seem even more out of proportion to doing it yourself.   I think this approach corresponds to an unbalanced sharing of risk between them and you: you're taking a year's worth of risk, and they are taking none.  A small number of vendors, including ENKI, are finding that utility billing (paying for what you use on a monthly basis with only an installation or setup charge up-front) is a more equitable sharing of risk in what is expected to be a successful partnership between you and your vendor.   Whether you fully outsource your operations, or just a few tasks, asking for utility billing is a way to enter into a relationship of equality with your vendor that builds collaboration because you both have skin in the game.

In creating and providing our Computing Utility and Operations as a Service offerings, which roughly correspond to the two columns of the table above, we've had to ask ourselves many of the same questions I've discussed above.  Since ENKI itself isn't on the scale of a large corporate IT department, we've chosen to only develop critical capabilities in-house, and build relationships with trusted partners for our customers' remaining requirements.  A unified CRM system ties us together with our partners to provide seamless service to our customers.  I don't claim that we have canned solutions for any new customers' needs - whether covered in the table above or not - but I do have confidence that between ourselves and our partners, we can rapidly determine the best solutions if we do not.  And as an additional benefit, we don't keep resources on hand to meet every possible need, which allows us to beat the competition - including our customers' previous solutions - on cost.  This approach corresponds to negotiating utility billing with our own vendors, and passing the savings on to you.

Finally, ENKI has virtual data center grid-based computing technology that allows us to deliver computing services to our clients with automatic reliability that avoids the need for us to have a large staff.  Because our grid-based computing is scalable, as our customers' needs grow we simply turn up a knob and they recieve more computing power in one or more virtual servers, paying only for what they use each month. 

I would be very interested to hear from you about the challenges you face in meeting your company's IT needs, and what you need from an outsourced IT provider to meet those needs.