One of the questions I get asked most often by prospective customers is how they will be able to release new versions of their custom software to their websites under AppLogic or VMWare.  Generally, ENKI does not allow customers to have root access to their own production websites if they place responsibility for keeping them up onto us, so that we can keep track of any changes that are made in order to provide maximum uptime and minimize flailing about to find out who changed what.   Database access is provided, however.  I'll describe our release processes in this context.  If you have your own Cloud installation, you'll want to partition responsibility in a similar way between your developers and your operations staff in order to have reliable and short release windows.

Our Operations as a Service (OaaS) basic release processes are described below, but bear in mind that the actual process has to be flexibly defined depending on the structure and design of your application, and may require a custom process.

No Database Migration Needed

1. We create a copy of the production site to make a test environment.
2. The customer copies the release candidate code via scp/ssh into the test environment and tests it.
3. When the testing is complete, the customer stops the production environment and notifies us that the test environment is ready to be released.
4. We rsync any changes to the live site data from the production environment into the test environment where appropriate.
5. We alter the IP addresses to make the test environment be the new production environment and restart it to complete the release. 

Database Migration Needed

1. We create a copy of the production site to make a test environment.
2. The customer copies the release candidate code via scp/ssh into the test environment, migrates the data, and tests the release candidate.
3. The customer creates a patch process document describing the necessary steps to update the code and run the migration, and provides an SCP server for ENKI to retrieve the updated code - usually in tarball form - from (which may just be the test environment.)
4. When the testing is complete and the release is determined to be ready, the customer notifies ENKI, which stops the production environment and restarts it on a temporary IP address.  A site-down server is started to notify users.
5. ENKI deploys the code updates and runs the migration according to the patch process document (or, alternatively, the customer runs the migration).
6. Customer tests the production site and notifies ENKI when they are satisfied.  ENKI alters the IP addresses of the production environment to bring it live and  restarts it to complete the release. 

Minor patch

Customer notifies ENKI and provides an scp server with the patch on it, together with patching instructions. We stop the live site, perform the patch and restart the live site.

Minimizing Down Time

Clearly there are are quite a few variations that can be made to these processes, depending on the customer requirements.  For example, if the live site cannot be allowed to go down for more than an absolute minimum, the migration-based process could be modified to include keeping transaction logs on the live site while the patching and migration took place, followed by a database update from the logs while the live site was down. This of course depends on selecting a database that can reliably work with transaction logs.

We're well aware that many of our customers are in the volatile social networking, SaaS, or mobile application space where downtime can cause a permanent loss of customers due to impatience or bad publicity, so it makes sense to create a custom release process that minimizes the risk.  One of the great benefits of AppLogic or VMWare when combined with on-demand or utility computing service is that you can work out all the kinks in your release process on an instantly-provisioned copy of your live site while customers are still using the previous software version.

Well here it is six months after I wrote the first article about ENKI's search for a CRM system, and we finally did choose one and started our implementation.  After a lot of looking around and testing, we ended up choosing NetSuite, a SaaS service.  It wasn't an easy choice: I was quite certain that none of the systems I looked at would meet 100% of our needs, but NetSuite with its full spectrum of features (from web store to CRM to accounting) together with script-based customization, seemed like it would fit the bill best.  I haven't changed that opinion after 5 months of using it, but I'm also not completely happy with our purchase.  Read on...

After living with NetSuite for 5 months, in which we moved our accounting records for 2007 to it, switched to using it for sales automation, and have started to use it for support automation, I have to say that I'm very impressed with what it can do out of the box.  If you've read this site carefully, you will know that I worked there earlier in its history, but even though I was in charge of the development department, I never had a chance to experience the full range of its capabilities.  In most cases, whatever I want to do is much easier than QuickBooks, and offers more capability.  Reports, for example, offer a great deal more customizability, and the ability to monetize almost any interaction with the customer is incredible.

However, it hasn't been a honeymoon.  Like most lower-end ERP and CRM products, NetSuite has a built-in business process for everything, which unfortunately doesn't match the processes my business needs to succeed.   Or, it has no process at all and you're stuck with having to remember how to do something right.  Unlike Fortune-500 ERP and CRM systems, Netsuite isn't built on general purpose software frameworks that support the underlying essential "atomic" elements of business process automation such as event handling, event scheduling, and conditional event flow. This leads to trouble in getting it to do what you want, even with it's scripting customization capability.

There are two main areas that I am concerned with.  The first is billing: we bill our customers on a pay-as-you-go basis for services rendered and resources consumed each month.  This neither fits into NetSuite's memorized transactions or their Advanced Billing(available on an upcharge) models, since the billing is recurring, but always different.  The way around it is for us to write custom software in NetSuite's Javascript customization language that calculates the customer's bill and creates an invoice using NetSuite's web services access.  This is in process, and I have hope that it will work out.  For now I stay up late at the end of each month doing billing manually!

The second area is our support process.  We offer a variety of service level agreements (SLAs) to our customers, some of which have guaranteed response times of as little as two hours.  During that time, the system has to capture the customer's request, notify a responsible on-duty support engineer, and find an alternate if there is no acknowledgement.  However, NetSuite's capabilities for measuring elapsed time and acting on time-based settings is only accurate to hours, not minutes, and they only check the status of time-based tasks every half hour.  The result is that the process I described above, under worst-case conditions, can take up to 1 1/2 hours to notify an alternate support engineer, which could cause us to miss our guaranteed response window.   Worse yet, there are bugs in NetSuite that keep the case escalation from working as documented so we don't reliably receive email notification that our cell phones can react to and start beeping for attention. So for now, we at ENKI can't take any time away from our cell phones, and have to check them constantly for customer support cases.

Also, some critical functionality we need to access and respond to cases with our cell phones via email is not part of the product.  I've written some JavaScript to work around this, but there are more bugs that keep that from working as expected.  With NetSuite's poor record on fixing bugs that might be critical to only a small segment of their customer base (according to posts on their users' group bulletin board), I despair of any quick resolution.  These problems are so severe that with ENKI's continued growth, we may need to replace NetSuite just to be able to deliver our core value proposition to our customers.  This would be a shame, since in general I really like the service - and I don't have time for another migration and implementation. 

We've also had heart-stopping excitement from time to time in other areas.  A few days before Christmas, I wanted to use a NetSuite Campaign to send my customers a Christmas thank-you card.  Unfortunately, I missed un-setting the "new customers are unsubscribed to campaigns by default" checkbox when I set our implementation up, and now I can't mail my customers, and I can't change the setting for existing customers due to NetSuite's ill-considered anti-spam policies.  I had to export the list to excel, and then use a third-party mail program.  The cards went out on time.  I just didn't get any sleep.

Since our implementation, one of our partners has suggested Enterprise Wizard as a solution to our problems.  A quick tour through that service showed me that they're focused on implementing their customers' unique business processes through an easy graphical editor.  This would be perfect for us, but it's missing NetSuite's financial aspects and its webstore, which I've started to use to sell our off-the-shelf virtual private server products.  Writing an accounting system in Enterprise Wizard isn't my idea of fun!

This experience has shown me that even with today's Software-as-a-Service ERP and CRM applications, automating your company's processes is still a difficult, expensive, and frustrating process - as it was 10 years ago.  When it starts working, however, the benefits can be incredible, saving you money on staffing and allowing big-company marketing, sales, and support services that you couldn't achieve any other way. 

Some learnings I take away from this humbling experience so far include:

1. Don't be in a hurry.  Sticking with what you have a little longer is always better than switching systems if you feel under pressure.  The switch itself is a lot of work, and working around an incomplete evaluation is expensive or could even cause your roll-out to fail.
2. No system will do everything you want, or if it does, it's oriented for a budget that SMBs can't afford.
3. Ask your vendors and customers how they solve the problems you're considering buying this software or service to address.  Use networking opportunities as well, such as online groups, professional groups, and associations.
4. If you get a system that does 90% of what you need, be prepared to hire consultants for extensive programming to achieve the last 10%, or spend some time learning today's web2.0 technologies to do it yourself.
5. Be creative.  If it turns out the system doesn't do something you need, there's usually a workaround if it allows you to export data or access it remotely via web services.  You may be able to change your business processes to match those of the system as well.
6. Implement as much of your needs as you can in your test drive or trial of the product before buying.  However, don't let a vendor charge you for any sort of testing or application consulting prior to purchase: it sets a bad precendent for your future relationship.
7. Pick your implementation partners well, or hire your own programmer for customization.  It doesn't make any sense to implement a customization five times because your partner doesn't understand your business.
8. Document your actual and/or desired business processes prior to buying.  This will help you with your evaluation, and you may be able to extract a money-back guarantee from the system vendor's sales staff that they can meet these requirements.  This is not easy - but you'll need to do it sooner or later anyway if you are going to hire consultants to customize your system for you, or if you want to go for SAS70 or ISO certification.

As usual, I'd love to hear from you about your experiences, or advice!

I came across an interesting blog article today about making a reliable MySQL database cluster on EC2.  The author goes over the pain of working around two limitations that EC2 has: lack of immutable (unchangable) storage associated with an EC2 instance, and the lack of a static IP address for an EC2 instance.  However, EC2 isn’t the only game out there for utility computing.  Services such as ENKI's Computing Utility, based on 3Tera's AppLogic grid operating system, can provide scalable utility-billed cloud computing at higher levels of reliability than EC2.

With AppLogic, you get 3-nines “right out of the box”, and 4 or 5-nines can be achieved using some of the techniques mentioned in the blog I was reading. AppLogic’s 3 nines comes from its ability to automatic restart on a different, working server any virtual servers that were hosted on failed hardware. AppLogic is oriented around “Applications” which are networks of Appliances in a private address space - essentially a virtual private data center. Even if an Appliance is restarted after a failure, it gets the same DNS name inside the Application that it had before (as maintained by AppLogic) so other Appliances in the Application can continue to refer to it successfully. The automatic switchover usually takes just a few minutes after a server fails.

Where AppLogic goes beyond EC2 is that virtual server images, called “Appliances” in AppLogic, are object-oriented in the sense that the configuration information for the server (application, setup files, patches, etc) is maintained separately from the server instance itself, so if the server fails, it’s restarted from its parent “class.” Clustering or parallel instances are supported because each Appliance's unique configuration information (including config file settings entered from the AppLogic user interface) are loaded into the Appliance when it restarts, yet all the Appliances in a cluster can share a single NAS Appliance and hence share a single deployment of code or data.  To get better performance in, for example, a Java/Tomcat environment you need only add another Tomcat instance to your Application.

User data is stored in a RAIDed volume or on a NAS appliance with its own RAIDed volumes within the grid of redundant servers that AppLogic runs on, which means that it is available without interruption even if a server goes down.  Again, this storage is more reliable (and available at a lower cost) than Amazon's S3 offering - and it's within the same grid of hardware, connected by gigabit links for better performance.  

There are currently 3 ways to use AppLogic. and ENKI can help you get started with any of them. You can purchase utility computing from ENKI on a pay-as-you-go basis which allows you to use a virtual private data center (AppLogic Application) and scale it on demand. In this case, you don’t need to know anything about AppLogic to use it since we manage it for you. You can also lease a stack of servers from a managed service provider offering AppLogic and manage it yourself or hire us to manage it for you.  Or, if you’re working for a big enough company, 3Tera will lease AppLogic licenses to you for use in your data center, and we can supply training and support.