Over the last few months, I've written about how to analyze and reduce power consumption in your data center.   I'll continue to write updates as I learn more that I can share with you.  What I've noticed is that this topic is of great interest to you - I get more hits on this website from power consumption searches than from any other kind of search.  It's almost as though some of you are desperate to reduce power consumption.  Or is it cost?  (Leave me a comment or email - I'd love to hear from you.) 

I've avoided giving stock, off-the-shelf solutions for this problem up until now.  Depending on the design of your data center and how your company is structured, the answer is obviously different for each of you.  In fact, you may want to engage a consultant to help you with it.  However, there is one answer that is so big, so obvious, so landscape-changing that it is the Elephant in the Living Room, and I didn't think to suggest it until now.  It's to outsource your data center!  Now, you'll probably say to yourself that I must sound insincere, that I couldn't possibly have missed this, considering that it is ENKI's business.  However, ENKI wasn't always in this business: we used to focus on helping clients with their own data centers, which is why these articles got started.  Also, like a good engineer, I was focused on reducing power - but not to zero!

What's really in play here is that if you're interested in reducing power in your data center, you're really looking at *cost.*  For those of you who want to do the Earth a favor and run a green business, I applaud you but in my opinion the choice to run a green business isn't an option anymore: nowadays, efficient business *is* green business.   So what we're really talking about is cost, which drives green considerations as well.  And that's where outsourcing your data center comes in.  For many companies, the staff, management attention, and necessary skills to run a small to medium-sized data center efficiently and reliably are not economically justifiable anymore.  You can see an example I worked up based on the experience of Digg here.

Instead, you should be looking for an OaaS (outsourced operations as a service) provider that can partner with your company in a way that allows you to trust them with your information technology infrastructure.  We have created ENKI to be just such a provider.   Our committment to win-win business relationships means that we are always open to negotiating a solution that works for you, even if it doesn't appear in our website or documentation.  Our virtual private data center technology means that your data center architecture can be moved to our hosting environment without change, but with the added advantages of automatic failover and nearly instant provisioning of multiple copies of your application - perfect for testing, pre-deployment, or software-as-a-service providers.

And of course, we're following the power consumption best practices that I've been writing about, not the least of which is using virtualization to use our hardware as efficiently as possible while providing you service at the best possible value.  So you can sleep well and be green at the same time.

Recently, we partnered with 3Tera to provide support to AppLogic customers, and with another company as well for professional services and support.  We need a way to share customer issues between the three companies, moving them to appropriate people for resolution and yet keeping other company confidential information private.  In addition, we would like to empower our customers to create or check on the status of issues they submit to us.  My CEO would dearly like a way to do more than keep track of sales leads, but rather automate the sales process so that every prospective customer gets adequate sales attention to be sure we've explored the possibilities for a win/win relationship.   So far, ad-hoc tools like excel, Outlook, and email are doing some of the job for us, with great effort from all concerned.  I decided to start looking for a solution that would cost us about $5000 for the first year, for six users.

Having implemented a worldwide CRM system for Hewlett-Packard about 6 years ago, I was sure that by now the benefits of CRM would be easily available to small and medium businesses.  What I have been finding is that there are a lot of products claiming to provide the benefits of CRM, but very few of them deliver on that promise.  And none of the SMB CRM products support barrier-free collaboration of the type that ENKI is looking for, though some come close.  I see small businesses constantly collaborating as I speak with my customers, but I get told repeatedly by CRM vendors that they've never heard of such a thing.   Teamwork, maybe.  Collaboration, no.  

Being a big proponent of open-source software, I thought I'd start with Sugar CRM, a supposedly complete open-source system.  Sugar is developed as open source, but then the company which employs most of the Sugar developers, SugarCRM, adds another 50% functionality and sells it as a product.  The open source version is almost a toy in comparison to the for-pay version, which is priced at close to $500 per seat per year.   Both versions are a bit behind state-of-the-art Web2.0 technology and  are a bit clunky to use.  Essential features which ENKI needs, like a customer Portal for self-service support, cost extra.  Not only that, but SugarCRM sells you the software to install on your server, but then charges you the same amount every year.  That doesn't feel like win/win to me, and it's a strange Software-as-a-Service model - without the service.  Sugar's hand always seemed to be out, begging for more money, but even worse they seemed to be interfering with the open source aspect of the product by releasing an enhancement and then mysteriously having their employees stop working on the previously free open-source version of the same feature.  Hey, everyone's gotta make a living, but if you say something's open source...

I also looked at Info At Hand, another proprietary customization of Sugar CRM open source code, with at least 50% more features added, including a spiffy simple cell phone interface, enhanced customizability, and some features for front-of-the-house financial processes that would greatly enhance our Quickbooks-based accounting.  However, the Long Reach corporation, a small company that provides Info At Hand, seemed overwhelmed by my pre-sales support questions.  Their "free trial" online version was neither: I ran into bugs or intentionally disabled features, which when added to spotty documentation meant that I couldn't get some important features to work.  They suggested that I purchase the software and install it, and then they'd support me.  Somehow, I couldn't bring myself to buy and install something just to see if it was right for me, especially since my interaction with them produced doubts about how well they'd be able to support me after the sale.  While their prices for support are reasonable, the limit of one case per seat per year had me wondering what would happen if I asked more than six questions for my six seats, especially when I already had more than six questions written down.  I may yet follow their suggestion to buy and try out of frustration with the alternatives, but I haven't gotten that far yet.  

There's another open-source CRM product, called Centric.  I'm evaluating it now.  I don't think it will meet my needs because it's not as configurable for some special requirements I have, such as tracking which hardware a customer of ENKI's is being hosted on.  Also reports and workflow customizations have to be written in Java, something nobody here at ENKI has time to do.  The jury is out, but at least the company is considerably more engaging and listening-oriented than SugarCRM or Long Reach has been.

I also took a look at the two "biggies" in hosted CRM: Salesforce.com and NetSuite.  I was struck by how unmotivated the SalesForce sales team was, taking days to contact me or respond to questions.  Well hey, it is August.  Or maybe their product is selling so well that they don't need to call back.  I can believe it: the tool itself is very user-friendly and intuitive.  I instantly fell in love with it, but at $1500/seat/year, you need more than love to make a purchase, you need what you need to run your business.  I haven't been able to answer that question yet.  What's nice about their product offering is that the pricing is very clear.  I can tell what I'll be paying a year for now if I grow by a few seats.

I also took a "test drive" of the NetSuite product.  If you've read this site, you know that I used to manage their development team.  So, I have some knowledge about how the product works and more than a little loyalty to the company and the people in it as I remember my time there with great fondness.  However, the product is very expensive, so I can't let my feelings get in the way of making a sound judgment on whether the software is right for my business.  My "test drive" was quite successful and I was able to demonstrate that the system could do what ENKI needs.  Also, NetSuite's sales people went out of their way to make sure I was successful with the trial, including spending some time with me on a Saturday.  I'd really like to buy the product, since its features both support my partner-collaboration model as well as provide a great customer interface.  But their pricing model is confusing with lots of extra-cost options that you might find necessary as you try to get the system to do what you want.  The cost of the system climbs rapidly and unpredictably as your company grows.  So, it's really difficult to know what I'll be paying in a year, or even six months.  That's pretty scary, and I have sympathy for their salespeople:  it must be quite difficult to sell the product.  The initial investment for ENKI is probably around $10,000 - including implementation assistance and fixed yearly charges that disappointingly contradict the pay-as-you-go promise of Software-as-a-Service.

I also looked at some expensive traditional boxed-software products, but they are well over my $6000 desired budget, often heading north of $20,000 for fewer features than the other products I looked at.   Or, you can go to Fry's (the local computer store) and buy boxed software that will act as a fancy contact manager, which is far short of my needs.  I even tried a free trial CD of Microsoft's Dynamics CRM, but couldn't figure out how to use it.  It looks totally unconfigured, like a blank slate.  I called them to confirm that what I was seeing was what they were offering either as a boxed product or as a hosted service and they said yes.  In my business, I need to be able to run with the product out of the box, even if I know there is lots of configuration in its future, so that choice was out.  There are also a gaggle of CRM systems which focus mostly on sales and marketing - often little more than intelligent spam generators.  The worst offender I found was a product called "Infusion" whose vendor not only bombarded me with chatty overly-familiar sales entreaties, but also seems to be responsible for a lot of the other chatty spam that I get on a daily basis from other companies.  It must work for them, as the chatty ads claim, but it doesn't work for me!

It's been a pretty discouraging journey so far.  Either I have to get in bed with a vendor who clearly doesn't understand ENKI's win-win business model and isn't likely to be a successful partner for us, or I can spend much more than I'd like to get something that will do the job.  My other option - a suite of single-point solutions for sales, finance, support, and knowledge management is looking better and better, though I learned rather painfully at Hewlett-Packard how difficult it is to manage "islands of automation" as your company grows and needs change.  In any case, the cost of setting up a CRM system isn't just in the software: it's also in the initial customization you have to make to the system (and your employees) to support your own company's workflows, as well as all the data you have to enter before it becomes useful as a business tool.  

So for the moment, there isn't a solution that I like.  I'll keep you up-to-date on my adventures here in the blog, or - if all goes well - you will simply see the CRM system in action on ENKI's web site.  Or so I hope! 

Here at ENKI, we keep up a dialog with our customers about what we can do to make them more successful.  From the time we founded ENKI, we have envisioned providing IT operations services to customers who didn't want to build their own operations department.  Over time, we have added capabilities and products, such as our Computing Utility, to achieve this goal.

This evening, I was having dinner with a principal from one of my consulting customers, a Web 2.0 startup.  He told me about his critical dissatisfaction with his current hosting vendor.  His analysis was that they just want to lease him hardware, and that they can't support that mysterious interface between his web application and the hardware and operating system they're selling him, leaving him to develop IT operations skills in-house.  He wants his company's IT operations to just work, so he can concentrate on making his business succeed and prepare for his go-live event.  His CEO is putting pressure on him to solve the problem, fast.

Sure, he knows someone who used to set up data centers for banks, and he could call this guy and get a data center built for his company.  But the cost, delays, and hassle seem overwhelming to him.  But he'd do it if he had to.  Or, he could continue to lease machines from a hosting company, and hire an administrator to configure the hardware and networking to support his application - which could include clustering and other configurations that the hosting vendor might not support.  But in any case, he would be responsible for designing and supporting the infrastructure, which he finds costly and difficult since his expertise is in creating software applications.

Over the course of the dinner, I heard more and more about what he really wanted from his hosting vendor:

1. Computing delivered as a utility, on a pay-for-what-you-use basis, so that he doesn't have to invest in capital equipment or excess capacity.
2. A predictable sales model and pricing so he knows what he'll be paying as he grows and he can trust his vendor to be a partner.
3. Ability to rapidly scale up hardware resources in his data center, such as if his company has a successful launch.
4. High reliability of the hosting service.  This could include rapid failover, data backup, and more.
5. No restrictions on how his software is written, so that he can simply deploy it into the data center.
6. Adequate support to ensure that any problems he has are addressed quickly, whether they are handled by the vendor or his staff.
7. Deep technical knowledge regarding IT operations from his operations vendor, so that he can get any assistance he needs to quickly roll out his application.
8. Infrastructure technology that allows easy management, scaling, failover, and provisioning of his computing infrastructure and applications. 
9. A sincere desire on the part of the vendor to partner with him for his success, with the flexibility to match his particular needs
10. Any and all services from the vendor so that he'll never need to build an IT operations capability himself, nor seek funding to pay for it.

I've started calling this set of offerings outsourced Operations as a Service, or OaaS - sort of a counterpart to SaaS, software as a service.  I've modified ENKI's website to communicate this value proposition.  I'd be interested and grateful to hear from you about what you think are the essential elements of an OaaS strategy that would meet your company's needs.

Being a science fiction fan and a computer enthusiast, I was bound to come across Neil Stephenson's novel, "Snow Crash" sooner or later.  It is a story of the near future, in which large corporations control the world and the governments of the world are subservient to them.  Much of the true action in the world actually happens in the "Metaverse", a simulated world akin to Second Life.  The story revolves around a real-world hero, named Hiro Protagonist, who struggles to save the world - real and simulated - from a techno-fundamentalist guru who is controlling the minds of his followers using ancient Sumerian knowledge, originally developed by the Sumerian god, Enki.  It's a great story, and I won't spoil it for you... but it had enough information in it about Enki to pique my curiosity. 

Later, I read more about about translations of the ancient Sumerian history and cosmology in books by Zecharia Sitchin, who translated ancient fired clay tablets that told the story of how Enki created humans through gene splicing of apes and an alien species, of which Enki was a member.  While Sitchin's translations are often questioned, they do tell a beautiful story of how Enki, a supremely capable scientist, created humanity, fell in love with it, and tried to save it from a catastrophic flood - the precursor to the story of Noah in the Bible.  And the lack of other competent translations that challenge Sitchin's interpretations leaves a delicious, lingering question about Enki's actual existence hanging in the air.

Enki actually translates in Sumerian to EN.KI, or honorable ruler of Earth.  His playful demeanor, technological savvy, love for humanity, and example of capable and enlightened leadership all are reflected in the intentions behind the founding of ENKI.

I was having lunch the other day with the CTO of a successful web startup which is on its way to financial break-even, usually a good sign that the business model is working and the venture capitalists will support the company until it goes public.  He asked me about ENKI's Computing Utility offering and why it would be advantageous for a company like his to use it.

Ultimately, for a start-up company living off of investor cash, the name of the game is to be able to break even with the smallest investment possible, which allows the founders, investors, and employees to make the largest profit.  Even with a great business model and happy customers, this means controlling costs.  In the late nineties, wild parties, huge speculative marketing experiments, and vast, optimistic computing infrastructure build-outs were the rage - but most of those companies ran out of money, contributing to the dot-bomb crash.   Working in that world, I always wondered what happened to business fundamentals: careful investments and cost control.  Nowadays, these best practices are back in style even in the high-flying world of web startups, in large part because investors are more careful about how they spend their money.

This is where utility computing comes in.  Instead of investing in your own data center and staff to maintain it, you just buy as much computing as you need.   Large enterprises talk about depreciation expense, taxes, and capital cost as part of their break-even financial models, but for the startup it boils down to "how much money is in the bank and how long can I run with that?"  So buying computing as a service makes a lot of sense.  But how much sense does it make?

Recently I was at the web2.0 conference and attended a talk by one of the founders of Digg, who shared a little about how much computing resource they were applying to their business.  After getting a rough idea of what they were doing, I asked myself how much it was costing them, and if buying utility computing from ENKI would have helped them out.  A little time with Excel and some numbers I put together from published colocation costs got me a spreadsheet that said that a Digg-like company would save about a million dollars over a year and a half on data center costs.

The spreadsheet can be viewed here .

I had to make some assumptions, particularly that the start-up was trying to run very lean, working their IT staff hard, in my experience to the edge of burnout.  This seemed in line with Digg's staffing count and my experience with web start-ups.  I also had to make some assumptions about how the startup's application would map into ENKI's grid technology, which allows for much higher levels of server utilization and failover capability with less hardware.  So the numbers might be off by 20-30%, especially depending on the startup's IT philosophies.  But no matter how you cut it, paying as you go is what a start-up should do.  

Another thing I discussed with my CTO friend was if a start-up could outsource their computing needs with confidence.  In my experience working with start-ups, their computing infrastructure succeeded or failed based on the quality of their IT staff and management.   At ENKI, we have an experienced team, but we also make the interface with our customers easy by not just selling hosting, but rather consulting with our customers to implement their applications successfully on our computing resources.  I accounted for this cost as best I could as a per-server monthly labor charge.  It is surprisingly low, but that's the benefit of the grid technology we're using.

In the quest for ever higher computing efficiency, we see vendors touting more and more "green" hardware, which I talked about a bit in the last data center power consumption article.  This might lead one to believe that having the most effficient hardware is the best way to reduce data center power consumption.  However, it is not.  The true key to efficient computing is your software.  Software approaches to saving data center power fall into two categories: virtualization, and optimal applications.

Optimize your Applications!

Have you ever wondered how NASA can send a probe to Jupiter with multiple computers in it, running off just a few tens of watts?  Aside from special-purpose computers, those probes have software in them that's written for efficiency, usually in a very efficient fully-compiled language.  Engineers have pored over the code at length, removing every inefficiency, and using the best algorithms for the job.  If you're building a digital services company, like a web startup or Saas provider, it pays to think about efficiency up-front.  In my personal experience as a coder and software manager, I've seen efficiency improvements through careful coding of 1000% or more!  You can take that to the bank.

The keys to achieving high software efficiency are a deep knowledge of programming techniques and the languages and environments you're planning to implement your applications in, as well as good data on the performance of any server applications you plan on buying.  Often, efficiency can be assumed to be the same thing as speed: the faster the program executes on a given computer, the more it gets done for the amount of power and equipment you have invested in running it.  For software you write or have written for you, an execution profiler is an indispensable tool.  It will show your developers where their code is spending most of its time, and allow them to focus on speeding up the problem areas.  If an execution profiler isn't practical, you can still get similar information by putting timing checks into suspect code. Your choice of language can greatly affect the efficiency of your software, so it can pay to choose your implementation language carefully, or implement compute-intensive sections of code in a more efficient language. While today's rapid application development pace may let you get to market quickly, it can also cause your team to skip the optimizations that will save significant costs on IT infrastructure later.  Many startups find themselves "refactoring" (rewriting) their applications within a year or two just to be able to meet customer demand, not to mention containing IT costs.

For acquired software, it pays to do evaluations under the types of loads you expect to stimulate them with, or read software reviews very carefully.  If you're purchasing based on reviews, be sure the reviewer has done throughput testing on the software that is similar to the conditions you plan to run it under.    Databases in particular can eat up lots of computing resources, so which database you choose may heavily influence how much computing power you will need to run your application.  Proper setup parameters are critical to getting the most out of purchased software, including setting up the operating system correctly and normalizing the data in your database.  Often, application execution environments, such as the popular LAMP (Linux, Apache, MySQL, PHP) can be tuned to have a wide range of performance on the same hardware - and, you can often replace the database or another component with one that is radically more efficient.   Finally, when your application is running, don't forget to circle back and make sure that your hardware is tuned to support it: applications are either RAM-limited or CPU-limited, so building your servers to take this into account will allow you to reduce the number of servers, often dramatically.   Another important tuning technique for today's multimedia applications is to choose the storage solution that matches your software's data throughput requirements: different types of RAID or NAS appliances can greatly influence how much hardware you need to get good performance.

Server utilization is the key to saving on power and real estate

So, now that your application is running on its servers as efficiently as possible, what more can you do to increase efficiency?  Well, most CTOs will find themselves choosing to run their servers at 5-25% average load, to make sure that peak loads don't cause excessive response times for their users.  However, if you remember from our hardware discussion, much of the power drawn by a server is independent of load.  Our standard dual-core double-Opteron 1U servers draw about 200 watts under full load, but still use about 130W when they're idling.  (We put a minimum of disks and other accessories in our servers, so they use less power.)  So, you can see that our server running at 10% usage still uses 66% of its full power.  In other words, if you were able to keep the server busy at 100%, you'd get six times higher energy efficiency, and ten times more space efficiency!

Server virtualization allows you to run multiple operating system environments, each potentially with multiple applications, on the same server.   Virtualization splits up the compute and memory resources of the server, making each virtual machine think it has the server to itself.  Clearly, you will have to provision your server with adequate resources (including CPU and memory) to allow each virtual machine to adequately run its applications.  There are a couple big names in the virtualization space: VMWare, which is proprietary code, and Xen, which is open source.  Xen is also resold with support, utilities, and management software by Virtual Iron and XenSource.  These systems are not cheap, though the Xen based systems can be more cost effective.  They are also problematic to set up and operate.  Not the least problem of which is knowing which server contains which applications in case of hardware failure!  Also, your administrators will have to pay more attention to server utilization, and move applications on and off them in order to use them optimally.  For this, I recommend  a good monitoring application that can tell you when things are headed south, as well as some of the virtualization management suites that are becoming common.

Despite the drawbacks, the payoff for virtualization is huge.  For example, if you run 5 applications that have an average CPU usage of 10% on a virtualized server, you'd still have 50% headroom and enjoy a 5X hardware cost savings, including energy usage.  Many large companies are virtualizing their whole data centers, together with new high-density blade servers, and getting giant payoffs in space and energy savings.  But even if you just have a rack of 4 servers, virtualization can bring you significant benefits.

Here at ENKI, we use a virtualization scheme based on Xen to run our data center.  However, on top of the virtualization, we use AppLogic, a Grid Operating System from 3Tera that allows us to build Virtual Private Data Centers on top of a grid of identical servers.  AppLogic can dynamically scale the resources each application is allocated as well as bring in cold standby machines automatically in case of failures.  This allows us to provide our utility computing product.  As a result, we only deploy as much computing power as our customers need, allowing us to run green and pass the savings on to them.  It also allows us to run our servers at even higher levels of utilization, since we can easily provision additional hardware resources and move virtual servers to them to handle spikes in demand.

To continue the discussion of data center power consumption, I'd like to take a quick look at alternatives for addressing power utilization by optimizing the hardware side of the problem.

Before I begin, I wanted to share an interesting link  with you about the power consumption of each avatar (simulated person) in the online simulation Second Life.  I wanted to say it is amusing, but it actually raises worrisome questions about our whole IT infrastructure and who ends up paying these energy costs.  The article points out that the average avatar consumes about as much power as a Brazilian citizen.  You can read more here.  In any case, the article reminded me how important the link is from a company's business model through its software implementation, then through its hardware, then the electricity bill and finally the bottom line that investors will see.

As I mentioned in the first article, power supply efficiency has a large impact on server power consumption and heat generation.  If you are buying servers in small lots, it's hard to have control over what's inside your servers.  For example, the average server power supply is 65% efficient.  Compare this to Google, which custom-builds their own servers from selected off-the-shelf components and achieves claimed power supply efficiencies of 90%.  In ENKI's data center, we're also building our own servers to optimize the hardware environment for our business.  After optimizing for the cost of physical space for the server and equipment costs, we don't always have the flexibility to choose the highest efficiency power supplies.  However, whenever possible we choose 80+% efficiency power supplies.   For a 500-watt server, this translates to a 144-watt reduction, or about a $150 yearly savings (at an assumed electricity cost of about $0.12 per kilowatt hour.)  Recently a number of large vendors including Dell and IBM have started to deliver servers with power supply efficiencies of up to 91%, but at a price premium.  Given the inevitable electricity rate increases and the fact that you will also save on cooling costs, these servers can save you a significant amount over the life of the equipment.

 
If you're buying larger quantities of servers, it may make sense to buy them with DC power supplies which are much more efficient, often exceeding 90%.  A few vendors offer this option but before you go for it, make sure that you have DC power available and it is reasonably priced.  Many data centers charge a stiff premium for supplying DC power, including a provisioning fee.  Also, you'll need to rewire your entire power supply distribution system for the higher currents that low-voltage DC requires.  You'll want to balance these extra costs against the energy savings!  Another option is making your own DC power.  Rackable Systems, for example, offers AC-DC converters (rectifiers) that go in the top of a rack of servers, and supply redundant DC power to each server.   Rackable claims a total of 30% power savings using this configuration (implying that their overall power supply efficiency is close to 95%), as well as increased server reliability because a lot of the waste heat is confined to the rectifiers in the top of the rack.  Once again, you'll need to sharpen your pencil to determine if the considerable extra cost of the Rackable solution will save you money during your planning horizon.

 
After the power supply, the next largest source of heat in a server is the processor.  Both Intel and AMD claim to have lower power utilization than the other, and each has an interesting story to back it up.  The current Intel processors use less power than equivalent-performance AMD chips, but when you factor in the power from their "chipset" (external support circuitry and memory controller) and the special FB-DIMM memory they require, AMD's systems may use less power overall.  Sun's "Niagara" processor was specifically designed to save power, and some reviews show it to be three to five times more efficient than the latest AMD or Intel processors.  Between different rating metrics and measurement philosophies from each processor vendor, it can be hard to tell which will be more efficient; so many experts recommend that you actually measure the power usage of each server you are interested in buying under your typical loads to determine which has the lowest power consumption.  We use a "clamp-on" current meter and test our servers with actual application software loads.  In general, you can save power by increasing the number of processor cores in a server, since the power-hungry memory and support circuitry is shared.  Blade systems offer some of the highest processing power per watt efficiency, but because of the large number of processors per case, they also draw a lot of power so you will want to calculate the total power draw per rack before you assume your current infrastructure can handle the current.  Don't forget that with the large power density of blade solutions, power supply efficiency becomes even more important. 

 
Finally, there are things you can do to reduce incidental power consumption from cooling.  HP, APC, EMC, and IBM have thermal management systems and services that can make a new or remodeled data center more efficient by controlling the amount of cooling sent to each rack of equipment and hence reducing cooling requirements.  These are expensive options best applied to an entire large data center, but these vendors claim to be able to save you significant sums.  On a smaller scale, Rackable has optimized cooling in their servers and fully-configured racks containing back-to-back servers with passive airflow technology.  This saves power used to run cooling fans and reduce temperature variation in your data center by eliminating the ubiquitous hot aisle/cold aisle layout.  Chatsworth Products Inc. has some nice retrofit racks that can do the same thing with your choice of server hardware.

 
One thing to remember is that data center equipment has a finite life.  Depreciation rules allow you to set the financial life of a server, and now that manufacturers understand that power consumption matters, you can expect increases in computing efficiency to occur regularly over time.  Thus, it makes sense to rotate your equipment through your data center with regular purchases to keep the average power consumption down, much like a good investor would purchase stock monthly rather than all at once.  This means you will want to plan for any growth and make arrangements to only purchase what you need in each period.

 
In my next article on data center power consumption, we'll move on to software and systems architecture options to reduce power usage, including but not limited to virtualization.

If you have read through the material on our website, you may have noticed that we emphasize the win/win business paradigm.  This term may be new to you, and I wanted to share some background and detail with you about the intentions behind it.  In a nutshell, the idea is that if we can help your business win, we will win as well, so our intention is to do all we can to position our customers' business to win. 

If your experience in the Information Technology industry has been anything like mine, you have probably found very few vendors who you could completely trust based on their true intentions and approach to business.  Instead, you may have often found yourself concerned whether they were really serving you in their interactions with you.  You may have felt lied to, manipulated, coerced, betrayed, or abandoned at various points as it became obvious that your vendor placed their own interests ahead of your needs, often without even letting you know this was the case.  I personally have been responsible for large IT projects in which (of course) a few days before rollout one of my vendors suddenly announced that their deliverable was significantly different from what we'd agreed upon.  They had interpreted a portion of the contract in their favor without telling me, never putting the learnings they had gained from months of contact with my organization together with a sincere desire to support my success by working out any details that were uncomfortable for them.  The result was that my project was late and I was forced to pay them much more than I'd budgeted in order to close the gap.  I felt quite betrayed, and certainly didn't want to use or recommend them again.  This was definitely not an outcome in which we both came out ahead.  In fact, despite their getting more money, in the end both of us lost out because they didn't build the goodwill that they needed to grow their business.

After numerous such interactions, I realized that my personal and professional life was only going to be meaningful to me if I could embody a different way of doing business based on a genuine respect for anyone I established a relationship with - starting with myself - and a sincere commitment to finding a way that the relationship could support both parties in equality.   I don't take this commitment to building a win/win business lightly: it means in many cases reinventing every part of my business and avoiding some of the quick paths to success that I learned in the past.  Figuring out some of the hows and whys definitely keeps me up at night!  But it's also exciting: my entire experience of being in business is fresh, new and uplifting, and I'm seeing that the vendors and customers who I connect with begin to feel as excited as I do as they slowly start to understand what this means.  But this understanding takes time, for all of us.  In many cases we have to allow that the way things have always worked in the past isn't the only way that they CAN work.

I'd like to share a few of the insights I've had on my journey in this article, but I'm sure that there will be opportunities to share more of them with you in the future.  For example, I've had to redefine what sales means for a company committed to win/win relationships.  Traditional sales techniques theoretically revolve around finding a need and filling it, which seems like a win/win activity.  But all too often they involve creating an artificial need, subtly or overtly coercing buying decisions, using discounts as a means to get the customer to ignore their own business need, creating a personal relationship just to get a sale, and so on.  Even more confusing is the fact that something like a time limit or discount may be grounded in a legitimate business requirement for the vendor, but ends up being coercive because of the intention behind it.  So I've come to the conclusion that bringing sales into the win/win paradigm is really a matter of intention. My intention is my commitment to win/win and equality in relationships which is reflected in Enki's charter with statements like  "...to create wealth for our clients and hence for ourselves."

While this statement of intention is simple, some of the differences in my behavior with respect to sales that I want to see are quite complex and nuanced.  I don't even begin yo claim that I have all the answers yet.  For example:

Technology Used

• Cisco 1841 Integrated Services Router
• Cisco WIC-4ESW a 4-port 10/100BaseTx Ethernet switch interface card
• Cisco IOS 12.4 software

Network Diagram

Router Configuration

 
version 12.4
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Cisco1841
!
logging buffered 8192 debugging
!
resource policy
!
clock timezone PST -8
clock summer-time PDT recurring
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
ip subnet-zero
no ip source-route
ip cef
!
!
ip domain name foo.com
!

!--- This is the connection to the Wireless ISP. We mark this interface as external
!--- so NAT will translate using our external NAT address space pool.

interface FastEthernet0/0
 description wireless-isp Wireless Interface
 ip address 172.16.0.194 255.255.255.240
 ip nat outside
 ip virtual-reassembly
 speed 100
 full-duplex
 no cdp enable
!

!--- This is the connection to the T1 ISP. Once again we mark this interface as external.

interface FastEthernet0/1
 description att Communications Interface
 ip address 172.16.1.130 255.255.255.240
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed 100
 no cdp enable
 no mop enabled
!

!--- Since this router only has 2 native Ethernet ports, and we need more that that
!--- it was necessary to add this switch module.  In order to get the traffic into the
!--- switch, we need to configure a Vlan. (see below)  The default Vlan for all
!--- ports is Vlan 1.

interface FastEthernet0/0/0
!
interface FastEthernet0/0/1
!
interface FastEthernet0/0/2
!
interface FastEthernet0/0/3
!

!--- This is where we configure the Vlan.  For the purposes of this example
!--- we are going to treat the Vlan interface just like an Ethernet Interface.
!--- We give it an IP address and tell NAT that is an inside interface.  This means that traffic
!--- coming into this interface from an "outside" interface will need to be translated from
!--- public address space to private address space.

interface Vlan1
 ip address 10.10.10.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
!
ip classless
!

!--- Since we have 2 active connections we need to have 2 default routes.  The wireless
!--- connection is much faster than the T1, so the wireless connection has a lower routing
!--- metric (0).  The lower the cost (metric) the higher the priority that route has when the
!--- router is making routing decisions.  In addition, if the 172.16.0.192 circuit should fail,
!--- the router will automatically take the 172.16.1.129 route because the other route
!--- would be deleted from the routing table until the circuit came back up.

ip route 0.0.0.0 0.0.0.0 172.16.0.193
ip route 0.0.0.0 0.0.0.0 172.16.1.129 10
!
!

!--- These are some NAT timings.  Since we don't have much address space here, we need to
!--- keep the table small and the timings short.  This client has about 45 employees and since
!--- we only have 8 addresses for each network, we need to be mindful of NAT table size and
!--- IP addresses.

ip nat translation timeout 3600
ip nat translation tcp-timeout 3600
ip nat translation dns-timeout 300

ip nat pool wireless-isp-pool 172.16.0.199 172.16.0.206 prefix-length 28
ip nat pool att-pool 172.16.1.136 172.16.1.142 prefix-length 28

!--- The NAT pool we use for public/private address space mapping is determined
!---by which network the traffic is going to take.

ip nat inside source route-map wireless-isp-nap-map pool wireless-isp-pool overload
ip nat inside source route-map att-nap-map pool att-pool overload

!--- This is where the fun begins and this section of the configuration is where everything
!--- comes together.  Normally there would be no need for the route-map here.  But because
!--- we have a requirement to run in active-active with the two ISPs we need to use route-maps
!--- to control the NAT process.  These route-maps look to see which interface the traffic is
!--- entering the router from, then they look to see which host the traffic is destined for.  If the
!--- incoming traffic is destined for one of our servers, then we need to force a NAT translation
!--- such that when the traffic needs to leave the router it will take the proper route.

ip nat inside source static 10.10.10.21 172.16.1.131 route-map exchange-att extendable
ip nat inside source static 10.10.10.22 172.16.1.132 route-map webserver-att extendable
ip nat inside source static 10.10.10.23 172.16.1.134 route-map commserver-att extendable
ip nat inside source static 10.10.10.21 172.16.0.195 route-map exchange-wireless-isp extendable
ip nat inside source static 10.10.10.22 172.16.0.196 route-map webserver-wireless-isp extendable
ip nat inside source static 10.10.10.23 172.16.0.197 route-map commserver-wireless-isp extendable
!
ip access-list extended nat-list
 permit ip 10.10.10.0 0.0.0.255 any

!--- The next three ACLs are for the route-maps.  They define what interesting traffic is.
!--- Here are the inside addresses of our servers: Exchange, Web and Communications.

ip access-list extended exchange-acl
 permit ip host 10.10.10.21 any
!
ip access-list extended commserver-acl
 permit ip host 10.10.10.23 any
!
ip access-list extended webserver-acl
 permit ip host 10.10.10.22 any
!

!--- The next six route maps determine the server-to-network matching for NAT addresses.

route-map webserver-wireless-isp permit 10
 match ip address webserver-acl
 match interface FastEthernet0/0
!
route-map commserver-att permit 10
 match ip address commserver-acl
 match interface FastEthernet0/1
!
route-map commserver-wireless-isp permit 10
 match ip address commserver-acl
 match interface FastEthernet0/0
!
route-map webserver-att permit 10
 match ip address webserver-acl
 match interface FastEthernet0/1
!
route-map exchange-wireless-isp permit 10
 match ip address exchange-acl
 match interface FastEthernet0/0
!
route-map exchange-att permit 10
 match ip address exchange-acl
 match interface FastEthernet0/1
!

!--- These are the default NAT route-maps.  These are used for all traffic that is not covered
!--- by the above server route-maps.

route-map att-nap-map permit 10
 match ip address nat-list
 match interface FastEthernet0/1
!
route-map wireless-isp-nap-map permit 10
 match ip address nat-list
 match interface FastEthernet0/0
!
!
line con 0
line aux 0
line vty 0 4
!
end

Troubleshooting

It is possible to verify that everything is working correctly by looking at the NAT translation table.  Use the following commands:

 
show ip interfaces brief
show ip nat translation
show ip nat statistics
debug ip nat [list] [detailed]

Last month a client asked us to configure a wireless Internet connection to augment their preexisting T1 connection from a local ISP. The client had been experiencing high latency in their Internet traffic with occasional losses of service.  Fault tolerance in their Internet connection is very important to them and their business processes because their business model is highly dependent on worldwide network connectivity.

They decided to purchase a wireless connection from a local ISP to augment their existing bandwidth and solve their reliability problems.  Once installed, this would give the client an additional 2-3 Mbits of wireless connectivity.  They purchased a Cisco 1841 router with a NM-4ESW WIC as the network gear that would be managing the connections.

What's very interesting about this situation is that the company wanted both circuits to be active and configured in a fault tolerant manner such that if one circuit failed all internet traffic would transparently be routed through the working connection.  In addition, when the failed circuit came back up, the traffic would transparently return to the active-active flow model.

In addition, they had 3 servers that needed to be exposed to the Internet as part of their day-to-day operations.  They were the Microsoft Exchange server, a web server, and a communications server.  The challenge with this setup is that the systems needed to be NAT-ed to 2 separate public address spaces simultaneously, and the router needed to be configured in such a way as to keep it all straight.

One of the most important things is to configure the router to avoid routing loops.  This is where traffic enters the router from Network A but leaves the router from an interface connected to Network B, which can result in lost data, timing problems, or intermittent traffic flow failures.

In order to keep this from happening I used  route-maps to determine which interface originated the traffic and an access control list (ACL) to determine if the traffic was interesting (actionable.)  These two criteria would then be used to create the NAT mapping between source and destination, which in turn would determine which router interface the traffic would leave through.

For example, a user on the Internet wishes to talk to the Client's Web Server.  The routing for this user is such that the network traffic will flow through the wireless link (see network diagram.)  When the user's packet hits the router interface, the router will determine if it is destined for one of the Client's three servers.  If there is a match, it will then look to see which interface the packet originated from.  In this example, that would be FastEthernet0/0.  The router would then build a NAT translation entry mapping the Web Server into the wireless ISP's address space.  When the Web Server replies to the users' request, the router would look at the NAT table, see the address translation, and know which interface to send the traffic back to the user on. 

The Cisco IOS keeps a NAT table of inside/outside address pairs, where inside is the private address space and outside is the global address space.  So all I have to do is to control how the traffic gets NAT-ed and the router will take care of the rest.

Click here for the details of the implementation including router configuration and network diagram.

This technique is a cost-effective way to increase bandwidth as well as reduce dependence on a single network provider and its physical infrastructure.  Coupled with redundant routers, this approach provides a solution for fully redundant branch office connectivity.