Your Continuity Plan Treats Every Cooling Unit the Same. The Air Doesn't.

I got into business continuity because a customer asked me questions I couldn't answer.

Business continuity is the work of planning for what happens when something critical fails, so the failure is a procedure you follow instead of a fire you fight. This was back in my BCGI days. The customer started asking hard questions about what would happen if something failed, and our sales team couldn't keep up, so the questions landed on my desk. I didn't have good answers either. That bothered me enough to go do something about it. I got certified, and a while later I helped found a local chapter of business continuity planners. Because I was running a data center at the time, I could walk them through the actual data hall and show them the infrastructure we were talking about. Most of them had never seen the floor up close.

That's the thing about business continuity. You spend your time looking at what happens when things break, and it teaches you to see the whole ocean instead of just the patch of water in front of you. It also taught me where the planning tends to stop, right at the data hall door, where cooling failure actually plays out.

What the planners get right, and what they miss

Business continuity teams are good at running tabletop exercises. They'll sit a room down and say, all right, you lose a cooling unit. What does your plan do? Do you shift load to another site? Do you have remediation? It's a good exercise and more teams should run it.

Here's where it falls short. Most business continuity planners don't have the depth on the data center floor to know that losing Unit A and losing Unit B are not the same event. One unit can fail and cause almost no problem. Another can fail and create all kinds of havoc. The plan treats them the same because the people writing it can't see the difference. They're working from a floor plan, not from what the air actually does when a unit drops offline.

A failure becomes an emergency when you're guessing

The best way to avoid this issue is to ask three questions.

How long do you have before a cooling failure becomes a critical issue? Which unit, if you lost it right now, would hurt the most? Is there a written plan that tells the team how to triage and restore?

In a lot of facilities, nobody can answer all three. The team is sharp and they'll respond fast when something goes wrong, but they're responding to a situation they've never measured. They don't know the window. They don't know which failure is the most dangerous one. So a cooling failure turns into an emergency, with people scrambling and guessing, when it could have been a procedure somebody follows off a sheet.

The difference between an emergency and a procedure is whether you did the work before the failure or during it.

How you actually find your window

You can measure all of this before anything fails. It's called resiliency testing, and it's straightforward.

You place sensors across the data center floor. Then you shut down cooling units one at a time and watch what happens. Our resiliency software collects the sensor data in real time and tracks the temperature rise as it happens. It calculates how fast the room is heating and predicts when it will reach the critical threshold you've set. You run that for each unit, each scenario.

What you get is a timeline from failure to critical for every test. Lose this unit, you have this many minutes. Lose that one, you have a different number, and now you know which failure is the one to worry about. That timeline is the thing you build a real response plan on. Not a guess. A measured window with a number attached.

It answers the questions the planners should be asking. What if the unit fails. How much time do I have? When do I hit the point of no return?

Look at the whole ocean

If I were still running a floor, I'd want to know where my biggest risk lived before it found me. I'd want the failure that keeps me up at night identified, measured, and written into a plan my team could follow at two in the morning without calling me.

That's the view from 40,000 feet instead of 5,000. You stop reacting to the wave in front of you and start seeing the water you're actually swimming in. A cooling failure is going to happen eventually. Whether it's an emergency or a procedure is mostly decided long before the unit ever quits. It’s worth testing before you need it.

About the Author

Gregg Haley is a data center and telecommunications executive with more than 30 years of leadership experience. Most recently served as the Senior Director of Data Center Operations - Global for Limelight Networks. Gregg provides data center assessment and optimization reviews showing businesses how to reduce operating expenses by identifying energy conservation opportunities. Through infrastructure optimization energy expenses can be reduced by 10% to 30%.

In addition to Gregg's data center efforts, he has a certification from the Disaster Recovery Institute International (DRII) as Business Continuity Planner. In November of 2005, Gregg was a founding member and Treasurer of the Association of Contingency Planners - Greater Boston Chapter, a non-profit industry association dedicated to the promotion and education of Business Continuity Planning. Gregg had served on the chapter's Board of Directors for the first four years. Gregg is also a past member of the American Society of Industrial Security (ASIS).

Gregg currently serves as the Principal Consultant for Purkay Labs.