Peter M. Curtis - Maintaining Mission Critical Systems in a 24/7 Environment

Critical Load: Requires 100% uptime. Must have uninterrupted power input to safeguard against facility damage or losses, prevent danger and injury to personnel, or keep critical business functions on line.

Essential Load: Supports routine site operations. Able to tolerate power failures without data loss or affecting overall business continuity. Critical cooling can be classified as essential load when critical spaces have a “ride through time,” allowing chillers, cooling towers, and pumps to restart automatically in a limited period.

Discretionary Load: Load that indirectly supports the operation of the facility, such as administrative, cafeteria/pantries/fitness centers/retail spaces, and office functions. Can be “shed” without affecting overall business continuity in order to keep critical loads on line.

The average human heart beats approximately 70 times a minute, or a bit more than once per second. Imagine if a heart missed three beats in a minute. This would be considered a major power line disturbance if we were to compare it to an electrical distribution system. Take the electrical distribution system that feeds your facility, or better yet, the output of the UPS system, and interrupt it for 3 seconds. This is an eternity for computer hardware. The critical load is disrupted, your computers crash, and your business looses two days’ worth of labor or worse, is fined $10 to $20 million by the federal government because they did not receive the quota of $500 billion dollars of transaction reports by the allocated time.

All this could have been prevented if electrical maintenance and testing were performed on a routine basis, and the failed electrical connections were detected and repaired. Repairs could have been quickly implemented during the annual infrared scanning program that takes place before building maintenance shutdowns.

What can the data processing or facility manager do to ensure that their electrical system is as reliable as possible?

The seven steps to improved reliability and maintainability are:

Planning and impact assessment

Engineering and design

Project management

Testing and commissioning

Documentation

Education/training and certifications with annual re‐certification

Operations and maintenance

When designing a data processing center, it is important to hire competent professionals to advise each step of the way. If the data processing center is being installed in an existing building, you do not have the luxury of designing the electrical system from scratch. A proficient electrical engineer will design a system that makes the most out of the existing electrical distribution. Use electrical contractors who are experienced in data processing installations. Do not attempt to save money using the full 40% capacity for a conduit; because as quickly as new, state‐of‐art equipment is installed, it is de‐installed. Those same number 12 wires will need to come out of the conduit without disturbing the working computer hardware.

Have an experienced electrical testing firm inspect the electrical system, perform tests on circuit breakers, and use thermal‐scan equipment to find “hot spots” due to improper connections or faulty equipment. Finally, plan for routine facility shutdowns to perform preventative maintenance on all critical equipment. Facility managers must not underestimate the cost‐effectiveness of a thorough preventative maintenance program, nor must they allow senior management to do so. Critical system maintenance is not a luxury; it is a necessity. Again, do you want electrical outages to be scheduled or unscheduled?

Integrating the ideal critical infrastructure is just about impossible. Therefore, seek out the best possible industry authorities to solve your problems. Competent consultants will have the knowledge, tools, testing equipment, training, and experience necessary to understand the risk tolerance of your company, as well as recommend and implement the proper and most advanced proven designs.

Equipment manufacturers and service providers are challenged to find and retain the industry’s top technicians. As 24/7 operations become more prevalent, the available talent pool will diminish. This could cause response times to increase from the current industry standard of 4 hours. Therefore, the human element has a significant impact in risk and reliability.

No matter which firms you choose, always ask for sample reports, testing procedures, and references. Your decisions will determine the system’s ultimate reliability, as well as how easy the system is to maintain. Seek experienced professionals from both your own company, and third parties for information systems, property and operations managers, space planners, and the best consultants in the industry for all engineering disciplines. The bottom line is to have proven organizations working on your project. Systems that are not designed, installed, and operated optimally by your operations team will only hurt the operations of your company and cause discontent down the road.

To date, the mission critical facilities manager has not achieved high levels of prestige within the corporate world. This means that if the requirements are 24/7, forever, the mission critical facilities manager must work hard to have a voice in the boardroom. The board can then become a powerful voice that supports the facilities manager and establish a standard for managing the risks associated with older equipment or maintenance cuts. For instance, relying on a UPS system that has reached the end of its useful life, but is still deployed due to cost constraints increases the risk of failure. The facilities manager is in a unique position to advise and paint vivid scenarios to the board. Imagine incurring losses due to downtime, plus damage to the capital equipment that is keeping the Fortune 1000 company in business.

Board members understand this language: it is comparable to managing and analyzing risk in other avenues, such as whether to invest in emerging markets in unstable economies. The risk is one and the same; the loss is measured in the bottom line.

The facilities engineering department should be run and evaluated just like any other business line; it should show a profit. But instead of increased revenue, the business line shows increased uptime, which can be equated monetarily, plus far less risk. It is imperative that the facilities engineering department be given the tools and the human resources necessary to implement the correct preventative maintenance training, and document management requirements, with the support of all company business lines.

Data center reliability ultimately depends on the organization as a whole weighing the dangers of outages against available enhancement measures. Reliability modeling is an essential tool for designing and evaluating mission critical facilities. The conceptual phase, or programming, of the design, should include a full Probabilistic Risk Assessment (PRA) methodology. The design team must quantify performance (reliability and availability) against cost in order to push fundamental design decisions through the approval process.

Reliability predictions are only as good as the ability to model the actual system. In past reliability studies, major insight was gained into various electrical distribution configurations using IEEE Standard 493‐2007 Recommended Practice for the Design of Reliable Industrial and Commercial Power Systems , or, The IEEE Gold Book. It is also the major source of data on failure and repair rates for electrical equipment. There are, however, aspects of the electrical distribution system for a critical facility that differ from other industrial and commercial facilities. Therefore, internal data accumulated from the engineer’s practical experience is needed to complement the Gold Book information.