Peter M. Curtis - Maintaining Mission Critical Systems in a 24/7 Environment

In new 24/7 facilities, it is imperative to not only design and integrate the most reliable systems, but also to keep them simple. When there is a problem, the facilities manager is under enormous pressure to isolate the faulty system without disrupting any critical electrical loads and does not have the luxury of time for complex switching procedures during a critical event. An overly complex system can be a quick recipe for failure via human error if key personnel who understand the system functionality are unavailable. When designing a critical facility, it is important that the building design does not outsmart the facilities manager. Companies can also maximize profits and minimize cost by using the simplest design approach possible or integrate automatic recovery or “self‐healing” automatic controls to recover from a failure. One prevalent example is the current use of Static Transfer Switches (STS’s) discussed in a later chapter. The STS will automatically and within milliseconds switch power sources to critical equipment.

In older buildings, facility engineers and senior management need to evaluate the cost of operating with obsolete electrical distribution systems and the associated risk of an outage. Where a high potential for losses exists, serious capital expenditures to upgrade the electrical distribution system are monetarily justified by senior management. The cost of downtime across a spectrum of industries exploded in recent years, as businesses have become completely computer‐dependent, and systems have become increasingly complex ( Table 3.2).

Table 3.2 The Cost of Downtime

( Source: Data from Information Technology Intelligence Consulting).

* Prepared by a disaster‐planning consultant of Contingency Planning Research

Imagine that you are the manager responsible for a major data center that provides approval of checks and other on‐line electronic transactions for American Express, MasterCard, and Visa. On the biggest shopping day of the year, the day after Thanksgiving, you find out that the data center has lost its utility service. Your first reaction is that the data center has a UPS and standby generator, so there is no problem, right? However, the standby generator is not starting due to a fuel problem, and the data center will shut down in 15 minutes, the amount of time the UPS system batteries can supply power at full load. The penalty for not being proactive is the loss of revenue, potential loss of major clients, and if the problem is large enough, your business could be at risk of financial collapse. You, the manager, could have avoided this nightmare scenario by exercising the standby generator every week for 30 minutes – the proverbial ounce of prevention.

There are about ten times as many UPS systems in use today than there were 10 years ago, and many more companies are still discovering their worth after losing data during a power line disturbance. Do you want electrical outages to be scheduled or unscheduled? Serious facilities engineers use comprehensive preventative maintenance procedures to avoid being caught off‐guard.

Many companies do not consider installing backup equipment until after an incident has already occurred. During the months following the U.S. Northeast Blackout of 2003, the industry experienced a boom in the installation of UPS systems and standby generators. Small and large businesses alike learned how susceptible they are to power disturbances and the associated costs of not being prepared. Some businesses that are not typically considered mission critical learned that they could not afford to be unprotected during a power outage. For example, the Blackout of 2003 destroyed $250 million of perishable food in New York City alone. 1 Businesses everywhere, and of every type, are reassessing their level of risk tolerance and cost of downtime.

Mission critical facilities cannot be susceptible at any time to an outage, including during maintenance of the subsystems. Therefore, careful consideration must be given in evaluating and implementing redundancy in systems design. Examples of redundancy are classified as (N+1) and (N+2) configurations and are normally applied to the systems below:

Utilities service

Power distribution

UPS

Emergency generator

Fuel system supplying emergency generator

Mechanical Systems

HVAC Distribution including piping, AHU’s and CRAH units

Fire Protection Systems

A standard system (N+1) is a combination of two basic schemes that meet the criteria of furnishing an essential component plus one additional component for backup. This design provides the best of both worlds at a steep price, with no economies of scale considered. A standard system protects critical equipment and provides long‐term protection to critical operations. In a true (N+1) design, each subsystem is configured in a parallel redundant arrangement such that full load may be served even if one system is off‐line due to scheduled maintenance or a system failure. The alternate (+1) component is also fed from a different power source.

The next level of reliability is a premium system. The premium system meets the criteria of an (N+2) design by providing the essential component plus two components for backup. It also utilizes diverse electric services from different power distribution boards and two different utility substations. Under this configuration, any one of the backup components can be taken off‐line for maintenance and still retain (N+1) reliability.

Another level resiliency can be gained by having building electric services installed underground as opposed to aerial. Aerial services are susceptible to lightning strikes, fallen trees, or damaged utility poles.

Facilities can also be classified into different Tiers based on their required level of reliability and maintainability. Tiers for mission critical facilities range from I to IV, with Tier IV being the most redundant, reliable, and maintainable.

Table 3.3Uptime Tiers

(Reference: Uptime Institute)

Load Classifications