Peter M. Curtis - Maintaining Mission Critical Systems in a 24/7 Environment

The large footprint now required for reliable power without planned downtime also affects the planning and maintenance of data center facilities. Over the past two decades, the cost of the facility relative to the computer hardware it houses has not grown proportionately. Budget priorities that favor computer hardware over facilities improvement can lead to insufficient performance. The best way to ensure a balanced allocation of capital is to prepare a business analysis that shows the costs associated with the risk of downtime.

There is no such thing as plug and play when critical infrastructure is deployed, or existing systems are overhauled to support a company's changing business mission. Reliability is not guaranteed simply by installing new equipment or even building an entirely new data center. An aggressive and rigorous design, failure mode analysis, testing/commissioning process, and operations plan proportional to the facility's criticality level are a necessity and not an option.

Of particular importance is the actual commissioning process and developing a detailed operations plan. More budget dollars should be allocated to testing/commissioning, documentation, education/ training, and operations/maintenance because more than 50 percent of data center downtime can be traced to human error. Due to the facility’s 24/7 mission critical status, this will be the construction team’s sole opportunity to integrate and commission all of the systems. At this point in the project, a competent, independent test engineer familiar with the equipment has witnessed testing of all installed systems at the factory.

Commissioning is a systematic process of ensuring, through documented verification, that all building systems perform according to the design intent and to the future owner's operational needs. The goal is to provide the owner with a safe and reliable installation. A commissioning agent who serves as the owner's representative usually manages the commissioning process. The commissioning agent's role is to facilitate a highly interactive process of verifying that the project is installed correctly and operating as designed. This is achieved through coordination with the owner, design team, construction team, equipment vendors, and third‐party commissioning provider during the various phases of the project. ASHRAE’s Commissioning Guideline 0‐2500 is a recognized model and a good resource that explains this process in detail and can be applied to critical systems.

Prior to installation at the site, all major equipment should undergo factory acceptance testing that is witnessed by an independent test engineer familiar with the equipment and the testing procedures. However, relying on the factory acceptance test is not sufficient. Once the equipment is delivered, set in place, wired, and functional testing completed, integrated system testing begins. The integrated system test verifies and certifies that all components work together as a fully integrated system. This is the time to resolve all potential equipment problems. There is no “one size fits all” formula.

Before a new data center or renovation within an existing building goes on‐line, it is crucial to ensure that the systems are burned‐in and failure scenarios are tested no matter the schedule, milestones, and pressures. You won't have a chance to do this phase over, so get it right the first time. A tremendous amount of coordination is required to fine‐tune and calibrate each component. For example, critical circuit breakers must be tested and calibrated prior to exposing them to any critical electrical load. After all tests are complete, results must be compiled for all equipment and the certified test reports prepared, establishing a benchmark for all future testing.

Scheduling time to educate staff during systems integration testing is not considered part of the commissioning process but is extremely important in order to reduce human error.

This activity can be considered part of the transitions‐to‐operations process. Hands‐on training is invaluable because it improves situational awareness and operator confidence, which in turn reduces human error. The training can also break through misplaced confidence. Sometimes we deem ourselves ready for a task, but we are really not. Handing off the new infrastructure or facility to fully trained and prepared operations teams improves success and uptime throughout the facility lifecycle. When you couple the right design with the right operations plan (training programs, documentation/IMOPs, and preventative maintenance), the entire organization will be much better prepared to manage through critical events and the unexpected. If proper training and preparation aren’t done during the commissioning stage, building engineers will not become familiar with various processes and procedures. Learning on the job increases operational risk. Knowing this up front, there is absolutely no reason not to commit the necessary budget for training, technical maintenance programs, accurate documentation/storage, and finally, some type of credible certification that is continually revisited. If done correctly, potential mishaps and near misses will be avoided, and the reduced risk will be like an annuity paying “Reliability/ Availability” dividends.

Whenever a fault occurs in an electrical power system, relatively high currents flow, producing large amounts of destructive energy in the forms of heat and magnetic forces. When an electrical fault exceeds the interrupting rating of the protective device, or the fault withstands rating of equipment such as switchgear or panelboards, the consequences can be devastating, including injury, damaged electrical equipment, and costly downtime. A Short Circuit Study (SCS) is required to establish minimum equipment ratings for power system components to withstand these mechanical and thermal stresses that occur during a fault, and thus SCS’s are mandated by article 110.9 of NFPA NEC 70 Code. Also required by the NEC are markings and nameplates of the SCS ratings on equipment. These include industrial control panels (409.110), industrial machinery (670.3[A]), HVAC equipment (440.4[B]), meter disconnect switches (230.82[3]), and motor controllers (430.8).

The short‐circuit calculations that are the basis of an SPS involve the reduction of the electric power distribution system supplying current to each fault location to a Thevenin equivalent circuit of system voltage sources and impedances at each bus. (Refer to ANSI/IEEE Std 399 Recommended Practice for Power System Analysis (Color Book Series ‐ the Brown Book)). These calculations can be done by hand, but since they can be very tedious for mid‐size to large distribution systems, they are usually performed using the same specialized software that models the power system under normal load flow conditions to determine system power flows and voltage drop. When the software is used for a SCS, the model is used to calculate the resultant fault current at each bus in the system. Typically, results are presented for both three‐phase and line‐to‐ground faults. (Software used by electric utility companies also calculates line‐to‐line fault current values.)

In order for the electrical distribution system analysis software to produce reliable results, the electrical distribution system being analyzed must be modeled accurately. All current sources must be investigated and inputted into the model, such as:

The Electric Utility – As a minimum, the service voltage and available 3‐phase and single line‐to‐ground short circuit current (usually given in amps or MVA) and the three‐phase and line‐to‐ground equivalent circuit reactance/resistance ratio, referred to as the X/R ratio, are required. Many utilities also provide the equivalent system impedances at the point of service connection. The system short circuit contribution data must be requested from the electric utility.