Peter M. Curtis - Maintaining Mission Critical Systems in a 24/7 Environment

A summary of the phases given in ASHRAE appears below, with some additions for mission critical facilities. These additions, along with all phases, should be included in all mission critical facility projects.

Pre‐Design phaseDocument Owner’s project requirements and basis of design

Design phaseCommission focused Design reviewWriting Cx specifications

Construction phaseFactory Acceptance Testing ‐ FATConstruction check listingStart‐up (Pre‐functional) testing

Acceptance phaseSite acceptance (Functional)Acceptance testing to verify the performance of critical equipmentIntegrated testingO & M document reviewStaff training oversightDevelop and prove out EOPs. SOPs and MOPs

Occupancy and Operations phaseContinual review and updating of materialsContinual training of O&M staffReliability Assurance Testing (continual commissioning)

A quality control process would never be overlooked in any valuable production project such as a new model automobile or TV. Why would we neglect quality control for critical facility projects? In the case of critical facility projects, quality control or commissioning starts in the pre‐design phase so we can make sure the Owner’s Project Requirements (OPR) are fully developed and track all relevant documents such as the Basis of Design (BOD). We do this so that during value engineering, we can evaluate any impact to the BOD and OPR, and if the proposed change does not meet the documented requirements, then the team must sign off on it. It should be clear that these documents are the foundation of your project and part of the quality control process required for the commissioning of a critical facility. Without them, the commissioning team is left without a clear set of criteria upon which to base their findings and actions.

In the design phase, we have a commissioning‐focused design review that should not be confused with a peer review. In a commissioning‐focused design review, the commissioning engineers should provide input on making the building and systems easier to commission and easier to maintain. They should also examine equipment layout, labeling schemes, and other ergonomic factors, which can result in a facility that operations team members find intuitively easy to work with. This, in turn, reduces the likelihood of operator error in emergency response situations. The CxA should also verify that bid documents adequately specify building commissioning as this will help reduce vendor change orders. It many cases it is better to have the CxA provide commissioning specifications and have them included in the pre‐purchase and other bid sets. The focused review also needs to verify that there are adequate monitoring and control points specified to facilitate commissioning and O&M (trending capabilities, test ports, control points, gages, and thermometers). A review needs to include a review of design as it pertains to the reliability and redundancy standards of the owner and industry standards and verifies that building O&M plan and documentation requirements specified are adequate.

During the construction phase, much of the rudimentary testing is accomplished. During this time, the factory acceptance testing is being conducted, and it is important to have the CxA involved to verify that the controls and interlocks will work with the complete system. During this time, equipment is being delivered and installed. During this installation, the vendors and GC should be verifying the installation using construction checklists provided by the CxA. These checklists basically track the construction process and verify that the vendor delivered what was paid for in good condition, installed properly, and has the proper clearance. The vendor startup will follow, and if performed in accordance with the agreed procedures, all the functions, including all the alarms, will be verified. It is important to track all these documents and have them signed off by the vendors as proof of proper startup. In some cases, the vendors will sign off the documents and not perform all the requirements, and that will slow down the acceptance and integrated testing. In this case of improper start‐up, the delays can be back charged to the responsible vendor.

In the Acceptance Phase, the CxA will first operate all the equipment in all configurations and verify the proper start‐up by the vendors. Once this is completed, the CxA should verify the performance of certain equipment without using vendor‐provided test gear. This is done to keep the quality control process in the hands of the CxA and make sure that only calibrated equipment is used, and the calculations are performed without bias. The equipment list below is recommended to be subjected to this extra acceptance test phase.

1 Emergency power systems and controls.

2 Uninterruptible power supply (UPS) systems and batteries.

3 Flywheel energy storage systems.

4 Static Transfer Switches (STS) and all associated controls.

If the acceptance testing is done properly, it should, at a minimum, verify the equipment is worthy of critical load. In some cases, deficiencies found during this process have forced the vendors to meet their own specifications and improve product quality.

We are now ready for integrated testing, and the intent of this test is to verify that the building and all the systems work together to meet the client’s design requirements. Some hints for having a successful integrated test that proves proper operation and no unwanted system interaction are:

1 Perform a full data center heat load test, including any enclosed cooling systems.

2 Perform integrated testing at 25%, 50%, 75%, and 100% of the design load.

3 Use data loggers on the data center floor to verify measured data and BMS controls.

4 Check all operating modes, including maintenance configurations.

Staff training and operations documents need to be provided before we can start operations. Proper training must be given to the staff for systems and integrated operations. I would suggest that the vendors provide system training, and the CxA provides overall operations documents. The operations documents should include Maintenance Operation Procedures (MOP), Emergency Operation Procedures (EOP), Standard Operation Procedures (SOP), and Alarm Response Procedures (ARP). With properly trained staff and proper operations documents, the human error faults can be minimized.

As we stated earlier, the commissioning process continues into the occupancy phase to maintain operational continuity. A yearly review and update of training as required due to system upgrade or operational requirements maintains staff and procedural quality. For mission critical facilities, a yearly verification of performance for critical electrical system prevents loss of productivity due to system degradation referred to as Reliability Assurance Testing. These tests should be similar to those performed in the system acceptance test procedures used during the acceptance phase and use the original data for trending any changes in the system. The Reliability Assurance Testing should be performed after the vendor has provided Preventive Maintenance (PM). The reason we perform these tests after the vendor preventive maintenance routine is that the vendor just interacted with a commissioned system and disassembled some portions. In some cases, they are providing updated software or control boards. The system now needs to be certified through Reliability Assurance Testing to be worthy of critical load. Remember that the vendor provided PM does not measure performance or track system degradation, so without a Reliability Assurance Testing program, the quality control process had been compromised.

Before the facility goes on‐line, it is crucial to resolve all potential equipment problems (Technology, Operations, etc.). This is the construction team’s sole opportunity to integrate and commission all the systems, due to the facility’s 24/7 mission critical status. At this point in the project, all systems installed were tested at the factory and witnessed by a competent Commissioning Authority (CxA) familiar with the equipment processes and procedures.