Peter M. Curtis - Maintaining Mission Critical Systems in a 24/7 Environment

Once the equipment is delivered, set in place, and wired, it is time for the second phase of certified testing and integration. The importance of this phase is to verify and certify that all components work together and to fine‐tune, calibrate, and integrate the systems. There is a tremendous amount of preparation in this phase. The facilities engineer must work with the factory, field engineers, and independent test consultants to coordinate testing and calibration. Critical circuit breakers must be tested and calibrated prior to placing any critical electrical load on them. When all the tests are completed, the facilities engineer must compile the certified test reports, which will establish a benchmark for all future testing. The last phase is to train the staff on each major piece of equipment and prepare for the transition to operations.

Many decisions regarding how and when to service a facility’s mission critical electrical/mechanical equipment are going to be subjective. The objective is easy: a high level of safety and reliability from the equipment, components, and systems. But discovering the most cost‐effective and practical methods required to accomplish this can be challenging. Network with colleagues, consult knowledgeable sources and review industry and professional standards and best practices before choosing the approach best suited to your maintenance goals. Also, keep in mind that the individuals performing the testing and service should have the best training and experience available. You depend on their conscientiousness and decision‐making ability to avoid potential problems with perhaps the most crucial equipment in your building. Most importantly, learn from your experiences and those of others. Maintenance programs should be continuously improving. If a scheduled procedure has not previously identified a problem, consider adjusting the schedule respectively. Examine your maintenance programs on a regular basis and make appropriate adjustments to improve constantly.

Acceptance and maintenance testing are pointless unless the test results are evaluated and compared to standards, and to previous test reports that have established benchmarks. It is imperative to recognize failing equipment and to take appropriate action as soon as possible. Common practice in this industry is for technicians to perform maintenance without reviewing prior work tickets and records. This approach defeats the value of benchmarking and trending and must be improved. The mission critical facility engineers can then keep objectives in perspective and depend upon his/her options when faced with a real emergency.

The importance of taking every opportunity to perform preventive maintenance thoroughly and completely ‐ especially in mission critical facilities‐cannot be stressed enough. If not, the next opportunity will come at a much higher price: downtime, lost business, lost potential clients, and not to mention the safety issues that arise when technicians rush to fix a maintenance problem. So, do it correctly ahead of time and avoid shortcuts because it will be very difficult to do it again.

The mission critical industry’s focus on physical infrastructure enhancements descends from the early stages of the trade when all efforts were placed solely in design and construction techniques to enhance mission critical equipment.

Twenty‐five years ago, the technology supporting mission critical loads was simple. There was little sophistication in the electrical load profile; at that time, the industry was in its infancy. Over time the data centers have grown from a few mainframes supporting minimal software applications to server farms that can occupy 100,000 ft 2or more – with Google and Microsoft being prime examples.

As more processing power is required to sustain our global economy, the electrical and mechanical systems supporting the critical load became increasingly complex. With businesses relying on this infrastructure, more capital dollars were invested to improve the uptime of the business’s lines. Today billions of dollars are invested on an enterprise‐level into the infrastructure that supports the business 24/7 applications; the major investments are normally in design, equipment procurement, and project management. Few capital dollars are invested in the documentation, change management, education/training, or operations and maintenance. The initial capital investment was just the tip of the iceberg ( Figure 1.1).

Figure 1.1 Hidden Costs of Operations

Figure 1.2Typical screenshot of Smart WALK™ dashboard

(Courtesy of PMC Group One, LLC.)

Years ago, most organizations relied heavily on their workforce to retain much of the information regarding the mission critical systems. A large body of personnel had a similar level of expertise. They remained with their company for decades. Therefore, little emphasis was placed on maintaining a living document for a critical infrastructure. Tables 1.4to 1.6identify questions with regards to managing the loss of personnel, documentation, and managing during a critical event.

Table 1.4 Managing Loss of Critical Personnel

Table 1.5Documentation Issues

Table 1.6 Managing During Critical Events

Figure 1.3Smart WALK™ mobile screenshot

(Courtesy of PMC Group One, LLC.)

Figure 1.4Screenshot of Smart TEAM® Learning Management System

(Courtesy of PMC Group One, LLC)

The mission critical industry can no longer manage their critical system as they did twenty‐five years ago. The requirements are very different today in that the sophisticated nature of the data center infrastructure requires constant refreshing and updating of documentation. One way to achieve this is to include a living document system that provides the level of granularity necessary to operate a mission critical infrastructure into a capital project. This will assist in keeping the living document current each time a project is completed, or a milestone is reached. Accurate information is the first level of support that provides first responders the intelligence they need to make informed decisions during critical events. It also acts like a succession plan as employees retire, and new employees are hired, thus reducing risk and improving their learning curve. Remember that greater than 50% of all downtime can be tracked to human error.