Peter M. Curtis - Maintaining Mission Critical Systems in a 24/7 Environment

Many organizations associate disaster recovery and business continuity only with IT and communication functions and miss other critical areas that can seriously impact their business. Within these areas may be a multitude of critical systems that require maintenance, the development of procedures, and appropriate documentation. Some of these systems are listed later in Table 1.3.

One major area that necessitates strategy development is the banking and financial service industry. The absence of strategy that guarantees recovery has an impact on employees, facilities, power, customer service, billing, and customer and public relations. All areas require a clear, well‐thought‐out strategy based on recovery time objectives, cost, profitability impact, and safety. The strategic decision is based on some of the following factors:

The maximum allowable delay time prior to the initiation of the recovery process.

The time frame required to execute the recovery process once it begins.

The minimum computer configurations required to process critical applications.

The minimum communication device and backup circuits required for critical applications.

The minimum space requirements for essential staff members and equipment.

The total cost involved in the recovery process and the total loss as a result of down time.

Developing strategies with implementation steps means no time is wasted in a recovery scenario. The focus is to implement the plan quickly, and successfully in order to accomplish this people must be properly trained. Is the person you hired three months ago up to this task? The right strategies implemented will effectively mitigate damages, minimize disruptions, reduce the cost of downtime, and remove the threat to life safety.

To assure reliable operation a Critical Environment Workflow and Change Management Process must be established and followed. Commensurate Roles and Responsibilities of the Engineering, Technology and Security groups will be developed, implemented, and adhered to in order to manage both planned and unplanned events and associated risks.

The Critical Environment (CE) is defined as the “physical space and the systems within a facility that are uniquely configured, sized and dedicated to supporting specific critical business operations as defined by the user”. There are many specific rooms and areas within facilities in today’s ever‐changing environment. Some are located within the buildings structure while others are located outside. Regardless of where a CE may be located, these locations have an immediate impact on the client’s ability to maintain business operations/continuity. Examples of some of these CE areas can be seen in Table 1.2.

Table 1.2 Critical Areas

Critical infrastructure systems are prevalent throughout a facility. Depending on the facility's size, there could be many redundant systems supporting the same critical environment. Knowing which systems that could impact the clients’ critical function/operation is paramount. Some of these systems are listed in Table 1.3.

Table 1.3 Critical Systems

Change Management is a process for managing and communicating change across relevant functions and business units to ensure and deliver integration of procedures and processes. Note that during emergency situations, established emergency response and escalation procedures shall be followed.

When work is contemplated for accomplishment within the critical environment, ranging from certain simple or routine cleaning and inspection tasks to very complex and detailed preventive maintenance, corrective maintenance or construction efforts, it is essential that an orderly and thorough approach to work planning and execution be undertaken. In every instance where work is planned in the critical environment, all departments must ensure that risk to company operations is thoroughly assessed and that appropriate risk mitigation is in place while the work is performed.

The level of detail required in a MOP (Method of Procedure) shall be correlated to the complexity of the work and magnitude of the potential risk. Relatively complex or high‐risk work shall be meticulously detailed in the MOP. The detail required for less complex work would not necessarily be as extensive. The bottom line: a properly developed, reviewed, and approved MOP will result in reduced risk to business operations. Required Change Request Information includes:

Who is doing the work?

What systems will be affected?

Which areas of the building will be affected?

Is there redundancy for the system being disrupted?

Detailed procedures for the proposed task.

Is assistance needed from other Lines of Business?

What hardware will be moved, added, or changed?

How long is the task duration?If an outage is required:

How long will power be out?

Are there any critical points during the process (High Risk) that can be identified?

Will those systems be protected by UPS, generators, or other redundancy?

What kind of backup systems are available if a problem arises?

Will utility power be taken down?

Are other feeds to the building affected?

If redundancy is to be reduced, what redundancy will be lost, and for how long?

The purpose of the Critical Escalation Procedures is to allow for the successful response to a critical site event. Following the escalation process, assures proper notification and timely response. By assessing the event first, critical information will be available early on. It is important that a chain of command be followed because when events arise, teams need to ensure that communication and reactions are escalated in the proper fashion.

The definition of commissioning (Cx) has developed over the past twelve years from being nothing more than vendor start‐up to the comprehensive quality control process it is today. Some still cling to the idea that commissioning is something that starts at the end of construction or after the design is complete. Indeed, for most of the 20 thCentury, when building systems were much less complex, and tradesmen did not need additional expertise to activate them, this approach was sufficient. Today, many commissioning professionals have found it helpful for themselves and their clients to organize the effort into “levels” of commissioning, and level one starts after the design is complete at Factory Acceptance Testing (FAT). The advent of the Leadership in Energy Efficient Design (LEED) program and its various certifications forced the industry once again to rethink the commissioning process. For several good reasons, LEED requires design reviews and other requirements that start earlier in the project than commissioning professionals were accustomed to. A widely accepted definition of commissioning can be found in ASHRAE’s Guideline 0. ASHRAE’s Commissioning Guideline 0‐2005 is a recognized model and good resource that explains commissioning as a quality control process in detail. Although its intent is to apply to all kinds of buildings, it can be applied to critical facilities with some embellishment to address the unique needs of verifying critical system performance. The quality control process given by ASHRAE is in phases starting with the Pre‐Design Phase and continues throughout the Occupancy and Operations phase.