Centralized control improves operating efficiency

The project began in April 2000. Upon completion at the end of 2001, total project cost was $8 million (Can.).

The new Edmonton control center houses consoles dedicated to liquid transmission pipelines and terminals as well as gas transmission and distribution.

The liquid pipeline consoles are responsible for 15 pipelines ranging in diameter from 12 in. to 48 in. and lengths up to 1,100 miles. The total system includes 8,200 miles of pipeline on 4,000 miles of right-of-way, powered by 184 pump stations with an average power consumption of about 300 Mw.

Overall, the pipeline system transports more than 2.2 million b/d of 75 different commodities of crude oil, NGL, and refined products.

The gas consoles are responsible for Vector Pipeline, which transports 1 bcfd with 30,000 hp of compression. In addition, the gas consoles control the Enbridge Consumers Gas system, which serves southern Ontario. This distribution system has 3,100 miles of transmission lines and 18,600 miles of distribution lines. The customer base is 1.5 million with a peak demand of 3 bcfd and 100 bcf of storage capacity.

The terminal consoles are responsible for 16 terminal sites with a total of 166 tanks. The working volume is 22 million bbl with typical storage of 10-15 million bbl.

Technically, the most difficult project component was consolidation of the terminal facilities. These centers all had existing supervisory control and data acquisition (SCADA) systems that were built under the assumption that the SCADA system would be local to the facility.

Progressive control systems

Enbridge created and has maintained its SCADA system since 1967. This initially was done because the SCADA industry did not exist at that time.

The first implementation was on IBM hardware in the control center and PDP-11s at the pump stations. The SCADA software, which consisted of many FORTRAN routines, migrated from the IBM platform to Digital Computers DEC-10 and VAX platforms.

In 1991, it was decided that a Windows-based control system should replace the aging SCADA system. After review, the in-house route was again followed as the best option for Enbridge Pipelines.

The result, an application known as PROCYS, has been implemented across the Enbridge system.

PROCYS is based on the Real Time Application Platform (RTAP) from Verano. This is a state-of-the-art SCADA development toolkit, from which the final production application has been developed. It provides a graphical user interface (GUI), display utilities, extensive hierarchical database management, communications, historical trends, alarm management, and event scheduling.

The hierarchical database is tuned to the needs of a high availability, real-time control system. PROCYS has been developed to address the SCADA requirements for the liquid hydrocarbon and gas transportation industries.

The system operates on the UNIX and NT operating systems. These are both fully functional operating systems that incorporate all necessary features, including real time extensions, POSIX compliance, and security. The configuration of the SCADA software and the operating system is carefully tuned to maximize system performance.

The system allows many features to be provided without programming, and additional software may be written to provide special functions.

The PROCYS database runs on Hewlett-Packard servers, operates in real time, and provides updates to calculations based on event triggers. These events include data changes, elapsed timers, and operator input.

The PROCYS RTAP database is built with a hierarchical structure that creates a logical representation of the field devices. PROCYS supports hot standby redundancy, protocol independent field-data forwarding, automatic client fail-over, long-term data archival, on-line multiple-database editing, extensive command structures with verification, network data distribution, state-of-the art alarm system, extensive display system, and comprehensive trending and reporting system.

Centralized control center

The Edmonton control center expanded from 5 operator consoles to 16 as a result of control center consolidation (Fig. 2). Of the 16 operator consoles, 14 are occupied 7 days a week, 24 hr/day. Two consoles are fully functional spares. The room can expand to 20 operator consoles.

The number of operators was increased to 67, and they work 12-hr shifts.

The console furniture was supplied by Evan Consoles and was designed to house eight monitors and two Hewlett-Packard workstations.

One workstation runs Windows NT with two monitors attached. It is used for office and corporate applications. The second workstation runs the LINUX operating system and is used for the SCADA operator interface. This workstation is equipped with six monitors controlled by a single keyboard and mouse.

In addition, the operator consoles are supported by a data center that houses 24 Hewlett-Packard UNIX server's (J6000). These SCADA servers run both the PROCYS application and leak- detection software from Stoner Associates, Carlisle, Pa.

Precentralization

Of the 10 terminal operations being centralized into Edmonton, 8 had existing RTAP-based SCADA systems (Fig. 3). Each was designed for a specific terminal facility. The screen design was customized to each site, and very little effort was spent making these systems common across Enbridge.

The SCADA systems were connected to the PLC network directly, and the terminal PLCs were polled as quickly as possible. Each site had a primary and backup workstation. The backup system was a cold standby with start-up taking about 5 min to complete.

In addition to the SCADA system, the terminals also used an independent GUI for interaction with the local flow computers.

The remaining two terminal facilities were not running the new PROCYS SCADA system. They used a GUI package called VXL from Control Systems International. This software is also connected directly to the terminal PLCs and act as an interface for operations staff.

A PROCYS replacement project was initiated because integration of the VXL systems into Edmonton was not needed.

Postcentralization

The terminal SCADA architecture changed dramatically with centralization of control into the remote control center (Fig. 4). The primary communications path was changed from one that was actively polling the PLC network to one that allowed for report-by-exception communication to the SCADA system.

To do this quickly (without implementing a new protocol at the PLC level), a communication gateway was added between the SCADA system and the PLC network. The gateway polls the PLCs the way the SCADA system did before centralization. Also, the gateway gathers data changes and sends only those items to the SCADA system.

Arcom Control Systems Inc.'s Director product was selected for the gateway. The Director Series Industrial Network Gateways are multiplatform products built around Arcom's APEX software core. This runs under an embedded operating system, which enables both operational and financial data to be acquired, networked, transformed, and utilized efficiently using the TCP/IP networks.

Three communication paths were implemented between the local site and the control center to strengthen the communication network. The primary data path is a wide area network (WAN) frame-relay connection. If this path is unavailable, the network routers automatically dialup the local site and establish a TCP/IP connection over a voice-grade phone line.

This dialup path provides full network visibility to the local site. This allows applications besides SCADA to interact with the terminal. In addition, the SCADA application can communicate with the Directors by dialup over a second phone line to the local site. This capability is operator-initiated and only provides SCADA communications to the site.

So far, frame-relay outages and dialup outages have not been experienced at the same time. This philosophy has the inherent weakness of relying on a single technology for communications, but we have not seen the need to go with a fully independent-separate backup solution.

Besides changes to communications, major work was done on the failover and redundancy systems.

A guarded walkway provides a safe travel corridor though the maze of pipes and valves at Enbridge's tank farm in Edmonton.

The number of SCADA servers was doubled from two to four. This provided SCADA systems in both the primary and backup control centers.

Enbridge feels that every control center must have at least two systems that can control a facility. In addition, the backup SCADA systems were modified from the traditional approach of cold standby to hot standby backup systems.

The failover software was modified to keep all terminal SCADA systems current with both field and operator data changes. This approach sped up system failover times from 5 min to less than 30 sec.

Project schedule

Moving control of 10 terminal facilities into one centralized control center affected several employees. Staffing and relocation for the project was as complex as the technical side. Employees were relocated from across Canada and the US, while new employees were hired and trained.

The company wanted to do everything possible to reduce the uncertainty and stress that this change would cause. This was done with accurate and timely communications and by keeping the transition time as short as possible. All facilities were centralized within 18 months.

An accurate project schedule was essential to ensure success. Delays would only keep staff in place longer. That would complicate the human resource side of the project and impact the financial picture.

In order to centralize the existing control systems and create two new ones, delivery of the facilities was staggered. This led to the creation of two technical teams.

One team concentrated on centralizing the eight existing terminal PROCYS systems, and the other was responsible for creating the two new terminal control systems. Both teams were involved in the overall design and stayed in constant communication during implementation.

Manpower for this project peaked at 16 people; 10 people were dedicated to terminal centralization. Table 1 shows the work schedule.

To maintain this project schedule, a priority was placed on establishing control in Edmonton. Only system changes that were required to centralize the facilities were made.

All other changes were delayed and performed after the operation was controlled from Edmonton. This prevented scope creep from affecting the project's schedule.

Expanded single control center

The expansion of responsibility to a single control center led to a number of design requirements.

The goal was to provide an environment in which control of facilities could be efficiently mixed and matched. This was broken down into the following functional requirements:

• Provide a single common operator interface.
• Allow view-and-control actions for all facilities at all consoles in the control center.
• Ensure the operation of one console does not interfere with any other.

To accommodate the first requirement, screen changes were made to make the operator interfaces look the same and to add the terminal name to all displays. The name was implied when the SCADA system was local to a site.

The difficulty came when the flow computer interface was taken into consideration. The metering systems were implemented and maintained by a separate department from the SCADA department. This system, which directly supports Enbridge revenues, is where all the batch initiations and terminations actions are performed.

This control activity was being relocated to Edmonton and had to be considered as part of the centralization project when looking for a single operator interface.

Each terminal metering system was implemented differently, and over time, some control functions were built into it. This process degraded to the point that, at some terminal locations, operators had to flip between the SCADA system and the metering system to complete certain control functions. This was not wanted in the centralized version.

Due to the tight project timeline and the sheer amount of functionality in the metering system, the work was broken into two phases.

The first task was to integrate all control functions into the SCADA system. Once this was completed, control of the facility could relocate to Edmonton.

All other metering system functions would initially stay in the metering systems GUI but would eventually be migrated into the SCADA system. This work will take place after the centralization project and when completed will provide a single operator interface.

The second design objective was to view and control any part of the Enbridge operation from any of the 16 operator consoles.

To meet this objective, a common set of start-up screens was created and placed on all of the operator consoles. These screens allow the operator to choose the facility to view or control. The operator can direct a request to the correct SCADA server and then redirect the SCADA displays back to the corresponding operator console.

The combination of PROCYS, RTAP, and UNIX "X-win dows" made this task relatively easy with some simple screen- management software.

In addition to displays, audible alarms were also redirected to ring the correct console. This redirection required a new alarm server program that followed the same client-server design as the display system.

The alarm system was the third area considered for possible console interference.

New alarm issues were created because of the increased number of consoles in the single control center. Issues addressed included alarms ringing more than one console and determining which console was alarming when operators were in other parts of the control center.

Duplicate alarm issues arose because the SCADA systems were initially isolated. If certain common alarms were active, the pipeline and terminal SCADA systems would signal the respective operators. When these operators are in the same room and can view each other's operations, these duplicate alarms can become a distraction.

Alarm reviews ensured that only proper alarms were presented. To help operators easily identify alarming consoles, visual indicators (lights) are being considered and will likely be implemented to reduce confusion further.

By addressing these requirements, Enbridge will have a system in which work tasks can be further refined and distributed to make the operation as efficient as possible.

Common operational procedures and common training programs will further reduce overhead costs and ensure that everything is running at its optimum capacity.

Communication infrastructure

The communication infrastructure was the largest technical hurdle encountered during the centralization project.

The existing control systems were connected to the local PLC network. They scanned devices on these networks as quickly as possible and reported all data values to the SCADA system. If the communications interface went down, then the PLCs were most likely not functioning, and the only control alternative would be manual control at the device level.

All of the remote sites are accessible by WAN based on frame-relay. A recently completed project had already moved pipeline communications over to TCP/IP, and this same architecture was needed for remote terminal control.

We did not want to continue having the SCADA system participate directly in the field PLC networks, and we also wanted to send only data changes from the terminal to the control center. To implement these changes, we installed communication gateways at the terminal facility.

The gateway took over polling terminal PLCs and reported only data items that changed from the last reported value (report by exception). The response times could be maintained at the same level as the local terminal control system but in a manner that works across a WAN link.

As a safeguard, the SCADA system polls terminals for full data sets on a timed basis, currently 10 min. Configuration of the Director was driven from the SCADA database and automated to prevent discrepancies and reduce maintenance loads.

Redundancy and failover

The terminal SCADA system in place before centralization consisted of a primary system and a cold-standby backup. If the primary server failed, then the switchover time would take 5 min. This switchover time was acceptable because operators were local to the operation.

Moving the terminal SCADA systems to a remote control center required the redesign of the failover components and increased redundancy.

Full redundancy and failover has been a part of the liquid pipeline control system for several years. This model was extended and implemented for the terminal control systems.

This philosophy calls for at least two redundant systems in each of the primary and backup control centers. It also requires backup SCADA systems to be in a "hot-standby" mode. "Hot-standby" means that the systems run with current data (both field and operator entered). Switchover from the primary SCADA system to a backup system takes less then 30 sec.

The terminal SCADA software was placed on two workstations at each control center. Software was added to distribute incoming field data to all SCADA servers along with software routines to synchronize operator-entered data and data destined for long-term storage.

The SCADA servers were placed on separate network links to remove the local area network (LAN) single point of failure. Directors were implemented in pairs to protect against hardware failures.

Switching is driven by operator-initiated commands. Redundant communication paths between the control center and terminal safeguard against potential communication outages.

The primary path is frame-relay, backed up by router-to-router dialup access. The router initiates this dialup mode when the primary link becomes unavailable. This switchover from frame to dialup takes less than 3 min.

In addition, the SCADA application could dialup the Directors and reroute SCADA data over a separate dialup link. This switchover to dialup Director communications is again driven by operator-initiated commands.

Implementation to this redundancy level protects the facilities and prevents both quality and environmental incidents. In case of disaster, the backup control center can take over all Edmonton control center operations. The backup control center is a complete replica of the Edmonton control center and can be staffed in less than 30 min.

Finally, a copy of the terminal SCADA system was left at the terminal. This system is primarily to facilitate maintenance activities by the local staff but can be used to control the site if all forms of remote control fail.

Local protection systems

Moving control of a facility to a remote location is risky if the control system loses the ability to control the site. Local protection systems were implemented at the lowest control level possible to mitigate this risk.

The pipeline-control system has used local protection systems for a number of years, and this experience helped create the local protection systems for the terminal sites.

Local protection systems monitor the health of communications between local and remote sites. The local protection logic is executed if a communication outage occurs.

Communication health is monitored by a watchdog process that checks both the SCADA system and the PLCs. Every 5 sec the SCADA system toggles a health bit in the local PLCs. The PLCs respond in a similar manner and toggle a separate bit that the SCADA system reads.

If the SCADA system sees this bit remain constant for a 45-sec period, then an operator alarm is triggered. This alarm reoccurs every 45 sec until the condition is cleared.

If the PLC does not see its communications bit change for 10 min, then the PLC executes its shutdown logic. This logic typically shuts down booster pumps as well as other protective actions, depending on the site and the activity currently in progress.

If a terminal goes into a shutdown condition, the pipeline is also driven to "safe" operation until the condition is cleared and the operators can restart the facilities.

Transition of control

The final step in moving terminal control to Edmonton involved a full point-by-point commissioning and system test. The commissioning process verified every analog and discrete status along with all commands. System testing ensured all failover systems were functioning as designed along with all other internal routines.

In order to accommodate these tests and keep commodities flowing, both the new and old systems were in place at the same time. Local terminal staff continued with the day-to-day operation on the old SCADA system until Edmonton staff took over on the new version.

Once the commissioning and systems tests were completed, operations staff monitored the SCADA system and slowly transitioned control into the Edmonton control center. This was usually started with control from Edmonton for day shifts.

Once confidence and training issues were resolved, full-time control from Edmonton was initiated. This was, on average, a 2-week process. The last step in transitioning control was the activation of the local PLC shutdown logic.

Completion

Overall the control center centralization project was a great success. The project naturally broke into the two components, staffing and SCADA, both of which relied on constant communications.

All SCADA systems were delivered on schedule, on budget, and without any safety or environmental incidents. This is attributed to strict management of scheduling and the scope and the expertise of the team.

The PROCYS system is based on powerful software components that are open and easily modified. This foundation, along with vast remote control experience on liquid pipelines, eased the relocation of the terminal control systems into the expanded single control center.

The author

Scott Clarke is supervisor of SCADA architecture for Enbridge Pipelines Inc., Edmonton. He has been employed by Enbridge for 12 years, holds a BSc (1990) in computer engineering from the University of Alberta, and is a registered professional engineer in Alberta. Clarke was the project manager for the SCADA portion of the Enbridge control center consolidation.