The true vision of automation

32 IEEE power & energy magazine may/june 20071540-7977/07/$25.00©2007 IEEE

TTHE RETAIL INDUSTRY IN THE UNITED STATES KNOWS THATevery year between Thanksgiving and Christmas the hustle and bustle is onto sell one more video game or one more sweater. For the North Americanelectric utility industry, every year between the beginning of May and thebeginning of June the hustle and bustle is on to get one more transformerhumming or one more substation online. It should come as no surprise thenthat the livening of a small transmission substation on a blue-sky Saturdaylate in May of 2006 in a small town in Michigan could go unnoticed. Thissubstation features the use of an Ethernet local area network (LAN), relianceon IEC 61850, the capture of nonoperational data, and a station human-machine interface (HMI). This is pretty obviously an example of “substationautomation” and yet that term may not tell the whole story.

To help appreciate how much “substation automation” has become a part of thelexicon, a recent Google of the phrase got the same approximate number of hits asthe phrase “protective relay” (145,000). Though the definition of “protective relay”can be debated, the term “substation automation” can be actively deceptive. Itimplies that this concept is within the walls of the substation and seems to refer tothe automatic operation of things like voltage control, load transfer, and tapchanges. The Michigan substation is meant to realize the organic implementationof automation technologies. It reflects how a system would evolve around thesetechnologies as opposed to having substation automation added to a system. Thissubstation was an attempt at realizing the vision of automation.

Strategy and MethodologyIn 2004, Michigan Electric Transmission Co. embarked on a program to devel-op a sweeping business and technical strategy to replace the aging protection

may/june 2007 IEEE power & energy magazine 33

and control equipment. This aggressive investment and replacement strategycan be the most cost-effective solution for system-wide upgrading.

The implementation methodology, addressing impact on capital and oper-ating costs, is described in more detail and is based on the following tasks:

✔ integrating all relaying, control, monitoring, automation, and enterprisefunctions through Ethernet LANs in the substations and EPRI’s com-mon information model (CIM)

✔ introducing IEC 61850 LAN integration system and protocol as rapidlyas is practical, to replace control wiring, and to simplify integration anddata flow

✔ organizing protective functions using the newest generations of relays toimprove dependability and security, while drastically reducing the num-ber of units required and complying with or exceeding all agencydesign requirements

✔ looking at recent operating issues, relaying problems, industry trends,and recent wide-area system events; and designing a solution that aimssquarely at improving performance on these specifics.

Key steps to implement the standardized substation protection and controldesign are being carried out. The first step was to develop a practical and inno-vative technical strategy for system-wide wholesale upgrading, includingstudying the existing system design and operating issues, recent industryevents (e.g., the August, 2003 Northeast U.S. blackout), and the designrequirements of North American Electric Reliability Council (NERC). Thestrategy describes how the functions in the latest microprocessor relays can bearrayed for the most cost-effective and fully redundant protection, whiledrastically reducing the amount of equipment from what was needed with

© EYEWIRE

preceding designs. The strategy shows how the IEC 61850LAN protocol is to be introduced into the system design.Monitoring functions that feed directly into the asset manage-ment program are included in the design.

Levels of ImplementationThese key steps in the strategy can be viewed at three levelsof implementation at the substation. The first level of imple-mentation is that the substation must be designed to operatesuccessfully. Recent technological advances were assessedfor the ability to be immediately and successfully used tooperate the system. The second level of implementation is toinsure reliability of the substation; how well it can continueto operate after the loss of any component. The third level ofimplementation is the visibility of the substation system tobeing able to track the other two levels. For example, whensubstations’ supervisory controls were limited and the pro-tection and control system was electromechanical, the sys-tem was adequately operable, often reliable enough, butalmost completely opaque. Presence on-site might increasethe visibility of the substation’s operations, but posteventanalysis often involved deductive reasoning based on cluesgathered, such as the overcurrent relay didn’t have a target,the negative sequence relay had a target, and the operatorsaid he smelled something in the yard.

OperabilityThe core function of a transmission substation is to facilitatethe flow of power through a bulk electrical system. Therefore,whatever design is used, be it fully automated or not, thebreakers have to close when power is needed to flow. Trans-formers need to be energized when the load requires it. The

newest technologies may promise much, but if they can’t beimplemented to reliably operate a system then they shouldn’tbe called on to do so. The design philosophy for the programwas summed up by the phrase “this design needs to be lead-ing edge, not bleeding edge.” Figure 1 summarizes the result-ing design reflecting the strategy.

The substation operating system features an EthernetLAN that allows data gathering along with protection andcontrol commands to be exchanged. This LAN connects allthe relays with most of the other intelligent electronic devices(IEDs) in the substation. The operating commands sent to theyard equipment are sent over hard-wire connections. Thedesign for the off-site operation of yard equipment uses a cor-porate wide-area network (WAN) via data communicationservices. Operation by personnel on-site can be performed atthe HMI by mousing over the elements featured on a one-linerepresentation of the yard (Figure 2). Personnel can alsooperate the 138-kV circuit breakers from buttons on the frontof relays (Figure 3).

Multifunction relays were used so even redundant protec-tion could be afforded without using large amounts of panelspace. The relays, being IEDs, not only performed the crucialfunction of protection but also specific features were requiredto execute the design of operation. These IEDs are enabledfor IEC 61850 and featured programmable front-panel but-tons (Figure 3) that could functionally take the place of testswitches. In addition, the design of the IEDs was flexibleenough that the core functions could be changed withoutremoving the relay from the panel. In other words, a trans-former relay can become a line relay by reconfiguring therelay and addressing the external connections; replacing thehardware platform is not required.

34 IEEE power & energy magazine may/june 2007

figure 1. New substation protection and control LAN architecture.

Managed Optical Ethernet Switches - LAN 1

Managed Optical Ethernet Switches - LAN 2

Line A Relay 2IEC 61850

and DNP 3.0

Local HMIGPS Clock

RoutersPhysical and Electrical Isolation of Redundant Protection Systems

Xfmr Relay 2IEC 61850 and DNP 3.0

Xfmr Relay 1IEC 61850

and DNP 3.0

Bus Relay 2IEC 61850

and DNP 3.0

Bus Relay 1IEC 61850

and DNP 3.0

PMU 2COMTRADE/IEEE

1344

SubstationAutomation Host

Local Historian

dDFR Host

METC Enterprise Service Providers

Monitoing IEDsSerial Comms Protocol

Connections for 1 msTime Stamp Synch

Corporate WANvia

Primary and Hot Standby DataCommunications Services

Other ControlCenters

SCADA/EMS

PMU 1COMTRADE/IEEE

1344

Line A Relay 1IEC 61850 and

DNP 3.0

may/june 2007 IEEE power & energy magazine

The end result of all these features was great freedom inthe physical design of the relay panels since the necessaryfunctions could be either hard wired, programmed into therelay, or executed over the LAN. With this great flexibility inphysical design, standards could be created that would beapplicable to more scenarios and wouldn’t change with everynew relay feature. Add to this a standardized entrance ofcables to the control house and how the cables are run to eachpanel and now standardization can go beyond the panel. Thelayout of the building could be standardized based on thefuture build of the site. Any changes to the bulk electrical sys-tem that may happen in the yard (the addition of a line ortransformer or a reassigned bus position) can be accommo-dated with minimal physical construction.

No matter how far technology can take us, it must beassumed that people will have to be able to operate the sys-tem on-site. To help insure that the implementation of thestrategy would be able to be operated, a human factors engi-neering evaluation was performed. Theobjective was to inform the final designand build out of the control houses andtheir control panels of any problemswith the human factors and ergonomicsof the workspace, as well as the userinterface with the control-house controlsand displays. Recommendations of thestudy were incorporated into the design.

ReliabilityIt is not enough that a substation operatesproperly; it must also operate reliablyunder credible contingency situations. Inthe U.S.-Canada Power System OutageTask Force report from April of 2004titled “Final Report on the August 14,2003 Blackout in the United States andCanada: Causes and Recommendations,”it was identified that one of the commoncauses of the significant outages of thelast 30 years on the bulk electrical systemlevel was a lack of “safety nets” where“A safety net is a protective scheme thatactivates automatically if a pre-specified,significant contingency occurs.” This isan important concept at the substationlevel as well and any good design needsto address “safety nets.” There are twoways to fulfill the requirement for ade-quate safety nets: either install those netsor eliminate the credibility of a signifi-cant contingency. It must be further iden-tified that there are two types of threats tothe substation that require safety nets.One is the internal threat to the systemdue to normal failure over time of power

system and control-house components. The second is theexternal threat to the substation instigated by forces from theoutside. The NERC identifies two types of electrical sectorthreats: cyber and physical (see http://www.nerc.com/cip.html). Cyber security addresses the attacks on the corpo-rate WAN that pose external threats to the communicationbetween the bulk operating system and the substation. Theproject strategy was to emphasize cyber security and design acomprehensive on-site security system to address the externalthreats to the substation’s physical plant.

Internal ThreatsThe goal was to maintain the reliable operation of the substa-tion by eliminating the effects of any single credible contin-gency on the control house. The decision was that at the345-kV level the control systems were to be fully redundant.At the 138-kV level the systems weren’t required to have fullredundancy, but they must have backups in place for each

35

figure 2. An HMI displaying the substation one-line diagram.

figure 3. Front view of an 11-1/RH30 IED.

credible contingency. The reason for the difference is that aredundant system reflects the criticality of the electrical sys-tem components at any one substation at this voltage. There-fore, a redundant system is one where even extremecontingencies have no effect on the operability of the substa-tion, whereas a backup system may have reduced operabilityunder equally extreme circumstances. These redundant sys-tems can be referred to as System 1 and System 2.

RedundancyWhat was found during design was that the technologiesadopted allowed the benefits of a standardized solution forboth levels of operation to outweigh the costs of most of theredundancy required at 345 kV. In the world of multifunctionprotective IEDs, the marginal cost between a redundant deviceand a backup device, one that may have fewer functions, is notsignificant; therefore, redundant devices were installed. Fur-thermore, it had been decided that the benefits of diversity ofmanufacturer was not significant enough to preclude evaluat-ing benefits gained from using the same manufacturer on allrelays. So the design was free to use an identical device as theredundant, driving standardization farther.

One of the early concepts behind the design was the identi-fied value of redundant battery systems. A common practice intransmission system protection is to protect for all credible sin-gle contingencies including battery failure. Since it is commonfor a substation to have one battery, its failure would leave thesubstation unable to take any action to clear a fault condition.Therefore, all remote sites have to act in place of the site withthe failed battery. With this requirement in place, distancerelays at the remote sites (sometimes referred to as Zone 3) hadto be set to see the other remote sites, which can be a very largesetting. The operation of the Zone 3 distance function duringperiods of high load was identified as a contributing cause formore than one of the major outages covered in the U.S.–Cana-da Power System Outage Task Force report. The redundantbattery minimizes the likelihood of a single credible event dis-abling all operations at a substation, removing one of the needsfor the Zone 3 distance relay to be set high.

In the final design, the largest difference between the 138-kV and the 345-kV systems is this requirement of two sepa-rate batteries (Figure 4) at the higher voltage substations andthe physical separation of the redundant systems. Beyond theneed for two batteries and physical separation, the differences

between the systems using redundancyor backup were subtle.

Since the control and protection sys-tem relies heavily on the Ethernet LAN,the failure of one of the switches or of afiber connection must have no effect oneither the ability or speed of communica-tion. To accomplish this, a design strate-gy similar to the redundant dc systemwas adopted. The implementation ofredundancy can be seen in the two viewsof the system architecture presented inFigure 1 and Figure 5, specifically in theapplication of redundant LANs.

Redundancy isn’t enough, though.You can have two eggs, but if they’re inthe same basket then the second egg maynot be worth much. An effort was madeto have physical separation between theredundant elements. In this design,redundant relays have a 6-ft aislebetween them. The two batteries are notonly in two different rooms but there aretwo battery chargers and two tray sys-tems for getting cables from the batteriesto the relay panels. The redundant Ether-net switches are also separated by anaisle. However, whereas pains were takento eliminate the close proximity of wiresfrom System 1 to wires from System 2, itwas recognized that the communicationinfrastructure had to be different. LAN 1needs to know the status of LAN 2 and


figure 4. Detail of the building layout showing two batteries.

EyeWash

125 VDC Battery Bank (Half)1 - 7 ft-0 in × 1 ft-81/2 in Rack




ExhaustFan

ExhaustFan

1 in Conduit(Weather Station)

FanControl

EyeWash

FanControl

WeatherStation

Heater

System 1Battery Room

System 2Battery Room

Louver

Louver

Heater


vice versa. Therefore, the design required a physical connec-tion between the two LANs.

This application of physical separation was extended to thetermination cabinet design. The termination cabinet is thepoint where all the cables come in from the yard that areassigned to a system. This is a large wall-mounted box filledwith columns of terminal blocks. It was observed that theeffects of the failure of any wire termination followed by a firecould wipe out an entire system. To address this, the physicaldesign of the box was revised to include metal plates betweenthe columns of terminal blocks. The blocks themselves weremounted on plates that raised them from the back of the box.This allowed the physical access to the blocks that the separat-ing plates took away. The results are that the separating plateswill limit the effect that the heat and smoke of a fire at the ter-minal blocks has on adjacent columns of blocks. Figure 6 is aclose shot of terminal block in the termination cabinet. Asyou’ll notice, there is space on either side of the block closerto the back wall of the cabinet. The separating plate is visibleas is a portion of a second block on the other side of the plate.

The pursuit of reliability has led us to the following: redun-dant dc systems: redundant relays and redundant communica-

tions. Combine these with the prevalence of redundant tripcoils on transmission system circuit breakers and the need forphysical separation between redundant elements and a virtual-ly complete System 1/System 2 approach is a natural out-growth. This redundancy is complete to the point that theentire System 1 could theoretically be taken out of service andthe bulk power system could still be operated through System2 at no loss of efficacy or speed. Redundancy not only bringsthe system to a high level of reliability but also further enablesthe modularity of design and increases the benefits of stan-dardization. A line panel on System 1 is identical to a linepanel on System 2. The panel line up for System 1 reflects thepanel line up for System 2. The termination cabinet where theSystem 1 cables come into the control house can be nearlyidentical to the termination cabinet for System 2. This conceptsimplifies the design efforts significantly.

External Threats: Cyber Security Plans for the ProjectThe many aspects and dimensions of cyber security for aproject like this are like the multiple ugly heads of themythical monster called the Hydra. Worst of all, when youthink you’ve dispatched one, another one grows to take its

37

figure 5. System architecture with greater connection detail.

Existing Control BuildingControlCenter 1

ControlCenter 2

Modem Modem

ModemSplitter

ModemSplitter

Existing RTU

SubstationNo

NewRTU

SecureServer

Master SiteServer

Runtime

MQTTMPLS

Network

Switch

T1 Relay

T1 Relay

Red BusRelay

Blue BusRelay

TransformerMonitors

Pots

PhoneSwitch

Telephone

T2 Relay

T2 Relay

B kr Relay

B kr Relay

B kr Relay

B kr RelayB kr Relay

B kr Relay B kr Relay

B kr Relay

L7 Relay L4 Relay

L4 Relay

L5 Relay

L5 Relay

L7 Relay

L8 Relay

L8 Relay

138 kV

L3 Relay

L2 Relay

L2 Relay

L1 Relay

L1 Relay

Switch

HMI No

MISC No(SecuritySystemAlarms)

Inverter

ServerRuntime

LocalSurveillance

Monitor

Switch

GPS Clock

DVMre Receiver

Switch

Server

Legend

Copper

10/100 Base FL10/100 Base TX100 Base FX

Gateway

WirelessBackup

Substation

PLC

PLC

PLC

PLC

PLC

PLC

PLC

PLC

PLC

345 kV

L3 Relay

place. Some of the ugly heads include security managementpractices, access control systems, network and telecommu-nications security, security architecture, encryption, applica-tion security, and physical security to name a few.

This section will focus on the network/telecommunica-tions security and the overall security architecture. Figure5 gives you an overall view of the architecture of the proj-ect. As with most cyber security architectures, much ofthe “defense in depth” comes from the multiple securitylevels or zones. The most secure area of the architectureis the local substation network that connects directly tothe relays. As designed, each relay has its own IP switchconnection. Not only does this eliminate any potentialcollisions but these switches have media access control(MAC) address filtering capability, thereby adding anoth-er level of security in the overall architecture. The relayshave multiple levels of passwords and audit logging toinsure access only by authorized personnel. The next levelup includes the computer that is responsible for scanningthe relays and then reporting the results “to the world.”This security layer isolates the relays from communicat-ing with multiple clients, leaving them free to do their jobof protecting the grid.

So how does the information go from “the IEDs to theBoardroom”? Well, enter the multiprotocol label switching(MPLS) network. Since a corporate SONET network was notavailable throughout Michigan, the use of a public/privatenetwork is the next logical alternative. Enter MPLS or net-works that have now become a fundamental building blockused by many of the large Internet service providers as theirbackbone. The key to the security of these networks is thatthe entire IP address space is available to each client that sub-scribes to the network service and that the core network rout-ing protocols are completely invisibleto the client and visa versa. MPLSnetworks have quietly been providingIP connectivity for several years tobusiness-critical applications. Thenext layer in the security architectureare the boundary routers that are pro-grammed to route only specific IPaddresses from one place to another.This network connects all the substa-tions in a many-to-many communica-tions network that includes thecorporate data center.

We now enter the corporate datacenter (Figure 7) and the notoriouscorporate firewall or, more appropri-ately, the corporate wall of Swisscheese. The classic firewall model isto close all ports until it is demon-strated that there is a need to have theport open. While this is a good strate-gy and represents our next security

layer, closing down all ports is a real pain operationally. Itvirtually guarantees that any new application won’t run untilthat port/service is enabled. However, cyber security wasnever about making life easy but rather minimizing risk, so“no pain no gain.” Recognize that once all the applicationsare implemented and the firewall is properly configured,there are numerous “holes” punched through the firewall,and hence the notion of a wall of Swiss cheese. Typically,the firewalls are used to implement several additional layersof security. First, the “real time” systems are fire walled offfrom the corporate network and the corporate network is firewalled off from the Internet. One of the key techniques tosupport this isolation is the use of dedicated/fixed IPaddresses for all servers. This allows firewall rules to bewritten that expressly allow traffic to and from a specificserver on a specific port. Corporate applications traffic suchas e-mail, file sharing, terminal server, etc., are prohibitedfrom entering the “real-time/substation network.” Most ofthe real-time data are stored in relational or specialized time-series databases, thereby further isolating the substationLAN from direct contact with the “outside” world.

Finally, we get to the “dreaded” Internet with all its virus-es and cyber threats. At a minimum, only those servers thatabsolutely need to serve Web traffic have a connection to theoutside world. Also, where possible, we’ve established a“DMZ” (demilitarized zone, a semiprotected LAN segment)where the Web servers operate, communicating through oneof the firewall ports back to the corporate servers.

This design provides layer upon layer of security. Onemore step in keeping the ugly Hydra/cyber security headsdispatched: AUDIT, AUDIT, AUDIT! No matter how manysleepless nights you have spent designing the most bullet-proof architecture, you don’t really know until you bring in

the “white hats” (friendly hackers)to attack the network and look forvulnerabilities on how good yourdesign really is. A key part of theplan is to perform these audits usingan independent firm and one that isfamiliar with recent work going onat the national labs. In particular, theDepartment of Energy jointly estab-lished the National SupervisoryControl and Data Acquisition(SCADA) Test Bed program atIdaho National Laboratory and San-dia National Laboratory.

Physical SecurityAmong the requirements NERC setsin CIP-006, which addresses physicalsecurity, are physical access controls,monitoring physical access, and log-ging physical access. The substationdesign leveraged technology to effec-


figure 6. Detail of a termination cabinet.


tively address these requirements. Critical control houses arebuilt with a card reader on the doors with the intent that thecontrol of the physical access can both monitor and log thataccess. Also at critical substations, perimeter cameras withmotion sensing are deployed along the fence line. The motionsensors catch suspected intruders and the cameras swing tomonitor and log the access. Logging is done by recording theimages to an on-site digital video recorder. Long-term storageof the images recorded is done as needed.

VisibilityOf the three levels of implementation discussed here, techno-logical advances in the digital realm have most benefited visi-bility. The same device that protects equipment can monitorthat equipment at no extra cost. It is now much easier to com-municate waves of instantaneous data along with videoimages, oscillography files, and event logs.

The first requirement for visibility is that remote operat-ing personnel have the real-time data at hand on which tobase informed decisions. Another benefit of redundancy andthe reliance on IEDs for system data is that if the relay thatis the source of current and voltage information fails, thenthe redundant device automatically takes on the duty of pro-viding the data. Likewise, if the device used to operate abreaker fails, its redundant partner takes over. This providesa nearly redundant SCADA system.

The cameras installed for security purposes also pro-vide literal visibility of the operation of the substation.The implementation of the cameras includes presets forcamera position on substation events. For example, if atransformer relay operates on a fault, nearby cameras willswing to the preset position pointed at the transformer.Now when remote personnel access the video images, thecamera is already in place to survey the equipment in dis-tress in real time or retrieve the most recently recordedvideo. The cameras can also be manipulated remotely to

39

figure 7. Pathways between the IEDs and the boardroom.

figure 8. Records and events collected from external sources.

ApplicationsServer

DMZ LAN

DeveloperWorkstation

OperatorWorkstation

DatabaseServer

RemoteWorkstation

Radio

Modem

Modem

FEPServer

RTU

Internet

ICCPServer

RTU

RTUSwitch

DMZFirewall

SwitchInternetFirewall

DMZFirewall

InternetFirewallSwitch

Corporate LAN

ICCPServer

WEBServer

HistorianDatabase

Server

SCADA LAN

CorporateWorkstation

check the surrounding area for indications of why the triptook place.

One of the first devices that greatly improved visibilityin substations was the digital fault recorder (DFR). DFRsare stand-alone devices connected to representative inputsthat provide crucial information regarding system condi-tions during events. The project design strategy has analternate approach. Since the relays installed alreadyrecord oscillographic data, a computer was installed dedi-cated to the retrieval of these files from all the relays.These files were then stored by time. As more substationscome online, the implementation plan for these computerswas to upload the oscillography to a central server organ-ized by substation. Figure 8 is a screen shot showingevents gathered from numerous sources arranged chrono-logically. Office analysis of system-wide events could nowhappen within minutes of the event.

Digital data acquisition affords a level of visibility neverconsidered possible in the electromechanical era of substa-tions. Exhaustive routine maintenance of every connectionbetween relays and auxiliary relays was a requirementbecause this was the only way the failure of wires or theirterminations could be discovered. With the adoption of IEC61850 for the delivery of protection related commands,every data connection between relays is under constantscrutiny. If, for whatever contingency, that protection func-tion is unavailable, not only is there a redundant function inplace but also this failed state is alarmed immediately. Thisis the equivalent of having every wire tested every fewminutes in an older substation.

With the dependence on IEDs for protection, control,and communication, it is crucial that the IEDs themselvesare visible. As described, the design of the project reducesphysical installation. Auxiliary relays have been eliminated,test switches are nearly extinct, and instead of hundreds ofwires strung between relays, there are now two pairs offibers from each relay to Ethernet switches. However, withthe elimination of the physical comes the proliferation ofthe digital. What was once communicated with detailed dcschematics and recorded relay settings now must be accom-plished with settings files and logic diagrams. And controlof the configuration of the relays is critical to the reliableoperation of the substation. These newer functions are beingsupported by IED manufacturers’ efforts to make the work-ings of multifunction microprocessor relays increasinglyvisible. The IEDs used in these substations have softwareavailable that automatically converts relay configurationinto logic diagrams and easily understandable settingsreports. The software will also document inter-relay rela-tionships, reducing the time spent on documentation. Figure9 shows a small portion of the logic diagram representationof the configuration for one of the relays.

Even though all this visibility exists, it is at a resolutionthat is not easily understood. It would be an input overloadif someone were trying to assess the data in a real-timemanner. So once you access the information, it must bestored for later analysis. It is at this point that the invest-ment in automation really comes through and the vision ofautomation is realized. The next step is to convert the datainto business intelligence.


The design philosophy for the program was summed up by the phrase “this designneeds to be leading edge, not bleeding edge.”

figure 9. Relay software-generated logic diagram.

BKR FAIL 1 TRIP OP 30H9BFR ON (VO39)

CONTROL PUSHBUTTON 2 ON

LATCH 1 ON

AND

116

BFR RST On (VI21)

LATCH 1 OFF

117

OR

118

30H9BFR OFF (VO40)

AND


Data Warehousing and Information AccessThe newest relays and communications systems selected forthe project present the enterprise with a massive stream ofsubstation data that must be automatically stored, managed,analyzed, and presented in useful forms that improve busi-ness and technical operations. The data warehousing fea-tures of modern information architectures are essential toprovide end users with easy access to the wealth of data andinformation substation devices provide. Three types of dataare being created and stored for later retrieval as needed:

✔ sequence of events records✔ high-speed time-series data records such as COM-

TRADE oscillography and phasor measurement files,

including binary status such as relay trip and close orunit line protection communications signals

✔ analog power system measurements and reports fromequipment monitors such as transformer analysis IEDs.

All the data are collected, organized, and archived at thedata-hosting center using the modeling standard CIM andproviding easy access by the staff. Today, CIM is embodiedwithin IEC standards 61968 and 61970 and the project isbenefiting from these standards through its use of readilyavailable adapters that can be used to rapidly integrate datafrom various applications. Various diagnostic tools, includingautomatic preprocessing and dashboard reporting, are beingdeveloped to aid the end user in analyzing this wealth ofinformation. In conjunction with the upgrade project is an

41

It is not enough that a substation operates properly; it must also operate reliably under credible contingency situations.

figure 10. The basic architecture of the decision support system.

Portlets

FinancialDB's

Utility Databases

EventDB

PerformanceDB

GISDB

CMMSDB

EMSDB

Sources:StaffContractorsSuppliersCustomersField ForceRegulators

DataHistorian

MonitorPoints

MonitorEvents

CalcEvents

CalcPoints

Substation Devices

Utility Real Time Data

Remote AssetMonitoring

Tools

ExpertGrid

Analyzers

Portal Server

WebPages

Dashboards,Scorecards,

OLAP, Reports

ApplicationServer

IEC 61968/61970 Compliant Middleware

CIMData

Warehouse

Real-TimeEvent

Analyzers

Analytics Stack

OptimizationComputations

DashboardTools

OLAP andHypercubes

CustomAnalytics

Data Miner

NotificationServer

ET

L

integral effort to develop information systems for the staff.These systems will benefit asset management, system plan-ning, and operations.

Information dashboards that support each of these busi-ness areas are currently under development. For example,operations will have ready access to fault location informa-tion, SER, lightning strikes, transformer monitors, weatherand video streams, and operating performance information.Planners will benefit with more accurate system and deviceloading information, better system event capture, weatherprofiles, and improved system models.

Asset management will have access to an immensearray of information that will be distilled from thedetailed operating data. For example, consider trans-former monitoring and life management. The transformerrelays report currents and voltages. Oil condition is moni-tored with installed gas-in-oil detectors. Top-oil tempera-ture sensors and accessory alarms connect to data-collection IEDs. A weather station reports ambient condi-tions. This body of data can support life assessment andemergency operating decisions. To get these results, thekey data is extracted and analyzed with modeling algo-rithms and stored as trend results for each transformer.Self-organizing neural networks preprocess the vastamounts of operational data for the operating and mainte-nance management personnel who get prioritized succinctinformation on which they can act, quickly if needed.Also, data or alarms indicating maintenance problems orrepair issues can act as triggers. These can originate in thesubstation or with back-office processing functions. Thesedrive notices to business partners; create work orders andstatus tracking, map issues to geographic information sys-tems, update asset management records, and search forpatterns or issues requiring broad action.

Converting Data intoBusiness IntelligenceRecently there have been a number of initiatives in the U.S.power industry around the notion of an intelligent grid.Along these lines, one of the core elements of this projectis to capture all of the data available at the substation andstream it to a central location for decision making as wellas operational support.

The substation data sources are the following:✔ advanced IEDs for circuit breaker operation✔ digital fault recorder

✔ transformer monitoring systems✔ security system✔ weather station (not shown on the diagram)✔ phasor measurement units (PMUs)The automated substations contain distributed data his-

torian servers that collect data from the substation datacollection system (DCS) and send them to the central datahistorian server. Unlike a relational database, data histori-ans provide an efficient means of storing temporal data(time-series data) using various algorithms to essentiallycompress the data. One needs to use caution in theirselection of historian vendors to assure that the compres-sion method is adequate for the intended use. We chose touse a vendor that implements a lossless method. The com-munications network and quality of service controls areused to prioritize data traffic from the substations, withSCADA having the highest priority. The data historianreal-time service receives IED data from the DCS and for-wards them to the historian server. This is the means bywhich real-time data are made available to the centralizeddata warehouse providing a means for business analyticsto be performed.

The project identified four levels of analytics:Level 1 Simple thresholds and alarms: monitoring trans-

former oil temperature.Level 2 Financial trends, basic system performance met-

rics: budget versus actual, TSAIDI, TSAIFI, etc.Level 3 Real-time event analysis, interpretation of event

sequences: diagnosing circuit breaker failuremodes from DFR waveform data.

Level 4 Analytics for optimization purposes: prioritiza-tion of asset maintenance, asset replacement.

The decision support system (Figure 10) implements thedata integration, analytics, and information distributionfunctions. The central data repository is a data warehousethat is structured in compliance with the CIM for utilities.Use of CIM is central to the concept of open standards.Information is disseminated from the data warehouse andanalytics stack throughout the enterprise via Web servicesand portals.

Figure 11 shows how the analytics provide decision sup-port for operations and business functions. The Level 1 ana-lytics (parameter thresholds and notifications) are quiteextensive and require sophisticated management to permiteach authorized user to subscribe to only those notificationsthat are of interest in the user’s job role. Any authorized user,


Redundancy not only brings the system to a high level ofreliability but also further enables the modularity of designand increases the benefits of standardization.


from maintenance engineer up to chief operations officer, cansubscribe to notifications as desired. This is likewise true forhigher-level analytics, key performance indicator dashboards,and decision support analyses.

The CIM data warehouse and data historian jointly sup-ply the data to drive a variety of analytics. The project hasdefined a large set of analytics, including 19 system per-formance metrics and many financial and operationalmeasures. The analytics architecture supports both opera-tional metrics and business key performance indicators andeach person in the organization can receive the relevantanalytics and support data and can customize his or herportal to show preferred information in preferred locationsand formats.

Through careful consideration of the relevant key busi-ness drivers, the project arrived at a suitable intelligentgrid strategy. Having created a business model that reliesheavily upon outsourcing, advanced automation, and theuse of analytics to support operational and business deci-sions, the team developed an architecture that uses tech-nology to support both the outsourcing strategy and

minimal staffing by making maximum use of informationsources and tools.

The business case for this approach shows the value ofadvanced automation and decision support tools to be con-tained in the following:

✔ reduced operation and maintenance expenditures ✔ reduced capital expenditures✔ low staffing requirements✔ increased transmission system reliability✔ preservation of the value of infrastructure through use

of open standards.Realization of these benefits is achieved by maintaining

a low headcount, using remote monitoring to reduce fieldmanpower through reduction of both scheduled andunscheduled visits to their widespread collection of substa-tions, reducing CAPEX and operation and maintenancecosts through improved information-based asset manage-ment and reliability-centered maintenance, and using openstandards to guide the selection of equipment, systems, andarchitectures that minimize the future impact of changes inany one system, component, or supplier.

43

figure 11. Analytics support for business functions.

Utility Databases

FinancialDB's

PerformanceDB's

MaintenanceDB

GISDB

CMMSDB

IncidentDB

Source:StaffContractorsSuppliersCustomersField ForceRegulators

DataHistorian

MonitorPoints

MonitorEvents

CalcPoints

CalcEvents

Utility Real Time Data

Analytics L.3

Real Time Event AnalysisAdvanced Diagnostics

Analytics L.1

Thresholdsand

Alarms

Integer Programming ML Estimators and ClassifiersIntegral Maximization, Linear and Nonlinear Programming,Dynamic Programming, ACO/PSO, Simulated Annealing,Search Techniques

Optimization Tools

Analytics L.4

Analytics L.2

Data Mining, Clustering,Regression, CART,Model Construction

Grid Meta Data

CIMDataWarehouse

OLAP,Simple Meterics

AssetNormalized

Models

Constraints

BudgetCash FlowRegional

ManpowerRegulatory

Inter-Department

Dashboards,Scorecards,Cube Views,Reports

Solutions:Prioritization,Subsetting

Strategic Functions

Asset Life Cycle ManagementGrid Expansion PlanningSystem Performance AnalysisCAPEX OptimizationFault Mitigation PlanningPerformance Metric ImprovementPost-Fault Analysis

Operational Functions

Work ManagementPredictive MaintenanceReal Time Event InterpretationGrid ControlAsset Utilization OptimizationEvent-Based Maintenance

Utility Front and BackOffice Functions

Implement Decisions and Control

Transform Data into Information

Collect Low Level Data and Events

ET

L

Future SightInvesting in a robust communication infrastructure, a nimbleLAN-based operating system and the computing power ofIEDs put into place a system that is flexible enough to beenhanced in the future, especially in the area of visibility.

PMUs are part of an emerging technology. The projectimplementation included the installation of PMUs on all345-kV buses. This arrangement provided for monitoring ofall lines and transformers associated with the station.Improved visibility and operator situational analysis werekey factors in implementing this technology. One of the keyfindings of the August 2003 blackout was the lack of opera-tor awareness during the time leading up to the blackout.PMUs offer substantially improved intelligence not only forreal-time operations but also for postevent analysis and sys-tem model validation and more.

Farther down the path of substation automation is theuse of IEC 61850 GOOSE messages over the corporateWAN to other substations. If this method doesn’t reach thespeed of communication over power line carrier or fibercommunications for carrying pilot protection data, then itmight be used as a backup. This might also be a way toperform inter-substation remedial action schemes and widearea protection or monitoring schemes.

The Vision of Substation AutomationRecognizing the limitations of the term “substation automa-tion,” the concept it represents is realizing all the benefitsthat digital technology can bring to the substation. Theoverall strategy is not only to automate substations but tooptimize them. Substation automation is not only a protec-tion issue, it’s not a metering issue, and neither is it a super-visory control or a data acquisition issue. The vision ofsubstation automation is not only system wide but systemdeep. In this case, it not only runs from the MackinacBridge to the “wrist” of the Michigan “mitten,” substationautomation affects the system from the terminal block to the345-kV circuit breaker. That three-breaker substation east ofthe Michigan Dunes is the beginning of the realization ofthe vision of substation automation.

AcknowledgmentsIEC is a registered trademark of Commission Electrotech-nique Internationale. Google is a registered trademark ofGoogle Technology, Inc. SONET is a registered trademark ofSONAT, Inc.

For Further ReadingR. Brantley, K. Donahoe, J. Theron, and E. Udren, “Theapplication of IEC 61850 to replace auxiliary devices includ-ing lockout relays,” presented at the 60th Annual GeorgiaTech Protective Relaying Conference, Apr. 2006.

U.S.-Canada Power System Outage Task Force, “Finalreport on the August 14, 2003 blackout in the United Statesand Canada: Causes and recommendations,” Apr. 2004[Online]. Available: http://www.nerc.com.

R. Krutz and R. Vines, The CISSP Prep Guide: Masteringthe CISSP and ISSEP Exams, 2nd ed. New York: Wiley, Apr.2004.

BiographiesPaul Myrda has 30 years of experience in electrical powersystems engineering. Most recently he was director of opera-tions and chief technologist for Trans-Elect Inc. He wasinstrumental in developing an overarching strategy in assetmanagement and championed an innovative protection andcontrol system upgrade project for the Michigan ElectricTransmission Company, a former affiliate of Trans-Elect. Thisproject fully leveraged the capability of IEC 61850-basedmicroprocessor relays, physical security, telecommunications,and data warehousing technologies using EPRI’s commoninformation model. His diverse background includes planning,engineering, information systems, and project management.He has an M.B.A. from Kellogg Graduate School of Manage-ment and an M.S.E.E. and a B.S.E.E. from Illinois Institute ofTechnology. He is a licensed professional engineer in Illinois,a member of CIGRE, and a Senior Member of the IEEE.

Kevin Donahoe has spent the last 25 years working in theelectric utility industry. The last 22 of those years have beenspent testing, installing, trouble shooting, specifying, setting,estimating, designing, reviewing, documenting, and settingstandards for protection and control schemes. He spent 20 years with Commonwealth Edison, an Exelon company,before moving to GE Energy. Though the majority of hisexperience has been with transmission and distribution sub-stations, he has significant experience with generation protec-tion and distribution protection with specific experience withinterconnection requirements. He received his B.S.E.E. fromthe Illinois Institute of Technology in 1981 and in 1993received an M.B.A. from Lewis University. Donahoe is amember of the IEEE Power System Relaying Committee andthe IEEE Standards Advisory. He is a licensed professionalengineer in Illinois, Oklahoma, and Michigan.


p&e

The many aspects and dimensions of cyber security for a project like this are like the multiple ugly heads of the mythical monster called the Hydra.

Documents

The true vision of automation