Ieee Pes 2013 Smart Grid Compendium

Smart Grid: Challenges & Opportunities

INT8684_PECover2013_8.indd 1 4/4/13 5:11 PM

You don’t have to be Thomas Edison to be a force for change.Just a PES member.Joining the IEEE Power & Energy Society can provide a big boost to your career by enabling you to:

• Tackle broad-reaching challenges

• Become recognized as a thought leader by your industry peers

• Develop contacts that will prove useful throughout all stages of your career

• Be a part of the very active and engaged global PES Community

We help our members to be successful by providing:

• Up-to-date information on current trends and the latest technology

• Industry insight through Power & Energy magazine, technical reports and peer-reviewed publications

• Compelling programs and networking opportunities at our conferences and events

• Opportunity to meet, network and collaborate with local members via our vibrant chapters

Over 30,000 members of the IEEE Power & Energy Society recognize that their membership is an exceptional, cost-effective way to acquire the latest information about all aspects of the fast-changing electric power and energy industry. You can too, if you join us now!

To learn more about the IEEE Power & Energy Society, including the many other membership benefits, please visit

www.ieee-pes.org.

IEEE Power & Energy Society 445 Hoes Lane Piscataway, NJ 08854 USA

IEEE_MemberAds.indd 1 4/1/13 3:53 PM

ieee power & energy magazine 1

www.ieee.org/power

cont

ents

m a g a z i n e

features5 Smart Grid — Safe, Secure, Self-Healing

Challenges and Opportunities in Power System Security, Resiliency, and PrivacyBy S. Massoud Amin and Anthony M. Giacomonis

13 A Virtual Smart GridReal-Time Simulation for Smart Grid Control and Communications DesignBy David Anderson, Chuanlin Zhao, Carl H. Hauser, Vaithianathan Venkatasubramanian, David E. Bakken, and Anjan Bose

22 Forward PassPolicy Changes and Technical Opportunties on the U.S. Electric GridBy Timothy D. Heidel, John G. Kassakian, and Richard Schmalensee

30 DC, Come HomeDC Microgrids and the Birth of the “Enernet”By Brian T. Patterson

41 Enhancing Grid MeasurementsWide Area Measurement Systems, NASPInet, and SecurityBy Rakesh B. Bobba, Jeff Dagle, Erich Heine, Himanshu Khurana, William H. Sanders, Peter Sauer, and Tim Yardley

49 Staying in ControlCybersecurity and the Modern Electric GridBy Julie Hull, Himanshu Khurana, Tom Markham, and Kevin Staggs

on the cover


...a 2013 reprint journal from PES

49

13

2 ieee power & energy magazine

m a g a z i n e

Digital Object Identifier 10.1109/MPE.2013.2239531

Editor in ChiefMelvin I. Olken245 East 19th Street #20KNew York, NY 10003-2665 USA+1 212 982 8286 (phone fax)[email protected]

Associate EditorsGerald B. Sheblé, Business SceneCarl L. Sulzberger, History

Editorial BoardS. Massoud Amin, L. Goel, A.P. Hanson, N. Hatziargyriou, M.I. Henderson, S.H. Horowitz, P. Kundur, R. Masiello, K.M. Matsuda, A.P.S. Meliopoulos, M.I. Olken, M. O’Malley, A.G. Phadke, R.J. Piwko, C.E. Root, H. Rudnick, P.W. Sauer, M. Shahidehpour, B.R. Shperling, S.S. Venkata, B.F. Wollenberg

AdvertisingBarry LeCerfBullseye International Group, Inc.+1 913 663 1112, fax +1 913 663 [email protected]

IEEE Power & Energy MagazineIEEE Power & Energy Magazine (ISSN 1540-7977) (IPEMCF) is published bimonthly by the Institute of Electrical and Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th Floor, New York, NY 10016-5997 USA. Responsibility for the contents rests upon the authors and not upon the IEEE, the Society, or its members. IEEE Operations Center (for orders, sub-scriptions, address changes): 445 Hoes Lane, Piscataway, NJ 08854 USA. Telephone: +1 732 981 0060, +1 800 678 4333. Individual copies: IEEE members US$20.00 (first copy only), nonmembers US$77.00 per copy. Subscription Rates: Society members included with membership dues. Subscription rates available upon request. Copyright and reprint permis-sions: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy beyond the limits of U.S. Copyright law for the private use of patrons 1) those post-1977 articles that carry a code at the bottom of the first page, provided the per-copy fee indicated in the code is paid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2) pre-1978 articles without fee. For other copying, reprint, or republication permission, write Copyrights and Permissions Department, IEEE Operations Center, 445 Hoes Lane, Piscataway, NJ 08854 USA. Copyright © 2013 by the Institute of Electrical and Electronics Engineers, Inc. All rights reserved. Periodicals postage paid at New York, NY, and at additional mailing offices. Postmaster: Send address changes to IEEE Power & Energy Magazine, IEEE Operations Center, 445 Hoes Lane, Piscataway, NJ 08854 USA. Canadian GST #125634188

Printed in U.S.A.

ieee power & energy society (pes)The IEEE Power & Energy Society is an organization of IEEE members whose principal interest is the advancement of the science and practice of elec-tric power generation, transmission, distribution, and utilization. All members of the IEEE are eligible for membership in the Society.Mission Statement: To be the leading provider of scientific and engineering information on electric power and energy for the betterment of society, and the preferred professional development source for our members.

OfficersN.N. Schulz, PresidentM.M. Begovic, President-ElectM. Selak, Vice President, ChaptersJ.H. Nelson, Vice President, Technical ActivitiesP.W. Sauer, Vice President, EducationS. Rahman, Vice President, PublicationsW. Rosehart, Vice President, MeetingsH. Louie, Vice President, Membership & ImageR. Podmore, Vice President, New Initiatives/ OutreachL. Bertling Tjernberg, TreasurerC. Root, SecretaryA.C. Rotz, Past-President

IEEE Division VII DirectorC. Warren

IEEE Division VII Director ElectW.K. Reder

Region RepresentativesM. Chaganti, Y. Chen, T. Hiemer, F. Lambert, M. Nissen, J. Skillman, United StatesM. Armstrong, Canada C. Vournas, Europe, Middle East, & AfricaN. Segoshi, Latin AmericaL. Goel, Asia & Pacific

Governing Board Members-at-LargeE. Gunther, M. Jensen, J. Giri, T. Prevost

PES Executive DirectorPatrick Ryan, +1 732 465 6618,fax +1 732 562 3881, e-mail [email protected]

Standing Committee ChairsM. Crow, Awards & Recognition N. Nair, Constitution & Bylaws L. Bertling Tjernberg, Finance & AuditA.C. Rotz, Nominations & AppointmentsK. Butler-Purry, Power Engineering EducationW.K. Reder, Scholarship Plus

Chapter RepresentativesB. Allaf, J. Ammentorp, A. Bakirtzis, J.G. Calderon, R. Cespiedes, C. Diamond, D. Drumtra, J. Fleeman, B. Gwyn, Z.F. Hussien, I. Kuzle, N. Logic, J.C. Montero, P. Naidoo, T. Rajagopalan, P. Pabst, D. Sharafi, G.N. Taranto, E. Tobin, E. Uzunovic, D. van Hertem

Chapter Committee ChairsS. Chakravorti, Chapter SecretaryN. Mariun, Chapter/Section RelationsM. Armstrong, Electronic CommunicationsE. Carlsen, Awards & ResourcesY. Chen, Distinguished Lecturer ProgramK. Hadzimahovic, Chapters Web site

Membership & Image Committee ChairsA. St Leger, GOLD CoordinatorA. Bonthron, Membership DevelopmentL. Fan, Membership DevelopmentOpen, Web Site DevelopmentS. Bahramirad, PES WIE Liaison W. Bishop, Marketing

Technical CouncilJ.H. Nelson, ChairS.S. Venkata, Vice-ChairK.S. Edwards, SecretaryD. Novosel, Past-ChairM. Maytum, Web Master

Technical Committee ChairsM. Sedlak, Electric MachineryR. Groves, Energy Development & Power GenerationJ. Smith, Insulated ConductorsG. Ballassi, Nuclear Power EngineeringS. Carneiro, Jr., Power System Analysis, Computing, & EconomicsD. Nordell, Power System Communications T. Van Cutsem, Power System Dynamic Performance R. Arseneau, Power System Instrumentation & Measurements A. Conejo, Power System OperationsM.L. Chan, Power System Planning & Implementation R. Hedding, Power System Relaying L. Varga, Stationary Battery M. Dood, Substations A.J. Surtees, Surge Protective Devices T.W. Olsen, Switchgear W. Chiu, Transformers W.A. Chisholm, Transmission & Distribution

Coordinating CommitteesB. Djokic, Emerging TechnologiesS. Pullins, Intelligent GridP. Bishop, Marine SystemsR.J. Piwko, Wind & Solar Power

Standing CommitteesJ. Randolph, AwardsS.S. Venkata, Technical SessionsS.S. Venkata, Meetings & MarketingK. Edwards, Organization & ProceduresW. Bartley, Standards Coordination

Additional PositionsD. Nordell, Editor in Chief of Conference Papers

IEEE Periodicals/Magazines Department445 Hoes Lane, Piscataway, NJ 08854 USA+1 732 562 3950, fax +1 732 981 1855www.ieee.org/magazines

Geraldine Krolin-Taylor, Senior Managing EditorJanet Dudar, Senior Art DirectorGail A. Schnitzer, Assistant Art DirectorTheresa L. Smith, Production CoordinatorPeter M. Tuohy, Production DirectorFelicia Spagnoli, Advertising Production ManagerDawn Melley, Editorial DirectorFran Zappulla, Staff Director, IEEE Publishing OperationsIEEE prohibits discrimination, harassment, and bullying. For more information, visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html.


Noel Schulz, President, IEEE PES

lead

er’s

corn

er

greetings from theIEEE Power & Energy Society (PES)

WWELCOME TO OUR NEW COMPEN-dium of articles that appeared in last year’s IEEE P&E Magazine.

If the past is prologue to the future, you’ll find an array of current issues treated in depth here in this useful digest. In fact, looking over past reprint issues, it’s clear that a handful of fundamental themes continue to challenge the IEEE PES as well as the power industry at large. Yet as our thinking advances, we’re see-ing these issues more clearly, discovering fresh approaches and our collective work at meeting these challenges continues.

I saw many colleagues at the Febru-ary 2013 IEEE PES Innovative Smart Grid Technologies (ISGT) North Amer-ica conference in Washington, D.C. and I look forward to seeing many more of you at our upcoming PES meetings across the globe including our ISGT conference suite of meetings in Sao Paulo, April 15-17, and Copenhagen, Oct. 6-9. ISGT conference programs reflect many of the issues presented here and will undoubtedly continue to drive discussion, research and efforts to reach solutions in 2013.

I’d like to remind all of our members that these conferences offer outstanding networking opportunities to engage with your colleagues and broaden your hori-zons. The ISGT conference in Washing-ton, DC involved participants from over 30 countries. I had the opportunity to recap PES-specific priorities, including our progress on standards, and our chal-lenge in persuading students to join not only the power field but our Society as well. I also announced a new PES publi-cation, Electrification.

Patricia Hoffman, assistant secretary at the U.S. DOE in the office of Elec-tricity Delivery and Energy Reliability, delivered a timely keynote on “Grid Modernization and Resiliency,” which articulated a theme that underlies much of our work and is reflected by the col-lection of articles you hold in your hands. No nation is without power-related chal-lenges in supporting its economic vitality and security and our international mem-bership guarantees that we maintain a broad perspective on these issues.

Together we’re working toward solu-tions and sharing our findings across the world. By solving our shared electricity-related challenges and sharing the fruits of our work that we can advance grid modernization for every member of the international community.

That brings me to the content of this new compendium. You’ll glean the details from the Table of Contents, but we’ve packaged here articles on the array of issues that remain relevant year after year. We bring you up-to-date explorations of:

✔ the fundamental drivers, chal-lenges and probable solutions of and for smart, modernized grids,

✔ federal and state regulatory issues that bear scrutiny and need carefully considered changes,

✔ the rebirth of interest in DC power and how its efficiencies and other characteristics will make it an excit-ing area for innovative solutions,

✔ cost-effective, software-based sim ulation models that save the time and cost of laboratory or real-world trial and error in advancing power systems,

✔ cyber-attack vectors and vulner-abilities and the value of simula-tion for developing solutions,

✔ the fundamental value of obtain-ing accurate data on the grid’s operational state, made possible by wide area measurement sys-tems, or WAMS,

✔ and more on cyber security and communication networks.

Of course, applying our knowledge, pursuing discoveries and sharing insights is what makes IEEE PES a world-class hub for innovation. So read this digest, share it with colleagues, explain the ideas to family and friends (it’s a good practice, they’re energy consumers, too). Share your work by writing an article for IEEE P&E Magazine, visit the IEEE Smart Grid Portal (http://smart-grid.ieee.org/), keep up with our IEEE PES and smart grid tweets (@ieee_pes and @ieeesmart-grid) and join IEEE Smart Grid LinkedIn discussion group.

The sustainable, efficient provision and wise use of electricity can pave the way for more productive and sustain-able societies. I invite you to partake of this volume’s thoughtful articles and advance the search for solutions.

Sincerely,

Noel N. SchulzIEEE PES President, [email protected]://www.ieee-pes.org/

March 15, 2013To: Recipients of the 2013 IEEE PES Smart Grid Reprint Journal:

IEEE Transactions on Smart Grid

The IEEE Transactions on Smart Grid is intended to be a cross disciplinary and internationally archival journal aimed at disseminating the results of research on smart grid that relates to energy generation, transmission, distribution and delivery. The journal will publish original research on theories, technologies, design, policies, and implementation of smart grid. The Transactions will welcome manuscripts on design, implementation and evaluation of energy systems that include smart grid technologies and applications. Surveys of existing work on smart grid may also be considered for publication when they propose a challenging perspective on the future of such technologies and systems. Topical issues considered by the Transactions include:

> Smart sensing, communication and control in energy systems

> Wireless communications and advanced metering infrastructure

> Smart grid for energy management in buildings and home automation

> Phasor measurement unit applications for smart grid

> Smart grid for plug-in vehicles and low-carbon transportation alternatives

> Smart grid for cyber and physical security systems

> Smart grid for distributed energy resources

> Smart grid for energy savings and fi nancial management

> Smart grid in interdependent energy infrastructures

> Smart grid for intelligent monitoring and outage management

If you are interested in reviewing papers for this journal, please sign upas a reviewer on the Manuscript Central site at: http://mc.manuscriptcentral.com/pes-ieee.

The Transactions on the Smart Grid can be accessed via the drop down menu on the PES portal site. If you are interested in reviewing papers for our new Transactions and you are currently a reviewer for PES Transactions, you can access your account in Manuscript Central and add smart grid to your keywords or areas of expertise. If you have an account in Manuscript Central and are not currently a reviewer for PES Transactions and would like to become a reviewer for PES Transactions, access your account and you will automatically be given a reviewer center, then update your areas of expertise. If you do not have an account, create a new user account and complete all the required fi elds, you will then be given an author center and a reviewer center.

About the Editor-in-Chief: If you are interested in participating in the publication activities, please contact the Editor-in-Chief, Dr. Mohammad Shahidehpour at: [email protected]. Shahidehpour (Fellow ‘01) has been affi liated with IEEE for the last thirty years. His is currently the Carl Bodine Distinguished Professor of Electrical and Computer Engineering at Illinois Institute of Technology. Dr. Shahidehpour is an IEEE Distinguished Lecturer who has lectured in 30 countries on issues related to power system operation and control. He has servedas the Vice President of Publications for the IEEE Power & Energy Society and an Editor of theTransactions on Power Systems.

1540-7977/11/$31.00©2012 IEEEjanuary/february 2012 IEEE power & energy magazine 33january/february 2012 IEEE power & energy magazine 33

Smart Grid—Safe, Secure,Self-HealingChallenges and Opportunities in Power System Security, Resiliency, and Privacy

By S. Massoud Amin and Anthony M. Giacomoni

TTHE EXISTING POWER DELIVERY system is vulnerable to both natural disasters and intentional attack. A suc-cessful terrorist attempt to disrupt the power delivery system could have adverse effects on national security, the economy, and the lives of every citizen. Secure and reliable operation of the electric system is fundamental to national and international economic systems, security, and quality of life.

This is not new: both the importance and the diffi culty of protecting power sys-tems have long been recognized. In 1990, the U.S. Offi ce of Technology Assess-ment (OTA) issued a detailed report, Physical Vulnerability of the Electric System to Natural Disasters and Sabo-tage. The report concluded: “Terrorists could emulate acts of sabotage in several other countries and destroy critical [power system] components, incapacitating large

© B

RA

ND

X P

ICT

UR

ES

Digital Object Identifi er 10.1109/MPE.2011.943112

Date of publication: 13 December 2011

1540-7977/12/$31.00©2012 IEEE ieee power & energy magazine 5

Reprinted from January/February 2012 issue of IEEE Power & Energy magazine

january/february 2012 IEEE power & energy magazine 35

enormous investment, including more than 15,000 genera-tors in 10,000 power plants and hundreds of thousands of miles of transmission and distribution lines. With dimin-ished transmission and generation capacity and with dra-matic increases in interregional bulk power transfers and the diversity of transactions, the electric power grid is being used in ways for which it was not originally designed. Grid congestion and atypical power fl ows have been increasing during the last 25 years, while customer expectations of reli-ability and cyber and physical security are rising to meet the needs of a pervasively digital world.

Upgrading the control and communication systems for the power grid will present many new security challenges that must be dealt with before extensive deployment and implementation of smart grid technologies can begin. The digitization of such systems may enable remote attacks to grow rapidly, potentially spanning countries or even con-tinents. Moreover, the number of threats against computer systems is rapidly increasing due to the increased availabil-ity of highly sophisticated hacker tools on the Internet and the decrease in technical knowledge required to use them to cause damage. While the digitization of such systems will present many new security challenges, it will also provide the grid with increased fl exibility to prevent and withstand potential threats.

Key Smart Grid Security Challenges

Physical ChallengesThe size and complexity of the North American electric power grid makes it impossible both fi nancially and logis-tically to physically protect the entire infrastructure. There currently exist more than 450,000 mi of 100-kV or higher transmission lines and many more thousands of miles of lower-voltage lines. As an increasing amount of electric-ity is generated from distributed renewable sources, the problem will only be exacerbated; the U.S. Department of Energy (DOE) has concluded that generating 20% of all electricity with land-based wind installations will require at least 20,000 square miles. Thus it is probable that a well-organized, determined group of terrorists could take out portions of the grid as they have previously done in the United States, Colombia, and other locations around the globe. Several such incidents in the United States have been publicly reported during the last 30 years, includ-ing saboteurs operating in the Pacifi c Northwest and those using power lines and transformers for target practice on the East Coast. Colombia, for example, has faced up to 200 terrorist attacks per year on its transmission infra-structure over the last 11 years, as reported in a recent IEEE Power & Energy Magazine article by Corredor and Ruiz. Such attacks, although troublesome and costly to the local region, affect only a small portion of the over-all grid, however. To cause physical damage equivalent to that from a small to moderate-size tornado would be

extremely diffi cult, even for a large, well-organized group of terrorists.

Data on terrorist attacks on the world’s electricity sec-tor from 1994–2004 from the Oklahoma-based Memorial Institute for the Prevention of Terrorism show that transmis-sion systems are by far the most common target in terms of the total number of physical attacks. Figure 2 shows the percentage of terrorist attacks aimed at each of the major grid components.

One possible means of increasing the physical security of power lines is to bury them. A 2006 study by the Edison Electric Institute (EEI) calculated that putting power lines underground would cost about US$1 million per mile, com-pared with US$100,000 per mile for overhead lines, making the idea fi nancially infeasible.

Cyber ChallengesThe number of documented cyberattacks and intrusions worldwide has been rising very rapidly in recent years. The results of a 2007 McAfee survey highlight the pervasiveness of such attacks. For example, Figure 3 shows the percent-age of IT and security executives from critical infrastructure enterprises located in 14 countries around the world report-ing large-scale distributed denial-of-service (DDoS) attacks and their frequency.

DDoS attacks utilize networks of infected computers—whose owners often do not even know that they have been infected—to overwhelm target networks with millions of fake requests for information over the Internet.

Due to the increasingly sophisticated nature and speed of malicious code, intrusions, and DoS attacks, human responses may be inadequate. Figure 4 shows the evolution of cyberthreats over the last two decades and the types of responses that can be used to combat them effectively.

In addition, adversaries often have the potential to ini-tiate attacks from nearly any location in the world. A July 2010 article in The Economist quoted one senior American military source as saying, “If any country were found to be planting logic bombs on the grid, it would provoke the equiv-alent of the Cuban missile crisis.” Furthermore, currently

GenerationSubstations

TransmissionAll Others

11%

14%

13%

62%

figure 2. Electric terrorism: grid component targets, 1994–2004 (source: Journal of Energy Security).

34 IEEE power & energy magazine january/february 2012

segments of a transmission network for months. Some of these components are vulnerable to saboteurs with explo-sives or just high-powered rifl es.” The report also docu-mented the potential costs of widespread outages, estimat-ing them to be in the range of US$1 to US$5 per kWh of disrupted service, depending on the length of the outage, the types of customers affected, and a variety of other factors. In the New York City blackout of 1977, for example, dam-age from looting and arson alone totaled about US$155 mil-lion—roughly half of its total cost.

During the 20 years since the OTA report, the situation has become even more complex. Accounting for all criti-cal assets includes thousands of transformers, line reactors, series capacitors, and transmission lines. Protecting all these diverse and widely dispersed assets is impractical. Moreover, cyber, communication, and control layers add new benefi ts only if they are designed correctly and securely.

Electricity Infrastructure: Increasing InterdependenciesEnergy, telecommunications, transportation, and fi nancial infrastructures are becoming increasingly interconnected, thus posing new challenges for their secure, reliable, and effi cient operation. All of these infrastructures are complex networks—geographically dispersed, nonlinear, and inter-acting both among themselves and with their human owners, operators, and users (see Figure 1).

Virtually every crucial economic and social function depends on the secure and reliable operation of these infra-structures. Indeed, they have provided much of the high standard of living that the more developed countries enjoy. With increased benefi t, however, has come increased risk. As these infrastructures have grown more complex in order to handle increasing demands, they have become increas-ingly interdependent. The Internet, computer networks, and our digital economy have all increased the demand for reli-able and disturbance-free electricity; banking and fi nance depend on the robustness of electric power, cable, and wire-

less telecommunications infrastructure. Transportation sys-tems, including military and commercial aircraft and land and sea vessels, depend on communication and energy net-works. Links between the power grid and telecommunica-tions systems as well as between electrical power lines and oil, water, and gas pipelines continue to be the lynchpins of energy supply networks. This strong interdependence means that an action in one part of an infrastructure network can rapidly create global effects by cascading throughout the same network and even into other networks.

In the aftermath of the tragic events of 11 September 2001 and recent natural disasters and major power outages, there have been increased national and international concerns expressed about the security, resilience, and robustness of critical infrastructures in response to an evolving spectrum of threats. There is reasonable concern that national and international energy and information infrastructures have reached a level of complexity and interconnection that makes them particularly vulnerable to cascading outages, whether initiated by material failure, natural calamities, intentional attack, or human error. The potential ramifi cations of net-work failures have never been greater, as the transportation, telecommunications, oil and gas, banking and fi nance, and other infrastructures depend on the continental power grid to energize and control their operations. Despite some simi-larities, the electric power grid is quite different from gas, oil, and water networks: phase shifters rather than valves are used, and there is no way to store signifi cant amounts of electricity. Providing the desired fl ow on one line often results in “loop fl ows” on several other lines.

Potential Route Ahead: A Smarter GridThe key challenge is to enable secure and very high-confi -dence sensing, communications, and control of a heteroge-neous, widely dispersed, yet globally interconnected system. It is even more complex and diffi cult to control it for optimal effi ciency and maximum benefi t to the ultimate consumers while still allowing all its business components to compete fairly and freely.

To achieve this goal, a new “megainfrastructure” is emerging from the convergence of energy, telecommunica-tions, transportation, the Internet, and electronic commerce. In the electric power industry and other critical infrastruc-tures, new ways are being sought to improve network effi -ciency by eliminating congestion problems without seriously diminishing reliability and security. Nevertheless, the goal of transforming the current infrastructures into self-healing energy delivery, computer, and communications networks with unprecedented robustness, reliability, effi ciency, and quality for customers and our society is ambitious.

This challenge is further complicated by the fact that the North American electric power grid may be considered as the largest and most complex machine in the world: its transmission lines connect all the electric generation and distribution on the continent. This network represents an

Excellent PowerSystem Reliability

Exceptional PowerQuality

IntegratedCommunications

Compatible Devicesand Appliances

A SecureEnergyInfrastructure

figure 1. A complex set of interconnected webs (source: EPRI, 2002–present).



enormous investment, including more than 15,000 genera-tors in 10,000 power plants and hundreds of thousands of miles of transmission and distribution lines. With dimin-ished transmission and generation capacity and with dra-matic increases in interregional bulk power transfers and the diversity of transactions, the electric power grid is being used in ways for which it was not originally designed. Grid congestion and atypical power fl ows have been increasing during the last 25 years, while customer expectations of reli-ability and cyber and physical security are rising to meet the needs of a pervasively digital world.

Upgrading the control and communication systems for the power grid will present many new security challenges that must be dealt with before extensive deployment and implementation of smart grid technologies can begin. The digitization of such systems may enable remote attacks to grow rapidly, potentially spanning countries or even con-tinents. Moreover, the number of threats against computer systems is rapidly increasing due to the increased availabil-ity of highly sophisticated hacker tools on the Internet and the decrease in technical knowledge required to use them to cause damage. While the digitization of such systems will present many new security challenges, it will also provide the grid with increased fl exibility to prevent and withstand potential threats.

Key Smart Grid Security Challenges

Physical ChallengesThe size and complexity of the North American electric power grid makes it impossible both fi nancially and logis-tically to physically protect the entire infrastructure. There currently exist more than 450,000 mi of 100-kV or higher transmission lines and many more thousands of miles of lower-voltage lines. As an increasing amount of electric-ity is generated from distributed renewable sources, the problem will only be exacerbated; the U.S. Department of Energy (DOE) has concluded that generating 20% of all electricity with land-based wind installations will require at least 20,000 square miles. Thus it is probable that a well-organized, determined group of terrorists could take out portions of the grid as they have previously done in the United States, Colombia, and other locations around the globe. Several such incidents in the United States have been publicly reported during the last 30 years, includ-ing saboteurs operating in the Pacifi c Northwest and those using power lines and transformers for target practice on the East Coast. Colombia, for example, has faced up to 200 terrorist attacks per year on its transmission infra-structure over the last 11 years, as reported in a recent IEEE Power & Energy Magazine article by Corredor and Ruiz. Such attacks, although troublesome and costly to the local region, affect only a small portion of the over-all grid, however. To cause physical damage equivalent to that from a small to moderate-size tornado would be

extremely diffi cult, even for a large, well-organized group of terrorists.

Data on terrorist attacks on the world’s electricity sec-tor from 1994–2004 from the Oklahoma-based Memorial Institute for the Prevention of Terrorism show that transmis-sion systems are by far the most common target in terms of the total number of physical attacks. Figure 2 shows the percentage of terrorist attacks aimed at each of the major grid components.

One possible means of increasing the physical security of power lines is to bury them. A 2006 study by the Edison Electric Institute (EEI) calculated that putting power lines underground would cost about US$1 million per mile, com-pared with US$100,000 per mile for overhead lines, making the idea fi nancially infeasible.

Cyber ChallengesThe number of documented cyberattacks and intrusions worldwide has been rising very rapidly in recent years. The results of a 2007 McAfee survey highlight the pervasiveness of such attacks. For example, Figure 3 shows the percent-age of IT and security executives from critical infrastructure enterprises located in 14 countries around the world report-ing large-scale distributed denial-of-service (DDoS) attacks and their frequency.

DDoS attacks utilize networks of infected computers—whose owners often do not even know that they have been infected—to overwhelm target networks with millions of fake requests for information over the Internet.

Due to the increasingly sophisticated nature and speed of malicious code, intrusions, and DoS attacks, human responses may be inadequate. Figure 4 shows the evolution of cyberthreats over the last two decades and the types of responses that can be used to combat them effectively.

In addition, adversaries often have the potential to ini-tiate attacks from nearly any location in the world. A July 2010 article in The Economist quoted one senior American military source as saying, “If any country were found to be planting logic bombs on the grid, it would provoke the equiv-alent of the Cuban missile crisis.” Furthermore, currently

GenerationSubstations

TransmissionAll Others

11%

14%

13%

62%

figure 2. Electric terrorism: grid component targets, 1994–2004 (source: Journal of Energy Security).



figure 4. Cyberthreat evolution (source: EPRI).

Seconds

Days

Months

Early 1990s Mid 1990s Late 1990s 2000 2003 Time

Con

tagi

on T

ime

Fram

e Class IIHuman Response: Difficult/ImpossibleAutomated Response: Possible

Class IHuman Response: Possible

Class IIIHuman Response: ImpossibleAutomated Response: UnlikelyProactive Blocking: Possible

File Viruses

Macro Viruses

E-Mail Threats

Blended Threats

Warhol Threats

Flash Threats

more than 90% of successful cyberattacks take advantage of known vulnerabilities and misconfi gured operating systems, servers, and network devices.

The security of cyber and communication networks is fundamental to the reliable operation of the grid. As power systems rely more heavily on computerized communications and control, system security has become increasingly depen-dent on protecting the integrity of the associated informa-tion systems. Part of the problem is that the existing control systems, which were originally designed for use with propri-etary, stand-alone communication networks, were later con-nected to the Internet (because of its productivity advantages

and lower costs) but without adding the technology needed to make them secure. Moreover, numerous types of commu-nication media and protocols are used in the communication and control of power systems. Within a substation control network, it is common to fi nd commercial telephone lines as well as wireless, microwave, optical fi ber, and Internet con-nections. The diversity and lack of interoperability among the various communication protocols cause problems for anyone who tries to establish secure communication to and from a substation.

Electric power utilities also typically own and operate at least certain portions of their own telecommunications

figure 3. Percentage of critical infrastructure enterprise executives reporting large-scale DDoS attacks and their frequen-cy (source: McAfee).

100

80

60

Per

cent

age

40

20

Bra

zil

Indi

a

Fra

nce

Spa

in

Italy

Ger

man

y

Uni

ted

Sta

tes

Aus

tral

ia

Chi

na

Japa

n

Mex

ico

Rus

sia

Tot

al

Uni

ted

Kin

gdom

Sau

di A

rabi

a/M

iddl

e E

ast

Multiple Occurrences Every DayMultiple Occurrences Every Week

Less Than Monthly OccurrencesLess Than Annual Occurrences

Multiple Occurrences Every Month



systems, which often consist of a backbone of fi ber optic or microwave links connecting major substations with spurs to smaller sites. Increased use of electronic automation raises signifi cant issues regarding the adequacy of operational security, if security provisions are not built in.

More specifi cally, the operation of a modern power sys-tem depends on complex systems of sensors and automated and manual controls, all of which are tied together through communication systems. While the direct physical destruc-tion of generators, substations, or power lines may be the most obvious strategy for causing blackouts, activities that compromise the operation of sensors, communications, and control systems by spoofi ng, jamming, or sending improper commands could also disrupt the system, cause blackouts, and in some cases result in physical damage to key system components.

Any telecommunication link that is even partially outside the control of the organization that owns and operates power plants, supervisory control and data acquisition (SCADA) systems, or energy management systems (EMSs) represents a potentially insecure pathway into the business operations of the company as well as a threat to the grid itself. The interdependency analyses done by most companies in the last 12–14 years (starting with the preparations for Y2K and continuing after the tragic events of 9/11) have identifi ed these links and the system’s vulnerability to their failure. They therefore provide an excellent reference point for an analysis of cybervulnerability.

While some of the operations on the system are automatic, human operators in system control centers ultimately make the decisions and take the actions that control the operations of the system. In addition to the physical threats to such cen-ters and the communication links that fl ow in and out of them, one must be concerned about two other factors: the reliabil-ity of the operators within the centers and the possibility that insecure code has been added to a program in a center com-puter. The threats posed by “insiders” are real, as is the risk of a “Trojan horse” embedded in the software of one of more of the control centers. A 2008 survey by the Computer Secu-rity Institute and the U.S. Federal Bureau of Investigation of data compiled from 522 computer security practitioners and senior executives of U.S. corporations, government agencies, fi nancial and medical institutions, and universities reported that within a 12-month period, 59% of the respondents expe-rienced an attack from a virus, 29% reported unauthorized use of computer services, and 44% reported insider abuse.

The threat of a “Trojan horse” embedded in the control center software can only be addressed by means of careful security measures within the commercial fi rms that develop and supply this software along with careful security screen-ing of the utility and outside service personnel who perform software maintenance within the centers. Today, security patches often are not supplied to end users, or users are not applying the patches, as they fear they will affect system performance. Current practice is to apply an upgrade or

patch only after SCADA vendors thoroughly test and vali-date it, and this sometimes causes deployment to be delayed by several months.

As a result, cybersecurity is just as important as physical security, if not more so. Due to the gravity of these threats, the Federal Energy Regulatory Commission (FERC) pol-icy statement on the smart grid states that cybersecurity is essential to the operation of the smart grid and that the development of cybersecurity standards is a key priority. The DOE has also stated that the ability to resist attack by identifying and responding to disruptions caused by sabo-tage is one of the smart grid’s seven crucial functions. Much work remains to be done, however, to create standards that, when implemented, will adequately protect the grid from cyberattacks. Emerging standards fall well short of achiev-ing this ultimate goal.

Smart Grid Security Needs

Layered SecurityIn order to protect electric infrastructure from the threats outlined above, several layers of security are needed to minimize disruptions to system operations. Layered secu-rity (or “defense in depth”) involves strategically combining multiple security technologies at each layer of a computing system in order to reduce the risk of unauthorized access due to the failure of any single security technology. It expo-nentially increases the cost and diffi culty of compromising a system by creating a much stronger defense than the use of any individual component alone, thus reducing the likeli-hood of an attack.

The trend of connecting electrical control systems to the Internet exposes all layers of a system to possible attack. Computing layers that must be considered include

✔ personnel ✔ networks ✔ operating systems ✔ applications ✔ databases.

The security features to be employed at each layer include examination, detection, prevention, and encryption. To pro-tect control systems, well-established information security practices must also be utilized.

DeceptionAn additional defense mechanism is the use of deception. Deception consists of two possible techniques: dissimulation (hiding the real) and simulation (showing the false). McQueen and Boyer describe several potential dissimulation and simu-lation techniques that can be used for control systems. Three of the dissimulation techniques described are:

✔ masking the real by making a relevant object unde-tectable or blending it into background irrelevance

✔ repackaging, which hides the real by making a rel-evant object appear to be something it isn’t



✔ dazzling, which hides the real by making the identi-fi cation of a relevant object less certain by confusing the adversary about its true nature.

Likewise, three of the simulation techniques described are:

✔ inventing the false by creating a perception that a rel-evant object exists when it doesn’t

✔ mimicking, which invents the false by presenting char-acteristics of an actual and relevant object

✔ decoying, which displays the false so as to attract at-tention away from a more relevant object.

Deception will need to play a key role in smart grid defense mechanisms. Since existing control system archi-tectures are not random and therefore response characteris-tics are reproducible, the strength of potential adversaries is amplifi ed. Defense mechanisms using deception can greatly increase the diffi culty of planning and conducting successful attacks on a system by portraying control system response characteristics as random to attackers. They can also alert operators to possible threats before any systems are harmed.

Additional security needs include rapid containment, restoration, and recovery strategies for times when systems are inevitably compromised. Either software patching or the ability to rapidly identify and isolate the exploited systems must be enabled in order to minimize downtime. This is extremely important, since the consequences of an attack are directly proportional to the length of time the service is disrupted.

Advanced Metering Infrastructure

VulnerabilitiesThe implementation of advanced metering infrastructure (AMI) is widely seen as one of the fi rst steps in the digi-tization of the electric grid’s control systems. Despite the increase in the utilization of AMI, there has been very lit-tle assessment or R&D effort to identify the security needs for such systems. Smart meters, however, are extremely attractive targets for exploitation, since vulnerabilities can be easily monetized through manipulated energy costs and measurement readings. Currently, in the United States alone it is estimated that US$6 billion is lost by electricity providers to consumer fraud in the electric grid. Possible threats to the electrical grid introduced by the use of AMI include:

✔ fabricating generated energy meter readings ✔ manipulating energy costs

✔ disrupting the load balance of local systems by sud-denly increasing or decreasing the demand for power

✔ gaining control of millions of meters and simultane-ously shutting them down

✔ sending false control signals ✔ disabling grid control center computer systems and monitors

✔ disabling protective relays.As more utilities move toward using Internet Protocol

(IP)–based systems for wide area communications and as the trend of using standardized protocols continues through-out the industry, maintaining the security of such devices will be critical. AMI introduces serious privacy concerns, as immense amounts of energy use information will be stored at the meter. Breaches into this data could expose customer habits and behaviors. Such arguments have led to the recent moratoriums on AMI installations in numerous northern California communities and other areas throughout the country. As a result, several key privacy concerns need to be addressed, including those outlined by the Cyber Security Working Group of the U.S. National Institute of Standards and Technology (NIST). These include:

✔ Personal profi ling: using personal energy data to determine consumer energy behavioral patterns for commercial purposes

✔ Real-time remote surveillance: using live energy data to determine whether people are in a specifi c fa-cility or residence and what they are doing

✔ Identity theft and home invasions: protecting per-sonal energy data from criminals who could use the information to harm consumers

✔ Activity censorship: preventing the use of energy for certain activities or taxing those activities at a higher rate

✔ Decisions based on inaccurate data: shutting off power to life-sustaining electrical devices or provid-ing inaccurate information to government and credit-reporting agencies.

In addition, AMI systems will need to be defended against more traditional cyberthreats such as mobile and malicious code, DoS attacks, misuse and malicious insider threats, accidental faults introduced by human error, and the problems associated with software and hardware aging.

Security NeedsIn order to defend against the vulnerabilities described above, several security features need to be incorporated into

Upgrading the control and communication systems for the power grid will present many new security challenges that must be dealt with.



the development of AMI, along with new privacy laws to protect consumers. Current privacy laws in the United States are fragmented and vague and do not specifi cally address consumer energy usage. Data stored at the meter and trans-mitted over communication networks must also meet stan-dard cybersecurity requirements, including confi dentiality, integrity, availability, and nonrepudiation.

One security feature alone, such as encryption, will not be able to cover all the possible security threats. Since it is imperative that the industry maintain 100% uptime, both the physical security of the AMI system hardware and multiple standard IT security features like encryption and authenti-cation must be provided for. Furthermore, since it will be impossible to protect against all threats, smart meters must be able to detect even the most subtle unauthorized changes and precursors to tampering or intrusion. Additional consid-eration must also be given to the cost and impact the secu-rity features will have on AMI system operations. Smart meters will need to be cost-effective, since millions will need to be purchased and installed to replace antiquated analog devices. And they must also be robust as they will be deployed in very insecure locations.

Current Security InitiativesSince the terrorist attacks of 11 September 2001, several steps have been taken and initiatives accomplished to enhance the security and reliability of the nation’s current electricity infrastructure. These include the Complex Interactive Net-works/Systems Initiative (CIN/SI), a joint program spon-sored by the Electric Power Research Institute (EPRI) and the U.S. Department of Defense (DOD); EPRI’s Enterprise Information Security (EIS) program; EPRI’s post–9/11 Infrastructure Security Initiative (ISI); and various North American Electric Reliability Corporation (NERC) initia-tives, such as its information sharing and analysis centers (ISACs), public key infrastructure (PKI), and spare equip-ment database. Information security frameworks for electric power utilities have also been developed by the International Council on Large Electric Systems (CIGRE). A security framework is considered as the skeleton on which various elements are integrated for the appropriate management of security risk. The various elements considered by CIGRE include security domains, baseline controls, and security processes.

Research and Development Needs

The Smart Infrastructure: A Smarter, More Secure I-35W BridgeWithin less than a year after the August 2007 collapse of the I-35W bridge in Minneapolis, Minnesota, a city of sorts on the south side of the former bridge took shape, complete with a host of heavy-duty equipment pieces, temporary on-site areas for casting and other tasks, and crews con-stantly at work. The days and months that followed required

extraordinary efforts from many, including alumni of the University of Minnesota’s infrastructure systems engineer-ing program. They incorporated a sensor network into the new I-35W bridge (at less than 0.5% of total cost) that pro-vides full situational awareness of stressors, fatigue, mate-rial, and chemical changes, so as to measure and understand the precursors to failure and to enable proactive and a priori corrective actions.

Analogously, customized and cost-effective advance-ments are both possible and essential to enable smarter and more secure electric power infrastructures. For example, advanced technology now under development or under con-sideration holds the promise of meeting the electricity needs of a robust digital economy. The end vision of the smart grid consists of a highly developed electrical platform that engages consumers, enhances effi ciency, ensures reliability, and enables integration of renewable energy and electric transportation.

One key money- and power-saving element of the smart grid is its ability to measure how and when consumers use the most power. This information allows consumers to be charged variable rates for energy, based upon supply and demand. This variable rate will incentivize consumers to shift their heavy use of electricity to times of the day when demand is low.

The total cost of a stronger transmission system would be about US$82 billion over the next decade. Additionally, to create a smarter end-to-end power delivery system, we must invest between US$17 and US$24 billion over the next 20 years.

Investment in a smart grid would nearly pay for itself by reducing stupendous outage costs, a savings of US$49 billion per year, and improving energy effi-ciency, a savings of US$20.4 billion per year. Likewise, through smart grid-enhanced energy efficiency, by 2030 carbon dioxide emissions from the electric sector would be reduced by 58%.

Americans should not accept or learn to cope with increasing blackouts, nor should we rest on the notion that the technical know-how, political will, or money to bring our power grid up to 21st century standards do not exist. The truth is that, as a nation, we must and absolutely can meet the power needs of a pervasively digital society if the United States wishes to maintain its role as a global economic and political leader. The best of American inno-vation is yet to come, and the smart grid must be part of our future. The potential exists to create an electricity system that provides the same effi ciency, precision, and interconnectivity as the billions of microprocessors that it will power.

From a strategic viewpoint, long-term developments and research issues relating to the defense of cyber and physical interdependent infrastructure networks must also be con-sidered. The driving scientifi c motivation is to further our understanding of adaptive self-healing and self-organizing



mechanisms that can be applied to the development of secure, resilient, and robust overlaid and integrated energy, power, sensing, communication, and control networks.

In addition to the above, further research and develop-ment needs include the following areas:

1) Enabling technologies for an end-to-end secure system of sensing and measurement, leading to im-proved analysis and visualization and eventually to automation and self-healing systems:• monitoring and analysis, automation and control,

materials science, power electronics, and integrated distributed energy resources (DERs)

• sensing, communication, data management, and mathematical and theoretical foundations to support a better, faster, and higher-confi dence understanding of what is going on, leading to improved state and topology estimation and fast look-ahead simulation.

2) Enabling a stronger and smarter grid by means of complex dynamical systems, systems science, con-trols, and applied mathematics:• modeling, robust control, dynamic interaction in in-

terdependent layered networks, disturbance propa-gation in networks, and forecasting and handling uncertainty and risk

• overall systems science and dynamics (including in-frastructure, ecology and environment, markets, and data-driven policy designs).

3) Strategic R&D:• digital control of the energy infrastructure • integrated energy, information, and communica-

tions for the end user• transformation of the meter into a secure, two-way

energy and information portal• robust advanced power generation portfolio.

Awareness, education, and pragmatic tool development in this vital area continue to remain challenges. Educating stakeholders and colleagues about the cyber and physical interdependencies has often been diffi cult, as those who are distinguished members of the community and understand power systems well but are less aware of their cybervulner-abilities routinely minimize the importance of these novel—and persistent—threats.

ConclusionCyberconnectivity has increased the complexity of the con-trol systems and facilities it is intended to safely and reliably control. In order to defend electric infrastructure against the impacts of cyber and physical attacks, signifi cant challenges must therefore be overcome before extensive deployment and implementation of smart grid technologies can begin. Cyber-security and interoperability are two of the key challenges of the smart grid transformation. As for security, it must be built in as part of its design, not glued on as afterthought.

Regarding recent cyberthreat reports, it is fundamental to separate the “hype” from the truth. What is most concerning

about such reports is mainly one portion of an early article: “The response to the alert was mixed. An audit of 30 util-ity companies that received the alert showed that only seven were in full compliance, although all of the audited com-panies had taken some precautions.” This is the reality that needs to be addressed.

Finally, no matter how many layers of security or how much sophistication is used in defense mechanisms, it is essential that the industry hire qualifi ed people. Research fi ndings sug-gest that human and organizational factors do affect computer and information security performance in a multilayered fash-ion. Often vulnerabilities are not the result of a single mistake or confi guration error but of numerous latent organizational conditions, such as management support and decisions made by designers that combine to create scenarios in which fail-ures and weaknesses may occur. In many complex networks, the human participants themselves are both the most suscep-tible to failure and the most adaptable in the management of recovery. Thus, staff members must be well trained to respond to a wide variety of emergencies since no amount of technol-ogy can replace well-trained personnel.

For Further ReadingJ. Clemente, “The security vulnerabilities of smart grid,” J. Energy Security, June 2009.

P. H. Corredor and M. E. Ruiz, “Against all odds,” IEEE Power Energy Mag., vol. 9, no. 2, pp. 59–66, Mar./Apr. 2011.

G. N. Ericsson, “Information security for electric power utilities (EPUs)-CIGRE developments on frameworks, risk assessment, and technology,” IEEE Trans. Power Delivery, vol. 24, no. 3, pp. 1174–1181, July 2009.

P. McDaniel and S. McLaughlin, “Security and privacy challenges in the smart grid,” IEEE Security Privacy, vol. 7, no. 3, pp. 75–77, May/June 2009.

M. A. McQueen and W. F. Boyer, “Deception used for cyber defense of control systems,” in Proc. 2nd Conf. Hu-man System Interactions, Catania, Italy, 2009, pp. 624–631.

NIST, “Guidelines for smart grid cyber security,” The Smart Grid Interoperability Panel—Cyber Security Work-ing Group, NISTIR 7628, Gaithersburg, MD, Aug. 2010.

S. M. Amin, “Securing the electricity grid,” Bridge, vol. 40, no. 1, pp. 13–20, Spring 2010

S. M. Amin, “Energy infrastructure defense systems,” Proc. IEEE, vol. 93, no. 5, pp. 861–875, May 2005.

S. M. Amin, “Balancing market priorities with security issues: Interconnected system operations and control under the restructured electricity enterprise,” IEEE Power Energy Mag., vol. 2, no. 4, pp. 30–38, Jul./Aug. 2004.

BiographiesS. Massoud Amin is with the University of Minnesota.

Anthony M. Giacomoni is with the University of Minnesota.

p&e



I


IT IS GENERALLY RECOGNIZED THAT A HIGH-BANDWIDTHand highly available networked communication system should overlay the transmission system topology in order to enable the control and pro-tection envisaged today to make the grid more effi cient and more reliable. The specifi cations for such a communication system have been diffi cult to develop, however, because it needs to support a great variety of ap-plications, many of which have not yet been developed. Organizations such as the North American SynchroPhasor Initiative (NASPI) are trying to build on this vision of a communication system that can utilize phasor measurement data to initiate fast controllers, including fl exible alternat-ing current transmission system (FACTS) devices.

A major hurdle in developing such fast, wide area controls has been the lack of design tools available to do so. In particular, the development

© IMAGESTATE

Digital Object Identifi er 10.1109/MPE.2011.943205 Date of publication: 13 December 2011

A Virtual Smart Grid

Real-Time Simulation for Smart Grid Control and Communications Design

By David Anderson, Chuanlin Zhao, Carl H. Hauser, Vaithianathan Venkatasubramanian,

David E. Bakken, and Anjan Bose

1540-7977/12/$31.00©2012 IEEE ieee power & energy magazine 13



of controls that depend on communications to carry the input and output signals and complex software to process these signals requires tools to simulate and analyze such controls. To accurately portray the behavior of such con-trols, design tools must integrate the dynamic behavior of the power system with the response of the communication and computation system.

We describe here a simulator—GridSim—that can simu-late in real time the electromechanical dynamic behavior of the power grid, the IT infrastructure that overlays the grid, and the control systems taking advantage of that IT infra-structure. This simulator was devised for designing and test-ing new wide area control and protection schemes. GridSim is able to represent a large portion of a grid and runs in real time so that various components running at different sam-pling rates can be tested together.

BackgroundThe use of time-synchronized, high-data-rate sensor tech-nology is widely viewed as a critical enabler for increasing the reliability of the power grid while allowing the integra-tion of many more stochastically variable renewable energy sources such as solar radiation and wind. For example, the deployment of phasor measurement units (PMUs) is becom-ing more commonplace. PMUs are capable of sampling frequency, voltage, and current thousands of times per sec-ond and outputting accurate, time-stamped measurements 30–120 or more times per second. It is diffi cult, however, for utilities to take full advantage of these devices due to a lack of tools for designing and evaluating the control sys-tems that exploit them. Furthermore, the behavior of such control systems will also depend on the performance of the wide area communications systems that connect the sen-sors, control logic, and actuators—wide area communica-tions systems whose design and specifi cations are them-selves still evolving.

Simulation is historically one the principal tools used in the design of power system controls. No existing simu-lation framework, however, can model at the scale of the power grid the combined behavior of the power system, the communications system that overlays it, and the con-trol system that relies on the latter to monitor and control the former. GridSim is intended to address these issues by providing a very fl exible simulation framework that incorporates power system simulation, data delivery, fl ex-ible sensor deployments, and the ability to incorporate actual power system components, protocols, and algo-

rithms. Using actual power system artifacts is important for two reasons. First, it allows the artifacts to be tested in the simulation environment, which is one way to increase confi dence in a design. Second, it allows existing artifacts such as the Grid Protection Alliance’s openPDC product and the GridStat communication framework to be used as building blocks for GridSim, speeding its implementation. From this decision comes another requirement: that Grid-Sim operate in real time so as to properly interface with these artifacts.

The Overall Design of GridSimGridSim is a real-time, end-to-end power grid simulation package designed using a default sample rate of 30 samples per second (per sensor). The goal of this project is to simu-late power grid operation, control, and communications at gridwide scale (e.g., the Western Interconnection) in order to give utilities the ability to explore new equipment and control system deployments. Possibilities include simulat-ing large-scale PMU installations and power applications able to utilize the vast quantities of data generated in such a situation. With the objective of providing tools to simulate real-world equipment usage and the ability to be used in con-junction with readily available utility industry equipment, GridSim uses the IEEE C37.118 data format standard for all streaming measurement data.

The GridSim platform consists of a number of compo-nents falling into four groups: power system simulation, substation simulation, communication and data delivery, and control center applications (see Figure 1). We fi rst describe the overall relationship between these groups and then look at each of them in detail.

The power system simulation calculates the electro-mechanical dynamics in real time. Sensor data from the simulated power system are fed in C37.118 format to the substation simulation processes at a rate of 30 samples per second. In the substations, data are optionally pro-cessed by substation applications and published, along with the outputs of the substation-level applications, to the data delivery component through simulated substa-tion gateways. Delivery to control center applications and other substations occurs via the data delivery system. Note the design choice here: the wide area data delivery system is not involved in connecting simulated sensors within the simulated substations where they are located. Although the substation-level processing of the data is simulated, the data communication within the substation

GridSim can represent a large portion of a grid and runs in real time so that various components running at different sampling rates can be tested together.



is assumed to be negligible for the current goals of wide area control design.

The data delivery component of GridSim is GridStat, a publish-subscribe, wide area data delivery framework designed from the ground up to meet the emerging needs of electric power grids. Once data are published, the fl exibility provided by the GridStat data delivery middleware allows subscribing applications to be easily integrated into the sys-tem without massive reconfi guration.

In the current GridSim implementation, published data are used by the two control center applications included in this project: the hierarchical state estimator and the oscilla-tion and damping monitor.

Power System SimulationPower system simulation in GridSim is provided by a mod-ifi ed version of TSAT, an industry-proven transient stabil-ity simulator produced by Powertech Labs, Inc. Unmodi-fi ed TSAT accepts power system topologies, initial values, and dynamic simulation variables (such as faults at spe-cifi c times) as inputs. On execution, the simulator loads the input values, then as quickly as possible computes the state of the system over time; on completion it writes the results to a fi le.

An off-line transient stability simulation such as TSAT does not perfectly meet the needs of GridSim. To obtain real-time performance, the simulator was modifi ed so that simulation time progresses no faster than wall-clock time. This is accomplished by pausing after computing each set of measurements (30 sets per second) until the correct wall-clock time arrives for that set to be published. To extract the measurement sets at the time they are produced by the simulation, certain TSAT functions are used. They directly implement simulated PMUs attached to particular points in the power system topology where they measure frequency, voltage, and current 30 times a second. These sensor data from the simulated PMUs are sent to the measurement gen-erator for postprocessing (see Figure 2).

Substation SimulationThe measurement generator also bridges the gap between the bus-branch power system model supported by TSAT and the more detailed bus-breaker model that represents the sub-stations. To do this, GridSim’s static data generator creates tables that map the FromBus/ToBus/EquipmentID measure-ment identifi cation information used in TSAT to the unique CircuitBreaker/BusID/PMUID numbers used throughout the rest of GridSim. Data from the static data generator also

PowertechTSAT Simulator

MeasurementGenerator

C37.118Generator

Static DataGenerator

Simulated PowerSystem

SubstationSimulation Substation

OM

SubstationSE

SubstationOM

SubSE

SubstationOM

SubSE

SubstationOM

SubSE

SubstationO

SuSE

SubstationGateway

GridStat

ControlCenterApplications

OscillationMonitor

OpenPDC

StateEstimator

Substation N

Substation 1

FE

FE

FE

FE

FE

FE

figure 1. GridSim architecture.



allow the measurement generator to synthesize additional measurements, such as breaker currents, from the TSAT outputs. Noise and other real-world attributes can be added within the measurement generator, if desired. Once these operations have been performed, the PMU measurements are sent to a C37.118 encoder and then to the substation sim-ulation processes.

The substation simulation processes host substation-level power applications and substation gateways. Power applications perform computations—both the applications described below have substation-level processing—and sub-mit results to the substation gateway. Measurement genera-tor output for each substation is also published to the data delivery component by the substation gateway.

Communication System and Data DeliveryData delivery latency and loss rate are important factors in the performance of wide area control and protection applications, but the data delivery infrastructure that will ultimately support those applications is still evolving. Grid-Sim’s data delivery component, GridStat, is a publish-subscribe middleware framework that has infl uenced the NASPInet effort led by NERC and the U.S. Depart-ment of Energy (DOE). Its design centers on the fact that sensor measurements are digitally repre-sented as a periodic stream of data points. Working from this data model, GridStat was designed to allow for effi cient, wide area, encrypted multicast delivery of data. GridStat as a component of GridSim is a realistic model for emerging power system data delivery services and at the same time provides great fl exibility for confi guring and evaluating poten-tial wide area control and protec-tion applications.

GridStat is designed to meet the requirements of emerging control and protection applica-tions that require data delivery latencies on the order of 10–20 ms over hundreds of miles with extremely high availability. The GridStat architecture consists of two communication planes: the data plane and the manage-ment plane (see Figure 3). The

data plane is a collection of forwarding engines (FEs) designed to quickly route received messages on to the next FE or termination point. The FEs are entirely dedi-cated to delivering messages from publishers to sub-scribers. Routing configuration information is delivered to the FEs from the management plane. The forwarding latency through an FE implemented in software is on the order of 100 µs, and with network processor hard-ware it is less than 10 µs. We believe that the perfor-mance of a custom hardware implementation of an FE could match or exceed that of a general-purpose Internet router. Thus, in a typical wide area configuration, Grid-Stat would not add more than 1 ms over the speed of the underlying network while providing quality-of-service

Static Data Generator GeneratesEquipment in Each Substation

Real-Time Data Generator:Acquire Data from TSAT Output

Type of Data?Generator or Load

Calculate Complex Voltage fromMagnitude and Angle. Active and

Reactive Power Are Given Insteadof Current Magnitude and Angle,Thus Calculate Complex Current

by Solving the Equationof Complex Power

Branch or Transformer

Calculate Complex Voltage andCurrent from Magnitude andAngle Given in TSAT Output

Assign Complex Voltage andCurrent as Measurement to

Respective Equipment

TSAT Output Ends?

Yes

Real-Time DataGenerator Ends

No

Assign Measurements to Circuit Breakers

figure 2. Measurement generator logic.



(QoS) guarantees tailored to rate-based control and protec-tion applications.

The management plane is a set of controllers, called QoS bro-kers, that manage the FEs of the data plane. The QoS brokers are organized in a hierarchy to refl ect the natural hierarchy in power grids. When a subscriber wishes to receive data from a publisher, it communicates with a QoS broker that designs a route for the data and delivers the routing information to the relevant FEs, creating the subscription. Since path computations are done out of band from data delivery, even heavy loads of new subscription creation do not adversely affect the performance of the data plane. Beyond this, QoS brokers have a privileged view of routing per-formance and the router graph that allows them to create opti-mal delivery paths. QoS brokers also implement policies for resource usage, cybersecurity, aggregation, and adaptation.

Because the entire purpose of GridStat is the effi cient delivery of data, it includes features providing confi gurable QoS per subscription while attempting to minimize data delivery costs. A subscriber can request quality-oriented parameters such as data delivery rate, temporal redundancy of data packets, and spatial redundancy of data streams (delivery over multiple independent delivery paths, each of which meets the end-to-end delay requirements). The QoS brokers ensure that each subscriber gets the resources it needs while preserving the needs of existing subscriptions. To con-serve network resources, the management plane identifi es any shared data paths between a publisher and two or more subscribers. If there is any overlap in these paths, the manage-ment plane ensures that data are only sent once for that leg of the journey before being duplicated at the split.

GridStat supports multicast delivery of a given sensor update stream whereby different subscribers can subscribe to different rates yet no update message is ever sent over a network link more than once and it is not forwarded on a link at all if not needed. FEs implement this via a mechanism

called rate fi ltering: only forwarding an update on an outgo-ing link at the highest rate that any subscriber downstream via that link requires. Some kinds of data place additional restrictions on the rate fi ltering. GridStat’s rate-fi ltering algorithms are coordinated across multiple PMU streams in order to ensure that subscribers receive sets of updates from different PMUs taken at the same instant. For example, con-sider PMUs that send updates at a rate of 120 Hz. While such a high rate would be useful for a few application programs, many applications would not need such frequent updates. For an application subscribing to two different PMU streams at a rate of 20 Hz, fi ve-sixths of the updates will be dropped before reaching it. But GridStat ensures that the same one-sixth of the updates are delivered from the two PMUs, so they can be used as a global snapshot. This synchronized rate fi ltering is set up when subscriptions are being added and is based on time stamps in the updates, so it does not require any inter-FE coordination when updates are being delivered. So scalability is not harmed by this strong deliv-ery property.

When used as the data delivery layer component of Grid-Sim, GridStat allows for virtual substations to be created or

figure 3. GridStat architecture.

QoS Broker

QoS Broker

Leaf QoS Broker

GridStat

FE

FE

FE

FE

FE

Pub1

Pub2

Sub1

Sub2Sub3

GridStat allows for virtual substations to be created or reconfigured and additional subscribers and power applications to be added with minimal changes.



reconfi gured and additional subscribers and power applica-tions to be added with minimal changes. This contrasts starkly with the current situation in the power grid, where even mini-mal changes to the number of sources or consumers of data can require the data delivery system to be completely re-archi-tected. Conversely, GridSim also allows for potential deploy-ments of GridStat to be tested with real-world volumes of data and with different network and power system topologies.

Control Center ApplicationsContinuing the theme of using existing artifacts as compo-nents of the GridSim environment, we now describe two control center applications that have been incorporated into GridSim thus far.

One of the main objectives of GridSim is to allow exper-imentation with and testing of wide area control and pro-tection applications using PMU and other high-rate, time-stamped data streams. Thus far, two prototype applications have been included in GridSim: a linear, hierarchical state estimator and an oscillation monitoring system.

Both applications were built using components of the Grid Protection Alliance’s openPDC product. Thus, one benefi t of incorporating these applications in GridSim is that other openPDC-based applications can easily be brought into the GridSim environment. The openPDC application

set is an open-source software system that collects PMU measurements from multiple sources, aligns them according to their time stamps, and processes them with user-defi ned functions. The openPDC applications also provide numer-ous advanced functions, such as cybersecurity and device management, that are necessary for industry use. Thus far, however, GridSim uses only the C37.118 protocol parser and the time-alignment functionality.

The openPDC applications contain three kinds of adapt-ers: input adapters, action adapters, and output adapters. GridSim’s applications, however, use only two of these. Input adapters read data and parse them. Although the open-PDC applications provide many built-in input adapters that can read data from fi les, databases, or the network, none of them supports the publish-subscribe communication pattern used in GridSim. New input adapters were therefore devel-oped supporting the GridStat publish-subscribe system. Action adapters receive time-aligned measurements and process them. In GridSim, all of the power system calcula-tions, including substation-level and control center–level state estimation as well as oscillation detection, are implemented using custom action adapters. These new functions embed-ded in the openPDC applications are not only useful in the simulation environment but can also be run in the real indus-try environment.

Since the openPDC applications were primarily designed and implemented for fi eld usage, which has different tech-nical requirements from GridSim, work was performed to adapt them for the simulation environment. For example, the openPDC applications provide a user interface for confi gur-ing devices, phasors, and measurements. Since GridSim is intended to simulate a variety of systems that may change frequently, manual confi guration is too cumbersome and error-prone. A program was therefore created to read the power fl ow fi le for TSAT and confi gure the whole system automatically, saving a lot of effort and simplifying the inte-gration of the openPDC and simulation software.

The Oscillation Monitoring SystemThe oscillation monitoring system (OMS) application has been developed at Washington State University for real-time monitoring of problematic electromechanical oscilla-tions using wide area PMU measurements. OMS combines advanced signal-processing algorithms with heuristic expert system rules to automatically extract the damping ratio, fre-quency, and mode shape of poorly damped electromechanical oscillations in a power system from power system measure-ments. A prototype OMS has been implemented as part of the phasor data concentrator at Tennessee Valley Authority (TVA) since 2007. It is also currently being implemented at Entergy in conjunction with a smart grid investment grant project.

In our GridSim project, the OMS is being used as a real-time application example, both serving to illuminate what GridSim must provide in order to incorporate actual applica-tions and demonstrating how executing an application with

FDD Analysisfor Ambient

Data

Event?

Prony Analysis forPostdisturbance

Data

Start

Read Datafrom PDC

Moving WindowCross-Check

Yes

No

Alarm

ControllerTrigger

PoorlyDamped Mode

Detected?

Yes

No

EventAnalysisEngine

DampingMonitorEngine

Moving WindowCross-Check

figure 4. Flowchart of an oscillation monitoring system.



simulated real-time test data can help validate the application. The OMS engines are integrated into an action adapter module of the openPDC applications. Thus, the OMS receives real-time simulated PMU data streams from TSAT, via the measurement generator and the data delivery system, which are buffered onto the internal signal-processing engines of the OMS. Results from the OMS can be exported to a custom SQL database that can be visualized and set to trigger alerts or alarms whenever damping levels of oscillatory modes fall below prespecifi ed thresholds. The operator can then take manual action to bring the damping back to acceptable levels.

Unlike the real power system, where the actual modal characteristics of the system are unknown values, the modal properties of the test system in TSAT can be accu-rately determined from model-based small-signal stability analysis. Comparing the outputs of the OMS engines with the respective model-based modal values is useful for test-ing and tuning the OMS engines for target power systems. Since GridSim includes communication models, such stud-ies also reveal the effects of communication delays, the loss of PMU channels, and network congestion on the resulting OMS modal estimates. We plan to use GridSim to test auto-matic control action by the OMS, although such closed-loop feedback will require further modifi cation of TSAT.

The OMS includes two engines, as shown in the fl ow chart in Figure 4. The event analysis engine, shown on the right side of the fl ow chart, carries out an expert system–based Prony-type ringdown analysis of system responses following disturbances in the system. The objective for this engine is fast detection of sudden changes in the damping of oscillatory modes from large disturbances in a power sys-tem, so that mitigating control actions can be initiated before the damping problems degenerate into widespread black-outs. Typical analysis uses 5–10 s of PMU data at a time, and the calculations are repeated over moving time windows and over different PMU signal groups to ensure the con-sistency of results. The event monitor engine can typically detect oscillatory problems by using 10–15 s of PMU data, starting from the instant the oscillations begin to appear in a power system.

The complementary damping monitor engine, shown on the left side of the fl ow chart, estimates the damping, fre-quency, and mode shape of poorly damped oscillatory modes from ambient PMU measurements. Unlike the event moni-tor engine, which only works when the system is subject to disturbances, the damping monitor engine is applicable all the time. By using natural power system responses to rou-tine random fl uctuations from load variations and generation changes, the damping monitor engine continuously tracks damping levels and mode shapes of poorly damped oscilla-tory modes. The damping monitor engine uses an extension of a frequency-domain algorithm called frequency domain decomposition (FDD). This engine is aimed at preventive detection of poorly damped oscillations. The damping moni-tor engine uses about four minutes’ worth of PMU data in

every computational run. As with the event monitor engine, the analysis is then repeated over moving time windows and over different signal groups to verify the consistency of modal analysis results.

Figure 5 shows the results from the two engines for a recent event near a major generating plant. The system encountered a routine event at about 830 seconds. The event analysis engine of the OMS carried out moving time-window analysis of the PMU measurements using real-time Prony analysis and concluded at 838 seconds (the vertical dotted line in Figure 5) that the oscillation was from a local 1.2-Hz

figure 5. Illustration of analysis results from OMS engines.

5.15

5.14

5.13

5.12

5.11

5.1

5.09

820 840 860 880Time (s)

Ambient Noise Analysis1.2 Hz at + 1.8% Damping. Local Mode.

900 920 940

EventAnalysis

1.2 Hz at + 1.5% Damping. Local Mode.

AmbAAAAmA bmbbmAmbbbAAAAAmbmbAmbbAAAmAmbbbAAA bbAmbAmAAmmmbmmmmmm ient Noise AAAAAAAAAAnalysis1.2 Hz at + 1.8% Damping. Local Mode.

EEEEEveEEEEE ntvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvAnalysisnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

111.2 Hz at + 1.5% Damping. Local Mode.22222222222222222222222222222222222222222222222

Control Center

TopologyProcessor

StateEstimator

Real-TimeDatabase

StaticDatabase

AnalogMeasurements

DigitalStatus

CB/NDConnections ND/Equipment

Connections

EquipmentParameters

SystemTopology

SubstationRTU

SubstationRTU

SubstationRTU

SCADA

Maintenance

figure 6. The two-level linear state estimator.



mode (i.e., one involving mainly one PMU or a few nearby PMUs) with a damping ratio of +1.5%. Subsequently, the damping monitor engine analyzed the real-time ambient PMU data and estimated the dominant oscillatory mode to be the same local mode at 1.2 Hz, with a damping ratio of +1.8%. Thus the results of ringdown analysis and ambient noise analysis match well for this example. The two engines serve as complementary techniques for identifying the domi-nant poorly damped oscillatory modes of a power system whenever such modes exist.

State EstimatorA two-level linear state estimator has been developed at Washington State University that is an excellent candidate

application for testing in the GridSim environment. It is based on PMU data and requires algorithmic processing at the sub-station level, fast communication of the substation results to the control center, and synchronization of the data at the control center before it fi nally calculates a state estimate (SE) for the whole system. The power system simulation produces PMU measurements 30 times per second, and the fi nal SE is also calculated at the same rate. Thus errors in the simulation, communication, synchronization, and SE calculation can all be checked during the testing of this application on GridSim.

The processing of this two-level SE is shown in Figure 6 for both the substation level and the control center level. At each substation, the local PMU data are processed using linear estimation algorithms for both current and voltage

1.21.00.80.60.40.20.0

Val

ue (

p.u)

1.21.00.80.60.40.20.0

Val

ue (

p.u)

1.21.00.80.60.40.20.0

Val

ue (

p.u)

1.21.00.80.60.40.20.0

Val

ue (

p.u)

0 1 2 3 4 5 6Time (s)

7 8 9 10 11

0 1 2 3 4 5 6Time (s)

7 8 9 10 11

0 1 2 3 4 5 6Time (s)

7 8 9 10 11

0 1 2 3 4 5 6Time (s)

(d)

(c)

(b)

(a)

Control Center Bus Voltage

Substation Bus Voltage

Generated Bus Voltage

TSAT Bus Voltage

7 8 9 10 11

Bus 1 Bus 2 Bus 3 Bus 4 Bus 5Bus 7 Bus 8 Bus 9 Bus 10 Bus 11

Bus 6


Bus 410


Bus 32


Bus 32

figure 7. GridSim results for an 11-substation system using the two-level linear state estimator.



phasor measurements. This pro-cessing has the advantage of esti-mating and eliminating errors from noise, bad analog data, and bad circuit breaker status data on a small set of measurements. The topology, current, and voltage esti-mates from each substation are then sent through the communica-tion network to the control center. At the control center, the data are synchronized for the same time stamp, and the whole system states are linearly estimated.

Figure 7 provides some results for this test as carried out on Grid-Sim for an 11-substation power system. For a small system like this, the simulation and communication speeds were not a problem, so the test’s purpose was mainly to check the computation processes and data delivery. When the SE was running perfectly, the fi gure shows that the bus voltages (a) calculated by the TSAT simulation, (b) generated by the PMU data generator, (c) estimated at the substation level, and (d) estimated at the control center all compare quite well 30 times a second for about eight seconds after a fault on the sys-tem. Many things can go wrong, however, as demonstrated in Figure 8 by introducing some jitter in the data delivery between the substation and the communication level, thus producing erroneous SE results at the control center.

ConclusionsA fast communication and computation system overlay-ing the power grid is a key enabler for applications taking advantage of PMUs and FACTS controllers to achieve the smart grid of the future. The tools needed to develop and test these new applications do not exist today, however. We have described such a tool—a simulation platform called Grid-Sim—that can be used to develop and test wide area control and protection schemes.

We have developed this platform to simulate the power grid in real time for electromechanical dynamics and to generate and stream PMU data in standard format. It also includes the ability to deliver measurements and processed data over a high-bandwidth networked communication sys-tem called GridStat. Finally, we have used GridSim to simu-late and test two new applications—oscillation monitoring and linear state estimation—that are quite different from each other but both utilize PMU streaming data in real time. We show that platforms such as GridSim can successfully and rapidly prototype new “smart” applications.

We should note that closed-loop control is not illustrated in this article. Both the OSM and the linear state estimator are real-time but open-loop applications, which means that the outputs are used by the operator to initiate manual control if necessary. Closed-loop control will be incorporated into

GridSim, and the signifi cant changes needed in the power system simulator to accomplish this are being developed.

AcknowledgmentsWe gratefully acknowledge the assistance of Powertech Labs and the Grid Protection Alliance in adapting their TSAT and openPDC products, respectively, for use in GridSim. This research was supported by a grant from the U.S. Department of Energy (Award #DE-OE0000032).

For Further Reading[Online]. Power Tech Labs. TSAT—Transient Security As-sessment Tool. 2011. Available: http://www.powertechlabs.com/software-modeling/dynamic-security-assessment- software/transient-security-assessment-tool

D. Bakken, A. Bose, C. Hauser, D. Whitehead, and G. Zweigle, “Smart generation and transmission with coherent, real-time data,” Proc. IEEE (Special Issue on Smart Grids), vol. 99, no. 6, pp. 928–951, June 2011.

[Online]. Grid Protection Alliance. The Open Source Phasor Data Concentrator. 2011. Available: http://openpdc.codeplex.com

G. Liu, V. M. Venkatasubramanian, and J. R. Carroll, “Oscillation monitoring system using synchrophasors,” in Proc. IEEE PES General Meeting, Calgary, Canada, July 2009, pp. 1–4.

T. Yang, H. B. Sun, and A. Bose, “Transition to a two-level linear state estimator, part I: Architecture, part II: Al-gorithm,” IEEE Trans. Power Syst., vol. 26, no. 1, pp. 46–62, Feb. 2011.

BiographiesDavid Anderson is with Washington State University.

Chuanlin Zhao is with Washington State University. Carl H. Hauser is with Washington State University. Vaithianathan Venkatasubramanian is with Washing-

ton State University. David E. Bakken is with Washington State University. Anjan Bose is with Washington State University. p&e

1.2

1.0

0.8

0.6

0.4

0.2

0.00 1 2 3 4 5

Time (s)V

olta

ge (

p.u)

6 7 8 9 10 11

SE Bus Voltage Curve 2

Bus 11 Bus 12 Bus 71 Bus 22Bus 81 Bus 42 Bus 41Bus 32 Bus 31 Bus 21

Bus 91

figure 8. State estimator results with jitter in the communication system.


Policy Challenges and Technical Opportunities on the U.S. Electric Grid

By Timothy D. Heidel, John G. Kassakian, and Richard Schmalensee

may/june 2012

PPUBLIC POLICIES AT BOTH THE STATE AND FED ERAL LEVELS in the United States and a variety of technological and economic changes are poised to signifi cantly alter both the demand for and supply of electric-ity in the country over the next several decades. These changes will yield a wide range of new challenges and opportunities, including incorporating variable energy sources like wind and solar radiation; adjusting distribu-tion systems to accommodate small-scale, distributed generators; accom-modating the charging of electric vehicles and other changes in electricity demand; making the best use of new technologies to ensure reliability and effi ciency under changing conditions; responding to threats presented by the vast increase of data communications within the grid; and meeting chang-ing workforce needs.

A variety of technologies exists today that can help meet the emerging challenges effectively in the United States. In a recently completed two-year study on the future of the U.S. electric grid that we performed with a dozen other economists and engineers, however, we found that the promise of these new technologies will only be fully realized if a number of regulatory poli-cies are changed, if necessary research and development is performed, and if important data are compiled and shared. Maintaining system reliability, keeping electricity rates at acceptable levels, and achieving state and federal policy goals will depend to a large degree on a few key choices made—or not made—at the state and federal levels and within the industry over the next few years.

In this article, we fi rst discuss the performance of the U.S. grid today. Then, we describe several of the most important challenges and opportuni-ties that are likely to face the U.S. grid over the next several decades.

The U.S. Grid TodayPhysically, the U.S. electric grid consists of approximately 170,000 mi of high-voltage electric transmission lines (i.e., lines rated at >200 kV) and associated equipment and nearly 6 million mi of lower-voltage distribu-tion lines. In aggregate, the U.S. grid serves about 125 million residen-tial customers, 17.6 million commercial customers, and 775,000 industrial customers that account for 37%, 36%, and 27% of electricity use, respec-tively. At the highest level, the electric power system of the continental United States consists mainly of three independently synchronized grids: the Eastern Interconnection, the Western Interconnection, and the Electric Reliability Council of Texas (ERCOT). The three grids are linked by only a few low-capacity dc lines. Within these broad areas are 107 balancing authorities, responsible for balancing the supply and demand for power in specifi ed zones.

©A

RT

VIL

LE, L

LC.

Digital Object Identifi er 10.1109/MPE.2012.2188669 Date of publication: 19 April 2012

1540-7977/12/$31.00©2012 IEEE30 IEEE power & energy magazine may/june 201222 ieee power & energy magazine

Reprinted from May/June 2012 issue of IEEE Power & Energy magazine

Policy Challenges and Technical Opportunities on the U.S. Electric Grid

By Timothy D. Heidel, John G. Kassakian, and Richard Schmalensee

may/june 2012

PPUBLIC POLICIES AT BOTH THE STATE AND FED ERAL LEVELS in the United States and a variety of technological and economic changes are poised to signifi cantly alter both the demand for and supply of electric-ity in the country over the next several decades. These changes will yield a wide range of new challenges and opportunities, including incorporating variable energy sources like wind and solar radiation; adjusting distribu-tion systems to accommodate small-scale, distributed generators; accom-modating the charging of electric vehicles and other changes in electricity demand; making the best use of new technologies to ensure reliability and effi ciency under changing conditions; responding to threats presented by the vast increase of data communications within the grid; and meeting chang-ing workforce needs.

A variety of technologies exists today that can help meet the emerging challenges effectively in the United States. In a recently completed two-year study on the future of the U.S. electric grid that we performed with a dozen other economists and engineers, however, we found that the promise of these new technologies will only be fully realized if a number of regulatory poli-cies are changed, if necessary research and development is performed, and if important data are compiled and shared. Maintaining system reliability, keeping electricity rates at acceptable levels, and achieving state and federal policy goals will depend to a large degree on a few key choices made—or not made—at the state and federal levels and within the industry over the next few years.

In this article, we fi rst discuss the performance of the U.S. grid today. Then, we describe several of the most important challenges and opportuni-ties that are likely to face the U.S. grid over the next several decades.

The U.S. Grid TodayPhysically, the U.S. electric grid consists of approximately 170,000 mi of high-voltage electric transmission lines (i.e., lines rated at >200 kV) and associated equipment and nearly 6 million mi of lower-voltage distribu-tion lines. In aggregate, the U.S. grid serves about 125 million residen-tial customers, 17.6 million commercial customers, and 775,000 industrial customers that account for 37%, 36%, and 27% of electricity use, respec-tively. At the highest level, the electric power system of the continental United States consists mainly of three independently synchronized grids: the Eastern Interconnection, the Western Interconnection, and the Electric Reliability Council of Texas (ERCOT). The three grids are linked by only a few low-capacity dc lines. Within these broad areas are 107 balancing authorities, responsible for balancing the supply and demand for power in specifi ed zones.

©A

RT

VIL

LE, L

LC.

Digital Object Identifi er 10.1109/MPE.2012.2188669 Date of publication: 19 April 2012

1540-7977/12/$31.00©2012 IEEE30 IEEE power & energy magazine may/june 2012 ieee power & energy magazine 23

Reprinted from May/June 2012 issue of IEEE Power & Energy magazine

32 IEEE power & energy magazine may/june 2012

As a result of the layering of historical policy decisions and the lack of a comprehensive, shared vision of system structure or function, the U.S. electric power system today operates under a fragmented and often inconsistent policy regime. For instance, organized wholesale markets for power play a central role in some areas, but in others the traditional vertically integrated utility model remains domi-nant. Generation facilities are variously owned by investor-owned utilities, rural cooperatives, municipal utilities, fed-eral government entities, and independent power producers. Subsidies of various sorts for public and cooperative entities are important in some regions but not at all in others. Trans-mission and distribution voltage levels also vary regionally. Several hundred entities currently own parts of the trans-mission or bulk power system and, at the distribution level, approximately 3,200 organizations provide electricity to retail customers.

Assessing the performance of a system as complex as the U.S. electric grid is not a simple task. International com-parisons and even comparisons within the United States are diffi cult because of differing geography, rates of growth, and defi nitions of performance measures. Systems that have grown more rapidly in recent years, for instance, will on average have newer equipment. Comparisons over time may reveal nothing more than the advance of technology driven by vendor R&D. Moreover, because there are diminishing returns to investing to increase effi ciency and reliability and because perfection is unattainable at any cost, it is possible not just to underinvest but also to overinvest in these and other dimensions of performance.

An important measure of the performance of a transmis-sion and distribution system is the fraction of energy gener-ated that is lost due to heating of transmission and distribu-tion lines and of other components. That fraction has fallen signifi cantly over time in the United States. As Figure 1 shows, losses in transmission and distribution decreased from more than 16% in the late 1920s to less than 7% today. This refl ects investments in transmission and distribution systems, the development and deployment of more effi -cient transformers and other equipment, and transmission at higher voltages.

Reliability is also an important dimension of perfor-mance. Increases in transmission voltage and many other, less visible, technological advances have contributed to improved reliability over time. Protective relaying enabled the detection and isolation of system faults, for instance, and high-speed reclosing circuit breakers and relaying allowed transmission lines to be reenergized after a fault automati-cally and in only a few seconds. Lightning arrestors allowed the effects of lightning strikes to be contained automatically.

At the bulk power level, data on major disturbances and unusual occurrences have been reported to the U.S. Depart-ment of Energy (DOE) since the 1970s and to the North American Electric Reliability Corporation (NERC), which has responsibility for the reliability of the bulk power system, since 1984. These data are not consistent, complete, or nec-essarily accurate, however, and they cannot reliably be used to assess changes in the reliability of the bulk power system over time. Most outages in the United States occur within distribution systems, but a recent study from Lawrence

Berkeley National Lab found that only 35 U.S. states require utili-ties to report data on the impact of all outages on consumers, and reporting standards and practices differ. It is accordingly impossible to make comprehensive compari-sons across space or over time. In particular, the treatment of very short interruptions varies among U.S. states and among different countries, so outage counts can-not be usefully compared.

Nonetheless, the data that are available suggest that U.S. reli-ability is on a par with that of other industrialized countries. According to a recent study from the Electric Power Research Institute (EPRI), U.S. customers can expect to experience between one and a half and two power interruptions and between two and eight hours without power each year. This is on a par with

figure 1. U.S. transmission and distribution losses, 1926–2009. Losses are measured as the difference between energy generated and energy delivered to customers and thus in practice include losses due to theft. Theft is not considered to be important in the United States today, but it is significant in some other nations.

18

16

14

12

10

T &

D L

osse

s (%

of T

otal

Gen

erat

ion)

8

6

4

2

0

1926

1929

1932

1935

1938

1941

1944

1947

1950

1953

1956

1959

1962

1965

1968

1971

1974

1977

1980

1983

1986

1989

1992

1995

1998

2001

2004

2007

Year


may/june 2012 IEEE power & energy magazine 33

most European countries, where customers generally expe-rience from less than one interruption per year to almost three, based on data from the Council of European Energy Regulators. Of course, there is great variation in reliability between urban and rural areas. Such comparisons cannot reveal whether U.S. reliability is too low, too high, or opti-mal, given the benefi ts of reducing outages and the costs of doing so.

A fi nal dimension of performance involves the use of new technology to increase productivity. The U.S. electric utility industry has historically devoted a very small fraction of its revenues to R&D, instead relying primarily on its suppliers for innovation. U.S. utilities have sometimes collaborated with vendors on R&D activities and have participated in col-laborative research through EPRI. In recent years, however, utilities have shifted away from longer-term, collaborative projects and toward shorter-term proprietary efforts. More-over, investor-owned utilities, which account for almost all nonfederal utility R&D spending, reduced their R&D bud-gets beginning in the 1990s, spending on average less than 1% of their revenues on R&D. The decrease in utility R&D funding refl ects, in part, reluctance among utilities to incur (and regulators to approve) R&D expenditures as U.S. fed-eral and state policies pursued more industry competition during those years.

New Challenges and Opportunities

Grid-Scale Variable GenerationOwing to strong federal and state policy support, wind and solar generation are almost certain to become more impor-tant in the United States over the next several decades—though perhaps not as important in many U.S. regions as they already are in some European Union countries. Effi -ciently increasing the penetration of grid-scale renewable generation while maintaining reliability will require modifi -cations to power system planning and operation.

At high penetration levels, the variable and imper-fectly predictable power output of these “variable energy resources,” or VERs, causes the demand minus VER gen-eration—that is, the net load that must be met by other gen-erators—to become noticeably more variable and diffi cult to predict. To maintain reliability despite this variability, the system and its operation must be modifi ed at some cost. Wind and solar forecasts will have to be fully integrated into system operations and planning. Indeed, utilities and system operators in many U.S. regions are already actively working on improving their forecasting capabilities. Power system fl exibility will also become more important, and incentives for investments that add generation fl exibility or for operat-ing generation resources in a fl exible manner may be needed in regions with organized markets. Full or virtual consolida-tion of small balancing areas would facilitate VER integra-tion, as would requiring new VER generators to meet perfor-

mance specifi cations, such as low-voltage ride-through and inertial response, appropriate for operation in the high-VER future they are likely to encounter.

In the United States, many of the most attractive wind resources are located in the “wind belt” that stretches north from Texas through the Dakotas to the Canadian border and offshore on both coasts. While the offshore resources are closer to major load centers, the costs of offshore wind installations are generally considerably greater than those for onshore facilities in good locations. Similarly, the prime locations for solar power are in the nearly cloud-free and sparsely populated desert Southwest.

Exploiting these resources will require building more transmission than if fossil-fueled or nuclear generating plants built relatively close to load centers were supporting system expansion. The use of very long transmission lines can cause technical issues and compromise system stability; such issues will have to be monitored carefully in the years to come. In addition, adequate planning tools that can deal with complex networks and that take uncertainty rigorously into account do not exist today, and research to develop them is needed. For such research to be most productive, detailed data covering the major interconnections—data that are now deemed proprietary—must be made appropriately available to researchers.

As VER penetration increases, more of the new trans-mission lines will cross state borders or the 30% of U.S. land managed by federal agencies. Cost allocation and sit-ing have been particularly contentious for these transmis-sion facilities. When boundary-crossing lines are proposed today, they tend to be evaluated in isolation rather than as part of a wide-area planning process, and allocation of the costs involved is often done via facilities-specifi c nego-tiations. Under current law, the siting of all transmission lines is a matter for the states rather than FERC. Lines that cross land managed by federal agencies also need the approval of those agencies. Consequently, the construction of interstate transmission facilities requires the consent of multiple state regulators and, sometimes, of one or more federal agencies. FERC Order No. 1000, issued in July 2011, should signifi cantly increase wide-area planning of transmission systems, make routine the allocation of the costs of boundary-crossing transmission facilities, and, by explicitly adopting the “benefi ciaries pay” principle, rationalize the allocation of those costs. Establishing per-manent and collaborative planning processes at the inter-connection level and a single cost allocation procedure for boundary-crossing projects in each interconnection would further enhance the ability of the United States to effi ciently and reliably achieve its renewable energy goals. Expanding FERC’s authority for siting boundary-crossing transmission facilities would help facilitate transmission expansion for the integration of VERs. Even with this change, however, siting transmission facilities will remain a diffi cult challenge to manage.



Distributed GenerationPolicies at both the U.S. state and federal levels favor distrib-uted generation from low-carbon sources, and these policies seem likely to continue. At the federal level, personal and corporate tax incentives encourage distributed generators. Most states have programs that subsidize distributed gen-eration. The DOE has also established the goal that all new commercial and industrial construction should be energy-neutral by 2030. That is, such buildings must generate as much energy as they use. Furthermore, “net-metering” pro-grams in 46 states and the District of Columbia compensate end users for generating their own energy at the retail elec-tricity rate rather than the wholesale cost of energy. Cus-tomers who generate electricity on-site in these programs save both the energy charge, or the wholesale cost of energy, and the distribution charge for that electricity. The utility, however, saves only the corresponding energy cost. In this way, recovering network costs through per-kWh charges provides an additional subsidy to distributed generation that can encourage its uneconomic penetration.

At low levels of penetration, distributed generation sim-ply reduces the load at individual substations. At high levels of penetration, however, distributed generation can exceed load at the substation level, causing unusual distribution fl ow patterns. In some cases, power many even fl ow from the substation into the transmission grid. Many distribution systems are currently not designed to handle such reverse fl ows, however, and customer power quality can sometimes suffer. High levels of penetration can also add to the stress on electrical equipment, such as circuit breakers, and com-plicate the operation of the distribution system, particularly during emergencies. Additional monitoring and new sys-tems for the operation, protection, and control of distribu-tion systems will be necessary if U.S. distributed generation penetrations grow signifi cantly. And since much of this dis-tributed generation will be in the form of VERs, there will be an impact on the control of central generating resources. Enabling such penetration in a cost-effective manner will

require investment by the distribution utility. Current regu-latory frameworks may not provide adequate incentives for such investments, however, as growth in distributed genera-tion will often reduce utilities’ sales and profi ts. Transition-ing away from recovering the largely fi xed costs associated with transmission and distribution networks through volu-metric (US$/kWh) charges would help alleviate this incen-tive misalignment and could have a signifi cant impact on the growth of U.S. distributed generation.

Changes in Electricity DemandUnlike some other regions of the world, electricity demand growth in the United States is not likely to emerge as an important source of disruption in the next few decades. Based on data from the U.S. Energy Information Admin-istration (EIA), between 1949 and 1973 U.S. electricity use grew at an average annual rate of 8.3%. The system was able to meet that demand growth with only sporadic diffi culty. With rising prices after 1973, electricity use grew at an aver-age annual rate of 2.5% between 1973 and 2006. In contrast, EIA’s most recent reference case projection is for growth to average only about 0.9% per year between 2010 and 2030.

U.S. electricity demand has changed, however, and is likely to continue to change in ways that pose challenges to the system. Over the past several decades, due in part to the increased penetration of air conditioning and the rela-tive decline of industrial loads, there has been a substan-tial increase in the ratio of system peak loads to average loads. Because power systems need to be sized to meet peak demand with a margin for reliability, the peakier demand becomes (all else being equal), the lower capacity utilization becomes, and thus the higher rates must be raised to cover all costs.

Figure 2 illustrates this change, showing load dura-tion curves for New England and New York expressed as percentages of peak hour demand. The fi gure shows, for instance, that in the 1980–84 period in both New York and New England, demand exceeded 80% of its peak for only about 1,000 hours—about 11.4% of the time. By the 2005–09 time frame, demand in both New York and New England exceeded only 70% of its peak for about 1,000 hours, so that more than 30% of capacity was in use less than 12% of the time. This trend raises average costs because of the need to pay for capital that is idle most of the time and, by increasing capacity requirements,worsens the problem of siting genera-tion plants and transmission lines.

Electric vehicles (EVs)—including plug-in hybrids and pure electric vehicles—could exacerbate these trends. Although their penetration is generally projected to be slow at the national level, EVs are expected to achieve high levels of penetration quickly in some high-income areas with envi-ronmentally conscious consumers. If EVs are charged when commuters return home, as seems most likely under current policies, they could add signifi cantly to system peak loads, worsening the problem of increasing peakiness of demand.

figure 2. Normalized load duration curves for New England and New York.

10.90.8

0.70.60.50.40.30.20.1

01 2,001 4,001 6,001 8,001

Hours of Year

Nor

mal

ized

Loa

d

New England Average 1980–1984New England Average 2005–2009NY Average 1980–1984NY Average 2005–2009



On the other hand, measures that encourage overnight charg-ing could increase demand when it would otherwise be low, thus tending to fl atten load duration curves.

Making other loads similarly responsive to system con-ditions could also shift demand off-peak, helping to slow the trend depicted in Figure 2. Dynamic pricing—in which retail prices vary over short time intervals to refl ect changes in the actual cost of providing electricity—could induce such responses. U.S. demand-response programs have grown sub-stantially in recent years. Most demand-response programs in place today, however, use other approaches and focus on responding to occasional emergencies rather than system-atic load leveling. In some regions with organized wholesale markets that include a capacity market, demand response has been allowed to bid in as a proxy for capacity, illustrating its potential value to the economic effi ciency of the system.

A variety of new and emerging technologies, including advanced metering systems, can receive price information based on the real-time cost of providing electricity and can transmit usage information every few minutes. This makes it possible to provide real-time incentives to reduce system peaks caused by central air-conditioning, vehicle charging, and other loads, resulting in more effi cient use of grid assets and thus lower rates. Many large commercial and industrial customers already operate under dynamic pricing. Such pricing regimes likely will also be widespread options—if not the default—for residential U.S. consumers by 2030.

Existing studies suggest that regulators can achieve sub-stantial load shifting—and perhaps overall demand reduc-tion—when dynamic pricing is combined with the use of technology to automate responses to price changes. Residen-tial dynamic pricing also requires substantial investment in advanced metering infrastructure (AMI) to measure usage over short time intervals. Substantial AMI investments have recently been funded through the American Recovery and Reinvestment Act of 2009 (ARRA), and some state regula-tors have mandated universal AMI deployment. But thus far, there has been little if any movement toward the dynamic pricing regimes that AMI enables. As long as the results are shared widely, ARRA-supported and regulator-mandated investments in AMI will provide important learning oppor-tunities to develop effi cient paths to universal dynamic pric-ing. Where wholesale electricity markets exist, effective competition in the retail sales of electricity might stimulate innovation in ways that make dynamic pricing both accept-able to consumers and regulators and effective in modify-ing demand. Response automation technologies are not yet

mature, however, and further research into the behavior of U.S. residential consumers faced with dynamic pricing is needed. In this later regard, although there have been many dynamic pricing pilot programs, few have been structured so as to produce reliable data, and results have been highly variable.

Innovative Technologies for Increased Reliability and EfficiencyInnovative technologies that can improve system perfor-mance, offering enhanced reliability, increased capac-ity, and the ability to better accommodate new resources (VERs, EVs, and so on) are poised for signifi cant growth in the United States. The integration of such technologies into comprehensive networks of sensors, communications infrastructure, control equipment, and intelligent manage-ment systems will be a major focus of the U.S. electric power industry over the next several decades. These technology opportunities, often referred to as the “smart grid,” are likely to provide signifi cant benefi ts.

In the transmission system, phasor measurement units (PMUs) are powerful devices that provide rich streams of frequent, time-stamped data on transmission system con-ditions. PMUs with appropriate analysis tools that turn the measured data into actionable information could allow system operators to anticipate contingencies, reduce the risk of wide-area blackouts, enhance system effi ciency, and improve system models. While PMU hardware exists and is currently being installed more widely in the United States as a result of ARRA funding, the software and analysis tools necessary to fully capitalize on this investment are yet to be developed and deployed. More widespread PMU data shar-ing among utilities, system operators, and researchers will also be essential for the development and effective use of these tools.

In addition to PMUs, fl exible alternating current trans-mission system (FACTS) devices based on advances in power electronics will provide greater control of voltages and power fl ows throughout the bulk power system and could allow more power to be transmitted on existing lines with-out increasing the risk of failure. Historically, deployment of the most versatile FACTS devices has been limited by their relatively high cost. Costs are falling, however, and higher penetration of VERs is likely to increase the value of deploy-ing these technologies within the U.S. transmission system. Furthermore, integrating FACTS devices with PMUs and emerging wide-area measurement systems will allow their

Due to a variety of recent policy changes and technical innovations, the U.S. electric grid will encounter significant opportunities and challenges over the next several decades.



control capabilities to be leveraged so as to provide even greater benefi ts. Ongoing research into new system control algorithms, software, and communication systems that fully utilize PMUs, FACTS devices, and other new transmission system technologies is likely to create a high payoff and could accelerate the deployment of these technologies.

Many technologies are also available to enhance the reli-ability and effi ciency of distribution systems. Coping effi -ciently with the integration of distributed generation, elec-tric vehicles, and demand response will require signifi cant investments in new and emerging technologies, including distribution management software systems, equipment that is capable of more accurately monitoring and controlling voltages, automatic reconfi guration of distribution circuits, and advanced metering.

The benefi ts of deploying these technologies are less well known and may be more diffi cult to quantify relative to most recent investments in distribution systems; they will aim to provide new capabilities, not just expand capacity. To reduce perceived uncertainties and make possible better system-spe-cifi c decisions, it is critical that detailed information about the results of technology pilots and early deployments is shared as widely as possible. The DOE has recently funded a variety of smart grid demonstrations and technology pilots. These projects provide an important opportunity for learning.

The electric utility industry has traditionally relied pri-marily on its suppliers for the innovation that has driven its productivity growth. Supplier R&D has naturally focused on equipment that can be sold to utilities. Therefore, although research in the several non-equipment-related research areas mentioned above is likely to bring substantial payoffs, these are unlikely to attract equipment vendors. The electric utility industry itself should be able to support the efforts required, however, even if federal support does not materialize. For this to happen, regulators will need to recognize that techni-cal progress benefi ts consumers broadly and to permit mod-

est increases in utility R&D budgets. It will also likely be necessary for the industry to reverse the downward trend in cooperative R&D spending and make appropriate use of cooperative funding through EPRI, one or more independent system operators, and project-specifi c coalitions.

Finally, utilities and regulators in the United States and elsewhere have historically tended to avoid investments in unfamiliar technologies perceived to have uncertain payoffs. The tendency of traditional regulatory systems to encourage excessively conservative behavior is likely to become more and more expensive over time if the increasingly attractive opportunities to enhance effi ciency and reduce cost through the deployment of unfamiliar technology are not exploited. Regulatory innovations are necessary in the United States to provide adequate incentives for investments in unfamiliar technologies while also ensuring that the returns on these investments are shared appropriately with ratepayers. This is an important problem—but one without an obvious solution.

Data Communications, Cybersecurity, and Information Privacy ChallengesThe increasing use of communications systems, sensing and control equipment, AMI, and distribution automation technologies will enhance reliability and effi ciency but will also give rise to new challenges. As the U.S. grid evolves, increasing amounts of data will be exchanged among meters, other sensors, and various computers and control facilities through complex communications systems. The National Institute of Standards and Technology (NIST), with sub-stantial industry input, is overseeing the critical process of developing the interoperability standards that are needed to ensure these systems are compatible not only with each other but also with future generations of technology. In addition, there are ongoing debates in the United States about the use of spectrum and the roles of public and private networks.

Since no communications system can be completely free from errors, the future grid must be designed to mitigate the consequences of data errors. More chilling is the pos-sibility of deliberate sabotage via computers and data com-munications, the sort of cyberattacks that other industries have experienced. The existence of more communications nodes and channels facilitates the insertion of malicious data into the system; in addition, a greater reliance on automated responses to system conditions that may be misreported can make it more diffi cult to prevent serious damage.

As illustrated in Figure 3, cybersecurity involves more than protecting against attacks. In fact, as communica-tions systems expand into every facet of grid control and operations, their complexity and continuous evolution will preclude perfect protection from cyberattacks. Response and recovery, in addition to preparedness, will therefore be important components of cybersecurity, and it is impor-tant for the government agencies involved to work with the private sector and publicly owned utilities in a coordinated fashion, to support the research necessary to develop best figure 3. The cybersecurity life cycle.

AssessVulnerabilities,

Threats,Impacts

ReduceVulnerabilities,

Threats,Impacts

PreventAttacks,

Incidents,Other Outages

RespondDuring Attack

Recover andRestore

Mitigation



practices for response to and recovery from cyberattacks on transmission and distribution systems, and to deploy those practices rapidly and widely.

NERC is responsible for cybersecurity standards devel-opment and compliance for the U.S. bulk power system, but no entity currently has comparable nationwide respon-sibility for distribution systems. State public utility com-missions (PUCs)—which generally are responsible only for investor-owned distribution systems—usually lack cybersecurity expertise, and the same is true of munici-pal utilities, cooperatives, and other public systems. While the consequences of a successful attack on the bulk power system are potentially much greater than an attack at the distribution system level, the boundary between transmis-sion and distribution has become increasingly blurred, and distribution-level cybersecurity risks will require serious attention. Though NIST is facilitating the development of cybersecurity standards broadly, it does not have an opera-tional role, and no single agency currently has responsibil-ity for cybersecurity across all aspects of grid operations, including distribution systems. This is an unsolved prob-lem, but one that the federal government is actively focus-ing on. The DOE and the U.S. Department of Homeland Security recently announced an initiative to work together with industry to develop a comprehensive approach to cybersecurity. But even if this joint effort proves workable or if a single agency is ultimately given appropriate regula-tory authority, cybersecurity preparedness, response, and recovery efforts across the electric power sector, including both bulk power and distribution systems, will be critical. A variety of federal government agencies, NERC, NIST, state PUCs, utilities, public power authorities, and such expert organizations as IEEE and EPRI will need to be involved if these efforts are to be effective.

With the collection, transmission, processing, and stor-age of increasing amounts of information about customer electricity usage comes heightened concern for protecting the privacy of those customers. As advanced metering is implemented, information on personal habits will be avail-able to electric companies at a level never before envisioned by utilities or policy makers. Information about the opera-tion of the electric grid itself will soon be available at a level of detail that will be of interest to those with both commer-cial and malicious interests.

Deciding who has access rights to these data and ensur-ing consumers’ privacy will be important considerations in the design and operation of grid communications networks. Many governments have passed laws protecting the privacy of personal information, though this legislation as yet does not specifi cally target electricity usage information. Utilities and related organizations will have to develop systems and procedures to protect the privacy of grid information so as to satisfy the concerns of customers and their governments. The complex issues involved are being actively debated in several U.S. states. Coordination across states will be neces-

sary to mitigate the concerns of companies that operate in multiple jurisdictions and the concerns of their customers, as data on both companies and their customers regularly cross state boundaries.

A Changing WorkforceEven if it faced none of the challenges discussed above, the electric power industry would need to rejuvenate its work-force in order to maintain current levels of performance. The challenge of an aging technical workforce, a problem made more serious by the decline in university power engineering programs, could have a signifi cant impact on the ability of the grid to meet the new challenges and seize the new oppor-tunities described above. The IEEE U.S. Power and Energy Engineering Workforce Collaborative (PWC) has reported that approximately 45% of U.S. electric utility engineers will be eligible for retirement or could leave engineering for other reasons in the next fi ve years. While it is diffi cult to predict exactly how many new engineers will be needed between now and 2030, there appears to be a signifi cant gap between anticipated industry demands and both the pipeline of stu-dents entering power engineering and the faculty in place to train them. Fortunately, U.S. industry workforce challenges have received increasing attention in the past several years. Despite these efforts, this will likely remain an important area of focus in the years to come.

ConclusionDue to a variety of recent policy changes and technical innovations, the U.S. electric grid will encounter signifi cant opportunities and challenges over the next several decades. As we have described above, various policy and system-level issues will need to be addressed and new technologies will need to be fully developed and used appropriately for the U.S. grid to evolve along an effi cient path with minimal dis-ruption and to ensure electricity rates and levels of reliability remain acceptable. The journey to the electric grid of 2030 has begun, and there will be plenty of surprises along the way. Much can and should be done now to smooth the road ahead.

For Further ReadingMassachusetts Institute of Technology. (2011). The Future of the Electric Grid. Cambridge, MA. [Online]. Available: http://web.mit.edu/mitei/research/studies/the-electric-grid-2011.shtml

BiographiesTimothy D. Heidel is with the Massachusetts Institute of Technology.

John G. Kassakian is with the Massachusetts Institute of Technology.

Richard Schmalensee is with the Massachusetts Insti-tute of Technology.

p&e


november/december 2012 ieee power & energy magazine 61

key enabler of today’s Internet.) Such a network should have the means to value and effi ciently utilize electrons produced by small private or community-owned renewable generators on an equal footing with those served up by huge private or publicly owned and regulated utilities. This also liberates us from the constrained practice of required behaviors imposed

by the dominant use of highly regulated central power gen-eration and one-way distribution and moves us strongly toward a more democratic, user-centric view that includes distributed local generation and multidirectional networked distribution and use. Such systems are capable of reshaping the prevailing notion that quality of life around the world

© STOCKBYTE

60 ieee power & energy magazine november/december 20121540-7977/12/$31.00©2012IEEE


Date of publication: 18 October 2012

MMOST DISCUSSIONS ABOUT AC VERSUS DC ELECTRICITY INCLUDE A RETELLINGof the famous technical and commercial battle between Edison and Westinghouse/Tesla. It’s a story about everything from electrocuting elephants at state fairs to the ambitious work of electrifying both urban and rural America. It’s the tale of one of man’s greatest engineering feats. It tells of a centralized power generation system based on the dominant use of incandescent light bulbs and ac constant-speed motors. In the end though, it is a retelling of history—and unfortunately, it is a history that doesn’t project well into the future.

This article is about making history in the power world. It’s about the rebirth of the earli-est form of electrical power—dc power—and its potential to change the world once again. It is being reborn with the help of modern solid-state power electronics technology.

The story is also about the work of EMerge Alliance (EA), a nonprofi t open industry association that is creating and promoting new standards based on the contemporary use of dc technology for power generation, storage, distribution, and use. This quickly growing alliance—it already includes more than 100 organizations from industry, government, and academia—was conceived by and is

populated with thought leaders motivated by the need for a phase change in the way we think about electric power. EA was born into a world searching for ways to move away from its almost exclusive dependence on synchronous fossil-fueled centralized power generation and ac macro grid transmission and distribution toward a system that can adaptively and effi ciently include highly distributed, native dc electrical power generation and storage and deliver it an evolved predominance of natively dc loads. In the end, it’s about a new energy network, or “enernet.”

The future of civilized progress is increasingly underwritten by our use of electrons to do work. So their sourcing, distribution, and effi cient use is as fundamental as it is critical to our continued existence on this planet. While seeking better and cleaner ways of collecting and returning energy to and

from the environment, it should be fundamentally recognized that electrons play a valuable role in utilizing energy from sustainable sources that can be used to do the vast majority of the work we desire

The members of EA propose an expanded use of hybrid ac-dc power systems that are more akin to today’s adaptive and information-rich Internet than they are to yesteryear’s hard-wired party-line telephone system. The application standards they are creating include a family of application area–specifi c dc microgrids that, when interconnected with the “soon to be smart” ac grid, will com-bine to form the aforementioned “enernet.” (This term was fi rst used in a presentation made at the Massachusetts Institute of Technology (MIT) by Bob Metcalfe, the well-known inventor of Ethernet, a

DC, Come Home

DC Microgrids and the Birth of the “Enernet”

By Brian T. Patterson


Reprinted from November/December 2012 issue of IEEE Power & Energy magazine


key enabler of today’s Internet.) Such a network should have the means to value and effi ciently utilize electrons produced by small private or community-owned renewable generators on an equal footing with those served up by huge private or publicly owned and regulated utilities. This also liberates us from the constrained practice of required behaviors imposed

by the dominant use of highly regulated central power gen-eration and one-way distribution and moves us strongly toward a more democratic, user-centric view that includes distributed local generation and multidirectional networked distribution and use. Such systems are capable of reshaping the prevailing notion that quality of life around the world

© STOCKBYTE

60 ieee power & energy magazine november/december 20121540-7977/12/$31.00©2012IEEE


Date of publication: 18 October 2012

MMOST DISCUSSIONS ABOUT AC VERSUS DC ELECTRICITY INCLUDE A RETELLINGof the famous technical and commercial battle between Edison and Westinghouse/Tesla. It’s a story about everything from electrocuting elephants at state fairs to the ambitious work of electrifying both urban and rural America. It’s the tale of one of man’s greatest engineering feats. It tells of a centralized power generation system based on the dominant use of incandescent light bulbs and ac constant-speed motors. In the end though, it is a retelling of history—and unfortunately, it is a history that doesn’t project well into the future.

This article is about making history in the power world. It’s about the rebirth of the earli-est form of electrical power—dc power—and its potential to change the world once again. It is being reborn with the help of modern solid-state power electronics technology.

The story is also about the work of EMerge Alliance (EA), a nonprofi t open industry association that is creating and promoting new standards based on the contemporary use of dc technology for power generation, storage, distribution, and use. This quickly growing alliance—it already includes more than 100 organizations from industry, government, and academia—was conceived by and is

populated with thought leaders motivated by the need for a phase change in the way we think about electric power. EA was born into a world searching for ways to move away from its almost exclusive dependence on synchronous fossil-fueled centralized power generation and ac macro grid transmission and distribution toward a system that can adaptively and effi ciently include highly distributed, native dc electrical power generation and storage and deliver it an evolved predominance of natively dc loads. In the end, it’s about a new energy network, or “enernet.”

The future of civilized progress is increasingly underwritten by our use of electrons to do work. So their sourcing, distribution, and effi cient use is as fundamental as it is critical to our continued existence on this planet. While seeking better and cleaner ways of collecting and returning energy to and

from the environment, it should be fundamentally recognized that electrons play a valuable role in utilizing energy from sustainable sources that can be used to do the vast majority of the work we desire

The members of EA propose an expanded use of hybrid ac-dc power systems that are more akin to today’s adaptive and information-rich Internet than they are to yesteryear’s hard-wired party-line telephone system. The application standards they are creating include a family of application area–specifi c dc microgrids that, when interconnected with the “soon to be smart” ac grid, will com-bine to form the aforementioned “enernet.” (This term was fi rst used in a presentation made at the Massachusetts Institute of Technology (MIT) by Bob Metcalfe, the well-known inventor of Ethernet, a

DC, Come Home

DC Microgrids and the Birth of the “Enernet”

By Brian T. Patterson


Reprinted from November/December 2012 issue of IEEE Power & Energy magazine

62 ieee power & energy magazine november/december 2012

will be constrained by the limits and harmful effects of our current electrical energy systems. It is a view that stimulates innovation and investment in a far more resilient and flexible network with far less impact on the environment in the short term—and one that seeks harmony with it in the longer term.

To accomplish this, EA’s vision includes a system topol-ogy that links electrical elements starting at the chip level to electrical elements at the public utility’s generation plants and everything in between in different ways than they are currently configured. In this context, it thus redefines both the physical topology (how things are connected) and the logical topology (how things behave). The essential new physical ingredients of this vision include the concept of semiautonomous microgrids and the recognition that dc power is the technologically preferred form of electricity to be used within these grids. It seeks to minimize the wasteful impact of unnecessary power conversions and recognize that the increasing majority of new sources and uses of electri-cal power are, for the most part, natively dc or, at the least, are not constant-frequency ac and that they make use of dc-based power electronics.

EA acknowledges the technical and social challenges certain to be raised during the pursuit of its vision. When it comes to energy—and especially electrical energy and the marvel of our existing 100-year-old ac electrical energy sys-tem—many are tempted to disown the challenge of creating a better future as represented by this more balanced vision of the role dc can play. But what EA’s members envision is no more (or less) dramatic, demanding, or risky an undertak-ing than that associated with the recent transformations of our telephony, information, and computing systems during the creation of today’s Internet. In some ways it should be a far less ominous transformational job, as the lessons learned from crafting the Internet are still fresh in our minds. The reward: an electricity network that can enhance business and personal economic growth and ecological well-being in a way that rivals the positive effects of the Internet.

The Future of Zero-Net-Energy Buildings The future starts today, so EA’s vision is directly connected to the widely discussed contemporary goal of creating zero-net-energy buildings (ZEBs). ZEBs, at least in the context of this article, are buildings that “cleanly” generate enough energy on-site to equal the energy they use, thus creating a “net zero” balance at the building level. This further creates the opportunity to lessen the overall impact of energy gen-eration on our economy, climate, and ecology.

Even if one is not swayed by the desirability of the improved economics or ecology related to ZEBs—the U.S. government, for example, has called for all new commercial buildings by 2030 and 50% of existing buildings by 2040 to qualify as ZEBs—perhaps with the addition of local power storage, the prospect of making buildings less vulnerable to technical and external threats to our national electric grid sys-tem is enough of a motivator. It should be noted that ZEBs are not necessarily islanded from the grid—in fact, they are typically connected to the power network with the concept of having the grid provide back-up power supply in the case that a ZEB can not meet net zero energy for some reason. Also, during periods of excess generation to load at a ZEB, a point of interconnection to the grid is provided for the ZEB to sell back to the grid under certain circumstances.

And it’s not just the federal government that’s involved. There’s a large movement in the architectural and engineer-ing community, called the 2030 Challenge, that is focus-ing on building and renovating our way to climate-neutral buildings by 2030. Many leading firms have already joined this effort, and it is supported by the American Institute of Architects. Trying to combine forces, the U.S. Department of Energy (DOE) has funded a Zero-Energy Commercial Build-ings Consortium (CBC) to bring industry leaders, building owners, designers, and manufacturers together to identify the challenges and obstacles facing us on this path.

The biggest aspect of this challenge is that we are not starting from scratch. We can’t just concern ourselves with new buildings. Of all the commercial buildings that will exist in 2030, 85% are already built. So we need ways of taking existing buildings and improving their energy use dramati-cally. Some of these existing buildings are pretty old. And nationally, more than 95% of our building stock is small: under 50,000 ft2. In New York City, for example, the average age of commercial buildings is 50 years. In the mid-Atlantic region, nearly 50% are that old, and they tend to be small, less than 100,000 ft2. Many haven’t been renovated significantly, particularly for energy retrofits, in decades. Even in Califor-nia, a bellwether state for energy efficiency, there are no effi-ciency standards for existing buildings. This is particularly problematic in office buildings, where 37% of all commercial electrical energy is consumed. These statistics indicate the challenge we face in transforming today’s building stock.

Fortunately, many individuals and groups are now begin-ning to focus on the challenge of existing buildings in terms of sustainability. One group is creating strategies for exist-ing buildings in Philadelphia, where DOE has funded an

It’s about the rebirth of the earliest form of electrical power—dc power—and its potential to change the world once again.



We believe the dc-empowered “enernet” will be seen as the heart of what’s coming: a new electric energy age.

“innovation hub” for existing buildings, originally called the Greater Philadelphia Innovation Cluster (GPIC) and now known as the Energy Efficient Buildings Hub (EEB Hub). Another effort, on the same campus as the EEB Hub, is the GridSTAR Center, another DOE-assisted program that is coordinated in part by the Penn State Center for Sustainability.

Several common approaches to designing for low- or zero-net-energy buildings, whether they’re new or existing, are emerging from these and other similar efforts around the world. Lighting is often a primary target, both in terms of increas-ing day lighting and making the remaining electric light more energy efficient. And mechanical and heating, ventilation, and air-conditioning (HVAC) systems are seeing a range of new design strategies, including revised ventilation schemes, the use of new technologies like chilled beams and radiant panels, and the expanded use of variable-speed drive motors for pumps and air handlers. So-called “smart building” approaches add controls and building automation. Another focus is on-site power generation and storage, including using solar, wind, and other clean energy generation and more efficient power distri-bution throughout a building. In general, design strategies for new building and deep renovation projects are changing, with a growing focus on the 2030 challenge.

What’s a DC Microgrid?One of the least publicized but most significant ways a build-ing’s design can change is in the way it is powered. Chang-ing basic infrastructure has never been the glamorous part of any design challenge. But a building’s power infrastruc-ture is one of the key facets linking building design and renovation to the national electrical “smart grid” effort. A new approach to the way we generate and use power in our buildings—using an infrastructure called dc microgrids—is linked to how we should make and distribute power at the national electrical grid level—the “macro grid.”

The use of microgrids is partly motivated by the increas-ing concern for the strain on and vulnerability of our elec-trical macro grid system. Witness the 2011 blackout in Southern California due to a utility worker’s mistake in Yuma, Arizona, and the blackout in the northeastern United States in 2003. And these are only the sensationalized events reported by the media; there are thousands of lower-level events, power disturbances, and failures recorded each day. These random disturbances and linear dynamic failures in the power delivery system are putting their own emphasis on creating independent, building-level power self-sufficiency via such microgrids.

It is also believed by a growing number of proponents that “smart” dc microgrids can help us make better use of the energy generated, stored, and used at a local level. Whether they are for new on-site energy generation (e.g., solar instal-lations) or adding smart devices to monitor energy use or intelligently connecting power to electric vehicles and battery storage, such approaches give us added control of energy use at the building level, thus making buildings bet-ter “partners” with the nation’s smart grid efforts. They also provide a way to buy centrally generated energy at times of the day when it is more abundant, temporarily store it, and then use it during peak demand periods.

DC microgrids interconnect a localized grouping of electricity sources and loads that predominately gener-ates, distributes, and uses electrical power in its native dc form at low voltages (up to 1,500 Vdc) and operates either connected to the traditional centralized grid or func-tions autonomously as physical and/or economic condi-tions dictate. Such microgrids are typically connected to and operate in conjunction with ac macro grids to form a smart grid. The macro grids are typically utility-operated, centralized generation, wide-area transmission, and local distribution electricity grids that predominately use elec-trical power in its alternating current (ac) form at high and medium voltages (above 1,500 V) that otherwise require waveform, phase, and voltage synchronization for mul-tiple power source interconnection. A pictogram of typi-cal macro grid-to-microgrid interconnection is shown in Figure 1.

Regarding the potential use of such dc microgrids, the DOE-sponsored Zero Energy CBC has reported that dc power may hold the key. The consortium cited dc power and dc microgrids as a next-generation technology and application that could fundamentally change the way we power commercial buildings. They noted that dc power can reduce or eliminate ac-to-dc conversions at the equipment and building level so that we can save more of the energy we need.

But how much dc power is being used in commercial buildings? DC power is already used in most of the elec-tronic devices you’re familiar with and use in your every-day work environment, from smartphones to computers and printers to your iPad and even the lighting over your head. But it is also used in the racks and racks of equipment in data centers that support your information technology systems. And dc is fundamental to the variable-speed motor drives that help deliver your heating and air conditioning and to



the electric vehicles you drive, or are planning on driving, to and from your buildings in the future. More and more of what uses electricity is utilizing solid-state and semiconduc-tor power electronics based on dc.

The challenge is this: for those dc devices to use the ac electricity that is delivered to them, they have to convert ac to dc. Simply put, these conversions waste energy.

The Plague of Wasteful Power ConversionA telltale sign of these wasteful conversions from ac to dc are the ubiquitous power bricks and chargers cluttering our work spaces. Every time you plug in your laptop charger, you’re converting the ac available in the building to the dc power that your computer needs to run. The same thing applies to your smartphone and other personal electronic devices. When you feel these converters get hot, that’s the energy lost in the conversion process. The amount of energy lost differs with various devices, but is generally 10–25%. And what’s worse, many of these converters consume nearly as much energy when the associated device they’re attached to is off as when it is on.

There are other, less obvious ac-dc conversions going on in buildings. One is in the electrically ballasted fluorescent ceiling lights you see overhead. Another takes place within solar installations. For example, in a typical photovoltaic (PV)

system, the native dc power produced by the solar panels is inverted to ac power, just so it can be distributed in the building. Then the ac power gets converted back to dc for specific device uses, such as lighting. This double conversion wastes even more energy. After these double conversions, 15% or more of the solar energy generated is lost.

The trend toward the use of dc devices has been increasing for decades, and there’s no end in sight. Data center growth alone approaches a compound average annual rate of nearly 30%. The simple reality is that almost everything based on semiconductor electronics is also based on the use of dc power, not ac power. In fact, Virginia Tech’s Center for Power Electronics Systems in Blacks-burg estimates that more than 80% of the electricity used in office buildings

passes through power electronics and experiences one or more conversions between ac and dc electricity. And yet we don’t have comprehensive standards for how best to gener-ate, distribute, and use dc power, the form of electricity most of these devices need. Such standards could provide the opportunity to reduce or eliminate unnecessary power conversions. They would also help simplify and improve the reliability of the electronic equipment involved, reduce the waste generated when these chargers and converters are put into landfills, and help make the “user experience” simpler by eliminating the many different adapter plugs now necessary. Defining common interfaces and standards for our dc devices at multiple building levels could help us simplify how we use power while saving energy, offering the potential for 5–15% savings or more, depending on the ac-dc conversions we reduce or eliminate.

The Critical and Clarifying Role of Standards and Codes Standards and codes play critical roles in moving us toward improved energy use. Organizations such as National Fire Protection Association (NFPA), Underwriters Labora-tory (UL), National Electric Manufacturing Association (NEMA), and newer ones (including EA) are working together and have established task groups to address criti-cal issues for alternative energy, including dc microgrid

The dc microgrid-enabled “enernet” vision represents a certain level of decentralization of the nation’s grid and is intended to facilitate the current smart grid overhaul.

Generation

Smart Grid

BuildingMicrogrids

Why Microgrids?

Increase Renewable Energy Availability

Create Open Environment for Energy Innovation

Improve Availability in Underserved Markets

Improve Reliability and Security

Transmission Distribution SmartMeters

Smart Buildings

Onsite RenewableEnergy Generation

Local Energy Storage

figure 1. Pictogram of macro grid-to-microgrid interconnection.



distribution systems and electric vehi-cle charging as well as dc distributed electricity storage, natively dc genera-tion systems, and other new dc elec-trical uses. Model installation codes such as the National Electrical Code (NEC) help assure safety and other important attributes of energy sys-tems; they therefore become critically important to energy use improve-ments. These organizations have com-mitted to addressing these new issues proactively and aggressively. Already, new sections have been added to the NEC to cover small wind turbine elec-trical systems and solar PV systems in ways that minimize any associated safety risk. And for the next code cycle, hundreds of proposals have been submitted and are being con-sidered regarding alternative energy systems, new battery technologies for distributed energy storage, electric vehicle systems, fuel cells, and low-voltage dc power distribution systems.

Product and system standards also play an essential role is supporting the effective deployment of products for alter-native energy equipment and systems. Proactive develop-ment of the requirements for appropriate application, design and test requirements, code compatibility, and the definition of standardized product interoperability, system attributes, and usage outcomes are all a part of their clarifying roles. In the case of dc power distribution systems, UL and EA have directly teamed up in a number of formal and informal ways to develop these much-needed standards. Combining UL’s extensive technical, research, and government collabo-ration competencies with EA’s group of visionary and moti-vated leaders in industry has been essential in helping define the preferred alternatives for beginning the fundamentally transformative national shift to native dc electricity genera-tion, distribution, and use. And together with NEMA and Electric Power Research Institute (EPRI), they have begun to lay the groundwork for North American and global har-monization activities.

A good deal work has been done and yet more begun, while much work remains for standards organizations. But thus far, many of the key standards organizations are embracing the challenge before them. Maintaining this early momentum and velocity in this regard is vital.

Getting from Here to ThereThe challenge in doing this, of course, lies in the details of defining what’s needed. Both standards and ecosys-tem development rely heavily on use cases. What types of energy generation should be used? What loads need to be addressed? How do we create a new architecture or new

standards for “dc power distribution” that can transport and distribute energy safely and effectively between new energy sources and uses? What are the likely use cases?

EA formally—and enthusiastically—took on this chal-lenge just three short years ago. Based in California, with more than 100 member organizations that include national labs, universities, manufacturers, UL, NEMA, and other industry liaisons, EA has been identifying and creating tech-nology application standards that promote the safe and effi-cient use of dc electricity for all types of applications within and around buildings. EA has set out to create open, nonpro-prietary dc application standards in each of four key areas in buildings as well as dc microgrid standards that interconnect all the pieces. Each application area is defined as a potential microgrid that can be implemented by itself, much the way you can buy a laptop computer and not connect it to a data network but still enjoy improved productivity. In this way, any or all of the subgrids can be opportunistically created in whatever order makes sense for either new or existing buildings.

Figure 2 shows how the EA member organizations see the potential for a larger common bus collecting and distrib-uting dc power in buildings. It shows a common dc bus that can directly connect a variety of power sources such as solar, wind, fuel cells, and rectified utility ac power, when needed, to serve multiple electrical loads—at a number of different dc voltages, high and low, throughout a building.

The key application areas (shown in Figure 3) for stan-dardization of dc power use in buildings include:

✔ interiors and occupied spaces where lighting and con-trol loads dominate the need for dc electricity

✔ data centers and telecom central offices with their dc-powered information and communications technology (ICT) equipment

Solar PV Wind

WindContr.

MPPT

Facility Power Server and Common Distribution / Collector (380 Vdc Nom) Bus

Electrical Loads

Gen SetFuel Cell

Other

UtilityMeter

Power Sources

380 VdcConverter

BatteryStorage

Data Center380 Vdc

ICTDesktop24 Vdc

ElectronicLoads

380/24 Vdc

HVACLoads

380 Vdc

PlugLoads

380 Vdc

LightingLoads24 Vdc

EVCharger380 Vdc

figure 2. New microgrid power distribution topologies in buildings.



✔ outdoor electrical uses, including electric vehicle charging and outdoor light-emitting diode (LED) lighting

✔ building services, utilities, and HVAC with vari-able-speed drive (VSD) and electronic dc motorized equipment.

The thought leaders and major companies involved in this groundbreaking work of setting new power standards for buildings include power system and information tech-nology networking leaders, lighting and building products innovators, and electromechanical and solar companies. The collective focus of leaders across technology and appli-cation areas has jump-started this broad effort, enabling it to quickly reach the kind of critical mass necessary to meet our building efficiency and security challenges.

Much of the focus is on using clean, renewable power generation (in its native dc generating form), whether that’s biofuel, solar PV, or wind, and on electrical power use, such as green IT and low-energy lighting schemes. Just as we’ve leveraged hybrid power systems for cars, we can leverage hybrid power systems for buildings. The transformational coexistence of both ac and dc systems will let us focus on existing buildings as well as new buildings. It also seems best to take a modular approach, as the timing of the opportuni-ties to use hybrid power or dc power may differ in various areas and types of buildings. Some areas—such as data cen-ters—represent a significant potential for dc use when they are new, significantly expanded, or considerably updated. Others—such as interior lighting—are already recognized as big energy consumers that can be updated area by area to use dc. Still others—such as plug loads—may have to await standards for the conversion of existing branch wiring to reach all your small miscellaneous equipment uses.

DC Interiors: The Occupied SpaceBut buildings are not designed by engineers concerned with energy use alone. They are principally designed by archi-tects, who are also focused on how all aspects of their build-ings will perform for the owners and occupants who are their clients. It is important to appreciate that it is not just about energy and energy efficiency but about effective and produc-tive spaces for working, learning, healing, and so on. The sacrifice of good design simply translates into inefficiency of energy in a high order, i.e., poor productivity, under-utilized space, etc. Running a crane motor at a lower horsepower can be more efficient, providing the crane can still safely lift a prescribed load of the correct weight and articulation. The case for properly designed buildings is similar in principle. Electrical system design strategies in ZEBs that implement new standards for power distribution should also help meet a building’s overall goals. A pictogram of EA’s dc standards as implemented in the building interior is shown in Figure 4.

An example of an implementation of this standard is the headquarters of the U.S. Green Building Council in Washington, D.C. Another is the new Sustainability Resource Center (SRC) at the University of California, San Diego (UCSD), which was looking for innovation in green building strategies. As a leader in promoting new energy approaches and a regular user of the solar power already on campus, the center decided to implement direct dc distribu-tion through a new array created just for this project.

Figure 5 shows a solar array put in place for a new com-mercial interior. The goal was to use this clean energy source directly whenever it was available and not invert it to ac power, avoiding the typical 7–15% energy loss from the conversion process. The loads for the solar dc power were energy efficient but otherwise ordinary lighting and interior controls. This use of direct dc power led SRC to better-qual-ity power and greater lighting efficiency. In fact, SRC won several awards and a U.S. Green Building Council Leader-ship in Energy and Environmental Design (LEED) Gold rat-ing under commercial interiors (CI) for the project, which included an innovation credit for its “high-efficiency dc microgrid.”

LEED has also started to recognize the importance of incorporating flexibility into interior design. Proposed 2012 credit areas include a specific credit focused on flexible design. Although the credit is currently envisioned as relating partic-ularly to health care, the importance of design flexibility in many types of buildings is being more generally recognized.

DC Data CentersThere are flexible dc power design strategies for other spaces within buildings as well. Data and telecom centers are great candidates. “Green” data centers and “green IT” have become hot topics. Data centers are huge and grow-ing energy users in buildings, and there are data centers in nearly every building, not just the huge server farms created for organizations like Facebook and Google. In fact, 99% of

OccupiedSpace

DataCenters

OutdoorBuildingServices

dc PowerMicrogrids

figure 3. EA’s key dc microgrid building application segments.



ac BranchPower

dc CellingGrid

380 Vdc Bus

Occupied SpaceInfrastructure:

© 2011 EMerge Alliance

P1 = Ceiling

P1

P2 = Walls

P2

P2

P3 = FurnitureP3

P4 = FloorsP4

24 Vdc Bus

Occupancy andDaylight Sensors

OptionalOnsite dc Power HVAC

Actuator

IT WirelessAccess Device

AV Devicesand Security

PowerSupply

Lights

RoomControls

208-277 VacInteriors

figure 4. Pictogram of the EA dc standard as implemented for building interiors.

dc Source: Dedicated Solar Array

dc Loads: Lighting and Controls

figure 5. Lighting and controls on a solar-powered dc microgrid at UCSD. (Source: Armstrong World Industries.)



all data centers are considered “small.” But they contain the majority of servers using power, according to EPRI.

The challenge is that smaller data centers are operated in organizations that often don’t have the internal resources to focus on best practices for power distribution and effi-cient energy use, as they are busy focusing on making sure the system performs the data management and processing work it is intended to do. But the U.S. Environmental Pro-tection Agency (EPA) has estimated that 6 billion kWh of energy could be saved each year with only a 10% efficiency improvement in these data centers.

Again, there are new application standards starting to appear for this dc power application. EA’s technical standards group, led by EPRI and including such companies as ABB, Cisco, Delta, Emerson, Intel, Juniper, and others, is nearing completion of a new standard whose key elements are shown in Figure 6.

While these standards are being finalized, leading orga-nizations and institutions have started to implement proto-type approaches for dc in data centers. These include Duke Energy, Lawrence Berkeley National Laboratory, and, once again, UCSD—a national pioneer in new energy research and innovation.

In particular, the experience of Duke Energy is instruc-tive for those interested in looking at dc data center design. Duke’s is a typical “small-to-medium-size” data center. The owner has years of experience with ac-based data center systems. It worked with EPRI to set up a rigorous compara-tive study.

A review of some highlights of the study follows; a full report is available on the EPRI Web site, along with a video that displays technical details. The big takeaway for Duke was a 15% increase in the electrical efficiency of this data center when running on dc. In its report, EPRI noted that average reductions for other smaller data centers could fall anywhere within a 10–30% range.

Barriers: The Challenges of Increased DC Use in BuildingsThe use of dc power is not without it challenges. These fall into five major categories:

1) lack of application and equipment standards for dc power distribution

2) lack of common understanding and basic application knowledge of building distribution-level dc

acInput

A

B

100–600 Vdc

380 Vdc Busway(or Cabling)

> ac Flow> dc Flow (Native) dc Flow (Converted) Physical Data Center

Optional Onsitedc Power

Optional Onsite dcGenerator and/or Storage

Optional Onsitedc Power

ICT Racks

Point of CommonConnection

ac-dcConverter

dcECC

dcUPS

MPPT

Optional Onsiteac Generator

Copyright EMerge Alliance. All rights reserved.

figure 6. Pictogram of EA’s dc standards as implemented in a data center.



3) differences in safety and power protection device application

4) lack of a robust ecosystem to support the use of dc in building-level electrification

5) an unclear pathway for moving from ac-centric power distribution to dc-inclusive distribution schemes.

The first three challenges are being addressed with increasing resources by such standards and trade organiza-tions as EA, the European Telecommunications Standards Institute (ETSI), the International Electrotechnical Com-mission (IEC), IEEE, NEMA, NFPA, the Power Sources Manufacturers Association (PSMA), the Smart Grid Interop-erability Panel (SGIP) of the National Institute of Standards and Technology (NIST), UL, and others. As awareness of and interest in the potential benefits of dc power use increases, so do the resources each of these organizations is willing to dedicate to resolving these challenges. Currently, each of the above-named organizations has a dedicated and clearly iden-tified project or program addressing these needs.

The fourth challenge, the lack of an ecosystem, is a clas-sic “chicken or egg” issue. The power industry, following Darnell Research and Pike Research, has begun formally forecasting and tracking the ecosystem growth opportunity associated with dc microgrids. The numbers they are begin-ning to report suggest the egg is beginning to hatch.

The fifth challenge, the transformational path forward, is perhaps the least clear of all. But EA, via its strategic plan, has plotted a path with a layered approach that allows the transforma-tion to be opportunistic, especially with respect to transforming existing building stock. Dividing building power applications into blocks of subdistribution microgrids, the plan calls for a section-by-section approach over time. Each of the application “standards” lays out a subsection that can be converted if and when that part of the building is due for a renovation or updat-ing for other reasons, so the cost of the transformation is largely offset by normal capital or leasehold improvement spending.

While this means that a complete transformation may take decades, early adopters with fast-churning buildings could be done much sooner. This timing and approach is reminiscent of similar transformations seen with the Inter-net and with wireless telephony.

The dc microgrid-enabled “enernet” vision represents a certain level of decentralization of the nation’s grid and is intended to facilitate the current smart grid overhaul. The dc microgrid changes the model from an almost exclusively cen-tralized generation and distribution system of electrical power delivery to one that is significantly more flexible and accom-modating of both new alternative sources of on-site electricity generation and storage and the new mix of loads that have increasingly become the norm. It better recognizes that future electrical loads will be even more electronic, more distributed, and more essential to our economy and way of life.

By designing electric power systems that focus better on the needs of digital devices, we improve the networks in which they operate (both power and control) so as to benefit

from—or indeed require—the operational duplicity that comes with efficient electrical storage devices such as bat-teries and capacitors.

But extensive employment of dc microgrids will not hap-pen without human intervention. The impediments to their full deployment, as outlined herein, must be dealt with. The standardization and ecosystem development work EA is doing with the help of others will continue in the areas of dc microgrid–supported electric vehicle charging, building services (HVAC, water and waste pumping, compressed air, and so on), and the definition of dc microgrid and smart grid connectivity standards. And although this work can be viewed as disruptive unto itself, it is motivated by the desire to bring new order and logic to the very disruptive technologies it intends to serve and optimize

The aggregated and continuously growing use of elec-tronic data and telephony; electric vehicles; solid-state and electronically driven lighting, motors, and controls; and personal electronics—coupled with the increasing use of natively dc distributed “clean-tech” electricity production—has already and hurriedly pushed us past a logical tipping point in the ac-dc electrical energy equation. It’s a time for true innovation, not the reiterative extension of our past ways. For as surely as the digitally empowered Internet will be viewed as the heart of the information age, we believe the dc-empowered “enernet” will be seen as the heart of what’s coming: a new electric energy age.

For Further ReadingH. Kakigano, Y. Miura, and T. Ise, “Low-Voltage Bipolar-Type DC Microgrid for Super High Quality Distribution, IEEE Trans. Power Electron., vol. 25, no. 12, pp. 3066–3075, Dec. 2010.

C. Marnay and S. Vossos. LBNL/DOE Webinar: Direct DC power systems for efficiency and renewable energy inte-gration [Online]. Available: http://efficiency.lbl.gov/news/lbnl_doe_webinar_direct_dc_power_systems_for_effi-ciency_and_renewable_energy_integration_0

B. Nordman. What the real world tells us about saving energy in electronics. Lawrence Berkeley National Labora-tory Symposium [Online]. Available: http://eetd.lbl.gov/ea/nordman/docs/e3s_nordman.pdf

P. Savage, R. R. Nordhaus, and S. P. Jamieson, “DC Microgrids: Benefits and Barriers,” Yale School of Forestry & Environmental Studies, 2010.

K. Shenai and K. Shah, “Smart DC mircro-grid for effi-cient utilization of distributed renewable energy,” in Proc. of IEEE EnergyTech, Cleveland, OH, 2011, pp. 1–6.

M. Ton, B. Fortenbery, and W. Tschudi, “DC power for improved data center efficiency,” Lawrence Berkeley National Lab, Report, Mar. 2008.

BiographyBrian T. Patterson is with Armstrong World Industries, Lancaster, Pennsylvania. p&e



Meet Global Modernization Experts at the

Worldwide IEEE PES Innovative Smart Grid

Technologies Conference Series

The vision of a modernized electrical delivery system — The Smart Grid — promises to revolutionize the production, delivery and use of electricity worldwide.Experts around the world gather annually at the IEEE Power & Energy Society's global ISGT Conferences to discuss state-of-the-art innovations in smart grid technologies. Each of the ISGT conferences feature special sessions and tutorials on wide ranging topics related to grid modernization, including:

• Impact of Smart Grid on Distributed Energy Resources (electric cars, demand response, distributed generation, storage)

• Smart Sensors and Advanced Metering Infrastructure Cyber Security Systems (intelligent monitoring and outage management)

• Wide Area Protection, Communication, and Control in Energy Systems

• Power and Energy System Applications (generation, transmission, distribution, markets, operations, planning)

• Energy Management Systems (with applications to smart buildings and home automation)

• Smart Grid Devices and Standards

• And More...

The ISGT Conferences present the very best of smart grid technology to the global community with events held annually in North America, Europe, Asia and every other year in Latin America.

For information on future ISGT events, plus other PES conferences, events publications and membership, please visit www.ieee-pes.org.

Networking: Meet and speak directly with utilities, business decision makers, industry leaders, regulators, and entrepreneurs working in grid modernization

Research: Noted academics and industry professionals come together to explore ways to make smart grid a cost-effective proposition

Opportunities: International speakers report on real success stories and pitfalls – as well as current business opportunities in their region

Results: Learn about real-practice technology, deployment experience, and customer acceptance related to grid modernization

ISGT2013_PE5.indd 1 4/4/13 2:09 PM


© C

RE

ATA

S


DDEREGULATION, MARKET TRANSACTIONS, CONGESTION MANAGE-ment, and the separation of functions have created increasing complexity that is making it diffi cult to maintain situational awareness and supervision of power sys-tem performance over large areas. Past reliability events (such as blackouts) have highlighted the need for better situational awareness and advanced applications to improve planning, operations, and maintenance. The deployment of a continent-wide wide area measurement system (WAMS) is an important part of the solution to these complex problems, but it faces challenges with respect to communications and security.

Enhancing Grid Measurements

By Rakesh B. Bobba, Jeff Dagle, Erich Heine, Himanshu Khurana, William H. Sanders,

Peter Sauer, and Tim Yardley

Wide Area Measurement Systems, NASPInet, and Security

1540-7977/12/$31.00©2012 IEEE




Wide Area Measurement System (WAMS)In its recent book A Century of Innovation, the National Academy of Engineering listed widespread electrifi cation fi rst on its list of the top 20 engineering achievements of the 20th century. Although the highly interconnected North American electrical power grid is rightly hailed as a great engineering feat, managing and operating it in a reliable and safe way remains a challenge that involves many com-plex technical tasks that must be accomplished at different time and geographic scales. Such tasks include continuous feedback control, protection and control mechanisms that operate every few milliseconds at substations, state estima-tors and contingency analysis processes that operate every few minutes, and generation dispatch decisions to bring power plants online or take them off-line based on load or expected demand. In earlier years, control areas were ver-tically integrated in all respects and acted as quasi islands responsible for fl ow control. The interconnections among control areas enabled emergency fl ow paths and occasional economic benefi ts. Knowledge beyond control area bound-aries was limited and often depended on slow point-to-point communications. Modern operations are far more complex, as reliability constraints require extensive congestion man-agement with signifi cant economic consequences. Further, given that various parts of the system are owned and oper-ated by many independent entities, reliable operation of the grid depends on those tasks being accomplished at a range of geographic granularities and with a high level of coordi-nation among the various entities that manage and operate the grid. With the passage of the Energy Policy Act of 2005 in the United States, the Federal Energy Regulatory Com-mission (FERC) and North American Electric Reliability Corporation (NERC) have been given additional authority to regulate electric power entities to ensure reliable opera-tion of the grid.

Historically, the grid has been very reliable. While minor outages have been fairly common, large-scale and wide-spread outages have been rare, and most customer inter-ruptions occur within relatively localized distribution infra-structure. An increasing demand for electricity has not been accompanied by increases in transmission capacity, how-ever, putting growing pressure on the reliability and safety of the grid. Recent large blackouts and outages, such as the 14 August 2003 blackout in the Northeast and the 26 February 2008 outage in Florida, stand as evidence. The fi nal report by the U.S.-Canada Power System Outage Task Force on the August 2003 blackout pointed out that the job of main-taining the system reliably had become harder because of reduced transmission margins. The report recommended the development and adoption of technologies, such as WAMS, that could improve system reliability by providing better wide area situational awareness.

Traditionally, sensor readings from substations in utili-ties are sent via a communication network to the supervi-sory control and data acquisition (SCADA) systems in the

local utility and exchanged regionally with other utilities and reliability coordinators using the Inter-Control Cen-ter Communications Protocol (ICCP). Typically, SCADA systems acquire sensor data every 2–4 s. Since the data are not time-stamped at the point of measurement or acquired synchronously, they do not capture the state of the system at a given moment in time. Rather, the data can provide a good estimate of the system state, assuming that the system is in quasi-steady state. While the grid operates in quasi-steady state most of the time, increased stress on the sys-tem means that operators’ views of it must be more fi ne-grained and cover a wider area, moving across multiple organizations in order to improve the reliability and stabil-ity of the grid.

A WAMS can be defi ned as a system that takes measure-ments in the power grid at a high granularity, over a wide area, and across traditional control boundaries and then uses those measurements to improve grid stability through wide area situational awareness and advanced analysis. Certain power system measurements cannot be meaning-fully combined unless they are captured at the same time. An important requirement of a WAMS, therefore, is that the measurements be synchronized. A high sampling rate—typically, 30 or more samples per second—is particularly important for measuring system dynamics and is another important requirement of a WAMS. Certain elements of a WAMS have existed in rudimentary forms in the Western Interconnection since the early 1990s, and the cascading outage of 1996 provided the impetus for further WAMS development.

Many advanced applications can take advantage of the measurement capability provided by a WAMS, including:

✔ Wide area monitoring: High-speed, real-time mea-surement data and analysis are essential to achieve wide area visibility across the bulk power system for entire interconnections. Time-synchronized mea-surements from geographically dispersed locations throughout a large region enable better operational awareness of the real-time condition of the grid and allow operators to make better-informed decisions.

✔ Real-time operations: Real-time operations improve operators’ understanding of how to take advantage of the newfound visibility of grid dynamics, including interarea oscillatory modes and methods for damping and stabilizing frequency oscillations.

✔ Improved accuracy of models: Time-synchronized wide area measurements continue to be very valu-able for improving the accuracy of planning models by precisely correlating simulation output with ob-served system behavior under a variety of conditions. Improved planning models enable better assessment of system behavior and will permit a more complete assessment of dynamic performance issues, such as disturbance response, voltage and frequency response, and stability performance.



✔ Forensic analysis: Synchronized measurement data collected at high sampling rates are also helpful for forensic analysis of blackouts and other grid distur-bances. Because the data are collected at high speed and are time-synchronized, their analysis can lead to faster and better understanding of precise sequences of events.

NASPInetPhasor measurement units (PMUs), developed in the early 1990s, were among the fi rst devices that could monitor the grid in a synchronized way and produce coordinated phasor measurements, also known as synchrophasors. A GPS clock signal is the most commonly used mechanism for providing the time reference needed for synchronizing PMU measure-ments. Another distinguishing feature of PMUs, in addition to synchronized measurements, is their sampling rate, which ranges from 30 samples per second up to 120 samples per second in current implementations. Even the low end of that spectrum, 30 samples per second, is an order of magnitude higher than the sampling rate of SCADA systems, meaning that PMU devices are capable of measuring system dynamic performance in a manner that is not possible with traditional SCADA systems. Their synchronized monitoring and high sampling rate make PMUs the ideal class of monitoring device for a WAMS. Traditionally, PMUs were stand-alone devices, but today many devices such as relays and digital fault recorders (DFRs) also have the ability to produce syn-chrophasors at high sampling rates.

Realizing the need for wide area measurement, moni-toring, and control across the continent and the potential of synchrophasor technology to enable these functions, the U.S. Department of Energy (DOE), the National Electric Reliability Council (NERC), and a range of electric utilities and other organizations formed the North American Syn-chroPhasor Initiative (NASPI) in 2007. NASPI’s vision is “to improve power system reliability through wide area mea-surement, monitoring and control,” and its mission is “to cre-ate a robust, widely available and secure synchronized data measurement infrastructure for the interconnected North American electric power system with associated analysis and monitoring tools for better planning and operation and improved reliability.”

Realizing a continent-wide WAMS requires not only synchronized measurement but also a high-speed commu-nication infrastructure that enables secure sharing of syn-chronized monitoring data among control centers. There is

an effort under way at NASPI to develop such an infrastruc-ture, known as the NASPI Network (or NASPInet); it is being designed to be secure, standardized, distributed, and capable of supporting future needs. One of the key require-ments for this communication infrastructure is that it must be able to support different classes of applications with varying levels of latency, accuracy, availability, message rate, and time-alignment requirements. For example, one class of applications, such as feedback control, places strict requirements on the latency, availability, and accuracy of data, while another class of applications, such as post-event analysis, values accuracy, availability, and sampling or message rate more than latency. The communication infra-structure should therefore be able to support different qual-ity-of-service (QoS) classes for traffi c and should be able to prioritize one class over another. Conceptually, as shown in Figure 1, NASPInet is made up of two components: the phasor gateway (PGW) and the data bus (DB). The PGW is envisioned as a utility’s or control center’s sole point of access to the DB. It will let the utility or control center share its synchrophasor data and obtain synchrophasor data from other utilities or control centers. The idea is that the data sharing will follow a publish-subscribe pattern, according to which a gateway that wishes to share data will publish them so that authorized gateways may subscribe to the pub-lished stream and receive the data. Each PGW will need to manage QoS and administer cybersecurity and access rights for the data it is sharing. The DB is envisioned as a wide area network that connects all the PGWs and provides the associated services for basic connectivity, QoS manage-ment, performance monitoring, and cybersecurity.

NASPInet’s Cybersecurity Requirements and Challenges It is crucial to secure a WAMS in order to ensure the avail-ability and integrity of the data it carries, which in turn affect the reliability of the power grid, since monitoring and con-trol applications may rely on those data. The core security goals of a WAMS are to ensure the availability, integrity, and confi dentiality of the data and the underlying computing and communication infrastructure. Furthermore, the data secu-rity should be ensured end to end, that is, from the time of data origination at the sensor to the time of use by a control or monitoring application. Achieving these security objec-tives is easier within a single organization (that is, from the measurement sensor to the control center owning or manag-ing the sensor) than it is for an infrastructure distributed over

It is crucial to secure a WAMS to ensure the availability and integrity of the data it carries since monitoring and control applications may rely on those data.



a wide area like NASPInet, which is envisioned as enabling data sharing across organizational boundaries and helping to realize a continent-wide WAMS. Here we highlight the security requirements of NASPInet, the many security func-tions and mechanisms needed to meet them, and the chal-lenges of realizing them.

Authentication, Authorization, and Access ControlOwners of sensor data would not want anyone other than authorized data-sharing partners to gain access to their data. Toward that end, they need to be able to ensure that an entity with which they are communicating is what it claims to be and that it is an authorized data-sharing partner. In other words, they need to be able to authenticate the entity with which they are communicating and verify that it is an autho-rized entity before they share their data. Similarly, a data receiver may want to authenticate the entity from which it is receiving data to make sure that the incoming data are legitimate.

A naive strategy for authentication is to create an out-of-band security and communication context and use it to establish communications and perform authentication. A more dynamic and scalable approach is desirable, how-ever, and could include leveraging a trusted third-party service to establish trust and long-term cryptographic keys among the WAMS entities, such as their PGWs. The third-party service could be a Kerberos-like service that helps establish long-term symmetric cryptographic

keys among entities, a certifi -cate authority that issues digital certifi cates that are trusted by members of NASPInet, or just a simple secure and authenticated directory service in which enti-ties like gateways can post their public keys or digital certifi cates. Once an entity is authenticated, its authorization to access the data needs to be verifi ed. Access control lists (ACLs) associated with data are often used to spec-ify the list of entities authorized to access the data. In such a case, authorization checking involves ensuring that the authenticated entity is listed in the ACL asso-ciated with the data. In addition to access control for data, which is enforced by the data owner’s gateway, there must be an access control mechanism at the net-work level to limit access only to authorized entities. In the case of NASPInet, the network-level

access control is to be administered and enforced by the DB function.

Integrity and Confidentiality of Measurement Data When sending data to an authenticated and authorized entity, it is necessary to protect the data’s confi dentiality and integ-rity. It is important to protect measurement data confi denti-ality from malicious eavesdroppers because such data may contain information sensitive for the market or reveal sen-sitive information about the grid that could be exploited to disrupt grid operation. Encryption primitives are commonly used to protect data confi dentiality. Similarly, it is important to protect measurement data integrity as inadvertent or mali-cious modifi cation of measurement data could lead opera-tors or applications to make catastrophic decisions. A typical approach is to use symmetric-key-based cryptographic mes-sage authentication (or integrity) codes to detect data tam-pering and to ensure that only legitimate data are accepted for use. Another notion, closely related to data integrity protection, is that of data origin or source authentication, which assures a receiver that data indeed originated at the entity from which the receiver was expecting data. As the symmetric key used to compute the cryptographic message authentication code is shared only between the sender and receiver, in a two-party setting (one sender to one receiver), verifi cation of a message authentication code assures the receiver that the data were not tampered with in transit and that the data originated at the expected sender. Thus, in a

NASPInet

Utility A

PGW PGW PGW

Data Bus

APPSAPPS

HistorianHistorian

MonitoringCenter 1

PDC

PMU PMU PMU

PMUs

PGW PDCPhasor Gateway Phasor Data Concentrator ApplicationsAPPS

OtherUtilities andMonitoring

CentersPDC

figure 1. A continent-wide WAMS and NASPInet concept.



two-party setting symmetric-key-based cryptographic message authentication codes provide both data integrity protection and data origin authentication.

Since a data owner might share the data with multiple enti-ties at the same time, for effi -ciency reasons, a WAMS should support not just unicast or two-party data sharing (that is, one sender to one receiver), but also multicast or multiparty data shar-ing (that is, one sender to multi-ple receivers). So multicast integ-rity and confi dentiality issues must also be addressed. Whereas the cryptographic key is shared between two parties in two-party or unicast data sharing, in a mul-ticast setting, the cryptographic keys used for encryption and message integrity protection are shared among a group of entities. Support for multicast data sharing can make data sharing effi cient, as shown in Figure 2. With support for multicast data sharing, the sender only needs to encrypt the data and compute the message authentication code once using the group keys; the sender then transmits the data only once, using the underlying multicast primitive. In contrast, with-out support for multicast data sharing, a sender will have to encrypt the data and compute the message authentication code separately for each receiving entity, with different keys for each receiver; then the sender must transmit the data as many times as there are receivers, increasing both communication and computation costs.

While multicast data sharing reduces communication and computation costs, it adds additional complexity for key management and for data origin authentication. Specifi cally, in a multicast setting, when a symmetric-key-based message authentication code is verifi ed as valid by the receiver, the receiver is assured that the data haven’t been tampered with by anyone outside the multicast group. The receiver cannot be sure, however, that the data originated at any particular mem-ber of the group, as the symmetric key used to compute the cryptographic message authentication code is shared among all the multicast group members and any one of the members is technically capable of generating a valid message authen-tication code. As a result, the receiver may have to rely on other means of data origin authentication. In a secure, well-confi gured, and well-monitored network, the receiver may be able to rely on the network layer to provide assurances about the origin of data packets.

One straightforward way to achieve data origin authenti-cation in a multicast setting without relying on network-layer

guarantees is to use digital signatures, which use asymmetric keys (public-private key pairs), instead of symmetric-key-based message authentication codes. The data sender would digitally sign the data using a private key. When the signature is verifi ed as valid using a public key, which corresponds to the private key and is distributed to all group members, the receivers can be sure that the data originated at that sender, as only that sender had access to the private key used to gener-ate the signature. Unfortunately, digital signatures are expen-sive in terms of both computation and communication, and it is a challenge to meet real-time requirements when every measurement is digitally signed.

Schemes to amortize the signature cost over multiple measurements exist and could reduce the overhead associ-ated with digital signatures. By defi nition, however, those schemes provide data source authentication for a group of measurements, and the group size must be picked care-fully to reduce the costs per measurement while provid-ing data source authentication at a meaningful granular-ity. Furthermore, loss of one or more measurements in the group might mean that the signature cannot be verifi ed. To pursue this approach, it would be necessary to design mechanisms to prevent or deal with loss of measurements.

An alternative to schemes that rely on asymmetric-key-based (or public-key-based) cryptographic primitives would be schemes that use symmetric-key-based cryptographic primitives but use time synchronization between entities to create the asymmetry necessary for data origin authentica-tion. But such schemes often introduce a great deal of key management complexity and, like the amortized signature schemes, result in verifi cation delays.

figure 2. (a) Unicast versus (b) multicast data sharing.

Encrypt{Data, K1}

Encrypt{Data, K2}

Encrypt{Data, Kn}Encrypt{Data, KG}

PGW1

PGW2

PGWS

PGWS

PGWS

PGWn PGWn

PGW1

PGW2

PGW1

Sending Phasor Gateway Receiving Phasor Gateway

K1 Through Kn: Pairwise Keys KG: Group Key

(a) (b)



NonrepudiationReliability coordinators and other regional entities may need to make decisions based on data from a WAMS that will have economic consequences for their members. They may be held accountable for those decisions, and they might have to defend them. They may therefore need to use an approach that not only protects data integrity but also pre-vents the data source or sender from denying having sent the data. In other words, they need a nonrepudiation property. Digital signatures are commonly employed to provide non-repudiation. But as mentioned earlier, digital signatures are expensive in terms of both computation and communication, and it is diffi cult to meet real-time requirements when every measurement is digitally signed. While signature amorti-zation schemes could perhaps be applied here as well, it is in general harder to provide nonrepudiation via alternative schemes that use symmetric-key-based cryptographic prim-itives but rely on time synchronization to create asymmetry.

Key ManagementAn important aspect of NASPI’s network security solutions will be key management: the ability to generate, distribute, revoke, and update cryptographic keying material among NASPInet entities. The cryptographic keying material might be used to provide various security properties such as entity authentica-tion, data confi dentiality or integrity protection, and nonrepu-diation. Long-term cryptographic keys established between entities, either with the help of a trusted third party or using an out-of-band mechanism, are often used for secure distribution of keys for confi dentiality and data integrity protection.

Key management is more complex in multicast set-tings than in unicast settings, as the cryptographic keys are shared among a group of entities. When the group composi-tion changes (that is, when a member of the group leaves or a new member joins), group keys need to be updated in a timely manner. Existing group keys need to be revoked and new group keys distributed rapidly, without disrupting the real-time measurement streams. Furthermore, in a multicast setting, a sending gateway may have to maintain a group key for every data stream that it is sharing; that would not be necessary in a unicast or pairwise setting, in which a pair-wise key between the sending and receiving gateways might be suffi cient to protect all data shared between them.

While multicast networks and their associated security challenges are common to several problem domains (such as audio and video conferencing, mobile ad hoc networks, and wireless sensor networks), NASPInet presents more strin-gent real-time requirements on data delivery. For certain control applications, latency requirements can range from ten to a few hundred milliseconds for continent-scale appli-cations. Such real-time delivery requirements have a signifi -cant impact on security solutions. As discussed above, it is not feasible to digitally sign every data packet for data source authentication and integrity protection, as the computation and communication overhead could lead to unacceptable

delays in data delivery. That further complicates multicast security solutions, especially for message source authentica-tion and integrity protection, and could suggest a need to deploy sophisticated key management solutions that allow timely source authentication of each data packet, utilizing symmetric-key solutions that are signifi cantly more effi cient than asymmetric-key-based digital signatures.

Data and Infrastructure Availability To ensure data availability, it is necessary to ensure the integrity and availability of the underlying computing and communication infrastructure. While a carefully thought-out fault-tolerant design will help, such a design by itself will not be suffi cient. As part of a critical infrastructure, NASPInet will be an attractive target and must be resilient against cyberattacks and intrusions by adversaries rang-ing from novices to nation-states. There should be mecha-nisms in place to protect against cyberattacks, to monitor for and detect cyberattacks and intrusions, and to respond to and recover from cyberattacks and intrusions in a timely manner. Network access control (NAC) is an example of a network-layer protective mechanism that prevents any-one other than authenticated and authorized devices and entities from accessing the measurement communication infrastructure. Secure logging, along with the associated auditing or monitoring functions, is an example of a mech-anism that can help with investigation and recovery from intrusions.

While it is clear that data and infrastructure need to be pro-tected, the level and kind of protection depend on each situa-tion’s relevant threat model and risk assessment. For instance, must data be kept confi dential from everyone other than the intended recipients, or is it suffi cient to keep the data confi -dential from anyone outside the measurement network? The latter scenario might require simpler multicast security solu-tions than the former. Furthermore, even if data must be kept confi dential from everyone besides the intended recipients, does the trust model assume that NASPInet organizations are honest, or does it assume that they are potentially malicious? The former scenario might lead to simpler security solutions than the latter. Likewise, depending on the kind of security services available from the underlying network layer or the level of trust in the underlying network layer, security solu-tions at higher layers such as the application layer may end up being simpler. For example, if the network layer is able to provide data origin authentication, then symmetric-key-based schemes may be suffi cient to provide data confi dentiality and integrity protection, thereby reducing complexity at the appli-cation layer. The requirements, threat model, and risk must therefore be carefully analyzed, as they have major implica-tions for the security design of the system, including the poli-cies, components, and tools needed for an appropriate solu-tion. That said, once a security solution has been deployed, it is far easier to relax its security requirements than to make them more rigorous.



NASPInet: Current Status and Future DirectionsThe NASPI community is making steady progress toward achieving the vision of a continent-wide WAMS. Through its Smart Grid Investment Grant (SGIG) awards, the DOE is investing signifi cantly in the deployment of hundreds of PMUs, along with the associated communications infra-structure, across the United States.

The realization of the vision of a continent-wide NASPInet will not be trivial, however. It faces many challenges, both technical and business-related. Potential options for creat-ing such a network range from leveraging the public Internet to leased multiprotocol label switching (MPLS) circuits to utility-controlled fi ber networks to completely isolated high-speed optical networks. Using the public Internet or other shared media poses QoS and security challenges. On the other end of the spectrum, one can provision and manage a completely isolated, private, high-speed network, but doing so could be prohibitively expensive and still retain security and QoS issues such as the identifi cation of a trustworthy entity that would own and/or manage that network. In rec-ognition of the challenges, many of the SGIG awardees with PMU projects are focusing on increasing PMU deployment and utilizing the data from those deployments at a regional level. The idea is to grow these regional systems into a conti-nent-wide WAMS enabled by NASPInet in the future.

As part of the ongoing PMU data infrastructure devel-opment and deployment efforts, several key cybersecurity requirements will be addressed. In the preceding section there was a progression from basic security and functional requirements to more advanced ones, e.g., from unicast security to multicast security and from data origin authenti-cation provided or supported by the network layer to applica-tion-level data origin authentication. Correspondingly, solu-tions that address the basic requirements are less expensive and better understood than those that meet more advanced requirements. Since the infrastructure is at an early stage of development, there is an opportunity to carefully consider a wide range of threats and security requirements so that secu-rity solutions can be built in from the ground up. This will let the WAMS be realized as a resilient critical infrastructure that can withstand sophisticated, targeted cyberattacks.

For Further ReadingG. Constable and B. Somerville, A Century of Innovation: Twenty Engineering Achievements That Transformed Our Lives. Washington, DC: National Academy Press, 2003.

North American SynchroPhasor Initiative. (2009, May). Data bus technical specifi cations for North American Synchrony-Phasor Initiative network. [Online]. Available: https://www.naspi.org/site/Module/Team/dnmtt/naspinet/naspinet_databus_fi nal_spec_20090529.pdf

North American Synchro Phasor Initiative. (2009, May). Phasor gateway technical specifi cations for North Ameri-can synchro-phasor initiative network. [Online]. Available: https://www.naspi.org/site/Module/Team/dnmtt/naspinet/naspinet_phasor_gateway__fi nal_spec_20090529.pdf

D. Novosel, V. Madani, B. Bhargava, K. Vu; and J. Cole. (2008, Jan.–Feb.). Dawn of the grid synchronization. IEEE Power Energy Mag. [Online]. 6(1), 49–60. Available: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4412940&isnumber=4408514

A. G. Phadke and R. M. de Moraes. (2008, Sept.–Oct.). The wide world of wide-area measurement. IEEE Power Energy Mag. [Online]. 6(5), 52–65. Available: http://ieeex-plore.ieee.org/stamp/stamp.jsp?tp=&arnumber=4610295&isnumber=4610275

R. Bobba, E. Heine, H. Khurana, and T. Yardley. (2010, Jan.). Exploring a tiered architecture for NASPInet. Presented at the Innovative Smart Grid Technologies Conf. (ISGT) [Online]. pp. 1–8, 19–21. Available: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5434730&isnumber=5434721

D. E. Bakken, A. Bose, C. H. Hauser, D. E. Whitehead, and G. C. Zweigle. (2011, June). Smart generation and trans-mission with coherent, real-time data. Proc. IEEE [Online]. 99(6), 928–951. Available: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5768095&isnumber=5768087

BiographiesRakesh B. Bobba is with the University of Illinois at Urbana-Champaign.

Jeff Dagle is with the Pacifi c Northwest National Labora-tory.

Erich Heine is with the University of Illinois at Urbana-Champaign.

Himanshu Khurana is with Honeywell Automation and Control Systems Labs.

William H. Sanders is with the University of Illinois at Urbana-Champaign.

Peter Sauer is with the University of Illinois at Urbana-Champaign.

Tim Yardley is with the University of Illinois at Urbana-Champaign.

p&e

The NASPI community is making steady progress toward achieving the vision of a continent-wide WAMS.



Join the IEEE Power & Energy Society and Don’t be Left in the Dark

Life with Power & Energy.

Life without it.

IEEE Power & Energy Society 445 Hoes Lane Piscataway, NJ 08854 USA

To learn more about the IEEE Power & Energy Society, including the many other membership benefits, please visit www.ieee-pes.org.

Over 30,000 members of the IEEE Power & Energy Society recognize that their membership is an exceptional, cost-effective way to acquire the latest information about all aspects of the fast-changing electric power and energy industry. You can too, if you join us now!

Your membership in IEEE PES enables you to:

• Tackle broad-reaching challenges

• Become recognized as a thought leader by your industry peers

• Develop contacts that will prove useful throughout all stages of your career

• Be a part of the very active and engaged global PES Community

We help our members to be successful by providing:

• Up-to-date information on current trends and the latest technology

• Industry insight through Power & Energy magazine, technical reports and peer-reviewed publications

• Compelling programs and networking opportunities at our conferences and events

• Opportunity to meet, network and collaborate with local members via our vibrant chapters

IEEE_MemberAds.indd 2 4/1/13 3:53 PM


Staying in Control

By Julie Hull, Himanshu Khurana, Tom Markham, and Kevin Staggs

© BRAND X PICTURES & LUSHPIX


TTHE USE OF SUPERVISORY CONTROL AND DATAacquisition (SCADA) became popular in the 1960s due to the expense of manual monitoring and control and an increase in the complexity of the systems. The blackout of 1965 in the northeastern United States prompted the U.S. Federal Power Commission to urge passage of the Electric Power Reliability Act of 1967, which would have mandated closer coordination among regional coordination groups. The National Electric Reliability Council was formed in 1968. These events also drove the development of large energy management systems for transmission SCADA. Early SCADA protocols were built on electromechanical telephone switching technology. At that time, the goal of communications security was to ensure that

Cybersecurity and the Modern Electric Grid

1540-7977/12/$31.00©2012 IEEE




the command got to the mechanism for control (this secu-rity was typically implemented through repetition).

Subsequently, SCADA moved to digital communica-tions, and the use of parity bits and checksums became prevalent for error checking and is still common today in the fi eld. Many protocols were in use; typically, each manu-facturer created its own, and some end users did the same. The network architecture was typically hierarchical, with the substations isolated. In the 1980s, a number of groups began working toward a common set of standards for proto-cols. The introduction of master stations and RTUs neces-sitated local area networks (LANs) and wide area networks (WANs), both of which can utilize more than one linking technology (e.g., satellite, telephone, wireless, power line carrier, fi ber optics, or microwave) to connect RTUs to mas-ter stations. The RTUs typically perform actions requested by the master station and report out-of-bounds conditions; some also perform local control, logging, and reporting. This diversity of communication media and protocols has left its legacy in the fi eld and has made it diffi cult to secure the infrastructure.

More recently, there has been a merging of the automa-tion and business networks, with a linking of the automa-tion WAN to the corporate network and, in some cases, an extension of these networks into customer sites. The use of intelligent electronic devices (IEDs) has also become com-mon and has caused yet another shift in the communica-tions architecture. Traditionally, the system was serial and hierarchical in nature: users communicated with the sub-station through an RTU or data concentrator (which then communicated with meters, relays, equipment, and so on), or users communicated directly with feeder devices (reclos-ers, switch controllers, and other equipment). With the advent of IEDs, there is much more networked information, which then fl ows up to substations and/or feeder devices using serial, direct-connect, wireless, and packet-switched circuits. The substation communication is often through a router on a LAN, along with the human-machine interface (HMI), data concentrator, equipment, and relays and may offer remote access to feeder-level devices. Figure 1 illus-trates a typical architecture for modern SCADA systems. Since many electric grid systems are now built using tra-ditional IT hardware and software, their attack surface is much larger, making them more vulnerable to cyberattack. With that in mind, deployed systems use a layered protec-tion approach, with multiple levels of fi rewalls and “demili-tarized zones,” as seen in Figure 1.

Even with this type of layered protection, the system is still vulnerable. The National Electric Sector Cyber Secu-rity Organization (NESCO) has published a white paper, “DNS as a Covert Channel Within Protected Networks,” that demonstrates DNS data exfi ltration techniques that do not require direct connectivity to any external resource from the targeted device. An attacker can get information from the RTU out through the corporate fi rewall and create a com-munication path back to that device, highlighting the need to watch outgoing fi rewall data.

There is an increasing amount of evidence showing that attackers are now focusing on control systems. They are operating with varying motivations and intentions, including cybercrime, extortion, and warfare. In the area of cyberextortion, for example, we have been warned for years about the increased cyberextortion being practiced on elec-tric utilities in Africa, Europe, India, and Mexico, where criminals threaten to cut off power if they are not paid. In a recent paper published by McAfee and the Center for Strategic and International Studies (CSIS), “In the Dark, Crucial Industries Confront Cyberattacks,” 200 industry executives from critical electricity infrastructure enter-prises in 14 countries were surveyed. The survey group was composed of IT executives in the energy, oil and gas, and water sectors whose primary responsibilities include IT security, general security, and industrial control systems. According to the paper, “One in four survey respondents have been victims of extortion through cyberattacks or threatened cyberattacks.” And it follows that once a crimi-nal fi nds an avenue of attack that works, the attacker tends to use it again and expand the list of victims. Nation-states have also been accused of using cyberattacks on control systems; such intrusions include the Russian cyberattack on Georgia’s pipelines and the alleged 2007 Russian attack on Estonia. In Kenneth Geers’s paper “Cyberspace and the Changing Nature of Warfare,” the author outlines the stra-tegic reasons why cyberwarfare is on the rise with respect to the electric power sector, including the fact that the Inter-net is vulnerable to attack. Many may argue that the elec-tric power system is not on the Internet. In many cases it is, however. Even more common is the scenario in which a device without a direct Internet connection is connected to the Internet at some point in its life cycle for software or fi rmware updates, confi guration, or maintenance. Or the device may interface with another device (e.g., a laptop or USB drive) that has been on the Internet and carries an infection or malicious code.

There is an increasing amount of evidence showing that attackers are now focusing on control systems, operating with varying motivations and intentions.



The methods used for a cyberattack vary depending on the attacker and the motivation. Some attackers are physically able to access a site through local surveillance, by browsing wireless networks within close physical proximity or even by accessing the site physically as part of the cyberattack; some perform the entire attack from a computer that could be 10,000 mi away. In any case, typically the fi rst step is to gather as much information as possible through publicly available sources (say, from the Internet). The Internet can provide names, phys-ical layouts, installed equipment, data useful for social engineering, and port scanning for other data. After this reconnaissance, adver-saries target specifi c components and systems using malware that exploits vulnerabilities to gain access to the system. There are many attack vectors for obtaining access to a SCADA system, from a brute-force attack through the business network to intercepting nonencrypted communications and playing them back, either to mimic control actions or to mask from the operator’s view the con-trol actions that are really being performed. Attacks can vary from the relatively simple—such as that of the disgruntled former contrac-tor who used existing privileges and gained access to the control system of a sewage treatment facility in Australia, then fl ooded the surrounding area with mil-lions of liters of untreated sew-age—to the Stuxnet worm, which was purportedly an attack on the Iranian nuclear industry using highly sophisticated malware and several zero-day vulnerabilities.

In the rest of this article we look at cybersecurity objectives and properties and discuss meth-ods for minimizing cyberattacks as well as detecting and respond-ing to attacks that do succeed. We then describe some cryptographic protocols commonly used to real-ize desired security properties such as confi dentiality and integ-rity. With this background in mind, we explore the challenges of

realizing secure control systems and some approaches that might work. We discuss control system security in general and use the example of modern SCADA systems to illustrate certain ideas. Finally, we review some key ongoing efforts in the control system security area involving the U.S. govern-ment, industry, and academia.

Legend

Denotes Attack Point

Note: There Are Many AttackVectors Not Noted on ThisDiagram, Including Drivers, Etc. Internet

Workstation ServerCorporateFirewall/DMZ

Business/CorporateNetwork

ControlCenter

Firewall

SCADAServer

Front-EndProcessor

FeederDevices

FeederDevices

RemoteAccess

Meters Relays

Subdivision BSubdivision A

WAN SCADANetwork

Local HMI Local HMI

Input/OutputPoints

Input/OutputPoints

Input/OutputPoints

EquipmentMonitor

NetworkInterface

NetworkInterface

RTU RTULAN

figure 1. Typical security architecture for SCADA systems.



What Are the Goals and Objectives of Cybersecurity? Cybersecurity tools and techniques are aimed at achieving three primary properties, namely, confi dentiality, integrity, and availability (CIA). Confi dentiality is the property that ensures that only authorized entities have access to sensitive information. For example, electricity market data and trans-action information are considered sensitive and should only be accessible to authorized market agents and not to other entities such as system operators. Integrity is the property that ensures that any unauthorized modifi cations to data and information are detected. For example, an adversary should not be able to modify sensor data without detection. Avail-ability is the property that ensures that critical systems and information must be available when needed. For example, communication networks supporting wide area measure-ment systems must be available to deliver data and informa-tion (e.g., synchrophasor measurements) even in the pres-ence of malicious activity such as an adversary launching a denial-of-service (DoS) attack . For critical infrastructure such as the electric grid, availability and integrity are typi-cally considered to be more important than confi dentiality.

Other security properties of interest to control sys-tems include nonrepudiation and privacy. Nonrepudiation involves assurances that a particular command or message was actually sent, as the receiving entity claims, and is typi-cally realized using digital signatures. Privacy, as a special form of confi dentiality, refers to adequate protection of per-sonally identifi able information and functions so that only authorized entities have access to this data. For example, consumer energy consumption data need to be kept private as AMI systems are realized. Achieving these properties for all computing and communication systems supporting the electricity grid is a major research, development, deploy-ment, and maintenance challenge.

A common approach to achieving these properties is to design, develop, and deploy cybersecurity technologies for protection, detection, and response. Protection sys-tems devise security components such as key management, authentication and authorization, and perimeter defense that help ensure the CIA properties against a range of

attacks. For example, encryption tools help provide con-fi dentiality, cryptographic message authentication tools help provide integrity, and redundancy helps provide avail-ability. Secure software and hardware development tech-niques are also an essential form of protection. Given the complexity of today’s systems, vulnerabilities are likely to remain after development that can be exploited by adver-saries despite the use of advanced protection systems. To deal with this, detection tools observe network and system behavior to identify malicious activities and attacks. For example, intrusion-detection systems may look for mal-ware signatures on the network. Finally, response tools are employed to enable administrators to deal with detected attacks and activities. For example, such tools may allow dynamic changes in fi rewall policies in order to limit infor-mation fl ow to and from adversaries to contain an attack. Collectively these protection, detection, and response sys-tems create an ecosystem in which secure and trustworthy operations can be executed. Typically, these technical solu-tions are used in conjunction with appropriate training for people and the use of well-defi ned processes to form a com-prehensive solution.

What Are Some Common Security Components?Earlier, we discussed the three objectives of security, namely, confi dentially, integrity, and authentication. Cryptography is used to provide confi dentiality and integrity.

The workhorse of secure communications systems is symmetric cryptography. This is often called secret-key cryptography because the keys, which are the same at both ends of the communications link, must be kept secret. These algorithms are frequently identifi ed by the length of their keys, e.g., the 128-bit Advanced Encryption Standard (AES). They can be thought of as codebooks that take a block of input data and encrypt it in a unique way based on the secret key. Figure 2 illustrates how a symmetric cipher could be used to protect data moving from a control center to a substa-tion. The process unfolds as follows:

1) The secret keys are generated, transported to the ends of the communications link, and loaded into crypto-graphic devices (often part of a larger computing de-vice) so that they are only known to the authorized sender and receiver. If attackers are able to obtain a copy of this key, they could also decrypt the data, ren-dering the system insecure.

2) The sender’s plaintext message is then passed through the codebook algorithm, where it is transformed into ciphertext. The output of the codebook is a function of both the key and the plaintext.

3) The ciphertext is transmitted over the communication link.

4) An eavesdropper listening in on the communications is able to intercept the ciphertext, but without the key the eavesdropper cannot decrypt the data and recover

figure 2. Symmetric key cryptography provides confidentiality.

Plaintext“Set 247 On”

Plaintext“Set 247 On”

Ciphertext“k3>A+zLcb+”

Ciphertext“k3>A+zLcb+”

Internet

Key Key

Eavesdropper

Codebook Codebook



the plaintext. Thus, the symmetric cryptography pro-vides confi dentiality.

5) The receiver passes the ciphertext through the code-book algorithm in reverse, using the secret key. The output of the codebook is the original plaintext.

Securely distributing the keys for symmetric-key cryp-tography is cumbersome, so asymmetric-key (also called “public-key”) cryptography, a newer form, is used to trans-port the secret keys and perform other types of authentica-tion. Three common public-key systems are RSA, El-Gamal, and elliptic curve cryptography (ECC). The underlying mathematics of these algorithms are signifi cantly different. All three, however, have a private key used to encrypt or sign a message and a related public key used to decrypt or verify messages, as shown in Figure 3. The originator of a message (e.g., a control center) signs the message with its encryption key, which is kept private. It then distributes its public key to everyone, including potential attackers. The legitimate receiver (e.g., a substation) uses the public key to verify that the message indeed came from the claimed source. An attacker could also use the public key to verify the message. But if an attacker attempts to forge a message, the verify operation will fail. Thus, public-key cryptography can be used to provide integrity and nonrepudiation. Nonre-pudiation lets a third party verify that a message came from the entity holding the associated private key. Public-key cryptography may also be used to provide confi dentiality for small messages (e.g., a key for symmetric encryption) by encrypting them with the public key and then having the intended recipient decrypt them with its private key.

Hash functions, such as the Secure Hash Algorithm with 256-bit output (SHA-256), are used to produce a mathemati-cal fi ngerprint of a message or fi le. The hash function takes in a fi le of arbitrary size (often quite large) and produces a fi xed-length output. Hash functions have the following properties:

✔ Given a fi le and its corresponding hash, it is very dif-fi cult to fi nd another fi le that will produce the same hash output.

✔ It is very diffi cult to produce two fi les that when hashed will yield the same hash output.

The hash output may then be signed using asymmet-ric cryptography. The resulting signed hash lets a receiver check the integrity of a large fi le by recalculating the hash and comparing it with a hash signed with the private key of the sender.

Certifi cation authorities are organizations that verify the credentials of a user, device, or software and then use asymmetric cryptography together with a hash function to issue the entity a digital certifi cate (e.g., under the X.509 standard) that may then be used for authentication over a net-work. Public-key infrastructure, using certifi cation authori-ties, hash functions, and of course public-key cryptography, is often used to build authentication and key management systems.

Cryptography is helpful in addressing many security issues. But the use of cryptography within the power grid is challenging for the following reasons:

✔ Legacy systems often lack the computing power and bandwidth necessary to support strong cryptography. SCADA systems often remain in the fi eld for years, making it impractical to support the newer, more computationally intensive algorithms required as the attacker’s computing power increases over the years.

✔ Cryptography often relies on random number genera-tors with high entropy. Many embedded devices lack the means to produce good random numbers.

✔ The key distribution and revocation process can be labor-intensive and prone to errors. This is especially true when multiple organizations are involved in the process. Mistakes made in the key management pro-cess may reduce the ability to communicate, which affects availability.

There are many other security functions used to enhance the integrity and availability of systems. Antitamper mecha-nisms are frequently used to protect hardware accessible to potential attackers (e.g., smart meters). These mechanisms deter the reverse-engineering of devices to recover crypto-graphic keys or fi rmware that would disclose how a device operates.

Why Is Cybersecurity for Control Systems Challenging?There are several contributing factors that make cyberse-curity of control systems a challenge. Three of these chal-lenges are:

✔ the clash between the operations team and IT team cultures

✔ the porting of legacy control software to common off-the-shelf (COTS) platforms

✔ the long life cycle of control systems.The fi rst is a cultural issue. The SCADA system engi-

neers are responsible for the confi guration and operation of any process. This includes a requirement to assure that cer-tain control systems, such as SCADA systems, are always available. In many cases, a control system is expected to

figure 3. Asymmetric-key (public-key) cryptography can provide integrity.

Key

InternetSign = Encrypt Verify = Decrypt

X

KeyOpen Breaker #3 Open Breaker #3

Open Breaker #1Public Key

Attacker



operate a plant over periods of many years with no shutdown or reduction in product manufactured by that control system. This means that availability is one of the most important requirements for any control system. Today’s modern con-trol systems are built using open-standard IT technologies such as Microsoft Windows–based computers and Ethernet networks that include commercial routers, switches, and fi rewalls. Because the SCADA system engineers are respon-sible for the operation of the process, they feel responsible for all of the equipment required to run the process. Because IT systems are now part of the equipment required to run the process, the IT department feels it is responsible for the IT equipment running that process. This leads to a clash between the IT department and the process engineering department. Among the factors contributing to this clash are items related to the management and maintenance of those IT assets. One example concerns the installation of secu-rity updates in the IT equipment. IT typically pushes out security updates shortly after they are available, and most security updates require a reboot of the computers being updated. These reboots are usually done at a time controlled by the IT department. A reboot of a control system computer can severely affect a process operator’s ability to operate a process safely, and so the process engineering team wants more control over when the updates are installed.

Another example results from migration to Ethernet net-works. Many modern control systems integrate the status of Ethernet components such as switches and routers into the overall system status displays. IT wants to manage and mon-itor the Ethernet equipment, and this can result in a loss of view of that equipment status to the SCADA operators. One way to sum up the clash is that IT is focused on the protec-tion of the intellectual assets of the company while SCADA system engineering focuses on the protection of the physical assets and manufacturing capabilities of the company. The priorities of the two can easily confl ict, leading to a clash between the two organizations.

Standards organizations such as the ISA99 standards development committee have recognized the unique secu-rity management needs of SCADA and control systems and are drafting security standards for those systems. The intent of ISA99’s proposed standards is to complement the IT standards that already exist while addressing those areas that need special attention for control systems. The North American Electric Reliability Corporation (NERC), the suc-cessor to the National Electric Reliability Council, has also realized the need for standards for control systems that con-

trol the generation and distribution of power and has created the NERC-CIP standards, which help guide the owners and operators of critical SCADA power systems.

The migration from proprietary control systems to open systems–based control has also contributed to some of the challenges. The IT industry and the control industry have evolved at different rates. While the IT industry was moving to PCs and servers, the control industry was still produc-ing proprietary systems on proprietary networks. The con-trol industry’s shift to open systems followed that of the IT industry by approximately seven years, and the control sys-tem industry is approximately that far behind in understand-ing how to develop and deploy secure systems. Many secu-rity issues that existed in IT systems six or seven years ago are now just starting to appear in control systems. One rea-son for this is that the way the migration of control systems to open systems occurred was to port as much of the propri-etary software to open system–based platforms as possible. Because the proprietary control systems had an implicit trust in the communications among devices in those systems, very few checks were performed in the code. Once ported to an open system, an application or device may become compro-mised by invalid input. Control device protocols were also developed with implicit trust, meaning that as they were moved to Ethernet, there was no attempt to add such things as authenticated and authorized communications.

Users of control systems expect them to last for a long time. It is not unusual for a control system to operate a plant for a period of 20 years or more. Most operators don’t expect to have to change the control system during that period. This period far exceeds the life cycle of any modern piece of open-systems hardware or software. The IT industry has a turnover rate of new systems every three to fi ve years, while the turnover rate for control systems has traditionally exceeded 20 years. As the control industry evolves further, the turnover rate will have to decrease. This will be a signifi -cant challenge for the industry as we move forward.

How Does One Design Secure Systems?There are several steps that can be taken to design secure control systems. First, consider procuring components that were designed with security in mind. Designing with secu-rity in mind means, for example, that the vendor of those components can demonstrate that it has integrated a security development life cycle (SDL) into its development process. The SDL will include security steps at all phases of devel-opment. This means there are security requirements for the

Many control devices will require security devices in the network that act as compensating controls to assist in securing them.



table 1. Representative efforts in the area of best practices for control systems security.

Type Description Title and URL

Organization DHS Industrial Control Systems Joint Working Group (ICS JWG); Cross Sector Cyber Security Working Group (CSCSWG); IT Sector Coordinating Council (IT SCC); Communications Sector Coordinating Council (CommSCC)

Organization with enforced standards

NERC Cyber Attack Task Force (CATF) and several related task forceshttp://www.nerc.com/filez/catf.htmlSecurity guidelines: NERC 1300, CIP-002-1 through CIP-009-1http://www.nerc.com/page.php?cid=2%7C20http://www.nerc.com/docs/standards/sar/Draft_Version_1_Cyber_Security_Standard_1300_091504.pdf

Publication National Institute of Standards and Technology (NIST) SP800-53R3

NIST Special Publication 800-53, Revision 3http://csrc.nist.gov/publications/PubsSPs.html

Publication NISTIR 7628 NIST publication on guidelines for smart grid cybersecurity

Publication DOE-supported and industry-led roadmap

Roadmap to Secure Control Systems in the Energy Sector http://www.oe.energy.gov/DocumentsandMedia/roadmap.pdf

Working Groups/Research

DOE Office of Electricity Delivery and Energy Reliability; Control Systems Security; Cyber Security for Energy Delivery Systems (CEDS)http://www.oe.energy.gov/controlsecurity.htmNational SCADA Test Bed (NSTB)http://www.oe.energy.gov/nstb.htmhttp://www.sandia.gov/ccss/home.htmDraft road map:http://energy.gov/oe/downloads/roadmap-achieve-energy-delivery-systems-cybersecurity-2011

Working Group NIST Smart Grid Interoperability Panel (SGIP)

NIST Smart Grid Interoperability Panel, the Cyber Security Working Group (CSWG)http://www.nist.gov/smartgrid/

Working Group UCA International Users Group OpenSG

Open SG Security Working Group’s Advanced Security Acceleration Project (ASAP-SG)

Standards/Working Group

International Electro-technical Commission (IEC) Technical Committee 57 Working Group 15

Data and Communications Security; focused on security for protocols 60870-5, 60870-6, 61850, 61970, and 61968

Standards AGA 12 Cryptographic Protection of SCADA Communications Part 1:http://www.aga.org/our-issues/security/Documents/0603REPORT12.PDFPart 2, Performance Test Plan:http://cipbook.infracritical.com/book3/chapter8/ch8ref4.pdf

Standards API 1164 Pipeline SCADA Securityhttp://engineers.ihs.com/document/abstract/BPZBGBAAAAAAAAAA

Standards FIPS 140-2 Security Requirements for Cryptographic Modules

Standards IEC 62210 Power System Control and Associated Communications—Data and Communication Securityhttp://webstore.iec.ch/preview/info_iec62210%7Bed1.0%7Den.pdf

Standards IEC 62351 Power Systems Management and Associated Information Exchange—Data and Communications Security, Part 1 (there are seven parts, all of which can be found on the IEC Web site):http://webstore.iec.ch/preview/info_iec62351-1%7Bed1.0%7Den.pdf

Standards IEEE 1686, IEEE 1402

Standard for Intelligent Electronic Devices (IEDs) Cyber Security CapabilitiesIEEE Guide for Electric Power Substation Physical and Electronic Security

Standards ISA-99 Manufacturing and Control Systems Securityhttp://www.isa.org/MSTemplate.cfm?MicrositeID=988&CommitteeID=6821

Academic Research

Trustworthy Cyber Infrastructure for the Power Grid

Trustworthy Cyber Infrastructure for the Power Grid http://tcipg.org/



product. Roles are defi ned for the confi guration, operation, and administration of control systems. These roles should include privileges for each role and identifying how the device responds when a user attempts to perform an opera-tion on the device that the user does not have privileges to perform. Providing a role with only those privileges neces-sary to perform the associated functions is commonly called least privilege. The device should be deployed with least privilege already confi gured, so that the end user or inte-grator does not have to perform any additional steps for the device to be secure.

Many control devices will require security devices in the network that act as compensating controls to assist in securing them. When this is the case, the device speci-fication should define the compensating control, how to configure it, and an explanation of why it is required. The device vendor should be following secure coding practices. Finally, the device vendor should have pro-cesses in place to respond to a security vulnerability disclosure if one ever occurs for its product. These are just some of the steps required. There are many good examples of SDLs available, including Microsoft’s Secu-rity Development Lifecycle, the Open-Web Application Security Project, and the Common Lightweight Applica-tion Security Process.

Once components are procured, system integrators also need to have methodologies for developing and confi gur-ing control systems for end users. The system integrator is responsible for integrating all of the pieces that together form a control system. As a control system is integrated, it will consist of multiple devices connected to multiple areas of a process with multiple functions. A model for how a control system is to be confi gured and information is to fl ow within it exists within the international ISA-95 stan-dard. This model provides a topology to be applied while designing and confi guring a control system. This topology provides a natural defense-in-depth approach to help pro-tect the more vulnerable components of a control system. In addition to ISA-95, the International Society of Auto-mation (ISA) standards committees have formed the previ-ously mentioned ISA99 standards development committee, which is developing the security requirements for indus-trial automation and control systems. The ISA-99 standards build on the reference models in ISA-95 and create security reference models for a typical SCADA system and a typi-

cal digital control system, the two classic types of control systems.

What Is Being Done to Secure Control Systems Today?It is important to note that if the attackers and attack vectors are studied, a common set of high-ranking vulnerabilities can be created that will signifi cantly affect the success of the attack. There are many good studies that can be found on common vulnerabilities and recommendations. Here are several:

✔ “Common Cyber Security Vulnerabilities Observed in Control System Assessments by the INL NSTB Pro-gram” (November 2008, U.S. Department of Energy)

✔ “Catalog of Control Systems Security: Recommen-dations for Standards Developers” (June 2010, U.S. Department of Homeland Security; www.us-cert.gov/control_systems)

✔ “Common Cyber Security Vulnerabilities Observed in DHS Industrial Control System Assessments” (July 2009, U.S. Department of Homeland Security).

Many organizations and governments have spent millions of dollars and years’ worth of effort in studying and rec-ommending good practices for control systems security. In addition, most vendors today are actively including security in the design of their products. Table 1 provides examples of representative work in this area, rather than an exhaustive list of the many activities currently taking place.

ConclusionsThis article provides an introduction to relevant cyberse-curity concepts and issues pertaining to emerging modern electric grid systems. We looked at the history of these sys-tems, the objectives of cybersecurity, challenges in address-ing security for control systems, common security tools and components, processes for designing secure grid systems, and some key efforts under way today.

BiographiesJulie Hull is with Honeywell ACS Research Labs.

Himanshu Khurana is with Honeywell ACS Research Labs.

Tom Markham is with Honeywell ACS Research Labs.Kevin Staggs is with Honeywell ACS Research Labs. p&e

This article provides an introduction to relevant cybersecurity concepts and issues pertaining to emerging modern electric grid systems.


IEEE Electrification Magazine is the only publication dedicated to disseminating information on all matters related to microgrids onboard electric vehicles, ships, trains, planes and off-grid applications.

Published quarterly starting in mid-2013, each issue will provide:

•News, analysis and insights on electric vehicles, electric ships, electric trains and electric planes

•Feature articles that allow you to stay current and connected to the challengesandopportunitiesforelectrificationinremotepartsoftheworld

•Access to comprehensive, in-depth technical analysis from engineers inthefieldofadvancedelectrification

• Industry insights, public sector programs and case studies on electric transportation

Thisisalimitedtimeoffer.Toensureyouareamongthefirsttosamplethisexcitingnewpublication, be sure to join the IEEE Power & Energy Society by July 3, 2013 and we will sendyouthefirstissueabsolutelyFREE.

There are a lot of exciting things to come from IEEE Electrification Magazine, so be sure to join PES now and reserve your issue TODAY!

NEW

Limited Time Offer. First Issue: $0.00

IEEE Electrification Magazine Launching in 2013!

www.ieee-pes.org/electrification

www.ieee-pes.org/electrification

Join PES by July 3, 2013 and receivethefirstissue FREE

IEEE8691_ElectraAd_2.indd 1 4/18/13 2:57 PM


INT8684_PECover2013_8.indd 1 4/4/13 5:11 PM