DynamicData-DrivenandReal-TimeVerifica4onforIndustrialControlSystemSecurity

PI:Dong(Kevin)JinPh.D.Students:ChristopherHannonandXinLiu

ProgramDirector:Dr.FredericaDaremaDDDASProgramPIMee4ng,January2016

1

@IITCampusMicrogrid

IndustrialControlSystems(ICS)

2

•  Controlmanycri4calinfrastructures–  e.g.,weaponssystems,aerospace,gasandoildistribu4onnetworks,wastewatertreatment,transporta4onsystems…

•  ModernICSincreasinglyadoptInternettechnologytoboostcontrolefficiency,e.g.,smartgrid

NextGenera4onofPowerGrid

LOADS SITESDISTRIBUTIONTRANSFORMER

DISTRIBUTIONSUBSTATION TRANSMISSION GENERATION

MoreEfficientorMoreVulnerable?

3 Picturesource:NISTFrameworkandRoadmapforSmartGridInteroperabilityStandards

Distribu5onOpsTransmissionOps

Opera4ons ServiceProviders

BulkGenera4on Distribu4on Customer

MarketsRTO/ISOOps

DMS AssetMgmt

EnterpriseBus

EMS

RTOSCADA

EMSWAMS

MDMSDemandResponse

Retailer/Wholesaler

Transmission

ISO/RTOPar4cipa4on

Aggregator

EnergyMarketClearinghosue

MarketServicesInterface

PlantControlSystem

Generators Substa4onDevice

FieldDevice

DistributedGenera4on

U5lityProvider

Third-PartyProvider

CIS

Billing

Home/BuildingManager

Aggregator

ElectricVehicle

DistributedGenera4on

ElectricStorage

Appliances

ThermostatCustomer

EMS

CustomerEquipment

Meter

Others

CIS

Billing

RetailEnergyProvider

PremisesNetworks

EnergyServicesInterface

MeteringSystem

Distribu4onSCADA

EnterpriseBus

TransmissionSCADA

EnterpriseBus

WideAreaNetwork

Substa5onLANs

Internet/e-business

FieldAreaNetworksData

Collector

Substa4onController

ElectricStorage

Internet/e-business

Communica4onPath Network

CyberThreatsinPowerGrids

4

Picturesource: 1.Na4onalCybersecurityandCommunica4onsIntegra4onCenter(NCCIC).ICS-CERTMonitorSep2014–Feb20152.hep://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-aeributed-hackers-take-down-power-grid/

•  245incidents,reportedbyICS-CERT

•  32%inenergysector

•  80,000residentsinwesternUkraine

•  6hours,lostpoweronDec23,2015

Protec4onofIndustrialControlSystems

5

•  Commercialof-the-shelfproducts– e.g.,firewalls,an4virussohware– fine-grainedprotec4onatsingledevicesonly

•  Howtochecksystem-widerequirements– Securitypolicy(e.g.,accesscontrol)– Performancerequirement(e.g.,end-to-enddelay)

•  Howtosafelyincorporateexis4ngnetworkingtechnologiesincontrolsysteminfrastructures?–  real-4me,large-scale,nointerferencewithnormalopera4ons…

OurApproach:DDDAS-basedReal-TimeSystemVerifica4on

6

ICSApplica5onModels

NetworkModels

PolicyEngine

topologynetwork-layerstates

(e.g.,forwardingtables)

Diagnosis•  Vulnerabili*es•  Errors

SystemFramework

DynamicModelUpdate/Selec3on Verifica3on

DynamicNetworkData(topology,forwardingtables…)DynamicApplica4onData(controlupdates…)User-specifiedPolicy(security,performance…)

VerifiedSystemUpdates

VeriFlow

New rules

VeriFlow Operation

4/3/2013 Department of Computer Science, UIUC 11

Network Controller

Generate equivalence

classes

Generate forwarding

graphsRun queries

Diagnosis report• Type of invariant

violation• Affected set of

packets

Rules violating network invariant(s)

Good rules

Network-LayerVerifica4on

7

PriorWork•  FlowChecker

[Al-Shaeretal.,SafeConfig2010]•  HeaderSpaceAnalysis

[Kazemianetal.,NSDI2012]•  Anteater

[Maietal.,SIGCOMM2011]•  VeriFlow

[Khurshidetal.,NSDI2012]

8

Switch'A' Switch'B'

Controller'Remove&rule&1& Install'rule'2'

rule%1%

rule%2%

Challenges—TimingUncertaintyNetworkdevicesareasynchronousanddistributedinnature

Packet'

Challenges—TimingUncertainty

9

Switch'A' Switch'B'

Controller'

Install'rule'2'

rule%1%

rule%2%

Remove&rule&1&(delayed)&

Loop-freedomViola4on

Uncertainty-awareModeling•  Naively,representeverypossiblenetworkstateO(2^n)•  Uncertaingraph:representallpossiblecombina4ons

10

UpdateSynthesisviaVerifica4on

Enforcingdynamiccorrectnesswithheuris4callymaximizedparallelism

11

AshouldreachB

2 1 3 4

WenxuanZhou,DongJin,JasonCroh,MaehewCaesar,andP.BrightenGodfrey.“EnforcingCustomizableConsistencyProper4esinSohware-DefinedNetworks.”NSDI2015.

OK,but…

12

Canthesystem“deadlock”?•  Provedclassesofnetworksthatneverdeadlock•  Experimentallyrareinprac4ce!•  Lastresort:heavyweight“fallback”likeconsistentupdates[Reitblaeetal,SIGCOMM2012]

Isitfast?

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

25000$

20000$

15000$

10000$

5000$

0$7/22/2014$22:00:00$

7/22/2014$23:00:00$

7/23/2014$0:00:00$

7/23/2014$1:00:00$

//$

//$

//$

//$

//$

//$

Time$

Num

ber$o

f$Rules$

in$th

e$Network$

7/22/2014$22:00:02$

7/22/2014$23:00:02$

7/23/2014$0:00:02$

7/23/2014$1:00:02$

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent Updates 0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent UpdatesEndEndEnd

Comple?on$Time$} CCG

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent UpdatesEndEndEnd

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate UpdateGCC

Consistent UpdatesEndEndEnd

CyberResources

SCADAServers

FieldDevices

Communica4onNetworks Rou4ng

PowerControlApplica5ons

DemandResponse

FrequencyControl

StateEs4ma4on

TopologyControl

…

…

•  Instability•  LossofLoad•  Synchroniza4onFailure•  Con4ngency•  LossofEconomics

Impact

DenialofService

FalseDataInjec4on Malware Insider

Aeack…

CyberAMacks

(a)CurrentPowerGrid:Poten4alCyberAeacksandTheirImplica4ons

VirtualizedU5lityNetwork1FrequencyControl

VirtualizedU5lityNetwork2DemandResponse

VirtualizedU5lityNetwork3StateEs4ma4on

VirtualizedU5lityNetwork4TopologyControl

(b)FutureSDN-enabledPowerGrid:ACyber-Aeack-ResilientPlauorm

ControlCenter

Cross-LayerVerifica5on

IntrusionDetec5on

What’snext?

13

•  Detec4on=>Mi4ga4on–  Example,Self-healingPMUnetworks

•  In-houseresearchidea=>Realsystemdeployment–  SDN-enabledIITMicrogrid

•  Networklayer=>Applica4onlayer,andCross-layerverifica4on

Task1:Self-HealingPMUNetworks(OngoingWork)

14PMU–PhasorMeasurementUnit

VideoDemo

“Self-HealingAeack-ResilientPMUNetworkforPowerSystemOpera4on,”SubmieedtoIEEETransac4onofSmartGrid,2016

SolarPV

GasGenerator

ChargingSta4on

WindTurbine

ComEdComEd

PershingSubsta4on(12.47kV)

FiskSubsta4on(12.47kV)

Task2:Transi4ontoanSDN-EnabledIITMicrogrid(OngoingWork)

•  Real-4mereconfigura4onofpowerdistribu4onassets•  Real-4meislandingofcri4calloads•  Real-4meop4miza4onofpowersupplyresources

15

ControlCenter

Exis4ngMasterController

SDNMasterController

SDNApplica*ons

GridApplica*onsLocalSDNController1

PMU

LocalSDNController2BuildingControl

LocalSDNControllern

…

Communica4onNetworks

SolarPV

GasGenerator

ChargingSta4on

WindTurbine

ComEdComEd

PershingSubsta4on(12.47kV)

FiskSubsta4on(12.47kV)

16

Task2:Transi4ontoanSDN-EnabledIITMicrogridACo-Simula4onFramework

17

Windows Linux

Power Coordinator● Setup Simulator ● Communicates Requests

between Emulator and Simulator

COM Port

Network &

IEDConfiguration

Network Coordinator● Configure Network

and Hosts● Synchronize with

Simulator

Synchronization Events

zmq socketKernel

Input or Import

Named Pipe

TCP Socket

Windows COM Port

Legend

DSSnetConfiguration

Processes/Elements

Components

Virtual Time System

IED Configuration

Power Element Configuration Mininet

HOSTS SWITCHES

CONTROLLER

Settings

OpenDSSElementsElements

MonitorsMonitors Controls

Circuit

Interface

Figure 2: DSSnet system architecture diagram. Note that the power simulator runs on a Windows machine and the networkemulator runs on a Linux machine.

to advance the simulation’s clock to the time stamp of thecurrent event request and to solve the power flow at thattime. Additionally, some elements of the power grid maybe modeled in the power coordinator as a function of time,such as loads and generation. These elements are not nec-essarily represented in the communication network, but canstill operate on DSSnet’s virtual clock.

3.1.5 Virtual Time System

Unlike simulation, the emulation clock elapses with thereal wall clock. Therefore, pausing the emulation requiresmore than just stopping the execution of the emulated enti-ties, but also pausing their clocks. Virtual time can be usedto achieve this goal [9, 19]. We choose to extend the workof [9], in which Mininet is patched with virtual time support.However, their motivation is di↵erent from ours.

In general, virtual time has at least two categories of ap-plication. The first one is to slow down emulation so thatit appears to emulated entities that they have su�cient vir-tual resources. Slowing down execution also alleviates theproblems caused by resource multiplexing. Another usage ofvirtual time is for emulation-simulation synchronization. InDSSnet, we assign every container a private clock, insteadof using the global time provided by the Linux OS. The con-tainers now have the flexibility to slow down, speed up orstop its own clock when synchronizing with the simulator.

However, the emulator needs to manage the consistencyacross all containers. This is achieved by a centralized time-keeper in [19], and by a two-layer consistency mechanism [9].In practice, the emulator configuration guarantees that all

containers are running with one shared virtual clock; Simi-larly, the container leverages the Linux process hierarchy toguarantee that all the applications inside the container areusing the same virtual clock. The two-layer consistency ap-proach is well-suited to this work for pausing and resumingbecause:

1. All hosts should be paused or resumed when we stopor restart the emulation.

2. All processes inside a container should be paused orresumed when we stop or restart the emulation.

The first task is done by the network coordinator. The sec-ond task is implemented based on the fact that processesinside a container belong to the same process group.

3.2 SynchronizationA key challenge in DSSnet is the synchronization between

connecting the emulated communication network and thesimulated power system. The root cause is that two di↵er-ent clock systems are used to advance experiments. Ordi-nary virtual-machine-based network emulators use the sys-tem clock, and a simulator often uses its own virtual clock.This di↵erence would lead to causality errors as shown inthe following example.In Figure 3, there are three cross-system events (E

i

), eachwith a response (R

i

). E1 occurs before E2, however, E2 mayrequire information from R1. Since the response occurs afterthe second event, the global causality is violated, and thusreduces experiment fidelity. An example of E1 is a request

“DSSnet:ASmartGridModelingPlauormCombiningElectricalPowerDistribu4onSystemSimula4onandSohwareDefinedNetworkingEmula4on,”SubmieedtoACMSIMSIGPADS,2016

Task3:Cross-layerVerifica4onFramework

18

Communica4onNetworklayer

PowerControlApplica4onlayer

Anetworkenvironmentwithdesiredproper4es(performance,security…)

Correctappbehaviors

19

6) Guaranteed realization of model predictive control

“MPC strategies are quite appealing for energy management of microgrids, since they allow for the implementation of control actions that anticipate future events such as variations in power outputs from non-dispatchable DER units, energy prices and instantaneous demand.” [2]

The general concept of MPC is illustrated by Figure 4. For instance, using MPC to eliminate the thermal overload of certain line(s) is shown in Figure 5. In such emergency conditions, it is essential to quickly lower the line flow(s) before tripping(s) that make(s) the system conditions more severe. Here, multiple control actions are available, e.g. generation adjustment, topology control, load shedding and both the choices and their sequences need to be optimally determined. In this case, the following control actions are assumed to be implemented: Action 1 (disconnection with PCC (Point of Common Coupling)), Action 2 (topology change of microgrid network), Action 3 (increase of generation output of unit n), Action 4 (shedding load at bus m). If some of these control actions are missed or mistakenly placed, the microgrid is likely to suffer from frequency or voltage oscillation, resulting more severe system conditions.

... Action'NAction'2Action'1

Time

Emergency'Occurs

Emergency'Detected

Maximum'Response'time

ConditionDeteriorates

Figure 4 Sequence of control actions by MPC

Action'4Action'2Action'1

Time

Emergency'Occurs

Emergency'Detected

Action'3

Emergency'Mitigated

!(a) Desired sequence of control actions

Action'4Action'2Action'1

Time

Emergency'Occurs

Emergency'Detected

Action'3

Condition'Deteriorates

System'Crashes

lost'or'delayed !(b) Loss or delay of control actions

Action'4Action'1Action'2

Time

Emergency'Occurs

Emergency'Detected

Action'3

Condition'Deteriorates

System'Crashes

disordered !(c) Disorder of control actions

Figure 5 Sequence of control actions

ModelPredic4veControl(MPC)Example:IncorrectPowerApplica4onControlduetoNetworkTemporalUncertainty

Task3:Cross-layerVerifica4onFramework

AchievementHighlights•  JournalPapers–  1toappear(ACMTOMACS),1underreview(IEEESmartGrid)

•  ConferencePapers–  2published,1underreview(ACMSIMGSIMPADS,ACMSOSR)

•  Awards–  BestPaperAward(PADS’15)–  BestPosterAward(PADS’15)–  Student,AdnanHaider(co-advisedwithDr.Xian-HeSun),namedfinalistforCRAOutstandingUndergraduateResearcherAward

20

DDDASWorkshopinconjunc4onwiththeACMSIGSIMPADSConference•  When:May16–17noon,2016•  Where:Banff,Alberta,Canada•  Keynotespeaker:Dr.FredericaDarema•  Co-chairs:RichardFujimoto,Dong(Kevin)Jin•  PaperSubmission:February1,2016

21

22