TowardsaSecureandResilientIndustrialControlSystemwithSo7ware-DefinedNetworking

Dong(Kevin)JinDepartmentofComputerScienceIllinoisInsEtuteofTechnology

TSS/SoSSeminar,March15,2016

1

PartoftheSoSLabletwith

2

•  DavidNicol

•  BillSanders•  MaRhewCaesar

•  BrightenGodfrey

Workwith…

3

WenxuanZhou JasonCro7 MaRhewCaesar BrightenGodfrey

ChristopherHannon JiaqiYan HuiLin ChenChen

JianhuiWang JunjianQi ZhiyiLi MohammadShahidehpour

Referencestopapersinthistalk

4

•  WenxuanZhou,DongJin,JasonCro7,MaRhewCaesar,andBrightenGodfrey."EnforcingCustomizableConsistencyProperEesinSo7ware-DefinedNetworks."NSDI,2015

•  ChristopherHannon,JiaqiYanandDongJin."DSSnet:AMicrogridModelingPlaeormwithElectricalPowerDistribuEonSystemSimulaEonandSo7wareDefinedNetworkingEmulaEon.”ACMSIGSIMPADS2016(toappear)

•  HuiLin,ChenChen,JianhuiWang,JunjianQiandDongJin."Self-HealingARack-ResilientPMUNetworkforPowerSystemOperaEon."IEEETransacEonsonSmartGrid(submiRed)

•  DongJin,ZhiyiLi,ChristopherHannon,ChenChen,JianhuiWang,MohammadShahidehpour."TowardsAResilientandSecureMicrogridUsingSo7ware-DefinedNetworking."IEEETransacEonsonSmartGrid,SpecialIssueonSmartGridCyber-PhysicalSecurity(submiRed)

IndustrialControlSystems(ICS)

5

•  ControlmanycriEcalinfrastructures–  e.g.,powergrids,gasandoildistribuEonnetworks,wastewatertreatment,transportaEonsystems…

•  ModernICSesincreasinglyadoptInternettechnologytoboostcontrolefficiency,e.g.,smartgrid

NextGeneraEonofPowerGrid

LOADS SITESDISTRIBUTIONTRANSFORMER

DISTRIBUTIONSUBSTATION TRANSMISSION GENERATION

MoreEfficientorMoreVulnerable?

6 Picturesource:NISTFrameworkandRoadmapforSmartGridInteroperabilityStandards

Distribu(onOpsTransmission

Ops

OperaEons ServiceProviders

BulkGeneraEon DistribuEon Customer

MarketsRTO/ISOOps

DMS AssetMgmt

Enterprise

Bus

EMS

RTOSCADA

EMSWAMS

MDMSDemandResponse

Retailer/Wholesaler

Transmission

ISO/RTOParEcipaEon

Aggregator

EnergyMarketClearinghosue

MarketServicesInterface

PlantControlSystem

Generators SubstaEonDevice

FieldDevice

DistributedGeneraEon

U(lity

Provider

Third-Party

Provider

CIS

Billing

Home/BuildingManager

Aggregator

ElectricVehicle

DistributedGeneraEon

ElectricStorage

Appliances

ThermostatCustomer

EMS

CustomerEquipment

Meter

Others

CIS

Billing

RetailEnergyProvider

Premises

Networks

EnergyServicesInterface

MeteringSystem

DistribuEonSCADA

Enterprise

Bus

TransmissionSCADA

Enterprise

Bus

WideArea

Network

Substa(on

LANs

Internet/

e-business

FieldArea

NetworksData

Collector

SubstaEonController

ElectricStorage

Internet/

e-business

CommunicaEonPath Network

CyberThreatsinPowerGrids

7

Picturesource: 1.NaEonalCybersecurityandCommunicaEonsIntegraEonCenter(NCCIC).ICS-CERTMonitorSep2014–Feb20152.hRp://dailysignal.com/2016/01/13/ukraine-goes-dark-russia-aRributed-hackers-take-down-power-grid/

•  245incidents,reportedbyICS-CERT

•  32%inenergysector

•  80,000residentsinwesternUkraine

•  6hours,lostpoweronDec23,2015

ProtecEonofIndustrialControlSystems

8

•  Commercialof-the-shelfproducts– e.g.,firewalls,anEvirusso7ware– fine-grainedprotecEonatsingledeviceonly

•  Howtochecksystem-widerequirements– Securitypolicy(e.g.,accesscontrol)– Performancerequirement(e.g.,end-to-enddelay)

•  HowtosafelyincorporateexisEngnetworkingtechnologiesincontrolsysteminfrastructures?

ARepresentaEveSmartGridControlNetwork

9

ModernSubsta(onNetwork

SCADADataAggregator

WorkstaEonMonitor

...

relay breakerrelay breakerrelay breaker

RemoteTerminalUnit

RemoteTerminalUnit

modem

Real-Time

Control/

Monitoring

EnergyManagementSystem

DataHistorian

Time-criEcalControlUpdatesNetworkUpdates

ChallengesandOpportuni(es

DifferencesandSimilariEes

10

Similari(es

•  blackholeavoidance•  loopmiEgaEon•  fastconvergencespeeds•  prioritycontrol•  mulEpleservicesonasingle

physicalchannel•  …

AUElityControlNetwork AnEnterpriseNetwork

Differences

•  strictlydefinedforwardingpaths•  end-to-endperformance

guarantee•  system-widevisualizaEon•  real-Ememonitoring•  adeny-by-defaultsecuritymodel•  …

ProblemStatement

11

•  MinimizethegapswithanSDN-enabledcommunicaEonarchitectureforICS

•  CreateinnovaEveapplicaEonsforICSsecurityandresiliency– Real-EmenetworkverificaEon– Self-healingnetworkmanagement– Context-awareintrusiondetecEon– Manymore...

ICS–industrialcontrolsystemSDN–so7ware-definednetworking

12

Source:NickMcKeown,OpenNetworkingSummit2012

SpecializedControlPlane

SpecializedHardware

SpecializedFeatures

HorizontalOpeninterfacesRapidinnovaEon

VerEcallyintegratedClosed,proprietarySlowinnovaEon

AppAppAppAppAppAppAppApp

ControlPlane

ControlPlane

ControlPlane or or

OpenInterface

MerchantSwitchingChips

OpenInterface

SDNArchitecture

SDNArchitecture-ConEnue

Net3

Net4

Net5

Net6

Net1 Net2

OpenFlowProtocolDataPlane

ControlPlane

Applica0ons QoS AccessControl VPN

OpenFlowController

OpenFlowSwitches

13

CyberResources

SCADAServers

FieldDevices

CommunicaEonNetworks RouEng

PowerControlApplica(ons

DemandResponse

FrequencyControl

StateEsEmaEon

TopologyControl

…

…

•  Instability•  LossofLoad•  SynchronizaEonFailure•  ConEngency•  LossofEconomics

Impact

DenialofService

FalseDataInjecEon Malware Insider

ARack…

CyberANacks

CurrentPowerGrid:PotenEalCyberARacksandTheirImplicaEons

FutureSDN-enabledPowerGrid:ACyber-ARack-ResilientPlaeorm

14

AnSDN-EnabledPowerGrid

PowerGridComponentLayer

PowerNetworkLayer

CommunicaEonNetworkLayer

SDNControlLayer

ApplicaEonLayer GridApplicaEon

ControlManagemen

tMonitoring

Commun

icaE

onSystems

Powergrid

Systems

SDNApplicaEon

IDSVerificaEon

Self-healingNetwork

SolarPV

GasGenerator

ChargingStaEon

WindTurbine

ComEdComEd

PershingSubstaEon(12.47kV)

FiskSubstaEon(12.47kV)

TransiEontoanSDN-EnabledIITMicrogrid•  Real-EmereconfiguraEonofpowerdistribuEonassets•  Real-EmeislandingofcriEcalloads•  Real-EmeopEmizaEonofpowersupplyresources

15

ControlCenter

ExisEngMasterController

SDNMasterController

SDNApplica0ons

GridApplica0onsLocalSDNController1

PMU

LocalSDNController2BuildingControl

LocalSDNControllern

…

CommunicaEonNetworks

SolarPV

GasGenerator

ChargingStaEon

WindTurbine

ComEdComEd

PershingSubstaEon(12.47kV)

FiskSubstaEon(12.47kV)

16

TransiEontoanSDN-EnabledMicrogrid

•  SDN-basedApplicaEons– Real-EmeVerificaEon– Self-healingPMU

•  HybridTestbed– SDNemulaEon+PowerDistribuEonSystemSimulaEon

17

ApplicaEon1:NetworkVerificaEon–MoEvaEon

18

•  Unauthorizedaccess•  UnavailablecriEcalservices•  Systemperformancedrop•  Instability•  Lossofload•  SynchronizaEonFailure

•  …

89%ofoperatorsneversurethatconfigchangesarebug-free1

82%concernedthatchangeswouldcauseproblemswithexisEngfuncEonality1

1.  Surveyofnetworkoperators:[Kim,Reich,Gupta,Shahbaz,Feamster,Clark,USENIXNSDI2015]2.  PicturesborrowedfromVeriFlowslides[Khurshid,Zou,Zhou,Caesar,GodfreyNSDI2013]

VerificaEonSystemDesign

19

ICSApplica(onModels

NetworkModels

PolicyEngine

topologynetwork-layerstates

(e.g.,forwardingtables)

Diagnosis

•  Vulnerabili0es•  Errors

SystemFramework

DynamicModelUpdate/Selec3on Verifica3on

DynamicNetworkData(topology,forwardingtables…)DynamicApplicaEonData(controlupdates…)User-specifiedPolicy(security,performance…)

VerifiedSystemUpdates

VeriFlow

New rules

VeriFlow Operation

4/3/2013 Department of Computer Science, UIUC 11

Network Controller

Generate equivalence

classes

Generate forwarding

graphsRun queries

Diagnosis report• Type of invariant

violation• Affected set of

packets

Rules violating network invariant(s)

Good rules

Network-LayerVerificaEon

20

PriorWork•  FlowChecker

[Al-Shaeretal.,SafeConfig2010]•  HeaderSpaceAnalysis

[Kazemianetal.,NSDI2012]•  Anteater

[Maietal.,SIGCOMM2011]•  VeriFlow

[Khurshidetal.,NSDI2012]

PicturesborrowedfromVeriFlowslides[Khurshid,Zou,Zhou,Caesar,GodfreyNSDI2013]

21

Switch'A' Switch'B'

Controller'Remove&rule&1& Install'rule'2'

rule%1%

rule%2%

Challenges—TimingUncertaintyNetworkdevicesareasynchronousanddistributedinnature

Packet'

Challenges—TimingUncertainty

22

Switch'A' Switch'B'

Controller'

Install'rule'2'

rule%1%

rule%2%

Remove&rule&1&(delayed)&

Loop-freedomViolaEon

Uncertainty-awareModeling•  Naively,representeverypossiblenetworkstateO(2^n)•  Uncertaingraph:representallpossiblecombinaEons

23

UpdatesynthesisviaverificaEon

EnforcingdynamiccorrectnesswithheurisEcallymaximizedparallelism24

AshouldreachB

2 1 3 4

SlideborrowedfromWenxuanZhou,“CCG”NSDI2015

OK,but…

25

Canthesystem“deadlock”?•  Provedclassesofnetworksthatneverdeadlock•  ExperimentallyrareinpracEce!•  Lastresort:heavyweight“fallback”likeconsistentupdates[ReitblaRetal,SIGCOMM2012]

Isitfast?

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

25000$

20000$

15000$

10000$

5000$

0$7/22/2014$22:00:00$

7/22/2014$23:00:00$

7/23/2014$0:00:00$

7/23/2014$1:00:00$

//$

//$

//$

//$

//$

//$

Time$

Num

ber$o

f$Rules$

in$th

e$Network$

7/22/2014$22:00:02$

7/22/2014$23:00:02$

7/23/2014$0:00:02$

7/23/2014$1:00:02$

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate Update

GCC

Consistent Updates 0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate Update

GCC

Consistent Updates

End

End

End

Comple?on$Time$} CCG

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate Update

GCC

Consistent Updates

End

End

End

0

5000

10000

15000

20000

25000

0 2 4 6 8 10 12 14 16

Immediate Update

GCC

Consistent Updates

End

End

End

SlideborrowedfromBrightenGodfrey,TSSSeminar,Sep2015

ApplicaEon2:Self-HealingPhasorMeasurementUnit(PMU)Networks

26

IntegraEonofACommunicaEonNetworkandAPMUNetwork

Self-HealingPMUNetworks

27

VideoDemo

•  Isolatecompromiseddevices

•  “Self-heal”thenetworkbyquickly

re-establishingroutes

–  Torestorepowersystemobservability

–  Usinganintegerlinearprogrammodel

AHybridTesEngPlaeorm

28

Windows Linux

Power Coordinator● Setup Simulator ● Communicates Requests

between Emulator and Simulator

COM Port

Network &

IEDConfiguration

Network Coordinator● Configure Network

and Hosts● Synchronize with

Simulator

Synchronization Events

zmq socketKernel

Input or Import

Named Pipe

TCP Socket

Windows COM Port

Legend

DSSnetConfiguration

Processes/Elements

Components

Virtual Time System

IED Configuration

Power Element Configuration Mininet

HOSTS SWITCHES

CONTROLLER

Settings

OpenDSSElementsElements

MonitorsMonitors Controls

Circuit

Interface

Figure 2: DSSnet system architecture diagram. Note that the power simulator runs on a Windows machine and the networkemulator runs on a Linux machine.

to advance the simulation’s clock to the time stamp of thecurrent event request and to solve the power flow at thattime. Additionally, some elements of the power grid maybe modeled in the power coordinator as a function of time,such as loads and generation. These elements are not nec-essarily represented in the communication network, but canstill operate on DSSnet’s virtual clock.

3.1.5 Virtual Time System

Unlike simulation, the emulation clock elapses with thereal wall clock. Therefore, pausing the emulation requiresmore than just stopping the execution of the emulated enti-ties, but also pausing their clocks. Virtual time can be usedto achieve this goal [9, 19]. We choose to extend the workof [9], in which Mininet is patched with virtual time support.However, their motivation is di↵erent from ours.

In general, virtual time has at least two categories of ap-plication. The first one is to slow down emulation so thatit appears to emulated entities that they have su�cient vir-tual resources. Slowing down execution also alleviates theproblems caused by resource multiplexing. Another usage ofvirtual time is for emulation-simulation synchronization. InDSSnet, we assign every container a private clock, insteadof using the global time provided by the Linux OS. The con-tainers now have the flexibility to slow down, speed up orstop its own clock when synchronizing with the simulator.

However, the emulator needs to manage the consistencyacross all containers. This is achieved by a centralized time-keeper in [19], and by a two-layer consistency mechanism [9].In practice, the emulator configuration guarantees that all

containers are running with one shared virtual clock; Simi-larly, the container leverages the Linux process hierarchy toguarantee that all the applications inside the container areusing the same virtual clock. The two-layer consistency ap-proach is well-suited to this work for pausing and resumingbecause:

1. All hosts should be paused or resumed when we stopor restart the emulation.

2. All processes inside a container should be paused orresumed when we stop or restart the emulation.

The first task is done by the network coordinator. The sec-ond task is implemented based on the fact that processesinside a container belong to the same process group.

3.2 SynchronizationA key challenge in DSSnet is the synchronization between

connecting the emulated communication network and thesimulated power system. The root cause is that two di↵er-ent clock systems are used to advance experiments. Ordi-nary virtual-machine-based network emulators use the sys-tem clock, and a simulator often uses its own virtual clock.This di↵erence would lead to causality errors as shown inthe following example.In Figure 3, there are three cross-system events (E

i

), eachwith a response (R

i

). E1 occurs before E2, however, E2 mayrequire information from R1. Since the response occurs afterthe second event, the global causality is violated, and thusreduces experiment fidelity. An example of E1 is a request

PowerDistribuEonSystemSimulaEon+SDN-basedNetworkEmulaEon

AHybridTesEngPlaeorm

29

•  Challenges– TemporalfidelityinnetworkemulaEon– SynchronizaEonbetweentwosub-systems•  EmulaEon–execuEng“naEve”so7waretoproducebehaviorinwall-clockEme•  SimulaEon–execuEngmodelso7waretoproducebehaviorinvirtualEme

Ourapproach:VirtualTime

•  Keyidea:tradeexecuEonEmewithfidelity•  TimedilaEonfactor(TDF)[Gupta,2011]

•  TDF=10–  10secondsinrealEme<=>1secondinaEme-dilatedemulatedhost

–  a100Mbpslinkisscaledtoa1Gbpslink

D.Gupta,K.V.Vishwanath,etal.“Diecast:TesEngdistributedsystemswithanaccuratescalemodel”.ACMTransacEonsonComputerSystems,29(2):1–48,2011

36

VirtualTimeSystemArchitectureforaContainer-basedNetworkEmulator

Virtual Time ClockOpen vSwitch

Virtual Network

Virtual Time Keeping

……LinuxContainer

TimeDilation

LinuxContainer

LinuxContainer

Time Freezer

Freezer Command Flow

Dilation Command Flow

Sourcecode:hRps://github.com/liRlepreRy/VirtualTimeForMininet

37

VirtualTimeisUseful

38

1.EmulaEonFidelityEnhancement

2.SimulaEon/EmulaEonSynchronizaEon

FutureWork

39

•  MoreapplicaEons– e.g.,SpecificaEon-basedIntrusionDetecEon

•  NetworklayeràApplicaEonlayerandCross-layerverificaEon

•  In-houseresearchideaàRealsystemdeployment–  IITMicrogrid– FirstClusterofMicrogridsinUS(12MWIIT+10MWBronzeville)

VirtualizedU(lityNetwork1

FrequencyControlVirtualizedU(lityNetwork2

DemandResponse

VirtualizedU(lityNetwork3

StateEsEmaEonVirtualizedU(lityNetwork4

TopologyControl

ControlCenter

Cross-Layer

Verifica(on

Intrusion

Detec(on

40

SpecificaEon-basedIntrusionDetecEon

Cross-layerVerificaEon

41

CommunicaEonNetworklayer

PowerControlApplicaEonlayer

AnetworkenvironmentwithdesiredproperEes(performance,security…)

Correctappbehaviors

43