Real-Time�Internet of Things

ChenyangLuCyber-PhysicalSystemsLaboratoryh7p://www.cse.wustl.edu/~lu/

InternetofThings

Ø Convergence of q  Miniaturized devices: integrate processor, sensors and radios.

q  Low-power wireless: connect millions of devices to the Internet.q  Data analytics: make sense of sensor data.

q  Cloud: scalable computing.

Ø Large-scale IoT-driven controlq  Smart manufacturing, transportation, power grid, healthcare…q  Real killer apps of IoT!

q  Closed-loop control requires real-time performance!

ClinicalWarning

3

R.Dor,G.Hackmann,Z.Yang,C.Lu,Y.Chen,M.KollefandT.C.Bailey,ExperienceswithanEnd-To-EndWirelessClinicalMonitoringSystem,ConferenceonWirelessHealth(WH'12),October2012.

Rapid Response

IndustrialIoT

4

Offshore Onshore

WirelessHART in Process Industries[Courtesy: Emerson Process Management]

Closed-loop industrial automation à latency boundsØ  Frontend: wireless sensor-actuator networks

Ø  Backend: edge and central clouds

Real-TimeIoT

Ø Miniaturized devices à real-time embedded systems

Ø  Low-power wireless à real-time wireless

Ø Data analytics à real-time analyticsØ Cloud à real-time data processing

5

à End-to-End Real-Time Performance

Real-TimeIoT

Ø Miniaturized devices à real-time embedded systems

Ø  Low-power wireless à real-time wirelessØ Data analytics à real-time analyticsØ Cloud à real-time data processing

6

à End-to-End Real-Time Performance

IndustrialIoT

7

Offshore Onshore

WirelessHART in Process Industries[Courtesy: Emerson Process Management]

Closed-loop industrial automation à latency boundsØ  Frontend: wireless sensor-actuator networks

Ø  Backend: edge and central clouds

WirelessHART

Industrial wireless standard for process automation

8

Ø  Reliability and predictabilityq  Multi-channel TDMA MACq  One transmission per channelq  Redundant routesq  Over IEEE 802.15.4 PHY

Ø Centralized network managerq  Collect topology informationq  Generate routes and scheduleq  Change when devices/links break

TheControlChallenge

9

sensordata

Sensor

Actuator

controlcommand

Controller

Dependable control requires•  real-time •  resilience to loss

•  control performance

Most of today’s industrial wireless networks are for monitoring.

TheReal-TimeProblem

Ø A feedback control loop incurs a flow Fi q  Route: sensor à … à controller à … à actuator

q  Generate packet every period Pi

q  Multiple control loops share a network

Ø  Each flow must meet deadline Di (≤ Pi)q  Stability and predictable control performance

Ø  Research problemsq  Real-time transmission scheduling à meet end-to-end deadlinesq  Fast schedulability analysis à adapt to wireless dynamics through

admission control and rate adaptation

10

DelaysinWirelessHART

A transmission is delayed by

Ø  channel contention: all channels are assigned to other transmissions

Ø  transmission conflict over a same nodeq  contributes significantly to latency!

11

2 1

•  1 and 5 conflict•  4 and 5 conflict•  3 and 4 do not

4 5 3

FastDelayAnalysis

Ø Compute upper bound of the delay for each flowq  Sufficient condition for real-time guarantees

Ø Channel contention à multiprocessor task schedulingq  A channel à a processorq  Flow Fi à a task with period Pi, deadline Di, execution time Ci q  Leverage existing response time analysis for multiprocessors

Ø Account for delays due to transmission conflicts

12

A. Saifullah, Y. Xu, C. Lu and Y. Chen, End-to-End Communication Delay Analysis in Industrial Wireless Networks, IEEE Transactions on Computers, 64(5): 1361-1374, May 2015.

!"#$%&'"(!"#

&!"#$%&'"(!"$

)*$%+*,

DelayduetoConflict

Ø  Low-priority flow Fl and high-priority flow Fh, conflict à delay Fl

Ø  Q(I,h): #transmissions of Fh sharing nodes with Fl q  In the worst case, Fh can

delay Fl by Q(l,h) slots

Ø  Conflicts contributes significantly to delaysq  Must be considered in

algorithms and analysis!

13

Fl delayed by 2 slots

Fl delayed by 2 slots

Fl delayed by 1 slot

Real-TimeWirelessNetworkingØ  WirelessHART stack in TinyOS [EWSN 2015]

q  Implementation on a 69-node testbedq  Network manager (scheduler + routing)

Ø  Fast delay analysesq  Fixed priority [RTAS 2011, TC 2015, RTSS 2015]q  Earliest Deadline First [IWQoS 2014]

Ø  Conflict-aware real-time schedulingq  Fixed priority [ECRTS 2011, RTSS 2015]q  Dynamic priority [RTSS 2010, IWQoS 2014]

Ø  Energy-efficient routing [IoTDI 2016]

Ø  Emergency communication [ICCPS 2015]

Ø  Channel selection [INFOCOM 2017]

14

C. Lu, A. Saifullah, B. Li, M. Sha, H. Gonzalez, D. Gunatilaka, C. Wu, L. Nie and Y. Chen, Real-Time Wireless Sensor-Actuator Networks for Industrial Cyber-Physical Systems, Proceedings of the IEEE, 104(5): 1013-1024, May 2016.

Real-TimeIoT

Ø Miniaturized hardware à real-time embedded systems

Ø  Low-power wireless à real-time wireless

Ø Data analytics à real-time analyticsØ Cloud à real-time data processing

15

à End-to-End Real-Time Performance

Ø  Virtualization platforms provide no guarantee on latencyq  Xen: credit scheduler, [credit, cap]

q  VMware ESXi: [reservation, share, limitation]

q  Microsoft Hyper-V: [reserve, weight, limit]

Ø Clouds lack service level agreement on latencyq  Amazon, Google, Microsoft cloud services: #VCPUs

16

Cloudisreal-Dmetoday

Current clouds provision resources, not latency!

TowardsReal-TimeCloudØ  Support real-time applications in the cloud.

q  Latency guarantees for tasks running in virtual machines (VMs).

q  Real-time performance isolation between VMs.

q  Resource sharing between real-time and non-real-time VMs.

Ø Multi-level real-time performance provisioning.q  RT-Xen à real-time VM scheduling on a virtualized host.

q  VATC à real-time network I/O on a virtualized host.q  RT-OpenStack à real-time cloud resource management.

17

VATC: Real-Time Communication

RT-O

penS

tack

Xen

Ø Xen: type-1, baremetal hypervisor q  Domain-0: drivers, tool stack to control VMs.

q  Guest Domain: para-virtualized or fully virtualized OS.

Ø  Scheduling hierarchyq  Xen schedules VCPUs on PCPUs.

q  Guest OS schedules threads on VCPUs.

q  Xen credit scheduler: round-robin with proportional share.

18

PCPUs

OS Sched

Xen Scheduler

OS Sched OS Sched

VCPUReal-TimeTask

RT-Xen

Ø  Real-time schedulers in the Xen hypervisor.Ø  Provide real-time guarantees to tasks in VMs.

Ø  Incorporated in Xen 4.5 as the real-time scheduler.

19

RT-Xen

h7ps://sites.google.com/site/real^mexen/

S. Xi, M. Xu, C. Lu, L. Phan, C. Gill, O. Sokolsky and I. Lee, Real-Time Multi-Core Virtual Machine Scheduling in Xen, ACM International Conference on Embedded Software (EMSOFT'14), October 2014.

ComposiDonalScheduling

Ø Analytical real-time guarantees to tasks running in VMs.Ø  VM resource interfaces

q  A set of VCPUs each with resource demand <period, budget >

q  Hides task-specific informationq  Computed based on compositional scheduling analysis

20

Resource Interface Resource Interface

Resource Interface

Hypervisor

Virtual Machines

Workload Workload

Scheduler

Scheduler

Scheduler

Real-TimeSchedulerDesignØ  Global Scheduling

q  Allow VCPU migration across cores

q  Work conserving – utilize any available cores

q  Migration overhead and cache penalty

Ø  Partitioned Schedulingq  Assign and bind VCPUs to cores

q  Cores may idle when others have work pending

q  No migration overhead or cache penalty

Ø  Enforce resource interface through budget managementq  Periodic Server vs. Deferrable Server

Ø  Priority: Earliest Deadline First vs. Deadline Monotonic

21

Real-TimeDeferrableServer(RTDS)inXen

Ø  Global Scheduling q  Allow VCPU migration across cores

q  Work conserving – utilize any available cores

q  Migration overhead and cache penalty

Ø  Partitioned Schedulingq  Assign and bind VCPUs to cores

q  Cores may idle when others have work pending

q  No migration overhead or cache penalty

Ø  Enforce resource interface through budget managementq  Periodic Server vs. Deferrable Server

Ø  Priority: Earliest Deadline First vs. Deadline Monotonic

22

RT-Xenvs.Xen

23

•  Xen (credit scheduler) misses deadlines at 22% of CPU capacity.

•  RT-Xen delivers real-time performance at 78% of CPU capacity.

VirtualizedNetworkI/O

Ø Xen handles all network traffic through Dom0Ø  Real-time and non-real-time traffic share Dom0

q  CPU and network contention

Ø  Long delays for real-time traffic in virtualized hosts

XenHypervisor

NetworkComponents

Non Real-Time

App

Dom2

CPUMemory Storage

Real-Time App

Dom1Dom0

NIC

24

NetworkI/OinVirtualizedHostsØ  Linux Queueing Discipline

q  Rate-limit and shape flows

q  Prioritization or fair packet scheduling

Ø  Priority inversion in virtualization componentsq  between transmissionsq  between transmission and reception

Ø  VATC: Virtualization-Aware Traffic Controlq  Process packets in prioritized kernel threadsq  Dedicated packet queues per priority NIC

QueueingDiscipline

Dom0

Real-Time App

Dom1Non- Real-Time App

Dom2

VirtualizaDonComponents

25

C. Li, S. Xi, C. Lu, C. Gill and R. Guerin, Prioritizing Soft Real-Time Network Traffic in Virtualized Hosts Based on Xen, RTAS 2015.

Real-TimeTrafficLatency

VATC reduces priority inversion à lower latency for real-time traffic.

26

10 16 32 64 128 256 512 1024 Dyn Cons Dyn0

0.5

1

1.5

2

2.5

3

3.5

4

Roun

d−tri

p La

tenc

y ( m

s)

Interrupt Interval (µs)

Prio, Dom0−3.18FQ_CoDel, Dom0−3.18VATC

•  Median round-trip latency of real-time traffic.•  CPU contention from two small-packet interfering streams.

VirtualizedHostàCloud

Ø  Provide real-time performance to real-time VMsØ Achieve high resource utilization

27

OpenStackLimitaDons

Ø  Popular open-source cloud management system

Ø  VM resource interfaceq  Number of VCPUsq  Not real-time

Ø  VM-to-host mappingq  Filtering (admission control)

•  VCPU-to-PCPU ratio (16:1), max VMs per host (50)

•  Coarse-grained admission control for CPU resources

q  Ranking (VM allocation)

•  Balance memory usage

•  No consideration of CPU resources

28

Manager

Host Host Host

VM VM VM VM VM

RT-OpenStack

Ø Co-hosting real-time VMs with non-real-time VMs

Ø Deliver real-time performanceq  Support RT-Xen resource interfaceq  Real-time-aware VM-to-host mapping

Ø Achieve high resource utilizationq  Co-locate non-real-time VMs with real-time VMsq  Non-real-time VMs consume remaining resources without affecting the

real-time performance of real-time VMs

29

S. Xi, C. Li, C. Lu, C. Gill, M. Xu, L. Phan, I. Lee, O. Sokolsky, RT-OpenStack: CPU Resource Management for Real-Time Cloud Computing, IEEE International Conference on Cloud Computing (CLOUD'15), June 2015.

RT-OpenStack:VM-to-HostMapping

Ø Admission control: RT-Filterq  Accept real-time VMs based on schedulability and memory

q  Consider only accepted real-time VMs

Ø  VM allocation: RT-Weigherq  Balance CPU utilization

q  Consider only accepted real-time VMs

30

ResourceInterface AdmissionControl VMAllocaDon

Real-TimeVMs {<period,budget>} Schedulability+Memory CPUU^liza^on

Non-Real-TimeVMs BestEffort Memory Memory

OpenStack

31

13%

36%

31% 61% 37%

75%30%29%

47% 32%

73%Hadoopfinish^me:314seconds

Ø  Overload four hosts with real-time VMs à deadline misses. Ø  Two hosts running non-real-time VMs only.

Ø  Unbalanced distribution of real-time domains.

RT-OpenStack

32

0%

0%0%

0%

0% 0%

0%

0%0% 0%

0%

Hadoopfinish^me:435seconds

Ø  Schedulability guarantees for real-time VMs à no deadline miss.Ø  Distribute real-time VMs across hosts.

Ø  Hadoop makes progress using remaining CPU resources.

Real-DmeCloudStack

Ø  RT-Xen: real-time VM scheduling in virtualized hosts.Ø  VATC: real-time network I/O in virtualized hosts.

Ø  RT-OpenStack: real-time cloud resource management.

33

VATC: Real-Time Communication

RT-O

penS

tack

Latency guarantees

IoT:NewHorizonforReal-Time!

Ø Real “killer apps”: large-scale IoT-driven controlq Smart manufacturing, transportation, grid…

Ø From best effort to real-time IoTq Devices, wireless, cloud, analytics…

Ø Grand challenges and opportunities for real-time!q End-to-end latency across multi-tier architectureq Scalability with regard to devices and computational load

q Dependable control over IoT

34