Latency- and Resilience-Aware NetworkingSPP 1914: “Cyber-Physical Networking”http://larn.systems

Andreas Schmidt, Pablo Gil Pereira, Thorsten HerfetTelecommunications LabSaarland Informatics Campus - Saarbrücken

Stefan Reif, Timo Hönig, Wolfgang Schröder-PreikschatDepartment of Computer Science 4 (Distributed Systems and Operating Systems)Friedrich-Alexander-Universität Erlangen-Nürnberg

November 30th, 2018

Recapitulation: Goals

Hardware

Operating System

PRRT

ApplicationTailor PRRT to control applications

Improve PRRT’s performance

OS for predictable communication

Tailor network to control applications

2 / 20

Outcome (since April 2018)

Software, Hardware & AlgorithmsPRRT Predictable Reliable Real-time Transport protocol

- Improved Packet Loss Measurement- Coding Configuration Optimization- Cross-Layer Pacing + Rate Control- Released stable version 0.3.1

X-Lap Cross-Layer Timing Analysis∆elta System Modification AnalysisRNA Reliable Networking Atom

- Drone- Quality of Control Metrics- Preparing release 0.9.0

3 / 20

Outcome (since April 2018)

Software, Hardware & AlgorithmsPRRT Predictable Reliable Real-time Transport protocol

- Improved Packet Loss Measurement- Coding Configuration Optimization- Cross-Layer Pacing + Rate Control- Released stable version 0.3.1

X-Lap Cross-Layer Timing Analysis∆elta System Modification Analysis

RNA Reliable Networking Atom- Drone- Quality of Control Metrics- Preparing release 0.9.0

3 / 20

Outcome (since April 2018)

Software, Hardware & AlgorithmsPRRT Predictable Reliable Real-time Transport protocol

- Improved Packet Loss Measurement- Coding Configuration Optimization- Cross-Layer Pacing + Rate Control- Released stable version 0.3.1

X-Lap Cross-Layer Timing Analysis∆elta System Modification AnalysisRNA Reliable Networking Atom

- Drone- Quality of Control Metrics- Preparing release 0.9.0

3 / 20

Outline

Status

PRRT

X-Lap & ∆elta

INTspect

Conclusion

4 / 20

PRRT: Update

Changelog▶ v0.3.1 released in November

▶ Congestion control inspired by BBR 1.x▶ Error control with reliable timing for ARQ▶ Improved network estimation capabilities

▶ v0.2.0 released in May▶ API enhancements.▶ Significant extension of the Wiki, README and other pieces of documentation.

Components under Development

▶ Rate Control & Pacing:Reduced latencies and jitter.

▶ Improved loss estimation▶ Performance optimizations

2 × 101 3 × 101 4 × 101

E2E Delivery Time [ms]

0.0

0.2

0.4

0.6

0.8

1.0

CDF

TCP-CUBICTCP-BBRPRRTDTopt

5 / 20

PRRT: Update

Changelog▶ v0.3.1 released in November

▶ Congestion control inspired by BBR 1.x▶ Error control with reliable timing for ARQ▶ Improved network estimation capabilities

▶ v0.2.0 released in May▶ API enhancements.▶ Significant extension of the Wiki, README and other pieces of documentation.

Components under Development

▶ Rate Control & Pacing:Reduced latencies and jitter.

▶ Improved loss estimation▶ Performance optimizations

2 × 101 3 × 101 4 × 101

E2E Delivery Time [ms]

0.0

0.2

0.4

0.6

0.8

1.0

CDF

TCP-CUBICTCP-BBRPRRTDTopt

5 / 20

Outline

Status

PRRT

X-Lap & ∆elta

INTspect

Conclusion

6 / 20

∆elta: Approach

Implement

Evaluate

Analyze

Approach:▶ Measurement-based protocol

evaluation

▶ Derive meaningful information▶ Support protocol design

Challenges:□ Identify critical code parts

→ Latency, Jitter, Energy

□Analyze code modifications□Automate feedback

7 / 20

∆elta: Approach

Implement

Evaluate

Analyze

Approach:▶ Measurement-based protocol

evaluation▶ Derive meaningful information▶ Support protocol design

Challenges:□ Identify critical code parts

→ Latency, Jitter, Energy

□Analyze code modifications□Automate feedback

7 / 20

∆elta: Approach

Implement

Evaluate

Analyze

Approach:▶ Measurement-based protocol

evaluation▶ Derive meaningful information▶ Support protocol design

Challenges:□ Identify critical code parts

→ Latency, Jitter, Energy

□Analyze code modifications□Automate feedback

7 / 20

Control-Flow Reconstruction

E1

E2

E3

E4

E5

Goal:▶ Reconstruct protocol information from

timestamps

Control flow reconstruction:1. Compute happens-before relation of events

2. Compute happens-directly-before relationof events

Edges represent:▶ Code segments ⟨Ei,Ej⟩▶ Local communication▶ False-positives

8 / 20

Control-Flow Reconstruction

E1

E2

E3

E4

E5

Goal:▶ Reconstruct protocol information from

timestamps

Control flow reconstruction:1. Compute happens-before relation of events2. Compute happens-directly-before relation

of events

Edges represent:▶ Code segments ⟨Ei,Ej⟩▶ Local communication▶ False-positives

8 / 20

Control-Flow Reconstruction

E1

E2

E3

E4

E5

Goal:▶ Reconstruct protocol information from

timestamps

Control flow reconstruction:1. Compute happens-before relation of events2. Compute happens-directly-before relation

of events

Edges represent:▶ Code segments ⟨Ei,Ej⟩▶ Local communication▶ False-positives

8 / 20

Latency Criticality Analysis

0.0 0.1 0.2 0.3 0.4 0.5 0.6latency criticality

<CopyOutputEnd, PrrtDeliver>

<PrrtSubmitPackage, PrrtSendEnd>

<HandlePacketStart, SendFeedbackStart>

<CopyOutputStart, CopyOutputEnd>

<PrrtReturnPackage, HandlePacketEnd>

<DecodeStart, DecodeEnd>

<HandlePacketEnd, PrrtReceivePackage>

<SendFeedbackEnd, PrrtReturnPackage>

<DecodeEnd, HandlePacketStart>

<PrrtSendStart, PrrtSubmitPackage>

<LinkTransmitStart, LinkTransmitEnd>

<LinkReceive, DecodeStart>

<PrrtTransmitStart, LinkTransmitStart>

<SendFeedbackStart, SendFeedbackEnd>

<PrrtSendEnd, LinkTransmitStart>

Goals:▶ Filter out false-positive segments▶ Identify code segments worth

optimizing

Detect relevant code segments:1. Compute the correlation between the

segment latency and the E2E latency2. Ignore segments with negative

correlation

9 / 20

Latency Criticality Analysis

0.0 0.1 0.2 0.3 0.4 0.5 0.6latency criticality

<CopyOutputEnd, PrrtDeliver>

<PrrtSubmitPackage, PrrtSendEnd>

<HandlePacketStart, SendFeedbackStart>

<CopyOutputStart, CopyOutputEnd>

<PrrtReturnPackage, HandlePacketEnd>

<DecodeStart, DecodeEnd>

<HandlePacketEnd, PrrtReceivePackage>

<SendFeedbackEnd, PrrtReturnPackage>

<DecodeEnd, HandlePacketStart>

<PrrtSendStart, PrrtSubmitPackage>

<LinkTransmitStart, LinkTransmitEnd>

<LinkReceive, DecodeStart>

<PrrtTransmitStart, LinkTransmitStart>

<SendFeedbackStart, SendFeedbackEnd>

<PrrtSendEnd, LinkTransmitStart>Goals:

▶ Filter out false-positive segments▶ Identify code segments worth

optimizing

Detect relevant code segments:1. Compute the correlation between the

segment latency and the E2E latency2. Ignore segments with negative

correlation

9 / 20

Code Modification Analysis

foo(0); foo(1);

... ...

??

foo(0); foo(1);

... ...

??

Goals:▶ Evaluate the influence of code changes▶ Interferences are complex

Modification impact analysis:1. Evaluate the original and the modified

versions of the protocol2. Apply the k-sample Anderson-Darling-Test

on the traces3. For each code segment, test decides:

“similar” or “different”

10 / 20

Code Modification Analysis

foo(0); foo(1);

... ...

??

foo(0); foo(1);

... ...

??

Goals:▶ Evaluate the influence of code changes▶ Interferences are complex

Modification impact analysis:1. Evaluate the original and the modified

versions of the protocol2. Apply the k-sample Anderson-Darling-Test

on the traces3. For each code segment, test decides:

“similar” or “different”

10 / 20

Code Modification Analysis

foo(0); foo(1);

... ...

??

foo(0); foo(1);

... ...

??

Goals:▶ Evaluate the influence of code changes▶ Interferences are complex

Modification impact analysis:1. Evaluate the original and the modified

versions of the protocol2. Apply the k-sample Anderson-Darling-Test

on the traces3. For each code segment, test decides:

“similar” or “different”

10 / 20

Code Modification Analysis

<PrrtSendPacketStart,PrrtSendPacketEnd> has not changed<CopyOutputStart,CopyOutputEnd> has not changed<SendFeedbackStart,SendFeedbackEnd> has not changed...

<PrrtSubmitPackage,PrrtSendEnd> has changed:2.85±0.91 µs → 4.94±0.58 µs

<LinkReceive,DecodeStart> has changed:2.36±3.80 µs → 3.56±4.88 µs

...

11 / 20

Outline

Status

PRRT

X-Lap & ∆elta

INTspect

Conclusion

12 / 20

Interrupts in Linux

Lack of Data▶ Hardware complexity complicates interrupt latency predictability▶ Unknown overhead for software interrupt handling mechanisms

Lack of Understanding for Delay Sources

▶ Unknown reasons for delays▶ Reasons for delays vary among interrupt handling mechanisms

Lack of Tool Support

▶ Tool for precise and flexible overhead measurements▶ Support of different hardware architectures

13 / 20

Linux Interrupt Handling

</>Device Processor Top Half Bottom Half:

Softirqs/Tasklets

Bottom Half:Workqueues

Application

1 2 3 4

T1

T2

Linux Interrupt Handling

▶ Top half▶ Bottom half

→ Softirq, Tasklet, Workqueue→ Variation in flexibility

INTspect Analysis

▶ Latency and Jitter analysis▶ Software overhead

→ Latency T1 and T2→ Variation in latency & jitter

14 / 20

INTspect Architecture & Evaluation

spawn

call

call

call

sysfs

use

User-Space

Kernel

INTSIGHT

Syscalls

Appl. 1 Appl. 2 INTSPECT

Load

generator

Kernel Module▶ Support all bottom half mechanisms▶ Minimal run-time interference▶ Portable

User-Space Component

▶ Analysis & Control framework▶ Plugin interface

→ load generators, hardware states

Hardware Support

▶ x86▶ ARM → Atmel SAMA5D3 Xplained

15 / 20

Comparison of Bottom Handler Mechanisms

0 10 20 30 40

1e+001e+011e+021e+031e+04

1e+001e+011e+021e+031e+04

1e+001e+011e+021e+031e+04

Delay [µs]

Occ

urre

nces

Softi

rqTa

skle

tW

orkq

ueue

IRQ OOB

16 / 20

Sources of Interference jk

0 10 20 30 40

1e+001e+011e+021e+031e+04

1e+001e+011e+021e+031e+04

1e+001e+011e+021e+031e+04

Delay [µs]

Occ

urre

nces

Non

eEn

tropy

Sche

d.

IRQ OOB

17 / 20

Outline

Status

PRRT

X-Lap & ∆elta

INTspect

Conclusion

18 / 20

Outcome (since April 2018)

Accepted Publications▶ Reif, Stefan; Schmidt, Andreas; Hönig, Timo; Herfet, Thorsten; Schröder-Preikschat,

Wolfgang: “∆elta: Differential Energy-Efficiency, Latency, and Timing Analysis forReal-Time Networks”, International Workshop on Real-Time Networks (RTN), 2018

▶ Herzog, Benedict; Gerhorst, Luis; Heinloth, Bernhard; Reif, Stefan; Hönig, Timo;Schröder-Preikschat, Wolfgang: INTSPECT: Interrupt Latencies in the Linux Kernel,Brazilian Symposium on Computing Systems Engineering (SBESC), 2018.

▶ Reif, Stefan; Gerhorst, Luis; Bender, Kilian; Hönig, Timo: “Towards Low-Jitter andEnergy-Efficient Data Processing in Cyber-Physical Information Systems”, HawaiiInternational Conference on System Sciences (HICSS), 2019

Publications Under Review▶ Schmidt, Andreas; Reif, Stefan; Gil Pereira, Pablo; Hönig, Timo; Herfet, Thorsten;

Schröder-Preikschat, Wolfgang: “Blinded”, Blinded, 2019.

Publications in the Writing

▶ Joint publication based on BarCamp IV results.

19 / 20

Outcome (since April 2018)

Accepted Publications▶ Reif, Stefan; Schmidt, Andreas; Hönig, Timo; Herfet, Thorsten; Schröder-Preikschat,

Wolfgang: “∆elta: Differential Energy-Efficiency, Latency, and Timing Analysis forReal-Time Networks”, International Workshop on Real-Time Networks (RTN), 2018

▶ Herzog, Benedict; Gerhorst, Luis; Heinloth, Bernhard; Reif, Stefan; Hönig, Timo;Schröder-Preikschat, Wolfgang: INTSPECT: Interrupt Latencies in the Linux Kernel,Brazilian Symposium on Computing Systems Engineering (SBESC), 2018.

▶ Reif, Stefan; Gerhorst, Luis; Bender, Kilian; Hönig, Timo: “Towards Low-Jitter andEnergy-Efficient Data Processing in Cyber-Physical Information Systems”, HawaiiInternational Conference on System Sciences (HICSS), 2019

Publications Under Review▶ Schmidt, Andreas; Reif, Stefan; Gil Pereira, Pablo; Hönig, Timo; Herfet, Thorsten;

Schröder-Preikschat, Wolfgang: “Blinded”, Blinded, 2019.

Publications in the Writing

▶ Joint publication based on BarCamp IV results.

19 / 20

Outcome (since April 2018)

Accepted Publications▶ Reif, Stefan; Schmidt, Andreas; Hönig, Timo; Herfet, Thorsten; Schröder-Preikschat,

Wolfgang: “∆elta: Differential Energy-Efficiency, Latency, and Timing Analysis forReal-Time Networks”, International Workshop on Real-Time Networks (RTN), 2018

▶ Herzog, Benedict; Gerhorst, Luis; Heinloth, Bernhard; Reif, Stefan; Hönig, Timo;Schröder-Preikschat, Wolfgang: INTSPECT: Interrupt Latencies in the Linux Kernel,Brazilian Symposium on Computing Systems Engineering (SBESC), 2018.

▶ Reif, Stefan; Gerhorst, Luis; Bender, Kilian; Hönig, Timo: “Towards Low-Jitter andEnergy-Efficient Data Processing in Cyber-Physical Information Systems”, HawaiiInternational Conference on System Sciences (HICSS), 2019

Publications Under Review▶ Schmidt, Andreas; Reif, Stefan; Gil Pereira, Pablo; Hönig, Timo; Herfet, Thorsten;

Schröder-Preikschat, Wolfgang: “Blinded”, Blinded, 2019.

Publications in the Writing

▶ Joint publication based on BarCamp IV results.

19 / 20

Conclusion

jkNetworkjk

✓ PRRT – Predictable protocol✓ Network pacing

Operating System

✓ X-Lap, ∆elta, INTspect✓ Wait-free synchronisation

Reliable Network Atom (RNA)(Embedded Edge-Server, Car, Drone, ...)

+

▶ Integration▶ Optimisation▶ Cross-layer pacing

Thank you for your attention. Questions?

20 / 20

Conclusion

jkNetworkjk

✓ PRRT – Predictable protocol✓ Network pacing

Operating System

✓ X-Lap, ∆elta, INTspect✓ Wait-free synchronisation

Reliable Network Atom (RNA)(Embedded Edge-Server, Car, Drone, ...)

+▶ Integration▶ Optimisation▶ Cross-layer pacing

Thank you for your attention. Questions?

20 / 20

Conclusion

jkNetworkjk

✓ PRRT – Predictable protocol✓ Network pacing

Operating System

✓ X-Lap, ∆elta, INTspect✓ Wait-free synchronisation

Reliable Network Atom (RNA)(Embedded Edge-Server, Car, Drone, ...)

+▶ Integration▶ Optimisation▶ Cross-layer pacing

Thank you for your attention. Questions?

20 / 20