CROSS-LAYERCROSS-LAYERLATENCY-AWARE AND -PREDICTABLELATENCY-AWARE AND -PREDICTABLE

DATA COMMUNICATIONDATA COMMUNICATIONAndreas Schmidt

Apr 07th, 2020

Anywhere on Earth

Doctoral Colloquium

復

1

CYBER-PHYSICAL SYSTEMSCYBER-PHYSICAL SYSTEMS

2 . 1

DISTRIBUTED CYBER-PHYSICAL SYSTEMSDISTRIBUTED CYBER-PHYSICAL SYSTEMS

2 . 2

A REAL-WORLDA REAL-WORLDDISTRIBUTED CPSDISTRIBUTED CPS

Proprietary SolutionEnhanced Shock Burst (ESB)

Our Open SolutionWi-Fi, IP, PRRT

Publication-in-Progress [BSGPH20]0:00 / 0:41

2 . 3

2008 2008

Cyber Physical Systems:Design Challenges [Lee08]

"Passing of time is inexorable"

Most abstractions insufficiently capture timing

Edward A. LeeUC Berkeley

2 . 4

LATENCY-*LATENCY-* AWARENESS AWARENESS

Know what latency you requireKnow what latency you causeShare this with others

PREDICTABILITY PREDICTABILITY

Built upon awarenessImprove confidenceShare confidence with others

2 . 5

STATE OF AFFAIRS IN NETWORKINGSTATE OF AFFAIRS IN NETWORKINGThe Internet

most dominant interconnecting communication systemsbest effort, i.e. no predictable timing (by design)

Link: Transmission latencies known, but not exposed

IP: No timing at all

Transport Layer:TCP: UDP: RTP:

Let's go back to proprietary, closed networks? No! Why?

2 . 6

Internet Architecture [Cla18] provides

INTEROPERABILITYINTEROPERABILITY

2 . 7

RESEARCH QUESTIONRESEARCH QUESTION**

CAN WE MAINTAINCAN WE MAINTAIN INTEROPERABILITY INTEROPERABILITY AND AND AT THE SAME TIME AT THE SAME TIME

ACHIEVEACHIEVE LATENCY-AWARENESS AND -PREDICTABILITY? LATENCY-AWARENESS AND -PREDICTABILITY?

* as you might assume, this is a question too big for a PhD thesis

2 . 8

TODAY'S RESEARCH QUESTIONTODAY'S RESEARCH QUESTION**

CAN WE BUILD CAN WE BUILD TRANSPORT LAYERSTRANSPORT LAYERSWITHWITH

LATENCY-AWARENESS AND -PREDICTABILITY?LATENCY-AWARENESS AND -PREDICTABILITY?

*

2 . 9

OUTLINEOUTLINE

PRRTPRRT X-LAPX-LAP X-PACEX-PACE TTSTTS OUTLOOKOUTLOOK

2 . 10

PRRTPRRT[SH16a, GGG+19]

3 . 1

WHAT IS THE PROBLEM?WHAT IS THE PROBLEM?Full reliability infinite time [Sha48, Fei54]

Most applications...

... have (some) time constraint

... can tolerate (some) missing data

(Today's) transport protocols do not consider that

⇒

3 . 2

FLAVOURS OF RELIABILITYFLAVOURS OF RELIABILITY

Full Partial

3 . 3

PRRT PROTOCOLPRRT PROTOCOLPredictably Reliable Real-time Transport

Fundamental work in [Gor12]

Redesign for control applications since 2015

3 . 4

PRRT'S PLACE IN THE ISO MODELPRRT'S PLACE IN THE ISO MODELDefault

Control/Multimedia App

PRRT

UDP

IP

Any MAC

In GeneralControl/Multimedia App

PRRT

Any Layer withProcess-to-ProcessDatagram Delivery

3 . 5

PRRT CHANNEL MEASUREMENTPRRT CHANNEL MEASUREMENTParameter Approach

Round-trip Time NTP Algorithm [MMBK10]

Delivery Rate IETF Dra� [CCYJ17]

Loss Rate & Correlation Statistics over success-failure sequences

Selective ACKs with feedback

Measurements exposed via API

3 . 6

PRRT'S RECEIVE MODESPRRT'S RECEIVE MODES

3 . 7

PRRT'S FUNCTIONSPRRT'S FUNCTIONSError Control

Adaptive Hybrid Error Coding [Gor12] (not covered in this talk)

Congestion ControlBased on BBR ideas [CCG+16]Utilization & measurementLatency-avoidance

Rate and Flow ControlLater: X-Pace

3 . 8

EVALUATIONEVALUATION

[GGG+19]

3 . 9

USABILITYUSABILITY Policy

Publicly available at

MIT licensed

Technology

C implementation & API

Python API (via Cython)

Rust API (via bindgen)

prrt.larn.systems

3 . 10

http://prrt.larn.systems/

X-LAPX-LAP[RSH+17, RSH+18]

4 . 1

WHAT IS THE PROBLEM?WHAT IS THE PROBLEM? CPS demand predictable latency and timing

Cross-layer, intra-host profiling is requiredvalgrind, wireshark, packetdrill... insufficient

Microsecond regime is increasingly relevant [BMPR17]

4 . 2

STAMPSSTAMPS Wall-Time

CLOCK_MONOTONIC

expensive to capture (≈ 70ns)

Processor Cyclesrdtsc

cheap to capture (≈ 10ns)non-trivial relation to wall-time

4 . 3

WHAT IS X-LAP?WHAT IS X-LAP?X = Cross, Lap = time to do one round

X-Lap is a

cross-layer, intra-host, timing & latency

analysis tool

download at xlap.larn.systems

4 . 4

http://xlap.larn.systems/

STAMPS IN X-LAPSTAMPS IN X-LAP

e.g.: ChannelTransmit

 [ms]                          

 [1]    

↑t = 5 ms ↓

c = 7 156 294

4 . 5

X-LAP IN ACTIONX-LAP IN ACTION

4 . 6

TRACES DATA FORMATTRACES DATA FORMATSN ... ChannelTransmit_T ChannelTransmit_C ...

42 ... 5.012ms ...

43 ... 5.029ms ...

... ... ... .... ...

7 156 294

47 961 303

4 . 7

RUNTIME TRACING CODERUNTIME TRACING CODE

XlapStamp_Cycle(LinkTransmitStart, pkt->seqno);

XlapStamp_TimeValue(ChannelTransmit, timestamp, pkt->seqno); XlapStamp_CycleValue(ChannelTransmit, cyclestamp, pkt->seqno);

XlapStamp_Cycle(LinkTransmitEnd, pkt->seqno);

void send_packet(PrrtPacket* pkt) {1 pace(); // delays sending of data using cross-layer pacing2 wait_for_free_congestion_window_space();3 compute_next_send_time(); // for cross-layer pacing4 char[] bytes = serialize_packet_to_bytes(pkt);5

6 struct timespec timestamp;7 uint64_t cyclestamp;8 send(sock_fd, bytes, & timestamp, &cyclestamp);9

1011

track_outstanding_packet(pkt);1213

}14

4 . 8

CYCLE TO TIMECYCLE TO TIME

4 . 9

PACKET TRACEPACKET TRACE

4 . 10

TRACE JITTERTRACE JITTER

4 . 11

MULTISERIES CORRELATIONMULTISERIES CORRELATION

4 . 12

X-PACEX-PACE[SRGP+19]

5 . 1

WHAT IS THE PROBLEM?WHAT IS THE PROBLEM? Bufferbloat [GN12]

Unpaced Paced

5 . 2

PACEPACE PACINGPACINGintentionally delaying

a transmissionP , [P ] = time

unit of work

5 . 3

CROSS-LAYER PACINGCROSS-LAYER PACINGevery step considers

and adapts its pace to thebottleneck pace

PACED SYSTEMSPACED SYSTEMS

i ∈ [0 : n − 1]P (i)

P (btl)

S is paced iff

∀i, j ∈ [0 : n − 1] :

i < j ⇒ ≥P (i) P (j)

5 . 4

PROPAGATE PACESPROPAGATE PACES

5 . 5

REDUCE 無駄REDUCE 無駄*** muda or waste, originally described by Toyota Production System [Ohn88]

Inventory: only a delivered message is useful

Waiting: a message loses value over time

Overprocessing: as-fast-as-possible causes losses

5 . 6

X-PACE IN PRRTX-PACE IN PRRT

5 . 7

CROSS-LAYER APICROSS-LAYER API

send_sync()Passive

socket.btl_paceActive

5 . 8

EVALUATION APPROACHEVALUATION APPROACHPRRT with X-Pace

vs.

Optimized TCP Variants (CUBIC, BBR) NODELAY, QUICKACKlow_latencySNDBUF & RCVBUF lowwrite() not send()

5 . 9

INTERNET SCENARIOINTERNET SCENARIOtraceroute to 79.199.28.123 (79.199.28.123), 30 hops max, 60 byte packets 1 vlan-herfet-neu.nt.uni-saarland.de (134.96.86.1) 0.400 ms 0.358 ms 0.420 ms 2 c65eb36-win.net.uni-saarland.de (134.96.6.54) 0.328 ms 0.314 ms 0.384 ms 3 cr-dui1-te0-5-0-2-4.x-win.dfn.de (188.1.241.185) 8.687 ms 8.688 ms 8.680 ms 4 cr-fra2-be16.x-win.dfn.de (188.1.144.178) 9.618 ms 9.732 ms 9.727 ms 5 ffm-b12-link.telia.net (213.248.97.40) 9.226 ms 9.563 ms 9.191 ms 6 ffm-bb3-link.telia.net (62.115.142.46) 10.069 ms 9.761 ms 9.704 ms 7 ffm-b4-link.telia.net (62.115.120.6) 9.802 ms 9.792 ms 10.024 ms 8 dtag-ic-319284-ffm-b4.c.telia.net (213.248.93.187) 10.374 ms 10.175 ms 10.17 9 91.23.246.213 (91.23.246.213) 13.230 ms 13.252 ms 13.234 ms 10 * * * ... 30 * * *

5 . 10

RESULTSRESULTS

5 . 11

TRANSPARENT TRANSMISSIONTRANSPARENT TRANSMISSIONSEGMENTATIONSEGMENTATION

[SH16b, SH17b]

6 . 1

WHAT IS A PROBLEM?WHAT IS A PROBLEM? Small receiver buffer flow-limited communication ⇒

6 . 2

TRANSPARENT TRANSMISSIONTRANSPARENT TRANSMISSIONSEGMENTATIONSEGMENTATION

Relay: Transport-Layer Performance-Enhancing-Proxy [ ] RFC3135

6 . 3

https://tools.ietf.org/html/rfc3135

TTS IN ACTIONTTS IN ACTION

6 . 4

IDEAL RECEIVER BUFFERIDEAL RECEIVER BUFFERderived from a theoretical flow control model

= ⋅BrelayRT T1

RT T2Brecv

Λ = = + 1Reff,TTS

Reff,E2E

RT T1

RT T2

6 . 5

EVALUATIONEVALUATION= 4 KiB,    RT = 20 ms,    RT = 5 msBrecv T1 T2

⇒ Λ = 5,     = 4 ⋅ 4KiBBrelay

6 . 6

BENEFITS OF TTSBENEFITS OF TTSLossy Last MileHigh Reordering due to JitterInsufficient Receiver Buffer

6 . 7

OUTLOOKOUTLOOKe.LARN, [SGPH20]

7 . 1

FUTURE WORKFUTURE WORK

Energy-Awareness

StatisticalShaping

MulticastSupport

7 . 2

WRAP-UPWRAP-UP

NETWORKED CPSNETWORKED CPS

PRRTPRRT[SH16a, GGG+19]

X-LAPX-LAP[RSH+17, RSH+18]

X-PACEX-PACE[SRGP+19]

TTSTTS[SH16b, SH17b]

OUTLOOKOUTLOOKe.LARN, [SGPH20]

8 . 1

QUESTIONSQUESTIONS

8 . 2

APPENDIXAPPENDIX

9 . 1

PUBLICATIONS PUBLICATIONS[SH16a] Schmidt, Andreas; Herfet, Thorsten: “Approaches for Resilience- and Latency-Aware Networking“, InternationalSymposium on Networked Cyber-Physical Systems (NetCPS, Poster Session), Munich, Germany, September 2016

[SH16b] Schmidt, Andreas; Lange, Tobias; Herfet, Thorsten: “Low-Latency Multimedia Streaming using OpenNetworking Environments“, International Conference on Computer and Communications (ICCC), Chengdu, China,October 2016

[SH17b] Schmidt, Andreas; Herfet, Thorsten: “Transparent Transmission Segmentation in So�ware-Defined Networks”,IEEE Conference on Network So�warization (NetSo�), Bologna, Italy, July 2017

[RSH+17] Reif, Stefan; Schmidt, Andreas; Hönig, Timo; Herfet, Thorsten; Schröder-Preikschat, Wolfgang: “X-Lap: ASystems Approach for Cross-Layer Profiling and Latency Analysis for Cyber-Physical Networks”, 15th InternationalWorkshop on Real-Time Networks (ECRTS RTN), Dubrovnic, Croatia, June 2017

[RSH+18] Reif, Stefan; Schmidt, Andreas; Hönig, Timo; Herfet, Thorsten; Schröder-Preikschat, Wolfgang: “∆elta:Differential Energy-Efficiency, Latency, and Timing Analysis for Real-Time Networks“, 16th International Workshop onReal-Time Networks (ECRTS RTN), Barcelona, Spain, July 2018

[GGG+19] Gallenmüller, Sebastian; Glebke, René; Günther, Stephan; Hauser, Eric; Leclaire, Maurice; Reif, Stefan; Rüth,Jan; Schmidt, Andreas; Carle, Georg; Herfet, Thorsten; Schröder-Preikschat, Wolfgang; Wehrle, Klaus: “Enabling wirelessnetwork support for gain scheduled control“. 2nd International Workshop on Edge Systems, Analytics and Networking(EdgeSys), Dresden, Germany, March 2019

[SRGP+19] Schmidt, Andreas; Reif, Stefan; Gil Pereira, Pablo; Hönig, Timo; Herfet, Thorsten; Schröder-Preikschat,Wolfgang: “Cross-layer Pacing for Predictably Low Latency”. 6th International IEEE Workshop on Ultra-Low Latency inWireless Networks (ULLWN), Paris, France, April 2019

[SGPH20] Schmidt, Andreas; Gil Pereira, Pablo; Herfet, Thorsten: "Predictably Reliable Real-time Transport Services forWireless Cyber-Physical Systems", IFAC World Congress, Berlin, Germany, July 2020

9 . 2

REFERENCES REFERENCES[Sha48] Claude Elwood Shannon. A Mathematical Theory of Communication. 1948.

[Fei54] Amiel Feinstein. A new basic theorem of information theory. 1954.

[Lee08] Edward A. Lee. Cyber Physical Systems: Design Challenges. In Proceedings of the 11th IEEE InternationalSymposium on Object and Component-Oriented Real-Time Distributed Computing (ISORC), 2008.

[MMBK10] D. Mills, J. Martin, J. Burbank, and W. Kasch. Network time protocol version 4: Protocol and algorithmsspecification. RFC 5905, RFC Editor, June 2010. http://www.rfc-editor.org/rfc/rfc5905.txt.

[GN12] Jim Gettys and Kathleen Nichols. Bufferbloat: Dark Buffers in the Internet. Communications of the ACM, 55(1):57– 65, 2012.

[Gor12] Manuel Gorius. Adaptive Delay-constrained Internet Media Transport. PhD thesis, Saarland University, 2012.

[CCG+16] Neal Cardwell, Yuchung Cheng, C Stephen Gunn, Soheil Hassas Yeganeh, and Van Jacobson. BBR: Congestion-based congestion control. ACM Queue, 14(5):50, 2016.

[BMPR17] Luiz Barroso, Mike Marty, David Patterson, and Parthasarathy Ranganathan. Attack of the Killer MicrosecondsCommunications of the ACM, 60(4):48–54, March 2017.

[CCYJ17] Yuchung Cheng, Neal Cardwell, Soheil Hassas Yeganeh, and Van Jacobson. Delivery rate estimation - IETFDRAFT, 2017.

[Cla18] David D Clark. Designing an Internet. MIT Press, 2018.

9 . 3

IMAGE SOURCES IMAGE SOURCESEdward Lee

9 . 4

https://ptolemy.berkeley.edu/~eal/