Upload
amaya-wand
View
212
Download
0
Tags:
Embed Size (px)
Citation preview
www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009
Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA
Wilfried Steiner
TTTech Computertechnik AG
MAPLD 2009 – Session A
CoMMiCS
Page 2www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Problem Statement
1. Reliable Synchronization
2. Multi-Hop Network Topology
3. TTEthernet - Realization in FPGA
4. Model-Based Development
TTE
1588
1588
Eth
TTE
TTE
TTE
Eth
TTE
TTE
TTE
TTE
TTE
TTE
TTE
Eth
www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009
Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA
Page 4www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Synchronization Strategy
Perfe
ct Cloc
k
Real Time
Com
pute
r T
ime
Slow Clock
Fast Clock
R.int
Mes
sage
Exc
hang
e
R.int
Mes
sage
Exc
hang
e
Clock Synchronization Service
Startup/Restart Service
Clock Synchronization Service is executed during normal operation mode to keep the local clocks synchronized to each other.
Startup/Restart Service is executed to reach an initial synchronization of the local clocks in the system.
Integration/Reintegration Service is used for components to join an already synchronized system.
Clique Detection Services are used to detect loss of synchronization and establishment of disjoint sets of synchronized components.
Page 5www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Two-Step Clock Synchronization Service
For now, let us assume that we operate in a single-hop network:
End Systems operate as Synchronization Masters/Clients.
Switches are configured as Compression Masters.
Synchronization Strategy operates in two steps for clock synchronization during normal operation mode.
Step1: Synchronization Masters send synchronization messages to Compression Masters.
Step2: Compression Masters send synchronization messages back to the Synchronization Masters and Clients.
SynchronizationMaster
Compression Master
ID 1 ID 2
ID 3
Step 1
SynchronizationMaster
SynchronizationClient
SynchronizationMaster
SynchronizationMaster
Compression Master
ID 5 ID 5
ID 5ID 5
Step 2
SynchronizationMaster
SynchronizationClient
SynchronizationMaster
ES 101dispatch
permanence
permanence
ES 102
ES 103
ES 104
Sw
itch
201
791 792 794
permanence
permanence
permanence380
380
380
380
380
380
380
380
dispatchpermanence
dispatchpermanence
dispatchpermanence
dispatch
dispatch
dispatch
dispatch
1110
1110
1110
1110
1000
0100
0010
0001
1000
0100
0010
1110
1110
1110
1110
scheduleddispatch_point_in_time round_trip_delay acceptance_window
dispatch_point_in_time
calculation of dispatch_point_in _time
collection oflocal clock readings
re-synchronization interval x x + 1x - 1
Page 6www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Four-Step Startup/Restart Service
Port 1send
receive
sendreceive
sendreceive
sendreceive
Port 2
Port 3
Port 4Fau
lty S
witc
h 20
1
Port 1send
receive
sendreceive
sendreceive
sendreceive
Port 2
Port 3
Port 4Cor
rect
Sw
itch
202
ES 101
sendreceive
Fau
lty E
S 1
01
sendreceive
ES 102
sendreceive
sendreceive
ES 103
sendreceive
sendreceive
ES 104
sendreceive
sendreceive
CS
CS
CS
CS
CA
CA
CA
CA
CA
CA
CA
CA
CA
CA
CA
CA
3101
3101
3101
3101
CSO
4103
4202
4202
4202
4202
4103
4103
4103
4202
4202
4202
4202Step1: Synchronization Masters send out Coldstart Frames (CS)
Step2: Compression Masters forward all CS frames to all ports
Step3: Each Synchronization Master that receives a CS frame will answer with a Coldstart Acknowledgement Frame (CA)
Step4a: Compression Masters will forward all CA frames to all ports (for multiple failure tolerant configurations).
Step4b: Compression Masters will forward only one compressed CA frame (for single failure tolerant configurations).
Page 7www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Formal Verification Activities (Synchronization Strategy)
TTEthernet Executable Formal Specification Using symbolic and bounded model checkers sal-smc and sal-bmc
Focus on Interoperation of Synchronization Services (Startup, Restart, Clique Detection, Clique Resolution, abstract Clock Synchronization)
Verification of Lower-Level Synchronization Functions Permanence Function
verified with the infinite-bounded model checker sal-inf-bmc
using disjunctive invariant and k-induction
Compression Function verified with the infinite-bounded model checker sal-inf-bmc
using abstraction and 1-induction
Formal Methods have been applied as early as in the requirements capturing phase
Finalization and Completion of the formal assessment within the CoMMiCS Project
Complexity Management for Mixed-Criticality Systems
European Communities FP7 project [FP7/2007-2013] no. 236701]
CoMMiCS
www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009
Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA
Page 9www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
TTEthernet in a Nutshell
… is all about smart ways to integrate data flows with differentcharacteristics in a single communication infrastructure.
While various groups started with standard Ethernet and aimat enhancing it with real-time and fault tolerance properties,TTTech comes from the opposite direction:
Our know-how is in real-time and fault-tolerant data flows.
We are applying our know-how on top of Ethernet and optimize ourconcepts with respect to Ethernet features (switches, full-duplex, etc).
In principle we can enable any carrier protocol with time-triggered technology, …
… but there are some good reasons for using Ethernet.
Page 10www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
The Motivation for Ethernet
Ethernet hardware is low cost.
Ethernet is a well-established open-world standard and very scaleable.
The OSI reference model gives a well-structured classification of concepts that can be built on top of Ethernet.
Existing tools can be leveraged as cost-efficient diagnosis tools.
As all messages in TTEthernet are standard Ethernet compliant, existing tools can be leveraged for time-triggered messages as well.
Standard web servers can be leveraged for maintenance and configuration.
TTEthernet allows hosting of IEEE 1588 clock synchronization clients.
Engineers learn about Ethernet at school.
Ethernet compatibility enables the usage of technology that is established, tested, and verified.
Page 11www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Different applications have different communication needs: fault, tolerance, real time, high speed, high bandwidth, low latency, low jitter, active redundancy, hot standby, …
TTEthernet provides following traffic classes for applications:
Time-Triggered (TT): Bandwidth in TTEthernet networks can be highly utilized due to the possibility of strictly deterministic (vs. probabilistic) traffic scheduling of high-priority periodic traffic.
Rate-constrained (RC): Event-triggered traffic with priority levels (ARINC 664) and transmission guarantees.
Non-critical standard Ethernet (BE): Low priority traffic (e.g. data download) served during network idle times; can also be scheduled explicitly.
Dataflow in TTEthernet
TTEthernet allows applications with different communication requirements to share a single physical network.
Page 12www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
TTEthernet Integrated Dataflow
TT TTTT TT TTTT TT
3ms cycle
2ms cycle
3ms cycle 3ms cycle
2ms cycle 2ms cycle 2ms cycle
6ms Cluster Cycle
RC BE BE BE RC BE t
Sender
1 Switch/RouterReceiver
Sender
2
TT TT TT
3ms cycle 3ms cycle 3ms cycle
BE BE BE t
TT TT TT
2ms cycle 2ms cycle2ms cycle
BE BE RC BE
t
Dataflow – Integration - Time-Triggered (TT)
- Rate-Constrained (RC) - Standard Ethernet (BE)
Protocol Control Frames (PCFs) are transmitted on the same physical network as dataflow.
Page 13www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Integration Problem
When two (or more) messages compete for relay to the sameoutgoing port, the switch (or ES) has to serialize these messages.
If there are messages of same priority the messages will be serviced according First-Come First-Served.
What happens if there is a Data message (Data) in relay, when a Protocol Control Frame (PCF) becomes ready for relay?
If a Data message is relayed by a switch when a PCF arrives, the PCF message is delayed until the relay process of the Data message is finished.
Hence, in the worst case the PCF is delayed for a maximum sized Data message (plus other queued PCFs).
Protocol Control Frames record their delay through the network.
All components in the network that impose a delay on a PCF will add this delay into a field in the PCFs.
PCF
Data
PCF Data
Data PCF
PCF Data
Preemption:
Timely Block:
Shuffling:real-time
Contention:
Page 14www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Single-Hop Synchronization Flow
Compression MasterSynchronization
Master/Client
Frame Sendsm_send _pit
SynchronizationMaster
Frame Dispatchsm_dispatch _pit
Real Time
Local Clock
Frame Receivecm_receive _pit
Frame Scheduled Receivecm_scheduled_receive _pit
Frame Permanencecm_permanence _pit
Frame Compressedcm_compressed _pit
Frame Dispatchcm_dispatch _pit
Frame Sendcm_send _pit
Frame Receivesmc_receive _pit
Frame Permanencesmc_permanence _pit
Frame Scheduled Receivesmc_scheduled_receive _pit
(2 * max_transmission_delay + compression_master_delay)
(2 * max_transmission_delay + compression_master_delay) + clock_corrDelays
Transparent Clock
TC TC
Acceptance Window Acceptance Window
Compression Function
Message PermanenceFunction
Message PermanenceFunction
1
2
3
4
5
6
7
8 9
11
10
The (Dispatch, Permanence) – Pair hides the network jitter for PCFs almost entirely.
~constantdynamic
Page 15www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Multi-Hop Synchronization Flow
Compression Master Synchronization Client
Frame Sendsmc_send _pit
Synchronization Client
Real Time
Local Clock
Frame Receivecm_receive _pit
Frame Scheduled Receivecm_scheduled_receive _pit
Frame Permanencecm_permanence _pit
Frame Compressedcm_compressed _pit
Frame Sendcm_send _pit
Frame Receivesmc_receive _pit
Frame Permanencesmc_permanence _pit
Frame Scheduled Receivesmc_scheduled_receive _pit
Transparent Clock
TC TC
Frame Receivesmc_receive _pit
Frame Sendsmc_send _pit
TC
From Synchronization
Master
To Synchronization
Master/Client
Frame Dispatchcm_dispatch _pit
Compression Function
Acceptance Window Acceptance Window
Message PermanenceFunction
Message PermanenceFunction
12
TC+
Multi-hop requires only a small add-on to the existing mechanism: an intermediate switch will not only consume the PCF, but also forward it.
www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009
Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA
Page 17www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Synchronization Master / Client i
FilterFrame
PermanenceSync StateMachine
Protocol Control Frames (PCFs) from the Compression Master flow in from the left.
PCFs produced by the Synchronization Master flow out on the right.
Filter: removes PCFs that are semantically incorrect (e.g. wrong type, outdated, etc.).
Frame Permanence: holds PCFs in buffers until their age equals the configured max. transparent clock value (respecting the transparent clock field as part of their age).
Sync State Machine: this module will process all the PCFs that made it past the Frame Permanence module as specified by the synchronization services. In particular this module is in charge of maintaining the local copy of the synchronized cluster time.
Page 18www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Synchronization Master / Client ii
The size of the Synchronization Unit depends upon others on the number of Synchronization Masters that have to be supported.
For sixteen Synchronization Masters:
the complete block (incl. interfaces to MACs) takes ~5,000 flip-flops (including diagnostics) and ~7,000 ALUTs
this is 12% and 15%, respectively, of the total size of a 3 channels GBits/s End System controller in an Altera Stratix III FPGA.
For four Synchronization Masters:
the block needs ~2,200 flip-flops and ~3,500 ALUTs.
Page 19www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
TTEEvaluation System (2 channels, 1 Gbit/s)
Page 20www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
TTEEvaluation System
Takes you from first evaluation steps to full-blown prototypes
4 Linux-based PCs with triple channel TTEthernet capability
Optical Gigabit Ethernet (1000Base-X) compliant interfaces
One or more high-performance TTEthernet switches with 4 ports and configuration, monitoring and diagnostic interfaces
Optical cables supplied for easy connection to existing LAN infrastructure or directly to the user’s development PC (needs 1000Base-X interface)
TTEthernet end system software libraries offering an API (application programming interface) to the TTEthernet protocol hardware
TTEthernet tools for network planning, schedule inspection, and configuration
PC-based monitoring tool supports logging/recording for analysis of network traffic and network-based applications
Power supply 110-240 VAC
Example application, documentation
Page 21www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
TTEthernet Hardware Products - Summary
TTEDevelopment Switches TTEDevelopment Switch 1Gbit/s
TTEDevelopment Switch 100Mbit/s
TTEEnd Systems TTEEnd System PCIe
TTEEnd System PMC
TTEEnd System XMC
TTEEvaluation Systems TTEEvaluation System 1Gbit/s
TTEEvaluation System 100 Mbit/s
TTETest Equipment TTEMonitoring Switch 1Gbit/s
TTEMonitoring System
www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009
Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA
Model-Based Development
Page 23www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
TTEthernet Executable Formal Specification i
SM1
SMk
SM2
CM1
CM2
CONNECTIONS
…
Current State
Ideal State
DIAGNOSIS
= or /=
FilterFrame
PermanenceCompression
MasterFrame
PermanenceSync StateMachine
FilterFrame
PermanenceSync StateMachine
www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved; may be published with permission by MAPLD 2009
Reliable Synchronization for Multi-Hop Networks and its Realization in FPGA
Summary & Conclusion
Page 25www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
Summary
Reliable Synchronization We showed the two-step clock synchronization service and the four-step
startup/restart service and discussed their failure-tolerance capabilities.
for Multi-Hop Networks We discussed the problem of integrating Control-Flow and Dataflow on the same
physical network.
We showed that the permanence function is a very simple mean to establish a strong interface with clean semantics for the synchronization services.
and its Realization in FPGA We introduced the conceptual design and discussed the implementation details in an
Altera Stratix III FPGA.
Model-Based Development We introduced the Executable Formal Specification and sketched how the worst-case
stabilization time of TTEthernet has been calculated.
Page 26www.tttech.com Copyright © 2009, TTTech Computertechnik AG. All rights reserved;may be published with permission by MAPLD 2009
QUESTIONS?