Linux Foundation – End User Summit May 1, 2012 Precision ... · Time From PTP Master on network...

Precision Time Protocol, andSub-Microsecond SynchronizationMike KravetzIBM Linux Technology Centerkravetz@us.ibm.com

Linux Foundation – End User Summit May 1, 2012

Agenda

● Background/History leading to Precision Time Protocol

● Results of Experimentation

● Building out a Complete PTP Solution

IBM LTC Low Latency Team Involvement With PTP

Customer Queries

RealTime Linux Suitable for Financial Environments?

High Frequency Trading?

IBM LTC RealTime Team

Real requirement is Low Latency, rather than RealTime

Customer Queries

Time Synchronization in Financial Environments?

IBM Low Latency Team

Exploration of Precision Time Protocol

High Frequency Trading Environment

Market Feeds

Network Switch

Feed Handlers

Algorithm Engines

Trade Management

Time Synchronization

Current Methods

Synced External Clocks● GPS or Similar time source attached to each system● New Connection Infrastructure● New Management Issues● Expense

Network Time Protocol – NTP● Linux/Unix Standard

● Works “everywhere”● Designed for WANs● Sync Accuracy may not be good enough for Low Lat environments

Emerging Method

Precision Time Protocol - PTP● Uses existing Network Infrastructure● Designed for LANs● Targeting sub-microsecond accuracy

GPS for Time Synchronization (Receiver per-System)

GPS for Time Synchronization (Distribute Signal)

Network Time Protocol

USER SPACE

KERNEL

HARDWARE

SYSTEMCLOCK

ADAPTER

NETWORK

SYSTEM CALL API

NetworkStack

Time From Network Time Sources

Network Time Protocol – The Original Standard

● Hierarchical Time Distribution amongst hosts● Intended for synchronization of hosts on WAN● Designed to sync clocks within a few milliseconds● Poll interval 64 sec default 8 sec min

● (a lifetime at fast LAN timeframes)● User space daemon slews system clock

● (slow operation, e.g. adjtime())● Network latencies, scheduling latencies, clock drift, kernel preemption, all introduce inaccuracies

Jitter Source

adjtime()

NTP CLIENT

gettime()

USER SPACE

KERNEL

HARDWARE

SYSTEMCLOCK

ADAPTER

NETWORK

SYSTEM CALL API

Network Stack

Time From Network Time Sources

Network Time Protocol – The Current Standard

Current Implementation

● SO_TIMESTAMP socket option● Time noted in Network Stack (or lower)

Jitter Source

adjtimex()

NTP CLIENT

timestampfrom packet

Measuring Clock Deltas – A Example of Perfect

Environment

● Protocol Used● Load

● Beginning 15 minutes after start● Range : 0 to 0 nanoseconds● Mean : 0.0● StdDev: 0.0

Network Time Protocol – LAN (minimal load)

Environment

● Simple 10Gb LAN for NTP traffic● Poll interval 8 sec● Minimal CPU and Network Activity

● Beginning 15 minutes after start● Range : -9 to 11 microseconds● Mean : 0.154792● StdDev: 3.352142

Adding CPU Load

Kernel Build

while true; domake cleanmake -j 32

Network Time Protocol – LAN (CPU load)

Environment

● Simple 10Gb LAN for NTP traffic● Poll interval 8 sec● Heavy CPU Activity, Minimal Network Activity

● Beginning 15 minutes after start● Range : -108 to 21 microseconds● Mean : 0.335034● StdDev: 6.050032

Precision Time Protocol (PTP)

Assumptions

● Master to Slave Transit time = equals = Slave to Master Transit Time● Delays (Transit times) Remain Constant● Master and Slave can accurately compute time to send or receive messages.

USERSPACE

KERNEL

HARDWARE

ADAPTER

NETWORK

SYSTEM CALL API

NetworkStack

Time From PTP Master on network

Precision Time Protocol (PTP) – SW Only Implementation

PTP – SW Only

● Intended for host synchronization in LANs● Hierarchical Time Distribution● Slaves sync to Master● Master syncs to GPS or similar time source● Query for time sync default is 1 sec● Uses timestamps on packets to compute latency information

Sources of jitter similar to NTP● Packets timestamped in Network stack● Socket option SO_TIMESTAMP

Requires● Linux kernel 2.6.30 or later● OpenSource PTP daemon

Jitter Source

SYSTEMCLOCK

PTP CLIENT

Precision Time Protocol SW Only – LAN (Minimal Load)

Environment

●Simple 10Gb LAN for PTP traffic● Sync interval 1 sec● Minimal CPU and Network Activity

● Beginning 15 minutes after start● Range : -783 to 807 nanoseconds● Mean : 18.975601 ● StdDev: 248.732998

Precision Time Protocol SW Only – LAN (CPU Load)

Environment

●Simple 10Gb LAN for PTP traffic● Sync Interval 1 sec● Heavy CPU Activity, Minimal Network Activity

● Beginning 15 minutes after start● Range : -45917 to 199103 ns● Mean : -48.299971● StdDev: 5793.242824

Adding Network Load

SCP and NETPERF

while true; do netperf -l 1 -H 10.0.0.11 # 7.2Gb (1 Second) scp -l 102400 to.11 10.0.0.11:work # 100Mb (10 Seconds)done

Precision Time Protocol SW Only – LAN (CPU & Net Load)

Environment

●Simple 10Gb LAN for PTP traffic● Sync interval 1 sec● Heavy CPU and Network Activity

● Range : -115106 to 578730 ns● Mean : -0.637253● StdDev: 13670.112245● Delay Updates Aborted : 13507

USER SPACE

KERNEL

HARDWARE

ADAPTER

NETWORK

SYSTEM CALL API

Precision Time Protocol (PTP) - Adapter TimeStamp Support

PTP - Adapter Support

● Network adapter timestamps packets● Adapter has separate clock

● Closer to events, less jitter● Socket SO_TIMESTAMP

● Hardware option● microsecond or better synchronization

Requires● Linux kernel 2.6.30 or later● Adapter with PTP support● PTP daemon modified for HW timestamping

Jitter Source

Time From Master Sources

HW packet timestamp

adjtimex()

NetworkStack

SYSTEMCLOCK

New Problem – Multiple Clocks on Same System

Clock on Adapter

Provides TimeStamp for:● Incoming Packets● Outgoing Packets

System Clock

Provided view of Time to:● Kernel● Applications

USER SPACE

KERNEL

HARDWARE

ADAPTER (Clock)

SYSTEM CALL API

Multiple PTP Domains

Jitter Source

adjtimex()new API

NetworkStack

SYSTEMCLOCK

PTP Adapter Clock Domain PTP System Clock Domain

USER SPACE

KERNEL

HARDWARE

ADAPTER

NETWORK

Linux PTP Clock Support - Future Direction

PTP – Linux PTP Clock support

● Framework for Accessing PTP Clocks● User API for PTP clock access● PTP daemon access PTP Clocks in standard way● No need for special daemons● Applications can use PTP clock

Requires● Current Linux kernel 3.0.+● Driver for PTP clock● PTP daemon modified for PTP Clock

HW packet timestamp

NetworkStack

SYSTEM CLOCK

ANYAPPLICATION

PTP Clock API

PTP Adapter Clock – LAN (Minimal Load)

Environment

● Adapter Clock deltas from Master● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Minimal CPU and Network Activity

●Beginning 15 minutes after start● Range : -59 to 60 nanoseconds● Mean : 0.061463● StdDev: 16.908153

PTP System Clock (from Adapter) – LAN (Minimal Load)

Environment

● System Clock deltas from Adapter● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Minimal CPU and Network Activity

● Beginning 15 minutes after start● Range : -438 to 336 nanoseconds● Mean : 9.825280● StdDev: 89.881447

PTP System Clock (from Master) – LAN (Minimal Load)

Environment

● System Clock deltas from Master● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Minimal CPU and Network Activity

PTP Adapter Clock – LAN (CPU Load)

Environment

● Adapter Clock deltas from Master● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Heavy CPU Activity, Min Net Activity

● Beginning 15 minutes after start● Range : -58 to 63 nanoseconds● Mean : -0.052723● StdDev: 17.305986

PTP System Clock (from Adapter) – LAN (CPU Load)

Environment

● System Clock deltas from Adapter● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Heavy CPU Activity, Min Net Activity

PTP System Clock (from Master) – LAN (CPU Load)

Environment

● System Clock deltas from Master● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Heavy CPU Activity, Min Net Activity

PTP Adapter Clock – LAN (CPU & Net Load)

Environment

● Adapter Clock deltas from Master● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Heavy CPU Activity, Heavy Net Activity

● Beginning 15 minutes after start● Range : -64 to 80 nanoseconds● Mean : -0.031430● StdDev: 17.836507

PTP System Clock (from Adapter) – LAN (CPU & NET Load)

Environment

● System Clock deltas from Adapter● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Heavy CPU Activity, Heavy Net Activity

PTP System Clock (from Master) – LAN (CPU & NET Load)

Environment

● System Clock deltas from Master● Simple 10Gb LAN for NTP traffic● Sync interval 1 sec● Heavy CPU Activity, Heavy Net Activity

● Beginning 15 minutes after start● Range : -326 to 178 nanoseconds● Merge : 0.586328● StdDev: 42.471485

Same Test – Bad Network Cable

Environment

● Bad Network Cable● Do not try at home :)

● Range -7,320,339 to 82,032,290 ns● Packets delayed up to 5,444,813 ns

PTPD SLAVE

ADAPTER

PTP – Transparent Clock/Switch Support Addressing Jitter in the LAN

SWITCH

Transparent

Transit Time Correction

ADAPTER

PTP – Transparent Clock

● Accounts for Switch delays● Delay time used by PTP algorithms

Requires● Switch with PTP Transparent Clock Support

SYSTEMCLOCK

Jitter Source

PTP MASTER

PTP – Boundary Clock/Switch Support. Bridging Multiple LANs

SWITCH(Transparent)

PTP MASTER

PTP – Boundary Clock

● Multiple PTP ports● Slave on one LAN/ port● Master on one LAN/PTP port● Ideal topology minimizes Boundary Clock Traversals

Requires● Switch with PTP Boundary Clock Support

PTP SLAVE

SWITCH(Transparent)

PTP SLAVE

SWITCH(Boundary)

Slave Master

Acknowledgments

■ Solarflare Corporation– PSFN5322F 10GbE Adapters with PTP Support

■ Symmetricom Corporation– SyncServer S350 Time Server with PTP support

Legal Statement

■ This work represents the view of the author and does not necessarily represent the view of IBM.

■ IBM is a registered trademark of International Business Machines Corporation in the United States and/or other countries.

■ UNIX is a registered trademark of The Open Group in the United States and other countries

■ Linux is a registered trademark of Linus Torvalds.

■ Other company, product, and service names may be trademarks or service marks of others.

Linux Foundation – End User Summit May 1, 2012 Precision ... · Time From PTP Master on network...

Documents

IEEE 1588 Precision Time Protocol (PTP) for Mellanox Onyx® Switches Design Guide · 2020-04-24 · Time transfer requirements are evolving towards better precision and accuracy

CAMBIUM NETWORKS - Advantec · 3 PTP Overall Portfolio PTP 550 Introduction PTP 550 Features DSO Spectrum Analyzer Channel Bonding PTP 670 vs PTP 550 Product launch information and

Tutorial on the Precision Time Protocol taken from IEC 62439-3...2021/02/12 · The basic principle of PTP is shown in Figure 2. – 4 –Tutorial on PTP from IEC 62439 IEC 2021

High-precision, highly efficient binder discharge Binding ...€¦ · Binding agent spreader SW 12 MC Binding agent spreader SW 16 MC Binding agent spreader SW 20 MC Container Container

Protein tyrosine phosphatases PTP-1B and TC-PTP play

Configuring Precision Time Protocol - cisco.com · Precision Time Protocol (PTP) is defined in IEEE-1588 as Precision Clock Synchronization for Networked Measurements and Control

eCPRI Intel® FPGA IP User Guide...Precision Time Protocol (1588 PTP) hardware timestamp. Full hardware support, and required 1588 PTP software stack. • Supports 25 Gbps and 10 Gbps

Wireless Fabric Connectivity Solutions - doubleradius.com · PTP AP: 120 Subscribers PTP Bridge-in-a-Box: PTP SM PTP SM PTP SM PTP ... Products Supported • For Ubiquiti® XW/XM

PTP Overview

PTP Services

Charles River Proficiency Testing Program (PTP) | … Charles River Proficiency Test Program (PTP) ... have been using the Charles River PTP to ensure ... PTP; endosafe Created Date:

INSTALLATION GUIDE PTP 820G and PTP 820G Assured

CAMBIUM NETWORKS - Advantec · PTP Overall Portfolio PTP 550 Introduction PTP 550 Features DSO Spectrum Analyzer Channel Bonding PTP 670 vs PTP 550 ... • ARQ Protocol • Metal

PTP Presentation

The Precision Timing Protocol - Internet2...Sep 23, 2016 · – Spidercloud • Cellular rebroadcast, proprietary implementation of PTP – Service provider monitoring • Y.1731

PTP#200#SERIES#SOLUTIONS# - DoubleRadius · PTP#200#SERIES## Platform# Feature PTP 200 PTP 230 PTP 250 Models PTP 49200 PTP 54230 PTP 58230 PTP 5X250 RF Bands (GHz) 4.9 Public Safety

Introduction to PTP PTP Basics - Nettimelogic GmbH Basics.pdfWhat is PTP? • Precision Time Protocol standardized in IEEE1588-2008 (or IEC61588-Ed.2) • Packet based time synchronization

AppNote - PTP IEEE1588 with Prosilica GT - aprolink.jp · IEEE1588 Precision Time Protocol (PTP) manages clock synchronization of multiple devices across an Ethernet network. Once

SOLUTIONS - Netwell Ltd · 2020. 5. 7. · SOLUTIONS E-band (71-76GHz, 81-86GHz) PtP PtP PtP PtP PtP PtP PtP PtP PtP PtMP PtMP FDD FDD FDD FDD TDD TDD TDD TDD TDD 13W13W license license

Advanced Guiding Systems for Web Handlingttemsa.com/pdf/2016GuidingSolutions.pdf · 2017. 2. 27. · PTP IEE-1588 Precision Time Protocol • Incorporates the latest IEEE-1588 Precision