Nanosecond Accuracy Timing Solution for Data Center …d2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKDCT-2215.pdf · Nanosecond Accuracy Timing Solution for Data Center Fabric BRKDCT-2215

© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public

Nanosecond Accuracy Timing Solution for Data Center Fabric BRKDCT-2215

1


Agenda

Timing requirements and solution

IEEE 1588 clock type and how does it work

IEEE 1588 message details

IEEE 1588 functions and operation on Nexus switch

Nexus 3000 PTP performance benchmarking

PTP network design considerations

PTP performance validation in solution testing lab


Precision Timing is Fundamental

Clock is the one of the most important component of any modern electrical system

Network and applications also need accurate timing information to correlate all the events

‒ Network diagnostics

‒ Application transactions

‒ Digital forensics

‒ Data and event-log analysis are all based on accurate timing information

Accurate timestamp is also a mandatory requirement for compliance


Timing Challenge for Today’s Network

Financial institutes are building faster trading systems

‒ To gain competitive advantage to increase profitability

Switches can forward the packet in a matter of microsecond

‒ Cisco Nexus 3000 ultra low latency switch

HPC clusters need accurate timing synchronization solution

‒ Some HPC applications have strict timing synchronization requirement, such as performance analysis on parallel process. This requirement becomes more important today as the cluster size is getting bigger and bigger


Timing Challenge for Today’s Network

Switches can forward the packet in a matter of microsecond

• Nexus 3000 ultra low latency switch can forward the packet (64B) in 800ns!

© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 6 Cisco Confidential 6 © 2010 Cisco and/or its affiliates. All rights reserved.

Timing Challenge for Today’s Network Electronic Trading Environment

~100us

Sub 100us !


Need an Accurate Timing Synchronization Solution

7

Have a precise local only clock is not practical ‒ It can be very expensive

‒ Sill have the risk to run out of sync

Time need to be precise and synchronized ‒ Everyone will have the same timing information

‒ Solve the problem for distributed system

‒ Should be easily distributed

‒ Should meet the accurate requirement


Comparison of Different Timing Source

NTP (Network Time Protocol) o Traditionally NTP is used to provide timing information on packet network.

However, its accuracy usually is limited at milliseconds level

GPS (Global Position System) o GPS with proper installation and calibration can provide 100ns accuracy. However,

It requires separate network and media (not Ethernet). It’s costly, not straight forward to deploy in a large scale

IRIG (And other serial timing protocols) o Legacy protocol mostly used in environment requires millisecond accuracy. Also

facing the same challenge as GPS

PTP (Precision Timing Protocol) o Defined in IEEE1588, distributed time synchronization protocol for packet network.

Can provide nanosecond accuracy.


IEEE 1588v2 Precision Time Protocol

IEEE 1588 Precision Time Protocol (PTP) is a highly accurate distributed time synchronization protocol for packet network

IEEE 1588-2008, as known as IEEE 1588v2 or PTPv2 is the latest IEEE 1588 standard.

• Can direct map to Ethernet, or UDP IPv4.

• Packet based timing distribution and synchronization.

• Nanosecond to sub-microsecond accuracy

• Low administrative effort, easy to manage and maintain

• Low cost and low resource use, works on high-end or low-end device

• Support redundant and fault-tolerant

• No need to implement costly GPS or other dedicated timing network


Industry Benefitted by IEEE 1588v2

Telecommunications PTP Telecom Profile ITU G.8265.1 ITU G.8275.1

Industry automation (Default profile)

Financial

Smart grid (Power profile, IEEE C37.238)


IEEE 1588v2 For Financial and DC Network

Industries

• Financial Trading • High Performance Computing • Massively Scalable Data Centers

Applications @ Server

• Application latency measurement • Event & Log correlation between systems • Measure “time to trade”

• Switch & Hop-by-Hop Latency Measurements • Accurate Timestamp of Monitored Traffic for ERSPAN

Applications @ Switch


Agenda

Timing requirement and solution



IEEE 1588 features supported on Nexus switch and configurations

Nexus 3000 IEEE 1588 performance benchmarking

IEEE 1588 PTP network design considerations



IEEE 1588v2 Clock Type

Master Clock (MC)

• Master clock provide reference time for one or more slave clocks.

• A grandmaster (GM) is the highest-ranking clock within its PTP domain, it’s the primary reference source (PRS) for all other PTP elements

Slave Clock (SC) • Each slave clock synchronizes itself to the master clock

Ordinary Clock (OC) • Has a single PTP port in a domain and maintains the timescale used in

the domain. It could be a master clock, or a slave clock


IEEE 1588v2 Clock Type – Cont’d

Boundary Clock (BC)

• Has multiple PTP ports in a domain and maintains the timescale used in the domain. It could be a master or slave clock in the same time on different port

Transparent Clock (TC) • A device that measures the time taken for a PTP event message to

transit the device, and compensate the packet delay by updating the timestamp.

• End-to-End transparent clock

• Peer-to-Peer transparent clock


Master Clock Selection

Based on IEEE 1588v2 BMCA

Announce messages are exchanged among potential grand masters. BMCA (Best Master Clock Algorithm) runs locally on each port. It compares its own data set with the received data set to determine the best clock based on the attributes with following priority:

1. Priority1(0-255)

2. Class (clockClass)

3. Accuracy (clockAccuracy)

4. PTP variance (offsetScaledLogVariance)

5. Priority2 (0-255)

6. Identifier (IEEE EUI-64) (clockIdentity)

The number of hops between local clock and master is also used to as a tie-breaker

The status of the port can be master, slave or passive


Hierarchy Network Clock Topology

1 Elect the grand master, form a master-slave hierarchy. Grand master is selected based on Best Master Clock selection Algorithm (BMCA). (Master clock 1 is selected as Grand Master in the diagram)

2 Each slave clock synchronizes itself to the master clock

Master Clock 1 Master Clock 2

Boundary Clock (1) Nexus switch


Slave Slave Slave Slave Slave Slave

Grand Master


Server farms

Nexus Switches

Clock source


IEEE 1588v2 Clock Synchronization

Clock Offset

t2 - t1- mean path delay – Sync Message correction field

Mean Path Delay

((t2 - t1) + (t4 – t3)) / 2

After the synchronization

Slave clock derives Time of Day, phase and frequency signals from the master


Agenda




IEEE 1588 features supported on Nexus switch and configurations





IEEE 1588v2 Message Types

General messages (not time stamped) • Announce

• Follow_Up

• Delay_Resp

• Pdelay_Resp_Follow_Up

• Management and signaling

Event messages (need to be accurately time stamped) • Sync

• Delay_Req

• Pdelay_Req

• Pdelay_Resp


IEEE 1588v2 Message Types

The Announce message is used to establish the synchronization hierarchy

The Sync, Delay_Req, Follow_Up, and Delay_Resp messages are used to synchronize ordinary and boundary clocks

The Pdelay_Req, Pdelay_Resp, and Pdelay_Resp_Follow_Up messages are used to measure the link delay in transparent clocks

The signaling messages are used for communication between clocks for all other purposes


IEEE 1588v2 Packet Details

Nexus switches support PTP over UDP over IPv4 multicast • Communication between master and slave use IPv4 multicast group

address 224.0.1.129

• Event messages use UDP Port 319

• General message use UDP port 320

• Above applies to both unicast and multicast

Nexus switches current doesn’t support PTP unicast communication

IANA also reserved additional multicast address for PTP, currently it’s not used

• 224.0.1.130

• 224.0.1.131

• 224.0.1.132


Time-of-Day

IEEE 1588v2 PTP is capable of frequency, phase and time-of-day synchronization

Telecommunication industry requires the synchronization of frequency, phase and time-of-day

Most of the applications in financial institute and data center networks are interested in Time-of-Day synchronization


Agenda




IEEE 1588 functions on Nexus switch and configurations





PTP Feature Supported on Nexus Switches

PTP is supported on all Nexus 3000 platforms

• Nexus 3048 (10/100/1000M with 10G uplinks)

• Nexus 3064 (100M/1G/10G with 10G/40G uplinks)

• Nexus 3016 (16 x 40G with flexible 10G/40G template)

PTP is supported on all F series line cards in Nexus 7000 platforms

PTP is supported on all Nexus 5500 platforms

Interface supported by PTP function

• All L2 / L3 physical interface

• Port-Channel interface (including VPC interface)

• Not supported on sub-interface and SVI

Nexus platforms support 2-step boundary clock


IEEE 1588 Implementation in Nexus 3000

1. 1588 packet is time-stamped at ingress of ASIC to record the t2,

2. Timestamp points to the first bit of the packet (following SFD)

3. Packet is copied to CPU with timestamp and destination port

4. The packet is going through PTP stack and other process

5. The packet is sent to egress port.

6. The corresponding timestamp for the TX packet is available from the FIFO TX time stamp

PCI-Local Bus

SPI eUSB Flash

USB Conn

Console

Management Ethernet

48 x XFI XLAUI/XFI

48 x SFP + Cages 4 x QSFP

Switching ASIC

PTP Stack, Clock Servo,

Frequency Synchronizer

4GB DRAM

CPU

1 2

3 4

5

Nexus 3000 Data Center Switch

6


PTP Configuration on Nexus Switches

Required steps:

Enabling PTP feature (no license required)

Configuring an IP address for PTP communication, one for each PTP node

Enabling PTP on corresponding interfaces

n3k(config)# feature ptp n3k(config)# interface loopback0 N3k(config-if)#ip address 1.2.3.4/32 n3k(config-if)# ptp source 1.2.3.4 n3k(config-if)# interface e1/1 n3k(config-if)# ptp


PTP Configuration on Nexus Switches

Optional steps:

Configuring priority values, it’s a global value, lower number has higher priority

Configure PTP packet rate on interface

Use PTP to update system calendar

Currently Nexus 3000 requires 4 PPS or higher sync packet rate from neighboring master

n3k(config)# ptp priority 1 <priority 1> n3k(config)# ptp priority 2 <priority 2> n3k(config)# interface e1/1 N3k(config-if)#ptp sync interval ? <-6-1> Log seconds


Verify PTP Operation on Nexus Switches

Verify PTP interface status Verify PTP clock status

n3048-1# sh ptp brief PTP port status ----------------------- Port State ------- -------------- Eth1/16 Master Eth1/12 Master Eth1/11 Slave Eth1/9 Master Eth1/6 Disabled Eth1/4 Master Eth1/3 Master n3048-1#

n3k-p1# sh ptp clock PTP Device Type: Boundary clock Clock Identity : 0: 5:73:ff:ff:ee:cb:41 Clock Domain: 0 Number of PTP ports: 4 Priority1 : 255 Priority2 : 255 Clock Quality: Class : 248 Accuracy : 254 Offset (log variance) : 65535 Offset From Master : 528 Mean Path Delay : 412 Steps removed : 1 Local clock time:Fri Apr 13 06:51:48 2012 n3k-p1#


Verify PTP Operation on Nexus Switches

Verify PTP master clock Verify PTP clock status

n3k-p1# sh ptp parent PTP PARENT PROPERTIES Parent Clock: Parent Clock Identity: 0:b0:ae:ff:fe: 2:a8:9b Parent Port Number: 1 Observed Parent Offset (log variance): N/A Observed Parent Clock Phase Change Rate: N/A Grandmaster Clock: Grandmaster Clock Identity: 0:b0:ae:ff:fe: 2:a8:9b Grandmaster Clock Quality: Class: 6 Accuracy: 33 Offset (log variance): 25600 Priority1: 128 Priority2: 128

n3k-p1# sh ptp time-property PTP CLOCK TIME PROPERTY: Current UTC Offset valid: 1 Current UTC Offset: 34 Leap59: 0 Leap61: 0 Time Traceable: 1 Frequency Traceable: 1 PTP Timescale: 1 Time Source: 0x20(GPS) n3k-p1#


PONG - Pulse of the Network

switch# configure terminal switch(config)# pong destination-swid 3506 destination-mac 001b.54c2.9a43 vlan 1 count 3 Packet No. 1 --- ----------------- -------------------------- Hop Switch-id Switching time (sec, nsec) --- ----------------- -------------------------- 1 0-1b-54-c2-9a-41 0 4752 2 0-1b-54-c2-9a-43 0 544258088 3 0-1b-54-c2-9a-41 0 4792 Round trip time: 0sec 15416 nsec Packet No. 2 --- ----------------- -------------------------- Hop Switch-id Switching time (sec, nsec) --- ----------------- -------------------------- 1 0-1b-54-c2-9a-41 0 4752 2 0-1b-54-c2-9a-43 0 522744240 3 0-1b-54-c2-9a-41 0 4736 Round trip time: 0sec 15368 nsec Packet No. 3 --- ----------------- -------------------------- Hop Switch-id Switching time (sec, nsec) --- ----------------- -------------------------- 1 0-1b-54-c2-9a-41 0 4752 2 0-1b-54-c2-9a-43 0 521869920 3 0-1b-54-c2-9a-41 0 4800 Round trip time: 0sec 15360 nsec

PONG measures the switch port to port delay by utilizing the underlying PTP infrastructure


Troubleshoot - SPAN

Use Switched Port Analyzer (SPAN) to monitor PTP packet

Use Wireshark to decode the captured packets

SPAN configuration:

monitor session 1 source interface Ethernet1/11 both destination interface Ethernet1/8


Troubleshoot - Enanalyzer

n3048-1# ethanalyzer local interface inbound-hi detail display-filter ptp limit-captured-frames 1 Capturing on eth4 … snap … Ethernet II, Src: 54:7f:ee:14:6d:fc (54:7f:ee:14:6d:fc), Dst: 01:00:5e:00:01:81 (01:00:5e:00:01:81) Destination: 01:00:5e:00:01:81 (01:00:5e:00:01:81) Address: 01:00:5e:00:01:81 (01:00:5e:00:01:81) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Source: 54:7f:ee:14:6d:fc (54:7f:ee:14:6d:fc) Address: 54:7f:ee:14:6d:fc (54:7f:ee:14:6d:fc) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) Type: IP (0x0800) Internet Protocol, Src: 11.11.55.1 (11.11.55.1), Dst: 224.0.1.129 (224.0.1.129) Version: 4 Header length: 20 bytes Differentiated Services Field: 0xd4 (DSCP 0x35: Unknown DSCP; ECN: 0x00) 1101 01.. = Differentiated Services Codepoint: Unknown (0x35) .... ..0. = ECN-Capable Transport (ECT): 0 .... ...0 = ECN-CE: 0 Total Length: 82 Identification: 0x0000 (0) Flags: 0x00 0... = Reserved bit: Not set .0.. = Don't fragment: Not set ..0. = More fragments: Not set

Ethernet Header

IP Header

CLI: ethanalyzer local interface inbound-hi


Troubleshoot - Enanalyzer [Continued] Fragment offset: 0 Time to live: 64 Protocol: UDP (0x11) Header checksum: 0x944d [correct] [Good: True] [Bad : False] Source: 2.2.2.1 (2.2.2.1) Destination: 224.0.1.129 (224.0.1.129) User Datagram Protocol, Src Port: 319 (319), Dst Port: 319 (319) Source port: 319 (319) Destination port: 319 (319) Length: 52 Checksum: 0x45e7 [correct] [Good Checksum: True] [Bad Checksum: False] Precision Time Protocol (IEEE1588) versionPTP: 2 versionNetwork: 44 subdomain: messageType: Unknown (0) sourceCommunicationTechnology: PROFIBUS (5) sourceUuid: 73:ff:ff:ee:cb:41 (73:ff:ff:ee:cb:41) sourcePortId: 8 sequenceId: 36660 control: Sync Message (0) …… snap …..

UDP Header

PTP Header


Troubleshoot - CoPP

PTP CoPP policer is set to 1000 pps (packet per second) by default

NX-OS provides configurable CoPP to allow user to tune the CoPP parameters to meet their particular need.

n3k-p1# sh policy-map interface control-plane | section ptp class-map copp-s-ptp (match-any) police pps 1000 OutPackets 100054646 DropPackets 0 n3k-p1# n3k-p1(config)# policy-map type control-plane copp-system-policy n3k-p1(config-pmap)# class copp-s-ptp n3k-p1(config-pmap-c)# police pps ? <0-20000> Packet per second value n3k-p1(config-pmap-c)# police pps


Agenda









PTP Performance Validation

Two important metrics are used to verify the PTP performance* • Accuracy: Clock (Time of the Day) offset between GMC and slaves

• PDV (Packet Delay Variation)

Accurate Time-of-day is what the financial trading environment and data center applications are looking for

Factors need to be considered during performance validation • Traffic load and patterns

• Network topology, number of hops

• Type of PTP grand master, slaves

• PTP protocol parameters (packet rate, etc *Currently there is no standard on how to benchmark PTP BC or TC devices


PTP Performance Validation - HW client

PTP Grandmaster

Oscilloscope

PTP HW client (Cisco 2941)*

1PPS 1PPS

PTP Protocol over 1GbE Ethernet

*Cisco 2941 is used because it’s the only available PTP hardware client with GE interface when the test was performed

22.9ns

GPS satellite

GPS Antenna

Step 1: Baseline

Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Std Dev (ns)

4.5 hours 22.9 -3.5 47.5 14.2

Counter


PTP Performance Validation – HW client

PTP Grandmaster

Oscilloscope


1PPS 1PPS

PTP Protocol over 1GbE Ethernet

265 ns

GPS satellite

GPS Antenna

Step 2: Add 1 Nexus 3064


12 hours 265.29 82 418.5 76.19

Counter



PTP Grandmaster

Oscilloscope


1PPS 1PPS

389 ns

GPS satellite

GPS Antenna

Step 3: Add 2 Nexus 3064s


12 hours 389.2 117.5 569.8 92.4

Counter



PTP Grandmaster

Oscilloscope


1PPS 1PPS

391 ns

GPS satellite

GPS Antenna

Step 4: Congest the network link

Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Stdev (ns)

4.5 hours 391.5 117.5 569.8 84.6

Counter

Congested Traffic tool



PTP Grandmaster

Oscilloscope


1PPS 1PPS

-1.05 ns with large variance

GPS satellite

GPS Antenna

Step 5: Stop the traffic, disable PTP on Nexus switches

Test Duration Mean offset (ns) Min offset (ns) Max offset (ns) Stdev (ns)

12 hours -1.05 3743 4985 126.7

Counter

PTP function disabled


PTP Performance – PDV Validation PDV (Packet Delay Variation) could be introduced by queueing delays,

data path changes, congestion, etc.

PDV can reduce the accuracy of the network timing protocol

Sync PDV and Delay_Req PDV are the two most significant PDVs

PDV can be accumulated in a chain of PTP clocks

Boundary clock can help reduce the PDV accumulation in the network


PTP Performance – PDV Validation

PTP Grandmaster

GPS satellite

GPS Antenna

PTP is enabled

PDV generator and measurement tool

PDV should be measured under different traffic load


PTP Performance – PDV Validation

80% traffic load PDV=~400

Link congested PDV=~400

One-hop 7 hour PDV test


PTP vs. NTP Performance Comparison


PTP vs. NTP Performance Comparison

Average offset: NTP: 263us PTP: 0.412us


Components of End-to-End Solution

Grandmaster Clock • GPS (or equivalent) clock input

• What’s the oscillator options

• What and how many interfaces it support

• What’s the protocol it supported

• What kind of I/O output it provide

• What’s the management capability

Oscillator Interface Protocol I/O output Management

Rubidium 10GbE* Unicast/Multicast 1PPS Web GUI

OCXO 1GbE 1-step / 2-step 10Mhz SSH / SNMP

TCXO 100M NTP GbE (fibber or copper) Authentication

*Not available on current market


Components of End-to-End Solution

PTP Clients • Hardware-assistant or software only

• Oscillator options (hardware-assistant)

• What I/O interfaces it supports

• What’s the protocol it supported

• Fault-tolerant capability

• Management capability

Oscillator * I/O Options * OS requirement Protocol Fault-Tolerant

OCXO 1PPS in/out Windows Unicast/Multicast Switch-over

TCXO Other time code Linux/Unix 1-step / 2-step Fail back to NTP

Ethernet 32 / 64 bits NTP Never go back!

* Applicable to hardware-assistant client only


Agenda









PTP Network Design Considerations

Site survey & requirement

gathering

Define benchmarking methodology

Define network

topology and solution

component

POC testing and/or pilot

run

In production, continue

monitoring the network

It’s time to build an End-to-End PTP network!


PTP Network Design Considerations

Site survey & requirement gathering ‒ application requirement

‒ server requirement

‒ network load, traffic pattern, etc

Benchmarking methodology: PDV or offset? ‒ application requirement

Where is the grand master?

How many layer does network have?

What’s the scalability requirement?

Does it require dedicated PTP switch?


PTP Network Design Considerations In-band Solution

• Install & configure precision

timing source (e.g., GPS and grand master)

• Enable PTP on all Nexus switches in the existing data network

• Install & configure PTP client on existing servers

• For greenfield implementation

Core Layer

Server Farms


• Install & configure precision

timing source (e.g., GPS and grand master)

• Install & configure dedicated

PTP switches (boundary clock) as a PTP distribution switch

• Install & configure PTP client

on existing servers

• For retrofit implementation

Core Layer

PTP Network Design Considerations Out-of-band Solution

Server Farms


PTP Network Design Considerations PTP as a Service From Exchange or DC IT

• PTP can be provided by the

Exchange as precision time synchronization service to its customers

• Data center service team can also provide PTP as a service to its client and applications.

• Can be done via in-band, out-

of-band or hybrid mode

Backbone

Customer A Customer B Customer C

Customers

Provider Site A Site B


Agenda









PTP Validated in Cisco Customer Solution Lab

Precision Time Source – Grandmaster with GPS receiver

Dedicated PTP distribution switches – 1st hop, 2 x nexus 3048s

Spine/Aggregation layer – 2nd hop, all Nexus 3064s

Leaf/Access layer 3rd hop, all nexus 3064s, each N3064 has 20-30 servers

In-band Solution


PTP Test Results in Cisco Solution Lab

57

Server location PTP Client Nexus Platform Accuracy (usec)

1st Hop Cisco PTP Client (v3.8) Nexus 3048 pos avg 0.351768496 neg avg -1.541941669

2nd Hop Cisco PTP Client (v3.8) Nexus 3064-E pos avg 0.511412221 neg avg -1.487674687

3rd Hop Cisco PTP Client (v3.8) Nexus 3064-E pos avg 1.269494697 neg avg -1.299382558

Open Source PTP client (OS-2.2.1) is also tested. Its performance results are similar but experienced failover problems.

NX-OS Server Spec Test Duration Sync packet rate (PPS)

5.0(3)U2(2) UCS 200-M1(1x Intel E5504 processor)

Redhat Enterprise 5.4 12 hours

4 PPS (packet per second) between Nexus switches and

GM, 1 PPS towards server

Configurations Test bed is fully loaded with unicast/muticast traffic. All major L2/L3, multicast, QoS features are configured to simulate a customer production network

Performance test results


Additional Sessions

BRKSPG-2170 ‒ Synchronization in Packet-based Networks (SyncE & IEEE1588-2008)

BRKDCT-2214 ‒ Ultra Low Latency Data Center Design


Looking forward

ITU standard update (lots of work need to be done )

Security will be a concerns when PTP deployment become mainstream


Key Take Away

We need a highly accurate timing synchronization solution in sub-microsecond level

IEEE 1588 PTP is a widely used timing synchronization protocol in packet network

Cisco Nexus data center switches support PTP in hardware today

Cisco Nexus switch can delivery accurate PTP timing information to client under heavy network load

PTP solution need to be carefully designed and reviewed before enabled in production network

Large-scale in-band PTP deployment is verified in Cisco solution lab


Complete Your Online Session Evaluation Give us your feedback and you

could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.

61


Documents

Nanosecond Accuracy Timing Solution for Data Center …d2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKDCT-2215.pdf · Nanosecond Accuracy Timing Solution for Data Center Fabric BRKDCT-2215