Differences between Financial and Telecom Network Environment

Kamatchi Gopalakrishnan

Distinguished Engineer

Agenda

Telecom

versus

Financial

Network

Summary

Network

Time-sync

Time-sync

Profile

comparison

Time-sync basics and requirements 1

Synchronization overview

TA=1/fA

TB=1/fB

fA=fB

t

t

TA=1/fA

TB=1/fB

fA=fB

Frequency Synchronization

Phase Synchronization

A

B

01:00:00

fA=fB

Time Synchronization

01:00:10

01:00:00 01:00:10

A

B

Aligning clocks with respect to frequency

Aligning clocks with respect to phase

Aligning clocks with respect to time.

A

B

FDD application TDD application FSI application

Synchronization methods and options

GNSS (GPS, GLONAS,

Galileo, BeiDou)

NTP

SONET/SDH, Sync-E

PTP

- Jamming, Spoofing, RF factors

- Expensive

- Line of site issue

- Good for milliseconds

- Not good for high precision

- Not for phase or time sync

- Every node must support

- High precision – Phase,

Frequency and Time

Telecom versus FSI network timing requirements

Telecom network FSI network

Frequency and/or phase sync Time of day (ToD) sync

UTC traceability not a must. Arbitrary time source is acceptable

UTC traceability is must (ESMA) or NIST traceability in US (SEC)

Ranges from 100nsec to order of microseconds – depends on MBH applications

Must be less than 100usec from UTC (ESMA)

Operational requirements Regulatory requirements

Timing in Telecom versus Financial network 2

Telecom versus FSI network model comparison

Telecom network model FSI network model

Multiple administrative domain (Mostly) Single or few cases multiple administrative domain

Network services and transports not in control of single admin

Controlled network services and transports

No direct leverage to upgrade network segments

Leverage to upgrade/modify network segment

Downtime causes financial impact Downtime causes both financial and regulatory impact

Timing is operational requirement Timing is a regulatory requirement

Telecom – Multi Operator network

Access

Network

Backhaul Network

Mobile Core

Network

GNSS

Core GM

Operator-1 Operator-2 Operator-3

End to End 1.5 microsecond

Mobile Phone roaming between cell-sites

BS2

BS1

BS2 BS1

+/-50 ppb

+/-50 ppb

BS2 drifts outside 50ppb window

Mobile cannot lock to BS2, call dropped

Phase (TDD) and Frequency (FDD) sync

Cell tower

Handset

Send to handset at frequency F1

Send to tower at frequency F2

Frequency Division Duplex

Tower Hand Tower Hand Tower F1

Time

Cell tower

Handset

Time: 12:00.01 Send to handset at frequency F1

Time: 12:00.05 Send to tower at frequency F1

Time Division Duplex

Tower F1

Time

Handset F2

Requires: accurate frequency

Requires: accurate frequency & phase

Mobile application phase sync requirements

(Frequency : 16 ppb/ 50 ppb)

Application Phase

CDMA2000 +/-3 us to +/- 10 us

LTE-TDD +/- 1.5 us (< 3KM cell radius) +/- 5 us (> 3KM cell radius)

LTE MBMS (LTE-FDD and LTE-TDD) +/- 10 us

LTE-A CoMP +/- 0.5 us to +/- 1.5 us

LTE-A eICIC +/- 1.5 us to 5 us

E911 and Locating services +/- 0.1 us

Small cells +/- 3 us (1 to 5 us) with 100 to 250 ppb

Financial Network – Single Operator Domain

• 10 / 100 thousand servers.

• Boundary Clocks to serve group of servers

•

MARKET DATA FEED

Compute Cluster Interconnect

HFT SERVERS

BACK END COMPUTE CLUSTER

EXCHANGE

MONITORING APPLICATIONS

EXCHANGE CUSTOMERS

Synchronization flow model

Spine

Switches Leaf

Switches

Gateway

routers

Time Legality

• How do you prove that something

happened before (or after) a certain

time?

• How do you correlate events across a

large/global network?

Time-sync requirements in FSI (MiFID-2/ESMA)

• RTS 25 of Regulatory and Implementing standards – annex 1

• Adopted regulatory requirements

Reference Time – UTC traceable

Compliance with maximum divergence requirements

Level of accuracy for operator of trading venue

Gateway-to-Gateway latency

Maximum divergence from UTC

Time stamp Granularity

> 1 millisecond 1 millisecond 1 millisecond or better

=< 1 millisecond 100 microseconds 1 microsecond or better

Profile comparison 3

Telecom Financial/Ente

rprise

What are profiles?

• Timing profiles are subset of requirements derived from IEEE1588 specification.

• Targeted for particular application to achieve required synchronization goals in most reliable and interoperable way.

• Different Timing profiles: Telecom profile – Mobile Backhaul applications

Enterprise profile – Enterprise/DC and financial application

802.1AS/AVB profile – Audio/Video in bridged network

SMPTE-2059-2 profile – IP based video broadcasting network

Telecom versus Enterprise profile

• PTP over Ethernet Multicast

• Sync-E + PTP combined mode

• Phase accuracy in microseconds

• Fixed PTP packet rates

• Alternate Best Master Clock Algorithm (ABMCA)

• No Unicast Negotiation

• PTP over IPv4 or IPv6 Multicast

• Plain PTP mode

• Time of day (less than 100us UTC)

• Configurable packet rates

• 1588 default Best Master Clock Algorithm

• Unicast negotiation allowed

Enterprise Profile Telecom Profile

Summary 4

Summary - Precision Timing Challenges

• Packet Delay Variation

• Scaling - Number of PTP clients support

• Number of hops between GM and End Slave nodes

• Precision Performance Monitoring

• Overlay versus Inline synchronization flow

Thank you

Backup slides

Case1 - Accuracy in Hi-FREQ algorithmic trading

Algorithmic Trading Servers

Algorithmic Trading Servers

T1 Trade Execution

Timestamp

Market Data Creation Servers

Market Feed Generator Servers

When a trade executes, a timestamp is generated by the trade execution server

1 The trade execution data is sent to the market data creation servers

2 T2 Market Data Creation

Timestamp

The market data creation servers generate a timestamp when new market data is created

3 The market data is sent to the feed generator servers

4 T3 Market Feed Timestamp

The market feed generator servers timestamp the feed which they send out

5

The market feed reaches the customers’ algorithmic servers with the three timestamps embedded

6

T3 Market Feed Timestamp

T2 Market

Data T1 Trade Data

With NTP precision, often T3 < T2 < T1 – ie, market data is sent before it is created and even before the trade has been settled! Algorithms are confused leading to lost business and angry customers for the exchange

7

With PTP, exchanges can achieve better precision (1 us or less) that will let them fix this problem

Trade Execution Servers

Algorithmic Trading Servers

Case-2: End To End Latency Analysis

Market data feed BGP/OSPF IP Multicast

HFT servers HFT servers

• In HFT, latency is king

• Different latency components

• PTP based timing to correlate

measurements across nodes

Case 3: logging for regulatory reasons

Market data feed BGP/OSPF IP Multicast

HFT Algorithmic Trading Servers

Precise timing provided by 1588 is needed to achieve accuracy in logging operations to alleviate regulatory problems

Customer Trade Requests Buy request arrives from Customer X for IBM options at

server A – timestamped with T1, record sent to the logging server

1

T1 Customer X Buy IBM @250

Logging Server

A B C

Trade is executed (“filled”) by server A at time T2 – log record sent to logging server 2

T2 Customer X Fill IBM @250

Buy request arrives from Customer Y for IBM options at server C – timestamped with T3, record sent to the logging server

3

T3 Customer Y Buy IBM @251

With NTP precision, it might be that T3 < T1 < T2, so this is how the log looks like when read after the fact: 4

Raises regulatory concerns around fair trading

MiFID-2/ MiFIR and ESMA • MiFID – Markets in Financial Instrument Directive

• MiFIR – Markets in Financial Instrument Regulation

• ESMA – European Security Market Authority

• Regulatory Technical Standard (RTS)

• Implementing Technical Standard (ITS)

• Imposes: Fairer, safer and more efficient markets

Greater transparency

Stronger investor Protection