TNC18 Intelligent networks, cool edges?www.geant.org

Click to edit Master title style

• Click to edit Master text styles• Second level

• Third level

• Fourth level

• Fifth level

08/12/2019 1

GÉANT network update

www.geant.org

Rick HavernHead of Network Engineering

Tech Ex, New Orleans LA USA, 201912

Topics, at speed• Talked about SDN a few years ago, harder than it looks

• GN4-3N IRU Fibre/spectrum procurement

• DCI deployment

• OLS procurement/disaggregation

• Automation, hired a NetDevOps person • IP/MPLS testing: Ansible+Salt, Git, Robot framework, pyEZ• Transmission testing: Groove G30, DTN-x, Exfo OSA & Packet Blazer

• Streaming telemetry, lab testing to come, software selection, dimensioning

• Observium is pretty cool

• Segment routing, running in production as secondary since Jan19, script comparing LDP/SR tables

• RPKI is coming, lab testing in progress

• IP/MPLS layer, 400G needed soon

• P layer perhaps, due to the cost of 400G interfaces on MX platform

3

SDN, the buzz of a few years ago

Turns out, SDN is harder to implement than it looks and with that, it is less useful than expected, so we stopped

4

GN4-3N: IRU SGA

Go beyond the state-of-the-art by restructuring the

backbone network through exploration and

procurement of long-term IRUs and associated

equipment to increase the footprint, stimulating the

market in cross-border communications infrastructure

whilst decreasing the digital divide and reducing

costsFrom EC objectives

6

The process

Regional Studies

Technology strategy

Traffic projections

Model built to understand and forecast capacity

requirements and analyse costs

€Reference topology

IRU Budget € K

Lowest cost share topology option € 48,294

10% contingency € 4,829

Total € 53,123

Overview of costs

Impact on running costs € k 2018 Budget Reference topology Variance % reduction

Total 6,427 4,193 2,234 -34.8%

This budget covers both new dark fibre/spectrum and new open optical line system for all elements lit and operated by GÉANT.

8

➢Dark fibre core

➢Leased capacity regions

➢Some rings, some hub and spoke

➢Short procurement cycles

➢Long implementation cycles

➢Short term requirements

➢Higher cost impact

Blue – Dark fibreBlack– Leased capacityOrange - Spectrum

The GEANT network today

Reference topology2018

9

• Fibre/spectrum to more NRENs

• Increase of meshing

• Tiering

• Better regional connectivity

• PoPs in strategic locations

OPEX cost efficiency

Technology choices

- Dark Fibre- Obvious choice for long term investment - Well known product- Not very cost efficient when BW requirement is low

- Spectrum- Gaining traction in the market

- 2018 RFI, 3 providers offering - 2019 RFI, up to 17

- Widely used in submarine world- Significantly more cost efficient when bandwidth requirements are low for fibre- Not yet as standard as fibre- Various flavour of it available

Spectrum – what we would like to see

PROVIDER NMS

GEANT NMSGEANT WSS GEANT WSS

PROVIDER WSS

PROVIDER WSS

… …

Transponder

Transponder

Transponder

Transponder

Transponder

Transponder

FlexMUX

FlexMUX

Other WSSsOther WSSs

- Best solution for GEANT- contiguous portion of the C-Band (considered 500GHz+)

- direct connection to the Provider WSS

- GEANT manages slicing

12

Updating the topology – Market engagement

• Oct 2018 - Capacity Europe 2018 to raise awareness and find new possible suppliers

• Mar 2019 - Webinar to explain scope, steps and process for engaging

• Apr 2019 - 2nd RFI to market and NRENs to refresh quotes/ explore alternatives• 51 suppliers answered

• 460 quotes

215 DF

54 Spectrum (more pending and being received)

191 Lease

• Mar –> May 2019 - Face to faces meetings with providers, 33 meetings, ~135h of intense discussions to further explore alternatives

• May 2019 - Re-model based on new information

• Oct 2019 - Capacity Europe 2019 to raise awareness and find new possible suppliers

13

Reference Topology 2019• Similar principles applied when

creating the first reference topology applied

• Cost share impact over 10 years

• Missed opportunity value of Capital to cost share reduction is considered*

• Offers enough potential capacity for the 2030 projected traffic levels

• When difference below 10% look at other aspects such as End to End and link length

• Map completely based on 2019 received quotations.

• Exception: Bru-Fra added as Nordunet may be able to provide this link, however full quote is not available yet

• Reference 2019 is not optimised in all areas. This is a new starting point, not the end

Dark fibre/spectrum: solution choice

Cost relation heat map

Timelines - what needs to happen

15

• From tendering to building

• Integration into existing OSS/BSS stack

The current GEANT DWDM/OTN network

Based on Infinera DTNxsystem – Network elements provide large capacity point to point circuits across defined routes

OTN and GMPLS makes solution easy to operate and very fast to turn up new services with high availability

GÉANT Point-to-Point –10G and 100G lambda services as well as our IP trunks are provided on this infrastructure DTN-X in operation with good results since 2012

Interesting trends

• Commodity DWDM pluggables

• Data centre interconnects

• Open line systems/ Alien waves maturity

• Disaggregation

TIP reference architecture

Data Centre Interconnects

• Typically based on next gen of commodity pluggable optics

• Point to point connectivity over a fibre link

• Data centre style 1 RU stackable form factor.

• Large reduction in cost over traditional telecoms equipment architectures

• Significant increase in density and reduction in power consumption

• Modular – easy to scale up

• Easy upgrade path to new technology

Loss of equipment integrity – not designed to be highly available as per ETSI etc.

• No internal hardware redundancy

• No in-service upgrades

• Restricted temperature operation

GÉANT’s DCI choice

• GÉANT is using Coriant Groove G30.

• 1 RU stackable

• 4 sleds, each up to 4 x 100G

• Tuneable optics:

• 200G up to 1000km with 16 QAM modulation

• 150G up to 2000km with 8 QAM modulation

• 100G up to 5000km with DP-QPSK

• Client side is QSFP28

• Next generation 1.2G per slot with 600Gbps per CFP using 64QAM

DCI deployment 2018-2019

DCI boxes deployed instead

of additional expenditure on

the existing Infinera DTN-x

estate

Estimated savings of

~€2.3M using this technique

Typically able to close using

8 or 16QAM, giving 300 or

400G per G30 sled

GÉANT uses one G30

chassis per degree for

resilience

GÉANT Transmission Lab Automation

Infinera OCGBand mux

EDFA EDFA

splitter splitterGroove Groove

Add span noise

Plot Rx sensitivity

Infinera OCGBand mux

50km SMFVOA 1

VOA 2

•We have automated our network model in Cambridge lab

•Automated tests can cycle through hundreds of power/gain/modulation/noise/FEC parameters

Vary power/

modulation/FEC

Measure pre-FEC

BER

Infinera OCG Groove

Spectrum

GÉANT Transmission Lab Automation

• The lab automation controller communicates with optical network elements and test boxes

• Config file interface allows the user to rapidly create new test scenarios

• Script set the Groove G30 parameters – power, wavelength etc. and controls the attenuation on the digital VoAs

• Acquires optical performance monitoring data from DTN-x and G30 and our Exfo OSA and Exfo Packet Blazer

• Utilizes multiple communication protocol connectors including TL1, RS232, REST API, SCPI to interact with devices

• Nested loops of test configurations allows multiple parameters too be varied

• Many thanks to Michal Altmann (CESNET) in developing this code.

Automation / Configuration Control: “Golden Config”

• We have been automating our testbed setup for IP/MPLS type certification testing using the Robot framework and pyEZ

• Went from weeks/months to under an hour; now we run tests nightly in the lab

• “Infrastructure as code”

• Use software methods to control configurations

• “Eliminates configuration drift, through automation, thereby increasing the speed and agility of infrastructure deployments” *

Tools - large devops community:

• Collaboration and version control: Gitlab

• Configuration management:

• Ansible (for the automation systems)

• Saltstack (for the network devices)

• CI/(CD): Gitlab-CI

• Jsnapy to collect operational status checks for before/after action comparison

* Infrastructure as Code (July 2017), AWS Whitepaper, https://d0.awsstatic.com/whitepapers/DevOps/infrastructure-as-code.pdf

Golden Config - Operational

• Need analysis of each stanza of configuration • What should the policy really be vs inherited cruft

• Once a service is defined, the policy to create that service will also be defined, documented and enforced.

• Operational concerns: Ensuring we have a flexible, extensible infrastructure prior to controlling the production network is paramount.

Streaming Telemetry

• SNMP based polling inefficient: everything polled at the same frequency• Without considering (likelihood of) data change• Currently polling at 5 min therefore microbursts are invisible.

• Netflow data has one-minute averages, better view of elephants, but still no view of microbursts

• Streaming Telemetry: Subscription based, adjustable for the expected useful granularity

• Infrequent changes (line card/hostname changes) set with a longer interval, couple of times per day.

• Network interfaces every couple of seconds - finally giving level of granularity to capture microbursts!

• Streaming Telemetry: Highly granular data from the network, for as long as we can store it

• No smoothing of data as it ages, brilliant for when new traffic analysis tool is introduced

• Streaming Telemetry: Base for future path computation/routing adjustments• Near real-time network data allow PCE controller to make near real-time adjustments

• Avoid congestion, dynamically make space for long duration elephant flows

Streaming Telemetry to do list

• Determine what to measure and how frequently

• Software packages for collection, storage and analysis

• Proof of concept in lab

• Does it work as expected or break any existing service

• Gather business requirements for data retention

• Dimensioning hardware required to meet requirements for production

• Production POC

Observium

• Auto-populated view of the active network interfaces

• Works “out of the box”

• Alerts very easily configurable can be silenced until a specific date, or suppressed until the element comes back up.

• Aiming to replace Cacti, but challenges with ingesting historical data

• GEANT Software Development have written custom script to ingest Cacti rrdinto a time series database

• Support cost very reasonable, may even spin up separate versions for private networks

Segment Routing

• RSVP-TE in a large number of nodes is hard work to set up and maintain in a dynamic environment

• However we need a way to explicitly define a path for specific circumstances, such as low latency applications

• RSVP-TE is still a pain in the butt for networks with many nodes

• Segment routing give us this functionality without the pain of RSVP-TE

• Recent versions of Junos give the option of running Topology Independent Loop-free Alternate Fast Re-route (TI-LFA)

• GÉANT running SR in parallel with LDP since Jan 2019, but not yet preferred; script compares the two and alerts if not same

RPKI is gaining traction

• To gain experience operating, GÉANT have installed two instances in the lab

• Routinator (nlnetlabs)

• Octo-RPKI (cloudflare) + GoRTR (cloudflare)

• No issues found to date, need to determine if dropping INVALID routes will impact any of GÉANT’s customers before deploying to production network

• Probably only apply to Internet Access, not R&E peers

IP/MPLS layer 2019

3x100G in

the Western

and Eastern

rings;

200 & 100G

surrounding;

Nx10G

further afield

IP/MPLS layer 2025

• Forecast in

2025 according

to current

growth rate and

3N expansion

• 400G+

interfaces

become a

necessity in

core locations

TNC18 Intelligent networks, cool edges?www.geant.org

Click to edit Master title style

• Click to edit Master text styles• Second level

• Third level

• Fourth level

• Fifth level

08/12/2019 32

Thank you!

www.geant.org

Any questions?

© GÉANT Association on behalf of the GN4 Phase 2 project (GN4-2).The research leading to these results has received funding fromthe European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 731122 (GN4-2).