www.geant.org 1 |

www.geant.org

Mian Usman Network Architect

GÉANT Network Evolution

www.geant.org

Runs a membership association for Europe's National Research & Education Networks (NRENs) GÉANT Association

Coordinates and participates in EC-funded projects Under Horizon 2020 the financial instrument for implementing the Innovation Union, a Europe 2020 flagship initiative aimed at securing Europe's global competitiveness

Operates a pan-European e-infrastructure GÉANT network

Manages a portfolio of services for research & education EduX

Organises and runs community events & working groups TNC, task forces & special interest groups

To support collaboration and development amongst researchers, the dissemination of information & knowledge, and provide access to a portfolio of services and infrastructure resources:

2 |

www.geant.org

Membership Association

GÉANT Association supports and represents over 40 NRENs across Europe.

Together they support over 10,000 institutions and 50 million academic users.

3 |

www.geant.org

Network The GÉANT network interconnects research, education and innovation communities worldwide, with secure, high-capacity networks.

We design, plan, build and operate the large-scale, high-performance GÉANT network that connects European NRENs to each other and the rest of the world for sharing, accessing and processing the high data volumes generated by research and education communities and for testing innovative technologies and concepts.

Interconnecting Europe’s NRENs over a 500Gb highly-resilient

pan-European backbone. NRENs serve 50 million users at 10,000 institutions across

Europe. Network services: IP, Point-to-Point Services, VPN, Testbeds,

performance monitoring

4 |

Data transfer tests in 2017 between 10G servers in GÉANT and AARNET achieved 9.73Gbps over 48h through R&E networks, whereas over commercial links

this was only 1.77Gbps.

www.geant.org

GÉANT operates an advanced, high-performance network supporting Europe’s NRENs with cost-effective, highly available, high-capacity services customised for and dedicated to research and education

Availability Targets 99.999% - 27 seconds downtime/month

99.99% - 4.5 minutes/month 99.4%* - 4 hours/month

GÉANT network services

GÉANT IP – Ultra-high performance uncontended IP connectivity at up to 100Gbit/s

GÉANT VPN – Layer 3 VPN services for NRENs and institutions supporting private networking needs

GÉANT Point-to-Point – High performance dedicated connectivity up to 100Gbit/s for the most demanding applications

GÉANT Open – Allows NRENs and approved commercial organisations to exchange connectivity in a highly efficient and flexible manner

eduroam – Seamless WiFi access around the world

Network Services

5 |

www.geant.org

GÉANT IP network

Based on Juniper MX Platform – All layer 2 and layer 3 services are provided on IP layer

GÉANT Point-to-Point – 10G and 100G lambda services are provided on optical infrastructure

Makes use of GÉANT DWDM Network for its core links and uses leased capacity where dark fibre isn’t available.

6 |

GÉANT Network Topology

www.geant.org 7 |

0

500

1000

1500

2000

2500

3000

2015 2016 2017 2018

GÉANT traffic PB/year

IP/MPLS Lambdas

• 2.4 EB of data received in 2018

• 26.4% growth from 2017 (29.9% for IP/MPLS)

• Longer term trend - 30%

0

100

200

300

400

500

600

700

800

Q12015

Q22015

Q32015

Q42015

Q12016

Q22016

Q32016

Q42016

Q12017

Q22017

Q32017

Q42017

Q12018

Q22018

Q32018

Q42018

GÉANT traffic PB – Quarterly view

• Data growth x 3 over four years period

Network and Capacity Growth

www.geant.org 8 |

Network and Capacity Growth - Traffic Distribution

LAMBDAS 39%

R&E 23%

LHCONE 15%

IAS 14%

OTHERS 9%

GÉANT traffic - by category

LAMBDAS – 10G and 100G Ethernet point to point services provisioned directly over the Infinera OTN switching layer

R&E – IP/MPLS routing for Global R&E prefixes

LHCONE – Dedicated L3VPN for LHC

IAS – GÉANT on-net internet access service

OTHERS – includes other L3VPNs, L2VPNs, MDVPN, GÉANT open and LANs

www.geant.org 9 |

• Support exponential bandwidth increase • With highly unpredictable traffic • Cost optimisation for backbone link upgrades

• Improve network visibility and control operational costs

• Provide programmability • offering the ability to use APIs to link applications to the network • Increasing agility and speed of service development

• Reduce dependence on vendor (software) • Preference for using open and standard hardware APIs

• Short Funding Cycle – Higher cost of change every 7-8 years

Network Evolution - Challenges

www.geant.org

Fiber Renewal Plan Starting Point – Current Network

• Current Fiber Network:

• Short term leases

• 14 countries connected on fiber

• UK, BE, FR, CH, DE, AT, NL, HU, HR, IT, SI, SK, CZ, ES (spur)

• Other countries on leased lines

• Typical capacity multiple 10Gs

10 |

www.geant.org

GÉANT Topology - IRU SGA

EC created a funding vehicle to procure infrastructure on long terms contracts and with 100% funding.

Go beyond the state-of-the-art by restructuring the backbone network through exploration and procurement of long-term IRUs and associated equipment to increase the footprint, stimulating the market in cross-border communications infrastructure whilst decreasing the digital divide and reducing costs

Extract from objective for the IRU SGA

www.geant.org

Regional Studies - to Agree Topology

12 |

Ireland UK

UK/Ireland Led by Guy Roberts

Belarus Latvia Estonia Lithuania

Russia

Finland

Sweden Norway Denmark

Baltic - Led by Ieva Muraškienė (LITNET)

Poland

Greece Hungary Bulgaria

Romania Serbia

South East Europe - Led by Pavel Skoda (CESNET)

Albania

Slovakia

Israel

Cyprus Turkey

Macedonia

Montenegro

Croatia

France Croatia

Central Europe region - Led by Sebastiano Buscaglione

Poland Germany

Italy

Switzerland

Austria Czech Republic

Slovenia

Malta Holland Belgium UK

Spain

Hungary

Luxembourg

+ Nordunet

Portugal

Spain

Iberian Peninsula Led by Emanuel Massano (GARR)

France

Italy

www.geant.org

Stakeholders review

Network Blueprint

IRU budget €

Financial constraints

Desirable outcome?

Community Engagement and

Workshops

Regional Studies

Technology Strategy Traffic

Projections

Market RFI

26 |

Infrastructure Sharing Equipment

Cost

GÉANT topology – a network blueprint

www.geant.org

Straw Man approved at GÉANT General Assembly June 2018 • REPLACEMENT NETWORK

proposal

• De-risked: Included links that can be built with sufficient certainty

• Based on regional studies

• Investment required: 48 M€

14 |

Result: On fiber: 24 countries: UK, IE, PT, ES, FR, BE, NL, DE, EE, LV, LT, PL, AT, CZ, SK, CH, HU, IT, SI, HR, RO, BG, GR, RS CYNet, IUCC, MARNET, MREN, RASH, RESTENA, ULAKBIM: • standard leased capacity (minimally 10GE,

might be 100GE by end of project • or additional DF/spectrum projects as part

of regional improvement / extension

Commercial DF or spectrum NREN DF or spectrum

www.geant.org

Network Infrastructure and Services Operations

15 |

€-

€1

€2

€3

€4

€5

€6

€7

1.19 4.19 7.19 10.19 1.20 4.20 7.20 10.20 1.21 4.21 7.21 10.21 1.22 4.22 7.22 10.22

An

nu

al O

&M

(M

Eur)

Evolution of connectivity O&M in positive "Flip"(*) scenario

OLD Commercial DF NEW Commercial DF OLD Line System NEW Line System

NEW DCI OLD Commercial Leased NEW Commercial Leased OLD NREN Spectrum

OLD NREN Leased NEW NREN DF NEW NREN Spectrum

(*) “Flip”: Quickly changing contracts for existing fibres to IRU

www.geant.org 16 |

• Vendor and Technology Agnostic Architecture

• Disaggregate: de-couple innovation cycles and accelerate innovation • Enable GÉANT to use best-in-class solution at different layers

• Decrease dependence on a single vendor

• Increase options to connect NRENs to different platform depending on their needs • Optimize investments to align with requirements

• Reduce space and power requirements by allowing smaller form factors

Open and disaggregated architecture allows to evolve, and to differentiate from commercial providers and develop and deliver advanced networking services

for R&E users

Network Evolution Plans - Flexibility and Control

www.geant.org 17 |

Powerful planning tools - understand impact Support current investment choices and understand future impact

• Network modelled and analysed at component level • Understand impact of growth on costs

• Where, when and how expenditure is expected

• Better understanding of interplay between technical choices and network growth at both technical and financial level allows for identifying areas for improvement

• Detailed, flexible modelling allows benchmarking and evaluating alternatives

Better/More transparent decision making

Supporting evidence for review and process improvement

www.geant.org 18 |

Vendor Proprietary Network Controller

Multi-vendor Open Network Controller/Orchestrator

• Proprietary and Closed APIs • Vendor-Specific data model

Vertically integrated single vendor solution

• Open and Standard APIs

• Open data model

Multi-vendor, disaggregated

solution

NMS NMS

Open Optical Line SYSTEM

Disaggregate: Open Optical Line System

DCIs

Network Evolution - Opening up the optical transport layer

www.geant.org 19 |

• Supporting growth on current

Infinera DTNx leads to very high costs, and is not effective • Major changes expected on Network in

next few years – DTNx components may be obsolete

• Requirement-to-Technology mapping not optimal for DTNx – 70% of capacity does not use (expensive) OTN switching capabilities

• Modelling used to understand how and where DCIs should be used to integrate the Infinera system to offset short term issues and optimize investment

0 €

1 €

2 €

3 €

4 €

5 €

6 €

7 €

8 €

2 018 2 019 2 020 2 021 2 022M

ilio

ny

Cost comparison Pure Infinera vs Infinera + DCI hybrid solution

Cumulative costs (€)

Infinera DTNx only Infinera DTNx + DCI

Powerful planning tools - understand impact Guide introduction of new technology

www.geant.org 20 |

• From a monolithic block to a open and disaggregated optical layer in GÉANT network

• Procured DCI (Infinera Groove G30) • Significantly reduces the cost of upgrading the GÉANT backbone links • Significantly reduces the space and power requirements • Protects GÉANTs investment by re-using the DCI’s with different optical line system

• Next steps: OLS

Vendor Meetings

Facebook Voyager Lab testing and Field Trial

Lab Testing field trial

DCI equipment

Technical Specification

for DCI Procurement

Launch DCI

Procurement

ConcludeDCI

Procurement

Deploy DCI

Launch OLS

Procurement

Deploy OLS

Network Evolution - Opening up the optical transport layer

www.geant.org 21 |

Plan for deployment completed during 2018 - based on modelling and analysis

2 phases:

•Phase 1 (2018-2019) – Western ring IP trunks move to DCI and upgrade to 300G

•Phase 2 (2020) – Milan - Marseille – Geneva IP trunks move to DCI

First 300G link live in GÉANT from December 2018 (Lon1-Lon2)

Phase 1 overview

London Amsterdam

Frankfurt Geneva

Paris

Planning and Optimisation – Planning optimal DCI Deployment

www.geant.org 22 |

Options: (cost) effective solution?

Network Evolution - Opening up the packet layer

www.geant.org 23 |

Network Evolution - Opening up the packet layer

• Investigated alternative solutions

• Incumbent vendor’s response to changing market: a cost-effective small device • Low cost of change, as no integration work is required, no forklift migration. • Fewer new skills to learn for the Ops and NOC teams • Both software and hardware support provided by a single vendor.

• Introduction of MX204 where appropriate • Power consumption -66%, Yearly maintenance cost -79%

• Does not preclude later introduction of other vendors!

Software and

Hardware Vendor

Meetings

Worked with JRA1 to develop

evolution options

Worked with Metaswitch and Corsa to create a

new solution

Loan kit from Dell, Arista and Agema

for testing

Testing of different white boxes with IPInfusion

and Metaswitch

New Juniper Product MX204

Testing of MX204 and Cost Benefit

Analysis

Order Placed

Replace MX480 with

MX204 in 14 Locations

www.geant.org

Planning and Optimisation – Introducing the MX204

MX 204

4x QSFP – 1x100G or 4x10G

8x SFP – 1G or 10G

400G total switching capacity Enhanced QoS PTP support Dual PSU Single routing engine

~80% OPEX reduction

~70% less power

Maintenance cost could be optimised for small PoPs in GÉANT

MX204 tested for operational deployment and procured

Design for implementation of first few sites completed

Deployment planned for Q2 - 2019

www.geant.org

Infinera OTN

Juniper MX-960 and MX-480

Juniper MX204

25

NRENs

Dark Fibre Leased Capacity

Infinera

Infinera OTN

Coriant DCI

New Interconnection between routers

Juniper MX-960

Replace MX-480s With MX-

204

Open Line System

Coriant DCI

Deploy OLS and remove OTN

Packet Core Devices

Edge Devices

Simplify Packet Layer

DCI – External Transponders

Vendor Agnostic Optical Layer

Increase Options for

NREN Connectivity

Low Capex and Reduced Opex

Low space and power

requirements

66% reduction in power and

70% reduction in maintenance

Network Evolution - Architectural Impact

www.geant.org 26 |

Network Configuration

Automation and Change Management

Orchestration Policy-based

Automation of Configuration

Intent based

Significant work carried out on Automation

Engaged with several vendors - trial with

vendors planned in 2019

Long Term Goal

Users intent and network configuration

in sync

Network Evolution – Management Software and Systems

www.geant.org 27 |

Lab built to replicate the production network 1 • MPLS

• IS-IS

• Multicast

• IPv4 & IPv6

• Netflow v9

• MDVPN

• L2 circuits

• EVPN

• NSR/GRES

• DWDM

• 1/10/100G links & LAGs

• LDP

• RSVP

• QoS

• Elephant flows

• MBGP

• Logical systems

• BFD

• 1.8m routes

• 50Gbps flows

• Simulation of 8 NRENs

• Simulation of Internet

exchange points

• Enough load to test

convergence and

protocols

2 Traffic generation and full routing table

200+ tests

• Physical flapping

• Protocol behaviours

• Route propagation

• Interoperability

• Stability

• Memory leaks

3

Outcome: No service-affecting bugs have been introduced into the network for the last four years

4 Troubleshoot exceptions – report to vendor

5

6

Repeat test until all pass

Certify code for release into network

Check for expected behaviour through a series of tests

Test suite automated using Python and Robot

framework, saving three man months per

iteration. Tests can be run nightly in the lab and

any variation in results will be highlighted. Robot

keeps track of configuration changes and when

they occurred.

Testing can also be used to

verify that new

services/features do not

interfere with existing

services on GÉANT

network.

Network Test Automation – JUNOS code certification, benchmarking and

qualification

www.geant.org 28 |

Infinera OCG Band mux

EDFA EDFA

splitter splitter DCI - Groove DCI - Groove

Infinera OCG Groove

Add span noise

Plot Rx sensitivity

Infinera OCG Band mux

50km SMF VOA 1

VOA 2

0,0000002

0,000002

-20 -15 -10 -5 0

pre

-FEC

BER

Rx power dBm

Rx sensitivity v. noise @ 8QAM Spectrum

• Scripting allows to set/read all parameters • Dozens of optical settings for new transponders • Automation allows hundreds of tests to be run overnight

Vary power/ modulation/

FEC

Infinera DTN Infinera DTN

Network Test Automation – Optical Lab Automation

www.geant.org 29 |

Deliver network evolution and

forecast long-term cost of the GÉANT

network

Plan for the replacement of the

existing GÉANT fibre network

Deliver new functionality to the

existing GÉANT platforms and

document how the GÉANT network will

evolve

Develop plans for how new network

infrastructure can be used in the GÉANT

network

Evolution Plans Architecture

Cost modelling & Long-term financial

forecasting capabilities

Technology assessments

Fibre Renewal Plan

Network Evolution Plan – Summary

www.geant.org 30 |

Thank you

www.geant.org

Any questions?

© GEANT Limited on behalf of the GN4 Phase 2 project (GN4-2). The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement No. 731122 (GN4-2).