Design and Procurement of GRNET4 - TERENA and Procurement of the new Layer2/3 network ... * A competitive dialog process could be very helpful ... GPL price > 25M € for Juniper/Cisco

Design and Procurement of the new Layer2/3 network

Andreas PolyrakisNOC Technical Manager

[email protected]

Terena Network Architects Workshop21‐22 November 2012, Copenhagen

GRNET4

Terena Network Architects Workshop, 21-22 November 2012, Copenhagen

2

What to expect

“Design and Procurement of the new Layer2/3 network”

What to expect: Design principles and decisions Approach, methodologies, lessons learned

What not to expect: Not a pure presentation about technologies Not a presentation about the procurement of our fiber

network and DWDM equipment


3

GRNET DWDM and optical network >9.000km of dark fiber

(15-year IRUs) DWDM in most parts

RED:ALU 1626LM LH, up to 21x10Gbps

GREEN:ALU 1626LM Reg, up to 16x10Gbps

YELLOW: ADVA FSP2000, up to 4x1Gbps

BLACK:No DWDM, just Ethernet switches with1GE links

The DWDM equipment and Fiber Network are also about to be re-procured


4

GRNET L2/L3 POPs

>40 PoPs all over Greece Large, Medium: IP router(s) Ethernet

Switch(es)

Small: Ethernet

Switch(es)

2 (+½…) DataCenters +1 planned…


5

Why Upgrade? (1) Lack of new features on the old hardware

No reason to exist if we offer the same as commercial ISPs No funding unless we produce new services or get involved into GEANT/EU

projects No fun for ourselves Also: EOL platforms

Cost of 10GE ports / no 100G support

Vendor/Platform mess Routers:

Cisco 12000, Cisco 7200, Cisco 7300, Juniper T, Juniper MX Switches:

Cisco 2970, Cisco 3750, Cisco 6500, Extreme X450/X450a/X350, Juniper 4200 Protocol interoperability issues No feature parity between platforms, eg RPKI, BGP flowspec Increased OPEX (daily ops, contracts & support, cases) Difficult or impossible to deploy new services (eg AutoBahn/BoD)


6

Why Upgrade? (2)

URGENT: Get rid of Ethernet as a WAN transport technology Spanning Tree problems Very poor implementations of MST, awful interoperability

Common denominator, rapid spanning tree. Awful use of capacity, impossible to T.E.

Hard to safeguard from loops within customers

Hard to monitoring & troubleshooting Ethernet services Vlan propagation (VTP/GVRP/MRP) Scalability, exposure to customer MACs Downtime, increased OPEX


7

ApproachSet up a team (6 members + 3 reviewers) with the following duties: Prepare: 2-4Q2011

Gather requirements Carry out the dimensioning Examine alternative technologies & topologies Choose the architecture, draft the solution, produce the requirements for the RFP Present to the administration, receive approval

Design: Discuss with vendors, finalize technical details design and requirements Draft the deployment/upgrade/replacement procedure

Procure: Produce the RFP 1Q2012 Publish the RFP 3Q2012 Evaluate the proposals Award the contract 4Q2012 (estimated)

Life Span of GRNET4: at least 5+years, estimated ~2020. Financial crisis in Greece may have an impact on this!

The same team would examine the procurement of the optical network. At that time it was unclear if those two procurements would be published in parallel or if one of them (and which one) would proceed. That was also to be decided by the team.


8

Services Internet connectivity

IP feed (through BGP). IPv4/v6 unicast/multicast Various features (protocols/functionality) are necessary. Very big and heterogeneous list

Layer 2 (Ethernet) VPN services connect sites of the same customer connect to other customers connect to GRNET datacenters

(combined with our VPS service)

Comments: No requests for L3 VPNs (service decommissioned) Sparse QoS requests However, good to be able to support both…


9

Architecture

Service oriented approach

Separate IP, Transport (Carrier and Access) and DataCenterequipment.

Rules about roles. eg when a CPE switch needs to be upgraded to a Carrier Node.


10

Dimensioning - Questions

A Traffic Analysis and projections was conducted in order to dimension the network Customers’ capacity needs Number and locations of POPs of the IP network Capacities of the IP nodes, categorization and

capacities of Carrier nodes and CPE switches Capacities of IP/Carrier backbone, access, uplink, IX

links is 40G/100G necessary?


11

Dimensioning - Results Capacity planning

Customers: Majority@1Gbps, few @10Gbps. 2x-3x10Gbps might be seen in the future (2015+)

Backbone for Carrier/IP: Minimum 10Gbps links (although this is over-engineering in many cases) Multiple of 10Gs on the “core”

GEANT & IX projection (2015): 2x10G No 40G support. 100G may be used in the mid-term future.

Location of IP POPs 93% of traffic is sourced from/destined to Athens (incl. IX & GEANT traffic) Max. one-way transmission delay from Athens to any POP: <7ms Makes sense to keep IP POPs in Athens only

… but Thessaloniki is kept as a future option (potential 2nd exit point)

Sizing of Carrier Nodes: 2 Large, 13 Medium, 26 Small 2 flavors of CPE switches: Large and Small


12

GRNET4 Topology


13

IP Network - Functionality Collapsed IP network

a small number of nodes with advanced functionality: protocols & features, esp. Routing Firewalling / (c.g.)NAT / IPS & IDS Monitoring/netflow/IPFIX

Decreased cost, easier upgrade of feature-set Carrier and DWDM networks deliver connectivity to IP network Backbone links, Access links, Datacenter connectivity, Upstream(s) and IX

Design 2 POPs in Athens for redundancy; one IP router per POP Active/active setup, identical hardware, software, configuration Each resource (customer, IX, GEANT, datacenter) will be double-

homed (over Carrier or DWDM network)


14

Carrier Network – Functionality

Basic functionality: Transfer of Ethernet Frames between two points (E-LINE) or point-to-multipoint (E-LAN) E-Lines would be used to connect customers to our IP network E-Lines and E-LANs would be used to connect sites of customers

(VPN functionality) (E-trees: a neat feature…?)

Traffic Engineering Fast Reroute

Support for QoS / Shaping / Traffic Profiles OAM functions to monitor and troubleshoot Smooth handover and cooperation with Access Network

(esp. in terms of management and OAM)

MEF standards, terminology etc should be used


15

Carrier Network – Technology candidates

Technology candidates: MPLS-TP, PBB/T, EoMPLS Choice of the technology had to be done on an early stage* Great impact on procurement Had to be certain about the choice

Evaluation/Validation Method: Participation in conferences and Fora Feedback from GEANT community:

Ethernet Architecture Workshop with NORDUnet and Surfnet (Copenhagen, March 2012).

Feedback from GEANT and other NRENs (eg PSNC). Endless discussions with vendors PBB/T evaluation on our own lab (Nortel equipment) Studying, studying, studying…

* A competitive dialog process could be very helpful – the technology would not have to be chosen at an early stage. Unfortunately this was ruled out by the administration and legal department.


16

Carrier Network – Technology choice

And the winner is: EoMPLS Already familiar with this technology, no risks Easy deployment/upgrade path Small learning curve GEANT and many NRENs are on the same path

MPLS-TP Too early, products seemed immature Promising but high risk

Not so well suited to our environment PBB/T

Poor vendor support, somehow abandoned Not convinced that it fits our needs


17

CPEs

(not so) Low-cost Ethernet switches Should cooperate smoothly with the Carrier

Network i.e, OAM

Two flavors, large and small Large: +2x10GE interfaces

20/06/2011Ομάδα δικτύου ΕΔΕΤ‐4

17


18

NMS

One NMS to manage the entire network OS management Configuration management, templates Service provisioning, esp E-LINE/E-LAN services, T.E,

OAM

Web interface & Northbound API Customers view & requests Integration with provisioning tools, eg BoD Data Mining


19

Uniformity of equipment

Decision to go with a single vendor Interoperability of carrier/access OAM Better integration with (vendor) NMS

Same platform for IP/Carrier Extra credits for any uniformity

eg single OS, same SDK etc

Attempt to minimize different types of optics (SFPs+/XFPs)


20

Bill of Material

Α/Α Type Quantity

1 CARRIER Small 382 CARRIER Medium 103 CARRIER Large 34 IP Router 25 CPE Small 506 CPE Large 307 DataCenter Switches 58 NMS 1


21

Procurement Budget: 2.6M€,

all inclusive (licenses, features, etc) 5 years of support

GPL price > 25M € for Juniper/Cisco €/$ rate was a big headache…

Status: RFP on air: August 1st, 2012

Quite delayed, due to political and financial situation in Greece… Closing date, Sept 25th, 2012 About to finish the technical evaluation

Rollout: 1-2Q 2013 Optical Network procurement & implementation is necessary to

implement the final topology


22

Thank you

Questions?

Documents

Design and Procurement of GRNET4 - TERENA and Procurement of the new Layer2/3 network ... * A competitive dialog process could be very helpful ... GPL price > 25M € for Juniper/Cisco