39
Deutsches Forschungsnetz

Deutsches Forschungsnetz

  • Upload
    lilly

  • View
    48

  • Download
    0

Embed Size (px)

DESCRIPTION

Deutsches Forschungsnetz. Roll-out of a National Research Network - a Field Report. Spring Internet2 Member Meeting Martin Wilhelm German National Research Network DFN [email protected]. Why a new network for DFN ?. - PowerPoint PPT Presentation

Citation preview

Page 1: Deutsches  Forschungsnetz

Deutsches Forschungsnetz

Page 2: Deutsches  Forschungsnetz

Roll-out of a National Research Network -a Field Report

Spring Internet2 Member Meeting

Martin WilhelmGerman National Research Network DFN

[email protected]

Page 3: Deutsches  Forschungsnetz

Seite 3

• Contract with operator (TSI, subsidiary of Deutsche Telekom) ended at the end of 2005

• Market situation had changed significantly (availability of dark fiber, lower prices)

• Requirements of users had changed a lot. In particular increased demand for– high capacities between relatively few sites– improved reliability and availability– more capacity for same / lower price

Why a new network for DFN ?

Page 4: Deutsches  Forschungsnetz

Seite 4

The „Old“ Network G-WiN

G-WiN

Leipzig

Berlin

Frankfurt

Karlsruhe

Garching

Kiel

Dresden

Aachen

Regensburg

Kaiserslautern

Augsburg

Hannover

ErlangenHeidelberg

IlmenauWürzburg

Magdeburg

Oldenburg

Essen

St. Augustin

Core Nodes10 Gbit/s2,4 Gbit/s2,4 Gbit/s622 Mbit/s

Status: July 2005

Rostock

Global Upstream Hamburg

Stuttgart

Braunschweig

Marburg

Bielefeld

Göttingen

Page 5: Deutsches  Forschungsnetz

Seite 5

Pre-Tender: Market Survey

• Market Survey started in early 2004

• Constraint: No budget for investments, so:– asking for offers of DWDM-services (no

purchase of equipment)– asking for offers of rental of DF (no IRUs with

upfront payment)

• Results of Survey:– there is a market for DF in Germany (not the

case when tendering for the „old“ Network)– a number of competitors can offer DWDM

services at reasonable prices

Page 6: Deutsches  Forschungsnetz

Seite 6

Preparing Call for Tender

• The top 60 sites (by volume) were defined as potential end points for dark fiber (DF) as well as for Managed Wave Lengths (MWL), resulting in a 60 x 60 matrix

• Bidders were asked to offer options for DF and / or MWL between any of two sites

• In another lot bidders were asked to offer DWDM-services on DF provided by DFN

• In a 4th lot offers for network supervision were solicited (24 * 7 * 365)

Page 7: Deutsches  Forschungsnetz

Seite 7

Results of Tender (1)

• DFs were offered between a significant number of DFN sites at reasonable prices

• Comparing prices for MWL and cost for DF plus DWDM-services showed in many cases economic benefits for DF-solution from 2nd wavelength on (surprise, surprise ...)

• For a few areas there was either no offer for DF or offers were economically not reasonable (the German DF badlands ;-))

Page 8: Deutsches  Forschungsnetz

Seite 8

Results of Tender (2)

• As a nice side effect, offers of DF could be arranged to build a number of rings. In total X-WiN now spans across approx. 5,500 fiber kilometers (approx. 3440 miles)

• These rings then were complemented by a number of MWLs

• Increased number of nodes reduced number of access lines cost savings

Page 9: Deutsches  Forschungsnetz

Seite 9As of 18.01.2006

X-WiN-Topology(Fibers and Managed Wavelengths)

Fiber KPN

Fiber GL

Fiber GC

Fiber DFN

Managed WL

ESF46 Core Sites (+ 3 CBF)

•31 connected by Fiber

•10 connected by Managed WL

• 9 connected by fiber and MWL

KOP

ENS

STB

BAY

BIR

GSI

WUE

GOEKAS

MAR

GIE

GRE

GAR

ERL

MUE

FZJ

AAC

HAM

DES

POTTUB

FZK

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

HUB

ADH

EWE

AUG

FFO

AWI

SAA

KEH

Richtung Basel

BAS

Page 10: Deutsches  Forschungsnetz

Seite 10

Another topological view

TUB

POT

ADHHUB

MAG

BRA

BAY

ILM

LEICHE

DRE

JEN

FRA

BIR

FZJ

AAC

DUI

MUE

BIE

HAN

BRE DESHAM

ROS

REG

GAR

STUFZKHEI

GSI

27

39

32

69.7

168.5

62.68

121.73

132.30

274.42155,2060.50

91.00

69.3

188.98

138.10

55.00

268.8

141.6

145.0

139.35

156.9

196.0010.00

249.00

380.11

107.0

76.88

179.70

102.2

120.20

144.9

104.3

300.24

16

138.00

230.4

KIE

B

THU

E

F

G

I

J

K

L

M

N

O

P

Q

R

S

T U V

W

X

Y Z

AA

AC

AD

AL

AM

AE

AFAG

AIAJ

AK

ZIB

ESFL-S1

79.71

A

CAH

ERL

D1

D2

265.58

158.30

H

FFO

AWI

operational

Under construction

AN AO115.00

132

Kehl-S2102

154.00AP

BAS

Page 11: Deutsches  Forschungsnetz

Seite 11

DWDM Equipment

• Contract (provision of DWDM-services) awarded to Huawei

• Includes installation, operation and maintenance of DWDM equipment (carrier grade DWDM) as well as amplification (OLA)

• Contract defines SLAs (and penalties) and obligation for pro-active 24*7 monitoring

• Access to performance data (read-only) via open non-proprietary interface is part of contract

Page 12: Deutsches  Forschungsnetz

Seite 12

Wavelength Planning

• Task 1: Substitute G-WiN by X-WiN as hassle-free as possible (minimize reconfigu-ration and relocation of existing routers)– done by (almost) 1:1 copy of existing SDH-links

to wavelengths (STM-16, STM-64)

• Task 2: Integrate new DFN sites– this required a number of new routers. Some

reconfiguration could not be avoided

• Task 3: Optimise wavelength plan after the dust has settled (current work)

Page 13: Deutsches  Forschungsnetz

Seite 13

Roll Out Stage 1

• Co-ordination of providers (DF and DWDM)– exchange of fiber info (lengths of fibre spans,

attenuation of spans, location and access regulations of amplifier huts, contact details as phone, fax, e-mail of hotlines, ...)

• Preparation of DFN-sites– creating space for additional equipment (DWDM)

at „old“ sites– preparing grounds (sometimes from scratch) in

„new“ sites: installation of cabinets, power, UPS, A/C, patch fields, out-of-band access, ...

Page 14: Deutsches  Forschungsnetz

Seite 14

Roll Out Stage 2

• Verification of WL routing– Optimal WL routes proposed by Huawei experts– Cross-checked by DFN and verified against IP

routing

• Building of the (in)famous last mile– Almost all DFN-sites are on university / research

lab campuses. All (but two) sites have dual connections. Cross-check that all fiber routes are built with full redundancy (sometimes difficult because of unadequate infrastructure or stubborn administrations)

Page 15: Deutsches  Forschungsnetz

Seite 15

Roll Out Stage 3

• Installation of Optical Line Amplifiers (OLA)– Early installation was possible, because all long

distance spans of fibers as well as amplifier huts were already in place

• Installation of DWDM equipment at DFN sites

Page 16: Deutsches  Forschungsnetz

Seite 16

Some Installation Impressions

Page 17: Deutsches  Forschungsnetz

Seite 17

More Impressions

Page 18: Deutsches  Forschungsnetz

Seite 18

Even more Impressions

Page 19: Deutsches  Forschungsnetz

Seite 19

Roll Out Stage 4

• Link Commissioning– five teams working in parallel to set up basic

interworking between adjacent DWDM locations. Since in most cases OLAs were required, this was done step by step: DWDM A -> OLA1 -> OLA2 -> DWDM B -> OLA3 -> DWDM C, then back to set up the other direction.

– Detailed planning was necessary to minimize travel time between sites (problem of traveling sales man ;-)).

– Average fuel consumption per working day: approx. 70 l per team

Page 20: Deutsches  Forschungsnetz

Seite 20

Roll Out Stage 5

• Service Commissioning– again five teams in parallel working to set up

end-to-end wavelengths including the digital interfaces (STM-64, STM-16, 10 GE, 1 GE), fine tuning of lasers etc.

– again quite a bit of travel was involved, although remote set up and calibration was possible at this stage

Page 21: Deutsches  Forschungsnetz

Seite 21

Roll Out Stage 6

• Switch over from „old“ G-WiN SDH-links to „new“ X-WiN wavelengths

• The first approx. 10 links could be set up in parallel to existing SDH-links. This worked well. Good for building confidence ;-)

• Since interfaces at all of our routers could not simply be doubled, most of the core links had to be „hot swopped“ (no old links in parallel standby). Some nervosity could be noticed ...

Page 22: Deutsches  Forschungsnetz

Seite 22

Timeframe was a bit tight ...

• The only thing we were sure of was the cut off date for G-WiN: 31 December 2005 ...

• Fortunately, remarkably few things went wrong:– 4 or 5 wrong patches, a few faulty patch cables,

three faulty boards, one weak laser etc etc

• Last G-WiN link switched off on 29/12/05

• A German proverb says: „Luck is with the competent“ :-)

• The following slides show how time went by

Page 23: Deutsches  Forschungsnetz

Seite 23

FFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Roll out X-WiN (15/09/05)

KEH

BAS

Page 24: Deutsches  Forschungsnetz

Seite 24

FFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Roll out X-WiN (26/09/05)

KEH

BAS

Page 25: Deutsches  Forschungsnetz

Seite 25

FFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

Roll out X-WiN (29/09/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 26: Deutsches  Forschungsnetz

Seite 26

FFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

Roll out X-WiN (02/10/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 27: Deutsches  Forschungsnetz

Seite 27

FFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

Roll out X-WiN (13/10/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 28: Deutsches  Forschungsnetz

Seite 28

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

Roll out X-WiN (31/10/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 29: Deutsches  Forschungsnetz

Seite 29

Roll out X-WiN (07/11/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

KOP

Page 30: Deutsches  Forschungsnetz

Seite 30

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

KOPRoll out X-WiN (18/11/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 31: Deutsches  Forschungsnetz

Seite 31

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

KOPRoll out X-WiN (21/11/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 32: Deutsches  Forschungsnetz

Seite 32

Roll out X-WiN (30/11/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

KOP

Page 33: Deutsches  Forschungsnetz

Seite 33

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

KOPRoll out X-WiN (05/12/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 34: Deutsches  Forschungsnetz

Seite 34

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

KOPRoll out X-WiN (14/12/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 35: Deutsches  Forschungsnetz

Seite 35

EWEFFO

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

KIE

ROS

LEI

JEN

ESF

HUB

ADH

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

BAS

KOPRoll out X-WiN (19/12/05)

DF delivered

DF ordered

MWL ordered

MWL delivered

DFN-Site

Site infrastructure ready

DWDM installed

Page 36: Deutsches  Forschungsnetz

Seite 36

Today:

• Everything now on X-WiN

• First links of old network were cut off on 2 January 2006

• No operational issues so far ...

Page 37: Deutsches  Forschungsnetz

Seite 37

Cross Border Fibers

FFO

KIE

HUB

ADH

EWE

GIE

GAR

ERL

BAY

MUE

FZJ

AAC BIR

HAM

DES

POTTUB

FZK

GSI

DUI

BRE

HAN

BRA MAGBIE

FRA

HEI

STU

REG

DRE

CHE

ZIB

ILM

ROS

LEI

JEN

ESF

AWI

GOEKAS

MAR

GRE

WUE

AUG

SAA

KEH

PSNC

Switch/GARR

Renater

CPH

(PL)

(NL)

(FR)

CBFDFN PoPDark fibreWavelength

Surfnet

Page 38: Deutsches  Forschungsnetz

Seite 38

The Idea behind CBFs

• Connectivity between NRENs usually via GÉANT

• Cross Border Fibers complement existing GÉANT infrastructure and can add resilience

• CBFs have a big potential for savings

• Will be part of European infrastructure - not a testbed !

• Challenge: Operational issues. Requires definition of technical, operational, organisational and financial framework

Page 39: Deutsches  Forschungsnetz

Seite 39

Questions ...?

? ??