“Internet2 Presentation”

Brian Stengel, Director of Operations, Kinber

Peter J. Heverin, Project Manager, KinberMike Carey, Network Engineer, Kinber

Jon Paul Herron, Director of Engineering, GlobalNOCLuke Fowler, Senior Manager, Systems Engineering, GlobalNOC

Marianne Chitwood, Director of Operations, GlobalNOC

September 20th, 2012

What is PennREN

• Capital Budget - $128,958,031– Federal Stimulus Funds - $99,660,678– Matching Funds - $29,297,353

• Outside Plant Infrastructure Constructed for PennREN – 48 Stands of NZD Fiber optic Cable– 1700± Route Miles– Outsourced Fiber Maintenance

• 13 Optical Regeneration Service Nodes

• 56 Service Distribution Access Nodes

a high-speed, state-wide, research & education serving healthcare, K-20 and the public good

Project Route

Total PennREN Fiber route is estimated at 1,613 miles.

The route consists of the following;

• 1,086 miles of new aerial construction.• 486 miles of aerial overlash.• 14 miles of new underground construction.• 27 miles of leased underground conduit.

Network Backbone Engineering is 95% complete overall.

3

Project Route

4

PennREN LocationsService Nodes

Lehigh University

401 N. Broad

Penn State Hershey Medical Center

Indiana University of Pennsylvania

Allegheny Center Mall

Slippery Rock University of PennsylvaniaPenn State Erie (Behrend)

Clarion University of Pennsylvania

Penn State Dubois

Penn State – State College

Windstream

Bucknell University

University of Scranton

Amp NodesEast Stroudsburg University of Pa

Kutztown University of Pa

West Chester University of Pa

Millersville University of Pa

Shippensburg University of Pa

Pitt Johnstown

Community College of Beaver Co

Allegheny College

Pitt Titusville

Penn State Hazelton

Access Nodes46 Locations on PennREN fiber

Project Teams

• Hardware procurement• Design• Integration• Project Management• Professional Services

• Plan/Design• Optical engineering• Field services• Professional services

• Staging• Shipping• Configuration• Testing• Turn-up• Field Services

• Plan/Design• Consulting• Engineering• Support• Training

• Fiber OSP• Building entrances• Cabinets, Installation• Splicing• Testing• Last mile solutions

• Earthlink sourcing for OOB MPLS Network

• Last mile solutions

• Service Desk, Tier 1, Tier 2• Systems and Network Tools• Network Engineering• Net, Perf, Change Management• Operations support

Node Installation Status

Segment in Production

Equipment Installed

Under Construction

Co-Location at Member Sites

Construction Diagrams

Optical Core/DWDM System

Packet Core – MPLS

Plan – Build – Integrate

Peering Points (Initial)

Internet

R&E

R&E

Internet

Network Management

External Networks

External Networks

Out-of-band access

In-band management

Performance Measurement Servers at every Service Node

PS1PS2PS3

DB

Operations - SupportGRNOC provides to KINBER:• Service Desk – 24x7x365 call center support, ticket management, technical

support coordination, and workflow support• Network Engineering – Expert network engineers work with the Service Desk

to ensure fast problem resolution, provisioning, and strategic engineering and planning

• Software and Systems – Provides support through a fully integrated system of network management, measurement, and visualization tools

The GlobalNOC at Indiana University provides carrier-grade operations, tools, and network expertise while placing a singular focus on the unique requirements of the research and education (R&E) Community

GRNOC supports 20+ R&E networks across the country

Operations - Maintenance

• Warranty• Hardware replacement

(pre-ship)• Software maintenance

• Warranty• Hardware

replacement (pre-ship)

• Software maintenance

• Fiber Maintenance• Emergency Restoral• Routine

Maintenance• OSP Records

• Host IT/Site teams provide remote eyes and hands support upon request

• KINBER Network Engineers

Foundation for Services

West DWDM Ring

East DWDM Ring

MPLS PE Switches

ASBR Router

Multi-DegreeROADMs

External Networks

External Networks

3rd Party Providers

On-Net: Delivery

Internet

R&E R&E

Internet

AccessNode

AccessNode

AccessNode

AccessNode

Access Nodes

Off-Net: Delivery

Internet

R&E R&E

Internet

MemberOff-Net

First/Last MileCommercial

Co-Lo

CXCarrier

Private

NNI

PennREN

EPC – Ethernet Port Connection Service

Customer Router

Service Node: PE Switch

Customer Subscribes to 1/10GE Ethernet Port(s)

R&E networks

VPLS (pt – pt, multipoint)

KMEX

Commodity Internet

Services• Services available to a customer with an EPC

R&E Transit

KINBER Member Exchange

VPLS – Pt-Pt, Multipoint

Commodity Internet

Member A

Member AMember C

Member B

Services• Community-wide, distributed service for members to

exchange traffic across a common network. Similar to an Internet Exchange

• Member-to-Member peering, ad-hoc R&E activities• Best-effort traffic exchange within the community

R&E Transit

KINBER Member Exchange

VPLS – Virtual Private LAN Service

(Point-to-PointMulti-Point)

Commodity Internet

• Transit service to major R&E networks such as Internet 2, ESNet, NLR…

• Provided by KINBER affiliates

• Virtual private networks with committed bandwidth can be established using VPLS instances in the PennREN network

• VPWS – Virtual Private Wire Service – Point to Point• VPLS – Virtual Private LAN Service - Multipoint

• Access to commodity Internet service is available over the PennREN network

Optical – Wave/Lambda Services

• Optical Waves 10G can be provisioned across the networkMember AMember A

Member AMember B Member B

• PerfSONAR Measurement Archives allow exchange of data with other network operators

• Regularly scheduled testing across the backbone

• User-initiated testing for applications such as problem diagnosis

• Multiple routing tables allow us to support both 1G and 10G testpoints on a single host

Performance Measurement

Performance Measurement

Deploying performance measurement servers at each service node

3 servers per site:– Active throughput measurement (1G and 10G)– Active latency measurement– local data collection / ad hoc performance measurement

Specific performance measurement tools include:

BWCTL

OWAMP

MaDDash

24

Network Measurement

• GlobalNOC tool-set updated to support all PennREN devices, including:– SNAPP – High-resolution SNMP-based network

utilization data– LLAMA – DWDM layer performance measurement

data– Central storage of other passive data like syslog,

configuration, flow, etc. for regular and ad-hoc processing & analysis.

PennREN Service Desk

• GlobalNOC Specialized Support Technician • Footprints PennREN project created – integrated to

TickMon, Operations Calendars, Trouble Tickets• Telephone number for PennREN customers integrated

into shared GlobalNOC phone queue• Email established noc@pennren.net to receive customer

inquiries and or communications from vendors….this is monitored 24x7x365

PennREN Service Desk

• Network and Member impact guidelines defined

• Web form for customers/vendors to submit trouble tickets into Footprints

• Change management process/form created• Internal documentation developed/published

for staff training and reference

PennREN Service Desk

• Pro-active network monitoring• Support for scheduled maintenances and

changes• Vendor coordination• Customer install process• Reporting• Security• Tools and Communications

Weighing network needs

A new network is more than a construction project

• It’s a system:– Goals– Infrastructure– People– Services– Operations– Business

• Make decisions and plans early (also, there will be more decisions than you think, so watch for bottlenecks)

• The work on all of these areas starts right away

Service Definition

• Unrealistic to imagine services will be completely defined from the beginning of the design

• But, enough understanding is needed to guide the design and build plan

• Early “anchor” users help a lot!

Communications

• Communication needs:– to be high-bandwidth– Early is good– Changing staff can be disruptive, do it carefully.– Informal is good– Multi-channel is good– reliable technologies are good– Face to face is good

Documentation

• Documentation:– Have a place for documents– Keep the place for documents clean– Keep the purpose of each document clear/distinct

Operations

• Operations Preparedness:– Start early, there are a LOT of things to think

about, especially:• Turn-up/acceptance process• Expectations for facilities• How and where to keep network data

Build-outs make for lots of information

• Need to be ready to put it somewhere where it’ll be usable later

• Data entry is cheap• Entering/documenting is best when the

information and its context are fresh

The lowly management network

• Sometimes doesn’t get enough attention• It can get crazy complicated or crazy expensive• NOTE: in an SDN world, this becomes even

MORE important!