Campus Network Design

Brittany Cunningham Victor AntonovTrevor Marsh 8 December 2009

Campus Network Design 2

Table of Contents

1. Design Decisions2. Population &

Needs3. Wide-Area

Network 4. Routing Protocol5. Main Campus6. Satellite

Campuses7. Remote Campuses

7. Remote Access 8. VoIP9. Wireless10. Security and

Authentication 11. Network

Management12. Costs Evaluation

2009.12.08

Design DecisionsBrittany Cunningham

Campus Network Design 4

Why a Hierarchical Design?

Route summarization Distributed routing and switching Simplified implementation and

management Broadcast domain control Infrastructure changes Quality of Service

2009.12.08

Campus Network Design 5

Core and Distribution Layers

2009.12.08

Campus Network Design 6

Population and NeedsVictor Antonov

2009.12.08

User Groups

Students WWW, e-mail, multimedia access

Staff E-mail, VoIP, WWW

Faculty E-mail, VoIP, multimedia/WWW

Research VoIP, e-mail, multimedia

Students

Most student access will come from the dorms but some will be from academic access points

Student needs will be mostly in download bandwidth

Upload (disregarding video upload) is not expected to be great. Illegal upload needs to be discouraged.

Campus Network Design 9

Student Traffic Estimations

2009.12.08

Type of Object Size inKb # objects DL # objects UL traffic DL (MB) traffic UL (MB)

intrainte

rtotal intra inter total intra inter total intra inter total

E-mail message 10 5 30 35 2 10 12 732 4,395 5,127 293 1,465 1,758

Web page 50 10 190 200 1 2 3 7,324 139,160 146,484 732 1,465 2,197

Spreadsheet 100 2 1 3 1 1 2 2,930 1,465 4,395 1,465 1,465 2,930

Word processing document 200 2 2 4 2 1 3 5,859 5,859 11,719 5,859 2,930 8,789

Image view/upload 500 5 50 55 3 20 23 36,621 366,211 402,832 21,973 146,484 168,457

Presentation document 2,000 1 1 2 1 0 1 29,297 29,297 58,594 29,297 0 29,297

5 min songs @ 96 kbps 3,600 0 100 100 0 20 20 0 5,273,438 5,273,438 0 1,054,688 1,054,688

2 hrs of movie @ 256 kbps 230,400 0 1 1 0 1 1 0 3,375,000 3,375,000 0 1,687,500 1,687,500

80.8 8979.3 9060.1 58.2 2828.1 2886.3 GB

859.0359 273.6681total Mbps(24 hrs)

1288.5539 410.5021

total Mbps(16 hrs)

* Estimated 15,000 students

Public Access Traffic Estimations

Type of Object Size in Kb # people # objects DL # objects UL traffic DL (MB) traffic UL (MB)

intra inter total intra inter total intra inter total intra inter total

Terminal screen 4 2,000 30 0 30 20 0 20 234 0 234 156 0 156

E-mail message 10 15,000 5 15 20 2 10 12 732 2,197 2,930 293 1,465 1,758

Web page (including simple GIF and JPEG graphics) 50 15,000 15 30 45 1 2 3 10,986 21,973 32,959 732 1,465 2,197

Spreadsheet 100 15,000 2 1 3 1 1 2 2,930 1,465 4,395 1,465 1,465 2,930

Word processing document 200 15,000 2 2 4 2 1 3 5,859 5,859 11,719 5,859 2,930 8,789

Graphical computer screen 500 5,000 3 0 3 0 0 0 7,324 0 7,324 0 0 0

Presentation document 2,000 15,000 1 1 2 1 0 1 29,297 29,297 58,594 29,297 0 29,297

High-resolution (print-quality) image 50,000 10,000 1 1 2 0 0 0 488,281 488,281 976,563 0 0 0

1 hrs of video stream @ 256 kbps 115,200 5,000 0 1 1 0 0 0 0 562,500 562,500 0 0 0

VoIP 2,400 5,000 5 3 8 5 3 8 58,594 35,156 93,750 58,594 35,156 93,750

532.9 1085.5 1618.4 36.9 7.2 44.1 GB

153.4460 4.1784total Mbps (24 hrs)

230.1690 6.2676total Mbps (16 hrs)

Staff / Administration

Least amount of traffic generated VoIP telephony important Higher UL rate because of audio and

video links

Staff / AdministrationType of Object

Size in Kb # people # objects DL # objects UL traffic DL (MB) traffic UL (MB)

intra inter total intra inter total intra inter total intra inter total

E-mail message 10 200 7 8 15 7 8 15 14 16 29 14 16 29

Web page (including simple GIF and JPEG graphics) 50 200 10 25 35 0 0 0 98 244 342 0 0 0

Spreadsheet 100 200 1 1 2 1 1 2 20 20 39 20 20 39

Word processing document 200 200 3 2 5 2 3 5 117 78 195 78 117 195

Graphical computer screen 500 100 1 0 1 0 0 0 49 0 49 0 0 0

Presentation document 2,000 100 1 1 2 1 1 2 195 195 391 195 195 391

High-resolution (print-quality) image 50,000 100 0 1 1 0 1 1 0 4,883 4,883 0 4,883 4,883

VoIP 2,400 200 10 20 30 5 10 15 4,688 9,37514,06

3 2,344 4,688 7,031

5.1 14.5 19.5 2.6 9.7 12.3 GB

1.8509 1.1637

total Mbps (24 hrs)

5.5528 3.4912

total Mbps (8 hrs)

Research

Most research organizations and universities are connected via Internet2 – a research network

Internet2 is developing and deploying advanced network applications and technologies for research and higher education

Internet2 recreates the partnerships of academia, industry, and government that helped foster today’s Internet in its infancy.

Research partnership gives access to (anonymized) traffic data unavailable from commercial networks

Research Needs

Some areas of research can generate huge amounts of data

A separate line will be dedicated to the research needs and access to Internet2

Needs for some areas of research are described in the next slides

Physics Research

Dependant on the area of physics but usually produces large amounts of data

Russian example on High Energy Physics research In 2003 produced ~30 TB Predicted needed connectivity for 2006 was 1-2.5 Gbps While a university might not produce all this data and

exchange it with the world, it is safe to assume that in 2009-2010 all educational physics research might need ~2 Gbps connection

Some examples of physics research applications: Large, high-quality images of the sky (astrophysics) Complex 3D models (fluid/air dynamics)

Biology/Medicine

Audio and visual information on species, habitats, conditions

DNA models, genetic sequences Neuroinformatics - neuroimaging

resources, including multi-scale imaging

Protein identification, characterization, quantification

Other Areas

Other areas of research that will produce a lot of traffic over the network: Weather science High-performance computing Chemistry Geography

Wide-Area Network Victor Antonov

Wide-Area Network

Main Campus 4 Secondary Campuses

In the same metro area as main campus 50+ satellite campuses

Nationwide Connections to the Internet and Internet2

Serving main and secondary campuses Redundancy of the WAN

WAN Connection

Metro Ethernet technology to connect smaller campuses

EVPL (Ethernet Virtual Private Line) topology with point-to-point Ethernet virtual connections

Multiple EVCs to enable hub and spoke configuration

Bandwidth of 1Gb (which can be later scaled up for growing bandwidth needs)

Two providers for redundancy: COX and Verizon

Metro Ethernet

Cost-effectiveness Scalable bandwidth (1Gb and higher) Low operating, maintenance,

administration costs Simplicity of native Ethernet format

over traditional WAN technologies Customer controls IP addressing and

routing

MAN Implementation

Layer 2/3 switches and/or routers Highly redundant network

Full mesh topology MPLS backbone

Costly Highly reliable and scalable

Multiprotocol Label Switching

Benefits of MPLS (basic) Node-to-node connections (virtual links) Highly scalable Independent of any Data Link layer

technology Less overhead (no segmentation and

reassembly) Highly compatible with IP

MPLS

Benefits of MPLS Connections are unidirectional

▪ A bi-directional traffic will use two connections which allows a link failure to ideally affect only one of the traffic directions

Multi-level tunneling Fast recovery time – MPLS Fast Reroute

offers recovery time of <50 ms▪ Geared towards real-time application (VoIP)

support

MPLS-based Ethernet MAN Ethernet interface on fiber

(100BASE-FX) Ethernet over MPLS over Ethernet

Customers’ Ethernet packets are transported over MPLS and the service provider network uses Ethernet again as the underlying technology to transport MPLS

Fast Reroute Implemented

Advantages of an MPLS-based Metro Ethernet

Scalability pure Ethernet MAN are limited to a maximum of 4,096

VLANs for the whole network, when using MPLS, Ethernet VLANs have local meaning only

Resiliency 30 to 1 sec convergence for pure Ethernet vs 50 msec for

MPLS-based MAN (Fast Reroute) Multiprotocol convergence

an MPLS-based Metro Ethernet can backhaul not only IP/Ethernet traffic but virtually any type of traffic coming from customer networks or other access networks

End to End administration and maintenance MPLS-based MAN offers a wider set of troubleshooting and

OAM MPLS-based tools which can effectively troubleshoot and diagnose network problems

MAC ping, MAC traceroute, LSP ping etc.

MAN Design

University is the provider itself It will receive internet access and provide it to main and

secondary campuses Can provide access for closely related organizations –

research foundation , R&D sites, high schools Operates and administers its own network

▪ Can freely implement policies Main campus is closely connected with the core

network Customers are secondary campuses and an

related organizations (see above)

WAN Redundancy

Two providers of the metro-ethernet services COX and Verizon

Ethernet solutions: EVPL (Ethernet Virtual Private Lines) topology with point-to-point Ethernet virtual connections (EVCs) Multiple EVCs will be used to enable hub-and-

spoke configuration to interconnect campuses.

Satellite Campuses

Separate internet access OC-1 lines offering ~50Mbps

transmission speeds Main BW consumer is distance learning

video links▪ Assuming roughly 120 students per remote

campus, this is 30 Mbps traffic at peak times Access to university resources

achieved through VPN

WAN Overview

MetroEthernet Area Network

(main and secondary campuses

)

Cox

VerizonSatellite Campuses

Routing ProtocolBrittany Cunningham

Campus Network Design 33

Convergence

What determines convergence time? Time to detect path loss Time to detect new best path Time to update routes and tables

2009.12.08

Campus Network Design 34

How does EIGRP help?

Stubby areas Hierarchical design limits queries Fast convergence Cisco hardware is optimized for

EIGRP

2009.12.08

Campus Network Design 35

Route Summarization

Fewer queries to core Allows traffic filtering Control multicast traffic Smaller routing tables Naturally synergizes with

hierarchical design

2009.12.08

Campus Network Design 36

Keeping Multicasts to a Minimum

Rendezvous point near multicast source

Auto-rendezvous on all other L3 switches

IGMP snooping No cross-campus VLANs

2009.12.08

Main CampusBrittany Cunningham

Campus Network Design 38

Main Campus Considerations

15 buildings Approximately 750 faculty and staff Approximately 15,000 students Electronic records VoIP phone system Complete wireless coverage Research

2009.12.08

Campus Network Design 39

Access Layer in a Single Building

2009.12.08

Campus Network Design 40

Server Farm

2009.12.08

Campus Network Design 41

Research Considerations

WAN links to partnered universities High-performance computing

clusters

2009.12.08

Satellite CampusesBrittany Cunningham

Campus Network Design 43

Satellite Campuses

1-4 buildings each Approximately 250 faculty and staff Approximately 8,000 students VoIP phone system Complete wireless coverage Backups from main server farm WAN links to main campus

2009.12.08

Remote Campuses and Access

Brittany Cunningham

Campus Network Design 45

Remote Campuses

50+ remote sites Approximately 2,000 students Local staff with access to university

resources

2009.12.08

Campus Network Design 46

Remote Access

Faculty and Staff must have secure access to files and other resources

Access must be available anywhere with an internet connection

Solution: VPNs

2009.12.08

Campus Network Design 47

VPNs

Consider: What resources should require a VPN? What resources could be supported by

web VPNs? How can we make connecting as easy as

possible? Adaptive Security Appliance

2009.12.08

VoIPBrittany Cunningham

Campus Network Design 49

VoIP

Main and satellite campuses only Traffic is in separate traffic VLAN 802.1Q VLAN tagging to ensure QoS

2009.12.08

WirelessTrevor Marsh

Main architecture

Cisco’s Unified Wireless Network Quality name Guaranteed support won’t end in a year because company

bankrupts Provides easy and proven configurations Offers:

▪ Context Aware: Track assets, perform condition monitoring, improve process flow, and use location and other contextual information

▪ Wireless Network Security: Proactive threat protection, RF visibility, and wired network security help ensure that data remains private and secure and that the network is protected from unauthorized access.

▪ Radio Frequency (RF) Solutions: Spectrum analysis can help detect and eliminate sources of RF interference in wireless networks.

Main Components

Cisco Catalyst 6500 or 7600 series switch After placement of a Cisco Wireless

Service Module(CiSM) you can have up to 2100 access points

Use Cisco Aironet 1250 series access point Allows for upgrade to 802.11n

Centralized Management

Management of all of the access points is easier due to Cisco’s use of LWAPP (Lightweight Access Point Protocol) Handles all of the access points at once Can assign each access point with a

primary and secondary controller Each wireless controller will be

bundled with the switch which will allow access to the distribution layer

WLAN connected to the LAN

This allows for the usage of the same DCHP server and access to anything else in the Distribution Layer, provided properly accessed, such as storage and others.

Broadcast

802.11n is not yet popular enough 802.11a 5.2Ghz band will be

primarily used while 802.11b/g (2.4Ghz) will be sparingly used for legacy devices

802.11a

Potentially less interference Provides at least eight, and potentially

up to 22, non-overlapping channels, compared with three for 802.11b/g

Allows for auto-configuration of channels and power to access points

Failsafe

There will be one controller per switch, which means two controllers per building Placement in all buildings will allow for

enough coverage for all of ODU If one fails the other will automatically

cover the slack Automatic reboot after 3 minutes

If any access point fails, the CiSM will increase the power to the others

Security and Authentication

Brittany Cunningham

Campus Network Design 59

Access Control Lists

Located in Distribution Layers Additional ACLs may be on Access

Layer No ACLs in Core-Why? Careful planning is necessary during

design and implementation

2009.12.08

Campus Network Design 60

Intrusion Detection and Prevention

DHCP snooping Intrusion Detection Systems (IDS) Port security

2009.12.08

Campus Network Design 61

Where should firewalls be?

Resnet gateway Server gateway Between core and exterior gateways Remote site gateways VPN connection gateway

2009.12.08

Network ManagementBrittany Cunningham

Campus Network Design 63

Network Management

TACACS+ for networked devices Authentication Authorization Accounting

Locally-configured credentials as backup

Solarwinds Network Monitoring System

2009.12.08

Costs EvaluationBrittany Cunningham

Campus Network Design 65

Hardware Costs

Item Quantity

Cost per Unit

Total Cost

Catalyst 4500 Series Switch 75 $8,000 $600,000

Catalyst 6500 Series Switch 16 $20,000 $320,000

ASA 5500 Series 2 $3,000 $6,000

Wireless Access Points 1,200 $800 $960,000

Cisco 6500 Wireless Services Module

8 $30,000 $254,000

Cabling Estimate * 1 $1,000,000 $1,000,000

Hardware Overhead (40%) $1,256,000

TOTAL $4,396,000

2009.12.08

* University will hire a contractor for all cabling.

Campus Network Design 66

Non-Hardware Costs

Item Cost

Orion Network Performance Monitor (500 devices) $8,475

Orion Netflow Traffic Analyzer (500 devices) $5,995

Orion IP SLA Manager 1 (25 IP SLA source devices) $3,995

Orion Network Configuration Manager (1000 nodes) $10,495

LANsurveyor $1995

IPv4 Allocation and Assignment (ARIN; /20) $2,250

IPv6 Allocation and Assignment (ARIN; /40) Free w/ IPv4

AS Number Assignment (ARIN) $500

ARIN Maintenance Fee (Per Year) $100

Non-Hardware Overhead (40%) $13,522

TOTAL $47,327

2009.12.08

Campus Network Design 67

Resources

http://www.uwec.edu/hiltonts/101/CBAsample/projectsample.htm

http://cisco.com http://www.ciscopress.com http://www.netcraftsmen.net/resources/archived-

articles/431.html http://etutorials.org/Networking/Lan+switching+first-step http://www.engr.wisc.edu/computing/security.html http://www.solarwinds.com http://www.arin.net Rizwan Bhutta, Network Systems Senior Engineer Sheila Brink, Network Systems Senior Engineer Jeff Spyker, Network Systems Senior Engineer Robert Perry, Network Systems Senior Engineer

2009.12.08

Questions?