The ShadowNet ProtoGENI Measurement Infrastructure Jim Griffioen Lab for Advanced Networking University of Kentucky Lexington, KY Kobus Van der Merwe AT&T

The ShadowNet ProtoGENI Measurement Infrastructure

Jim GriffioenLab for Advanced

NetworkingUniversity of Kentucky

Lexington, KY

Kobus Van der MerweAT&T Labs - Research

Florham Park, NJ

Other Project MembersZongming Fei (Kentucky)

Eric Boyd (Internet 2)

OutlineProtoGENI ShadowNet

Leveraging AT&T ShadowNet

March 17, 2010GEC7

ProtoGENI ShadowNet

March 17, 2010GEC7

Project OverviewProblem: ProtoGENI backbone router resources are

limited and can be challenging to use.

Idea: Leverage the logical router features of Juniper routers to dynamically create virtual routers (slivers) in the backbone that provide carrier-grade performance and services.

Challenge 1: Creating the control software needed to virtualize the Juniper M7i and integrate with the ProtoGENI network

Challenge 2: Make it easy for users to “see” what is happening on their backbone router slivers.

March 17, 2010GEC7

Project Goals1. Deploy “virtualizable” commercial routers

(Juniper m7i) in the ProtoGENI backbone that support commercial OS/software.

2. Add software support to these virtual routers that will enable per-slice monitoring and measurement.

3. Develop tools and interfaces that will allow slice users to use the measurement infrastructure in simple and easy ways.

March 17, 2010GEC7

March 17, 2010GEC7

Source: http://groups.geni.net/geni/attachment/wiki/presentations/protogeni_Ricci_gec3.pdf

ProtoGENI Network

March 17, 2010GEC7

Source: http://groups.geni.net/geni/attachment/wiki/presentations/protogeni_Ricci_gec3.pdf

ProtoGENI Shadownet Sites

Year 1

Year 2

ProtoGENI Backbone Node Architecture

March 17, 2010GEC7

Sli

ver

1

Net

FP

GA

Net

FP

GA

Net

FP

GA

Sli

ver

n

General PurposeSlivers

Gigabit Ethernet Switch

Internet 2

Non-sliced PCSliced PC


March 17, 2010GEC7

LogicalRouter 1

LogicalRouter 2

LogicalRouter n

Juniper M7i Router

Sli

ver

1

Net

FP

GA

Net

FP

GA

Net

FP

GA

Sli

ver

n

ShadowBoxController

JuniperComponent

Manager

VirtualServer



Internet 2


ShadowBox Router


March 17, 2010GEC7

LogicalRouter 1

LogicalRouter 2

LogicalRouter n

Juniper M7i Router

Sli

ver

1

Net

FP

GA

Net

FP

GA

Net

FP

GA

Sli

ver

n

ShadowBoxController

JuniperComponent

Manager

VirtualServer

perf

SO

NA

R 1

perf

SO

NA

R n


MeasurementSlivers


Internet 2


ShadowBox Router

Leveraging AT&T ShadowNet

March 17, 2010GEC7

Why ShadowNet? ShadowNet is roughly addressing same problem as GENI, however

Less clean slate… Focus on services and network management…

Need the ability to more rapidly evolve the way we run our network and the services we offer in our network (pull): Inherently difficult:

– Potential impact to existing services Networks are shared, new service/feature might negatively interact with existing services

Gets worse with time: networks are “cumulative” (hardly ever gets switched off)

Very long test cycles

– Need for support systems Configuration management, network management, service monitoring, provisioning, customer interfaces,

billing, fault management

Legacy lock in: Existing (complicated) systems need to be modified to support new services

Extremely long development time

New vendor technologies (push): Programmability and virtualization available from major vendors

– Allow non-vendor code to execute on routers

– Loosen the tight coupling between physical boxes and logical functions

Rethink the way we deploy services and operate our network

ShadowNet as (part of) a solution “National footprint” network/platform/testbed for research and service

trials– Connected to, but separate from production network

Limit impact on operational network

Look like a customer to AT&T network

– In between lab and production Stable enough for service trials

Open/flexible enough for research experiments

– “General purpose”, shareable testbed facility Would like to make this a widely available/useful facility, akin to general purpose computing facilities

The role of ShadowNet: Operational (but non-production) environment to enable:

– Evaluation of new technologies/vendor capabilities No impact on existing network/services

– Service testing/trials in a realistic environment (including customer trials) Utilize virtualization and partitioning capabilities to limit interaction and reduce risk

– Evolution of network support systems Free from legacy lock

– Research in operational setting Both networking and “Internet services”

Safe playground for network evolution– This model might become the way we want to build our network

ShadowNet node architecture

Operational nodes:

Richardson, TX

Pleasanton, CA

Chicago, IL

Waiting for network connectivity:

Middletown, NJ

Page 14

JuniperM7i

Router

Router

Sun Fire X4150 Server







Cisco Catalyst 3560G-48TS

GigE

ShadowNetrack

• Each node:

– “Gateway” router, Juniper M7i

– 2 X GigE connectivity to AT&T network

– 7 X SunFire x4150 servers

– 2 X “multiservice” routers, Juniper M7i

– Cisco GigE switch (Catalyst 3560)

– OOB access

• AS 5105:

– Full BGP table

– 4 /24 prefixes

– Advertise up to /32

Set of building blocks that can be flexibly combined into an operational network (or networks)

Page 15

ShadowNet

•Sharable and composable infrastructure

•Strong separation between physical and logical devices:

•Physical machines -> virtual machines

•Physical routers -> logical routers

•Physical links -> logical gigE links: pseudowires, tunnels, VLANs etc

•ShadowNet slices consist of logical devices that have been plumbed together

•However, allow allocation of physical devices to a slice

March 17, 2010GEC7

Life cycle of ShadowNet devices

March 17, 2010GEC7

Using ShadowNet

•CloudNet experimentation• Combining cloud computing with VPN

• Fairly elaborate setup involving many components• Create VPLS VPN between three sites• Prototype dynamic VPN connectivity• Experiment with (live) virtual machine and storage migration• Mechanisms for optimizing WAN migration

In the works:• Cloud control architecture• Slice with bunch of VMs for “architectural support for network debugging”• Declarative approach to network management• Extend to provide mobility functionality

"The interesting thing about cloud computing is that we've redefined cloud computing to include everything that we already do. I can't think of anything that isn't cloud computing with all of these announcements.”

Larry Ellison, CEO Oracle

Wall Street Journal, September 26, 2008

Enterprise Cloud Challenges

Existing cloud platforms do not meet the needs of enterprise customers

Insufficient security controlsNeed isolation at server and network level

Deployment is difficult - transparencyCloud resources are completely separate from local onesCan’t make VMs look like part of existing enterprise

network

Limited control over network resourcesCannot specify network topology or IP addressesCannot reserve bandwidth or request QoS guarantees for

network links

Page 18

Page 19

CloudNet Enterprise-Ready Virtual Private Clouds

•Use VPNs to separate customer resources

•Customer’s cloud resources are only reachable from other VPN end points

•More flexible control of how IP addresses are assigned

•Physical network is transparent to customer

•Assume a virtual machine abstraction

CloudNet writeup (intranet):

http://www.research.att.com/~kobus/docs/cloudnet.pdf

VPNs provide both network resource isolation and strong security

Page 20

Virtual Private Clouds

Virtual Private Cloud:

• Collection of cloud resources presented to customer as a private set of cloud resources, transparently and securely connected to customer VPN

• Manage network resources in the same dynamic manner as cloud resources

Server

PE

PE

VPN AVPN A

VPN AVPN A

VPN BVPN B

VPN BVPN BServer

Server

Cloud Site X

AT&T Backbone

PE

PE

Server

Cloud Site Y

Server

VPC A

VPC B

Page 21

System/Architecture Components

Server

Server

Server

Server

Server

PE

CERouter

PE

VPN AVPN A

VPN AVPN A

VPN BVPN B

VPN BVPN B

NetworkManager

High level abstraction:

• Create compute resources

• Map into VPN

• Cross domain interaction

CloudNet Portal

Cloud Manager

Cloud Platform

AT&T Backbone PE

Network DomainCloud Domain

Cloud Manager:

• Create compute resources

• Map into VPN (cloud side)

Network Manager (IRSCP):

• VPN management (network side)

Page 22

Cloudnet in ShadowNet:Physical nodes involved CloudNet slice

AT&T backbone(7132)

AT&T backbone(7132)

JuniperM7i









ShadowNetrack

JuniperM7i









ShadowNetrack

GRE tunnels

JuniperM7i









ShadowNetrack

PLTN

CHCG

RCSN

Page 23

Cloudnet in ShadowNet:VPLS MPLS VPN in a slice

P1 P3

CiscoSwitchP1 Juniper

RouterCiscoSwitch P3Juniper

Router

VLAN-CCCVLANVLAN VLAN VLAN

Logical link: VLAN cross connect example

PLTN5

RCSN6

CHCG6

PE1 P1

RR/IRSCP P3 PE3

P2 PE2Logical tunnel

VLAN circuit cross connect

Physical ethernet

PLTN

RCSN

CHCG

Page 24

VM migration across WAN

•Ipsec client on laptop provides remote access to VPN•Run game server on VM•Run game client on laptop•Game server move with VM

•Application very sensitive to network impairments•Client monitor typically shows game detects minor changes

•VM migration across WAN “just works” using VPLS VPNs•Optimize for WAN conditions:

•Storage: moving between asynchronous and synchronous replication•VM: optimizing migration logic + redundancy elimination

PLTN5

RCSN6

CHCG6

PE1 P1

RR/IRSCP

P3 PE3

P2 PE2

VpnRemap

PLTN

CHCG

RCSN

r0

r0

drbd

VM0

GameServer

VM0

GameClient

Laptop

ipsec

Thank You!

Questions?

This material is based upon work supported in part by the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of GPO Technologies, Corp, the GENI Project Office, or the National Science Foundation.

Documents

The ShadowNet ProtoGENI Measurement Infrastructure Jim Griffioen Lab for Advanced Networking University of Kentucky Lexington, KY Kobus Van der Merwe AT&T