By L. Peterson, Princeton T.Anderson, UW D. Culler, T. Roscoe, Intel, Berkeley HotNets-I (Infrastructure panel), 2002 Presenter Shobana Padmanabhan Discussion

bybyL. Peterson, PrincetonL. Peterson, Princeton

T.Anderson, UWT.Anderson, UWD. Culler, T. Roscoe, Intel, BerkeleyD. Culler, T. Roscoe, Intel, Berkeley

HotNets-I (Infrastructure panel), 2002HotNets-I (Infrastructure panel), 2002

PresenterPresenterShobana PadmanabhanShobana Padmanabhan

Discussion leaderDiscussion leaderMichael WilsonMichael Wilson

Mar 3, 2005Mar 3, 2005CS7702 Research seminarCS7702 Research seminar

A blueprint for introducing A blueprint for introducing disruptive technology into Internetdisruptive technology into Internet

OutlineOutline

• IntroductionIntroduction• ArchitectureArchitecture• PlanetLabPlanetLab• ConclusionConclusion

IntroductionIntroduction

Widely-distributed applications make own forwarding Widely-distributed applications make own forwarding decisionsdecisions

Network-embedded storage, peer-to-peer file sharing, content Network-embedded storage, peer-to-peer file sharing, content distribution networks, robust routing overlays, scalable object distribution networks, robust routing overlays, scalable object location, scalable event propagationlocation, scalable event propagation

Network elements (layer-7 switches & transparent caches) Network elements (layer-7 switches & transparent caches) do application-specific processingdo application-specific processing

But Internet is ossified..But Internet is ossified..

Until recently: Recently:

Internet

Figures courtesy planet-lab.org

This paper proposes using This paper proposes using overlay networks to achieve it..overlay networks to achieve it..

Overlay network Overlay network

A virtual network of nodes & logical A virtual network of nodes & logical links, built atop existing network, to links, built atop existing network, to implement a new serviceimplement a new service

Provides opportunity for innovation as Provides opportunity for innovation as no changes in Internetno changes in Internet

Eventually, ‘weight’ of these overlays Eventually, ‘weight’ of these overlays will cause emergence of new will cause emergence of new architecturearchitecture

Similar to Internet itself (an overlay) Similar to Internet itself (an overlay) causing evolution of underlying telephony causing evolution of underlying telephony networknetwork

Figure courtesy planet-lab.org

This paper speculates what this new architecture will look like..

OutlineOutline


GoalsGoals

Short-term: Support experimentation with new Short-term: Support experimentation with new servicesservices TestbedTestbed

Experiment at scale (1000s of sites)Experiment at scale (1000s of sites)Experiment under real-world conditions Experiment under real-world conditions

diverse bandwidth/ latency/ lossdiverse bandwidth/ latency/ loss wide-spread geographic coveragewide-spread geographic coverage

Potential for real workloads & usersPotential for real workloads & usersLow cost of entryLow cost of entry

Medium-term: Support continuous services that Medium-term: Support continuous services that serve clientsserve clients Deployment platformDeployment platform

support seamless migration of application from prototype to support seamless migration of application from prototype to service, through design iterations, that continues to evolveservice, through design iterations, that continues to evolve

Long-term: Microcosm for next generation Internet!Long-term: Microcosm for next generation Internet!

ArchitectureArchitecture

Design principlesDesign principles• Slice-abilitySlice-ability• Distributed control of resourcesDistributed control of resources• Unbundled (overlay) managementUnbundled (overlay) management• Application-centric interfacesApplication-centric interfaces

Slice-abilitySlice-ability

A slice is horizontal cut of A slice is horizontal cut of global resources across nodesglobal resources across nodes

Processing, memory, storage..Processing, memory, storage..

Each service runs in a sliceEach service runs in a slice Service is a set of programs Service is a set of programs

delivering some functionalitydelivering some functionality

Node slicing mustNode slicing must• be securebe secure• use resource control mechanismuse resource control mechanism• be scalablebe scalable


Slice ~ a network of VMsSlice ~ a network of VMs

Virtual MachineVirtual Machine

VM is the environment where a program VM is the environment where a program implementing some aspect of the service runsimplementing some aspect of the service runs

Each VM runs on a single node & uses some of Each VM runs on a single node & uses some of the node’s resourcesthe node’s resources

VM must be VM must be No harder to write programs, protection from other VMs, No harder to write programs, protection from other VMs,

fair sharing of resources, restriction of traffic generationfair sharing of resources, restriction of traffic generation

Multiple VMs run on each node withMultiple VMs run on each node with VMM (Virtual Machine Monitor) arbitrating node’s VMM (Virtual Machine Monitor) arbitrating node’s

resourcesresources

Virtual Machine Monitor (VMM)Virtual Machine Monitor (VMM)

a kernel-mode driver running in the host operating a kernel-mode driver running in the host operating systemsystem

Has access to the physical processor & manages Has access to the physical processor & manages resources between host OS & VMsresources between host OS & VMs prevents malicious or poorly designed applications prevents malicious or poorly designed applications

running in virtual server from requesting excessive running in virtual server from requesting excessive hardware resources from the host OShardware resources from the host OS

With virtualization, two interfaces nowWith virtualization, two interfaces now API for typical services & API for typical services & Protection Interface used by VMMProtection Interface used by VMM

VMM used here is Linux VServer..VMM used here is Linux VServer..

A node..A node..


Across nodes (ie. across network)Across nodes (ie. across network)

Node manger (one per node; part of VMM)Node manger (one per node; part of VMM) When When service managersservice managers provide valid provide valid ticketstickets

Allocates resources, creates VMs & returns a leaseAllocates resources, creates VMs & returns a lease

Resource Monitor (one per node)Resource Monitor (one per node) Tracks node’s available resources (using VM’s interface)Tracks node’s available resources (using VM’s interface)

Tells Tells agentsagents about available resources about available resources

Agents (centralized)Agents (centralized) Collect resource monitor reportsCollect resource monitor reports

Advertise Advertise ticketsticketsIssue Issue ticketstickets to to resource brokersresource brokers

Resource Broker (per service)Resource Broker (per service) Obtain tickets from agents on behalf of Obtain tickets from agents on behalf of service managersservice managers

Service Managers (per service)Service Managers (per service) Obtain Obtain ticketstickets from from brokerbroker Redeem Redeem ticketstickets with with node managersnode managers to create VM to create VM Start serviceStart service

ObtainingObtaining a Slice a Slice

Agent

Service Manager

Broker

Courtesy Jason Waddle’s presentation material


Agent

Service Manager

Broker

Resource Monitor

Resource Monitor

Resource Monitor



Agent

Service Manager

Broker

Resource Monitor

Resource Monitor

ticket

ticket

Resource Monitor

ticket



Agent

Service Manager

Broker

ticket

ticket

ticket



Agent

Service Manager

Broker

ticket

ticket

ticket



Agent

Service Manager

Broker

ticket

ticket

ticket



Agent

Service Manager

Broker

ticket

ticket

ticket



Agent

Service Manager

Broker

ticket

ticket

ticketNode Manager

Node Manager



Agent

Service Manager

Broker

ticket



Agent

Service Manager

Broker

ticket


Architecture Architecture

Design principlesDesign principles• Slice-abilitySlice-ability• Distributed control of resourcesDistributed control of resources• UnbundledUnbundled (overlay) management(overlay) management• Application-centric interfacesApplication-centric interfaces

Distributed control of resourcesDistributed control of resources

Because of dual role of testbed, two types of usersBecause of dual role of testbed, two types of users ResearchersResearchers

Likely to dictate how services are deployed &Likely to dictate how services are deployed &

Node propertiesNode properties Node owners/ clientsNode owners/ clients

Likely to restrict what services run on their nodes & how resources are Likely to restrict what services run on their nodes & how resources are allocated to themallocated to them

De-centralize control between the twoDe-centralize control between the two• Central authority provides credentials to service developersCentral authority provides credentials to service developers• Each node independently grants or denies a request, based on local Each node independently grants or denies a request, based on local

policypolicy



Unbundled (overlay) managementUnbundled (overlay) management

Independent sub-services, running in own slicesIndependent sub-services, running in own slicesdiscover set of nodes in overlay & learn their capabilitiesdiscover set of nodes in overlay & learn their capabilitiesmonitor health & instrument behavior of these nodesmonitor health & instrument behavior of these nodesestablish a default topologyestablish a default topologymanage user accounts & credentialsmanage user accounts & credentialskeep software running on each node up-to-date &keep software running on each node up-to-date &extract tracing & debugging info from a running nodeextract tracing & debugging info from a running node

Some are part of core system (user a/c..)Some are part of core system (user a/c..)Single, agreed-upon versionSingle, agreed-upon version

Others can have alternatives, with a default, replaceable over timeOthers can have alternatives, with a default, replaceable over time

Unbundling requires appropriate interfacesUnbundling requires appropriate interfacesEg. hooks in VMM interface to get status of each node’s resourcesEg. hooks in VMM interface to get status of each node’s resources

Sub-services may depend on each other Sub-services may depend on each other Eg. resource discovery service may depend on node monitor serviceEg. resource discovery service may depend on node monitor service



Application-centric interfacesApplication-centric interfaces

Promote application development by letting it run Promote application development by letting it run continuously (deployment platform)continuously (deployment platform)

Problem: difficult to simultaneously create testbed Problem: difficult to simultaneously create testbed & use it for writing applications& use it for writing applications API should remain largely unchanged while underlying API should remain largely unchanged while underlying

implementation changesimplementation changes If alternative API emerges, new applications must be If alternative API emerges, new applications must be

written to it but original should be maintained for legacy written to it but original should be maintained for legacy applicationsapplications

OutlineOutline


PlanetLabPlanetLab

Phases of evolutionPhases of evolution

1.1. Seed phaseSeed phase 100 centrally managed machines100 centrally managed machines Pure testbed (no client workload)Pure testbed (no client workload)

2.2. Researchers as clientsResearchers as clients Scale testbed to 1000 sitesScale testbed to 1000 sites Continuously running servicesContinuously running services

3.3. Attracting real clientsAttracting real clients Non-researchers as clientsNon-researchers as clients

PlanetLab todayPlanetLab today

Services• Berkeley’s OceanStore – RAID distributed over Internet• Intel’s Netbait – Detect & track worms globally• UW’s ScriptRoute – Internet measurement tool• Princeton’s CoDeeN – Open content distribution network

Courtesy planet-lab.org

Related workRelated work

Internet2 (Abilene backbone)Internet2 (Abilene backbone) Closed commercial routers -> no new functionality in the middle of networkClosed commercial routers -> no new functionality in the middle of network

EmulabEmulab Not a deployment platformNot a deployment platform

Grid (Globus)Grid (Globus) Glues together modest number of large computing assets with high bandwidth Glues together modest number of large computing assets with high bandwidth

pipes but pipes but planetlab emphasizes on scaling the less bandwidth applications across wider planetlab emphasizes on scaling the less bandwidth applications across wider collection of nodescollection of nodes

ABONE (from active networks)ABONE (from active networks) Focuses on supporting extensibility of forwarding function but Focuses on supporting extensibility of forwarding function but

planetlab is more inclusive ie. apps throughout the network including those involving planetlab is more inclusive ie. apps throughout the network including those involving storage componentstorage component

XBONEXBONE Supports IP-in-IP tunneling, w/ GUI for specific overlay configurationsSupports IP-in-IP tunneling, w/ GUI for specific overlay configurations

Alternative: package as desktop applicationAlternative: package as desktop applicationEg. Napster, KaZaaEg. Napster, KaZaa

Needs to be immediately & widely popularNeeds to be immediately & widely popular Difficult to modify system once deployed unless compelling applicationsDifficult to modify system once deployed unless compelling applications Not secureNot secure

KaZaa exposed all files on local systemKaZaa exposed all files on local system

Conclusion Conclusion

An open, global network test-bed, for pioneering An open, global network test-bed, for pioneering novel planetary-scale services (deployment).novel planetary-scale services (deployment).

A model for introducing innovations (service-A model for introducing innovations (service-oriented network architecture) into the Internet oriented network architecture) into the Internet through overlays.through overlays.

Whether a single winner emerges & gets Whether a single winner emerges & gets subsumed into Internet or subsumed into Internet or services continue to define their own routing, services continue to define their own routing, remains a subject of speculation..remains a subject of speculation..

ReferencesReferences

• PlanetLab: An overlay testbed for broad-coverage PlanetLab: An overlay testbed for broad-coverage services by B. Chun et. al., Jan 2003services by B. Chun et. al., Jan 2003

Backup slidesBackup slides

Overlay construction problemsOverlay construction problems

• Dynamic changes in group membership

– Members may join and leave dynamically

– Members may die

• Dynamic changes in network conditions and topology

– Delay between members may vary over time due to congestion, routing changes

• Knowledge of network conditions is member specific

– Each member must determine network conditions for itself

Testbed’s mode of operation as Testbed’s mode of operation as deployment platformdeployment platform

Documents

By L. Peterson, Princeton T.Anderson, UW D. Culler, T. Roscoe, Intel, Berkeley HotNets-I (Infrastructure panel), 2002 Presenter Shobana Padmanabhan Discussion