Sponsored by the National Science Foundation GENI Instrumentation and Measurement WG: 2 nd I&M Workshop and Priority Architecture Topics GENI Engineering

Sponsored by the National Science Foundation

GENI Instrumentation and Measurement WG:

2nd I&M Workshop and Priority Architecture Topics

GENI Engineering Conference 8UC San Diego

GPO System Engineer: Harry MussmanJuly 22, 2010

www.geni.net

Sponsored by the National Science Foundation 2July 22, 2010

2nd I&M Workshop

• Objective:– Gather contributors from key I&M prototyping projects that

have already implemented pieces of I&M functionality in a manner consistent with GENI goals.

– Define priority pieces of the I&M architecture by consensus– Assemble teams to complete the discussion and

documentation– Draft a roadmap for implementations during Spirals 2 and 3.– See http://groups.geni.net/geni/wiki/2ndInstMeasWork#References

• Initial set of priority topics:– GENI I&M use cases – GENI I&M services – GENI measurement plane – Interfaces, protocols and schema for measurement data in

GENI


(continued)

• Approach:– Each priority topic was discussed in a structured manner– The organizers outlined an approach or solution, and representatives from

the key projects discussed it– The group identified where they had a consensus, and where there were

issues that needed further discussion– One new topic was identified: how to discover, specify, authorize and

assign all of the various types of resources required for I&M capabilities– The interfaces topic was split into four topics, divided upon function – Finally, teams were assembled to discuss each priority topic, come to a

consensus, and write a revised section for the architecture document

• Initial set of priority topics:– GENI I&M use cases – GENI I&M services – GENI measurement plane – Interfaces, protocols and schema for measurement data in GENI


(continued)

• Organizing Committee:– Paul Barford - University of Wisconsin – Madison– Bruce Maggs – Duke University and Akamai – Harry Mussman – BBN/GPO – Vic Thomas - BBN/GPO– Evan Zhang – BBN/GPO

• Key projects who sent representatives:– OML (ORBIT Measurement Library) OMF (ORBIT Management Framework) – Instrumentation Tools – perfSONAR – Scalable Sensing Service (S3) – OnTimeMeasure for network measurements – GENI Meta-Operations Center and NetKArma– Virtual Machine Introspection (VMI) – Experiment Management Service – Digital Object Registry


(continued)

• Final set of priority topics:– Topic 1: GENI I&M Use Cases– Topic 2: GENI I&M Services– Topic 3: GENI I&M Resources– Topic 4: GENI I&M Measurement Plane and Interfaces– Topic 5: GENI I&M Interfaces and Messages/Flows/APIs : Manage

Services– Topic 6: GENI I&M Interfaces and Messages/Flows/APIs : Data

Flows and Data File Transfers– Topic 7: GENI I&M Interfaces and Messages/Flows/APIs : Service

Registration and Discovery– Topic 8: GENI I&M Interfaces and Messages/Flows/APIs : GUIs– Topic 9: GENI Measurement Data Schema


Topic 1: GENI I&M Use Cases

• Team members: – Paul Barford - University of Wisconsin – Madison – Jim Griffioen - Univ Kentucky – Prasad Calyam* - Ohio Supercomputing Ctr – Camilo Viecco - Indiana Univ – Brian Hay – Univ Alaska *agreed to organize first writing and discussion

• Identify all user groups, and provide basic use cases for each user group: – Experiment researchers – Experiment (opt-in) users– Central (i.e., GMOC) operators – Cluster and aggregate providers and operators – Archive service providers and operators – Researchers that use archived measurement data


Topic 2: GENI I&M Services

• Team members:– Harry Mussman* and Evan Zhang – BBN/GPO – Giridhar Manepalli – CNRI– Chris Small and Beth Plale - Indiana Univ *agreed to organize first writing and discussion

• Summarize current view of services– Measurement Orchestration (MO) Service – Measurement Point (MP) Service – Measurement Information (MI) Service – Measurement Collection (MC) Service – Measurement Analysis and Presentation (MAP) Service– Measurement Data Archive (MDA) Service

• Need basic definition of a Measurement Data Archive (MDA) service


(continued)

• Have identified three different types of services:– Type 1: Dedicated service platform for customized Information.

(Completely dedicated to an experiment)– Type 2: Common service platform with dedicated slivers for

customized Information. (Common portion, plus parts associated with different experiments)

– Type 3: Common service for common or customized information. (Common service, with data provided to multiple experiments)


GENI I&M Architecture:Services and Interfaces


Topic 3: GENI I&M Resources

• Team members:– Vic Thomas - BBN/GPO – Jim Griffioen* - Univ Kentucky – Martin Swany - Univ Delaware – Camilo Viecco - Indiana Univ – Brian Hay – Univ Alaska – Giridhar Manepalli - CNRI *agreed to organize first writing and discussion

• How are each of these resources discovered, specified, authorized and assigned: – Hosts, VMs, etc. – Network connectivity – Software, e.g., I&M software that can be included in an

experiment – I&M services – I&M data flows and file transfers – I&M data files stored in archives


(continued)

• Options:– Always by mechanisms provided by the CF? – With CF plus additional mechanisms? – Consider example of LS in perfSONAR – Consider example of data file stored in archive, owned by an

experimenter – Need to understand interop with CF for each option– Does CF setup secondary authorization mechanisms in some cases? If so,

how?

• For each item, consider– Unique name?– Unique and persistent identifier?– Ownership– What sort of policies the owner may want to apply– How to create– How to name– How to register and discover– How to authorize and assign


(continued)

• Chart:–


Topic 4: GENI I&M Measurement Plane and Interfaces

• Team members: – Harry Mussman* – BBN/GPO– Ezra Kissel – Univ Delaware – Chris Small - Indiana Univ *agreed to organize first writing and discussion

• Assume: Use IP backbone network (with Internet access) for most measurement traffic (like control traffic)

• Consider: Use Layer 2 (VLAN) connections only when necessary

• Issues with measurement traffic:– Which protocols are active? minimize number to simplify?– Need access to resources in aggregates, even when resources are in

private address space, via proxies (i.e., gateways)– Need to provide authentication and authorization, without and with

proxies– Need to be able to reserve bandwidth (provide QoS) to protect

measurement traffic from other traffic, and vice-versa– Which of these requires the use of Layer 2 (VLAN) connections?–


GENI Measurement Traffic Flows


GENI I&M Interfaces and Messages/Flows/APIs

• 1) Discover Resources and Assign Slivers– EC srvc uses CF to discover resources, and then assign slivers to slice/researcher

for I&M srvc’s

• 2) Configure and Program Slivers– EC srvc uses CF and/or ssh to load std or customized software images for I&M

srvc’s– Note: 1) and 2) are not specific to I&M services

• 3) Manage Services– EC srvc and MO srvc use CF and/or https to check status of I&M srvcs, receive

event notifications, and execute functions such as start, stop, reset, reboot, and checkpoint

• 4) Measurement Data Flows and Data File Transfers– Measurement data flows and data file transfers between I&M srvcs. Options: Pull,

Push, Pub/Sub.

• 5) Register I&M Service– Operator configures I&M srvc to register with Lookup Srvc, advertising name,

location, and available metadata

• 6) Discover I&M Service and Establish Meas Data Flow– ECS or I&M srvc discovers I&M srvc advertisement, and establishes data flow

• 7) Conduct and Observe Experiment– Researcher uses browser to interact with and observe services via web portals

(GUIs)


GENI I&M Architecture:Services and Interfaces


Topic 5: GENI I&M Interfaces and Messages/Flows/APIs: Manage Services

• Team members: – Vic Thomas - BBN/GPO – Ivan Seskar – Rutgers WINLAB – Max Ott – NICTA – Sonia Fahmy* – Purdue – Giridhar Manepalli – CNRI *agreed to organize first discussion and writing

• Define an approach based on OMF/OML and S3

• Consider:– HTTP(S)– REST vs SOAP– Authorization by credentials or ? If credentials, how to revoke?– Pass XML fragments– Define basic API


Topic 6: GENI I&M Interfaces and Messages/Flows/APIs:

Data Flows and Data File Transfers • Team members:

– Harry Mussman* – BBN/GPO– Ivan Seskar – Rutgers WINLAB– Max Ott – NICTA– Ezra Kissel – Univ Delaware– Prasad Calyam - Ohio Supercomputing Ctr– Michael Zink - UMass Amherst*agreed to organize first writing and discussion

• Consider data flows and data file transfers between all services– Fundamental to I&M, and not yet defined in GENI– Linked to measurement data schema, which is primary


(continued)

• Consensus to allow a wide range of options: – Both data flows and data files transfers– Types: Pull, Push, Pub/Sub– Protocols: SNMP, SCP, FTP, gridFTP, HTTP, XMPP, TCP,

SCTP– Consider: Naming, Discovery, Connectivity ,

Authentication and authorization mechanisms

• Goal is to define an initial set:– Minimum set required for GENI– Mapped to current projects– Permit later additions


Topic 7: GENI I&M Interfaces and Messages/Flows/APIs: Registration and

Discovery of Serviceswith Available Measurement Data

• Team members:– Jason Zurawski* – Internet2 (yes)– Prasad Calyam - Ohio Supercomputing Ctr (yes)*agreed to organize first writing and discussion

• Consider approach used in perfSONAR

• Summarize for:– Services with available data flows– Also sources of file transfers?– Also GUIs?


Topic 8: GENI I&M Interfaces and Messages/Flows/APIs:

GUIs • Team members:

– Jeremy Reed and Zongming Fei - Univ Kentucky (yes) – Guilherme Fernandes* – Univ Delaware (yes)– Puneet Sharma - HP Labs (yes)– *agreed to organize team

• Consider all types of GUIs:– Control experiments– Display I&M results– Report status– View archive service

• Define overall goals for GENI GUIs

• Consider portal, for access to multiple GUIs

• Consider need for authentication and authorization


Topic 9 GENI Measurement Data Schema

• Team members:– Bruce Maggs – Duke University and Akamai– Max Ott – NICTA– Ivan Seskar – Rutgers WINLAB– Martin Swany* - Univ Delaware– Camilo Viecco - Indiana Univ– Michael Zink - UMass Amherst– Jim French – CNRI*agreed to organize first discussion and writing

• Consider:– Measurement data schema– Metadata schema– Metadata contents


(continued)

• Goal is to define an initial set:– Minimum set required for GENI– Mapped to current projects– Permit later additions

• Consider measurement data schema and/or metadata schema from:– perfSONAR– GMOC-provided– Current OML approach– Proposed OML approach using IPFIX– NetCDF (as used by DI Cloud)

• Provide overall template for GENI metadata:– Which items in GENI metadata template are required?– Invariant?


GENI I&M Architecture Priority Topics:Current Status

Topic Group discussion

First writing

WG review

Topic 1: GENI I&M Use Cases

Topic 2: GENI I&M Services

Topic 3: GENI I&M Resources

Topic 4: GENI I&M Measurement Plane and Interfaces

Topic 5: GENI I&M Interfaces and Messages/Flows/APIs : Manage Services

Topic 6: GENI I&M Interfaces and Messages/Flows/APIs : Data Flows and Data File Transfers

Topic 7: GENI I&M Interfaces and Messages/Flows/APIs : Service Registration and Discovery

Topic 8: GENI I&M Interfaces and Messages/Flows/APIs : GUIs

Topic 9: GENI Measurement Data Schema


GENI I&M Architecture Document:Current Status

• Status:– By GEC7 : v0.1 DRAFT completed, by GPO; see

http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurementsArchitecture

– By GEC8: v0.5 DRAFT completed, by GPO, with contributions from WG; see

• Plan:– By GEC9: v1.0 draft, reviewed by WG


GENI I&M Architecture Priority Topics and Document: Next Steps

• Complete group discussion and first writing for each priority topic:– When? – Add members to each group?

• Review each priority topic with WG, and rewrite:– How?– When?

• Assemble and review revised Architecture Document as a whole– How?– Before GEC9

• Draft a roadmap for implementations during Spiral 3.

Documents

Sponsored by the National Science Foundation GENI Instrumentation and Measurement WG: 2 nd I&M Workshop and Priority Architecture Topics GENI Engineering