26
Open Innovation Platforms: Network Testbeds, Cloud Testbeds and StarLight Software Define Network Exchange(StarLight SDX) Jim Chen Associate Director International Center for Advanced Internet Research(iCAIR) Northwestern University APAN41-Cloud WG Manila, Philippines January 26, 2016

Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

Open Innovation Platforms: Network

Testbeds, Cloud Testbeds and StarLight Software Define Network Exchange(StarLight SDX)

Jim ChenAssociate Director

International Center for Advanced Internet Research(iCAIR)Northwestern University

APAN41-Cloud WGManila, PhilippinesJanuary 26, 2016

Page 2: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

• Creation and Early Implementation of Advanced Networking Technologies - The Next Generation Internet All Optical Networks, Terascale Networks, Networks for Petascale Science

• Advanced Applications, Middleware, Large-Scale Infrastructure, NG Optical Networks and Testbeds, Public Policy Studies and Forums Related to NG Networks

• Three Major Areas of Activity: a) Basic Research b) Design and Implementation of Prototypes c) Operations of Specialized Communication Facilities (e.g., StarLight)

Accelerating Leading Edge Innovation

and Enhanced Global Communications

through Advanced Internet Technologies,

in Partnership with the Global Community

Introduction to iCAIR:

Page 3: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

Open Innovation Platforms

• GENI: Global Environment for Networking Innovations, “a virtual laboratory for exploring future internet technologies and protocols at scale”.

• NSF13602 NSFCloud: “enable the academic research community to develop and experiment with novel cloud and cloud-like architectures that can support a diversity of innovative applications”.

• NSF14554 IRNC RXP: “platforms of network innovation supporting the research and education needs of global scientific collaborations”.

Page 4: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument
Page 5: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument
Page 6: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

Multiple HPC Cloud Computing Testbeds

Specifically Designed for Science Research

At Scale Experimentation

Integerated With High Performance Networks

=> Open Commons Consortium

Page 7: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument
Page 8: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

Will Be Contiguous To

the StarLight SDX

Page 9: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

Global Research Platform

• Currently – A Concept

• A Specialized Globally Distributed Platform For Science Discovery and Innovation

• Based On State-Of-the-Art-Clouds

• Interconnected With Computational Grids, Supercomputing Centers, Specialized Instruments, et al

• Also, Based On World-Wide 100 Gbps Networks

• Leveraging Advanced Architectural Concepts, e.g., SDN/SDX/SDI – Science DMZs

• Ref: Demonstrations @ SC15, Austin Texas November 2015

Page 10: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument
Page 11: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

JANUARY 26 , 2016 11

CHAMELEON: A LARGE-SCALE, RECONFIGURABLE EXPERIMENTAL ENVIRONMENT FOR CLOUD RESEARCHPrincipal Investigator: Kate Keahey

Co-PIs: J. Mambretti, D.K. Panda, P. Rad, D. Stanzione

Presented by Jim Chen, associate Director,

International Center for Advanced Internet Research, Northwestern University

APAN 41

Manila Philippines

JANUARY 25-29, 2016

Page 12: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT

Large-scale: “Big Data, Big Compute, Big Instrument research” ~650 nodes (~14,500 cores), 5 PB disk over two sites, 2 sites

connected with 100G network

Reconfigurable: “As close as possible to having it in your lab” From bare metal reconfiguration to clouds

Support for repeatable and reproducible experiments

Connected: “One stop shopping for experimental needs” Workload and Trace Archive

Partnerships with production clouds: CERN, OSDC, Rackspace, Google, and others

Partnerships with users

Complementary: “Can’t do everything ourselves” Complementing GENI, Grid’5000, and other experimental testbeds

Page 13: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

CHAMELEON HARDWARE

SCUs connect to core and fully connected to each other

Heterogeneous Cloud Units

Alternate Processors and Networks

SwitchStandard

Cloud Unit42 compute

4 storage

x10

Chicago

To UTSA, GENI, Future Partners

AustinChameleon Core Network

100Gbps uplink public network(each site)

Core Services3.6 PB Central File Systems, Front End and Data Movers

Core ServicesFront End and Data

Mover Nodes 504 x86 Compute Servers48 Dist. Storage Servers102 Heterogeneous Servers16 Mgt and Storage Nodes

SwitchStandard

Cloud Unit42 compute

4 storage

x2

Page 14: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

CAPABILITIES AND SUPPORTED RESEARCH

Virtualization technology (e.g., SR-IOV, accelerators), systems, networking, infrastructure-level resource management, etc.

Repeatable experiments in new models, algorithms, platforms, auto-scaling, high-availability, cloud federation, etc.

Development of new models, algorithms, platforms, auto-scaling HA, etc., innovative application and educational uses

Isolated partition, bare metal reconfiguration: OpenStack and Grid’5000

Isolated partition, Chameleon Appliances

Persistent, reliable, shared clouds

Page 15: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

SOFTWARE: CORE CAPABILITIES

Chameleon Appliance Catalog

A library of generic, special-purpose, and educational environments

Persistent Clouds

(OpenStack)

Discovery, Provisioning, Configuration, and Monitoring

Testbed representation and discovery (Grid’5000)

Nova/Blazar, Ironic, Neutron, Ceilometer

(OpenStack, Rackspace OnMetal)

Persistent Cloud User Clouds

Page 16: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

EXPERIMENTAL WORKFLOW

discover

resources

provision

resources

configure

and interactmonitor

analyze, discuss,

and share

design the

experiment

Page 17: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

SELECTING AND VERIFYING RESOURCES

Complete, fine-grained and up-to-date representation Machine parsable, enables match makingVersioned

“What was the drive on the nodes I used 6 months ago?”

Dynamically VerifiableDoes reality correspond to description? (e.g., failures)

Grid’5000 Registry Automated resource description, automated export to RM

G5K-checksRun at boot, acquire information, compare with resource

catalog description

Page 18: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

PROVISIONING RESOURCES

Resource leases

Allocating a range of resourcesDifferent node types, switches, etc.

Multiple environments in one lease

Advance reservations (AR) Sharing resources across time

Eventually: match making, Gantt chart displays

OpenStack Nova/BlazarExtensions to support working with more resources, match

making, and displays

Page 19: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

CONFIGURE AND INTERACT

Map multiple appliances to a leaseAllow deep reconfiguration (incl. BIOS)SnapshottingEfficient appliance deploymentHandle complex appliances

Virtual clusters, cloud installations, etc.

Interact: reboot, power on/off, access to consoleShape experimental conditions

OpenStack Ironic, Glance, and meta-data servers

Page 20: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

MONITORING

Enables users to understand what happens during the experiment

Types of monitoringUser resource monitoringInfrastructure monitoring (e.g., PDUs)Custom user metrics

High-resolution metricsEasily export data for specific experiments

OpenStack Ceilometer

Page 21: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

OUTREACH AND ENGAGEMENT

Early User ProgramCommitted users, driving and testing new capabilities,

enhanced level of support

Chameleon WorkshopAnnual workshop to inform, share experimental

techniques solutions and platforms, discuss upcoming requirements, and showcase research

Advisory BodiesResearch Steering Committee: advise on capabilities

needed to investigate upcoming research challengesIndustry Advisory Board: provide synergy between industry

and academia

Page 22: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

PARTNERSHIP WITH GENI COMMUNITY

Chameleon Will Enable the GENI Virtual Laboratory For Networking and Distributed Systems Research and Education To Extended Significantly With Many New Types of Resources.

This Blending of Resources Will Enable Investigations Of New Types Of Innovative Highly Distributed Environments at Scale.

The Architectural Options For Blending Chemeleon and GENI Are Being Discussed

Page 23: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

EXPERIMENTER RESOURCES

Unique Deeply Programmable Blending Of Resources. Extremely Close Integration (Ideally, Seamless) of

Programmable Networks With Programmable CloudsA Richly Resourced Platform For Experimentation

With Exceptional Novel Architecture.Enables the Creation of New, Highly Innovative

Distributed Environments, Including Specialized/Customized vs Generic Environments

Highly Granulated Levels of Experimental Research Control, Measurement, Analytics, Visualization, and Reproducibility.

Page 24: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

PROJECT SCHEDULE

Q1 2015: FutureGrid@ChameleonChameleon Technology Preview OpenStack FutureGrid-style cloud43 projects, 81 users, 29 institutions

Summer 2015: New hardware: large-scale homogenous partitions available to Early Users

Fall 2015: Large-scale homogenous partitions and bare metal reconfiguration generally available

2015/2016: Refinements to experiment management capabilities, higher level capabilities

Fall 2016: Heterogeneous hardware available

Page 25: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

TEAM

Kate Keahey

Chameleon PI

Science Director,

Software Development

Joe Mambretti

Programmable networks

DK Panda

High-performance

networks

Dan Stanzione

Facilities Director

Paul Rad

Industry Liason

Page 26: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument

www. chameleoncloud.org

THE TESTBED IS THERE – “JUST” ADD RESEARCH!

Large-scale, responsive experimental testbed

Targeting critical research problems at scale

Reconfigurable environment

Support use cases from bare metal to production clouds

One-stop shopping for experimental needs

Trace and Workload Archive

Engage the community The most important element of any experimental testbed is users and

the research they work on

Come and visit us at www.chameleoncloud.org!