Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Open Innovation Platforms: Network
Testbeds, Cloud Testbeds and StarLight Software Define Network Exchange(StarLight SDX)
Jim ChenAssociate Director
International Center for Advanced Internet Research(iCAIR)Northwestern University
APAN41-Cloud WGManila, PhilippinesJanuary 26, 2016
• Creation and Early Implementation of Advanced Networking Technologies - The Next Generation Internet All Optical Networks, Terascale Networks, Networks for Petascale Science
• Advanced Applications, Middleware, Large-Scale Infrastructure, NG Optical Networks and Testbeds, Public Policy Studies and Forums Related to NG Networks
• Three Major Areas of Activity: a) Basic Research b) Design and Implementation of Prototypes c) Operations of Specialized Communication Facilities (e.g., StarLight)
Accelerating Leading Edge Innovation
and Enhanced Global Communications
through Advanced Internet Technologies,
in Partnership with the Global Community
Introduction to iCAIR:
Open Innovation Platforms
• GENI: Global Environment for Networking Innovations, “a virtual laboratory for exploring future internet technologies and protocols at scale”.
• NSF13602 NSFCloud: “enable the academic research community to develop and experiment with novel cloud and cloud-like architectures that can support a diversity of innovative applications”.
• NSF14554 IRNC RXP: “platforms of network innovation supporting the research and education needs of global scientific collaborations”.
Multiple HPC Cloud Computing Testbeds
Specifically Designed for Science Research
At Scale Experimentation
Integerated With High Performance Networks
=> Open Commons Consortium
Will Be Contiguous To
the StarLight SDX
Global Research Platform
• Currently – A Concept
• A Specialized Globally Distributed Platform For Science Discovery and Innovation
• Based On State-Of-the-Art-Clouds
• Interconnected With Computational Grids, Supercomputing Centers, Specialized Instruments, et al
• Also, Based On World-Wide 100 Gbps Networks
• Leveraging Advanced Architectural Concepts, e.g., SDN/SDX/SDI – Science DMZs
• Ref: Demonstrations @ SC15, Austin Texas November 2015
www. chameleoncloud.org
JANUARY 26 , 2016 11
CHAMELEON: A LARGE-SCALE, RECONFIGURABLE EXPERIMENTAL ENVIRONMENT FOR CLOUD RESEARCHPrincipal Investigator: Kate Keahey
Co-PIs: J. Mambretti, D.K. Panda, P. Rad, D. Stanzione
Presented by Jim Chen, associate Director,
International Center for Advanced Internet Research, Northwestern University
APAN 41
Manila Philippines
JANUARY 25-29, 2016
www. chameleoncloud.org
CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT
Large-scale: “Big Data, Big Compute, Big Instrument research” ~650 nodes (~14,500 cores), 5 PB disk over two sites, 2 sites
connected with 100G network
Reconfigurable: “As close as possible to having it in your lab” From bare metal reconfiguration to clouds
Support for repeatable and reproducible experiments
Connected: “One stop shopping for experimental needs” Workload and Trace Archive
Partnerships with production clouds: CERN, OSDC, Rackspace, Google, and others
Partnerships with users
Complementary: “Can’t do everything ourselves” Complementing GENI, Grid’5000, and other experimental testbeds
www. chameleoncloud.org
CHAMELEON HARDWARE
SCUs connect to core and fully connected to each other
Heterogeneous Cloud Units
Alternate Processors and Networks
SwitchStandard
Cloud Unit42 compute
4 storage
x10
Chicago
To UTSA, GENI, Future Partners
AustinChameleon Core Network
100Gbps uplink public network(each site)
Core Services3.6 PB Central File Systems, Front End and Data Movers
Core ServicesFront End and Data
Mover Nodes 504 x86 Compute Servers48 Dist. Storage Servers102 Heterogeneous Servers16 Mgt and Storage Nodes
SwitchStandard
Cloud Unit42 compute
4 storage
x2
www. chameleoncloud.org
CAPABILITIES AND SUPPORTED RESEARCH
Virtualization technology (e.g., SR-IOV, accelerators), systems, networking, infrastructure-level resource management, etc.
Repeatable experiments in new models, algorithms, platforms, auto-scaling, high-availability, cloud federation, etc.
Development of new models, algorithms, platforms, auto-scaling HA, etc., innovative application and educational uses
Isolated partition, bare metal reconfiguration: OpenStack and Grid’5000
Isolated partition, Chameleon Appliances
Persistent, reliable, shared clouds
www. chameleoncloud.org
SOFTWARE: CORE CAPABILITIES
Chameleon Appliance Catalog
A library of generic, special-purpose, and educational environments
Persistent Clouds
(OpenStack)
Discovery, Provisioning, Configuration, and Monitoring
Testbed representation and discovery (Grid’5000)
Nova/Blazar, Ironic, Neutron, Ceilometer
(OpenStack, Rackspace OnMetal)
Persistent Cloud User Clouds
www. chameleoncloud.org
EXPERIMENTAL WORKFLOW
discover
resources
provision
resources
configure
and interactmonitor
analyze, discuss,
and share
design the
experiment
www. chameleoncloud.org
SELECTING AND VERIFYING RESOURCES
Complete, fine-grained and up-to-date representation Machine parsable, enables match makingVersioned
“What was the drive on the nodes I used 6 months ago?”
Dynamically VerifiableDoes reality correspond to description? (e.g., failures)
Grid’5000 Registry Automated resource description, automated export to RM
G5K-checksRun at boot, acquire information, compare with resource
catalog description
www. chameleoncloud.org
PROVISIONING RESOURCES
Resource leases
Allocating a range of resourcesDifferent node types, switches, etc.
Multiple environments in one lease
Advance reservations (AR) Sharing resources across time
Eventually: match making, Gantt chart displays
OpenStack Nova/BlazarExtensions to support working with more resources, match
making, and displays
www. chameleoncloud.org
CONFIGURE AND INTERACT
Map multiple appliances to a leaseAllow deep reconfiguration (incl. BIOS)SnapshottingEfficient appliance deploymentHandle complex appliances
Virtual clusters, cloud installations, etc.
Interact: reboot, power on/off, access to consoleShape experimental conditions
OpenStack Ironic, Glance, and meta-data servers
www. chameleoncloud.org
MONITORING
Enables users to understand what happens during the experiment
Types of monitoringUser resource monitoringInfrastructure monitoring (e.g., PDUs)Custom user metrics
High-resolution metricsEasily export data for specific experiments
OpenStack Ceilometer
www. chameleoncloud.org
OUTREACH AND ENGAGEMENT
Early User ProgramCommitted users, driving and testing new capabilities,
enhanced level of support
Chameleon WorkshopAnnual workshop to inform, share experimental
techniques solutions and platforms, discuss upcoming requirements, and showcase research
Advisory BodiesResearch Steering Committee: advise on capabilities
needed to investigate upcoming research challengesIndustry Advisory Board: provide synergy between industry
and academia
www. chameleoncloud.org
PARTNERSHIP WITH GENI COMMUNITY
Chameleon Will Enable the GENI Virtual Laboratory For Networking and Distributed Systems Research and Education To Extended Significantly With Many New Types of Resources.
This Blending of Resources Will Enable Investigations Of New Types Of Innovative Highly Distributed Environments at Scale.
The Architectural Options For Blending Chemeleon and GENI Are Being Discussed
www. chameleoncloud.org
EXPERIMENTER RESOURCES
Unique Deeply Programmable Blending Of Resources. Extremely Close Integration (Ideally, Seamless) of
Programmable Networks With Programmable CloudsA Richly Resourced Platform For Experimentation
With Exceptional Novel Architecture.Enables the Creation of New, Highly Innovative
Distributed Environments, Including Specialized/Customized vs Generic Environments
Highly Granulated Levels of Experimental Research Control, Measurement, Analytics, Visualization, and Reproducibility.
www. chameleoncloud.org
PROJECT SCHEDULE
Q1 2015: FutureGrid@ChameleonChameleon Technology Preview OpenStack FutureGrid-style cloud43 projects, 81 users, 29 institutions
Summer 2015: New hardware: large-scale homogenous partitions available to Early Users
Fall 2015: Large-scale homogenous partitions and bare metal reconfiguration generally available
2015/2016: Refinements to experiment management capabilities, higher level capabilities
Fall 2016: Heterogeneous hardware available
www. chameleoncloud.org
TEAM
Kate Keahey
Chameleon PI
Science Director,
Software Development
Joe Mambretti
Programmable networks
DK Panda
High-performance
networks
Dan Stanzione
Facilities Director
Paul Rad
Industry Liason
www. chameleoncloud.org
THE TESTBED IS THERE – “JUST” ADD RESEARCH!
Large-scale, responsive experimental testbed
Targeting critical research problems at scale
Reconfigurable environment
Support use cases from bare metal to production clouds
One-stop shopping for experimental needs
Trace and Workload Archive
Engage the community The most important element of any experimental testbed is users and
the research they work on
Come and visit us at www.chameleoncloud.org!