APAN Cloud WG (2015/3/2)

Building High-Performance Inter-Cloud Infrastructure in Japan

Masaharu Munetomo

Professor & Vice Director,Information Initiative Center,Hokkaido University, Sapporo, JAPAN.

[email protected]

HOKKA IDO UN IVERS ITY GU IDEBOOK 2014-2015

46Campus Maps

Campus Maps

1

2

3

4

5

6

7

8

9

10

11

P

P

P

P

P

To Hakodate Station

N

20

19

18

17

16

15

1413

12

11

10

9

8

7

8

6

5

4

3

2

1

... we are cosmopolitan, and accessible...

Picturesque Hakodate is home to Hokkaido University’s Faculty of Fisheries Science and is located on the south-west of the island.

With a population of approximately 280,000 people, the coastal city is at the base of Mount Hakodate, which boasts amazing natural beauty. The

view from the summit is renowned for having one of the most beautiful views in Japan, particulary at night. Since it opened in 1935, the Hakodate

Campus has had a proud history of being at the forefront of fisheries research and is one of the largest campuses of its kind in Japan.

Main Gate Main Building Annex BuildingMarine Bioresources Research Building Marine Frontier Research Building Marine Science Creative Research Building Lecture-room BuildingStudent Laboratories Controlled Environment RoomsTowing Tank RoomAuditorium

Hakodate Campus Map

Library Aquatic Biological Specimen House (Nakabe Hall)Fisheries Museum (main building) Fisheries Museum (annex) Gymnasium Student Center Student Activities Building Swimming Pool Athletic Field

12

13

14

15

16

17

18

19

20

Sapporo Campus

Hokkaido

Hakodate Campus

1

Masaharu Munetomo

• Professor & Vice director, Information Initiative Center, Hokkaido university, Sapporo, JAPAN.

• Chief examiner, Cloud computing research group of national supercomputing centers in Japan.

• Chief examiner, SIG Cloud, Academic eXchange for Information Environment and Strategy (AXIES) in Japan.

• Chief examiner, SIG Mathematical Problem-Solving, Information Processing Society of Japan (IPSJ)

• General advisor, Cloud Utilization Promotion Agency (CUPA) & Managed Service Providers associations in Japan (MSPJ)

• Founding member and of steering committee, Open Compute Project in Japan (OCPJ)

2

Information Initiative Center, Hokkaido University

• Founded in 1962 as a national supercomputing center.

• A member of High Performance Computing Infrastructure (HPCI) and Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructure (JHPCN) in Japan.

• University R&D center for Supercomputing, Cloud computing, Networking, IT systems for education

• Supercomputer (172TFlops) & Academic Cloud System (43TFlops)

@�(�*��[�( %!*��.9��

�7W1)�'*�%1�L�XN��"*��2G [@�(�*

��89=)+Y=)H4:<��"*��$*(�#�([,P=)

ZCQ=��"*��&(��#�(��B?��

;D�7S>3�RN EI�V��[57J��MO?A�/

-�Q0K�UF�@�(�*�6�T��

��

��

3

HPCI (High Performance Computing Infrastructure)

• Collaboration of national supercomputing centers in Japan.

• RIKEN AICS (K computer) & Supercomputing Centers (University, Research Institutes) connected via academic high-speed network (SINET4)

• Federations of users & systems management (GSI-SSH, Gfarm supported)

http://hpci-office.jp/4

Hokkaido University Academic Cloud System

• Largest Academic Cloud System in Japan started services from Nov. 2011: 43TFlops (5,000 cores), and more than 2,000 VMs can be deployed.

• Employing CloudStack to provide cloud management portal.

• High-performance cloud system: each physical node has 40-cores, 128GB memory. Network: 10GbE x 2, Shared Storage: 260TB (SAN) + 500TB (NAS) + 2PB (WebDAV, S3, Gfarm)

Hitach BladeSymphony BS2000 Xeon E7 8870 2.4GHz (10-core) x 4

128GB memory / 10GbE x 2

Hitachi NAS Storage AMS2300: 260TB AMS2500: 500TB 5

Use case: “Big Data” processing systems

• We provide “Big Data” service VM package consisting of Hadoop, Hive, Mahout, and R.

• Automated deployment of VM clusters, customizing scheduling policies in CloudStack to balance I/O overheads for cluster packages (Hadoop / MPI / Torque).

Storage #3�Virtual(Disk�



Zone!POD!

Shared Storage #1�Resource Pool #1�

HyperVisor #2�

HyperVisor #1�Virtual(Disk�VM(

Balancing!overheads!of!disk!I/O!with!round8robin!assignment!of!Virtual!disks.!�

Storage #1�

VM(

VM(

VM(

VM(

Virtual(Disk�Hadoop Cluster�Shared Storage #2�

Resource Pool #2�

HyperVisor #4�

HyperVisor #3�

Virtual(Disk�

VM(

Shared Storage #3�Resouce Pool #3�

HyperVisor #6�

HyperVisor #5�

Virtual((Disk�

VM(

Shared Storage #4�

Resouce Pool #4�

HyperVisor #8�

HyperVisor #7�

Virtual(Disk�

VM(

6

Use case: simulation environment to replace in-house computing servers or clusters

• Replacement of in-house clusters of laboratories employing L (10-core) or XL (40-core) project servers.

• Filling in the gap between PCs and super-computers.

7

Use case: development of in-silico screening system for drug design

• Center for Research and Education on Drug Discovery builds a Structure Based Drug Design (SBDD) system for in-silico screening with the academic cloud system

• A virtual private cloud system using XL servers (40-core): modeFRONTIER® and AutoDock are installed as docking applications.

AutoDock[1]

AutoDock[2]

AutoDock

AutoDock

AutoDock

AutoDockContinuous execution of analysis

servers

8

Use-case: Fishing ground prediction system

• Researchers in department of fishery build a fishing ground prediction system on Hokkaido university academic cloud system

• The system provides information on promising sea area for fishing boats to catch squids, employing satellite images and data assimilation results.

Portal System

Satellite image processing Data assimilation

Fishing ground prediction

INMARSAT

Satellite Earth station

Satellite Communications

Squid Fishing Boats Fishing ground prediction system portal

9

Use-case: Employing PaaS for scalable interactive evolutionary computation

• Building a scalable interactive evolutionary computation framework to evolve solutions according to the preferences of millions of users.

CloudStack

VM

Ubuntu

instance

VM

Ubuntu

Redis

VM

Ubuntu

Redis

VM

Ubuntu

Redis

Database

・・・

VM

Ubuntu

instance

VM

Ubuntu

instance

・・・

Applycation resource

iGA iGA iGA

Load Balancer

CloudFoundry

Sever

・・・

Interactive Evolutionary Computation using PaaS

Users select solutions according to their preferences

Present cadndates of solutions from the system

10

Japanese academic inter-cloud infrastructure

• Development of the inter-cloud system over Japanese universities to collaborate private clouds from Kitami (Northernmost) to Ryukyu (Southernmost) universities through Japanese academic high-speed network (SINET4).

Hokkaido University

Kitami Institute of Technology

University of Ryukyus (Okinawa)

National Institute of Informatics (NII)

@�(�*��[�( %!*��.9��

�7W1)�'*�%1�L�XN��"*��2G [@�(�*

��89=)+Y=)H4:<��"*��$*(�#�([,P=)

ZCQ=��"*��&(��#�(��B?��

;D�7S>3�RN EI�V��[57J��MO?A�/

-�Q0K�UF�@�(�*�6�T��

��

��

11

Related projects

• Remote collaborations of distributed cloud systems (JHPCN)

• Federations technologies development toward academic inter-cloud (Collaborative research project, National Institute of Informatics)

• Large-scale Distributed Design Exploration Framework (JHPCN)

• Development of distributed database infrastructure across Japan

• Inter-cloud resource optimization with multi-objective evolutionary algorithms

• Designing the next-generation Hokkaido university high-performance inter-cloud system

12

Remote collaborations of distributed cloud systems

• Prototyping an inter-cloud manager and authentication infrastructure for federation of academic cloud systems managed by different cloud middleware (CloudStack, OpenStack, etc.)

• Designing a VPC (Virtual Private Cloud) management framework in the distributed inter-cloud systems.

Cloud A IaaS �

Cloud B IaaS �

Cloud C IaaS �

User�

VPC 1�

Internet�

VM�VM�

VM�

VPC 2�

220km�

13

Large-scale Distributed Design Exploration Framework (LDDEF)• To establish a framework to support “parameter surveys” by

supercomputing simulations collaborating design engineers sharing information on promising solutions with distributed DBs

• “Multi-objective designexploration” explores Pareto-fronts stored indistributed DBs

• Optional info. Isstored in objectstorages forvisualization andanalysis Solutions DB

(distributed)

Automated replication for DR and

load balancing

Visualization

Simulation (Supercomputer)

Optimization & DB management (Cloud system)

Distributed Database

Product

14

Grid Unified Framework for Optimization (Grid-UFO) & MHGRID (Asim, Wahib, Munetomo, 2008-2010)

• A unified framework collaborating optimization algorithms libraries and simulation programs to evaluate “fitness” values registered by different developers in GRID computing distributed exec. environment.

GridUFO(Checks(compa3bility(of(sovler:obj(func(pair(

(Solvers(Database(

(Obj(Func(Database(

User(Develops(&(Registers(an((Op3miza3on(Problem(

User(Develops(&(Registers(a(Solver(

Solver(Developer(

User(Selects(a(Solver(&(an(Objec3ve(Func3on(

GridUFO(Deploys(the(Job(over(Grid(

Solver(Obj(Func(

MHAPI(

Ninf:IDL(

Distributed(Implementa3on(over(Grid(

MHML(

Obj(Func(Developer(

Ordinary(User(

Subm

its(Op3

miza3

on(Jo

b(

MHML(

15

LDDEF: System architecture overview

• Fully distributed and scalable architecture consisting of simulators in supercomputers, optimization engines, analyzers object storages and distributed database nodes in the inter-cloud environment.

DB�

Object)Storage(s)�

DB�

DB�

Simulator �

Optimizer �Simulator �

Optimizer �

<s,:f> �

<s,:?> �

<s,:f> �

<s,:?> �

replication �

<p>�<p>�

{:<s,:f>:} �

<s’> �

{:<s,:f>:} �

<s’> �

Analyzer:/ �Visualizer)

Controller:&)User:Interface �

Distributed:DBs �

{:<p>:}�

{:<s,:f>:} �(feedback) �

replication �

16

Cassandra distributed database nodes deployed across Japan

• We have built a testbed of Cassandra distributed database nodes across Japan from Kitami (Hokkaido) to Okinawa connected via SDN (Vyatta).

• We have tested performance with/without replications and availability and resiliency in cases of node and network faults.

0"

1000"

2000"

3000"

4000"

5000"

6000"

1" 11" 21" 31" 41" 51" 61" 71" 81"

0"

1000"

2000"

3000"

4000"

5000"

1" 11" 21" 31" 41" 51" 61" 71" 81"

Num

ber"of"requests�Num

ber"of"requests�

write8latency"(ms)�

read8latency"(ms)�

with"replicaCons� without"replicaCons�Hokkaido'University'Informa3on'Ini3a3ve'Center�

Kitami'Ins3tute''of'Technology�

University'of'the'Ryukyus'

70ms�

60ms�

10ms�

17

Cloud Resource Deployment Optimization (CReDO) in the Inter-Cloud Environment• Optimizing deployment of virtualized systems requested from

users according to their system specifications using multi-objective evolutionary algorithms such as NSGA-II/III.

• Semi-automated scheduling policy to “recommend” a variety of system deployment patterns at Pareto-front to users.

CReDO Solver / Optimizer

DB

Request with Spec. info

Response with Deploy. info

Public Cloud A Public Cloud B Private Cloud

System info., Accounting, etc

18

Multi-objective inter-cloud resource optimization using multi-objective evolutionary algorithms.

• We employ multi-objective evolutionary algorithms such as NSGA-II and NSGA-III to solve resource optimization problems in the inter-cloud environment.

• Solving multi-objective optimization considering cost, performance(response time), and greenness (CO2 emission) simultaneously.

19

Toward the next generation of Hokkaido university academic cloud as high-performance inter-cloud

• We are planning to develop a high-performance inter-cloud system as the next generation Hokkaido university academic cloud

• Inter-cloud (service layer): multi-cloud controller & broker with cloud exchange

• Inter-cloud (infrastructure layer): Inter-cloud connector with SDN controller

Private Cloud with Supercompter & BigData Storage�

Inter-Cloud Portal (multi-cloud controller)

�

Public Cloud A�

VPN (SDN)�Public Cloud B�

Inter-Cloud Connector�

Community Cloud C�Public/Comunity Clouds�

Cloud Exchange�

HPC�

20

High-performance inter-cloud design: an example

SW Super'computer40GbE'(x'142)

FC'or'IB

Tape'Archive

IaaS/'HaaS'

IaaS/'HaaS'

IaaS/'HaaS'

Cloud'Shared'Storage

App WebDAV'S3/Gfarm

HPC'Storage

Campus'LAN

FW'Router

SW

Management'servers'

Baremetal,'VMWare) Baremetal,'CloudStack'or'OpenStack) Baremetal)

100G'x'1 Tape'ArchiveIaaS/'

HaaS'IaaS/'HaaS'

Public'clouds'Community'clouds'

Remote'site'#1 Remote'site'#2

SW SW SW SW

IPS'

40GbE'(x'142)

SINET5'

Campus'DC

SDN' ontroller

22

Roadmap & Future direction

• 2016Q2: Upgrade network infrastructure (SINET5: 100Gbps)

• 2017Q2-Q3: Replacing inter-cloud infrastructure (including remote sites) & supercomputer at Hokkaido university

• Regional inter-cloud collaborations in Hokkaido

• National inter-cloud collaborations with other universities, NII and other research institutes to establish academic community cloud federations

• International inter-cloud collaborations (Asia-Pacific?)

• Investigations on future trends in inter-cloud applications such as IoT/IoE, extreme-scale parallel and distributed computing including big data processing and machine learning.

23

CloudWeek2015@Hokkaido University

• A collection of symposium, conference, and workshop related cloud computing technologies, sponsored by information initiative center, Hokkaido University.

• Sep.7th - 9th or 10th, 2015, at Hokkaido University, Sapporo, Japan.

• Academic Inter-Cloud Symposium 2015 for Universities, Research institutes

• Open Cloud Conference 2015 for Cloud service providers, vendors, etc.

• ITRC RICC (Regional Inter-Cloud Committee) Workshop

• Call for international speakers!

Joint usage/research base

Cloud Week@Hokkaido University (cloud symposium)

Promotion of open-type joint research

Introduction of Petabyte-class Data Science Unified Cloud Storage

Innovative high-performance computing infrastructure (HPCI)

Administration organization of the Information Initiative CenterMain activities of the Information Initiative Center

Cloud Week 2013@Hokkaido University （as of Jul. 1, 2014）

6142（2）

1141116153（2）

55

ProfessorSpecially appointed professorAssociate professorAssistant professor(Visiting researcher)Research support promotion workerClerical workerTechnical workerSpecific professionalPart-time workerResearch support promotion workerClerical assistantClerical support worker

Research divisions

Total

Clerical divisions (Information Environment Promotion Headquarters, Information Promotion Division)

Director of the Information Initiative Center

University of Tokyo Information TechnologyCenter

Kyoto UniversityAcademic Center for Computing and Media Studies

Tsukuba University Center for Computational Science

Tokyo Institute of Technology Global Scientific Information and Computing Center

Japan Agency for Marine-Earth Science and Technology Center for Earth Information Science and Technology

Kyushu University Research Institute for Information Technology

Osaka University Cybermedia Center

Nagoya University Information Technology Center

Hokkaido UniversityInformation InitiativeCenter

Tohoku UniversityCyber Science Center

RIKEN Advanced Institutefor Computational Science “K”

Institute of Statistical Mathematics Center for Engineering and Technical Support

「京」Faculty members

Organization chart Research divisions　As a nationwide joint use facility, the Center aims to promote the advancement of education and research and the implementation and support of education taking advantage of information media, through R&D to facilitate informatization and the development and operation of an information infrastructure.

● Hokkaido University Information Initiative Center● Tohoku University Cyber Science Center● University of Tokyo Information Technology Center (core base)● Tokyo Institute of Technology Global Scientific Information and　 Computing Center● Nagoya University Information Technology Center● Kyoto University Academic Center 　 for Computing and Media Studies● Osaka University Cybermedia Center● Kyushu University Research Institute　 for Information Technology

　The Center is acknowledged as one of the joint use facilities among eight universities constituting the Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures (JHPCN), a network-type joint usage/re-search base established in accordance with the provisions of the Enforcement Regulations of the School Education Law.

■ Facilities constituting the network-type joint usage/　 research base

　In addition to open-type joint research conducted according to the purposes of JHPCN, the Center is conducting its own open-type joint research, in which the Center plays a central role in grand challenge-type research using information infrastructure and academic joint research projects to promote such grand challenges.　In FY 2014, research projects in the following fields were sought from the public and 19 joint research projects are now being conducted in collaboration with teachers and researchers of Japanese and international research institutes.

　Operation of HPCI was commenced in September 2012 by establishing an innovative computation base for joint use to meet diverse user needs, by connecting the next generation supercomputer “K” at RIKEN and supercomput-ers of universities and research institutes nationwide via a high speed network (SINET4). 　The Research Organization for Information Science & Technology (RIST) is in charge of the selection of projects, management of common receptions and other utilization promotion services.　Our Center is participating as a system configuration organization and is providing computation resources. In FY 2014, six research projects using the Center were adopted.Purposes of HPCI● To develop and provide a system and mechanism so that a wide range of high-performance computing (HPC) users nationwide with different individual computation resource provision organizations can efficiently use HPC resources anywhere in the country● To accelerate extensive HPC utilization ranging from seeds to large-scale research, and to develop/expand the industrial use computational science community and return the results to society by enabling nationwide matching of needs and resources

　The Center launched Petabyte-class Data Science Unified Cloud Storage service in April 2014. The system consists of a high-capacity storage (total physical capacity: approx. 2 PB) and a cloud system with 500 physical cores. As its purpose is to provide resources to HPCI (innovative high-performance computing infrastructure) and JHPCN (Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures), the service is provided mainly to those who were selected for these projects.

　As the Hokkaido University Academic Cloud, which is one of Japan’s largest academic clouds, was established in November 2011, symposiums that contribute to exchange of opinions on the current status and future develop-ment of cloud research have been held yearly since FY 2012, by gathering cloud-related researchers from Japan and abroad.　The FY 2013 symposium, which was held for three days and involved more than 300 participants contributed to the development of research technology by inviting leaders of various fields in cloud-related technology, holding lectures and exchanging detailed information.

■ Effective period of acknowledgment: 　 April 1, 2010 – March 31, 2016

■ Purpose of the base　The purpose of this network-type base is to contribute to further advancement and constant development of Japan’s academic and research bases through interdisciplinary joint usage/research concerning so-called grand challenges, which have been considered to be extremely difficult to solve or clarify, using super-large-sized computers, super-high-capacity storages/net-works and other information infrastructures. It covers information processing fields in general, including the global environment, energy, substances/materi-als, genome information, web data, academic information, time-series data from sensor networks, image data and program analysis.

■ Operation of the base　JHPCN is operated by the Steering Committee and Joint Research Project Screening Committee established at the University of Tokyo’s Information Technology Center, which is its core base.

■ Promotion of open-type joint research　For interdisciplinary research using large-scale information infrastructure, JHPCN is conducting 34 joint research projects in FY 2014 (of which 7 is related to our Center), by seeking research projects concerning application of ultra-large-scale numerical calculation and data processing systems and ultra-high-capacity network technology, as well as the field of ultra-large-scale information systems integrating these technologies from the public.

● Large-scale computer simulation

● Large-scale data science

● Digital contents

● Basic technology for　 solving large-scaleproblems

● Networks and cloud　 technology

● Educational information　 media

Board of delegates

Director Vice Director

Large-Scale Computation System Research Division

Information Network Research Division

Digital Contents Research Division

Research divisions

Media Education Research Division

Joint Usage/Research Committee

Special Committee of System Usage

Special Committee of System Technology

Special Committee of Libraries/Databases

Special Committee of Joint Research



Chief of the Information Promotion Section

Information Environment Promotion Headquarters

Head of the IT promotion group

System operation team

Information networkteam

Information security team

Software support team

Committee for On-Campus Joint Use of Educational Information Systems



Committee for On-Campus Joint Use of Information Networks Systems

In charge of joint usage/research

In charge of general affairs/planning

In charge of media education

In charge of business system operation

In charge of accounting

Assistant Section Chief

Large-scale Computation System Research Division

Information Network Research Division

Sophistication of large-scale scientific computing, R&D of application programs and research

on design technologies for large-scale information systems

Development of application software using large-scale/ul-tra-high-speed computer systemsResearch on the design and construction methods of large-scale computational system

●

●

●

●

●

●

●

●

Accumulation, processing, distribution, utilization and other handling of digital data contents

Research on information system infrastructure for accumu-lation, evaluation, processing and distribution of digital data contents centering around an academic cloudResearch on accumulation, evaluation, processing, distribution, utilization and other handling of digital data contents related to natural science, humanities, etc.

　The Center has four research divisions as presented below, where cutting edge research on information technology, development of relevant technologies to support information infrastructure, education activities using information media and related research activities are conducted.

Research on advanced technologies for next-generation information networks,

research on information security and information distribution infrastructure

R&D of basic technologies for next-generation information network systems and basic research on ubiquitous computing and media applicationTheoretical research on the information distribution infrastructure in general, practical research on data analysis and research on the confidentiality and completeness of distribution information

Media Education Research Division

Digital Contents Research Division

Research on sophistication of education using ICT and information education and promotion

of informatized/open education in the global society

Research on sophistication of information education, infor-matization of education, open education, education/learn-ing using ICT and systems to support education/learningResearch on fostering communication abilities based on collaborative learning using ICT

24