Intercloud object storage: colony
2012.4.18
Shigetoshi Yokoyama
National Institute of Informatics
@jxta - Trial to use swift
in wide area network -
1
Cloud projects @ NII
・・・
Univ. -A Cloud Univ.-B Cloud
Univ.-X Cloud
NIINII CloudCloud
Academic
Community Cloud Education Cloud Education Cloud
Research Cloud Research Cloud
Science Information Network
Goal: academic community cloud
2
Intercloud services Intercloud services
dodai dodai
Cluster Cluster
IaaS IaaS
VM VM VM VM
Cluster Cluster
IaaS IaaS
VM VM VM VM Virtual
Machines
Physical
Machines
SINET (L2VPN)
Cluster Cluster
IaaS IaaS
VM VM VM VM
Cluster Cluster
IaaS IaaS
VM VM VM VM
colony VMI VMI VMI VMI: Virtual Machine Image
VM: Virtual Machines
Launch machine images
Colony and dodai in a scenario
3
4
Colony
Intercloud object storage service
5
Swift for
intercloud
use
Swift
Swift
Swift
Swift for
intercloud
use
Swift for
intercloud
use
Swift for
local use
Swift for
intercloud use
Nova
Nova
Nova
Glance
Glance Glance
Colony is a project which
federates cloud object
storage services, like swift
to realize intercloud object
storage service.
Swift-I
Cloud-A
Swift-A
Container A1
Container A2
Container A3
Inter-cloud Container I1
Inter-cloud Container I4
Object A1-1
Object A1-2
Object A1-3
Object I4-1
Object I4-2
Object I4-3
Cloud-B
Container B1
Container B2
Container B3
Inter-cloud Container I1
Inter-cloud Container I8
Object B1-1
Object B1-2
Object B1-3
Object I1-1
Object I1-2
Object I1-3
Inter-cloud object storage service : colony
Cloud Services
Inter-cloud Container
I1 Inter-cloud Container I2
Inter-cloud Container I3
Inter-cloud Container I13
Inter-cloud Container I10
Inter-cloud Container I4
Swift-B
Geographically
Distributed Object I4-1
Object I4-2
Object I4-3
Object I1-1
Object I1-2
Object I1-3
Users’ points of view
6
Colony realizes the federation
7
C o l o n y Apache
mod_wsgi mod_shib
Colony-horizon
Colony-keystone
Colony-dispatcher
Squid
Slapd
Ubuntu
Colony-Keystone
Slapd
Swift
Colony-Keystone
Slapd
Swift
Provide seamless access to
multiple swifts
Authenticate with
Shibboleth IdP
Shibboleth IdP
Cloud-A User
Swift-I Swift-A
Developed software components in colony
• Colony-Horizon – based on diablo/stable Horizon with some enhancements
• Multi-region support – Users can choose which swift is used to store/retrieve objects
• Swift Container’s ACL ,metadata support
• Swift Object’s metadata support
• >5G segment upload support …
• Colony-Keystone – based on diablo/stable Keystone with some enhancements
• Authenticate with Shibboleth
• %{tanant_name} can be used for endpointTemplates in addition to %{tenant_id} for
federating cloud services
• Colony-Dispatcher - new
• Relay requests to multiple object services (and merge response for clients)
• Relay requests to a specific object service indicated by URI
• Choose the “nearest” swift-proxy server to relay requests
• Copy objects among different swifts
• Utilities - new
• Tools to make admin task easier to provide the federation of object storage services
8
Swift Swift --AA
Colony-horizon
9
Swift Swift --II Users can choose swift
Shibboleth SPShibboleth SP
10
ColonyColony--HorizonHorizon ColonyColony--HorizonHorizon
Shibboleth IdP
Colony-
Keystone
Colony-
Keystone
Colony – keystone
1. ID/passwd 2. Attribute: ePPN, mail_addr
3. Attribute: ePPN
4. auth_token
0-1. User registration by mail_addr
0-2 . Associate ePPN to mail_addr
by initial access
Modifications to keystone
• Add ePPN field to keystone schema
•
('/ /email')
•
('/users/{
• Add ePPN field to keystone schema
• ADD rest api services to create token by ePPN
('/token_by/eppn') and email address('/token_by/email')
• Add a rest api service to register/update ePPN
('/users/{user_id}/eppn')
Colony-dispatcher
Swift Proxy
Colony Dispatcher
Swift Proxy Swift Proxy
Swift-A (local) Swift-I (intercloud )
A:container1
A:container2
I:container1
I:container2
Swift Client
1. Swift client can send requests to Swift-A and Swift-I through Swift Dispatcher
2. Swift Dispatcher merges the response from each Swift and send it to Swift Client
Requests modified for merging responses.
•Account Info
•Container List
•X-Copy-from/to
Response merged by
Colony Dispatcher has
a prefix to indicate
which Swift is used to
store.
Response merged by
Colony Dispatcher has
a prefix to indicate
which Swift is used to
store.
11
A:container1
A:container2
I:container1
I:container2
Swift Client
Swift Dispatcher can use cache proxy (like squid) per
swift proxy to retrieve objects from remote swifts.
Caching
12
Swift Proxy
Colony Dispatcher
Swift Proxy Swift Proxy
Swift-A (local) Swift-I (intercloud )
Cache(Proxy)
Download performance with cache
13
0
50,000,000
100,000,000
150,000,000
200,000,000
250,000,000
300,000,000
1K(bytes) 1M(bytes) 10M(bytes) 100M(bytes) 1G(bytes) 5G(bytes)
Normal 13,201 11843208 77113446 152263382 163115324 154252275
Cache 382,004 135938185 226800796 256492362 261189612 260956927
byte
s/se
c
Download performance (5times, average)
・AMD Opteron(tm) Processor 6128 2000Mhz 16core ・32GB Memory ・10000baseT 10000Mb/s Full Duplex
Ubuntu Server 11.04 x86_64 Client Colony-Dispatcher Squid Swift-Proxy
・AMD Opteron(tm) Processor 6128 2000Mhz 16core ・32GB Memory ・10000baseT 10000Mb/s Full Duplex
Ubuntu Server 11.04 x86_64 Swift-Storage(account, container, object)x 3
Cache is
effective,
especially for
small size
objects.
Performance : original swift
14
Swift
@Tokyo Swift@ Chiba
Swift @Sapporo
6.15Mbits/sec
6.29Mbits/sec
941.23Mbits/sec
929.15Mbits/sec
7.80Mbits/sec
7.73Mbits/sec
1 2 3 4 5
1K 1,755 2,165 395 3,178 1.375
1M 431,840 446,824 537,768 412,542 43,307
10M 710,179 741,350 737,565 723,446 731,824
100M 780,548 775,131 788,620 782,503 786,224
1 2 3 4 5
1K 11,608 13,293 12,916 14,691 12,126
1M 3,427,827 3,427,402 3,455,788 673,614 3,386,009
10M 958,018 959,098 7,188,901 974,072 949,389
100M 1,109,440 98,400,555 99,641,184 15,411,221 92,400,774
•Upload – Always effected by worst case
•Download – worse case is ramdomly choosed (about 1/3)
Client
Swift cannot perform well when
confgirure it in inter-region.
10G
1G
netperf
Performance : with cache
15
1 2 3 4 5
1K 2,388 2,027 1,629 2,718 2,423
1M 521,456 479,606 488,142 455,359 541,622
10M 727,240 678,511 724,562 730,127 699,518
100M 786,893 777,227 787,707 782,546 784,544
1 2 3 4 5
1K 398,133 415,753 397,979 446,576 406,510
1M 9,238,473 202,193,598 177,694,628 161,992,275 134,450,057
10M 55,186,177 288,776,403 319,892,614 331,691,392 307,500,293
100M 473,135,008 471,484,134 467,654,981 412,153,419 451,464,516
•Upload – Always effected by worst case
•Download (through Cache)
Cache works
Swift
@Tokyo Swift@ Chiba
Swift @Sapporo
6.15Mbits/sec
6.29Mbits/sec
941.23Mbits/sec
929.15Mbits/sec
7.80Mbits/sec
7.73Mbits/sec
Client
10G
1G
netperf
Performance : with zone-awareness
16
1 2 3 4 5
1K 11,356 13,157 13,074 12,758 12,680
1M 9,824,750 11,205,249 7,599,312 10,931,206 11,199,982
10M 52,294,403 51,437,092 51,050,686 52,641,471 52,300,141
100M 97,937,987 101,847,002 102,385,002 102,413,801 101,462,855
•Upload
Modifications for zone-awareness (just a hack) Modifications for zone-awareness (just a hack)
•Uploading • Calculate the number of unfinished tasks in the send queue for each area and when one area are much worth than others, quit the uploading job for its area.
•Downloading • Check connection performance of object servers and try to retrieve an object from the fastest one.
Swift should be zone-aware
for geographically distributed use.
Swift
@Tokyo Swift@ Chiba
Swift @Sapporo
6.15Mbits/sec
6.29Mbits/sec
941.23Mbits/sec
929.15Mbits/sec
7.80Mbits/sec
7.73Mbits/sec
Client
10G
1G
netperf
• Prototyping is done in NII research cloud
https://github.com/nii-cloud/colony/
• Future plan
- Performance enhancements
- Real deployment and evaluation
17
Project status of colony
18
Dodai
Server resource pool
Clusters for each private cloud
Allocate servers
from server resource pool
Release servers
to server resource pool
Private cloud - A Private cloud - B Private cloud - C
CaaS : Cluster as a Service
Elastic
Private
Cloud
19
OpenStack-1 Eucalyptus
Hadoop SunGrid
Engine
Hadoop MPI Hadoop MPI
CaaS
IaaS
PaaS
OpenStack-2
Hadoop MPI
Dynamically and securely separated clusters
…
Upper layer: Deploying software [dodai-deploy]
Lower layer: Preparation of nodes [dodai-compute]
Dodai = dodai-deploy + dodai-compute
20
Dodai-deploy • It is a framework for deployment tool
– New software can be added by anyone
– Now supports
• Openstack Essex
• Openstack Diablo
• Hadoop
• SGE
• It is fast
– Do installation parallel on different machines.
• It is easy to use
– Not just Web UI
– CLI for maintainer
– Rest API for integration
21
How to use dodai-deploy • Install
– Just one command to install all-in-one
• curl https://raw.github.com/nii-cloud/dodai-deploy/master/setup-env/setup-
all-in-one.sh | sudo sh
• Use
– Web UI
• Learn step-by-step guidance at
http://$dodai_deploy_server:3000/
– REST APIs
• Try at http://$dodai_deploy_server:3000/rest_apis/index.html
– CLI
• script/cli.rb list SERVER
22
What’s inside in dodai-deploy
23
Ruby on Rails
MCollective
puppet dodai-deploy server
dodai-deploy nodes
dodai-compute
• In a word
–Bare metal version nova
• Futures
–Provide the same interface as nova to
operate on physical machines
–Separate network segments with Open
Flow Controller
24
What’s inside in dodai-compute
OpenFlow
controller modified nova
OpenFlow
switch
cobbler
25
1. Dodai-deploy
- One of the OpenStack compute automated installations tools http://docs.openstack.org/trunk/openstack-compute/admin/content/openstack-compute-deployment-tool-with-puppet.html
https://github.com/nii-cloud/dodai-deploy/
- Used in NII research cloud development
2. Dodai-compute
https://github.com/nii-cloud/dodai-compute/
- Used in NII research cloud development
3. Future plan
- Enahance based on Feedbacks from NII researchers
- Experiments on intercloud compute service
26
Project status of dodai
27
NII research cloud
gunnii
NII academic cloud : gunnii
dodai-compute
dodai-deploy
Clu
ste
r as a
Serv
ice
on
demand
on
demand
on
demand
28
Deployment Tool
Compute
Based on
bare metal
supported
generalized
“Elastic Private Cloud”
Object storage (Swift)
OpenFlow
Switch/controller
OpenFlow
Switch/controller
NFS
L3 switch
Researchers Researchers
NII shibboleth IdP NII shibboleth IdP
Compute nodes×40 (initial)
GUI server
Deployed Hadoop, OpenStack, Sun Grid Engine
Storage nodes X 8 (initial)
gunnii prototype system
29
Demo ?
30
gunnii is shibboleth ready.
31
Thank you
https://github.com/nii-cloud/