Openstack days taiwan 2016 0712

1

~ Architecture of our public clouds ~

OpenStack Days TaiwanJul 12, 2016Naoto Gohko (@naoto_gohko)GMO Internet, Inc.

How is GMO Ineternet using OpenStack

for Public CloudSlide URLhttp://www.slideshare.net/chroum/openstack-days-taiwan-2016-0712-public-cloud-arch

ConoHa public cloud (lang zh)https://www.conoha.jp/zh/

ConoHa public cloud (lang en)https://www.conoha.jp/en/

http://www.slideshare.net/chroum/openstack-days-taiwan-2016-0712-public-cloud-arch



https://www.conoha.jp/zh/





2

Public Clouds

We are offering multiple public cloud services.

3

Physical Servers

Running VMPhysical Server

1508

25294

Created VM

Running Infrastructure (2015/10)

137223

4

OpenStack service development team

5

Cloud service development team: (abount 30 people)– OpenStack Neutron team: 4 people

• Neutron driver / modification / engineering– Cloud API development team: 5 people

• Public API validation program• OpenStack modification / scaduler programing / keystone

– Cloud Infra. development team: 11 people• Security engineering / glance driver / cinder driver / nova

additional extensions / construction of OpenStack infra.– Applicatoin cloud service development team: 5 people

• Billing engineering / staff tools / GMO AppsCloud web GUI

Additional engineering team: many people (30 ~) – QA Team / Server Engineering Team / GUI development Team– Network Engineering Team / SaaS development Team– CRM backend and billing Team

Cloud service development team: Now(2016)

6

Cloud service development team: Office(2016) #1

Neutron TeamAnd

Cloud API Team

Cloud Infra. TeamAnd

AppsCloud Team

7

Cloud service development team: Office(2016) #2

Neutron TeamAnd

Cloud API Team

Cloud Infra. TeamAnd

AppsCloud Team

8

Limied number of people.But, we have to run a lot of OpenStack service clusters.

9

Service developmemt historyby OpenStack

10

Swift cluster

GMO Internet, Inc.: VPS and Cloud servicesOnamae.com VPS (2012/03) : http://www.onamae-server.com/Forcus: global IPs; provided by simple "nova-network"

tenten VPS (2012/12)http://www.tenten.vn/Share of OSS by Group companies in Vietnam

ConoHa VPS (2013/07) : http://www.conoha.jp/Forcus: Quantam(Neutron) overlay tenant network

GMO AppsCloud (2014/04) : http://cloud.gmo.jp/OpenStack Havana based 1st regionEnterprise grade IaaS with block storage, object storage, LBaaS and baremetal compute was provided

Onamae.com Cloud (2014/11)http://www.onamae-cloud.com/Forcus: Low price VM instances, baremetal compute and object storage

ConoHa Cloud (2015/05/18) http://www.conoha.jp/Forcus: ML2 vxlan overlay, LBaaS, block storage, DNSaaS(Designate) and original services by keystone auth

OpenStack Diablo

on CentOS 6.x

NovaKeystoneGlance

Nova network

Shared codes

Quantam

OpenStack Glizzly

on Ubuntu 12.04

NovaKeystoneGlance

OpenStack Havana

on CentOS 6.x

KeystoneGlance

Cinder

Swift

Swift

Shared cluster

Shared codes KeystoneGlanceNeutron

Nova SwiftBaremetal compute

NovaCeilometer

Baremetal compute

Neutron LBaaS

ovs + gre tunnel overlay

Ceilometer

Designate

SwiftOpenStack Junoon CentOS 7.x

NovaKeystoneGlanceCinder

Ceilometer Neutron LBaa

SGMO AppsCloud (2015/09/27) : http://cloud.gmo.jp/2nd region by OpenStack Juno based Enterprise grade IaaS with High IOPS Ironic Compute and Neutron LBaaS

Upgrade Juno

GSLB

SwiftKeystone Glance

CinderCeilometer

NovaNeutron

IronicLBaaS

http://www.onamae-server.com/

http://www.tenten.vn/

http://www.conoha.jp/

http://cloud.gmo.jp/



http://cloud.gmo.jp/

11

Dark age for the Cloud suppliers

12

OpenStack Swift: shared cluster

13

Swift Hardware: Object nodes• Boot: SSD x2• HDD: 4TB x12• E3-1230 v3 @ 3.30GHz• Memory 16GB• 10GbE x2 (SFP+)

(Intel NIC)ASUSTeK COMPUTER INC.RS300-H8-PS12

14

Hardware: LVS-DSR and reverse-proxy(Layer7) nodes• Boot: SSD x2

• E3-1230 v3 @ 3.30GHz• Memory 16GB• 10GbE NIC x1 (Intel NIC)Supermicro microblade8 blade nodes type

15

Hardware: swift-proxy nodes• Boot: HDD x6 (1.7TB)

– Ceilometer Log disk– (Swift all request billing data)

• E5620 @ 2.40GHz x2 CPU• Memory 64GB• NIC: 10GbE SFP+ x2(Intel NIC)System x3550 M3 (old IBM)

Hardware: account/container-server nodes• Boot: HDD x2• Account/Container storage: SSD x2• E5620 @ 2.40GHz x2 CPU• Memory 64GB• NIC: 10GbE SFP+ x2(Intel NIC)System x3550 M3 (old IBM)

16

Swift cluster (Havana to Juno upgrade)

SSD storage:container/account server at every zone

18

swift proxy

keystone

OpenStack Swift cluster (5 zones, 3 copy)

swift proxy

keystoneLVS-DSRLVS-DSR HAProxy(SSL)HAProxy(SSL)

Xeon E3-1230 3.3GHzMemory 16GB

Xeon E3-1230 3.3GHzMemory 16GB

Xeon E5620 2.4GHz x 2CPUMemory 64GB

swift objects

swift objects

Xeon E3-1230 3.3GHz

swift accountswift container

Xeon E5620 2.4GHz x 2CPUMemory 64GB, SSD x 2

swift objects

swift objects

Xeon E3-1230 3.3GHz



swift objects

swift objects

Xeon E3-1230 3.3GHz



swift objects

swift objects

Xeon E3-1230 3.3GHz



swift objects

swift objects

Xeon E3-1230 3.3GHz



19

swift objectsswift objects





swift proxy keystone

Havana AppsCloudswift proxy keystone

Grizzly ConoHa

HavanaTo Juno

swift account

swift container

swift account

swift container

swift account

swift container

swift account

swift container

swift account

swift container


Juno ConoHaswift proxy keystone

Juno AppsCloud

Swift cluster: multi-auth and multi-endpoint


Juno Z.com

20

Swift shared cluster: ex)

21

OpenStack history of computing environment

22

Oname.com VPS(Diablo) • Service XaaS model:

– VPS (KVM, libvirt)• Network:

– 1Gbps• Network model:

– Flat-VLAN (Nova Network), without floting IP(no L3)

– IPv4 only• Public API

– None (only web-panel)• Glance

– Public image only.

OpenStack service: Onamae.com VPS(Diablo)

23

ConoHa(Grizzly)• Service XaaS model:

– VPS + Private networks (KVM + ovs)• Network model:

– Flat-VLAN + Quantam ovs-GRE overlay

– IPv6/IPv4 dualstack• Network:

– 10GE wired(10GBase-T)

• Public API: None (only web)• Glance

– Only Public image• Cinder: None• ObjectStorage

– Swift (After Havana)

OpenStack service: ConoHa(Grizzly, 2013/07)

24

Grizzly• Quantam Network:

– It was using the initial version of the Open vSwitch full mesh GRE-vlan overlay network with LinuxBridge Hybrid

ButWhen the scale becomes large, Localization occurs to a specific node of the communication of the GRE-mesh-tunnel(with under cloud network(L2) problems)(Broadcast storm?)

OpenStack service: ConoHa(Grizzly)

25

• Service XaaS model:– KVM compute + Private VLAN networks + Cinder + Swift

• Network:– 10Gbps wired(10GBase SFP+)

• Network model: – IPv4 Flat-VLAN + Neutron LinuxBridge(not ML2) + Cisco Nexsu L2 sw/port

driver– Brocade ADX L4-LBaaS original driver

• Public API– Provided the public API

• Ceilometer (Billing)• Glance : Provided(GlusterFS)• Cinder : HP 3PAR(Active-Active Multipath original) + NetApp• ObjectStorage : Swift cluster • Bare-Metal Compute

– Modifiyed cobbler bare-metal deploy driver – Cisco Nexsus switch bare-metal networking driver (L2 tenant NW)

OpenStack service: GMO AppsCloud(Havana)

26

OpenStack service: GMO AppsCloud model

compute

vm

NIC

Vlan network

bridge

NIC vlan

tap

vNIC

Vlan network

vNIC

bridge

vlan

tap

compute

NIC

bridge

NIC vlan

bridge

vlan

public networkNeutron LinuxBridge model(very Fast, simple is Best) this cloud is optimized services for the GAME server.

27

Cisco Nexsus L2 sw/Port manage driver(self made)• L2 resource is limited / SW CPU

– MAC ADDRESS– VLAN per Network– VLAN per Port

Allowed VLAN to trunked port is allowed only VLAN to be used in LinuxBridge in VM/Baremetal Compute node.

– Baremetal : link aggregation port– Port discovery using by lldp

• Cisco Nexsus NX-OS– Server:

LACP : port-ChannelActive-Active link aggreration

29

Nova-baremetal(havana)/Ironic(juno) ansibleBaremetal networking• Bonding NIC + lldp discovery• Taged VLAN• allowd VLAN + dhcp native VLAN

30

GMO AppsCloud(Havana/Juno)

31

Public API security and load balance:• LVS-DSR• L7 reverse-proxy• API validation wrapper

32

public API

Web panel(httpd, php)

API wrapper proxy(httpd, phpFramework: fuel php)

Nova API

Customer sys API

Neutron API Glance API

OpenStack API for input validation

Customer DB

Keystone API

OpenStack API

Cinder APICeilometer API

Endpoint L7:reverse proxy

Swift Proxy

33

public API: step 1, step 2)

step 1) LVS-DSR (L4) is received https(tcp/443) packet, then forward api-reverse-proxy real IP’s.

step 2) HAProxy has valid API ACL and backend server configurations.IF HAProxy allowed POST “/v2.0/tokens”, then the request call to ext-api-wrapper0[12].

34

public API: step 3), step 4)

step 3) ext-api-wrapper0 [12], it is a php program.request URI and header, and the input value of json of the body was confirmed by php, and then call the real OpenStack API as the next processing.

step 4) OpenStack API that is checked the input value will be run.

35

OpenStack Juno cluster: • ConoHa (Juno) and Z.com

cloud• AppsCloud (Juno)

36

Tokyo

Singapore

Sanjose

# ConoHa has data centers in 3 Locations

37

Tokyo Singapole

User/tenant User/tenant

API ManagementKeystone API

API Management

Keystone API

API ManagementKeystone API

Token Token

Tokyo SanJoseSingapore

API Management

Keystone API

API Management

Keystone API READ/

WRITEREAD READ

TokenToken Token

Do not create/delete

users

Do not create/delete

users

Our Customer baseUser administration

# User-registration is possible in Japan only

DB Replication DB ReplicationUser/tenant User/tenantUser/tenant

R/W R/W

Yuya Matoba

38

OpenStack Juno: 2 service cluster, released

Mikumo ConoHa Mikumo Anzu

Mikumo = 美雲 = Beautiful cloud

New Juno region released: 10/26/2015

39

• Service model: Public cloud by KVM• Network: 10Gbps wired(10GBase SFP+)• Network model:

– Flat-VLAN + Neutron ML2 ovs-VXLAN overlay + ML2 LinuxBridge(SaaS only)

– IPv6/IPv4 dualstack• LBaaS: LVS-DSR(original)• Public API

– Provided the public API (v2 Domain)• Compute node: ALL SSD for booting OS

– Without Cinder boot • Glance: provided• Cinder: SSD NexentaStore zfs (SDS)• Swift (shared Juno cluster)• Cobbler deply on under-cloud

– Ansible configuration• SaaS original service with keystone auth

– Email, web, CPanel and WordPress



– L4-LB-Nat + Neutron ML2 LinuxBridge VLAN– IPv4 only

• LBaaS: Brocade ADX L4-NAT-LB(original)• Public API

– Provided the public API• Compute node: Flash cached or SSD• Glance: provided (NetApp offload)• Cinder: NetApp storage• Swift (shared Juno cluster)• Ironic on under-cloud

– Compute server deploy with Ansible config• Ironic baremetal compute

– Nexsus Cisco for Tagged VLAN module– ioMemory configuration

40

OpenStack Cinder Block storage:

ConoHa: NexentaStor(SDS)AppsCloud: NetApp

41

NexentaStor zfs cinder: ConoHa cloud(Juno)

Compute

42

NetApp storage: GMO AppsCloud(Havana/Juno)If you are using the same Cluster onTAP

NetApp a Glance and Cinder storage, it is possible to offload a copy of the inter-service of OpenStack as the processing of NetApp side.

• Create volume from glance image

((glance the image is converted (ex: qcow2 to raw) required that does not cause the condition)

• Volume QoS limit: Important function of multi-tenant storage• Uppper IOPS-limit by volume

43

OpenStack Ironic: Only AppsCloud:• Undercloud Ironic deploy• Multi-tenant Ironic deploy

44

Ironic with undercloud: GMO AppsCloud(Juno)For Compute server deployment.Kilo Ironic and All-in-one• Compute server: 10G boot• Clout-init: network• Compute setup: Ansible

Under-cloud Ironic(Kilo):It will use a different network and Ironic Baremetal dhcp for Service baremetal compute Ironic(Kilo).(OOO seed server)

Trunk allowed vlan, LACP

45

Ironic(Kilo) baremetal: GMO AppsCloud(Juno)Boot baremetal instance• baremetal server

(with Fusion ioMemory SanDisk)• 1G x4 bonding + Tagged allowed

VLAN• Clout-init: network + lldp• Network: Nexsus Cisco

Allowd VLAN security

Ironic Kilo + Juno: Fine• Ironic Python driver• Whole Image write• Windows: OK

46


– Flat-VLAN + Neutron ML2 ovs-VXLAN overlay + ML2 LinuxBridge(SaaS only)

– IPv6/IPv4 dualstack• LBaaS: LVS-DSR(original)• Public API

– Provided the public API (v2 Domain)• Compute node: ALL SSD for booting OS

– Without Cinder boot • Glance: provided• Cinder: SSD NexentaStore zfs (SDS)• Swift (shared Juno cluster)• Cobbler deply on under-cloud

– Ansible configuration• SaaS original service with keystone auth

– Email, web, CPanel and WordPress



– L4-LB-Nat + Neutron ML2 LinuxBridge VLAN– IPv4 only

• LBaaS: Brocade ADX L4-NAT-LB(original)• Public API

– Provided the public API• Compute node: Flash cached or SSD• Glance: provided (NetApp offload)• Cinder: NetApp storage• Swift (shared Juno cluster)• Ironic on under-cloud

– Compute server deploy with Ansible config• Ironic baremetal compute

– Nexsus Cisco for Tagged VLAN module– ioMemory configuration

47

Fin.