58
1 © Copyright 2012 EMC Corporation. All rights reserved. CloudFoundry: Elastic Architecture Design and Deploy Based on OpenStack 郭郭 (Golfen Guo) @ 郭郭郭郭郭郭 [email protected] Cloud Platform and Application Lab, EMC Labs China

Cloud foundry elastic architecture and deploy based on openstack

Embed Size (px)

Citation preview

Page 1: Cloud foundry elastic architecture and deploy based on openstack

1© Copyright 2012 EMC Corporation. All rights reserved.

CloudFoundry: Elastic Architecture Design and Deploy Based on OpenStack

郭峰 (Golfen Guo)@ 教授也是浮云[email protected]

Cloud Platform and Application Lab, EMC Labs China

Page 2: Cloud foundry elastic architecture and deploy based on openstack

2© Copyright 2012 EMC Corporation. All rights reserved.

EMC Labs China

Advanced Technology Research and Development

Big Data Lab

Cloud Infrastructure and System Lab

Cloud Platform and Applications Lab

University Collaboration

Industry Standards Office

IP Portfolio Development

VisionBecome an elite research and

advanced technology institute in China

-Become the model

for future EMC Labs worldwide

Page 3: Cloud foundry elastic architecture and deploy based on openstack

3© Copyright 2012 EMC Corporation. All rights reserved.

Agenda

• What is PaaS

• CloudFoundry Overview

• Elastic Architecture in CloudFoundry

• Introduce to BOSH

• CPI and OpenStack

• Deploy PaaS with BOSH

Page 4: Cloud foundry elastic architecture and deploy based on openstack

4© Copyright 2012 EMC Corporation. All rights reserved.

What is PaaS? (from Search Cloud Computing)

• Platform as a Service (PaaS) is a way to rent hardware, operating systems, storage and network capacity over the Internet.

• The service delivery model allows the customer to rent virtualized servers and associated services for running existing applications or developing and testing new ones.

RENT, not BUILD one!

Virtual Server and Associated Service

Page 5: Cloud foundry elastic architecture and deploy based on openstack

5© Copyright 2012 EMC Corporation. All rights reserved.

How PaaS Benefit Us?

Before: Now:

Page 6: Cloud foundry elastic architecture and deploy based on openstack

6© Copyright 2012 EMC Corporation. All rights reserved.

How PaaS Benefit Us? (Cont.)

Traditional IT

Storage

Servers

Networking

O/S

Middleware

Virtualization

Data

Applications

Runtime

You

Man

age

O/S

PaaS

Storage

Servers

Networking

Middleware

Virtualization

Data

Applications

Runtime

You

Man

age

Man

ag

ed b

y P

latfo

rm

Agility and Cost

Savings

O/S

IaaS

Storage

Servers

Networking

Middleware

Virtualization

Data

Applications

RuntimeY

ou M

anag

eM

an

ag

ed b

y P

latfo

rm

From Accelerating your Journey to Application Transformation, EMC World 2012

PaaS powered

Page 7: Cloud foundry elastic architecture and deploy based on openstack

7© Copyright 2012 EMC Corporation. All rights reserved.

Cloud Foundry – The Open PaaS

Page 8: Cloud foundry elastic architecture and deploy based on openstack

8© Copyright 2012 EMC Corporation. All rights reserved.

PaaS in Action…

$ cat hw.c#include <stdio.h>main() { printf(“Hello World\n”);}

$ cc hw.c; ./a.out

Another analogy from Mark Lucovsky,

Every one knows, “Hello World”:

Page 9: Cloud foundry elastic architecture and deploy based on openstack

9© Copyright 2012 EMC Corporation. All rights reserved.

PaaS in Action... (Cont. )

$ cat hw.rb require 'rubygems' require 'sinatra'

$hits = 0 get '/' do $hits = $hits + 1 "Hello World - #{$hits}" end

$ vmc push hw

“Hello World” in the Cloud:

Page 10: Cloud foundry elastic architecture and deploy based on openstack

10© Copyright 2012 EMC Corporation. All rights reserved.

Scale App On Demand

Step 1: Scale instance to 10:

$ vmc instances hw 10

Step 2: Change the codes:

get '/' do $hits = $redis.incr(‘hits’) "Hello World - #{$hits}" end

Redis

Step 3: bind Redis service to App:

$ vmc create-service redis –bind hw

Implement a Scalability “Hello World” in Cloud:

Page 11: Cloud foundry elastic architecture and deploy based on openstack

11© Copyright 2012 EMC Corporation. All rights reserved.

Scale App On Demand(Cont.)

From Cloud Foundry Launch Event, April, 2011 PaaS powered

Page 12: Cloud foundry elastic architecture and deploy based on openstack

12© Copyright 2012 EMC Corporation. All rights reserved.

Simplify Deployment [mysqld]user = foobarport = 3306basedir = /usrbind-address = 172.58.77.101key_buffer = 16Mthread_stack = 128Kthread_cache_size = 8…

[nginx]http.include mime.types;default_type: application/octet-stream;log_format: main ‘$remote_addr - $remote_user []…’keepalive_timeout 65;

[tomcat]<Connector redirectPort=“8443” emptySessionPath…/><bean id=“sessionFactory” class=“org.springframework…/> [frontend]dependencies:- mysqlclient- rubyfiles:- core/app/fe/**/*- core/common/**/*

[blah]- blah blah blah

Web App

Before:

Page 13: Cloud foundry elastic architecture and deploy based on openstack

13© Copyright 2012 EMC Corporation. All rights reserved.

Simplify Deployment (Cont.)

Web App

# to target and login to cloud foundryvmc target http://api.cloudfoundry.comvmc login

# to create and boot the app for the first timevmc push myapp –instances 2 –mem 64M –path ../code

# to create the database and bind it to the appvmc create-service mysql –name mydb –bind myapp

# update live app with new codevmc update myapp –path ../code

Now:

Page 14: Cloud foundry elastic architecture and deploy based on openstack

14© Copyright 2012 EMC Corporation. All rights reserved.

First Touch CloudFoundry

prompt> gem install vmcprompt> vmc target api.cloudfoundry.comprompt> vmc loginprompt> vmc push

Would you like to deploy from the current directory? [Yn] Yes Application Name: hello Application Deployed URL: 'hello.cloudfoundry.com'? hello-bob.cloudfoundry.com Detected a Sinatra Application, is this correct? [Yn] Yes Memory Reservation [Default:128M] (64M, 128M, 256M, 512M or 1G) (Press Enter to take default) Would you like to bind any services to 'hello'? [yN]: No Uploading Application: Checking for available resources: OK Packing application: OK Uploading (0K): OK Push Status: OK Staging Application: OK Starting Application: OK

Page 15: Cloud foundry elastic architecture and deploy based on openstack

15© Copyright 2012 EMC Corporation. All rights reserved.

First Touch CloudFoundry(Cont.)Upload your app

Your apps deployed

Your all created services

Memory of each instance

Change served instances

Start, stop, update, restart

Services of current app

Information of app

Page 16: Cloud foundry elastic architecture and deploy based on openstack

16© Copyright 2012 EMC Corporation. All rights reserved.

PaaS Architecture Pattern

RabbitMQ

Redis

MongoDB

MySQL

Page 17: Cloud foundry elastic architecture and deploy based on openstack

17© Copyright 2012 EMC Corporation. All rights reserved.

PaaS Architecture Pattern (Cont. )

Page 18: Cloud foundry elastic architecture and deploy based on openstack

18© Copyright 2012 EMC Corporation. All rights reserved.

PaaS Architecture Pattern (Cont. )

Page 19: Cloud foundry elastic architecture and deploy based on openstack

19© Copyright 2012 EMC Corporation. All rights reserved.

Conclusion

Abstract to three layers:

• Routers for finding right endpoint of Apps

• Runtime for Apps

• Services provided by platform, consumed by Apps

Page 20: Cloud foundry elastic architecture and deploy based on openstack

20© Copyright 2012 EMC Corporation. All rights reserved.

The Keys of Design…

• Elastic

• Failover/System Robust

• Resource Recycling

Page 21: Cloud foundry elastic architecture and deploy based on openstack

21© Copyright 2012 EMC Corporation. All rights reserved.

Elastic Architecture in CloudFoundry

Design principals:1. Each components can be run standalone;2. Each components can be scale-out, and notify the peers with

message;3. The components communicate only with message or RESTful

API.

Self-government and Loose Couples :• Easy to add new components. eg Stager, UAA, ACM…;• Easy to evolve each component. eg. CC_ng, Router v2…

Page 22: Cloud foundry elastic architecture and deploy based on openstack

22© Copyright 2012 EMC Corporation. All rights reserved.

Elastic Architecture in CloudFoundryA request comes…

Page 23: Cloud foundry elastic architecture and deploy based on openstack

23© Copyright 2012 EMC Corporation. All rights reserved.

Pluggable Runtime Support

Refers to https://github.com/cloudfoundry/vcap-staging

• stager -> vcap-staging

Stager StagingPlugin

PHP Rails3Java_We

bPlay …

klass = StagingPlugin.load_plugin_for(plugin_name)plugin = klass.from_file(config_path)plugin.stage_application

extends StagingPlugin

Start Script

Stop Scrip

tServe

r

App

Page 24: Cloud foundry elastic architecture and deploy based on openstack

24© Copyright 2012 EMC Corporation. All rights reserved.

Pluggable Runtime Support (Cont.)

So what we need to do is…• Extends Class StagingPlugin in

Common.rb

Line3 ~ Line 62, 59 lines of codes to support PHP.

Page 25: Cloud foundry elastic architecture and deploy based on openstack

25© Copyright 2012 EMC Corporation. All rights reserved.

Pluggable Runtime Support (Cont.)

Key methods to rewrite:• stage_application

• start_command

• startup_script

• stop_command

• stop_script

Page 26: Cloud foundry elastic architecture and deploy based on openstack

26© Copyright 2012 EMC Corporation. All rights reserved.

Pluggable Services SupportRefers to a nice presentation by Nicholas

Kushmerick

Cloud Foundry Services in last forum:• Service advertisement

• Service Gateway -> Cloud Controller

• POST /services/v1/offerings

• DELETE /services/v1/offerings/:label

• Instance management

• Cloud Controller -> Service Gateway

• Provision: POST /gateway/v1/configurations

• Bind: POST /gateway/v1/configurations/:id/handles

• Unbind: DELETE /gateway/v1/configurations/:id/handles/:handle

• Unprovision: DELETE /gateway/v1/configurations/:id

Page 27: Cloud foundry elastic architecture and deploy based on openstack

27© Copyright 2012 EMC Corporation. All rights reserved.

But …

• Modular Design

• Version Tolerance

• Flexible Runtime/Service

• Elastic Architecture

• Many kinds of nodes

• Many nodes each kind

Complex deployment process like other distribution system

Trade Off

Page 28: Cloud foundry elastic architecture and deploy based on openstack

28© Copyright 2012 EMC Corporation. All rights reserved.

Practical problem at CloudFoundry.com

40+ unique node types

75+ unique software packages

500-5,000 VMs

2x/week cf.com updates

24*7*365 non-stop operation

No-downtime deployments

Reliable ,robust ,repeatable deployments ,updates , capacity adjustments

Small teams manage many instances

Page 29: Cloud foundry elastic architecture and deploy based on openstack

29© Copyright 2012 EMC Corporation. All rights reserved.

How to deploy CloudFoundry?

Page 30: Cloud foundry elastic architecture and deploy based on openstack

30© Copyright 2012 EMC Corporation. All rights reserved.

CloudFoundry BOSH CloudFoundry BOSH is an open source tool-chain for release

engineering, deployment, and lifecycle management of large scale distributed services

– Prescriptive way of creating releases and managing systems and services

– It is not a collection of shell scripts, not a pile of Perl

Built to deploy and manage production-class, large scale clusters– Production grade Cloud Foundry clusters: 500+ VMs, 40+ jobs, 75+

packages– Multi-node, multi-tier, complex clusters

Built for DevOps usage and scale by a crack team of veterans– A project, not a product: command line interface, YAML, etc.– Continuous improvement, iterative development, rough edges

https://github.com/cloudfoundry/bosh

Page 31: Cloud foundry elastic architecture and deploy based on openstack

31© Copyright 2012 EMC Corporation. All rights reserved.

CloudFoundry BOSH (Cont.)

Built from the need to operate cloudfoundry.com– Over 40 different types of VMs– Hundreds of running instances of VMs– Pushing at least 2 changes per week

End-to-end management– Release engineering– Deployment– Lifecycle management

Generic solution– Any service– Any cloud– Any operating system

Page 32: Cloud foundry elastic architecture and deploy based on openstack

32© Copyright 2012 EMC Corporation. All rights reserved.

BOSH Architecture

1. Upload Stemcell1. CLI -> Director2. Director -> Blobstore

2. Upload Release1. CLI -> Director2. Director -> Blobstore

3. Deploy1. CLI -> Director2. Director -> A3. Agent -> B4. Agent -> C

Page 33: Cloud foundry elastic architecture and deploy based on openstack

33© Copyright 2012 EMC Corporation. All rights reserved.

BOSH Concepts Stemcell

– VM template– BOSH Agent– IaaS Plugin

Release– Jobs

Job– Packages– Templates (scripts, confs)– Monitoring

Package– Source/blobs– Dependencies– Packaging (scripts)

Page 34: Cloud foundry elastic architecture and deploy based on openstack

34© Copyright 2012 EMC Corporation. All rights reserved.

IaaS Neutral

Cloud Foundry BOSH

Cloud Provider Interface(CPI)

vSphere: battle tested implement AWS: code complete

https://github.com/piston/openstack-bosh-cpi

OpenStack: testable release

Page 35: Cloud foundry elastic architecture and deploy based on openstack

35© Copyright 2012 EMC Corporation. All rights reserved.

Cloud Provider Interface

Stemcell– create_stemcell (image, cloud_properties)– delete_stemcell (stemcell)

VM– create_vm (agent_id, stemcell, resource_pool, networks,

disk_locality, env)– delete_vm (vm)– reboot_vm (vm)– configure_networks (vm, networks)

Disk– create_disk (size, vm_locality)– delete_disk (disk)– attach_disk (vm, disk)– detach_disk (vm, disk)

Page 36: Cloud foundry elastic architecture and deploy based on openstack

36© Copyright 2012 EMC Corporation. All rights reserved.

VM

Cloud Provider Interface (Cont.)For OpenStack

Stemcell– create_stemcell– delete_stemcell

VM– create_vm– delete_vm– reboot_vm– configure_networks

Disk– create_disk– delete_disk– attach_disk– detach_disk

Cloud Provider Interface

OpenStack Compute

OpenStackImage Service

VMVM

VMVM

Page 37: Cloud foundry elastic architecture and deploy based on openstack

37© Copyright 2012 EMC Corporation. All rights reserved.

Deploy PaaS with BOSH

IaaS

BOSH

CLI

Page 38: Cloud foundry elastic architecture and deploy based on openstack

38© Copyright 2012 EMC Corporation. All rights reserved.

Deployments

Release

Network

Resource pools

Jobs

Properties

Update concurrency

Compilation workers

Cloud properties

Page 39: Cloud foundry elastic architecture and deploy based on openstack

39© Copyright 2012 EMC Corporation. All rights reserved.

Deployments for CloudFoundyCloudfoundry.ymlname: cloudfoundry

release: name: cloudfoundry version: 89.1-dev

compilation: workers: 4 network: default cloud_properties: ram: 1024 disk: 2048 cpu: 2

update: canaries: 1 canary_watch_time: 3000-90000 update_watch_time: 3000-90000 max_in_flight: 2 max_errors: 1

Page 40: Cloud foundry elastic architecture and deploy based on openstack

40© Copyright 2012 EMC Corporation. All rights reserved.

Deployments for CloudFoundy (Cont.)Cloudfoundry.yml

networks: - name: default subnets: - static: - 192.168.2.50 - 192.168.2.89 range: 192.168.2.0/24 gateway: 192.168.2.1 dns: - 10.254.174.10 cloud_properties: name: PrivateNetwork- name: lb subnets: - static: - 192.168.2.90 - 192.168.2.99 range: 192.168.2.0/24 gateway: 192.168.2.1 dns: - 10.254.174.10 cloud_properties: name: PrivateNetwork

Page 41: Cloud foundry elastic architecture and deploy based on openstack

41© Copyright 2012 EMC Corporation. All rights reserved.

Deployments for CloudFoundy (Cont.)Cloudfoundry.ymlresource_pools:

- name: infrastructure network: default size: 29 stemcell: name: bosh-stemcell version: 0.6.2 cloud_properties: ram: 256 disk: 2048 cpu: 1 env: bosh: password:

Page 42: Cloud foundry elastic architecture and deploy based on openstack

42© Copyright 2012 EMC Corporation. All rights reserved.

Deployments for CloudFoundy (Cont.)Cloudfoundry.ymljobs:

- name: cloud_controller template: cloud_controller instances: 1 resource_pool: infrastructure networks: - name: default static_ips: - 192.168.2.60

- name: nats template: nats instances: 1 resource_pool: infrastructure networks: - name: default static_ips: - 192.168.2.52

Page 43: Cloud foundry elastic architecture and deploy based on openstack

43© Copyright 2012 EMC Corporation. All rights reserved.

Deployments(CloudFoundy)Cloudfoundry.yml

properties: domain: cflocal.com

env: {}

networks: apps: default management: default

nats: user: nats password: aaa3ij3122 address: 192.168.2.52 port: 4222

Page 44: Cloud foundry elastic architecture and deploy based on openstack

44© Copyright 2012 EMC Corporation. All rights reserved.

Use Case

bosh deployment cf.yml

git pull

bosh upload releaseRun

bosh deployDone!

bosh target cf

Bug report to Dev

Page 45: Cloud foundry elastic architecture and deploy based on openstack

45© Copyright 2012 EMC Corporation. All rights reserved.

Acknowledgments

VMware China R&D Center Network & Information Center, Shanghai Jiao Tong Univ.

CloudFoundry Communityhttp://www.cloudfoundry.org

Piston Communityhttps://github.com/piston/openstack-bosh-cpi

Page 46: Cloud foundry elastic architecture and deploy based on openstack

46© Copyright 2012 EMC Corporation. All rights reserved.

Q & A

Page 47: Cloud foundry elastic architecture and deploy based on openstack

47© Copyright 2012 EMC Corporation. All rights reserved.

Backup Slides

Page 48: Cloud foundry elastic architecture and deploy based on openstack

48© Copyright 2012 EMC Corporation. All rights reserved.

Management Path– vmc info

health manager

service

cloud controller

dea

app app

db

nats

router

Page 49: Cloud foundry elastic architecture and deploy based on openstack

49© Copyright 2012 EMC Corporation. All rights reserved.

Deployment Path– vmc push

health manager

service

cloud controller

dea

app app

db

nats

router

Page 50: Cloud foundry elastic architecture and deploy based on openstack

50© Copyright 2012 EMC Corporation. All rights reserved.

Data Path– Client Request

health manager

service

cloud controller

dea

app app

db

nats

router

Page 51: Cloud foundry elastic architecture and deploy based on openstack

51© Copyright 2012 EMC Corporation. All rights reserved.

Evolving Cloud Foundry - Kernel

stager

Package cache

uaa

v2

v2

v2

v2

caldecott

cloud controller

router

nats health manager

dea

app appservice

db

warden

v2

v2

v2

v2

Page 52: Cloud foundry elastic architecture and deploy based on openstack

52© Copyright 2012 EMC Corporation. All rights reserved.

How to write a custom service?

• Public API for advertising services and managing service instances

• Endpoints:– Cloud Controller– Service Gateway

(one per service)• REST / JSON / HTTP(S)

Cloud Controll

er

MySQL Gatewa

yMongoD

BGatewa

y

Page 53: Cloud foundry elastic architecture and deploy based on openstack

53© Copyright 2012 EMC Corporation. All rights reserved.

How to write a custom service? (Cont.)

Step 1: Service advertisement

Service Gateway -> Cloud Controller, two methods: POST /services/v1/offerings DELETE /services/v1/offerings/:label

:label => 'mysql-5.1':url => 'http://12.34.56.78:4321',:plans => ['free', 'bronze', 'silver', 'gold']:tags => ['mysql', 'mysql-5.1', 'database']:active => true,:description => 'MySQL database service':acls => { :wildcards => '*@vmware.com', :plans => { :bronze => { :users => ['[email protected]', '[email protected]'] } }}

Page 54: Cloud foundry elastic architecture and deploy based on openstack

54© Copyright 2012 EMC Corporation. All rights reserved.

How to write a custom service? (Cont.)

Step 2: Instance management

Cloud Controller -> Service Gateway, four methods:

Provision: POST /gateway/v1/configs Bind: POST /gateway/v1/configs/:id/handles Unbind: DELETE /gateway/v1/configs/:id/handles/:handle Unprovision: DELETE /gateway/v1/configs/:id

Page 55: Cloud foundry elastic architecture and deploy based on openstack

55© Copyright 2012 EMC Corporation. All rights reserved.

Private API App

Service Gateway

Service Node(eg, PostgreSQL)

Cloud Controller

vmc, STS

Developer

End Users

Avmc create-service postgresql

BPOST /services/v1/configurations

ECREATE DATABASE abc123CREATE ROLE pqr456

CPOST /gateway/v1/configurations

G{host: 12.34.56.78, port: 3456, db: abc123, user: pqr456}

HVCAP_SERVICES = [postgres: {host: 12.34.56.78, port: 3456, db: abc123, user: pqr456}]

I GET /foo/bar

SELECT *FROM FOO

host 12.34.56.78Service

(eg, PostgreSQL)

DNATS: provision

F{host: 12.34.56.78, port: 3456, db: abc123, user: pqr456}

Reference implementation

Page 56: Cloud foundry elastic architecture and deploy based on openstack

56© Copyright 2012 EMC Corporation. All rights reserved.

For legacy service – Service Broker

• Cloud Foundry users may already have existing services• Legacy database that hasn’t / won’t / can’t be moved

CloudFoundry.com• Services not yet supported

• Service Broker exposes an arbitrary legacy service instance to Cloud Foundry so that it can be used just like any other instance

Page 57: Cloud foundry elastic architecture and deploy based on openstack

57© Copyright 2012 EMC Corporation. All rights reserved.

For legacy service – Service Broker (Cont.)

AppService Gateway

Core Service(eg, PostgreSQL)

Cloud Controllervmc, STS

Service Broker

Brokered Service(eg, Oracle DB)

Service Broker Admin Tools

Bregister hrdb {host: 87.65.43.21, port: 6543, db: def123, user: stu456}

CPOST /broker/v1/configuration

EPOST /services/v1/configurations

Dvmc create-service hrdb

F

POST /gateway/v1/configurationsG{host: 87.65.43.21, port: 6543, db: def123, user: stu456}

HVCAP_SERVICES = [hrdb: {host: 87.65.43.21, port:6543, db: def123, user: stu456}]

IGET /foo/bar

SELECT *FROM FOO

host 12.34.56.78

ACREATE DATABASE def123CREATE USER stu456

Connection

Details

Page 58: Cloud foundry elastic architecture and deploy based on openstack