Beyond 1001 Dedicated Data Service Instances

Introduction

The Challenge

Given: Application platform based on

Cloud Foundry to serve thousands of apps

Application Runtime

App App App App App App App App App App

• Many platform users - who don’t

know each other

• Different app langs & frameworks

• 100% on-demand self-service: no

involvement of the platform

operator necessary.

• Instant scalablity & self-healing

$> cf push myapp

Easy Deployment

Runtime abstraction

ExecutionStagingJava

Droplet

Bildpack

Droplet

ExecutionStaging

Bildpack

DropletRuby Ruby

Droplet

Droplet DropletJava

Droplet Ruby

Droplet

In front of the Cloud

Foundry Runtime

all droplets are equal

Droplet Droplet

$START_CMD $START_CMD

DropletContainer

Droplet

$START_CMD

Something you can

execute in a container.

Something you can

$START_CMD

Abstraction enables further

assumptions & automation

Scaling Apps

Assuming this to be our status quo.

Cloud Foundry Runtime

Instance#1

Instance#2

$> cf scale -i 3 app#1

App Scalability

Scaling Apps

Instance#1

Instance#3

Instance#1

Instance#2

Two additional instances have been created.

App Self-Healing

Everything is healthy.

Instance#1

Instance#3

Instance#1

Instance#2

App Self-Healing

App #1 Instance #2 is failing.

Instance#1

Instance

Instance#3

Instance#1

Instance#2

App Self-Healing

App #1 Instance #2 - gone temporarily.

Instance#1

Instance#3

Instance#1

Instance#2

App Self-Healing

App #1 Instance #2 re-created.

Instance#1

Instance#3

Instance#1

Instance#2

How does the paradise for

Backing Services look

Missing: A solution to serve thousands of

data services

Data ServicesApplication Runtime

The Mission

Providing a growing number of data services with full lifecycle automation of

thousands of data service instances across a wide range of infrastructures

a9s PostgreSQL a9s MongoDB a9s RabbitMQ

a9s Elasticsearch a9s Redis a9s LogMe

thousands of data service instances across a wide range of infrastructures.

in both:

in both:public

in both:public and on-premise clouds

and integrate well with multiple platforms

Requirements

Portability

Robustness

Scalability

On-demand self-service

Performance Maintainability

Flexibility

Extensibility

Usability

Manageability

Security

Multi-tenancy

Portability

Scalability

Production-Readiness

Design

How to build it?

Data Service Provisioning API Automation Middleware Data Service Automation

Open Service Broker, a new industry standard

for data service provisioning.

• Google

• Pivotal

• IBM

• RedHat

• Fujitsu

• SAP

Open Service Broker API Supporters

• Cloud Foundry

• OpenShift

• Kubernetes

• More to come

Supporting Platforms

• Cloud Foundry

• OpenShift

• Kubernetes

• More to come

Supporting Platforms

• Get Service CatalogGET /v2/catalog

• Provision Service - Create Service InstancePUT /v2/service_instances/:id

• Bind ServicePUT /v2/service_instances/:instance_id/

service_bindings/:id

• Unbind ServiceDELETE /v2/service_instances/:instance_id/

• Unprovision ServiceDELETE /v2/service_instances/:id

http://docs.cloudfoundry.org/services/api.html#api-overview

HTTP Verb Action

Service Catalog GET /v2/catalog

Deliver meta data about the data service.

Create Service Instance PUT /v2/service_instances/:id

Provision a VM, install and configure a data service VMs

/ Cluster representing a service instance.

Create Service Binding PUT /v2/service_instances/:instance_id/

Create a data service user and return credentials representing a service

binding.

Delete Service Binding DELETE /v2/

service_instances/:instance_id/service_bindings/:id

Remove credentials associated with the service binding.

Delete Service Instance DELETE /v2/service_instances/:id

Destroy the VMs and data associated with the service instance.

Data Service Provisioning API Automation Middleware Data Service

The Open Service Broker API does not define what a service instance is.

Applying the design pattern: On-Demand Provisioning

of Dedicated Data Service Instances

Result

Using a Service Broker with Cloud Foundry

$> cf create-service

$> cf create-service mongodb single-small my-single-mongo-1

Easy Deployment

my-single-mongo-1

MongoDB

$> cf create-service mongodb cluster-small my-3node-mongo-

cluster-2

Easy Deployment

Newly created service instance

my-3node-mongo-cluster-2

MongoDB

my-single-mongo-1

MongoDB

Technical Challenges

State is handled differently in each backing service.

Operational model will be different. Replication, failure detection, failover.

State is handled differently in each backing service.

Operational model will be different. Replication, failure detection, failover.

The data service automation will be different.

Where to store state?

App Self-Healing

Everything is healthy.

Instance#1

Instance#3

Instance#1

Instance#2

App Self-Healing

App #1 Instance #2 is failing.

Instance#1

Instance

Instance#3

Instance#1

Instance#2

App Self-Healing

App #1 Instance #2 - gone temporarily.

Instance#1

Instance#3

Instance#1

Instance#2

App Self-Healing

App #1 Instance #2 re-created.

Instance#1

Instance#3

Instance#1

Instance#2

App self-healing is easy because there is NO STATE.

How to store state but still being able to perform self-healing?

Store state on a remotely attached block device = persistent disk.

Infrastructure as a Service (IaaS), e.g. OpenStack

VIRTUAL DATACENTER

Router

STORAGE

Storage Node Storage Node Storage Node

HDD HDD

IaaS API

VIRTUAL DATACENTER

Router

STORAGE

HDD HDD

Storage Volume

Operating System

VIRTUAL MACHINE

IaaS API

Persistent disk has a file system. Filesystems may fail

⇒ replication / clustering & backupsare still very important.

The data lifecycle has been decoupled from the VM lifecycle ⇒ The VM becomes disposable.

Storing state ✅

What needs to be automated?

Data Service Instance Lifecycle

Provision a data service server Install data service software

Configure data service software Consume data service with apps

Debug data service issues Update data service version

Update operating system Backup & recover data

Scale out data service VM(s) Destroy data service & DB VM(s)

Lifecycle of a Data Service Instance1

Can you do that x * 1000 times?

You either automate it or delegate it to the app developer.

Automation

„BOSH let’s you orchestrate the

lifecycle of

large-scale deployments of stateful distributed systems

to infrastructure.“

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH API

BOSH CLI $> bosh deploy

IaaS API

BOSH CPI

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH API

IaaS API

BOSH CPI

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH API

IaaS API

BOSH CPI

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH API

Storage Volume

Operating System

BOSH Agent

VIRTUAL MACHINE

IaaS API

BOSH CPI

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH APIOperating

System

BOSH Agent

VIRTUAL MACHINE

IaaS API

BOSH CPI

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH APIOperating

System

BOSH Agent

VIRTUAL MACHINE

IaaS API

PostgreSQL

BOSH CPI

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH APIOperating

System

BOSH Agent

VIRTUAL MACHINE

IaaS API

PostgreSQL

BOSH CPI

VIRTUAL DATACENTER

VIRTUAL MACHINE

Router

STORAGE

HDD HDD

BOSH APIOperating

System

BOSH Agent

VIRTUAL MACHINE

IaaS API

BOSH Agent

VIRTUAL MACHINE

PostgreSQL Operating System

BOSH Agent

VIRTUAL MACHINE

Cloud Controller

Operating System

BOSH Agent

VIRTUAL MACHINE

BOSH CPI

BOSH Automation

BOSH Releases

contain the automation

A BOSH-Deployment depends on 1 .. *

Stemcells

A BOSH-Deployment is described by a

Release & Manifest

A Release describe 1 .. * Jobs

A Release contains 1 .. * Package

A BOSH Deployment’s

settings are contained in a

Manifest

Infrastructure settings

settings are contained in the

Cloud Config

BOSH makes your

deployments…

Infrastructure

Independent

• A BOSH release contains the

main-automation (software packages, how to run

processes)

• BOSH releases can be re-used

on every* infrastructure

Automate once,

deploy everywhere.

VMware

BOSH CLI

OpenStack

VMware

BOSH CLI

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

OpenStack

BOSH CLI

VMware AWS OpenStack

BOSH BOSH BOSH

BOSH CLI

$> bosh target http://bosh-on.aws.com

BOSH BOSH BOSH

BOSH CLI

BOSH BOSH BOSH

BOSH CLI

$> bosh deploy

BOSH BOSH BOSH

BOSH CLI

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

Some Service / App

BOSH Agent

VIRTUAL MACHINE

BOSH BOSH BOSH

• Switch deployment between

clouds

• Keep the same release

• Use a stemcell specific to the

new cloud

• Adapt the cloud config

Operating System

Independent

A BOSH release does

not depend on the OS

The only dependency to

the OS is a BOSH

stemcell

Operating System Image

BOSH Agent

VIRTUAL MACHINE

BOSH Agent

VIRTUAL MACHINE

BOSH Agent

VIRTUAL MACHINE

} OS image +BOSH agent= Stemcell

Ubuntu Stemcell

BOSH Agent

VIRTUAL MACHINE

• Changing the OS of a BOSH

deployed system

• Change the stemcell

• Change the manifest

Scalable

Horizontal Scaling

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

Some Service

BOSH Agent

VIRTUAL MACHINE

• Scaling-out a BOSH deployed

system

• Use the same stemcell

• Change the manifest

Vertical Scaling

4GB RAM 1 vCPU

10GB persistent disk

BOSH Agent

VIRTUAL MACHINE 4 GB RAM, 1 vCPU

10 GB Persistent Disk

PostgreSQL

BOSH Agent

PostgreSQL

BOSH Agent

PostgreSQL

BOSH Agent

PostgreSQL

VIRTUAL MACHINE 8 GB RAM, 2 vCPUs

BOSH Agent

PostgreSQL

BOSH Agent

PostgreSQL

BOSH Deployments are Predictable

• Src code is compiled in a freshly

created VMs

• VMs always contain exact the

software, specified in the release

• No left-overs of prior

deployments as new VMs are

BOSH Deployments are

Repeatable

Executing a specific BOSH deployment always leads to exact same deployed system.

Monitored &Self-Healing

Self-healingprocess failures

BOSH Installation BOSH Managed Infrastructure Resources

Some process

BOSH Agent

Process Monitor

Yet another process

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

BOSH Agent Process Monitor

BOSH Registry

BLOB Store

Process MonitorProcess Monitor

Some process

BOSH Agent

Process Monitor

Yet another process

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

BOSH Registry

BLOB Store

Some other process

Self-healingprocess monitor failures

Some process

BOSH Agent

Process Monitor

Yet another process

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

BOSH Registry

BLOB Store

Some other process

Some process

BOSH Agent

Yet another process

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

BOSH Registry

BLOB Store

Some other process

Some process

BOSH Agent

Yet another process

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

BOSH Registry

BLOB Store

Some other process

Some process

BOSH Agent

Yet another process

BOSH Agent

BOSH Health Monitor

BOSH Director

NATS Message Bus

Some other process

BOSH Registry

BLOB Store

Some other process

Process Monitor

Self-healingVM failures

Some process

BOSH Agent

Process Monitor

Yet another process

NATS Message Bus

Some other process

Process Monitor

BOSH Registry

BLOB Store

🎖 BOSH Director

BOSH Health Monitor❤

Some process

BOSH Agent

Process Monitor

Yet another process

NATS Message Bus

Process Monitor

BOSH Registry

BLOB Store

🎖 BOSH Director

Some other process

Self-healingBOSH Agent failures

BOSH Managed Infrastructure Resources

Yet another process

Process Monitor

BOSH Installation

Some process

BOSH Agent

Process Monitor

BOSH Agent

NATS Message Bus

Some other process

BOSH Registry

BLOB Store

🎖 BOSH Director

BOSH Agent

BOSH Managed Infrastructure Resources

BOSH Installation

Some process

BOSH Agent

Process Monitor

NATS Message Bus

Some other process

BOSH Registry

BLOB Store

🎖 BOSH Director

BOSH Agent

Yet another process

Process MonitorBOSH Agent

Reference Architecture

a9s Deployer

Templates Deployments

a9s Service Broker

my-3node-postgres-cluster-2

Postgresql

my-single-postgres-1

Postgresql

Middleware Adapter

Cloud Foundry Adapter

a9s PostgreSQL SPI

Service InstanceService Instance

Postgresql

Service Instance

Cloud ControllerCF Clientcreate service

create service

create deployment from template xy with attributes {…}

deploy release abc & deployment manifest xyz

Execute deployments

create

service specific credentials

create binding

PortabilityBOSH:

Multi-infrastructure support

Open Service Broker API:

Multi-platform support

Scalability On-demand provisioning of dedicated service instances

BOSH: Scale existing service instances

vertically, solo & clustered

instances

Production Readyness

Dedicated data service instances / Strong instance

isolation

BOSH: Self-healing, clustered service

instances, backup & restore

Open Service Broker API, On-demand provisioning, on-

demand updatesOn-demand backup &

restore

a9s PostgreSQL a9s MongoDB a9s RabbitMQ

a9s Elasticsearch a9s Redis a9s LogMe

Operations

Continuous Data Service Delivery

Testa9s

Release

Delivering Data Service Patches

BuildUpstream

Release

Platform #1

Platform #2

Platform #n

Updating the Data Services

Update Data Service Instances

a9s PostgreSQLOpen Source

PostgreSQL Building new Data Service Releases

Common Maintenance Tasks to performed at Scale

• Create VM

• Install and start data services

Create Service Instance

• Destroy old VM

• Create new VM

• Mount old persistent disk

• Create and mount new persistent disk

• Copy data

• Optional: reintegrate into the cluster

Vertical Scale Service Instance

• Destroy old VM

• Create new VM based on new

Stemcell (\w new OS version)

• Attach persistent disk

OS Update

Ultimate Question

Can you handle more than 1001 Data Service Instances?

Excerpt from our perf tests:

Provisioning 1001 instances in sequence over a greater period of time does not

expose any significant bottleneck.

For large highly-frequented platforms the amount of simultaneous deployments

may become relevant.

Data Service Instances

BOSH Queue Time

Avg. time to provision VM

Total time needed to

create instances

250 14 min 6:57 min 21 min

500 29 min 7:01 36 min

750 46 min 7:13 53 min

… … … …

Optimization Task: Manage BOSH queueing time to an acceptable

level.

Scaling BOSH is key to deal with simultaneous provisioning.

Sum Up

• Full lifecycle automation is feasible

• Open Service Broker API, a new

standard

• Choosing the right automation

technology is key, e.g. BOSH

• CI/CD based dev and ops are essential

Sum Up

Questions?

A. Shared VM cluster

B. Dedicated containers

C. Dedicated VMs / VM clusters

Common Data Service Design Patterns

Scaling a shared VM cluster

MongoDB Cluster

MongoDB VM #1 MongoDB VM #2 MongoDB VM #3

MongoDB Cluster

Service Instance #1 = database #1

Service Instance #… = database #…

Service Instance # n-max = database #n-max

MongoDB Cluster

Low costs per service instance

Simple Service Broker Logic

create service → create a database

create service binding → create a database user

Weak Isolation!

Structural Limitation!

What to do when the shared cluster is full?

MongoDB Cluster

MongoDB Cluster #1

MongoDB Cluster

MongoDB Cluster #1

MongoDB Cluster #2

Service Instance #n+1

Service Instance #…

Service Instance # 2*n-max

Service Broker Logic

Simple

Complex

Fragmentation

Caused by frequent creation and / deletion of

service instances

Fragmentation

MongoDB VM #2 MongoDB VM #3

MongoDB Cluster #1

MongoDB Cluster #2

service instances

Fragmentation

MongoDB Cluster #1

MongoDB Cluster #2

Placement Problem

MongoDB Cluster #1

MongoDB Cluster #2

Placement Problem

MongoDB Cluster #1

MongoDB Cluster #2

New Service Instance?

Strategy to place new service instances is required

and may require data service specific logic.

Placement Problem

MongoDB Cluster #1

MongoDB Cluster #2

New Service Instance

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

A unbalanced set of clusters wastes

infrastructure resources.

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

A cluster rebalance freeing infrastructure

resources would desirable.

Cluster Rebalancing

MongoDB Cluster #1

Shared Cluster Conclusion

• Scalability issued can be addressed

• Isolation issues are heavily data-service

specific > A generic solution is not

possible.

Scaling Dedicated Containers

Better Isolation

2 VMs across 2 AZs

PosgreSQL Cell

Docker host VM #1 Docker host VM #2

2 VMs across 2 AZs

PosgreSQL Cell

Service Instance #1 = 2 Docker containers

+ 2 PostgreSQL processes

+ 2 PostgreSQL databases asynchronously replicated

2 VMs across 2 AZs

PosgreSQL Cell

How to scale?

2 VMs across 2 AZs

PosgreSQL Cell

2 VMs across 2 AZs

PosgreSQL Cell

What to do when the Cell/Cluster is full?

2 VMs across 2 AZs

PosgreSQL Cell #1

2 VMs across 2 AZs

PosgreSQL Cell #2

Same Service Broker Challenge

New Challenge: How to add Cell-VMs on-demand?

On-Demand VM provisioning is unavoidable.

Why not delegate most challenges?

On-Demand Dedicated VMs and Clusters

Architecture

a9s Deployer

a9s Service Broker

my-3node-mongodb-cluster-2

MongoDB

my-single-mongodb-1

MongoDB

Middleware Adapter

a9s MongoDB SPI

MongoDB

Service Instance

create service

Execute deployments

create

create binding

Let BOSH do the VM orchestration!

Let the infrastructure solve the placement and fragmentation challenge!

Shared Data Services

• Single VM or single

cluster of VMs

• Single PostgreSQL

server or single

PostgreSQL cluster

• Isolation limited to

PostreSQL multi-

tenancy capabilities

Shared PostgreSQL Cluster > Bad idea

Postgresql

PostgreSQL Cluster

Single PostgreSQL Server

Service Instance

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Shared PostgreSQL = SPOF

PostgreSQL Cluster

Postgresql

VM#3Service Instance

Service Instance

PostgreSQL Cluster

Service Instance

App App App App

App App App App App

App App App App App App

App App

App App App

App App App App App App App

App App App App App

App App App App

App App App App App App App App

App App App App App App App App App

App App App

Service Instance

PostgreSQL Cluster

Service Instance

App App App App

App App App App App

App App

App App App

Service Instance

PostgreSQL Cluster

Service Instance

App App App App

App App App App App

App App

App App App

Service Instance

Your shared PostgreSQL cluster goes down, all your PostgreSQL database instances go down.

Beware of bad neighborhood

PostgreSQL Cluster

Postgresql

Service Instance

PostgreSQL Cluster

Postgresql

Service Instance

👹 👹

PostgreSQL Cluster

Postgresql

Service Instance

👹 👹

PostgreSQL Cluster

Postgresql

Service Instance

Service Instance ⚠

👹 👹

PostgreSQL Cluster

Postgresql

Service Instance

Service Instance ⚠ Service Instance ⚠

👹 👹

Shared clusters are vulnerable to bad neighbors

Dedicated

• Service instance =

dedicated VM or

dedicated cluster of

• Uses infrastructure

isolation to enable

multi-tenancy support

Dedicated PostgreSQL instances > Good idea

Postgresql

Service Instance

and / or

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

PostgreSQL failures are contained. Only one service instance affected.

Bad neighborhood protection with dedicated service instances

PostgreSQL Cluster = Service Instance #2

Postgresql

VM#3Service Instance Service Instance Service Instance

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

Service Instance ⚠ Service Instance ⚠ Service Instance ⚠

👹 👹

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

Infrastructure isolation

Dedicated clusters isolate bad neighbors

Automation

Robustness

Automation

Self-Healing

Robustness

Automation

Self-Healing

Scalability

Robustness

Automation

Self-Healing

Scalability

Robustness

VM vs. Container

Resource TypePre-provisioned

vs. On-demand-provisioning

Provisioning

Single replica vs.

Data redundancy / Multiple replicas

Resurrection Failover vs.

Standby-Failover

Failover Strategy

Data Redundancy

Automation Technology

BOSH vs. Chef vs. Puppet

Service InstancesShared Dedicated

Infrastructure Reliability

Perfect. HA VMs. No SPOFs. Never fails. VMs cost more than a

Design to fail. Fails from time to time.

Saves money.

Seconds, minutes, hours?

Desired time to repair Service instance availability

Service broker availability

Availability

Service broker performance (ops/s) Service instance performance

Time to provision service instance.

PerformanceAccessibilityRemote log-in to service instances.

ConfigurabilityAdapt to local network

and security policies. Integrate existing

infrastructure.

SecurityNetwork security.

Encryption.

TransparencyAccessing metrics and logs.

OperabilityEasyness to operate and

maintain.

1. Define a service instance! 2. Define total # service instances! 3. Define # service instance CRUD ops / min!

Common Data Service

Design Patterns

A. Shared VM cluster

B. Dedicated containers

C. Dedicated VMs / VM clusters

Common Data Service Design Patterns

MongoDB Cluster

Low costs per service instance

Simple Service Broker Logic

create service → create a database

create service binding → create a database user

Weak Isolation!

What to do when the shared cluster is full?

MongoDB Cluster

MongoDB Cluster #1

MongoDB Cluster

MongoDB Cluster #1

MongoDB Cluster #2

Simple

Complex

Fragmentation

service instances

Fragmentation

MongoDB Cluster #1

MongoDB Cluster #2

service instances

Fragmentation

MongoDB Cluster #1

MongoDB Cluster #2

Placement Problem

MongoDB Cluster #1

MongoDB Cluster #2

Placement Problem

MongoDB Cluster #1

MongoDB Cluster #2

New Service Instance?

Strategy to place new service instances is required

and may require data service specific logic.

Placement Problem

MongoDB Cluster #1

MongoDB Cluster #2

New Service Instance

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

A unbalanced set of clusters wastes

infrastructure resources.

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

Cluster Rebalancing

MongoDB Cluster #1

MongoDB Cluster #2

A cluster rebalance freeing infrastructure

resources would desirable.

Cluster Rebalancing

MongoDB Cluster #1

Shared Cluster Conclusion

• Scalability issued can be addressed

• Isolation issues are heavily data-service

specific > A generic solution is not

possible.

Better Isolation

2 VMs across 2 AZs

PosgreSQL Cell

2 VMs across 2 AZs

PosgreSQL Cell

2 VMs across 2 AZs

PosgreSQL Cell

How to scale?

2 VMs across 2 AZs

PosgreSQL Cell

2 VMs across 2 AZs

PosgreSQL Cell

What to do when the Cell/Cluster is full?

2 VMs across 2 AZs

PosgreSQL Cell #1

2 VMs across 2 AZs

PosgreSQL Cell #2

Same Service Broker Challenge

New Challenge: How to add Cell-VMs on-demand?

On-Demand VM provisioning is unavoidable.

Why not delegate most challenges?

On-Demand Dedicated VMs and Clusters

Architecture

a9s Deployer

a9s Service Broker

MongoDB

my-single-mongodb-1

MongoDB

Middleware Adapter

a9s MongoDB SPI

MongoDB

Service Instance

create service

Execute deployments

create

create binding

Let BOSH do the VM orchestration!

Let the infrastructure solve the placement and fragmentation challenge!

Shared Services Instances

• Single VM or single

cluster of VMs

• Single PostgreSQL

server or single

PostgreSQL cluster

• Isolation limited to

PostreSQL multi-

tenancy capabilities

Shared PostgreSQL Cluster > Bad idea

Postgresql

PostgreSQL Cluster

Single PostgreSQL Server

Service Instance

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Service Instance 2

Service Instance 3

Service Instance 1

Shared PostgreSQL = SPOF

PostgreSQL Cluster

Postgresql

Service Instance

PostgreSQL Cluster

Service Instance

App App App App

App App App App App

App App

App App App

App App App App App

App App App App

App App App App App App App App

App App App App App App App App App

App App App

Service Instance

PostgreSQL Cluster

Service Instance

App App App App

App App App App App

App App

App App App

Service Instance

PostgreSQL Cluster

Service Instance

App App App App

App App App App App

App App

App App App

Service Instance

Your shared PostgreSQL cluster goes down, all your PostgreSQL database instances go down.

Beware of bad neighborhood

PostgreSQL Cluster

Postgresql

Service Instance

PostgreSQL Cluster

Postgresql

Service Instance

👹 👹

PostgreSQL Cluster

Postgresql

Service Instance

👹 👹

PostgreSQL Cluster

Postgresql

Service Instance

👹 👹

PostgreSQL Cluster

Postgresql

Service Instance

👹 👹

Shared clusters are vulnerable to bad neighbors

Dedicated Service Instances

• Service instance =

dedicated VM or

dedicated cluster of

• Uses infrastructure

isolation to enable

multi-tenancy support

Dedicated PostgreSQL instances > Good idea

Postgresql

Service Instance

and / or

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM#1

Postgresql VM#2

Postgresql VM#3

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

Postgresql

Service Instance

Postgresql VM

Service Instance

Postgresql VM

Service Instance

PostgreSQL failures are contained. Only one service instance affected.

Bad neighborhood protection with dedicated service instances

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

👹 👹

Postgresql

Infrastructure isolation

Dedicated clusters isolate bad neighbors

Pre- vs. On-Demand Provisioning

Pre-Provisioned

Pre-provisoned service instances

Postgresql

Service Broker

$> cf create-service postgresql single-small my-single-postgres-1

Easy Deployment

Pre-provisioned service instances

Postgresql

Service Broker

$> cf create-service postgresql cluster-small my-3node-postgres-

cluster-2

Easy Deployment

Pre-provisioned service instances

Postgresql

Service Broker

On-Demand-Provisioned

On-demand provisioned service instances

Service Broker

Some Automation

Service Broker

Some Automation

Postgresql

Service Instance

Service Broker

Some Automation

Postgresql

Service Instance

Service Broker

Some Automation

Pre-provisoned service instances

Postgresql

Service Broker

Some Automation

Beyond 1001 Dedicated Data Service Instances - … Service Instances. Introduction. The Challenge....

Documents

Cloud Foundry and OpenStack - events.static.linuxfound.org · Communication Hub IRC (#openstack-dev) Mailing lists ... ③Stage application ④Deploy application ⑤Manage application

How to Build a High Performance Application Using Cloud Foundry and Redis (Cloud Foundry Summit 2014)

How to deploy your application to heroku and cloud foundry

Application of OBO Foundry Principles in GO

Application of Various Equipment Used In Foundry for ... · Application of Various Equipment Used In Foundry for Prevention of Pollution - A Review Raunak Gupta1, M. Kannan2, Raghuraj

SQL Server Best Practices on Tintri – Stand-Alone Instances · included. SQL Server 2012 introduced application level HA with AlwaysOn Capabilities including Server Failover Instances

Hargreaves Foundry Drainage...The Hargreaves Foundry group of companies includes Hargreaves Foundry Drainage, Hargreaves Foundry and Hargreaves Lock Gates. Hargreaves Foundry is a

Foundry Management System Desktop Application

syntax cloud application persistent instances distributed objects … · 2011-01-26 · A cloud application is a java serial program that interacts with persistent instances of distributed

INFORMATION SECURITY & PRIVACY WHITEPAPER...Web/ Application Instances Database Instances 2 Customer 3 The Ivalua cloud features a “multi-instance” architecture that delivers logical

Application description 07/2018 Integration of S7-1500 Package … · 2018-07-06 · No. Description 6. Creating ASO instances Create the ASO instances. These instances are always

Frank Konieczny AF Chief Technology Officer · 2017-01-14 · –Multiple application instances due to geographic performance limitations or COOP –Localized application instances

Application and multi instances

Cloud Foundry User Survey · Cloud Foundry ser urvey April 2018 2 Cloud Foundry is the prevailing open source cloud application platform used by half of the Fortune 500 and a third

State Toolkit for Developing Beneficial ... - Foundry Gate state Toolkit for Developi… · concrete products. In more limited instances, foundry sand is being used in manufactured

TIBCO BusinessWorks Container Edition Samples · Cloud Foundry has commands for creating and updating user-provided service instances (cups). Once created, user-provided service instances

Application Services Design - EdgeX Foundry

A Windows Azure application runs multiple instances of each role A Windows Azure application behaves correctly when

Foundry 1. Foundry - Cluster Observatoryclusterobservatory.in/report/foundry.pdf · 1 Foundry 1. Foundry Foundry is a factory which produces metal castings from either ferrous or

Rheumatism Foundry Workersoem.bmj.com/content/oemed/23/1/42.full.pdf · Rheumatism in Foundry Workers ... Foundry Expected Foundry Expected Foundry ... significantly less frequent