Puppet Camp Sydney 2014 - Evolving Design Patterns in AWS

E V O LV I N G D E S I G N PAT T E R N S I N A W S

John Painter James DymondSolutions Architect Infrastructure Engineer

sourcedgroup.com.au

www.linkedin.com/in/painterj www.linkedin.com/in/jamesdymond

Sourced Group architect and deploy a globally distributed multi-master puppet solution for a large investment bank

February 2011

O P E R AT I N G S Y S T E M S

A P P L I C AT I O N C O N F I G U R AT I O N

Converge the infrastructure to one point of truth

A U D I T I N G G O V E R N A N C E S I M P L I F I E D C O N T R O L

Control and report on the point of truth

Scale that convergence to the entire enterprise

A U D I T I N G

G O V E R N A N C E

S I M P L I F I E D C O N T R O L

E U R O P E : 6 P M - 8 A MU S A : 6 P M - 8 A MA S I A : 6 P M - 8 A M

Global risk management, simplified releases

Sourced Group architect and deploy a globally distributed multi-master puppet solution for a large investment bank

February 2011

“Building a Multimaster Environment”

Greg Cockburn

PuppetCamp Sydney 2012

http://www.slideshare.net/gergnz/puppetcamp-sydney-2012-building-a-multimaster-environment

http://www.slideshare.net/gergnz/puppetcamp-sydney-2012-building-a-multimaster-environment

“Building a Multimaster Environment”

Greg Cockburn

PuppetCamp Sydney 2012

Sourced are the first to onboard an AsiaPac telecommunication company onto AWS, powered by Puppet Enterprise

February 2012

E V E R Y T H I N G I S P R O G R A M AT I C

S TA N D A R D I S AT I O N O F I N F R A S T R U C T U R E

Puppet CloudPack

V I R T U A L P R I VAT E C L O U D P R I VAT E I P S PA C E

C O M P U T E C O M P U T E

C O M P U T E C O M P U T E

C U S T O M E R

V P N

C O M P U T E C O M P U T E

C O M P U T E C O M P U T E

Public IP Public IP

Public IP Public IP

Public IP

Sourced are the first to onboard an AsiaPac telecommunication company onto AWS, powered by Puppet Enterprise

February 2012

“Using Puppet Enterprise Edition as Heterogeneous

Cloud Glue”

PuppetConf 2012

C L O U D PA C K U P L I F T

• Uplifted the existing Puppet CloudPack to support EC2 instances inside the AWS VPC

• Left Puppet Enterprise in the middle of the provisioning and lifecycle management

• Supported Multi-cloud provisioning

• http://www.youtube.com/watch?v=mwiBjJZWraA

S TA N D A R D I S AT I O N A N D C O N S O L I D AT I O N

“Using Puppet Enterprise Edition as Heterogeneous

Cloud Glue”

PuppetConf 2012

Sourced implement a masterless Enterprise Puppet solution for a financial services organisation

March 2013

C L O U D C O M P U T I N G I S F U N D A M E N TA L LY D I F F E R E N T

T R A N S I E N T C O M P U T E

V 1 . 0 V 1 . 0

myapp.com.au

V 1 . 0 V 1 . 0

myapp.com.au

V 2 . 0 V 2 . 0

myapp.com.au

V 2 . 0 V 2 . 0V 1 . 0 V 1 . 0

Infrastructure, deployment, and configuration have standardised

!

It was not just servers that were transient, it was entire application stacks

Enterprises want to leverage this agility to drive their application iteration harder and respond to

market conditions.

Cloud computing has significantly changed the application lifecycle

Weekly/Monthly -> Hourly

+

C L O U D F O R M AT I O N

Tasks for AWSV 2 . 0 V 2 . 0

• Cloudformation (AWS)

• Cloud Init (Puppet)

• CFN Init (AWS)

• Completely isolated application delivery

• Aware of the other AWS elements around them

– N I G E L K E R S T E N - P U P P E T C A M P S Y D N E Y 2 0 1 4

“18 months really sucks”

- E V E R Y S I N G L E P R O D U C T M A N A G E R . E V E R .

“18 months really sucks”

Business teams are willing to accept standardisation to gain agility and control over deployments

A P P L I C AT I O N

A P P L I C AT I O N

A P P L I C AT I O N

A P P A P P A P P

A P P A P P A P P

A P P A P P A P P

Standard Framework Lots of AWS Features

Application Configuration

Configuration Managent

Governance and Compliance

Configuration Managent

Governance and ComplianceApplication Configuration

Load

Time

Load

Time

Load

Time

$

$

$ $

$

$

G I T

I N S TA N C E I N S TA N C E

C D T O O L

AW S S 3

L O G G I N G

I N S TA N C EI N S TA N C E

S O L U T I O N 4

– N I G E L K E R S T E N - P U P P E T C A M P S Y D N E Y 2 0 1 4

“If you not already using PuppetDB, you should be.”

H Y B R I D C L O U D

H Y B R I D C L O U D

P U P P E T A W S A U T O S C A L I N G B R O K E R

I N S TA N C E I N S TA N C E I N S TA N C E

Simple Notification Service (SNS)

Autoscaling Group

SNS Topic

Simple Queue Service (SQS)

…

A U T O S C A L E B R O K E R

D E M O

S O L U T I O N 5

O P E R AT I N G S Y S T E M S

O P E R AT I N G S Y S T E M S

O P E R AT I N G S Y S T E M S

A P P L I C AT I O N

A P P L I C AT I O N

A P P L I C AT I O N

Framework

Rather than making the OS a good candidate for the application, we are now tailoring the applications to be a good citizen of a standardised compute platform.

H A R D WA R E

O P E R AT I N G S Y S T E M

R U N T I M E

A P P L I C AT I O N

H A R D WA R E

O P E R AT I N G S Y S T E M

R U N T I M E

A P P L I C AT I O N

O P E R AT I N G S Y S T E M S

O P E R AT I N G S Y S T E M S

O P E R AT I N G S Y S T E M S

A P P L I C AT I O N

A P P L I C AT I O N

A P P L I C AT I O N

Framework

- T H E N E W P R O D U C T M A N A G E R S

“1 week really sucks”

O P E N S H I F T

• Standardised runtime container

• Solution that accepts application artefacts and deploys, runs, scales, and manages their lifecycle

• Drives application portability

• OS and application containers are a blackbox

T H E F R A M E W O R K S S T I L L N E E D T O G O V E R N E D

A P P A P P A P P A P P

Puppet + Autoscale Broker

A M A Z O N W E B S E R V I C E S - A U T O S C A L E D

A P P A P P A P P A P P

Puppet + Autoscale Broker + Global Distribution

AW S - S Y D N E Y AW S - S I N G A P O R E

A P P A P P A P P A P P

Puppet + Autoscale Broker + Global Distribution + Multi Cloud

AW S - S Y D N E Y AW S - S I N G A P O R E< C L O U D > - S Y D N E Y < C L O U D > - S I N G A P O R E

A P P A P P A P P A P P

Puppet + Autoscale Broker + Global Distribution + Multi EVERYTHING!

AW S - S Y D N E Y AW S - S I N G A P O R E< C L O U D > - S Y D N E Y

John Painter James DymondSolutions Architect Infrastructure Engineer

sourcedgroup.com.au

www.linkedin.com/in/painterj www.linkedin.com/in/jamesdymond