Loft Your Web Platform Into the Clouds with Immutable Infrastructure

Steven Merrill

© 2016 Phase2

OVERVIEW

4© 2016 Phase2

TOPICS COVEREDImmutability in software development

Immutable Infrastructure

Building immutable artifacts

Using immutable artifacts with AWS autoscaling

Using immutable artifacts with containers

Real-world examples

© 2016 Phase2

MUTABILITY

6© 2016 Phase2

im·mu·ta·ble

adjective

unable to be changed

7© 2016 Phase2

IMMUTABILITY IN SOFTWARE DEVELOPMENTFirst-class immutable variables in language runtimes

No need to synchronize read access in parallel systems

Immutable data stores

Elm’s time-travel debugging

Mercurial’s immutable history / write-ahead log

Immutable data structures in React.js

© 2016 Phase2

IMMUTABLE INFRASTRUCTURE

9© 2016 Phase2

IMMUTABLE INFRASTRUCTURE? WHAT?Continuation of infrastructure as code

Core tenet: build immutable artifacts and promote them

As in development, try to limit mutation of deployed infra

Unchanging artifacts (AMIs, containers) eliminate drift

10© 2016 Phase2

AN EXAMPLEA client had autoscaling troubles

Provisioned from scratch from a plain Ubuntu AMI

Problems:

Drift

Time to provision / autoscaling timeouts

Hard dependency on configuration management server

11© 2016 Phase2

COUNTERPOINTSMost servers may not have read-only disks

Container systems can allow read-only rootfs

Will you really disable SSH?

One theory: contaminate instances if you do

12© 2016 Phase2

WHY?Immutable infra practices enable autoscaling

Rallying cry: have cattle, not pets

It can allow for easier rollback

Possibility for canary or blue/green deployment

From here on out, we’ll discuss autoscaling readiness

© 2016 Phase2

FOUNDATIONS: BUILD PROCESSES

14© 2016 Phase2

BUILD THEORYA modern web app likely has several build steps

Pull dependencies down

Source compilation

CSS/JS compilation and assembly

Phase2 has grunt-drupal-tasks to automate this

Build from drush make, npm theme tasks

Has a package command to make a folder or tarball

15© 2016 Phase2

VENDORINGConsider vendoring your dependencies

Great discussion about this in “Who Owns Your Availability?” episode of Arrested DevOps

16© 2016 Phase2

BUILD TARGETSDeployment repository

Tarball

Machine image

Amazon AMI / Google Private images

Rkt / Docker tagged container image

JENKINS: FORCE FOR GOOD OR EVIL

18© 2016 Phase2

BUILD PROCESS EVOLUTIONAwful

Build in-place on each of n web heads

Better

Build once in workspace, rsync to docroots

Good

Build once, store artifact, capistrano-style deploy

Goal

Build artifact, build image around artifact

19© 2016 Phase2

BUILDING IMAGESAutoscaling-ready image builds have 2 components:

Tagged code releases

A combination of Linux packages on a filesystem

Linux vulnerabilities happen fairly frequently

Be ready to build for new code releases and package CVEs

© 2016 Phase2

AUTOSCALING

21© 2016 Phase2

WHY AUTOSCALE?Reduces stress

Automatic healing

Cost benefits

Performance benefits

Automatic scaling based on metrics

Manual adjustment up and down

© 2016 Phase2

AUTOSCALING DRUPAL IN AMAZON WEB SERVICES

23© 2016 Phase2

DRUPAL SITE COMPONENTSBig 3 components of a running Drupal site

Codebase on an app server

Relational database

Files directory

Optional

Object cache (?)

Varnish / CDN

24© 2016 Phase2

AUTOSCALING GROUPS IN AWSAn autoscaling group is attached to one or more ELBs

ELB health checks and timeouts control ASG actions

You use CloudWatch alarms to trigger autoscaling

Avg CPU > 70% for 5 min, avg CPU < 40% for 30 min

A LaunchConfig ties together an AMI and instance size

e.g. ami-beefbeef on c4.xlarge w/ 100 GB disks

One ASG is set to one LaunchConfig at a time

25© 2016 Phase2

AWS RECIPE FOR DRUPALJenkins instance building tarballs + AMIs

Autoscaling group with public Drupal instances

RDS for MySQL

S3 bucket for files directory via amazons3.module

ElastiCache for memcache via memcache.module

CloudFront for page caching / distribution

SES for email via smtp.module

26© 2016 Phase2

EXAMPLE AMI AUTOSCALING SETUPHave one build process to create a tarball / build artifact

Use Packer to start from a base AMI

File provisioner copies code up

Run Ansible to install packages

Create new LaunchConfig

Flip instances over one at a time

27© 2016 Phase2

ELIDED PACKER EXAMPLE{ "variables": { "aws_access_key" : "xxxxxxxxxxxxxxxxxxxx", "aws_secret_key" : "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "aws_instance_type" : "t2.micro", "east_aws_ami" : "ami-00000000", "west_aws_ami" : "ami-11111111", "aws_ssh_username" : "ec2-user", “east_aws_subnet" : "subnet-00000000”, “west_aws_subnet" : "subnet-11111111", "east_aws_vpc": "vpc-00000000", "west_aws_vpc": "vpc-11111111" },

28© 2016 Phase2

ELIDED PACKER EXAMPLE "provisioners": [{ "type": "file", "source": "/path/to/my/build.tgz", "destination": "/opt/build.tgz" }, { "type" : "shell", "inline": [ "/usr/bin/ansible-playbook -ilocal /opt/playbook.yml” ]}], ],

29© 2016 Phase2

ELIDED PACKER EXAMPLE "builders": [{ "type": "amazon-ebs", "name": "east-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-east-1", "source_ami": "{{user `east_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-00000000"], "subnet_id" : "{{ user `east_aws_subnet` }}", "vpc_id": "{{user `east_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_east" },

30© 2016 Phase2

ELIDED PACKER EXAMPLE { "type": "amazon-ebs", "name": "west-ami", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "us-west-2", "source_ami": "{{user `west_aws_ami`}}", "instance_type": "{{user `aws_instance_type`}}", "ssh_username": "{{user `aws_ssh_username`}}", "security_group_ids" : ["sg-11111111"], "subnet_id" : "{{ user `west_aws_subnet` }}", "vpc_id": "{{user `west_aws_vpc`}}", "ami_name": "autoscale_{{user `build_tag`}}_west" }]}

31© 2016 Phase2

S3 SHARP EDGESS3 files with amazons3/s3fs.module use memory_limit

You may be tempted to try the s3fs daemon

Instead, s3fs_cors/amazons3_cors modules

32© 2016 Phase2

AUTOSCALING QUIRKSSchema updates

Whitescreens on new/old as schema changes flip

Add double capacity, let ELB health checks do the work

Maintenance mode

Separate editorial instance

Build one image and switch behavior on ENV

© 2016 Phase2

CLIENT EXAMPLE

34© 2016 Phase2

MLS DIGITAL - MLSSOCCER.COM AND CLUBS21-site Drupal 7 platform

League and all club sites

Game day scaling patterns - huge traffic spikes

No autoscaling, separate docroots

Moved to multisite in prep

Build tarballs from 500 MB to 40 MB

Total opcache / APC savings of 1.6 GB

35© 2016 Phase2

MOVING MLS TO AUTOSCALINGOld setup provisioned from a base Ubuntu with Salt

Timeout problems

Moved to Packer AMI builds with all software & code

An AMI can come up if everything else is down

Systems of record for Drupal state are elsewhere

MySQL database - Amazon RDS

Memcache - Amazon ElastiCache

Files - Amazon S3 via s3fs.module

36© 2016 Phase2

SUCCESSFrom 4 pets to 2-5 cattle

Major hosting cost reduction

Later Drupal optimizations lowered instance size

A TYPICAL MONTH

DRUPAL OPTIMIZATIONS

AFTER TUNING

© 2016 Phase2

CONTAINERS

41© 2016 Phase2

POTENTIAL DOCKER BENEFITSIncreased density

Ability to run heterogenous workloads

Native CPU and disk performance

Resource limits

42© 2016 Phase2

DOCKER IN PRODUCTIONDocker on single hosts is easy

Docker in production requires a lot of coordination

Overlay networking / network security

Service discovery

Intelligent scheduling and capacity management

Routing / load balancing

Secrets management

Monitoring and performance management

43© 2016 Phase2

DOCKER BUILDSBuild considerations

Dockerfile vs configuration management

Dynamic templating

Build tools

docker build Packer

Configuration management system

44© 2016 Phase2

DOCKERDocker daemon allows for tagging

Ideally, use tags from canonical repo and as branches

p2/site:v1.0.0, p2/site:v1.1.0 p2/site:dev, p2/site:stage, p2/site:prod

Build script

docker build -t p2/site:v1.1.0 . docker tag p2/site:v1.1.0 p2/site:dev docker push p2/site:v1.1.0; docker push p2/site:dev

45© 2016 Phase2

KUBERNETESOpen-source container scheduler from Google

“Manage a cluster of Linux containers as a single system to accelerate Dev and simplify Ops.”

Based on Borg design principles

Large pool of contributors and upstream/downstream users

Red Hat (OpenShift 3)

Deis (Helm)

CoreOS (Tectonic)

46© 2016 Phase2

KUBERNETESRequires you to bring overlay networking

Features

Service discovery, load balancing, cluster DNS

Pod scaling and autoscaling, node draining

Resource-aware scheduling and capacity limits

REST API and CLI clients

Secrets management

YAML or JSON definition of k8s API objects

47© 2016 Phase2

KUBERNETES OBJECTSNodes

Labels

Pods

DaemonSets

Replication controllers

Services

Deployments

48© 2016 Phase2

KUBERNETES POD YAMLapiVersion: v1 kind: Pod metadata: name: lamp app: lamp

spec: containers: - name: web image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: "2000m”

49© 2016 Phase2

KUBERNETES RC YAMLapiVersion: v1 kind: ReplicationController metadata: name: lamp spec: replicas: 5 selector: app: lamp template: metadata: labels: app: lamp spec: containers:

- name: lamp image: phase2/apache24php55 resources: requests: memory: "512Mi" cpu: "1000m" limits: memory: "1024Mi" cpu: “2000m” ports: - containerPort: 8080 name: lamp protocol: TCP

50© 2016 Phase2

KUBERNETES SERVICE YAMLapiVersion: v1 kind: Service metadata: name: lamp spec: selector: app: lamp ports: - name: lamp port: 8080

© 2016 Phase2

SUMMARY

52© 2016 Phase2

IMMUTABLE INFRASTRUCTUREBuild immutable artifacts for deployment

Limit configuration drift in your infrastructure

Be ready to build new images for code releases or CVEs

Autoscaling can help with cost, stress, and self-healing

© 2016 Phase2

QUESTIONS?

So How Was It? - Tell Us What You ThinkEvaluate th is session - ht tps: / /events.drupal .org/neworleans2016/schedule

Thanks!