Rise of the MachinesContinuous Delivery at SEEK
CO PRESENTED BY:
CD @ SEEK“To the cloud”
Scorched Earth Deployments
● Create a pristine environment for every release.● No need to migrate state of environment.
○ ∴ Configuration Management tools not required.
● State should not be changed at runtime.○ ∴ Immutable Server
● Blue/Green deployment strategy:○ Monitor health during deployment.○ Auto-rollback if new environment is unhealthy.
Automate all the things!
● Delivery team is responsible for the automation of:○ Server installation○ Infrastructure creation○ Deployment
● Automation is impossible without team managed infrastructure.
Infrastructure as Code● SEEK use AWS CloudFormation to automate their infrastructure.● Operations are responsible for the network-level infrastructure,
including:○ VPC○ Routing○ Users and Roles○ Shared services
● Delivery team is responsible for infrastructure within the VPC, such as:○ Elastic Load Balancer○ Security Group○ Auto Scaling Group○ Launch Configuration○ Instance Profile
Server automation
● Provisioning Linux:○ SSH ⇒ Easy!
● Provisioning Windows:○ WinRM ⇒ Hard!○ AWS makes bootstrapping easier:
■ EC2 User Data■ cfn-init
○ Packer Windows Plugins can be used to create Windows Amazon Machine Images
● Combination of Packer and CloudFormation can be used regardless of target platform
Packaging● Applications should be self-installable.● Application installation should be separate from server automation.● Installation should be independent of Continuous Deployment tool.● Native package manager is ideal solution:
○ Linux: apt-get or yum○ Windows:
■ One-Get (available in next version of Windows)■ Chocolatey (available now!)
● Package manager is platform independent pattern○ <package-manager> install <package-name>
● Windows Coding Horror:○ Interacting with Windows resources is not trivial, custom
PowerShell modules or PowerShell DSC Resources make this easier.
Pre-Baked Images
● Install a package on a base image.○ But… don’t include configuration.
● Then, create an image (AMI).● AMI is our environment independent deployable
artifact.
● Future is containerised...○ Docker is the next evolution of our approach.
Late-binding of context
● Keep your artifact environment agnostic:○ Frameworks want you to apply configuration at build or
package-time.○ Deploy tools want to apply configuration at install-time.○ When deploying to the cloud, you should apply configuration at
launch-time.● When AMI is launched it doesn’t know where it is:
○ Need to inject environment-specific context at launch-time.○ How do you do this?
Environment VariablesA platform independent way of specifying environment-specific config that’s been around since 1979● Use environment variables for:
○ DB connection parameters.○ Location of service endpoints.
● Don’t use them for:○ Altering runtime behaviour.
● Overriding .NET configuration with environment variables:○ .NET 4.5.x: write your own Configuration Provider.○ .NET 4.6: works out of the box.○ Azure + .NET 4.5.x: works out of the box.
● Service discovery is on the horizon:○ SEEK is investigating.○ Externalised config is first step.
Confidence to Release
● Put enough steps and checkpoints in pipeline to get enough confidence to release.
● Too many checks will result in a slow pipeline and kill the benefits of Continuous Delivery.
● Take things away until you cry.○ Then... compensate some other way○ For example, consumer-driven contract testing makes sure we
honour API contracts without the need to integrate.
Ship It!
● Distinction between Deployment and release:○ Deploy: create Blue environment and deploy application.○ Release: cut-over from Green to Blue environment.
● Smoke test in Blue environment before release using synthetic transaction.
● Same smoke test can be used to monitor health after release.● Choose to enable manual release if change set is risky and additional
manual testing is required.
Towards Polyglot Architecture
● Continuous Delivery + Microservices gives teams the ability to choose the right stack for the job.
● For SEEK, CD is opening up the possibility migrating legacy apps from Windows to Linux.○ Who knows, in the future, running Windows containers will be a
viable alternative.● SEEKs approach to Continuous Delivery provides better logical
separation automation steps.● Time to first commit still a problem and this will only get worse with
the proliferation of different stacks and tools.
Machine Factories
Manual Development Environments
■ Are a bad experience for your developers.■ Are time consuming and therefore costly.■ Make it difficult for devs to move between projects.■ Are rarely accurately reconstructed.■ Encourages manual configuration rather than the more
reliable and repeatable practice of Infrastructure-as-code.
Can lead to environments that have all of the parts, but not quite in the right order.
“It works on my box”
■ Sign of environmental divergence.■ The Butterfly effect: Minor differences in environment
configuration can produce obscure bugs in Prod.■ This issue is compounded when deployments carried out by
another person (e.g. Ops).
Dev vs Production Parity
■ Environment Parity = Confidence.■ Overcompensate lack of parity with more, possibly manual tests.■ Environment Parity…
○ enables continuous delivery .○ requires diligence and automation for accurate replication.○ encourages infrastructure-as-code.
Wouldn’t it be great if we...
■ Could use declarative formats for setup automation, to minimize time and cost for new developers joining the project.
■ Had a clean contract with the underlying operating system, offering maximum portability between execution environments.
Wouldn’t it be great if we...
■ Built apps suitable for deployment on modern cloud platforms, obviating the need for servers and systems admin.
■ Could minimize divergence between development and production, enabling continuous deployment.
■ Could scale up without significant changes to tooling, architecture, or development practices.
The “12 Factor App”
■ These 5 principles are what loosely describe the 12 factor App, a guideline for modern SaaS products.
■ 12 Factor Apps, Factor #10: Dev/prod parity
■ “Keep development, staging, and production as similar as possible”.
■ Vagrant can help us solve the environment parity issue.
Vagrant, Docker Compose & FriendsOnward to immutable development environments
Scorched Earth Dev Environments
■ Dev automation typically come in 2 flavours:○ Use of CM Tools like Puppet, Chef (Boxen, Babushka)○ Virtual Machine Runners (Vagrant, Docker Compose)
■ CM Tools fraught with challenges:○ Not isolated○ Hard to automate across OS/Platforms○ Doesn’t force developers to automate
What is Vagrant?
“A tool to help create and configure lightweight, reproducible, and portable development environments.”
■ A replacement for manually configured boxes. ○ Enforces infrastructure-as-code.
■ A machine runner (typically a VM or a Container)■ Separates authoring environment from development runtime■ Isolated & repeatable
What is Vagrant?
The following is a logical view of how Vagrant interacts with your authoring environment:
■ Vagrant has spun up Virtualbox VM with the name ‘Web’, and is running it on a Mac OSX desktop
■ Code is synchronised from /c/proj/x -> /vagrant of the running environment
What is Vagrant?
■ The Web application spins up a server using the code in /vagrant, which binds to a private network on the VM on port 80
■ Port 8080 is exposed to the outside ‘world’ (in this case, a Mac OSX desktop) which forwards requests to port 80 in the running VM
■ The Developer is free to author in whichever IDE of her choosing
The TAO of Vagrant
■ vagrant up to get a fully working dev runtime.■ Development occurs in preferred IDE: Visual Studio,
Sublime, ViM, with your terminal, your shortcuts - etc.■ vagrant destroy && vagrant up will destroy the
machine and create a new, clean, isolated environment.■ Workflow repeatable and transferrable to other projects.
Developing on EcosystemsFrom monoliths to ecosystems
From Monolith to Microservices
■ Applications are usually more interesting than a single UI, they are usually connected to things.○ They form part of a functioning ecosystem.
■ Vagrant good for Monoliths and ecosystems.■ Explosion of microservices highlights this need.
From Monolith to Microservices
Commentary on Architecture:
■ Multiple components, in disparate technologies■ Distributed system■ Layered architecture■ = complexity
Increased complexity increases the number of things that can go wrong. We can combat this with parity and local simulation.
From Monolith to Microservices
■ Vagrant can replicate ecosystems locally, while still having the benefits of isolation, repeatability, immutability etc.
■ Running ecosystems locally gives us flexibility.■ Vagrant can simulate or test a number of scenarios, for
example to:○ Model distributed systems behaviour○ Perform disaster testing; outages, network partitions ...○ Replace components independently○ Test OS/DB upgrades○ Run entire production replica ecosystems, locally
Local Ecosystem Setup
The following is a Logical View of this fictional ecosystem with Web and API applications exposed to the ‘World’ and a database hidden ‘behind the firewall’:
Local Ecosystem Setup
■ Note that we are running multiple virtualisation technologies here as if they were all running locally
■ Vagrant will spin up 3 local VMs: ○ Web and API applications exposed to the ‘World’ and a
database box hidden ‘behind the firewall’:■ The 4 VMs are able to communicate through a private
network on their native ports (80, 80 and 27017 respectively).
Native to Docker and is fast, but some caveats:
■ Unless on linux, networking is not automatically exposed to the host machine.
■ You need to ensure that your apps are compiled/built within the Docker containers - otherwise you’ll still need to bootstrap your dev box anyway!
■ Hybrid VM/Docker setups not supported.
Docker Compose
Machine Image Pipelines with PackerWhere environments are born
Cattle in Production, Pets elsewhere
■ Dev == Prod - but what about our build/other servers?■ We don't want cattle in production and pets elsewhere.■ Quality starts before coding begins
○ Must think about environments we develop, build and deploy to.
■ How do we ensure other environments are configured identically?-> Immutable servers everywhere!
The Machine Factory
■ How do we go about creating and managing all of these machine images?
■ At SEEK, we needed a way to generate all of these machine images in a sensible, standardised and organised way.
■ Enter: The Machine Factory:○ “A process for generating machine images for use in
different contexts such as development, test, build, prod and so on.”
■ So what does this look like?
[Insert diagram here]
Windows Server 2012
AWS AMI
Machine Factory
■ Assembly-line style manufacturing:○ Input images (AMI, ISO, Docker Image etc.)○ Apply Build Steps with shared infrastructure code, using
CM tools, shell scripts etc.○ Output Images (AMI, ISO, Docker etc.)
■ Images put in stasis for their destined environment/context where they will be reanimated.
■ CM tools as a code smell:○ Your application might be too big. For this step,
lightweight options usually suffice.
Tuning the assembling line: Pipelines
■ Booting machines and provisioning from scratch is slow.■ 2 main strategies: Intermediate images & enrichment.■ Create intermediate images to shortcut future builds:
○ Install common infra such as log forwarders, statsd, AV, patches/updates etc.
○ Engineering wide (Platform), team wide (Base) or specific Application images.
Tuning the assembling line: Enrichment
■ Prod should have only the runtime dependencies○ Better Security.○ Better Performance.
■ Dev will have the most. This can be expressed as:
Prod ⊊ Build Server ⊊ Dev
■ Enrich machines from left to right in your pipeline, feeding previous image into next step.
Machine Factory @ SEEKWalking the walk
Applying CD to Talent Search
■ Talentsearch previously part of .NET monolith.■ Talent Search is a web-based tool that Recruiters use to
connect with candidates, and was previously part of the monolith Warner discussed earlier.
Carving out the relevant bits
■ Isolate scope, cut out vertical slices from the monolith.■ Enables us to Focus our automation efforts.■ Broadly this is achieved using 2 approaches:
○ Mocking out downstream APIs for local dev.○ Consumer Driven Contracts, using Pact.
■ These give us isolation and speed.
Automating Windows
■ Windows tooling support in past year is much improved.■ AWS & Vagrant all have first class support.■ Seek have contributed a number of plugins for Vagrant to
make Windows development easier,■ ...a number of useful chocolatey packages,■ ...and a large suite of DSC resources to assist configuring
Windows machines.■ And then there is Packer...
Automating Windows: Packer
“A tool for creating identical machine images for multiple platforms from a single source configuration”
■ Packer is the core of our Machine Factory.○ It will boot a machine from an image such as an ISO○ Configure it with scripts or CM tools. ○ Export it into multiple formats - such as a Vagrant box,
Docker image or an AMI.■ Uses a simple JSON DSL to easily comprehend and visualise
the steps applied in the manufacturing process.
Automating Windows: Packer
■ Packer uses SSH, which is very fragile on Windows.■ SEEK have helped create a port of Packer to communicate
natively to Windows over WinRM.■ The result is Packer Community, a set of plugins for Packer
that make Windows machine automation a breeze.■ Using Packer we have automated:
○ our local dev boxes in Vagrant,○ our build/test/prod servers in AWS,○ ...across Linux, Windows Server and Mac OSX.
#winning
■ Dev env < 10 mins for .NET API + UI■ Dev -> Build -> Prod from Factory
○ Vagrant: Vbox, Parallels, Mac + Windows
○ AWS■ Commit to Release in 7 mins.■ Proven ‘vertical slices’ strategy.
Bonus: Challenges & Lessons on WindowsTips on effective use of Vagrant, Packer and DSC.
Vagrant
■ Other authoring environment dependencies = anti-pattern. ○ Installing VM provider (e.g. Virtualbox) + Vagrant should
be the goal○ Consider scripting the install (DSC, BoxStarter,
Powershell scripts etc.) if you can’t avoid.■ Shared folders and IIS Applications don’t play nicely
○ Use rsync synced folder type instead■ When provisioners get too big/slow, move into a Packer
script - vagrant up should be fast and reliable
Vagrant
■ Always set $VAGRANT_HOME on multi-partitioned machines due to disk space restrictions (n+1 space requirements)
■ Distributing Images○ Use Versions + Atlas, if possible○ S3 also good option
■ GUI Testing (e.g. Protractor tests)○ Consider having a GUI fallback (at the cost of slightly
bigger images)○ X-Forwarding on *nix hosts not ideal
Vagrant
■ Cross-OS concerns do occasionally come into play:○ NPM for example has issues writing to a linux
networked file system where the underlying host is Windows.
■ Domain access can be problematic and increases complexity - avoid if possible.○ Use Vagrant Windows Domain plugin if you can’t.
■ Use multiple NICs with private_network if application requires multiple SSL certificates.
Vagrant
Box cleanup:
■ Clean up after yourself and remove any binaries/installers/cruft not required for day-to-day use
■ Defrag and compress virtual disk images if possible■ Matt Wrock has a number of good tips on how to cut it right
down.
Vagrant
Build .NET apps in Vagrant without Visual Studio:■ Install MS Build Tools and VS Agents■ Package reference assemblies from VS installation into
guest machine.
Install the remote debugger and port forward on 4018 to debug from Visual Studio
DSC
■ Expect pain/effort, it’s in its infancy and is immature. ■ Error messages can be very unhelpful/confusing, and you
should be prepared to get comfortable with Windows debugging in the event viewer and the CLI.
■ IMHO, DSC is still a better choice than the *nix open source alternatives such as Chef/Ansible (although support is improving).
DSC
■ Use the Vagrant and the DSC Plugin to iterate on a box; Vagrant allows you to test your DSC configurations in a clean environment and test any potential issues with conditions/ordering, without polluting your host environment
■ Prefer the feature rich SEEK-Jobs/DSC community resources over the Powershell team (experimental) xModules
■ Avoid the Script resource - It is rarely a good idea to use it. Write your own custom resource instead.
Packer■ Don't try to do it all in one go. Build complexity■ When building a new image, create scripts, run them
manually and then automate once you’re comfortable they’re working as expected.
■ Iteration times can be slow, particularly when booting and installing an OS○ Create intermediate boxes to build off to avoid that
overhead.■ Using Vagrant to test those scripts is often a good starting
point.
Packer
■ Sysprep Windows machines so licenses don’t expire.■ If you’re creating a Windows image, Use Packer Community
(only if you can’t use Linux! :)).■ Don’t forget to disable Windows Updates (at least for dev
machines).■ Windows boxes often need restarting, use the restart-
windows Packer Community provisioner to do so safely.
Packer
■ Build Packer images in pipelines - feed output of Packer runs into the start of another Packer run.
■ For example, use the Virtualbox ISO builder to boot and install a Windows machine with updates applied.
■ That box can now be fed into the Virtualbox OVF builder which runs the application specific provisioners (typically this is the more error prone step).
Thank you
@SEEK_Geek
CO PRESENTED BY:
@terrencebenade
@dius_au @warnergodfrey@matthewfellows
Resources
■ Vagrant (http://vagrantup.com)■ The TAO of Vagrant (http://mitchellh.com/the-tao-of-vagrant)■ Vagrant DSC Plugin (https://github.com/mefellows/vagrant-dsc/)■ Vagrant Windows Domain Plugin (https://github.com/SEEK-
Jobs/vagrant-windows-domain)■ DSC (http://blogs.technet.
com/b/privatecloud/archive/2013/08/30/introducing-powershell-desired-state-configuration-dsc.aspx)
■ Read the free DSC eBook (https://onedrive.live.com/?cid=7f868aa697b937fe&id=7F868AA697B937FE!107) for a small but comprehensive guide to DSC
Resources
■ SEEK-Jobs/DSC resource (https://github.com/SEEK-Jobs/DSC/)■ Setting up a server for use with DSC (see packer templates https:
//github.com/mefellows/packer-community-templates)■ Packer (http://packer.io)■ Packer Windows Plugins (aka Packer for Windows https://github.
com/packer-community/packer-windows-plugins)■ Sample Windows Packer Templates (https://github.
com/mefellows/packer-community-templates)■ Legacy Windows Packer Templates and provisioning scripts (Uses
OpenSSH, however there is plenty of inspiration to be found - https://github.com/joefitzgerald/packer-windows)