Upload
amazon-web-services
View
494
Download
0
Embed Size (px)
Citation preview
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
James Saull, Solutions Architect – Amazon Web Services
Emerson Loureiro, Senior Software Engineer - Gilt Groupe
July 2016
DevOps on AWS:Deep Dive on Continuous Delivery
and the AWS Developer Tools
https://secure.flickr.com/photos/mgifford/4525333972
Why are we
here today?
Software moves faster today
Software creation and distribution is
easier and faster than ever:
• Startups can now take on giants with little to
no funding ahead of time
• Getting your software into the hands of
millions is a download away
• Your ability to move fast is paramount to your
ability to fight off disruption
What tools do you need to move fast?
Releasing software in this new software driven world
requires a number of things:
• Tools to manage the flow of your software development
release process
• Tools to properly test and inspect your code for defects
and potential issues
• Tools to deploy your applications
First, we need to
understand a little bit about
software release processes
https://www.flickr.com/photos/jurvetson/5201796697/
Release processes levels
Source Build Test Production
Continuous integration
Continuous delivery
Continuous deployment
Continuous Delivery Benefits
Improve developer
productivity
Find and address
bugs quickly
Deliver updates fasterAutomate the software
release process
A look back at
development at
Amazon..
https://secure.flickr.com/photos/pixelthing/15806918992/
2001
Development transformation at Amazon: 2001-2009
2009
monolithic
application + teams
microservices + 2 pizza teams
In 2009, we
ran a study to
find out where
inefficiencies
might still exist
We were just waiting.
WaitWrite Code WaitBuild
Code WaitDeploy to Test
Deploy to
Prod
Weeks
Mins Days Mins Days Mins Days Mins
Automated actions and transitions; from check-in to production
Development benefits:
• Faster
• Safer
• Simplification & standardization
• Visualization of the process
Pipelines
This has continued to work out really well:
In 2014:
• Thousands of service teams across Amazon
• Building microservices
• Practicing continuous delivery
• Many environments (staging, beta, production)
50 million deploys
Continuous delivery service for fast and
reliable application updates
Model and visualize your software release
process
Builds, tests, and deploys your code every time
there is a code change
Integrates with 3rd party tools and AWS
AWS CodePipeline
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
JavaApp
Elastic Beanstalk
PipelineStage
Action
Transition
CodePipeline
MyApplication
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
JavaApp
Elastic Beanstalk
NotifyDevelopers
Lambda
CodePipeline
MyApplication
Parallel actions
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
JavaApp
Elastic Beanstalk
NotifyDevelopers
Lambda
TestAPI
Runscope
CodePipeline
MyApplication
Sequential actions
8. Retrieve build artifact
EC2 instance
CodePipeline
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
JavaApp
Elastic Beanstalk
Source Artifact
S3
Build Artifact
S3
5. Get source artifact
1. Get Changes
6. Store build artifact
3. Poll for Job
4. Acknowledge Job
7. Put Success
9. Deploy build artifact
Elastic Beanstalk
Web container
Java App
MyApplication
We have a strong partner list, and it’s growing
Source Build Test Deploy
AWS service integrations
Source Invoke Logic Deploy
AWS Elastic BeanstalkAmazon S3
AWS CodeDeploy
AWS Lambda
AWS CodeCommit
DEMO
https://www.flickr.com/photos/seattlemunicipalarchives/12504672623/
Deploying your
applications
https://secure.flickr.com/photos/simononly/15386966677
Automates code deployments to any instance
Handles the complexity of updating your
applications
Avoid downtime during application deployment
Deploy to Amazon EC2 or on-premises
servers, in any language and on any operating
system
Integrates with 3rd party tools and AWS
AWS CodeDeploy
appspec.yml Exampleversion: 0.0os: linuxfiles:
- source: /destination: /var/www/html
permissions:- object: /var/www/html
pattern: “*.html”owner: rootgroup: rootmode: 755
hooks:ApplicationStop:
- location: scripts/deregister_from_elb.shBeforeInstall:
- location: scripts/install_dependencies.shApplicationStart:
- location: scripts/start_httpd.shValidateService:
- location: scripts/test_site.sh- location: scripts/register_with_elb.sh
appspec.yml Exampleversion: 0.0os: linuxfiles:
- source: /destination: /var/www/html
permissions:- object: /var/www/html
pattern: “*.html”owner: rootgroup: rootmode: 755
hooks:ApplicationStop:
- location: scripts/deregister_from_elb.shBeforeInstall:
- location: scripts/install_dependencies.shApplicationStart:
- location: scripts/start_httpd.shValidateService:
- location: scripts/test_site.sh- location: scripts/register_with_elb.sh
• Remove/add instance to ELB
• Install dependency packages
• Start Apache
• Confirm successful deploy
• More!
• Send application files to one
directory and configuration
files to another
• Set specific permissions on
specific directories & files
v2 v2 v2 v2 v2 v2
one at a time
half at a time
all at once
v2 v2 v2 v1 v1 v1
v2 v1 v1 v1 v1 v1 Agent Agent
Dev Deployment group
ORProd Deployment group
Agent
AgentAgent
Agent Agent
Agent
Choose Deployment Speed and Group
https://www.flickr.com/photos/spacex/16510243060/
Launching to
Production
Launching to Production
After you’ve built and tested your code and
hopefully gone through a few pre-production
deploys, its time for the real thing!
You’ll want think about:
• Impact to customers
• Impact to infrastructure
• Impact to business
How can we track these and communicate
deploys?
Extend AWS CodePipeline Using Custom Actions
Update tickets Provision resources
Update dashboards Send notifications Security scan
Mobile testing
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
Action
Custom Action
JavaApp
Elastic Beanstalk
MyApplication
CodePipeline
Deploy
JavaApp
Elastic Beanstalk Job Worker3. Perform Job
1. Poll for Job
2. Acknowledge Job
4. Put Success
Amazon
EC2
With custom actions,
the job worker drives the interaction
between AWS CodePipeline
and other applications or services
Source
Source
GitHub
Build
JenkinsOnEC2
Jenkins
Deploy
MyAction
AWS Lambda
JavaApp
Elastic Beanstalk
2. Perform Job
1. Invoke Lambda function
3. PutJobSuccessResult
MyApplication
CodePipeline With AWS Lambda-based actions,
AWS CodePipeline
drives the integration with Lambda,
which then connects with other
applications or services
AWS
Lambda
Micro-services
Deployment at
Gilt with AWS
Emerson Loureiro,
Senior Software
Engineer, Gilt
Gilt...
● Sales go live everyday at
noon
Micro-services
Move fast
Phased rollout
Rollback just as fast
Deployment tooling is important for us...
Can we do it with AWS?
1st Generation (DiscoDeploy)
● SBT pluginawsAccountName: gilt-xxxxxxx
instanceType: m3.xlarge
servicePort: 9000
instanceCount: 2
jvm:
- -Dnewrelic.config.file=/opt/gilt/service/conf/newrelic.yml
- -Dnewrelic.config.agent_enabled=true
- -Dlogger.resource=logback.production.xml
- -Dconfig.resource=application.production.conf
- -Xms12G
- -Xmx12G
- -XX:NewSize=6G
- -XX:MaxNewSize=6G
...
1st Generation (DiscoDeploy)
Docker
Registry
ASG
EC2 EC2 EC2 EC2
ELBsbt deploy-setup
sbt release-docker
sbt deploy-service
sbt deploy-scale
● Good to begin with
1st Generation (DiscoDeploy)
Docker
Registry
ASG
EC2 EC2 EC2 EC2
ELB
2nd Generation (Ionblaster)
● Based on CloudFormation
[
{
"service": "SERVICE",
"domain": "SERVICE.gilt.com",
"instances": { "min": 3, "max": 3 },
"type": "t2.small"
"run": {
"image": "gilt/SERVICE",
"cmd": ["-Dhttp.port=9000"],
"args": ["-p", "9000:9000"]
},
"health_path": "_internal_/healthcheck"
...
}
]
● Based on CloudFormation
sbt docker:publish
ionblaster new SERVICE 0.0.1
2nd Generation (Ionblaster)
0.0.1
DNS
2nd Generation (Ionblaster)
● Based on CloudFormation
sbt docker:publish
ionblaster new SERVICE 0.0.2
ionblaster traffic SERVICE 0.0.2 10
0.0.1
DNS
0.0.2
2nd Generation (Ionblaster)
● Based on CloudFormation
ionblaster traffic SERVICE 0.0.2 100
0.0.1
DNS
0.0.2
2nd Generation (Ionblaster)
● Based on CloudFormation
ionblaster kill SERVICE 0.0.1DNS
0.0.2
● Traffic migration could be better
● No logical grouping among stacks
2nd Generation (Ionblaster)
DNS
0.0.2
● Based on Elastic Beanstalk
ionroller release SERVICE 0.0.1
3rd Generation (Ionroller)
0.0.1
DNS
Application
● Based on Elastic Beanstalk
ionroller release SERVICE 0.0.2
3rd Generation (Ionroller)
0.0.1
DNS
0.0.2
Application
● Based on Elastic Beanstalk
ionroller release SERVICE 0.0.2
3rd Generation (Ionroller)
0.0.1
DNS
0.0.2
Application
● Based on Elastic Beanstalk
ionroller release SERVICE 0.0.2
3rd Generation (Ionroller)
0.0.1
DNS
0.0.2
Application
● Based on Elastic Beanstalk
ionroller release SERVICE 0.0.2
3rd Generation (Ionroller) DNS
0.0.2
Application
What we didn’t like about those...
“We could solve this now, or, just wait six
months, and Amazon will provide a solution.”
Andrey Kartashov, Distinguished Engineer,
Gilt
● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR)
make create stack
Development Canary
4th Generation (Mobile Service Tools - MST)
Production
● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR)
make publish
make deploy-development
make deploy-production-canary
make deploy-production
Development Canary
4th Generation (Mobile Service Tools - MST)
Production
● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR)
make deploy-production
Development Canary
4th Generation (Mobile Service Tools - MST)
Production
● Based on CodeDeploy + CloudFormation + AWS CLI (+ ECR)
make deploy-production
Development Canary
4th Generation (Mobile Service Tools - MST)
Production
● Mostly AWS tooling
● Quick deployments
Development Canary
4th Generation (Mobile Service Tools - MST)
Production
● Boilerplate scripts
● Docker registry problems
● No cross-account support
● Shared CloudFormation template
Development Canary
4th Generation (Mobile Service Tools - MST)
Production
● Boilerplate scripts
● Docker registry problems
● No cross-account support
● Shared CloudFormation template
4th Generation (Mobile Service Tools - MST)
Base
template
● Based on CodeDeploy + CloudFormation + S3
nova stack create ENVIRONMENT
Development Canary
5th Generation (Nova)
Production
Environment
● Based on CodeDeploy + CloudFormation + S3
nova deploy ENVIRONMENT development
nova deploy ENVIRONMENT canary
nova deploy ENVIRONMENT production
Development Canary
5th Generation (Nova)
Production
Environment
● Versioned CloudFormation templates
5th Generation (Nova)
V2V1
Happiness
FIN, ACK
We’ve seen a quick run through today of the benefits of
continuous delivery on our software release process:
• Continuous integration (build/test) helps shrink our
feedback loop greatly
• We can get our software out in front of our users much
more rapidly
• By moving faster we can actually ensure better quality
• CodePipeline allows for integration with almost any
service or tool you can think of!
• Plus visualization of what’s going on!
Resources to learn more:• DevOps: https://aws.amazon.com/devops/
• Continuous integration: https://aws.amazon.com/devops/continuous-integration/
• Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/
• CodePipeline
• https://aws.amazon.com/codepipeline/
• https://aws.amazon.com/documentation/codepipeline/
• CodeDeploy
• https://aws.amazon.com/codedeploy/
• https://aws.amazon.com/documentation/codedeploy/
• https://github.com/awslabs/aws-codedeploy-samples
• Code Services Starter Kit: http://bit.ly/AWSCodeStarterKit
But wait, there’s more!
Please remember to rate this
session under My Agenda on
awssummit.london
Thank you!