View
155
Download
0
Category
Tags:
Preview:
Citation preview
Speakers
• Tim Miller
• VP of Engineering
• Daniel Onorato
• Director of QA
• Robert Sulway
• Director of Development
Agenda
• Defining Continuous Delivery
• Motivations and Strategies
• Defining the Weekly Release
• Keys to Success
• Automating Is a “Must”
• The Dashboard
• (Surprising?) Benefits of Shorter Release Cycles
Your Roadmap to Application Agility
Code Build Integrate Test Release Deploy Operate
Agile Development
Continuous Integration
Continuous Delivery
Continuous Deployment
DevOps
Broker Cloud Services with RightScale
Self-Service Cloud Analytics
RightScale Cloud Portfolio Management
Cloud Management
Design
Virtualized
Environments
Public
Clouds
Other
Services
Private
Clouds
Automate
Multi-Cloud Orchestration & Governance
Operate Deploy Report Optimize
• Complex Cloud Management code base, lives in over 200 repositories
• Some code going back 7 years
• Running on ~600 servers in Production
• 7-8 week release cycles
• Developers, QA & Ops all set in their ways, process gets layered on top of
process
Landscape
6
• Get planned features to market more quickly
• Respond to urgent requests
• Reduce WIP
• Master Branch Instability – not getting tested during
regression
• Increase quality, decrease escalations
• Details of features “fresh in minds” on release
• Mindset: My feature is going into production soon
• Think about deployment!
Motivations
• Go Big
• Teams would need to innovate
• Required to shed the old and do things differently
• Add in some constraints to make things more interesting
• Can’t lower the quality bar in any way
• Can’t add human resources to achieve the goal
• Can’t reduce velocity
• If your are going to do it weekly, can’t have downtime during releases
Strategies
8
• Spoiler Alert: These became benefits
• The biggest challenge was getting past “We can’t..” • …because it would lower quality bar, QA always chasing the next
release
• ...roll out deployments in parallel, need to roll out serially
• …because it will take too much time/people for release process
The Challenges
What it is:
• A vehicle to release a collection of features on a weekly basis when
they are “done”
What it is not:
• A mandate to plan, design, implement, test and deploy a given feature
in a week’s time
Defining the Weekly Release
• Process
• Relentless adherence to our “contracts”
• Contract for the definition of “done” for features
• Contract for the “ready to ship” weekly features collection
• Dashboards (Jira/Travis) to see daily status of “ready to ship”
• Environments
• Branching strategy
• Automation to stand up functional test environments for given features
• Staging environment to test the weekly features collection as a whole
Keys to Success
Keys to Success...continued
12 12
• Automation
• Write once, run anywhere system level test automation (feature test,
stage integration, production)
• Matrix driven automation test execution for release deployment
validation in production with dashboard
• Automation to facilitate rolling production deployment
• Organization
• Embedded QA Engineering on scrum teams
• Tightly integrated Dev/Ops team
• For Libraries:
• Unit test coverage in (ci) for new feature, either measured or manually verified
to exist by QA
• For Apps:
• Unit test coverage running in (ci) for new feature(either measured or manually
verified to exist by QA)
• Functional test coverage in (ci) for new feature (either measured or manually
verified to exist by QA)
• Automated system test coverage run on a mini-rs and ready for merge into
master test collateral repo
• Note: we will require either the unit, functional or system test. The QA Eng/Devs will
determine required coverage at the time a feature is worked.
• Adequate end user documentation has been created.
• Product owner sign-off for feature
Definition of “done”
• For both Libraries and Apps:
• Downtime window is not required
• Compatible with a rolling system restart
• Documented migration instructions exist (if necessary)
• Documented deployment instructions exist (if necessary)
• Any required ServerTemplate updates must be coded and integrated on
a mini-rs
Definition of “done”
A weekly features collection is deemed “ready to ship” when:
• Verification the new feature(s) unit test(ci), functional test(ci) and/or
system level tests (staging) are passing on staging branch
• Verification of a full nightly regression run on staging env from staging
branch with no impact on existing features and pass/fail trends
• All regression issues are resolved
Definition of “ready to ship”
The RightScale DevOps Process
Mini-RightScale
Dev Environment
Staging
Environment Production
Environment
Deployment
Automation
Deployment
Automation
Pull
code &
test
Deployment
Automation
Operations Automation
Pull
code &
test
Pull
code &
release
Monitor
alarm
react
Develop
Templates
• Write once, run anywhere test collateral
• Sharing test frameworks and collateral across products
• Tying test frameworks into matrix based invocation system.
Streamlining QA Automation
• Automation frameworks and collateral support running a given
test in each of our possible test targets without updating test
collateral once written. (mini-rs, staging, production).
• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb
SETTINGS=mini-rs,nightly_settings SHARD=93
• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb
SETTINGS=staging,nightly_settings SHARD=93
• $ bundle exec rake SPEC=ui/functional/server_array_functional.rb
SETTINGS=us_3,nightly_settings SHARD=3
Write once, run anywhere test collateral
• Gemify (Ruby) test collateral repos to allow sharing among
various automation frameworks.
• Can speed up test execution in many cases
• Speeds up test suite implementation
• Example UI test using API test resources for setup
Sharing Test Collateral Repos
require 'rester'
require 'right_api_15’
describe "Verifies CRUD operations on the new ELB browser:", :crud => true do
before(:all) do
Session.login(:account_name => :test_account)
url = config(:LOCATION)
email = config(:EMAIL)
password = config(:PASSWORD)
RightApi15.default_login(login_info)
end
context "Setup" do
it "creates Network required for this test" do
@clouds.each do |cloud|
@api_created['network'] = NetworkManager::Network.create(:cloud => cloud, :cidr_block => '13.0.0.0/16’)
end
…
it "creates a load balancer using network/subnet with no name" do
@clouds.each do |cloud|
cloud_href = RightApi15::MetaData.get_cloud_href(:cloud_name => cloud)
network = @api_created['network'].select {|n| n.cloud_href == cloud_href}.first
subnet = @api_created['subnet'].select {|sn| sn.href.include?(cloud_href)}.first
security_group = @api_created['security_group'].select {|sg| sg.cloud_href == cloud_href}.first
LoadBalancer.create(:cloud => cloud,
:load_balancer_name => "#{@name}-api",
:VPC => { :vpc_option => network.resource_uid,
:subnets => subnet.resource_uid,
…
Benefits of shorter release cycles
34 34
• Master code branch very stable at all times
• Fewer regression bugs found in integration cycle
• Automation test coverage guaranteed to grow and exist (definition of
done)
• Forced us to keep pushing on automation and tying systems together to
lessen the human resource tax associated with each release (work in
progress)
• Forced us to engineer feature updates/migrations such that they can be
performed in a rolling fashion with no system maintain window requiring
down time.
• Releases happen in the middle of the our normal work day as opposed to
Friday nights / Weekends. Everyone much happier with this.
• Features are deployed very close in time to there development. Details
still fresh in our minds
35
Next Steps and Q&A
E-book:
Continuous Integration and Delivery in the Cloud: How We
Do It at RightScale: How RightScale Does It
www.rightscale.com/ci-cd-ebook
Recommended