Upload
ethan-ram
View
1.217
Download
0
Embed Size (px)
Citation preview
Making the Agile Leap to Continuous Deployment
Ethan Ram / 15 Feb 2014V1.1
© All rights reservedWant to use slides from this presentation? Ask for permission: [email protected]
Ethan Ram - Bio• Chief architect of Videobet• Leading Videobet’s R&D and OPs
teams Agility efforts• As VP R&D of GameGround
transformed the R&D to run in Continuous Deployment environment
• Hacker in soul and a Windows Internals hobbyist
Find me here: • [email protected]• LinkedIn• The Null Terminator
blog
2/15/2014 © All rights reserved 2
Videobet• A Playtech division • Innovative Gaming
Platform Provider– Established 9 years ago– 150 employees in Tallinn
and Kiev– Operate in 9 countries– 60% of UK market
• R&D– Front-end teams (C++)– Server teams (Java)– Game development teams
2/15/2014 © All rights reserved 3
• Why Continuous Deployment• Setting the goals • Deployment technics• Development technics• Making the leap• Some unexpected results • Q&A
Agenda
2/15/2014 © All rights reserved 4
Agile Manifesto Principle #1
Our highest priority is to satisfy
the customer through early and
continuous delivery of valuable
software.
At SlideShare, instead of big launches, we release code all the time... it reduces risk.
Since we are putting out small changes at a time, if it breaks we know what it is and can
roll back.Rashmi
We have been doing continuous deployment at Plex since our inception in 1995. We
formalized the process with our own “deploy” button in 2003, and hit the 25k deployment
mark less than two years thereafter. We currently average between 50-100
deployments per day to our massive online ERP system.Jerry Foster
Goal: Continuous Deployment
2/15/2014 © All rights reserved
WordPress.com - we’ve averaged about 16 product releases a day, every day for the last
four and a half years… Everyone in our company has access to a deploy button that releases the latest checked in code to about 400 production servers in our web tier in less
than 30 seconds (across 3 data centers). Toni Schneider
One of the exciting things… about Digg is continuous deployment - when developers fix a bug or add a new feature, there's no need
to wait for a scheduled release… the turnaround time for a change drops
dramatically.Andrew Bayer
5
Wix “From Jan ‘13 to Jun ’13 we did 1500 Deployments, 470 A/B Tests,
200 Feature Toggles”Yoav Avrahami
Flickr "10+ Deploys Per Day" Steve Conover
2/15/2014 © All rights reserved 6
Major ReleaseMajor Release
No. I Can’t Do That • “We’re starting this campaign next week and
I need the product to do X”• “We need to stop showing this feature to
non-US customers”• “I think this idea would work. When can we
start A/B testing live?”• “I need to know if users are clicking on this
link and if they see this error message”• “Forget about the other features. We have to
go live by the end of the month!”
2/15/2014 © All rights reserved 7
2/15/2014 © All rights reserved
Continuous Deployment: Continuous Deployment: Yes We Can!Yes We Can!
8
How to Streamline Deployment?
• Speed-up Dev-ops• Lower risk of deploying something broken• Streamline feature implementation
2/15/2014 © All rights reserved 9
???
Monitoring & Rollback• Must haves – real-time– Monitor that features are working,
not just servers– Monitor the business results– Visualize and put it on the big
screen– One-click rollback
• Rollback: the ultimate risk mitigation– Prepare to fail!
• Will the rollback work?– Rollback drills
2/15/2014 © All rights reserved 10
Deployment & Servers SetupGo from – 8 hrs downtime on upgrade• Physical machines• IP based network topology• Layered servers: web-apps,
backend, cache, db• Feature servers• Each team has a different
deployment setup and scripts• Complex database schema upgrades
To – 8 mins deploy, no downtime• Virtual machines• DNS based network topology• All servers on each VM
– DB too!!!• Same VM setup on all
environments – With different config– With different data-set
• One deployment script run on dev machine >> production– Target tag and environment name
as input• Versioned mini schema
migrations
2/15/2014 © All rights reserved 11
IT’S EASY!
DB Versioning With Flyway DB• Mini schema migrations kept with source
code– Applying migrations added between v1 to v2 is
easy– Roll-back of migrations supported built-in
• Each developer has her own DB • Merging team-branch migrations is even
simpler than merging code• Integrative with Maven/Ant/scripts and Java
APIs– Server can upgrade/lower DB schema by itself
• Data for unit-testing can be added and removed on the fly
2/15/2014 © All rights reserved 12
Flyway DB makes
you Agile!
Staging on Production• Get real users to use new version ASAP– Call it Beta, Early Availability, Version Trial, Test-
bed
• Deploy new version to part of the servers and direct it traffic of part of the user-base– Deploy to a subset of servers– Run manual tests & automated tests– Direct some traffic to new version– Monitor! Measure!– Deploy to the rest after some time
• Not only for consumer-based internet products!
2/15/2014 © All rights reserved 13
• Integration Test Automation– Focus first on covering main data flows–More main features, less details
• Part of the Continuous Integration– Build > Unit-test > Integration Tests– All tests must pass to allow build promotion
• Run the same tests on all environments– Developers’ & production environments too
• Don’t give-up on manual testing• ‘Test frenzy’ as version signs-off– One hour session, structured, lead by QA– ALL employees join. The secretary too.
Quality Assurance
2/15/2014 © All rights reserved 14
…not so easy…
2/15/2014 © All rights reserved
BACKLOG SELECTED IN PROGRESS RESOLVED VERIFIED TESTBED DONEBacklog
• Bugs• Improvements.• Daily
prioritization
Selected
•Feature owner assigned•Task Breakdown•User stories•Design/gfx•Test plan
In Progress
• Coding• Unit testing• Integration
testing
Resolved
• Need verification
• Drop build to QA
• Automated integration testing
• Manual testing
Verified
• Verified• Ready to be
installed on TestBed
Testbed
• Optional step: install on TestBed
• Regression Testing
Done
• Running on production
• Done / Not relevant
R&D Kanban Board: CD Workflow
• Enforce WIP limits• One owner for each column• Feature owner responsible for feature end-to-end…
• QA plan and test completion (no need for bugs!)• Unit testing and integration testing• Use DBA/HTML/FLASH and QA resources as needed• Test it on testbed/production
16
Developing on Trunk
2/15/2014 © All rights reserved 17
Use feature toggle flag Use feature toggle flag 2/15/2014 © All rights reserved 18
• Use feature toggle flags• Each dev has a full system - update, compile, run it all, test-it• Write backwards compatible code
– Assume some servers/services do not have the new feature/code yet
– DB fields may be missing– Mark older APIs with [deprecated/obsolete]
• Dynamic Configuration– Central config, dynamically loaded– Plan for no app/system restarts
• Schema changes are added to code as DB versioning scripts• Larger features that require a branch are rare!• Education: Do not break the trunk!
Developing on the Trunk
2/15/2014 © All rights reserved 19
Thorn Free?!?
Product: From Revolution to Evolution
2/15/2014 © All rights reserved 20
V2.2
V2.3 / R403
R417
Leaping into CD• Initiative must come from high mgmt.
– It’s a large investment… and risky• It’s not all or nothing >>>
– First milestone: weekly release– Set a deadline and commit: “we start
releasing once a week right after the next major release in September”
• It’s a project to deliver– Multi-function task-force: devs, QA,
deployment– Full time!– Plan a lot – detailed task list etc.
• Training to all teams: process, tools• Decide on a CD weekly schedule
– Tuesday afternoon is best time for a release• Make results visible early!
– And celebrate success!
2/15/2014 © All rights reserved 21
money time
The first CD release felt a bit like this…The first CD release felt a bit like this…..Just Go For It!..Just Go For It!
2/15/2014 © All rights reserved 22
Some Unexpected Results• … the sky did not fall on us • Excellent process transparency
– R&D stopped fighting with Product Mgmt.– Team leaders stopped blaming each other– Devs thought it’s much more fun– Everyone got more involved
• We managed to attract much better engineers• R&D managers thought its awesome!
2/15/2014 © All rights reserved 23
Q&A
Want to use slides from this presentation? Ask for permission: [email protected]
2/15/2014 © All rights reserved 24
Thanks!
Thanks for contributing!Yoav Avrahami @ WixMike Mcgarr @ Excella