Upload
devops-israel
View
3.334
Download
3
Tags:
Embed Size (px)
DESCRIPTION
We will go over the motivations for wix.com R&D to move to a CI/CD/TDD model, how the model was implemented and the impact on Wix R&D. We will cover the tools used (developed in-house and 3rd party), change in methodologies, what we have learned during the transformation and the unexpected change in working with product and the rest of the company. Presented in the Continuous Delivery track at DevOps Con Israel 2013
Citation preview
Continuous Delivery @ Wix
DevOpsCon
2013
Yaniv Even-Haim
VP R&D
Wix Intro
Wix in numbers • 29,000,000 registered users from 233 countries
• ~1,200,000 new users every month
• ~ 30,000 new websites every day
• 26,000,000 web sites
• 800,000 mobile sites
• 1Tbyte of media uploaded every day
• More than 1 Billion users media files
• 250+ servers in 3 data centers
• 350 employees
• 100 R&D people
A story on Wix time machine
CD is business concern
Traditional “Economies of scale” threaten by cloud technology
Be faster than small startups
CD “Economies of Agility “ = Velocity X Size
20
10
-8
20
10
-9
20
10
-…
20
10
-…
20
10
-…
20
11
-1
20
11
-2
20
11
-3
20
11
-4
20
11
-5
20
11
-6
20
11
-7
20
11
-8
20
11
-9
20
11
-…
20
11
-…
20
11
-…
20
12
-1
20
12
-2
20
12
-3
20
12
-4
Revenue per month
CD is culture & mindset • Trust the developers
– Empower developers to change production
– Developer knows his system best
CD is culture & mindset • Automation as a default choice
– no more “is it worth to automate ? ”
– Everything should be automated
• Product/Dev/QA/Ops boundaries are going down
– Everyone responsible for everything
Welcome to the twilight zone
• Move fast - Less formality
– Corridor - IN , Meeting Room - Out
Welcome to the twilight zone
Get out of thought land
• The Law of Failure
– Most new “its” will fail even if they are flawlessly executed
• Invest less, in-touch less , better ability to admit it fail
– Data beats opinions - Let the market decide
make sure you building the right it before build it right
Some Best Practices
• Everyone develops on the Trunk/Master
– Release frequent – small pieces of functionality
– It’s the developer responsibility not to break anything
• Code can get to production at anytime
– TDD & integration test
• Use Feature Toggle
– Unbaked new code can go to production – no harm done
– New code goes with a guard – use new or old code
Some Best Practices
• Backwards and Forwards compatible
– Each component has to function with latest, next or prior version of other components (including DBs)
• Gradual Deployment & Self Test
– Deploy new version to one server and perform self test. If it passes, continue deployment to other servers
• A/B Testing
– Open a new feature to a percent of your users. Is it better?
– Deployment is an engineering decision
CD – prepare to invest….. • Dev infrastructure - Refactor , Refactor, Refactor
• Testing infrastructure & know how
• Deployment infrastructure & tools
• Automation , Automation , Automation
• Monitoring (business and technical)
– hundreds of aspects
– thresholds use is a Must
– Monitor business KPIs
– Internal & external
– Endless Tuning & learning
• Deployment complexity
– More services
– More dependencies
– Multi data centers
• Development complexity
– Very hard to work on master/trunk with 30+ dev on same code base
– backward and forward comptability is not easy
• Wider set of technologies ,tools & languages
– Higher level of infrastructure investments
– Harder to have all around players
– More communication/sync required
Scaling challenges
Scaling challenges – Quality
– Testing coverage challenge
• Use cases increases
• Browsers wild west
• Tablets and mobile variety
– Feature toggle combinations
• Dozens of parallel feature toggles
– Production frequent changes
• Services are updated daily
• Toggles switch on/off on daily basis
Scaling challenges - Availability • Service Availability
• Cost of fault/downtime increasing as business grow
• Service downtime during deployment is unacceptable
• Human mistakes are inventible when changing production daily
Scaling challenges – Monitoring
• Monitoring in multi technology env
• Unified dashboard require major effort
• Reduce alert noises resulted by deployments
• Identify root cause is not simple
– Some faults does not have immediate impacts
– Cascading effects
• Reliable business KPI without false positive
–Marketing changes
– Special events (holidays, sport event etc)
Scaling challenges – Product • Product MVP practices
– Does MVP meet your product standards?
• What about tooltip,help,first time ux, etc.. ?
– How to define a product that can be developed in a day ?
– And that can win in a/b test …
To Be
Implemented
Scaling challenges – Org • Complex Matrix Management
- People moving between teams frequently
- One can have several managers on different tasks
- Hard both to managers and employees
- Everyone should master the matrix management skills
Wix Developer Lifecycle
Wix Developer Lifecycle
New Relic - Monitoring
Application dashboard
Chix – Staging Deployment
It is not all or nothing game - it is
continuous journey