Embed Size (px)
Citation preview
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alan Milford, Systems Architect, AOL Platforms
Ilan Rabinovitch, Director of Technical Community, Datadog
November 30, 2016
DEV202
From Dial-up to DevOpsAOL's Migration to the Cloud
1983
A few things have
changed....
acquisitions
68
advertising companies
15
Online advertising
across desktop,
mobile, and TV
What that looks like internally
• Many complex systems
• High traffic, low latency response
• Downtime isn't an option
• Every transaction has a cost and value
What does DevOps
mean to us?
CultureAutomationMeasurementSharingJohn Willis (@botchagalupe), Damon Edwards (@damonedwards)
6 years ago...
Starting a new team
• New advertising product
• New R&D engineering team
• New (untested) software
• No dedicated QA
A recipe for
disaster...?
Not necessarily
Time to scale
Moving into production
• Rough estimate of full traffic load
• Look at current CPU/memory/IO
• Do some maths
• Purchase appropriate hardware
Done
Made it to production
• Everything was stable
• Traffic was steady
• System worked as expected
• Product owners & engineers were happy
Everything's running smoothly
Tra
ffic
Time
Wait, what?
Tra
ffic
Time
What happened?
• Complete data centre outage (not our facility)
• Servers powered up in random order
• Data corruption
• Recovery time was out of our hands
"Success consists of going from failure to
failure without loss of enthusiasm."
Winston Churchill
This isn’t going to work
Making our teams better
• Need to speed up development cycle
• Agile project management
• Remove layers of communication
• Remove manual steps
Agile is changing thinking
DevOps is changing doing
CultureAutomationMeasurementSharing
Culture
• People and process first
• Teamwork is key
• Initiative to find solutions
• More important than any tool
Automation
• Replaced people with shell scripts*
• Scaled with code rather than people
• Automated build and deploys
• Made things virtual not physical
* I guarantee someone here is wearing this t-shirt
To the cloud
Internal private cloud
• Improved flexibility
• Management all in-house
• Development speed increased
• Parity between environments
Internal private cloud
• Strong internal security
• Still limited by capacity
• Provisioning more hypervisors is difficult
• Responsible for own geo distribution
That capital cost
didn't just go away
Building on AWS
• Capital expense to variable expense
• No need to worry about hardware refresh
• Lead time to spin up reduced
• Time to deployment reduced
Building on AWS
• Open flexibility to engineers
• Much quicker to try something new
• Iteration cycles speed up
• No waiting on hardware
Treat servers like
cattle, not pets
but…
Do you know what this cow is doing?
Collecting data is cheap;
not having it when you
need it can be expensive
Measurement/metrics
• If it moves, measure it
• How is the application performing?
• Are things getting better? Or worse?
• Remove the guesswork from decisions
@honest_update
Sharing
• Encourage teams to share their data
• Help people to solve their own issues
• Everyone plays a part in getting to production
• Play nice with others
Datadog Slack
What did we learn?
• Push hard to refactor applications
• Changing mindsets is harder than changing tech
• Empower people to make their own decisions
• Give them the data to do it
This is a learning process
• Don't search for a silver bullet
• Data is key to knowing what's going on
• Understand workloads
• You wouldn't heat your home all day in summer
Build a culture that can
adapt quickly to change
Process should make life
easier, not be a checklist
Don't get complacent;
technology moves quicker
than people
@alz
@irabinovitch
Thank you!
Remember to complete
your evaluations!
