Embed Size (px)
DESCRIPTION
Both Formula 1 and DevOps teams are driven by innovation, technology, agile mindsets and money. However, when it comes to managing change, Formula 1 teams excel in being agile to deliver superior levels of performance and reliability. In comparison, DevOps teams strive to deliver innovative applications for the business, but more often that not struggle to make them fast or reliable once they are deployed in live environments. This session takes a look at how the people, processes, collaboration and tools behind a Formula 1 team can inform how we approach DevOps, and how DevOps teams can better manage the performance and reliability of their applications throughout the application lifecycle. Stephen Burton is Tech Evangelist at AppDynamics, and is also the alter ego of increasingly popular "App Man" character. Steve is responsible for promoting best practice application performance management (APM) for distributed applications running in cloud, virtual and physical environments. Before joining AppDynamics, Steve held senior product management positions at OpTier and Precise leading innovation and creative solutions to help customers better manage the performance of their applications. Steve has previously worked in pre-sales and also spent many years as a senior developer and application support engineer when his career began at Sapient. Steve received his BS in Computer Science from Lancaster University, UK. He enjoys speed and is a keen motorsport/formula 1 fan and hopes one day to become The Stig.
Citation preview
Dev OPS
Stephen Burton, Tech Evangelist@BurtonSays
What can DevOps Learn from F1?
Tuesday, June 19, 12
• App Support
•Developer
• Systems Engineer
• Pre-Sales
• Product Management
• Tech Evangelist
About Me
Tuesday, June 19, 12
2011 Formula 1 World ChampionSebastian Vettel
Who is this?
Tuesday, June 19, 12
• 20 Races on the World Calendar
• 12 Teams, 24 Cars & Drivers
• Cost is ~$7.5 Million per car
• Spend $75 - $300 Million per year
• 0-100-0 in < 5 seconds
Formula 1
Tuesday, June 19, 12
• F1 car has 80,000 components,1KM wiring &100+ sensors
• 99.9% assembled correctly still means 80 POF’s
•Monaco GP - Manholes are welded down
• Brakes on F1 Car is like hitting a brick wall at 200mph (5G)
•Downforce a baby elephant sitting on drivers head
Interesting F1 Facts
Tuesday, June 19, 12
“DevOps is about being agile,and going from
A-Ha to Cha Chingas quickly as possible”
John WillisDevOps Evangelist
Why F1 and DevOps?
Tuesday, June 19, 12
Innovation. Agility. Change.
Tuesday, June 19, 12
Simple?
Tuesday, June 19, 12
Tuesday, June 19, 12
Hidden Complexity
Tuesday, June 19, 12
Cutting Edge Technology
Tuesday, June 19, 12
Cool Innovation
Tuesday, June 19, 12
Simple?
Tuesday, June 19, 12
ActiveDirectory
Tibco BW
3rd PartyWeb Service
Oracle
Cassandra
Tomcat
Tomcat
JBoss ASP.NET
3rd PartyWeb Service
SQLServer
3rd PartyJava App
JBoss
JDBC
SOAP
ADO.NET
HTTP
SOAP
SOAP
LDAP
ADO.NET
THRIFT
JMS THRIFT
JMS
HTTP
Hidden Complexity
Tuesday, June 19, 12
REALLY
Really.
Tuesday, June 19, 12
Really. Really.
Tuesday, June 19, 12
Seriously.
Tuesday, June 19, 12
What you can’t see...
Tuesday, June 19, 12
Cool Integration
Tuesday, June 19, 12
SUCCESS IN FORMULA 1
is Winning RacesSuccess.
Tuesday, June 19, 12
Ideas & Agility Wins RacesTuesday, June 19, 12
Dev
Developers wantAgility & Change
OPS
Operations wantAvailability & Stability
WHAT IS SUCCESS FOR DEVOPS ?
Tuesday, June 19, 12
COMPARING TEAMS
Formula 1
•Driver•Sponsors•Architects•Designers•Engineers•Testing•Operations•Support
Applications
•End User•Business•Architects•Designers•Developers•Testing•Operations•Support
Tuesday, June 19, 12
MANAGING CHANGEISN’T EASY
Design > Build > Test > Deploy > Support
SlowdownsOutages
Mis-configuration
Bottlenecks
Bugs
End UserComplaints
BusinessRequirements
ProjectPlans
More BusinessRequirements
Tuesday, June 19, 12
CHANGE
Doesn’t Happen Overnight Though
Tuesday, June 19, 12
Change or Your Competition Will
Tuesday, June 19, 12
•Cars Evolve• Up to 30 new parts per race
F1 Teams Collaborate
•Engineering• Aero, Engine, Transmission, ....
•Operations• Mechanics, Telemetry, Pit Crew
Tuesday, June 19, 12
Change is Constant
Tuesday, June 19, 12
Develop Test SupportDeployDesign
Operations work hand in hand with Engineering
Race Weekend
Formula 1 Lifecycle
Tuesday, June 19, 12
MEASURING SUCCESS
• Downforce
• Performance
• Reliability
• Race Wins
Telemetry &Monitoring
Driver Results
Monitoring is critical to Managing Change
Tuesday, June 19, 12
End User Experience is Critical
Tuesday, June 19, 12
Monitoring is also Critical
Tuesday, June 19, 12
Measure Impact
Tuesday, June 19, 12
The NOC
Tuesday, June 19, 12
MONITOR & MANAGEIMPACT OF CHANGE
Fast Slow Fail
Tuesday, June 19, 12
Tools are Important
Tuesday, June 19, 12
SO, DO YOU KNOWHOW FAST AND RELIABLE
YOUR CODE ISIN PRODUCTION?
<1% of Developers actually do
Tuesday, June 19, 12
CODE CAN BE FUNCTIONALLY PERFECT
Text
But how Fast and Reliable will it be?
Tuesday, June 19, 12
Concurrency Data Volume Resource
3 ThingsImpact Application Performance
Tuesday, June 19, 12
DEV
Data Volume ResourceConcurrency
Not Often
QA/TEST
Data Volume ResourceConcurrency
Less Often
PRODUCTION
Data Volume ResourceConcurrency
Often
WHERE DO YOU FIND REAL BOTTLENECKS?
Tuesday, June 19, 12
F1 TESTING
• 3 Pre-season Tests (Integration & Soak testing)
• 1 Mid-season Test (For Ferrari to catchup)
•Wind Tunnel & Simulator (Unit & Functional & Load Testing)
• Race Weekends (Production Testing)
Tuesday, June 19, 12
HANDS UP IF YOU DOPERFORMANCE TESTING
ON EVERY RELEASE
(Its tough with Agile cycles)
Tuesday, June 19, 12
SUCCESSISN’T HANDING OVER A
RELEASE TO OPS
DEPLOYME
Tuesday, June 19, 12
GIFTS CAN HAVE SURPRISES
NOBODY LIKES SURPRISES
Tuesday, June 19, 12
DEV SUCCESSISN’T THE APP LASTING
1 HOUR IN PRODUCTION
Tuesday, June 19, 12
•Ops has a 48 hour warranty period with Dev
• If the App experiences consistent slowdowns or outages
• All Tickets get re-assigned to Development
The DevOps Warranty
Tuesday, June 19, 12
REAL-LIFE SURPRISESWhen Things go Wrong
Tuesday, June 19, 12
BLUE - Concurrency (Trx/min)GREEN - Avg Response TimeRED - Resource (CPU %)
Dealing with Spikes
Tuesday, June 19, 12
BLUE - Concurrency (Trx/min)GREEN - Avg Response Time
Not Dealing with Spikes
Tuesday, June 19, 12
Trx/min
Avg RTPool LimitPool Usage
Trx Stalls
Production Ground to a halt for 2 hours And again the next day
Who owns JVM Configuration? Ops or Dev?
Not Learning from Failure
Tuesday, June 19, 12
Whose responsible for this? Ops?
Bad Implementation
Tuesday, June 19, 12
Whose responsible for this? Ops?
Missing Configuration
Tuesday, June 19, 12
Whose responsible for this? Ops?
Inefficient Code
Tuesday, June 19, 12
Root Cause: Cache wasn’t thread safe
46,463 Checkouts in the Day
2,492 were impacted
Avg. $100 per Checkout
Cost of Deadlock: ~ $249,000
Code Deadlock
Tuesday, June 19, 12
IMAGINE ENGINEERSWITH NO RACE VISIBILITY
Develop Test SupportDeployDesign
Car would never win races or finish
Race Weekend
Tuesday, June 19, 12
Before
After
Tuesday, June 19, 12
Before
After
Tuesday, June 19, 12
• # Agile Releases or % Availability is pointless
• End User Experience?
• Business Transaction Throughput?
• MTBF and Business Impact?
DevOps Need Common Goals
Tuesday, June 19, 12
• Feedback Loop from Ops to Dev
• Share Goals, Tools & Metrics
• Continuous Improvement
• Learn from Failure
•Win and Loose Together
Lessons Learned•Don’t write slow SQL Queries•Remember to use caching•Don’t rely on ORM•Assume everything will fail•Logging everything is not a good idea•Remember to do performance testing
DevOps - Collaborate & Learn
Tuesday, June 19, 12
•Define Success
• Embrace Change, don’t fear it
•Monitor, Measure and Manage Impact
• Learn from Failure
• Have Fun and remember
Top Tips for DevOps
Tuesday, June 19, 12
DevOps is about Team Work
Tuesday, June 19, 12
+
Visit the AppDynamics Booth
Free T-shirt Free Application Monitoringin Production
www.appdynamics.com
Tuesday, June 19, 12
We’re Hiring!
Tuesday, June 19, 12
