Implementing Metrics-Driven DevOpsWhy and How!Andreas Grabner: @grabnerandi, [email protected] Slides: http://www.slideshare.net/grabnerandiPodcast: https://www.spreaker.com/show/pureperformance
@grabnerandi
@grabnerandi
@grabnerandiAND MANY MORE
@grabnerandihttps://dynatrace.github.io/ufo/
“In Your Face” Data!
@grabnerandi
Time of D
eployment
Availability dropped to 0%
#1: Availability -> Brand Impact
@grabnerandi
New Deployment + Mkt Push
Increase # of unhappy users!
Decline in Conversion Rate
Overall increase of Users!
#2: User Experience -> Conversion
Spikes in FRUSTRATED Users!
@grabnerandi
#3: Resource Cons -> Cost per Feature
@grabnerandi
App with Regular Load supported by
10 ContainersTwice the Load but 48 (=4.8x!) Containers! App doesn’t scale!!
#4: Scalability -> Cost per User
@grabnerandi
#5: Performance -> Behavior
@grabnerandi
@grabnerandi
DevOps @ Targetpresented at Velocity, DOES and more …
http://apmblog.dynatrace.com/2016/07/07/measure-frequent-successful-software-releases/
“We increased from monthly to 80 deployments per week
… only 10 incidents per month …
… over 96% successful! ….”
“We Deliver High Quality Software,Faster and Automated using New Stack“
„Shift-Left Performance to Reduce Lead Time“
Adam Auerbach, Sr. Dir DevOps
https://github.com/capitalone/Hygieia & https://www.spreaker.com/user/pureperformance
“… deploy some of our most critical production workloads on the AWS platform …”, Rob Alexander, CIO
2 major releases/yearcustomers deploy & operate on-prem
26 major releases/year170 prod deployments/dayself-service online sales SaaS & Managed
2011 2016
@grabnerandi
Not only fast delivered but also delivering fast!
-1000ms +2%
Response Time Conversions
-1000ms +10%
+100ms -1%
Why most (will) fail!
@grabnerandi
@grabnerandi
It‘s not about blind automation of pushing more bad code on new stacks through a pipeline
@grabnerandi
It‘s not about blindly adding new features on topof existing withouth measuring its success
@grabnerandi
I learning from
others
@grabnerandi
http://bit.ly/sharepurepath
@grabnerandi
Scaling an Online Sports Club Search Service
2015201420xx
Response Time
2016+
1) 2-Man Project 2) Limited Success
3) Start Expansion
4) Performance Slows Growth Users
5) Potential Decline?
@grabnerandi
Early 2015: Monolith Under Pressure
Can‘t scale vertically endlessly!
May: 2.68s 94.09% CPU Bound
April: 0.52s
@grabnerandi
From Monolith to Services in a Hybrid-Cloud
Front Endto Cloud
Scale Backendin Containers!
@grabnerandi
Go live – 7:00 a.m.
@grabnerandi
Go live – 12:00 p.m.
What Went Wrong?
@grabnerandi
26.7s Load Time5kB Payload
33! Service Calls
99kB - 3kB for each call!
171! Total SQL Count
Architecture ViolationDirect access to DB from frontend service
Single search query end-to-end
@grabnerandi
The fixed end-to-end use case“Re-architect” vs. “Migrate” to Service-Orientation
2.5s (vs 26.7) 5kB Payload
1! (vs 33!) Service Call
5kB (vs 99) Payload!
3! (vs 177) Total SQL Count
@grabnerandi
@grabnerandi
You measure it! from Dev (to) Ops
@grabnerandi
Build 17 testNewsAlert OK
testSearch OK
Build # Use Case Stat # API Calls # SQL Payload CPU
1 5 2kb 70ms
1 35 5kb 120ms
Use Case Tests and Monitors Service & App Metrics
Build 26 testNewsAlert OK
testSearch OK
Build 25 testNewsAlert OK
testSearch OK
1 4 1kb 60ms
34 171 104kb 550ms
Ops#ServInst Usage RT
1 0.5% 7.2s
1 63% 5.2s
1 4 1kb 60ms
2 3 10kb 150ms
1 0.6% 3.2s
5 75% 2.5s
Build 35 testNewsAlert -
testSearch OK
- - - -
2 3 10kb 150ms
- - -
8 80% 2.0s
Metrics from and for Dev(to)Ops
Re-architecture into „Services“ + Performance Fixes
Scenario: Monolithic App with 2 Key Features
@grabnerandi
your tool of choice
Baseline Key Metrics
#SQL, #Threads, Bytes Sent, # ConnectionsWPO Metrics, Objects Allocated, ...
@grabnerandihttps://github.com/Dynatrace/Dynatrace-Test-Automation-Sampleshttps://dynatrace.github.io/ufo/
Regression Detected
Fail the build!
@grabnerandi
Dev&Test: Check-In Better Code
Performance: Production Ready Checks! Validate Monitoring
Ops/Biz: Provide Usage and Resource Feedback for next
Sprints
Test / CI: Stop Bad Builds Early
Build & Deliver Apps like the Unicorns!With a Metrics-Driven Pipeline!
@grabnerandi
12:00 a.m – 11:59 p.m.
QuestionsSlides: slideshare.net/grabnerandiGet Tools: bit.ly/dtpersonalWatch: bit.ly/dttutorialsFollow Me: @grabnerandiRead More: blog.dynatrace.comListen: http://bit.ly/pureperfMail: [email protected]
Andreas GrabnerDynatrace Developer Advocate@grabnerandihttp://blog.dynatrace.com