DevOpsGuys
@DevOpsGuys www.devopsguys.com 1
DevOpsGuysThe benefits of using an
APM solution while performance testing
Why Perf testing without APM is like Corona without the Lime…
DevOpsGuys
@DevOpsGuys www.devopsguys.com 2http://www.flickr.com/photos/lendog64/5781379311/
DevOpsGuys
@DevOpsGuys www.devopsguys.com 3
Perf Testing & APM in 30mins
• Why Performance Test? • What is APM anyway? • Benefits of APM when Perf Testing• With some case studies along the
way!• Q&A
DevOpsGuys
@DevOpsGuys www.devopsguys.com 4
Why Performance Test?
1. Ensure that we meet our customer’s expectations of website response time– So that they convert and we make £££
2. Ensure that we have sufficient computing capacity to meet #1 under load (load testing)
3. Understand how the system reacts when things go wrong (stress testing)
DevOpsGuys
@DevOpsGuys www.devopsguys.com 5
DevOpsGuys
@DevOpsGuys www.devopsguys.com 6
APM Defined
“In the fields of information technology and systems management, application performance management (APM), is the monitoring and managing of performance and availability of software applications. APM strives to detect and diagnose application performance problems to maintain an expected level of service. APM is the translation of IT metrics into business meaning (i.e. value).”http://en.wikipedia.org/wiki/Application_Performance_Management
DevOpsGuys
@DevOpsGuys www.devopsguys.com 7
“5 Dimensions of APM*”
1. End user experience monitoring - (Active and passive)
2. Application runtime architecture discovery and modeling
3. User-defined transaction profiling (also called business transaction management)
4. Application component monitoring5. Reporting & Application data analytics
*According to Gartner
DevOpsGuys
@DevOpsGuys www.devopsguys.com
CA
QuestHP
OpTier
Opnet
IBM
Compuware
Oracle
MicrosoftSL
Inetco
NetScout
eG InnovationsProgress
ASG
IderaVisual networks
Info vista
Network Instruments
Knoa
Aternity
Coradient
CorrelsenseNastel
dynaTracePrecise
Arctures Technologies
BMC
Managed Engine
New RelicAppDynamics
2011Riverbed
OpTier
2012
AppNeta
Dell
2013
Gartner APM Magic Quadrant
Legac
y Ven
dors
DevOpsGuys
@DevOpsGuys www.devopsguys.com 9
BENEFITS OF LOAD TESTING WITH APM TOOLS
Never leave home without it…
DevOpsGuys
@DevOpsGuys www.devopsguys.com 10
#1See the Big Picture
(aka Systems Thinking)
DevOpsGuys
@DevOpsGuys www.devopsguys.com 11
Systems Thinking 101
“Systems thinking is a framework for seeing interrelationships rather than things, for seeing patterns rather than static snapshots. It is a set of general principles spanning fields as diverse as physical and social sciences, engineering and management.” - Peter Senge
DevOpsGuys
@DevOpsGuys www.devopsguys.com 12
Command & Control vs Systems Thinking
http://www.systemsthinking.co.uk/2-1.asp
DevOpsGuys
@DevOpsGuys www.devopsguys.com 13
CASE STUDY #12 WEEKS WASTED…
“any improvement not made at the constraint is an illusion”
DevOpsGuys
@DevOpsGuys www.devopsguys.com 14
The Scenario
• E-commerce customer• New website • High data-start times on page load• Poor load testing results• Initial conversation with client• 2 week delay while hosting provider
scaled out the hardware…• Problem still not fixed…
DevOpsGuys
@DevOpsGuys www.devopsguys.com 15
What’s the root cause of the ~1sec Data Start?
Waterfall Diagram of page load times
DevOpsGuys
@DevOpsGuys www.devopsguys.com 16
DevOpsGuys
@DevOpsGuys www.devopsguys.com 17
#2Drill down to the
Detail(Code-level)
DevOpsGuys
@DevOpsGuys www.devopsguys.com 18
Find the slow code!
DevOpsGuys
@DevOpsGuys www.devopsguys.com 19
#3Faster Iteration= Better Value
DevOpsGuys
@DevOpsGuys www.devopsguys.com 20
Maximise Testing Value
Test
AnalyseFix
Deploy
DevOpsGuys
@DevOpsGuys www.devopsguys.com 21
CASE STUDY #156X FASTER IN A DAY…
9am to 9pm. 12hrs. 56x faster!
DevOpsGuys
@DevOpsGuys www.devopsguys.com 22
The Scenario
• New website pre-launch testing• Customer Portal (SaaS solution)• Start Load testing 9am• 4 “Test & Fix” cycles in the day • Final testing completed at 9pm
DevOpsGuys
@DevOpsGuys www.devopsguys.com 23
The End Results = 56x Average
ResponseThroughput
(Req/sec)Initial Testing 3024ms 16/minFinal Result 54ms 59/min
56x faster 3.5x throughput
Before 3024ms After
54ms
DevOpsGuys
@DevOpsGuys www.devopsguys.com 24
Nice flat line!
Before
After
DevOpsGuys
@DevOpsGuys www.devopsguys.com 25
#4Team Work“The Enemy is Performance,
not each other…”
DevOpsGuys
@DevOpsGuys www.devopsguys.com 26
The Blame Game…
Ops DevQA
DevOpsGuys
@DevOpsGuys www.devopsguys.com 27
The Fix-It Game…
Ops
DevQA
DevOpsGuys
@DevOpsGuys www.devopsguys.com 28
DevOpsGuys
@DevOpsGuys www.devopsguys.com 29
Summary
1. See the Big Picture (systems thinking)
2. Drill down to the details3. Faster Iteration = Better Value4. Stop the “Blame Game”– aka “There’s your Problem!”
DevOpsGuys
@DevOpsGuys www.devopsguys.com 30
DevOpsGuys
Thank YouWorkshop Session @
1345End-to-end testing & APM demooperations | availability | performance | scalability | migration | strategy
DevOpsGuys
@DevOpsGuys www.devopsguys.com 31
DevOpsGuys provide development & operations support (DevOps) services
for your online applications.
You run your business. We run your website.
It’s as simple as that.
What We Do
DevOpsGuys
@DevOpsGuys www.devopsguys.com 32
Our service portfolio
• 24 x 7 Application-Centric web operations
• DevOps Transformation & Training
• Continuous Delivery implementation
• Infrastructure Automation & “Infrastructure as Code”
• Application Performance Management
• AppDynamics Implementation & Consultancy
• Web Performance consulting
• Web monitoring• Load Testing• Test Automation and
scripting• JetBrains Product
Implementation & Consultancy
• Octopus Deploy implementation
DevOpsGuys
@DevOpsGuys www.devopsguys.com 33
James Smith (@TheDevMgr)
• 16 years of software delivery experience.
• Proven track record– Pizza Hut– KFC– Dell– RAC– Aviva– Fujitsu Siemens– Whitbread
Steve Thair (@TheOpsMgr)
• 25 years of IT Operations experience.
• Proven track record– Vodafone– Credit Suisse Group– IBM– BNP Paribas– Zurich Financial Services– Totaljobs Group– TSL Education
The Leadership Team
DevOpsGuys
@DevOpsGuys www.devopsguys.com 34
Contact UsVisit us at www.devopsguys.com
Call our team on +44 (0)20 8798 3079
Email our engineers at [email protected]