View
221
Download
0
Category
Preview:
Citation preview
8/14/2019 Ptss Aptd Print
1/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems Page 1
Derived from:
Microsoft patterns & practicesPerformance Testing Guidance for Web Applications
By: J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
2007 Microsoft Corporation. All rights reserved.
http://www.codeplex.com/PerfTestingGuide
Scott BarberChief Technologist
PerfTestPlus, Inc.
Performance Testing Software Systems:
Analyzing Performance Test Data
8/14/2019 Ptss Aptd Print
2/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems Page 2
Scott Barber, CTO, PerfTestPlus, Inc.
sbarber@perftestplus.comwww.perftestplus.com
Co-Founder,Workshop On Performance and Reliability
www.performance-workshop.org
Executive Director,Association for Software Testingwww.associationforsoftwaretesting.org
Co-Author,Performance Testing Guidance for Web Applicationswww.codeplex.com/PerfTestingGuidewww.amazon.com/gp/product/0735625700
Performance Testing Software Systems
mailto:sbarber@perftestplus.comhttp://www.perftestplus.com/http://www.performance-workshop.org/http://www.associationforsoftwaretesting.org/http://www.codeplex.com/PerfTestingGuidehttp://www.codeplex.com/PerfTestingGuidehttp://www.codeplex.com/PerfTestingGuidehttp://www.codeplex.com/PerfTestingGuidehttp://www.codeplex.com/PerfTestingGuidehttp://www.codeplex.com/PerfTestingGuidehttp://www.associationforsoftwaretesting.org/http://www.performance-workshop.org/http://www.performance-workshop.org/http://www.performance-workshop.org/http://www.perftestplus.com/mailto:sbarber@perftestplus.com8/14/2019 Ptss Aptd Print
3/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 3
Credits
Some of this material was developed for, or inspired by, Performance TestingGuidance for Web Applications, a Microsoft patterns & practices book by
J.D. Meier, Scott Barber, Carlos Farre, Prashant Bansode, and Dennis Rea.
Many ideas in this course were inspired or enhanced by colleagues includingAlberto Savoia, Roland Stens, Richard Leeke, Mike Kelly, Nate White, RobSabourin, Chris Loosley, Ross Collard, Jon Bach, James Bach, JerryWeinberg, Cem Kaner, Dawn Haynes, Karen Johnson, and the entire
WOPR community.
Most of the concepts in this presentation are derived from publications,presentations, and research written and/or conducted by Scott Barber.
Many ideas were improved by students who took previous versions of this
course, back to 2001.This course has been heavily influenced by:
Rapid Software Testing(James Bach & Michael Bolton, 1995-2007 Satisfice, Inc.)Just-In-Time Testing(Robert Sabourin, 1998-2007 Amibug, Inc.)
8/14/2019 Ptss Aptd Print
4/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 4
I Assume That You:
Test software performance or manage someone(s) who does.
Have at least some control over the design of your tests andsome time to create new tests.
Have at least some influence over your test environment.
Are worried that your test process is spending too much timeand resources on things that arent important AND/OR
Are worried that your test process doesnt leave enough timeand resources to determine what IS important.
Believe that good testing requires thinking.
Test under uncertainty, resource limitations and timepressure.
Have a major goal to find important problems quickly.
Want to get very good at testing software performance.
8/14/2019 Ptss Aptd Print
5/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 5
There is no such thing as a
junior performance tester
but there are people who are new
to performance testing.
--Scott Barber
8/14/2019 Ptss Aptd Print
6/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 6
Instructional Methods That I Use
The Class Presents My Editorial Opinions: I do not make appeals toauthority; I speak only from my experiences, and I appeal to your
experience and intelligence.Not All Slides Will be Discussed: There is muchmore material here
than I can cover in detail, so I may skip some of it. (If you want me togo back to something that I skipped, just ask.)
I Need to Hear from You: You control what you think and do, so I
encourage your questions aboutand challenges tothe lecture. (Talk tome during the break, too.)
If You Want Specifics, Bring Specifics: I invite you to bring realexamples of testing problems and test documents to class. (I am happyto show you how I would work through them.)
The Exercises are the Most Important Part: I sometimes use immersivesocratic exercisesthat are designed to fool you if you dont askquestions. I usually do not provide all the information you need. Askingquestions is a fundamental testing skill!
Slide Adapted from Rapid Software Testingby James Bach & Michael Bolton, 1995-2007, Satisfice, Inc.
8/14/2019 Ptss Aptd Print
7/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 7
Instructional Methods That I Use
If I call on you,
and you dont want to be put on the spot,just say Pass!
Slide Adapted from Rapid Software Testingby James Bach & Michael Bolton, 1995-2007, Satisfice, Inc.
8/14/2019 Ptss Aptd Print
8/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 8
What Not to Expect From Me
Untested theory.
Marketing fluff.
Pulled punches to protect the guilty.
The One True Answer to anything.Every concept to apply, precisely as
presented, to every context.
Over simplifications withoutacknowledgement.
A disimpassioned, boring instructor!
8/14/2019 Ptss Aptd Print
9/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 9
Primary Goal of this Course
To teach you how to
think about, organize, manage, and report
performance test data effectively,
under time and resource constraints.
8/14/2019 Ptss Aptd Print
10/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 10
Lets face the truth, performance testing
*IS* rocket science.
--Dawn Haynes
8/14/2019 Ptss Aptd Print
11/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 11
Performance Testing Principles
Project context is central to successful performance testing.
Business, project, system, & user success criteria.
Identify system usage, and key metrics; plan and design tests.
Install and prepare environment, tools, & resource monitors.
Script the performance tests as designed.
Run and monitor tests. Validate tests, test data, and results.
Analyze the data individually and as a cross-functional team.
Consolidate and share results, customized by audience.
"Lather, rinse, repeat" as necessary.
8/14/2019 Ptss Aptd Print
12/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 12
8/14/2019 Ptss Aptd Print
13/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 13
Do you know your performance testing mission?
Do you know the Commanders Intent?
Can you find out?
Might COPE in PUBS help?
Example from my days as a U.S. Army LT:
Mission: Secure hilltop 42 NLT 0545 tomorrow.
Commanders Intent: It is my intent that the supplyconvoy safely cross the bridge spanning the gorgebetween hilltop 42 and hilltop 57 between 0553 and0558 tomorrow.
8/14/2019 Ptss Aptd Print
14/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 14
8/14/2019 Ptss Aptd Print
15/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 15
Goals: Soft Boundaries(User Satisfaction)
Requirements: Firm Boundaries(Business or Legal)
Thresholds: Hard Boundaries(Laws of Physics)
Constraints: Arbitrary Boundaries(Budget or Timeline)
Performance Criteria are boundariesdictated orpresumed by someone orsomething that matters.
8/14/2019 Ptss Aptd Print
16/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 16
+ =>
8/14/2019 Ptss Aptd Print
17/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 17
Performance Testing Objectives
What we actually hope to gain by testing performance
Are sometimes completely unrelated to statedrequirements, goals, thresholds, or constraints
Should be the main drivers behind performance testdesign and planning
Usually indicate the performance-related priorities ofproject stakeholders
Will frequently override goals in go-live decisions
How do we know if were meeting our objectives?
8/14/2019 Ptss Aptd Print
18/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 18
8/14/2019 Ptss Aptd Print
19/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 19
Sorry, this slide is busy now. Try a differentcourse by Scott Barber.
For now, remember to design tests that areprovide information about your objectives.
--SlideMaster
8/14/2019 Ptss Aptd Print
20/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 20
8/14/2019 Ptss Aptd Print
21/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 21
Sorry, this slide is busy now. Come back fora different course by Scott Barber.
For now, remember to instrument yourenvironment to collect information about
your objectives.
--SlideMaster
8/14/2019 Ptss Aptd Print
22/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 22
8/14/2019 Ptss Aptd Print
23/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 23
MacGyver is a super-hero,
*not*
a career path.
--Scott Barber
8/14/2019 Ptss Aptd Print
24/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 24
--SlideMaster
Yes, this slide is busy as well. It is alsoavailable in other courses by Scott Barber.
For now, remember that for your data to beuseful, the scripts must accurately represent
your design.
8/14/2019 Ptss Aptd Print
25/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 25
8/14/2019 Ptss Aptd Print
26/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 26
You guessed it, this slide is busy too. It ishaving a party with its friends, but is happyto display itself during other courses by
Scott Barber.
For now, remember to validate that yourtests did what you intended and that you
execute enough times to haveconfidence in your data.
--SlideMaster
8/14/2019 Ptss Aptd Print
27/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 27
8/14/2019 Ptss Aptd Print
28/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 28
With an order of magnitude fewer variablesperformance testing could be a science,but for now,
performance testing is at bestascientific art.
--Scott Barber
8/14/2019 Ptss Aptd Print
29/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 29
All three have an average of 4.
Which has the best performance?
How do you know?
8/14/2019 Ptss Aptd Print
30/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 30
All three have an average of 4.
Which has the best performance?
How do you know?
8/14/2019 Ptss Aptd Print
31/60
www.PerfTestPlus.com 2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 31
Now which has the best performance?
8/14/2019 Ptss Aptd Print
32/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems Page 32
Results are meaningless without technical context.
Dont over-trust results until you can repeat them.
Within the test run, across tests, across data, etc.
If you can repeat it or its >1%, its not an outlier.
Graph, blink, overlay, compare, and contrast.
If it can get you sued, check it every time.
How well do the results represent reality.
This is where users care and symptoms are found.
If its broken, performance doesnt matter.
8/14/2019 Ptss Aptd Print
33/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems Page 33
Methods:
BlinkDe-Focus & Re-Focus
Overlay
Plot
Bucket
Look for Odd
Be Derivative
Ditch the DigitsUn-average Averages
Manual
8/14/2019 Ptss Aptd Print
34/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems Page 34
Facts:
Analysis is a team sport.
We cannot proveanything.
Focus on patterns, trends, and feelings.
Numbers are meaningless out of context.
Qualitative feedback is at leastas relevant asquantitative feedback.
8/14/2019 Ptss Aptd Print
35/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems
Page 35
Instructions:
Reassemble into your group.
Pay attention, Im going to explain this group ofexercises orally.
Be prepared to describe your findings with the class.
8/14/2019 Ptss Aptd Print
36/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems
Page 36
Lets play a game that I call...
8/14/2019 Ptss Aptd Print
37/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems
Page 37
8/14/2019 Ptss Aptd Print
38/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software Systems
Page 38
8/14/2019 Ptss Aptd Print
39/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.Performance Testing Software SystemsPage 39
8/14/2019 Ptss Aptd Print
40/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 40
8/14/2019 Ptss Aptd Print
41/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 41
8/14/2019 Ptss Aptd Print
42/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 42
8/14/2019 Ptss Aptd Print
43/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 43
8/14/2019 Ptss Aptd Print
44/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 44
8/14/2019 Ptss Aptd Print
45/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 45
8/14/2019 Ptss Aptd Print
46/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 46
8/14/2019 Ptss Aptd Print
47/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 47
Linear extrapolationof performance test results is,at best, black magic.
Dont do it (unless your name is Connie Smith, PhD.or Daniel Menasce, PhD.)
--Scott Barber
8/14/2019 Ptss Aptd Print
48/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 48
Stakeholders need data to make decisions.Many decisions cant wait until tomorrow.
Reports are only interesting if they contain datathat is useful.
A great report for developers is probably a lousyreport for executives.
Try to use pictures over numbers and numbersover words. Save words for recommendations.
Strive to make reports compelling without
explanation.
Unless you are hiding something, make thesupporting data available to the team.
8/14/2019 Ptss Aptd Print
49/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 49
Facts:
Most people will never read performance testresults docs.
Most people dont really understand the underlyingcomponents to performance.
It is our job to make it easy for them to understand,and understand quickly.
Being skilled at graphical presentation of technicalinformation is critical for us to help othersunderstand the message we are delivering.
Confusing charts and tables lead to wrongdecisions causing lost $ and ruined reputations.
8/14/2019 Ptss Aptd Print
50/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 50
What consumers of reports want:
Answers NOW! (They might not even know the question)
To understand information intuitively.
Simple explanations of technical information.
To be able to make decisions quickly and have theinformation to support those decisions.
Trigger phrases to use with others.
Concise summaries and conclusions.
Recommendations and options.
8/14/2019 Ptss Aptd Print
51/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 51
What consumers of reports usually get:
8/14/2019 Ptss Aptd Print
52/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 52
Strive for something better:
Concise verbal descriptions.Well formed, informative charts (pretty pictures).
Focus on requirements and business issues.
Dont be afraid to make recommendations ordraw conclusions!
Make all supporting data available to everyone,all the time (Dont sit on data cause they wont understand it).
Report DocumentReport *AT LEAST* every 48 hours duringexecution.
8/14/2019 Ptss Aptd Print
53/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 53
Inspired by ET:
Edward Tufte, Ph.D., Professor Emeritus of politicalscience, computer science and statistics, andgraphic design at Yale.
According to ET:
Power Corrupts...
8/14/2019 Ptss Aptd Print
54/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 54
PowerPoint Corrupts Absolutely.
8/14/2019 Ptss Aptd Print
55/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 55
Appendix Level
8/14/2019 Ptss Aptd Print
56/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 56
8/14/2019 Ptss Aptd Print
57/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 57
Guess where I am...
For now, remember that you can neverhave too much data, and that youll rarely
have enough time to analyze the data youdo have.
--SlideMaster
Performance Testing Principles
8/14/2019 Ptss Aptd Print
58/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 58
Performance Testing Principles
Project context is central to successful performance testing.
Business, project, system, & user success criteria.
Identify system usage, and key metrics; plan and design tests.
Install and prepare environment, tools, & resource monitors.
Script the performance tests as designed.
Run and monitor tests. Validate tests, test data, and results.
Analyze the data individually and as a cross-functional team.
Consolidate and share results, customized by audience.
"Lather, rinse, repeat" as necessary.
Questions
8/14/2019 Ptss Aptd Print
59/60
www.PerfTestPlus.com
2006-8 PerfTestPlus, Inc. All rights reserved.
Performance Testing Software Systems Page 59
Questions
Contact Info
8/14/2019 Ptss Aptd Print
60/60
P fT Pl Performance Testing Soft are S stems P
Contact Info
Scott BarberChief Technologist
PerfTestPlus, Inc
E-mail:
sbarber@perftestplus.com
Web Site:
www.PerfTestPlus.com
Recommended