Embed Size (px)
Citation preview
CLOUD TESTINGRewriting the Rules of Performance Testing
RULE 3: Performance Testing Across the Lifecycle
SOASTA Webinar Series
Performance Has Been a Lab Exercise
Dev & Test Lab Production EnvironmentDEPLOY
Performance Testing is often considered hard & non-repeatable due to real constraints:
• Linear, multi-step approaches• Non-correlated, offline analysis• Disconnected teams• Costly test hardware• Time-intensive test setup• Inability to hit realistic scale
Status Quo:small test window prior
to release
Time-consuming testing limits deployment options
Do We Spend Time Where It Counts?
TIME
TESTHW Setup FIXANALYZE RETEST REPEAT?
BUILD TEST
ANALYZE
FIX
Traditional Approaches
Eliminate constraints and TEST MORE, at SCALE, in LESS TIME
Many approaches are process and pre-test heavy
Agile approaches deliver results fast
TEST BUILDING
TEST
ANALYZE
FIX
HW
Setup
HW
Setup
TEST
ANALYZE
FIX
HW
Setup
TEST
ANALYZE
FIX
HW
Setup
TEST
ANALYZE
FIX
HW
Setup
Extra time!
Continuous Performance Testing is Progressive
Dev & Test Lab Production Environment
Start small, early to root
out code-level issues
Include performance focus in iterations
Hit expected scale in
staging and maintenance windows
Test in Production
Recheck regularly for regression
and capacity
Plan for growth
Keys to agile performance testing? SPEED – SCALE – COST
s
Memory leaks
Scale and Time
Sta
ge /
Team
Performance Bugs Know No BordersIssues span teams, time and scale
Release& Deploy
Network& Ops
CDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Slow pages
Conflict with other apps
Dev & Test
Test Lab Staging Production (100% +++)
Method-level tuning
Max sockets exceeded
Firewall max capacity
Memory leaks
Caching issues and benefits
Capacity limits
Code bugs
Database thread counts
Inefficient database queries
Search technology limits
Security bottlenecks
Global latency variance
Complete Performance TestingProactive and Predictive
Performance Coverage Status
Begin to track performance coverage across a range of
critical objectives!
Track performance regression (and progression) over time!
Let’s Look at the NEW RULES
Matt Heusser
Performance Testing: Early, Often, Always
• Three concrete techniques
• No out of pocket cost
• Learn something new
• Expand your options for performance testing
Goals Today
Assumptions• Software Under Test is a web-based system• Prefer: Running Continuous Integration• Prefer: With some browser-driving automated checks
What has been will be again, what has been done will be done again;
there is nothing new under the sun. - Ecclesiastes 1:9 (NIV)
No, this is not new
•XP Explained: 2000
•Agile Manifesto: 2001
•XP Installed: 2002
•Selenium Released: 2004
•Scrum Gets Popular: 2005
But there is a little new here
Ten years of improvement in iterative development and incremental functional tests, yet …
* Performance testing tends to remain at the end, driving up cost and causing
delays *
Approach #1: Test Concurrency
• Script takes 10-15 minutes to execute
• Test in groups of 4 to 14
• Test for 15-45 minutes
• Add randomization at the end
A Sample Script …
1. Log-In2. Search for Part ABC33. Add Part to your Cart4. Search/Add part ABC45. Add to your cart6. Change Qty of ABC3 to 37. Checkout8. Logout9. Log-in10. etc
#2: Monitor and Aggregate Functional Test Results
•Wrap logs with timing
•Run the automated tests
•Report your finding
#2: Monitor and Aggregate:An Implementation
#3: Run functional tests under load
•Hook up a load testing tool running in the background
•Run the automated tests “just” like before
•Performance Tests become load tests!
•Bonus: Mine Staging and Prod for load test data!
Putting it all together:
• We covered three ways to incrementally test system performance
• Some performance testing earlier is better than none
• Most obvious performance tests are concurrency issues and can be found by a small group quickly.
• So go find them early!
• Finding perf/load earlier means the programmer will recognize what he did to introduce it – easy fix!
Q&A
Contact SOASTA:www.soasta.com/cloudtest/[email protected] us:
twitter.com/cloudtest
facebook.com/cloudtest
Contact [email protected] Matt:
twitter.com/mheusser
Questions & Answers will post on CloudLink: http://cloudlink.soasta.com
Previous & Upcoming SOASTA Webinars:Stop Cheating & Start Running Realistic Tests (w/ Scott Barber)
Run More Tests & Find More Issues (w/ Neil Gunther)http://www.soasta.com/info-center/webinars/
