Embed Size (px)
DESCRIPTION
This talk covers how to apply the Automated Load Testing for Continuous Delivery approach
Citation preview
BugDay Bangkok 2014
BUGDAY BANGKOK 2014 SPEAKER
Nitisak
Nitisak Mooltreesri
DST Worldwide Services
Facebook: Nitisak Mooltreesri
Twitter: @theimagine
Automated Load Testing for Continuous Delivery.
Consider how much time you spend finding and fixing problems in production. Even with the most comprehensive functional testing process, you still discover elusive bugs under simultaneous load of thousands of users after going live. Early testing and rapid feedback, as well as effective communication, are cornerstones of the Agile approach to software development.
This talk covers how the BPMS performance team at DST have done the Automated Load Testing for Continuous Delivery approach.
The Approach and techniques for early testing can also be applied to non Agile projects, so even if you haven’t jumped on the Agile bandwagon, you and your team can benefit from this session.
BugDay Bangkok 2014
Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014
Why automated Load testing is important ?
• Most users click away after 8 seconds of delay
• Business revenue loss due to poor web applications performance
• Failure of US Based Airlines computerized system to maintain flight plans in Nov 2009
• Aberdeen found that inadequate performance could impact revenue by up to 9%
• Business performance begins to suffer at 5.1 seconds of delay in response times of web applications and 3.9 for critical applications
BugDay Bangkok 2014
Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014
What are the risks we are testing for ? (1)
Scalability :
• Operation Problems ? Solve with hardware ?
• Tall Stacks --> Wirsth’s Law : “Software is getting slower more rapidly than hardware get faster”
• Subject to use patterns and user models.
What Does a Problem Look Like ?
• Longer response times is a clue
• “High” CPU/Memory/Storing/Network Utilization
BugDay Bangkok 2014
What are the risks we are testing for ? (2)
Capacity : System can’t support the expected load structurally/as engineered.
• Response time very sensitive to load
• Growing Queues
• Hard or Soft Resource Limitations
• High CPU Limitation
• Increasing I/O Latency
• Run out of database threads
BugDay Bangkok 2014
What are the risks we are testing for ? (3)
Concurrency : ( Race Conditions, Database locks, contention points)
• Infrequent functional issues that seem to only occur under load.
• Process crashes
• Not easily reproducible.
BugDay Bangkok 2014
What are the risks we are testing for ? (4)
Reliability: Degradation over time, system becomes slower, less predictable, or eventually fails.
• Memory or Object Leaks
• More frequent Garbage Collection
• Decaying response time
BugDay Bangkok 2014
Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014
“Realism” vs “Simulation Tests” Approaches.
Traditional testing we have to test realistic things!!!
• Will the completed, deployed system support :
• [a,b…) users
• Performing ( e,f…] activities
• At (j,k…] rates
• On mn… configuration Uder rs … external conditions,
• Meeting x,y… response time goals ?
BugDay Bangkok 2014
“Realism” vs “Simulation Tests” Approaches.
Simulations Tests
• Maybe the answer is more control
• Horizontal Scalability makes assumptions let’s use them
• Test Subsets: Single servers, single components, cheaply and repeatedly
• Calibration tests.
BugDay Bangkok 2014
“Realism” vs “Simulation Tests” Approaches.
Suggestions: Build easily repeatable, reliable rapid test.
• Login ( Measure session overhead/footprint)
• Simple Workflows, avoid data caching effects.
• Control variables
• Be ready to repeat/troubleshoot when you find anomalies.
BugDay Bangkok 2014
Agenda
• Why automated Load testing is important ?
• What are the risks we are testing for ?
• “Realism” vs “Simulation Tests” Approaches.
• Performance Testing Incomplete Systems.
• Q/A
BugDay Bangkok 2014
Performance Testing Incomplete Systems.
Showcase : Daily performance testing in BMPS business unit at DSTWS and DSTSystem.
BugDay Bangkok 2014
Approach: Simulation Approach
Suggestions: Build easily repeatable, reliable rapid test.
• Login ( Measure session overhead/footprint)
• Simple Workflows, avoid data caching effects.
• Control variables
• Be ready to repeat/troubleshoot when you find anomalies.
Cheap and Fast Performance Tests :
BugDay Bangkok 2014
Automated Tools
Tools:
• HP Performance Center 11.52
• HP Vugen IDE
• Electric Cloud for CI
• In-house software for generating report.
BugDay Bangkok 2014
Performance Testing Incomplete Systems.
Process and Lifecycle:
BugDay Bangkok 2014
HP Performance Scripting.
BugDay Bangkok 2014
Create Performance Test Scenarios
BugDay Bangkok 2014
Create a Performance Test Workflow.
BugDay Bangkok 2014
Integrate to Continues Integration
BugDay Bangkok 2014
Reporting Early Performance Results.
BugDay Bangkok 2014
The benefits of daily performance testing
• Reduce performance tickets by 70% compare with the previous version.
• Increase performance test efficiency by 150%.
• Reduce performance test effort by 80+% (Compared with the previous project)
• Increase the performance test machines utilization by 450%
BugDay Bangkok 2014
Some Recommendation for Agile Load Testing
• Establish performance goals
• Include performance tests in the build process
• Test incrementally
• Reuse functional tests
BugDay Bangkok 2014
Q/A
