Performance testing for Automation QA – why and how?

Kovalenko Andrey

Andrey Kovalenko6 last years in IT - again 3 years of developing and implementing distributed protected networks3 last years - performance engineer kovalenko.andrey.odessa

About

What is performance?

Load testing - to verify application behavior under normal and peak load conditions.

Stress testing - To determine or validate an application’s behavior when it is pushed beyond normal or peak load conditions.

Capacity testing - To determine how many users and/or transactions a given system will support and still meet performance goals.

Why do we need performance?

• Based non-functional aspects of the software system• Non functional testing should be performed after

functional testing• Almost impossible to run manually• Describes HOW GOOD the product works

Performance VS Functional testing

• Based on functional specification• Functional testing is executed first• Easy to do manual testing• Describes WHAT the product does

Functional manualFunctional manual

PerformancePerformance

Functional AutomationFunctional Automation• Based on functional specification• Automation should be performed after manual

functional testing• Impossible to run manually• Describes WHAT the product does

Software Testing Life Cycle

STLC for performance testing

1. Identify requirements1. Features related to performance – with devs2. Modules related to performance – with devs and FA/SA3. Analyzing application’s tiers

2. Acceptance criteria (Performance KPIs):1. Main TRT values – from your FA and public sources2. Main system resources threshold – from your experience

3. Strategy planning and design:1. Customer scenario use – from your FA2.2. Transaction defining Transaction defining – from your experience and FA3. Scripts writing4.4. Production-like environment establishing Production-like environment establishing – can be a problemcan be a problem5.5. Monitors for resources – native, from specific toolsMonitors for resources – native, from specific tools

STLC for performance testing

4. Execute & Analyze1. Run performance scenario2. Collect results and resources’ consumption

data3. Analyze results, comparing with KPIs4.4. Analyze source code to find possible Analyze source code to find possible

bottlenecksbottlenecks5.5. Analyze logs, heap GC, dumps Analyze logs, heap GC, dumps 6.6. Create recommendationsCreate recommendations for application for application

tuning – for application server, code, tuning – for application server, code, hardware, softwarehardware, software

7. Prepare reports for stakeholder

• Multi-user scenario• Interested in response time• Should be ran on production-like env• Resources must be monitored,

application should be profiled

Performance VS Automation

• Single user• Interested in response data• Can be ran on any hardware• No needs to monitor resources

AutomationAutomation PerformancePerformance

Automation->Performance(pretty straight ahead)

1. Exclude from you automation scripts data validation2. Allocate transactions you will measure in your code3. Frame these pieces of code in time measurement

constructions4. Get environment (real or cloud)5. Prepare set of monitors for system and application

resources6.6. Get a headache with running n scripts in parallel Get a headache with running n scripts in parallel 7.7. After running collect results and prepare report for After running collect results and prepare report for

stakeholderstakeholder

Automation->Performance(tricky way)

1. If you use NUnit, Junit, Selenium or HP UFT2. If you use Eclipse or VisualStudio3. HP LoadRunner has plug-in with ability to transfer your

automation scripts to performance – in this case you will be able to run performance scenario and create reports automatically

4. Get environment (real or cloud)5. Run scenario and get reports automatically

**LoadRunner has community license for 50 users for freeLoadRunner has community license for 50 users for free

Automation->Performance(tricky way)

Automation->Performance(tricky way)

Thank You!