Load/Performance Testing Tools and Techniques

Copyright 2006 by Cordell Vail

Presentation by Joe Towns & Cordell Vail

Load/Performance Testing Tools and Techniques

Caesars Palace, Las Vegas, NV – 4th to 7th June 2006

J


Some of the joke slides you saw in the actual presentation as transition slides, have been removed from this web version of the presentation due to copyright laws. They can only be displayed in a class room setting and not distributed to the public with out permission from the cartoonist. Therefore, the ones I do not have permission to distribute have been removed.

COPYRIGHT NOTICE:

Copyright 2006 by Cordell Vail2

Methodology• Definition of terms

• Creating a baseline for your application

• Know the objective

• Know the development environment

• Only buy what you need

c

Copyright 2006 by Cordell Vail3

• Comparing baseline to results• Uniformity of results

• Trusting the test results

• Possible solutions

• Can we trust market share?

• Testing tool comparisons

• Demo of tools with our results

Methodology

J


Definition Of Terms

It is not so important how you define testing terms….

just that everyone in your organization uses the same definitions

4c


Stress Testing

Tests the server –

Peak volume over a short span of time

5


Stress Testing


Stress Testing


Load Testing

Tests the database –

Largest load the database can handle at one time

6


Load Testing


Volume Testing

Tests the server & the database

Heavy volumes of data over time

(combination of Stress Testing and Load Testing over time)

7


Volume Testing


Performance Testing

Tests user response time

With web applications this is normally the main consideration

8

Performance testing does not include tuning, debugging and fixing, but provides the information to direct those efforts.

Dale Perry


Performance Testing


Benchmark Testing

Compares your testing

standards to the same testing

standards in other similar

organizations in the industry

9J


Benchmark Testing

J


Baseline Testing

Setting testing standards to be

used as a starting point for

comparison later within your

own organization

10J


Baseline Testing


Creating an Application Baseline

Baseline levels of performanceFor individual modulesFor entire application

Used to compare Client/Server to WebSpeed®

Helps identify potential risk due to performance implications

11J


Know the Objective

targets causesStress Test Stress Attack Point(s)

Stress Failure

Buffer, cache Resource leak

Data corruption

Unhandled exception

Race condition\deadlock

Async pattern

p\invokes

Fault Injection

Synchronization

Duration

Torture

Breakpoint

Limit

Load

This example taken froma presentation at the Seattle SASQAG meeting 21 Apr 2005 by Keith Stobie. Diagram created by Reji John. Both work at Microsoft.(Used with permission)

12c


Know the Development Environment

NOTE: The Client Server and the 2 Web Servers areHP Proliant BL 20 PG2 Dual 3GH Xeon with 2GB Ram

13c


Only Buy What You Need

Does the tool matter?

Is it compatible with Progress®?

Is the price right?

14J


Only Buy What You Need

J


Comparing Baseline to Results

How big is this equipment?

15c


Comparing Baseline To Results

Now how big is it?

16c


Comparing Baseline To Results

Apples to Apples

Baseline is your standardfor comparison

17c


Uniformity of Results

Consistent

Repeatable

Understandable

Usable

18J


Uniformity of Results

J


Trusting the Test Results

Here is your typical waterfall19

c



You have to wonder how a bear daresstand there …

20c



Unless you know how high it really is!

21c



What is our perspective?

Can we trust a new tool?

What is our expectation?

22c



It is all a matter of PERSPECTIVE

23c


Possible Solutions

Compare to a known application

Use more than one tool

Outsourcing

24J


Possible Solutions

J


Can We Trust Market Share?

To find the right tool…

Who is using what tools?

How much do we want to pay?

25C


Source:http://www.mercury.com/us/pdf/company/newport_load2000.pdf


26J


Finding a tool

93% of the testing toolmarket shareis controlledby 7 vendors

all chargingvery high prices


27

3

c


Due to budget considerations

we looked at the “OTHER” 7%

for tools that would work with Progress ®


28c


Testing Tool Comparison Links

http://www.vcaa.com/testengineer/links.htm

http://www.testingfaqs.org/t-load.html

http://hammerhead.sourceforge.net/

http://opensourcetesting.org/performance.php

http://www.grove.co.uk/Tool_Information/Choosing_Tools.html

http://www.softwaregatest.com/gatweb1.html#LOAD

http://www.sqa-test.com/toolpage.html

http://www.webservices.org/index.php/ws/content/view/full/102

http://opensourcetesting.org/performance.php

http://sourceforge.net/projects/dieseltest/

29

Testing Tool Comparisons

c


Testing Tool Vendor LinksHere are links to those 7 larger vendors that are holding 93% of the market share

Check out the prices for yourself!

www-306.ibm.com/software/awdtools/tester/performance/index.htmlwww.segue.com/products/load-stress-performance-testing/index.aspwww.mercury.com/us/products/performance-center/loadrunner/www.radview.com/products/WebLOAD.aspwww.quotium.com/qpro_overview_load_testing.htmlwww.empirix.com/default.asp?action=article&ID=418www.compuware.com/products/qacenter/performance.htm

30


c


TestMaker – PushToTest: FREEhttp://www.pushtotest.com/Downloads/

WAST – Microsoft: FREEhttp://www.microsoft.com/downloads/details.aspx?FamilyID=E2C0585A-062A-439E-A67D-5A89AA36495&displaylang=en

LoadTester – AppPerfect: FREEhttp://www.appperfect.com/products/devsuite/lt.html

Testing Tool Findings

31


c


Site Tester 1 – Pilot: $29http://www.pilotltd.com/eng/index.html

Portent Supreme –Loadtesting.com: $279www.loadtesting.com

WAPT - Logasoft: $299http://www.loadtestingtool.com

32


c


Webserver Stress Tool 7 – Paessler: $625www.paessler.com

HOLODECK - SISE: $1,500http://www.sisecure.com/holodeck/learn.shtml

NOTE: Holodeck is a fault injection tool not a normal virtual user testing tool

33


c


Other Options

Testing Onshore Outsourcinghttp://www.veritest.com/services/load_stress.asp

Online Hosted Serviceshttp://servers.aplus.net/loadbalance.htmlhttp://www.webpartner.com/products/st_main.htmlhttp://www.keynote.com/http://www.webmetrics.com/loadtesting.html

NOTE: Hosted or online testing may be available free from your ISP

34


J


• Define Terms

• Baselines

• Uniformity

• Objective

• Environment

• Buying Tools

• Comparing Results

• Trusting Results

• Possible Solutions

• Market Share

• Tool Comparisons

• Demo

35c

TAKE HOME’S


TAKE HOME’S

36


QUESTIONS?

38


Demo of Tools With Our Results

Our load testing tool results

with WebSpeed®

and Progress® Language

39c


Joe Towns, Applications Development Manager/ Project Manager. Joe grew up in the Seattle area, attained his degree in Computer Science and has worked in the field of software development for over 18 years. Joe has worked in various phases of software development, with focus on quality assurance, quality control and configuration management. Joe is currently the Applications Development Manager and Project Manager for the Financial Management development team at the Washington School InformationProcessing Cooperative (WSIPC), where he is pursuing PMI certification. Joe brings a development perspective to the presentation.

The Authors

Cordell Vail, cste, sbs – Quality Assurance Analyst.Cordell grew up in Utah and graduated from the Brigham Young University. He has completed two years of graduate school work at the University of Utah in Interpersonal Communications. He is a Certified Software Test Engineer and Certified School Business Specialist with 10 years experience in manual and automated testing. Cordell has made several presentations on Improving Testing Processes at both local and national conferences. Cordell brings to the presentation a test engineer’s [email protected]

40

mailto:[email protected]


NOTE: The information contained in this presentation and the handout is for use only by the participants who attend our seminar at the Progress Software Exchange 2006 Conference held 4th to the 7th of June, 2006 in Las Vegas, Nevada.

Distribution of this information to anyone other than those attending the seminar is not authorized by the authors. It is for educational purposes of the seminar attendees only. Due to copyright laws, the jokes that were in the original presentation are not included in the handout.

Credits

41


DEMO SLIDES


DEMONSTRATING HOW THE WAPT TOOL WORKS

WITH A WEBSPEED APPLICATION


The tool we selected for our testing was WAPT


To use WAPT first you need to createa profile, which becomes the individuallyrecorded test scripts


Name the PROFILE (this will by your test script)


A profile named TEST 1 is now createdthe actual name in the example below will be605-303-01 - Monroe JH - log in - Allan Shell - SAVE Attendance af

Copyright 2006 by Cordell VailYou will now have to record the test script


When you record the profile you can use thename you want to use


It will then ask you to enter the URL for the web page to be recorded from


It then begins to record all of the Web page mouse clicks as you navigate through the application


Now you have arecoded test scriptthat you can play back

You place the loginscript in the “INITIAL”folder and the logoffscript in the “FINAL”folder so they only runone time in the sequence


Each test script (or as they call them - profiles) has set properties that control it and can be changed


Now, after you have recorded a whole series of profiles,you can run them in different combinations in a scenario

Each scenario is just a shell that holds profiles.

You can run one profile or many in each scenario






You can then add the profiles that you have created that youwant to have run in this particular scenario



The test scripts for each profile you put in the scenario is displayed


Each profile has set properties that can be changedthat control how the profiles are to be run


With the properties in the scenario you can control the test run


You can also control where the logs will be sent and set properties for them


As the test runs you can watch the results being built


The test results are then displayedWAPT has a wonderful reporting system


There are numerous test results that you can then display, print or email to management

Documents

Load/Performance Testing Tools and Techniques