Non-Functional Testing

Last update: 21-05-2014

Philippe CHARMANcharman@fr.ibm.com

http://users.polytech.unice.fr/~charman/

Non exhaustive list of non-functional testing

• Performance Testing– Load testing– Stress Testing– Endurance Testing

• Usability Testing• Accessibility Testing• Compatibility Testing • Compliance Testing• Internationalization and localization Testing• Doc & Training Testing• Recovery Testing• Etc.

Load Testing

• Load testing is the process of putting demand on a system or device and measuring its response

• Some examples:– simulating multiple users accessing a web server– giving a huge document to a word processor– subjecting a mail server to a large amount of e-mail

traffic– writing and reading data to and from a hard disk

continuously• It helps to identify the maximum operating capacity of an

application as well as any bottlenecks and determine which element is causing degradation

• Synonyms: volume testing, scalability testing, etc.

Load Testing

Stress Testing

• Testing used to determine the stability of a system

• Testing beyond normal operational capacity, often to a breaking point, in order to observe the results

• The goals may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space),

Endurance Testing

• Kind of load testing over a longer period of time

• During endurance tests, memory utilization is monitored to detect potential leaks

• Synonyms: reliability testing, soak testing

Performance Tests

Main types What we are mainly concerned at

Load Tests

Scalability Tests

System response time, big memory peak, bottlenecks

Stress Tests Will it crash ?

Endurance Tests Reliability TestsSoak Tests

System response time, memory/resource consumption over a long period of time

Usability Evaluation Methods

• Usability : how well users can learn and use a product to achieve their goals and how satisfied they are with that process

• How to evaluate usability:– Ease of learning– Efficiency of use– Memorability– Error frequency and severity– Subjective satisfaction

• Some evaluation methods:– Inspection– Testing– Inquiry– Rapid prototyping

Usability Inspection

• Usability inspection is aimed at finding usability problems in the design,

• An evaluator inspects a user interface

• Some inspection methods:– Cognitive walkthrough– Heuristic evaluation

Usability Testing

• Technique used to evaluate a product by testing it on real users

• How many users to test ?• Eye Tracking

Eye Tracking Demo

Accessibility Testing

• Software accessibility is a general term used to describe the degree to which an application is available to as many people as possible

• Access issues:– Visual impairments – Motor skills – Hearing impairments– Photosensitive epilepsy

Accessibility Testing

• Typical test cases:– Make sure that all functions are available via keyboard only (do

not use mouse)– Make sure that information is visible when display setting is

changed to High Contrast modes– Make sure that screen reading tools can read all the text

available and every picture/Image have corresponding alternate text associated with it

– Make sure that product defined keyboard actions do not affect accessibility keyboard shortcuts

– Make sure that Web pages do not contain anything that flashes more than two times in any one second period

– etc ...

Accessibility Testing

• Do a simple test:– Start menu > Control Panel > Accessibility Options >

Display > – Check the checkbox "Use High Contrast"– Are your main applications (web browser, email reader,

etc.) still usable ?

• Another test:– Install the tool Color Oracle from http://colororacle.org/– Is it still easy to distinguish colors differences with your

main applications?

Color impaired vision

Compatibility Testing

• Testing conducted on the application to evaluate the application's compatibility with the computing environment that can be composed of:– Hardware platform : x86, x64, UltraSPARC-I,

POWER7 etc. – OS: Windows XP 32-bit, Debian 4.0, Solaris 11, etc.– Database: Oracle, Sybase, DB2, etc.– Browser: IE, Firefox, Chrome,– Peripheral: printer, scanner, – Etc.

• Compatibility Matrix• Pairwise Testing methods

Compatibility Matrix

Backward & Forward Compatibility Testing

• Backward compatibility

• Forward compatibility

Compliance Testing

• Some exemples:– Blue Wash d’IBM– HTLM5 Conformance Tests – Coding rules in C/C++ from MISRA, JSF, etc.– Etc.

C/C++ Standards

• Code should be:– Portable– Reliable– Maintenable– Testable– Reutilisable– Extensible– Readable

C/C++ Standards

• Some examples

• Avoid long and complex functions. • Rationale: long functions are difficult to understand and

to test

• Do not use #define to use a constant• Use "const«  instead

• Use #include to include .h only and not .cpp files

Compliance Tests

• More:

• MISRA web site (Motor Industry Software Reliability Association)

http://www.misra.org.uk/

• JSF web site (Joint Strike Fighter)http://www.jsf.mil/downloads/down_documentation.htm

• Web site of the tool Understandhttp://www.scitools.com/

Internationalization& Localization Testing

• Globalisation = i18n + l10n• I18N & l10n

– OS– Keyboard– Display text– Date format– Number format– Etc.

Internationalization& Localization Testing

• To be performed once functional testing has been completed and all the functionality related defects are identified and removed

• 2 phases:First, to make sure that application’s GUI or

functionality will not be broken with the translated text

Second to make sure that translation of all the strings have happened properly

Recovery Testing

• Testing how well an application is able to recover from crashes, hardware failures and other similar problems.

• Recovery testing != reliability testing

References

• Srinivasan Desikan & Gopalaswamy Ramesh, Software Testing: Principles and Practice, Pearson Education

• J. Nielsen, "Heuristic Evaluation". In Jakob Nielsen and Robert L. Mack, editors, "Usability Inspection Methods". John Wiley and Sons, Inc. 1994.

• Mauro, C.L.: Professional usability testing and return on investment as it applies to user interface design for web-based products and services

More information …

• http://www.usability.gov/• http://www.usabilityhome.com/• http://www.testinggeek.com/

• http://www.misra.org.uk/