149/038 13-LXE 110 0048 Uen Rev PA1 Standardized Risks and Tours for

Exploratory Testing2011-10-10 1149/038 13-LXE 110 0048 Uen Rev PA1 Standardized Risks and Tours for

Exploratory Testing2011-10-10 1

Facilitating exploratory testing for junior testers and stakeholders

Introduction

• In exploratory testing, a charter sets the agenda or goal of a test session

• This charter is selected to be executed based on current risks

• Both the creation of charters and the identification and understanding of risks is a complex task that can only be performed by senior testers

• One way to simplify these tasks is some form of standardization

• This is a way to transfer knowledge from senior testers to their junior counterparts

Overview

Standardized Risk List

• By standardizing risks in a list it will give junior testers a way of not missing risks that are obvious to more senior testers

• Of course you will never cover the complete risk space with a standardized list, but it is a place to start a risk analysis from

• This standardize list will differ from organization to organization because of difference in complete risk space

• Often certain feature specific risks will also be identified which are outside of the standard risk list

Standard Risk List Example [2]

• Complex: anything disproportionately large, intricate, or convoluted.• New: anything that has no history in the product.• Changed: anything that has been tampered with or "improved“.• Upstream Dependency: anything whose failure will cause cascading

failure in the rest of the system.• Downstream Dependency: anything that is especially sensitive to

failures in the rest of the system.• Critical: anything whose failure could cause substantial damage.• Precise: anything that must meet its requirements exactly.• Popular: anything that will be used a lot.• Strategic: anything that has special importance to your business, such

as a feature that sets you apart from the competition.• Third-party: anything used in the product, but developed outside the

project.• Distributed: anything spread out in time or space, yet who elements

must work together.• Buggy: anything known to have a lot of problems.• Recent failure: anything with a recent history of failure.

Standardized Charter List

• Standardized charters will not only make it easier for junior testers to select charters themselves, but reporting will be streamlined and more easily accessible to managers and project leaders

• Sometimes you will need to create a feature/application specific charter that is unique, but the goal is to minimize the need for these kinds of unique charters by creating a standardized list that cover a large percentage of all possible charters

• The standardized charter list should be a living document that is continuously updated

Standard Charter List Example [1]

• Documentation Tour: Look in the online help or user manual and find some instructions about how to perform some interesting activity. Do those actions. Improvise from them.

• Sample Data Tour: Employ any sample data you can, and all that you can. The more complex the better. • Variability Tour: Tour a product looking for anything that is variable and vary it. Vary it as far as possible, in every dimension

possible. Exploring variations is part of the basic structure of my testing when I first encounter a product. • Continuous Use: While testing, do not reset the system. Leave windows and files open. Let disk and memory usage mount.

You’re hoping that the system ties itself in knots over time. • Feature tour: Move through the application and get familiar with all the controls and features you come across. • Complexity tour: Find the five most complex things about the application. • Claims tour: Find all the information in the product that tells you what the product does. • Configuration tour: Attempt to find all the ways you can change settings in the product in a way that the application retains

those settings. • User tour: Imagine five users for the product and the information they would want from the product or the major features

they would be interested in. • Testability tour: Find all the features you can use as testability features and/or identify tools you have available that you can

use to help in your testing. • Scenario tour: Imagine five realistic scenarios for how the users identified in the user tour would use this product. • Variability tour: Look for things you can change in the application – and then you try to change them. • Interoperability tour: What does this application interact with? • Data tour: Identify the major data elements of the application. • Structure tour: Find everything you can about what comprises the physical product (code, interfaces, hardware, files, etc…). • Money tour: Test the features that users purchase the app for• Rained-out tour: Start and stop tasks, hit cancel, etc. • Obsessive compulsive tour: Perform tasks multiple times, perform tasks multiple times, perform tasks multiple times • Back alley tour: Test the least-used features • All-nighter tour: Keep the app open overnight• Feature/Application specific tour: A tour that is unique for a specific feature application

Traceability between Risk and Charter

• It should be clear in the test session report what risks triggered the execution of a certain standardized charter

Improved Standardized Test Session Report

• In the test session report, it will be clear which standardized charter has been tested, as well as what risks triggered that charter – but also some additional information

• This is not meant to replace qualitative information in the test session reports, only to complement this information and in some ways quantify it

Software Quality Attributes [3]

• Each test session is mapped towards one or more software quality attributes depending on which attributes the session covered

• Which attributes are used should depend on the software characteristics that are important for the organization

• Example Software Quality Attributes could be:

• Functionality - A set of attributes that bear on the existence of a set of functions and their specified properties. The functions are those that satisfy stated or implied needs.

• Reliability- A set of attributes that bear on the capability of software to maintain its level of performance under stated conditions for a stated period of time.

• Usability - A set of attributes that bear on the effort needed for use, and on the individual assessment of such use, by a stated or implied set of users.

• Efficiency - A set of attributes that bear on the relationship between the level of performance of the software and the amount of resources used, under stated conditions.

• Maintainability - A set of attributes that bear on the effort needed to make specified modifications. • Portability - A set of attributes that bear on the ability of software to be transferred from one

environment to another.

Aggregated Project Status Report

• One of the main advantages of standardization is to generate reports that are easily understood by stakeholders without test experience

• The numbers in the cells in the matrixes on the following slides indicate how many charters have been executed within a specific test area, categorized by either standardized charter type or software quality attribute

• The color of the cells indicate the quality status of the test area in that specific category

Aggregated Project Status ReportStandardized Charter View

Documentation Tour

All-nighter tour

Feature tour

Complexity tour

User tour Back alley tour

Test Area 1 1 1 3 2 4 1

Test Area 2 2 1 4 3 5 1

Test Area 3 1 2 3 1 3 2

Test Area 4 1 0 2 2 3 1

Test Area 5 1 1 1 1 1 1

Test Area 6 1 1 5 2 1 2

Test Area 7 2 0 4 3 1 1

Further Testing Needed

Failed Passed

Aggregated Project Status ReportSoftware Quality Attribute View

Functionality Reliability Efficiency Usability Portability Maintainability

Test Area A 10 4 8 2 1 1

Test Area B 20 7 5 3 1 0

Test Area C 13 3 7 1 0 0

Test Area D 15 0 4 2 0 0

Test Area E 17 4 3 1 0 0

Test Area F 10 7 5 2 1 1

Test Area G 24 2 7 3 1 1

Further Testing Needed

Failed Passed

Defect Connection to Charters

• It should be possible to click on of the cells in the matrix and then receive the corresponding list of charters and issues connected to those charters

7

Charters Defects Defect Status

Charter A Defect 1 Fixed

Charter B Defect 2, 3

Fixed, Ongoing

Charter C -

Charter D -

Charter E Defect 4 Under Investigation

Charter F -

Charter G -

Conclusion

• For a senior tester, a standardized risk list is not necessary, but will at least not hinder the work – for a junior tester it can be a great help in not missing common critical risks

• Standardized reporting will help project leaders and managers better understand the project status as a complement to the qualitative information the tester provides

• Easy traceability from risk to test session to test results to defect is also valuable

Reference

[1] Of Testing Tours and Dashboardshttp://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards/

[2] Heuristic Risk-based Testinghttp://www.satisfice.com/articles/hrbt.pdf

[3] ISO/IEC 9126http://en.wikipedia.org/wiki/ISO/IEC_9126