Exploratory Testing · Exploratory testing: A Definition Testing is context-dependent Testing as a sapient activity Heuristics and questioning strategies . What is a computer program?

Exploratory Testing

Software Testing: INF3121 / INF4121

27.04.2017 / Yulai de Meer Fjeld

Learning Outcomes

Understanding the system to be tested

Building quality, not only checking for it

Scripted testing vs. exploratory testing

Let’s play: The Witness

Exploratory testing: A Definition

Testing is context-dependent

Testing as a sapient activity

Heuristics and questioning strategies

What is a computer program?

Computer Program: A Definition

“A set of instructions for a computer”

“A collection of instructions to perform a

specific task when executed by a

computer”

“A list of instructions that tell a computer

what to do”

Is the definition complete?

What is a house?

House: A Definition

“A collection of construction materials

assembled according to house-design

patterns”

House: Other Definitions

“A building for human habitation”

“A building that functions as a home”

“A building in which people meet for a

particular activity”

“A residence for a community”

Computer Program: A New Definition What are computer programs?




Missing factors



Missing factors

Intent → Why?

Stakeholders → Who?



Missing factors

Intent → Why?

Stakeholders → Who?

Cem Kaner’s alternative definition

The purpose of a program is to provide value to people

“A communication among several individuals and computers, who are

distributed across space and time, that contains instructions that can be

executed by a computer”

Quality To provide value, we expect a certain degree of quality

A computer program is far more than its code

A software product is far more than the instructions for the device




Quality is far more than the absence of errors

Have you ever bought a program simply because it was bug-free?






Most likely, you have not

Instead, we use programs because we believe they provide some benefit






Most likely, you have not

Instead, we use programs because we believe they provide some benefit

Quality is value experienced

Improvement | Solution

How to strive for quality?

Quality Assurance

What is software testing?

Software Testing: A Definition Generally defined as:

“A process of executing a program or application with the intent of finding

software bugs”

Software Testing: A Definition Generally defined as:

ISTQB defines software testing as:

“A process of executing a program or application with the intent of finding

software bugs”

“The process consisting of all lifecycle activities, both static and dynamic,

concerned with planning, preparation and evaluation of software products

and related work products to determine that they satisfy specified

requirements, to demonstrate that they are fit for purpose and to detect

defects”

Testing: An Investigation Testing is far more than writing code

Testing is investigative work



Testers examine

Code and documents

Systems

People

And the relationships between them



Testers examine

Code and documents

Systems

People

And the relationships between them

We need to understand the objective

Why are we testing in the first place?

Testing: A Search for Information Testing is always a search for information

What information do we search for?


Find important bugs in order to get them fixed

Assess the quality of the product

Assess conformance to specifications

Find safe scenarios for use of the product






How do we search for information?






How do we search for information?

We examine the work products

We compare the results to expected outcomes

We explore!

Scripted testing

Scripted Testing A test script specifies

Test data and operations

Expected results

Comparisons to be made by humans / machines

Scripted Testing A test script specifies

Test data and operations

Expected results

Comparisons to be made by humans / machines

Scripts can control

Manual testing by humans

Automated test execution or comparison by a machine

Scripted Testing: Example Testing login functionality on a simple site

Test data

?


Test data

Username and Password

Test procedure

?


Test data

Username and Password

Test procedure

Go to site

Type in username and password

Click “Login”

Scripted Testing: Example Let us write a manual script for testing login functionality

Note that we have not included error handling (e.g. incorrect username / password)

Scripted Testing Benefits of scripted testing

What are the benefits of using scripts in testing?


Division of labour

Planning / Analysis / Implementation / Execution / Evaluation


Division of labour


Careful thinking about the design of each test

Created from requirements, code, etc.

Covers all attributes of the system that have been defined


Division of labour


Careful thinking about the design of each test

Created from requirements, code, etc.

Covers all attributes of the system that have been defined

Review by other stakeholders

Reuse- and repeatability

Other testers can replicate the test in no time

Scripted Testing Problems with test scripts

What are the problems with test scripts?


Programs fail in many ways, some of which may seem inexplicable



Scripts are “hit and miss”

Unintentional blindness

Pay attention to certain things, not to others






Computers focus only on what they are programmed to look at

Blind by design → Can leave many stones unturned

With a script, you miss the same things every single time






Computers focus only on what they are programmed to look at

Blind by design → Can leave many stones unturned

With a script, you miss the same things every single time

We have to explore!

The need for exploration

The Witness: An Example

“Explore an abandoned island”

A video game by Jonathan Blow

Released in 2016

Gameplay

Walk around an island

Solve puzzles

Let’s play!

The Witness: On Exploration What can we learn from The Witness?

The Witness: On Exploration

Failure

Without failing, there can be no learning

The need to embrace


Failure


Curiosity

Exploring leads to discovery

The need to embrace


Failure


Curiosity


Logic

Acknowledge, and adhere to the rules

The need to embrace


Failure


Curiosity


Logic

Acknowledge, and adhere to the rules

Nonsense

Discard the rules; they are meant to be broken

The need to embrace

What is Exploratory Testing?

Exploratory Testing As defined by the ISTQB

“An informal test design technique where the tester actively controls

the design of the tests as those tests are performed, and uses

information gained during testing to design new and better tests”

Exploratory Testing As defined by the ISTQB

However, it was a term first coined by Cem Kaner

“An informal test design technique where the tester actively controls

the design of the tests as those tests are performed, and uses

information gained during testing to design new and better tests”

“A style of software testing that emphasises the personal freedom

and responsibility of the individual tester, to continually optimise the

value of his/her work, by treating test-related learning, test design,

test execution, and test result interpretation as mutually supportive

activities that run in parallel throughout the project”

Testers Explore For each software / application:

Doing Exploratory Testing Anything that can guide us in

What to test / How to test / Recognise a problem



Creating / fashioning / constructing according to plan

Design is not scripting





Executing the test and collecting the results

Can be automated or manual





Executing the test and collecting the results

Can be automated or manual

What we learn from the system under test

Information about the system

Information about how we are testing the system

Testing is Context-Dependent Software systems are different

How are they different?


Intended for different users

Used in different ways

Pose different risks


Intended for different users

Used in different ways

Pose different risks

Require different focus in testing

We cannot test all systems the same way

We need information about the system under test

We let this information guide the test effort

Testing is Context-Dependent Testing depends on the following

Testing is Context-Dependent Testing depends on the following

Even for the same product → Testing differs from release to release

First release → Positive testing | Subsequent releases → Performance testing

Patch → Negative, regression testing

Testing is Context-Dependent Questions to understand the context and build quality

What do we need to understand?


Who are the customers and who are the users?

Is the customer the same as the user?

How can we describe the typical user?





What will the product be used for?

How important is feature X to the customers?

Can they live without this feature?


Who are the customers ad who are the users?






What is the time to market?

How much is invested in the development / testing of the product?








What is the time to market?

How much is invested in the development / testing of the product?

What are the customers’ and users’ expectations?

Testing: A Sapient Activity Sapience

“The ability to act with judgement”

Homo sapiens → “The Wise Man”



Testing always happens under conditions of uncertainty

Good testing stands up to scrutiny

Without scrutiny, software testing would be an easy task

One could test badly, and no one would ever notice / care



Testing always happens under conditions of uncertainty

Good testing stands up to scrutiny

Without scrutiny, software testing would be an easy task

One could test badly, and no one would ever notice / care

The test effort will (at some point) be judged

Either by users, management, peers, etc.

This makes testing difficult!

Sapience in Testing Example requirement

“The system shall operate at an input voltage range of 100-250 VAC”

How would you test this?




Poor answer

“Test with an input voltage in the range of 100-250 VAC”




Poor answer

“Test with an input voltage in the range of 100-250 VAC”

An improved, but still poor answer

First: “Test with input voltage below 100 VAC”

Then: “Within 100-250 VAC”

Last: “Above 250 VAC”

Sapience in Testing Good testing should answer some fundamental questions

What are some good questions to ask?


What happens when the system receives voltage outside its range?

Does it shut down? Blow up? Give erroneous results / error message?

If it is designed to pause and recover; what actually happens?





Is there a special gear that can supply voltage at different levels?

If so; we could probably use it

What does it cost? / What are the risks?





Is there a special gear that can supply voltage at different levels?

If so; we could probably use it

What does it cost? / What are the risks?

What happens if there is a power spike?

What happens if the power input suddenly drops and then recovers?


How long can the product run above or below the rated levels?

Run the product for a long time at the upper and lower voltage ranges




What happens when we turn the product on and off?

Does the product need time to reach some state before use?

If so; how long does that take?







Run the system with varying input voltages

Varying slowly and rapidly as we run other tests







Run the system with varying input voltages

Varying slowly and rapidly as we run other tests

What are the correlations between input voltage and output?

Can we improve this process?

Heuristics How can we structure our problem-solving skills?

Why is this important?


Testing happens under constraints → Limited resources

Exhaustive testing is impossible

Need to speed up the process of finding satisfactory solutions





Heuristic → Heuriskein (greek) → Find

Experience-based technique

Problem-solving, learning, and discovery





Heuristic → Heuriskein (greek) → Find

Experience-based technique

Problem-solving, learning, and discovery

Example heuristics

Rule of thumb | Educated guess | Intuitive judgement | Stereotyping

Heuristics: Different Types

Guideword Heuristics Simple strategy for making decisions and finding solutions

Does not provide the answers

Instead: Suggests key elements to consider

Keywords direct attention to the details that are most likely to matter





Based on experience and probability

Ensure most crucial aspects are covered

Regardless of how little knowledge the tester has about a program





Based on experience and probability

Ensure most crucial aspects are covered

Regardless of how little knowledge the tester has about a program

Words or labels to access your full spectrum of knowledge

Used to effectively analyse and approach a problem

Heuristics: SFDPO What is it?

What files does it have? Is it one program or many?



What does it do?

What are its functions? What kind of user interface does it have?



What does it do?


What does it process?

What does the input / output look like? What are the modes / states?



What does it do?




What does it depend on?

What operating systems does it run on? Special configurations?



What does it do?




What does it depend on?

What operating systems does it run on? Special configurations?

How will it be used?

Who will use it? Where and how will it be used?

Heuristics: Cheat Sheet

http://testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

Trigger Heuristics Ideas associated with an event or condition

Recognise when it is time to take further action

Shift focus to think in a different way

Trigger Heuristics Ideas associated with an event or condition

Recognise when it is time to take further action

Shift focus to think in a different way

Emotions and feelings are powerful sources for trigger heuristics

Impatience → An intolerable delay?

Frustration → A poorly conceived workflow?

Surprise → Inconsistency with expectations?

Confusion → Unclear interface? Poor testability?

Annoyance → A missing feature?

Boredom → An uninteresting feature / test?

Heuristics for The Witness Guideword heuristics

Dots → Cover all

Different coloured squares → Separate them

Trigger heuristics

Frustration → Use the environment, find the right perspective

Confusion → Check your assumptions

Tiredness → Time for a break?

Heuristics in Testing We use heuristics the same way when testing


Notice what you do when trying to approach a problem

What do you do?

What made you succeed?

What made you fail?



What do you do?


What made you fail?

Find an evocative label or phrase for that thought or process

Should be easy to remember



What do you do?


What made you fail?

Find an evocative label or phrase for that thought or process

Should be easy to remember

Try using the labels next time you approach a problem

See how effective they can be!

Questioning Strategies Each test case is viewed as a question we ask the product

“Can you do this?”

“What happens if I do that?”




Shifting the focus from designing test cases

To designing a set of questions

Use heuristics to come up with appropriate questions




Shifting the focus from designing test cases

To designing a set of questions

Use heuristics to come up with appropriate questions

Phoenix Checklist

Developed by the CIA → Guides the operatives when facing a problem

Context-free questions → Can be applied to exploratory software testing

Phoenix Checklist

The problem

Why is it necessary to solve the problem?

What is the unknown?

What is it you don’t yet understand?

What is not the problem?

What is the information you have?

Sufficient? Redundant? Contradictory?

Should you draw a diagram of the problem?

Can you separate various parts of the problem?

Have you seen this problem before?

Do you know of related problems?

Can these methods be used?

What are the most probable cases you can imagine?

The plan

Can you solve the whole problem? Part of it?

Can you derive something useful from the given info?

Have you used all the information?

Can you separate steps …

in the problem-solving process?

What should be done?

How should it be done?

What do you need to do at this time?

Who will be responsible for what?

What milestones can best mark your progress?

How will you know when you succeed?

How will you know when you are done?

Phoenix Checklist

The problem

Why is it necessary to solve the problem?

What is the unknown?

What is it you don’t yet understand?

What is not the problem?

What is the information you have?

Sufficient? Redundant? Contradictory?

Should you draw a diagram of the problem?

Can you separate various parts of the problem?

Have you seen this problem before?

Do you know of related problems?

Can these methods be used?

What are the most probable cases you can imagine?

The problem: Testing Wikipedia

The world’s most popular online encyclopaedia

Do not know all of the site’s functionality

The internal structure of the site

Testing the graphical user interface

Access to the site

Access to the database

Could be helpful

Yes → Content / User profiles / Tools

Yes

Testing other websites

Of course

Information search and editing

Phoenix Checklist

The Plan: Testing Wikipedia

Yes

Given info provides an overview of the functionality

Not as of yet

Yes

Plan, Analyse, Implement, Execute, Evaluate

Categorise functionality into areas of use

Create specific tests for each usage area

Examine Wikipedia’s functionality to identify patterns

Has yet to be decided

Completed testing of various usage areas

When my efforts pass someone else’s scrutiny

When all usage areas have been tested

The plan

Can you solve the whole problem? Part of it?

Can you derive something useful from the given info?

Have you used all the information?

Can you separate steps …

in the problem-solving process?

What should be done?

How should it be done?

What do you need to do at this time?

Who will be responsible for what?

What milestones can best mark your progress?

How will you know when you succeed?

How will you know when you are done?

Assessing Exploratory Testing Exploratory testing can prove hugely beneficial

However, it must be applied in the right context

When can it be useful?



Lacking specifications and documentation

When requirements are not available

When under extreme time pressure






Other considerations

What are the disadvantages of exploratory testing?

What should you pay extra attention to?






Other considerations

Quality of test effort depends on the tester’s skill

Limited by the domain knowledge of the tester

Should be used to complement other test techniques

Summary

Scripted testing Exploratory testing

Closing Remarks

The lecture slides were based on

Previous lecture slides by Raluca Florea (2016)

Black, R., van Veenendal, E., Graham, D. (2012). Foundations of Software

Testing: ISTQB Certification 3E. Cengage Learning.

IEEE 829: Standard for Software and System Documentation

Documents

Exploratory Testing · Exploratory testing: A Definition Testing is context-dependent Testing as a sapient activity Heuristics and questioning strategies . What is a computer program?