Exploratory testing – basics, experiences and future? · Exploratory testing – basics, experiences and future? Knowit 1 18 April 2018 at TestCon Moscow, Russia ... • Treasurer

Exploratory testing –basics, experiences and future?

Knowit 1

18 April 2018 at TestCon Moscow, Russia

Kari Kakkonen

Knowit Oy (Finland)

[email protected]

Twitter @kkakkonen

Linkedin karikakkonen

Kari Kakkonen

ROLES

• Knowit Oy, Director/Quality and Competences, Lead Consultant, Trainer and Coach

• Treasurer of ISTQB Executive Committee

• Chairman of Finnish Software Testing Board (FiSTB)

• Auditor of Robot Framework Foundation.

ACHIEVEMENTS

• Influencing testing since 1996

• Ranked in 100 most influential IT persons in Finland (Tietoviikko magazine)

• Great number of presentations in Finnish and international conferences

• TestausOSY/FAST founding member.

• Co-author of Agile Testing Foundations book

• Regular blogger in Tivi-magazine

EDUCATION

• ISTQB Expert Level Test Management Full & Advanced Full & Agile Tester certified

• DASA DevOps Fundamentals, Scrum Master and SAFe certified

• SPICE provisionary assessor certified

• M.Sc.(Eng), Helsinki University of Technology (present Aalto University), Otaniemi, Espoo

• Marketing studies, University of Wisconsin-Madison, the USA.

BUSINESS DOMAINS

• Wide spread of business domain knowledge

• Embedded, Industry, Public,

• Training, Telecom, Commerce,

• Insurance, Banking, Pension

SERVICES

• ISTQB Advanced, Foundation and Agile Testing +

Knowit Quality Professional

• DASA DevOps Fundamentals

• Quality & Test process and organization

development, Metrics

• Agile testing, Scrum, Kanban, Lean

• Leadership

• Test automation, Mobile, Cloud, DevOps

• Quality, Cost, Benefits.

Twitter: @kkakkonen

LinkedIn: fi.linkedin.com/in/karikakkonen/

Knowit 2

What is it?

Knowit 4

Characterization of exploratory testing

• ”Planning and execution of testing is done at the same time”

(After James Bach)

• Test cases are not necessarily documented even afterwards (Cem Kaner)

• In exploratory testing, test design and test execution occur at the same time, guided by a prepared test

charter. (ISTQB Agile Tester syllabus)

• Testing is done iteratively piece by piece

• Continuous learning and interpretation

• Utilization of knowledge gained from experience

5© Copyright Knowit Oy 2015 | Confidential

Exploratory testing - terms

• Test Charter

• Divide your testing areas and assign missions

to each

• Testing area: a bunch of functionalities

• Exploration

• Adventure may go to sidetrack as long as you

come back to mainroad again (Kaner)

• Testing session

• Duration about ½ - 2 hours

• Why?

• Time span of concentrated work is about

20 minutes

• Getting back to work takes about 20

minutes


Learning in exploratory testing


Testing

Opinion-forming

Reporting

Designing actions

Observations

After reference: Psychology of Usability, Sinkkonen et al.

Learning from the system and customers

• What has changed or changes frequently?

• What do the customers want?

• What has been defined in an ambiguous way?

• Where do faults cluster?

• Weaknesses in the platform or programming language

• System dependencies


Reference: Lessons learned in software testing. Kaner, Bach, Pettichord

Learn from other testers, designers, from yourself

• What kind of errors do certain programmers make and how to report to and communicate with them

• What typical errors can there be in the system

• What functionalities have been built in a hurry

• What have you misunderstood and what is typically misunderstood

• How can the system be tested (especially in pair testing!)


Why exploratory testing?Knowledge-based perspective

• Exploit the natural diversity of people in testing *)

• ”Do not plan for store”

• Systematic variation of testing

• Quick feedback to designers / developers intensifies learning process *)

• Spread the knowledge of a tester of a special area


*) Reference: Exploratory testing: A multiple case study. Itkonen,

Rautiainen

Why exploratory testing?Testers’ approach

• Want to add more test cases and increase the coverage of the tests *)

• For defining the degree of automation of tests

• Quick overview of the quality *)

• Testing the side effects of changes – scripted testing can end up testing only the documented features *)

• The problem in regression testing is the selection of test cases, which requires user experience and

understanding of the system

• When the test documentation can not be written in advance *)



Rautiainen

Why exploratory testing?Management approach

• Low management costs of test documentation

• Finding out the features of a poorly-documented component

• Optimizing the productivity of testing?

• When the available workload is limited *)

• When you want to train the customer support at the same time



Rautiainen

Where exploratory testing fits

13

Different styles of testing

Adhoc testing

Exploratory Testing

And Session-Based

Test Management

High level test cases

design and

execution

Detailed test cases

design and execution

ET + SBTM

High level TC

Detailed TCAutomation

Automated testing

and ATDD

Exploratory

Testing

ET

Syste

ma

ticn

ess o

f te

stin

g

Adhoc

ET Scripted testing

Exploratory testing as part of test strategy

14© Knowit Oy Confidental 2012

Pure (100%)

scripted testing

Scripted testing

Exploratory testing

Pure (100%)

exploratory testing

Where do you put the

focus of your own

testing?Maybe here?

How is it done?

Knowit 15

Step by step

© Copyright Knowit Oy 2015 | Confidential

Plan

• Test charter

Test sessions

• Notes / Log

• Bugs

Debriefings

• Dashboard

”1 hour”Most of

the time

5 min

regularly

Exploratory testing in Prague – find a park

• Test charter

• ”look for green”

1713.4.2018 © Copyright Knowit Oy 2013 | Confidential | Version 1.0


• Test execution log

• Dashboard



• Defect report


Purpose of test charter(s)

• What will be tested?

• What documents are available?

• What kind of errors are being sought?

• Tasks and what test techniques will be used?

• What will be expected from test sessions?

• A dashboard?

• Defect reports?

• Some key test cases?


Reference: A practioner’s guide to software test design. Copeland

Ref. Exploratory testing: A multiple case study. Itkonen, Rautiainen

Test design within test charter

• Define the testing areas of the test object

• E.g. 20-40 areas covering whole system under test

• Divide each area to one or more test sessions

• Test charter works as a roadmap for each test session

• Define test cases to be documented as a by-product, if any

• heuristic: less than 10% of all tests

• Write down test ideas and/or high level test cases


Charter classical example

Area Coverage

and

working

hours

Practice Documents Result possible

errors

Risks

R1. Customer’s

all selected

items are not

added to order,

Effect: 20

eur/buyer,

probability 5%

R2. Order can

not be

completed after

interruption, 5,

probability: ??

Main page 100%

Path coverage

(direct paths) and

the most common

(80% used) loops

10h

Scripting

with

Functional

Tester-tool

Main page display

description

document, navigation

map (COMING

FROM

DEVELOPMENT)

All pages and

the shopping

cart are

available

Shopping

cart

5h Shopping cart-

UC.doc (use case)

Shopping cart

can be used in

the same way

as a real

shopping cart

The same

product can

not be added

to shopping

cart several

times

Emptying the

shopping cart

causes an

execption

R1

Order ? Order-UC.doc? R2


Reference: James Bach

Test Charter example ISTQB


• Actor: intended user of the system

• Purpose: the theme of the charter including what particular objective the

actor wants to achieve, i.e., the test conditions

• Setup: what needs to be in place in order to start the test execution

• Priority: relative importance of this charter, based on the priority of the

associated user story or the risk level

• Reference: specifications (e.g., user story), risks, or other information

sources

• Data: whatever data is needed to carry out the charter

• Activities: a list of ideas of what the actor may want to do with the system

and what would be interesting to test (both positive and negative tests)

• Oracle notes: how to evaluate the product to determine correct results

• Variations: alternative actions and evaluations to complement the ideas

described under activities

Reference: ISTQB Agile Tester syllabus

Charter as an excel


Test charter and test log in mindmap (Xmind)


Documents supporting testing

• Charter

• List of different testing strategies

• Lists of heuristics

• List of typical errors

• Kaner’s bug taxonomies*)

• Legal notices, standards, de facto-standards

• Requirements and design documentation of the system

• Self made description and models of the system behavior

• User guide *)


*) Reference: Testing Computer Software. Kaner et al.

**) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen

Kaner’s bug taxonomies

• User interface errors

• Error handling

• Boundary-related errors

• Calculation errors

• Initial and later states

• Control Flow errors

• Errors in handling or interpreting data

• Race conditions

• Load Conditions

• Hardware

• Source and version control

• Documentation

• Testing errors


Reference: Testing computer software, p. 60 – 64

Exploratory Testing - Heuristics

• A set of heuristics can be applied when testing.

• A heuristic can guide the tester in how to perform the testing and to evaluate the results, few examples

below


Heuristics Examples

Boundaries: Approaching the Boundary e.g. almost too big, almost too

small, at the Boundary

CRUD Create, Read, Update, Delete

Configuration variations Varying the variables related to configuration e.g. Screen

Resolution; Network Speed, Latency, Signal Strength; etc.

Interruptions e.g., log off, shut down, or reboot

Source: Elisabeth Hendrickson www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

http://www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

Performance attitude - professional working

• Keep the mission, your targets of testing in mind

• Write down observations and questions about the system

• Report in a disciplined and systematical way

• During the execution, write only the most essential test cases and in high level

• Test cases can be refined later, if needed


Tests are designed during execution

• Design test on the basis of charter its initial test

ideas

• Define tests from any questions in your mind

• A surprising situation may indicate an error:

Utilize the surprise effect!

• Use heuristics e.g.

• ”Backwards thinking”: ”This button saves

the definition text. I wonder what other ways

are there for saving the text?”


Note taking: test execution logs

• Keep a test execution log / notes

• Keep track of the tests carried out

• Main thing is that executed tests are noted down, not the detail

• You may create scripted test cases from some of the tests, add detail

• Note what test data you used in your tests

• You may record your execution with a suitable tool, only need to add your comments

• Add status to your notes e.g. with color-coding

• Write down also notes for test session post-mortem and your own learning


Note taking: defect logs

• After session work out defect reports for defects you found, based on your notes

• In defect reporting, traceability to the requirements must be maintained so that coverage can be

evaluated

• A well written execution log is the best evidence of the existence of a fault

• Report a bug clearly, so that the failure can be repeated

• You may use defect reporting systems


Testing dashboard as a test report - an example


Test area Workload Coverage Quality

level/risks

Comment

Main page !Interrupted High,

5hVery high [all

parts + stress

tests etc..]

49: 1435, 36: 1469,

42: 1501

wait for more

pictures of

user interface

Shopping

cart

!Started

High, 2h (reserved

4h)

Low [main parts

to testing] (High)

81: 1425

[probability 9 x

effect 9; error

number. 1425]

Order !Done

6h (reserved 4h)High [all parts]

Feedback !Not done

Low (reserved 1h)Low [main parts

to testing]

Reference: James Bach

Test charter and test log in mindmap (Xmind)


Measuring exploratory testing

• The duration of the session

• The relative change in the number of test cases by the same tester

• Coverage of testing per session

• The number of interruptions (Suspension criteria)

• Number of rejected defects in defect database

• …

• Metrics are the eyeglasses of testing that you need in order to be fully aware of the situation and

potential problems in testing

• It is recommended that you choose metrics that are suitable for the challenges of exploratory

testing


Some Tools

• Notetaking:

• Screenshots With Annotation

• Video with Annotation

• Intergrated bug reporting

• Test Management tool suite add-ons

• Micro Focus / HPE Sprinter

• QA Symphony qTest eXplorer

• Telerik Test Studio Explore

• qMetry Voyager

• Steps Recorder

• Recording Browser extensions

• Bug Magnet

• Notepad++ (pad tools in general)

• Xmind, Mindmup (mindmap tools in general)

• Rapid Reporter


• Test charter planning

• Charters

• Priorization

• Test Management tool suites

• Trello

• Jira Agile

• Micro Focus / HPE Agile Manager

• Pivotal Tracker

• Excel

Who can do it?

Knowit 37

Who to recruit? The profile of an exploratory tester

• Exploratory testing is particularly well suitable for a person, who…

• likes to take risks

• is not afraid of changes or new things

• is open-minded

• sets challenges and goals to themselves

• is smart and quick in finding test conditions

• ”Pioneer-style”

• Everyone can learn to be an exploratory tester


Reference: Choosing and Managing the Ideal Test Team. Lloyd Roden

Exploratory testing requires learning skills

• Outline the functionality of the system on paper

• Aim at understanding

• Don’t force yourself to remember facts, use documents

• Ask questions about the functionality of the system

• Recognize the items on which you need more information


Reference: Tutkiva oppiminen. Hakkarainen et al.

Improve your testing thinking skills

• Observe your own testing habits

• Recognize your own ways of thinking

• Learn from misunderstandings and mistakes

• Select the testing techniques according to the situation *)

• Improve your skills to deduce the states of the system *)


Adapted from the reference: Tutkiva oppiminen. Hakkarainen et al.,

*) Reference: Rapid Software Testing. James Bach

What in future?

Knowit 44

Exploratory future

• Exploratory testing will become the norm in agile testing

• And agile will be the norm for all development

• Testing tools suites will include exploratory testing charters and logging

• Coverage will be proven with both last quantities of well-logged exploratory testing and some formally

designed test cases

• With test automation to make sure the basics work, testers will be free to find important defects with

exploratory testing

• People will start exploring also with test automation

• Keyword driven and data-driven approaches

• AI will help with some of Exploratory Testing

© Copyright Knowit Oy 2015 | Confidential

Questions?

[email protected]

@kkakkonen

mailto:[email protected]

4/13/2018 Knowit 47

Creating digital opportunities