AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 1 Toward an Exploratory Testing Culture Robert Sabourin President & Principal consultant

AmiBug.Com, Inc.

© Robert Sabourin, 2008April 19, 2023 Slide 1

Toward an Exploratory

Testing Culture

Robert Sabourin

President & Principal consultant

AmiBug.Com, Inc.

Montreal, Canada

[email protected]

AmiBug.Com, Inc.


Toward an Exploratory Testing Culture

Traditional testing teams often agonize over exploratory testing. How can they plan and design tests without detailed up-front documentation? Stubborn testers may want to quit because they are being asked to move out of their comfort zone. Can a team’s testing culture be changed? Rob Sabourin describes how several teams have undergone dramatic shifts to embrace exploratory testing. Learn how to blend cognitive thinking skills, subject matter expertise, and “hard earned” experience to help refocus your team and improve your outcomes. Learn to separate bureaucracy from thinking and paperwork from value. Explore motivations for change and resistance to it in different project contexts. Leverage Parkinson’s Law—work expands to fill the time available—and Dijkstra’s Principle—testing can show the presence of bugs, but not their absence—to inspire and motivate you and your team to get comfortable in the world of exploratory testing.

AmiBug.Com, Inc.


– Defined– Philosophy

• Fundamental Question• Dijkstra’s Principle• Parkinson’s Law• Covey

– Purpose– People– Learning– Context– Diverse Testing Ideas– Case Study


AmiBug.Com, Inc.


Exploratory Testing

Defined

AmiBug.Com, Inc.


• Concurrent– Learning– Design– Execution

ET

Def

ined


AmiBug.Com, Inc.


Exploratory Testing

An Example:

Bug Confirmation Testing

ET

Def

ined

AmiBug.Com, Inc.


Bug Confirmation

• Confirm that the “fixed” bug is really fixed

ET

Def

ined

AmiBug.Com, Inc.


Bug Confirmation

• Was bug fixed?

• Did bug move?

• Did developers accidentally inject a new bug?

ET

Def

ined

AmiBug.Com, Inc.


Bug Confirmation

Was bug really fixed?1. Use repro steps from bug report

2. Confirm bug was in old build

3. Show bug is not in new build

ET

Def

ined

AmiBug.Com, Inc.


Bug Confirmation

Explore: Did the bug move?1. Exercise related capabilities

2. Vary steps to reproduce

3. Investigate features that share code

ET

Def

ined

AmiBug.Com, Inc.


Bug Confirmation

Explore: Did developers inject a new bug?1. Exercise application capabilities

2. Vary data

3. Walk through scenarios

ET

Def

ined

AmiBug.Com, Inc.


Bug Confirmation

Jason’s Approach?1. Speak with developer2. Confirm bug was in old build (repro)3. Confirm bug is not in new build (repro) 4. Identify a usage scenario which would have

exposed the bug5. Walk through the scenario many times with

variation of steps and user data6. Stop when perturbation is absurd

ET

Def

ined

AmiBug.Com, Inc.


Exploratory Testing

Philosophy

AmiBug.Com, Inc.


Fundamental Question

• How do you know when you are finished?

Phi

loso

phy

AmiBug.Com, Inc.


Edsger W. Dijkstra

• “Program testing can be used to show the presence of bugs, but never to show their absence”

Phi

loso

phy

AmiBug.Com, Inc.


C. Northcote Parkinson

Parkinson’s Law:“…work expands so as to fill the

time available for its completion…”

Phi

loso

phy

AmiBug.Com, Inc.


“…begin with the end in mind …

“…first things first …"

Steve Covey

Phi

loso

phy

AmiBug.Com, Inc.


"No! Try not, Do. Or do not.

There is no try."

Yoda

Phi

loso

phy

AmiBug.Com, Inc.


Exploratory Testing

Purpose

AmiBug.Com, Inc.


Purpose of Testing

• Common definition:– To find bugs before our customers do!

• Broader definition:– The role of testing is to provide objective input to

facilitate business decisions!– Keeps stakeholders aware of all issues or

concerns that relate to shipping a product!

Pur

pose

AmiBug.Com, Inc.


Bug Defined

• To make our job more fun, whenever we have a concern with software, we call it a “bug”.

Pur

pose

AmiBug.Com, Inc.


Exploratory Testing

People

AmiBug.Com, Inc.


• It’s all about people! (and the occasional bug too)

Peo

ple


AmiBug.Com, Inc.


Leadership

• Different Strokes for different folks at different times!– Adapt leadership style to the situation– Choose leadership styles deliberately!

Peo

ple

AmiBug.Com, Inc.


Leadership

• Basic Leadership Styles• Directing• Coaching• Supporting• Delegating

Peo

ple

AmiBug.Com, Inc.


Leadership

• Directing Leadership• Tell people specifically what to do• Provide constant feedback, praising and

redirection• Used when someone is new to a task and

uncertain as to how to successfully achieve the task

• Used sometimes in an emergency situation

Peo

ple

AmiBug.Com, Inc.


Leadership

• Coaching Leadership• Provide guidance and advice on how to achieve

goals • Does not need close direction but needs to learn

how to achieve success• Used when someone has a proven track record

but is new to this specific task• Team member is mature enough to ask for

assistance

Peo

ple

AmiBug.Com, Inc.


Leadership

• Supporting Leadership• Staff participate in decision making with leader• Staff works with leader to establish goals and

milestones• Person can work quite autonomously but needs

leaders help• Team member is mature

Peo

ple

AmiBug.Com, Inc.


Leadership

• Delegating Leadership• Staff is given broad goal and takes full

ownership of task• Constant feedback is not required• Confirm big picture business drivers have not

changed• Team member is autonomous

Peo

ple

AmiBug.Com, Inc.


Exploratory Testing

Learning

AmiBug.Com, Inc.


Unprepared

Learning

Lear

ning

AmiBug.Com, Inc.


Just-In-Time Testing

Sharpen Testing Skills• Thinker• Detective• Reporter• Diplomat• Negotiator• Cheer Leader• Pragmatist

Lear

ning

AmiBug.Com, Inc.


Exploratory Testing

Context

AmiBug.Com, Inc.


Context Drivers - BTO

• Business– Value– To whom?– Why?

• Technology– Solutions

• Organization– Corporate Structure– Team Structure– Roles and Responsibilities

Con

text

AmiBug.Com, Inc.


Con

text

AmiBug.Com, Inc.


Exploratory Testing

Diverse Testing Ideas

AmiBug.Com, Inc.


Testing Ideas

• Collect testing ideas• From testing ideas build a

series of testing objectives– Each can be assigned as

work to testers– Each can include all, part of,

or multiple testing ideas

Div

erse

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Investigative approaches– We become truffle

snorting pigs and try to find useful information in all evidence we discover

– We can even get good ideas from out of date sources

Div

erse

tes

ting

idea

s

AmiBug.Com, Inc.


Test Idea Sources

• Capabilities• Failure Modes• Quality Factors• Usage Scenarios• Creative Ideas• States• Data• Environments• White Box• Taxonomies

Div

erse

tes

ting

idea

s

AmiBug.Com, Inc.


Test Triage

• Allocate Testing Assignments to Testers– Make sure testers know context– Best thing to test– Best person to test it– Best people to explore it– Best lead– Are subject matter experts required

Div

erse

tes

ting

idea

s

AmiBug.Com, Inc.


Thank You

Questions?

AmiBug.Com, Inc.


Out of the Frying Pan into the Fire

The case of the distracting documentation

AmiBug.Com, Inc.


The case of the distracting

documentation • Corporate goal

– respond to market pressure without destroying ongoing development projects

Cas

e of

the

Dis

trac

ting

Doc

umen

tatio

n

AmiBug.Com, Inc.



• Before– In Testing

• Heavily scripted • Subject Matter Experts• Independent team of

system testers

Cas

e of

the

Dis

trac

ting

Doc

umen

tatio

n

AmiBug.Com, Inc.



• Before– In Development

• Teams torn apart to react to emergencies

• Heroic Knights in shining armor• Development uses several levels

of documentation• Tons of "integration testing”

before release to test team• Inconsistent unit testing

Cas

e of

the

Dis

trac

ting

Doc

umen

tatio

n

AmiBug.Com, Inc.



• Before– In Product Management

• Many different roles managing work• Many different requirement lists• Dominant consensus management from

highest levels

Cas

e of

the

Dis

trac

ting

Doc

umen

tatio

n

AmiBug.Com, Inc.



documentation • Setup

– Urgent project selected as pilot– Storyboards replaced

traditional requirements– Assigned team members by

surprise– No training (wing it)

Cas

e of

the

Dis

trac

ting

Doc

umen

tatio

n

AmiBug.Com, Inc.



• Problems– Testing team very uncomfortable– Discovered exploratory testing– Tons of overtime– Product management

not active– Treated as a heroic effort

Cas

e of

the

Dis

trac

ting

Doc

umen

tatio

n

AmiBug.Com, Inc.



documentation • Remedy

– Active product management– Consolidate backlog– Train team in method– Involve testers in planning sessions– Coaching on exploratory and Just in time

testing– Consistently use unit test framework– Teach team how to use "stories" for

requirements and testing– Team was committed during iteration

Cas

e of

the

Dis

trac

ting

Doc

umen

tatio

n

Documents

AmiBug.Com, Inc. © Robert Sabourin, 2008September 15, 2015Slide 1 Toward an Exploratory Testing Culture Robert Sabourin President & Principal consultant