Santa Barbara Agile: Exploratory Testing Explained and Experienced

1

Exploratory Testing Explained and Experienced

Maaret PyhäjärviEmail: <maaret@iki.fi> | Twitter: maaretp

Maaret PyhäjärviNimeä | Attribution (Finland)http://creativecommons.org/licenses/by/1.0/fi/http://creativecommons.org/licenses/by/1.0/fi/deed.en

2

TESTING IS ABOUT FEEDBACK:

QUALITY-RELATED INFORMATION

3

Realizations about Nature of Testing

2016

1639

5±2

4

20DYNAMICALLY ADAPT FOR LIMITED BUDGET

OPPORTUNITY COST

5

16 EXPECT THEUNEXPECTED

6

1639ROUTES ARE RELEVANT

NOT ALL BUGS ARE EQUAL

7

5±2TAKE NOTES

CREATE CHECKLISTS

8

Things Can Look Different from Different Perspectives

Exploratory Testing:Better tests, better testers!

• An approach, not a technique• Find unknown unknowns• Disciplined• Test is a performance, not

artifact– Artifacts support human

memory– Many forms: e.g. checklists

and automation• Exploratory performance

testing, Exploratory test automation, Exploratory regression testing

Test-related learning

Design of new tests

Test executionResult interpretation

9

10

There’s a Difference!• A comment I hear often on the ET

course I give:– ”I’ve always been doing this just did not

give it a name”• I must emphasize that:

– I require more out of exploratory testing than just ”going where you feel like while testing”

• Ask yourself:– Can you defend your choices of strategy

and tactics?– Can you explain what you’ve done in

words that don’t just explain numbers of bugs found?

– How do you know if you’re done or not?

”My testing would be

unsystematic ad hoc testing if I could not

tell the story of my tests, remember what I’ve

tested or what my strategy

was, or relate that back to

my mission” –James Bach

11

Exploratory Testing Self-Management

Vision (“Sandbox”) Current Charter

Other Charters Details

12

Exploratory Testing: Frame of Management

”A day’s work”

Vision (“Sandbox”) Current Charter

Other Charters Details

Bug Reports

Perception of quality

and coverage

Quality ReportDebriefin

g

Tester

Test Manager

PastResultsObstaclesOutlookFeelings

?#

xCharter backlog of the future testing

Out of budget

Next in importanc

e!#, ?, x, +20:20:60

Session sheets of the past testing

Idea of exploratio

n

Metrics summary

Coaching

PlaybooksCoverage outlines

14

Test Ideas / Quick-and-Dirty

Download the full 2-page Cheat Sheet with ideas from Elisabeth Hendrickson, James Lyndsay, and Dale Emery on Qualitytree.com

15

(Exploratory) Testing Dynamics Source: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing

Dynamics. v.2.2. 2009

Evolving

work produc

ts

Skills and

tactics

Testing

polarities

Test strate

gy”A set of

considerations designed to help you

test robustly or evaluate someone

else’s testing.”

” To develop ideas or search a complex space quickly yet thoroughly, not only must you look at theworld from many points of view and perform many kinds of activities”

”Exploratory testing spirals upward

toward a complete and professional set

of test artifacts”

” …skills that comprise professional and cost effective exploration of technology. Each is distinctly observable and learnable, and each is necessary to exploratory work.”

16

Exploration SkillsSource: Adapted from James Bach, Jon Bach, Michael Bolton. Exploratory Testing

Dynamics. v.2.2. 2009

Self-manageme

nt

Developing ideas

Examining product

DoneTo DoIssuesCoverage

All sources available Best use of time – effective and efficient work

Making modelsTool support – creative solutionsRisk-based testing – scientific approachKeeping one’s eyes open

17

Test Automation / Tools in ET• Any form of acquiring quality-related

information fits into exploratory testing• It’s not manual, it’s brain-engaged – and for

making that happen, you need to be smart with automation and tools!

• Example: you might not at first know what you’re looking for...

Search "<ns1:Koodi>" (443 hits in 169 files) in VE  Y:\ELLU\LAPA\Testaus\Kokonaiseläketurvan-ote\Ansaintatiedot-VE\HaeAnsaintatiedot-VE__ 20110307 14-22-15.xml (4 hits)                 Line 1313:                <ns1:Koodi>YL130I</ns1:Koodi>                Line 1317:                <ns1:Koodi>LAPA_172_011</ns1:Koodi>                Line 1321:                <ns1:Koodi>67</ns1:Koodi>                Line 1324:                <ns1:Koodi>67</ns1:Koodi> A lot of text cut away from here…

18

Favoring Mindmaps (Mindmup, Xmind) and Notetaking (Rapid Reporter, iTester) tools

19

Testers don’t break your code, they

break your illusions about the code.

-- adapted from James Bach

20

Empirical Evidence over Speculation

VALUE

ILLUSIONS

•Code doing what it’s supposed to•Product doing what it would need to•Your process is able to deliver with change in mind•Business growing with uninformed risks on the product and the business model around it.

Serendipity, Perseverance …and Love of Testing

• Serendipity = Lucky accident

• Just my luck?– Luck favors the ones

who intentionally vary their actions

• ”The more I practice, the luckier I get”

• Perseverance = Keep trying

• Testing takes time – keep trying with more ideas, stop giving up so easily

• “It's not that I'm so smart, it's just that I stay with problems longer.”

22

The World Has Already Changed

COMMODITY TESTERS• Manual

checkers• Tests are an

artifact

SKILLED TESTERS• Explorers of

products and businesses

• Testing is a performance

Question, Thoughts?Feedback?