Experiences with Exploratory Testing

Alan Richardson@eviltester

www.eviltester.comwww.compendiumdev.co.uk

www.seleniumsimplified.com

Future of Testing: New Techniques and Methodologies● Exploratory Testing is not new,

●but people still don't do it.● The future will be the same, as now,

●unless we take responsibility for it.● Learn from experience,

●base experiments on observation.

Some things I've learned by doing and teaching Exploratory Testing● Adopt a personal approach ● Exploration & experiment is natural● Environment can kill exploratory testing● Evidence based documentation● Use Automation in your exploration

What does "Exploratory Testing" mean?

"An Interactive Process of Simultaneous learning, test design, and test execution"

James Bach

http://www.satisfice.com/glossary.shtml

"If you master the source, but not the teachings, whenever you open your mouth you will speak at random; if you master the teachings but not the source, you will be like a one-eyed dragon."

The story of Chinese Zen, by Nan Huai-Chin

Adopt a personal approach to Testing

Take responsibility for your test approach:

● What does it mean to me?● How do I do it?● Why did I start?● Why do I do it?

What do I mean by exploratory testing?

● Continual Evaluation & Re-evaluation

● Observation & Evidence

● Tool Augmented

● Low Overhead & Fast

● Skill Based● Team Based● Intent

Characterized by:

Low Overhead & Fast● Fast paced environments, becoming norm● Success based on outcome, not compliance● Maximize the time spend "doing"

Techniques:● Think through risks and target those. System

risks, business risks. ● Chunk testing into 'charters' and 'sessions'.

Log the testing against those. Raise defects.

Skill BasedRather than just relying on 'Techniques'...● Assume the tester knows the techniques

and applies them on an adhoc skill based basis

● Train people where risks and bugs exist and how to look for them

● Train people to be adaptable● Trust people, give them responsibility

Team Based

● Use the whole team to review everyone else's work

● Write logs that others can read● Review coverage of others testing and

suggest new scope● Pair to come up with new ideas● Debrief rather than report● The Manager tests as well

Continual Evaluation & Re-evaluation

● Start with minimal scope● Test to learn what you need to test● Write down what you know and don't know● Continually revisit your approach and

testing scope● Revisit your use of tools● Question your effectiveness● Model the system and the process

Intent●Know what you are testing for.●Decide on what to test next.

●What risk are you concerned about?●What requirement are you investigating?

●What data are you covering?●Don't do it because you planned to.

●Do it because it is important now.

Observation & Evidence● Focus / Defocus

● Go Meta● Go Deeper● Peripheral Observation● Relate to existing evidence

● Produce Evidence● screenshots, system logs, ● recordings, manual logs

Tool Augmented

●Tools augment our testing to help us test better

●Tools should never drive or dictate our process

●Exploratory Testing has heavy use of tools for:

Observation, Data Creation, Logging, State Manipulation

My Exploratory Process

● Modelling● Observing● Reflective● Intentional● Manipulative

Why should you do exploratory testing?

• Because you liked the definition?• Because you liked the characteristics?• Because you want to test more effectively?

Personal Approach I don't know why you should do it. I only know why I do it, and why I started doing it.

Why did I start doing exploratory testing?

"There is a risk that the test team will waste a lot of time and money by creating scripts that are wrong and will never be run"

• Frustration Mainly• Heavily Scripted environment which wasn't

working well

In order to be effective we had to Subvert the processStep#

Description Expected Result

1 I've only added this because the tool made me

I'll test the conditions in the case and write down my observations

Why do I keep doing Exploratory Testing?

Because:● It works● I get better at it, the more I do it● I can train people to do it● It adapts to different environments,

and processes

● Stop Following● Do and Learn● Change What you do

Take Responsibility to succeed

Develop Ability & enable

Response Ability

Response Able Adapt Able

Adapt Ability

Account Ability

Account Able

What stops Exploratory Testing?

● Environment● Personal Fear● Existing Learning

Automation & Exploratory Testing

● Tool Augmented● Exploratory Automated Testing

● Models & Checks● Automation leads to Exploration

An example: Random Coverage of  Defined Scope

● New Registration Process● Manual Testing kept finding problems, ● Every app change introduced new errors

● Randomize Coverage of a Model● Model registration as a process of equivalent steps

● Fill Name, Fill Password, Fill PostCode etc.● Model data as a set of templates to randomly fill

● {a-z,A-Z, }*3-25, {a-z,A-Z,0-9,!"£$%^&*()}*6-18● Leave to run, Automated Checks flag issues

● When run is clean, perform exploratory testing

When Automation Leads to exploration

●Sometimes you want to get somewhere known fast, ●so that you can explore unknowns

● Automate the 'getting to'● Randomise to maximise chance that this shows a problem

● When there, engage brain and test

Explore and Experiment with your Testing● The Future of Testing requires us to Unlearn. ● Change our existing beliefs based on evidence.● Make it easy to experiment with new

approaches.● Evaluate Qualitatively as well as Quantitatively.● Take responsibility, own your test approach.

27

Blogs and Web Sites

● SeleniumSimplified.com

● EvilTester.com

● Testing Papers and Tools

– CompendiumDev.co.uk

@eviltester

Online Training Courses

● Technical Web Testing 101

– Unow.be/at/udemy101

● Intro to Selenium

– Unow.be/at/udemystart

● Selenium 2 WebDriver API

– Unow.be/at/udemyapi

Videos

youtube.com/user/EviltesterVideos

Books

Selenium Simplified

Unow.be/rc/selsimp

Alan Richardson is an Independent Test Consultant based in the UK. He offers training and consultancy in Selenium WebDriver, exploratory and technical web testing.

– uk.linkedin.com/in/eviltester

Contact Alan for training and consultancy tailored to your needs:

alan@compendiumdev.co.uk