1

2

3

4

Objectives:

In this talk, I want to talk about:

a. What does it mean when folks say, “I am doing Exploratory Testing”, why is

it important in Agile, etc

b. Along the road, debunk some of the myths surrounding Exploratory Testing

Key Takeaways:

1. How to make your Exploratory testing sessions more effective so that you achieve customer value

5

James Bach defines it succinctly as simultaneous test definition and execution. Of course, exploratory testing has been very popular and most testers do exploratory testing, perhaps calling it by another name or not calling it anything at all

6

What pops in your mind as you think about exploratory testing? There are many reasons why exploratory testing might be interesting to testers in today’s world: Find bugs fast and early without test overhead – exploratory testing is lightweight and reasonably good test hunches will let you find defects quick and early without the documentation time overhead Centered on customer value rather than spec compliance – Tester is an end user proxy on the team and is key to the product being useful to the end user. Exploratory testing is done on working software and focused on customer value. Crucial component of Agile testing – Everyone tests on an Agile team : the developers, Product owner, SCRUM master. The iterations are typically short and defects need to be found and fixed to deliver optimal customer value within a short turnaround time. This is an ideal situation to use a lightweight and effective style of testing like exploratory testing When you know less about a system and testing is open ended When you have to find maximum bugs in minimum time

7

So, despite so many advantages, why do many people still have reservations with exploratory testing? Exploratory testing has rapidly gained popularity over the past few years as one of the most efficient forms of testing in an agile lifecycle. Exploratory testing has a proven advantage of finding bugs faster, with less wastage in terms of test documentation and covering large areas of the application under test in a shorter time period. Why then, do we still find exploratory testing not being a mainstay in large scale enterprise product testing? Here are some of the myths associated with Exploratory testing: a. Exploratory testing is ad-hoc, monkey testing b. Exploratory testing creates poorly documented bugs c. There is no way to track exploratory testing

Lets look at each one of them in detail.

8

Exploratory testing is NOT random, undirected testing. On the contrary, unlike scripted testing, you cannot just follow a set of pre-determined steps that might arrive at a defect. Testers often have hunches about where defects could be – a particular dev tends to be bad at exception handling, or an app that is weak in dealing with frequently disconnecting connections, or a website doesn’t do effective load balancing when stressed. As you learn more about the system under test, you get new hunches, new test ideas that could lead to unfound defects. Vectoring exploratory testing with some of these test ideas will help explore the system more effectively and completely. Two great ways of vectoring exploratory testing can be found in session based test management by James Bach and exploratory testing tours by James Whittaker. SBTM will help direct exploratory testing by getting the tester to define a mission upfront and help keep exploration focused in that area. Testing tours help define various paths to explore in the system with each tour leading to different kinds of defects in different areas.

9

How we use XT within our product development – using Charters/Themes

10

We use the concept of testing tours extensively in our in house testing – check out some of our testing tours look like: My favorite tour is the Back Alley tour where you exercise some of the lesser known paths in the application, the Fedex tour where you can track a data structure from point to point to see how it gets transmuted, the All Nighter tour that has caught some nasty memory leaks we fixed recently The Landmarks tour where sales folks do DEMOs of the product The Guidebook tour or F1 tour – following documentation to the letter The Intellectual tour – asking the software hard questions – Inputs & data that system accepts that streches its limits The Anti-social tour – providing least likely inputs or bad inputs The couch potato tour – accepting all default values The sabotage tour – find out the resources used, then restrict access to those resources The collectors tour – collect up all the outputs that can be generated from the system The Supermodel tour – UI Interface

11

Apart from the tours we discussed, we also use XT for the user stories that got completed in this sprint. This provides us the traceability to answer questions such as: did all the user stories in the sprint got covered in the Testing; which stories yielded the most number of bugs etc. This allows us to tie the progress back to the customer value instead of just the deviations from the spec. We also use XT for our Bug bashes – we divide up the team into different roles and either use the Tours Or User Stories to drive the bug bash. Demo of VS11 for doing exploratory testing on user stories

12

Lets look at the 2nd myth – XT creates poorly documented bugs

13

Since in exploratory testing, there is no prescribed path/spec that you are following, it becomes critical to have all the information needed to reproduce the bug, once it has been found. So, how do you capture all the data that helps you in filing a Rich bug. Using some of the tools that capture your actions as you are exploring the application would be very nice. Similarly collecting all the event logs, system info logs, screenshots that show what you just saw, an entire video recording of what paths you were running through are some of the items that could be captured to generate a Rich bug. Well, it might even include an entire debugging log to analyze what you were doing for the developer – effectively capturing the entire state of the bug.

14

Here is some of the information we collect as part of bug filing during exploratory testing Demo of VS11 for rich bug filing

15

Some of the key things you saw in the Demo include: - How easy it is to create test cases from you exploratory session – this could be

either during the session or post the session completion - How the tests that you create are automatically added to the right Test Suites

which get rolled up into the feature area - How each iteration of exploration generates test cases which get automatically

rolled up in the Test plan you are executing - How with each sprint, regression suites are automatically being created

16

Lets look at the 3rd myth – there is no way to track exploratory testing

17

Here are some of the metrics that can be used to track Exploratory testing: - Trending of bug priorities and bug types that were filed during exploratory testing - Ratio of time spent on exploration to the number of bugs that were filed - Ratio of time spen on exploration to the number of test cases that were created - Coverage of functionality across charters/themes - Coverage of Code while doing exploratory testing - Coverage across User stories that have been taken up for current / past sprints - The main thing is that there are NO absolute metrics – you have to finetune/tailor

to suit the quality of your application

18

Here are some of the metrics we use to track our XT work Demo of VS11 for tracking purposes (Code Coverage, Bugs files, Test cases filed etc)

19

Exploratory testing done with the right tools enables you to: Define and test what is appropriate for your app Test more frequently and find bugs faster Create a trail of structured test cases for regression

20

Objectives:

In this talk, I want to talk about:

a. What does it mean when folks say, “I am doing Exploratory Testing”, why is

it important in Agile, etc

b. Along the road, debunk some of the myths surrounding Exploratory Testing

Key Takeaways:

1. How to make your Exploratory testing sessions more effective so that you achieve customer value

21