Session-Based Test Management

An approach to testing with– No test cases– No expected results– No pass/fail metrics

Testing on a project is divided up into focussed “sessions” of exploratory testing.

Before we start…

• I’m no expert• A lot to cover. I will circulate links with more detail• Definitions/opinions are as per “Context-Driven

Testing”– No best practices, depends on context– Very influenced by social sciences and related fields– All testing is “sapient” and exploratory, or it’s not testing– “Good software testing is a challenging intellectual process”

What problem is this solving?

• Exploratory testing the most valuable and powerful approach – especially in Agile

BUT• Perceived as vague, unstructured, unmanageable– Hard to gauge coverage– The tester themselves may not be able to clearly explain

what testing they did – Managers want metrics, estimates and line-graphs (even if

these can be meaningless)• “If you can’t measure it, you can’t manage it”

Exploratory Testing – the CDT view

• Defined as “Simultaneous learning, test design and test execution”

• An experimental, scientific approach• Fits the natural human approach to learning or

problem-solving• Exploratory testing is not random• Exploratory testing is not a luxury• Exploratory testing is not what you do if you have

spare time after executing all the test cases

Session-Based Test Management

• Formulated by Jonathan and James Bach• Manages and tracks Exploratory Testing• Testing tasks on a project divided into sessions• A test session:– A dedicated and uninterrupted period of exploratory

testing – Focussed on set objectives defined in a “Charter”– Generates notes and lightweight reporting on the testing

carried out• The testing task is expected to change and evolve

Session-Based Test Management: Planning

PLANNING: The Charter

• Each test session is described by a Charter• A single sentence that sets the scope of the testing– What you’ll test– How you plan to test it– The information you hope to provide

• That scope should be achievable in around 90 mins• Charters are basis of estimation and coverage

PLANNING: Drawing up Charters

• Charters can come from many sources:– What are the elements to be tested? What are the

requirements? What are the risks?– Backlog/sprint planning meetings

• Elisabeth Hendrickson’s “Explorer” format:“Explore [target]with [resources, techniques]to discover [information valuable to the project]”Eg. “Explore unexpected failures with power and network interruptions to discover if the product recovers gracefully”

PLANNING: Drawing up Charters

……Read Chapter 2 of “Explore It!” By Elisabeth Hendrickson.

(It’s only 13 pages)

Examples….

Session-Based Test Management: Testing

TESTING: Focus

• Charter as scope– Some flexibility possible

• A dedicated and uninterrupted period of testing– No meetings, no email, no drive-bys

• Session length will vary but perhaps 90 mins

TESTING: Test Ideas

• Use testing experience or instinct; use your domain knowledge

• Test scenarios could be ok, but not pre-scripted stepsRemember it should be a fluid process …• Use the software, observe it and be alert to new

questions or test ideas – Follow those new thoughts now?– Note them as ideas for follow-up?

TESTING: Test Idea Frameworks

• Personas• Tours• Heuristics(Could be with the “with” part of your Charter)

•Encourage thinking beyond specific features or tasks in isolation. – how parts of the software work together– user flows

TESTING: Idea Frameworks - Personas

• Fully imagine and describe users of the software. Be that person while you test the product– Role– Age– Education – What devices do they use?– What environment do they use your product in?– What are their goals in using the product?– What motivates them, What annoys them? – What websites would they use? What TV programmes would they

watch?

TESTING: Idea Frameworks - Tours

• Mike Kelly: touring an application– Claims tour– Configuration tour

• James Whittaker: exploratory testing tours– The Museum tour– The Supermodel tour– The Bad Neighbourhood tour

TESTING: Idea Frameworks - Heuristics

• Shortcut approaches to complex questions or problems which are usually based on experience. They’re usually effective, but not guaranteed to be.– Eg. “Trial and error”

• In Testing:“Test at the boundaries” ; “CRUD” ; “Some, none, all” ; “OFAT (One Factor At a Time)” ; “Break the rules” ; “The Iceberg Heuristic”You will have your own ….

• Test Heuristic Cheat Sheet

TESTING: Note Taking

• Notes on your testing are crucial• Not just a Pass/Fail count– What you did. Why you did it. What you saw. What

questions and ideas you had. Any frustrations/blocks.• Style of note taking can be personal to you

– I tend to use Rapid Reporter or just pen/paper

• You need to be able to summarise and discuss the notes from your session

Session-Based Test Management: Reporting

REPORTING: Session De-brief

• Tester reviews session findings with Test Manager/Test Lead/Other Tester– A bit like reporting in a stand-up– Is more testing needed in this area?

• May lead to ideas for more charters– Are there issues or blocks affecting the testing?– Are there questions which need to be referred to a

product owner?• Raise bugs as normal• In my case: lightweight reporting

REPORTING: Project Status

• Managers can have metrics on ET, including:– Charters/Sessions completed and remaining– Sessions completed per day or week (velocity)– Product areas/functionality coverage– Bugs raised– Time spent testing vs setup, admin etc

• See links to articles by Jonathan Bach and Rob Lambert

The End

To summarise - in SBTM:• Testing described in session charters, not test cases• Exploratory Testing is not random• Exploratory Testing is creative and requires intelligent

thought. A number of techniques and approaches are available.

• More information in the links I’ll distribute• For anyone interested, I can demo the Rapid Reporter

note-taking tool