Exploratory Test Designlets-test.com/wp-content/uploads/2012/05/REdgren... · 2012-05-17 · Exploratory Test Design Lets Test, Runö, 7-May-2012 Rikard Edgren Qamcom Karlstad [email protected]

Qamcom Research & Technology AB |www.qamcom.se| +46 31 721 1730

Exploratory Test Design

Let’s Test, Runö, 7-May-2012

Rikard Edgren Qamcom Karlstad

[email protected]


“Exploratory software testing is a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

[Cem Kaner]

Why is this good?

at start we don’t know everything important

we want to know more

we will design tests and learn as we go




Agenda

1. Test Analysis Sources for Test Ideas

2. Test Design Quality Characteristics

3. Test Execution Modeling


Part 1: Test Analysis

You always have requirements of some sort.

Understanding and questioning them is a good start, but no more than that.

It’s about learning, and finding out what is important.

Exploratory Test Design looks at many places.


PR

OD

UC

T 1. Capabilities – requirements, examples et.al.

2. Failure Modes – “what if…” - question everything

3. Models – many, if invisible models count

4. Data – exploit dependencies

5. Surroundings – environment / granularity

6. White Box – developer perspective + tester mindset

7. Product History – error catalogues

8. Rumors – kill them or prove them right

9. Actual Software – gulp your Pommac

10. Technologies – things that tend to go wrong

11. Competitors – also in-house, analogue solutions

Sources For Test Ideas


BU

SIN

ESS

12. Purpose – benevolent start

13. Business Objectives – product vision, value drivers

14. Product Image – what should/would users think?

15. Business Knowledge – learn, or co-operate

16. Legal Aspects – what must be avoided?

17. Creative Ideas –products worth building are unique

18. Internal Collections – product-specific quicktests

19. You – you are a user, you matter

Sources For Test Ideas TE

AM


Exercise: Your Information Sources

Write down one or two actual sources of information that improved your testing (do not use requirements!)

Example: Interaction designers gave me their background material they collected from real customers. It helped me design more realistic tests, and also to understand which types of problems that are important.


PR

OJE

CT

20. Project Background – what happened last time?

21. Information Objectives – the purposes of testing

22. Project Risks – test risky areas early

23. Test Artifacts – other’s testing

24. Debt – test against shortcuts

25. Conversations – people talk and collaborate

26. Context Analysis – what should effect your testing?

27. Many Deliverables – test objects and/or inspiration

28. Tools – a starting point for exploration

Sources For Test Ideas


STA

KEH

OLD

ERS

29. Quality Characteristics – in the back of your head

30. Product Fears – capture stakeholder’s worries

31. Usage Scenarios – what people want to do

32. Field Information – environment, needs, feelings

33. Users – some we like, some we don’t like

34. Public Collections – Appendix, Cheat Sheet, Not Done

35. Standards – read, understand, use…

36. References – as oracle and inspiration

37. Searching – Altavista, Volunia et.al.

Sources For Test Ideas EX

TER

NA

L


Capabilities

Failure Mode

Models

Data

Surroundings

White-

box

Product History

Actual software

Technologies

Competitors

Purpose

Image

Business

Knowledge

Legal

aspects

Creative Ideas

Internal

Collections

You

Project

Background

Information

Objectives

Risks

Test Artifacts

Debt

Conversations

Context Analysis

Many

Deliverables

Tools

Quality Characteristics

Fears

Usage

Scenarios

Field Information

Users

Public

Collections

Standards

References

Searching


Test Analysis Questions

Yes, we have all of these, but what should we do with them?

Understand, and use as appropriate Some become straightforward test ideas

Some need a lot of elaborations

Some make other tests richer

We don’t even have time to do the requirements-based tests; how should we have time for all of these?

Judgment, some of these give more important information skip the existing tests someone (you?) already has run

try a few that looks promising

change the ways you test from time to time


Testworthy


Part 2: Test Design

A process of synthesizing test ideas.

This involves: many information sources

judgment of what’s important

experience and creativity

taking advantage of what we learn

practical limitations/opportunities

The number one heuristic: Diverse half-measures


One-Liner Test Ideas

Continuously jot down your test ideas in short format, leaving details to execution.

Don’t necessarily think in terms of Pass/Fail.

Let other people review to get more ideas, better prioritization (and skip test ideas that don’t seem important.)

Use the list for planning HOW and WHEN to test

When to apply techniques, and which to run without flourishes

Things to keep in the back of your head all the time


Capability. Can the product perform valuable functions?

Reliability. Can you trust the product in many and difficult situations?

Usability. Is the product easy to use?

Charisma. Does the product have "it"?

Security. Does the product protect against unwanted usage?

Performance. Is the product fast enough?

IT-bility. Is the product easy to install, maintain and support?

Compatibility. How well does the product interact with software and environments?

Supportability. Can customers’ usage and problems be supported?

Testability. Is it easy to check and test the product?

Maintainability. Can the product be maintained and extended at low cost?

Portability. Is transferring of the product to different environments and languages enabled?


Examples: Usability

Is the product easy to use? Intuitiveness: it is easy to understand and explain what the product can do.

Learnability: it is fast and easy to learn and rememember how to use the product.

Operability: an experienced user can perform common actions very fast.

Clarity: is everything stated explicitly and in detail, with a language that can be understood, leaving no room for doubt?

Errors: there are informative error messages, difficult to make mistakes and easy to repair after making them.

Control: the user should feel in control over the proceedings of the software.

Tailorability: default settings and behavior can be specified for flexibility.

Accessibility: the product is possible to use for as many people as possible, and meets applicable accessibility standards.

Documentation: there is a Help that helps, and matches the functionality.


Examples: Reliability

Can you trust the product in many and difficult situations? Stability: Run the product for a long time, without restarts.

Robustness: Provoke error messages; hit hard, hit many times.

Stress handling: Exceed various limits.

Recoverability: Turn off the power; restart and look at behavior.

Data Integrity: Use all types of data, in different sizes on different systems through all parts.

Safety: Thorough brainstorming around areas where people can get hurt.

Disaster Recovery: Test that the back-up actually can restore everything.

Trustworthiness: Note moments when you are unsure what the product is up to.


Exercise: Important Characteristics

This is an ongoing exercise for the rest of Let‘s Test:

Which characteristics are important for you?

How do you test these?

http://www.youtube.com/watch?v=aNi17YLnZpg


Part 3: Test Execution

want to see problems

see a lot of things

look at many places

look often

focus on what’s important

look for others

The Test Eye


Variation & Complexity

Running the same tests over and over again is good to spot things that break (regression testing)

To find new information, you should perform new tests, or do variations on your old tests.

Adding more complexity gives richer tests with bigger chance to find new, important information

Data

Environment

Sequences

Users


Test Execution Heuristics

Boundary value analysis

Error-Prone Machine

Be ready for serendipity

Do one more thing

Fresh eyes find failure

Follow the scent


Modeling

It might not suffice to keep all the information in your head.

A great framework for getting structure is to use SFDPOT from James Bach’s Heuristic Test Strategy Model.

Structure – what the product is

Functions – what the product does

Data – what the product operates on

Platform – the environment the product depends on

Operations – what the users want to accomplish

Time – relations between the product and time

http://www.testingeducation.org/BBST/foundations/Bach_satisfice-tsm-4p-1.pdf







Modeling

People have different ways of modeling, and that’s a strength.

We need several models in order to capture what’s important to different users.

Models can come from

developers

data

surroundings

usage

quality characteristics

your detailed knowledge about the product


Invisible Models

Skilled testers make many models all the time.

Your understanding (regardless of origin) is a sort of model.

The diversity in models bring more, and richer test ideas.

These don’t need to be documented, but it’s a good exercise.


Exercise: Your first models

Have a look at the software I will show.

How many models do you get?


Perfect Age Models

Long Time Ago

Modern Time

Now Future


Perfect Age Models


Perfect Age Models


Testers’ Trait – Many Perspectives

Testers bring new perspectives, that’s why we find important problems others don’t see.

If you train your ability to think in many different ways, this doesn’t cost a lot of time… …as long as you can judge what is important.


Find Five Faults

...but you don’t have a key, and don’t know how many problems there are

...and there might be other things that are much more important


Finale

You are this generation’s thinking testers

You have to find YOUR best ways

Do your best, collaborate, learn to understand

what is important


Questions

???

Further reading: Exploratory Testing Dynamics (Bach, Bach, Bolton)

BBST Test Design (Kaner, Fiedler)

The Little Black Book on Test Design (Edgren)

[email protected] www.thetesteye.com

http://www.satisfice.com/blog/wp-content/uploads/2009/10/et-dynamics22.pdf

http://www.testingeducation.org/BBST/testdesign/BBSTTestDesign2011pfinal.pdf

http://www.thetesteye.com/papers/TheLittleBlackBookOnTestDesign.pdf

mailto:[email protected]

http://www.thetesteye.com/

Documents

Exploratory Test Designlets-test.com/wp-content/uploads/2012/05/REdgren... · 2012-05-17 · Exploratory Test Design Lets Test, Runö, 7-May-2012 Rikard Edgren Qamcom Karlstad [email protected]