1

Software Testing

Based on personal experience and what I have learned from the RST course, and other materials from JB,MB and others.

Kristjan Uba

kristjanuba.wordpress.com kristjan@uba.ee

2012

v 0.1.2 James Bach – www.satisfice.com Michael Bolton – www.developsense.com

1

Acknowledgments

Much of material of this course is based on works by James Bach, Michael Bolton. Direct outtakes from RST course are with their permisson.

I have also been influenced by the works of Gerald M. Weinberg, Cem Kaner, Jonathan Bach, and others.

The course material is updated with the feedback I get from attendees. And from online discussion about the craft.

This is my personal understanding of the testing craft. Not a final truth.

2

2

Disclaimer

I exepect that all that attend wish to learn about software testing. And to become very good at it.

I need to hear from you in order to be more relevant to your needs.

Exercises are the most important part. They are designed to fool you, unless you ask

questions. All of the needed information is not provided at the start. Asking questions is essential skill.

3

Important Note

During this course I will ask questions and want you to answer. You can say “pass” if you do not want to be in

spotlight.

I might ask you to defend your answer, pushing you out of your comfort zone.

You are welcome to challenge me as well.

4

3

Quick Question

Code coverage 97% or 56 449 Test Cases, which is better?

And the answer is:

5

Alternative: http://www.projectcartoon.com/create/ From: http://blog.thingsdesigner.com/uploads/id/tree_swing_development_requirements.jpg 6

4

Terminology

Bug Anything about the product that threatens quality.

Quality Value to some person (that matters).

Check Compering actual result to expected value, resulting in binary

answer – pass or fail. Test

Asking and answering a question: “Is there a problem here?” Can consist of multiple checks.

Testing Questioning a product in order to evaluate it.

Definitions from RST course – some from JW and CK. 7

The First thing

Testing is easy If you have endless amount of time If you have complete knowledge If working without scrutiny - without anyone

judging your work (even yourself).

In reality, testing is hard.

8

5

Nature of Testing 1

We can only show that there are bugs We can not show that there are not any more

bugs. Every product is a tool, a solution for something,

providing a service, etc. A product is a part of social interaction,

therefore the quality of the product depends on the evaluator.

Quality is subjective. 9

Nature of Testing 2

When is testing done? When we have found all the bugs? When we have found all important bugs? When time is up? When we feel confident that (we have covered as

much as we could (with given mission/time/resources) and) additional testing would not uncover any more (significant) errors (with reasonable amount of time/resources)

How to measure testing? 10

6

Lack of Resources

Time – we will never have time for exhaustive testing. (Dimensions, environment variables, timings, etc).

Budget - not enough for tools/prototypes/test environments/etc.

Tools – not right tools for the job. Signal generators, virtual server parks, etc. Nor time, to create/purches/train to use them. Purpos built tools are more efficient then off-the-shelf ones.

11

Lack of Resources 2

People – teambuilding, finding the best experienced people.

Skills – keeping oneself up to date with new technology, learning and improving the craft.

12

7

Lack of Information

Specifications –missing, outdated, or both. According to what the developers created the

code if the specifications are messy?

Oracles – How can testers know if there is a bug or not. We need oracles to assess whether we have a

problem or not. Where do oracles come from ?

13

Bonus Question

How would you test this requirement ? “The product shall operate at an

input voltage range of nominal 100-250 VAC.”

14

8

Lack of Information 2

Can we make assumptions? What makes assumptions important?

Impact (what will happen if the assumtion is false) Probability (of the assumption being false) Relevance (if someone else makes a different one ?)

How to deal with assumptions?

Assumptions – mother of all successes.

If used correctly.

15

Complexity

“Any sufficiently advanced technology is indistinguishable from magic. ” - Arthur C. Clarke

Layers of technology can hide problems. (Dependecies, 3rd party apps/dll, hw)

Understanding the technology helps to investigate more efficiently.

Television set

16

9

Complexity 2

The functionality (and purpose) of a product needs to be understood in order to test it.

The functionality might be totally from different context (medicine, space ships, accounting, protocol analysis, etc).

Inputs - outputs Complex input-output (non-human readable) Non-defined usability (fast, reliablity, secure,etc)

17

Complexity 3

Working in realtime - Clock syncing Concurrency Timezone handling Legacy support for older versions Legal issues with product usage Interface to other systems

18

10

Comics

http://xkcd.com/927/ 19

Organizational

How to create a good team for testing Working in team (also with developers) Testing management (doing testing rather than

test documentation) Organizational support for testing and learning

(budget, political, tools, etc) Testing in house, as a 3rd party Legal issues between partners

20

11

Social

Testing is inherently with negative undertone. Our role is to find problems and communicate them. People skills Even when speaking the same langugae people can

misunderstand eachother. Not to mention when speaking non-native language with non-native speaker. (Consider Finnish and Indians speaking english)

Must be aware of cultural differences Presentation of information is vital so it is not misleading Fighting with misconceptions about testing

21

Personal

Having imagination Dealing with stress Dealing with boredom, repetitivness Knowing that one is fallible Intentional blindness (misleading eyes),

misleading randomness, built-in oracle, built-in context box, etc

Circles on a paper

22

12

Personal 2

Demonstrate fallibility Intentional Blindness Models/Oracles in your head

Cultural Linguistual Habitual

Non-random randomness Not paying attention (to the right things)

Magic tricks work for the same reason that bugs exists.

Illusion

23

The problems

Imprecise nature of testing Lack of resources (Time, tools, people, etc) Lack of information (specifications, oracles, etc) Complexity (technology, functionality, etc) Organizational (teams, managment, etc) Social (language, culture, etc) Personal (discipline, human fallibility, etc)

24

13

What is Testing?

“Questioning a product in order to evaluate it.” – by James Bach.

Questioning – asking questions about the goal or design of the product. Or ‘questioning’ the product by operating it in various ways.

25

What is Testing?

The product answers by behaving in a certain manner that testers ‘observe ’ and ‘evaluate ’.

Evaluation is process in which testers try to infer the behavior of the product in the field to pinpoint possible problems.

26

14

What is Testing?

Question Set it up in

different ways Try different

versions Try simple things Try sequences Try combinations Try weird things Try them again

Observe What it does? Where to look? How to look? What’s there? What is Not there? What changed? How often changes? Will it change?

Evaluate Is it ok? Read the spec Find the spec Find inconsistencies Find (obvious)

problems Find obscure

problems

Procedures Test Coverage Oracles

27

What is Testing?

The goal of testing is ... ? to find lot of bugs ? to find important bugs first ? to say when the product is ready for release?

The goal of testing is to give feedback to

project so that the stakeholders could make informed decisions.

28

15

What is Testing?

So what do testers do ? Test Design

Model the test space Determine coverage Determine oracles Determine test procedures

Test Execution Configure test system Operate test system Obeserve test system Evaluate test results Report test results

Mostly driven by risk assessment – what could go wrong and what would happen then?

Reporting back if and how did the risks came true. (And if there is anything else to try)

29

How to be efficient at testing?

In order to be effiecient we must cut waste and optimize.

Stop wasting time on pointless activities: Testing what does not need to be tested Scripted test planning (test cases paper) Collecting useless and misleading statistics

Optimizing testing Prioritizing testing according to mission Encouraging critical thinking Communicating clearly and honestly Flexible variation between structured and freestyle Improving testers skills

30

16

Dealing with problems

Knowing the context Knowing the mission Stating your problems Being friendly Asking questions Keeping open mind Being curios Improving your skills

Analyzing know info Asking for testability Asking for tools and

resources Trusting your team Using heuristics Knowing the risks Applying ‘critical

thinking’

31

Context

Who are: users, clients, stakeholders, developers?

Product: what is the products goal, what problem is it solving, what service is it providing?

Technical: what technologies are used, what 3rd party components are used?

Culture: company, developing teams, users? Legislation: are there any standards, limits, legal

issues?

Lateral thinking puzzle

32

17

Know your mission

What kind of feedback do they need? Finding out what is the mission goes a

long way into being efficient -> means doing things that are needed.

Ask questions, find out priorities, make (written) assumptions, etc

Feedback Q

33

Know your mission 2

Why are we testing this product? What part should we focus on? What is our main goal? What constraints to we have? What resources do we have? Do stakeholders agree with our view of

the testing mission?

34

18

Asking questions

First ask if it is ok to ask questions Question the product, mission, specifications,

context Question (your) assumptions Ask challenging and/or provoking questions What if we cannot ask a question ?

Make written assumptions State your view of the problem

And present that to stakeholdres, before starting to work or later on with the report.

35

Asking questions 2

For difficult questions it is advisable to ask in manner which is solution seeking: State the questions clearly Draw out the subquestions by points Add your understanding of the problem Provide one (or more) possible solutions Add pro’s and con’s to your solutions Add the possible impact to the project if neccessary

This way the other party can answer much more

quickly and can take more informed decision.

36

19

Social Interaction

Communication Tell what you want clearly Repeat to others what you understood so

that they could correct you. Everyone makes assumptions (called

‘common sense’) and expects everyone else to share it. And people are surprised when they are misunderstood.

Folding Exercise

37

Being friendly

Working with friendly people is more productive than with hostile people.

To keep unwanted emotions at minimum: Know the culture (avoids accidental insults) Be calm Talk neutrally Be open to discussion Remember that everyone makes mistakes (even you)

Being friendly does not mean being buddies.

38

20

Giving advice

“No matter how it looks, everyone is trying to be helpful.” Gerald M. Weinberg

Make sure people want your advice, present it in a manner that is not assaulting.

39

Keep open mind

You might not know the best There might be things you do not know

that change the context Question yourself too Have you forgotten something Remember that sometimes, just

sometimes, the boss really is right.

40

21

Improving your skills

Are you doing the best you can? In a difficult spot don’t freeze

Share problems Review assumptions Do something now Make a counteroffer

Learn in safe environment Practise thinking

Open combination lock How many moves? What’s the maximum parking fee?

41

How to do right tests at the right time?

Treating test design and execution as parallel process. Using heuristics to guide and structure the process. Consider cost vs value in all testing activities. Diversify tactics and team members. Dynamically manage the focus of your work. Manage the tests dynamically (do what is needed, not

what was planned). Evolve the tests through testing as your knowledge of

the context and product grows. 42

22

Test Design Functional testing Boundary testing White box Black box Automated testing Scripted testing Smoke testing State testing State Transaction testing Unit Testing Integration Testing System testing Load testing Security testing Portability testing Maintainability testing Confrimation testing

Non-Functional testing Grey box Domain testing Stress testing Static testing Dynamic testing Reliablity testing Beta testing Integration testing Scenario Based Testing Error correction testing Manual Testing Risk Based Testing Equivalence class Testing Regression testing Spec-based testing Tour-based testing Paired testing

The list is incomplete. 43

Test Design 1

Some are more general than others Some are overlapping

Some are the same with different name.

Ways to think about the product

Prompts to think about certain aspects

44

23

Test Design 2

Every one of those has its pros and cons. And knowing the limitations of each is most important.

Key to efficient testing is to know when to apply certain kind of technique.

Analyzing the situation is a must. Just doing what has been done before is most probably suboptimal (not to say wasteful).

45

Test Design 3

Choosing the one(s) to use depends on: Intended coverage Tester skills Testability of the system Current state of the software Available resources Overall mission

46