Testing Awareness

8/11/2019 Testing Awareness

1/52

1

Testing Awareness


2/52

3

Agenda

Testing?

A Sample Test Process

A few Testing DefinitionsTest Process Flow

Test roles

Testing realities


3/52

4

What is Testing ?Three integer values representing the lengths of the sides of a triangle, are

the input of the application. The output or result of the application is amessage that the triangle is scalene or equilateral or isocele

Define a set of tests that, to your feeling, would test this application

properly. Enrich your test descriptions by specifying the used set of data

?A B

C

Scalene : A < B + C

Isocele : B = C and A < B + C

Equilateral : A=B=C and A < B + C


4/52

5

Test cases1. Do you have a test that represents a valid scalene triangle ?

2. Do you have a test that represents a valid equilateral triangle ?

3. Do you have a test that represents a valid isoceles triangle ?

4. Do you have all three permutations of the two equal sides of test 3 (e.g. 3,3,4; 3,4,3; 4,3,3

5. Do you have a test in which one input value is zero ?

6. Do you have a test in which one input is a negative integer ?

7. Do you have a test with all three inputs greater than zero, but with the sum of two of theinput values equal to the third (if the program sais that 1,2,3 represents a scalene trianglewould contain a bug) ?

8. Do you have at least all permutations of the previous test ?

9. Do you have a test with three integers greater that zero but with the sum of two of the

numbers less then the third ?

10. Do you have at least all permutations of the previous test ?

11. Do you have a test with all input values being zero ?

12. Do you have at least one test with non-integer inputs ?

13. Do you have a test with the number of input values (two rather then three) ?14. Did you specify for each test the expected output ?


5/52

6

What is testing ?

Testing is executing a software with the

intent to find defect

Exercice feed Back

- Expected result must be define in advance

- Program Algorithm is different from test description

- Valid and Invalid statement must be tested

- Exhaustive testing can be expensive

-


6/52

7

But where to stop ?

EXTENT OF TESTING

NUMBER OF

DEFECTS OPTIMUM

TEST

COST OF

TESTING

OVER

TESTUNDER

TEST


7/52

8

Why is Test important ?

1 2,5 510

25

100

0

20

40

60

80

100

120

Require

ment

Design

Coding

UnitTe

st

Acceptance

Maintenance

Cost to remove an error


8/52

9

Errors are unavoidable

What the client wanted What the contract said What the analyst designed

What the developer wrote What was delivered What was really needed


9/52

10

Why Testing?

Defects are discovered sooner

Reduced correction costs!

Quality is evaluated early and iteratively in the

project

Increased visibility!


10/52

11

Agenda

Testing ?


A few Testing ConceptTest Process Flow

Test roles

Testing realities


11/52

12

Valid for : Corporate Development(Finance, Governement, distribution,)

1 application is composed of 800 modules imbeded in asystem of 80 000 modules.

A project involve in average between 10 & twenty

people. Lot of activities on the same system in the same time

User involve in project and elaborate detailled

requirement while you develop You start always late but you must deliver on time.


12/52


13/52

14

Test stages and Test scenarios

creation

System Tests

Integration Tests

Unit Tests

Acceptance Tests

Software

Architecture (SAD)

Components

Design

Software

Requirements (SRS)

Business needs(Vision)

SCENARIOS CREATION

SCENARIOS CREATION

SCENARIOS CREATION

SCENARIOS CREATION


14/52

15

Test stages

The test stages are the key to validate the code alongdifferent point of view and different level of details

Business

Technical The test stages have a huge impact on how the scenario

are written

Business oriented

Technically oriented

The test stages are related to the maturity of thesoftware only and independent of the phase

The 4 test stages may be present in each and every iteration


15/52

16

Test concepts 2 : Test types

Most common test types Performance tests (Mainframe response time, Scalability)

Functional tests (Conformance to functionalrequirements)

Configuration tests (Different software and hardware onwhich the application will run)

Security tests (Role Based Security)

Non-regression tests (No new defects caused by new

code) Stress tests (Operate the system to its limit)

Recovery tests (Behavior in case of system break-down)

Usability tests (Help on-line, ergonomy,)


16/52

17

Test types

Test types are the key to translate testexpectations in a concrete test strategy :

What tests will be performed.

When will which tests be performed.

What skills are needed.

What tools are needed.


17/52

18

Test concepts 3 : Test Strategy

Test Strategy is

Best combination off test stages, test types and

application feature to test in order to :

- Anticipate RiskHigh risk first

- Address test requirement based on priority- Show quality achievement regularly during

project execution


18/52

19

Agenda

Testing ?


A few Testing Concept

Test Process Flow

Test roles

Testing realities


19/52

20

Test process overview

Test Strategy & plan

[Not last

cycle of

the

iteration]

[Last

iteration

of the

project]

[Not last

iteration of theproject]

Test Strategy and

Planning

Test Design and

Build

Test

ExecutionTest

Reporting

Close Test

Session

[Last test

cycle of the

iteration]

Plan

Designand Build

Executeand Report

Define testscenarios

Test model Frequent

reports

Completed during eachiteration to gain

maturity


20/52

21

RUP Lifecycle - Test discipline


21/52

22

Test Strategy and Planning During inception and

elaboration

Define a test strategy able tomeet stakeholders testexpectations.

Define the resources needed toundertake this strategy

Deliverables

An approved test plan

Optionally a Test Initiation

Checklist

During construction andtransition

Update and detail initial plan

Detailed schedule

Updated scope

[Not last

cycle of

the

iteration]

[Last

iteration

of the

project]

[Not last


Test Strategy and

Planning

Test Design and

Build

Test

ExecutionTest

Reporting

Close Test

Session

[Last test

cycle of the

iteration]

T S d


22/52

23

Test Strategy and Planning

Output =Test Plan Summary of quality

expectations.

Test strategy

Test exit criteria

Test schedule

Reporting facilities

Resources needed bythe strategy

Which test types in which test

stages for which features?

When is testing finished?

Skills, metrics reported, audience,

periodicity

People, budget, tools, test

environments.

Quality Objectives


23/52


24/52

25

Each test scenario describes

Unique identifier

Detailed objective for thescenario

Steps to be executed

Corresponding expectedresults

Scenario priority

Unambiguously identify the scenario

for further reference (e.g. defect

reporting)

Assess success or failure of the

scenario

Procedure to be followed by the tester

To be compared against the actual

behavior, base for defect reporting.

Test Design and BuildOutput =Test scenarios

Importance of the scenario


25/52

26

Test Execution and Reporting

Execute test scenarios

Register the defects

Report execution results

Deliverables Defects

Reports

[Not last

cycle of

the

iteration]

[Last

iteration

of the

project]

[Not last


Test Strategy and

Planning

Test Design and

Build

Test

ExecutionTest

Reporting

Close Test

Session

[Last test

cycle of theiteration]

T E i d R i


26/52

27

A defect is reported in a defect tracking tool Difference

Status in the defect life-cycle

Unique identifier

eg. Test tracker, QAA, test director

Link with failed scenario Id

Defect tracking tool

Communication ! Registration

Statistics

A defect is linked to a particular scenario


Output =Defects

Defect ARE Key :-Communication-Quality Measurment-Process Improvment


27/52

28


Output = ReportDefect Status Nb of defects

Open 15

Under correction 14

To be re-tested 35

Under re-test 3

Closed 45

#Defects per status

Open

Under correction

To be re-tested

Under re-test

Closed

There is a

bottleneck there

+ Written conclusions of the analysis


28/52

29

Close Test Session

Publish in the test sign-offdocument to which extend

the quality expectations aremet

Defects pending

Features tested/ not tested

Acceptance criteria

[Not last

cycle of

the

iteration]

[Last

iteration

of the

project]

[Not last


Test Strategy and

Planning

Test Design and

Build

Test

ExecutionTest

Reporting

Close Test

Session

[Last test

cycle of theiteration]


29/52

30

Activities workload in each phase

Test strategy

and planning

Test Design andBuild

Test Execution

and Reporting

Close test

session


30/52

31

Agenda

Testing ?


A few Testing Concept

Test Process Flow

Test roles

Testing realities


31/52

32

Test Roles

Test coordinator

Test analyst

Tester


32/52

33

Roles - Test Coordinator

Plan and coordinate test activities and resources.

Control the test execution and report on a regular basis, thetest status and progress.

Assertive - Independent minded Owner of the test plan and the test model

Good relational skills

High workload during

Strategy and planning

Execution and reporting


33/52

34

Roles - Test Analyst

Delivers the test model and test scenarios.

Deep understanding and knowledge of the project technicalspecifications and business requirements.

Skilled in Test Specification Techniques (Boundary analysis,decision tables, equivalence classes).

Owner of the test scenarios

Details oriented

High workload during design and build


34/52

35

Roles - Tester

Execute scenarios, report execution results, and register defects.

Correct understanding and knowledge of the applicationtechnical and functional specifications and businessrequirements.

Eye for details and rigorous in his/her execution of the testscenario and defect reporting.

Good relational skills (will have to report defects and work ingood cooperation with their authors).

High workload during execution and reporting


35/52

36

Summary: The building blocks

A test plan The project test objectives, the strategy to meet

them and the means required by this strategy

Test scenarios A detailed description of the steps to execute

and pre-defined expected behavior for each ofthem

A test model

The link between the features and the testscenarios

Frequent reporting

A clear description of what does and does notwork in the system

Role definition Who does what and when

Test strategy and testorganization

Detailed test execution

procedures and

corresponding expectations

Tracability and input

to reporting

Visibility

Ownership


36/52

37

Some Testing realities !!


37/52

38

1. Quality versus deadlines

what if deadlines were expressed in the same wayquality often is?

Yes, were really committed to deadlines. We need to finish

early -we mustnt be late. Exactly when? Well, Ill know it whenthe date arrives, but you cant tie it down like that. Oh yes, wehave a plan, but we dont actually put dates on it - but its veryimportant to achieve timely delivery.

Deception: I cant see it, so it isnt important

Need to make quality visible (quantified attributes)

Need to evaluate projects (and project managers) onquality achieved, not just on deadlines met


38/52

39

Symptoms: Test Planning

2 weeks?

Content of test plans 6 weeks

Shorter one looks exactly the same inside

therefore it must not be any different in nature

2 weeks is better for my schedule

Cant see any difference, therefore there is none Solution: make detailed test activities visible

Not shorten a piece of string, cut off arm or leg!

testing

testing


39/52

40

2. What does testing produce?

Development versus testing

can tell when development is finished

it produces something concrete and demonstrable

positive effect

testing doesnt change the product at all

neutral effect: still the same product

negative effects adds extra work when were pushed

didnt want to know about bugs anyway

Why testing is a hard sell

Illusion - testing

adds confidence

- or opportunity toimprove quality


40/52

41

Sample: Y2K a waste of time

Nothing happened (much)

Therefore, there was never a real serious problem

The doom merchants were wrong

All that effort was a waste of money just like immunisation programmes

house/car/health insurance

national defense Does this remind you of anything?

Software testing in general?


41/52

42

3. Relationship problem

What causes stress?

Developers

Have I done a good job? (want to show off)

Will it work (properly)/(forefront of technology)

Project managers

Will we meet our deadlines? (promised resources)

How do good testers affect these concerns?

Find lots of bugs, makes everything worse! worse for developers: find bugs, have to fix not move on

worse for manager: delay project, need more resources


42/52

43

Symptoms: developers

What do you mean, it doesnt work? It works on my machine

It worked yesterday

I see no bugs Defensiveness: thats not a bug, its a feature

People who find bugs are not nice

Dont find any [more], it will slow us down

Why dont you find something important instead?

You mis-used the system - a real user would never dothat


43/52

44

What causes stress for testers?

Will we find all the bugs?

How much testing is enough?

Have we tested everything?

Not enough time to do a proper job

Why should I have to fight the developers I just report their own bugs, why should they hate me?

Will I get any recognition How can anyone tell Ive done a good job?

Guilt about bugs not found

What if bugs I found werent fixed - is it my fault?


44/52

Found it Found it Found it

Im sure

this is the

last one!

Oops!

Oh dear!

#@&*%!

The last bug

Oh No!


45/52

46

Symptoms: testers

I want to be your friend

feel guilty about finding bugs in my friends work

Its never enough

want to keep testing forever / Im not done yet

Its all my fault

feel guilty about any bugs missed in testing

we should continually improve, but cant get them all

remember, you didnt put them in!


46/52


47/52

48

Symptoms: test managers

Conflict of interest

want to encourage my team to find faults

do a good job if we find lots

my manager doesnt want us to delay the project do a good job if we dont find many

paradox

if you do your job properly, youre not a team player if youre not a team player, youre not doing your job

properly

dont want to challenge developers (too much)


48/52

49

And finally ...

Some recent research into brain function

testers brain

developers brain

managers brain

Sales hype resistance


49/52

The Managers Brain

Deadline

awareness

hemisphereGolf

centre

Listening skills

Quality awareness cell

Testing awareness micron Unknown

AMB

Concern for staff

Belief in

teams ability

Belief in own ability

Logical deduction particle

Sales hype resistance

Meet-

ings

Neurological Research

Grove Consultants,2000

New feature enthusiasm zone


50/52

Molehill to

mountain

converter

Stupid

question

generator

Alarmist area

Unlikely inputgenerator

Nit-

picking

zone

Diplomacy neurone

Positive

attitude

particle

New feature enthusiasm zone

Appreciation of

developer genius

Accepting

criticism

cell

Detailed technical

understanding

of code

Deadline

awarenes

micron

The Testers Brain

Enjoying

bad news

Bug

sniffing

segment

Boredom

nucleus



Belief in own fallibility


51/52

The Developers Brain

New

feature

area

Over-complication

centre

Bug to feature

conversion

hemisphere

Bug

insertion

lobe

Magneticattraction to

newest coolest

development

tools

Optimism gland

Documentation

particle

Testing skill nucleus

Users wouldntdo that nerve centre

Bug removalneurone

Accurate

estimation

nerveDesire to learn from

previous project

Interpersonal skills

synapse




52/52

Questions & Answers

Thanks You

Documents

Testing Awareness