8/11/2019 Testing Awareness
1/52
1
Testing Awareness
8/11/2019 Testing Awareness
2/52
3
Agenda
Testing?
A Sample Test Process
A few Testing DefinitionsTest Process Flow
Test roles
Testing realities
8/11/2019 Testing Awareness
3/52
4
What is Testing ?Three integer values representing the lengths of the sides of a triangle, are
the input of the application. The output or result of the application is amessage that the triangle is scalene or equilateral or isocele
Define a set of tests that, to your feeling, would test this application
properly. Enrich your test descriptions by specifying the used set of data
?A B
C
Scalene : A < B + C
Isocele : B = C and A < B + C
Equilateral : A=B=C and A < B + C
8/11/2019 Testing Awareness
4/52
5
Test cases1. Do you have a test that represents a valid scalene triangle ?
2. Do you have a test that represents a valid equilateral triangle ?
3. Do you have a test that represents a valid isoceles triangle ?
4. Do you have all three permutations of the two equal sides of test 3 (e.g. 3,3,4; 3,4,3; 4,3,3
5. Do you have a test in which one input value is zero ?
6. Do you have a test in which one input is a negative integer ?
7. Do you have a test with all three inputs greater than zero, but with the sum of two of theinput values equal to the third (if the program sais that 1,2,3 represents a scalene trianglewould contain a bug) ?
8. Do you have at least all permutations of the previous test ?
9. Do you have a test with three integers greater that zero but with the sum of two of the
numbers less then the third ?
10. Do you have at least all permutations of the previous test ?
11. Do you have a test with all input values being zero ?
12. Do you have at least one test with non-integer inputs ?
13. Do you have a test with the number of input values (two rather then three) ?14. Did you specify for each test the expected output ?
8/11/2019 Testing Awareness
5/52
6
What is testing ?
Testing is executing a software with the
intent to find defect
Exercice feed Back
- Expected result must be define in advance
- Program Algorithm is different from test description
- Valid and Invalid statement must be tested
- Exhaustive testing can be expensive
-
8/11/2019 Testing Awareness
6/52
7
But where to stop ?
EXTENT OF TESTING
NUMBER OF
DEFECTS OPTIMUM
TEST
COST OF
TESTING
OVER
TESTUNDER
TEST
8/11/2019 Testing Awareness
7/52
8
Why is Test important ?
1 2,5 510
25
100
0
20
40
60
80
100
120
Require
ment
Design
Coding
UnitTe
st
Acceptance
Maintenance
Cost to remove an error
8/11/2019 Testing Awareness
8/52
9
Errors are unavoidable
What the client wanted What the contract said What the analyst designed
What the developer wrote What was delivered What was really needed
8/11/2019 Testing Awareness
9/52
10
Why Testing?
Defects are discovered sooner
Reduced correction costs!
Quality is evaluated early and iteratively in the
project
Increased visibility!
8/11/2019 Testing Awareness
10/52
11
Agenda
Testing ?
A Sample Test Process
A few Testing ConceptTest Process Flow
Test roles
Testing realities
8/11/2019 Testing Awareness
11/52
12
Valid for : Corporate Development(Finance, Governement, distribution,)
1 application is composed of 800 modules imbeded in asystem of 80 000 modules.
A project involve in average between 10 & twenty
people. Lot of activities on the same system in the same time
User involve in project and elaborate detailled
requirement while you develop You start always late but you must deliver on time.
8/11/2019 Testing Awareness
12/52
8/11/2019 Testing Awareness
13/52
14
Test stages and Test scenarios
creation
System Tests
Integration Tests
Unit Tests
Acceptance Tests
Software
Architecture (SAD)
Components
Design
Software
Requirements (SRS)
Business needs(Vision)
SCENARIOS CREATION
SCENARIOS CREATION
SCENARIOS CREATION
SCENARIOS CREATION
8/11/2019 Testing Awareness
14/52
15
Test stages
The test stages are the key to validate the code alongdifferent point of view and different level of details
Business
Technical The test stages have a huge impact on how the scenario
are written
Business oriented
Technically oriented
The test stages are related to the maturity of thesoftware only and independent of the phase
The 4 test stages may be present in each and every iteration
8/11/2019 Testing Awareness
15/52
16
Test concepts 2 : Test types
Most common test types Performance tests (Mainframe response time, Scalability)
Functional tests (Conformance to functionalrequirements)
Configuration tests (Different software and hardware onwhich the application will run)
Security tests (Role Based Security)
Non-regression tests (No new defects caused by new
code) Stress tests (Operate the system to its limit)
Recovery tests (Behavior in case of system break-down)
Usability tests (Help on-line, ergonomy,)
8/11/2019 Testing Awareness
16/52
17
Test types
Test types are the key to translate testexpectations in a concrete test strategy :
What tests will be performed.
When will which tests be performed.
What skills are needed.
What tools are needed.
8/11/2019 Testing Awareness
17/52
18
Test concepts 3 : Test Strategy
Test Strategy is
Best combination off test stages, test types and
application feature to test in order to :
- Anticipate RiskHigh risk first
- Address test requirement based on priority- Show quality achievement regularly during
project execution
8/11/2019 Testing Awareness
18/52
19
Agenda
Testing ?
A Sample Test Process
A few Testing Concept
Test Process Flow
Test roles
Testing realities
8/11/2019 Testing Awareness
19/52
20
Test process overview
Test Strategy & plan
[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of theproject]
Test Strategy and
Planning
Test Design and
Build
Test
ExecutionTest
Reporting
Close Test
Session
[Last test
cycle of the
iteration]
Plan
Designand Build
Executeand Report
Define testscenarios
Test model Frequent
reports
Completed during eachiteration to gain
maturity
8/11/2019 Testing Awareness
20/52
21
RUP Lifecycle - Test discipline
8/11/2019 Testing Awareness
21/52
22
Test Strategy and Planning During inception and
elaboration
Define a test strategy able tomeet stakeholders testexpectations.
Define the resources needed toundertake this strategy
Deliverables
An approved test plan
Optionally a Test Initiation
Checklist
During construction andtransition
Update and detail initial plan
Detailed schedule
Updated scope
[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of theproject]
Test Strategy and
Planning
Test Design and
Build
Test
ExecutionTest
Reporting
Close Test
Session
[Last test
cycle of the
iteration]
T S d
8/11/2019 Testing Awareness
22/52
23
Test Strategy and Planning
Output =Test Plan Summary of quality
expectations.
Test strategy
Test exit criteria
Test schedule
Reporting facilities
Resources needed bythe strategy
Which test types in which test
stages for which features?
When is testing finished?
Skills, metrics reported, audience,
periodicity
People, budget, tools, test
environments.
Quality Objectives
8/11/2019 Testing Awareness
23/52
8/11/2019 Testing Awareness
24/52
25
Each test scenario describes
Unique identifier
Detailed objective for thescenario
Steps to be executed
Corresponding expectedresults
Scenario priority
Unambiguously identify the scenario
for further reference (e.g. defect
reporting)
Assess success or failure of the
scenario
Procedure to be followed by the tester
To be compared against the actual
behavior, base for defect reporting.
Test Design and BuildOutput =Test scenarios
Importance of the scenario
8/11/2019 Testing Awareness
25/52
26
Test Execution and Reporting
Execute test scenarios
Register the defects
Report execution results
Deliverables Defects
Reports
[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of theproject]
Test Strategy and
Planning
Test Design and
Build
Test
ExecutionTest
Reporting
Close Test
Session
[Last test
cycle of theiteration]
T E i d R i
8/11/2019 Testing Awareness
26/52
27
A defect is reported in a defect tracking tool Difference
Status in the defect life-cycle
Unique identifier
eg. Test tracker, QAA, test director
Link with failed scenario Id
Defect tracking tool
Communication ! Registration
Statistics
A defect is linked to a particular scenario
Test Execution and Reporting
Output =Defects
Defect ARE Key :-Communication-Quality Measurment-Process Improvment
8/11/2019 Testing Awareness
27/52
28
Test Execution and Reporting
Output = ReportDefect Status Nb of defects
Open 15
Under correction 14
To be re-tested 35
Under re-test 3
Closed 45
#Defects per status
Open
Under correction
To be re-tested
Under re-test
Closed
There is a
bottleneck there
+ Written conclusions of the analysis
8/11/2019 Testing Awareness
28/52
29
Close Test Session
Publish in the test sign-offdocument to which extend
the quality expectations aremet
Defects pending
Features tested/ not tested
Acceptance criteria
[Not last
cycle of
the
iteration]
[Last
iteration
of the
project]
[Not last
iteration of theproject]
Test Strategy and
Planning
Test Design and
Build
Test
ExecutionTest
Reporting
Close Test
Session
[Last test
cycle of theiteration]
8/11/2019 Testing Awareness
29/52
30
Activities workload in each phase
Test strategy
and planning
Test Design andBuild
Test Execution
and Reporting
Close test
session
8/11/2019 Testing Awareness
30/52
31
Agenda
Testing ?
A Sample Test Process
A few Testing Concept
Test Process Flow
Test roles
Testing realities
8/11/2019 Testing Awareness
31/52
32
Test Roles
Test coordinator
Test analyst
Tester
8/11/2019 Testing Awareness
32/52
33
Roles - Test Coordinator
Plan and coordinate test activities and resources.
Control the test execution and report on a regular basis, thetest status and progress.
Assertive - Independent minded Owner of the test plan and the test model
Good relational skills
High workload during
Strategy and planning
Execution and reporting
8/11/2019 Testing Awareness
33/52
34
Roles - Test Analyst
Delivers the test model and test scenarios.
Deep understanding and knowledge of the project technicalspecifications and business requirements.
Skilled in Test Specification Techniques (Boundary analysis,decision tables, equivalence classes).
Owner of the test scenarios
Details oriented
High workload during design and build
8/11/2019 Testing Awareness
34/52
35
Roles - Tester
Execute scenarios, report execution results, and register defects.
Correct understanding and knowledge of the applicationtechnical and functional specifications and businessrequirements.
Eye for details and rigorous in his/her execution of the testscenario and defect reporting.
Good relational skills (will have to report defects and work ingood cooperation with their authors).
High workload during execution and reporting
8/11/2019 Testing Awareness
35/52
36
Summary: The building blocks
A test plan The project test objectives, the strategy to meet
them and the means required by this strategy
Test scenarios A detailed description of the steps to execute
and pre-defined expected behavior for each ofthem
A test model
The link between the features and the testscenarios
Frequent reporting
A clear description of what does and does notwork in the system
Role definition Who does what and when
Test strategy and testorganization
Detailed test execution
procedures and
corresponding expectations
Tracability and input
to reporting
Visibility
Ownership
8/11/2019 Testing Awareness
36/52
37
Some Testing realities !!
8/11/2019 Testing Awareness
37/52
38
1. Quality versus deadlines
what if deadlines were expressed in the same wayquality often is?
Yes, were really committed to deadlines. We need to finish
early -we mustnt be late. Exactly when? Well, Ill know it whenthe date arrives, but you cant tie it down like that. Oh yes, wehave a plan, but we dont actually put dates on it - but its veryimportant to achieve timely delivery.
Deception: I cant see it, so it isnt important
Need to make quality visible (quantified attributes)
Need to evaluate projects (and project managers) onquality achieved, not just on deadlines met
8/11/2019 Testing Awareness
38/52
39
Symptoms: Test Planning
2 weeks?
Content of test plans 6 weeks
Shorter one looks exactly the same inside
therefore it must not be any different in nature
2 weeks is better for my schedule
Cant see any difference, therefore there is none Solution: make detailed test activities visible
Not shorten a piece of string, cut off arm or leg!
testing
testing
8/11/2019 Testing Awareness
39/52
40
2. What does testing produce?
Development versus testing
can tell when development is finished
it produces something concrete and demonstrable
positive effect
testing doesnt change the product at all
neutral effect: still the same product
negative effects adds extra work when were pushed
didnt want to know about bugs anyway
Why testing is a hard sell
Illusion - testing
adds confidence
- or opportunity toimprove quality
8/11/2019 Testing Awareness
40/52
41
Sample: Y2K a waste of time
Nothing happened (much)
Therefore, there was never a real serious problem
The doom merchants were wrong
All that effort was a waste of money just like immunisation programmes
house/car/health insurance
national defense Does this remind you of anything?
Software testing in general?
8/11/2019 Testing Awareness
41/52
42
3. Relationship problem
What causes stress?
Developers
Have I done a good job? (want to show off)
Will it work (properly)/(forefront of technology)
Project managers
Will we meet our deadlines? (promised resources)
How do good testers affect these concerns?
Find lots of bugs, makes everything worse! worse for developers: find bugs, have to fix not move on
worse for manager: delay project, need more resources
8/11/2019 Testing Awareness
42/52
43
Symptoms: developers
What do you mean, it doesnt work? It works on my machine
It worked yesterday
I see no bugs Defensiveness: thats not a bug, its a feature
People who find bugs are not nice
Dont find any [more], it will slow us down
Why dont you find something important instead?
You mis-used the system - a real user would never dothat
8/11/2019 Testing Awareness
43/52
44
What causes stress for testers?
Will we find all the bugs?
How much testing is enough?
Have we tested everything?
Not enough time to do a proper job
Why should I have to fight the developers I just report their own bugs, why should they hate me?
Will I get any recognition How can anyone tell Ive done a good job?
Guilt about bugs not found
What if bugs I found werent fixed - is it my fault?
8/11/2019 Testing Awareness
44/52
Found it Found it Found it
Im sure
this is the
last one!
Oops!
Oh dear!
#@&*%!
The last bug
Oh No!
8/11/2019 Testing Awareness
45/52
46
Symptoms: testers
I want to be your friend
feel guilty about finding bugs in my friends work
Its never enough
want to keep testing forever / Im not done yet
Its all my fault
feel guilty about any bugs missed in testing
we should continually improve, but cant get them all
remember, you didnt put them in!
8/11/2019 Testing Awareness
46/52
8/11/2019 Testing Awareness
47/52
48
Symptoms: test managers
Conflict of interest
want to encourage my team to find faults
do a good job if we find lots
my manager doesnt want us to delay the project do a good job if we dont find many
paradox
if you do your job properly, youre not a team player if youre not a team player, youre not doing your job
properly
dont want to challenge developers (too much)
8/11/2019 Testing Awareness
48/52
49
And finally ...
Some recent research into brain function
testers brain
developers brain
managers brain
Sales hype resistance
8/11/2019 Testing Awareness
49/52
The Managers Brain
Deadline
awareness
hemisphereGolf
centre
Listening skills
Quality awareness cell
Testing awareness micron Unknown
AMB
Concern for staff
Belief in
teams ability
Belief in own ability
Logical deduction particle
Sales hype resistance
Meet-
ings
Neurological Research
Grove Consultants,2000
New feature enthusiasm zone
8/11/2019 Testing Awareness
50/52
Molehill to
mountain
converter
Stupid
question
generator
Alarmist area
Unlikely inputgenerator
Nit-
picking
zone
Diplomacy neurone
Positive
attitude
particle
New feature enthusiasm zone
Appreciation of
developer genius
Accepting
criticism
cell
Detailed technical
understanding
of code
Deadline
awarenes
micron
The Testers Brain
Enjoying
bad news
Bug
sniffing
segment
Boredom
nucleus
Neurological Research
Grove Consultants,2000
Belief in own fallibility
8/11/2019 Testing Awareness
51/52
The Developers Brain
New
feature
area
Over-complication
centre
Bug to feature
conversion
hemisphere
Bug
insertion
lobe
Magneticattraction to
newest coolest
development
tools
Optimism gland
Documentation
particle
Testing skill nucleus
Users wouldntdo that nerve centre
Bug removalneurone
Accurate
estimation
nerveDesire to learn from
previous project
Interpersonal skills
synapse
Neurological Research
Grove Consultants,2000
8/11/2019 Testing Awareness
52/52
Questions & Answers
Thanks You