Upload
dao-anh-hien
View
232
Download
0
Embed Size (px)
Citation preview
8/6/2019 ICST Tutorial
1/83
Software Testing:A Newcomers Guide
Paul Ammann
Software Engineering Group
George Mason University
Presented at ICST
April 2, 2009
Denver, Colorado
Slides Co-Produced with Jeff Offutt
8/6/2019 ICST Tutorial
2/83
Software Testing: A Newcomers Guide 2009 22
Here! Test This!
Testing two decades ago
A stack of computer printoutsand no documentation
Testing a decade agoTesting Now?
A disk and no documentationA complex conglomeration only low level documentation
8/6/2019 ICST Tutorial
3/83
Software Testing: A Newcomers Guide 2009 33
A Talk in 4 Parts
1. Why do we test ? Costs and Attitudes
2. What should we do during testing ? Test Activities
Software Testing Terms
Evolving Notion of Coverage Criteria
3. How do we get to this future of testing ? Beizers Maturity Levels
4. Background for ICSTsessions
We are in the middle of aWe are in the middle of a revolutionrevolution in how software is testedin how software is tested
ResearchResearch isis finallyfinally meeting practicemeeting practice
8/6/2019 ICST Tutorial
4/83
Software Testing: A Newcomers Guide 2009 44
Part 1 : Why Test?
Written test objectives are rare or useless
The software shall be easily maintainable What fact is each test trying to verify?
Requirements definition teams should include testers!
Ensures testing can satisfy test objectives
Provides rationale for tests
How much testing is enough?
If you dont knowIf you dont know whywhy youre conductingyoure conductinga test, it wont be very helpfula test, it wont be very helpful
8/6/2019 ICST Tutorial
5/83
Software Testing: A Newcomers Guide 2009 55
Cost of Testing
In the real-world, testing is the principle post-design activity
Restricting early testing usually increases cost
Extensive hardware-software integration requires more testing
Youre going to spend at least half ofYoure going to spend at least half ofyour development budget on testing,your development budget on testing,whether you want to or notwhether you want to or not
8/6/2019 ICST Tutorial
6/83
Software Testing: A Newcomers Guide 2009 66
Cost of Not Testing
Not testing is even more expensive
Planning for testing after development is prohibitivelyexpensive
Hardware developers spend huge amounts on testing
Software test tools arent that expensive. Many are free!!!
Program Managers often say:Program Managers often say:Testing is too expensive.Testing is too expensive.
8/6/2019 ICST Tutorial
7/83
Software Testing: A Newcomers Guide 2009 77
Caveat: Impact of New Tools andTechniques
Theyre teaching a new wayof plowing over at the Grangetonight - you going?
Naw - I alreadydont plow as goodas I know how...
Good Techniques and Tools are Not Enough:Good Techniques and Tools are Not Enough:
Practitioners Need to Adopt Them Too!Practitioners Need to Adopt Them Too!
8/6/2019 ICST Tutorial
8/83
Software Testing: A Newcomers Guide 2009 88
Part 2 : What ?
But But WWhathat should we do ?should we do ?
1. Types of test activities
2. Software testing terms
3. Changing notions of testing
test coverage criteria
criteria based on structures
8/6/2019 ICST Tutorial
9/83
Software Testing: A Newcomers Guide 2009 99
Testing in the 21st Century
We are going through a time of change Software Defines Behavior
network routers
financial networks
telephone switching networks
other infrastructure
Embedded Control Applications
airplanes, air traffic control
spaceships
watches
ovens remote controllers
Safety critical, real-time software
Web apps must be highly reliable
And of course security is now all about software faults !
PDAs
memory seats
DVD players
garage door openers
cell phones
Testing ideas haveTesting ideas have
matured enough tomatured enough to
be used in practicebe used in practice
8/6/2019 ICST Tutorial
10/83
Software Testing: A Newcomers Guide 2009 10
Test Activities Testing can be broken up into four general types of activities
1. Test Design2. Test Automation
3. Test Execution
4. Test Evaluation
Each type of activity requires different skills, backgroundknowledge, education and training
No reasonable software development organization uses the samepeople for requirements, design, implementation, integration
and configuration control
10
Why do test organizations still use the same peopleWhy do test organizations still use the same people
for all four test activities??for all four test activities??
This is clearly aThis is clearly a wastewaste of resourcesof resources
1.a) Criteria-based1.b) Human-based
8/6/2019 ICST Tutorial
11/83
Software Testing: A Newcomers Guide 2009 11
1. Test Design (a) Criteria-Based
This is the most technicaljob in software testing
Requires knowledge of :
Discrete math Programming
Testing
Requires much of a traditional CS degree
This is intellectually stimulating, rewarding, and challenging
Test design is analogous to software architecture on thedevelopment side
Using people who are not qualified to design tests is a sure way toget ineffective tests
11
Design test values to satisfy coverage criteriaDesign test values to satisfy coverage criteria
or other engineering goalor other engineering goal
8/6/2019 ICST Tutorial
12/83
Software Testing: A Newcomers Guide 2009 12
1. Test Design (b) Human-Based
This is much harder than it may seem to developers
Criteria-based approaches can be blind to special situations
Requires knowledge of : Domain, testing, and user interfaces
Requires almost no traditional CS
A background in the domain of the software is essential
An empirical background is very helpful (biology, psychology, )
A logic background is very helpful (law, philosophy, math, )
This is intellectually stimulating, rewarding, and challenging
But not to typical CS majors they want to solve problems and buildthings
12
Design test values based on domain knowledge ofDesign test values based on domain knowledge of
the program and human knowledge of testingthe program and human knowledge of testing
8/6/2019 ICST Tutorial
13/83
8/6/2019 ICST Tutorial
14/83
Software Testing: A Newcomers Guide 2009 14
3. Test Execution
This is easy and trivial if the tests are well automated
Requires basic computer skills
Interns
Employees with no technical background
Asking qualified test designers to execute tests is a sure way toconvince them to look for a development job
If, for example, GUI tests are not well automated, this requires alot ofmanual labor
Test executors have to be very careful and meticulous withbookkeeping
14
Run tests on the software and record the resultsRun tests on the software and record the results
8/6/2019 ICST Tutorial
15/83
Software Testing: A Newcomers Guide 2009 15
4. Test Evaluation
This is much harder than it may seem
Requires knowledge of :
Domain
Testing
User interfaces and psychology
Usually requires almost no traditional CS
A background in the domain of the software is essential
An empirical background is very helpful (biology, psychology, )
A logic background is very helpful (law, philosophy, math, )
This is intellectually stimulating, rewarding, and challenging
But not to typical CS majors they want to solve problems and buildthings
15
Evaluate results of testing, report to developersEvaluate results of testing, report to developers
8/6/2019 ICST Tutorial
16/83
Software Testing: A Newcomers Guide 2009 16
Other Activities
Test management : Sets policy, organizes team, interfaces withdevelopment, chooses criteria, decides how much automation isneeded,
Test maintenance : Tests must be saved for reuse as softwareevolves
Requires cooperation of test designers and automators Deciding when to trim the test suite is partly policy and partly technical
and in general, very hard !
Tests should be put in configuration control
Test documentation : All parties participate
Each test must document why criterion and test requirement satisfiedor a rationale for human-designed tests
Traceability throughout the process must be ensured
Documentation must be kept in the automated tests
16
8/6/2019 ICST Tutorial
17/83
Software Testing: A Newcomers Guide 2009 17
Approximate Number of Personnel
A mature test organization may need only one test designer towork with several test automaters, executors and evaluators
Improved automation will reduce the number of test executors
Theoretically to zero but not in practice
Putting the wrong people on the wrong tasks leads to
inefficiency, lowjob satisfaction and lowjob performance A qualified test designer will be bored with other tasks and look for a job
in development
A qualified test evaluator will not understand the benefits of test criteria
Test evaluators have the domain knowledge, so they must be freeto add tests that blind engineering processes will not think of
17
8/6/2019 ICST Tutorial
18/83
8/6/2019 ICST Tutorial
19/83
Software Testing: A Newcomers Guide 2009 19
Applying Test Activities
19
To use our people effectivelyTo use our people effectively
and to test efficientlyand to test efficiently
we need a process thatwe need a process that
lets test designerslets test designersraise their level of abstractionraise their level of abstraction
8/6/2019 ICST Tutorial
20/83
Software Testing: A Newcomers Guide 2009 20
Model-Driven Test Design
20
softwareartifact
model /structure testrequirements
refined
requirements /test specs
inputvalues
testcases
testscripts
testresults
pass /fail
IMPLEMENTATION
ABSTRACTION
LEVEL
DESIGN
ABSTRACTION
LEVEL
8/6/2019 ICST Tutorial
21/83
8/6/2019 ICST Tutorial
22/83
Software Testing: A Newcomers Guide 2009 22
Model-Driven Test Design Activities
22
softwareartifact
model /structure testrequirements
refined
requirements /test specs
inputvalues
testcases
testscripts
testresults
pass /fail
IMPLEMENTATION
ABSTRACTION
LEVEL
DESIGN
ABSTRACTION
LEVEL
Test DesignTest Design
TestTest
ExecutionExecutionTestTest
EvaluationEvaluation
Raising our abstraction level makestest design MUCH easier
8/6/2019 ICST Tutorial
23/83
Software Testing: A Newcomers Guide 2009 23
Software Testing Terms
Like any field, software testing comes with a large number ofspecialized terms that have particular meanings in this context
Some of the following terms are standardized, some are used
consistently throughout the literature and the industry, but somevary by author, topic, or test organization
The definitions here are intended to be the most commonly used
23
8/6/2019 ICST Tutorial
24/83
Software Testing: A Newcomers Guide 2009 2424
Important TermsValidation & Verification (IEEE)
Validation : The process of evaluating software at the end ofsoftware development to ensure compliance with intendedusage
Verification : The process of determining whether the productsof a given phase of the software development process fulfill therequirements established during the previous phase
IV&V stands for independent verification and validation
8/6/2019 ICST Tutorial
25/83
Software Testing: A Newcomers Guide 2009 2525
Test Engineer & Test Managers
Test Engineer : An IT professional who is in charge of one ormore technical test activities
designing test inputs
producing test values
running test scripts
analyzing results
reporting results to developers and managers
Test Manager : In charge of one or more test engineers
sets test policies and processes
interacts with other managers on the project otherwise helps the engineers do their work
8/6/2019 ICST Tutorial
26/83
Software Testing: A Newcomers Guide 2009 2626
Test Engineer Activities
Test
Designs
Output
Executable
Tests
Computer EvaluateP
Test
Manager
Test
Engineer
Test
Engineer
design instantiate
execute
8/6/2019 ICST Tutorial
27/83
Software Testing: A Newcomers Guide 2009 2727
Static and Dynamic Testing
Static Testing : Testing without executing the program This include software inspections and many forms of analysis
Very effective at finding certain kinds of problems especially potential faults,that is, mistakes that may only be visible after the program evolves
Example faults identifiable with static analysis:
Overriding equals(), but not hashCode()
Implementing clone() by calling a constructor
Using foreign data prior to scrubbing
Hypertext inputs
Parts of SQL queries
Array indices
Dynamic Testing : Testing by executing the program with realinputs
Mostly what this talk is about
8/6/2019 ICST Tutorial
28/83
8/6/2019 ICST Tutorial
29/83
Software Testing: A Newcomers Guide 2009 2929
Testing & Debugging
Testing : Finding inputs that cause the software to fail
The principle topic of this lecture
Debugging : The process of finding a fault given a failure
Fault localization is a huge issue
Some faults have failures that are hard to reproduce
Often desirable to recast a problem report as a short test case that also
reveals the failure General principle: Faults usually have short revealing test cases
These test cases are excellent candidates for regression tests
8/6/2019 ICST Tutorial
30/83
Software Testing: A Newcomers Guide 2009 3030
Fault & Failure Model
Three conditions necessary for a failure to be observedThree conditions necessary for a failure to be observed
1. Reachability : The location or locations in the program thatcontain the fault must be reached
2. Infection : The state of the program must be incorrect
3. Propagation : The infected state must propagate to cause someoutput of the program to be incorrect
8/6/2019 ICST Tutorial
31/83
Software Testing: A Newcomers Guide 2009 3131
Test Case
Test Case Values : The values that directly satisfy one testrequirement
Expected Results : The result that will be produced whenexecuting the test if the program satisfies it intended behavior
8/6/2019 ICST Tutorial
32/83
8/6/2019 ICST Tutorial
33/83
Software Testing: A Newcomers Guide 2009 3333
Inputs to Affect Controllability andObservability
Prefix Values : Any inputs necessary to put the software intothe appropriate state to receive the test case values
Postfix Values : Any inputs that need to be sent to the softwareafter the test case values
Two types of postfix values
1. Verification Values : Values necessary to see the results of the test case values
2. Exit Commands : Values needed to terminate the program or otherwise return itto a stable state
Executable Test Script : A test case that is prepared in a formto be executed automatically on the test software and producea report
Example: Junit tests
8/6/2019 ICST Tutorial
34/83
Software Testing: A Newcomers Guide 2009 3434
Top-Down and Bottom-Up Testing
Top-Down Testing : Test the main procedure, then go downthrough procedures it calls, and so on
Bottom-Up Testing : Test the leaves in the tree (procedures thatmake no calls), and move up to the root.
Each procedure is not tested until all of its children have been tested
8/6/2019 ICST Tutorial
35/83
Software Testing: A Newcomers Guide 2009 3535
White-box and Black-box Testing
Black-box testing : Deriving tests from external descriptions ofthe software, including specifications, requirements, and design
White-box testing : Deriving tests from the source code internalsof the software, specifically including branches, individualconditions, and statements
This view is really out of date.This view is really out of date.
The more general question is:The more general question is:from what level of abstractionfrom what level of abstractionto we derive teststo we derive tests??
8/6/2019 ICST Tutorial
36/83
Software Testing: A Newcomers Guide 2009 3636
Evolving Notion of Coverage Criteria
Old view of testing is of testing at specificsoftware development phases
Unit, module, integration, system
New view is in terms ofstructures and criteria
Graphs, logical expressions, syntax, input space
8/6/2019 ICST Tutorial
37/83
8/6/2019 ICST Tutorial
38/83
Software Testing: A Newcomers Guide 2009 3838
Beizers Insight : Find a Graph and Cover It
Tailored to:
a particular software artifact
code, design, specifications
a particular phase of the lifecycle
requirements, specification, design, implementation
But graphs do not characterize all testing techniques well
Four abstract models seem to suffice
8/6/2019 ICST Tutorial
39/83
8/6/2019 ICST Tutorial
40/83
Software Testing: A Newcomers Guide 2009 4040
New : Criteria Based on Structures
1. Graphs
2. Logical Expressions
3. Input DomainCharacterization
4. Syntactic Structures
(not X or not Y) and A and B
if (x > y)
z = x - y;
else
z = 2 * x;
Structures : Four ways to model software
A: {0, 1, >1}
B: {600, 700, 800}
C: {swe, cs, isa, infs}
8/6/2019 ICST Tutorial
41/83
8/6/2019 ICST Tutorial
42/83
8/6/2019 ICST Tutorial
43/83
8/6/2019 ICST Tutorial
44/83
Software Testing: A Newcomers Guide 2009 4444
2. Logical Expressions
((a> b) orG) and(x
8/6/2019 ICST Tutorial
45/83
8/6/2019 ICST Tutorial
46/83
Software Testing: A Newcomers Guide 2009 4646
2. Logic Active Clause Coverage orMultiple Condition/Decision Coverage (MCDC)
((a>b)orG)and(x
8/6/2019 ICST Tutorial
47/83
8/6/2019 ICST Tutorial
48/83
8/6/2019 ICST Tutorial
49/83
8/6/2019 ICST Tutorial
50/83
8/6/2019 ICST Tutorial
51/83
8/6/2019 ICST Tutorial
52/83
8/6/2019 ICST Tutorial
53/83
Software Testing: A Newcomers Guide 2009 5353
Two Ways to Use Test Criteria
1. Directly generate test values to satisfy the criterion oftenassumed by the research community most obvious way to usecriteria very hard without automated tools
2. Generate test values externally and measure against thecriterion usually favored by industry
sometimes misleading
if tests do not reach 100% coverage, what does that mean?
Test criteria are sometimes called metrics
8/6/2019 ICST Tutorial
54/83
8/6/2019 ICST Tutorial
55/83
8/6/2019 ICST Tutorial
56/83
8/6/2019 ICST Tutorial
57/83
8/6/2019 ICST Tutorial
58/83
Software Testing: A Newcomers Guide 2009 5858
Beizers Maturity Levels
Level 0 : Theres no difference between testing and debugging
Level 1 : The purpose of testing is to show correctness
Level 2 : The purpose of testing is to show that the software
doesnt work Level 3 : The purpose of testing is not to prove anything specific,
but to reduce the riskof using the software
Level 4 : Testing is a mental discipline that helps all ITprofessionals develop higher quality software
8/6/2019 ICST Tutorial
59/83
8/6/2019 ICST Tutorial
60/83
Software Testing: A Newcomers Guide 2009 6060
Level 1 Thinking
Purpose is to show correctness
Correctness is impossible to achieve
What do we know ifno failures?
Good software or bad tests?
Test engineers have no: Strict goal
Real stopping rule
Formal test technique
Test managers are powerless
This is what hardware engineers often expectThis is what hardware engineers often expect
8/6/2019 ICST Tutorial
61/83
8/6/2019 ICST Tutorial
62/83
8/6/2019 ICST Tutorial
63/83
8/6/2019 ICST Tutorial
64/83
Software Testing: A Newcomers Guide 2009 6464
Summary
More testing saves money Planning for testing saves lots of money
Testing is no longer an art form Engineers have a tool box of test criteria
When testers become engineers, the product gets better The developers get better
8/6/2019 ICST Tutorial
65/83
8/6/2019 ICST Tutorial
66/83
8/6/2019 ICST Tutorial
67/83
Software Testing: A Newcomers Guide 2009 67
Part 4: Testing Topics by ICST Section
ICST has 16 Sessions with Papers: 13 Research and 3 Industry R1: GUI Testing
R2: Model Checking
R3: Embedded and Real-Time Testing
R4: QA and Test Management
R5: Model-Based Testing
R6: Static Analysis
R7: Security Testing
R8: Empirical Studies
R9: Test Case Generation
R10: Web Testing
R11: Aspects and Faults
R12: Mutation and Non Functional Testing
R13: Assertions and Failure States
I3: Real World Testing I2: Test Management
I3: Automation
Following slides briefly overview foundations for each Session Tall Order!
Ill be brief
67
8/6/2019 ICST Tutorial
68/83
Software Testing: A Newcomers Guide 2009 68
Section R1: GUI Testing
Basic Issues
Many systems have only a GUI interface
Problems may exist in GUI, or in underlying application
Structural Tests
Legal event sequences / hiding of uninteresting events
Model Based Tests Operational / Behavioral / Graph models
Research Foci
Case studies on industrial software
Reuse and scalability are huge issues
Particularly in the presence of maintenance
Crash testing
68
8/6/2019 ICST Tutorial
69/83
Software Testing: A Newcomers Guide 2009 69
Section R2: Model Checking
Basic Issues
Define a labeled state space with transitions: A state machine
Check (temporal) properties over paths in that state space
Bonus: Counterexamples
Standard Problem: Scalability
Research Foci Show two descriptions consistent (or identical)
Test case generation
Interpreting counterexamples as test cases
Using model checker as test generation engine
Model checking real code
Direct application to programs (eg JavaPathFinder)
69
8/6/2019 ICST Tutorial
70/83
Software Testing: A Newcomers Guide 2009 70
Section R3: Embedded and Real-Time Testing
Basic Issues
Embedded software has a huge observability problem
How do you know the exact state of the SUT?
Testing timeliness constraints as well as functional correctness
Research Foci
Abstract models for SUT Notions such as Partial Observability
Passive Testing
Non interference of test harness with observations
Test specification frameworks for specific domains
TTCN-3
70
8/6/2019 ICST Tutorial
71/83
8/6/2019 ICST Tutorial
72/83
8/6/2019 ICST Tutorial
73/83
8/6/2019 ICST Tutorial
74/83
8/6/2019 ICST Tutorial
75/83
S ti R9 T t C G ti
8/6/2019 ICST Tutorial
76/83
Software Testing: A Newcomers Guide 2009 76
Section R9: Test Case Generation
Basic Issues
Old, but very important problem
We need tools that can automatically generate good test data!
Research Foci
Generating feasible paths in finite state machines
Hard problem whether a given transition is, in fact, feasible Generating test inputs to follow paths
Key factors
Constraint solving
Path explosion
Relation of paths to other coverage goals
Test inputs to satisfy complex predicates
Efficient algorithms
76
8/6/2019 ICST Tutorial
77/83
8/6/2019 ICST Tutorial
78/83
8/6/2019 ICST Tutorial
79/83
S ti R13 A ti d F il St t
8/6/2019 ICST Tutorial
80/83
Software Testing: A Newcomers Guide 2009 80
Section R13: Assertions and Failure States
Basic Issues
Assertions are statements that must be true at given points in execution
Extremely powerful approach to fault detection
Failure states are program states after something bad happens.
Research Foci
Extracting tests from runtime failures
Automatically turning any failure (in operation) into a test
Huge potential practical impact
Clearly, overhead needs to be low
Assertion-based test validation
When software is modified, which assertions need to be checked?
Metamorphic testing
Solving the oracle problem by relating multiple program runs
Experience with implementing this approach via assertions
80
S ti I1 R l W ld T ti
8/6/2019 ICST Tutorial
81/83
Software Testing: A Newcomers Guide 2009 81
Section I1: Real World Testing
Basic Issue
Testability:
The degree to which components are easy to test
Components can be designed to be easier to test
The efficiency of the test process
Research Foci Why testability is hard inpractice
Organizational factors
Technical Factors
81
8/6/2019 ICST Tutorial
82/83
S ti I3 A t ti
8/6/2019 ICST Tutorial
83/83
Section I3: Automation
Basic Issues
Test generation
Code for multi-core processors
Specialized languages
Research Foci
Test generation to satisfy coverage criteria
Application to safety critical system
How to test code optimized for multi-core?
Testing multi-threaded is much harder than single-threaded!
Developing high level languages for specific domains
High level test approaches are a necessary complement