Upload
cnshariff
View
221
Download
0
Embed Size (px)
Citation preview
7/30/2019 Ch1_A Perspective on Testing
1/41
Ch. 1 A Perspective on Testing
Definitions and Concepts of Testing and Quality
What is
Testing?
What is
Quality ?
HowRelated?
1. Does finding a Lot of Problems through testing improveQuality? Ch. 1 A Perspective on Testing -
Sum2012Bernal
1
7/30/2019 Ch1_A Perspective on Testing
2/41
Introduction to Testing Software Testing is a relatively new discipline although
programmers always debugged their programs.
Testing was conducted to show that software works.
In the 1970s Glen Myers wrote a book, Art ofSoftware Testing (1978).
He believed that the main purpose of testing is to findfaults.
Are these (works .vs. faults) opposing views?
2. Why do we and why should we Test ?Ch. 1 A Perspective on Testing -
Sum2012Bernal
2
7/30/2019 Ch1_A Perspective on Testing
3/41
Why Test ?
Because we cant assume that the software works
How do we answer the question:Does this software work?
Is it reliable ? (system is continuously functioning > 720hrs) Is it functional ?
Complete Consistent
Is it responsive ? (response time < 1 second)
In general, what is the Quality of our software?
How do we answer this?
3. How would you answer this about the program that you wrote?How would you answer this about the program your friend wrote?
Ch. 1 A Perspective on Testing -
Sum2012Bernal
3
7/30/2019 Ch1_A Perspective on Testing
4/41
Quality of Software?
Depends on what we include as software
Just the executable code ----- How about ?
the pre-populated data in the database
the help text and error messages
the source logic code
the design document
the test scenarios and test results
the reference manuals
the requirements document
When we talk about quality how much of the above(do we / should we) include?
How would you test these different artifacts?
Ch. 1 A Perspective on Testing -
Sum2012Bernal
4
7/30/2019 Ch1_A Perspective on Testing
5/41
Approaches to ---Testing Software
Some of us hope that our software works as
opposed to ensuring that our software works?Why? Just foolish
Lazy
Believe that its too costly (time, resource, effort, etc.)
Lack of knowledge
DO NOT use the I feel lucky or I feel confident approachto testing - - - - although you may feel that way sometimes.
Use a methodical approach to testing to back up the
I feel lucky/confident feeling
Methods and metrics utilized must show VALUE Value, unfortunately, often are expressed in negative terms
Severe problems that cost loss of lives or business
Problems that cost more than testing expenses and effort5
7/30/2019 Ch1_A Perspective on Testing
6/41
3 major Testing approaches
1. Traditionally, testing includes executing the code with test cases.
(assuming - code is the main software artifact)
2. What do we do with the non-executable software artifacts?
Reviews and Inspections
- Expensive in terms of human resources- A lot to maintain and keep updated
3. Can we provethat the software works or is defect free?
- Theoretically, given an arbitrary program we can not show that it has nobug.
- We can use Formal Proofs to show the behavior of code
6
7/30/2019 Ch1_A Perspective on Testing
7/41
An Informal Survey in the 90s Professionals taking a course in testing:
60% were new to testing 20% had 1 to 5 years of experience in testing 20 % expert testers
Metric used in testing a) Most regularly used: Counting bugs found and ranking them by
severity b) Small number used : bug find rate or bug fix rate
Formal Methods used Almost none formally trained in inspection or analysis
Test tool: 76 % had been exposed to some automated test tool
Test definitions Most of the practicing testers could not supply good definitions of
testing terms; they just did the work!Ch. 1 A Perspective on Testing -
Sum2012Bernal
7
7/30/2019 Ch1_A Perspective on Testing
8/41
Historical Progression of Attitudes onSoftware Quality
-PC and Desktop Computingbecame ubiquitous-multiple vendors-quicker product development &shorter term investment-systems ran by non-computerindividuals
1990s
**New product was fashionable
& reboot became acceptable.**
-Web availability to many-Business conducted on the Web
-Software and systems are nothobbies but a business again
Late 1990s & 2000s
**Product Reliability & Qualityis once again important** --
especially SECURITY--
-Large Host & Centralized
Systems-Single Vendor (hdw-sw-serv)-Long term development andlong term investment(10 yrs)-Single platform-Systems ran by computerprofessionals
1980s & Before
**Product Reliability & Qualitywas required and expected**
Even though we didntget good quality
Ch. 1 A Perspective on Testing -
Sum2012Bernal
8
7/30/2019 Ch1_A Perspective on Testing
9/41
Current State (2000-2010s) on Testing
1. Software is written by many with the
entrepreneurial spirit: Speed to market New & innovation is treasured Small organization that cant afford much more than
coders
2. Embracing Agile process and mistaking it as fastproduction regardless of what:
Not much documented (requirements/design) Hard to test without documented material
3. Lack of Trained/Good/Experienced Testers Testers are not quite rewarded as equals to designers,but definitely gaining on and sometimes surpassingprogrammers(takes good cops to catch the thieves)
4. Improvement in tools and standards makingdevelopment easier and less error prone
Why ?
How ?Ch. 1 A Perspective on Testing -
Sum2012Bernal
9
7/30/2019 Ch1_A Perspective on Testing
10/41
Is Quality still an Issue, today?
What is Quality?
Some common comments:
I know it when I see it
ReliableMeets requirements
Fit for use
Easy to use
ResponsiveFull function / full feature
Classy and luxurious
How would you define quality?Ch. 1 A Perspective on Testing -
Sum2012Bernal
10
7/30/2019 Ch1_A Perspective on Testing
11/41
Some U.S. pioneers on Quality
Pioneers:Joseph M. Juran
Quality =>Fitness for use
W. Edward Deming Quality => Non-faulty system
More recently: Philip Crosby
Quality => conformance with requirements Achieve quality via prevention of error
Target of Quality is zero defect
Measure success via cost of qualityCh. 1 A Perspective on Testing -
Sum2012Bernal
11
7/30/2019 Ch1_A Perspective on Testing
12/41
ISO/IEC 9126 (1994) Quality model
Quality is composed of severalcharacteristics:
FunctionalityReliabilityUsabilityEfficiencyMaintainability
Portability
How do you know if it is achieved & how would you test for these?Do these have to be specified in the requirements? ---- if they arenot-------- do we ask for these during review/inspection?
What do these mean ?
Ch. 1 A Perspective on Testing -
Sum2012Bernal
12
7/30/2019 Ch1_A Perspective on Testing
13/41
Quality
Quality is a characteristic or attribute
Needs to be clearly defined and agreed to
May have sub-attributes (e.g. previous page)
Needs specific metrics for each of the sub-attributesNeeds to be measured with the defined metric(s)
Needs to be tracked and analyzed
Needs to be projected
Needs to be controlled
Testing and Measurement are two keyactivities that would help us manage quality.
Explain ----- the relationship ?
Imagine what it is likewithout this.
Ch. 1 A Perspective on Testing -
Sum2012Bernal
13
7/30/2019 Ch1_A Perspective on Testing
14/41
Summarizing major Competitors to Quality
Schedule (first to market)
Requirements (bad or missing)
New and Exciting (demands of WANT not need)
Price and Availability (retail customers)
Ch. 1 A Perspective on Testing -
Sum2012Bernal
14
7/30/2019 Ch1_A Perspective on Testing
15/41
Some Precept Concerning Quality & QA Quality requirements Does Not always dictate schedule !
Market Conditionoftendictates schedule (especially for smallcompanies)
BUT For large and multiple release software, quality is still a factor and may
affect schedule ---- albeit schedule is seldomly changed for quality Software development process today incorporates both the need for
speed and quality (incorporating the notion of ) a) service cost and b) rewrite for a replacement new product.
Quality does not require zero defect Reliability
Commercial (non-life critical or mission critical) products are notdeveloped with zero defect goal in mind. They are much more marketdriven --- market prefers but does not demand zero defects.
Focus on proper support Focus on main functions and heavily used areas (not all defects are the same) Focus on customer productivity (e.g. easy to learn and use) Zero Defect is very expensive proposition ( time & resource)Ch. 1 A Perspective on Testing -
Sum2012Bernal
15
7/30/2019 Ch1_A Perspective on Testing
16/41
Users Do Not Always Know What
They Want Users may not know all the requirements (especially for large,
complex systems which require professional or legalknowledge.)
Users may not have the time or interest to really focus on
the requirements at the time when asked (timing problem).Users have their own fulltime jobs.
Users may not know how to prioritize needs from wishes
Users may not know how to articulate clearly all therequirements. (They are non-software development people.)
Developers may not listen well or properly interpret the usersstatements. (They are not industry specialists.)
Requirements is a key factor in software development ---- why?How does it affect software quality? ----- think about definitions of Quality--- meets requirements
Ch. 1 A Perspective on Testing -
Sum2012Bernal
16
7/30/2019 Ch1_A Perspective on Testing
17/41
Requirements are not always Clear,
Stable, and Doable Not all requirements are technically feasible; sometimes the
desired new technology needs to be prototyped first.
Sometimes the requirements are changed, causing re-design or re-code without proper assessment of schedule
impact. Requirements are not always reviewed and signed off, but
sometimes given in verbal form --- especially small changes.
People mistake iterative development to mean continuouschange of requirements.
Whats the danger here? cost, schedule, quality
Ch. 1 A Perspective on Testing -
Sum2012Bernal
17
7/30/2019 Ch1_A Perspective on Testing
18/41
Customers often go for New and
Exciting Product If the product has all the needed features it would sell ---
is not necessarily true; people often WANTnew & extrafeatures.
Reliability is not always enough; sometimes customers willsacrifice quality for new and exciting features.
The story of IBM OS/2 operating system and Microsofts DOSoperating system (even though both was commissioned byIBM).
IBM went for Reliability of the old Host Machines for desktop PCs
Microsoft went for exciting individual user interfaces
Over-emphasis of exciting features is one reason why weare regressing a little in software quality in the last ten years !
**Still, consider Apple i-phone success in spite of its activation & other problemsCh. 1 A Perspective on Testing -
Sum2012Bernal
18
7/30/2019 Ch1_A Perspective on Testing
19/41
Price and Availability is sometimesmore important to customers
(especially for commodity levelsoftware) than Product Maturity orQuality.
At the commodity level software, the customers are
individuals who wants the product NOW at an competitiveprice. (much like shopping for a home appliance such as acoffee maker, T.V. or an i-phone)
Sophisticated and full feature software needs to bebalanced and sometimes traded off for price and speed.
Customers dont always need all the functions andproduct maturity they think they require ---- if the price isright!
Ch. 1 A Perspective on Testing -
Sum2012Bernal
19
7/30/2019 Ch1_A Perspective on Testing
20/41
Summarizing: majorCompetitors and Adversaries
to Qualitywhen developing software
Schedule (first to market)
Requirements (bad or missing)
New and Exciting (demands of WANTnot need)
Price and Availability (retail customers)
Ch. 1 A Perspective on Testing -
Sum2012Bernal
20
7/30/2019 Ch1_A Perspective on Testing
21/41
Start of Quality Assurance
As software size and complexity increased in the1960s and 1970s, many software projects started tofail.
Many software did not perform the required functions Others had performance (speed) problems
Some had large number of defects that prevented users fromcompleting their work; some just flat out wont even install !
Software Quality Crisis was recognized and QualityAssurance was born, along with the term SoftwareEngineering (1968 NATO conference).
Ch. 1 A Perspective on Testing -
Sum2012Bernal
21
7/30/2019 Ch1_A Perspective on Testing
22/41
Software Quality Assurance (QA)
Software QA focused on 2 main areas
Software productSoftware process
The focus on the process areas borrowedmany techniques from the traditionalmanufacturing area
Concept of reliability (number of defects, mean timeto failure, probability of failure, etc. metrics)
Concept of process control in terms of looking atrepeatability of process--- repeatable processproduces similar product (or controllable results).
emphasis today : e.g. CMM & CMMI
Ch. 1 A Perspective on Testing -
Sum2012Bernal
22
7/30/2019 Ch1_A Perspective on Testing
23/41
QA, Process Control, and Documentation
A period of heavy emphasis on softwaredevelopment process and excessivedocumentation dominated QA --- initially thisimproved the software crisis.
1. Process included multiple steps of reviews2. Process included multiple steps of testpreparation,
testing, and test result analysis3. Process was controlled by many documentsand
document flow which also improved projectcommunications
But ----- the price paid was a) speed and b) innovation.
Ch. 1 A Perspective on Testing -
Sum2012Bernal
23
7/30/2019 Ch1_A Perspective on Testing
24/41
Software Development is NOTManufacturing (or is it?)
Software Development is extremely laborintensive
People are not uniform like machines used in
manufacturing.
Software Development often requiresinnovation
Every software seems to be a one of its kindalthough more and more are becomingstandardized
The same set of people do not get to repeatedlydevelop the exactly same software multiple times.Ch. 1 A Perspective on Testing -
Sum2012Bernal
24
7/30/2019 Ch1_A Perspective on Testing
25/41
Improve Documentation (Availability andPresentation Technique)
Improve the creation and maintenance of documentsvia an on-linerepository or configuration manager
Centrally protected data Sharable data
Managed data (data relationships are managed)
Improve the usability and productivity byproviding more and better visualization of data
Replace numbers with graphs and figures Replace words with pictures
track, project, control
Ch. 1 A Perspective on Testing -
Sum2012Bernal
25
7/30/2019 Ch1_A Perspective on Testing
26/41
Improve Testers Tools and Methodology
Test Methodology Improvements Test coverage analysis
Test case generation
Test-Fix-Integrate Process
Test results analysis
Test metrics definition and measurements process Etc.
Test tools improvements Test coverage computation
Test trace
Test script generator Test result records keeping and automated analysis
Build and integration (daily builds)
Etc.
Ch. 1 A Perspective on Testing -
Sum2012Bernal
26
7/30/2019 Ch1_A Perspective on Testing
27/41
Perspective on Testing
Today we test because we know that systemshave problems - - - we are fallible.
1. To find problems and find the parts that do notwork
2. To understand and show the parts that do work
3. To assess the quality of the over-all product (Amajor QA and release management responsibility)
Ch. 1 A Perspective on Testing -
Sum2012Bernal
27
7/30/2019 Ch1_A Perspective on Testing
28/41
Some Talking Definitions(IEEE terminology based)
Error
A mistake made by a human The mistake may be in the requirements, design, code, fix,
integration, or install
Fault
Is a defect or defects in the artifact that resulted from an error
There may be defects caused by errors made that may or maynot be detectable (e.g. error of omission)
Failure
Is the manifestation of faultswhen the software is executed.
Running code
May show up in several places
May be non-code related (e.g. reference manual)
Incident
Is the detectable symptom of failures
(Not in the text)
(includes error of omission and no-code?)
Note
Example? (bank accnt)
Ch. 1 A Perspective on Testing -
Sum2012Bernal
28
7/30/2019 Ch1_A Perspective on Testing
29/41
Testing in the Large
Testing is concerned with all, but may not beable to detect allErrorsFaultsFailures
Incidents Testing utilize the notion of test cases to
perform the activities of test(s) Inspection of non-executablesExecuting the codeAnalyzing and formally proving the non-
executables and the executable in a businessworkflow (or user) setting
Ch. 1 A Perspective on Testing -
Sum2012Bernal
29
7/30/2019 Ch1_A Perspective on Testing
30/41
Software Activities and Error Injections, FaultPassing, and Fault Removal
Requirements
Design
Code
Testing
Error
Error
Error
Fixing
Error
faults
faults
Faults/failures
Inspection
Inspection
Fixing
Note that in fixing failures, one can commit errors and introduce faultsCh. 1 A Perspective on Testing -
Sum2012Bernal
30
7/30/2019 Ch1_A Perspective on Testing
31/41
Recording each Test Case
Test Case number
Test Case author
A general description of the test purpose
Pre-condition
Test inputs
Expected outputs (if any)
Post-condition
Test Case history:
Test execution date
Test execution person
Test execution result (s)
Ch. 1 A Perspective on Testing -
Sum2012Bernal
31
7/30/2019 Ch1_A Perspective on Testing
32/41
Specification vs Implementation
Specification Implementation
Expected Actual
The ideal place where expectation and actual matchesThe other areas are of concern ---- especially to testers
Ch. 1 A Perspective on Testing -
Sum2012Bernal
32
7/30/2019 Ch1_A Perspective on Testing
33/41
Specification vs Implementation vs Test Cases
Area S = Yellow=SpecificationArea P = Blue=Programmed-Implementation
Area T = Pink=Test CasesWhat do thesenumbered regionsmean to you?
S=Spec->Expected P=Program->Actual
T=Tested
56
2
4
1
3
7
8
Ch. 1 A Perspective on Testing -
Sum2012Bernal
33
7/30/2019 Ch1_A Perspective on Testing
34/41
Black Box vs White Box code testing
Black box testing (Functional Testing)Look at mainly the input and outputs
Mainly uses the specification as the source fordesigning test cases.
The internal of the implementation is not includedin the test case design.
Hard to detect missing specification
White box testing (Structural Testing)Look at the internals of the implementation
Design test cases based on the design and codeimplementation
Hard to detect extraneous implementation thatwas never specified
Ch. 1 A Perspective on Testing -
Sum2012Bernal
34
7/30/2019 Ch1_A Perspective on Testing
35/41
Recording Test Results
Use the same form describing the test case --- seeearlier slide on recording test case and expand theresults to include:
State Pass or Fail on the Execution Result line
If failed:
1. Show output or some other indicator to demonstrate the faultor failure
2. Assess and record the severity of the fault or failure found
Ch. 1 A Perspective on Testing -
Sum2012Bernal
35
7/30/2019 Ch1_A Perspective on Testing
36/41
Failure Classification (Tsui)
Very High severity brings the systemsdown or a function is non-operational andthere is no work around
High severity a function is not operationalbut there is a manual work around
Medium severity a function is partiallyoperational but the work can be completed
with some work around Low severity minor inconveniences but the
work can be completed
Ch. 1 A Perspective on Testing -
Sum2012Bernal
36
7/30/2019 Ch1_A Perspective on Testing
37/41
Fault Classification (Beizer)
Mild misspelled word
Moderate - misleading or redundant info
Annoying truncated names; billing for $0.00
Disturbing some transactions not processed
Serious - lose a transaction Very serious - incorrect transaction execution
Extreme Frequent very serious errors
Intolerable - database corruption
Catastrophic System shutdown
Infectious - Shutdown that spreads to others
Ch. 1 A Perspective on Testing -
Sum2012Bernal
37
Increasingseverity
7/30/2019 Ch1_A Perspective on Testing
38/41
IEEE list of anomalies (faults)
Input/output faults
Logic faults
Computation faults
Interface faults
Data faults
Why do you care about these types of faults (results of errors made)?
Because they give us some ideas of what to look for indesigning future test cases --- in process of for otherproducts developed by same people
Ch. 1 A Perspective on Testing -
Sum2012Bernal
38
7/30/2019 Ch1_A Perspective on Testing
39/41
Different Levels of Testing
Program unit A
Program unit B
Program unit T
.
.
.
Function 1
Function 2
Function 8
.
.
Component 1
WholeSystem
Component 3
.
Unit Testing Functional Testing Component TestingSystemTesting
Ch. 1 A Perspective on Testing -
Sum2012Bernal
39
7/30/2019 Ch1_A Perspective on Testing
40/41
Still Need to Demonstrate Value of Testing Catastrophic problems (e.g. life or business ending ones)
do not need any measurements---but--- others do:
Measure the cost of problems found by customers Cost of problem reporting/recording
Cost of problem re-creation
Cost of problem fix and retest
Cost of solution packaging and distribution
Cost of managing the customer problem-to-resolution steps
Measure the cost of discovering the problems and fixing them priorto release
Cost of planning reviews and testing
Cost of executing reviews and testing
Cost of fixing the problems found and retest
Cost of inserting fixes and updates
Cost of managing problem-to-resolution steps
Compare the above two costs AND include loss ofcustomer good-will
Ch. 1 A Perspective on Testing -
Sum2012Bernal
40
7/30/2019 Ch1_A Perspective on Testing
41/41
Goals of Testing?
Test as much as time allows us
Execute as many test cases as (schedule) time allows?
Validate all the key areas
Test only the designated keyrequirements?
Find as much problems as possible
Test all the likely error prone areas and maximize testproblems found?
Validate the requirements
Test all the requirements?
To reach a quality target
State your goal(s) for testing. - - - what would you like people to
say about your system ? Your goals may dictate your testing process
Very important tothink about this ---
Quality Target?
Ch. 1 A Perspective on Testing -
Sum2012Bernal
41