Upload
beverley-carter
View
220
Download
3
Embed Size (px)
Citation preview
Chapter 12: Software Testing & Quality Assurance
Instructor: Paul K Chen
TopicsTopics
I. An Overview
II. The Testing Process
III. Software Quality Assurance
VI. Quality Measurement
V. Concluding Remarks
I. An OverviewI. An Overview
I.1 Definitions
I.2. Related Terms and Definitions
I.3 Typical Software Quality Factors
I.4. Testing vs. Quality Assurance
I.5 The Three Levels of Testing
I.1 DefinitionsI.1 Definitions
Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results.
Quality is not intangible
The purpose of testing is to make quality visible.
Testing is the measurement of software quality.
I.2. Related Terms and I.2. Related Terms and DefinitionsDefinitions
Alpha Testing Alias for functional testing performed by developers. Beta Testing Alias for functional testing performed by users. Regression Testing Testing is performed after functional improvements or
repair to the program. Its purpose is to verify that a modified system still meets requirements.
Black Box Testing The tester is completely unconcerned with the internal
behavior and structure of the program.
Related Terms and DefinitionsRelated Terms and Definitions
Functional Testing
Testing is performed to find discrepancies between the program and its external specifications.
White Box Testing
Testing is focused on the internal processing and structure of the program.
Quality Assurance
A planned and systematic pattern of all actions necessary to provide confidence the product conforms to established requirements.
I.3. I.3. Typical Software Quality Typical Software Quality FactorsFactors
Functionality (External factors)
Correctness; Reliability; Usability; Integrity Engineering (Internal quality)
Efficiency; Testability; Documentation; Structure Adaptability (added dimension)
Reusability; Expandability (flexibility);
Maintainability
I.4. Testing vs. Quality I.4. Testing vs. Quality AssuranceAssurance
Testing and quality assurance must work together for the continual improvement of the developmental and maintenance process.
The test personnel should record the defect by type which should be given to quality assurance at appropriate testing checkpoint
Testing vs. Quality AssuranceTesting vs. Quality Assurance
Quality assurance should use these defect lists to identify troublesome parts of the test, development, and maintenance process. The sharing of this information is to eradicate the cause of the most frequently occurring defects.
In addition, quality assurance must use this information only for improvement in the process, and should not present it to other individuals as this would be a breach of trust in the partnership relationship.
I.5. The Three Levels of Testing-I.5. The Three Levels of Testing-Unit TestingUnit Testing
Unit testing Testing of individual programs as they are written.
Objective: Confirm that module is coded correctly. Who does it: usually the programmer. What is tested: functions are exercised (black box);
code may be exercised (white box); Extremes and boundaries are explored.
When completed: Usually when programmer feel comfortable and has no known defects.
Records: Usually not recorded.
The Three Levels of Testing-System The Three Levels of Testing-System TestingTesting
System Testing Testing of groups of programs Objective: Assemble modules into a working system.
Determine readiness for acceptance testing. Who does it: Team leader or test group. What is tested: system requirements; internal and external
interfaces. When completed: Usually when majority of requirements
met; no major defects remain. Tools or Aids: Test cases; test data generator. Records: Defects discovered usually logged; test cases
maintained.
The Three Levels of Testing-The Three Levels of Testing-Acceptance TestingAcceptance Testing
Acceptance Testing Acceptance testing begins when system testing is complete.
Objective: Evaluate readiness for use. Who does it: End users or their agents. What is tested: Major functions, documentation and
procedures. When completed: Usually when user is comfortable or
when test set runs successfully. Tools or Aids: Comparators. Records: Spotty; some highly formalized.
II. The Test ProcessII. The Test Process
II.1. Define Testing Strategy
II.2 Develop Testing Plan
II.3 Execute Tests
II.4 Summarize and Report Testing Results
II.1. Define Testing StrategyII.1. Define Testing Strategy
Definition:
Testing strategy is the planning of test activities and the
subsequent systematic control of these activities. Both the
management and key staff must agree upon a purpose
and direction.
Testing StrategyTesting Strategy
Test strategies encompass at least:
Objective: What you try to achieve.
Scope: Define what areas you want to cover.
Approach: How testing is performed to achieve the goal.
Testing StrategyTesting Strategy
Specifically test strategies include the following:
Testing Methods- Inspection; Analysis; Demonstration; Test
a. Inspection using visual examination. b. Analysis: Verification using comparison with
expected results. c. Demonstration: Verification by performing the
function the result of which is observable. d. Test: Verification by systematic control and
exercising.
Testing StrategyTesting Strategy
Cost & Benefits
Cost: Equipment; personnel; education; schedule. Benefits: Reliability; maintainability; measurable
Limitations
Time; cost; system complexity
Organizations
Development; user; independent
Testing StrategyTesting Strategy
Verification strategies by area Requirements tractability; design conformance to
standards, architecture, directions, and code.
Verification strategies by approach Walkthrough; top down; bottom up; thread; hybrid Walkthrough: An informal meeting held to verify the
validity of a product. Area to review: Functional and/or technical format; 3-7 people among peers using team dynamic.
ISO 9001ISO 9001
The ISO 9001 series of International Standards for Quality management and quality assurance has been adopted in more than 90 countries and is being implemented by thousands of manufacturing or service organizations in both public and private sectors. One of the most successful series of standards in the history of ISO (International Organization for Standardization), ISO 9000 has generated much publicity and made the name of ISO known to a wider business community than the specialists directly concerned with technical standards.
II.2. Develop Test PlansII.2. Develop Test Plans
What is it? Testing plans are the realization of testing strategies. It is an approved statement of work for all testing activities.Test plans can be written for various levels of testing:
High level Overall project testing plan System test plan Subsystem test plan Integration test plan Low level Unit test plan
Develop Test Plans-ContinuedDevelop Test Plans-Continued
Selection of test items Do a work breakdown structure by product and task. Test personnel and organization Responsibility matrix Test facilities and resources Hardware and software requirements Testing schedule Testing libraries Test data Test training Test tools
Task Breakdown StructureTask Breakdown Structure
System
Task Component
A
Task Component
B
Task A Task B Task C Task D Task E
Develop Test Plans-ContinuedDevelop Test Plans-Continued
Test (control) procedures
Test readiness review Certification process Tractability matrix
Measurements Validation methods Configuration management policies Testing problem reporting procedures Testing logs
II.3. Executing TestsII.3. Executing Tests
Design test case
Select test tools
Select test conditions and data
Set up test case folios
Test Case DesignTest Case Design
Black box approach: (requirements-based)Characteristics: Testing functionality of the system and ignoring the
internal structure of the system. Tests are derived from specifications.
White (glass) box approach: (design-based)Characteristics: Focusing on the structure of the system and ignoring its
specifications Tests are derived from program structure
Black Box ApproachBlack Box Approach
Advantages: Functions are forced to be known and enumerated.Disadvantages: Usually requires a lot of tests. Specifications may not be complete.Techniques Decision tree diagram State transition diagram (for real time system) Input/output space partitioning (probabilities/combination) Boundary value analysis Walkthrough
Black Box Approach-Black Box Approach-TechniquesTechniques
Scientific Cause/effect graph
Input/output space partitioning
Boundary analysis
Random testing
Non-scientific Error guessing
Start with most scientific approach; build a base of test
cases; Complement your base by adding additional cases
derived from the other techniques.
Black Box Techniques- Black Box Techniques- Cause/Effect GraphingCause/Effect Graphing
Cause/effect graphing (decision tree/decision table)
Method: Analyze the specifications to identify all explicit and
implicit causes and effects. Give each cause and effect an arbitrary number.
Draw the cause/effect graph (decision tree). Convert the graph (tree) to a decision table.
Decision TableDecision Table
It shows that transformation between a set of inputs and a
set of outputs. Decision tables are a useful mini-
specification technique when the control processing is
better expressed as a “look-up tables” than as an
algorithmic procedures.
1. $ amt S small (0-$1.999) . L large (at least $2,000)2. Per share cost C low cost (0 -$19/share) X expensive (at least $20/share3. Type of lot E even lot (multiple of 100 share)
Dollar AmountPer share costType Lot
1 2 3 4 5 6 7 8
S S S S L L L L
C C C CX X X XE E E EO O O O
5 % of base
3 % of base
10 % of base
5 % of base
1 % of base
yes yes yes yes
yes
yes
yes
yes
yes
yes
yes
yesyes
yes
yes
YesYes YesYes
Yes
Yes
Decision TreeDecision Tree
The decision tree is a variant of the decision table that
depicts a graphical view of the conditional selection
process used to determine a set of actions from a set of
inputs. The decision points are the nodes in the tree; the
actions are found at the leaves.
CommissionPolicy
SmallAmount
LargeAmount
LowCost
expCost
LowCost
expCost
Even Lot
Even Lot
Odd Lot
Odd Lot
Even Lot
Odd Lot
Even Lot
Odd Lot
5 %+10 % of Base
5 %+11 % of Base
5 %+6 % of Base
5 %+5 % of Base
5 %+10 % of Base
5 %+11 % of Base
5 %+5 % of Base
5 %+6 % of Base
Black Box Black Box Techniques-Input/Output Space Techniques-Input/Output Space PartitioningPartitioning
Methods: 1. Analyze the specifications to identify all input and output
conditions. Partition these conditions into classes. These classes are further divided into two types, valid vs. invalid.
Case in point:If input conditions have a range 1-99, specify one valid class (1-99) and two invalid classes (<1 and > 99)If input conditions have a set of values, (for ex., passing grades A, B,C), specify one valid equivalence class for each and one invalid equivalence class.
Black Box Black Box Techniques-Input/Output Space Techniques-Input/Output Space PartitioningPartitioning
2. Identify the equivalence classes for output conditions and partition into classes. These will also fall into two types-valid vs. invalid.
3. Identify “cases” that will be representative for each of the classes identified.
Black Box Techniques- Black Box Techniques- Boundary value analysisBoundary value analysis
Boundary value analysis
Boundary conditions are those values directly above or
below the edges of the input equivalence or output
equivalence classes. For example:
If a range 1-99 is specified, test for 0,1,99,100. If a number 1 is specified, test for 1,1, -1, 1, 1.001, .999,
etc.
Black Box Techniques- Black Box Techniques- Random Random TestingTesting
Random testing
Uses a randomly selected subject or all possible input.
Better yet, use statistical sampling techniques.
Error guessing
White Box ApproachWhite Box Approach
Advantages: Integration dependent errors can be found.
Disadvantages: Tests can not find the functionality of the program. Number of tests may be prohibitive.
White Box TechniquesWhite Box TechniquesTechniques Path testing Each executable path through a program is tested at
least once. Total path coverage is best but not enough. Branch testing each entry point is invoked at least once; Special values
testing. Boundaries in the input space are explored and potential exceptions or end-point conditions in the code are targeted at by the choice of input data.
Walkthrough A review process in which a designer leads one or more
others through a segment of design or code he or she has written.
Select Test ToolsSelect Test Tools
Program logic analyzer
Produces hierarchical charts depicting program logic structure (structure chart)
Analyzes source code for logical structure and produces reports flagging logic discrepancies.
Test Data Generator
Collects data from existing files.
Collects data based on file attributes and program. For example, Pro/Test.
Select Test ToolsSelect Test Tools
Captive/Replay Tools
Recreates online user transactions by capturing screen input, for ex., Traps.
File comparators
Compares contents of two files and identifies discrepancies.
Select Test DataSelect Test Data
Criteria: reliability and validity
Reliability: If any set of test data selected will reveal any incorrectness of the program.
Validity: If it is simply able to select a test data set that will demonstrate the incorrectness of the program.
Two orthogonal but complementary approaches to test data selection: Black box treatment vs. White box treatment.
Set Up Test Case FoliosSet Up Test Case Folios
The test case folio is simply a folder or a document
containing information on a case or related group of
cases. Possible contents:
1. Identification for the test group or test case.
2. Testing objectives.
3. Description of the situation being tested.
4. Test data (or a reference to where it is or how generated).
Set Up Test Case FoliosSet Up Test Case Folios
5. Expected or required results.
6. Entrance criteria (What has to happen before the test can start).
7. Setup and execution procedures.
8. Exit criteria.
9. Responsibility.
10. Active logs.
11. Actual results.
II.4. Summarize and Report II.4. Summarize and Report Testing ResultsTesting Results
Recording the results of analysis.
Reporting the analysis to involved parties.
Serving as barometer of effectiveness for the test plan.
Could be produced at various levels (system to unit).
Get OrganizedGet Organized
Project procedure
Specific assignments of responsibility
Communication medium
Project handbook
Support library
Project library
III. Software Quality Assurance III. Software Quality Assurance (SQA)(SQA)
III.1 Erroneous Assumptions
III.2 What’s Software Quality Assurance?
III.3 System Quality Dimensions
III.4 Steps for Installing a Basic SQA
III.5 Metrics
III.6 Total Quality Management (TQM)
III.1 Erroneous Assumptions III.1 Erroneous Assumptions About QualityAbout Quality
Quality is intangible and not measurable.
Quality products are too expensive.
Quality problems originate with programmers.
Quality comes from the quality control department.
III.2 What’s Software Quality III.2 What’s Software Quality Assurance (SQA)?Assurance (SQA)?
Software quality is conformance to customer
requirements when such requirements have been
explicitly documented as a foundation for software
development (Functional issues).
Software quality is conformance to a set of implicit
requirements that are desired for all computer-based system. Implicit requirements include testability, maintainability, reliability efficiency, transportability, and security. (Design for internal issues)
What’s Software Quality What’s Software Quality Assurance (SQA)?Assurance (SQA)?
Software quality assurance is a set of procedures that are applied throughout the software engineering process. In essence, the SQA organization assures that standards are properly applied; reviews are conducted; testing is through; changes are properly tracked and recorded; and overall documentation and record keeping are appropriate.
III.3 System Quality DimensionsIII.3 System Quality Dimensions
Three dimensions to system quality.
Quality of design: design it right.
Quality of conformance: build it right.
Quality of performance: it has to work right.
Quality of DesignQuality of Design
Three aspects: Functionality- from the non-technical perspective
Works right; easy to use; within schedule and budget; understandably documented.
Engineering-from the technical perspective
Runs efficiently; conforms to specs; uses the latest techniques and technology.
Adaptability
Flexibility; reusability; maintainability.
Quality of Conformance & Quality Quality of Conformance & Quality of Performanceof Performance
Quality of Conformance
Conformance to design specifications (which relates to the specific applications).
Conformance to standards and conventions (which relates to all applications).
Quality of Performance Uses resources efficiently.
III.4 Steps for Installing a Basic III.4 Steps for Installing a Basic SQASQA
1. Develop a set of software engineering guidelines
Software engineering standards and procedures range from simple informal guidelines to mandated procedural requirements enforced by government edict. These standards can be modified for local needs.
SQA Standards:
DOD-STD-2167; FAA-STD-018; IEEE-STD-8200.1
Steps for Installing a Basic SQASteps for Installing a Basic SQA2. Establish formal technical review guidelines Review the product, not the producer. Set an agenda and maintain it. Limit debate and rebuttal. Enumerate problem areas, but do not attempt to solve every
problem that is noted. Take written notes. Limit the participants and insist upon advance preparation. Develop a checklist for each product that is likely to be
reviewed. Allocate resources and time resources Conduct meaningful training for all reviewers.
Steps for Installing a Basic SQASteps for Installing a Basic SQA
3. Define a testing strategy and distinct black box and white box testing methods.
Black box testing method focuses on the functional requirements, and attempts to find errors in the following categories:
A. Incorrect and missing functions.
B. Interface errors.
C. Errors in data structures
D. Performance errors.
Steps for Installing a Basic SQASteps for Installing a Basic SQA
White box testing method focuses on the close examination of procedural detail (on the module’s internal workings). Four traditional criteria: A. Path testing where each executable path through a program
is tested at least once.B. Branch testing which exercises all outcomes of each predicate.C. Structured testing where loops are tested to upper bounds of
iterations.D. Special values testing where ‘singularities’ and boundaries in
the input space are explored and potential exceptions or end-point conditions in the code are targeted at by the choice of input data.
Steps for Installing a Basic SQASteps for Installing a Basic SQA
4. Establish a set of procedures for change control.
Establish a set of procedures for documenting and
evaluating the effect of these changes.
Prior to freeze-not subject to managerial approval
After freeze-some formal change evaluation procedure
should exist.
The activity of change control is part of the general
process of software configuration management.
III.5 III.5 MetricsMetrics
It refers to any measurement related to software development. Measurements motivate a organization to improve its process. Useful Metrics:
a. Size: total lines code written; total comment lines; total function points.
b. Productivity: Number of work hours spent on the project; number of work hours spent in each routine.
c. Maintainability:Line of code in each routine; number of decision points in each routine.
d. Overall Quality: Total number of defects; Average defects per thousand of line of codes
III.6 Total Quality Management III.6 Total Quality Management (TQM)(TQM)
It is a conception of quality as an evolutionary process toward perfection instead of conceiving quality as controlling the number of defective products produced.
For TQM to become a reality, the full organizational support of management and early commitment to quality from the analyst and from the business are necessary.
One of the strongest quality assurance actions the system analysis team can take is:
Structured Walkthrough—to use peer reviewers to monitor the system’s programming and overall development.
Quality System DesignQuality System Design
The bottom-up Approach; The top-down approach; and Modular programming concept.
Modules falls into three classes: 1. Control modules- determining the overall program
logic 2. Transformational modules – changing input into output
3. Specialized modules—performing detailed, functional work.
Quality System Design-Quality System Design-(continued)(continued)
Modularity in the Windows Environment:
Dynamic Data Exchange (DDE) updates data in one program based data in another program.
Object Linking and Embedding (OLE) is where an object in a second program retains the properties of an object in the first program.
Object Linking and Embedding Object Linking and Embedding for Databases (OLE DB)for Databases (OLE DB)
Microsoft has defined set of data objects, collectively known as OLE DB.
Allows OLE-oriented applications to share and manipulate sets of data as objects.
OLE DB is an object-oriented specification based on C++ API.
Components can be treated as data consumers and data providers. Consumers take data from OLE DB interfaces and providers expose OLE DB interfaces.
OLE DBOLE DB
Common Gateway Interface (CGI)Common Gateway Interface (CGI)
Specification for transferring information between a Web server and a CGI program.
Server only intelligent enough to send documents and to tell browser what kind of document it is.
But server also knows how to launch other programs. When server sees that URL points to a program
(script), it executes script and sends back script’s output to browser as if it were a file.
Common Gateway Interface (CGI)Common Gateway Interface (CGI)
CGI defines how scripts communicate with Web servers.
A CGI script is any script designed to accept and return data that conforms to the CGI specification.
Before server launches script, prepares number of environment variables representing current state of the server, who is requesting the information, and so on.
Script picks this up and reads STDIN.
Common Gateway Interface (CGI)Common Gateway Interface (CGI)
Then performs necessary processing and writes its output to STDOUT.
Script responsible for sending MIME header, which allows browser to differentiate between components.
CGI scripts can be written in almost any language, provided it supports reading and writing of an operating system’s environment variables.
Common Gateway Interface (CGI)Common Gateway Interface (CGI)
Four primary methods for passing information from browser to a CGI script:
Passing parameters on the command line. Passing environment variables to CGI programs. Passing data to CGI programs via standard input. Using extra path information.
IV. Quality MeasurementIV. Quality Measurement
IV.1 Quality Control & Testing SDLC
IV.2 Quality Metrics
IV.3 SEI Process Maturity Model
IV.1. Quality Control & Testing IV.1. Quality Control & Testing SDLCSDLC
The purpose of testing is to measure quality.
Good testing is a prerequisite to quality control.
Quality control defines what the requirements are (quality is); involves measuring quality; and acts on the results.
Achieving significant testing improvements depends on successfully imposing a new testing life cycle on top of the development cycle.
IV.2. Quality MetricsIV.2. Quality Metrics
The term “Metrics” refers to any measurement related to Software development.Two reasons to measure your process: Any way of measuring the process is superior to not
measuring it at all. Statements like “This new method seems more productive” aren’t good enough.
To argue against metrics is to argue that it’s better not to know what’s really happening on your project.
Useful MetricsUseful Metrics
Size
Total lines of code written
Total comment lines
Total data declarations
Total blank lines
Defect Tracking
Work hours spent on correcting each defect.
Average time required to find a defect.
Quality MetricsQuality Metrics
Metrics Quality Use
Product defect rate
# of defects
----------------
Project size
Quality of product
Cost performance
Total proj cost
-----------------
Project size
Function point
Quality MetricsQuality Metrics
Select metrics to manage.
Select measures to support the metrics.
Function points
Time
Defects
Costs
Metrics must be clearly defined.
Quality MetricsQuality Metrics
schedule = total on-time schedules*
-----------------------------
total schedules**
* sum of al function points active on-time projects times
The number of elapsed days to date in each on time
project.
** sum of all function points in active projects times the number of days in each elapsed projects.
IV.3 Process Maturity ModelIV.3 Process Maturity ModelSEI (Software Engineering Institute) funded by DOD to study software development and disseminate information about effective development practices. It identifies five levels of software process maturity:1. Anarchy- Operate without formal planning2. Folklore- Tribe Knowledge3. Standards-Methodology written down, but no one has collected data to measure its effectiveness4. Measurements- Data is collected to assess the process
effectiveness.5. Optimization- An organization focuses repeatability and
measurement primarily to improve product quality.
Process Maturity ModelProcess Maturity Model
Three factors are needed to improve process
maturity:
1. Repeatability (Methodology)
2. Measurability (Metrics)
3. Feedback (Optimization)
Repeatability (Methodology)Repeatability (Methodology)
A process is made repeatable by the use of a standard methodology, thus laying the groundwork for the measurement at the higher levels.
Feedback (Optimization)Feedback (Optimization)
The salient characteristics of the SEI organizational maturity model at its highest maturity are repeatability, measurability, and controlled variations and feedback.
V. Concluding Remarks- Elements V. Concluding Remarks- Elements of A Software Quality Programof A Software Quality Program
Software quality objectives: external vs. internal
characteristics. Explicit quality activities: no more quick and dirty
programming. Testing strategy. Software engineering guidelines. Informal technical reviews: desk checking Formal technical reviews: a peer review, a
walkthrough
Elements of A Software Quality Elements of A Software Quality ProgramProgram
External audits: audits outside the organization. Development processes: effective risk management. Change control procedure. Measurements of results.
Quality ActivitiesQuality Activities
Requirements
Design
Code construction
System implementation Maintenance
Verify requirements; Test Strategy; Test Plan Verify design; prepare
test cases; place cases under CM
Verify code; create test cases; execute testing Alpha test; beta test Regression test; manage
change control
System Phase Activity