Advanced Software Engineering 1 Advanced Software Engineering: Software Testing COMP 3705 (Lecture1) Sada Narayanappa Anneliese Andrews (Chair DU) Thomas

Advanced Software Engineering 1

Advanced Software Engineering: Software

TestingCOMP 3705 (Lecture1)

Sada Narayanappa

Anneliese Andrews (Chair DU)Thomas Thelin

Carina Andersson


Lectures Theory + discussions Cover the basic parts of software testing

1. Introduction

2. Black-box, Reliability, Usability

3. Inspections, white-box testing

4. Lifecycle, documentation

5. Organization, tools

6. Metrics, TMM

7. Research presentation

Technical

Overview

Technical / Manager

Managerial

Economic


Examination

Written exam based on the book and lab sessions

Lab sessions (approved) Project/presentations are grades See class web site for Assignment policy


This week

Read course program Project

• Read Projects in Software Testing/Pick research• Decide Research/subject• Discuss papers with me – describe why is it interesting

Lab• Prepare lab 1

Read Burnstein 1-3• Prepare Burnstein 4,12


Schedule

Read

• Course program

• Projects in Software Testing Check homepage If needed we will have

• Extra Lab dates

• Ask TA with Lab work


Facts about testing

System development: 1/3 planning 1/6 coding 1/4 component test 1/4 system test[Brooks75] Planning

Coding

Component test

System test


Why use testing?

• Risk mitigation• Faults are found early• Faults can be prevented• Reduce lead-time• Deliverables can be reused• …


Why do faults occur in software? Software is written by humans

• Who know something, but not everything

• Who have skills, but aren’t perfect

• Who don’t usually use rigorous methods

• Who do make mistakes (errors) Under increasing pressure to deliver to strict deadlines

• No time to check, assumptions may be wrong

• Systems may be incomplete Software is complex, abstract and invisible

• Hard to understand

• Hard to see if it is complete or working correctly

• No one person can fully understand large systems

• Numerous external interfaces and dependencies


TMM – Test Maturity Model

Stages or levels to move from

• Chaotic, unstructured testing model

TO

• Systematic, process driven approach Software is an engineering discipline Requires

• Quality in Process

• Quality in Product

Are important


TMM Test process Evaluation

• Guided by Capability Maturity model (CMM)• Stages or levels to evolve from and to• Each level, except Level 1, has structure• Level 2 and above have:

• Structure

• Goals

• Organization structure


Internal structure of TMM

Level

Testing

Capability

indicateMaturity Goals

Maturity Sub goals

Activities/tasks/responsibilities

Implementation

and Organizational

adaptation 3 Critical views

Manager Developer User/Client

Supported By

Achieved By

Organized By

Addresses

Contains


Test Maturity Model



TMM Levels• Level1- No process defined; debug/testing are

same• Level2- Test & Debug tools/Test plan/Basic test

process• Level3- Test Org/Technical Training/Lifecycle/

Control & Monitor, support V Model• Level4- Review process/test

measurement/Software quality Evaluation – Test Logging with severity

• Level5-Defect prevention/Quality/Process optimization


Good enough quality

To claim that any given thing is good enough is to agree with all of the following propositions:

• It has sufficient benefits

• It has no critical problems

• The benefits sufficiently outweigh the problems

• In the present situation, and all things considered, further improvement would be more harmful than helpful

James Bach, IEEE Computer, 30(8):96-98, 1997.


Quality attributes – ISO 9126


Quality attributes


Origins of defects

Defect sources

Lack of educationPoor communicationOversightTranscriptionImmature process

Impact of software artifactsErrors

Faults / Defects

FailuresImpact from user’s

viewPoor quality software

User dissatisfaction

Fault model


Whoops, that’s my calculator


Testing, Verification & Validation

Definition 1 Verification

• is the product right? Validation

• is it the right product?

Definition 2 Verification

• satisfies the conditions at the start of the phase

Validation

• satisfies the requirements

TestingThe process of evaluating a program or a system


Definitions

Failure is an event, fault is a state of the software caused by an error

• Error – human mistake

• Fault / Defect – anomaly in the software

• Failure – inability to perform its required functions

• Debugging / Fault localization – localizing, repairing, retesting.


• A TEST CASE consists of:• A set of inputs

• Execution conditions

• Expected outputs

• A Test is:• Group of related test cases

• Procedures needed to carry out the test case

Definitions

IEEE Definition

Organization may define additional attributes


Scripted and non-scripted testing

• In scripted testing test cases are pre-documented in detailed, step-by-step descriptions

• Different levels of scripting possible

• Scripts can be manual or automated

• Non-scripted testing is usually manual testing without detailed test case descriptions

• Can be disciplined, planned, and well documented exploratory testing

• or ad-hoc testing


Test oracle

• An oracle is the principle or mechanism by which you recognize a problem

• Test oracle provides the expected result for a test, for example• Specification document• Formula• Computer program• Person

• In many cases it is very hard to find an oracle• Even the customer and end user might not be able to tell

which is the correct behaviour


Test Bed

• Environment contains

•all the hardware and software to test software component/system

• Examples:

•Simulators

•Emulators

•Memory checkers


Other Definitions

• Important to understand the following definitions

•Quality– degree of meeting specified requirement

•Metric – quantitative measure

•Quality metric•Correctness – perform the function

•Reliability –perform under stated condition

•Usability – effort to use the system

•Integrity – withstand attacks

•Portability/maintainability/interoperability …


Principle 1 – purpose of testing

Testing is the process of: exercising a software component using a

selected set of test cases, with the intent of

1.Revealing defects

2.Evaluating quality


Principles

2: A good test case – When the test objective is to detect defects, then a good test case is one that has high probability of revealing a yet undetected defect(s)

3: Test result – The results should be inspected meticulously

4: Expected output – A test case must contain the expected output


Principles

5: Input – Test cases should be developed for both valid and invalid input conditions

6: Fault content estimation – The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component

7: Test organization – Testing should be carried out by a group that is independent of the development group


Principles

8: Repeatable – Tests must be repeatable and reusable

9: Planned – Testing should be planned

10: Life cycle – Testing activities should be integrated into the software life cycle

11: Creative – Testing is a creative and challenging task


Goals of the course

A test specialist - trained engineer- have knowledge of test-related

Principles/processes/measurements, standards, plans, tools, and methods, and

learn how to apply - testing tasks to be performed.

Knowledge

Skills

Attitudes


www.swebok.org


www.swebok.org


Defect classes and Defect repository

Functional DescriptionFeatureFeature interactionInterface description

Requirement/SpecificationDefect Classes

Algorithmic and processingControl, Logic, and sequence DataModule interface descriptionExternal interface description

Design Defect Classes

Algorithmic and processingControl, Logic, and sequence typographical data flow DataModule interface Code documentationExternal hardware, software

CodingDefect Classes

Test HarnessTest designTest procedure

TestingDefect Classes

Defect ClassesSeverityOccurrences

Defect Repository

Defect reports/analysis

Defect reports/analysis


Lab sessions

Preparation, Execution, Report

1.Black-box testing

2.Usage-based testing and reliability

3.White-box testing

4. Inspection and estimation

5.Software process simulation


Project: Option 1

• Learn a specific area of software testing

• Collect and summarize research information

• Critical thinking beyond the written information

• Present information in a structured way

• Peer review


Project: Option 1

Research: solve a research problem; survey the state-of-the-art and identify the research problems in some area; develop and justify an extension to an existing technique; etc.

Evaluation: apply and evaluate a technique or evaluate a commercial testing or, analysis tool.

Practical: Use an existing technique to test a system or design and implement a prototype for a system.


Project: Option 1

Read Projects in Software Testing Divide in groups (2-3 persons)

Discuss with me

http://www.cs.du.edu/~snarayan/sada/teaching/COMP3705/FilesFromCD/Project/Project_SwTest.pdf


Project: Option 2

Research paper presentation Find an interesting paper Many research papers we come across

during class – pick one for presentation Have four paper presentation Choose your team and prepare Reading paper takes time – start early

Documents

Advanced Software Engineering 1 Advanced Software Engineering: Software Testing COMP 3705 (Lecture1) Sada Narayanappa Anneliese Andrews (Chair DU) Thomas