testing from basics

7/27/2019 testing from basics

1/25

Fundamentals of Testing

Speaker: Jerr y Gao Ph.D.

San Jose State Un iversity

email : jerrygao@email .sjsu.edu

URL : http://www.engr.sjsu.edu/gaojerry


2/25

Topic: F undamentals of Testing

- Software Error s

- Software Test Planing

- Software Test Requi rements

- Software Test Design- Review for Software Testing

- Software Test Executi on

- Problem Tracking System

- Software Test Management

-Software Test Engineers Tasks

- Test Automati on and Tools

Jerry Gao Ph.D. 7/20002

Presentation Outl ine

Al l Rights Reserved


3/25


What i s a software error?

One common defi ni tion of a software error is a mismatch between the program

and its specif ication.

Defi niti on #1:

A mismatch between the program and its specification is an errorin the program if and only if the specification exists and is correct.

Defi niti on #2:

A software error is present for when the program does not do what

its end user reasonability expects to do. (Myers, 1976)

Defi niti on #3:There can never be an absolute definition for bugs, nor an

absolu te determination of their exi stence. The extent to which a program has

bugs is measured by the extent to which it fail s to be usefu l . Thi s is a

fundamentally human measure. (Besizer, 1984)


Software Errors


4/25


- User inter face errors, such as output errors, incorrect user messages.

- Function error s - Defect hardware

- I ncorrect program version - Testing errors

- Requirements errors - Design errors

- Documentation err ors - Architecture error s

- Module in terf ace error s - Per formance error s

- Er ror handling - Boundary-related error s

- L ogic errors, such as calculation errors

- State-based behavior errors - Communi cation errors

- Program structure err ors, such as control -fl ow error s


Categories of Software Errors


5/25


L ike other activi ties in software engineer ing phases, i t i s impossible to have a

cost-effective software test process without a very good planing,

The major objective of software test planing:

- generate a well-defined software test plan.

What content should be included in a software test plan?

- Testing activi ties and schedule

- Testing tasks and assignments

- Selected test strategy and test models

- Test methods and cr i teri a

- Requir ed test tools and envir onment

- Problem tracking and reporting- Test cost estimation

Other needed items: qual i ty contr ol process and standards


Software Test Planing


6/25

Jerr y Gao Ph.D. 7/2002


Software Test Requi rements

Befor e star ting test design, we must identify our test objectives, focuses, and test i tems.

The major purpose is to help us understand what are the targets of software testing.

Th is step can be done based on:

- Requi rements specif ications

- I nputs fr om developers- F eedback f rom customers

Benef its are:

- I denti fy and rank the major focus of software testing

- Check the requi rements to see if they are corr ect, completed, and testable

- Enhance and update system requi rements to make sur e they are testable

- Support the decision on selecting or def ining test strategy

For example,

- for performance testing, we need clear requi rements on system per formance.


7/25



Software Test Requi rements

The essenti als of testing r equi rements include:

- Specif ied testing methods

- Required test types am d test coverage cr i teri a

- Selected or requi red test tools

- Testing focuses and test i tems for each type of software testing

An example of performance testing requi rements:

Check the system performance to make sure it meet 99% system

reliability requirements

A typical example for requi red test i tems is:

Test item #I: Test the call waiting feature (REQ #j) during system testing based on the

given requirements specifications.


8/25



Software Test Design

Software test design is an impor tant task for software test engineers.

A good test engineer always know how to come out quali ty test cases and perf orm

eff ective tests to uncover as many as bugs in a very tight schedule.

What do you need to come out an eff ective test set ?

- Choose a good test model and an eff ective testing method

- Apply a well -defi ned test cri ter ia

- Generate a cost-eff ective test set based on the selected test cr i teria

- Write a good test case specif ication document

What i s a good test case?

- I t must have a high probabil i ty to discover a software error

- I t is designed to aim at a specif ic test requi rement- I t is generated by fol lowing an eff ective test method

- I t must be wel l documented and easil y tracked

- I t is easy to be per formed and simple to spot the expected results

- I t avoids the redundancy of test cases


9/25



Software Test Design

What content should be included in a test case?.

Test Case ID: Test I tem:

Wrote By: (tester name) Documented Date:

Test Type: Test Suite#:

Product Name: Release and Version No.:

Test case descr iption:

Operation procedure:

Pre-conditions: Post-conditions:

I nputs data and/or events: Expected output data and/or events:

Requi red test scripts:


10/25

Test execution can be performed:

- using manual approach

- using a systematic approach

- using a semi-automatic approach

Basis activi ties in test execution are:

- Select a test case

- Set up the pre-condi tions for a test case

- Set up test data

- Run a test case foll owing i ts procedure

- Track the test operations and resul ts- Moni tor the post-conditions of a test case & expect the test resul ts

- Ver ify the test resul ts and report the problems if there is any

- Record each test execution



Software Test Execution


11/25

What do you need to per form test execution?

- a test plan

- test design specif ication with test case specif ications

- a test sui te with documented test cases (optional)

- test suppor ting facil i ty, such as test dr ivers, test stubs, simulators

What are the outcome of an test execution:

- Text executi on r ecord and report

- Problem and bug reports

- Er ror logs and

With automatic test execution tools (or test control tools), we can do:

- automatic test r unner

- record and replay



Software Test Execution


12/25

Problem Analysis and Report



When do we issue a problem?

Whenever a bug or problem is found, we need to wr i te down a problem report immediately.

What ar e the content of a problem report?

Problem ID cur rent software name, release no. and version no.

Test type Reported by Reported date Test case ID

Subsystem (or module name) Feature Name (or Subject)

Problem type (REQ, Design, Coding, ) Problem severi ty (Fatal, Major, M inor, ..)

Problem summary and detail ed descri ption

Cause anal ysis How to reproduce? Attachments


13/25

Problem Analysis and Report



How to track, control , and manage issued problems?

- A systematic solut ion is needed to tr ack and maintain the reported problems in a

repository.

- Defi ne and implement a problem control and analysis process to control problem

tracking, reporting, analysis, fi xing and resoluti ons.

Character istics of a problem report:

- Simple and understandable - Traceable and numbered

- Reproducible - Non- judgmental

Problem analysis:-F inding the most ser ious consequences

-F inding the simplest and most general condi tions

-F inding alternative paths to the same problem

-F inding related problems


14/25



Software Test Review

What i s a review?

- A review is a ver i f ication activity to assur e a corr ect method has been used to

create a software product.

- Parti cipants in a review take ful l responsibili ty for resul ts.

There are two types of r eviews:

- Formal r eviews:

- use a well -defi ned review method (or technique)

- generate formal review resul ts

- I nformal reviews

- use a desk checking approach- conduct an i nformal r eview

- generate information review resul ts


15/25



Software Test Review

Products shoul d be reviewed dur ing software testing:

Test Plan Test Design Specif ication

Test Procedur e Specif ication

Test Report Problem Reports

What do reviews accompli sh?

- Reviews provide the primary mechanism for r el iably evaluati ng progress.

- Reviews train and educate the parti cipants and have a signi fi cant posit ive eff ect on

staff competence.

- Reviews give earl y feedback and prevent more ser ious problems fr om ar ising.

- Reviews bring individual capabili ty to light.


16/25



Test Management

Test management encompass:

- management of a test team

- management of a test pr ocess

- management of test projects

A test managers role:

- Play as a leadership role in :

- plani ng projects - setting up a dir ection -motivating people

- bui ld a team - manage engineers

- Play as a controller in :

- product evaluation - performance evaluati on

- changing to a new dir ection

- Play as a supporter in:

- assist and train engineers - train engineers

- enforce and control test methods, standards, and cr iter ia

- select and develop test tool s


17/25



Test Management

Test management functions:

- Management - Manage test projects

- M anage team members

- M anage test processes

- Motivation - Motivate quali ty work fr om team members

- Simulate for new ideas and creative solu tions

- Methodology - Contr ol of setting up test methodology, process, standards.

- Control of establi shing test cri ter ia

- Mechanization - Control the selection and development of test tools

- M echanism for the conf iguration management of test sui tes

- Contr ol of setting up an integrated test environment

- Measur ement - Measur e test cost, complexi ty and eff orts

- M easure engineer per formance

- Measur e test effectiveness

- Measure product quali ty


18/25



Test Management

Test management encompass:

- management of a test team

- management of a test pr ocess

- management of test projects

A test managers role:

- Play as a leadership role in :

- plani ng projects - setting up a dir ection -motivating people

- bui ld a team - manage engineers

- Play as a controller in :

- product evaluation - performance evaluati on

- changing to a new dir ection

- Play as a supporter in:

- assist and train engineers - train engineers

- enforce and control test methods, standards, and cr iter ia

- select and develop test tool s


19/25



Test Management

Questions for managers who claim to be managing software testing:

Management: Do you plans address testing?

Do you know who is responsible?

Have you publ ished your testing poli cy?

Motivation: Do you provide incentive for people do quali ty work?

Do you encour age people to take advantage of train ing

oppor tuniti es in testing methods?

Methodology: Ar e your testing methods procedurali zed and are people

trained in their use?Ar e you aware of new testing techniques and are you

worki ng to in troduce them?


20/25



Test Management

Questions for managers who claim to be managing software testing:

Mechanization: Do you suf fi cient hardware and equipment to support testing?

Have you provided appropri ate software testing tools and aids?

Do you evaluate automated testing aids on an ongoing basis?

Measurement: Do you track err ors, faul ts, and fai lu res?

Do you know what testing costs?

Do you quanti tatively measur e testing per formance?


21/25



Test Engineers Tasks

What does a test engineer do?

- Ensure that testing is per formed

- Ensure that testing is documented

- Ensure that testing methodology, techniques, standar ds are

establ ished and developed

The basic ski l ls for a qual if ied test engineer :

Controll ed - Organized individual, systematic plani ng

- Good planing on testing

Competent - Technical awareness of testing methods, tools, and cr iteri a

Cri tical - I nner determination to discover problems

Comprehensive - Total understanding of the given system and specif ications

- Pay good attention to the detai ls

Considerate - Abi l i ty to related to others and resolve confl icts


22/25



Basic Test Engineers Tasks

The basic tasks of a test engineer include:

- Prepare testing plans and organi ze testing activiti es

- Design test cases and document them using well -defined test methods

- Develop test pr ocedur es and test data

- Wri te problem repor ts and text executi on records

- Use testing tools and aids

- Review test designs, resul ts and problems

- Test maintenance changes

- Oversee acceptance tests


23/25



Test Automation and Tools

What is software test automation?

Software test automation refers to a process and activities that reduce manual testing

activi ties and efforts in a software development l if ecycle.

Why software test automation?

- Reduce software testing time, cost, and efforts- I ncrease the quali ty of software testing

- Apply well -defi ned test methods thr ough tools

- Reli eve the complicated and redundant work fr om test engineers

What do we need to automate software testing?

- L imited cost and schedule

- Select and intr oduce quali ty test tools

- Develop necessary eff ective and eff icient test tools and facil i ty

- Apply wel l-defined testing methods and coverage

- F orm an integrated test envi ronment supporti ng vari ous software testing activi ties

for products on a production line


24/25




Basic steps for test automation:

Level #1: Automatic test management

- test case & sui te management, and documentati on

- test scr ipt management

Level #2: Au tomatic test execution- black-box test control and execution

- white-box test control and execution

Level #3: Au tomatic test generation

- black-box test generation

- wh ite-box test generation

Level #4: Automatic test measur ement

- test coverage and metr ics measur ement

- test cost and complexi ty measur ement


25/25




Classif ication of software test tools:

- GUI record and replay tools

- Program specif ication -based test tools

- Test management and conf iguration management tools

- Test generation tools, such as random test tools

- Program-based test tools (or whi te-box test tools)

- Program test metr ics tools

- Program per formance test tools

- performance monitor ing tools

- performance evaluati on tools

- Program load test tools

- Protocol-based conf irmation tool tools

- Program regression test tools (such as program change analysis tool)

Documents

testing from basics