Testing Process Models - University of Kansas · Process Model of a Test Factory. Agenda 1. ... Testing Process Models ... •Acceptance Tests Testing Report 12/12/2009 29

EECS 814 – Fall 2009

Jennifer Kaufman

Testing Process Models

Process Model of a Test Factory

Agenda

1. Introduction & Abstract

2. Organizational Models

3. Testing Process Models

5. Conclusion

4. Process Model of a Test Factory

12/12/2009 2

Introduction

“A proposal of a process model to create a

Test Factory” by:

Ana Sanz

Javier Garcia

Javier Saldana

Antonia Amescua

Presented at the ICSE „09 Workshop on

Software Quality

12/12/2009 3

Abstract

Software testing is being outsourced to

“Test Factories”

Test Factories must implement an

organizational model that includes a

defined testing process model

This research presents a testing process

model to manage a Test Factory

12/12/2009 4

Software Quality Assurance

According to Wikipedia, Software Quality

Assurance (SQA) consists of a means of

monitoring the software engineering

processes and methods used to ensure

quality.

SQA strives to minimize the cost of

guaranteeing quality throughout the

development process

12/12/2009 5

SQA & Testing

Validation ensures the right functions are

performed

Verification ensures the correct

performance of these functions

Testing is part of validation and verification

Inspections

Technical review

Unit testing

Integration testing

All are SQA activities12/12/2009 6

Outsourcing Testing

Software reliability is demanded by

customers

Complex systems require efficient SQA

activities – including more testing

Companies that do not have the testing

resources are beginning to outsource

testing

These organizations have been termed

“Test Factories”

12/12/2009 7

Test Factories

Designed to correctly perform validation

and verification activities

Dedicated resources and infrastructure for

complex testing

Can help reduce software development

costs

Lack organizational model to manage the

testing process

12/12/2009 8

Hypothesis

“If we have a well-defined organizational

model which allows the creation of Test

Factories, outsourcing testing-related

activities could be managed efficiently.”

A well-defined organizational model

includes:

Organizational structure

Process model

Sanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69).

Vancouver, Canada: IEEE.

12/12/2009 9

Organizational Models

Software testing can be organized in

various ways

Developers as testers

Independent test teams

Integrated test teams

QA/QC

Outsourcing & Test Factories

12/12/2009 10

Testing Process Models

Testing process improvement increases

client satisfaction, reduces defect rates,

and increases fulfillment of quality

objectives

There are numerous testing process

reference models

12/12/2009 11

Framework Quagmire

Ahern, D. M., Clouse, A., & Turner, R. (2001). CMMI Distilled. New York: Addison-Wesley.

12/12/2009 12

CMMI

Capability Maturity Model Integrated

Developed by the Software Engineering

Institute (SEI) at Carnegie Mellon

University

Works organizations to continually improve

software-intensive systems

Helps organizations to develop or acquire the

right software, defect free, within budget and

on time, every time

Carnegie Mellon University. (2009). CMMI Overview. Retrieved October 31, 2009, from Software Engineering Institute:

http://www.sei.cmu.edu/cmmi/index.cfm

.12/12/2009 13

CMMI

WHAT to do

Not HOW and WHO should do it

A benchmark of the maturity of an

organization‟s processes with industry

state of practice

Ad Hoc & Immature

Mature & Discipline

Business Goals

14



.12/12/2009

CMMI

It helps integrate once separate

organizational functions

Set process improvement goals and

priorities

Provide guidance for quality processes

Provide a point of reference for appraising

current processes

CMMI does not fully support testing

process improvement 15



.12/12/2009

TMM

Testing Maturity Model

Developed at the Illinois Institute of

Technology by professors Bob Carlson

and Ilene Burnstein and their graduate

students

It provides a guideline for test process

improvement

Compliments Capability Maturity Model

IIT Center for Professional Development. (2009). Software Testing Maturity Model (TMM). Retrieved December 10, 2009, from IIT Center

for Professional Development: http://www.iit.edu/cpd/itm/research/software_testing.shtml

12/12/2009 16

TMM

Six levels of maturity

1. Initial

Rush to code from requirements

No clear separation between design and code

phases

No attempt is made to use well known design

techniques or methods

2. Phased

Clear separation between the requirements

phase and design is phase

IIT Center for Professional Development. (2009). Software Testing Maturity Model (TMM). Retrieved December 10, 2009, from IIT Center

for Professional Development: http://www.iit.edu/cpd/itm/research/software_testing.shtml

12/12/2009 17

TMM

3. Systematic

Defined design process with a methodical

approach to design and the incorporation of

design activities in the project plan

Process can be taught and mastered, highly

repeatable

4. Integrated

Defined design process integrated with

development and testing processes

Portion of the code can be automatically

generated and is consistent in structure

Can be consistently maintainedIIT Center for Professional Development. (2009). Software Testing Maturity Model (TMM). Retrieved December 10, 2009, from IIT Center

for Professional Development: http://www.iit.edu/cpd/itm/research/software_testing.shtml12/12/2009 18

TMM

5. Managed

Design process managed by measures

Large scale reuse and use of components leads

to a high level of consistency

Minimize the knowledge gap between

management and software development staff

6. Optimal

Systems are based on composition of designs

Methodology is repeatable across people

Process lends itself to self-analysis seeking

corrections, efficiency, and improvements to the

product

12/12/2009 19

TMMi

Developed and promoted by the TMMi

Foundation

Compliments CMMI

Based on TMM

Guideline and reference framework for test

process improvement

TMMi Foundation. (2009). Test Maturity Model Integration. Retrieved December 10, 2009, from TMMi Foundation:

http://www.tmmifoundation.org/downloads/tmmi/TMMi%20Framework.pdf

12/12/2009 20

TMMi

• Defect Prevention

• Test Process Optimization

• Quality Control5. Optimization

• Test Measurement

• Software Quality Evaluation

• Advanced Peer Reviews4. Management

and Measurement

• Test Organization

• Test Training Program

• Test Life Cycle and Integration

• Non-Functional Testing

• Peer Reviews

3. Defined

• Test Policy and Strategy

• Test Planning

• Test Monitoring and Control

• Test Design and Execution

• Test Environment

2. Managed

1. Initial

TMMi Foundation. (2009). Test Maturity Model Integration. Retrieved December 10, 2009, from TMMi Foundation:

http://www.tmmifoundation.org/downloads/tmmi/TMMi%20Framework.pdf12/12/2009 21

TPI

Test Process Improvement model

Develop by Sogeti, an IT service provider

Supports the improvement of test

processes

Offers insight into the maturity of the test

processes within your organization

Helps define gradual and controllable

improvement steps

Sogeti. (2009). Test Process Improvement (TPI®): A step-by-step guide for improving your test process. Retrieved December 10, 2009, from

Sogeti:

http://www.ie.sogeti.com/Documents/Publications/Sogeti_Testing_Test_Process_Improvement_TPI_Step_by_step_guide_150609.pdf

12/12/2009 22

TPI

20 key areas for improvement

Life cycle model

Moment of involvement

Estimating and planning

Test specification

techniques

Static test techniques

Metrics

Test automation

Test environment

Office environment

Commitment and motivation

Test functions and training

Scope of methodology

Communication

Reporting

Defect management

Testware management

Test process management

Evaluation

Low-level testing


Sogeti:


12/12/2009 23

TPI

Each key area is given a maturity level

ranging from A to D

A Test Maturity Matrix established the

relations among areas

A set of checkpoints and improvements

are made after the assessment


Sogeti:


12/12/2009 24

TPI


Sogeti:


12/12/2009 25

TestPAI

Integrated with CMMI level 3 engineering

processes

Includes and defines all practices related

to testing

Same structure as CMMI

Proposes 5 specific goals and related

practices



12/12/2009 26

What‟s Lacking

The 4 models discussed are helpful to

create a Test Factory, but not complete

No model examined integrate

organizational structure and a well-defined

process model

The authors proposed a process model of

a Test Factory



12/12/2009 27

Proposed Process Model

3 categories of testing processes

Management

Technical

Support

Each process was defined by identifying

these elements:



Purpose

Description

Activities

Inputs/Outputs

Roles

Work Elements

Work Products

Technical Instructions

12/12/2009 28

Proposed Process Model

Support

Quality Assurance

Control and Monitoring

Configuration Management

Measurement and Analysis

Management

Testing Requirement Development

Testing Planning

Testing Design

TechnicalTesting Execution

• Validation Tests

• Verification Tests

• Acceptance Tests

Testing Report

12/12/2009 29

Management

Testing requirement development

Inputs: Analysis document, requirement

document, specification of client needs

Outputs: Contract between the organization

and the Test Factory, testing requirements

Description: If the feasibility study and project

scope are agreed upon, a contract is signed

and testing requirements are captured and

managed



12/12/2009 30

Management

Testing Planning

Inputs: Analysis document, testing

requirement document

Outputs: Test plan

Description: A test plan is developed and

maintained. In addition to the normal

elements of a test plan, it includes sections of

the project plan which are necessary manage

the testing phase.



12/12/2009 31

Management

Testing Design

Inputs: Analysis document, design document,

testing requirement document, test plan

Outputs: Test plan, Test design specification,

test procedure specification

Description: Establish the information to

execute testing effectively and efficiently

including test plan tasks, schedule, risk,

resources, and cost



12/12/2009 32

Technical

Testing Execution

Inputs: Test work package including artifacts

to be tested, project documentation and

testing documentation

Outputs: Result reporting

Description: Test cases and scripts and data

are required to prepare and execute tests. A

result report is prepared after execution. This

is comprised of verification tests, validation

tests and acceptance tests.



12/12/2009 33

Technical

Verification Tests

Inputs: Artifacts to be tested


Description: Inspections, walkthroughs and

technical reviews are performed to review the

artifacts. Checklists can be used as a simple

and useful verification tool.



12/12/2009 34

Technical

Validation Tests

Inputs: Project documentation and testing

documentation


Description: Unit tests, integration tests,

system tests, regression tests, stress tests,

performance tests and usability tests are

executed to check the system.



12/12/2009 35

Technical

Acceptance Tests

Inputs: Requirement document and testing

document


Description: Acceptance tests are executed

by the customer or end user to ensure the

system meets the client‟s needs.



12/12/2009 36

Technical

Testing Report

Inputs: Result reporting

Outputs: Result summary reporting

Description: Testing results are analyzed and

evaluated and a brief report is drafted. This

process controls and manages incidents.



12/12/2009 37

Support

Quality Assurance

Inputs: Test work package

Outputs: Result audit reporting

Description: Verification that the activities

defined in the current process have been

carried out and that the appropriate

intermediate or final product is obtained.



12/12/2009 38

Support

Control and Monitoring

Inputs: Test plan, result monitoring reporting

Outputs: Result monitoring reporting

Description: To control and monitor the state

of the projects in regards to schedule, costs,

resources, milestones, commitments and

risks. This is a continuous process.



12/12/2009 39

Support

Configuration Management

Inputs: Any element involved in the project

Outputs: Information about the change

Description: Changes in the configuration

items must follow a formal process of change.

The impact is assessed and the change is

accepted or rejected.



12/12/2009 40

Support

Measurement and Analysis

Inputs: Project and process specifications

Outputs: Measurement reporting

Description: To control the processes

implemented in the organization. Information

collected is measured and analyzed based on

set indicators. A measurement report is

drafted.



12/12/2009 41

Use of The Model

The proposed model has been

implemented at several companies in

Spain

The proposed process model was adapted

for each implementation to include:

Justification for using a Test Factory

Metrics to measure the performance of the

model

Methodological guides to be considered when

testingSanz, A., Garcia, J., Saldana, J., & Amescua, A. (2009). A proposal of a proces model to create a Test Factory. WoSQ '09 (pp. 65-69).


12/12/2009 42

Conclusion

Test Factories is one way to structure

testing in organizations

Successfully managing a Test Factory

requires a customized management model

that includes

A process model

An organizational structure

12/12/2009 43

Conclusion

The proposed model defines:

The processes required to manage a testing

factory

Key elements to develop a contract for

outsourcing testing activities

The proposed model has been

implemented in several companies but the

results were not presented in this article

12/12/2009 44

Future of Software Testing

12/12/2009 45