Evaluation of Quality Assurance Factors in Agile Methodologies

International Journal of Advanced Computer Science, Vol. 2, No. 2, Pp. 73-78, Feb. 2012.

Manuscript Received:

Revised:

Accepted:

Published:

Keywords

Agility,

Software

Quality

assurance,

Quality

factors,

Testing

Techniques

Abstract__ The agile software development

claims to improve the quality of the software

products. This claim has increases its usage

in the new industry of the software and

information technology. But, due to its

immature nature, very few frameworks for

the evaluation of quality exist. The proposed

research has introduced a tool for the

evaluation of quality in agile software

development. Various factors addressing the

quality have been evaluated among different

phases in various practiced methodologies of

agile software development. The proposed

tool has provided a new platform for

application of various types of testing in agile

methodologies. It has opened a new horizon

of research in software industry

1. Introduction

The traditional methodologies for the software

development focused on waterfall approach. After

achieving the freezing point of requirements the system was

developed in increments. However, volatile nature of

market, rapid change in customer requirements ,dynamic

infrastructure of business organization , exhaustive need for

short time deliverables and highly variations in pace of

development technologies and tools added the competitive

pressure to sustain in the market. As a result many

organizations adapted agile method for software

development. Practical implementation of software

development is highly dependent on the practices followed

by different organizations. The introduction of Agile as

software methodology in literature was introduced in late

90's.After that several methodologies have been proposed.

With the above mention factors the conformance to

quality standards was also challenging task. The agile

development promises to provide quality assurance. In fact

agile development is popular for quality delivery of the

project in small deliverables. The element of the proposed

research also targets the issues of quality in agile

development.

This work was supported by the National University of Science and

Technology , Rawalpindi ,Pakistan

M.Sirshar, with National University of Science and Technology,

Software Engineering MS Research scholar ([email protected])

F.Arif, with National University of Science and Technology,Rawalpindi

Pakistan([email protected])

The paper is organized into five sections. Section contains

literature review. Section 3 includes quality assurance

(QA) factors being evaluated in the research. Section 4

introduces tools and applications on agile methodologies.

Section 5 discusses the analysis .Section 6 draws

conclusion

2. Literature Review

Many definitions have been proposed for defining

quality and agile development. The literature survey reveals

very few definitions regarding agile and quality together.

A. Defining Quality

Many researchers agree on the consensus that quality is

conformance to customer needs. Many researches enhance

this view to define quality as characteristics bear by the

software product which must reflect the needs specified by

the customer.

Meyer [1] defines software quality according to an

adapted number of quality factors as defined by McCall [2],

which are; correctness, robustness, extendibility, reusability,

compatibility, efficiency, portability, integrity, verifiability,

and ease of use etc.

Pressman agrees with Crosby and defines quality as

“conformance to explicitly stated functional requirements,

explicitly documented development standards, and implicit

characteristics that are expected of all professionally

developed software” [3]. Sommerville [4] defines software

quality as a management process concerned with ensuring

that software has a low number of defects and that it reaches

the required standards of maintainability, reliability,

portability and so on. Pfleeger [5] agrees with Garvin who

views quality from five different perspectives namely:

• The transcendental meaning that quality can be recognized

but not defined,

• User view meaning that quality is fitness for purpose,

• Manufacturing meaning that quality is conformance to

specification,

• Product view meaning that quality is tied to inherent

product characteristics

• The value-based view meaning that quality depends on the

amount the customer is willing to pay for the product.

A. Defining agility

In literature, agility is commonly refers to the readiness

to motion. The Agile manifesto was published by group of

software engineers and consultants in 2001. It is stated as

Evaluation of Quality Assurance Factors in Agile

Methodologies

Mehreen Sirshar & Dr. Fahim Arif


International Journal Publishers Group (IJPG)©

74

follows [6]-[7]:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

Ambler [8] considers agile quality to be a result of

practices such as effective collaborative work, incremental

development, and iterative development as implemented

through techniques such as refactoring, test-driven

development, modeling, and effective communication

techniques. The definition of quality from view of agile

development is still dependent on organizational practices

and the agile methodology being followed.

From the agile perspective quality has been defined as

McBreen [9] defines agile quality assurance as the

development of software that can respond to change as the

customer requires it to change. This implies that the

frequent delivery of tested, working, and

customer-approved software at the end of each iteration is

an important aspect of agile quality assurance.

B. Quality assurance factors in agility

Table 1, defines different quality attributes followed in

agile development. Each quality factor is evaluated in

different agile methodologies. Some organizations follow

mixed methodologies to achieve different quality

requirements. Analysis of methodologies reveals some

practices which are suggested to be followed to achieve

high quality.

All agile methodologies do not address all of the quality

assurance factors .Several methodologies address multiple

factors. However combinations of methodologies are

followed to ensure the required quality factors. Quality

factors are non functional attributes of the system which are

implicitly stated .Therefore, they are most difficult to define

and evaluate.

Compatibility

Platform independence can be achieved by multiple

agile methodologies especially open source development

methodologies. It can also be achieved by following OO

(object Oriented) design in local practices of an

organization.

Cost Effectiveness

Iterative and incremental nature helps to achieve cost

effectiveness by delivering the prioritized requirements in

small increments.

Correctness

Many agile methodologies involve active involvement

of the customer therefore correctness in requirements is

handled and the correctness in the behavior of the desired

system is also achieved.

Ease of use

Active involvement of the customers increases the

understanding of the system thus increasing in ease of use

of the system when designing its interface. Interface is

highly important when web layouts are designed. However,

this may create problems when open source development is

followed but it is easily achieved through scrum, XP

(extreme programming), RUP (rational unified process) and

JAD (joint application development) techniques.

Extendibility

OOP design, COTS and modular development help to

cater changing requirements with greater flexibility the

system can be easily extended by achieving low coupling.

Almost all of the agile methodologies follow OOP designs

to achieve it.

Maintainability

Modular design and compatibility between interfaces

help to achieve maintainability. Feature driven development

and crystal methodologies play important role to achieve

maintainability.

Performance

Sprint, OO designs, code ownerships and common class

activities help to increase performance in almost all of the

existing agile methodologies.

Portability

Distributed computing and web service design has

TABLE 1:

QUALITY ASSURANCE FACTORS AND THEIR DEFINITIONS [20]

Quality

Parameters

Definitions

Correctness The ability of a system to perform according to defined specification.

Robustness Appropriate performance of a system under cases not covered by the specification. This is complementary to correctness.

Extendibility A system that is easy to adapt to new specification.

Reusability Software that is composed of elements that can be used to construct different applications

Compatibility Software that is composed of elements that can easily combine with other elements.

Efficiency The ability of a system to place as few demands as possible to hardware resources, such as memory, bandwidth used in

communication and processor time.

Portability The ease of installing the software product on different hardware and software platforms

Timeliness Releasing the software before or exactly when it is needed by the users.

Integrity How well the software protects its programs and data against unauthorized access

Verification and

validation

How easy it is to test the system.

Ease of use The ease with which people of various backgrounds can learn and use the software.

Maintainability The ease of changing the software to correct defects or meet new requirements

Performance Low utilization of resources, lower response time and mean time of failure and recovery define the performance of the system.

Cost effectiveness The ability of a system to be completed within a given budget.

Mehreen Sirshar et al.: Evaluation of Quality Assurance Factors in Agile Methodologies.


75

increased portability in agile methodologies.

Integrity

Low coupling among design increments help to achieve

integrity of the data as well as the integrity of system

Reusability

OO (object Oriented) designs are based on the principle

of reusability. Furthermore, organizations following agile

development keep track of all design increments to make

them available in short time boxes.

Robustness

Robustness is ensured by following development

standards however robustness is implicit. It cannot be

explicitly stated.

Timeliness

Iterative development and short cycles are followed to

achieve timeliness.

Validation and verification

Unit testing, integration testing and regression testing

and interface testing are commonly followed methodologies

to ensure validation and verification. Further more user

involvement ensures verification with each deliverable.

3. Tool for evaluation of quality

for agile methodologies

This section provides a brief overview of existing agile

methodologies, quality factors in agile development and

different quality assurance factors. The proposed tool

provides assurance factors. The proposed tool provides

different testing techniques that can be applied to different

quality attributes supported by specific methodology

through quality assurance factors. One such tool was

proposed by Hou et al [10] based on comparative study

between waterfall methodologies and existing agile

methodologies. The tool had certain limitations found in

analysis as mentioned below:

• Singles out two main aspects of quality management

namely; quality assurance and verification and validation.

• Overlooks other vital techniques used in agile processes to

achieve higher quality management.

• And is mainly based on Extreme Programming which on it

own is not a complete representation of agile

methodologies.

To address these limitations another tool was proposed

by Mankandla et al [11] .But the tool failed in its

applicability to all agile methodologies. It focuses only on

two of the widely used agile methodologies. In the

subsequent development effort similar tool has been

proposed in this research paper to address the limitations

provided by proposed tool. The proposed tool is applicable

to all agile methodologies.

4. Agile Methodologies.

Agile methodologies take the advantage of object

Oriented programming and component based software

development paradigms. Commonly followed agile

methodologies have been discussed below:

A. Extreme Programming

Beck et al [12] defines five phases of extreme

Programming: Exploration, Planning, Iteration to release,

Productionizing, Maintenance and death. In exploration

phase user requirements are collected on story cards,

planning focuses on making plan for deliverable defined

through story cards, extra testing and checking is done in

productionizing phase, maintenance phase deals with

keeping interest of market, customer and product in

connection to changing requirements where as death phase

is end or complete delivery of the product. It is notable that

the focus of the maintenance phase is on execution of

quality assurance activities.

B. Scrum

Scrum focuses on self organizing teams especially in the

scenarios where the environment and resources are highly

meant to variant. According to Schwaber[13] Scrum follows

three phases. First phase is pre-game phase which includes

planning and architecture development. Second is

development phase and third is post-game phase which

deals with closure of release. Major focus is on carrying the

activity known as Sprint in the development phase. Sprint is

iterative in nature and focus is on dealing with changing

demands and requirements.

C. Crystal Methodologies

Crystal Methodology is among recently developed

methodologies. Two well known methodologies are Crystal

Clear Methodology and Crystal Orange Methodology.

Crystal methodologies follows incremental delivery, direct

user involvement, timeliness defined with milestones,

regression testing , two user reviews per release, workshop

and interviews for product and methodology tuning, quality

and design standards, version control strategies and

compilation tools. According to CockBurn [14] each

increment of crystal include the development of

requirement specification document ( not detailed )

followed by the phase of staging, includes planning and

scheduling, followed by several iterations development,

monitoring, stability with milestones and review before

the final release.

D. Feature Driven development.

Feature driven development focuses on design and

implementation features in five steps using object oriented

methodology. Palmer et al [15] defines, develop and build

overall model, build feature list, plan by feature, and design

by feature and build by feature as five distinct stages.

Domain or context modeling is an important aspect of this

methodology which helps to increase performance by clear

understanding of the system. Inspection is commonly

followed defect prevention strategy. Feature defines the

functionalities required from the product by the customer.

E. Rational Unified process (RUP)

RUP has been developed to complement UML modeling

while following the agile methodologies. It is an iterative

paradigm based on OO (Object Oriented) Modeling.

Kructhen [16] states inception, elaboration, construction and



76

transition as major processes followed in RUP methodology.

Business Modeling and environment workflows are

important part of RUP methodology. The transition phase in

RUP is totally dedicated to testing to ensure quality.

Inception is focused on gathering requirements, elaboration

on building models and construction on development of the

systems based on modeling content.

F. Dynamic System Development Methodology

(DSDM)

DSDM has been evolved to support RAD (Rapid

application Development framework).DSDM is based on

learn, speculate and collaborate phases. [17] DSDM is

component oriented approach. It is preferably followed for

large and complex systems. In speculation phase the project

is instantiated and planning is done. In collaboration phase

various component based methodologies are coordinated. In

last phase of learning quality reviews are carried out, quality

assurance is achieved and product is release.

G. Open Source Software Development(OSSD)

OSS is volunteer based distributed development

paradigm .It is based on versioning control systems located

on geographically distributed zones. Apache Server, Perl

and Linux are common examples of it. No project planning

and documentation is done. However the code is tested

before launching an open source platforms. Two common

open source platforms are 'Sourcefourge' and 'Freshmeat'

which provide free web based hosting. Sherma et al [18]

proposed problem discovery, finding volunteers, solution

identification, code testing and reviewing, code change view,

code commit documentation and release management as

commonly followed phases.

H. Agile Modeling or Lean Development

It is among most recently introduced methodologies. It

is based on repairing acute problems in design and

documentation, and support vast cultural boundaries and

geographically distributed organizations [19].

I. Pragmatic Programming

Pragmatic programming has no defined processes, tools

and standards .It is at the liberty of the organization

following some internal methods as well upon the

collaboration and communication among geographically

dislocated teams. Automation of tasks is preferred in

TABLE 2: EVALUATION OF METHODOLOGIES AGAINST QUALITY FACTORS.

Quality Parameters Agile Methodologies and quality activities

XP (extreme

programming)

Srcum Crystal methodologies Feature driven

development

RUP

Correctness User stories Review meetings Interviews and formal

meetings, printing

whiteboards

Feature lists prepared

with customers

Inception phase.

Interviews with

stakeholders

Robustness Generic OO design OO Design Tuning methods OO design OO design

Extendibility Simple design Prioritization Versioning systems Simple design +

versioning system

UML design

Reusability OO design Existing design Configuration management

tools

OO design UML design, COTS

development

Compatibility Inherit in OO deign OO design Version control Version control Inherit in OO design

Efficiency Pair programming Team meeting White boards Class code ownership UML modeling

Portability OO design practices OO designs Compilation tools Configuration

management support

Component Based architecture

Timeliness Incremental and iterative

development

Iterative Incremental 4-6 weeks Iterative development Iterative development

Integrity OO design OO design Common object model Class code ownership UML modeling

Verification and

validation

Unit testing Testing and reviews Regression testing , Interface

testing

Inspection Acceptance testing

Ease of use Simple design Simple design GUI Designs, User manuals Feature list UML modeling

Maintainability Simple design Compatible with

existing system

Versioning system OO design and versioning

control

UML compliances

Performance Productioniz-ing phase Sprint phase Common object Model Class code ownership OO design

Cost effectiveness Iterative development Iterative development Incremental delivery on

regular basis

Reusable design features

in iterative manner

Iterative development with OO

design

Correctness User stories Review meetings Interviews and formal

meetings, printing

whiteboards

Feature lists prepared

with customers

Inception phase.

Interviews with

stakeholders

Robustness Generic OO design OO Design Tuning methods OO design OO design

Extendibility Simple design Prioritization Versioning systems Simple design +

versioning system

UML design

Reusability OO design Existing design Configuration management

tools

OO design UML design, COTS

development

Compatibility Inherit in OO deign OO design Version control Version control Inherit in OO design

Efficiency Pair programming Team meeting White boards Class code ownership UML modeling

Portability OO design practices OO designs Compilation tools Configuration

management support

Component Based architecture

Timeliness Incremental and iterative

development

Iterative Incremental 4-6 weeks Iterative development Iterative development

Integrity OO design OO design Common object model Class code ownership UML modeling

Verification and

validation

Unit testing Testing and reviews Regression testing , Interface

testing

Inspection Acceptance testing

Ease of use Simple design Simple design GUI Designs, User manuals Feature list UML modeling

Maintainability Simple design Compatible with

existing system

Versioning system OO design and versioning

control

UML compliances

Performance Productioniz-ing phase Sprint phase Common object Model Class code ownership OO design

Cost effectiveness Iterative development Iterative development Incremental delivery on

regular basis

Reusable design features

in iterative manner

Iterative development with OO

design

Mehreen Sirshar et al.: Evaluation of Quality Assurance Factors in Agile Methodologies.


77

pragmatic programming.

Table 2 provides a comparison among different

discussed methodologies and quality assurance factors

obtained from them in different processes. Pragmatic

Programming has been intentionally left blank due to its

immature nature and unavailability in literature.

5. Analysis and Suggestions

Testing techniques which are suggested to be followed

in particular methodology are tabulated in Table 3.

Complex documentation is avoided. Automated testing

techniques are preferred due to short time boxes. Beta

testing, regression testing and integration testing are usually

followed.

Analysis of the table reveals the fact that apart from

following complex and formal testing techniques simple

and commonly followed techniques are used in nature. It is

due to the fact that almost all of the agile methodologies

are based upon design simplicity , to promote reuse and

efficiency as well as they are based on Object oriented

Methodologies.

6. Conclusion

This research has introduced a innovative tool for

evaluation of agile methodologies against quality assurance

factors. The quality management in agile methodologies is

also new era of research. The limitation of tool is that is

applicability in industry has not been verified. It may be due

to difference in practices followed from organization to

organization. The proposed tool would spark new era of

research in field of agile software development especially

the scenarios of agile modeling architecture for enterprise or

offshore development.

References

[1] Meyer, B., Object-Oriented Software Construction, Prentice

Hall PTR, pp.4-20, 2000

[2] McCall, J.A., Richards, P. K., & Walters, G. F., Factors in

Software Quality, Vols. 1, 2 and 3, National Technical Information

Service, 1977

[3] Pressman, R.S., Software Engineering a Practitioner’s

Approach, Mcgraw-Hill, 2001

[4] Sommerville, I., Software Engineering. Addison-Wesley,

2004

[5] Pfleeger, S. L., Software Engineering: Theory and Practice,

Prentice Hall, 2005

[6] CockBurn, A., Agile Software Development, Boston,

Addison –Wisely, 2002 (a)

[7] Beck, k., Beedle, M., Van Bennekum, A., CockBurn, A.,

Cunningham, W., Folwer, M., Gernning, J., Highsmith, j., Hunt, A.,

Jeffries, R., Kern, J., Maerik, B., Martin, R., Mellor, S., Schwaber,

K., Sutherland, J., & Thomas, D., Manifesto for software Agile

Development (22.3.2002) ,2001, http//: agileManifesto.org

[8] Ambler, S., Quality in an Agile World, Software Quality

Professional, Vol. 7, No. 4, pp. 34-40, 2005

[9] McBreen, P., Quality Assurance and Testing in Agile Projects,

Available from: http://www.mcbreen.ab.ca/talks/CAMUG.pdf

[Accessed 26 July 2006].

[10] M. Huo, J. Verner, Y. Zhu, & M.A. Babar, "Software Quality

and Agile Methods," (2004) Proceedings of the 28th Annual

International Computer Software and Applications Conference

(COMPSAC04), IEEE Computer

[11] E. Mnkandla, & B. Dwolatzky, "Defining Agile Software

Quality Assurance," (2006) Proceedings of the International

Conference on Software Engineering Advances (ICSEA'06), IEEE

[12] K. Beck, "Extreme Programming Explained: Embrace

Change, Reading Mass, Addison Wesely," (1999) IEEE Computer

vol. 32, no. 10, pp. 70-77

[13] Schwaber, K., & Beedle, M., Agile Software Development

with Scrum, Upper Saddle River, NJ, Prentice-Hall, 2002

[14] CockBurn, A., Writing Effective Use cases, The crystal

collection for software professionals, Addison-Wisely Professional,

2000 (b)

[15] Palmer, S.R., & Fesling, J.M., A practical Guide to Feature

driven Development, Upper saddle River, Prentice Hall, 2002

[16] Kructhen, P., the Rational Unified Process, an Introduction,

Addison-Wesely, 2000

[17] Highsmith, J. A., Adaptive software Development, a

Collaborative Approach to managing complex systems, New York,

NY, Dorset House Publishing, 2000

[18] S. Sharma, V. Sugumaran, & B. RajaGopalan, "A

Framework for creating hybrid open source software

communities," (2002) Information system Journal, vol. 12, no. 1,

pp. 7-25

[19] S. Ambler, "Lessons in agility from Internet based

development," (2002) International proceedings on information

systems, IEEE

[20] IEEE Standard Glossary of Software Engineering

Terminology, IEEE Std 610, 1990

TABLE 3:

TESTING TECHNIQUES AND AGILE METHODOLOGIES

Agile methodology Testing techniques

Extreme Programming Unit Testing, acceptance testing, integration testing, functional testing by tester or user, customer

Feedbacks

Scrum Review meetings, unit testing , functional testing, and sprint backlogs

Crystal Methodologies

Crystal clear and crystal Orange)

Regression testing , interface testing and repeated GUI testing

Feature driven Development Inspection, formal testing and unit testing

Rational Unified Process Alpha testing, Beat testing, component integration testing

DSDM (dynamic system Development

Methodology)

Regression testing, Converge testing

Open Source software development(OSSD) Beta testing

Lean development or agile Modeling Beta testing, Inspection



78

Mehreen Sirshar , was born in Rawlpindi , Pakistan ,in 1985. She

Received her B.S.E degree from Fatima Jinnah

University ,Rawalpindi Pakistan.She is Currently Reseach

Scoholar for M.S (S,E) at National University of Science and

Technology Rawalpindi ,Pakistan.She is Serving as Lecturer in

Software Engineering in Fatima Jinnah Women University .She is

author of 3 international publications, reviewer of various

international conferences and supervisor of couple of graduate

degrees. She is currently conducting her Ms-Phd courses in

Millitary College of Signals.(a campus of National Univeristy of

Science and Technology) Rawalpindi Pakistan.She is currently

conducting her research in Digital video tracking projects.

Dr Fahim Arif was commissioned in

Signal Corps of Pakistan Army in 1988.

He has performed duties in Army units,

Research and Development setup of Corps

of Signals and National University of

Sciences and Technology. He has served

as Telecommunication Engineer in Special

Communication Organization and Frontier

Works Organization in southern Sindh

area. He has done Bachelors in Telecommunication from Military

College of Signals (UET Lahore) in 1995 and Master in Sciences

in Computer Software Engineering from National University

Science and Technology in 2003. He has won NUST Endowment

fund scheme scholarship for NUST in 2003 and International

Research Support Initiative Program Fund from HEC in 2007. He

has completed his PhD degree from National University Science

and Technology in 2009. His contribution to international research

in recent few years is excellent. He has presented/published eight

research papers in different international conferences including

USA and Canada. In addition to his research publications, he is

doing as reviewer for various international conferences. He work

as international researcher in System and Computer Engineering

Department, Carleton University, Ottawa, Canada in 2007 and

participated numerous research and academic activities. Recently,

his biography has been published by South Asian Publication

Who’s Who in the World 2008 Edition and awarded with Star

Laureate 2008 in recognition to his contributions to knowledge and

research. Higher Education Commission has nominated him as

their official PhD supervisor in 2010. Currently, he is teaching

various academic courses at MCS and supervising one PhD

student and different MS/ BE students in their final projects/thesis

in NUST.

Documents

Evaluation of Quality Assurance Factors in Agile Methodologies