[AgileCMMI] Practical Report: CMMI Measurements and Analysis in Agile environment

CMMI® 9th Technology Conference and User Group | www.ndia.org/meetings/0110 2009

Practical Report: CMMI Measurements and Analysis practices

based on Agile (Scrum) Method

Ahmed Mahdy

Senior Software Quality Engineer, Agile Coach

Raya Corporation, Egypt

[email protected]

[email protected]

Sree Yellayi

SCAMPI Lead Appraiser, SEI Authorized CMMI Instructor, Certified Scrum Master

Siemens, Greater New York City Area

[email protected]

[email protected]

Reviewed by Hillel Glazer

Certified High Maturity SCAMPI Lead Appraiser, SCAMPI B&C Team Leader, and an Introduction to

CMMI® instructor for both CMMI for Development and CMMI for Services

[email protected]

Abstract Software industry is one of the most empirical industries due to its high dependence on technology and people. Each

software company adopts a specific development methodology, and furthermore seeks to build a system for its process

improvement by adopting one or more Software Process Improvement (SPI) models. CMMI is a process framework

which is widely adopted by software and systems development companies, while Scrum is one of the more

recent project management agile method whose adoption is growing rapidly. CMMI is basically a process

improvement framework which provides a set of processes for software and system development management, Scrum

can be thought of as an iterative project management framework for development activities, CMMI has a wider scope

and different aims to those of Scrum and covers production support, maintenance, product implementation and

application transition type projects as well.

Scrum and other agile methods have clearly appeared in 2001, this paper shows how to implement the Measurements

and Analysis (M&A) process area of CMMI model and clarify how M&A can be achieved in the agile (Scrum)

organization.

Most of the promising objectives of any software development method or process are delivering working software on

time, quality and budget. Software Engineering Institute (SEI) has paid a lot of efforts to mostly satisfy these

objectives in its process improvement models, and lately CMMI version 1.2 has been released.

Nevertheless, the world becomes convinced with adding another objective, which is delivering a business value to the

customer. Along the years, software engineers have proposed several methodologies. Agile is the most known and

latest proposed development method that can achieve this objective beside the other mentioned objectives.

Our objective in an agile environment is not to do software measurement. We must learn to build reliable software

measurement process based on valid software measurement tools. If we try to do too much too soon, we will likely

fail.

Introduction Software industry is one of the most empirical industries due to its high dependence on technology and people. Each

software company adopts a specific development methodology, and furthermore seeks to build a system for its process

improvement by adopting one or more Software Process Improvement (SPI) models. CMMI is a process framework

which is widely adopted by software and systems development companies, while Scrum is one of the more recent

project management agile method whose adoption is growing rapidly. CMMI is basically a process improvement

framework which provides a set of processes for software and system development management, Scrum can be


thought of as an iterative project management framework for development activities, CMMI has a wider scope and

different aims to those of Scrum and covers production support, maintenance, product implementation and application

transition type projects as well [1]. In 1996, Kent Beck joined the Chrysler C3 payroll project. It was in this context

that the full set of XP practices matured, with some collaboration by Ron Jeffries and inspiration from earlier 1980s

work at Tektronix with Ward Cunningham. XP went on to garner significant public attention because of its emphasis

on communication, simplicity, and testing, its sustainable developer-oriented practices, and its interesting name. [2]

Scrum and other agile methods have clearly appeared in 2001, Alan MacCormack reported a study of key success

factors in recent projects; first among these was adopting an IID life cycle: Now there is proof that the evolutionary

approach to software development results in a speedier process and higher-quality products. […] The iterative process

is best captured in the evolutionary delivery model proposed by Tom Gilb [3]. Scrum appeared when a group of 17

process experts—representing DSDM (Dynamic Systems Development Method), XP (extreme Programming) ,

Scrum, FDD (Feature Driven Development), and others— who are interested in promoting modern and simple IID

(Iterative and Incremental Development) methods and principles, they met in Utah to discuss common ground. From

this meeting came the Agile Alliance (www.agilealliance.org) and the now popular catch phrase “agile methods,” all

of which apply IID. And in 2002, Alistair Cockburn, one of the participants, published the first book under the new

appellation [4]. Moreover, prominent software-engineering thought leaders from each succeeding decade supported

IID practices, and many large projects used them successfully [5]. Agile is all about value individuals and interactions

over processes and tools, working software over comprehensive documentation, customer collaboration over contract

negotiation, and responding to change over following a plan [15]. This paper shows how to implement the

Measurements and Analysis (M&A) process area which is an important Process Area (PA) in CMMI model; the

organization cannot be appraised without satisfying this PA, and clarifying how M&A can be achieved in the agile

(Scrum) organization.

Basically, software engineering measurement is not a resource issue; it is a commitment issue. The bottom line is that

all the usable measures from practicing Scrum on a project could be used to address the practices of M&A process

area. In fact, the alignment of these measures to the information needs is very visible due to the very nature of “value-

based” focus of the scrum method. In our research, we found that there need not be any additional measures that are

required to be “invented” to fulfill the M&A goals from CMMI model. Usage of existing ones is enough and we will

share those mappings and arguments with you for your application and subsequent extension to other process areas in

the model.

Agile CMMI Why Agile CMMI

Most of the promising objectives of any software development method or process are delivering working

software on time, quality and budget. Software Engineering Institute (SEI) has paid a lot of efforts to mostly

satisfy these objectives in its process improvement models, and lately CMMI version 1.2 has been released.

Nevertheless, the world becomes convinced with adding another objective, which was somehow neglected

more than highly considered, which is delivering a business value to the customer. Along the years, software

engineers have proposed several methodologies (see figure 1),


Waterfall (Royce): Requirements,

Design, Implementation, Verification

and maintentance

V-Model (Anon): Align Testing to

waterfall development.

Spiral Model (Barry Boehm)

Iterative

RAD (James Martin)

Prototyping, iterative, time-boxed, and

user driven

RUP (Rational)

Object Oriented, Iterative, time-boxed, and user

driven

Agile (Kent Beck)

Incremental, user driven, customer interaction

1965

1970

1980

1985

1991

1998

1999

Years

(Figure-1: History of Software Development Methodologies)

Agile is the most known and latest proposed development method that can achieve this objective beside the

other fore mentioned objectives.

Feasible together?

Yes! [6]

Requirement of CMMI Measurements and Analysis (M&A)

Model Components [7]


(Figure-2: CMMI Model Components)

Measurements and Analysis Required Objectives and Expected Practices

o SG 1: Align Measurement and Analysis Activities � SP 1.1: Establish Measurement Objectives

� SP 1.2: Specify Measures

� SP 1.3: Specify Data Collection and Storage Procedures

� SP 1.4: Specify Analysis Procedures

o SG 2: Provide Measurement Results � SP 2.1: Collect Measurement Data

� SP 2.2: Analyze Measurement Data

� SP 2.3: Store Data and Results

� SP 2.4: Communicate Results

Implementation of M&A M&A Procedure Flow


(Figure-3: Measurements & Analysis Procedure Flow)

Identify Measurement Objectives

A child must learn to crawl before he walks. He must learn to walk before he runs. So it will be for an

incipient software measurement program. We must learn to crawl first. At the outset, the complexity of the

measurement problem space appears astonishingly large. There are many products, such as requirements,

design, and code. Each of these products was produced by a process, in a development environment, by

people [8]. It is very difficult to measure a software process. This is not a good place to start. It is very

dangerous to measure people. This measurement data is so easily misused. Until we have gained some real

sophistication in measuring people and process, we will have little success in trying to measure aspects of

the software development environment. [9]

Our objective in an agile environment is not to do software measurement. We must learn to build reliable

software measurement process based on valid software measurement tools. If we try to do too much too

soon, we will likely fail. Basically, software engineering measurement is not a resource issue; it is a

commitment issue. We are going to build a measurement process that will generate great volumes of data.

These data must be converted to information that can be used in the software development decision-making

process. The principle is a simple one. Even a small amount of measurement data that can be converted to

useful information is better than large volumes of measurement data that have little or no information value.

We must begin with simple tools and focus most of our attention on the measurement and management

processes. We must remember that we learned how to measure distances in the first grade with a very crude

ruler. Our teachers did not give us micrometers to learn measurement. Ultimately, we learned that our rulers

could be used to quantify size attributes of the objects in our environment. These rulers, in fact, had utility

beyond their obvious use as bludgeons that we could use on our classmates. [10] In order to monitor and evaluate process performance we must consider views of different stakeholders that take part in the process. The best performance is

achieved when the objectives of all stakeholders are satisfied. Kueng [11] defines process performance as “the degree of

stakeholder satisfaction”.


The objectives are subjective and should be defined from the organization; however, here are some

examples of what objectives vs. stakeholders can be established (table 1):

Objectives Stakeholder

• Timely information on project information performance and progress

• Product Quality Improvement Progress

• Process Quality Improvement Progress

IT Management

• Time Utilization and Management

• Roles and Responsibilities Commitment

Project Team Members

• Efficient and effective resolutions for the impediments

• Process Compliance

Project Scrum Master and

Quality Assurance

• Customer Satisfaction Customer

Table 1: Practical Example of Measurement Objectives and Stakeholders

To illustrate the meaning of who could most likely be the fore mentioned stakeholders:

IT Management: General Manager, Project Manager who are concerned with the traditional aspects of

software development from the perspective of time, cost, and quality.

Project Team Members: Developers, Testers, Team Leader, and Technical Writers

Project Quality Assurance & Coach: Concerned with facilitating the use of SCRUM and CMMI, creating

conditions for smooth running of the development process, hence the main goal is to provide an efficient

impediments resolution.

Customer: the customer, customer representative, and/or other third parties.

Specification of Measures

The measures are obtained based on the defined objectives, by thinking of each objective and all related

links that negatively or positively affect it.

According to CMMI, measures may be either “base” or “derived”. While data for base measures are

obtained by direct measurement, data for derived measures come from other data, typically by combining

two or more base measures. Derived measures serve as performance indicators showing the achievement of

particular goals. Originally, Scrum had only one base measure: the estimate of the amount of work

remaining that needs to be done in order to complete a Product Backlog item or a task in the Sprint Backlog

(SB). At the task level, this measure is collected every day for each task in the Sprint Backlog separately.

[12]

Fortunately, the daily Scrum meetings give an outstanding help to achieve Measurements & Analysis,

especially the data collection.

Here are some proposed measures as a practical example:

Objective Measures

• Timely information on project information performance and progress

• Remaining work from Sprint S

• Remaining work on day d for each task in the Sprint Backlog (SB)

• Each task progress for day d in the SB


• Time Utilization

and Management • Number of working hours per day d

• Efficient and effective resolutions for the impediments

• Number of impediments

• Number of resolved impediments

• Customer Satisfaction

• Customer Acceptance

• Customer Survey Result

• Product Quality Improvement Progress

• The number of bugs found during the Sprint S.

• The number of bugs reported by the user in a fixed period after Release R.

• Total number of Product Backlog Items (PBIs) or done stories committed in the Sprint S.

• The number of PBIs completed in the Sprint S.

• Total number of tasks in the Sprint

• The number of tasks completed during the Sprint S.

• Actual Vs Planned in any Process Improvement Change

• Number of Process Improvement Requests

• Roles and Responsibilities Commitment

• The number of completed tasks for each team member (for each Sprint Backlog)

• Process Compliance

• Number of Non-Compliances (NCs) reported by Quality Assurance per Sprint S.

• Number of resolved NCs per Sprint S.

Table 2: Practical Example of Measurement Objectives and its Measures

Data Collection Points

All base measures are based on data

(Figure-3: Data Collection Points for the specified measures)

It’s not recommended to propose a mapping between the measures and its collection points in a general

proposal, to allow more flexibility to different environments and practices.


The collection points should be defined in the process and can be customized per project (if needed).

Moreover, the use of automated tools is much recommended after gaining the main concept of Agile and

deeply knowing the goals of CMMI.

Measurements Analysis

Derived measures are derived from the base measures, such derived measures support the decision

makers in different management levels that serve for analyzing software process performance in

comparison to target values set by software development organization.

As practical examples: The objective “Timely Information on Project Progress” can be analyzed using the following indicators:

- Work Effectiveness It’s the ratio between the decrement of remaining work and done work. The decrement of remaining work between days d1 and d2 of Sprint should be equal or greater than the done work in the same interval, meaning that the best value for this indicator is 1 or more; however, the values significantly greater than 1 may be a sign of poor planning.

d

d

dWP

WSWE = (Eq. 1)

Where dWE denotes the Work Effectiveness of a specific day d in a Sprint, dWP denotes the planned

work that should be accomplished in the beginning of the day d, dWS denotes the work spent in the

day d. The calculation of Works can be in terms of tasks’ number or the average percentages of

tasks’ completion.

However, burn-up or burn-down charts can do this job in an easier visualized way, and it can be done

weekly or every 2 days.

- Schedule Performance Index (SPI) It’s the ratio between the Earned Value (EV: the value of all completed tasks) and Planned Value (PV: the

estimated planned value of all tasks to be completed in a certain time during the project) [13]. The best value for SPI is 1 or more; however, if it’s greater than 1, then the project is ahead of schedule.

And to calculate the Schedule Performance Index using the work remaining and work spent measures we assume that the amount of tasks that must be accomplished at a certain point in the Sprint is proportional to the time elapsed from the beginning of the Sprint. The work remaining and work spent measures allow a

precise definition of the Earning Value equation jdEV , for each task j in the Sprint Backlog on the day d of a

Sprint. It can be computed as a ratio between the amount of work already spent and all the work required (spent and remaining) to accomplish the task:

jd

d

i

ji

d

i

ji

jd

WRWS

WS

EV

,

1

1

,

1

1

,

,

+

=

∑

∑−

=

−

= (Eq. 2)

jiWS , denotes the amount of work spent for task j on day i, i=1,2,…,d-1, and jdWR , denotes the amount of

work remaining for task j on day d.

By using the earning value, we can get the SPI by the following formula:


DE

SL

WR

WREV

SPIn

j

jinitial

n

j

jinitialjd

d •=

∑

∑

=

=

1

,

1

,,

(Eq. 3)

Where jinitialWR , denotes the initial estimate of the work remaining for task j, SL the Sprint Length in the unit

of days, and DE the number of days elapsed.

- Cost Performance Index of employee costs (CPI)

It’s the ratio between the Earned Value (currency unit) and actual costs. The best value for CPI is 1 or more, and this means that the cost of completing the work is planned rightly or less than planned respectively [14].

And for the other objectives:

- The achievement degree of work per Sprint that shows what percentage of the completed tasks

compared with the planned tasks, the ratio between the number of tasks completed in the Sprint and

total number of tasks in the Sprint Backlog or between the number of PBIs completed in the release

and total number of PBIs committed (per Release).

- The average amount of overtime at Sprint/release/project level considering the expected hours, the

amount of work spent and administrative days.

- The average number of projects that each employee works (or participates) in parallel.

- Qualitative evaluation of working conditions or working values like communication and teamwork,

physical discomfort, psychological well-being, workload, supervision, opportunities for growth,

transparency etc. and easily the team can get these measures by practicing Scrum retrospectives per

Sprint or/and Release, in more detail:

1. At the beginning of project or Release, specify the working values (or conditions) 2. At each retrospective, add a part to get a self-assessment for each value from the team

members in one time, and ask for justification for each low and far different assessments,

finally get the recommendations and suggestions from the team members on how to improve

these assessments next Sprint.

Summary

In fact, there is no practice in scrum like the expected practices of Measurements and Analysis (MA).

Nevertheless, in scrum you have the opportunities to get the measures that implement MA easily.

And to summarize the specific practices (table3) and generic practices (table4) of MA process area [16]:

SP 1.1 Establish and maintain

measurement objectives that

are derived from identified

information needs and

objectives.

• Team Kickoff or Chartering of the project

• The more important thing in any measurement system is

the settlement of objectives and values/purposes for each

measure

• The measures can be tailored by project also to add the

best value

SP 1.2 Specify measures to address

the measurement objectives. • Sprint Burndown chart that tracks effort remaining.

• Release Burndown or Burnup chart that tracks story

points that have been completed. This shows how much

of the product functionality is left to complete.

SP 1.3 Specify how measurement

data will be obtained and

stored.

• This can be declared/defined in Team Kickoff or

Chartering of the project


SP 1.4 Specify how measurement

data will be analyzed and

reported.

• The Scrum process does describe the purpose and use

the Sprint and Release burndown charts.

• Define the analysis procedure clearly and the mechanism

of reporting.

SP 2.1 Obtain specified

measurement data. • Daily Scrum meeting where Sprint and Release

burndown/burnup data are collected.

SP 2.2 Analyze and interpret

measurement data. • Daily Scrum meeting where Sprint and Release

burndown data are analyzed.

SP 2.3 Manage and store

measurement data,

measurement specifications,

and analysis results.

• This can be declared/defined in Team Kickoff or

Chartering of the project

SP 2.4 Report results of

measurement and analysis

activities to all relevant

stakeholders.

• Daily Scrum meeting where Sprint and Release

burndown charts are reviewed.

Table 3: Summary of MA specific practices in Scrum

GP 1.1 Perform the specific

practices of the measurement

and analysis process to

develop work products and

provide services to achieve

the specific goals of the

process area.

• Table 3 satisfies this practice

GP 2.1 Establish and maintain an

organizational policy for

planning and performing the


process.

• Specify the goals and objectives in the organizational

policy to be aligned with the plan of measurements and

the intended measures along the project.

GP 2.2 Establish and maintain the

plan for performing the


process.

• The Scrum lifecycle definition and the releases (or

milestones) to perform Scrum.

• The defined measures and data collection points

GP 2.3 Provide adequate resources

for performing the

measurements and analysis

process

• The resources and schedule time allocated to perform

Scrum planning, monitoring and requirements activities

(for example: the internal and external chartering that

shows the environment, roles and resources)

GP 2.4 Assign responsibility and

authority for performing the

process, developing the work

products, and providing the

services of the MA

• The resource roles allocated to perform the activities in

process documents and in the chartering as well (i.e.

define who is responsible for measurement planning,

collection… and all activities)

GP 2.5 Train the people performing

or supporting the MA

process as needed

• Scrum team training for the aspects of Scrum that relate

to measurements & analysis. (i.e. training materials,

…etc)

GP 2.6 Place designated work

products of the measurement

and analysis process under

appropriate levels of control.

• Here is CM role.

• To make it simpler, you can take pictures of measures,

make an excel sheet for the measures and measurement

files, and define their location in CM based on project

repository structure

• Using any version control will help you to achieve this

practice (like SharePoint, VersionOne, …etc)


• Include in the repository of measurement & analysis: the

used tools for measurement analysis, selected measures

for this project, data collection points…etc

GP 2.7 Identify and involve the

relevant stakeholders of the


process as planned.

• The list of Scrum team members, their roles and

allocations in the chartering (the beginning of the

project)

• Report to the relevant stakeholders with the results of

measurement analysis.

GP 2.8

Monitor and control the


process against the plan for

performing the process and

take appropriate corrective

action.

• Scrum master follows the plan & schedule of

measurements, and data collection points.

• Scrum master assures that the measurements are

collected and analyzed according to the plan & schedule.

GP 2.9 Objectively evaluate

adherence of the


process against its

process description,

standards, and

procedures, and address

noncompliance.

• Provide measurement results.

• Align measurement & analysis activities

• Specifications of base and derived measures

• Data collection and storage procedures

GP2.10 Review the activities,

status, and results of the

measurement and

analysis process with

higher level management

and resolve issues.

(other generic goals for

level 4 and 5)

• The senior manager has a visibility into scrum teams’

work and progress by burnup or burndown charts.

• The senior manager can be reported with the reports of

measurement & analysis results.

• The actions that are taken by the senior manager

according to the reported measurement & analysis

results.

Table 4: Summary of MA generic practices in Scrum

Conclusion

The bottom line is that all the usable measures from practicing Scrum on a project could be used to address

the practices of M&A process area. In fact, the alignment of these measures to the information needs is very

visible due to the very nature of “value-based” focus of the scrum method. In our research, we found that

there need not be any additional measures that are required to be “invented” to fulfill the M&A goals from

CMMI model. Usage of existing ones is enough.

References [1] Scrum and CMMI: A High level assessment of compatibility, Srinivas Chillara and Pete Deemer

[2] K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, 1999

[3] A. MacCormack, “Product-Development Practices That Work,” MIT Sloan Management Rev., vol. 42,

no. 2, 2001

[4] A. Cockburn, Agile Software Development, Addison-Wesley, 2002


[5] Craig Larman, Victor R. Basili , Iterative and Incremental Development: A brief History

[6] Hillel Glazer, Jeff Dalton, David Anderson, Mike Konrad, Sandy Shrum, CMMI or Agile: Why Not

Embrace Both, Nov 2008

[7] CMMI Product Team, CMMI for Acquisition Ver 1.2, Nov 2007

[8] Juran, J.M., Juran on Quality by Design: The New Steps for Planning Quality into Goods and Services,

Free Press, New York, 1992

[9] Deming, WE., Out of the Crisis, MIT Press, Cambridge, MA, 2000

[10] Munson, J.C. and Coulter, N.S., "An Investigation of Program Specification Complexity," Proceedings

of the ACM Southeastern Regional Conference, April 1988

[11] P. Kueng, “Process performance measurement system: a tool to support process-based organizations,”

Total Quality Management, 2000

[12] J. Sutherland et al., “Scrum and CMMI Level 5: The Magic Potion for Code Warriors,” in Proc.

AGILE 2007

[13] Project Management Knowledge, Schedule Performance Index [online], http://www.project-

management-knowledge.com/definitions/s/schedule-performance-index , 2008

[14] Project Management Knowledge, Cost Performance Index [online], http://www.project-management-

knowledge.com/definitions/c/cost-performance-index, 2008

[15] Manifesto for Agile Software Development [online], http://www.agilemanifesto.org , 2008

[16] Neil Potter and Mary Sakry, Process Group Newsletter Volume 16 No 2, March 2009