January 20, 2000 CSE 7315 - SW Project Management / Chapter 13 – SW Process Maturity, Appraisal & Imp Copyright © 1995-2000, Dennis J. Frailey, All Rights

January 20, 2000

CSE 7315 - SW Project Management / Chapter 13 – SW Process Maturity,

Appraisal & ImpCopyright © 1995-2000, Dennis J. Frailey,

All Rights Reserved

Slide # 1

Chapter 13

Software Process Maturity, Appraisal and Improvement

Humphrey, chapters 3-4

NOTICE: This material is copyrighted and may be copied or downloaded ONCE ONLY by students who are registered in this course at Southern Methodist University or National Technological University.

January 20, 2000



All Rights Reserved

Slide # 2

Outline

13.1 Software Process Maturity13.2 The SEI Capability Maturity

Model13.3 Reviewing the Five Levels13.4 A Detailed Look at SEI Level 213.5 Process Appraisal Chapter Summary

January 20, 2000



All Rights Reserved

Slide # 3

13.1 - Software Process Maturity

Desirable Characteristics of a Software Process (*):

• Predictable– Cost and schedule commitments can be

made on the basis of reliable techniques, and then can be counted on

• Productive– Efficient relative to other processes

(*) These are also desirable characteristics of software

January 20, 2000



All Rights Reserved

Slide # 4

More Desirable Characteristics

• Effective– Requirements or expectations can be met, if

committed to– Quality meets accepted standards

• Improvable– Can increase predictability, productivity,

effectiveness in an orderly and reliable manner

January 20, 2000



All Rights Reserved

Slide # 5

How to Achieve These Characteristics?

• The principles of process management, such as statistical quality control, have been used to manage other processes to achieve these characteristics

• In concept, there seems to be no reason why the principles should not be applicable to software development– The techniques might vary – And the techniques may be less well established

• However there are some unique issues to be taken care of– Software really is different in many ways

January 20, 2000



All Rights Reserved

Slide # 6

Basic Steps of Statistical Quality Control

• Define measurements that tell you something about the process

• Measure what you are doing– Minimize disruption – If the measurement changes the process

too much, it is not telling you what you need to know

• Learn from the measurements• Apply what was learned to improve the

process– DO NOT USE TO PUNISH THE

PRACTITIONERS!

January 20, 2000



All Rights Reserved

Slide # 7

Measure the Right Thing

Watts Humphrey’s

Model of Why it is

Easy to Measure the Wrong Thing

WhatActually

Happened

What is Supposed

toHappen

What You Think

ActuallyHappened

January 20, 2000



All Rights Reserved

Slide # 8

Don’t Focus on the Practitioners

Example - Chicken RecipeDavid,

I will be watching you as you cook

to see how well you are doing and

help you improve.

David is more likely to: -- be more careful than usual (and therefore NOTbehave as usual) -- be nervous and thereforepossibly error prone

NowI’m

ReallyNervous

!

January 20, 2000



All Rights Reserved

Slide # 9

Don’t Punish or Blame the Practitioners

David, this disaster

isall your fault!

I’ll Show You!

David is likely to be: -- defensive -- uncooperative

January 20, 2000



All Rights Reserved

Slide # 10

Focus on the Process

David,the recipe is flawed

andwe need your help to

improve it.

I had a lot oftrouble with some

parts.And I think I know

somebetter spices.

David will helpto improve andwill become abetter cook aswell.

January 20, 2000



All Rights Reserved

Slide # 11

Frailey’s Maxim forLife in the Real World

You must succeed with normal, average people. Unlike the

university, you cannot reward the top students and flunk out

the rest.

January 20, 2000



All Rights Reserved

Slide # 12

The Normal Distribution Curve

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

Mean

+ 1 s- 1 s

+ 2 s

Individual Capability

Number of

People with This

Capability

January 20, 2000



All Rights Reserved

Slide # 13

The Process for Improving a Process (last step of SQC)

1) Understand the current status of the process

2) Develop a vision of the desired process3) Establish a (prioritized) list of required

process improvements4) Produce a plan to accomplish these

improvements5) Commit the resources to execute the

plan6) Carry out the plan7) Repeat with step 1

January 20, 2000



All Rights Reserved

Slide # 14

STUDENT CHALLENGE• Apply the process improvement process to

the chicken recipe several times. • Each time, DOCUMENT the improvements

and the results

Example:1) Current status: the chicken is overdone

and bland2) Vision: somewhat spicier, cooked just right3) a) don’t cook so long; b) try some other spices.

January 20, 2000



All Rights Reserved

Slide # 15

Example, continued

4) a) try turning after 15 minutes b) try adding sage to the sauce5) Buy some sage, buy the other

ingredients; plan to cook it again6) Cook it again and see how it goes Not crispy enough Spices still not right7) Repeat with a revised plan

January 20, 2000



All Rights Reserved

Slide # 16

Possible Exam Question

List the steps of the process improvement process; give an example of how this process might be applied to the software design process.

January 20, 2000



All Rights Reserved

Slide # 17

Other Pitfalls of Measurement

If you measure people without their knowing it– You may get accurate data until

they find out about it– Then you lose accuracy AND the

trust of the peopleA preferred approach is:– Educate them in the principles of

process improvement

January 20, 2000



All Rights Reserved

Slide # 18

Other Pitfalls of Measurement

A preferred approach is:– Work with them to develop effective

measures that they do not fear and will not “fudge”

– Collect and analyze data in a cooperative manner

– Use the data ONLY to evaluate the process

– Demonstrate use of the data• so people see it in action, as you told them

it would be used

January 20, 2000



All Rights Reserved

Slide # 19

More PitfallsYou can measure against goals orYou can measure to determine facts

about the process

If you measure against goals, your staff may change behavior and the process to improve the metric

But you want them to change the process to improve actual performance

Make sure you are asking for the right things.

January 20, 2000



All Rights Reserved

Slide # 20

Organizational Process Maturity

How Mature is Your Organization?This is Measured by How Effectively You Utilize

Processes• Philip Crosby first introduced the concept of

a five level maturity model• Crosby’s model was a measure of

Management Maturity • Crosby believed that this could be measured

by evaluating effective use of processes, based on the work of Edwards Deming

January 20, 2000



All Rights Reserved

Slide # 21

Philip Crosby’s Five Patterns of Management Attitude

1 Uncertainty -- Do not understand the task

2 Awakening -- Understand the problems, not

the solutions

3 Enlightenment -- Understand HOW to solve

known problems

4 Wisdom -- Understand WHY the solutions

work

5 Certainty -- Can solve unexpected problems

January 20, 2000



All Rights Reserved

Slide # 22

Humphrey’s Model of Software Organizational

Maturity (1)• Humphrey’s Model of Process Maturity is based on the work of Crosby and Deming

• Humphrey revised Crosby’s Model to form a model of Software Process Management Maturity

• Other authors have developed similar models but we will not discuss them here(1) See Humphrey for additional information. Chapters 1.2.1

through 1.2.5 cover each level in detail.

January 20, 2000



All Rights Reserved

Slide # 23

Humphrey’s Five Levels1) Initial - Ad-hoc and unmanaged; not

under control; depends on heros.2) Repeatable - Stable via rigorous

management and control; can be predicted provided you use the same people and the same kind of problem

3) Defined - Stable due to knowledge and definition of processes. Predictable even if entirely new people are used. Automation begins to pay off.

January 20, 2000



All Rights Reserved

Slide # 24

Humphrey’s Five Levels4) Managed - Comprehensive

measurement and analysis. Manage by fact.

5) Optimizing - Significant improvements on a regular basis in a controlled fashion.

January 20, 2000



All Rights Reserved

Slide # 25

Humphrey’s ModelFocus Key Process Areas ResultLevel

Productivityand Quality

RISK

4

Managed

3

Defined

2

Repeatable

1

Initial

5

Optimizing

ProjectManagement

EngineeringProcess

Product andProcess Quality

ContinuousProcessImprovement

Heroes

Process Meas/Anal

Quality Management

Process Focus&Def’n;

Integrated SW Mgmt;

Peer Reviews; Training;

Intergroup Coord; SW

Product Engineering

Project Mgmt; Planning;

Rqmts Mgmt; SQA;SCM; Subcont. Mgmt

Defect Prevention

Technology Innovation

Process Change Mgmt

January 20, 2000



All Rights Reserved

Slide # 26

What You Know at Each Level1 -- You Make Guesses and Rely on Heroism2 -- You Know What to Do3 -- You Know Why it Should be Done

(based on theory); -- You Know When NOT to do it

(exceptions that prove the rule)4 -- You Know Exactly Why and What (based

on factual data to supplement the theory)5 -- You Keep Up to Date; -- You Know What Changes are Coming

and How to Cope with Them

January 20, 2000



All Rights Reserved

Slide # 27

Rationale for the Five Levels• Historical experience (Crosby, et. al.)

with other processes• A reasonable sequence of measurable

and attainable improvement goals– You know what to do first

• Interim improvement plateaus– You don’t take on too much at a time

• They assist in setting priorities• Each level is the foundation for the next

CAVEAT: do one level at a time. Skipping levels risks ultimate failure.

January 20, 2000



All Rights Reserved

Slide # 28


Name and describe the essential characteristics of each level in Humphrey’s five-level scale of software process maturity.

Note: you must read the text to answer properly. The class notes only cover the highlights. This question is traditionally one of the ones I use to see if the student read the book.

January 20, 2000



All Rights Reserved

Slide # 29

Why Get to Level 5?

• Because it’s there• Because your competitor will do it• Because it will make you one of the

most productive and highest quality organizations

Ultimately, the only reason most companies strive to improve is

that they must do so to survive.

January 20, 2000



All Rights Reserved

Slide # 30

The Test of Commitment

Does the organization provide the necessary:– Resources– Management Support and

Reinforcement– Willingness to Change– Actual Change of the Culture of the

Organization

January 20, 2000



All Rights Reserved

Slide # 31

13.2 - The SEI Software Capability Maturity

Model (CMM)• This is an effort by the SEI to flesh

out the Humphrey model with best practices from the software development community

• There have been two releases to date:– Release 1.0 (1991) - Paulk91 (CMU/SEI-

91-TR-24), Weber91 (CMU/SEI-91-TR-25)– Release 1.1 (1993) - Paulk93a/b

(CMU/SEI-93-TR-24/25)

January 20, 2000



All Rights Reserved

Slide # 32

Release 2.0 of The SEI Software Capability Maturity

Model (CMM)• Release 2.0 was scheduled for 1997,

but was delayed pending integration with other CMMs (CMM Integration Project)– Significant changes in structure from

Release 1.0– Consistent with CMMs for other

disciplines– Too soon to tell whether it is an

improvement or not

January 20, 2000



All Rights Reserved

Slide # 33

Further Information on CMM• Information available from:

http://www.sei.cmu.edu/managing/managing.html

• Current release of CMM available from:

http://www.sei.cmu.edu/cmm/ cmm.html

January 20, 2000



All Rights Reserved

Slide # 34

Electronic Access to SEI

ftp.sei.cmu.edu

128.237.2.179

January 20, 2000



All Rights Reserved

Slide # 35

Why The SEI SW Capability Maturity Model

(CMM)• The idea is to characterize an

organization at each level in terms of goals, practices, etc.

• The model is used as the basis for appraisals and other evaluations (covered in a later section of this chapter)

January 20, 2000



All Rights Reserved

Slide # 36

Why Develop Such a Model

• To help organizations achieve a more capable software development organization

• The basic problem is that most organizations are immature in their software development practices, and this leads to expensive, unreliable software

• SEI was created to help industry extract itself from this condition

January 20, 2000



All Rights Reserved

Slide # 37

What are the Characteristics of an Immature Organization?• “In an immature organization, there is no

objective basis for judging product quality or for solving product or process problems.

• “Therefore, product quality is difficult to predict.

• “Activities intended to enhance quality such as reviews and testing are often curtailed or eliminated when projects fall behind schedule.” (1)

(1) Paulk, 1993a.

NOTE: much of what follows is derived from Paulk, 1993a and 1993b (SEI CMM 1.1)

January 20, 2000



All Rights Reserved

Slide # 38

What are the Characteristics of a Mature Organization?

• “... a mature software organization possesses an organization-wide ability for managing software development and maintenance processes.

• “The software process is accurately communicated to both existing staff and new employees, and work activities are carried out according to the planned process.

• “The processes mandated are fit for use (1) and consistent with the way the work actually gets done.

January 20, 2000



All Rights Reserved

Slide # 39

Mature Organization (continued)

• “These defined processes are updated when necessary, and improvements are developed through controlled pilot-tests and/or cost benefit analyses.

• “Roles and responsibilities within the defined process are clear throughout the project and across the organization.” (2)

(1) Humphrey, 1991 (2) Paulk, 1993.

January 20, 2000



All Rights Reserved

Slide # 40

Basic Terms

• The software process capability of an organization provides one means of predicting the most likely outcomes to be expected from the next software project the organization undertakes

• With a capable organization, you have higher confidence that the outcome will be as predicted

Software process capability describes the range of expected results that can be achieved by following a software

process.

January 20, 2000



All Rights Reserved

Slide # 41

Basic Terms

• Thus, software process performance focuses on the results achieved, while software process capability focuses on results expected.

• Recall that you need a good process model (capability) and good execution (performance) to have a good process

Software process performance represents the actual results achieved

by following a software process.

January 20, 2000



All Rights Reserved

Slide # 42

Achievement May or May Not Reflect Capability

• The capability of the project is constrained by the specific attributes of the project and the context or environment in which it is carried out.

• Capability is defined in a specific context, such as “building client-server tools in Cobol for the defense industry.”

January 20, 2000



All Rights Reserved

Slide # 43

Some Reasons Why Performance May Not Live Up

to Capability• Heavy use of new personnel• Radical changes in the application or

technology

• Either of these may place a project's staff on a learning curve that causes their project's capability and performance to fall short of the organization's full process capability.

January 20, 2000



All Rights Reserved

Slide # 44

Process Maturity

• Maturity implies a potential for growth in capability and indicates both the richness of an organization's software process and the consistency with which it is applied in projects throughout the organization.

Software process maturity is the extent to which a specific process is

explicitly defined, managed, measured, controlled, and effective.

January 20, 2000



All Rights Reserved

Slide # 45


Explain the Difference between Process Maturity, Process Capability, and Process Performance

January 20, 2000



All Rights Reserved

Slide # 46

Structure of the CMM

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

January 20, 2000



All Rights Reserved

Slide # 47

Structure (continued)• “Each level comprises a set of process

goals that, when satisfied, stabilize an important component of the software process.

• “Achieving each level of the maturity framework establishes a different component in the software process, resulting in an increase in the process capability of the organization.”

As one moves up the levels, one exhibits greater maturity and capability and one EXPECTS to achieve better performance

January 20, 2000



All Rights Reserved

Slide # 48

Effects of Higher Levels

At higher levels, one expects the following characteristics:

• Better visibility into what is happening

• Less variability in outcomes

• Less risk associated with software development, due to more accurate planning and better management

January 20, 2000



All Rights Reserved

Slide # 49

Better Visibility

Level 1Input Outp

ut

Level 2 Process Step



Input Output




January 20, 2000



All Rights Reserved

Slide # 50

Lower Variability / Less Risk

Level 1

Level 2

Level 3

etc.

January 20, 2000



All Rights Reserved

Slide # 51

What kind of Model is the CMM?

• The CMM is a descriptive model and a normative model

• A descriptive model acts as an example or a paradigm or an ideal– A model home– A fashion model

• A normative model acts as a way to compare two or more instances of something– A model of the human body– A model of an automobile engine

January 20, 2000



All Rights Reserved

Slide # 52

CMM as a Descriptive Model• The CMM describes essential (or key)

attributes that would be expected to characterize an organization at a particular maturity level.– Example: an organization at level 3 has

documented its process for configuration management

• One can use these attributes to evaluate one’s organization or to establish goals for an organization– Example: if we do not document our process for

CM, we are probably not at level 3– If we want to be at level 3, we should probably

document our process for CM

January 20, 2000



All Rights Reserved

Slide # 53

Uses of the ModelAppraisal teams will use the CMM to

identify strengths and weaknesses in the organization.– The SEI has defined a process for doing self-

assessments and other forms of appraisalsEvaluation teams will use the CMM to

identify the risks of selecting among different contractors for awarding business and to monitor contracts.– The SEI has defined a process for doing

capability evaluations– There are many less extensive ways the CMM

can be used to evaluate contractors

January 20, 2000



All Rights Reserved

Slide # 54

Uses (continued)Managers and technical staff will use the

CMM to understand the activities necessary to plan and implement a software process improvement program for their organization– The CMM is sometimes followed too rigidly,

but the goals tend to be universally applicableProcess improvement groups, such as an

SEPG [software engineering process group], will use the CMM as a guide to help them define and improve the software process in their organization.

January 20, 2000



All Rights Reserved

Slide # 55

The CMM as A Normative Model

• The detailed practices in the CMM characterize the normal types of behavior that would be expected in an organization doing large-scale projects in a government contracting context.

• A given organization can compare itself with the CMM to determine how it “stacks up” with what is considered an example of best practices.

January 20, 2000



All Rights Reserved

Slide # 56

Does the CMM Fit a Small Organization?

• There is considerable debate about the extent to which these practices should be expected in other kinds of organizations.

• Watts Humphrey (1) has shown that the principles, if not always the specific practices, are applicable to

individual, single-person projects (very small scale) that are

not in a government contracting mode(1) Humphrey, Watts, A Discipline of Software Engineering, Addison Wesley, 1994.

January 20, 2000



All Rights Reserved

Slide # 57

The Principles seem to be Universally Applicable

• The intent is that the CMM is at a sufficient level of abstraction that it does not unduly constrain how the software process is implemented by an organization

• Rather, it describes what the essential attributes of a software process would normally be expected to be

• One must keep this in mind when applying the CMM

January 20, 2000



All Rights Reserved

Slide # 58

Structure of the CMM

Maturity LevelsMaturity Levels

Key Process AreasKey Process Areas

Common FeaturesCommon Features

Key PracticesKey Practices

Goals

Implementation

Infrastructureor Activities

ProcessCapability

Address

Describe

Achieve

Contain

Organized by

Contain

Indicate

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

January 20, 2000



All Rights Reserved

Slide # 59

ExampleLevel 2:

RepeatableLevel 2:

Repeatable

SW Project PlanningSW Project Planning

Activities PerformedActivities Performed

DocumentedProcedure forSize Estimates

DocumentedProcedure forSize Estimates

Goal:

SW Estimatesare Documented

...

Implementation:

ImplementationActivities


ProcessCapability:

DisciplinedProcess

Address

Describe

Achieve

Contains

Organized by

Contains

Indicate

January 20, 2000



All Rights Reserved

Slide # 60

Key Process Areas• Except for Level 1, each level of the CMM is

defined in terms of a series of Key Process Areas or KPAs

• Key process areas indicate the areas an organization should focus on to improve its software process.

• Key process areas identify the issues that must be addressed to achieve a maturity level.– Level 2 KPAs are what you focus on when you are at

level 1– Level 3 KPAs are what you focus on when you are at

level 2– And so forth.


January 20, 2000



All Rights Reserved

Slide # 61

Structure of KPAsEach key process area identifies a

cluster of related activities that, when performed collectively,

achieve a set of goals considered important for enhancing process capability.

– Note that KPAs are artificially associated with specific levels in the CMM

– In actual fact, each KPA evolves as an organization moves up the maturity scale


January 20, 2000



All Rights Reserved

Slide # 62

KPA DefinitionEach KPA is defined in terms of the

following:

• Goals - why you do it - what you hope to achieve– The goals signify the scope, boundaries, and

intent of each key process area.


January 20, 2000



All Rights Reserved

Slide # 63

Five Common Features• Certain features or attributes are

common to each KPA• These attributes indicate whether the

implementation and institutionalization of a key process area are effective, repeatable, and lasting.

• The five common features are:– Commitment to Perform– Ability to Perform– Activities Performed– Measurement and Analysis– Verifying Implementation


January 20, 2000



All Rights Reserved

Slide # 64

Key Practices

• Key Practices - what you (typically) do to achieve the goals– Infrastructure elements, such as policies or

practices or resources– Activities, such as reviews or processes


January 20, 2000



All Rights Reserved

Slide # 65

Commitment to Perform• Commitment to Perform describes

the actions the organization must take to ensure that the process is established and will endure– Example: establishing a policy that

mandates a particular activity

We WILL do

this!


Commitment to Perform typically involves

establishing organizational policies

and demonstrating senior management sponsorship.

January 20, 2000



All Rights Reserved

Slide # 66

Ability to Perform• Ability to Perform describes the

preconditions that must exist in the project or organization to implement the software process competently


Boss

SEPGSW

Manager

Ability to Perform typically involves resources, organizational structures, and

training

January 20, 2000



All Rights Reserved

Slide # 67

Activities Performed

• Activities Performed describes the roles and procedures necessary to implement a key process area


Activities Performed typically involve: -- Establishing plans and procedures, -- Performing the work, -- Tracking it, and -- Taking corrective actions as necessary

January 20, 2000



All Rights Reserved

Slide # 68

Measurement and Analysis

• Measurement and Analysis describes the need to measure the process and analyze the measurements


Measurement and Analysis typically includes examples of the measurements that could be taken to determine the status and effectiveness of the activities performed.

January 20, 2000



All Rights Reserved

Slide # 69

Verifying Implementation

• Verifying Implementation describes the steps to ensure that the activities are performed in compliance with the process that has been established


Verification typically encompasses reviews

and audits by management and software quality

assurance.

January 20, 2000



All Rights Reserved

Slide # 70

Key Practices

• The key practices describe "what" is to be done

-- Example: a key practice for risk management might be periodic evaluation of risks and the status of known risk areas

• But they should not be interpreted as mandating "how" the goals should be achieved


January 20, 2000



All Rights Reserved

Slide # 71

Alternative Practices• Alternative practices may

accomplish the goals of the key process area

• The key practices should be interpreted rationally to judge whether the goals of the key process area are effectively, although perhaps differently, achieved.


Key Practice per CMM Example

Key Practice for Our Situation

January 20, 2000



All Rights Reserved

Slide # 72

13.3: Reviewing the Five Levels

• The five levels correspond to Humphrey’s five levels (see text)

• But the CMM provides additional detail and many “best practices”

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

January 20, 2000



All Rights Reserved

Slide # 73

Level 1- Initial• The software process is characterized as

ad hoc, and occasionally even chaotic. • Few processes are defined, and success

depends on individual effort (heroes).• Unstable environment is subject to

catastrophe if key people leave– “Bus sensitive projects”

• Planning is ineffective -- reaction is the order of the day

• Plans are routinely ignored or abandoned

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

“You can be very successful, but you cannot count on it.”

January 20, 2000



All Rights Reserved

Slide # 74

Level 2 - Repeatable• Basic project management processes

are established to track cost, schedule, and functionality.

• The necessary process discipline is in place to repeat earlier successes on projects with similar applications.

• Stability is present, even when heroes leave, so long as the overall environment is consistent

• Everyone knows how to do it, although they do not always know why

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

“You can expect consistent behavior.”

January 20, 2000



All Rights Reserved

Slide # 75

Key Process Areas for Level 2

• Configuration Management• Software Project Planning• Quality Assurance• Requirements Management• Subcontractor Management (*)• Project Tracking and Oversight (**)

(*) Acquisition Management in V 2.0(**) Project Control in V 2.0

January 20, 2000



All Rights Reserved

Slide # 76

Level 3 - Defined• The software process for both

management and engineering activities is documented, standardized, and integrated into a standard software process for the organization.

• All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software.

• People know why, not just how to do the job

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

“You can expect stability in a changing environment.”

January 20, 2000



All Rights Reserved

Slide # 77

Key Process Areas for Level 3• Organizational Process Focus• Organization Process Definition• Organizational Training Program• Integrated Software Management– All activities are linked together

• Intergroup Coordination (*)• Software Product Engineering– Good software engineering techniques;

effectively used

• Peer Reviews– Defect removal at all steps of the

process

(*) Product Interface

Coordination in V 2.0

January 20, 2000



All Rights Reserved

Slide # 78

Level 4 - Managed• Detailed measures of the software process

and product quality are collected. • Both the software process and products are

quantitatively understood and controlled.• Decisions are based on fact• Process behavior is quantified• Processes are instrumented to facilitate

data collection• Measurements are applied across the

organization so that norms and exceptions can be identified

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

January 20, 2000



All Rights Reserved

Slide # 79

Key Process Areas for Level 4• Quantitative Process Management (*) -- Metrics are defined, collected, analyzed

and acted upon• Software Quality Management (**) -- Quality is managed, not just controlled(*) Statistical Process Management in V 2.0(**) Organization Process Performance in V

2.0

• Organization Software Asset Commonality (new KPA for V 2.0)

January 20, 2000



All Rights Reserved

Slide # 80

Level 5 - Optimizing

• Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

January 20, 2000



All Rights Reserved

Slide # 81

Key Process Areas for Level 5• Defect Prevention– You not only find defects, you correct or

remove the process steps that cause them

• Technology Change Management (*)– You insert technology in an orderly and

managed fashion– New methods and tools do not disrupt

performance

… continued

January 20, 2000



All Rights Reserved

Slide # 82

Key Process Areas for Level 5

• Process Change Management (*)– Not only do you change the process, but

you do so in a managed way, so the outcome is predictable within narrow limits

(*) Merged into Organization Process and Technology Innovation in V 2.0

• Organization Improvement Deployment (V 2.0)

January 20, 2000



All Rights Reserved

Slide # 83

Types of Changes

0

10

20

30

40

50

60

70

80

Time ----->

Capability

Do Nothing to Improve

January 20, 2000



All Rights Reserved

Slide # 84

Types of Changes

Time ----->

Capability

0

10

20

30

40

50

60

70

80

Do NothingContinuousImprovement

January 20, 2000



All Rights Reserved

Slide # 85

Types of Changes

Time ----->

Capability

0

10

20

30

40

50

60

70

80 Do Nothing

ContinuousImprovement

Reengineering

January 20, 2000



All Rights Reserved

Slide # 86

13.4 - A Detailed Look at SEI Level 2 (CMM 1.1) (see note)

• Configuration Management• Project Planning• Quality Assurance• Requirements Management• Subcontractor Management• Project Tracking and Oversight

Level 5Maturity

Level 4 Maturity

Level 3 Maturity

Level 2 Maturity

Level 1 Maturity

Note: V2.0 document shows differences from V1.1 via notations in margin.

January 20, 2000



All Rights Reserved

Slide # 87

Each KPA of the CMM has the Following Elements

(Ci) - Commitments that establish the organization’s willingness to do the activity

(Abi) - Abilities that demonstrate the organizations capability to do the activity

(Aci) - Activities that demonstrate the organization is doing it

(Mi) - Monitors that measure the effectiveness of the activity

(Vi) - Verifications that make sure it is being done and being done right

Some of these have changed from CMM 1.0 to CMM 1.1

January 20, 2000



All Rights Reserved

Slide # 88

Format of Following Viewgraphs

• Review of Typical Symptoms that indicate a problem in this KPA

• Discussion of the Purpose and Typical Practices

• Discussion of the Goals• Discussion of individual Activities, Abilities,

Commitments, etc. - organized in the form of specific policies, practices, etc.

• Each item may be followed by a code, such as (Ab3), indicating that the item is called for by “Ability 3” of the CMM for this KPA

January 20, 2000



All Rights Reserved

Slide # 89

13.4.1 -- Configuration Management

•We don’t know which version of module AB17 goes with release 2.1 of the product.

•The user reference manual does not correspond to the actual software.

TYPICAL SYMPTOMS

•We fixed that bug -- but it came back!

– Why did it return?– How can we get rid of it for good?

•The source code of the released product will not compile without errors.

– Or, perhaps -- when we compile it the resulting program doesn’t work.

January 20, 2000



All Rights Reserved

Slide # 90

Configuration ManagementPurpose

• To maintain integrity of the product I.e., all the parts fit together and we know

which parts go with each other

January 20, 2000



All Rights Reserved

Slide # 91

Configuration ManagementPractices (typical)

• Unique identifier for each component of the product

• Unique identifier for each version of the product

• All changes are documented and controlled– system integrity is preserved

• At any time we know the status of all parts

January 20, 2000



All Rights Reserved

Slide # 92

Configuration ManagementGoals I

1. Software CM activities are planned– Planning occurs at project

commencement– Plans are documented and

followed

January 20, 2000



All Rights Reserved

Slide # 93

Configuration ManagementGoals II

2. Selected software work products are identified, controlled, and available– Includes documentation,

specifications, code, tests, etc.• Anything that might have multiple

versions and users– Does not necessarily include all

work products• For example, test reports or code not

yet tested– Not necessarily a highly formal

control procedure

January 20, 2000



All Rights Reserved

Slide # 94

Configuration Management

Goals III

3. Changes to identified software work products are controlled

4. Affected groups and individuals are informed of the status and content of software baselines

January 20, 2000



All Rights Reserved

Slide # 95

Configuration ManagementAbilities I

• A Software Configuration Control Board (SCCB) is established to manage software baselines– All changes must have SCCB approval

• A person or group is responsible for coordinating and implementing configuration management– Could be a software engineer on a small

project– Usually a separate group on a large project

January 20, 2000



All Rights Reserved

Slide # 96

Configuration ManagementAbilities II

• Adequate resources and funding are provided

• Training in CM duties is provided (for all affected individuals)

January 20, 2000



All Rights Reserved

Slide # 97

Sample Behaviors• Program manager:– Reviews and approves CM plan

• Project lead or software manager:– Documents CM plans– Typically chairs the software configuration control board.

• Software engineers:– Informal control of software prior to release to the formal

CM process– Release to formal CM after unit test

• Configuration manager:– Establish a workable CM system and plan– Select the right tools <<< HARD TO DO <<<

January 20, 2000



All Rights Reserved

Slide # 98

Policies and Procedures

Organizational Policy: – SCM Implementation (C1)

Documented Procedures:– For preparation of an SCM Plan (Ac 1)– For handling change requests & problem reports (Ac 5)– For changes to baselines (Ac 6)– For creation & release of products to baselines (Ac 7)– For recording status of units and configuration items

(Ac 8)– For conducting baseline audits (Ac10)

Policies&

Procedures

Code: Ci - Commitment #i in CMM Description Mi - Measurement #i Ac i - Activity #i Vi - Verification #i Ab i - Ability #i

January 20, 2000



All Rights Reserved

Slide # 99

Standards:– Criteria for selecting configuration items and units

(Ac 4) – Standard reports on SCM Activities (Ac 9)

Documents:– SCM Plan (Ac 1)

Training:– SCM Group - standards, tools, procedures, etc. (Ab 4)

– SW Engineers and Others - SCM procedures, etc that they must follow (Ab 5)

Standards, Documents & Training

SCMPlanfor

ProjectXXX

January 20, 2000



All Rights Reserved

Slide # 100

InfrastructureEstablish:

– Responsibility for software configuration management (Ab2)

– A software configuration control board (SCCB)

(Ab1)

– A software configuration management library (Ab3)

January 20, 2000



All Rights Reserved

Slide # 101

Special Issues• CM Tools are often a major problem– Commercially available tools tend to be immature.– CM tools must integrate well with other tools such as

compilers, editors, etc. – Tool evaluation and selection takes knowledge and time.

Look for the features you will need the most:• Control of source code, documents, object code, tests, etc.• Reuse support• Automatic recompile of all code affected by a change• Problem reporting support (e.g., metrics and status)

• Delays from poor CM can endanger a project schedule– Good planning and execution are required.

January 20, 2000



All Rights Reserved

Slide # 102

13.4.2 - Software Project Planning

TYPICAL SYMPTOMS• The project is way behind schedule• The project is way over budget• All the key people have quit the project• The new relational data base did not work for our

application -- we have to recode the whole thing• We’re ready for testing but there is no test

system available• The compiler has bugs and we spend all our time

getting around them• . . .

January 20, 2000



All Rights Reserved

Slide # 103

Software Project Planning• Purpose:

To establish reasonable plans for performing and managing the project

• Practices:– Plans are written and followed– Size and effort are estimated using

consistent methods– Commitments are established and

documented– The job to be done is clearly defined– Risks are assessed and managed– Schedules and resources are determined

January 20, 2000



All Rights Reserved

Slide # 104

Software Project Planning -- Goals

1.Software estimates are documented– For use in planning and tracking

2.Activities and commitments are planned and documented– Everyone works to the same plan

3.Affected groups and individuals agree to their commitments– They must know what they are expected

to do and believe they can do it

January 20, 2000



All Rights Reserved

Slide # 105

Software Project Planning - Abilities

• A statement of work is documented and approved– Lists items to be delivered and specific

tasks– Defines scope, goals, customers, end

users– Specifies standards, responsibilities,

dependencies– May define cost goals, schedule,

resource constraints

January 20, 2000



All Rights Reserved

Slide # 106

Software Project Planning - Abilities

• Responsibilities for software development are assigned (for a team, this means roles)


• Planners are trained in estimating and planning procedures

January 20, 2000



All Rights Reserved

Slide # 107

Sample Behaviors• Program manager:– Assigns a software manager or

planner before key decisions and commitments are made

– Reviews commitments with the software manager

– Leads organization to alignment on commitments

… continued

January 20, 2000



All Rights Reserved

Slide # 108

Sample Behaviors (continued)

• Project lead or software manager:– Participates in project planning – Develops software estimates– Completes and documents software plans

• Software engineers:– Contribute to planning and

estimating tasks

January 20, 2000



All Rights Reserved

Slide # 109

Policies, Procedures & StandardsOrganizational Policy:

For planning a software project (C2)

Documented Procedures:For deriving estimates of size, effort, cost, critical

computer resources, and schedule (Ac 9-12)

For developing a project software development plan (Ac 6)

For review of project commitments by senior management (Ac 4)

Standards:Those applicable to project should be identified,

documented and made available to all affected participants

Policies&

Procedures

January 20, 2000



All Rights Reserved

Slide # 110

Documents, Training &Infrastructure

Documents:Statement of work for the software project (Ab 1) Software development plan (Ac 7)

Software risk management plans (Ac 13) Plans for software engineering facilities and support

tools (Ac 14)

Training:Software planners -- trained on estimating &

planning procedures (Ab 4)

Infrastructure:Establish a designated software manager (C 1) Identify a software lifecycle (Ac 5)

January 20, 2000



All Rights Reserved

Slide # 111

Special Issues I

• It is hard to do too much planning - but projects usually do too little– Data from real projects show that the

most expensive problems result from mistakes made very early

January 20, 2000



All Rights Reserved

Slide # 112

Special Issues II• Planning often occurs in a stressful

environment (Proposals, short schedules, limited

resources, etc.)– It is very easy to overlook important

issues– Pay attention to interfaces with other

disciplines• Beware of inconsistent system concepts that

lead to incompatible interfaces!

– Define a process for planning and a set of planning tools and procedures

January 20, 2000



All Rights Reserved

Slide # 113

13.4.3 - Software Quality Assurance

TYPICAL SYMPTOMS• Management was surprised that the

product did not work correctly after release.– No one mentioned that it was not really

ready.

• The customers complain that the software does not work as advertised.– But we tested it!

January 20, 2000



All Rights Reserved

Slide # 114

Software Quality Assurance

MORE TYPICAL SYMPTOMS• We have a procedure to

prevent that!– Why did it happen?– Why wasn’t the procedure

followed?

• That project violated company policy.– Did they know about the policy?– Who authorized them?

January 20, 2000



All Rights Reserved

Slide # 115

Software Quality AssurancePurpose

To provide management with visibility into the process being followed and the products being built.

To provide a quality control mechanism.

January 20, 2000



All Rights Reserved

Slide # 116

Software Quality AssurancePractices (typical)

– Reviews– Audits– Communication– Measurements– Independent confirmation of

compliance• Standards, requirements, procedures

January 20, 2000



All Rights Reserved

Slide # 117

Software Quality AssuranceGoals I

1.Software quality assurance activities are planned– Planning begins at project

commencement– Plans are documented and followed

January 20, 2000



All Rights Reserved

Slide # 118

Software Quality AssuranceGoals II

2.Adherence of software products and activities to applicable standards, procedures and requirements is verified objectively– Independent reporting chain for

verifiers– Quality assurance helps keep you

honest -- they are part of the team, not the enemy

January 20, 2000



All Rights Reserved

Slide # 119

Software Quality AssuranceGoals III3.Affected groups and individuals are informed

of software quality assurance activities and results– Knowing the facts helps everyone do the

best job

4.Noncompliance issues that cannot be resolved within the project are addressed by senior management– Senior management is responsible for

compliance to policies, commitments, etc.

January 20, 2000



All Rights Reserved

Slide # 120

Software Quality AssuranceAbilities• A group exists that is responsible for coordinating

and implementing SQA for the project– Could be a part-time job on a small project– Usually several full-time people on a large project– A key characteristic is an independent reporting

chain - but this can be relaxed for more mature organizations

• Adequate resources and funding are provided• Training is provided in SQA activities (for all affected

individuals)

January 20, 2000



All Rights Reserved

Slide # 121

Sample Behaviors - I• Senior management:– Provides an organizational

structure with an independent reporting channel for SQA personnel

– Periodically reviews SQA activities and results

• Program manager:– Reviews and approves SQA plan– Periodically reviews SQA activities

and results

January 20, 2000



All Rights Reserved

Slide # 122

Sample Behaviors - II• Project lead or software manager:– Participates with SQA staff in

developing SQA plan• Software engineers:– Cooperate with SQA staff

January 20, 2000



All Rights Reserved

Slide # 123

Sample Behaviors - III

• SQA staff:– Prepares SQA plan– Provides consultation and review in

preparation of software development plans and procedures

– Evaluates plans, processes and products– Conducts audits and reviews to verify

compliance to plans, standards and procedures

January 20, 2000



All Rights Reserved

Slide # 124

Sample Behaviors - IV

• SQA staff:– Documents and tracks

noncompliance issues– Verifies that corrections are made – Periodically reports results of

activities to software engineering group, management and (sometimes) customer

January 20, 2000



All Rights Reserved

Slide # 125

Policies, Procedures & Standards

Organizational Policy:For Implementing SQA (C1)

Documented Procedures:For preparing an SQA plan (Ac 1) For documenting and handling deviations in

software activities and work products (Ac 7)

Standards:None specific to SQA, but all standards

applicable to the project must be identified and followed

Policies&

Procedures

January 20, 2000



All Rights Reserved

Slide # 126


Documents:SQA Plan (Ac 1)

Training:SQA Group - procedures, activities, etc. that they

are required to do (Ab 3) SW Engineers and Others - SQA responsibilities (Ab

4)

Infrastructure:Establish an SQA Group or responsible party (Ab 1)

January 20, 2000



All Rights Reserved

Slide # 127

Special Issues - ISQA can degenerate into box checking

and “nit picking”– Quality is not perfection! “Fitness

for intended use.”– SQA personnel must understand the

goals, processes and procedures in order to apply proper judgment.

– This works best if they are part of the team rather than totally independent

January 20, 2000



All Rights Reserved

Slide # 128

Special Issues - II

Software engineers may lack respect for SQA staff

– Nobody likes to hear about problems.

– Some organizations use the least experienced staff for SQA positions• Compare with the model used in the

best Japanese companies (see next slide)

January 20, 2000



All Rights Reserved

Slide # 129

Pride in Professionalism

If the SW group takes pride in the quality of its work, SQA can be seen as a

valuable aide in making the product right the first time.

– In Japan’s best companies, SQA is typically staffed by the most senior software engineers -- experienced and highly respected staff!

– Athletes respect the coaches who measure them because they help them get better

January 20, 2000



All Rights Reserved

Slide # 130

13.4.4 - Requirements Management

TYPICAL SYMPTOMS• The product is perfect, but the customer doesn’t

like it– Often because the developers “know what’s best for

the customer”

• Finger pointing – Multiple versions of the requirements– Multiple interpretations of the requirements

• The system will not integrate– The parts do not fit together– Some functions are duplicated in software &

hardware– Some functions are missing entirely

January 20, 2000



All Rights Reserved

Slide # 131

Requirements Management• Purpose:

To establish a common understanding between the customer and the software project

• Practices:– Reaching agreement on the requirements– Documenting the requirements– Controlling changes to the requirements– Communicating changes to the requirements– Allocating requirements - to software, hardware,

etc. - in a clear, unambiguous manner

January 20, 2000



All Rights Reserved

Slide # 132

Requirements Management

Goals I1.System requirements allocated to

software are controlled to establish a baseline– Requirements may come directly from

the customer on a software-only project– Requirements may come from a system

engineering function on a hardware & software project

– “Controlled” means you don’t change without approval and communication

January 20, 2000



All Rights Reserved

Slide # 133

Requirements Management

Goals II2.Software plans, products and

activities are kept consistent with the requirements– If you change the requirements, you must

consider changing the plans, etc.

January 20, 2000



All Rights Reserved

Slide # 134

Requirements ManagementAbilities I

• Responsibility is established for analyzing the system requirements and allocating them to hardware, software and other system components– This is a prerequisite to software

engineering– Includes responsibility for managing

changes

• Allocated requirements are documented– Includes technical and non-technical

requirements

January 20, 2000



All Rights Reserved

Slide # 135

Requirements ManagementAbilities II


• Members of software engineering group (and other related groups) are trained in requirements management activities

January 20, 2000



All Rights Reserved

Slide # 136

Sample Behaviors• System engineer or project lead:– Allocates requirements to all parts of

the system• Don’t forget support, installation,

documentation, etc.• Include platform selection, network

configuration, and other tasks that are not part of software development

– Controls and communicates changes• May use a system configuration control

board– Listens to understand the impact of

changes

January 20, 2000



All Rights Reserved

Slide # 137


• Software manager:– Participates in system configuration

control process– Determines the impact of changes on

software cost and schedule– Communicates the impact to affected

parties (system engineers, customers, program managers, etc.)

January 20, 2000



All Rights Reserved

Slide # 138


• Software engineers:– Review requirements (and changes) – Communicate the cost/impact of

requirements/changes– For example, which modules need to

be redesigned, recoded, retested, etc.

– (This works best if the software engineer can document the estimated cost and schedule impact of each change.)

January 20, 2000



All Rights Reserved

Slide # 139

Policies, Procedures & StandardsOrganizational Policy:

For managing system requirements allocated to software (C1)

Documented Procedures:No specific procedures are designated by the CMM, but

you may want to consider the following, especially for projects involving both hardware and software:

-- A documented procedure for allocating, reviewing and controlling system and software requirements

-- A documented procedure for changing system and sw requirements

Standards:No specific standards are designated by the CMM

January 20, 2000



All Rights Reserved

Slide # 140

Documents

This should be done in a SINGLE ARTIFACT such as a system design document -- NOT in a series of independent documents (sw requirements; hw requirements; support

requirements; etc.)

Allocation of requirements is documented (Ab 2)

January 20, 2000



All Rights Reserved

Slide # 141

Training &Infrastructure

Training:Software engineers, etc -- requirements

management procedures, documents, etc. (Ab 4)

Infrastructure:Establish responsibility for system

requirements analysis, allocation to software, and control (Ab 1)

January 20, 2000



All Rights Reserved

Slide # 142

Special Issues - I• The basic responsibility for requirements management

may lie outside of the software engineering function– Software may be only part of a larger system

product– Many large projects use a system engineer

responsible for design, control and integration of the whole system

Subsystem(segment)

Subsystem(segment)

Subsystem(segment)

Product or Item(configuration

item)


item)


item)

System

Subsystem(segment)

Requirements Management Responsibility

Requirements Implementation Responsibility

}

}

January 20, 2000



All Rights Reserved

Slide # 143

Special Issues - II

• The symptoms of poor requirements management tend to show up toward the end of the project -- when it costs the most to fix the problems– Often the fixes require extensive

software changes, but the “fault” may lie outside the software function

January 20, 2000



All Rights Reserved

Slide # 144

Special Issues - III• The cost of proper requirements

management comes at the front end of the project

Integrate& Test

Design theArchitecture& Allocate

Requirementsto

Parts

Buildthe

System

AnalyzeRequirements

• The penalty for improper requirements management comes at the back end

January 20, 2000



All Rights Reserved

Slide # 145

13.4.5 - Software Subcontract Management

TYPICAL SYMPTOMS

They were supposed to deliver it today. They just called and told us

they have a three month delay!

They always delivered on time in the

past!

January 20, 2000



All Rights Reserved

Slide # 146

More Typical Symptoms

We’ve been hit by a $25,000 fine because

our user interface does not meet EPA

standards.

The subcontractor did not

know about the

regulations.

The big cost will be the

recall - about

$200,000

January 20, 2000



All Rights Reserved

Slide # 147

Further Typical Symptoms

They delivered the software

module, but no test cases and

no documentation

The contract was vague

on this point.

They thought that was our responsibilit

y.

January 20, 2000



All Rights Reserved

Slide # 148

Software Subcontract Management

• Purpose:– To select qualified subcontractors– To manage subcontractors effectively

• Practices:– Establish policies for software subcontract

management– Select subcontractors based on ability to do the job– Communicate effectively with subcontractors– Document commitments– Flow down standards, processes, etc.– Track and review subcontractor performance and

results.

January 20, 2000



All Rights Reserved

Slide # 149

Subcontract Management -- Goals

1. Select qualified subcontractors– Include software management in

selection process

2. Agree to commitments– Get them in writing

3. Maintain ongoing communication

4. Track performance against commitments– Reviews, audits, etc.

January 20, 2000



All Rights Reserved

Slide # 150

Subcontract Management -- Abilities

• Adequate resources and funding for selection and management of subcontractors

• Training is provided in subcontract management for all individuals involved

• Orientation in technical aspects of project for subcontract management staff

January 20, 2000



All Rights Reserved

Slide # 151

Sample Behaviors• Program manager:– Includes software manager in subcontractor

selection process– Establishes policies and procedures for managing

subcontracts

• Software manager and subcontract manager:– Track subcontractor schedule, effort, size, risks,

etc. – Take corrective action when there are significant

deviations from plan– Hold periodic reviews of subcontract project status

January 20, 2000



All Rights Reserved

Slide # 152


Organizational Policy: – For managing software subcontracts (C1)

Documented Procedures:– For defining and planning work to be

subcontracted (Ac 1)

– For for selecting subcontractors based on ability (Ac 2)

– For resolving changes to subcontractor’s statement of work, terms and conditions, and other commitments (Ac 6)

January 20, 2000



All Rights Reserved

Slide # 153


Documented Procedures:– For formal reviews of subcontractor’s

software engineering accomplishments (Ac 9)

– For SQA monitoring of subcontractor’s SQA activities (Ac 10)

– For SQA monitoring of subcontractor’s SCM activities (Ac 11)

– For acceptance test of subcontractor products (Ac 12)

January 20, 2000



All Rights Reserved

Slide # 154

Standards and Documents

Standards:– For subcontractors are established (derived

from project standards) (Ac 1, 2)

Documents:– Plan for selecting subcontractors (Ac 1)

– Statement of work for each subcontractor (Ac 1)

– Contractual agreement (Ac 3)

– Subcontractor’s software development plan (Ac 4)

January 20, 2000



All Rights Reserved

Slide # 155

Training:– Software manager and subcontract

managers - how to perform required subcontract management activities (Ab 2)

– Software subcontract managers - technical aspects of the project (Ab 3)

Infrastructure:– Establish a subcontract manager (C2)

Training and Infrastructure

January 20, 2000



All Rights Reserved

Slide # 156

Special Issues

• Subcontractors may not like to be reviewed and audited – Diplomacy, tact, and firm management skills

are needed– A cooperative approach based on mutual

goals for quality can lead to more openness and more accurate information

– Standards and regulations can provide leverage in negotiations

“We trust you, but ISO 9000 requires that we do an audit”

January 20, 2000



All Rights Reserved

Slide # 157

Special Issues

• Other parts of your own company are subcontractors too!– They may not require all the formality, but

the principles are still the same:• Statement of work, commitment, tracking, etc.

January 20, 2000



All Rights Reserved

Slide # 158

13.4.6 - Project Tracking and Oversight

• They’ve been working on that module for eight weeks and everyone else is waiting for it.– Did the developers in charge know how

many people depend on the module?

• We’re six months behind schedule and nobody realized it!–Why did it take so long to find out?

January 20, 2000



All Rights Reserved

Slide # 159

13.4.6 - Project Tracking and Oversight

• The project manager promised a new feature to the customer -- but never told any of the programmers!– “I thought you knew about this!”

• The software takes up too much disk space.– Nobody ever thought it would get that big.

January 20, 2000



All Rights Reserved

Slide # 160

Project Tracking and OversightPurpose

To provide adequate visibility into actual progress so that management can take effective actions when the software project’s performance deviates significantly from the software plans

January 20, 2000



All Rights Reserved

Slide # 161

Project Tracking and Oversight

Practices (typical)• Establish policies for software project management

• Use a software development plan for tracking and communicating status

• Track schedule, size, effort, computer resources, technical activities, risks

• Hold periodic reviews and take corrective action

• Revise plans and schedules to reflect changes -- using a defined procedure

• Review customer commitments on a regular basis

January 20, 2000



All Rights Reserved

Slide # 162

Tracking and Oversight -- Goals

1. Actual results and performance are tracked against software plans– Plans are revised to reflect actual performance

and changes in requirements or commitments

2. Corrective actions are taken and managed to closure when actual performance deviates significantly from software plans

3. Changes to software commitments are communicated to all affected groups and individuals

January 20, 2000



All Rights Reserved

Slide # 163

Tracking and Oversight -- Abilities

• A software development plan is documented and approved

• The software manager assigns explicit responsibility for all work products and activities

• Adequate resources and funding are provided for project tracking

• Training is provided in management of technical and personnel aspects of the project (for all affected managers/leads)

January 20, 2000



All Rights Reserved

Slide # 164

Sample Behaviors• Program manager:– Reviews and approves the software

development plan– Communicates changes to commitments– Attends formal software reviews

• Software managers and leads:– Track schedule, effort, size, risks, etc. – Take corrective action when there are

significant deviations from plan– Hold periodic reviews of project status

• Software engineers:– Cooperate with tracking and review activities

January 20, 2000



All Rights Reserved

Slide # 165

Policies, Procedures & Standards

Organizational Policy: For managing software projects (C2)

Documented Procedures:For revising the software development plan (Ac

2)

For review of commitments and changes to commitments with senior management (Ac 3)

For conducting formal reviews of the software project (Ac 13)

Standards:No specific standards are called out by the CMM

January 20, 2000



All Rights Reserved

Slide # 166

Documents:Software Development Plan (Ab 1)

Training:Software manager - management of technical

and personnel aspects of the project (Ab 4)

Software leads and first-line managers - technical aspects of the project (Ab 5)

Infrastructure:Establish a project software manager (C1)

Establish periodic internal reviews (Ac 12)


January 20, 2000



All Rights Reserved

Slide # 167

Special Issues• Selection of metrics is very important– It takes effort to collect, analyze and respond to

metrics– The recommended approach is to focus on a small

number of useful metrics• Track the things that relate to significant risks, such as

size, effort, schedule

• Metrics are often seen as intrusive by individual software developers– Get them involved in definition of metrics– Do not use metrics to find fault with individuals

• Generally, the fault is with the process, the procedures, the tools, or the estimates

January 20, 2000



All Rights Reserved

Slide # 168

Possible Exam Question Relate the various elements of the CMM to topics

discussed in this course.Note which topics are and are not coveredDo a cross reference between the course and the

CMM

January 20, 2000



All Rights Reserved

Slide # 169

13.5: Process Appraisal“If you don’t know where you are, a map

won’t help” (1)

An appraisal is a way to determine where you are

(1) Humphrey (textbook)

January 20, 2000



All Rights Reserved

Slide # 170

Why Do Appraisals?

Most organizations suffer because they cannot see their problems, not because they cannot solve them.

Most organizations are focused on defining solutions, not on defining problems.

Most organizations tend to solve symptoms, not the fundamental, underlying problems.

Why not Just Solve the Problems?

January 20, 2000



All Rights Reserved

Slide # 171

Responding to Symptoms

Problem

Symptom

Symptom

Symptom

Response LongerTerm Cure

Response

TemporaryCure

January 20, 2000



All Rights Reserved

Slide # 172

Types of “Appraisals”• Reviews

• Audits

• Capability Evaluations

• Self Assessments

• Other Appraisals

January 20, 2000



All Rights Reserved

Slide # 173


Compare and contrast the different forms of appraisal with respect to:–Cost–Length–Who Performs–Objectives–Advantages–Drawbacks–etc.

January 20, 2000



All Rights Reserved

Slide # 174

ReviewsPurpose:

To learn the status of the project

Performed by:

Managers

Method: Practitioners report on

the status and plans of their projects, following specified formats and reporting on specific metrics

January 20, 2000



All Rights Reserved

Slide # 175

Reviews (continued)Typical Duration: A few hours to several daysAchieves: Uncovers problems (or, at least, symptoms)Drawbacks: Does not identify solutions May motivate hiding of problemsAdvantages: Relatively inexpensive Tends to get everyone back on the same

track

January 20, 2000



All Rights Reserved

Slide # 176

AuditsPurpose: To study a project in detail and find problems. Also to keep things on trackPerformed by: Independent technical experts, often outsidersMethod: Experts question practitioners and examine artifacts

of their process to determine what is happeningTypical Duration: Several days to several weeks

January 20, 2000



All Rights Reserved

Slide # 177

Audits (continued)

Achieves:

Uncovers real problems more often

Informs staff that management cares about the results

Advantages:

Tends to uncover real problems

Tends to confirm or disprove suspected problems

January 20, 2000



All Rights Reserved

Slide # 178

Audits (continued)Drawbacks: More expensive than a review Does not identify solutions May motivate hiding of problems Can generate hostility and lack of cooperation Can de-motivate Outsiders often do not understand and even if they do, they are not believed

You don’ttrust us!

January 20, 2000



All Rights Reserved

Slide # 179

Capability EvaluationsPurpose: To determine if an organization is capable of doing a

job wellPerformed by: Outside experts in all affected areasMethod: -- Many methods have been used for years -- For example, each of the armed services has ways

of doing contractor capability evaluations before selecting contractors

-- SEI has defined the SCE method, based on the CMM

January 20, 2000



All Rights Reserved

Slide # 180

SEI Definition

Typical duration: several days

A software capability evaluation is an appraisal by a trained team of

professionals to identify contractors who are qualified to perform the software work or to monitor the

state of the software process used on an existing software effort.

January 20, 2000



All Rights Reserved

Slide # 181

Capability Evaluations (continued)

• The SEI SCE (Software Capability Evaluation) is like an organized audit where the CMM is the model against which the program is evaluated

CMM 1.1

January 20, 2000



All Rights Reserved

Slide # 182

SEI Capability Evaluations What They Achieve

• They are based on well established best practices for software development

• So the results can be seen in comparison with accepted best practices

• They uncover problems (or, at least, symptoms)

• They note lack of problems• They note capabilities

January 20, 2000



All Rights Reserved

Slide # 183

SEI Capability Evaluations Drawbacks

• They do not identify solutions• They may motivate hiding of

problems• They are relatively expensive• Inexperienced evaluators can

misinterpret the findings, resulting in incorrect or unfair results– Example: focusing on practices rather

than goals and commitments

January 20, 2000



All Rights Reserved

Slide # 184

SEI Capability Evaluations Drawbacks

• Inexperienced or immature managers may not understand the conclusions, interpret them correctly, or act appropriately– Example: instituting metrics when the

problem is communication

January 20, 2000



All Rights Reserved

Slide # 185

“Self” AssessmentsPurpose: To help an organization learn where it is, how to

improve, and what to do nextPerformed by: A team of inside and/or outside technical and management expertsMethod: There are many methods of self assessment SEI has defined the two methods of software capability

assessment based on the CMM There have been several versions of the SEI method,

which can cause confusion

January 20, 2000



All Rights Reserved

Slide # 186

SEI Definition (from Paulk)

“A software process assessment is an appraisal by a trained team of

software professionals to determine the state of an organization's current software process, to determine the

high-priority software process-related issues facing an organization, and to obtain the organizational support for

software process improvement.”

January 20, 2000



All Rights Reserved

Slide # 187

“Self” Assessments (continued)Achieves:

• Uncovers problems • Identifies recommended solutions• Can be used to motivateDrawbacks:• Can mislead if too many internal participants• Can be misused as a sort of audit or proof of

capability• Can be very expensive• Can do more harm than good if practitioners

are motivated to change and management does not support the changes

January 20, 2000



All Rights Reserved

Slide # 188

Assessment vs. Evaluation

CMM 1.0

Evaluations

• Find Problems

• Evaluate Capability

• For External Use

Assessments

• Find Problems

• Find Solutions

• Motivate Improvement

• For Internal Use

CMM 1.1

CMM 1.0

Humphrey5-LevelModel

January 20, 2000



All Rights Reserved

Slide # 189

SEI Assessment Methods

• SPA– Software Process Assessment

• CBAIPI– CMM based Appraisal for Internal

Process Improvement

With both methods, SEI receives results and records confidentially. Results are published in aggregate form, quarterly.

January 20, 2000



All Rights Reserved

Slide # 190

SPA Method• Version 1 was based on a questionnaire or

survey, derived from Watts Humphrey’s model• Typical duration: 1 week• Several intermediate versions have been

based on intermediate versions of the CMM• The most recent and current version is based

on CMM 1.1• In each case there have also been several

degrees of “certification” of assessors, leading to many misunderstandings and unverifiable claims

January 20, 2000



All Rights Reserved

Slide # 191

Training for SEI Assessments • Initial Assessment Method (1988-93)– Assessors were trained by SEI– Later, Assessors were given “certification” by SEI– An assessment was deemed “more reliable” if one

or more trained and/or certified assessors were involved

– But SEI would not release results of any assessment or confirm or deny the validity of any assessment

– See Humphrey, Chapter 3, for more details

• Later Assessment Methods– Individuals and/or Organizations certified as

assessment team leaders and/or as trainers

January 20, 2000



All Rights Reserved

Slide # 192

CBAIPI Method• Typical duration: 1-2 weeks• More comprehensive than SPA• Places increased emphasis on CMM, when

compared with the SPA method• Intended to foster improvement for an

organization, not just focused on software• May be conducted by internal or external

personnel• Must be “led” by an “SEI Certified Lead Appraiser”

January 20, 2000



All Rights Reserved

Slide # 193

CMM-based Appraisals

Maturity LevelsMaturity Levels


Common FeaturesCommon Features


Goals

Implementation


ProcessCapability

Address

Describe

Achieve

Contain

Organized by

Contain

Indicate

Questions

January 20, 2000



All Rights Reserved

Slide # 194

SPA and CBA-IPI IssuesThese appraisals are intended to help an

organization evaluate itself– But too often, organizations will

publicize their appraisal results in the form of an achieved SEI maturity level

– SEI states that “when the focus is primarily on achieving a maturity level, it can distort the purpose of process improvement by diverting attention from genuine process improvement activities”

January 20, 2000



All Rights Reserved

Slide # 195

SPA and CBA-IPI Issues(continued)

– Contracting agencies, especially in the government, often encourage incorrect use of assessments by insisting on specific SEI maturity levels for contract eligibility

– SEI specifically states that “finding a maturity level is optional” for SPA and CBA-IPI and “not intended or appropriate” for an SCE

January 20, 2000



All Rights Reserved

Slide # 196

More SPA and CBA-IPI IssuesAssessment expense has generated many

lesser approaches– Typical appraisal cost: $80,000 (large org)• Assessment team (external): $45,000• Preparation $15,000• Time for participants $20,000

– Alternatives are OK for individual use, but the less extensive the less reliable the results

– Lack of external participation, while saving money, can also result in too-harsh or too-lenient appraisals

January 20, 2000



All Rights Reserved

Slide # 197

Chapter Summary• Maturity is a measure of management

effectiveness as measured by effective use of processes

• Managers must succeed with average people

• Humphrey’s process maturity model is derived from Crosby’s model of management effectiveness

• The SEI CMM is an elaboration of Humphrey’s model with specific recommendations for best practices and other details

January 20, 2000



All Rights Reserved

Slide # 198

Chapter Summary (continued)

• The CMM can be used as a descriptive or a normative model

• Each level has key process areas and other common elements

• The level 2 key process areas are fundamental to good management and correspond to major sections of this course

January 20, 2000



All Rights Reserved

Slide # 199

Chapter Summary (continued)

• Appraisals of various kinds use the CMM or other models to evaluate an organization’s maturity

• The more effective appraisal methods identify solutions as well as problems and foster buy-in and acceptance at all levels in the organization

January 20, 2000



All Rights Reserved

Slide # 200

End ofCHAPTER

13