7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 1/20
PSM Users ’ Grou p Conference
20 July 1999
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSMSoftware Perform ance
Evaluat ion Maturi ty Model
(SPE-MM)
Guy Mercur io Defense Contract Management Command
Softw are Center
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 2/20
GM 7/20/99 2
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSMSoftware Perform ance Evaluat ion
Matur i ty Model
w Introduction
w Development
w Goalsw Structure
w Pilot Test Approach & Issues
w Reporting Techniques
w Next Stepw Questions
w Contact
Agenda
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 3/20
GM 7/20/99 3
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
w Presentation at last years PSM Users’ Group Conference
focused on training DCMC software workforce on
measurement need and application
w Need to determine effectiveness of training
w Has the software measurement effort improved
w Need to develop a consistent method of evaluating
w Method needed to include DCMC policy on surveillance of
software development which includes software
measurement plus other best practices
Software Perform ance Evaluat ion
Matur i ty Model
Introduction
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 4/20
GM 7/20/99 4
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
Development of the DCMC mod el used p ort ions o f the fo l lowing:
Software Acquisition Capability Federal Aviation Administration Integrated
Model (SA-CMM) Capability Maturity Model (FAA-iCMM)
Software Capability
Maturity Model (SW-CMM)
DCMC SPE-MM
Software Perform ance Evaluat ion
Matur i ty Model
SPE-MM derived from SA-CMM features
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 5/20
GM 7/20/99 5
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
w Concept document developed - approved Aug ‘98
w Project Plan developed
þ Three phased approachþ What were going to do
þ When were going to do it
þ How much will it cost
þ What is the benefit to DCMC
w Project Team establishedþ Various skill levels
w Model and profile questionnaire baseline Jan ‘99
Software Perform ance Evaluat ion
Matur i ty Model
SPE-MM Development
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 6/20
GM 7/20/99 6
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
Determine the “health” of DCMC Contract
Administration Office (CAO) activities in the area of
software Contract Administration Support (CAS)
performance.
w Allow a CAO Commander to identify existing level of
maturity and goals to improve upon software CAS activities
Software Perform ance Evaluat ion
Matur i ty Model
Goals
w Identify needs to adjust Command training,policy, and guidance
w Focus DCMC Software Center assistance
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 7/20
GM 7/20/99 7
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
Do any way to get the job done - Process undefined -.
Success based on individual effort
Process in place at lower levels - Repeatable results for
specific application
Organizational process established and documented addressing
all applications - Same process used across organization
Activities measured and analyzed - Data provides ability to predict performance
and adjustments are made when process deviates from control limits
Optimized: Continuous process improvement activities in place. Piloting of
innovative ideas and technologies are attempted
Performed
Repeatable
Defined
Quantitative
Software Engineering Institute’s Software Acquisition Capability
Maturity Model tailored to DCMC Mission
SPE-MM Structure
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 8/20
GM 7/20/99 8
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSMSoftware Perform ance Evaluat ion
Matur i ty Model
Performed
Repeatable
Defined
Quanti tat ive
Optimized
Software SurveillanceSoftware CAS Management
Software CAS Planning
Training
DCMC Risk Management
Supplier Performance Management
Software CAS Process Definition & Maintenance
Quantitative Software CAS
DCMC Continuous Process Improvement
No Key Process Areas
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 9/20
GM 7/20/99 9
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
w Standardized method and data collection tools
w Two trained/experienced teams (5 per team)
w 6 locations selected by Districts (East & West)
w Questionnaires provided in advance
w Additional feedback from pilot sites on model
improvement
w Preliminary feedback on evaluation from pilot sites was
positive
w Provided roll up analysis to pilot sites
Pilot Test Approach
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 10/20
GM 7/20/99 10
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
w SPE objective was not clearly understood
w Perception of added requirements imposed
þ Fear that CAO Commander will demand ultimate
level (Optimized)
þ Fear of a Command-wide performance level
þ More DCMC Policy requirements
þ Fear of Competition
w Perception of another audit/assessment
Pilot Test Issues
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 11/20
GM 7/20/99 11
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
Example Reporting Techniques
w KPA strengths & weaknesses
w High level maturity profile
w KPA common feature breakout
wKPA roll-up by CAO
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 12/20
GM 7/20/99 12
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
KPA : S/W CAS PLANNING (Examp le)
The purpose of Software CAS Planning is to ensure that all reasonable
planning for the Software acquisition is conducted and that all elements
of the project are included.
STRENGTHSS/W Surveillance Plans in place and have been distributed.
WEAKNESSES
No CAO S/W Facility Plan is currently in place.
No measurements are currently being made by management of the S/W CAS
planning activities.
No measurement process in place and measurement analysis not performed .
CAO management demonstrates a minimal level of awareness of S/W insight
activities.
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 13/20
GM 7/20/99 13
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
S W S u r v e i l l a
n c e
CAO-C
S W C P l a n
S W C M
g t
P r d D
e f & M a i n
S u p ’ r P e
r f M g t
D C M C
R M
T r a i n i n g
Q u a
n t S W
C
D C M C
C P I
KPA SATISFACTION PROFILE EXAMPLE
To identify Command-wide performance indicators in specific S/W CAS activities
M a t u r i t y
L e v e l
Performed
Repeatable
Defined
QuantitativeOptimized
= KPA CRITERIA
MET
= KPA CRITERIA
NOT MET
KPA RATING
LEVEL RATING
CAO-B
CAO-A
CAO-D
CAO-E
CAO-F
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 14/20
GM 7/20/99 14
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
Maturity Co1 Co2 Co3 Ab1 Ab2 Ab3 Ab4 Ac1 Ac2 Ac3 Ac4 Ac5 Ac6 Ac7 Me1 Me2 Ve1 Ve2
Performed
No KPAs
Repeatable
SWC Pln Y Y Y Y Y Y Y Y Y Y Y
SWC Mgt Y Y Y Y Y Y Y Y Y Y Y
SW SURV Y Y Y Y Y Y Y Y Y Y Y N
Defined
SWC PD&M N N Y N Y Y Y Y N Y Y Y N Y N
SPM Y N N N N Y Y Y N Y Y Y
DCMC RM N N Y Y Y N N N N N N N N
Trng Y Y N Y N N Y Y Y N N Y Y
Quantitative
QSWC N N Y N N N N N N N N N N
Optomized
DCMC CPI N N N N Y N N N N N N N N N N
ommitment Co
Ability Ab N Not Met
Activity Ac
easurement Me Y Met
Verification Ve
KPA Breakout Example
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 15/20
GM 7/20/99 15
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSMSoftware Perform ance Evaluat ion
Matur i ty Model
KPA roll-up by CAO ExampleMaturity Co1 Co2 Co3 Ab1 Ab2 Ab3 Ab4 Ac1 Ac2 Ac3 Ac4 Ac5 Ac6 Ac7 Me1 Me2 Ve1 Ve2
Performed
No KPAs
Repeatable
SWC Plan 2 1 0 3 3 3 1 1 5 1 5
SWC Mgt 4 0 0 2 2 4 5 6 4 5 2
SW Surv 0 3 2 4 4 2 2 1 1 2 2 6 6 5
Defined
SWC PD&M 4 3 3 3 3 3 5 5 5 6 4 4 5 4 5
SPM 0 2 1 5 5 4 6 0 3 4 2 3
DCMC RM 3 4 1 2 2 4 5 5 4 4 5 4 4
Trng 1 1 3 2 1 3 2 0 1 5 2 1 1
Quant
QSW CAS 5 5 5 4 5 6 6 6 6 5 6 6 6
Optomized
DCMC CPI 6 2 3 3 1 6 6 6 6 2 4 6 6 6 6
ommitment Co
Ability Ab "One Book"
Activity Ac
asurement Me Not "One-Book"
erification Ve Number of CAO occurances of SPE-MM "Not Met"
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 16/20
GM 7/20/99 16
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
Phase II
w Model updated based on
þ Pilot evaluationsþ Comments obtained
w Evaluate remaining CAOs
w Plan to complete by Sept ‘00
w Analyze data for possible policy change
w Model can be used as Internal Self Assessment
tool
Next Step
Software Perform ance Evaluat ion
Matur i ty Model
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 17/20
GM 7/20/99 17
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
w Performing follow-up with pilot evaluation for possible
action plan for process improvements
w DCMC Earned Value Center is tailoring the SPE-MM inorder to evaluate CAO performance related to DCMC
Earned Value activity
w DCMC is the Executive Agent for Earned Value
w Future plans is to combined the SPE-MM and EV-MMand include Systems Engineering into one integrated
model
Software Performance Evaluat ion
Matur i ty Model
Other Related Activity
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 18/20
GM 7/20/99 18
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
w CMM Based Surveillance initiative planned that will
attempt to standardize our process and enhanceProgram Office visibility
w Developing guidebook that will provide insight to the
activities performed by software personnel
Software Performance Evaluat ion
Matur i ty Model
Other Related Activity
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 19/20
GM 7/20/99 19
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSMSoftware Perform ance Evaluat ion
Matur i ty Model
Questions
Comments
Issues ...
7/27/2019 DCMC Software Performance Evaluation Maturity Model
http://slidepdf.com/reader/full/dcmc-software-performance-evaluation-maturity-model 20/20
GM 7/20/99 20
Defense Contract Management Comm and Defense Contract Management Comm and
PSMPSMPSMPSM
For additional information contact:
Guy Mercurio
DCMC Software Center
Boston, MA
(617) 753 - 4122
www.dcmc.hq.dla.mil
Software Perform ance Evaluat ion
Matur i ty Model