View
214
Download
1
Tags:
Embed Size (px)
Citation preview
© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
CMMISM Appraisal Overview
Southern California SPIN
December 7, 2001
SM CMMI, CMM Integration, and SCAMPI are service marks of Carnegie Mellon University ® Capability Maturity Model and CMM are registered with the U.S. Patent and Trademark Office
Jane Moon
Raytheon
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 2© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
CMMI Appraisal Method StatusV1.0 assessment products published October 2000
• Assessment Requirements for CMMI (ARC)• Standard CMMI Assessment Method for Process
Improvement (SCAMPISM) Method Definition
Several pilot appraisals performed in 2000 (Phase I) and 2001 (Phase II)
V1.1 primary objectives:• Performance improvements• Integrated appraisal method (assessments and
evaluations)• Detailed method definition
ARC and SCAMPI Method Definition documents currently in SEI publication cycle
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 3© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
AMIT Membership
Jim Armstrong(SPC)
Mary Busby(Lockheed Martin)
David Kitson(SEI)
Rick Barbour(SEI)
Geoff Draper(Harris)
Gene Miluk(SEI)
Dan Bennett(USAF STSC)
Bud Glick(Motorola)
Joseph Morin(ISD, Inc.)
Ben Berauer(Raytheon)
Will Hayes(SEI)
Paul Riviere(U.S. Army CECOM)
Tom Bernard(USAF ASC/EN)
Rick Hefner(TRW)
Charlie Ryan(SEI)
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 4© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Characterizing ARC Appraisal Method Classes
SomeMostAll
(15504 option)
ARC requirements applicable
SmallMediumLargeTeam size
NoNoYes (optional)
15504 conformance
LowMediumHighResource needs
NoNoYesRatings generated
LowMediumHighAmount of objective evidence gathered
Class CClass BClass ASummaryCharacteristic
• Consider a family of appraisal methods in determining overall appraisal needs
• Class A methods may not be the most appropriate choice for organizations early in their process improvement cycle
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 5© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
SCAMPI Assumptions and Design Principles
1. SCAMPI is positioned as a Class A benchmarking method.
2. Goal achievement is a function of the extent to which the corresponding practices are present in the planned and implemented processes of the organization.
3. Practice implementation at the organizational unit level is a function of the degree of practice implementation at the instantiation level (e.g., projects)
4. The aggregate of objective evidence available to the appraisal team is used as the basis for determination of practice implementation.
5. Appraisal teams are obligated to seek and consider objective evidence of multiple types in determining the extent of practice implementation.
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 6© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
What is not in the ARC or MDD?Form and content of data collection instruments
Tool dependencies
Authorization-related requirements for SCAMPI Lead Appraisers (SM)
Detailed reporting requirements to CMMI Steward following completion of appraisal
Government acquisition policy related issues.• Deployment issues and content described in FAR, RFP,
etc.• Reuse of evaluation results (to be addressed by
Evaluation IPT)• Source Selection Authority (SSA) interface
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 7© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
SCAMPIMDDv1.0
DoDSW Eval
IPT Detailed Method Definition:• Phases, Processes, Activities• Inputs, Outputs, Outcomes• Options
Implementation Guides:• Internal Process Improvement• Supplier Selection and Monitoring
CMMI Requirements(Revised)• A-Spec• ARC
• Change Requests• Performance Ideas• Best Practices • Pilot Feedback
OtherAppraisalMethods
SCAMPI Method Definition Document (MDD) Transition
OtherSources
• CBA IPI, EIA 731-2• SCE, SDCE, FAM• etc.
• Evaluation Requirements Group
SCAMPIV1.1
• Method description
Implem.Guides(v1.1+)
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 8© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
ARC / SCAMPI Improvement Strategy
Shift appraisal team focus from discovery to verification• Leverage pre-onsite analysis of organization model
implementation (documentation, mapping, etc.)
Integrated data collection and continuous consolidation• Prioritize areas for focused investigation based on data
collection, analysis, and sufficiency of coverage (i.e., “triage”)
Provide detailed method definition and implementation guidance
• Support clarity, consistency, repeatability
• Organize content for efficient usage in the field
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 9© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Summary of ARC Changes
Expand ARC to encompass appraisal application modes
Supplier feedback of evaluation results (preliminary findings, final findings), per Evaluation IPT
Defined appraisal input (a major portion of the appraisal plan) subject to sponsor approval and change control (per ISO/IEC 15504)
Greater clarity in Class A, B, C descriptions and relationships in overall appraisal strategy
15504 conformance is optional for Class A methods; selected 15504-specific requirements may not apply
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 10© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
What are Practice Implementation Indicators?
“The fundamental idea of practice implementation indicators (PIIs) is quite simple and broadly applicable to any practice or activity; it is based on the presumption that the conduct of an activity or the implementation of a practice will result in “footprints” which are attributable to the activity or practice.”
- SCAMPI Method Definition, V1.1
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 11© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Objective Evidence Data Types
Direct Artifacts• Tangible outputs resulting directly from
implementation of a practice (e.g., Typical Work Products)
Indirect Artifacts• Artifacts that are a consequence or indicative of
performing a practice(e.g., meeting minutes, reviews, logs, reports)
Affirmations• Oral or written statements confirming or
supporting implementation of the practice(e.g, interviews, questionnaires)
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 12© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Characterizing Practice Implementation
Assign characterization values reflecting the extent of practice implementation for each instance
• Fully Implemented (FI)
• Largely Implemented (LI)
• Partially Implemented (PI)
• Not Implemented (NI)
Aggregate practice characterizations to organizational unit level using defined method aggregation rules
Iterate and focus revisions to data collection plan
Generate findings based on aggregation of weaknesses and strengths
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 13© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Characterizing Practice Implementation - 2
• Any situation not covered by aboveNot Implemented (NI)
• Direct artifacts absent or judged inadequate
• Artifacts or affirmations indicate some aspects of the practice are implemented
• One or more weaknesses noted
Partially Implemented (PI)
• Direct artifacts present and appropriate
• Supported by indirect artifact and/or affirmation
• One or more weaknesses noted
Largely Implemented (LI)
• Direct artifacts present and appropriate
• Supported by indirect artifact and/or affirmation
• No weaknesses noted
Fully Implemented (FI)
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 14© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Example – Indicators of Practice Implementation
PP SP1.1-1:
Establish and maintain
a top-level work breakdown structure (WBS)
for estimating the scope of the project.
Primary artifact:-top-level WBS, with revision history-task descriptions-work product descriptions
Affirmation:- how is the WBS used?- how are estimates generated?
Indirect artifact:- project estimates aligned with WBS elements
Indirect artifact:-minutes of meetings at which WBS was generated or used to develop project estimates
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 15© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Data Collection and Rating Concepts
Corroboration
• Must have direct artifacts, combined with either indirect artifact or affirmation
Coverage
• Must have sufficient objective evidence for implementation of each practice, for each instance
• Must have face-to-face (F2F) affirmations (avoid “paper-only appraisals”):
– At least one instance for each practice (“one column”)
– At least one practice for each instance (“one row”)
– or 50% of practices for each PA goal, for each project, have at least one F2F affirmation data point
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 16© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Affirmation Coverage Rules - Summary
Column
Row
PA.SPx.4-1
PA.SPx.3-1
PA.SPx.2-1
PA.SPx.1-1
Project-4Project-3Project-2Project-1
1. “One Row, One Column”
or, 2. “50% rule”:
> 50% of PA practices for each goal, for each project, have at least one face-to-face (F2F) affirmation data point
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 17© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Aggregation and Consensus
Practice Implementation Characterizations
(practice instantiation level)
Practice Implementation Characterizations
(organizational unit level)
Goal Satisfaction Ratings
Capability Level and/or Maturity Level Ratings
Level ofConsensus
Mini-Team
Full Team
Full Team
Full Team
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 18© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
MDD v1.1 Outline
Phase I:Plan and Preparefor Appraisal------------ - --- --------- ---- ----- -------------------Phase II:ConductAppraisal------------ - --- --------- ---- ----- -------------------Phase III:ReportResults------------ - --- --------- ---- ----- -------------------
Primary ReferenceMaterial
IntroductoryProse------------ - --- --------- ---- ----- -------------------
Doc. Overview______________________________
Part1 Descriptive nameand information______________________________
Part2 Descriptive nameand information______________________________
...______________________________
Part N Descriptive nameand information______________________________
ExecutiveSummary for theAppraisal Sponsor------------ - --- --------- ---- ----- -------------------
Method Overviewwith audience-specific summaryof this document------------ - --- --------- ---- ----- -------------------
Front Matter
Glossary------------ - --- --------- ---- ----- -------------------
SCAMPIAppraisalDisclosureStatement (ADS)------------ - --- --------- ---- ----- -------------------
Role of PIIs inVerifying PracticeImplementation------------ - --- --------- ---- ----- -------------------ARC/MDD
Traceability------------ - --- --------- ---- ----- -------------------
Appendices
FocusedInvestigationElaboration andGuidance------------ - --- --------- ---- ----- -------------------
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 19© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
MDD Structure
Phases (3)
Processes (11)• Entry / exit criteria,
inputs, outputs, activities, etc.
Activities (43)• Activity Description• Required Practices• Parameters and
Limits• Optional Practices• Implementation
Guidance
3.1 Analyze Requirements
Purpose Understand the business needs of the organization for whom the appraisal is being requested. The Appraisal Team Leader will collect information and help the appraisal sponsor match appraisal objectives with their business objectives.
Entry CriteriaInitial contact between the appraisal sponsor and authorized SCAMPI Lead Appraiser has occurred. The Lead Appraiser has been given access to members of the sponsoring organization, and asked to develop a proposal.
Inputs Sponsor, Initial Requirements and Constraints, Process-related Legacy Information
Activities 3.1.1. Determine Appraisal Objectives3.1.2. Determine Appraisal Constraints3.1.3. Determine Appraisal Scope3.1.4. Determine Outputs3.1.5. Obtain Commitment to Appraisal Input
Outputs Appraisal Input
Outcome The decision to proceed with the appraisal based on a shared understanding of the appraisal objectives, constraints and scope.
Exit Criteria The appraisal input has been approved by the appraisal sponsor and placed under change management.
Sample MDD Activity
3.1.1 Determine Appraisal Objectives
Activity Description
The business needs for process improvement drive the requirements for the conduct of any given appraisal, and generally include one or more of three closely related factors: Reducing costs Improving quality, and Decreasing time to market
The fundamental premise of process improvement is that organizationalprocesses significantly impact these factors.
A fair and objective characterization of the process in use in the organization(s) is the essential reason for conducting an appraisal. In addition to this motivation, a sponsor' s desire to conduct an appraisal could be driven by one or more of the following business related goals: Document a credible benchmark that reflects successful process
improvement Evaluate areas of potential risk that may effect the performance of the
organization Involve members of the appraised organization in improving the
performance of the process Support specific decisions related to the direction of a new or existing
improvement program Motivate a supplier to focus on process issues that affect their performance
on a contract
Required Practices
Identify sponsor and relevant stakeholders, and establish communication.
Document business and appraisal objectives
Assure alignment of appraisal objectives to business objectives
Determine and document appraisal usage mode (internal process improvement, supplier selection, process monitoring).
Parameters and Limits
At least one communication between appraisal team leader and sponsor (Some usage modes may limit this significantly, others may re quire much more than one interaction).
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 20© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Potential Future EfficienciesIncremental and delta appraisals
Additional work aids (templates, checklists, “look fors / listen fors”)
Improved instruments and tools
Statistical sampling
Leverage and cross-correlate model built-in dependencies for improved appraisal data management.
• Relationships (threads) among practices (GPs, SPs), Goals, PAs– e.g. PP, PMC, IPM
• Single work products / indicators that satisfy multiple practices
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 21© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Appraisal Concept of Operations
ProjectImplementation
OrganizationImplementation
• Model / process mapping• Objective evidence
Inventory ofObjectiveEvidence
• Verified implementation• Strengths, weaknesses• Findings, ratings
DefinedProcesses
• Process improvement• Class B, C appraisals
Organization
CMMI Steward
AppraisalTeam
CMMIProductSuite
• Transition• Deployment• Tailoring
• Appraisal planning• Readiness review
• Verification• Validation• Aggregation• Focused Investigation
• Model• Method• Training
PAIS
• Appraisal results• Performance data
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 22© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
AMIT Initiatives Investigated
Integrated Data Collection and Verification Approach• Focused investigation - opportunistically use data collection (e.g.,
questionnaire, objective evidence) to narrow the focus for further investigation and team emphasis
• Leverage organizational assets reflecting implementation of model practices
• Greater appraisal team focus on verification rather than discovery
Incremental Appraisals (Deferred post-v1.1)• Pre-planned partitioning of appraisal scope across weeks or
months
Delta Appraisals (Deferred post-v1.1)• Partial re-assessment to focus on weaknesses identified in prior
SCAMPI appraisals
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 23© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
Summary - CMMI Appraisal Method Status
ARC and SCAMPI v1.1 revisions currently in SEI publication process
• Performance improvements• Integrated appraisal method (assessments and
evaluations)• Detailed method definition
Fundamental SCAMPI concepts• Indicator-driven appraisals• Focused investigation (Integrated data collection
and continuous consolidation)
CMMI Appraisal Method Overview - CMMI – 11/13/01 Page 24© 2001 by Carnegie Mellon University
C S a r n e g i e M e l l o n o f t w a r e E n g i n e e r i n g I n s t i t u t e
CMMISM
For More Information…
Contacts:
Geoff DraperHarris [email protected]
David KitsonManager, SEI Appraiser [email protected]
http://www.sei.cmu.edu/cmmi/