View
218
Download
0
Category
Tags:
Preview:
Citation preview
1
Convener: Houman Younessi
Convener: Houman Younessi
Software Engineering Management
Software Engineering Management
Course # CISH-6050Course # CISH-6050
Lecture 7: Lecture 7:
Software Process AssessmentSoftware Process Assessment
07/09/201207/09/2012
2 CISH-6050 - Software Engineering Management
AGENDA …AGENDA …
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay off?- Whistleblowers- Reference: SPICE Assessment
Documents
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay off?- Whistleblowers- Reference: SPICE Assessment
Documents
3 CISH-6050 - Software Engineering Management
SW-CMM Process AssessmentSW-CMM Process Assessment
• SEI SW-CMM Improvement Approach: IDEAL- Initiating- Diagnosing- Establishing- Acting- Leveraging
• Diagnose process, establish priorities, & act on them
• SEI SW-CMM Improvement Approach: IDEAL- Initiating- Diagnosing- Establishing- Acting- Leveraging
• Diagnose process, establish priorities, & act on them
4 CISH-6050 - Software Engineering Management
SW-CMM: IDEALSW-CMM: IDEAL
• Initiating:- Stimulus for improvement- Set context & sponsorship- Establish improvement infrastructure
• Diagnosing:- Appraise & characterize current practice- Develop recommendations- Document phase results
• Initiating:- Stimulus for improvement- Set context & sponsorship- Establish improvement infrastructure
• Diagnosing:- Appraise & characterize current practice- Develop recommendations- Document phase results
5 CISH-6050 - Software Engineering Management
SW-CMM: IDEAL …SW-CMM: IDEAL …
• Establishing:- Set strategy & priorities- Establish process action items- Plan actions
• Acting:- Define processes and measures- Plan & execute pilots- Plan, execute, & track installation
• Establishing:- Set strategy & priorities- Establish process action items- Plan actions
• Acting:- Define processes and measures- Plan & execute pilots- Plan, execute, & track installation
6 CISH-6050 - Software Engineering Management
SW-CMM: IDEAL …SW-CMM: IDEAL …
• Leveraging:- Document & analyze lessons- Revise organizational approach
• Leveraging:- Document & analyze lessons- Revise organizational approach
7 CISH-6050 - Software Engineering Management
SW-CMM Process AssessmentSW-CMM Process Assessment
• General Classes of SW-CMM Appraisal1. Software Process Assessment
Determine state of organization’s software process
2. Software Capability Evaluations Identify contractors qualified to
perform software work
• General Classes of SW-CMM Appraisal1. Software Process Assessment
Determine state of organization’s software process
2. Software Capability Evaluations Identify contractors qualified to
perform software work
8 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Software Process Assessment:- Identify improvement priorities within
organization- Assessment team uses CMM to guide
identifying & prioritizing findings- Findings & KPA guidance used to plan
improvement strategy for organization
• Software Process Assessment:- Identify improvement priorities within
organization- Assessment team uses CMM to guide
identifying & prioritizing findings- Findings & KPA guidance used to plan
improvement strategy for organization
9 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Software Capability Evaluations:- Identify risks associated with a project
or contract to build high quality on schedule & budget
- During acquisition process, capability evaluation may be performed on bidders
- Findings of an evaluation may be used to identify risk with using a contractor
- Performed on existing contracts to monitor process performance
• Software Capability Evaluations:- Identify risks associated with a project
or contract to build high quality on schedule & budget
- During acquisition process, capability evaluation may be performed on bidders
- Findings of an evaluation may be used to identify risk with using a contractor
- Performed on existing contracts to monitor process performance
10 CISH-6050 - Software Engineering Management
Software Process Assessment & Capability Evaluation Steps:
Software Process Assessment & Capability Evaluation Steps:
11 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Common Steps:- Team Selection
Select team trained in CMM Knowledgeable in SE & mgmt
- Maturity Questionnaire Site reps complete questionnaire
- Response Analysis Analyze results of questionnaire Investigation areas = KPAs
• Common Steps:- Team Selection
Select team trained in CMM Knowledgeable in SE & mgmt
- Maturity Questionnaire Site reps complete questionnaire
- Response Analysis Analyze results of questionnaire Investigation areas = KPAs
12 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Common Steps …- On-site Visit
Using results analysis, conduct on-site visit to view process areas
Using KPAs as guide, question, listen, review & synthesize info
Apply professional judgment Document rationale for situations
where KPAs not met
• Common Steps …- On-site Visit
Using results analysis, conduct on-site visit to view process areas
Using KPAs as guide, question, listen, review & synthesize info
Apply professional judgment Document rationale for situations
where KPAs not met
13 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Common Steps …- Findings
At end of on-site period, team produces list of findings
Identifies strengths & weaknesses of org’s software processes
Software Process Assessment -> Basis for PI recommendations
Software Capability Evaluation -> Findings part of risk analysis
• Common Steps …- Findings
At end of on-site period, team produces list of findings
Identifies strengths & weaknesses of org’s software processes
Software Process Assessment -> Basis for PI recommendations
Software Capability Evaluation -> Findings part of risk analysis
14 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Common Steps …- KPA Profile
Team prepares KPA profile, showing where KPAs satisfied / not satisfied by organization
KPA can be satisfied and still have associated findings, as long as findings don’t identify major problems achieving goals of KPA
• Common Steps …- KPA Profile
Team prepares KPA profile, showing where KPAs satisfied / not satisfied by organization
KPA can be satisfied and still have associated findings, as long as findings don’t identify major problems achieving goals of KPA
15 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Differences: Process Assessments & Capability Evaluation- Results of process assessment or
capability evaluation may differ - Assessment / evaluation scope may
vary: Different definitions of ‘Organization’ Org may be based on senior
management, geo location, common app, profit/loss center, etc.
Sample of selected projects
• Differences: Process Assessments & Capability Evaluation- Results of process assessment or
capability evaluation may differ - Assessment / evaluation scope may
vary: Different definitions of ‘Organization’ Org may be based on senior
management, geo location, common app, profit/loss center, etc.
Sample of selected projects
16 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Differences …- Motivation, objective, outcome & results
ownership differ These factors lead to differences in
dynamics of interviews, scope of inquiry, info collected, & results
Assessment & evaluation methods are different
Assessment training doesn’t prepare team to do evaluation, vice versa
• Differences …- Motivation, objective, outcome & results
ownership differ These factors lead to differences in
dynamics of interviews, scope of inquiry, info collected, & results
Assessment & evaluation methods are different
Assessment training doesn’t prepare team to do evaluation, vice versa
17 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Differences …- Process Assessment – performed in
open, collaborative environment Commitment from mgmt & staff to do
process improvement Objective: surface problems & help
improve organization Emphasis on interviews as tool for
understanding organization’s software process
• Differences …- Process Assessment – performed in
open, collaborative environment Commitment from mgmt & staff to do
process improvement Objective: surface problems & help
improve organization Emphasis on interviews as tool for
understanding organization’s software process
18 CISH-6050 - Software Engineering Management
SW-CMM Process Assessment …SW-CMM Process Assessment …
• Differences …- Capability Evaluation – performed in
audit-oriented environment Objective tied to monetary
considerations Emphasis on documented audit
trail that reveals software process actually implemented by organization
• Differences …- Capability Evaluation – performed in
audit-oriented environment Objective tied to monetary
considerations Emphasis on documented audit
trail that reveals software process actually implemented by organization
19 CISH-6050 - Software Engineering Management
AGENDA …AGENDA …
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay
off?- Whistleblowers- Reference: SPICE Assessment
Documents
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay
off?- Whistleblowers- Reference: SPICE Assessment
Documents
20 CISH-6050 - Software Engineering Management
Software Process AssessmentsSoftware Process AssessmentsSoftware Process AssessmentsSoftware Process Assessments
• Why do process assessment?- “Why Do Organizations Have
Assessments? Do They Pay Off?”, CMU SEI Technical Report, July, 1999
- Panel discussion from SEPG ’99 in Atlanta, GA on March 10, 1999
- Each panelist is one of most active lead assessor for SEI for CMM-Based Appraisals for Internal Process Improvement (CBA IPI)
• Why do process assessment?- “Why Do Organizations Have
Assessments? Do They Pay Off?”, CMU SEI Technical Report, July, 1999
- Panel discussion from SEPG ’99 in Atlanta, GA on March 10, 1999
- Each panelist is one of most active lead assessor for SEI for CMM-Based Appraisals for Internal Process Improvement (CBA IPI)
21 CISH-6050 - Software Engineering Management
Software Process Assessments …Software Process Assessments …Software Process Assessments …Software Process Assessments …
• Electronic Data Systems- Why do assessments?
Driven by customer: higher maturity to keep customers; extra bidding points for higher maturity; DoD requirements
- Do assessments pay off? Yes! Need exec sponsorship, funding Everyone learns, helps org culture
• Electronic Data Systems- Why do assessments?
Driven by customer: higher maturity to keep customers; extra bidding points for higher maturity; DoD requirements
- Do assessments pay off? Yes! Need exec sponsorship, funding Everyone learns, helps org culture
22 CISH-6050 - Software Engineering Management
Software Process Assessments …Software Process Assessments …Software Process Assessments …Software Process Assessments …
• Reuters- Why do assessments?
Framework for continuous process improvement; maturity levels build on previous level
Raises awareness in group- Do assessments pay off?
Clear view of cost, effort, resource Measure key performance items Method for introducing global change
• Reuters- Why do assessments?
Framework for continuous process improvement; maturity levels build on previous level
Raises awareness in group- Do assessments pay off?
Clear view of cost, effort, resource Measure key performance items Method for introducing global change
23 CISH-6050 - Software Engineering Management
Software Process Assessments …Software Process Assessments …Software Process Assessments …Software Process Assessments …
• U. S. Navy- Why do assessments?
Early SEI affiliate Beta test group for Maturity
Questionnaire Continued with SEI assessments
- Do assessments pay off? Invaluable to organization In 1997 determined 4.1 to 1 ROI Education for new team members
• U. S. Navy- Why do assessments?
Early SEI affiliate Beta test group for Maturity
Questionnaire Continued with SEI assessments
- Do assessments pay off? Invaluable to organization In 1997 determined 4.1 to 1 ROI Education for new team members
24 CISH-6050 - Software Engineering Management
Software Process Assessments …Software Process Assessments …Software Process Assessments …Software Process Assessments …
• Panel Q&A- Why pay cost of doing CBA IPI?- How often do you do full CBA IPI?- How small of organization to do CBA IPI?- How to reduce impact/stress on org when
doing CBA IPI?
• Panel Q&A- Why pay cost of doing CBA IPI?- How often do you do full CBA IPI?- How small of organization to do CBA IPI?- How to reduce impact/stress on org when
doing CBA IPI?
25 CISH-6050 - Software Engineering Management
AGENDA …AGENDA …
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay off?- Whistleblowers- Reference: SPICE Assessment
Documents
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay off?- Whistleblowers- Reference: SPICE Assessment
Documents
26 CISH-6050 - Software Engineering Management
Software Process Assessments …Software Process Assessments …Software Process Assessments …Software Process Assessments …
• Blowing the Whistle on Troubled Software Projects- Article by Mark Keil & Daniel Robey- Controversy behind going forward to
management when auditors find problems with projects
- Case study of IS Auditors 1999 Study 75 Auditors
• Blowing the Whistle on Troubled Software Projects- Article by Mark Keil & Daniel Robey- Controversy behind going forward to
management when auditors find problems with projects
- Case study of IS Auditors 1999 Study 75 Auditors
27 CISH-6050 - Software Engineering Management
Software Process Assessments …Software Process Assessments …Software Process Assessments …Software Process Assessments …
• Key Points- Auditor obligation vs. risk- Reluctance to blow the whistle- Reluctance to hear the whistle- Conditions that encourage or influence
whistle blowing
• Key Points- Auditor obligation vs. risk- Reluctance to blow the whistle- Reluctance to hear the whistle- Conditions that encourage or influence
whistle blowing
28 CISH-6050 - Software Engineering Management
AGENDA …AGENDA …
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay off?- Whistleblowers- Reference: SPICE Assessment
Documents
• Process Assessment- SEI SW-CMM Improvement Approach- Do organizational assessments pay off?- Whistleblowers- Reference: SPICE Assessment
Documents
29 CISH-6050 - Software Engineering Management
SPICE Assessment: ReferenceSPICE Assessment: Reference
• SPICE Software Process Assessment Information- SPICE documentation available free of
charge from Software Quality Institute web site: http://www.sqi.gu.edu.au/spice/suite/download.html
• SPICE Software Process Assessment Information- SPICE documentation available free of
charge from Software Quality Institute web site: http://www.sqi.gu.edu.au/spice/suite/download.html
30 CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …SPICE Assessment: Reference …
• SPICE Part 3 – Rating Process- Section 4.2: Defining Assessment Input - Section 4.4: Assessing and Rating
Process- Section 4.5: Recording the Assessment
Output
• SPICE Part 3 – Rating Process- Section 4.2: Defining Assessment Input - Section 4.4: Assessing and Rating
Process- Section 4.5: Recording the Assessment
Output
31 CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …SPICE Assessment: Reference …
• SPICE Part 4 – Guide to Conducting Assessment- Section 4.2: Process Rating Scheme- Section 4.3.1.1: Team-based Assessment- Section 4.4: Assessment Stages (8)- Section 4.5: Success Factors for Process
Assessment- Section 5.1: Reviewing Assessment
Inputs
• SPICE Part 4 – Guide to Conducting Assessment- Section 4.2: Process Rating Scheme- Section 4.3.1.1: Team-based Assessment- Section 4.4: Assessment Stages (8)- Section 4.5: Success Factors for Process
Assessment- Section 5.1: Reviewing Assessment
Inputs
32 CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …SPICE Assessment: Reference …
• SPICE Part 4 …- Section 5.2: Selecting Process Instances- Section 5.3: Preparing for Team-based
Assessment- Section 5.4: Collecting and Verifying
Information- Section 5.5: Determine Actual Ratings for
Process Instances- Section 5.6: Determining Derived Ratings- Section 5.7: Validating the Ratings- Section 5.8: Presenting Assessment Output
• SPICE Part 4 …- Section 5.2: Selecting Process Instances- Section 5.3: Preparing for Team-based
Assessment- Section 5.4: Collecting and Verifying
Information- Section 5.5: Determine Actual Ratings for
Process Instances- Section 5.6: Determining Derived Ratings- Section 5.7: Validating the Ratings- Section 5.8: Presenting Assessment Output
33 CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …SPICE Assessment: Reference …
• SPICE Part 5 – Construction, Selection, & Use of Assessment Tools- Section 4.1: Form and Purpose of an
Assessment Instrument- Section 4.2 (and Annexes A, B, C, D):
Implementation of Standard Indicators- Section 4.3: Tailoring of Indicators
Contained in Assessment Instrument- Section 4.4: Modular Assessment
Instruments
• SPICE Part 5 – Construction, Selection, & Use of Assessment Tools- Section 4.1: Form and Purpose of an
Assessment Instrument- Section 4.2 (and Annexes A, B, C, D):
Implementation of Standard Indicators- Section 4.3: Tailoring of Indicators
Contained in Assessment Instrument- Section 4.4: Modular Assessment
Instruments
34 CISH-6050 - Software Engineering Management
SPICE Assessment: Reference …SPICE Assessment: Reference …
• SPICE Part 5 …- Section 4.5: Capturing and Processing
Assessment Data- Section 4.6: Using an Assessment
Instrument
• SPICE Part 6 – Qualifications & Training of Assessors- Section 4.1: Role of the Assessor- Section 5: Assessor Competence
• SPICE Part 5 …- Section 4.5: Capturing and Processing
Assessment Data- Section 4.6: Using an Assessment
Instrument
• SPICE Part 6 – Qualifications & Training of Assessors- Section 4.1: Role of the Assessor- Section 5: Assessor Competence
35 CISH-6050 - Software Engineering Management
ReferencesReferences
• M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability Maturity Model for Software", Version 1.1, Software Engineering Institute, Carnegie Mellon University, February, 1993. Available at http://www.sei.cmu.edu/publications/documents/93.reports/93.tr.024.html
• W. S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, MA, 1989
• M. Paulk, C. V. Weber, B. Curtis, M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Boston, MA, 1995
• “Process Maturity Profile of the Software Community 2002 Mid-Year Update”, Software Engineering Institute, Carnegie Mellon University, August, 2002. Available at http://www.sei.cmu.edu/sema/pdf/SW-CMM/2002aug.pdf
• M. Paulk, B. Curtis, M. B. Chrissis, C. V. Weber, "Capability Maturity Model for Software", Version 1.1, Software Engineering Institute, Carnegie Mellon University, February, 1993. Available at http://www.sei.cmu.edu/publications/documents/93.reports/93.tr.024.html
• W. S. Humphrey, Managing the Software Process, Addison-Wesley, Reading, MA, 1989
• M. Paulk, C. V. Weber, B. Curtis, M. B. Chrissis, The Capability Maturity Model: Guidelines for Improving the Software Process, Addison-Wesley, Boston, MA, 1995
• “Process Maturity Profile of the Software Community 2002 Mid-Year Update”, Software Engineering Institute, Carnegie Mellon University, August, 2002. Available at http://www.sei.cmu.edu/sema/pdf/SW-CMM/2002aug.pdf
36 CISH-6050 - Software Engineering Management
References …References …
• D. Dunaway, R. Berggren, G. des Rochettes, P. Iredale, I. Lavi, G. Taylor, “Why Do Organizations Have Assessments? Do They Pay Off?”, Software Engineering Institute, Carnegie Mellon University, July, 1999. Available at http://www.sei.cmu.edu/publications/documents/99.reports/99tr012/99tr012abstract.html
• M. Keil, D. Robey, “Blowing the Whistle on Troubled Software Projects”, Communications of the ACM, Vol. 44, No. 4, April 2001, pp. 87-93. PDF file (ACMp87-Keil.pdf) available online via ACM Digital Library when
accessed from the RPI at Hartford Library • ISO/IEC TR 15504:1998 - Software Process Assessment (SPICE)
documentation suite. Available at http://www.sqi.gu.edu.au/spice/suite/download.html
• D. Dunaway, R. Berggren, G. des Rochettes, P. Iredale, I. Lavi, G. Taylor, “Why Do Organizations Have Assessments? Do They Pay Off?”, Software Engineering Institute, Carnegie Mellon University, July, 1999. Available at http://www.sei.cmu.edu/publications/documents/99.reports/99tr012/99tr012abstract.html
• M. Keil, D. Robey, “Blowing the Whistle on Troubled Software Projects”, Communications of the ACM, Vol. 44, No. 4, April 2001, pp. 87-93. PDF file (ACMp87-Keil.pdf) available online via ACM Digital Library when
accessed from the RPI at Hartford Library • ISO/IEC TR 15504:1998 - Software Process Assessment (SPICE)
documentation suite. Available at http://www.sqi.gu.edu.au/spice/suite/download.html
Recommended