BI-DIRECTIONAL GAP ANALYSES OF CMMI, AUTOMOTIVE SPICE · PDF fileBI-DIRECTIONAL GAP ANALYSES OF CMMI, AUTOMOTIVE SPICE, AND INCOSE Travis Foust, Matthew Ramirez, Regina Dorow October

BI-DIRECTIONAL GAP ANALYSES OFCMMI, AUTOMOTIVE SPICE, AND INCOSE

Travis Foust, Matthew Ramirez, Regina Dorow

October 2nd, 2017

2

BACKGROUNDCMMI for Development is a process reference model developed by the Software Engineering Institute (SEI) to be used by organizations developing systems (software or otherwise) to measure their maturity and/or capability.

Automotive SPICE, or ASPICE, is a process reference model developed by organizations within the automotive industry to create a more automotive-focused reference model compared to SPICE or CMMI. ASPICE is derived from SPICE.

INCOSE Systems Engineering Handbook is a handbook developed by the International Council of Systems Engineering to be a practical guide to Systems Engineering. INCOSE standards are be used to evaluate individual engineers while CMMI and ASPICE are used to evaluate organizations. The handbook adheres to ISO15288.

Objective - While CMMI and ASPICE are the two dominant sets of standards in the automotive industry, this study will attempt to understand the differences between all three of these standards. The results will be a valuable resource in determining what standard(s) should be mapped and adopted as best practice, required of suppliers, and/or used for internal appraisals.

3

CMMIProject Management Engineering

SupportProcess Management

REQM (2)

Requirements

Management

PP (2)

Project

Planning

PMC (2)

Project Monitoring

and Control

SAM (2)

Supplier Agreement

Management

IPM (3)

Integrated Project

Management

RSKM (2)

Risk

Management

QPM (4)

Quantitative Project

Management

RD (3)

Requirements

Development

TS (3)

Technical

Solution

PI (3)

Product

Integration

VER (3)

Verification

VAL (3)

Validation

MA (2)

Measurement

and Analysis

DAR (3)

Decision Analysis

and Resolution

PPQA (2)

Process and Product

Quality Assurance

CM (2)

Configuration

Management

CAR (5)

Causal Analysis

and Resolution

OPF (3)

Organizational

Process Focus

OPD (3)

Organizational

Process Definition

OT (3)

Organizational

Training

OPP (4)

Organizational

Process Performance

OPM (5)

Organizational

Performance Management

4

ASPICE

[1]

5

INCOSE

[2]

METHODOLOGY AND APPROACH

6

7

THE PROCESS

1. Become familiar with CMMI, ASPICE, and INCOSE.

2. Conduct literature review to understand similar studies in the field.

3. Define gap analysis scope and technique.

4. Execute gap analyses.

5. Review gap analyses.

6. Analyze results.

7. Determine and document conclusions.

8. Develop appropriate proposed actions.

6 Analyses (bi-directional x3)

CMMI – 167 Specific Practices

ASPICE – 261 Base Practices

INCOSE – 110 Process Activities

8

ANALYSES VOLUME

CMMI

ASPICE INCOSE

9

EXAMPLE ANALYSIS – ASPICE TO CMMI

ASPICE

System Engineering

Process Group (SYS)

SYS.1 Requirements

Elicitation

SYS.1.BP1: Obtain

stakeholder

requirements and

requests.

Search CMMI

specific practices

for match to

SYS.1.BP1.

CMMI

Engineering

Requirements

Development (RD)

RD 1.1: Elicit

stakeholder needs,

expectations,

constraints, and

interfaces for all

phases of the

product lifecycle.

ASPICE SYS.1

Requirements Elicitation

ASPICE BP CMMI SP

SYS.1.BP1 RD 1.1

SYS.1.BP2

SYS.1.BP3

SYS.1.BP4

SYS.1.BP5

SYS.1.BP6

Coverage 16.7%

“ASPICE to CMMI” may also be

interpreted as CMMI coverage of

ASPICE.

1 0

CHALLENGES

ASPICE

Develop software detailed

design.

CMMI

Develop a design for the

product or product

component.

Gap

Match

ASPICE

Identify relevant

regulations.

INCOSE

Use expert to help

determine relevant

regulations if needed.

Gap

1. Related practices may make assumptions and/or omit key actions…

2. Similar practices may vary in specificity…

3. Identical practices may have different subjects. E.g. “Identify risks” in System Design does not map to

“Identify risks” in Acquisition.

RESULTS

1 1

• Major Gap Areas

• Acquisition

• Supply

• Documentation

• Reuse

• Consistent gaps found in most processes for ensuring consistency of process and communicating results to impacted groups.

1 2

ASPICE TO CMMIContract Agreement 57.1%

Supplier Monitoring 100.0%

Technical Requirements 70.0%

Legal and Administrative Requirements 0.0%

Project Requirements 33.3%

Request for Proposals 0.0%

Supplier Qualification 60.0%

Supplier Tendering 0.0%

Product Release 46.2%

Requirements Elicitation 66.7%

System Requirements Analysis 62.5%

System Architectural Design 75.0%

System Integration and Integration Test 55.6%

System Qualification Test 71.4%

Software Requirements Analysis 62.5%

Software Architectural Design 66.7%

Software Detailed Design and Unit Construction 62.5%

Software Unit Verification 71.4%

Software Integration and Integration Test 55.6%

Software Qualification Test 71.4%

Quality Assurance 100.0%

Verification 80.0%

Joint Review 50.0%

Documentation 12.5%

Configuration Management 88.9%

Problem Resolution Management 55.6%

Change Request Management 87.5%

Project Management 90.0%

Risk Management 100.0%

Measurement 81.8%

Reuse Process Group Reuse Program Management 37.5% 37.5%

Process Improvement

Process Group

Process Improvement

87.5% 87.5%

Total 59.1%

Management Process

Group

ASPICE to CMMI Gap Analysis

40.0%

28.6%

89.3%

66.0%

64.6%

65.8%

Acquisition Process Group

Supply Process Group

System Engineering Process

Group

Software Engineering

Process Group

Supporting Process Group

• Major Gap Areas• Project Monitoring, Integrated Project Management, and Quantitative Project

Management• Verification and Validation• Decision Analysis• Process Management

• Some gaps in Engineering were found as a result of ASPICE’s higher level of specificity.

1 3

CMMI TO ASPICE

Maturity Total

Requirements Management 100.0% Measurement and Analysis 62.5%

Supplier Agreement Management 66.7% Process and Product Quality Assurance 75.0%

Project Planning 57.1% Configuration Management 100.0%

Project Monitoring and Control 10.0%

Integrated Project Management 0.0% Requirements Development 50.0% Decision Analysis and Resolution 0.0% Organizational Process Focus 0.0%

Risk Management 85.7% Technical Solution 50.0% Organizational Process Definition 0.0%

Product Integration 66.7% Organizational Training 0.0%

Validation 40.0%

Verification 25.0%

Level 4 Quantitative Project Management 0.0% Organizational Process Performance 0.0% 0.0%

Level 5 Causal Analysis and Resolution 100.0% Organizational Performance Management 30.0% 53.3%

Total 39.5%

61.1%

CMMI to ASPICE Gap Analysis

40.7% 47.5% 7.9%66.7%

29.1%

Level 2

Project Management Process ManagementEngineering Support

Level 3

• Major Gap Areas

• Architecture Definition

• System Analysis

• Operation

• Maintenance

• Disposal

• Supply

• Portfolio Management

• INCOSE to CMMI has the highest coverage results in this study.

1 4

INCOSE TO CMMIBusiness or Mission Analysis 100.0%

Stakeholder Needs and Requirements Definition 100.0%

System Requirements Definition 100.0%

Architecture Definition 33.3%

Design Definition 100.0%

System Analysis 0.0%

Implementation 100.0%

Integration 100.0%

Verification 100.0%

Transition 100.0%

Validation 100.0%

Operation 0.0%

Maintenance 0.0%

Disposal 0.0%

Project Planning 100.0%

Project Assessment and Control 66.7%

Decision Management 100.0%



Information Management 100.0%

Measurement 100.0%


Acquisition 80.0%

Supply 0.0%

Life-Cycle Model Management 100.0%

Infrastructure Management 100.0%

Portfolio Management 33.3%

Human Resource Management 66.7%

Quality Management 66.7%

Knowledge Management 100.0%

Total 73.6%

77.8%

40.0%

96.4%

66.7%

INCOSE to CMMI Gap Analysis

Technical Processes

Technical Management

Processes

Agreement Processes

Organizational Project-

Enabling Processes

• Major Gap Areas• Project Monitoring, Integrated Project Management, and Quantitative

Project Management• Causal Analysis and Resolution• Organizational Process Performance and Performance Management

• CMMI to INCOSE gaps mostly involve self-evaluation and process improvement.

1 5

CMMI TO INCOSE

Maturity Total

Requirements Management 80.0% Measurement and Analysis 100.0%

Supplier Agreement Management 100.0% Process and Product Quality Assurance 100.0%

Project Planning 50.0% Configuration Management 100.0%

Project Monitoring and Control 30.0%

Integrated Project Management 20.0% Requirements Development 90.0% Decision Analysis and Resolution 100.0% Organizational Process Focus 55.56%

Risk Management 85.7% Technical Solution 100.0% Organizational Process Definition 57.14%

Product Integration 77.8% Organizational Training 42.86%

Validation 100.0%

Verification 62.5%

Level 4 Quantitative Project Management 28.6% Organizational Process Performance 0.00% 16.7%

Level 5 Causal Analysis and Resolution 0.0% Organizational Performance Management 30.00% 20.0%

Total 62.3%

72.2%

CMMI to INCOSE Gap Analysis

50.8% 85.0% 39.5%83.3%

69.8%

Level 2

Project Management Process ManagementEngineering Support

Level 3

• Major Gap Areas

• Acquisition and Supply

• Software Engineering

• Joint Review

• Problem Resolution

• Change Management

• Reuse

• Most of the ASPICE to INCOSE engineering gaps were found due to INCOSE not covering engineering standards specific to software.

1 6

ASPICE TO INCOSEContract Agreement 100.0%

Supplier Monitoring 80.0%

Technical Requirements 40.0%

Legal and Administrative Requirements 40.0%

Project Requirements 20.0%

Request for Proposals 25.0%

Supplier Qualification 80.0%

Supplier Tendering 12.5%

Product Release 23.1%

Requirements Elicitation 83.3%

System Requirements Analysis 87.5%

System Architectural Design 50.0%

System Integration and Integration Test 88.9%

System Qualification Test 71.4%

Software Requirements Analysis 62.5%

Software Architectural Design 50.0%

Software Detailed Design and Unit Construction 0.0%

Software Unit Verification 0.0%

Software Integration and Integration Test 0.0%

Software Qualification Test 0.0%


Verification 100.0%

Joint Review 0.0%

Documentation 62.5%


Problem Resolution Management 0.0%

Change Request Management 0.0%

Project Management 70.0%


Measurement 54.5%

Reuse Process Group Reuse Program Management 12.5% 12.5%

Process Improvement

Process Group

Process Improvement

87.5% 87.5%

Total

System Engineering Process

Group

76.3%

ASPICE to INCOSE Gap Analysis

Acquisition Process Group

46.7%

Supply Process Group

19.0%

45.0%

Software Engineering

Process Group

19.1%

Supporting Process Group

41.5%

Management Process

Group

66.7%

• INCOSE to ASPICE has the lowest coverage results of this study.

• The large quantity of major gap areas is mostly due to INCOSE using broad, systems engineering standards while ASPICE uses specific, automotive engineering standards.

1 7

INCOSE TO ASPICEBusiness or Mission Analysis 40.0%

Stakeholder Needs and Requirements Definition 66.7%

System Requirements Definition 75.0%

Architecture Definition 33.3%

Design Definition 50.0%

System Analysis 0.0%

Implementation 0.0%

Integration 66.7%

Verification 100.0%

Transition 0.0%

Validation 0.0%

Operation 0.0%

Maintenance 0.0%

Disposal 0.0%

Project Planning 0.0%

Project Assessment and Control 100.0%

Decision Management 0.0%



Information Management 0.0%

Measurement 100.0%


Acquisition 0.0%

Supply 0.0%

Life-Cycle Model Management 0.0%

Infrastructure Management 0.0%

Portfolio Management 0.0%

Human Resource Management 0.0%

Quality Management 0.0%

Knowledge Management 0.0%

Total

INCOSE to ASPICE Gap Analysis

Technical Processes

Technical Management

Processes

Agreement Processes

Organizational Project-

Enabling Processes

31.8%

0.0%

0.0%

60.7%

33.3%

CMMI ASPICE INCOSE

CMMI 39.5% 62.3%

ASPICE 59.1% 45.0%

INCOSE 73.6% 31.8%

Results Summary - Row to Column

• CMMI has the highest average coverage of the other standards (66.4%).

• ASPICE has the lowest coverage of other standards due to its higher specificity.

• INCOSE has weaker coverage of other standards due its lack of software-specific processes.

1 8

RESULTS SUMMARY

CONCLUSIONS

1 9

2 0

GAP ANALYSIS SUMMARY – CMMI, ASPICE, INCOSE

• Supply Management

(outgoing)

• Legal and Administrative

Requirements

• Documentation

Management

• Decision Analysis and

Resolution

• Life-Cycle Model

Management

• Human Resource

Management

• Knowledge Management

• Causal Analysis and

Resolution

• Joint Review

• Process Performance

• Integrated Project

Management

• Quantitative Project

Management

• Request for Proposals

• Reuse Program Management

• Software-specific engineering

• System Analysis

• Operation

• Maintenance

• Disposal

CMMI

Unique Inclusions

Unique Gaps

ASPICE INCOSE

2 1

WHICH GAPS ARE IMPORTANT?

Weighing the importance of each major gap identified will lead us to determining which standard(s) should be mapped in Stages, adopted as best practice, required of suppliers, and/or used for internal appraisals.

Some identified gaps may have little significance in our business objective while others are critical.

Note that the weight of some gaps may differ for OEMs and suppliers. (E.g. supply management)

2 2

NEXT STEPS

1. Weigh the Gaps

Weigh each of the items identified in the Gap Analysis Summary. This should include all relevant engineering process leaders. These weights should be used to help determine action items regarding which standard(s) should be mapped in Stages, adopted as best practice, required of suppliers, and/or used for internal appraisals.

2 3

NEXT STEPS

2. Determine Best Standard/Model

It is crucial to pick a process model to adhere to. Without doing so, future processes can only reference back to previous processes, greatly diminishing potential improvement and increasing risk. Process requirements are necessary for process improvement to focus on the “what” rather than the “how.”

Choosing a process model to adhere to may involve using selections from multiple sets of standards in combination. The best option/combination should be selected and reviewed regularly.

QUESTIONS

2 4

ADDITIONAL INFORMATION

2 5

2 6

REFERENCES[1] Automotive SPICE PFM PAM v3.0

[2] INCOSE Systems Engineering Handbook 4e 2015 07

[3] CMMI for Development, Version 1.3

[4] Monzón, A. (2010, May). Bi-directional Mapping between CMMI and INCOSE SE Handbook. In Embedded Real-Time Software International Conference, Toulouse.

[5] Sabar, S. (2011). Software Process Improvement and Lifecycle Models in Automotive Industry.

[6] Baldassarre, M. T., Piattini, M., Pino, F. J., & Visaggio, G. (2009, May). Comparing ISO/IEC 12207 and CMMI-Dev: towards a mapping of ISO/IEC 15504-7. In Software Quality, 2009. WOSQ'09. ICSE Workshop on (pp. 59-64). IEEE.

[7] Sassenburg, H., & Kitson, D. (2006). A comparative analysis of CMMI and automotive SPICE. European SEPG, Amsterdam/Netherlands (June 2006).

Documents

BI-DIRECTIONAL GAP ANALYSES OF CMMI, AUTOMOTIVE SPICE · PDF fileBI-DIRECTIONAL GAP ANALYSES OF CMMI, AUTOMOTIVE SPICE, AND INCOSE Travis Foust, Matthew Ramirez, Regina Dorow October