SE Effectiveness Working Group 23-Nov-2010 Inaugural Meeting Presenter:Joseph P. Elm Software Engineering Institute

SE Effectiveness Working Group23-Nov-2010 Inaugural Meeting

Presenter: Joseph P. ElmSoftware Engineering Institute

SE Effectiveness Working Group23-Nov-2010

Agenda

SE Effectiveness Working Group

Initial Project: Support of NDIA / IEEE SE Effectiveness Study• Background and History

• Current Efforts

• SEEWG Role

3SE Effectiveness Working Group23-Nov-2010

Purpose

promote effective systems engineering by collecting and analyzing quantitative data on the impact of specific SE processes and practices on project performance.

This data will contribute to the development of a stronger business case for systems engineering.


Objectives

Identify principles and practices PROVEN to provide benefit to projects and programs

Develop guidance, training, and tools to• assist SE practitioners in applying these principles and practices

• assist system acquirers in ensuring that these principles and practices are applied to their projects

Establish means of monitoring / tracking the application of SE principles and practices and their impacts with the intent of continuously improving project performance through better SE


Scope

Address the collection, analysis, and distribution of quantitative evidence of the value of SE throughout the system life cycle

Address this question from the perspectives of both suppliers and acquirers

Seek data broadly across all market sectors• Defense

• Aerospace

• Transportation

• Telecommunications

• etc.


Membership, Roles, and Responsibilities

Position Responsibilities Incumbent Contact InfoSEEWG Chair: • Call and preside over WG meetings

• Manage and administer WG• Report to AD-Processes• Duties common to all WG members

Joseph P. Elm [email protected]

SEEWGCo-chair

• Preside over WB meetings in the absence of the Chair

• Assist the Chair in managing and administering the WG

• Duties common to all WG members

Eric Honour [email protected]

Sponsor • Resource advocacy• Status reporting to the INCOSE Technical

Director, BOD, and external stakeholders.)

Bob Swarz(AD-Processes)

[email protected]

SEEWG Members

• Planning• Research• Analysis• Technical writing• Presenting• Interviewing

Bruce ElliottJoseph ElmSummer FowlerEric HonourSteve MazeikaNancy RoseberryFrank SciulliRichard SidleyDale SmithTim Spencer

[email protected]@sei.cmu.edu [email protected]@[email protected]@[email protected]@[email protected]@iq-inc.com

SE Effectiveness Working Group23-Nov-2010

Agenda


Initial Project: Support of NDIA / IEEE SE Effectiveness Study• Background and History

• Current Efforts

• SEEWG Role


Agenda


Initial Project: Support of NDIA / IEEE SE Effectiveness Study

• Background and History

• Current Efforts

• SEEWG Role

Development Phasing

Life Cycle Integration

Systems Engineering

Process

Baselines Life Cycle Planning

Integrated Teaming

Systems Engineering Management


Background and History 1

The value of SE is appreciated by many, but disputed by some

Quantitative evidence of the value of SE is sparse• Greuhl, Walter: “Lessons Learned, Cost/Schedule Assessment Guide”.

NASA Comptrollers Office, 1992

• Honour, Eric; “Understanding the Value of Systems Engineering”. 2004

• NDIA SE Effectiveness Committee; “A Survey of Systems Engineering Effectiveness - Initial Results”. 2008



In 2006, NDIA embarked on a project to collect quantitative evidence of SE Value

• NDIA formed the SE Effectiveness Committee (SEEC)

• The SEEC conducted the SE Effectiveness Study– Developed a survey collecting information from defense contractors

• Assessed SE capabilities by cataloging artifacts of SE processes• Assessed project performance based on satisfaction of budget, schedule and

requirements• Assessed other factors influencing project performance

– Received responses from 64 projects– Analyzed the data and identified the strength of relationships between SE

activities and project performance– Results published results in 2007 and 2008

(http://www.sei.cmu.edu/reports/08sr034.pdf)

• Showed valuable relationships between many (but not all)SE activities and project performance


Artifact-based assessment of SE Practices

CMMI-SE/SW/IPPD v1.1

• 25 Process Areas• 179 Goals• 614 Practices• 476 Work Products


SystemsEngineering-related Filter


Size Constraint Filter

Considered significant to Systems Engineering

Survey content is based on a recognized standard (CMMI)


Assessment of Project Performance

Assess TOTAL Project Performance• Project Cost, Project Schedule, Project Scope

• Focus on commonly used measurements– EVMS, baseline management, requirements satisfaction, budget re-

baselining and growth, milestone and delivery satisfaction

Assessment of Other Factors• Project Challenge – some projects are more complex than others

• Acquirer Capability – some acquirers are more capable than others

• Project Environment – projects executed in and deployed to different environments have different needs


NDIA SE Effectiveness Study

The Bottom Line

For the projects that did the most SE,

56% delivered the best project performance

For the projects that did the least

SE, only 15% delivered the best project performance.

39%

46%

15%

29%

59%

12%

31%

13%

56%

BestPerformance( x > 3.0 )

ModeratePerformance( 2.5 x 3.0 )

Lower Performance( x < 2.5 )

Lower Capability

( x 2.5 )N = 13

Moderate Capability

( 2.5 < x < 3.0 )N = 17

HigherCapability

(x 3.0 )N = 16

Gamma = 0.32p = 0.04

PROJECT PERFORMANCE vs. TOTAL SE CAPABILITY

1.00

0.75

0.50

0.25

0.00


Relationship of SE Processes to Program Performance

-13%

13%

13%

21%

25%

28%

28%

32%

33%

34%

36%

37%

40%

49%

-20% -10% 0% 10% 20% 30% 40% 50% 60%

Project Monitor/Control

Project Planning

Config Mgmt

Product Integration

Verification

Risk Mgmt

Validation

Overall SE Capability

Reqts Devel & Mgmt

IPT Capability

Technical Solution

Trade Studies

Architecture

Reqts + Tech Solution

SE

Cap

ab

ilit

y

Gamma (strength of relationship)Composite Measures

Summary of Relationships

Strong RelationshipModerately Strong

to Strong Relationship

Moderately Strong

RelationshipWeak Relationship


Agenda




• Current Efforts

• SEEWG Role

Development Phasing


Systems Engineering

Process


Integrated Teaming




Presented study results and recommendations to OSD in 2007

Held discussions with IEEE in 2009 regarding extension of the study to a wider audience

Briefed new OSD leadership in May-2010• Positive reception of findings

• Interested in using findings to improve value of SE processes within DoD

• Supportive of follow-on study

So, Here we are today …


The Mission

Assist the DoD and the SE community in achieving a quantifiable and persistent improvement in project performance through appropriate application of systems engineering principles and practices

• Publishing research results is not enough. We need to:

– Identify principles and practices PROVEN to provide benefit

• This is an extension and a confirmation of the prior NDIA study

– Assist DoD in developing the guidance to implement the principles and practices identified in study findings

– Assist DoD in establishing a means of monitoring / tracking the results of efforts to improve SE

– Assist DoD in institutionalizing these efforts so that they become“the way DoD does business”.


The Team

National Defense Industrial Association

• Systems Engineering Division

Institute of Electrical and Electronics Engineers

• Aerospace and Electronic Systems Society

Carnegie Mellon University• Software Engineering Institute

(FFRDC)

The Systems Engineering Effectiveness Committee


The Plan 1

SEES proveneffective SE practices

SE Effectiveness Study

Business Case for SE

SE framework

Adoption by industry

Recommendations for DoD Acquisition

Policy Guidance Training

System AcquisitionSystem Development

Data collection and monitoring

Phase I

Phase II

Phase III

Aids


The Plan 2

Phase I: Identify SE activities that show net benefit (i.e., value exceeding cost) to program performance.

• Expand previous SE Effectiveness study to survey a larger population• Develop an SE Framework and a stronger Business Case for SE (BCSE)

Phase II: Develop recommendations to OSD for policy guidance, and training to implement the findings of Phase I.

• Develop recommendations for OSD review• Develop tools to implement policy and guidance

– Suggested RFP language – Suggested Contract language– Program office training – SE assessment methods and tools

(in collaboration with DAU, AFIT, et. al.) – SE artifact CDRLs

Phase III: Establish a mechanism for continuous improvement of the BCSE Framework.

• On-going data collection from DoD programs• Integration with program review processes


Survey Tenets

All data will be submitted anonymously• No data collected will identify the respondent, project, or organization\

All data will be handled confidentially• Data will be submitted directly to a secure web site managed by the SEI

– The SEI is a federally funded research and development center. It does not compete with any responding organizations, and frequently operates as a trusted broker in matters of confidential and proprietary information.

• Only authorized SEI staff will have access to the submitted data

Only aggregated data will be released to the participants and the public

• No released data will be traceable to a project, person, or organization.


Schedule

Phase IJun-2010 – Mar-2012

•Survey development, execution, and analysis

•BCSE Framework development

Phase IIMar-2012 – Jun-2013

•Recommendations to DoD for policy, guidance, and training

Phase IIIJun-2013 - Oct-2013

•Mechanisms for continued data collection and analysis


Agenda




• Current Efforts

• SEEWG Role

Development Phasing


Systems Engineering

Process


Integrated Teaming



Task I: Manage participation of INCOSE Members 1

The survey target audience is Project Managers, Chief Engineers, Lead System Engineers, etc. of projects delivering products (not services)

• Not limited to defense industries – all industries are welcome

• Not limited to US companies – all are welcome

Reaching potential respondents• Grass roots approach

– Reach our to organizational membership seeking participation

• Top down approach– Indentify and contact SE leadership at major companies represented in

participating organizations


Task I: Manage participation of INCOSE Members 2

Grass roots approach• Obtain INCOSE membership roster

• Broadcast an invitation to participate to members of INCOSE

Top down approach• Identify SE leadership at major companies

– Work from INCOSE membership roster– Identify SE leaders within their companies– Contact them directly and solicit their support to:

• Identify potential respondents within their company• Promote participation


Task II: Develop SE Framework and BSCW




SE framework






Phase I

Phase II

Phase III

Aids

INCOSE input


Task III: Support Industry Adoption




SE framework






Phase I

Phase II

Phase III

Aids

INCOSE input•Tools•Training•Consulting


Let’s Make this New Study a Success !

For more information, contact:

Joseph P. ElmSoftware Engineering [email protected]

Documents

SE Effectiveness Working Group 23-Nov-2010 Inaugural Meeting Presenter:Joseph P. Elm Software Engineering Institute