Cost Estimation of Human Systems

Integration

1stLt. Kevin Liu, USMCSM in Technology & Policy, June 2010

Research Advisors: R. Valerdi and D.H. Rhodes

The views expressed in this presentation are those of the author and do not reflect the official policy or position of the United States Air Force, Marine Corps, Department of Defense, or the U.S. Government.

This research was generously supported by the Air Force Human Systems Integration Office.

Motivation

seari.mit.edu © 2010 Massachusetts Institute of Technology 2

Research Questions/Hypothesis

seari.mit.edu © 2010 Massachusetts Institute of Technology 3

R1: How can the “right” amount of effort to invest in HSI be determined?

R2: How much does HSI effort cost?

R3: What is the relationship between HSI and systems engineering?

Hypothesis: HSI effort can be estimated as a function of Systems Engineering Effort.

Research Outline

seari.mit.edu © 2010 Massachusetts Institute of Technology 4

1. Review of Relevant Topics- Defense Acquisition- Systems Engineering

Hypothesis 1: HSI effort can be estimated as a function of SE.

2. Problem Formulation- HSI and Cost Estimation- Case Study: F119 Engine- HSI and COSYSMO

Hypothesis 2: HSI effort can be estimated by counting HSI-related requirements

3. Implementation and ValidationWorkshop 1: How can COSYSMO

requirements counting rules better account for HSI?

Workshop 2: How can an IPT reach a consensus about effort of HSI-related requirements?

4. Conclusions, Contributions, Recommendations

Review of Relevant Topics

seari.mit.edu © 2010 Massachusetts Institute of Technology 5

Image adapted from DoD 5000.02: Operation of the Defense Acquisition System

Stakeholder Requirements Definition

Requirements Analysis, Architecture Design

Implementation, Integration, Verification

Transition, Validation

Systems Engineering Technical Processes

Is a new program necessary?

Defense Acquisition Management

Problem Formulation-HSI

seari.mit.edu © 2010 Massachusetts Institute of Technology 6

HSI is tightly coupled to Systems Engineering

HSI came from Human Factors

““a comprehensive management and technical approach for addressing the human element in weapon system development and

acquisition,” (Air Force Instruction 10-601)

“HSI: interdisciplinary technical and management processes for

integrating human considerations within and across all system

elements; an essential enabler to systems engineering practice.”

INCOSE

HSI cost studies look at specifics, not prediction.

Bias & Mayhew - UsabilityDray & Karat - User Interface

Hypothesis: HSI effort can be estimated as a function of SE effort and used for predictive

purposes.

Bosert - HFEBooher (1997) –

Several Army Systems

Problem Formulation-Case Study: F119 Engine

seari.mit.edu © 2010 Massachusetts Institute of Technology 7

Research Questions1.How did Pratt & Whitney determine how much HSI effort would be needed?2.How much did HSI effort eventually cost?3.How did HSI fit into the larger SE picture?

Proposition: HSI effort can be isolated from total SE effort spent.

Takeaways1.HSI cost cannot be isolated from SE cost. 2.Requirements drive effort/focus spent on HSI3.Teams are important to SE/HSI process

Methodology-Case Study Research (Yin 2009)-Unit of Analysis: Early development at P&W- Method: Interviews and Literature- Interpretation of results: proposition vs. rival proposition Conclusion: Rival proposition supported.

Image: Pratt & Whitney

Problem Formulation-HSI and COSYSMO

seari.mit.edu © 2010 Massachusetts Institute of Technology 8

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

rementscesriosthms

- Application factors-8 factors

- Team factors-6 factors

Hypothesis 2: HSI effort can be estimated by counting requirements related to HSI.

RequirementsInterfacesAlgorithmsOperational scenarios

Determine System of Interest

Tested?Verified?Designed?

Yes

No

1

2

Sketch System of Interest’s 

Relationship to Rest of System

3

4Assess Complexity

Shall’sWill’sMust’s

5

Workshop 1: Application of Requirements Counting Rules

seari.mit.edu © 2010 Massachusetts Institute of Technology 9

Research Question: How can COSYSMO counting rules be modified to improve counting of HSI-related requirements?

Workshop Hypothesis #1:Using the cost estimation counting rules will produce requirements counts with high reliability across respondents.

Workshop 1: Application of Requirements Counting Rules

seari.mit.edu © 2010 Massachusetts Institute of Technology 10

Research Question: How can COSYSMO counting rules be modified to improve counting of HSI-related requirements?

Workshop Hypothesis #2: The cost estimation counting rules will help users quantify the number of HSI requirements to be input into COSYSMO.

Needs Identified

1.Better tools for consensus-building.

2.Further exploration of the impact of HSI on effort.

Workshop 2: Estimation of Relative HSI Effort

seari.mit.edu © 2010 Massachusetts Institute of Technology 11

Research Question: How can an IPT reach a consensus about the effort associated with an HSI-related requirement during early cost estimation?

1x       2x       4x       8x   

“What is the equivalent effort of the HSI-related requirement in question, as compared

to a nominal requirement?”

1. Complete Round 1 individuallya) Review and discuss

2. Round 2 of estimationa) Review and discuss

3. Round 3 of estimation

Building Consensus

“usually I think of training as something that you worry about after the system’s already

developed or kind of set, so you’re really not adding effort”

“[training] touches so many other folks and it has such a large human element in it, you really got to give yourself a lot longer schedule, and not unusually a lot more money”

Workshop 2: Estimation of Relative HSI Effort

seari.mit.edu © 2010 Massachusetts Institute of Technology 12

Consensus improved in 9 out of 10 cases

Conclusions and Recommendations

seari.mit.edu © 2010 Massachusetts Institute of Technology 13

• Modifications recommended for COSYSMO counting rules

• Disconnect between industry perception of HSI and HSI in DoD requirements

• Nonfunctional requirements must be considered early to avoid requirements creep

• Methodological contribution: quantification of Wideband Delphi results.

• Maturing understanding of how HSI effort is planned, tracked, and executed as part of SE in Air Force and industry